diff --git a/optional-skills/security/unbroker/README.md b/optional-skills/security/unbroker/README.md new file mode 100644 index 00000000000..e215e556ea2 --- /dev/null +++ b/optional-skills/security/unbroker/README.md @@ -0,0 +1,163 @@ +# unbroker + +An agent-native skill that finds a consenting person's exposed personal information across data +brokers and people-search sites and removes it. It runs automatically wherever it can, and hands off +to a human only where a site demands a CAPTCHA it cannot clear, a government ID, a phone call, or a +fax. + +

+ unbroker: autonomous removal pipeline (exposure field, the loop, ledger, re-scan horizon) +

+ +## About + +Hundreds of data brokers publish people's names, current and prior addresses, phone numbers, emails, +relatives, and property records. That exposure fuels doxxing, stalking, harassment, and identity +theft. Removing the data is the documented antidote, but it is high-volume work, full of dark +patterns, and perishable (brokers re-list you). Commercial services such as EasyOptOuts, Incogni, and +DeleteMe solve this for a fee, but they are closed, and you hand a company you know nothing about the +exact data you are trying to erase. + +unbroker brings those core capabilities together (EasyOptOuts' automation breadth, Incogni's +legal-request engine, DeleteMe's verification and reporting) as a transparent, auditable, +self-hosted skill that the user's own agent runs. It is **multi-tenant** (manage yourself, family, or +clients, each isolated), **consent-gated**, and built for **maximum automation with a human +fallback**. Scope is **US-first**, with EU/UK (GDPR) and global coverage on the roadmap. + +The design is **Hermes-native**: a small deterministic Python CLI (`scripts/pdd.py`) owns the state +(config, dossiers, broker DB, tier planning, ledger, drafts, reports), while the agent does the +scanning and submitting with native tools (`web_extract`, `browser_*`, email, `cronjob`, +`delegate_task`). [`SKILL.md`](SKILL.md) is the authoritative reference. + +## Install + +```bash +hermes skills install official/security/unbroker +``` + +Then start a new Hermes session and drive it (below). The skill works zero-config; a few optional +env vars unlock more automation (all documented in `SKILL.md` under Prerequisites): + +- `BROWSERBASE_API_KEY`: the recommended default browser. A real residential-IP cloud browser that + clears soft/managed CAPTCHAs (Turnstile, hCaptcha/reCAPTCHA checkbox) as normal operation, so + those brokers stay automated. It is not a solver and does not defeat hard challenges. +- Hands-off email, two ways: **browser mode** (`pdd.py setup --email-mode browser`, no stored + password; the agent sends opt-outs and opens verification links through your logged-in webmail), + or **`EMAIL_ADDRESS` + `EMAIL_PASSWORD`** for SMTP send + IMAP verification. Without either, it + falls back to writing drafts for you to send. +- the `age` binary: at-rest encryption of dossiers and ledgers. +- the `google-workspace` skill: a shared Google Sheets status tracker. + +## Usage + +Drive it from a Hermes session: + +> "Use the unbroker skill to remove my data from data brokers. Here is my consent. Run it hands-off +> and show me the human-task digest at the end." + +The agent configures itself (`setup --auto` selects programmatic email if `EMAIL_*` creds exist, the +cloud browser if available, and encryption if `age` is installed), records your consent, then drains +the autonomous queue: scan, opt out (parents first), send and verify emails, schedule re-checks. You +hear from it twice: at intake, and with one digest of anything only a human can do. + +The underlying CLI (run via `terminal`, as `python3 scripts/pdd.py `): + +| Command | Purpose | +|---|---| +| `pdd.py setup --auto` / `doctor` | Self-configure (most-autonomous valid config) and readiness check | +| `pdd.py intake` | Create a consenting subject (captures aliases, multiple emails/phones, prior addresses) | +| `pdd.py next` | The loop driver: ordered agent actions right now, the human digest, and the next wake time | +| `pdd.py brokers` / `refresh-brokers` | List people-search brokers, or pull the latest BADBOOL list plus the CA registry | +| `pdd.py registry` | State data-broker registry coverage (CA ~545 ingested; VT/OR/TX portals); `--search` to find one | +| `pdd.py drop` | The CA DROP one-shot: delete from all registered brokers in a single request | +| `pdd.py plan` | Per-broker tier, method, search vectors, and the exact fields to disclose | +| `pdd.py fanout` | Batch brokers into parallel `delegate_task` subagents | +| `pdd.py record` | Update the ledger (validated state machine); auto-stamps recheck dates | +| `pdd.py send-email` | Render and send an opt-out / CCPA / GDPR request (recipient locked to the broker's own address) | +| `pdd.py poll-verification` / `verify-link` | Resolve email-verification links (IMAP poll, or browser-mode from pasted text) | +| `pdd.py render-email` | Draft-only fallback (least-disclosure) | +| `pdd.py due` / `tasks` | Recheck queue for cron, and the consolidated human-task digest | +| `pdd.py status` / `report` | Per-subject status, plus optional Google Sheets rows | + +## How it works + +- **Autonomous by default.** After one human conversation (intake plus consent), the agent drains a + deterministic action queue (`pdd.py next`): scan, opt out parents-first, send and verify emails, + re-check on schedule, all without pausing to ask. Human-only work (gov-ID sites, phone callbacks, + hard-CAPTCHA sites) accumulates silently into a single end-of-run digest (`pdd.py tasks`). +- **Tiered automation (T0 to T3).** Every broker opt-out is classified from fully automated, to + automated with verification, to human-verified, to human-only. The agent always takes the highest + viable tier and escalates to a human task only when genuinely blocked. +- **Cluster parents first.** Many brokers are resold shells of a few parents, so one removal can + clear a dozen child sites. The planner orders parents ahead of standalone listings and ships + field-verified, per-parent playbooks that prefer the **right-to-delete** lane over mere suppression + (for example PeopleConnect's "delete my user data", or Whitepages' privacy email, which sidesteps + the phone-callback tool entirely). +- **Multi-identifier fan-out.** A person is indexed under every name/alias, phone, email, and + address. The planner expands all of them (filtered by what each broker supports) so listings under + a maiden name or an old address are found, not just "primary name plus current city". +- **Verify before you disclose.** Nothing is submitted until a real listing is confirmed, the match + is confirmed as the subject and not a namesake or relative, and only the exact fields a broker + requires are sent (least-disclosure; SSN and ID numbers are never volunteered). +- **Jurisdiction-aware.** Requests file under the framework that applies where the subject lives: + CCPA/CPRA in California, GDPR in the EU/UK, a general right-to-delete request otherwise. It never + cites a right the subject cannot invoke. +- **Coverage that matches or exceeds commercial services.** Two lanes: (1) people-search sites with + per-site opt-out mechanics (19 curated records, including FamilyTreeNow, Radaris, and Nuwber, plus + a live pull from [BADBOOL](https://github.com/yaelwrites/Big-Ass-Data-Broker-Opt-Out-List)), and + (2) the **state data-broker registries** as a distinct legal-coverage lane: the **California Data + Broker Registry** (~545 registered brokers, the authoritative universe the commercial services draw + from) is ingested, with Vermont, Oregon, and Texas surfaced as search portals. +- **The DROP one-shot.** California's Delete Request and Opt-out Platform is live: for a CA resident, + a single verified request deletes their data from **every registered broker at once**, and + `pdd.py next` surfaces it as the highest-leverage action. +- **Ledger, audit, and re-scan.** Every case is a validated state machine, every PII disclosure is + logged (field names only), and confirmed removals are re-scanned on a schedule so a re-listing is + caught and re-filed. Ledger writes are file-locked for safe concurrent runs. +- **Privacy by default.** Opaque subject ids (no name in ids, paths, or logs), optional `age` at-rest + encryption of dossiers, and everything local. The skill ships placeholder data only. + +## Tests + +85 hermetic tests (no network, browser, or email; SMTP and IMAP are exercised through injected +fakes): + +```bash +scripts/run_tests.sh tests/skills/test_unbroker_skill.py # CI-parity harness +python3 tests/skills/test_unbroker_skill.py # dependency-free fallback runner +``` + +## Safety and ethics + +- **Consent-gated.** The engine refuses to scan or act on a subject without a recorded + authorization. It is a removal tool, not a people-search aggregator. +- **Sanctioned browser only, no solver farms.** The default cloud browser clears soft/managed + CAPTCHAs the way any real browser would, but there is no CAPTCHA-solving service and no fingerprint + spoofing. Hard interactive challenges escalate to a human task. +- **Least-disclosure and honest reporting.** The skill submits only what a broker requires. "Hidden + from free search" is never reported as "deleted", and residual exposure (public records, paid-tier + retention) is disclosed. +- **PII handling.** Dossiers live under the Hermes home directory (`0600`, optionally + `age`-encrypted), with opaque ids. + +## Status + +**v1.0.** The deterministic engine, the autonomous loop, the verified cluster-parent deletion lanes, +and full broker-registry coverage (the CA Data Broker Registry plus the DROP one-shot) are built and +covered by 85 hermetic tests. The skill ships placeholder data only. Live agent-driven submission +against broker sites is the active field-testing frontier. + +## Credits and license + +- Broker dataset adapted from the **Big-Ass Data Broker Opt-Out List (BADBOOL)** by **Yael Grauer**, + licensed [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (attribution + required, non-commercial). See [yaelwrites.com](https://yaelwrites.com/). +- Code: MIT. + +## Disclaimer + +This is not legal advice. Only operate on people who have authorized removal of their own data. +Removing data from brokers reduces exposure but does not guarantee total erasure. Public records +(voter, property, court) and offline vectors are out of scope. diff --git a/optional-skills/security/unbroker/SKILL.md b/optional-skills/security/unbroker/SKILL.md new file mode 100644 index 00000000000..3888e04d02f --- /dev/null +++ b/optional-skills/security/unbroker/SKILL.md @@ -0,0 +1,285 @@ +--- +name: unbroker +description: Autonomously remove your info from data-broker sites. +version: 1.0.0 +author: SHL0MS (github.com/SHL0MS) +license: MIT +platforms: [linux, macos, windows] +prerequisites: + commands: [python3] +metadata: + hermes: + tags: [privacy, data-broker, opt-out, ccpa, gdpr, security, doxxing] + category: security + related_skills: [google-workspace, agentmail, himalaya, scrapling, osint-investigation] + homepage: https://github.com/NousResearch/hermes-agent +--- + +# unbroker + +Find where a person's personal information (name, addresses, phone, email, relatives) is exposed on +data brokers and people-search sites, then remove it - automatically where possible, with guided +human steps only where a site demands a CAPTCHA, government ID, phone call, or fax. Manages multiple +people independently. It does **not** defeat anti-bot systems, does **not** act on anyone without +recorded consent, and does **not** remove public records (voter/property/court) or accounts the +person controls. + +The Python CLI (`scripts/pdd.py`) owns the deterministic state - config, dossiers + consent, the +broker database, tier planning, the ledger, drafts, reports, **email sending (SMTP), verification-link +polling (IMAP), and the autonomous action queue (`next`)**. You (the agent) do the scanning and +form-driving with native tools: `web_extract` and `browser_navigate` for searching and web forms, and +`cronjob` for recurring re-scans. + +## Autonomy contract + +This skill is designed to run **hands-off**. After intake (+ recorded consent) there are exactly TWO +legitimate human touchpoints: (1) the intake conversation itself, and (2) ONE consolidated human-task +digest at the end of the run (`$PDD tasks`). Between those: + +- **Never ask the operator to choose configuration.** `$PDD setup --auto` detects capabilities and + picks the most autonomous valid config itself. +- **Never pause before individual submissions** when `autonomy=full` (the default): the consent + recorded at intake is standing authorization for T0-T2 opt-outs. (`autonomy=assisted` restores + per-submission confirmation for cautious operators - honor `confirm_first` flags in `next` output.) +- **Never interrupt the run for human-only work.** Record it (`record ... human_task_queued + --reason "..."`) and keep going; it all surfaces once in the final digest. +- **Drive the whole run as a loop over `$PDD next `** - it returns the exact ordered actions + to take right now (scan, poll verification, re-check, opt out parents-first, requeue blocked), plus + the human digest. Execute every action, record outcomes, re-run `next`, repeat until + `done_for_now`. Then present the digest, report, and schedule the cron. + +The hard limits that autonomy never overrides: no acting without recorded consent, no disclosure +beyond `disclosure_fields`, no CAPTCHA/anti-bot bypass, and `confirmed_removed` only after a +verifying re-scan. + +## When to Use + +- "Remove my (or my family member's) data from data brokers / people-search sites." +- "Opt me out", "delete me from Spokeo/Whitepages/etc.", "clean up after a doxxing." +- "Set up recurring privacy monitoring" (brokers re-list people). +- Checking which brokers still expose someone and why. + +## Prerequisites + +- `python3` (stdlib only; no extra packages needed for the core engine). +- **Optional upgrades** (the skill works zero-config without these; `setup --auto` turns on every + one it detects - each one converts a class of human tasks into agent actions): + - **Cloud browser (recommended default): `BROWSERBASE_API_KEY`.** `setup --auto` selects it + whenever the key is present, and it is the intended baseline: a real residential-IP cloud + browser **clears soft/managed CAPTCHAs (Cloudflare Turnstile, hCaptcha/reCAPTCHA checkbox) as + normal operation**, so those brokers stay automated (T1) instead of becoming human tasks. This + is not CAPTCHA "solving" - no solver service, no fingerprint spoofing; only interactive/behavioral + ("hard") challenges the browser genuinely cannot pass fall back to a human task. Without the key, + the plain agent browser is used and soft-CAPTCHA brokers drop to T2 (human). + - Email automation, two credential-free-or-not options: + - **Browser mode (no password): `setup --email-mode browser`.** The agent sends opt-out/CCPA + emails and opens verification links through the operator's **logged-in webmail** using + `browser_*` tools. Nothing is stored. Needs the inbox signed in in the browser Hermes uses + (a cloud browser like Browserbase won't hold the session; use a local/operator browser). + Falls back to drafts for an email if the inbox isn't reachable. + - **SMTP/IMAP (stored creds): `EMAIL_ADDRESS` + `EMAIL_PASSWORD`** (+ `EMAIL_SMTP_HOST` / + `EMAIL_IMAP_HOST` for non-mainstream providers; gmail/outlook/yahoo/icloud/fastmail inferred). + The CLI sends via `send-email` and reads verify links via `poll-verification`. The `agentmail` + skill (per-broker aliases) also counts. + - Google Sheets tracker: the `google-workspace` skill. + - The `scrapling` skill for stealth/Cloudflare-protected pages. + +## How to Run + +Run everything through the `terminal` tool. From this skill's directory: + +```bash +PDD="python3 scripts/pdd.py" +``` + +The engine stores data under `$PDD_DATA_DIR` (default `$HERMES_HOME/unbroker`), written +`0600`. Run via `terminal`, **not** `execute_code` (that sandbox scrubs env and redacts output, which +breaks reading the dossier). + +## Quick Reference + +| Command | Purpose | +|---|---| +| `$PDD setup --auto` | **Autonomous setup**: detect capabilities, pick the most autonomous valid config (no questions) | +| `$PDD doctor` | Readiness check: config, broker count, and which upgrades are on/available | +| `$PDD intake --full-name "..." [--alias ...] [--email ... --phone ...] [--city --state] [--prior-location "City,ST"] --consent` | Create a consenting subject; captures aliases + multiple emails/phones + prior locations; prints `subject_id` | +| `$PDD next ` | **The autonomous loop driver**: ordered agent actions right now + human digest + `next_wake_at` | +| `$PDD brokers [--priority crucial]` | List the people-search broker database (curated + live) | +| `$PDD refresh-brokers` | Pull the latest BADBOOL people-search list **and the CA Data Broker Registry** (`next` requeues this automatically when the cache is stale) | +| `$PDD registry [--search NAME]` | State registry coverage (CA ~545 ingested; VT/OR/TX portals surfaced); the DROP/email lane, not scanned | +| `$PDD drop [--filed]` | **The one-shot legal lever**: one CA DROP request deletes from ALL registered brokers; `--filed` records it | +| `$PDD plan [--priority crucial]` | Per-broker tier + method + `search_vectors` + the exact fields to disclose | +| `$PDD plan --batch` | **Reduce view**: overlays ledger state, groups brokers by next action (unscanned/found/indirect/blocked/in_progress/done), collapses ownership clusters, **orders `found` cluster-parents-first + emits a tailored `parent_playbook`**, prints `next_actions` | +| `$PDD fanout [--priority crucial] [--size 8]` | Batch brokers into parallel `delegate_task` subagents (auto for large runs) | +| `$PDD record [--found true] [--evidence JSON] [--disclosed F --channel C] [--reason "..."]` | Update the ledger (validated state machine); **auto-stamps `next_recheck_at`** | +| `$PDD send-email --listing [--kind ccpa_indirect ...]` | Render + record the request (recipient locked to the broker's own address). **browser** mode returns a `compose` payload to send via webmail (no password); **programmatic** mode SMTP-sends | +| `$PDD verify-link --text ''` | **browser mode**: extract a broker's verification link from webmail text you read (anti-phishing scored) | +| `$PDD poll-verification [--broker ]` | **programmatic mode**: poll IMAP for verification links (anti-phishing scored); auto-advances `submitted → verification_pending` | +| `$PDD render-email --listing ` | Draft only (fallback when no email mode is configured) | +| `$PDD due ` | Cases whose recheck window arrived (the cron re-scan queue) | +| `$PDD tasks ` | ONE consolidated human-task digest (present at END of run) | +| `$PDD status ` | Markdown status report | +| `$PDD report --sheets` | Rows for the Google Sheets tracker | + +## Batch operation (two-phase: crawl-all, then delete) + +For anything past a couple of brokers, run this as **map → reduce → act**, not broker-by-broker: + +- **Phase 1 - DISCOVER (read-only, parallel, idempotent).** Crawl *every* broker first and record a + verdict for each (`found` / `not_found` / `indirect_exposure` / `blocked`). Scanning has no side + effects, so it is safe to parallelize and retry. Getting the full exposure map *before* acting is + what unlocks cluster dedup and prioritization below. **Default: the parent drives `web_extract` + probes directly** - most people-search sites render name/phone/address results as static HTML that + `web_extract` reads in seconds. Escalate to `browser_*` only for the few JS-only sites, and to + `delegate_task` subagents only for genuinely *reasoning*-heavy work (large-scale namesake/relative + disambiguation). **Do NOT hand a browser-toolset subagent a big list of brokers to crawl** - in the + field this timed out repeatedly (600s, ~5-6 brokers each, no summary) because browser navigation is + heavy; the ledger writes that survived came at 10x the cost of parent `web_extract`. A `blocked` + (DataDome/Cloudflare/`antibot`) site is *not* a subagent job either: record `blocked` and requeue it + for a stealth/cloud browser (Browserbase) pass. Subagent reports are self-reports - the parent + re-fetches key URLs to confirm a `found` before trusting it (this cuts both ways: it caught a real + listing the parent had wrongly assumed was a false positive). +- **REDUCE - `$PDD plan --batch`.** Collapses the crawl into a phase-oriented plan: groups by + next action, **collapses ownership clusters** (a parent removal that clears children is ONE action, + not N - e.g. one Intelius/PeopleConnect suppression covers Truthfinder/Instant Checkmate/US Search/…), + and prints `next_actions`. `phase` is `discover` while anything is unscanned, else `delete`. +- **Phase 2 - DELETE (sequential, irreversible).** Work the reduced groups **parents first**: + `plan --batch` orders the `found` group cluster-parents-first (most children first) and emits a + `parent_playbook` with tailored, ordered steps per parent - follow that order and those steps + (full recipes in `references/methods.md` → "Ownership clusters - DO PARENTS FIRST"). Do the + cluster parents (skipping the covered children), **re-scan each parent's children after it confirms** + (they usually drop out), then the standalone listings; send the `indirect_exposure` cases as + CCPA/GDPR delete-my-PII emails (`send-email --kind ccpa_indirect`), and defer `blocked` to the + stealth-browser pass. Opt-outs hit CAPTCHAs, email-verification loops, and session binding - work + them **one at a time, carefully** (this is the opposite of fan-out), but do NOT stop to ask + permission per submission in `autonomy=full`; in `assisted`, confirm each one. **Prefer deletion + over suppression** where a broker offers both (e.g. PeopleConnect's "Right to Delete / DELETE MY + USER DATA" actually removes data; the suppression flow only hides it). + +Subagent reports are self-reports: the parent re-verifies key claims (listing URLs, match basis) before +recording `found` and before any deletion. + +## Procedure (the autonomous loop) + +1. **Setup (once, no questions).** Run `$PDD setup --auto` - it detects capabilities and configures + the most autonomous valid combination itself (programmatic email when `EMAIL_*` creds exist, + Browserbase when its key exists, `age` encryption when the binary exists, `autonomy=full`). Then + `$PDD doctor` and show the operator the readiness output **for information, not as a question** - + proceed immediately. Mention what would unlock more automation (e.g. email creds) but do not wait. +2. **Intake + consent (the ONE human conversation).** `$PDD intake ...` with `--consent` (and + `--consent-method`). Without consent the engine refuses to plan or act. Collect everything in one + pass - names/aliases, current + prior cities, emails, phones - so you never have to come back with + questions. For California subjects, also read `references/legal/drop.md`: `next` will surface a + `drop_submit` one-shot that deletes from every registered broker (~545) at once, which is the + single highest-leverage action. File it, then `drop --filed`. For non-CA subjects the + registry is covered by targeted CCPA/GDPR emails (`registry --search`, then `send-email`); the + people-search sites are worked directly in either case. +3. **Drain the queue.** Loop: + + ``` + while true: + q = $PDD next + if q.actions is empty: break + execute EVERY action in order; record each outcome via $PDD record + ``` + + `next` emits, in order: `refresh_brokers` (stale cache), `fanout_scan`/`scan_inline` (Phase 1 + crawl - see step 4), `poll_verification` (in-flight email confirmations), `verify_removal` (due + re-checks), `optout_web_form`/`optout_email_send` (Phase 2, parents-first with playbook steps), + `indirect_email_send`, and `stealth_rescan`. Human-only work never appears as an action - it + accumulates in `q.human_digest`. In `autonomy=full`, execute actions without pausing; honor + `confirm_first` in `assisted` mode. +4. **Scanning (when `next` says so).** For `fanout_scan`: run `$PDD fanout ` and **spawn one + `delegate_task` subagent per `batch`, in parallel, passing that batch's ready-made `brief`** - do + not scan all brokers yourself sequentially. For `scan_inline`: scan the few brokers yourself. + Either way, each broker gets **every** `search_vectors` entry via the `references/methods.md` + ladder (`web_extract` → `site:` probe → `browser_navigate` → `scrapling`), a 404 is INCONCLUSIVE + (not `not_found`), `blocked` is recorded when `antibot` is set and no stealth browser is available, + and subject vs namesake/relative is confirmed before recording: + `$PDD record --found --evidence '{"listing_urls":[...]}'`. + The parent re-verifies key `found` claims from subagents before trusting them. +5. **Opt-outs (when `next` says so).** Actions come pre-ordered parents-first with `steps` from each + broker record's own `optout.playbook` (field-verified; cluster parents like PeopleConnect, + Whitepages, BeenVerified, Spokeo have exact, live-checked recipes). **Deletion beats + suppression**: when an action carries `prefer_deletion`, complete the record's DELETION lane (e.g. + PeopleConnect's "DELETE MY USER DATA"), never just the hide-my-listing flow. Per method: + - **web_form** → drive `optout_url` with `browser_navigate`/`browser_type`/`browser_click`, submit + only `disclosure_fields`, screenshot the confirmation, then the action's `after` record command. + Playbooks may end with a right-to-delete `send-email` follow-up - do it (full erasure, not just + listing suppression). + - **email** → `$PDD send-email --kind --to + --listing ` records + discloses in one step (recipient locked to addresses the broker + record declares; `next` picks the kind from residency - never claim CCPA/GDPR for someone who + can't). In **browser** mode it returns a recipient-locked `compose` payload: compose a new + message to `compose.to` with `compose.subject`/`compose.body` exactly in the operator's webmail + via `browser_*` and send (no password); in **programmatic** mode it SMTP-sends. `next` also + routes human-gated forms (phone-callback/gov-ID) through a broker's deletion email when one + exists - the **rescue lane** (verified Whitepages pattern). Draft-only falls back to + `render-email` + a digest entry. + - **captcha** → soft/managed challenges clear automatically on the default cloud browser (proceed + as normal); only a hard interactive/behavioral challenge it can't pass is recorded `blocked` + (requeued for the stealth/operator-browser pass). Never a solver service. + - **phone_callback / account / gov_id / fax / mail / voice (T3)** *without a deletion email* → + never an agent action; `next` already routed these to the digest. Record them: + `$PDD record human_task_queued --reason "..."`. + 6. **Verification (when `next` says so).** In **programmatic** mode `$PDD poll-verification ` + finds arrived confirmation links via IMAP (anti-phishing scored, auto-advances state). In + **browser** mode, open the broker's confirmation email in the operator's webmail and run + `$PDD verify-link --text ''` to score the link. Either way **open the + link in the same browser** (several brokers bind the verification session to the browser that + opens it), finish the flow, then record `awaiting_processing`. `confirmed_removed` ONLY after a + verifying re-scan shows the listing gone - never off the submission flow's own confirmation page. +7. **Wrap up (once per run).** When `next` returns no actions: present `$PDD tasks ` (the + consolidated human digest) if non-empty, then `$PDD status `; if the Sheets tracker is + on, append `$PDD report --sheets` rows via the `google-workspace` skill. +8. **Schedule the next wake-up.** `next` returns `next_wake_at` (earliest due re-check). Create ONE + `cronjob` that re-runs this skill's loop for the subject (a prompt like: *"run the + unbroker loop for : `$PDD next` and execute all actions"*). Processing + windows, verification polls, and reappearance sweeps all flow through the same queue, so the case + keeps advancing with zero human attention. + +## Pitfalls + +- **Never disclose more than the broker already shows.** Submit only `disclosure_fields`. The engine + never volunteers SSN/ID numbers; you must not either. +- **No consent, no action.** The engine enforces this; do not work around it to "research" a third party. +- **`send-email` is idempotent + rate-limited.** It refuses to re-send a case already `submitted` + or beyond (use `--force` only if a genuine re-send is needed), and SMTP sends are paced by + `email_min_interval_seconds` (default 20s) with retry/backoff. Do not loop it to "make sure" - + a successful SMTP handoff is not proof of delivery; the due-queue re-scan is the real confirmation. +- **Ledger writes are locked.** Concurrent runs (cron + manual) serialize safely; if you ever see a + lock timeout, another run is mid-write - let it finish, don't delete the `.lock` by hand. +- **Autonomy ≠ improvisation.** Full autonomy means not *asking* between steps; it does not loosen any + gate. If a broker demands MORE than the planned `disclosure_fields` mid-flow, stop that case and + queue it (`human_task_queued --reason`) rather than deciding alone to disclose extra PII. +- **Don't interrupt the run with questions.** Config choices are `setup --auto`'s job; human-only work + goes to the digest. The only mid-run question that's ever warranted is a missing-identity fact that + blocks scanning (e.g. no city at all) - and that should have been collected at intake. +- **Use `terminal`, not `execute_code`** for `pdd.py` (secret scrubbing + output redaction break it). +- **Dossiers are plaintext by default** (JSON, `0600` under `HERMES_HOME`). For at-rest encryption run + `$PDD setup --encryption age` - it generates a local `age` key and encrypts dossiers + ledgers (the + audit log holds field names only and stays plaintext). It guards casual/backup/commit exposure, not + a full-`HERMES_HOME` read; set `PDD_AGE_IDENTITY` to a separate volume for real key separation. + `$PDD doctor` shows whether encryption is *actually* engaged (not just whether `age` is installed). +- **"Hidden from free search" ≠ deleted.** Only mark `confirmed_removed` after verifying the record is + actually gone; note paid-tier retention in the report. +- **Soft CAPTCHAs clear by default; don't fight the hard ones.** The default cloud browser passes + managed/soft challenges as normal operation (those brokers stay T1). For a hard interactive one it + genuinely can't pass, record `blocked` and let the stealth/operator-browser pass take it - never a + third-party solver service or fingerprint spoofing. +- **Broker pages change.** If a flow breaks, `$PDD record ... blocked` and flag the broker file in + `references/brokers/` for re-verification instead of guessing. +- **Verify non-field-verified records before submitting.** `confidence: auto` records came from + parsing BADBOOL (read `optout.notes`/`optout.links`, confirm the real opt-out URL). `confidence: + documented` records (several people-search sites) carry the correct published opt-out URL but have + **not** been field-verified (they 403 datacenter IPs), so confirm the live flow via the operator's + residential browser on first use, then set `last_verified`. Field-verified curated records (no + `confidence`, e.g. the cluster parents) have checked mechanics and take precedence. + +## Verification + +- `scripts/run_tests.sh tests/skills/test_unbroker_skill.py` (hermetic; no network), or the + dependency-free runner `python3 tests/skills/test_unbroker_skill.py`. +- Dry run: `$PDD setup --auto && $PDD doctor && SID=$($PDD intake --full-name "Test Person" + --email t@example.com --consent | python3 -c 'import sys,json;print(json.load(sys.stdin)["subject_id"])') + && $PDD next "$SID"` and confirm a readiness summary plus an ordered action queue. diff --git a/optional-skills/security/unbroker/assets/unbroker.png b/optional-skills/security/unbroker/assets/unbroker.png new file mode 100644 index 00000000000..18f7e3d5193 Binary files /dev/null and b/optional-skills/security/unbroker/assets/unbroker.png differ diff --git a/optional-skills/security/unbroker/references/brokers/advancedbackgroundchecks.json b/optional-skills/security/unbroker/references/brokers/advancedbackgroundchecks.json new file mode 100644 index 00000000000..ecc3932a61d --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/advancedbackgroundchecks.json @@ -0,0 +1,50 @@ +{ + "id": "advancedbackgroundchecks", + "name": "AdvancedBackgroundChecks", + "category": "people_search", + "priority": "high", + "jurisdictions": ["US"], + "search": { + "method": "url_pattern", + "url": "https://www.advancedbackgroundchecks.com", + "fetch": "web_extract", + "match_signal": "result", + "by": ["name", "phone", "address"], + "url_patterns": { + "name": "https://www.advancedbackgroundchecks.com/name/{first}-{last}", + "name_state": "https://www.advancedbackgroundchecks.com/name/{first}-{last}/in/{ST}", + "phone": "https://www.advancedbackgroundchecks.com/phone/{digits10}", + "address": "https://www.advancedbackgroundchecks.com/address/{num}-{street-with-type}-{city}-{ST}-{zip}", + "person_profile": "https://www.advancedbackgroundchecks.com/find/person/{first}-{last}-{22charID}" + }, + "url_format_quirks": [ + "All path segments lowercase, spaces -> hyphens. Name: /name/jane-public ; name+state: /name/jane-public/in/NY (state 2-letter UPPERCASE).", + "Phone: /phone/5551234567 (10 digits, NO punctuation).", + "Address: /address/123-main-st-anytown-ny-12345 (all hyphen-joined incl. ZIP; abbreviations like 'S' and 'Ave' kept verbatim, not expanded).", + "Removable unit is the person profile: /find/person/{first}-{last}-{22charID} (opaque mixed-case ID). Also /find/name/{first}-{last} and /find/name/{first}-{last}/in/{ST}.", + "NO 404s for plausible patterns: an empty search returns a soft-200 page with 'similar' fuzzy rows. The real 'not found' signal is the ABSENCE of an exact-match block in the results heading, NOT an HTTP error. Do not record not_found off a 404 here; read the heading.", + "No anti-bot gating on search/result/phone/address pages (web_extract reads them fine). Site self-brands as 'ActualPeopleSearch' in FAQ text.", + "Search tabs: /?tab=phone /?tab=email /?tab=address. Directories: /lastnames/{surname} /firstnames/{first} /people/{state-name}/{city-name} /people/zip/{zip} /people/areacode/{code}." + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.advancedbackgroundchecks.com/opt-out", + "requires": { + "profile_url": false, + "email_verification": true, + "captcha": true, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": ["full_name", "contact_email"], + "notes": "Two-step email-verification flow: initial form (radio 'I am: The subject of the request / An authorized agent of the subject', First name*, Middle name, Last name*, Email*, consent) -> emailed link -> full form -> confirmation (processed within 45 days). CAPTCHA-gated: Google reCAPTCHA ('Recaptcha requires verification'). Verified read-only 2026-06-30; not submitted.", + "est_processing_days": 3, + "reappearance_risk": "medium" + }, + "last_verified": "2026-06-30", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/beenverified.json b/optional-skills/security/unbroker/references/brokers/beenverified.json new file mode 100644 index 00000000000..8fad47fe34c --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/beenverified.json @@ -0,0 +1,56 @@ +{ + "id": "beenverified", + "name": "BeenVerified", + "category": "people_search", + "priority": "crucial", + "jurisdictions": ["US"], + "parent": "beenverified", + "owns": ["peoplelooker", "peoplesmart"], + "search": { + "method": "url_pattern", + "url": "https://www.beenverified.com/app/optout/search", + "fetch": "browser", + "match_signal": "result", + "by": ["name", "phone", "email", "address"] + }, + "optout": { + "tier": "T1", + "method": "web_form", + "url": "https://www.beenverified.com/svc/optout/search/optouts", + "email": "privacy@beenverified.com", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": ["full_name", "contact_email", "profile_url"], + "deletion": { + "via": "email_followup", + "email": "privacy@beenverified.com", + "kinds": ["ccpa", "generic"], + "notes": "privacy@beenverified.com is the documented address for opt-out, access, deletion, and correction requests (verified from the live privacy policy 2026-07-01). Controller is The Lifetime Value Co. -- a deletion request here can name their other properties too. DPO: dpo@beenverified.com. CCPA window: respond within 45 days (extendable to 90)." + }, + "playbook": [ + "Opt-out tool at beenverified.com/svc/optout/search/optouts ('Do Not Sell or Share' footer link; the legacy /app/optout/search path serves the same search) -- clears PeopleLooker + PeopleSmart.", + "Search the subject, open the matching listing, and submit with the confirmed profile URL + contact email. Email verification: poll-verification picks up the confirmation link; open it in the agent's own browser.", + "ONE opt-out per email address via the tool; for additional listings email support@beenverified.com or privacy@beenverified.com.", + "FULL DELETION follow-up (right-to-delete, beyond listing suppression): send-email --kind ccpa (CA) / generic to privacy@beenverified.com naming the listing URL(s); as the controller is The Lifetime Value Co., ask that the deletion cover affiliated LTV properties (NeighborWho, Ownerly, NumberGuru, Bumper) in the same request.", + "A separate property-search opt-out exists (NeighborWho/Ownerly are property-focused sisters); note residual exposure if relevant and re-scan the children after the parent confirms." + ], + "notes": "Opt-out form + deletion email both verified from the live privacy policy 2026-07-01 (policy dated 2025-10-21). Authorized agents: signed authorization letter or CA POA emailed to privacy@beenverified.com; agent-submitted delete/know requests must include the consumer's name, valid email, age, and address.", + "quirks": [ + "The 'Do Not Sell or Share My Personal Information' footer link now points to /svc/optout/search/optouts (observed 2026-07-01); records citing /app/optout/search are the same tool's older entry.", + "One opt-out per email address through the tool; email support for additional removals. The same browser/inbox must open the confirmation link.", + "Sister LTV Co. properties (NeighborWho, Ownerly, NumberGuru, Bumper) keep separate opt-out tools -- do NOT assume the BV tool cleared them; the privacy@ deletion request can name them, then verify by scanning each.", + "Right-to-know requests get an initial response within 10 days; deletion/access within 45 days (CCPA)." + ], + "est_processing_days": 7, + "reappearance_risk": "medium" + }, + "last_verified": "2026-07-01", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/clustal.json b/optional-skills/security/unbroker/references/brokers/clustal.json new file mode 100644 index 00000000000..c90a6fdfd99 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/clustal.json @@ -0,0 +1,47 @@ +{ + "id": "clustal", + "name": "Clustal", + "category": "people_search", + "priority": "high", + "jurisdictions": ["US"], + "search": { + "method": "url_pattern", + "url": "https://www.clustal.org/", + "fetch": "web_extract", + "match_signal": "record", + "by": ["name"], + "url_patterns": { + "name_index": "https://www.clustal.org/people-search/{letter}/{first-last}/", + "name_state": "https://www.clustal.org/people-search/{letter}/{first-last}/{st}/", + "record": "https://www.clustal.org/record/{first-last}-{8charID}/" + }, + "url_format_quirks": [ + "Name index: /people-search/{first-initial}/{first-last}/ e.g. /people-search/k/jane-public/ (lowercase, first letter of LAST name as the {letter} segment, hyphen-joined name). Optional state refine appends /{st}/ lowercase 2-letter.", + "Detail (removable unit): /record/{first-last}-{8charID}/ e.g. /record/jane-public-a1b2c3d4/ (opaque mixed-case 8-char id). The index page links each match to its /record/ URL.", + "Readable via web_extract (no hard bot gate as of 2026-07-01). Rich profile: age/DOB, current+prior addresses, phones, emails, relatives, neighbors, associates.", + "Name only: search is by name(+state); no reverse phone/email/address path observed. NOTE: clustal.org squats the 'Clustal' bioinformatics brand but IS a real people-search/data-broker site ('Find Your DNA Relatives'), not the sequence-alignment tool - do not dismiss it as a false positive." + ] + }, + "optout": { + "tier": "T0", + "method": "web_form", + "url": "https://www.clustal.org/privacy-control/", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": ["profile_url", "contact_email"], + "notes": "Opt-out at /privacy-control/ using the /record/ profile URL; support help@clustal.org. Verify the exact form fields + any CAPTCHA live before submitting (requires flags below are provisional from the site's opt-out copy, not yet a live form walk-through).", + "email": "help@clustal.org", + "est_processing_days": 7, + "reappearance_risk": "medium" + }, + "last_verified": "2026-07-01", + "source": "BADBOOL", + "confidence": "curated" +} diff --git a/optional-skills/security/unbroker/references/brokers/clustrmaps.json b/optional-skills/security/unbroker/references/brokers/clustrmaps.json new file mode 100644 index 00000000000..988aeddd6e9 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/clustrmaps.json @@ -0,0 +1,52 @@ +{ + "id": "clustrmaps", + "name": "ClustrMaps", + "category": "people_search", + "priority": "high", + "jurisdictions": [ + "US" + ], + "parent": "clustrmaps", + "owns": [], + "search": { + "method": "url_pattern", + "url": "https://clustrmaps.com/", + "fetch": "browser", + "match_signal": "result", + "antibot": "cloudflare", + "by": [ + "name", + "phone", + "address" + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://clustrmaps.com/bl/opt-out", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "contact_email", + "profile_url" + ], + "notes": "Address/resident aggregator. Opt-out at /bl/opt-out: submit the profile URL + email, confirm the link.", + "quirks": [ + "Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.", + "Needs the confirmed profile_url (the address/person page).", + "Email verification required." + ], + "est_processing_days": 3, + "reappearance_risk": "medium" + }, + "last_verified": null, + "source": "curated", + "confidence": "documented" +} diff --git a/optional-skills/security/unbroker/references/brokers/cyberbackgroundchecks.json b/optional-skills/security/unbroker/references/brokers/cyberbackgroundchecks.json new file mode 100644 index 00000000000..a592504f768 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/cyberbackgroundchecks.json @@ -0,0 +1,53 @@ +{ + "id": "cyberbackgroundchecks", + "name": "CyberBackgroundChecks", + "category": "people_search", + "priority": "high", + "jurisdictions": [ + "US" + ], + "parent": "cyberbackgroundchecks", + "owns": [], + "search": { + "method": "url_pattern", + "url": "https://www.cyberbackgroundchecks.com/", + "fetch": "browser", + "match_signal": "result", + "antibot": "cloudflare", + "by": [ + "name", + "phone", + "address" + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.cyberbackgroundchecks.com/removal", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "full_name", + "contact_email", + "profile_url" + ], + "notes": "Free people-search. Opt-out at /removal: find the record, submit email, confirm link.", + "quirks": [ + "Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.", + "Needs the confirmed profile_url.", + "Email verification required." + ], + "est_processing_days": 3, + "reappearance_risk": "medium" + }, + "last_verified": null, + "source": "curated", + "confidence": "documented" +} diff --git a/optional-skills/security/unbroker/references/brokers/familytreenow.json b/optional-skills/security/unbroker/references/brokers/familytreenow.json new file mode 100644 index 00000000000..5700d57d01c --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/familytreenow.json @@ -0,0 +1,50 @@ +{ + "id": "familytreenow", + "name": "FamilyTreeNow", + "category": "people_search", + "priority": "crucial", + "jurisdictions": [ + "US" + ], + "parent": "familytreenow", + "owns": [], + "search": { + "method": "url_pattern", + "url": "https://www.familytreenow.com/", + "fetch": "browser", + "match_signal": "result", + "antibot": "cloudflare", + "by": [ + "name", + "phone", + "address" + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.familytreenow.com/optout", + "requires": { + "profile_url": false, + "email_verification": false, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "full_name" + ], + "notes": "Notorious free people-search / doxxing site (no registry). Opt-out: search the record, select it, confirm removal on-site. No account, free.", + "quirks": [ + "Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.", + "Select the exact record from search results, then confirm removal; historically no email step, but re-lists periodically so keep the re-scan scheduled." + ], + "est_processing_days": 2, + "reappearance_risk": "high" + }, + "last_verified": null, + "source": "curated", + "confidence": "documented" +} diff --git a/optional-skills/security/unbroker/references/brokers/fastpeoplesearch.json b/optional-skills/security/unbroker/references/brokers/fastpeoplesearch.json new file mode 100644 index 00000000000..3cd7c90b1c4 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/fastpeoplesearch.json @@ -0,0 +1,43 @@ +{ + "id": "fastpeoplesearch", + "name": "FastPeopleSearch", + "category": "people_search", + "priority": "high", + "jurisdictions": ["US"], + "search": { + "method": "url_pattern", + "url": "https://www.fastpeoplesearch.com/", + "fetch": "browser", + "antibot": "datadome", + "match_signal": "result", + "by": ["name", "phone", "address"], + "url_patterns": { + "name": "https://www.fastpeoplesearch.com/name/{first}-{last}" + }, + "url_format_quirks": [ + "Name path /name/jane-public (lowercase, hyphen join) was ACCEPTED by the router under challenge, so the name pattern is confirmed even though content never rendered.", + "Sibling of truepeoplesearch (near-identical removal flow/branding); phone pattern is LIKELY /{digits} or /phone/{digits} and address /address/... but UNCONFIRMED - do not assume.", + "MOST aggressively gated of the people-search trio (2026-06-30): both server-side scraping (Firecrawl 504) AND headless browser (Cloudflare -> DataDome) fail on search AND /removal. Treat as fully blocked; needs residential-proxy/stealth browser to scan or opt out." + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.fastpeoplesearch.com/removal", + "requires": { + "profile_url": false, + "email_verification": true, + "captcha": true, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": ["full_name", "contact_email"], + "notes": "Opt-out NOT directly observed (2026-06-30): /removal is itself behind DataDome. By sibling-site pattern almost certainly mirrors truepeoplesearch (email-link flow, subject/agent toggle, name+email fields, hCaptcha) - INFERRED, not verified. Confirm before acting.", + "est_processing_days": 3, + "reappearance_risk": "high" + }, + "last_verified": "2026-06-30", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/intelius.json b/optional-skills/security/unbroker/references/brokers/intelius.json new file mode 100644 index 00000000000..1b3af8358ec --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/intelius.json @@ -0,0 +1,88 @@ +{ + "id": "intelius", + "name": "Intelius", + "category": "people_search", + "priority": "crucial", + "jurisdictions": [ + "US" + ], + "parent": "peopleconnect", + "owns": [ + "truthfinder", + "instantcheckmate", + "ussearch", + "zabasearch", + "classmates", + "peoplefinder", + "peoplelookup", + "addresses", + "anywho", + "publicrecords" + ], + "search": { + "method": "url_pattern", + "url": "https://www.intelius.com/", + "fetch": "web_extract", + "match_signal": "result", + "by": [ + "name", + "phone", + "address" + ], + "url_patterns": { + "name": "https://www.intelius.com/people-search/{First}-{Last}/", + "name_state": "https://www.intelius.com/people-search/{first}-{last}/{state-name}/", + "full_report": "https://www.intelius.com/search/?firstName={First}&lastName={Last}&city={City}&state={ST}&traffic%5Bsource%5D=INTSEO" + }, + "url_format_quirks": [ + "Name summary page: /people-search/{First}-{Last}/ (hyphen join; case-insensitive, both Jane-Public and jane-public work). Readable via web_extract (no hard bot gate as of 2026-06-30).", + "State refine: /people-search/{first}-{last}/{state-name}/ with the state SPELLED OUT lowercase, e.g. /jane-public/new-york/ (NOT the 2-letter code).", + "The summary shows last-known address + a 'possible relatives' list + a FAQ ('Where does X live?'). Corroborate the subject by address before acting; the 'Top People with the Last Name {Last}' block is unrelated namesakes.", + "Part of PeopleConnect: same data surfaces on Truthfinder/InstantCheckmate/USSearch; one suppression at suppression.peopleconnect.us covers the cluster." + ] + }, + "optout": { + "tier": "T1", + "method": "web_form", + "url": "https://suppression.peopleconnect.us/login", + "email": "privacy@peopleconnect.us", + "requires": { + "profile_url": false, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "contact_email" + ], + "deletion": { + "via": "in_flow", + "url": "https://suppression.peopleconnect.us/guided-mode", + "email": "privacy@peopleconnect.us", + "kinds": ["ccpa", "generic"], + "notes": "The portal's 'Right to Delete / DELETE MY USER DATA' (bottom of guided-mode) is the verified deletion path and covers the whole cluster. privacy@peopleconnect.us is the documented rights-request address (their 2025 CPRA metrics: 33,513 delete requests, median response < 1 day) - use it as the fallback if the portal breaks." + }, + "playbook": [ + "PeopleConnect portal (suppression.peopleconnect.us/login, privacy-center entry at /privacy-center) -- ONE flow here clears Truthfinder, Instant Checkmate, US Search, ZabaSearch, Classmates and ~15 more. DO THIS PARENT FIRST.", + "Step 1 asks ONLY for an email + consent checkbox (no name/DOB, no CAPTCHA) -> sends a verification email. Least-disclosure entry: just the contact email.", + "poll-verification will pick up the verify link. The link authenticates a SESSION bound to the browser that OPENS it: the SAME agent browser must open the link and drive /guided-mode (a different browser is bounced to /login).", + "SUPPRESSION != DELETION -- guided-mode's default flow only HIDES the report (data retained, can re-list). Scroll to the BOTTOM of guided-mode and use 'Right to Delete / DELETE MY USER DATA' (CCPA/CPRA) -- this actually deletes across the cluster and emails its own confirmation link; poll and open that too.", + "If the portal breaks: email privacy@peopleconnect.us (rights-request address, median response < 1 day per their published metrics), or sister addresses privacy@intelius.com / privacy@truthfinder.com / privacy@instantcheckmate.com / support@ussearch.com / privacy@classmates.com; phone 1-888-245-1655.", + "After the deletion confirms, re-scan the covered children (they normally drop out) before submitting any duplicate child opt-out." + ], + "notes": "PeopleConnect portal covers the cluster. Authorized-agent requests: signed written authorization (full name, address, phone, the email the consumer uses) or POA; for Right-to-Delete they verify agent authority with the consumer by email. Verified from the live privacy policy 2026-07-01 (policy dated 2026-06-30).", + "quirks": [ + "Step 1 (suppression.peopleconnect.us/login) asks ONLY for an email + a consent checkbox, then 'Continue' -> a verification email with a link. No CAPTCHA, no name/DOB at step 1. Least-disclosure entry: just the contact email. Verified live 2026-06-30.", + "The verification link authenticates a SESSION and lands on /guided-mode. That session is bound to the browser that OPENED it; a different browser hitting /guided-mode is redirected back to /login. So for hands-off automation the SAME agent browser must open the verify link (Mode B: read inbox -> agent browser navigates the link -> drive guided-mode).", + "SUPPRESSION != DELETION. The guided-mode/suppression flow only HIDES the background report (data retained, can re-list). At the BOTTOM of guided-mode there is a separate 'Right to Delete' section with a 'DELETE MY USER DATA' button (CCPA/CPRA) that removes the user data across the cluster - this is the stronger action and should be preferred. It also emails a confirmation link. (Confirmed via security.org Instant Checkmate guide + live flow 2026-06-30.)", + "Their published request metrics (2025, all US users regardless of state): 33,513 deletion requests, 26,603 complied, median response < 1 day - the deletion lane is real and fast. Verified from privacy policy 2026-07-01." + ], + "est_processing_days": 7, + "reappearance_risk": "medium" + }, + "last_verified": "2026-07-01", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/mylife.json b/optional-skills/security/unbroker/references/brokers/mylife.json new file mode 100644 index 00000000000..cdb739c1916 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/mylife.json @@ -0,0 +1,35 @@ +{ + "id": "mylife", + "name": "MyLife", + "category": "people_search", + "priority": "crucial", + "jurisdictions": ["US"], + "search": { + "method": "url_pattern", + "url": "https://www.mylife.com", + "fetch": "browser", + "match_signal": "profile", + "by": ["name"] + }, + "optout": { + "tier": "T3", + "method": "phone", + "url": "https://www.mylife.com/privacyrequest", + "requires": { + "profile_url": true, + "email_verification": false, + "captcha": false, + "gov_id": true, + "account": false, + "phone_callback": false, + "phone_voice": true, + "payment": false + }, + "inputs": ["full_name", "profile_url"], + "notes": "Often pushes a driver's-license upload or a call to (888) 704-1900. Emailing privacy@mylife.com with name + profile link is an alternative. Also covers Wink.com.", + "est_processing_days": 14, + "reappearance_risk": "high" + }, + "last_verified": "2026-06-28", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/nuwber.json b/optional-skills/security/unbroker/references/brokers/nuwber.json new file mode 100644 index 00000000000..f8d4424a3e8 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/nuwber.json @@ -0,0 +1,52 @@ +{ + "id": "nuwber", + "name": "Nuwber", + "category": "people_search", + "priority": "high", + "jurisdictions": [ + "US" + ], + "parent": "nuwber", + "owns": [], + "search": { + "method": "url_pattern", + "url": "https://nuwber.com/", + "fetch": "browser", + "match_signal": "result", + "antibot": "cloudflare", + "by": [ + "name", + "phone", + "address" + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://nuwber.com/removal/link", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "contact_email", + "profile_url" + ], + "notes": "People-search. Opt-out: submit the profile URL + email at /removal/link, confirm via the emailed link.", + "quirks": [ + "Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.", + "Needs the confirmed profile_url (paste the listing URL you recorded).", + "Email verification required." + ], + "est_processing_days": 3, + "reappearance_risk": "medium" + }, + "last_verified": null, + "source": "curated", + "confidence": "documented" +} diff --git a/optional-skills/security/unbroker/references/brokers/peekyou.json b/optional-skills/security/unbroker/references/brokers/peekyou.json new file mode 100644 index 00000000000..0c46810dc2d --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/peekyou.json @@ -0,0 +1,53 @@ +{ + "id": "peekyou", + "name": "PeekYou", + "category": "people_search", + "priority": "high", + "jurisdictions": [ + "US" + ], + "parent": "peekyou", + "owns": [], + "search": { + "method": "url_pattern", + "url": "https://www.peekyou.com/", + "fetch": "browser", + "match_signal": "result", + "antibot": "cloudflare", + "by": [ + "name", + "phone", + "address" + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.peekyou.com/about/contact/optout", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "full_name", + "contact_email", + "profile_url" + ], + "notes": "Aggregates social/web profiles. Opt-out: paste your PeekYou profile URL(s), pick a reason, provide email, confirm the link.", + "quirks": [ + "Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.", + "Needs the confirmed PeekYou profile_url.", + "Email verification required." + ], + "est_processing_days": 7, + "reappearance_risk": "medium" + }, + "last_verified": null, + "source": "curated", + "confidence": "documented" +} diff --git a/optional-skills/security/unbroker/references/brokers/peoplefinders.json b/optional-skills/security/unbroker/references/brokers/peoplefinders.json new file mode 100644 index 00000000000..6187f0c14d5 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/peoplefinders.json @@ -0,0 +1,53 @@ +{ + "id": "peoplefinders", + "name": "PeopleFinders", + "category": "people_search", + "priority": "high", + "jurisdictions": [ + "US" + ], + "parent": "peoplefinders", + "owns": [], + "search": { + "method": "url_pattern", + "url": "https://www.peoplefinders.com/", + "fetch": "browser", + "match_signal": "result", + "antibot": "cloudflare", + "by": [ + "name", + "phone", + "address" + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.peoplefinders.com/opt-out", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "full_name", + "contact_email", + "profile_url" + ], + "notes": "Standalone people-search (Confi-Chek family). Opt-out: find the listing, submit with a contact email, confirm via the emailed link.", + "quirks": [ + "Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.", + "Needs the confirmed profile_url from evidence.", + "Email verification: poll-verification picks up the link; open it in the agent browser." + ], + "est_processing_days": 7, + "reappearance_risk": "medium" + }, + "last_verified": null, + "source": "curated", + "confidence": "documented" +} diff --git a/optional-skills/security/unbroker/references/brokers/radaris.json b/optional-skills/security/unbroker/references/brokers/radaris.json new file mode 100644 index 00000000000..dbfb29a9a7d --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/radaris.json @@ -0,0 +1,58 @@ +{ + "id": "radaris", + "name": "Radaris", + "category": "people_search", + "priority": "crucial", + "jurisdictions": [ + "US" + ], + "search": { + "method": "url_pattern", + "url": "https://radaris.com/", + "fetch": "web_extract", + "match_signal": "View Profile", + "by": [ + "name", + "phone", + "address" + ], + "url_patterns": { + "name": "https://radaris.com/p/{First}/{Last}/" + }, + "url_format_quirks": [ + "Name profile page: /p/{First}/{Last}/ with First/Last Capitalized and a TRAILING slash, e.g. /p/Jane/Public/ . Readable via web_extract (no hard bot gate as of 2026-06-30).", + "The page aggregates: a 'phone numbers & home addresses' table (the DIRECT hit for the subject), a relatives/namesakes carousel ('Review the potential relatives'), and resume/CV records. The subject's removable row is in the address table; the carousel entries are OTHER people - disambiguate by address/age before acting.", + "Page is noisy with testimonials/reviews boilerplate; the signal blocks are 'phone numbers & home addresses N' and 'resumes & CV records N'." + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://radaris.com/control-privacy", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": true, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "profile_url", + "contact_email" + ], + "notes": "Multi-step control-privacy wizard: step 1 NEXT -> step 2 'identify your personal page' (paste the NUMBERED profile URL into the 'Or Enter URL of your page' field; typing reveals a NEXT submit button) -> then user_email + Google reCAPTCHA -> emailed verification link. If there is no View Profile button for the subject, email customer-service@radaris.com and reply to the auto-response until removed.", + "email": "customer-service@radaris.com", + "quirks": [ + "CAPTCHA: the form carries a Google reCAPTCHA (hidden field g-recaptcha-response) plus anti-bot fingerprint tokens (jfp, token). Tier is T2 without a captcha-clearing browser backend (T1 with Browserbase). (Record previously mis-declared captcha:false.)", + "The /control-privacy form REQUIRES the per-person NUMBERED profile URL. Pasting the aggregate /p/{First}/{Last}/ URL is rejected with the validation error: 'The URL must include unique number at the end'. The real removable URL looks like https://{region}.radaris.com/person/~First-Last/1234567890 (see the field placeholder).", + "The subject's own record may appear ONLY as a static row in the 'phone numbers & home addresses' table with NO 'View Profile' link, so no numbered profile URL is exposed for them (the /p/{First}/{Last}/ page deep-links only to relatives/namesakes via JS onclick, not static hrefs). When that happens the /control-privacy form cannot be used -- do NOT fabricate or submit a relative's or namesake's profile URL. Fall back to email: write customer-service@radaris.com with the subject's own listed details and request removal, replying to the auto-response until confirmed.", + "Pressing Enter in the URL field reloads the wizard to step 1 (does not submit); type into the field and click the revealed NEXT button instead." + ], + "est_processing_days": 14, + "reappearance_risk": "high" + }, + "last_verified": "2026-06-30", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/searchpeoplefree.json b/optional-skills/security/unbroker/references/brokers/searchpeoplefree.json new file mode 100644 index 00000000000..47be9653535 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/searchpeoplefree.json @@ -0,0 +1,53 @@ +{ + "id": "searchpeoplefree", + "name": "SearchPeopleFree", + "category": "people_search", + "priority": "high", + "jurisdictions": [ + "US" + ], + "parent": "searchpeoplefree", + "owns": [], + "search": { + "method": "url_pattern", + "url": "https://www.searchpeoplefree.com/", + "fetch": "browser", + "match_signal": "result", + "antibot": "cloudflare", + "by": [ + "name", + "phone", + "address" + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.searchpeoplefree.com/opt-out", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "full_name", + "contact_email", + "profile_url" + ], + "notes": "Free people-search. Opt-out: find the listing, submit with an email, confirm the link.", + "quirks": [ + "Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.", + "Needs the confirmed profile_url.", + "Email verification required." + ], + "est_processing_days": 3, + "reappearance_risk": "medium" + }, + "last_verified": null, + "source": "curated", + "confidence": "documented" +} diff --git a/optional-skills/security/unbroker/references/brokers/spokeo.json b/optional-skills/security/unbroker/references/brokers/spokeo.json new file mode 100644 index 00000000000..d63c0f101ba --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/spokeo.json @@ -0,0 +1,56 @@ +{ + "id": "spokeo", + "name": "Spokeo", + "category": "people_search", + "priority": "crucial", + "jurisdictions": ["US"], + "parent": "spokeo", + "owns": ["freepeopledirectory"], + "search": { + "method": "url_pattern", + "url": "https://www.spokeo.com/search", + "fetch": "browser", + "match_signal": "profile", + "by": ["name", "phone", "email", "address"] + }, + "optout": { + "tier": "T1", + "method": "web_form", + "url": "https://www.spokeo.com/optout", + "email": "privacy@spokeo.com", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": ["profile_url", "contact_email"], + "deletion": { + "via": "email_followup", + "email": "privacy@spokeo.com", + "kinds": ["ccpa", "generic"], + "notes": "privacy@spokeo.com is the documented direct privacy contact (verified live on /optout 2026-07-01). Use for full CCPA deletion beyond listing opt-out, for listings the form rejects, and when more listings keep surfacing." + }, + "playbook": [ + "Opt-out form at spokeo.com/optout -- clears FreePeopleDirectory. Inputs: the confirmed profile URL + contact email; the form emails a confirmation link (poll-verification picks it up; open it in the agent's own browser).", + "EVERY LISTING SEPARATELY: 'you may have multiple listings on Spokeo. Each one is identified by a unique URL and must be opted out individually' (their own wording). Run ALL search vectors first, collect every listing URL, and submit one opt-out per URL.", + "Fast: requests process in 24-48 hours per their page -- re-scan after 2 days and record the real outcome.", + "FULL DELETION follow-up: send-email --kind ccpa (CA) / generic to privacy@spokeo.com naming all listing URLs -- covers data retained beyond the free-search suppression.", + "Paid/account data may be retained even when hidden from free search -- verify actual removal via re-scan; never mark confirmed_removed off the free-search view alone." + ], + "notes": "Form + privacy@spokeo.com verified live 2026-07-01. Their page: opting out does not remove data from original sources and listings may reappear as new public records arrive -- keep the reappearance re-scan scheduled.", + "quirks": [ + "Profile URL formats the form accepts: listing URL like spokeo.com/{First}-{Last}/{City}/{ST}/p{id} or a purchase URL spokeo.com/purchase?q=... (examples shown on /optout).", + "Multiple listings per person are NORMAL (one per name x location x record cluster); each unique URL must be opted out individually -- feed every found listing URL through the form.", + "Processing is 24-48h ('depending on the nature of your request and the amount of data').", + "Paid accounts may retain data even when hidden from free search - verify actual removal, don't mark confirmed_removed off the free-search view alone." + ], + "est_processing_days": 2, + "reappearance_risk": "medium" + }, + "last_verified": "2026-07-01", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/thatsthem.json b/optional-skills/security/unbroker/references/brokers/thatsthem.json new file mode 100644 index 00000000000..dd6113313fa --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/thatsthem.json @@ -0,0 +1,46 @@ +{ + "id": "thatsthem", + "name": "That's Them", + "category": "people_search", + "priority": "crucial", + "jurisdictions": ["US"], + "search": { + "method": "url_pattern", + "url": "https://thatsthem.com/", + "fetch": "browser", + "match_signal": "result", + "by": ["name", "phone", "email", "address"], + "url_patterns": { + "name": "https://thatsthem.com/name/{First}-{Last}/{City}-{ST}", + "phone": "https://thatsthem.com/phone/{areacode}-{prefix}-{line}", + "email": "https://thatsthem.com/email/{email}", + "address": "https://thatsthem.com/address/{Street}-{City}-{ST}-{ZIP}" + }, + "url_format_quirks": [ + "ADDRESS path is fully hyphen-joined and joins street+city+state+ZIP with hyphens (e.g. /address/123-Main-St-Anytown-NY-12345) and REQUIRES the ZIP. The older slash form (/address/Street/City-ST) and any ZIP-less form 404.", + "NAME and PHONE and EMAIL paths use a slash before city/state and do NOT need a ZIP: /name/First-Last/City-ST , /phone/AAA-PPP-LLLL , /email/addr@host.", + "Street abbreviations are kept verbatim (Ave, Pl, St) - do NOT expand to Avenue/Place/Street; expanding 404s.", + "A 404 on a constructed URL means WRONG PATTERN, not 'no record present'. Treat as INCONCLUSIVE: fall back to the on-site search box (browser_type into the address/name field + submit) and read the resulting canonical URL." + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://thatsthem.com/optout", + "requires": { + "profile_url": false, + "email_verification": false, + "captcha": true, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": ["full_name", "street", "city", "state", "postal", "contact_email", "phone"], + "notes": "Opt-out form is gated by a Cloudflare Turnstile CAPTCHA (so tier is T2 without a captcha-clearing browser backend; T1 with Browserbase). All 7 fields are marked required by the form (Full Name, Street Address, City, State, ZIP, Email, Phone). Site sends a confirmation email and states ~72h processing (this is a completion notice, not a click-to-verify link). Least-disclosure tension: the form DEMANDS email+phone even though a public record may show less - disclose only to remove a record that is actually about the subject. Do not click the Spokeo identity-theft-protection link (paid product).", + "est_processing_days": 3, + "reappearance_risk": "low" + }, + "last_verified": "2026-06-30", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/truepeoplesearch.json b/optional-skills/security/unbroker/references/brokers/truepeoplesearch.json new file mode 100644 index 00000000000..0523fdab1c9 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/truepeoplesearch.json @@ -0,0 +1,43 @@ +{ + "id": "truepeoplesearch", + "name": "TruePeopleSearch", + "category": "people_search", + "priority": "high", + "jurisdictions": ["US"], + "search": { + "method": "url_pattern", + "url": "https://www.truepeoplesearch.com/", + "fetch": "browser", + "antibot": "datadome", + "match_signal": "result", + "by": ["name", "phone", "address", "email"], + "url_patterns": { + "name": "https://www.truepeoplesearch.com/results?name={First%20Last}&citystatezip={City,%20ST}" + }, + "url_format_quirks": [ + "Search results URL is QUERY-PARAM, not path: /results?name=Jane%20Public&citystatezip=Anytown,%20NY (space-encoded; comma between city and state). Confirmed as the canonical form the site's own search box generates.", + "On-site search tabs: Name / Phone / Address / Email / Neighbors (so name-, phone-, address-, and email-searchable).", + "HARD-BLOCKED for automated reads (2026-06-30): Cloudflare 'Just a moment...' interstitial -> DataDome device-check CAPTCHA (geo.captcha-delivery.com) on every results navigation. Page chrome (header/footer) may render while the result body stays blocked; submitting any search bounces to DataDome. web_extract/Firecrawl 504-timed out. Needs a residential-proxy/stealth browser (e.g. Browserbase) to scan; otherwise record 'blocked'." + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.truepeoplesearch.com/removal", + "requires": { + "profile_url": false, + "email_verification": true, + "captcha": true, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": ["full_name", "contact_email"], + "notes": "Removal page renders even when results are blocked. Flow: name+email+captcha -> emailed link -> fill opt-out form matching the record -> confirmation ('allow 3 days'). Fields: dropdown 'Exercise my rights as a: The subject of the request / An authorized agent of the subject', First Name*, Middle Name, Last Name*, Email Address*, consent checkbox. CAPTCHA-gated: hCaptcha ('I am human'). Contact support@truepeoplesearch.com; PO Box 7775 PMB 29296, San Francisco CA 94120-7775. Verified read-only 2026-06-30; not submitted. (Record previously mis-declared email_verification:false.)", + "est_processing_days": 3, + "reappearance_risk": "high" + }, + "last_verified": "2026-06-30", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/brokers/usphonebook.json b/optional-skills/security/unbroker/references/brokers/usphonebook.json new file mode 100644 index 00000000000..ff1d0082b6c --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/usphonebook.json @@ -0,0 +1,53 @@ +{ + "id": "usphonebook", + "name": "USPhoneBook", + "category": "people_search", + "priority": "high", + "jurisdictions": [ + "US" + ], + "parent": "usphonebook", + "owns": [], + "search": { + "method": "url_pattern", + "url": "https://www.usphonebook.com/", + "fetch": "browser", + "match_signal": "result", + "antibot": "cloudflare", + "by": [ + "name", + "phone", + "address" + ] + }, + "optout": { + "tier": "T2", + "method": "web_form", + "url": "https://www.usphonebook.com/opt-out", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": [ + "full_name", + "contact_email", + "profile_url" + ], + "notes": "Reverse-phone + people-search. Opt-out: paste the listing URL, provide an email, confirm via the emailed link.", + "quirks": [ + "Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.", + "Needs the confirmed profile_url.", + "Email verification required." + ], + "est_processing_days": 3, + "reappearance_risk": "medium" + }, + "last_verified": null, + "source": "curated", + "confidence": "documented" +} diff --git a/optional-skills/security/unbroker/references/brokers/whitepages.json b/optional-skills/security/unbroker/references/brokers/whitepages.json new file mode 100644 index 00000000000..ddc6a0362c8 --- /dev/null +++ b/optional-skills/security/unbroker/references/brokers/whitepages.json @@ -0,0 +1,57 @@ +{ + "id": "whitepages", + "name": "Whitepages", + "category": "people_search", + "priority": "crucial", + "jurisdictions": ["US"], + "parent": "whitepages", + "owns": ["411"], + "search": { + "method": "url_pattern", + "url": "https://www.whitepages.com/", + "fetch": "browser", + "match_signal": "listing", + "by": ["name", "phone", "address"] + }, + "optout": { + "tier": "T1", + "method": "email", + "url": "https://www.whitepages.com/suppression_requests", + "email": "privacyrequest@whitepages.com", + "requires": { + "profile_url": true, + "email_verification": true, + "captcha": false, + "gov_id": false, + "account": false, + "phone_callback": false, + "payment": false + }, + "inputs": ["full_name", "contact_email", "profile_url"], + "deletion": { + "via": "email", + "email": "privacyrequest@whitepages.com", + "url": "https://whitepagesprivacy.zendesk.com/hc/en-us/requests/new", + "kinds": ["ccpa", "generic"], + "notes": "privacyrequest@whitepages.com handles BOTH opt-out (removal from the site) and CCPA deletion requests, explicitly offered for people who do not want to provide a phone number for the automated tool. ~2 business day reply; opt-outs processed within 15 days. Verified from whitepages.com/privacy/consumer-rights 2026-07-01 (page dated 2026-06-22)." + }, + "playbook": [ + "EMAIL LANE (fully autonomous -- use this): send the removal + deletion request to privacyrequest@whitepages.com including the confirmed listing URL. This is Whitepages' own documented alternative 'if you would prefer not to provide a phone number'. Expect a reply within ~2 business days; they may ask identity-verification questions (name, email) -- answer with least-disclosure, never an ID number.", + "Include in the email: the listing URL(s), the subject's full name as listed, and the request to (a) remove the listing(s), (b) opt out of sale/sharing, and (c) delete personal data (CCPA 1798.105 for CA residents; they honor requests from other states per their consumer-rights page).", + "'Once a listing is removed, all known connected listings are also removed and the requester's information will not be sold by Whitepages in any capacity' -- one request covers connected listings; still re-scan afterward, and check Whitepages Premium + 411.com separately.", + "Alternative 1: webform at whitepagesprivacy.zendesk.com/hc/en-us/requests/new (same ~2-day handling; good fallback if the mailbox bounces).", + "Alternative 2 (only with an operator on the phone): the automated tool at whitepages.com/suppression_requests -- paste the listing URL, provide a phone number, answer the automated voice call and enter the 4-digit code. Fastest (est 1 day) but NOT autonomous.", + "Opt-out requests may take up to 15 days; verify with a re-scan before recording confirmed_removed." + ], + "notes": "Email/webform lane verified 2026-07-01: privacyrequest@whitepages.com or the Zendesk form replace the phone-callback tool ('if you would prefer not to provide a phone number... submit a request to a customer service agent'). Authorized agents: written signed permission required. General privacy contact: support@whitepages.com.", + "quirks": [ + "The automated suppression tool (whitepages.com/suppression_requests) REQUIRES a phone number + automated voice call with a 4-digit code + agreeing to their ToS -- that lane is phone_callback/T2. The email/webform lane exists precisely to avoid it; prefer email for autonomy.", + "Deletion exceptions they state: public records (property, court), fraud-prevention data, active-subscription data, legal holds. 'Hidden from search' vs 'deleted' distinction applies -- their opt-out removes the listing; the CCPA deletion covers non-public account data.", + "CCPA opt-out requests: up to 15 days to process. Right-to-know/access requests also via privacyrequest@whitepages.com or the Zendesk form." + ], + "est_processing_days": 15, + "reappearance_risk": "medium" + }, + "last_verified": "2026-07-01", + "source": "BADBOOL" +} diff --git a/optional-skills/security/unbroker/references/legal/ccpa.md b/optional-skills/security/unbroker/references/legal/ccpa.md new file mode 100644 index 00000000000..475a7ce012e --- /dev/null +++ b/optional-skills/security/unbroker/references/legal/ccpa.md @@ -0,0 +1,27 @@ +# CCPA / CPRA (California) + +Use for California residents (`residency_jurisdiction` starts with `US-CA`) and, in practice, many US +brokers that honor CCPA-style requests nationwide. + +## Rights invoked + +- **Delete** personal information (Cal. Civ. Code 1798.105). +- **Opt out** of sale/sharing of personal information (1798.120). + +## Request content + +Render with `legal.render_request("ccpa", broker, fields)` -> `templates/emails/ccpa-deletion.txt`. +Include only: full legal name, the contact email for correspondence, and the confirmed listing +URL(s). Do **not** include SSN or government IDs. + +## Authorized agent + +When acting for another consenting subject, use `render_request("ccpa_agent", ...)` +(`templates/emails/ccpa-authorized-agent.txt`) and attach the authorization artifact recorded in the +dossier (`consent.authorization_artifact`). The broker may separately verify the consumer's identity. + +## Notes + +- Brokers must respond within 45 days (extendable). Track as `awaiting_processing` until confirmed. +- "Hidden from free search" is not deletion - verify the record is actually gone before + `confirmed_removed`. diff --git a/optional-skills/security/unbroker/references/legal/drop.md b/optional-skills/security/unbroker/references/legal/drop.md new file mode 100644 index 00000000000..3b96333fc8e --- /dev/null +++ b/optional-skills/security/unbroker/references/legal/drop.md @@ -0,0 +1,34 @@ +# California DROP portal (highest-leverage lever) + +The California **Delete Request and Opt-out Platform** (`privacy.ca.gov/drop`) lets a California +resident demand deletion from **every registered data broker** with a single verified request, for +free. DROP is **live** (as of 2026); registered brokers must begin processing requests on +**2026-08-01**. The registered universe is the **California Data Broker Registry** (~545 brokers in +2025), which this skill ingests as its own coverage lane (`pdd.py registry`); one DROP request covers +all of them, which is how this skill reaches (and exceeds) the breadth of commercial services. + +## When to use + +For any subject with `residency_jurisdiction` starting `US-CA`, sequence DROP **first**: `pdd.py next` +surfaces a single `drop_submit` action covering the whole registry. Then handle the individual +people-search sites (which are also worked directly because they hold free, indexed listings). After +filing, run `pdd.py drop --filed` so the loop stops re-surfacing it. For non-CA subjects +DROP does not apply; cover the registry brokers with targeted CCPA/GDPR deletion emails +(`pdd.py registry --search`, then `pdd.py send-email`). + +## Flow (agent-assisted, mostly human verification) + +1. The operator creates/verifies a DROP account (identity verification is required by the state; this + is a human step - `human_task_queued`). +2. Submit one deletion request covering all registered brokers. +3. Record a single ledger case `case__drop` to track it; mark `submitted` -> + `awaiting_processing`. Registered brokers must process deletions on the state's schedule. +4. After the DROP cycle, re-scan the people-search long tail and only act on sites still showing data. + +## Caveats + +- DROP covers **registered data brokers**, not every people-search site. Keep doing the individual + opt-outs for non-registered sites. +- Identity verification means parts of this cannot (and should not) be fully automated. +- FCRA-regulated brokers (flagged in the registry, `optout.fcra`) hold consumer-report data with + separate rules; deletion may be limited and a dispute or security-freeze may apply instead. diff --git a/optional-skills/security/unbroker/references/legal/gdpr.md b/optional-skills/security/unbroker/references/legal/gdpr.md new file mode 100644 index 00000000000..d00d5469343 --- /dev/null +++ b/optional-skills/security/unbroker/references/legal/gdpr.md @@ -0,0 +1,20 @@ +# GDPR / UK-GDPR (roadmap - Phase 3) + +For EU/UK subjects. Not part of the P0 US-first scope; templates and routing land in Phase 3. + +## Rights invoked + +- **Erasure** ("right to be forgotten") - Article 17. +- **Object** to processing - Article 21. + +## Request content + +Render with `legal.render_request("gdpr", broker, fields)` -> +`templates/emails/gdpr-erasure.txt`. Address the controller's privacy/DPO contact. Include the data +subject's name, the contact email, and the listing URL(s); cite Article 17. + +## Notes + +- Controllers must respond within one month (Article 12(3)). +- EU-specific brokers and portals (e.g. Acxiom's EU consumer portals) are added in Phase 3 with + `jurisdictions: ["EU"]` records and residency-aware routing. diff --git a/optional-skills/security/unbroker/references/methods.md b/optional-skills/security/unbroker/references/methods.md new file mode 100644 index 00000000000..6e25f0a7bb6 --- /dev/null +++ b/optional-skills/security/unbroker/references/methods.md @@ -0,0 +1,235 @@ +# Opt-out method playbooks + +How the agent executes each broker `optout.method` using native Hermes tools. Always obey the +**verify-before-disclose** rule: confirm a real listing exists, then submit only the fields the broker +requires (`pdd.py plan` lists them per broker). + +**Autonomy:** `pdd.py next ` sequences all of this - it decides which method applies, orders +parents first, and routes human-only work to the digest. In `autonomy=full` (default), execute its +actions without pausing per submission; the consent recorded at intake is the authorization. These +playbooks are the HOW for each action type. + +## Scan ladder (all methods) + +Confirm exposure before acting, cheapest first. Run **every** `search_vectors` entry from `pdd.py +plan` (each name x location, phone, email, and address the broker's `search.by` supports) - different +vectors surface different listings for the same person; dedupe found URLs. + +1. `web_extract` on the broker `search.url` (fast HTML -> markdown). Look for `search.match_signal`. + Build per-vector URLs from `search.url_patterns` and heed `search.url_format_quirks` (see below). +1b. **`site:` search-engine probe (cheap, do it early and in parallel).** `web_search` with + `site: "First Last"` (add a city/ZIP or a unique phone/address to cut namesake + noise) often returns the **exact profile-slug URL** in one shot - which both confirms the listing + exists AND hands you the opaque `/find/person/` or `/p/` URL you'd otherwise have to + derive. Two big wins seen in the field: (a) it disambiguates namesakes fast - the SERP snippet + shows age/city so you can tell the subject from a same-name relative before fetching anything; and + (b) a broad `"First Last" ` search (no `site:`) surfaces **brokers not yet in + your DB** (e.g. information.com, peoplefinders.com) - record those as bonus exposures. Note: empty + `site:` results are INCONCLUSIVE (many broker pages aren't indexed / are `noindex`), not `not_found`. +2. If the page is JS-rendered or returns nothing useful, `browser_navigate` + `browser_snapshot` + (and `browser_type`/`browser_click` to run the site's search box). +3. If blocked by stealth/Cloudflare, use the `scrapling` skill via `terminal`. **If the broker record + has `search.antibot` set (e.g. `datadome`), results are behind a device-check CAPTCHA**: a + cloud/stealth browser (Browserbase) or `scrapling` may get through; if none is available, do **not** + burn attempts - `pdd.py record blocked` and move on (a re-scan with a stealth + backend can pick it up later). +3b. **Operator-browser path (the reliable unblock for anti-bot sites).** Cloudflare/DataDome key on + datacenter IPs + headless fingerprints, so `web_extract`, the proxyless agent browser, and even a + cloud browser often fail - but the **operator's own everyday browser (residential IP, real + fingerprint) sails straight through**. For any `blocked` site, hand the operator a paste-ready + search URL (built from `search.url_patterns`), give them the identity anchors to judge by (current + + prior addresses, age, a distinguishing detail) and the namesake/relative watch-list, and ask for + the verdict or a screenshot (the agent can read screenshots). This is a **first-class scan path, not + a fallback** - treat the operator's live check as authoritative and record the real verdict + (`found` / `not_found` / `indirect_exposure`), citing `scanned_via: operator_browser`. Same for + opt-out forms the agent's browser can't reach: guide the operator field-by-field (least-disclosure), + pausing before submit. (This is exactly why the same trick clears email-verification links the agent + can't open - see the Verification loop.) +4. Capture evidence: save listing URLs and a `browser` screenshot into the subject's `evidence/` dir, + then `pdd.py record found --found true --evidence '{"listing_urls":[...]}'`. + +If a listing genuinely does not exist: `pdd.py record not_found` and move on. + +### A 404 (or empty body) is INCONCLUSIVE, not "not_found" + +A constructed search URL that 404s almost always means the **URL pattern is wrong**, not that the +person is absent. Never record `not_found` off a 404. Instead: + 1. Re-check the broker's `search.url_patterns` / `url_format_quirks` and rebuild the URL. + 2. Fall back to the **on-site search box**: `browser_navigate` to the search page, `browser_type` + the raw query, `browser_click` Search, then read the **canonical result URL** the site lands on. + 3. Only after the site's own search returns an empty result set do you record `not_found`. + 4. If a pattern was wrong, fix it in `references/brokers/.json` (`url_patterns` + + `url_format_quirks`) so the next run is correct - see the rule below. + +### Log URL/format quirks for every site you scrape + +Whenever you discover how a broker's URLs are actually shaped (path layout, hyphen-vs-slash joins, +whether ZIP is required, abbreviation handling, query-param search, anti-bot gating), record it in +that broker's `references/brokers/.json` under `search.url_patterns` (the templates) and +`search.url_format_quirks` (the gotchas, including which forms 404). Bump `last_verified`. This makes +the deterministic URL path reliable across runs and subjects instead of rediscovered each time. If the +opt-out form's real requirements differ from the record (extra required fields, a CAPTCHA, an account), +fix `optout.requires` / `optout.inputs` / `optout.tier` too - those drive tier selection and +least-disclosure. Log opt-out mechanics gotchas (a broker that needs a profile URL but doesn't expose +one for the subject, an email-only fallback, an authorized-agent toggle) in `optout.quirks` - the +planner surfaces these as `optout_quirks` per broker. Example: Radaris sometimes shows the subject only +as a static address-table row with no "View Profile" link, so `/control-privacy` (which needs a profile +URL) can't be used - fall back to `optout.email` rather than submitting a namesake's URL. + +### Distinguish the subject from namesakes and relatives + +People-search sites are dense with namesakes and family clusters. Before recording `found`, confirm the +record is the **subject themselves** (corroborate via DOB, a known current/prior address, or the +identifier you searched). Two non-removable patterns to record as evidence but NOT as the subject's own +listing: + - **Namesake:** same name, different person (different DOB/location with no overlap). Not the subject. + - **Relative record:** the listing is about a *different* person (a relative) and merely *names* the + subject in a "Family" field, or carries the subject's email/phone as a secondary datum. This is a + third party's record - the consent gate correctly blocks acting on it. See "Indirect exposure" in + the web_form section for what the subject *can* still request. + +## web_form + +1. `browser_navigate` to `optout.url`; `browser_snapshot` to read the form. +2. Fill only the planned `disclosure_fields` with `browser_type`/`browser_click`; for `profile_url`, + paste the confirmed listing URL from evidence. +3. Submit; `browser_snapshot` to confirm the success state; screenshot to `evidence/`. +4. `pdd.py record submitted --disclosed --disclosed --channel web_form`. +5. If the broker requires email verification, follow **Verification loop** below. + +### Indirect exposure (named as a relative / your email on someone else's record) + +You asked the right question: if a broker lists a *relative* and names you in their "Family" field, or +shows **your** email/phone on **their** record, that IS personal information about you - even though the +record's primary subject is a third party. Resolve it in two distinct lanes: + +- **The self-service opt-out form does NOT cover this.** That form removes a record whose *primary + subject* is you. It has no notion of "scrub my identifiers from this other person's record," and + submitting it with the relative's address to force a match would be (a) disclosing data the listing + doesn't tie to you and (b) acting on a third party's record. Don't. The consent gate exists to stop + exactly that. +- **What you CAN do - a targeted "delete my personal information" request (CCPA 1798.105 / GDPR Art.17).** + These rights attach to *your* personal information *wherever the business holds it*, including as a + data point on another person's profile. So the subject may email the broker's privacy address and + request suppression of **their own specific identifiers** (this email address, this phone number, my + name in family/relative associations), citing the relative listings as the locations. This is a + narrower request than a full opt-out and does not require the relative's consent - you are only asking + them to delete data about *you*. Use `render-email` with the `ccpa`/`gdpr` template, list only the + subject's own identifiers + the URLs where they appear, and record it as a normal `submitted` → + `awaiting_processing` email case. Verify by re-scanning those identifier vectors (email/phone) after + the statutory window - `confirmed_removed` only when the subject's identifier no longer appears. +- **Caveat:** the broker may decline to alter a third party's record beyond removing your specific + identifiers, and "your name in a family graph" can be derived from public records they'll re-list. + Note residual exposure in the report rather than marking a clean removal. (Operational guidance, not + legal advice.) + +## email + +`pdd.py send-email --listing [--kind ccpa|gdpr|ccpa_indirect]` always does +the deterministic parts (recipient locked to an address the broker record declares, refusing anything +else; `--listing` mandatory; records `submitted`, logs disclosure, stamps `next_recheck_at`). How it +actually sends depends on `email_mode`: + +1. **browser mode (no password, autonomous):** the command returns a recipient-locked `compose` + payload (`to`/`subject`/`body`). Compose a NEW message in the operator's **logged-in webmail** via + `browser_*` (paste `compose.body` exactly, disclosing nothing beyond it) and send. No credentials + stored. Requires the inbox signed in in the browser Hermes uses. +2. **programmatic mode (SMTP creds):** the command SMTP-sends it directly, no human. +3. **draft_only fallback:** `pdd.py render-email --listing `; a digest entry + tells the operator to send it, and the agent records `submitted --channel email` afterward. + +Then follow the **Verification loop** if the broker emails a confirmation link. + +## Verification loop (email_verification brokers) + +- **browser mode (autonomous, no password):** open the broker's confirmation email in the operator's + webmail (`browser_*`), then `pdd.py verify-link --text ''` returns + the anti-phishing-scored link. `browser_navigate` it **in the same browser** (several brokers, e.g. + PeopleConnect, bind the session to the browser that opens the link), finish the flow, record + `awaiting_processing`. +- **programmatic mode (IMAP):** `pdd.py poll-verification ` polls IMAP for every in-flight + case, extracts the link (anti-phishing scored: only opt-out-looking links on the broker's own + domains), and auto-advances `submitted → verification_pending`. Then `browser_navigate` the link in + the agent's own browser, finish the flow, record `awaiting_processing`. +- **draft_only:** the digest tells the operator to click the link in the subject's inbox; the agent + records `awaiting_processing` on their word. +- Either way, the due queue (`pdd.py due`) brings the case back after the broker's processing window + for the verifying re-scan; only that re-scan justifies `confirmed_removed`. + +## phone_callback (e.g. Whitepages) + +Submit the web form, then the site places an automated call with a numeric code. If the operator is +available to read the code, capture it and complete the form (T2). Otherwise queue a human task. + +## phone (voice menu) / fax / mail / gov_id -> human task (T3) + +Do **not** attempt to automate. Create a `todo` task and `pdd.py record +human_task_queued` with exact instructions and an explicit **withhold** list (never SSN; never a +driver's-license number unless the subject chooses to and crosses out the ID number). Capture the +confirmation reference back into the ledger when the operator completes it. + +## captcha + +**Default: soft/managed CAPTCHAs clear automatically.** The recommended baseline backend is the +Browserbase cloud browser (`setup --auto` selects it when `BROWSERBASE_API_KEY` is set). Being a +real browser on a residential IP, it passes managed challenges - Cloudflare Turnstile, hCaptcha / +reCAPTCHA checkbox - as normal operation, so those brokers stay T1 and you just proceed. This is +**not** CAPTCHA solving: no solver service, no fingerprint spoofing. + +Only a **hard** challenge the browser genuinely can't pass (interactive image grids, behavioral +scoring that flags the session) becomes a fallback: `record ... blocked` and requeue it for the +stealth/operator-browser pass (`methods.md` → scan ladder 3b - the operator's own residential +browser is the reliable unblock). Without a cloud browser configured, soft-CAPTCHA brokers drop to +T2 and become human tasks. **Never use a third-party CAPTCHA-defeating service.** + +## Ownership clusters - DO PARENTS FIRST (playbooks live in the broker records) + +Many brokers are resold shells of a few parents, so **one parent removal clears a whole cluster of +children** (see `owns` in each record). In Phase 2 you MUST work the cluster **parents first**, then +the standalone listings - doing a child before its parent wastes a submission the parent would have +covered. `pdd.py plan --batch` **orders the `found` group parents-first** and emits a +`parent_playbook` whose `steps` come verbatim from each record's **`optout.playbook`** - the single +source of truth, field-verified, updated as live runs discover mechanics. What follows is the +operating doctrine; the exact steps are in `references/brokers/.json`. + +**Deletion beats suppression, email lanes beat forms.** Each parent record carries a structured +`optout.deletion` lane (`via: in_flow | email | email_followup`, plus the privacy address). The +autopilot routes accordingly: + +- **`in_flow`** (PeopleConnect): the deletion control lives INSIDE the web flow - complete the flow + and use the **Right to Delete / DELETE MY USER DATA** control, never just the suppression step. +- **`via: email`** (Whitepages): the fully-autonomous lane - `send-email` the request (residency-picked + kind: CCPA for US-CA, GDPR for EU/UK, generic otherwise), then `poll-verification` for their reply + and answer identity questions with least-disclosure. This is also the **rescue lane**: any broker + whose form demands a phone-callback/gov-ID/account but that declares a deletion email gets routed + here instead of the human digest. +- **`email_followup`** (BeenVerified, Spokeo): the opt-out form is the fast primary (it clears the + listing), and the playbook then sends a right-to-delete email for full erasure beyond suppression. + +Verified parent facts (live-checked 2026-07-01; details + steps in the records): + +- **Intelius/PeopleConnect** (~15+ sites in one flow): portal entry asks only email + consent → + verify link is **session-bound to the browser that opens it** → guided-mode → **DELETE MY USER + DATA** at the bottom (suppression alone re-lists). Fallback `privacy@peopleconnect.us` - their own + published metrics: 33.5k deletion requests, median response < 1 day. +- **Whitepages**: `privacyrequest@whitepages.com` (or the Zendesk form) handles removal + CCPA + deletion **without the phone-callback tool** - that phone call is only required by the automated + tool. One removal also drops "all known connected listings". ≤15 days; check 411.com + Premium. +- **BeenVerified**: opt-out tool (footer "Do Not Sell" link → `/svc/optout/search/optouts`) + email + verification; one opt-out per email address. Then `privacy@beenverified.com` deletion follow-up - + controller is The Lifetime Value Co., so name their sister properties (NeighborWho, Ownerly, + NumberGuru, Bumper) in the same request, and verify each separately. +- **Spokeo**: form takes ONE listing URL at a time and **each listing must be opted out + individually** - collect every listing URL from all search vectors first, then submit one opt-out + per URL. 24-48h processing. `privacy@spokeo.com` for full deletion beyond free-search suppression. + +After each parent removal is confirmed, **re-scan its children** before submitting anything for them - +usually they drop out and need no separate opt-out. + +### Any other parent +A parent without a hand-verified `optout.playbook` gets synthesised steps from its structured record +(URL/email, `requires` flags, deletion lane, notes/quirks). Follow those, and **write what you learn +back into `references/brokers/.json`** (`optout.playbook`, `optout.deletion`, `quirks`, +`last_verified`) so the next run is exact - that file, not this one, is where per-broker knowledge +accrues. + diff --git a/optional-skills/security/unbroker/references/state-machine.md b/optional-skills/security/unbroker/references/state-machine.md new file mode 100644 index 00000000000..feae5a3efbe --- /dev/null +++ b/optional-skills/security/unbroker/references/state-machine.md @@ -0,0 +1,43 @@ +# Case state machine + +One case = one (subject x broker). `pdd.py record` validates every transition against this table and +appends it to `audit.jsonl`. Authoritative definition lives in `scripts/ledger.py`. + +## States + +| State | Meaning | +|---|---| +| `new` | Case created, nothing done | +| `searching` | Scan in progress | +| `not_found` | Subject not listed (will be re-checked next cycle) | +| `found` | Listing confirmed; action needed | +| `indirect_exposure` | Subject's PII (email/phone/name) appears on a **third party's** record (e.g. named in a relative's "Family" field). Not removable via self-service opt-out; needs a targeted CCPA/GDPR delete-my-PII request | +| `action_selected` | Tier/method chosen | +| `submitted` | Opt-out submitted | +| `verification_pending` | Awaiting email/callback verification | +| `awaiting_processing` | Submitted, no verification needed; broker processing | +| `confirmed_removed` | Verified gone | +| `reappeared` | Was removed, now listed again | +| `human_task_queued` | Needs an operator step (captcha/ID/phone/fax/mail) | +| `blocked` | Broker dead / mechanics broken -> flag for DB re-verification | + +## Allowed transitions + +``` +new -> searching | found | not_found | indirect_exposure | blocked +searching -> not_found | found | indirect_exposure | blocked +not_found -> searching | found | indirect_exposure | blocked +found -> action_selected | submitted | human_task_queued | indirect_exposure | blocked +indirect_exposure -> submitted | human_task_queued | not_found | found | blocked +action_selected -> submitted | human_task_queued | blocked +submitted -> verification_pending | awaiting_processing | human_task_queued | blocked +verification_pending -> confirmed_removed | human_task_queued | blocked +awaiting_processing -> confirmed_removed | human_task_queued | blocked +confirmed_removed -> reappeared | confirmed_removed (recheck refreshes the date) +reappeared -> found | indirect_exposure +human_task_queued -> found | indirect_exposure | action_selected | submitted | verification_pending + | awaiting_processing | confirmed_removed | blocked +blocked -> searching | found | not_found | indirect_exposure | action_selected +``` + +A transition to the same state is always allowed (idempotent field updates). diff --git a/optional-skills/security/unbroker/scripts/autopilot.py b/optional-skills/security/unbroker/scripts/autopilot.py new file mode 100644 index 00000000000..0c900818a8d --- /dev/null +++ b/optional-skills/security/unbroker/scripts/autopilot.py @@ -0,0 +1,396 @@ +"""Autonomous action queue: what should the agent do RIGHT NOW for this subject? + +`next_actions` turns (dossier, broker DB, config, ledger) into an ordered queue of +concrete agent actions plus a human digest. The agent's whole run becomes a loop: + + while True: + q = pdd.py next + if not q["actions"]: break + execute each action, record outcomes + present q["human_digest"] once; schedule cron at q["next_wake_at"] + +Policy (cfg["autonomy"]): + full - intake consent is standing authorization; T0-T2 agent actions are + executed without pausing. Humans appear only in the digest. + assisted - same queue, but every submission action carries confirm_first=True. + +The queue is deterministic and side-effect free: it never mutates the ledger, it +only reads. Executing + recording stays with the agent (and the record command). +""" +from __future__ import annotations + +import datetime as _dt +import os +from pathlib import Path + +import brokers as brokers_mod +import emailer +import ledger as ledger_mod +import paths +import registry +import tiers + +CACHE_STALE_DAYS = 7 # refresh the live broker list after this +FANOUT_THRESHOLD = 8 # above this many unscanned brokers, use delegate_task fan-out + +# States with nothing left to do (absent a due recheck). +_TERMINAL = {"not_found", "confirmed_removed"} +_IN_FLIGHT = {"submitted", "verification_pending", "awaiting_processing"} + + +def cache_age_days(now: float | None = None) -> float | None: + """Age of the live BADBOOL cache in days, or None if never pulled.""" + p: Path = paths.brokers_cache_path() + if not p.exists(): + return None + now = now if now is not None else _dt.datetime.now().timestamp() + return max(0.0, (now - p.stat().st_mtime) / 86400.0) + + +def _now_iso() -> str: + return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def _min_future_recheck(ledger: dict, at: str) -> str | None: + future = [c.get("next_recheck_at") for c in ledger.values() + if c.get("next_recheck_at") and c["next_recheck_at"] > at] + return min(future) if future else None + + +def _digest(broker_row: dict, reason: str, steps: list[str], prep: list[str] | None = None) -> dict: + return { + "broker_id": broker_row.get("broker_id"), + "broker_name": broker_row.get("broker_name"), + "reason": reason, + "agent_prep": prep or [], # commands the agent runs BEFORE handing this to the human + "steps": steps, # what the human actually does + "withhold": ["SSN", "full driver's-license / passport numbers"], + } + + +def request_kind(dossier: dict, allowed: list[str] | None = None) -> str: + """Pick the honest legal basis for a deletion request from the subject's residency. + + ccpa only for California residents, gdpr only for EU/UK residents, generic otherwise. + `allowed` (from the broker's deletion.kinds) can restrict DOWN to generic but never + upgrades to a law the subject can't truthfully claim. + """ + res = (dossier.get("residency_jurisdiction") or "US").upper() + if res.startswith("US-CA"): + kind = "ccpa" + elif res.startswith(("EU", "UK", "GB")): + kind = "gdpr" + else: + kind = "generic" + if allowed and kind not in allowed and "generic" in allowed: + kind = "generic" + return kind + + +_HUMAN_GATES = ("gov_id", "fax", "mail", "phone_voice", "phone_callback", "account") + + +def _email_lane(row: dict) -> tuple[str | None, str]: + """(address, why) for the autonomous email lane of this broker, if one exists. + + Lane rules: + 1. the broker's primary opt-out method IS email; + 2. the record marks its deletion lane email-preferred (deletion.via == "email"); + 3. RESCUE: the primary flow is human-gated (gov ID / fax / phone / account) but a + right-to-delete email exists - the email lane restores full autonomy (this is the + verified Whitepages pattern: privacyrequest@ accepts requests precisely so people + don't have to do the phone-callback tool). + """ + deletion = row.get("deletion") or {} + req = row.get("optout_requires") or {} + if row.get("method") == "email": + addr = row.get("optout_email") or deletion.get("email") + return (addr, "primary opt-out method is email") if addr else (None, "") + if deletion.get("via") == "email" and deletion.get("email"): + return deletion["email"], "record prefers the right-to-delete email lane" + if (row.get("tier") == "T3" or any(req.get(k) for k in _HUMAN_GATES)) and deletion.get("email"): + return deletion["email"], "rescue: primary flow is human-gated; deletion email restores autonomy" + return None, "" + + +def _optout_action(row: dict, playbook: dict[str, dict], subject_id: str, dossier: dict, + email_mode: str, smtp_ok: bool, confirm_first: bool) -> tuple[dict | None, dict | None]: + """Map one actionable `found` row to (agent_action, human_digest_entry). + + Routing order maximizes autonomy: (1) the email lane (primary email method, preferred + right-to-delete email, or rescue from a human-gated form) beats everything when SMTP is + up; (2) genuinely human-only flows go to the digest; (3) web forms are driven with the + record's own field-verified playbook steps. + """ + bid = row["broker_id"] + req = row.get("optout_requires") or {} + tier = row.get("tier") + deletion = row.get("deletion") or {} + + # 1) The autonomous EMAIL LANE (right-to-delete by email + confirm the reply). + # Autonomous when SMTP is configured (programmatic/alias) OR in browser mode (agent sends via + # the operator's logged-in webmail; no password needed). + email_addr, lane_why = _email_lane(row) + can_email = (email_mode in ("programmatic", "alias") and smtp_ok) or email_mode == "browser" + if email_addr and can_email: + kind = request_kind(dossier, deletion.get("kinds")) + via = "browser" if email_mode == "browser" else "smtp" + then = ("send-email records it + returns a recipient-locked payload; compose and send it in " + "the operator's webmail via browser_*, then `verify-link` on the reply and open the link" + if via == "browser" else + "state auto-records as submitted; poll-verification picks up their verification reply, " + "open its link, then record") + return { + "type": "optout_email_send", + "broker_id": bid, "broker_name": row.get("broker_name"), "tier": tier, + "confirm_first": confirm_first, "send_via": via, + "to": email_addr, "kind": kind, "why": lane_why, + "command": f"python3 scripts/pdd.py send-email {subject_id} {bid} --kind {kind} " + f"--to {email_addr} --listing ", + "then": then, + }, None + if row.get("method") == "email": + return None, _digest(row, "email opt-out (draft mode: a human must hit send)", + ["Send the rendered draft from your own mail client", + f"Then: python3 scripts/pdd.py record {subject_id} {bid} submitted " + f"--disclosed contact_email --channel email"], + prep=[f"python3 scripts/pdd.py render-email {subject_id} {bid} --listing "]) + + # 2) Genuinely human-only work goes to the digest (no email lane could rescue it). + if tier == "T3": + return None, _digest(row, "human-only opt-out (gov ID / fax / mail / voice phone)", + [f"Follow the broker's process at {row.get('optout_url') or row.get('optout_email')}", + "Provide only the fields the listing already shows; cross out ID numbers on any document"]) + if req.get("phone_callback"): + return None, _digest(row, "phone-callback verification (operator must be on the phone)", + [f"Open {row.get('optout_url')} and submit with only the planned fields", + "Answer the automated call and enter the 4-digit code to finish"], + prep=[f"python3 scripts/pdd.py plan {subject_id} --batch # confirm fields first"]) + if req.get("account"): + return None, _digest(row, "requires creating/holding an account with the broker", + [f"Create/log in at {row.get('optout_url')} and submit the opt-out", + "Use the subject's contact email; no extra PII beyond the planned fields"]) + + # 3) web_form: drive the browser with the record's own playbook steps. + steps = (playbook.get(bid) or {}).get("steps") or list(row.get("optout_playbook") or []) \ + or tiers.synthesize_steps(row) + action = { + "type": "optout_web_form", + "broker_id": bid, "broker_name": row.get("broker_name"), "tier": tier, + "confirm_first": confirm_first, + "optout_url": row.get("optout_url"), + "clears_children": row.get("clears_children") or [], + "steps": steps, + "after": f"python3 scripts/pdd.py record {subject_id} {bid} submitted " + f"--disclosed ... --channel web_form", + } + if deletion: + action["prefer_deletion"] = ("this record has a right-to-delete lane -- complete the DELETION " + "flow, not just suppression" + (f" ({deletion.get('notes')})" + if deletion.get("notes") else "")) + if req.get("captcha"): + action["note"] = ("CAPTCHA-gated: attempt with the configured browser backend once; if it " + "does not clear, record blocked (do NOT retry-loop or bypass)") + return action, None + + +def next_actions(dossier: dict, brokers_list: list[dict], cfg: dict, + ledger: dict | None = None, env: dict | None = None) -> dict: + env = os.environ if env is None else env + ledger = ledger or {} + subject_id = dossier.get("subject_id", "") + autonomy = cfg.get("autonomy", "full") + confirm_first = autonomy == "assisted" + email_mode = cfg.get("email_mode", "draft_only") + mail = emailer.available(env) + at = _now_iso() + + batch = tiers.batch_plan(dossier, brokers_list, cfg, ledger, + browser_clears_captcha=cfg.get("browser_backend") == "browserbase" + or bool(env.get("BROWSERBASE_API_KEY"))) + groups = batch["groups"] + playbook = {p["broker_id"]: p for p in batch.get("parent_playbook") or []} + by_id = {b.get("id"): b for b in brokers_list} + + actions: list[dict] = [] + digest: list[dict] = [] + + # 0) keep the broker DB fresh (autonomously) + age = cache_age_days() + if age is None or age > CACHE_STALE_DAYS: + actions.append({ + "type": "refresh_brokers", + "why": "live broker cache missing" if age is None else f"cache is {age:.0f} days old", + "command": "python3 scripts/pdd.py refresh-brokers", + }) + + # 0b) DROP one-shot: for a CA resident, ONE request deletes from every registered + # broker (the whole CA Data Broker Registry) -- the highest-leverage removal there is. + registry_recs = brokers_mod.load_registry_cache() + residency = (dossier.get("residency_jurisdiction") or "US").upper() + drop_filed = bool((dossier.get("preferences") or {}).get("drop_filed_at")) + if registry_recs and residency.startswith("US-CA") and not drop_filed: + actions.append({ + "type": "drop_submit", + "one_shot": True, + "registry_count": len(registry_recs), + "url": registry.DROP_URL, + "command": f"python3 scripts/pdd.py drop {subject_id}", + "why": f"CA resident: one DROP request deletes from all {len(registry_recs)} registered " + "data brokers at once (superset of what commercial services cover).", + "after": f"python3 scripts/pdd.py drop {subject_id} --filed", + }) + + # 1) Phase 1 crawl: everything unscanned (read-only, parallel-safe) + unscanned = groups.get("unscanned") or [] + if unscanned: + ids = [r["broker_id"] for r in unscanned] + if len(ids) > FANOUT_THRESHOLD: + actions.append({ + "type": "fanout_scan", + "broker_ids": ids, + "command": f"python3 scripts/pdd.py fanout {subject_id}", + "how": "spawn ONE delegate_task subagent per batch IN PARALLEL with each batch's brief; " + "parent re-verifies key `found` claims before trusting them", + }) + else: + actions.append({ + "type": "scan_inline", + "broker_ids": ids, + "command": f"python3 scripts/pdd.py plan {subject_id}", + "how": "run every search_vector per broker via the methods.md ladder " + "(web_extract -> site: probe -> browser), record a verdict per broker", + }) + + # 2) in-flight email verifications: poll the inbox (or hand to the human in draft mode) + for st in ("submitted", "verification_pending"): + for bid, case in sorted(ledger.items()): + if case.get("state") != st: + continue + broker = by_id.get(bid) or {} + if not ((broker.get("optout") or {}).get("requires") or {}).get("email_verification"): + continue + if mail["imap"]: + actions.append({ + "type": "poll_verification", "via": "imap", + "broker_id": bid, + "command": f"python3 scripts/pdd.py poll-verification {subject_id} --broker {bid}", + "then": "browser_navigate the returned link IN THE SAME AGENT BROWSER (sessions are " + f"browser-bound), complete the flow, then record: awaiting_processing", + }) + elif email_mode == "browser": + actions.append({ + "type": "poll_verification", "via": "browser", "broker_id": bid, + "how": "open the broker's confirmation email in the operator's logged-in webmail " + f"(browser_*), then `python3 scripts/pdd.py verify-link {subject_id} {bid} " + "--text ''` to score the link, browser_navigate it in the SAME " + "browser, then record awaiting_processing", + }) + else: + digest.append(_digest( + {"broker_id": bid, "broker_name": (broker.get("name") or bid)}, + "verification email must be opened by a human (draft mode, no inbox access)", + ["Open the broker's verification email in the subject's inbox and click the link", + f"Then: python3 scripts/pdd.py record {subject_id} {bid} awaiting_processing"])) + + # 3) due rechecks: processing windows elapsed / reappearance sweeps + for case in ledger_mod.due(subject_id, at=at, ledger=ledger): + bid = case.get("broker_id") + st = case.get("state") + if st in ("awaiting_processing", "confirmed_removed"): + actions.append({ + "type": "verify_removal", + "broker_id": bid, + "why": "processing window elapsed" if st == "awaiting_processing" else "periodic reappearance re-scan", + "how": "re-run this broker's search_vectors; if gone record confirmed_removed; " + "if still listed record reappeared and requeue the opt-out", + }) + elif st in ("submitted", "verification_pending") and not mail["imap"]: + pass # already covered by the digest entry above + + # 4) Phase 2 opt-outs: parents first (batch_plan already ordered them) + for row in groups.get("found") or []: + action, task = _optout_action(row, playbook, subject_id, dossier, + email_mode, mail["smtp"], confirm_first) + if action: + actions.append(action) + if task: + digest.append(task) + + # 5) indirect exposure: targeted delete-my-PII requests + for row in groups.get("indirect_exposure") or []: + bid = row["broker_id"] + if (email_mode in ("programmatic", "alias") and mail["smtp"]) or email_mode == "browser": + actions.append({ + "type": "indirect_email_send", + "broker_id": bid, "confirm_first": confirm_first, + "send_via": "browser" if email_mode == "browser" else "smtp", + "command": f"python3 scripts/pdd.py send-email {subject_id} {bid} --kind ccpa_indirect " + f"--listing ", + }) + else: + digest.append(_digest(row, "indirect-exposure request (draft mode: a human must hit send)", + ["Send the rendered ccpa_indirect draft", + f"Then: python3 scripts/pdd.py record {subject_id} {bid} submitted " + f"--disclosed contact_email --channel email"], + prep=[f"python3 scripts/pdd.py render-email {subject_id} {bid} " + f"--kind ccpa_indirect --listing "])) + + # 6) blocked sites: stealth pass if we have one, else the operator-browser path + blocked = groups.get("blocked") or [] + if blocked: + ids = [r["broker_id"] for r in blocked] + if bool(env.get("BROWSERBASE_API_KEY")): + actions.append({ + "type": "stealth_rescan", + "broker_ids": ids, + "how": "retry these with the cloud/stealth browser backend, then record real verdicts", + }) + else: + for r in blocked: + digest.append(_digest(r, "site blocks automated access (anti-bot); a human browser gets through", + ["Open the paste-ready search URL from `plan` in your everyday browser", + "Report the verdict (or a screenshot) back to the agent", + f"Agent records: python3 scripts/pdd.py record {subject_id} " + f"{r['broker_id']} "])) + + # 7) anything already parked as a human task + for bid, case in sorted(ledger.items()): + if case.get("state") == "human_task_queued": + broker = by_id.get(bid) or {} + digest.append(_digest({"broker_id": bid, "broker_name": broker.get("name") or bid}, + case.get("human_task_reason") or "queued manual step", + ["See `pdd.py tasks` for the exact steps recorded with this case"])) + + # registry coverage summary (breadth beyond the scannable people-search sites) + coverage = None + if registry_recs: + coverage = { + "people_search_sites": len(brokers_list), + "registered_data_brokers": len(registry_recs), + "worked_via": "CA DROP one-shot" if residency.startswith("US-CA") else "targeted CCPA/GDPR email", + } + if not residency.startswith("US-CA"): + coverage["note"] = ("DROP is CA-only; for this subject the registry is covered by targeted " + "CCPA/GDPR deletion emails (`registry --search` then `send-email`), " + "not a single portal request.") + elif drop_filed: + coverage["note"] = "DROP already filed; registry deletions are in the brokers' hands." + + next_wake = _min_future_recheck(ledger, at) + return { + "subject": subject_id, + "autonomy": autonomy, + "phase": batch.get("phase"), + "counts": batch.get("counts"), + "actions": actions, + "human_digest": digest, + "coverage": coverage, + "done_for_now": not actions, + "fully_done": not actions and not digest and not next_wake, + "next_wake_at": next_wake, + "note": ("assisted mode: pause for operator confirmation on every action with confirm_first=true" + if confirm_first else + "full autonomy: recorded intake consent authorizes these submissions; do not pause. " + "Present human_digest ONCE at the end of the run, not per item."), + } diff --git a/optional-skills/security/unbroker/scripts/badbool.py b/optional-skills/security/unbroker/scripts/badbool.py new file mode 100644 index 00000000000..9a415ceef4b --- /dev/null +++ b/optional-skills/security/unbroker/scripts/badbool.py @@ -0,0 +1,177 @@ +"""Pull and parse the Big-Ass Data Broker Opt-Out List (BADBOOL) into broker records. + +BADBOOL (https://github.com/yaelwrites/Big-Ass-Data-Broker-Opt-Out-List) is a +maintained, frequently-updated markdown list. `refresh` fetches it and parses the +"People Search Sites" section into records that merge UNDER the curated DB (curated +records always win). Auto-parsed records carry source="BADBOOL-auto" and +confidence="auto" so the agent treats their URLs as best guesses to verify first. + +`parse()` is pure (markdown in, records out) so it is tested offline; `fetch()` is +the only network call and can be bypassed by passing markdown directly to refresh(). +""" +from __future__ import annotations + +import re +import urllib.request +from pathlib import Path + +import storage + +DEFAULT_URL = ( + "https://raw.githubusercontent.com/yaelwrites/" + "Big-Ass-Data-Broker-Opt-Out-List/master/README.md" +) +USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)" + +# BADBOOL legend symbols. +SYMBOLS = { + "crucial": "\U0001F490", # 💐 + "high": "\u2620", # ☠ + "gov_id": "\U0001F3AB", # 🎫 + "phone": "\U0001F4DE", # 📞 + "payment": "\U0001F4B0", # 💰 +} + +_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)") +_OPTOUT_HINT = re.compile( + r"opt[\- ]?out|optout|removal|remove|suppress|control-privacy|delete", re.I +) +_FIND_HINT = re.compile(r"find|your information|search|look ?up|look for", re.I) + + +def slug(name: str) -> str: + # Drop a trailing .com/.org/.info on the displayed name so "FastPeopleSearch.com" + # matches the curated id "fastpeoplesearch"; keep .net/.id so distinct sites differ. + n = re.sub(r"\.(com|org|info)\b", "", name.strip(), flags=re.I) + return re.sub(r"[^a-z0-9]+", "", n.lower()) + + +def _heading_flags(heading: str) -> tuple[str, dict]: + flags = {key: (sym in heading) for key, sym in SYMBOLS.items()} + name = heading + for sym in SYMBOLS.values(): + name = name.replace(sym, "") + name = name.replace("\ufe0f", "").strip() + return name, flags + + +def _priority(flags: dict) -> str: + if flags["crucial"]: + return "crucial" + if flags["high"]: + return "high" + return "standard" + + +def _pick(links: list[tuple[str, str]], hint: re.Pattern) -> str | None: + for _text, url in links: + if hint.search(url): + return url + for text, url in links: + if hint.search(text): + return url + return None + + +def _clean(text: str) -> str: + return re.sub(r"\s+", " ", text).strip()[:600] + + +def _build(name: str, flags: dict, body: str) -> dict: + links = _LINK_RE.findall(body) + web = [(t, u) for t, u in links if u.lower().startswith("http")] + mailtos = [u[7:] for _t, u in links if u.lower().startswith("mailto:")] + optout_url = _pick(web, _OPTOUT_HINT) + search_url = _pick(web, _FIND_HINT) or (web[0][1] if web else None) + + if flags["phone"]: + method = "phone" + elif optout_url: + method = "web_form" + elif mailtos: + method = "email" + else: + method = "manual" + + return { + "id": slug(name), + "name": name, + "category": "people_search", + "priority": _priority(flags), + "jurisdictions": ["US"], + "search": {"method": "url_pattern", "url": search_url, "fetch": "browser", + "match_signal": "result", "by": ["name", "phone", "address"]}, + "optout": { + "method": method, + "url": optout_url, + "email": mailtos[0] if mailtos else None, + "requires": { + "gov_id": flags["gov_id"], + "phone_voice": flags["phone"], + "payment": flags["payment"], + "email_verification": False, + "captcha": False, + "account": False, + "phone_callback": False, + }, + "inputs": ["full_name", "contact_email"], + "notes": _clean(body), + "links": [{"text": t, "url": u} for t, u in links], + "est_processing_days": 14, # unknown for auto records; drives next_recheck_at + }, + "source": "BADBOOL-auto", + "confidence": "auto", + "last_verified": None, + } + + +def parse(markdown: str) -> list[dict]: + """Parse the 'People Search Sites' section of BADBOOL into broker records.""" + records: list[dict] = [] + in_people = False + heading: str | None = None + body: list[str] = [] + + def flush() -> None: + nonlocal heading, body + if heading is not None: + name, flags = _heading_flags(heading) + if name: + records.append(_build(name, flags, "\n".join(body).strip())) + heading, body = None, [] + + for line in markdown.splitlines(): + if line.startswith("## "): + flush() + in_people = line[3:].strip().lower().startswith("people search") + continue + if not in_people: + continue + if line.startswith("### "): + flush() + heading = line[4:].strip() + elif heading is not None: + body.append(line) + flush() + return records + + +def fetch(url: str = DEFAULT_URL, timeout: int = 30) -> str: + req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) + with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 + return resp.read().decode("utf-8", errors="replace") + + +MIN_EXPECTED = 20 # BADBOOL's People Search section lists ~47; far fewer => upstream reorg, warn + + +def refresh(cache_path: Path, url: str = DEFAULT_URL, markdown: str | None = None) -> dict: + """Fetch (or accept) BADBOOL markdown, parse it, and write the snapshot cache.""" + md = markdown if markdown is not None else fetch(url) + records = parse(md) + storage.write_json(cache_path, records) + out = {"parsed": len(records), "cache_path": str(cache_path), "source_url": url} + if len(records) < MIN_EXPECTED: + out["warning"] = (f"only {len(records)} parsed (expected >{MIN_EXPECTED}); BADBOOL's " + "'People Search Sites' section may have moved/reorganized - check the parser") + return out diff --git a/optional-skills/security/unbroker/scripts/brokers.py b/optional-skills/security/unbroker/scripts/brokers.py new file mode 100644 index 00000000000..4cb63a5c6b0 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/brokers.py @@ -0,0 +1,77 @@ +"""Load and query the broker database (references/brokers/*.json). + +Each broker is one JSON file for clean diffs/PRs. Files beginning with `_` are +ignored (reserved for notes/scratch). +""" +from __future__ import annotations + +import json +from pathlib import Path + +import paths +import storage + +PRIORITY_ORDER = {"crucial": 0, "high": 1, "standard": 2, "long_tail": 3} + + +def _load_curated(directory: Path | None = None) -> list[dict]: + directory = directory or paths.brokers_dir() + out: list[dict] = [] + if not directory.exists(): + return out + for fp in sorted(directory.glob("*.json")): + if fp.name.startswith("_"): + continue + out.append(json.loads(fp.read_text(encoding="utf-8"))) + return out + + +def load_live_cache() -> list[dict]: + """Records pulled from BADBOOL via `refresh-brokers` (empty until refreshed).""" + return storage.read_json(paths.brokers_cache_path(), []) or [] + + +def load_registry_cache() -> list[dict]: + """CA Data Broker Registry records (separate coverage lane; empty until refreshed). + + Kept OUT of load_all() by default: these are not people-search sites to scan, they + are worked via the CA DROP one-shot + CCPA email. Consumers of the scan/plan/fanout + pipeline must not receive them; use this directly for coverage counts and the DROP/ + email lanes. + """ + return storage.read_json(paths.registry_cache_path(), []) or [] + + +def load_all(directory: Path | None = None, include_live: bool = True) -> list[dict]: + """Curated records, with live BADBOOL records merged underneath (curated wins).""" + merged: dict[str, dict] = {b["id"]: b for b in _load_curated(directory)} + if include_live: + for b in load_live_cache(): + bid = b.get("id") + if bid and bid not in merged: + merged[bid] = b + out = list(merged.values()) + out.sort(key=lambda b: (PRIORITY_ORDER.get(b.get("priority", "standard"), 9), b.get("id", ""))) + return out + + +def get(broker_id: str, directory: Path | None = None) -> dict | None: + for b in load_all(directory): + if b.get("id") == broker_id: + return b + return None + + +def by_priority(*levels: str, directory: Path | None = None) -> list[dict]: + wanted = set(levels) if levels else None + return [b for b in load_all(directory) if wanted is None or b.get("priority") in wanted] + + +def clusters(directory: Path | None = None) -> dict[str, list[str]]: + """Map a parent broker id -> child site ids it can clear (force-multipliers).""" + out: dict[str, list[str]] = {} + for b in load_all(directory): + owns = b.get("owns") or [] + if owns: + out[b["id"]] = list(owns) + return out diff --git a/optional-skills/security/unbroker/scripts/config.py b/optional-skills/security/unbroker/scripts/config.py new file mode 100644 index 00000000000..7f53554fe80 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/config.py @@ -0,0 +1,124 @@ +"""Install-wide configuration with easiest-first defaults. + +Everything works zero-config. `setup --auto` (the autonomous path) detects what +this environment can do and picks the MOST AUTONOMOUS valid configuration without +asking anyone; plain `setup` keeps the easiest-first defaults and only upgrades a +setting when a flag opts in. + +`autonomy` is policy, orthogonal to capability: + full - intake consent is standing authorization; the agent submits T0-T2 + opt-outs without pausing per submission (default). + assisted - the agent pauses for operator confirmation before each submission. +""" +from __future__ import annotations + +import os +from pathlib import Path +from shutil import which + +import emailer +import paths +import storage + +DEFAULT_CONFIG = { + "autonomy": "full", # hands-off after intake+consent + "email_mode": "draft_only", # zero credentials + "browser_backend": "auto", # auto = Browserbase when BROWSERBASE_API_KEY is set + # (recommended default; clears soft CAPTCHAs), else plain browser + "tracker_backend": "local-json", # no external dependency + "encryption": "none", # files still written 0600 + "default_rescan_interval_days": 120, + "email_min_interval_seconds": 20, # pace SMTP sends so a run can't torch the account +} + +VALID = { + "autonomy": {"full", "assisted"}, + # email_mode: + # draft_only - render drafts; the operator sends + clicks verify links (zero setup) + # browser - the agent sends + opens verify links through the operator's logged-in + # webmail via browser_* tools (NO password stored; needs a browser the + # operator's inbox is signed into) + # programmatic - CLI sends via SMTP + reads verify links via IMAP (needs EMAIL_* creds) + # alias - AgentMail agent-owned inboxes / per-broker aliases + "email_mode": {"draft_only", "browser", "programmatic", "alias"}, + "browser_backend": {"auto", "browserbase", "agent-browser", "camofox"}, + "tracker_backend": {"local-json", "google-sheets"}, + "encryption": {"none", "age"}, +} + + +def load_config() -> dict: + cfg = dict(DEFAULT_CONFIG) + cfg.update(storage.read_json(paths.config_path(), {}) or {}) + return cfg + + +def save_config(cfg: dict) -> Path: + merged = dict(DEFAULT_CONFIG) + merged.update(cfg) + for key, allowed in VALID.items(): + if merged.get(key) not in allowed: + raise ValueError(f"invalid {key!r}: {merged.get(key)!r} (allowed: {sorted(allowed)})") + return storage.write_json(paths.config_path(), merged) + + +def detect_capabilities(env: dict | None = None) -> dict: + """Report which opt-in upgrades are available without extra setup.""" + env = os.environ if env is None else env + home = paths.hermes_home() + google = ( + (home / "google_token.json").exists() + or (home / "skills" / "productivity" / "google-workspace").exists() + or (home / "skills" / "google-workspace").exists() + ) + mail = emailer.available(env) + return { + "browserbase": bool(env.get("BROWSERBASE_API_KEY")), + "agentmail": bool(env.get("AGENTMAIL_API_KEY")), + "email_imap_smtp": bool(env.get("EMAIL_ADDRESS") and env.get("EMAIL_PASSWORD")), + "smtp_send": mail["smtp"], # CLI can SEND opt-out emails itself + "imap_read": mail["imap"], # CLI can POLL verification links itself + "google_workspace": google, + "age": which("age") is not None, + } + + +def auto_configure(env: dict | None = None) -> dict: + """Pick the most autonomous configuration this environment supports (no questions). + + - email: programmatic when SMTP creds exist (CLI sends + IMAP-verifies itself); + alias mode when only AgentMail exists; draft_only as the capability floor. + - browser: browserbase when the key exists (clears soft CAPTCHAs -> more T1). + - encryption: age when the binary is installed (free privacy, zero human cost). + - tracker: stays local-json (google-sheets needs a sheet id -> a human choice). + """ + caps = detect_capabilities(env) + cfg = load_config() + cfg["autonomy"] = "full" + if caps["smtp_send"]: + cfg["email_mode"] = "programmatic" + elif caps["agentmail"]: + cfg["email_mode"] = "alias" + else: + cfg["email_mode"] = "draft_only" + cfg["browser_backend"] = "browserbase" if caps["browserbase"] else "auto" + if caps["age"]: + cfg["encryption"] = "age" + return cfg + + +def browser_clears_captcha(cfg: dict, env: dict | None = None) -> bool: + """True if the chosen browser backend can clear soft CAPTCHAs (shifts T2 -> T1). + + Browserbase is the recommended default: a real residential-IP cloud browser passes + soft/managed challenges (Turnstile, hCaptcha/reCAPTCHA checkbox) as normal operation. + This is NOT solving/spoofing - hard interactive challenges still escalate to a human. + `auto` inherits this whenever BROWSERBASE_API_KEY is present. + """ + backend = cfg.get("browser_backend", "auto") + if backend == "browserbase": + return True + if backend == "auto": + env = os.environ if env is None else env + return bool(env.get("BROWSERBASE_API_KEY")) + return False diff --git a/optional-skills/security/unbroker/scripts/crypto.py b/optional-skills/security/unbroker/scripts/crypto.py new file mode 100644 index 00000000000..98594f5e840 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/crypto.py @@ -0,0 +1,88 @@ +"""At-rest encryption for sensitive files via the `age` binary (optional). + +Engaged ONLY when config `encryption: age` AND an age identity key exists AND the +`age`/`age-keygen` binaries are available. When engaged, JSON docs under +`subjects/` (dossier, ledger) are written as `.age` ciphertext; the audit +log (field NAMES + states only, no raw PII values), `config.json`, and the broker +cache stay plaintext so the engine can read them. + +Threat model (be honest): this protects against casual disk inspection, accidental +`git add`/commits, screen-shares, and backup/cloud-sync leakage. The identity key +defaults to living beside the data at `$PDD_DATA_DIR/age-identity.txt` (0600); set +`PDD_AGE_IDENTITY` to a separate volume/token for true key separation. It does NOT +protect against an attacker who can already read your whole HERMES_HOME (they get +key + data together). +""" +from __future__ import annotations + +import json +import subprocess +from pathlib import Path +from shutil import which + +import paths + + +def age_available() -> bool: + return which("age") is not None and which("age-keygen") is not None + + +def encryption_setting() -> str: + """Read `encryption` straight from config.json (no config/storage import => no cycle).""" + cfg = paths.config_path() + if not cfg.exists(): + return "none" + try: + return (json.loads(cfg.read_text(encoding="utf-8")) or {}).get("encryption", "none") + except (ValueError, OSError): + return "none" + + +def identity_path() -> Path: + return paths.age_identity_path() + + +def ensure_identity() -> Path: + """Generate an age identity (X25519 keypair) if missing; return its path.""" + if not age_available(): + raise RuntimeError("`age`/`age-keygen` not found; cannot enable encryption") + p = identity_path() + if not p.exists(): + p.parent.mkdir(parents=True, exist_ok=True) + try: + p.parent.chmod(0o700) + except OSError: + pass + subprocess.run(["age-keygen", "-o", str(p)], check=True, capture_output=True) + try: + p.chmod(0o600) + except OSError: + pass + return p + + +def recipient() -> str: + """The age public key (recipient) for the identity, parsed from its header.""" + p = ensure_identity() + for line in p.read_text(encoding="utf-8").splitlines(): + s = line.strip() + if s.lower().startswith("# public key:"): + return s.split(":", 1)[1].strip() + if s.startswith("age1"): + return s + raise RuntimeError(f"no public key found in {p}") + + +def is_engaged() -> bool: + """True only when encryption is actually active (configured + available + key present).""" + return encryption_setting() == "age" and age_available() and identity_path().exists() + + +def encrypt(data: bytes) -> bytes: + out = subprocess.run(["age", "-r", recipient()], input=data, capture_output=True, check=True) + return out.stdout + + +def decrypt(data: bytes) -> bytes: + out = subprocess.run(["age", "-d", "-i", str(identity_path())], input=data, capture_output=True, check=True) + return out.stdout diff --git a/optional-skills/security/unbroker/scripts/dossier.py b/optional-skills/security/unbroker/scripts/dossier.py new file mode 100644 index 00000000000..50b4c8c6b10 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/dossier.py @@ -0,0 +1,135 @@ +"""Subject dossier management + consent gate + least-disclosure field selection.""" +from __future__ import annotations + +import datetime as _dt +import hashlib +import os +from pathlib import Path + +import paths +import storage + +# Identifiers we never volunteer in an opt-out (would expand exposure, not reduce it). +NEVER_VOLUNTEER = {"ssn", "social_security_number", "passport", "drivers_license"} + +VALID_CONSENT_METHODS = {"self", "written_authorization", "poa"} + + +def now() -> str: + return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def new_subject_id(full_name: str = "") -> str: + # Opaque id: derives NOTHING from the name, so PII never leaks into directory names, + # case ids, drafts, or the audit log. full_name kept only for call compatibility. + return "sub_" + hashlib.sha1(os.urandom(8)).hexdigest()[:10] + + +def create(identity: dict, consent: dict, residency: str = "US", prefs: dict | None = None) -> dict: + dossier = { + "subject_id": new_subject_id(identity.get("full_name", "subject")), + "consent": consent, + "identity": identity, + "residency_jurisdiction": residency, + "preferences": prefs or {"email_mode": "draft_only", "rescan_interval_days": 120}, + "created_at": now(), + } + save(dossier) + return dossier + + +def load(subject_id: str) -> dict | None: + return storage.read_json(paths.dossier_path(subject_id), None) + + +def save(dossier: dict) -> Path: + return storage.write_json(paths.dossier_path(dossier["subject_id"]), dossier) + + +def is_authorized(dossier: dict) -> bool: + c = dossier.get("consent") or {} + return bool(c.get("authorized")) and c.get("method") in VALID_CONSENT_METHODS + + +def require_authorized(dossier: dict) -> None: + if not is_authorized(dossier): + raise PermissionError( + f"subject {dossier.get('subject_id')!r} has no recorded authorization; refusing to act" + ) + + +def all_names(dossier: dict) -> list[str]: + """Primary name + aliases (maiden/married/nicknames), deduped, in priority order.""" + ident = dossier.get("identity", {}) + out: list[str] = [] + seen: set[str] = set() + for n in [ident.get("full_name"), *(ident.get("also_known_as") or [])]: + if n and n.lower() not in seen: + seen.add(n.lower()) + out.append(n) + return out + + +def all_addresses(dossier: dict) -> list[dict]: + """Current + prior addresses, each tagged with `kind` (current|prior).""" + ident = dossier.get("identity", {}) + out: list[dict] = [] + cur = ident.get("current_address") + if cur: + out.append({**cur, "kind": cur.get("kind", "current")}) + for a in ident.get("prior_addresses") or []: + out.append({**a, "kind": a.get("kind", "prior")}) + return out + + +def all_locations(dossier: dict) -> list[dict]: + """Distinct city/state pairs across all addresses (the vectors for name searches).""" + out: list[dict] = [] + seen: set[tuple] = set() + for a in all_addresses(dossier): + city = a.get("city") + key = ((city or "").lower(), (a.get("state") or "").lower()) + if city and key not in seen: + seen.add(key) + out.append({"city": city, "state": a.get("state")}) + return out + + +def contact_email(dossier: dict) -> str | None: + """The single email used for opt-out correspondence (designated, else the first).""" + ident = dossier.get("identity", {}) + prefs = dossier.get("preferences", {}) + emails = ident.get("emails") or [] + return prefs.get("contact_email_for_optouts") or (emails[0] if emails else None) + + +def select_disclosure(dossier: dict, inputs: list[str], override_email: str | None = None) -> dict: + """Return ONLY the dossier fields a broker's opt-out actually requires. + + Enforces least-disclosure: skips anything in NEVER_VOLUNTEER, and skips + `profile_url` (that is captured per-listing at submit time, not from the dossier). + A single contact email is used for correspondence even when the subject has several + (see all_names / all_addresses / search vectors for using every alternate to *find* listings). + """ + ident = dossier.get("identity", {}) + addr = ident.get("current_address") or {} + phones = ident.get("phones") or [] + available = { + "full_name": ident.get("full_name"), + "first_name": (ident.get("full_name") or "").split(" ")[0] or None, + "contact_email": override_email or contact_email(dossier), + "current_address": addr or None, + "street": addr.get("line1"), + "city": addr.get("city"), + "state": addr.get("state"), + "postal": addr.get("postal"), + "date_of_birth": ident.get("date_of_birth"), + "phone": phones[0] if phones else None, + } + out: dict = {} + for key in inputs: + if key in NEVER_VOLUNTEER or key == "profile_url": + continue + if available.get(key) is not None: + out[key] = available[key] + return out diff --git a/optional-skills/security/unbroker/scripts/email_modes.py b/optional-skills/security/unbroker/scripts/email_modes.py new file mode 100644 index 00000000000..d5b40eb8495 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/email_modes.py @@ -0,0 +1,76 @@ +"""Email modes A/B/C helpers + anti-phishing verification-link extraction. + +Mode A (default): render a ready-to-send draft to disk; the operator sends it. +Mode B/C: the agent SENDS via a Hermes email mechanism (IMAP/SMTP gateway, +`himalaya`, AgentMail, or Gmail via `google-workspace`) and READS the reply to +resolve the verification link with `extract_verification_link`. Those transports +are driven by the agent through native tools; this module stays network-free so +the hermetic tests pass. +""" +from __future__ import annotations + +import re +from pathlib import Path + +import legal +import paths + +_LINK_RE = re.compile(r"https?://[^\s\"'<>)\]]+", re.IGNORECASE) +_VERIFY_HINTS = ("opt", "remov", "verif", "confirm", "unsubscrib", "suppress", "delete", "privacy") + + +def render_draft(broker: dict, fields: dict, out_dir: Path | None = None) -> Path: + """Mode A: write a ready-to-send opt-out email for the operator to send.""" + body = legal.render_optout_email(broker, fields) + out_dir = out_dir or (paths.data_dir() / "drafts") + out_dir.mkdir(parents=True, exist_ok=True) + fp = out_dir / f"{broker.get('id', 'broker')}.txt" + fp.write_text(body, encoding="utf-8") + return fp + + +def render_request_draft(broker: dict, fields: dict, kind: str = "generic", + out_dir: Path | None = None) -> Path: + """Mode A: write a ready-to-send request of a specific KIND. + + kind: generic | ccpa | ccpa_agent | ccpa_indirect | gdpr. Used for indirect-exposure + (ccpa_indirect) and explicit legal requests, where the generic opt-out wording is wrong. + The filename is suffixed with the kind so an indirect request does not overwrite an opt-out draft. + """ + body = legal.render_request(kind, broker, fields) + out_dir = out_dir or (paths.data_dir() / "drafts") + out_dir.mkdir(parents=True, exist_ok=True) + suffix = "" if kind == "generic" else f"-{kind}" + fp = out_dir / f"{broker.get('id', 'broker')}{suffix}.txt" + fp.write_text(body, encoding="utf-8") + return fp + + +def extract_verification_link(email_body: str, broker: dict | None = None) -> str | None: + """Return the most likely opt-out/verification link from an email body. + + Anti-phishing: a link is only returned if its URL matches an opt-out hint + and/or the broker's own domain; arbitrary links score 0 and are ignored. + """ + candidates = _LINK_RE.findall(email_body or "") + if not candidates: + return None + + domain = "" + if broker: + url = (broker.get("optout") or {}).get("url") or (broker.get("search") or {}).get("url") or "" + m = re.search(r"https?://([^/]+)", url) + if m: + domain = m.group(1).replace("www.", "") + + best_score, best_link = 0, None + for link in candidates: + low = link.lower() + score = 0 + if any(h in low for h in _VERIFY_HINTS): + score += 2 + if domain and domain in low: + score += 3 + if score > best_score: + best_score, best_link = score, link + return best_link diff --git a/optional-skills/security/unbroker/scripts/emailer.py b/optional-skills/security/unbroker/scripts/emailer.py new file mode 100644 index 00000000000..927bc153565 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/emailer.py @@ -0,0 +1,342 @@ +"""Programmatic email (Mode B) via stdlib smtplib/imaplib - no human in the loop. + +This is what turns email opt-outs autonomous: `send()` delivers the rendered +request straight to the broker's known opt-out address, and `find_verification_link()` +polls the inbox for the broker's confirmation email and extracts the link (scored +by email_modes.extract_verification_link, so arbitrary/phishing links are ignored). +The agent still OPENS the link with its own browser - several brokers bind the +verification session to the browser that opens it (see the intelius record). + +Configuration comes from the same env vars the Hermes email gateway uses: + EMAIL_ADDRESS / EMAIL_PASSWORD (required for Mode B) + EMAIL_SMTP_HOST / EMAIL_SMTP_PORT (optional; inferred for common providers) + EMAIL_IMAP_HOST / EMAIL_IMAP_PORT (optional; inferred for common providers) + +Anti-misuse: `send()` refuses a recipient that is not the broker record's own +opt-out/privacy address - this module cannot be repurposed to email arbitrary people. +All network calls live behind small functions that the hermetic tests monkeypatch. +""" +from __future__ import annotations + +import email as _email +import email.utils +import imaplib +import json +import os +import re +import smtplib +import time +from email.message import EmailMessage +from pathlib import Path + +import email_modes +import paths + +# provider domain -> (smtp_host, smtp_port, imap_host, imap_port) +PROVIDERS = { + "gmail.com": ("smtp.gmail.com", 587, "imap.gmail.com", 993), + "googlemail.com": ("smtp.gmail.com", 587, "imap.gmail.com", 993), + "outlook.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993), + "hotmail.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993), + "live.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993), + "yahoo.com": ("smtp.mail.yahoo.com", 587, "imap.mail.yahoo.com", 993), + "icloud.com": ("smtp.mail.me.com", 587, "imap.mail.me.com", 993), + "me.com": ("smtp.mail.me.com", 587, "imap.mail.me.com", 993), + "fastmail.com": ("smtp.fastmail.com", 587, "imap.fastmail.com", 993), +} + + +def _domain(address: str) -> str: + return address.rsplit("@", 1)[-1].lower() if "@" in address else "" + + +def smtp_settings(env: dict | None = None) -> dict | None: + """SMTP connection settings, or None when sending is not configured.""" + env = os.environ if env is None else env + address, password = env.get("EMAIL_ADDRESS"), env.get("EMAIL_PASSWORD") + if not (address and password): + return None + inferred = PROVIDERS.get(_domain(address)) + host = env.get("EMAIL_SMTP_HOST") or (inferred[0] if inferred else None) + if not host: + return None # unknown provider and no explicit host + port = int(env.get("EMAIL_SMTP_PORT") or (inferred[1] if inferred else 587)) + return {"host": host, "port": port, "address": address, "password": password} + + +def imap_settings(env: dict | None = None) -> dict | None: + """IMAP connection settings, or None when inbox reading is not configured.""" + env = os.environ if env is None else env + address, password = env.get("EMAIL_ADDRESS"), env.get("EMAIL_PASSWORD") + if not (address and password): + return None + inferred = PROVIDERS.get(_domain(address)) + host = env.get("EMAIL_IMAP_HOST") or (inferred[2] if inferred else None) + if not host: + return None + port = int(env.get("EMAIL_IMAP_PORT") or (inferred[3] if inferred else 993)) + return {"host": host, "port": port, "address": address, "password": password} + + +def available(env: dict | None = None) -> dict: + return {"smtp": smtp_settings(env) is not None, "imap": imap_settings(env) is not None} + + +# --- sending ------------------------------------------------------------------ + +def broker_addresses(broker: dict) -> list[str]: + """Every address the broker record itself declares (the ONLY valid recipients). + + Includes the primary opt-out email, the right-to-delete lane's email + (optout.deletion.email), and any mailto: links parsed from BADBOOL. + """ + opt = broker.get("optout") or {} + out = [a for a in [opt.get("email"), (opt.get("deletion") or {}).get("email")] if a] + for link in opt.get("links") or []: + url = (link.get("url") or "") + if url.lower().startswith("mailto:"): + out.append(url[7:].split("?")[0]) + seen: set[str] = set() + deduped = [] + for a in out: + if a.lower() not in seen: + seen.add(a.lower()) + deduped.append(a) + return deduped + + +def _split_subject_body(text: str) -> tuple[str, str]: + """Templates start with a 'Subject: ...' line; split it out for the MIME header.""" + lines = text.splitlines() + if lines and lines[0].lower().startswith("subject:"): + return lines[0].split(":", 1)[1].strip(), "\n".join(lines[1:]).lstrip("\n") + return "Data removal request", text + + +def browser_send_payload(broker: dict, body_text: str, to: str | None = None) -> dict: + """Build a recipient-locked {to, subject, body} for the agent to send via browser webmail. + + No network and no credentials: the deterministic part (recipient-lock to the broker's own + declared address, subject/body split) happens here; the agent then composes and sends it in + the operator's logged-in webmail with browser_* tools. Same recipient guard as `send()`, so + the browser lane cannot be pointed at an arbitrary person either. + """ + allowed = broker_addresses(broker) + if not allowed: + raise RuntimeError(f"broker {broker.get('id')!r} declares no opt-out email address") + recipient = to or allowed[0] + if recipient.lower() not in {a.lower() for a in allowed}: + raise PermissionError( + f"refusing to target {recipient!r}: not an address the broker record declares " + f"(allowed: {allowed})" + ) + subject, body = _split_subject_body(body_text) + return {"to": recipient, "subject": subject, "body": body} + + +def _rate_limit_path() -> Path: + return paths.data_dir() / "email-rate.json" + + +def _respect_rate_limit(min_interval: float, sleep, now, state_path=None) -> None: + """Pace sends across CLI invocations so a run can't torch the sending account. + + Persists the last-send wall-clock time; if the next send is too soon, sleep the + remainder. Cross-process because each `send-email` is a separate invocation. + """ + if min_interval <= 0: + return + p = state_path or _rate_limit_path() + last = 0.0 + try: + last = float(json.loads(p.read_text(encoding="utf-8")).get("last", 0.0)) + except (OSError, ValueError, TypeError): + last = 0.0 + wait = min_interval - (now() - last) + if wait > 0: + sleep(min(wait, min_interval)) + try: + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(json.dumps({"last": now()}), encoding="utf-8") + except OSError: + pass + + +# SMTP errors that are permanent (don't retry) vs transient (retry with backoff). +_SMTP_PERMANENT = (smtplib.SMTPAuthenticationError, smtplib.SMTPRecipientsRefused, + smtplib.SMTPSenderRefused, smtplib.SMTPDataError) + + +def send(broker: dict, body_text: str, to: str | None = None, + env: dict | None = None, _smtp_factory=None, + min_interval: float = 0.0, max_retries: int = 3, + _sleep=time.sleep, _now=time.time, _rate_state=None) -> dict: + """Send an opt-out/legal request to the broker's own opt-out address. + + Recipient is locked to an address the broker record declares (PermissionError + otherwise). `min_interval` paces sends across invocations (deliverability / + account-safety); transient SMTP/socket failures retry with exponential backoff, + permanent ones (auth, recipient refused) raise immediately. NOTE: a successful + SMTP handoff is NOT proof of delivery - real bounces arrive later as inbound mail; + in programmatic mode `poll-verification`/inbox review surfaces them, and the + due-queue re-scan is the true confirmation. Returns send metadata. + """ + settings = smtp_settings(env) + if not settings: + raise RuntimeError( + "programmatic email not configured (need EMAIL_ADDRESS + EMAIL_PASSWORD, and " + "EMAIL_SMTP_HOST for non-mainstream providers); fall back to `render-email` drafts" + ) + allowed = broker_addresses(broker) + if not allowed: + raise RuntimeError(f"broker {broker.get('id')!r} declares no opt-out email address") + recipient = to or allowed[0] + if recipient.lower() not in {a.lower() for a in allowed}: + raise PermissionError( + f"refusing to send to {recipient!r}: not an address the broker record declares " + f"(allowed: {allowed})" + ) + + subject, body = _split_subject_body(body_text) + msg = EmailMessage() + msg["From"] = settings["address"] + msg["To"] = recipient + msg["Subject"] = subject + msg["Date"] = email.utils.formatdate(localtime=True) + msg["Message-ID"] = email.utils.make_msgid() + msg.set_content(body) + + _respect_rate_limit(min_interval, _sleep, _now, _rate_state) + + factory = _smtp_factory or smtplib.SMTP + attempts = 0 + while True: + attempts += 1 + try: + with factory(settings["host"], settings["port"], timeout=30) as smtp: + smtp.ehlo() + try: + smtp.starttls() + smtp.ehlo() + except smtplib.SMTPNotSupportedError: + pass # already-TLS ports / test doubles + smtp.login(settings["address"], settings["password"]) + smtp.send_message(msg) + break + except _SMTP_PERMANENT: + raise # auth / recipient refused: retrying won't help + except (smtplib.SMTPException, OSError) as exc: + if attempts > max_retries: + raise RuntimeError(f"SMTP send failed after {attempts} attempts: {exc}") from exc + _sleep(min(2 ** (attempts - 1), 30)) # 1s, 2s, 4s... capped + return {"to": recipient, "subject": subject, "message_id": msg["Message-ID"], + "from": settings["address"], "attempts": attempts, + "delivery_note": "SMTP accepted; not proof of delivery - a bounce would arrive as " + "inbound mail. The due-queue re-scan is the real confirmation."} + + +# --- inbox polling ------------------------------------------------------------ + +def _decode_part(part) -> str: + try: + payload = part.get_payload(decode=True) + if payload is None: + return "" + charset = part.get_content_charset() or "utf-8" + return payload.decode(charset, errors="replace") + except Exception: # noqa: BLE001 - malformed MIME must not kill the poll + return "" + + +def message_text(msg) -> str: + """All text/plain + text/html content of a parsed email message.""" + chunks: list[str] = [] + if msg.is_multipart(): + for part in msg.walk(): + if part.get_content_type() in ("text/plain", "text/html"): + chunks.append(_decode_part(part)) + else: + chunks.append(_decode_part(msg)) + return "\n".join(c for c in chunks if c) + + +def _broker_domains(broker: dict) -> list[str]: + """Domains this broker legitimately mails from (site domains + optout email domain).""" + domains: list[str] = [] + for section in ("optout", "search"): + url = ((broker.get(section) or {}).get("url")) or "" + m = re.search(r"https?://([^/]+)", url) + if m: + domains.append(m.group(1).lower().removeprefix("www.")) + opt_email = (broker.get("optout") or {}).get("email") + if opt_email and "@" in opt_email: + domains.append(_domain(opt_email)) + # strip subdomains to the registrable-ish tail (mailer.intelius.com -> intelius.com) + tails = {".".join(d.split(".")[-2:]) for d in domains if d} + return sorted(tails) + + +def fetch_recent(env: dict | None = None, since_days: int = 3, limit: int = 30, + _imap_factory=None) -> list[dict]: + """Fetch recent inbox messages: [{from, subject, date, text}], newest first.""" + settings = imap_settings(env) + if not settings: + raise RuntimeError("IMAP not configured (need EMAIL_ADDRESS + EMAIL_PASSWORD, and " + "EMAIL_IMAP_HOST for non-mainstream providers)") + import datetime as _dt + since = (_dt.date.today() - _dt.timedelta(days=max(0, since_days))).strftime("%d-%b-%Y") + + factory = _imap_factory or imaplib.IMAP4_SSL + conn = factory(settings["host"], settings["port"]) + try: + conn.login(settings["address"], settings["password"]) + conn.select("INBOX", readonly=True) + _typ, data = conn.search(None, "SINCE", since) + ids = (data[0].split() if data and data[0] else [])[-limit:] + out: list[dict] = [] + for mid in reversed(ids): # newest first + _typ, msg_data = conn.fetch(mid, "(RFC822)") + raw = next((p[1] for p in msg_data or [] if isinstance(p, tuple)), None) + if not raw: + continue + msg = _email.message_from_bytes(raw) + out.append({ + "from": msg.get("From", ""), + "subject": msg.get("Subject", ""), + "date": msg.get("Date", ""), + "text": message_text(msg), + }) + return out + finally: + try: + conn.logout() + except Exception: # noqa: BLE001 + pass + + +def link_from_messages(messages: list[dict], broker: dict) -> dict | None: + """Pure: find the broker's verification link in already-fetched messages. + + A message is only considered if its From domain OR any contained link matches + the broker's own domains; the link itself must pass the anti-phishing scorer. + """ + domains = _broker_domains(broker) + for m in messages: + sender = (m.get("from") or "").lower() + text = m.get("text") or "" + sender_match = any(d in sender for d in domains) + body_match = any(d in text.lower() for d in domains) + if not (sender_match or body_match): + continue + link = email_modes.extract_verification_link(text, broker) + if link: + return {"link": link, "from": m.get("from"), "subject": m.get("subject"), + "date": m.get("date")} + return None + + +def find_verification_link(broker: dict, env: dict | None = None, since_days: int = 3, + _imap_factory=None) -> dict | None: + """Poll the inbox and return the broker's verification link (or None yet).""" + messages = fetch_recent(env, since_days=since_days, _imap_factory=_imap_factory) + return link_from_messages(messages, broker) diff --git a/optional-skills/security/unbroker/scripts/ledger.py b/optional-skills/security/unbroker/scripts/ledger.py new file mode 100644 index 00000000000..e5ec331a1e4 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/ledger.py @@ -0,0 +1,164 @@ +"""Case ledger: opt-out state machine + append-only audit log. + +A "case" is one (subject x broker) record. State changes are validated against +TRANSITIONS and mirrored into audit.jsonl so every action is auditable. +""" +from __future__ import annotations + +import datetime as _dt +from pathlib import Path + +import paths +import storage + +STATES = [ + "new", "searching", "not_found", "found", "indirect_exposure", "action_selected", "submitted", + "verification_pending", "awaiting_processing", "confirmed_removed", "reappeared", + "human_task_queued", "blocked", +] + +TRANSITIONS: dict[str, set[str]] = { + "new": {"searching", "found", "not_found", "indirect_exposure", "blocked"}, + "searching": {"not_found", "found", "indirect_exposure", "blocked"}, + "not_found": {"searching", "found", "indirect_exposure", "blocked"}, + "found": {"action_selected", "submitted", "human_task_queued", "indirect_exposure", "blocked"}, + # indirect_exposure: subject's PII (email/phone/name) sits on a THIRD PARTY's record. The + # self-service opt-out form does not apply; the lever is a targeted CCPA/GDPR delete-my-PII + # request (-> submitted) or a human task. Re-scan can clear it (-> not_found) or upgrade it to a + # direct listing (-> found). + "indirect_exposure": {"submitted", "human_task_queued", "not_found", "found", "blocked"}, + "action_selected": {"submitted", "human_task_queued", "blocked"}, + "submitted": {"verification_pending", "awaiting_processing", "human_task_queued", "blocked"}, + # verification_pending -> awaiting_processing: the verify link was opened/acknowledged and the + # broker is now processing the removal (their stated window). confirmed_removed still requires a + # verifying re-scan, never the submission flow's own say-so. + "verification_pending": {"awaiting_processing", "confirmed_removed", "human_task_queued", "blocked"}, + "awaiting_processing": {"confirmed_removed", "human_task_queued", "blocked"}, + "confirmed_removed": {"reappeared", "confirmed_removed"}, + "reappeared": {"found", "indirect_exposure"}, + "human_task_queued": { + "found", "indirect_exposure", "action_selected", "submitted", "verification_pending", + "awaiting_processing", "confirmed_removed", "blocked", + }, + # blocked: automated tools (web_extract/proxyless browser) couldn't read the site. A later pass + # -- a stealth/cloud browser OR guiding the operator's own (residential) browser -- can resolve it + # to any real scan verdict, so blocked reaches not_found / indirect_exposure too, not just found. + "blocked": {"searching", "found", "not_found", "indirect_exposure", "action_selected"}, +} + + +def now() -> str: + return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def load(subject_id: str) -> dict: + return storage.read_json(paths.ledger_path(subject_id), {}) or {} + + +def save(subject_id: str, ledger: dict) -> Path: + return storage.write_json(paths.ledger_path(subject_id), ledger) + + +def new_case(subject_id: str, broker_id: str) -> dict: + return { + "case_id": f"case_{subject_id}_{broker_id}", + "subject_id": subject_id, + "broker_id": broker_id, + "state": "new", + "found": None, + "evidence": {}, + "disclosure_log": [], + "history": [], + } + + +def get_case(subject_id: str, broker_id: str) -> dict: + return load(subject_id).get(broker_id) or new_case(subject_id, broker_id) + + +def can_transition(old: str, new: str) -> bool: + return new == old or new in TRANSITIONS.get(old, set()) + + +def transition(subject_id: str, broker_id: str, new_state: str, **fields) -> dict: + if new_state not in STATES: + raise ValueError(f"unknown state {new_state!r}") + # Lock the whole load-modify-save so a concurrent cron re-scan / other tenant + # can't read a stale ledger and clobber this transition. + with storage.locked(paths.ledger_path(subject_id)): + ledger = load(subject_id) + case = ledger.get(broker_id) or new_case(subject_id, broker_id) + old = case.get("state", "new") + if not can_transition(old, new_state): + raise ValueError(f"illegal transition {old!r} -> {new_state!r} for broker {broker_id!r}") + case["state"] = new_state + for key, value in fields.items(): + case[key] = value + stamp = now() + case.setdefault("history", []).append({"at": stamp, "from": old, "to": new_state}) + ledger[broker_id] = case + save(subject_id, ledger) + storage.append_jsonl( + paths.audit_path(subject_id), + {"at": stamp, "broker_id": broker_id, "event": "transition", "from": old, "to": new_state}, + ) + return case + + +DEFAULT_PROCESSING_DAYS = 14 # when a broker record doesn't state est_processing_days +VERIFICATION_POLL_DAYS = 1 # how soon to re-poll for an unarrived verification email + + +def _plus_days(days: int, start: str | None = None) -> str: + base = _dt.datetime.strptime(start, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=_dt.timezone.utc) \ + if start else _dt.datetime.now(_dt.timezone.utc) + return (base + _dt.timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def followup_fields(new_state: str, broker: dict | None = None, + dossier: dict | None = None) -> dict: + """Auto-scheduling stamps for a transition, so nobody has to remember follow-ups. + + submitted / awaiting_processing -> recheck after the broker's stated processing window; + verification_pending -> re-poll the inbox quickly; + confirmed_removed -> periodic reappearance re-scan per subject preference. + """ + if new_state in ("submitted", "awaiting_processing"): + days = ((broker or {}).get("optout") or {}).get("est_processing_days") or DEFAULT_PROCESSING_DAYS + return {"next_recheck_at": _plus_days(int(days))} + if new_state == "verification_pending": + return {"next_recheck_at": _plus_days(VERIFICATION_POLL_DAYS)} + if new_state == "confirmed_removed": + interval = ((dossier or {}).get("preferences") or {}).get("rescan_interval_days") or 120 + return {"removal_confirmed_at": now(), "next_recheck_at": _plus_days(int(interval))} + return {} + + +def due(subject_id: str, at: str | None = None, ledger: dict | None = None) -> list[dict]: + """Cases whose next_recheck_at has arrived - the autonomous follow-up queue.""" + stamp = at or now() + out = [] + for case in (ledger if ledger is not None else load(subject_id)).values(): + when = case.get("next_recheck_at") + if when and when <= stamp: + out.append(case) + out.sort(key=lambda c: c.get("next_recheck_at") or "") + return out + + +def log_disclosure(subject_id: str, broker_id: str, fields: list[str], channel: str) -> dict: + """Record exactly which PII field *names* were disclosed to a broker.""" + with storage.locked(paths.ledger_path(subject_id)): + ledger = load(subject_id) + case = ledger.get(broker_id) or new_case(subject_id, broker_id) + stamp = now() + record = {"at": stamp, "fields": sorted(fields), "channel": channel} + case.setdefault("disclosure_log", []).append(record) + ledger[broker_id] = case + save(subject_id, ledger) + storage.append_jsonl( + paths.audit_path(subject_id), + {"at": stamp, "broker_id": broker_id, "event": "disclosure", + "fields": record["fields"], "channel": channel}, + ) + return record diff --git a/optional-skills/security/unbroker/scripts/legal.py b/optional-skills/security/unbroker/scripts/legal.py new file mode 100644 index 00000000000..325687b273d --- /dev/null +++ b/optional-skills/security/unbroker/scripts/legal.py @@ -0,0 +1,63 @@ +"""Render opt-out / legal request text from templates/ with safe substitution. + +Templates use {field} placeholders. Missing fields are left literal (never crash, +never inject blanks that look like real data). Field values come from the +least-disclosure selection in dossier.select_disclosure. +""" +from __future__ import annotations + +from pathlib import Path + +import paths + + +class _SafeDict(dict): + def __missing__(self, key): # leave unknown placeholders untouched + return "{" + key + "}" + + +def template_path(name: str) -> Path: + return paths.templates_dir() / name + + +def render(template_name: str, fields: dict) -> str: + text = template_path(template_name).read_text(encoding="utf-8") + return text.format_map(_SafeDict(fields)) + + +def _join_listings(value) -> str: + if isinstance(value, (list, tuple)): + return "\n".join(str(v) for v in value) + return str(value or "") + + +def _join_identifiers(value) -> str: + """Render the subject's OWN identifiers as a bullet list for an indirect-exposure request.""" + if isinstance(value, (list, tuple)): + return "\n".join(f" - {v}" for v in value if v) + return f" - {value}" if value else "" + + +def render_optout_email(broker: dict, fields: dict) -> str: + ctx = dict(fields) + ctx.setdefault("broker_name", broker.get("name", "the data broker")) + ctx["listing_urls"] = _join_listings(fields.get("listing_urls")) + ctx.setdefault("full_name", fields.get("full_name", "[your name]")) + ctx.setdefault("contact_email", fields.get("contact_email", "[your email]")) + return render("emails/generic-optout.txt", ctx) + + +def render_request(kind: str, broker: dict, fields: dict) -> str: + """kind: generic | ccpa | ccpa_agent | ccpa_indirect | gdpr""" + template = { + "generic": "emails/generic-optout.txt", + "ccpa": "emails/ccpa-deletion.txt", + "ccpa_agent": "emails/ccpa-authorized-agent.txt", + "ccpa_indirect": "emails/ccpa-indirect-deletion.txt", + "gdpr": "emails/gdpr-erasure.txt", + }.get(kind, "emails/generic-optout.txt") + ctx = dict(fields) + ctx.setdefault("broker_name", broker.get("name", "the data broker")) + ctx["listing_urls"] = _join_listings(fields.get("listing_urls")) + ctx["my_identifiers"] = _join_identifiers(fields.get("my_identifiers")) + return render(template, ctx) diff --git a/optional-skills/security/unbroker/scripts/paths.py b/optional-skills/security/unbroker/scripts/paths.py new file mode 100644 index 00000000000..887748f4175 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/paths.py @@ -0,0 +1,79 @@ +"""Filesystem paths for the unbroker skill (stdlib only). + +All per-subject data lives under PDD_DATA_DIR (default: $HERMES_HOME/unbroker), +which is the same trust boundary Hermes uses for .env and OAuth tokens. +""" +from __future__ import annotations + +import os +from pathlib import Path + + +def hermes_home() -> Path: + return Path(os.environ.get("HERMES_HOME") or (Path.home() / ".hermes")) + + +def data_dir() -> Path: + override = os.environ.get("PDD_DATA_DIR") + return Path(override) if override else hermes_home() / "unbroker" + + +def config_path() -> Path: + return data_dir() / "config.json" + + +def subjects_dir() -> Path: + return data_dir() / "subjects" + + +def subject_dir(subject_id: str) -> Path: + return subjects_dir() / subject_id + + +def dossier_path(subject_id: str) -> Path: + return subject_dir(subject_id) / "dossier.json" + + +def ledger_path(subject_id: str) -> Path: + return subject_dir(subject_id) / "ledger.json" + + +def audit_path(subject_id: str) -> Path: + return subject_dir(subject_id) / "audit.jsonl" + + +def evidence_dir(subject_id: str) -> Path: + return subject_dir(subject_id) / "evidence" + + +def skill_root() -> Path: + """The skill directory (parent of scripts/).""" + return Path(__file__).resolve().parent.parent + + +def brokers_dir() -> Path: + return skill_root() / "references" / "brokers" + + +def brokers_cache_path() -> Path: + """Live broker snapshot pulled from BADBOOL (merged under the curated DB).""" + return data_dir() / "brokers-cache" / "badbool.json" + + +def registry_cache_path() -> Path: + """CA Data Broker Registry snapshot (separate coverage lane; DROP/email, not scanned).""" + return data_dir() / "brokers-cache" / "ca-registry.json" + + +def age_identity_path() -> Path: + """age identity (private key) used for at-rest encryption when enabled. + + Defaults beside the data; point PDD_AGE_IDENTITY at a separate volume/token + for real key separation from the encrypted data. + """ + override = os.environ.get("PDD_AGE_IDENTITY") + return Path(override) if override else data_dir() / "age-identity.txt" + + +def templates_dir() -> Path: + return skill_root() / "templates" diff --git a/optional-skills/security/unbroker/scripts/pdd.py b/optional-skills/security/unbroker/scripts/pdd.py new file mode 100644 index 00000000000..09039ab4734 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/pdd.py @@ -0,0 +1,810 @@ +#!/usr/bin/env python3 +"""unbroker - deterministic CLI helper. + +The Hermes agent orchestrates scanning and opt-out submission with native tools +(`web_extract`, `browser_navigate`, email mechanisms). THIS CLI owns the +deterministic state: config, dossiers + consent, the broker DB, tier planning, +the ledger + audit log, draft/template rendering, and reports. + +Run it through the `terminal` tool (it can read PII files under HERMES_HOME); +do NOT run it through `execute_code` (that sandbox scrubs env and redacts output). + +Examples: + python pdd.py setup + python pdd.py intake --full-name "Jane Q. Public" --email jane@example.com \ + --city Oakland --state CA --residency US-CA --consent --consent-method self + python pdd.py plan sub_xxxx --priority crucial + python pdd.py record sub_xxxx spokeo found --found true \ + --evidence '{"listing_urls":["https://www.spokeo.com/..."]}' + python pdd.py render-email sub_xxxx spokeo --listing https://www.spokeo.com/... + python pdd.py status sub_xxxx +""" +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import autopilot # noqa: E402 +import badbool # noqa: E402 +import brokers as brokers_mod # noqa: E402 +import config as config_mod # noqa: E402 +import crypto # noqa: E402 +import dossier as dossier_mod # noqa: E402 +import email_modes # noqa: E402 +import emailer # noqa: E402 +import ledger as ledger_mod # noqa: E402 +import legal # noqa: E402 +import paths as paths_mod # noqa: E402 +import registry # noqa: E402 +import report as report_mod # noqa: E402 +import tiers # noqa: E402 + + +def _out(obj) -> None: + print(json.dumps(obj, indent=2, ensure_ascii=False)) + + +def _require_subject(subject_id: str) -> dict: + d = dossier_mod.load(subject_id) + if not d: + sys.exit(f"error: unknown subject {subject_id!r} (run `intake` first)") + return d + + +def cmd_setup(args) -> None: + if getattr(args, "auto", False): + # Autonomous path: detect capabilities and pick the most autonomous valid + # config without asking anyone. Explicit flags still win below. + cfg = config_mod.auto_configure() + else: + cfg = config_mod.load_config() + for key in ("autonomy", "email_mode", "browser_backend", "tracker_backend", "encryption"): + val = getattr(args, key) + if val: + cfg[key] = val + if cfg.get("encryption") == "age": + if not crypto.age_available(): + sys.exit("error: encryption=age requested but `age`/`age-keygen` not found. " + "Install age (e.g. `brew install age`) or use `--encryption none`.") + crypto.ensure_identity() # generate the key now so encryption is actually engaged + path = config_mod.save_config(cfg) + migrated = _migrate_subjects() # rewrite existing dossiers/ledgers into the new at-rest format + out = { + "config_path": str(path), + "config": cfg, + "encryption_engaged": crypto.is_engaged(), + "detected_upgrades": config_mod.detect_capabilities(), + "migrated_subjects": migrated, + "note": "Defaults are easiest-first (draft email, auto browser, local tracker, no encryption). " + "Pass flags to opt into upgrades, then run `doctor` for a readiness summary.", + } + if cfg.get("encryption") == "age": + out["age_identity"] = str(crypto.identity_path()) + _out(out) + + +def _migrate_subjects() -> int: + """Re-save each subject's dossier + ledger so they match the current at-rest format.""" + sd = paths_mod.subjects_dir() + if not sd.exists(): + return 0 + n = 0 + for child in sorted(sd.iterdir()): + if not child.is_dir(): + continue + sid = child.name + d = dossier_mod.load(sid) + if d is not None: + dossier_mod.save(d) + n += 1 + led = ledger_mod.load(sid) + if led: + ledger_mod.save(sid, led) + return n + + +def _check_writable(path) -> bool: + try: + path.mkdir(parents=True, exist_ok=True) + probe = path / ".write_test" + probe.write_text("x", encoding="utf-8") + probe.unlink() + return True + except OSError: + return False + + +def cmd_doctor(args) -> None: + import platform + + cfg = config_mod.load_config() + caps = config_mod.detect_capabilities() + data = paths_mod.data_dir() + writable = _check_writable(data) + curated = len(brokers_mod._load_curated()) + live = len(brokers_mod.load_live_cache()) + total = len(brokers_mod.load_all()) + + L = ["unbroker - readiness check", "=" * 42, + f"Python : {platform.python_version()}", + f"Data dir : {data} ({'writable' if writable else 'NOT writable'})", + f"Config : autonomy={cfg.get('autonomy', 'full')} email={cfg['email_mode']} " + f"browser={cfg['browser_backend']} " + f"tracker={cfg['tracker_backend']} encryption={cfg['encryption']}", + f"Brokers : {total} available ({curated} curated + {live} live" + + ("" if live else ", run `refresh-brokers` to expand to ~50") + ")", + "", "Opt-in upgrades:"] + rows = [ + ("Cloud browser (Browserbase) *RECOMMENDED*", caps["browserbase"], + "default backend: clears soft CAPTCHAs (Turnstile/hCaptcha) -> more T1", "set BROWSERBASE_API_KEY"), + ("Email auto (AgentMail)", caps["agentmail"], + "send + auto-verify, per-broker aliases (Mode B/C)", "install agentmail skill / set AGENTMAIL_API_KEY"), + ("Email send (CLI SMTP)", caps["smtp_send"], + "`send-email` delivers opt-outs itself (Mode B)", "set EMAIL_ADDRESS / EMAIL_PASSWORD (+ EMAIL_SMTP_HOST)"), + ("Verify-link poll (CLI IMAP)", caps["imap_read"], + "`poll-verification` reads confirmation links itself", "set EMAIL_ADDRESS / EMAIL_PASSWORD (+ EMAIL_IMAP_HOST)"), + ("Google Sheets tracker", caps["google_workspace"], + "shared status dashboard", "set up the google-workspace skill"), + ] + for name, ok, enables, how in rows: + L.append(f" [{'ON ' if ok else 'off'}] {name:<28} {enables}") + if not ok: + L.append(f" enable: {how}") + + # At-rest encryption: report TRUE engagement (configured + key present), not just binary presence. + engaged = crypto.is_engaged() + L.append(f" [{'ON ' if engaged else 'off'}] {'At-rest encryption (age)':<28} " + "encrypts dossiers + ledgers on disk") + if engaged: + L.append(f" key: {crypto.identity_path()} (0600) - guards casual/backup/commit " + "exposure, NOT a full-HERMES_HOME read") + elif cfg["encryption"] == "age": + L.append(" WARNING: encryption=age is SET but NOT engaged (age binary or key missing);" + " dossiers would be PLAINTEXT") + elif caps["age"]: + L.append(" off - dossiers are plaintext (0600). enable: `setup --encryption age`") + else: + L.append(" off - dossiers are plaintext (0600). install `age` first to enable") + + L += ["", "Verdict:", " Ready now in DRAFT mode (no setup needed): scan brokers, draft opt-out", + " emails for you to send, and track everything in the ledger."] + if caps["browserbase"]: + L.append(" Cloud browser ON (recommended default): soft/managed CAPTCHAs " + "(Turnstile/hCaptcha) clear automatically -> those brokers stay T1.") + else: + L.append(" No cloud browser: set BROWSERBASE_API_KEY (the recommended default) so soft " + "CAPTCHAs clear automatically; without it those brokers drop to T2 (human tasks).") + if cfg["email_mode"] == "draft_only": + L.append(" Email is draft-only: you send drafts + click verify links. For hands-off email " + "WITHOUT storing a password, run `setup --email-mode browser` (agent sends + opens " + "verify links via your logged-in webmail); or set EMAIL_* for SMTP/IMAP.") + elif cfg["email_mode"] == "browser": + L.append(" Email mode: browser (no password) - the agent sends opt-outs and opens verify " + "links via the operator's logged-in webmail. Ensure that inbox is signed in in the " + "browser Hermes uses (a cloud browser won't hold the session); else it falls back to drafts.") + if not crypto.is_engaged(): + L.append(" Storage: dossiers are PLAINTEXT JSON (0600 under HERMES_HOME). " + "Run `setup --encryption age` for at-rest encryption.") + if not live: + L.append(" Next: run `refresh-brokers` to load the full broker list.") + + # Freshness: warn when cached lists / curated mechanics are going stale (silent broker rot). + import time as _time + STALE_CACHE_DAYS, STALE_VERIFY_DAYS = 30, 180 + + def _age_days(p) -> float | None: + try: + return (_time.time() - p.stat().st_mtime) / 86400.0 + except OSError: + return None + + fresh = [] + for label, p in [("BADBOOL", paths_mod.brokers_cache_path()), + ("CA registry", paths_mod.registry_cache_path())]: + age = _age_days(p) + if age is None: + fresh.append(f"{label}: not pulled") + elif age > STALE_CACHE_DAYS: + fresh.append(f"{label}: {age:.0f}d old (stale, re-pull)") + stale_curated = documented = 0 + for b in brokers_mod._load_curated(): + conf = b.get("confidence") + lv = b.get("last_verified") + if conf == "documented" or not lv: + documented += 1 + continue + try: + if (_time.time() - _time.mktime(_time.strptime(lv, "%Y-%m-%d"))) / 86400.0 > STALE_VERIFY_DAYS: + stale_curated += 1 + except (ValueError, TypeError): + pass + if fresh: + L.append(" Freshness: " + "; ".join(fresh) + " (run `refresh-brokers`).") + if stale_curated or documented: + L.append(f" Freshness: {stale_curated} curated broker(s) last-verified >{STALE_VERIFY_DAYS}d ago; " + f"{documented} documented broker(s) awaiting first-use verification.") + print("\n".join(L)) + + +def cmd_intake(args) -> None: + if args.json: + data = json.loads(Path(args.json).read_text(encoding="utf-8")) + identity = data["identity"] + consent = data.get("consent", {}) + residency = data.get("residency_jurisdiction", "US") + prefs = data.get("preferences") + else: + if not args.full_name: + sys.exit("error: --full-name (or --json) is required") + identity = {"full_name": args.full_name, "emails": args.email or [], "phones": args.phone or []} + if args.alias: + identity["also_known_as"] = args.alias + if args.dob: + identity["date_of_birth"] = args.dob + addr = {k: v for k, v in {"line1": args.street, "city": args.city, + "state": args.state, "postal": args.postal}.items() if v} + if addr: + identity["current_address"] = addr + priors = [] + for loc in args.prior_location or []: + parts = [p.strip() for p in loc.split(",") if p.strip()] + if not parts: + continue + entry = {"city": parts[0]} + if len(parts) > 1: + entry["state"] = parts[1] + if len(parts) > 2: + entry["postal"] = parts[2] + priors.append(entry) + if priors: + identity["prior_addresses"] = priors + cfg = config_mod.load_config() + consent = {"authorized": bool(args.consent), "method": args.consent_method, "recorded_at": dossier_mod.now()} + residency = args.residency or "US" + prefs = { + "email_mode": args.email_mode or cfg["email_mode"], + "rescan_interval_days": cfg["default_rescan_interval_days"], + } + if args.contact_email: + prefs["contact_email_for_optouts"] = args.contact_email + d = dossier_mod.create(identity, consent, residency, prefs) + _out({"subject_id": d["subject_id"], "authorized": dossier_mod.is_authorized(d), + "residency": residency, "email_mode": (prefs or {}).get("email_mode"), + "names": dossier_mod.all_names(d), + "emails": len(d["identity"].get("emails") or []), + "phones": len(d["identity"].get("phones") or []), + "addresses": len(dossier_mod.all_addresses(d))}) + + +def cmd_brokers(args) -> None: + bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all() + _out([ + {"id": b.get("id"), "name": b.get("name"), "priority": b.get("priority"), + "method": (b.get("optout") or {}).get("method"), "owns": b.get("owns") or [], + "source": b.get("source"), "confidence": b.get("confidence", "curated")} + for b in bl + ]) + + +def cmd_refresh_brokers(args) -> None: + res = badbool.refresh(paths_mod.brokers_cache_path()) + curated_ids = {b["id"] for b in brokers_mod._load_curated()} + new = [b["id"] for b in brokers_mod.load_live_cache() if b["id"] not in curated_ids] + out = {**res, "curated": len(curated_ids), "new_from_live": len(new), + "people_search_total": len(brokers_mod.load_all()), + "note": "Live records have confidence=auto; verify their opt-out URL before acting."} + if not getattr(args, "no_registry", False): + try: + reg = registry.refresh_all(paths_mod.registry_cache_path()) + out["registry"] = {"total": reg["total"], "sources": reg["sources"], + "portals": reg["portals"], + "note": "Coverage lane worked via the CA DROP one-shot + CCPA email, " + "not the people-search scan. VT/OR/TX are search portals (no " + "bulk export); CA is the superset. See `drop` and `registry`."} + except Exception as exc: # noqa: BLE001 - registry pull is best-effort + out["registry_error"] = str(exc) + _out(out) + + +def cmd_registry(args) -> None: + recs = brokers_mod.load_registry_cache() + if not recs: + _out({"registered_brokers": 0, + "note": "registry empty - run `refresh-brokers` (pulls the CA Data Broker Registry)"}) + return + fcra = sum(1 for r in recs if (r.get("optout") or {}).get("fcra")) + out = {"registered_brokers": len(recs), "fcra_regulated": fcra, + "source": "CA Data Broker Registry (CPPA, 2025)", "drop_url": registry.DROP_URL, + "other_state_portals": registry.portals()} + if args.search: + q = args.search.lower() + hits = [r for r in recs if q in (r.get("name") or "").lower() + or q in (r.get("id") or "") or q in ((r.get("optout") or {}).get("email") or "").lower()] + out["matches"] = [{"id": r["id"], "name": r["name"], + "email": (r.get("optout") or {}).get("email"), + "url": (r.get("optout") or {}).get("url"), + "fcra": (r.get("optout") or {}).get("fcra")} for r in hits[:args.limit]] + out["match_count"] = len(hits) + _out(out) + + +def cmd_drop(args) -> None: + """The one-shot legal lever: CA DROP deletes from ALL registered brokers at once.""" + d = _require_subject(args.subject) + dossier_mod.require_authorized(d) + reg = brokers_mod.load_registry_cache() + res = (d.get("residency_jurisdiction") or "US").upper() + eligible = res.startswith("US-CA") + if args.filed: + prefs = d.setdefault("preferences", {}) + prefs["drop_filed_at"] = dossier_mod.now() + dossier_mod.save(d) + _out({"subject": args.subject, "drop_filed_at": prefs["drop_filed_at"], + "note": "recorded; `next` will stop surfacing the DROP one-shot"}) + return + _out({ + "subject": args.subject, + "eligible": eligible, + "residency": res, + "drop_url": registry.DROP_URL, + "covers_registered_brokers": len(reg), + "steps": ([ + "Go to privacy.ca.gov/drop and create/verify a DROP account (CA resident).", + "Submit ONE deletion request; it applies to EVERY registered data broker " + f"({len(reg)} in the current registry). Brokers must process starting 2026-08-01.", + "After filing, run `drop --filed` so the loop stops re-surfacing it.", + ] if eligible else [ + "DROP is a California mechanism; this subject's residency is not US-CA.", + "Parity path for non-CA: work the people-search sites via `next`, and send targeted " + "CCPA/GDPR deletion emails to registry brokers that hold this person's data " + "(`registry --search`, then `send-email`).", + ]), + "note": "DROP is the highest-leverage removal: one request covers the whole registry.", + }) + + +def cmd_plan(args) -> None: + d = _require_subject(args.subject) + dossier_mod.require_authorized(d) + cfg = config_mod.load_config() + bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all() + bcc = config_mod.browser_clears_captcha(cfg) + if getattr(args, "batch", False): + _out(tiers.batch_plan(d, bl, cfg, ledger_mod.load(args.subject), bcc)) + else: + _out(tiers.plan(d, bl, cfg, bcc)) + + +def cmd_fanout(args) -> None: + d = _require_subject(args.subject) + dossier_mod.require_authorized(d) + bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all() + grouping = tiers.fanout(bl, batch_size=args.size) + mode = "scan AND opt-out (operator authorized submissions)" if args.optout \ + else "READ-ONLY scan (submit nothing; reconnaissance only)" + batches = [] + for i, ids in enumerate(grouping["batches"], 1): + brief = ( + f"You are scan worker {i} of {len(grouping['batches'])} for the `unbroker` " + f"skill. First load the `unbroker` skill and read its references/methods.md. " + f"Subject id: {args.subject}. Handle ONLY these brokers: {', '.join(ids)}. " + f"For EACH broker: read references/brokers/.json; run EVERY search vector from " + f"`pdd.py plan {args.subject}` (filtered to your brokers); build URLs from search.url_patterns " + f"and heed url_format_quirks; a 404 is INCONCLUSIVE (rebuild/try the on-site search box), not " + f"not_found; confirm the SUBJECT vs namesakes/relatives before recording; if search.antibot is " + f"set and no stealth/cloud browser is available, record `blocked`. Record each outcome via " + f"`pdd.py record {args.subject} " + f"--found --evidence '{{\"listing_urls\":[...]}}'`. Mode: {mode}. " + f"Log any newly-discovered URL/format quirks into the broker JSON. " + f"Return a concise structured per-broker report." + ) + batches.append({"batch": i, "brokers": ids, "brief": brief}) + _out({ + "subject": args.subject, + "broker_count": grouping["broker_count"], + "batch_size": grouping["batch_size"], + "should_fanout": grouping["should_fanout"], + "batch_count": len(batches), + "batches": batches, + "instruction": ( + "If should_fanout is true you MUST spawn ONE delegate_task subagent per batch IN PARALLEL, " + "passing each batch's `brief`; do not scan all brokers yourself sequentially. Wait for every " + "report, consolidate, then proceed to opt-outs. If false, just scan the brokers inline." + ), + }) + + +def cmd_record(args) -> None: + d = _require_subject(args.subject) + dossier_mod.require_authorized(d) + broker = brokers_mod.get(args.broker) + # Auto-stamp follow-up scheduling (next_recheck_at / removal_confirmed_at) so the + # autonomous loop knows when to come back without anyone remembering to set it. + fields = ledger_mod.followup_fields(args.state, broker, d) + if args.found is not None: + fields["found"] = args.found + if args.evidence: + fields["evidence"] = json.loads(args.evidence) + if args.reason: + fields["human_task_reason"] = args.reason + case = ledger_mod.transition(args.subject, args.broker, args.state, **fields) + if args.disclosed: + ledger_mod.log_disclosure(args.subject, args.broker, args.disclosed, args.channel or "unknown") + _out({"broker": args.broker, "state": case["state"], + "next_recheck_at": case.get("next_recheck_at")}) + + +def _email_request(d: dict, b: dict, kind: str, listings, identifiers) -> tuple[dict, list[str]]: + """Least-disclosure (fields, disclosed_names) for an opt-out/legal email of KIND. + + A removal letter must self-identify. Name + a contact email are already known to the + broker (the name is displayed on the very listing being removed), so not extra exposure. + """ + fields = dossier_mod.select_disclosure(d, (b.get("optout") or {}).get("inputs", [])) + ident = d.get("identity", {}) + if ident.get("full_name"): + fields.setdefault("full_name", ident["full_name"]) + fields.setdefault("contact_email", dossier_mod.contact_email(d) or "") + if listings: + fields["listing_urls"] = listings + if kind == "ccpa_indirect": + # Indirect exposure: name ONLY the subject's own identifiers to scrub from a third party's + # record. Default to the contact email + the subject's name-as-relative if none specified. + # The indirect template renders ONLY these placeholders; do not over-report disclosure with + # unrelated dossier fields (phone/street/postal) that select_disclosure happened to populate. + ids = list(identifiers or []) + if not ids: + ids = [contact for contact in [dossier_mod.contact_email(d)] if contact] + ids.append(f'the name "{ident.get("full_name")}" where it appears as a relative/associated person') + fields = { + "full_name": fields.get("full_name"), + "contact_email": fields.get("contact_email"), + "listing_urls": fields.get("listing_urls"), + "my_identifiers": ids, + } + return fields, ["contact_email", "full_name", "my_identifiers"] + return fields, sorted(fields.keys()) + + +def cmd_render_email(args) -> None: + d = _require_subject(args.subject) + dossier_mod.require_authorized(d) + b = brokers_mod.get(args.broker) + if not b: + sys.exit(f"error: unknown broker {args.broker!r}") + kind = getattr(args, "kind", "generic") or "generic" + fields, disclosed = _email_request(d, b, kind, args.listing, getattr(args, "identifier", None)) + if kind == "generic": + draft = email_modes.render_draft(b, fields) + else: + draft = email_modes.render_request_draft(b, fields, kind=kind) + ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_draft:{kind}") + _out({"draft": str(draft), "kind": kind, "disclosed_fields": disclosed}) + + +def cmd_send_email(args) -> None: + """Mode B: render AND deliver the opt-out/legal request - no human in the loop. + + Sends ONLY to an address the broker record itself declares (emailer enforces it), + then records the ledger transition + disclosure and auto-stamps the recheck date. + """ + d = _require_subject(args.subject) + dossier_mod.require_authorized(d) + b = brokers_mod.get(args.broker) + if not b: + sys.exit(f"error: unknown broker {args.broker!r}") + cfg = config_mod.load_config() + mode = cfg.get("email_mode") + if mode not in ("programmatic", "alias", "browser"): + sys.exit("error: email_mode is draft_only; run `setup --email-mode browser` (no password; " + "sends via your logged-in webmail) or `--email-mode programmatic`, or use " + "`render-email` and send it yourself") + if not args.listing: + sys.exit("error: --listing is required (verify-before-disclose: never " + "email a broker about an unconfirmed listing)") + # Idempotency: don't re-send if this case is already submitted/beyond (prevents duplicate + # requests when an action is retried). --force overrides. + _POST_SUBMIT = {"submitted", "verification_pending", "awaiting_processing", "confirmed_removed"} + current = ledger_mod.get_case(args.subject, args.broker).get("state") + if current in _POST_SUBMIT and not getattr(args, "force", False): + _out({"skipped": True, "broker": args.broker, "state": current, + "note": "already submitted; not re-sending (idempotent). Use --force to re-send."}) + return + kind = getattr(args, "kind", "generic") or "generic" + fields, disclosed = _email_request(d, b, kind, args.listing, getattr(args, "identifier", None)) + body = legal.render_optout_email(b, fields) if kind == "generic" else legal.render_request(kind, b, fields) + + if mode == "browser": + # No network / no credentials: hand the agent a recipient-locked payload to send in the + # operator's webmail via browser_* tools. State still records deterministically here. + payload = emailer.browser_send_payload(b, body, to=args.to) + ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_browser:{kind}") + case = ledger_mod.transition(args.subject, args.broker, "submitted", + **ledger_mod.followup_fields("submitted", b, d)) + _out({"send_via": "browser", "compose": payload, "kind": kind, "disclosed_fields": disclosed, + "state": case["state"], "next_recheck_at": case.get("next_recheck_at"), + "instruction": "In the operator's logged-in webmail, compose a NEW email to compose.to " + "with compose.subject/body EXACTLY (disclose nothing beyond it) and send " + "it via browser_* tools. Then use `verify-link` on any confirmation reply.", + "note": "recipient is locked to the broker's declared address"}) + return + + result = emailer.send(b, body, to=args.to, + min_interval=float(cfg.get("email_min_interval_seconds", 0) or 0)) + ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_sent:{kind}") + case = ledger_mod.transition(args.subject, args.broker, "submitted", + **ledger_mod.followup_fields("submitted", b, d)) + _out({"sent": result, "send_via": "smtp", "kind": kind, "disclosed_fields": disclosed, + "state": case["state"], "next_recheck_at": case.get("next_recheck_at"), + "note": "if this broker verifies by email, `poll-verification` will pick up the link"}) + + +def cmd_verify_link(args) -> None: + """Extract a broker's verification link from email text the agent read in webmail (browser mode). + + IMAP-free counterpart to `poll-verification`: the agent opens the broker's confirmation email + in the operator's webmail, pastes the body here, and gets the anti-phishing-scored link back. + """ + _require_subject(args.subject) + b = brokers_mod.get(args.broker) + if not b: + sys.exit(f"error: unknown broker {args.broker!r}") + text = args.text + if args.file: + text = Path(args.file).read_text(encoding="utf-8", errors="replace") + if not text: + sys.exit("error: provide --text '' (or --file) from the broker's confirmation email") + link = email_modes.extract_verification_link(text, b) + _out({"broker": args.broker, "verification_link": link, + "next": ("browser_navigate the link IN THE SAME browser (sessions are browser-bound), " + f"complete the flow, then `record {args.subject} {args.broker} awaiting_processing`" + if link else + "no broker/opt-out-scoped link found in that text; confirm you opened the right email")}) + + +def cmd_poll_verification(args) -> None: + """Poll the inbox for brokers' verification links (Mode B) - replaces the human click-chase. + + For each in-flight case (submitted / verification_pending with email_verification), + extract the broker's link (anti-phishing scored). A found link auto-advances + submitted -> verification_pending (the email HAS arrived); the agent must then OPEN + the link in its own browser (sessions are browser-bound) and record the next state. + """ + d = _require_subject(args.subject) + dossier_mod.require_authorized(d) + led = ledger_mod.load(args.subject) + targets = [] + for bid, case in sorted(led.items()): + if args.broker and bid != args.broker: + continue + if case.get("state") not in ("submitted", "verification_pending"): + continue + b = brokers_mod.get(bid) + if b and (((b.get("optout") or {}).get("requires")) or {}).get("email_verification"): + targets.append((bid, case, b)) + if not targets: + _out({"subject": args.subject, "results": [], + "note": "no in-flight cases awaiting email verification"}) + return + results = [] + for bid, case, b in targets: + hit = emailer.find_verification_link(b, since_days=args.since_days) + if hit: + if case.get("state") == "submitted": + ledger_mod.transition(args.subject, bid, "verification_pending", + **ledger_mod.followup_fields("verification_pending", b, d)) + results.append({"broker": bid, "verification_link": hit["link"], + "email_from": hit.get("from"), "email_subject": hit.get("subject"), + "next": f"browser_navigate the link IN THE AGENT'S OWN BROWSER, complete " + f"the flow, then `record {args.subject} {bid} awaiting_processing` " + f"(or confirmed_removed only after a verifying re-scan)"}) + else: + results.append({"broker": bid, "verification_link": None, + "next": "no matching email yet; poll again later (next_recheck_at is set)"}) + _out({"subject": args.subject, "results": results}) + + +def cmd_next(args) -> None: + d = _require_subject(args.subject) + dossier_mod.require_authorized(d) + cfg = config_mod.load_config() + bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all() + _out(autopilot.next_actions(d, bl, cfg, ledger_mod.load(args.subject))) + + +def cmd_tasks(args) -> None: + _require_subject(args.subject) + print(report_mod.human_tasks_markdown(args.subject)) + + +def cmd_due(args) -> None: + _require_subject(args.subject) + cases = ledger_mod.due(args.subject) + _out({"subject": args.subject, "due_count": len(cases), + "cases": [{"broker_id": c.get("broker_id"), "state": c.get("state"), + "next_recheck_at": c.get("next_recheck_at")} for c in cases], + "note": "run `next` for the concrete follow-up action per case"}) + + +def cmd_status(args) -> None: + _require_subject(args.subject) + print(report_mod.render_markdown(args.subject)) + + +def cmd_report(args) -> None: + _require_subject(args.subject) + if args.sheets: + _out(report_mod.sheets_rows(args.subject)) + else: + print(report_mod.render_markdown(args.subject)) + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(prog="pdd", description="unbroker helper CLI") + sub = p.add_subparsers(dest="cmd", required=True) + + s = sub.add_parser("setup", help="write install config (easiest-first defaults; --auto = most autonomous)") + s.add_argument("--auto", action="store_true", + help="detect capabilities and pick the most autonomous valid config (no questions)") + s.add_argument("--autonomy", dest="autonomy", choices=sorted(config_mod.VALID["autonomy"])) + s.add_argument("--email-mode", dest="email_mode", choices=sorted(config_mod.VALID["email_mode"])) + s.add_argument("--browser-backend", dest="browser_backend", choices=sorted(config_mod.VALID["browser_backend"])) + s.add_argument("--tracker-backend", dest="tracker_backend", choices=sorted(config_mod.VALID["tracker_backend"])) + s.add_argument("--encryption", dest="encryption", choices=sorted(config_mod.VALID["encryption"])) + s.set_defaults(func=cmd_setup) + + s = sub.add_parser("doctor", help="readiness check: config, brokers, available upgrades") + s.set_defaults(func=cmd_doctor) + + s = sub.add_parser("intake", help="create a subject dossier (records consent)") + s.add_argument("--json", help="path to a dossier JSON file (overrides flags)") + s.add_argument("--full-name") + s.add_argument("--alias", action="append", metavar="NAME", + help="other name the subject is listed under (maiden/married/nickname); repeatable") + s.add_argument("--email", action="append", metavar="EMAIL", help="repeatable") + s.add_argument("--phone", action="append", metavar="PHONE", help="repeatable") + s.add_argument("--street", help="current street line1 (enables reverse-address search)") + s.add_argument("--city") + s.add_argument("--state") + s.add_argument("--postal") + s.add_argument("--prior-location", dest="prior_location", action="append", metavar="City,ST", + help="a past city/state (or City,ST,ZIP); repeatable") + s.add_argument("--dob", help="date of birth YYYY-MM-DD (only used if a broker requires it)") + s.add_argument("--contact-email", dest="contact_email", + help="which email to use for opt-out correspondence (default: first)") + s.add_argument("--residency", help="e.g. US, US-CA") + s.add_argument("--consent", action="store_true", help="subject authorizes removal on their behalf") + s.add_argument("--consent-method", default="self", choices=["self", "written_authorization", "poa"]) + s.add_argument("--email-mode", dest="email_mode", choices=sorted(config_mod.VALID["email_mode"])) + s.set_defaults(func=cmd_intake) + + s = sub.add_parser("brokers", help="list the broker database (curated + live)") + s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"]) + s.set_defaults(func=cmd_brokers) + + s = sub.add_parser("refresh-brokers", + help="pull the latest BADBOOL people-search list + the CA data broker registry") + s.add_argument("--no-registry", dest="no_registry", action="store_true", + help="skip the CA registry pull (BADBOOL people-search only)") + s.set_defaults(func=cmd_refresh_brokers) + + s = sub.add_parser("registry", + help="CA Data Broker Registry coverage (hundreds of brokers; DROP/email lane)") + s.add_argument("--search", help="find registered brokers by name / id / email substring") + s.add_argument("--limit", type=int, default=25, help="max matches to print (default 25)") + s.set_defaults(func=cmd_registry) + + s = sub.add_parser("drop", + help="CA DROP one-shot: delete from ALL registered brokers in one request") + s.add_argument("subject") + s.add_argument("--filed", action="store_true", help="mark DROP as filed (stops `next` surfacing it)") + s.set_defaults(func=cmd_drop) + + s = sub.add_parser("plan", help="compute per-broker tier + next action for a subject") + s.add_argument("subject") + s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"]) + s.add_argument("--batch", action="store_true", + help="phase-oriented batch view: overlays ledger state, groups by next action " + "(unscanned/found/indirect/blocked/in_progress/done), collapses ownership clusters") + s.set_defaults(func=cmd_plan) + + s = sub.add_parser("fanout", help="batch brokers into parallel delegate_task subagents (large runs)") + s.add_argument("subject") + s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"]) + s.add_argument("--size", type=int, default=8, help="brokers per subagent batch (default 8)") + s.add_argument("--optout", action="store_true", + help="brief authorizes opt-out submission (default: read-only scan)") + s.set_defaults(func=cmd_fanout) + + s = sub.add_parser("record", help="record a ledger state transition after an agent action") + s.add_argument("subject") + s.add_argument("broker") + s.add_argument("state", choices=ledger_mod.STATES) + s.add_argument("--found", type=lambda v: v.strip().lower() in ("1", "true", "yes", "y")) + s.add_argument("--evidence", help="JSON object stored as case.evidence") + s.add_argument("--disclosed", action="append", metavar="FIELD", help="field name disclosed") + s.add_argument("--channel", help="disclosure channel, e.g. web_form / email") + s.add_argument("--reason", help="for human_task_queued: why a human is needed (shown in `tasks`)") + s.set_defaults(func=cmd_record) + + s = sub.add_parser("next", help="autonomous action queue: exactly what to do right now") + s.add_argument("subject") + s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"]) + s.set_defaults(func=cmd_next) + + s = sub.add_parser("send-email", help="Mode B: render AND send the opt-out/legal request (records it)") + s.add_argument("subject") + s.add_argument("broker") + s.add_argument("--listing", action="append", metavar="URL", required=False, + help="confirmed listing URL (required: verify-before-disclose)") + s.add_argument("--kind", choices=["generic", "ccpa", "ccpa_agent", "ccpa_indirect", "gdpr"], + default="generic") + s.add_argument("--identifier", action="append", metavar="ID", + help="(ccpa_indirect only) a specific own-identifier to remove; repeatable") + s.add_argument("--to", help="override recipient (must be an address the broker record declares)") + s.add_argument("--force", action="store_true", help="re-send even if already submitted (default: idempotent skip)") + s.set_defaults(func=cmd_send_email) + + s = sub.add_parser("poll-verification", + help="Mode B (IMAP): poll the inbox for brokers' verification links (anti-phishing scored)") + s.add_argument("subject") + s.add_argument("--broker", help="only this broker (default: every in-flight verification case)") + s.add_argument("--since-days", dest="since_days", type=int, default=3) + s.set_defaults(func=cmd_poll_verification) + + s = sub.add_parser("verify-link", + help="browser mode: extract a broker's verification link from pasted webmail text") + s.add_argument("subject") + s.add_argument("broker") + s.add_argument("--text", help="the confirmation email body (read from the operator's webmail)") + s.add_argument("--file", help="path to a file with the email body (alternative to --text)") + s.set_defaults(func=cmd_verify_link) + + s = sub.add_parser("tasks", help="ONE consolidated human-task digest (present at end of run)") + s.add_argument("subject") + s.set_defaults(func=cmd_tasks) + + s = sub.add_parser("due", help="cases whose recheck window has arrived (cron re-scan queue)") + s.add_argument("subject") + s.set_defaults(func=cmd_due) + + s = sub.add_parser("render-email", help="render a Mode-A opt-out / legal-request draft (least-disclosure)") + s.add_argument("subject") + s.add_argument("broker") + s.add_argument("--listing", action="append", metavar="URL", help="confirmed listing URL") + s.add_argument("--kind", choices=["generic", "ccpa", "ccpa_agent", "ccpa_indirect", "gdpr"], + default="generic", + help="request type. 'ccpa_indirect' = delete MY identifiers from a third party's " + "record (indirect exposure); default 'generic' opt-out.") + s.add_argument("--identifier", action="append", metavar="ID", + help="(ccpa_indirect only) a specific own-identifier to request removal of " + "(e.g. an email or phone). Repeatable. Defaults to the contact email + " + "name-as-relative if omitted.") + s.set_defaults(func=cmd_render_email) + + s = sub.add_parser("status", help="print a Markdown status report") + s.add_argument("subject") + s.set_defaults(func=cmd_status) + + s = sub.add_parser("report", help="status report (default) or --sheets rows") + s.add_argument("subject") + s.add_argument("--sheets", action="store_true", help="emit Google Sheets rows as JSON") + s.set_defaults(func=cmd_report) + return p + + +def main(argv=None) -> None: + args = build_parser().parse_args(argv) + try: + args.func(args) + except (PermissionError, ValueError, RuntimeError, FileNotFoundError) as exc: + sys.exit(f"error: {exc}") + + +if __name__ == "__main__": + main() diff --git a/optional-skills/security/unbroker/scripts/registry.py b/optional-skills/security/unbroker/scripts/registry.py new file mode 100644 index 00000000000..5e42356deab --- /dev/null +++ b/optional-skills/security/unbroker/scripts/registry.py @@ -0,0 +1,293 @@ +"""Ingest the California Data Broker Registry into broker records (coverage breadth). + +The CA registry (CPPA, under the Delete Act) is the authoritative universe of data +brokers doing business with California residents -- ~545 businesses in 2025, each +required to publish a name, website, contact email, and a CCPA-rights/deletion URL. +This is the same universe commercial services (DeleteMe/Incogni/Optery) draw from, +plus the FCRA/GLBA-regulated and marketing/risk brokers most lists omit. + +These are NOT people-search sites you scan with a name -- most have no per-person +lookup UI. They are worked through the LEGAL lane: the CA DROP portal +(privacy.ca.gov/drop) is a single request that deletes from ALL registered brokers +at once (CA residents), and per-broker CCPA deletion emails to the contact address +are the fallback / non-CA path. So registry records are kept in their own lane +(loaded only when asked) and never dumped into the people-search scan pipeline. + +`parse()` is pure (CSV text in, records out) so it is tested offline; `fetch()` is +the only network call and can be bypassed by passing csv_text directly to refresh(). +""" +from __future__ import annotations + +import csv +import datetime +import io +import re +import urllib.request +from pathlib import Path + +import storage + +# CA CPPA registry CSVs are published per year (registry2024.csv, registry2025.csv, ...). +# 2025 is the latest COMPLETE dataset; the current year's file is empty until the Jan +# registration window closes. DEFAULT_URL is the known-good fallback; `ca_candidate_urls` +# probes newer years first so coverage auto-advances when the next year is published. +_CA_CSV = "https://cppa.ca.gov/data_broker_registry/registry{year}.csv" +_CA_FLOOR_YEAR = 2025 +DEFAULT_URL = _CA_CSV.format(year=_CA_FLOOR_YEAR) +DROP_URL = "https://privacy.ca.gov/drop" +USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)" + + +def ca_candidate_urls(today: datetime.date | None = None) -> list[str]: + """Newest-year-first CA registry URLs to try (auto-advances; never below the 2025 floor).""" + year = (today or datetime.date.today()).year + years = list(range(max(year, _CA_FLOOR_YEAR), _CA_FLOOR_YEAR - 1, -1)) + return [_CA_CSV.format(year=y) for y in years] + +# Multi-source registry lane. Only California publishes a clean bulk CSV (with contact email + +# CCPA-rights URL per broker) AND offers a one-shot deletion portal (DROP). Vermont, Oregon, and +# Texas maintain registries too, but only as searchable PORTALS (no reliable bulk export) and with +# no DROP-equivalent -- and they overlap CA heavily (CA is effectively the superset). So they are +# wired as first-class portal sources (official URL surfaced to the operator) rather than scraped. +# Adding any state that later publishes a CSV is a one-line "format: csv" entry (the parser is +# column-detection based, not CA-specific). +SOURCES = { + "ca": {"jurisdiction": "US-CA", "format": "csv", "url": DEFAULT_URL, "has_drop": True, + "name": "California Data Broker Registry (CPPA)"}, + "vt": {"jurisdiction": "US-VT", "format": "portal", "has_drop": False, + "url": "https://bizfilings.vermont.gov/online/DatabrokerInquire/", + "name": "Vermont Data Broker Registry (Secretary of State)"}, + "or": {"jurisdiction": "US-OR", "format": "portal", "has_drop": False, + "url": "https://dfr.oregon.gov/business/licensing/data-broker-registry/Pages/index.aspx", + "name": "Oregon Data Broker Registry (DCBS)"}, + "tx": {"jurisdiction": "US-TX", "format": "portal", "has_drop": False, + "url": "https://texas-sos.appianportalsgov.com/data-broker-registry", + "name": "Texas Data Broker Registry (Secretary of State)"}, +} + + +def portals() -> list[dict]: + """Registry sources that are searchable portals (no bulk export) -- surfaced to the operator.""" + return [{"key": k, "jurisdiction": s["jurisdiction"], "name": s["name"], "url": s["url"]} + for k, s in SOURCES.items() if s["format"] == "portal"] + +# Field label -> substring to locate its column on the header row (robust to +# year-to-year column shifts; the registry re-orders/adds columns between years). +_LABELS = { + "name": "data broker name:", + "dba": "doing business as", + "website": "data broker primary website:", + "email": "primary contact email", + "rights_url": "exercise their ca consumer privacy act rights", + "fcra": "regulated by the federal fair credit reporting act (fcra):", +} + + +def _norm(s: str) -> str: + """Registry CSVs use NBSPs and a BOM; normalize for matching + clean values.""" + return re.sub(r"\s+", " ", (s or "").replace("\ufeff", "").replace("\xa0", " ")).strip() + + +def slug(name: str, website: str = "") -> str: + base = re.sub(r"\.(com|org|net|io|ai|inc|co|us|info|llc)\b", "", (name or "").strip(), flags=re.I) + s = re.sub(r"[^a-z0-9]+", "", base.lower()) + if s: + return s + dom = re.sub(r"^https?://(www\.)?", "", (website or "").lower()) + return re.sub(r"[^a-z0-9]+", "", dom.split("/")[0]) or "broker" + + +def _domain(website: str) -> str: + dom = re.sub(r"^https?://(www\.)?", "", (website or "").strip().lower()) + return dom.split("/")[0] + + +def _find_colmap(rows: list[list[str]]) -> tuple[int, dict[str, int]]: + """Locate the label row (col0 == 'Data broker name:') and map fields to columns.""" + for i, row in enumerate(rows[:5]): + if row and _norm(row[0]).lower().startswith("data broker name:"): + colmap: dict[str, int] = {} + for field, needle in _LABELS.items(): + for j, cell in enumerate(row): + c = _norm(cell).lower() + if needle in c and not c.startswith("if the data broker"): + colmap[field] = j + break + return i, colmap + raise ValueError("CA registry: could not locate the header row") + + +def _get(row: list[str], idx: int | None) -> str: + return _norm(row[idx]) if idx is not None and idx < len(row) else "" + + +def _build(row: list[str], cm: dict[str, int], jurisdiction: str = "US-CA", + has_drop: bool = True) -> dict | None: + name = _get(row, cm.get("name")) + website = _get(row, cm.get("website")) + if not (name or website): + return None + email = _get(row, cm.get("email")) + rights = _get(row, cm.get("rights_url")) + dba = _get(row, cm.get("dba")) + fcra = _get(row, cm.get("fcra")).lower().startswith("y") + state = jurisdiction.split("-")[-1] + + method = "email" if email else ("web_form" if rights else "drop") + if has_drop: + notes = ("Registered CA data broker. One CA DROP request (privacy.ca.gov/drop) deletes from " + "this and every registered broker at once; or send a CCPA deletion request to the " + "contact email.") + else: + notes = (f"Registered {state} data broker (no one-shot delete portal in {state}). Send a " + "CCPA/state-law deletion request to the contact email.") + if fcra: + notes += (" FCRA-regulated: some data is credit-reporting data with separate rules -- deletion " + "may be limited; a consumer report dispute/security-freeze may apply instead.") + return { + "id": slug(name, website), + "name": name or _domain(website), + "dba": dba or None, + "category": "data_broker", + "priority": "long_tail", + "jurisdictions": [jurisdiction], + "search": {"method": "none", "url": website, "fetch": "none", "by": ["registry"]}, + "optout": { + "method": method, + "url": rights or website or None, + "email": email or None, + "requires": {"profile_url": False, "email_verification": False, "captcha": False, + "gov_id": False, "account": False, "phone_callback": False, "payment": False}, + "inputs": ["full_name", "contact_email"], + "deletion": { + "via": "drop" if has_drop else "email", + "email": email or None, + "url": rights or None, + "kinds": ["ccpa", "generic"], + "notes": ("Covered by the CA DROP one-shot (privacy.ca.gov/drop); CCPA email fallback." + if has_drop else "CCPA/state-law deletion email (no one-shot portal)."), + }, + "fcra": fcra, + "est_processing_days": 45, + "notes": notes, + }, + "source": f"{state}-registry", + "confidence": "registry", + "last_verified": None, + } + + +def parse(csv_text: str, jurisdiction: str = "US-CA", has_drop: bool = True) -> list[dict]: + """Parse a data-broker-registry CSV into broker records (deduped by id). + + Column detection is by header label, not fixed position, so any state that publishes a + registry CSV with name/website/email/rights columns parses without new code. + """ + rows = list(csv.reader(io.StringIO(csv_text))) + if not rows: + return [] + header_i, cm = _find_colmap(rows) + out: list[dict] = [] + seen: dict[str, int] = {} + for row in rows[header_i + 1:]: + if not any(c.strip() for c in row): + continue + rec = _build(row, cm, jurisdiction, has_drop) + if not rec: + continue + bid = rec["id"] + if bid in seen: # disambiguate id collisions by domain, then a counter + dom = re.sub(r"[^a-z0-9]+", "", _domain(rec["search"]["url"])) + cand = f"{bid}-{dom}" if dom and dom != bid else bid + while cand in seen: + seen[bid] += 1 + cand = f"{bid}-{seen[bid]}" + rec["id"] = cand + seen.setdefault(rec["id"], 0) + seen.setdefault(bid, 0) + out.append(rec) + return out + + +MIN_EXPECTED_CA = 100 # CA registry has ~500+; far fewer => wrong/empty file, warn + + +def fetch(url: str = DEFAULT_URL, timeout: int = 60) -> str: + req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) + with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 + return resp.read().decode("utf-8", errors="replace") + + +def _fetch_ca_latest() -> tuple[str, list[dict]]: + """Try newest CA registry year first; return (url, records) for the first non-empty.""" + last: tuple[str, list[dict]] = (DEFAULT_URL, []) + for url in ca_candidate_urls(): + try: + recs = parse(fetch(url), jurisdiction="US-CA", has_drop=True) + except Exception: # noqa: BLE001 - a missing year 404s; fall through to older years + continue + if recs: + return url, recs + last = (url, recs) + return last + + +def refresh(cache_path: Path, url: str = DEFAULT_URL, csv_text: str | None = None) -> dict: + """CA single-source refresh: fetch (or accept) the CA CSV and write the cache.""" + text = csv_text if csv_text is not None else fetch(url) + records = parse(text) + storage.write_json(cache_path, records) + fcra = sum(1 for r in records if (r.get("optout") or {}).get("fcra")) + return {"parsed": len(records), "fcra_regulated": fcra, + "cache_path": str(cache_path), "source_url": url} + + +def refresh_all(cache_path: Path, fetched: dict[str, str] | None = None) -> dict: + """Multi-source refresh: pull every CSV source, dedupe across states by domain, cache. + + `fetched` optionally supplies {source_key: csv_text} to bypass the network (tests). CSV + sources are ingested as broker records; portal sources contribute their URL for the operator + (no bulk export exists) but no records. CA is processed first so it wins domain collisions. + """ + all_recs: list[dict] = [] + seen_domains: set[str] = set() + per_source: dict[str, dict] = {} + for key, src in SOURCES.items(): + if src["format"] != "csv": + per_source[key] = {"jurisdiction": src["jurisdiction"], "format": "portal", + "url": src["url"], "records": 0, + "note": "searchable portal (no bulk export); operator/agent searches by name"} + continue + used_url = src["url"] + try: + if fetched is not None: + text = fetched.get(key) + if text is None: + raise RuntimeError("no CSV text supplied") + recs = parse(text, jurisdiction=src["jurisdiction"], has_drop=src["has_drop"]) + elif key == "ca": + used_url, recs = _fetch_ca_latest() # newest-year-first with fallback + else: + recs = parse(fetch(src["url"]), jurisdiction=src["jurisdiction"], has_drop=src["has_drop"]) + except Exception as exc: # noqa: BLE001 - one source failing must not sink the rest + per_source[key] = {"jurisdiction": src["jurisdiction"], "format": "csv", "error": str(exc)} + continue + added = 0 + for r in recs: + dom = _domain(r["search"]["url"]) + if dom and dom in seen_domains: + continue + if dom: + seen_domains.add(dom) + all_recs.append(r) + added += 1 + entry = {"jurisdiction": src["jurisdiction"], "format": "csv", "url": used_url, + "parsed": len(recs), "added_after_dedupe": added, + "fcra": sum(1 for r in recs if (r.get("optout") or {}).get("fcra"))} + if key == "ca" and len(recs) < MIN_EXPECTED_CA: + entry["warning"] = (f"only {len(recs)} parsed (expected >{MIN_EXPECTED_CA}); the CA " + "registry file may be empty/moved - verify the source URL") + per_source[key] = entry + storage.write_json(cache_path, all_recs) + return {"total": len(all_recs), "sources": per_source, "portals": portals(), + "cache_path": str(cache_path)} diff --git a/optional-skills/security/unbroker/scripts/report.py b/optional-skills/security/unbroker/scripts/report.py new file mode 100644 index 00000000000..1c38164a24f --- /dev/null +++ b/optional-skills/security/unbroker/scripts/report.py @@ -0,0 +1,161 @@ +"""Status dashboards, Markdown reports, human-task digest, and Google Sheets row export.""" +from __future__ import annotations + +import brokers as brokers_mod +import ledger as ledger_mod + +STATE_LABELS = { + "new": "Not started", + "searching": "Searching", + "not_found": "Not found", + "found": "Found (action needed)", + "indirect_exposure": "Indirect exposure (PII on a relative's record)", + "action_selected": "Action selected", + "submitted": "Submitted", + "verification_pending": "Awaiting verification", + "awaiting_processing": "Processing", + "confirmed_removed": "Removed", + "reappeared": "Reappeared", + "human_task_queued": "Human task", + "blocked": "Blocked", +} + + +def status_counts(subject_id: str) -> dict: + counts: dict[str, int] = {} + for case in ledger_mod.load(subject_id).values(): + state = case.get("state", "new") + counts[state] = counts.get(state, 0) + 1 + return counts + + +def metrics(subject_id: str) -> dict: + """Outcome metrics: what's actually confirmed vs merely claimed, and what's overdue. + + removal_rate is confirmed_removed over cases we actually acted on (found/submitted/... ), + NOT over the whole broker DB, so it reflects real progress on real exposure. `in_flight` + is 'claimed' (submitted/verifying/processing) but not yet re-scan-confirmed. `overdue` + counts cases whose recheck window has already passed (the cron backlog). + """ + c = status_counts(subject_id) + removed = c.get("confirmed_removed", 0) + in_flight = c.get("submitted", 0) + c.get("verification_pending", 0) + c.get("awaiting_processing", 0) + open_found = c.get("found", 0) + c.get("reappeared", 0) + c.get("action_selected", 0) \ + + c.get("indirect_exposure", 0) + acted = removed + in_flight + open_found + c.get("human_task_queued", 0) + c.get("blocked", 0) + return { + "confirmed_removed": removed, + "in_flight_claimed": in_flight, # submitted but NOT yet verified gone + "open_needs_action": open_found, + "blocked": c.get("blocked", 0), + "human_tasks": c.get("human_task_queued", 0), + "acted_total": acted, + "removal_rate": round(removed / acted, 3) if acted else 0.0, + "overdue_rechecks": len(ledger_mod.due(subject_id)), + } + + +def render_markdown(subject_id: str) -> str: + ledger = ledger_mod.load(subject_id) + counts = status_counts(subject_id) + total = sum(counts.values()) + removed = counts.get("confirmed_removed", 0) + + m = metrics(subject_id) + lines = [ + f"# unbroker - status for `{subject_id}`", + "", + f"**{removed} / {total} confirmed removed** · removal rate (of acted-on cases): " + f"{int(m['removal_rate'] * 100)}%", + "", + f"- Confirmed removed: {m['confirmed_removed']}", + f"- In flight (submitted, not yet re-scan-confirmed): {m['in_flight_claimed']}", + f"- Open / needs action: {m['open_needs_action']}", + f"- Blocked (anti-bot): {m['blocked']} · Human tasks: {m['human_tasks']}", + f"- Overdue rechecks (cron backlog): {m['overdue_rechecks']}", + "", + "| State | Count |", + "|---|---|", + ] + for state in ledger_mod.STATES: + if counts.get(state): + lines.append(f"| {STATE_LABELS.get(state, state)} | {counts[state]} |") + + tasks = [c for c in ledger.values() if c.get("state") == "human_task_queued"] + if tasks: + lines += ["", "## Outstanding human tasks"] + for c in tasks: + reason = c.get("human_task_reason", "manual step required") + lines.append(f"- **{c.get('broker_id')}** - {reason}") + + indirect = [c for c in ledger.values() if c.get("state") == "indirect_exposure"] + if indirect: + lines += ["", "## Indirect exposure (your PII on third-party records)", + "Not removable via the broker's self-service opt-out (the record is about someone " + "else). Lever: a targeted CCPA/GDPR delete-my-PII request naming only your own " + "identifiers."] + for c in indirect: + ev = c.get("evidence") or {} + note = ev.get("summary") or "subject's identifiers appear on another person's listing" + lines.append(f"- **{c.get('broker_id')}** - {note}") + return "\n".join(lines) + "\n" + + +def human_tasks_markdown(subject_id: str) -> str: + """ONE consolidated digest of everything that genuinely needs a human. + + The autonomous run accumulates human-only work silently (never interrupting); + this digest is presented once, at the end, so the operator clears it in a + single sitting. Includes queued tasks and blocked-site operator-browser checks. + """ + ledger = ledger_mod.load(subject_id) + tasks = [(bid, c) for bid, c in sorted(ledger.items()) if c.get("state") == "human_task_queued"] + blocked = [(bid, c) for bid, c in sorted(ledger.items()) if c.get("state") == "blocked"] + + lines = [f"# Human tasks for `{subject_id}`", ""] + if not tasks and not blocked: + lines.append("Nothing needs a human right now.") + return "\n".join(lines) + "\n" + + lines.append(f"{len(tasks)} manual step(s) + {len(blocked)} blocked site(s). " + "Everything else ran (or will run) autonomously.") + if tasks: + lines += ["", "## Manual steps"] + for bid, c in tasks: + b = brokers_mod.get(bid) or {} + opt = b.get("optout") or {} + lines.append(f"### {b.get('name', bid)}") + lines.append(f"- Why: {c.get('human_task_reason', 'manual step required')}") + where = opt.get("url") or opt.get("email") or "(see broker record)" + lines.append(f"- Where: {where}") + for q in (opt.get("quirks") or [])[:2]: + lines.append(f"- Note: {q}") + lines.append("- Withhold: SSN and full ID numbers - always.") + lines.append(f"- When done, tell the agent so it records the outcome for `{bid}`.") + if blocked: + lines += ["", "## Blocked sites (open in YOUR browser - it gets through where bots don't)"] + for bid, c in blocked: + b = brokers_mod.get(bid) or {} + url = ((b.get("search") or {}).get("url")) or "(see broker record)" + lines.append(f"- **{b.get('name', bid)}** - open {url}, search the subject, and report " + "the verdict (or a screenshot) back to the agent.") + return "\n".join(lines) + "\n" + + +def sheets_rows(subject_id: str) -> list[list[str]]: + """Header + one row per case for the optional Google Sheets tracker. + + The agent appends these via the `google-workspace` skill, e.g.: + google_api.py sheets append "Sheet1!A:F" --values + """ + rows = [["broker_id", "state", "found", "tier", "removed_at", "next_recheck"]] + for bid, c in sorted(ledger_mod.load(subject_id).items()): + rows.append([ + bid, + c.get("state", ""), + str(c.get("found", "")), + (c.get("automation") or {}).get("tier_used", ""), + c.get("removal_confirmed_at") or "", + c.get("next_recheck_at") or "", + ]) + return rows diff --git a/optional-skills/security/unbroker/scripts/scan.py b/optional-skills/security/unbroker/scripts/scan.py new file mode 100644 index 00000000000..3c91c30e774 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/scan.py @@ -0,0 +1,32 @@ +"""Stdlib fetch helper for simple url_pattern brokers (osint-style). + +For JS-rendered or anti-bot pages the agent should use the `web_extract` or +`browser_navigate` tools (and the `scrapling` skill for stealth/Cloudflare). +This helper only covers plain static pages and is intentionally network-light so +it can be mocked in tests. +""" +from __future__ import annotations + +import urllib.error +import urllib.request + +USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)" + + +def fetch(url: str, timeout: int = 20) -> tuple[int, str]: + req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 (https only by convention) + charset = resp.headers.get_content_charset() or "utf-8" + return getattr(resp, "status", 200), resp.read().decode(charset, errors="replace") + except urllib.error.HTTPError as exc: + return exc.code, "" + except (urllib.error.URLError, TimeoutError, ValueError): + return 0, "" + + +def looks_listed(html: str, match_signal: str | None) -> bool: + """Naive confirmation heuristic for static pages: does the match signal appear?""" + if not html or not match_signal: + return False + return match_signal.lower() in html.lower() diff --git a/optional-skills/security/unbroker/scripts/storage.py b/optional-skills/security/unbroker/scripts/storage.py new file mode 100644 index 00000000000..29a66bb80d3 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/storage.py @@ -0,0 +1,138 @@ +"""Storage helpers (stdlib only): atomic JSON, append-only JSONL, strict perms. + +Default backend is local-json. The optional google-sheets tracker is handled in +report.py by emitting rows for the `google-workspace` skill; this module stays +dependency-free so the hermetic tests never touch the network. +""" +from __future__ import annotations + +import contextlib +import json +import os +import time +from pathlib import Path +from typing import Any + +import crypto +import paths + + +@contextlib.contextmanager +def locked(target: Path, timeout: float = 10.0, stale: float = 30.0): + """Portable advisory lock via an O_EXCL lockfile next to `target`. + + Serializes read-modify-write on shared JSON (the ledger) across concurrent + processes - a cron re-scan overlapping a manual run, or multiple tenants - + so one writer can't clobber another's update. A lock older than `stale` + seconds is treated as abandoned (crashed writer) and broken, so a dead + process can never deadlock the queue. Works on macOS/Linux/Windows (O_EXCL). + """ + ensure_dir(target.parent) + lock = target.with_name(target.name + ".lock") + deadline = time.monotonic() + timeout + while True: + try: + fd = os.open(str(lock), os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600) + try: + os.write(fd, str(os.getpid()).encode()) + finally: + os.close(fd) + break + except FileExistsError: + try: + if time.time() - lock.stat().st_mtime > stale: + lock.unlink(missing_ok=True) + continue + except OSError: + pass + if time.monotonic() >= deadline: + raise TimeoutError(f"could not acquire lock {lock} within {timeout}s") + time.sleep(0.05) + try: + yield + finally: + with contextlib.suppress(OSError): + lock.unlink(missing_ok=True) + + +def _secure(path: Path, mode: int) -> None: + try: + os.chmod(path, mode) + except OSError: + pass # non-POSIX / unsupported FS; HERMES_HOME directory perms still apply + + +def ensure_dir(path: Path) -> Path: + path.mkdir(parents=True, exist_ok=True) + _secure(path, 0o700) + return path + + +def _is_sensitive(path: Path) -> bool: + """Per-subject docs (dossier, ledger) are sensitive; config/cache are not.""" + try: + Path(path).resolve().relative_to(paths.subjects_dir().resolve()) + return True + except (ValueError, OSError): + return False + + +def _age_path(path: Path) -> Path: + return path.with_name(path.name + ".age") + + +def _atomic_write(path: Path, data: bytes) -> Path: + tmp = path.with_name(path.name + ".tmp") + tmp.write_bytes(data) + _secure(tmp, 0o600) + os.replace(tmp, path) + _secure(path, 0o600) + return path + + +def write_json(path: Path, obj: Any) -> Path: + ensure_dir(path.parent) + data = (json.dumps(obj, indent=2, ensure_ascii=False) + "\n").encode("utf-8") + if _is_sensitive(path) and crypto.encryption_setting() == "age": + if not crypto.age_available(): + raise RuntimeError( + "encryption=age is configured but `age` is not available; " + "refusing to write PII as plaintext. Install age or run `setup --encryption none`." + ) + target = _atomic_write(_age_path(path), crypto.encrypt(data)) + if path.exists(): + path.unlink() # migrate plaintext -> ciphertext + return target + target = _atomic_write(path, data) + ap = _age_path(path) + if ap.exists(): + ap.unlink() # encryption turned off -> drop stale ciphertext + return target + + +def read_json(path: Path, default: Any = None) -> Any: + ap = _age_path(path) + if ap.exists(): + return json.loads(crypto.decrypt(ap.read_bytes()).decode("utf-8")) + if path.exists(): + return json.loads(path.read_text(encoding="utf-8")) + return default + + +def append_jsonl(path: Path, record: dict) -> Path: + ensure_dir(path.parent) + with path.open("a", encoding="utf-8") as fh: + fh.write(json.dumps(record, ensure_ascii=False) + "\n") + _secure(path, 0o600) + return path + + +def read_jsonl(path: Path) -> list[dict]: + if not path.exists(): + return [] + out: list[dict] = [] + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if line: + out.append(json.loads(line)) + return out diff --git a/optional-skills/security/unbroker/scripts/tiers.py b/optional-skills/security/unbroker/scripts/tiers.py new file mode 100644 index 00000000000..8d145b8cec3 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/tiers.py @@ -0,0 +1,269 @@ +"""Automation-tier selection and per-subject action planning. + +Tiers: + T0 fully automated, no verification loop + T1 automated submit + automated verification (email mode B/C, or backend-cleared captcha) + T2 automated submit, verification needs a human (hard captcha / phone callback / account) + T3 human-required end-to-end (gov ID, fax, mail, voice-only phone) +""" +from __future__ import annotations + +import dossier as dossier_mod +import vectors as vectors_mod + +HARD_HUMAN = ("gov_id", "fax", "mail", "phone_voice") + + +def select_tier(broker: dict, email_mode: str = "draft_only", + browser_clears_captcha: bool = False) -> str: + req = ((broker.get("optout") or {}).get("requires")) or {} + + if any(req.get(k) for k in HARD_HUMAN): + return "T3" + if req.get("account"): + return "T2" + + captcha = bool(req.get("captcha")) + if (captcha and not browser_clears_captcha) or req.get("phone_callback"): + return "T2" + + if req.get("email_verification"): + return "T1" if email_mode in ("programmatic", "alias") else "T2" + + if captcha and browser_clears_captcha: + return "T1" + return "T0" + + +def plan(subject_dossier: dict, brokers_list: list[dict], cfg: dict, + browser_clears_captcha: bool = False) -> list[dict]: + email_mode = (subject_dossier.get("preferences") or {}).get("email_mode") \ + or cfg.get("email_mode", "draft_only") + actions: list[dict] = [] + for b in brokers_list: + opt = b.get("optout") or {} + search = b.get("search") or {} + tier = select_tier(b, email_mode, browser_clears_captcha) + disclosure = dossier_mod.select_disclosure(subject_dossier, opt.get("inputs", [])) + svectors = vectors_mod.search_vectors(subject_dossier, b) + actions.append({ + "broker_id": b.get("id"), + "broker_name": b.get("name"), + "priority": b.get("priority"), + "method": opt.get("method"), + "tier": tier, + "human_required": tier == "T3", + "search_url": search.get("url"), + "fetch": search.get("fetch", "web_extract"), + "antibot": search.get("antibot"), + "search_by": vectors_mod.supported_by(b), + "search_vectors": svectors, + "optout_url": opt.get("url"), + "optout_email": opt.get("email"), + "disclosure_fields": sorted(disclosure.keys()), + "owns": b.get("owns") or [], + "notes": opt.get("notes", ""), + "optout_quirks": opt.get("quirks") or [], + "optout_requires": opt.get("requires") or {}, + # The DELETION lane (right-to-delete), distinct from listing suppression. Structured so + # the autopilot can route to it: {via: email|in_flow|web_form, email?, url?, kinds?, notes?} + "deletion": opt.get("deletion") or {}, + # Exact ordered opt-out steps maintained IN the broker record (field-verified knowledge + # lives with the data, not in code). + "optout_playbook": opt.get("playbook") or [], + }) + return actions + + +def fanout(brokers_list: list[dict], batch_size: int = 8) -> dict: + """Group brokers into batches for parallel `delegate_task` scan subagents. + + Scanning many brokers serially is slow and burns context; above `batch_size` + the agent is expected to spawn one subagent per batch (see SKILL.md). + """ + ids = [b.get("id") for b in brokers_list if b.get("id")] + batches = [ids[i:i + batch_size] for i in range(0, len(ids), batch_size)] + return { + "broker_count": len(ids), + "batch_size": batch_size, + "should_fanout": len(ids) > batch_size, + "batches": batches, + } + + +# States that mean "the crawl reached a verdict for this broker". +_SCANNED_STATES = {"found", "not_found", "indirect_exposure", "blocked", "submitted", + "verification_pending", "awaiting_processing", "confirmed_removed", "reappeared", + "action_selected", "human_task_queued"} +# States that still need a deletion action taken. +_ACTIONABLE_STATES = {"found", "indirect_exposure", "reappeared", "action_selected"} + + +def batch_plan(subject_dossier: dict, brokers_list: list[dict], cfg: dict, + ledger: dict | None = None, browser_clears_captcha: bool = False) -> dict: + """Reduce the per-broker plan into a phase-oriented batch view. + + Overlays the current ledger state on each broker, groups by what the operator + should DO next, and collapses ownership clusters so a parent removal that clears + children is ONE action, not N. Read-only: computes, never mutates the ledger. + """ + ledger = ledger or {} + actions = plan(subject_dossier, brokers_list, cfg, browser_clears_captcha) + + # child id -> parent id (only for parents present in this plan set) + child_to_parent: dict[str, str] = {} + for a in actions: + for child in a.get("owns") or []: + child_to_parent[child] = a["broker_id"] + + def state_of(bid: str) -> str: + return (ledger.get(bid) or {}).get("state", "new") + + groups: dict[str, list[dict]] = { + "unscanned": [], # no verdict yet -> Phase 1 crawl + "found": [], # direct removable listing -> Phase 2 opt-out (incl. reappeared/action_selected) + "indirect_exposure": [],# PII on a third party's record -> CCPA/GDPR delete email + "blocked": [], # anti-bot / needs stealth browser -> requeue + "in_progress": [], # submitted / verification_pending / awaiting_processing + "human": [], # human_task_queued -> the end-of-run digest, NOT re-scanning + "done": [], # confirmed_removed + "not_found": [], + } + covered_by_parent: dict[str, list[str]] = {} + + for a in actions: + bid = a["broker_id"] + st = state_of(bid) + # cluster collapse: if a parent in this set is already actioned, the child is covered + parent = child_to_parent.get(bid) + if parent and state_of(parent) in ("found", "reappeared", "action_selected", "submitted", + "verification_pending", "awaiting_processing", + "confirmed_removed", "human_task_queued"): + covered_by_parent.setdefault(parent, []).append(bid) + continue + + row = {"broker_id": bid, "broker_name": a["broker_name"], "priority": a["priority"], + "tier": a["tier"], "method": a["method"], "state": st, + "optout_url": a["optout_url"], "optout_email": a.get("optout_email"), + "clears_children": a.get("owns") or [], + "optout_requires": a.get("optout_requires") or {}, + "optout_quirks": a.get("optout_quirks") or [], + "deletion": a.get("deletion") or {}, + "optout_playbook": a.get("optout_playbook") or [], + "notes": a.get("notes", "")} + if st in ("submitted", "verification_pending", "awaiting_processing"): + groups["in_progress"].append(row) + elif st == "confirmed_removed": + groups["done"].append(row) + elif st in ("reappeared", "action_selected"): + groups["found"].append(row) # still needs the opt-out action + elif st == "human_task_queued": + groups["human"].append(row) # parked for the digest; never re-queued as work + elif st in groups: + groups[st].append(row) + elif st not in _SCANNED_STATES: + groups["unscanned"].append(row) + else: + groups.setdefault(st, []).append(row) + + # PARENTS FIRST: within the actionable 'found' group, order cluster parents (a removal + # that clears children) ahead of standalone listings, most-children first. Working a + # parent before its children is what makes the cluster dedup real -- do them in this order. + groups["found"].sort(key=lambda r: (-len(r.get("clears_children") or []), + {"T0": 0, "T1": 1, "T2": 2, "T3": 3}.get(r.get("tier") or "", 9), + r["broker_id"])) + + return { + "subject": subject_dossier.get("subject_id"), + "phase": "discover" if groups["unscanned"] else "delete", + "counts": {k: len(v) for k, v in groups.items()}, + "groups": groups, + "cluster_savings": {p: kids for p, kids in covered_by_parent.items()}, + "parent_playbook": _parent_playbook(groups["found"]), + "next_actions": _batch_next(groups, covered_by_parent), + } + + +def synthesize_steps(r: dict) -> list[str]: + """Generic ordered opt-out steps derived from an optout record's structured fields. + + Used for any broker without a hand-verified `optout.playbook`. Bespoke, field-verified + step lists live IN the broker JSON (`optout.playbook`) - single source of truth that + accrues knowledge as live runs discover mechanics (see methods.md logging rule). + """ + steps = [f"Opt out at {r.get('optout_url') or r.get('optout_email') or '(see broker record)'}" + + (f" -- clears {', '.join(r['clears_children'])}." if r.get("clears_children") else ".")] + req = r.get("optout_requires") or {} + if req.get("profile_url"): + steps.append("Needs the confirmed profile_url (paste the listing URL you recorded).") + if req.get("email_verification"): + steps.append("Email verification: the same browser/inbox must open the confirmation link.") + if req.get("phone_callback"): + steps.append("Phone-callback code required; queue a human task if no operator is available.") + if req.get("gov_id"): + steps.append("Government ID demanded (T3): human task; never send SSN or a full ID number.") + d = r.get("deletion") or {} + if d.get("email"): + steps.append(f"DELETION lane: a right-to-delete request can be emailed to {d['email']}" + + (f" ({d['notes']})" if d.get("notes") else "") + + " -- prefer deletion over suppression.") + if r.get("notes"): + steps.append(str(r["notes"])) + for q in (r.get("optout_quirks") or [])[:3]: + steps.append(str(q)) + return steps + + +def _parent_playbook(found_rows: list[dict]) -> list[dict]: + """Tailored, ordered opt-out instructions for each cluster PARENT in the found group. + + Steps come from the broker record's own `optout.playbook` (field-verified, maintained with + the data) with a synthesised fallback so the guidance is never empty. Standalone listings + are intentionally omitted -- the playbook exists to make the parents-first order concrete. + """ + playbook: list[dict] = [] + for i, r in enumerate([x for x in found_rows if x.get("clears_children")], start=1): + steps = list(r.get("optout_playbook") or []) or synthesize_steps(r) + playbook.append({ + "order": i, + "broker_id": r["broker_id"], + "broker_name": r["broker_name"], + "tier": r["tier"], + "clears_children": r["clears_children"], + "optout_url": r.get("optout_url"), + "optout_email": r.get("optout_email"), + "deletion": r.get("deletion") or {}, + "steps": steps, + }) + return playbook + + +def _batch_next(groups: dict, covered: dict) -> list[str]: + tips: list[str] = [] + if groups["unscanned"]: + tips.append(f"PHASE 1 (crawl): {len(groups['unscanned'])} broker(s) unscanned -- run `fanout` and " + "scan read-only before any deletion.") + if groups["found"]: + parents = [r for r in groups["found"] if r.get("clears_children")] + if parents: + order = " -> ".join(r["broker_id"] for r in parents) + tips.append(f"PHASE 2 (opt-out): {len(groups['found'])} direct listing(s). DO CLUSTER PARENTS " + f"FIRST, in this order: {order} (see `parent_playbook` for tailored per-parent " + "steps), then the standalone listings.") + else: + tips.append(f"PHASE 2 (opt-out): {len(groups['found'])} direct listing(s) to remove.") + if groups["indirect_exposure"]: + tips.append(f"{len(groups['indirect_exposure'])} indirect-exposure case(s): send a targeted " + "CCPA/GDPR delete-my-PII email (render-email --kind ccpa_indirect), do NOT use the opt-out form.") + if groups["blocked"]: + tips.append(f"{len(groups['blocked'])} blocked (anti-bot): requeue for a stealth/cloud browser " + "pass; don't burn subagent time fighting CAPTCHAs.") + if covered: + n = sum(len(v) for v in covered.values()) + tips.append(f"Cluster dedup: {n} child site(s) covered by parent removals -- skip separate opt-outs.") + if groups["in_progress"]: + tips.append(f"{len(groups['in_progress'])} in progress: resolve verification links, then confirm removal.") + if groups.get("human"): + tips.append(f"{len(groups['human'])} parked human task(s): present via `tasks` at end of run " + "(do not re-scan or re-queue them).") + return tips diff --git a/optional-skills/security/unbroker/scripts/vectors.py b/optional-skills/security/unbroker/scripts/vectors.py new file mode 100644 index 00000000000..4fcf006e1d3 --- /dev/null +++ b/optional-skills/security/unbroker/scripts/vectors.py @@ -0,0 +1,53 @@ +"""Enumerate the search queries to run per broker, across ALL of a subject's identifiers. + +People-search sites index a person under every name, phone, email, and address they +have. A subject with two names (maiden/married) and three past cities can have many +distinct listings on one broker, each found via a different search. `search_vectors` +expands the dossier into the concrete searches to run, filtered by what each broker +supports (`broker.search.by`, default ["name"]). +""" +from __future__ import annotations + +import dossier as dossier_mod + +# What a broker can be searched by; default if a record doesn't declare it. +DEFAULT_BY = ["name"] + + +def supported_by(broker: dict) -> list[str]: + return list((broker.get("search") or {}).get("by") or DEFAULT_BY) + + +def search_vectors(subject_dossier: dict, broker: dict) -> list[dict]: + """List of {by, query} searches to run for this subject on this broker.""" + by = set(supported_by(broker)) + ident = subject_dossier.get("identity", {}) + vectors: list[dict] = [] + + if "name" in by: + names = dossier_mod.all_names(subject_dossier) + locations = dossier_mod.all_locations(subject_dossier) + if locations: + for name in names: + for loc in locations: + vectors.append({"by": "name", + "query": {"full_name": name, "city": loc.get("city"), "state": loc.get("state")}}) + else: + for name in names: + vectors.append({"by": "name", "query": {"full_name": name}}) + + if "phone" in by: + for phone in ident.get("phones") or []: + vectors.append({"by": "phone", "query": {"phone": phone}}) + + if "email" in by: + for email in ident.get("emails") or []: + vectors.append({"by": "email", "query": {"email": email}}) + + if "address" in by: + for a in dossier_mod.all_addresses(subject_dossier): + if a.get("line1"): + vectors.append({"by": "address", + "query": {k: a.get(k) for k in ("line1", "city", "state", "postal")}}) + + return vectors diff --git a/optional-skills/security/unbroker/templates/consent/authorization.md b/optional-skills/security/unbroker/templates/consent/authorization.md new file mode 100644 index 00000000000..9815916727f --- /dev/null +++ b/optional-skills/security/unbroker/templates/consent/authorization.md @@ -0,0 +1,15 @@ +# Authorization to act on my behalf (data removal) + +I, **{full_name}**, authorize the operator of this tool to act as my agent for the limited purpose of +removing my personal information from data brokers and people-search websites, including submitting +opt-out, deletion, and do-not-sell/share requests under applicable privacy laws (e.g. CCPA/CPRA, +GDPR) on my behalf. + +This authorization is limited to data removal. It does not authorize any other use of my information. + +- Full name: {full_name} +- Date: {date} +- Signature: ______________________________ + +Store the signed copy at the path recorded in the dossier `consent.authorization_artifact`. Required +only when `consent.method` is `written_authorization` or `poa` (not for `self`). diff --git a/optional-skills/security/unbroker/templates/emails/ccpa-authorized-agent.txt b/optional-skills/security/unbroker/templates/emails/ccpa-authorized-agent.txt new file mode 100644 index 00000000000..e5e4a09a114 --- /dev/null +++ b/optional-skills/security/unbroker/templates/emails/ccpa-authorized-agent.txt @@ -0,0 +1,24 @@ +Subject: CCPA/CPRA request submitted by authorized agent (delete and opt out) + +To the {broker_name} privacy team, + +I am an authorized agent acting on behalf of the consumer named below, under Cal. Civ. Code +1798.135 and the CCPA/CPRA. Written authorization is on file and available on request. + +On the consumer's behalf I request that you: + + 1. DELETE all personal information you hold about them, and + 2. OPT them OUT of the sale and sharing of their personal information. + +Their information appears at: +{listing_urls} + +Consumer: + Name: {full_name} + Email for confirmation: {contact_email} + +Please confirm completion in writing to the email above. Do not request more sensitive information +than necessary to verify and process this request. + +Sincerely, +Authorized agent for {full_name} diff --git a/optional-skills/security/unbroker/templates/emails/ccpa-deletion.txt b/optional-skills/security/unbroker/templates/emails/ccpa-deletion.txt new file mode 100644 index 00000000000..db2d201c73c --- /dev/null +++ b/optional-skills/security/unbroker/templates/emails/ccpa-deletion.txt @@ -0,0 +1,22 @@ +Subject: CCPA/CPRA request to delete and opt out (do not sell or share) + +To the {broker_name} privacy team, + +Under the California Consumer Privacy Act as amended by the CPRA (Cal. Civ. Code 1798.105 and +1798.120), I request that you: + + 1. DELETE all personal information you hold about me, and + 2. OPT me OUT of the sale and sharing of my personal information. + +My information appears at: +{listing_urls} + +Identifying details for this request: + Name: {full_name} + Email: {contact_email} + +Please do not request more sensitive information than necessary to process this request. Confirm +completion in writing to the email above within the statutory timeframe. + +Sincerely, +{full_name} diff --git a/optional-skills/security/unbroker/templates/emails/ccpa-indirect-deletion.txt b/optional-skills/security/unbroker/templates/emails/ccpa-indirect-deletion.txt new file mode 100644 index 00000000000..e561e8d6e11 --- /dev/null +++ b/optional-skills/security/unbroker/templates/emails/ccpa-indirect-deletion.txt @@ -0,0 +1,30 @@ +Subject: Request to delete my personal information from third-party listings (CCPA/CPRA where applicable) + +To the {broker_name} privacy team, + +I am not the primary subject of the listings below, but each one currently exposes MY personal +information as a secondary data point (my email address and/or my name shown as a relative or +associated person). I am writing only about my own personal information, not about the individuals +who are the primary subjects of these records, and I am not requesting any change to their data. + +Please delete and suppress the following personal information about me wherever it appears in your +database and on Spokeo-operated sites, including Thatsthem.com: +{my_identifiers} + +These items currently appear at: +{listing_urls} + +To the extent the California Consumer Privacy Act as amended by the CPRA (Cal. Civ. Code 1798.105) +applies to my personal information, please treat this as a request to delete it and to opt me out of +its sale or sharing (1798.120). Where CCPA does not apply by residency, I ask that you honor this as a +standard removal of my personal information consistent with the policy you apply to such requests. + +Please do not request more information than is necessary to locate and remove these items, and please +do not add any new identifiers to my data in the course of processing this request. Confirm completion +in writing to the email below. + +Name: {full_name} +Contact email: {contact_email} + +Sincerely, +{full_name} diff --git a/optional-skills/security/unbroker/templates/emails/gdpr-erasure.txt b/optional-skills/security/unbroker/templates/emails/gdpr-erasure.txt new file mode 100644 index 00000000000..b1b7936af98 --- /dev/null +++ b/optional-skills/security/unbroker/templates/emails/gdpr-erasure.txt @@ -0,0 +1,19 @@ +Subject: Request for erasure under GDPR Article 17 + +To the {broker_name} data protection officer, + +Under Article 17 of the EU General Data Protection Regulation (and/or the UK GDPR), I request the +erasure of all personal data you hold about me, and under Article 21 I object to its processing. + +My personal data appears at: +{listing_urls} + +Identifying details: + Name: {full_name} + Email: {contact_email} + +Please confirm erasure in writing to the email above within one month, as required by Article 12(3). +Do not request more personal data than necessary to action this request. + +Sincerely, +{full_name} diff --git a/optional-skills/security/unbroker/templates/emails/generic-optout.txt b/optional-skills/security/unbroker/templates/emails/generic-optout.txt new file mode 100644 index 00000000000..3fef1bf986a --- /dev/null +++ b/optional-skills/security/unbroker/templates/emails/generic-optout.txt @@ -0,0 +1,15 @@ +Subject: Opt-out and data removal request + +To the {broker_name} privacy team, + +I am writing to request the removal of my personal information from {broker_name} and any sites you +operate or supply. My information currently appears at: +{listing_urls} + +Please suppress and delete the record(s) associated with my name, {full_name}, and do not sell or +share my personal information. + +Please confirm completion to this email address: {contact_email} + +Thank you, +{full_name} diff --git a/tests/skills/test_unbroker_skill.py b/tests/skills/test_unbroker_skill.py new file mode 100644 index 00000000000..920d0da799b --- /dev/null +++ b/tests/skills/test_unbroker_skill.py @@ -0,0 +1,1296 @@ +"""Hermetic tests for the unbroker skill. + +Stdlib + pytest only; NO live network, NO browser, NO email. Each test runs against +an isolated temp PDD_DATA_DIR. Runnable with pytest or directly: + + python3 -m pytest tests/test_unbroker_skill.py -q + python3 tests/test_unbroker_skill.py # portable fallback runner +""" +from __future__ import annotations + +import contextlib +import os +import shutil +import sys +import tempfile +from pathlib import Path + +# Resolve the skill's scripts dir across layouts: standalone dev repo (tests/) and hermes-agent +# (tests/skills/ -> optional-skills/security/unbroker/scripts). +_HERE = Path(__file__).resolve() +_REL = ("optional-skills", "security", "unbroker", "scripts") +_CANDIDATES = [ + _HERE.parent.parent / "skill" / "scripts", # standalone dev repo + _HERE.parent.parent.joinpath(*_REL), # standalone layout + _HERE.parent.parent.parent.joinpath(*_REL), # hermes-agent (tests/skills/) +] +SCRIPTS = next((c for c in _CANDIDATES if (c / "pdd.py").exists()), _CANDIDATES[0]) +sys.path.insert(0, str(SCRIPTS)) + +import autopilot # noqa: E402 +import contextlib as _ctx # noqa: E402 +import io as _io # noqa: E402 +import json as _json # noqa: E402 +import smtplib as _smtplib # noqa: E402 +import time as _time # noqa: E402 + +import badbool # noqa: E402 +import brokers # noqa: E402 +import config # noqa: E402 +import crypto # noqa: E402 +import dossier # noqa: E402 +import email_modes # noqa: E402 +import emailer # noqa: E402 +import pdd # noqa: E402 +import legal # noqa: E402 +import ledger # noqa: E402 +import paths # noqa: E402 +import registry # noqa: E402 +import report # noqa: E402 +import storage # noqa: E402 +import tiers # noqa: E402 +import vectors # noqa: E402 + +_AGE = bool(shutil.which("age") and shutil.which("age-keygen")) + + +@contextlib.contextmanager +def temp_env(): + """Isolate every test in a fresh PDD_DATA_DIR.""" + prev = os.environ.get("PDD_DATA_DIR") + with tempfile.TemporaryDirectory() as d: + os.environ["PDD_DATA_DIR"] = str(Path(d) / "pdd") + try: + yield Path(os.environ["PDD_DATA_DIR"]) + finally: + if prev is None: + os.environ.pop("PDD_DATA_DIR", None) + else: + os.environ["PDD_DATA_DIR"] = prev + + +def _consenting(full_name="Jane Q. Public"): + return { + "subject_id": "sub_test01", + "consent": {"authorized": True, "method": "self"}, + "identity": { + "full_name": full_name, + "emails": ["jane@example.com"], + "phones": ["+1-415-555-0137"], + "date_of_birth": "1987-04-12", + "current_address": {"city": "Oakland", "state": "CA", "postal": "94601"}, + }, + "preferences": {"email_mode": "draft_only"}, + } + + +# --- config ------------------------------------------------------------------- + +def test_config_defaults_are_easiest(): + with temp_env(): + cfg = config.load_config() + assert cfg["email_mode"] == "draft_only" + assert cfg["browser_backend"] == "auto" + assert cfg["tracker_backend"] == "local-json" + assert cfg["encryption"] == "none" + + +def test_config_roundtrip_and_validation(): + with temp_env(): + config.save_config({"email_mode": "programmatic"}) + assert config.load_config()["email_mode"] == "programmatic" + try: + config.save_config({"email_mode": "bogus"}) + except ValueError: + pass + else: + raise AssertionError("invalid email_mode should raise") + + +def test_browser_clears_captcha_logic(): + assert config.browser_clears_captcha({"browser_backend": "browserbase"}) is True + assert config.browser_clears_captcha({"browser_backend": "agent-browser"}) is False + assert config.browser_clears_captcha({"browser_backend": "auto"}, env={}) is False + assert config.browser_clears_captcha({"browser_backend": "auto"}, env={"BROWSERBASE_API_KEY": "x"}) is True + + +# --- storage ------------------------------------------------------------------ + +def test_storage_json_and_jsonl_roundtrip(): + with temp_env() as data: + p = data / "x.json" + storage.write_json(p, {"a": 1}) + assert storage.read_json(p) == {"a": 1} + assert storage.read_json(data / "missing.json", []) == [] + log = data / "audit.jsonl" + storage.append_jsonl(log, {"e": 1}) + storage.append_jsonl(log, {"e": 2}) + assert [r["e"] for r in storage.read_jsonl(log)] == [1, 2] + + +# --- at-rest encryption ------------------------------------------------------- + +def test_encryption_off_writes_plaintext(): + with temp_env(): + d = _consenting() + dossier.save(d) + p = paths.dossier_path(d["subject_id"]) + assert p.exists() and not Path(str(p) + ".age").exists() + + +def test_encryption_age_round_trip(): + if not _AGE: + return # age not installed -> effectively skipped (keeps hermetic CI green) + with temp_env(): + config.save_config({"encryption": "age"}) + crypto.ensure_identity() + assert crypto.is_engaged() + d = _consenting() + dossier.save(d) + plain = paths.dossier_path(d["subject_id"]) + enc = Path(str(plain) + ".age") + assert enc.exists() and not plain.exists() # only ciphertext on disk + assert not enc.read_bytes().lstrip().startswith(b"{") # not plaintext JSON + assert dossier.load(d["subject_id"])["identity"]["full_name"] == "Jane Q. Public" + + +def test_encryption_keeps_config_and_audit_plaintext(): + if not _AGE: + return + with temp_env(): + config.save_config({"encryption": "age"}) + crypto.ensure_identity() + # config.json must stay readable plaintext (crypto reads it to decide) + assert config.load_config()["encryption"] == "age" + assert not Path(str(paths.config_path()) + ".age").exists() + # audit log holds field NAMES only, kept plaintext by design + ledger.transition("sub_test01", "spokeo", "found", found=True) + assert paths.audit_path("sub_test01").exists() + + +# --- broker DB ---------------------------------------------------------------- + +def test_seed_broker_db_loads_and_is_well_formed(): + everyone = brokers.load_all() + assert len(everyone) >= 10 + ids = {b["id"] for b in everyone} + assert {"spokeo", "whitepages", "mylife"} <= ids + for b in everyone: + assert b.get("id") and b.get("name") and b.get("priority") in {"crucial", "high", "standard", "long_tail"} + assert (b.get("optout") or {}).get("method") + + +def test_clusters_expose_ownership(): + cl = brokers.clusters() + assert "freepeopledirectory" in cl.get("spokeo", []) + assert "peoplelooker" in cl.get("beenverified", []) + + +# --- tier selection ----------------------------------------------------------- + +def test_every_broker_resolves_to_valid_tier(): + for b in brokers.load_all(): + assert tiers.select_tier(b) in {"T0", "T1", "T2", "T3"} + + +def test_email_verification_tier_shifts_with_mode(): + spokeo = brokers.get("spokeo") + assert tiers.select_tier(spokeo, "draft_only") == "T2" + assert tiers.select_tier(spokeo, "programmatic") == "T1" + assert tiers.select_tier(spokeo, "alias") == "T1" + + +def test_captcha_tier_shifts_with_browser(): + tps = brokers.get("truepeoplesearch") + assert tiers.select_tier(tps, "programmatic", browser_clears_captcha=False) == "T2" + assert tiers.select_tier(tps, "programmatic", browser_clears_captcha=True) == "T1" + + +def test_hard_human_requirements_force_t3(): + assert tiers.select_tier(brokers.get("mylife")) == "T3" # gov_id + # thatsthem's opt-out is Cloudflare-Turnstile gated (captcha:true) -> T2 without a + # captcha-clearing browser backend, T1 with one. (Corrected 2026-06-30 after the + # live scan found the real form gated; the record previously mis-declared captcha:false.) + assert tiers.select_tier(brokers.get("thatsthem")) == "T2" + assert tiers.select_tier(brokers.get("thatsthem"), browser_clears_captcha=True) == "T1" + + +def test_plan_excludes_disallowed_fields(): + d = _consenting() + actions = tiers.plan(d, brokers.load_all(), config.DEFAULT_CONFIG) + for a in actions: + assert "ssn" not in a["disclosure_fields"] + assert "profile_url" not in a["disclosure_fields"] + + +def test_disclosure_maps_street_when_broker_requires_it(): + # thatsthem's opt-out form requires a street line; select_disclosure must surface it from + # current_address.line1 (regression: 'street' was in broker inputs but unmapped, silently dropped). + d = _consenting() + d["identity"]["current_address"]["line1"] = "123 Main St" + out = dossier.select_disclosure(d, ["full_name", "street", "city", "state", "postal"]) + assert out["street"] == "123 Main St" + # and when there is no street on file, it is simply omitted (never a blank/placeholder) + d2 = _consenting() + out2 = dossier.select_disclosure(d2, ["full_name", "street", "city"]) + assert "street" not in out2 + + +def _mini_broker(bid, owns=None, requires=None, notes="", quirks=None): + return {"id": bid, "name": bid.title(), "priority": "high", + "search": {"by": ["name"]}, + "optout": {"method": "web_form", "url": f"https://{bid}.example/optout", + "requires": requires or {}, "inputs": ["full_name"], "owns": owns or [], + "notes": notes, "quirks": quirks or []}, + "owns": owns or []} + + +def test_batch_plan_groups_by_ledger_state(): + d = _consenting() + bl = [_mini_broker("aaa"), _mini_broker("bbb"), _mini_broker("ccc"), _mini_broker("ddd")] + ledger = { + "aaa": {"state": "found"}, + "bbb": {"state": "not_found"}, + "ccc": {"state": "blocked"}, + # ddd absent -> unscanned/new + } + bp = tiers.batch_plan(d, bl, config.DEFAULT_CONFIG, ledger) + assert bp["phase"] == "discover" # ddd is unscanned + assert bp["counts"]["found"] == 1 + assert bp["counts"]["not_found"] == 1 + assert bp["counts"]["blocked"] == 1 + assert bp["counts"]["unscanned"] == 1 + assert any("PHASE 1" in t for t in bp["next_actions"]) + + +def test_batch_plan_collapses_ownership_clusters(): + # a parent that is being acted on (found/submitted/...) covers its children -> child dropped + d = _consenting() + bl = [_mini_broker("parent", owns=["kid"]), _mini_broker("kid")] + ledger = {"parent": {"state": "found"}, "kid": {"state": "found"}} + bp = tiers.batch_plan(d, bl, config.DEFAULT_CONFIG, ledger) + assert bp["cluster_savings"] == {"parent": ["kid"]} + # the child must NOT also appear as its own actionable 'found' row + found_ids = [r["broker_id"] for r in bp["groups"]["found"]] + assert "parent" in found_ids and "kid" not in found_ids + + +def test_batch_plan_orders_found_parents_first(): + # found group must be sorted parents-first, most-children-first, standalone last. + d = _consenting() + bl = [_mini_broker("standalone"), + _mini_broker("smallparent", owns=["c1"]), + _mini_broker("bigparent", owns=["c1b", "c2b", "c3b"])] + ledger = {"standalone": {"state": "found"}, "smallparent": {"state": "found"}, + "bigparent": {"state": "found"}} + bp = tiers.batch_plan(d, bl, config.DEFAULT_CONFIG, ledger) + order = [r["broker_id"] for r in bp["groups"]["found"]] + assert order == ["bigparent", "smallparent", "standalone"] + # PHASE 2 tip spells out the parents-first order and points at the playbook + phase2 = [t for t in bp["next_actions"] if "PHASE 2" in t] + assert phase2 and "PARENTS FIRST" in phase2[0] and "bigparent -> smallparent" in phase2[0] + + +def test_parent_playbook_has_bespoke_and_synthesised_steps(): + d = _consenting() + bespoke = _mini_broker("bespokeparent", owns=["truthfinder", "ussearch"]) + # bespoke steps live IN the broker record (optout.playbook), not in code + bespoke["optout"]["playbook"] = ["Step one from the record", "SUPPRESSION != DELETION warning"] + bl = [bespoke, + _mini_broker("newparent", owns=["k1", "k2"], + requires={"profile_url": True, "email_verification": True}, + notes="synth note", quirks=["q1"]), + _mini_broker("standalone")] + ledger = {b["id"]: {"state": "found"} for b in bl} + bp = tiers.batch_plan(d, bl, config.DEFAULT_CONFIG, ledger) + pb = {p["broker_id"]: p for p in bp["parent_playbook"]} + # standalone (no children) is NOT in the playbook + assert "standalone" not in pb + # bespoke recipe comes verbatim from the record's own playbook + assert pb["bespokeparent"]["steps"] == bespoke["optout"]["playbook"] + # synthesised recipe: newparent reflects its requires-flags + notes + quirks + steps = " ".join(pb["newparent"]["steps"]) + assert "profile_url" in steps and "verification" in steps.lower() + assert "synth note" in steps and "q1" in steps + # ordering is stamped on each entry, parents-first + assert [p["order"] for p in bp["parent_playbook"]] == [1, 2] + + +def test_batch_plan_phase_is_delete_when_all_scanned(): + d = _consenting() + bl = [_mini_broker("aaa"), _mini_broker("bbb")] + ledger = {"aaa": {"state": "confirmed_removed"}, "bbb": {"state": "not_found"}} + bp = tiers.batch_plan(d, bl, config.DEFAULT_CONFIG, ledger) + assert bp["phase"] == "delete" # nothing unscanned + assert bp["counts"]["unscanned"] == 0 + assert bp["counts"]["done"] == 1 + + +# --- ledger / state machine --------------------------------------------------- + +def test_ledger_valid_transition_and_audit(): + with temp_env(): + sid = "sub_test01" + ledger.transition(sid, "spokeo", "searching") + case = ledger.transition(sid, "spokeo", "found", found=True) + assert case["state"] == "found" and case["found"] is True + # found -> submitted must be allowed directly (action_selected is optional) + case = ledger.transition(sid, "spokeo", "submitted") + assert case["state"] == "submitted" + audit = storage.read_jsonl(__import__("paths").audit_path(sid)) + assert any(e["to"] == "found" for e in audit) + + +def test_new_can_record_scan_outcome_directly(): + with temp_env(): + assert ledger.transition("sub_test01", "thatsthem", "found", found=True)["state"] == "found" + assert ledger.transition("sub_test01", "radaris", "not_found")["state"] == "not_found" + # a scan that is bot-blocked on the very first hit must be recordable as blocked directly + # (no need to pass through 'searching' first) -- and not_found -> blocked when a re-scan is gated + assert ledger.transition("sub_test01", "spokeo", "blocked")["state"] == "blocked" + assert ledger.transition("sub_test01", "radaris", "blocked")["state"] == "blocked" + # a blocked site later scanned via the operator's own (residential) browser resolves to a + # real verdict, incl. not_found -- blocked -> not_found must be legal. + assert ledger.transition("sub_test01", "spokeo", "not_found")["state"] == "not_found" + + +def test_indirect_exposure_state_and_transitions(): + with temp_env(): + sid = "sub_test01" + # a scan can land directly on indirect_exposure (PII on a relative's record) + case = ledger.transition(sid, "thatsthem", "indirect_exposure", + evidence={"summary": "email on relative record"}) + assert case["state"] == "indirect_exposure" + # the lever from there is a targeted delete-my-PII request (-> submitted) + assert ledger.transition(sid, "thatsthem", "submitted")["state"] == "submitted" + # and a separate broker: not_found -> indirect_exposure is allowed (found on re-read) + ledger.transition(sid, "radaris", "not_found") + assert ledger.transition(sid, "radaris", "indirect_exposure")["state"] == "indirect_exposure" + # re-scan can clear it + assert ledger.transition(sid, "radaris", "not_found")["state"] == "not_found" + + +def test_ledger_illegal_transition_raises(): + with temp_env(): + try: + ledger.transition("sub_test01", "spokeo", "confirmed_removed") # new -> confirmed_removed + except ValueError: + pass + else: + raise AssertionError("illegal transition should raise") + + +def test_ledger_disclosure_log(): + with temp_env(): + ledger.log_disclosure("sub_test01", "spokeo", ["full_name", "contact_email"], "web_form") + case = ledger.get_case("sub_test01", "spokeo") + assert case["disclosure_log"][0]["fields"] == ["contact_email", "full_name"] + + +# --- dossier / consent / least-disclosure ------------------------------------ + +def test_consent_gate(): + assert dossier.is_authorized(_consenting()) is True + nope = _consenting() + nope["consent"] = {"authorized": False, "method": "self"} + assert dossier.is_authorized(nope) is False + try: + dossier.require_authorized(nope) + except PermissionError: + pass + else: + raise AssertionError("require_authorized should raise for non-consenting subject") + + +def test_least_disclosure_selection(): + d = _consenting() + got = dossier.select_disclosure(d, ["full_name", "contact_email", "profile_url", "ssn", "date_of_birth"]) + assert set(got) == {"full_name", "contact_email", "date_of_birth"} + assert "ssn" not in got and "profile_url" not in got + + +def test_designated_contact_email_overrides_first(): + d = _consenting() + d["identity"]["emails"] = ["first@x.com", "alias@x.com"] + assert dossier.contact_email(d) == "first@x.com" + d["preferences"]["contact_email_for_optouts"] = "alias@x.com" + assert dossier.contact_email(d) == "alias@x.com" + + +# --- alternates / search vectors --------------------------------------------- + +def test_all_names_and_locations_dedupe(): + d = _consenting() + d["identity"]["also_known_as"] = ["Jane Public", "Jane Q. Public"] # 2nd dups primary + d["identity"]["prior_addresses"] = [{"city": "Berkeley", "state": "CA"}, {"city": "Oakland", "state": "CA"}] + assert dossier.all_names(d) == ["Jane Q. Public", "Jane Public"] + assert [loc["city"] for loc in dossier.all_locations(d)] == ["Oakland", "Berkeley"] # current first, deduped + + +def test_search_vectors_fan_out_across_alternates(): + d = _consenting() + d["identity"]["also_known_as"] = ["Jane Smith"] + d["identity"]["prior_addresses"] = [{"city": "Berkeley", "state": "CA"}] + d["identity"]["emails"] = ["a@x.com", "b@y.com"] + d["identity"]["phones"] = ["+1-415-555-0137", "+1-510-555-0199"] + broker = {"id": "x", "search": {"by": ["name", "phone", "email", "address"]}} + v = vectors.search_vectors(d, broker) + assert len([x for x in v if x["by"] == "name"]) == 4 # 2 names x 2 locations + assert len([x for x in v if x["by"] == "phone"]) == 2 + assert len([x for x in v if x["by"] == "email"]) == 2 + assert len([x for x in v if x["by"] == "address"]) == 0 # no street line1 yet + + +def test_search_vectors_respect_broker_capabilities(): + d = _consenting() + d["identity"]["emails"] = ["a@x.com"] + v = vectors.search_vectors(d, {"id": "y", "search": {"by": ["name"]}}) + assert v and all(x["by"] == "name" for x in v) # broker can't search email -> no email vectors + + +def test_search_vectors_address_needs_line1(): + d = _consenting() + d["identity"]["current_address"] = {"line1": "123 Main St", "city": "Oakland", "state": "CA", "postal": "94601"} + v = vectors.search_vectors(d, {"id": "z", "search": {"by": ["address"]}}) + assert len(v) == 1 and v[0]["by"] == "address" and v[0]["query"]["line1"] == "123 Main St" + + +# --- opaque ids / fan-out / antibot ------------------------------------------ + +def test_subject_id_is_opaque_no_name_leak(): + sid = dossier.new_subject_id("Maiden Married Person") + assert sid.startswith("sub_") + assert "maiden" not in sid.lower() and "person" not in sid.lower() + assert dossier.new_subject_id("Maiden Married Person") != sid # not derived from the name + + +def test_fanout_batches_large_runs(): + g = tiers.fanout([{"id": f"b{i}"} for i in range(20)], batch_size=8) + assert g["broker_count"] == 20 and g["should_fanout"] is True + assert len(g["batches"]) == 3 and g["batches"][0] == [f"b{i}" for i in range(8)] + small = tiers.fanout([{"id": "x"}, {"id": "y"}], batch_size=8) + assert small["should_fanout"] is False and small["batches"] == [["x", "y"]] + + +def test_plan_surfaces_antibot(): + d = _consenting() + broker = {"id": "tps", "optout": {"requires": {}}, "search": {"antibot": "datadome", "by": ["name"]}} + actions = tiers.plan(d, [broker], config.DEFAULT_CONFIG) + assert actions[0]["antibot"] == "datadome" + + +def test_plan_surfaces_optout_quirks_and_email(): + d = _consenting() + broker = {"id": "radaris", "search": {"by": ["name"]}, + "optout": {"requires": {}, "email": "x@broker.test", "quirks": ["no profile URL -> email fallback"]}} + a = tiers.plan(d, [broker], config.DEFAULT_CONFIG)[0] + assert a["optout_email"] == "x@broker.test" + assert a["optout_quirks"] == ["no profile URL -> email fallback"] + + +# --- legal / templates -------------------------------------------------------- + +def test_legal_render_keeps_missing_placeholders_literal(): + out = legal.render("emails/generic-optout.txt", {"broker_name": "Spokeo"}) + assert "Spokeo" in out + assert "{full_name}" in out # missing field left literal, never blank-injected + + +def test_render_optout_email_includes_listing_and_name(): + b = brokers.get("spokeo") + out = legal.render_optout_email(b, {"full_name": "Jane Q. Public", + "contact_email": "jane@example.com", + "listing_urls": ["https://www.spokeo.com/jane"]}) + assert "Jane Q. Public" in out and "https://www.spokeo.com/jane" in out + + +def test_render_ccpa_indirect_request_names_only_own_identifiers(): + b = brokers.get("thatsthem") + out = legal.render_request("ccpa_indirect", b, { + "full_name": "Jane Q. Public", + "contact_email": "jane@example.com", + "my_identifiers": ["jane@example.com", 'the name "Jane Q. Public" where it appears as a relative'], + "listing_urls": ["https://thatsthem.com/email/jane@example.com"], + }) + # the request must frame this as the subject's OWN data on someone else's record + assert "not the primary subject" in out + assert "jane@example.com" in out + assert "https://thatsthem.com/email/jane@example.com" in out + # must NOT use the full-opt-out wording that claims the record is about the subject + assert "DELETE all personal information you hold about me" not in out + + +# --- email verification-link extraction -------------------------------------- + +def test_extract_verification_link_prefers_broker_optout_link(): + body = ("Hello,\nClick https://www.spokeo.com/optout/confirm?token=abc to confirm.\n" + "Unrelated: https://ads.example/promo\n") + link = email_modes.extract_verification_link(body, brokers.get("spokeo")) + assert link is not None and "spokeo.com" in link and "ads.example" not in link + + +def test_extract_verification_link_ignores_unrelated_only(): + assert email_modes.extract_verification_link("see https://example.com/news today") is None + + +# --- BADBOOL live-pull parser ------------------------------------------------- + +BADBOOL_FIXTURE = """ +## Search Engines +### Google +This is not a broker; ignore it. + +## People Search Sites + +### \U0001F490 BeenVerified +Find your information and opt out of [people search](https://www.beenverified.com/app/optout/search). + +### \U0001F490 \U0001F4DE MyLife +[Find your information](https://www.mylife.com), and then [opt out](https://www.mylife.com/privacyrequest). + +### \U0001F3AB PimEyes +To opt out, [upload an ID](https://pimeyes.com/en/opt-out-request-form). + +## Special Circumstances +### Not A Broker +Ignore this section entirely. +""" + + +def test_badbool_parses_people_search_section_only(): + recs = badbool.parse(BADBOOL_FIXTURE) + ids = {r["id"] for r in recs} + assert ids == {"beenverified", "mylife", "pimeyes"} # google + notabroker excluded + bv = next(r for r in recs if r["id"] == "beenverified") + assert bv["priority"] == "crucial" + assert "beenverified.com/app/optout" in (bv["optout"]["url"] or "") + assert bv["source"] == "BADBOOL-auto" and bv["confidence"] == "auto" + + +def test_badbool_symbols_map_to_requirements_and_tiers(): + recs = {r["id"]: r for r in badbool.parse(BADBOOL_FIXTURE)} + assert recs["mylife"]["optout"]["requires"]["phone_voice"] is True + assert recs["mylife"]["optout"]["method"] == "phone" + assert tiers.select_tier(recs["mylife"]) == "T3" + assert recs["pimeyes"]["optout"]["requires"]["gov_id"] is True + assert tiers.select_tier(recs["pimeyes"]) == "T3" + + +def test_badbool_merge_keeps_curated_and_adds_new(): + with temp_env(): + badbool.refresh(__import__("paths").brokers_cache_path(), markdown=BADBOOL_FIXTURE) + merged = {b["id"]: b for b in brokers.load_all()} + # curated record wins over the live one + assert merged["beenverified"]["source"] == "BADBOOL" + # a non-curated live record is added with auto confidence + assert "pimeyes" in merged and merged["pimeyes"]["confidence"] == "auto" + + +# --- report ------------------------------------------------------------------- + +def test_status_counts_and_markdown(): + with temp_env(): + sid = "sub_test01" + ledger.transition(sid, "spokeo", "searching") + ledger.transition(sid, "spokeo", "found") + ledger.transition(sid, "thatsthem", "searching") + ledger.transition(sid, "thatsthem", "not_found") + counts = report.status_counts(sid) + assert counts.get("found") == 1 and counts.get("not_found") == 1 + md = report.render_markdown(sid) + assert "status for" in md and "Count" in md + + +# --- autonomy: auto-configure --------------------------------------------------------------- + +def test_autonomy_default_is_full_and_valid(): + with temp_env(): + assert config.load_config()["autonomy"] == "full" + config.save_config({"autonomy": "assisted"}) + assert config.load_config()["autonomy"] == "assisted" + try: + config.save_config({"autonomy": "yolo"}) + except ValueError: + pass + else: + raise AssertionError("invalid autonomy should raise") + + +def test_auto_configure_picks_most_autonomous(): + with temp_env(): + # bare env -> draft_only floor, auto browser (still fully hands-off policy-wise) + cfg = config.auto_configure(env={}) + assert cfg["autonomy"] == "full" + assert cfg["email_mode"] == "draft_only" + assert cfg["browser_backend"] == "auto" + # SMTP creds -> programmatic email; Browserbase key -> cloud browser + cfg = config.auto_configure(env={"EMAIL_ADDRESS": "agent@gmail.com", + "EMAIL_PASSWORD": "app-pass", + "BROWSERBASE_API_KEY": "bb"}) + assert cfg["email_mode"] == "programmatic" + assert cfg["browser_backend"] == "browserbase" + # AgentMail only -> alias mode + assert config.auto_configure(env={"AGENTMAIL_API_KEY": "am"})["email_mode"] == "alias" + # encryption auto-on exactly when age is installed (free privacy, zero human cost) + assert config.auto_configure(env={})["encryption"] == ("age" if _AGE else "none") + + +# --- emailer: programmatic send + verification polling -------------------------------------- + +def test_emailer_settings_inference_and_floor(): + assert emailer.smtp_settings(env={}) is None + assert emailer.imap_settings(env={}) is None + env = {"EMAIL_ADDRESS": "a@gmail.com", "EMAIL_PASSWORD": "p"} + assert emailer.smtp_settings(env)["host"] == "smtp.gmail.com" + assert emailer.smtp_settings(env)["port"] == 587 + assert emailer.imap_settings(env)["host"] == "imap.gmail.com" + assert emailer.imap_settings(env)["port"] == 993 + # unknown provider without an explicit host -> NOT configured (never guess blind) + corp = {"EMAIL_ADDRESS": "a@corp.example", "EMAIL_PASSWORD": "p"} + assert emailer.smtp_settings(corp) is None + s = emailer.smtp_settings({**corp, "EMAIL_SMTP_HOST": "mail.corp.example", + "EMAIL_SMTP_PORT": "465"}) + assert (s["host"], s["port"]) == ("mail.corp.example", 465) + + +class _FakeSMTP: + sent: list = [] + + def __init__(self, host, port, timeout=None): + self.host, self.port = host, port + + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + def ehlo(self): + pass + + def starttls(self): + pass + + def login(self, user, password): + self.user = user + + def send_message(self, msg): + _FakeSMTP.sent.append(msg) + + +def test_emailer_send_locks_recipient_to_broker(): + env = {"EMAIL_ADDRESS": "agent@gmail.com", "EMAIL_PASSWORD": "p"} + broker = {"id": "radaris", "optout": {"email": "privacy@radaris.example"}} + _FakeSMTP.sent = [] + out = emailer.send(broker, "Subject: Remove my listing\n\nBody here", env=env, + _smtp_factory=_FakeSMTP) + assert out["to"] == "privacy@radaris.example" + assert _FakeSMTP.sent[0]["Subject"] == "Remove my listing" + assert "Body here" in _FakeSMTP.sent[0].get_content() + # arbitrary recipients are refused -- this tool cannot be repurposed to email people + try: + emailer.send(broker, "Subject: x\n\nb", to="victim@example.com", env=env, + _smtp_factory=_FakeSMTP) + except PermissionError: + pass + else: + raise AssertionError("non-broker recipient must be refused") + + +def test_emailer_send_requires_config_and_broker_address(): + broker = {"id": "x", "optout": {"email": "privacy@x.example"}} + try: + emailer.send(broker, "Subject: s\n\nb", env={}) + except RuntimeError: + pass + else: + raise AssertionError("unconfigured SMTP must raise (draft fallback, not a crash)") + try: + emailer.send({"id": "y", "optout": {}}, "Subject: s\n\nb", + env={"EMAIL_ADDRESS": "a@gmail.com", "EMAIL_PASSWORD": "p"}) + except RuntimeError: + pass + else: + raise AssertionError("broker without a declared address must raise") + + +def test_browser_send_payload_is_recipient_locked(): + broker = {"id": "radaris", "optout": {"email": "privacy@radaris.example"}} + p = emailer.browser_send_payload(broker, "Subject: Remove my listing\n\nBody here") + assert p["to"] == "privacy@radaris.example" + assert p["subject"] == "Remove my listing" and "Body here" in p["body"] + # the browser lane refuses arbitrary recipients too (same guard as SMTP send) + try: + emailer.browser_send_payload(broker, "Subject: x\n\nb", to="victim@example.com") + except PermissionError: + pass + else: + raise AssertionError("browser lane must refuse a non-broker recipient") + + +def test_browser_email_mode_is_autonomous_without_smtp_or_imap(): + with temp_env(): + assert config.save_config({"email_mode": "browser"}) # mode is valid + persists + d = _consenting() + d["residency_jurisdiction"] = "US-CA" + mailer = _mini_broker("mailer") + mailer["optout"]["method"] = "email" + mailer["optout"]["email"] = "privacy@mailer.example" + verifier = _mini_broker("verifier", requires={"email_verification": True}) + led = {"mailer": {"state": "found"}, + "verifier": {"broker_id": "verifier", "state": "submitted"}} + # browser mode with NO EMAIL_* creds -> still fully autonomous (agent uses webmail) + q = autopilot.next_actions(d, [mailer, verifier], _auto_cfg(email_mode="browser"), led, env={}) + sends = [a for a in q["actions"] if a["type"] == "optout_email_send"] + assert sends and sends[0]["send_via"] == "browser" and sends[0]["to"] == "privacy@mailer.example" + polls = [a for a in q["actions"] if a["type"] == "poll_verification"] + assert polls and polls[0]["via"] == "browser" + assert not q["human_digest"] # browser mode needs no human for these + + +def test_verification_link_from_messages_is_domain_scoped(): + broker = {"id": "spokeo", "name": "Spokeo", + "search": {"url": "https://www.spokeo.com/"}, + "optout": {"url": "https://www.spokeo.com/optout"}} + phish = {"from": "phisher@evil.example", "subject": "verify now", + "text": "click https://evil.example/optout/verify?x=1"} + real = {"from": "no-reply@spokeo.com", "subject": "Confirm your opt out", + "text": "Confirm here: https://www.spokeo.com/optout/verify/abc123"} + hit = emailer.link_from_messages([phish, real], broker) + assert hit["link"] == "https://www.spokeo.com/optout/verify/abc123" + # a phishing-only inbox yields nothing (domain scoping + link scoring) + assert emailer.link_from_messages([phish], broker) is None + + +# --- ledger: follow-up scheduling + due queue ------------------------------------------------ + +def test_verification_pending_to_awaiting_processing_is_legal(): + with temp_env(): + sid = "sub_test01" + ledger.transition(sid, "intelius", "found", found=True) + ledger.transition(sid, "intelius", "submitted") + ledger.transition(sid, "intelius", "verification_pending") + assert ledger.transition(sid, "intelius", "awaiting_processing")["state"] == "awaiting_processing" + + +def test_followup_stamps_and_due_queue(): + broker = {"optout": {"est_processing_days": 10}} + d = {"preferences": {"rescan_interval_days": 30}} + f_sub = ledger.followup_fields("submitted", broker, d) + assert "next_recheck_at" in f_sub + f_done = ledger.followup_fields("confirmed_removed", broker, d) + assert "removal_confirmed_at" in f_done + assert f_done["next_recheck_at"] > f_sub["next_recheck_at"] # 30d rescan > 10d processing + assert ledger.followup_fields("found", broker, d) == {} # scan verdicts get no stamp + led = { + "a": {"broker_id": "a", "state": "awaiting_processing", "next_recheck_at": "2000-01-01T00:00:00Z"}, + "b": {"broker_id": "b", "state": "confirmed_removed", "next_recheck_at": "2999-01-01T00:00:00Z"}, + } + assert [c["broker_id"] for c in ledger.due("sub_x", ledger=led)] == ["a"] + + +def test_badbool_auto_records_have_processing_estimate(): + recs = badbool.parse("## People Search Sites\n### Example\n[opt out](https://example.com/optout)\n") + assert recs[0]["optout"]["est_processing_days"] == 14 # drives next_recheck_at for live records + + +# --- autopilot: the autonomous action queue -------------------------------------------------- + +def _auto_cfg(**over): + cfg = dict(config.DEFAULT_CONFIG) + cfg.update(over) + return cfg + + +def test_next_actions_scan_first_then_optouts_parents_first(): + with temp_env(): + d = _consenting() + bl = [_mini_broker("parent", owns=["kid"]), _mini_broker("kid"), _mini_broker("solo")] + q = autopilot.next_actions(d, bl, _auto_cfg(), {}, env={}) + types = [a["type"] for a in q["actions"]] + assert "scan_inline" in types + assert not any(t.startswith("optout") for t in types) # never act before the crawl + assert q["phase"] == "discover" + led = {"parent": {"state": "found"}, "kid": {"state": "found"}, "solo": {"state": "found"}} + q2 = autopilot.next_actions(d, bl, _auto_cfg(), led, env={}) + opt = [a for a in q2["actions"] if a["type"] == "optout_web_form"] + assert [a["broker_id"] for a in opt] == ["parent", "solo"] # kid covered by parent + assert q2["phase"] == "delete" + + +def test_next_actions_fanout_above_threshold(): + with temp_env(): + d = _consenting() + bl = [_mini_broker(f"b{i:02d}") for i in range(12)] + q = autopilot.next_actions(d, bl, _auto_cfg(), {}, env={}) + assert any(a["type"] == "fanout_scan" for a in q["actions"]) + + +def test_next_actions_routes_human_only_to_digest(): + with temp_env(): + d = _consenting() + t3 = _mini_broker("faxer", requires={"fax": True}) + cb = _mini_broker("callbacker", requires={"phone_callback": True}) + led = {"faxer": {"state": "found"}, "callbacker": {"state": "found"}} + q = autopilot.next_actions(d, [t3, cb], _auto_cfg(), led, env={}) + assert not any(a["type"].startswith("optout") for a in q["actions"]) + reasons = " ".join(t["reason"] for t in q["human_digest"]) + assert "human-only" in reasons and "phone-callback" in reasons + + +def test_next_actions_email_send_vs_draft_digest(): + with temp_env(): + d = _consenting() + b = _mini_broker("mailer") + b["optout"]["method"] = "email" + b["optout"]["email"] = "privacy@mailer.example" + led = {"mailer": {"state": "found"}} + env = {"EMAIL_ADDRESS": "agent@gmail.com", "EMAIL_PASSWORD": "p"} + q = autopilot.next_actions(d, [b], _auto_cfg(email_mode="programmatic"), led, env=env) + assert any(a["type"] == "optout_email_send" for a in q["actions"]) + # draft mode: same case becomes a digest entry with the render command as agent prep + q2 = autopilot.next_actions(d, [b], _auto_cfg(), led, env={}) + assert not any(a["type"] == "optout_email_send" for a in q2["actions"]) + assert any("render-email" in " ".join(t["agent_prep"]) for t in q2["human_digest"]) + + +def test_next_actions_poll_verification_and_due_rechecks(): + with temp_env(): + d = _consenting() + b = _mini_broker("verifier", requires={"email_verification": True}) + led = { + "verifier": {"broker_id": "verifier", "state": "submitted"}, + "done1": {"broker_id": "done1", "state": "confirmed_removed", + "next_recheck_at": "2000-01-01T00:00:00Z"}, + } + env = {"EMAIL_ADDRESS": "agent@gmail.com", "EMAIL_PASSWORD": "p"} + q = autopilot.next_actions(d, [b, _mini_broker("done1")], + _auto_cfg(email_mode="programmatic"), led, env=env) + types = [a["type"] for a in q["actions"]] + assert "poll_verification" in types and "verify_removal" in types + # without IMAP, the verification click becomes a human digest entry instead + q2 = autopilot.next_actions(d, [b], _auto_cfg(), + {"verifier": {"broker_id": "verifier", "state": "submitted"}}, env={}) + assert not any(a["type"] == "poll_verification" for a in q2["actions"]) + assert any("verification email" in t["reason"] for t in q2["human_digest"]) + + +def test_next_actions_blocked_stealth_or_operator_browser(): + with temp_env(): + d = _consenting() + b = _mini_broker("gated") + led = {"gated": {"state": "blocked"}} + q = autopilot.next_actions(d, [b], _auto_cfg(), led, env={"BROWSERBASE_API_KEY": "bb"}) + assert any(a["type"] == "stealth_rescan" for a in q["actions"]) + q2 = autopilot.next_actions(d, [b], _auto_cfg(), led, env={}) + assert any("anti-bot" in t["reason"] for t in q2["human_digest"]) + + +def test_assisted_mode_flags_confirm_first(): + with temp_env(): + d = _consenting() + b = _mini_broker("solo") + led = {"solo": {"state": "found"}} + q = autopilot.next_actions(d, [b], _auto_cfg(autonomy="assisted"), led, env={}) + opt = [a for a in q["actions"] if a["type"] == "optout_web_form"] + assert opt and all(a["confirm_first"] for a in opt) + q2 = autopilot.next_actions(d, [b], _auto_cfg(), led, env={}) + assert all(not a["confirm_first"] for a in q2["actions"] if a["type"] == "optout_web_form") + + +def test_next_actions_refresh_then_done_flags(): + with temp_env(): + d = _consenting() + bl = [_mini_broker("solo")] + led = {"solo": {"state": "not_found"}} + q = autopilot.next_actions(d, bl, _auto_cfg(), led, env={}) + assert any(a["type"] == "refresh_brokers" for a in q["actions"]) # no cache yet + assert q["done_for_now"] is False + storage.write_json(paths.brokers_cache_path(), []) # fresh cache + q2 = autopilot.next_actions(d, bl, _auto_cfg(), led, env={}) + assert q2["actions"] == [] + assert q2["done_for_now"] and q2["fully_done"] + + +def test_parked_and_reappeared_states_group_correctly(): + # Regression: human_task_queued / action_selected / reappeared used to fall into "unscanned", + # so the autonomous loop would try to re-scan parked or already-actioned cases forever. + with temp_env(): + d = _consenting() + bl = [_mini_broker("parked"), _mini_broker("chosen"), _mini_broker("back")] + led = {"parked": {"state": "human_task_queued"}, + "chosen": {"state": "action_selected"}, + "back": {"state": "reappeared"}} + bp = tiers.batch_plan(d, bl, config.DEFAULT_CONFIG, led) + assert bp["counts"]["unscanned"] == 0 + assert bp["phase"] == "delete" + assert [r["broker_id"] for r in bp["groups"]["human"]] == ["parked"] + assert {r["broker_id"] for r in bp["groups"]["found"]} == {"chosen", "back"} + q = autopilot.next_actions(d, bl, _auto_cfg(), led, env={}) + assert not any(a["type"] in ("scan_inline", "fanout_scan") for a in q["actions"]) + assert {a["broker_id"] for a in q["actions"] if a["type"] == "optout_web_form"} == {"chosen", "back"} + + +# --- cluster parents: verified deletion lanes + data-driven playbooks ------------------------ + +def test_cluster_parents_have_playbook_and_deletion_lane(): + """Contract: every curated cluster parent must know EXACTLY how to remove the data. + + A parent record (owns children) must carry a non-empty field-verified optout.playbook + and a structured deletion lane -- deletion beats suppression, and the knowledge lives + in the record, not in code. + """ + for b in brokers._load_curated(): + if not b.get("owns"): + continue + opt = b.get("optout") or {} + bid = b["id"] + assert opt.get("playbook"), f"{bid}: cluster parent missing optout.playbook" + d = opt.get("deletion") or {} + assert d.get("email") or d.get("via"), f"{bid}: cluster parent missing deletion lane" + # every declared email must be a legal send-email recipient + for addr in [opt.get("email"), d.get("email")]: + if addr: + assert addr in emailer.broker_addresses(b), f"{bid}: {addr} not sendable" + + +def test_curated_intelius_playbook_prefers_deletion(): + b = brokers.get("intelius") + steps = " ".join(b["optout"]["playbook"]) + assert "SUPPRESSION != DELETION" in steps # the trap, encoded in the data + assert "DELETE MY USER DATA" in steps # the actual deletion control + assert "privacy@peopleconnect.us" in steps # the email fallback lane + assert b["optout"]["deletion"]["via"] == "in_flow" + + +def test_curated_whitepages_email_lane_is_autonomous(): + """The verified Whitepages pattern: privacyrequest@ bypasses the phone-callback tool.""" + b = brokers.get("whitepages") + opt = b["optout"] + assert opt["method"] == "email" + assert opt["email"] == "privacyrequest@whitepages.com" + assert opt["requires"]["phone_callback"] is False # the callback is only the ALT tool + # programmatic email -> fully automated (T1); draft mode -> needs a human for the verify loop + assert tiers.select_tier(b, email_mode="programmatic") == "T1" + assert tiers.select_tier(b, email_mode="draft_only") == "T2" + + +def test_request_kind_is_residency_honest(): + ca = {"residency_jurisdiction": "US-CA"} + tx = {"residency_jurisdiction": "US-TX"} + de = {"residency_jurisdiction": "EU-DE"} + assert autopilot.request_kind(ca) == "ccpa" + assert autopilot.request_kind(tx) == "generic" # never claim CCPA for a non-CA resident + assert autopilot.request_kind(de) == "gdpr" + assert autopilot.request_kind({}) == "generic" + # broker restriction can force DOWN to generic but never upgrade + assert autopilot.request_kind(tx, allowed=["ccpa", "generic"]) == "generic" + assert autopilot.request_kind(ca, allowed=["generic"]) == "generic" + assert autopilot.request_kind(ca, allowed=["ccpa", "generic"]) == "ccpa" + + +def test_email_lane_routing_and_rescue(): + with temp_env(): + d = _consenting() + d["residency_jurisdiction"] = "US-CA" + env = {"EMAIL_ADDRESS": "agent@gmail.com", "EMAIL_PASSWORD": "p"} + + # (a) primary email method -> email send action with residency-correct kind + mailer = _mini_broker("mailer") + mailer["optout"]["method"] = "email" + mailer["optout"]["email"] = "privacy@mailer.example" + # (b) RESCUE: T3 (gov_id) form but a deletion email exists (no via preference) -> + # email lane instead of the human digest + hard = _mini_broker("hardsite", requires={"gov_id": True}) + hard["optout"]["deletion"] = {"email": "privacy@hardsite.example", + "kinds": ["ccpa", "generic"]} + # (c) phone-callback form with deletion email -> email lane too + cb = _mini_broker("callback2", requires={"phone_callback": True}) + cb["optout"]["deletion"] = {"email": "privacy@callback2.example"} + led = {b: {"state": "found"} for b in ("mailer", "hardsite", "callback2")} + q = autopilot.next_actions(d, [mailer, hard, cb], + _auto_cfg(email_mode="programmatic"), led, env=env) + sends = {a["broker_id"]: a for a in q["actions"] if a["type"] == "optout_email_send"} + assert set(sends) == {"mailer", "hardsite", "callback2"} + assert sends["mailer"]["kind"] == "ccpa" # CA resident + assert sends["hardsite"]["to"] == "privacy@hardsite.example" + assert "rescue" in sends["hardsite"]["why"] + assert not q["human_digest"] # nothing left for a human + + # without SMTP the same brokers fall back honestly: email draft digest / human digest + q2 = autopilot.next_actions(d, [mailer, hard, cb], _auto_cfg(), led, env={}) + assert not any(a["type"] == "optout_email_send" for a in q2["actions"]) + assert len(q2["human_digest"]) == 3 + + +def test_send_email_accepts_deletion_lane_recipient(): + env = {"EMAIL_ADDRESS": "agent@gmail.com", "EMAIL_PASSWORD": "p"} + broker = {"id": "hardsite", + "optout": {"deletion": {"email": "privacy@hardsite.example"}}} + _FakeSMTP.sent = [] + out = emailer.send(broker, "Subject: Delete my data\n\nBody", env=env, _smtp_factory=_FakeSMTP) + assert out["to"] == "privacy@hardsite.example" + + +# --- human-task digest ------------------------------------------------------------------------ + +def test_human_tasks_digest_markdown(): + with temp_env(): + sid = "sub_test01" + ledger.transition(sid, "mylife", "found", found=True) + ledger.transition(sid, "mylife", "human_task_queued", + human_task_reason="gov ID demanded") + ledger.transition(sid, "fastpeoplesearch", "blocked") + md = report.human_tasks_markdown(sid) + assert "gov ID demanded" in md + assert "Withhold" in md + assert "fastpeoplesearch" in md.lower() + # empty ledger -> explicitly says nothing is needed + assert "Nothing needs a human" in report.human_tasks_markdown("sub_other") + + +# --- CA data broker registry (coverage breadth: DROP + email lane) --------------------------- + +def _registry_csv(): + """Mimic the CA registry CSV: junk row 0, label row 1 (with the real NBSP), data rows.""" + import csv as _csv + import io as _io + buf = _io.StringIO() + w = _csv.writer(buf) + w.writerow(["", "junk header the site hides", "", "", "", ""]) + w.writerow(["Data broker\xa0name:", "Doing Business As (DBA), if applicable:", + "Data broker primary website:", "Data broker primary contact email address:", + "Data broker's primary website that contains details on how consumers can exercise " + "their CA Consumer Privacy Act rights, including how to delete their personal information:", + "The data broker or any of its subsidiaries is regulated by the federal Fair Credit " + "Reporting Act (FCRA):"]) + w.writerow(["Acme Data LLC", "AcmeDBA", "https://acme.example", + "privacy@acme.example", "https://acme.example/ccpa", "No"]) + w.writerow(["Credit Bureau Co", "", "https://cbc.example", + "privacy@cbc.example", "https://cbc.example/rights", "Yes"]) + return buf.getvalue() + + +def test_registry_parses_ca_csv(): + recs = registry.parse(_registry_csv()) + assert len(recs) == 2 + assert len({r["id"] for r in recs}) == 2 # unique ids + acme = next(r for r in recs if "acme" in r["id"]) + cbc = next(r for r in recs if "cbc" in r["id"] or "credit" in r["id"]) + assert acme["optout"]["method"] == "email" + assert acme["optout"]["email"] == "privacy@acme.example" + assert acme["optout"]["deletion"]["via"] == "drop" # worked via DROP, not scanning + assert acme["confidence"] == "registry" + assert acme["category"] == "data_broker" + assert acme["optout"]["fcra"] is False and cbc["optout"]["fcra"] is True + + +def test_registry_refresh_isolated_from_people_search(): + with temp_env(): + res = registry.refresh(paths.registry_cache_path(), csv_text=_registry_csv()) + assert res["parsed"] == 2 and res["fcra_regulated"] == 1 + reg_ids = {r["id"] for r in brokers.load_registry_cache()} + assert len(reg_ids) == 2 + # CRITICAL: registry brokers must NOT leak into the people-search scan pipeline + assert reg_ids.isdisjoint({b["id"] for b in brokers.load_all()}) + + +def test_registry_multi_source_framework(): + # generic parser works for a non-CA state (proving multi-source, not CA-hardcoded) + vt = registry.parse(_registry_csv(), jurisdiction="US-VT", has_drop=False) + assert vt[0]["jurisdictions"] == ["US-VT"] + assert vt[0]["source"] == "VT-registry" + assert vt[0]["optout"]["deletion"]["via"] == "email" # no DROP outside CA + assert "no one-shot" in vt[0]["optout"]["deletion"]["notes"].lower() + # VT/OR/TX are surfaced as portals with official URLs (not fabricated rows) + ports = {p["jurisdiction"]: p for p in registry.portals()} + assert set(ports) == {"US-VT", "US-OR", "US-TX"} + assert all(p["url"].startswith("http") for p in ports.values()) + + +def test_registry_refresh_all_ingests_csv_and_lists_portals(): + with temp_env(): + res = registry.refresh_all(paths.registry_cache_path(), fetched={"ca": _registry_csv()}) + assert res["total"] == 2 + assert res["sources"]["ca"]["parsed"] == 2 and res["sources"]["ca"]["added_after_dedupe"] == 2 + assert res["sources"]["vt"]["format"] == "portal" # no bulk export, surfaced as portal + assert len(res["portals"]) == 3 + assert len(brokers.load_registry_cache()) == 2 + + +def test_next_surfaces_drop_for_ca_resident_only(): + with temp_env(): + registry.refresh(paths.registry_cache_path(), csv_text=_registry_csv()) + bl = [_mini_broker("solo")] + + ca = _consenting() + ca["residency_jurisdiction"] = "US-CA" + q = autopilot.next_actions(ca, bl, _auto_cfg(), {}, env={}) + assert any(a["type"] == "drop_submit" for a in q["actions"]) + assert q["coverage"]["registered_data_brokers"] == 2 + assert q["coverage"]["worked_via"] == "CA DROP one-shot" + + tx = _consenting() + tx["residency_jurisdiction"] = "US-TX" + q2 = autopilot.next_actions(tx, bl, _auto_cfg(), {}, env={}) + assert not any(a["type"] == "drop_submit" for a in q2["actions"]) + assert q2["coverage"]["worked_via"] == "targeted CCPA/GDPR email" + + ca["preferences"]["drop_filed_at"] = "2026-01-01T00:00:00Z" + q3 = autopilot.next_actions(ca, bl, _auto_cfg(), {}, env={}) + assert not any(a["type"] == "drop_submit" for a in q3["actions"]) + + +# --- hardening: locking / rate-limit / retry / idempotency / freshness / metrics ------------ + +def test_storage_lock_mutual_exclusion_and_stale_break(): + with temp_env() as data: + target = data / "x.json" + with storage.locked(target): # hold the lock + try: + with storage.locked(target, timeout=0.2): # second acquire must time out + raise AssertionError("second acquire should have timed out") + except TimeoutError: + pass + with storage.locked(target, timeout=0.2): # released -> acquires fine + pass + # a stale lock (old mtime) from a crashed writer gets broken + lock = target.with_name(target.name + ".lock") + lock.write_text("999999") + old = _time.time() - 120 + os.utime(lock, (old, old)) + with storage.locked(target, timeout=0.2, stale=30): + pass + + +def test_email_rate_limit_paces_sends(): + with temp_env() as data: + state = data / "rate.json" + slept, now = [], [1000.0] + emailer._respect_rate_limit(20, lambda s: slept.append(s), lambda: now[0], state) + assert slept == [] # first send: nothing to wait for + now[0] = 1005.0 # only 5s later + emailer._respect_rate_limit(20, lambda s: slept.append(s), lambda: now[0], state) + assert slept and abs(slept[0] - 15) < 0.01 # waited the remaining 15s of the 20s window + + +class _FlakySMTP: + attempts = 0 + + def __init__(self, host, port, timeout=None): + pass + + def __enter__(self): + _FlakySMTP.attempts += 1 + if _FlakySMTP.attempts < 3: + raise _smtplib.SMTPServerDisconnected("transient") + return self + + def __exit__(self, *a): + return False + + def ehlo(self): + pass + + def starttls(self): + pass + + def login(self, u, p): + pass + + def send_message(self, m): + _FlakySMTP.sent = m + + +class _AuthFailSMTP(_FlakySMTP): + def __enter__(self): + return self + + def login(self, u, p): + raise _smtplib.SMTPAuthenticationError(535, b"bad creds") + + +def test_email_send_retries_transient_then_succeeds(): + _FlakySMTP.attempts = 0 + env = {"EMAIL_ADDRESS": "agent@gmail.com", "EMAIL_PASSWORD": "p"} + broker = {"id": "x", "optout": {"email": "privacy@x.example"}} + out = emailer.send(broker, "Subject: s\n\nb", env=env, _smtp_factory=_FlakySMTP, + _sleep=lambda *_: None) + assert out["attempts"] == 3 and "delivery_note" in out + + +def test_email_send_does_not_retry_permanent_error(): + env = {"EMAIL_ADDRESS": "agent@gmail.com", "EMAIL_PASSWORD": "p"} + broker = {"id": "x", "optout": {"email": "privacy@x.example"}} + try: + emailer.send(broker, "Subject: s\n\nb", env=env, _smtp_factory=_AuthFailSMTP, + _sleep=lambda *_: None) + except _smtplib.SMTPAuthenticationError: + pass + else: + raise AssertionError("auth failure must raise immediately, not retry") + + +def _run(argv) -> dict: + buf = _io.StringIO() + with _ctx.redirect_stdout(buf): + pdd.main(argv) + return _json.loads(buf.getvalue()) + + +def test_send_email_is_idempotent_browser_mode(): + with temp_env(): + config.save_config({"email_mode": "browser"}) + sid = _run(["intake", "--full-name", "Jane Q. Public", + "--email", "jane@example.com", "--consent"])["subject_id"] + _run(["record", sid, "radaris", "found", "--found", "true"]) + first = _run(["send-email", sid, "radaris", "--listing", "https://radaris.com/p/x"]) + assert first.get("state") == "submitted" and first.get("send_via") == "browser" + again = _run(["send-email", sid, "radaris", "--listing", "https://radaris.com/p/x"]) + assert again.get("skipped") is True # not re-sent + + +def test_registry_candidate_urls_newest_first_with_floor(): + urls = registry.ca_candidate_urls(__import__("datetime").date(2027, 3, 1)) + assert urls[0].endswith("registry2027.csv") and urls[-1].endswith("registry2025.csv") + assert registry.ca_candidate_urls(__import__("datetime").date(2024, 1, 1))[0].endswith("registry2025.csv") + + +def test_registry_and_badbool_warn_on_too_few(): + with temp_env(): + res = registry.refresh_all(paths.registry_cache_path(), fetched={"ca": _registry_csv()}) + assert "warning" in res["sources"]["ca"] # 2 parsed < MIN_EXPECTED_CA + md = "## People Search Sites\n### One\n[opt out](https://one.example/optout)\n" + bres = badbool.refresh(paths.brokers_cache_path(), markdown=md) + assert bres["parsed"] == 1 and "warning" in bres + + +def test_report_metrics_removal_rate_and_overdue(): + with temp_env(): + sid = "sub_test01" + for st in ("found", "submitted", "awaiting_processing", "confirmed_removed"): + ledger.transition(sid, "a", st, **({"found": True} if st == "found" else {})) + ledger.transition(sid, "b", "found", found=True) # open + for st in ("found", "submitted", "awaiting_processing"): + ledger.transition(sid, "c", st, **({"found": True} if st == "found" else {})) + led = ledger.load(sid) + led["c"]["next_recheck_at"] = "2000-01-01T00:00:00Z" # force overdue + ledger.save(sid, led) + m = report.metrics(sid) + assert m["confirmed_removed"] == 1 + assert m["open_needs_action"] >= 1 and m["in_flight_claimed"] >= 1 + assert m["overdue_rechecks"] >= 1 and 0 < m["removal_rate"] <= 1 + + +if __name__ == "__main__": + failures = [] + tests = [(n, f) for n, f in sorted(globals().items()) if n.startswith("test_") and callable(f)] + for name, fn in tests: + try: + fn() + print(f"PASS {name}") + except Exception as exc: # noqa: BLE001 + failures.append((name, exc)) + print(f"FAIL {name}: {exc!r}") + print(f"\n{len(tests) - len(failures)}/{len(tests)} passed") + sys.exit(1 if failures else 0)