feat(skills): add security/unbroker (autonomous data-broker removal)

unbroker finds where a consenting person's info is exposed across data
brokers and people-search sites and files the removals, running as far as
each site allows and handing only genuinely human-only steps (hard CAPTCHA,
gov-ID, phone, fax) back as an end-of-run digest.

- Deterministic stdlib CLI (scripts/pdd.py) owns config, dossiers+consent,
  the broker DB, tier planning, the ledger, email, and the autonomous
  action queue; the agent scans/submits with native tools (web_extract,
  browser_*, delegate_task, cronjob, terminal).
- Verify-before-disclose, least-disclosure (never volunteers SSN), consent
  gate, opaque ids, optional age-at-rest encryption, file-locked ledger.
- Jurisdiction-aware (CCPA/CPRA, GDPR, generic); CA DROP one-shot covers
  the state registry (~545) in a single request; BADBOOL + curated
  people-search coverage; scheduled re-scan for re-listing.
- No CAPTCHA-solving services or anti-bot bypass; browser email mode needs
  no stored password.
- 85 hermetic tests (tests/skills/test_unbroker_skill.py; SMTP/IMAP via
  injected fakes, registry via CSV fixtures). Ships placeholder data only.

Broker dataset adapted from BADBOOL (Yael Grauer, CC BY-NC-SA 4.0).
This commit is contained in:
SHL0MS 2026-07-02 21:16:10 -04:00
parent 89acc19606
commit c2828f2b9b
52 changed files with 6703 additions and 0 deletions

View file

@ -0,0 +1,163 @@
# unbroker
An agent-native skill that finds a consenting person's exposed personal information across data
brokers and people-search sites and removes it. It runs automatically wherever it can, and hands off
to a human only where a site demands a CAPTCHA it cannot clear, a government ID, a phone call, or a
fax.
<p align="center">
<img src="assets/unbroker.png"
alt="unbroker: autonomous removal pipeline (exposure field, the loop, ledger, re-scan horizon)"
width="720">
</p>
## About
Hundreds of data brokers publish people's names, current and prior addresses, phone numbers, emails,
relatives, and property records. That exposure fuels doxxing, stalking, harassment, and identity
theft. Removing the data is the documented antidote, but it is high-volume work, full of dark
patterns, and perishable (brokers re-list you). Commercial services such as EasyOptOuts, Incogni, and
DeleteMe solve this for a fee, but they are closed, and you hand a company you know nothing about the
exact data you are trying to erase.
unbroker brings those core capabilities together (EasyOptOuts' automation breadth, Incogni's
legal-request engine, DeleteMe's verification and reporting) as a transparent, auditable,
self-hosted skill that the user's own agent runs. It is **multi-tenant** (manage yourself, family, or
clients, each isolated), **consent-gated**, and built for **maximum automation with a human
fallback**. Scope is **US-first**, with EU/UK (GDPR) and global coverage on the roadmap.
The design is **Hermes-native**: a small deterministic Python CLI (`scripts/pdd.py`) owns the state
(config, dossiers, broker DB, tier planning, ledger, drafts, reports), while the agent does the
scanning and submitting with native tools (`web_extract`, `browser_*`, email, `cronjob`,
`delegate_task`). [`SKILL.md`](SKILL.md) is the authoritative reference.
## Install
```bash
hermes skills install official/security/unbroker
```
Then start a new Hermes session and drive it (below). The skill works zero-config; a few optional
env vars unlock more automation (all documented in `SKILL.md` under Prerequisites):
- `BROWSERBASE_API_KEY`: the recommended default browser. A real residential-IP cloud browser that
clears soft/managed CAPTCHAs (Turnstile, hCaptcha/reCAPTCHA checkbox) as normal operation, so
those brokers stay automated. It is not a solver and does not defeat hard challenges.
- Hands-off email, two ways: **browser mode** (`pdd.py setup --email-mode browser`, no stored
password; the agent sends opt-outs and opens verification links through your logged-in webmail),
or **`EMAIL_ADDRESS` + `EMAIL_PASSWORD`** for SMTP send + IMAP verification. Without either, it
falls back to writing drafts for you to send.
- the `age` binary: at-rest encryption of dossiers and ledgers.
- the `google-workspace` skill: a shared Google Sheets status tracker.
## Usage
Drive it from a Hermes session:
> "Use the unbroker skill to remove my data from data brokers. Here is my consent. Run it hands-off
> and show me the human-task digest at the end."
The agent configures itself (`setup --auto` selects programmatic email if `EMAIL_*` creds exist, the
cloud browser if available, and encryption if `age` is installed), records your consent, then drains
the autonomous queue: scan, opt out (parents first), send and verify emails, schedule re-checks. You
hear from it twice: at intake, and with one digest of anything only a human can do.
The underlying CLI (run via `terminal`, as `python3 scripts/pdd.py <cmd>`):
| Command | Purpose |
|---|---|
| `pdd.py setup --auto` / `doctor` | Self-configure (most-autonomous valid config) and readiness check |
| `pdd.py intake` | Create a consenting subject (captures aliases, multiple emails/phones, prior addresses) |
| `pdd.py next` | The loop driver: ordered agent actions right now, the human digest, and the next wake time |
| `pdd.py brokers` / `refresh-brokers` | List people-search brokers, or pull the latest BADBOOL list plus the CA registry |
| `pdd.py registry` | State data-broker registry coverage (CA ~545 ingested; VT/OR/TX portals); `--search` to find one |
| `pdd.py drop` | The CA DROP one-shot: delete from all registered brokers in a single request |
| `pdd.py plan` | Per-broker tier, method, search vectors, and the exact fields to disclose |
| `pdd.py fanout` | Batch brokers into parallel `delegate_task` subagents |
| `pdd.py record` | Update the ledger (validated state machine); auto-stamps recheck dates |
| `pdd.py send-email` | Render and send an opt-out / CCPA / GDPR request (recipient locked to the broker's own address) |
| `pdd.py poll-verification` / `verify-link` | Resolve email-verification links (IMAP poll, or browser-mode from pasted text) |
| `pdd.py render-email` | Draft-only fallback (least-disclosure) |
| `pdd.py due` / `tasks` | Recheck queue for cron, and the consolidated human-task digest |
| `pdd.py status` / `report` | Per-subject status, plus optional Google Sheets rows |
## How it works
- **Autonomous by default.** After one human conversation (intake plus consent), the agent drains a
deterministic action queue (`pdd.py next`): scan, opt out parents-first, send and verify emails,
re-check on schedule, all without pausing to ask. Human-only work (gov-ID sites, phone callbacks,
hard-CAPTCHA sites) accumulates silently into a single end-of-run digest (`pdd.py tasks`).
- **Tiered automation (T0 to T3).** Every broker opt-out is classified from fully automated, to
automated with verification, to human-verified, to human-only. The agent always takes the highest
viable tier and escalates to a human task only when genuinely blocked.
- **Cluster parents first.** Many brokers are resold shells of a few parents, so one removal can
clear a dozen child sites. The planner orders parents ahead of standalone listings and ships
field-verified, per-parent playbooks that prefer the **right-to-delete** lane over mere suppression
(for example PeopleConnect's "delete my user data", or Whitepages' privacy email, which sidesteps
the phone-callback tool entirely).
- **Multi-identifier fan-out.** A person is indexed under every name/alias, phone, email, and
address. The planner expands all of them (filtered by what each broker supports) so listings under
a maiden name or an old address are found, not just "primary name plus current city".
- **Verify before you disclose.** Nothing is submitted until a real listing is confirmed, the match
is confirmed as the subject and not a namesake or relative, and only the exact fields a broker
requires are sent (least-disclosure; SSN and ID numbers are never volunteered).
- **Jurisdiction-aware.** Requests file under the framework that applies where the subject lives:
CCPA/CPRA in California, GDPR in the EU/UK, a general right-to-delete request otherwise. It never
cites a right the subject cannot invoke.
- **Coverage that matches or exceeds commercial services.** Two lanes: (1) people-search sites with
per-site opt-out mechanics (19 curated records, including FamilyTreeNow, Radaris, and Nuwber, plus
a live pull from [BADBOOL](https://github.com/yaelwrites/Big-Ass-Data-Broker-Opt-Out-List)), and
(2) the **state data-broker registries** as a distinct legal-coverage lane: the **California Data
Broker Registry** (~545 registered brokers, the authoritative universe the commercial services draw
from) is ingested, with Vermont, Oregon, and Texas surfaced as search portals.
- **The DROP one-shot.** California's Delete Request and Opt-out Platform is live: for a CA resident,
a single verified request deletes their data from **every registered broker at once**, and
`pdd.py next` surfaces it as the highest-leverage action.
- **Ledger, audit, and re-scan.** Every case is a validated state machine, every PII disclosure is
logged (field names only), and confirmed removals are re-scanned on a schedule so a re-listing is
caught and re-filed. Ledger writes are file-locked for safe concurrent runs.
- **Privacy by default.** Opaque subject ids (no name in ids, paths, or logs), optional `age` at-rest
encryption of dossiers, and everything local. The skill ships placeholder data only.
## Tests
85 hermetic tests (no network, browser, or email; SMTP and IMAP are exercised through injected
fakes):
```bash
scripts/run_tests.sh tests/skills/test_unbroker_skill.py # CI-parity harness
python3 tests/skills/test_unbroker_skill.py # dependency-free fallback runner
```
## Safety and ethics
- **Consent-gated.** The engine refuses to scan or act on a subject without a recorded
authorization. It is a removal tool, not a people-search aggregator.
- **Sanctioned browser only, no solver farms.** The default cloud browser clears soft/managed
CAPTCHAs the way any real browser would, but there is no CAPTCHA-solving service and no fingerprint
spoofing. Hard interactive challenges escalate to a human task.
- **Least-disclosure and honest reporting.** The skill submits only what a broker requires. "Hidden
from free search" is never reported as "deleted", and residual exposure (public records, paid-tier
retention) is disclosed.
- **PII handling.** Dossiers live under the Hermes home directory (`0600`, optionally
`age`-encrypted), with opaque ids.
## Status
**v1.0.** The deterministic engine, the autonomous loop, the verified cluster-parent deletion lanes,
and full broker-registry coverage (the CA Data Broker Registry plus the DROP one-shot) are built and
covered by 85 hermetic tests. The skill ships placeholder data only. Live agent-driven submission
against broker sites is the active field-testing frontier.
## Credits and license
- Broker dataset adapted from the **Big-Ass Data Broker Opt-Out List (BADBOOL)** by **Yael Grauer**,
licensed [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (attribution
required, non-commercial). See [yaelwrites.com](https://yaelwrites.com/).
- Code: MIT.
## Disclaimer
This is not legal advice. Only operate on people who have authorized removal of their own data.
Removing data from brokers reduces exposure but does not guarantee total erasure. Public records
(voter, property, court) and offline vectors are out of scope.

View file

@ -0,0 +1,285 @@
---
name: unbroker
description: Autonomously remove your info from data-broker sites.
version: 1.0.0
author: SHL0MS (github.com/SHL0MS)
license: MIT
platforms: [linux, macos, windows]
prerequisites:
commands: [python3]
metadata:
hermes:
tags: [privacy, data-broker, opt-out, ccpa, gdpr, security, doxxing]
category: security
related_skills: [google-workspace, agentmail, himalaya, scrapling, osint-investigation]
homepage: https://github.com/NousResearch/hermes-agent
---
# unbroker
Find where a person's personal information (name, addresses, phone, email, relatives) is exposed on
data brokers and people-search sites, then remove it - automatically where possible, with guided
human steps only where a site demands a CAPTCHA, government ID, phone call, or fax. Manages multiple
people independently. It does **not** defeat anti-bot systems, does **not** act on anyone without
recorded consent, and does **not** remove public records (voter/property/court) or accounts the
person controls.
The Python CLI (`scripts/pdd.py`) owns the deterministic state - config, dossiers + consent, the
broker database, tier planning, the ledger, drafts, reports, **email sending (SMTP), verification-link
polling (IMAP), and the autonomous action queue (`next`)**. You (the agent) do the scanning and
form-driving with native tools: `web_extract` and `browser_navigate` for searching and web forms, and
`cronjob` for recurring re-scans.
## Autonomy contract
This skill is designed to run **hands-off**. After intake (+ recorded consent) there are exactly TWO
legitimate human touchpoints: (1) the intake conversation itself, and (2) ONE consolidated human-task
digest at the end of the run (`$PDD tasks`). Between those:
- **Never ask the operator to choose configuration.** `$PDD setup --auto` detects capabilities and
picks the most autonomous valid config itself.
- **Never pause before individual submissions** when `autonomy=full` (the default): the consent
recorded at intake is standing authorization for T0-T2 opt-outs. (`autonomy=assisted` restores
per-submission confirmation for cautious operators - honor `confirm_first` flags in `next` output.)
- **Never interrupt the run for human-only work.** Record it (`record ... human_task_queued
--reason "..."`) and keep going; it all surfaces once in the final digest.
- **Drive the whole run as a loop over `$PDD next <subject>`** - it returns the exact ordered actions
to take right now (scan, poll verification, re-check, opt out parents-first, requeue blocked), plus
the human digest. Execute every action, record outcomes, re-run `next`, repeat until
`done_for_now`. Then present the digest, report, and schedule the cron.
The hard limits that autonomy never overrides: no acting without recorded consent, no disclosure
beyond `disclosure_fields`, no CAPTCHA/anti-bot bypass, and `confirmed_removed` only after a
verifying re-scan.
## When to Use
- "Remove my (or my family member's) data from data brokers / people-search sites."
- "Opt me out", "delete me from Spokeo/Whitepages/etc.", "clean up after a doxxing."
- "Set up recurring privacy monitoring" (brokers re-list people).
- Checking which brokers still expose someone and why.
## Prerequisites
- `python3` (stdlib only; no extra packages needed for the core engine).
- **Optional upgrades** (the skill works zero-config without these; `setup --auto` turns on every
one it detects - each one converts a class of human tasks into agent actions):
- **Cloud browser (recommended default): `BROWSERBASE_API_KEY`.** `setup --auto` selects it
whenever the key is present, and it is the intended baseline: a real residential-IP cloud
browser **clears soft/managed CAPTCHAs (Cloudflare Turnstile, hCaptcha/reCAPTCHA checkbox) as
normal operation**, so those brokers stay automated (T1) instead of becoming human tasks. This
is not CAPTCHA "solving" - no solver service, no fingerprint spoofing; only interactive/behavioral
("hard") challenges the browser genuinely cannot pass fall back to a human task. Without the key,
the plain agent browser is used and soft-CAPTCHA brokers drop to T2 (human).
- Email automation, two credential-free-or-not options:
- **Browser mode (no password): `setup --email-mode browser`.** The agent sends opt-out/CCPA
emails and opens verification links through the operator's **logged-in webmail** using
`browser_*` tools. Nothing is stored. Needs the inbox signed in in the browser Hermes uses
(a cloud browser like Browserbase won't hold the session; use a local/operator browser).
Falls back to drafts for an email if the inbox isn't reachable.
- **SMTP/IMAP (stored creds): `EMAIL_ADDRESS` + `EMAIL_PASSWORD`** (+ `EMAIL_SMTP_HOST` /
`EMAIL_IMAP_HOST` for non-mainstream providers; gmail/outlook/yahoo/icloud/fastmail inferred).
The CLI sends via `send-email` and reads verify links via `poll-verification`. The `agentmail`
skill (per-broker aliases) also counts.
- Google Sheets tracker: the `google-workspace` skill.
- The `scrapling` skill for stealth/Cloudflare-protected pages.
## How to Run
Run everything through the `terminal` tool. From this skill's directory:
```bash
PDD="python3 scripts/pdd.py"
```
The engine stores data under `$PDD_DATA_DIR` (default `$HERMES_HOME/unbroker`), written
`0600`. Run via `terminal`, **not** `execute_code` (that sandbox scrubs env and redacts output, which
breaks reading the dossier).
## Quick Reference
| Command | Purpose |
|---|---|
| `$PDD setup --auto` | **Autonomous setup**: detect capabilities, pick the most autonomous valid config (no questions) |
| `$PDD doctor` | Readiness check: config, broker count, and which upgrades are on/available |
| `$PDD intake --full-name "..." [--alias ...] [--email ... --phone ...] [--city --state] [--prior-location "City,ST"] --consent` | Create a consenting subject; captures aliases + multiple emails/phones + prior locations; prints `subject_id` |
| `$PDD next <subject>` | **The autonomous loop driver**: ordered agent actions right now + human digest + `next_wake_at` |
| `$PDD brokers [--priority crucial]` | List the people-search broker database (curated + live) |
| `$PDD refresh-brokers` | Pull the latest BADBOOL people-search list **and the CA Data Broker Registry** (`next` requeues this automatically when the cache is stale) |
| `$PDD registry [--search NAME]` | State registry coverage (CA ~545 ingested; VT/OR/TX portals surfaced); the DROP/email lane, not scanned |
| `$PDD drop <subject> [--filed]` | **The one-shot legal lever**: one CA DROP request deletes from ALL registered brokers; `--filed` records it |
| `$PDD plan <subject> [--priority crucial]` | Per-broker tier + method + `search_vectors` + the exact fields to disclose |
| `$PDD plan <subject> --batch` | **Reduce view**: overlays ledger state, groups brokers by next action (unscanned/found/indirect/blocked/in_progress/done), collapses ownership clusters, **orders `found` cluster-parents-first + emits a tailored `parent_playbook`**, prints `next_actions` |
| `$PDD fanout <subject> [--priority crucial] [--size 8]` | Batch brokers into parallel `delegate_task` subagents (auto for large runs) |
| `$PDD record <subject> <broker> <state> [--found true] [--evidence JSON] [--disclosed F --channel C] [--reason "..."]` | Update the ledger (validated state machine); **auto-stamps `next_recheck_at`** |
| `$PDD send-email <subject> <broker> --listing <url> [--kind ccpa_indirect ...]` | Render + record the request (recipient locked to the broker's own address). **browser** mode returns a `compose` payload to send via webmail (no password); **programmatic** mode SMTP-sends |
| `$PDD verify-link <subject> <broker> --text '<body>'` | **browser mode**: extract a broker's verification link from webmail text you read (anti-phishing scored) |
| `$PDD poll-verification <subject> [--broker <id>]` | **programmatic mode**: poll IMAP for verification links (anti-phishing scored); auto-advances `submitted → verification_pending` |
| `$PDD render-email <subject> <broker> --listing <url>` | Draft only (fallback when no email mode is configured) |
| `$PDD due <subject>` | Cases whose recheck window arrived (the cron re-scan queue) |
| `$PDD tasks <subject>` | ONE consolidated human-task digest (present at END of run) |
| `$PDD status <subject>` | Markdown status report |
| `$PDD report <subject> --sheets` | Rows for the Google Sheets tracker |
## Batch operation (two-phase: crawl-all, then delete)
For anything past a couple of brokers, run this as **map → reduce → act**, not broker-by-broker:
- **Phase 1 - DISCOVER (read-only, parallel, idempotent).** Crawl *every* broker first and record a
verdict for each (`found` / `not_found` / `indirect_exposure` / `blocked`). Scanning has no side
effects, so it is safe to parallelize and retry. Getting the full exposure map *before* acting is
what unlocks cluster dedup and prioritization below. **Default: the parent drives `web_extract`
probes directly** - most people-search sites render name/phone/address results as static HTML that
`web_extract` reads in seconds. Escalate to `browser_*` only for the few JS-only sites, and to
`delegate_task` subagents only for genuinely *reasoning*-heavy work (large-scale namesake/relative
disambiguation). **Do NOT hand a browser-toolset subagent a big list of brokers to crawl** - in the
field this timed out repeatedly (600s, ~5-6 brokers each, no summary) because browser navigation is
heavy; the ledger writes that survived came at 10x the cost of parent `web_extract`. A `blocked`
(DataDome/Cloudflare/`antibot`) site is *not* a subagent job either: record `blocked` and requeue it
for a stealth/cloud browser (Browserbase) pass. Subagent reports are self-reports - the parent
re-fetches key URLs to confirm a `found` before trusting it (this cuts both ways: it caught a real
listing the parent had wrongly assumed was a false positive).
- **REDUCE - `$PDD plan <subject> --batch`.** Collapses the crawl into a phase-oriented plan: groups by
next action, **collapses ownership clusters** (a parent removal that clears children is ONE action,
not N - e.g. one Intelius/PeopleConnect suppression covers Truthfinder/Instant Checkmate/US Search/…),
and prints `next_actions`. `phase` is `discover` while anything is unscanned, else `delete`.
- **Phase 2 - DELETE (sequential, irreversible).** Work the reduced groups **parents first**:
`plan --batch` orders the `found` group cluster-parents-first (most children first) and emits a
`parent_playbook` with tailored, ordered steps per parent - follow that order and those steps
(full recipes in `references/methods.md` → "Ownership clusters - DO PARENTS FIRST"). Do the
cluster parents (skipping the covered children), **re-scan each parent's children after it confirms**
(they usually drop out), then the standalone listings; send the `indirect_exposure` cases as
CCPA/GDPR delete-my-PII emails (`send-email --kind ccpa_indirect`), and defer `blocked` to the
stealth-browser pass. Opt-outs hit CAPTCHAs, email-verification loops, and session binding - work
them **one at a time, carefully** (this is the opposite of fan-out), but do NOT stop to ask
permission per submission in `autonomy=full`; in `assisted`, confirm each one. **Prefer deletion
over suppression** where a broker offers both (e.g. PeopleConnect's "Right to Delete / DELETE MY
USER DATA" actually removes data; the suppression flow only hides it).
Subagent reports are self-reports: the parent re-verifies key claims (listing URLs, match basis) before
recording `found` and before any deletion.
## Procedure (the autonomous loop)
1. **Setup (once, no questions).** Run `$PDD setup --auto` - it detects capabilities and configures
the most autonomous valid combination itself (programmatic email when `EMAIL_*` creds exist,
Browserbase when its key exists, `age` encryption when the binary exists, `autonomy=full`). Then
`$PDD doctor` and show the operator the readiness output **for information, not as a question** -
proceed immediately. Mention what would unlock more automation (e.g. email creds) but do not wait.
2. **Intake + consent (the ONE human conversation).** `$PDD intake ...` with `--consent` (and
`--consent-method`). Without consent the engine refuses to plan or act. Collect everything in one
pass - names/aliases, current + prior cities, emails, phones - so you never have to come back with
questions. For California subjects, also read `references/legal/drop.md`: `next` will surface a
`drop_submit` one-shot that deletes from every registered broker (~545) at once, which is the
single highest-leverage action. File it, then `drop <subject> --filed`. For non-CA subjects the
registry is covered by targeted CCPA/GDPR emails (`registry --search`, then `send-email`); the
people-search sites are worked directly in either case.
3. **Drain the queue.** Loop:
```
while true:
q = $PDD next <subject>
if q.actions is empty: break
execute EVERY action in order; record each outcome via $PDD record
```
`next` emits, in order: `refresh_brokers` (stale cache), `fanout_scan`/`scan_inline` (Phase 1
crawl - see step 4), `poll_verification` (in-flight email confirmations), `verify_removal` (due
re-checks), `optout_web_form`/`optout_email_send` (Phase 2, parents-first with playbook steps),
`indirect_email_send`, and `stealth_rescan`. Human-only work never appears as an action - it
accumulates in `q.human_digest`. In `autonomy=full`, execute actions without pausing; honor
`confirm_first` in `assisted` mode.
4. **Scanning (when `next` says so).** For `fanout_scan`: run `$PDD fanout <subject>` and **spawn one
`delegate_task` subagent per `batch`, in parallel, passing that batch's ready-made `brief`** - do
not scan all brokers yourself sequentially. For `scan_inline`: scan the few brokers yourself.
Either way, each broker gets **every** `search_vectors` entry via the `references/methods.md`
ladder (`web_extract``site:` probe → `browser_navigate``scrapling`), a 404 is INCONCLUSIVE
(not `not_found`), `blocked` is recorded when `antibot` is set and no stealth browser is available,
and subject vs namesake/relative is confirmed before recording:
`$PDD record <subject> <broker> <found|not_found|indirect_exposure|blocked> --found <bool> --evidence '{"listing_urls":[...]}'`.
The parent re-verifies key `found` claims from subagents before trusting them.
5. **Opt-outs (when `next` says so).** Actions come pre-ordered parents-first with `steps` from each
broker record's own `optout.playbook` (field-verified; cluster parents like PeopleConnect,
Whitepages, BeenVerified, Spokeo have exact, live-checked recipes). **Deletion beats
suppression**: when an action carries `prefer_deletion`, complete the record's DELETION lane (e.g.
PeopleConnect's "DELETE MY USER DATA"), never just the hide-my-listing flow. Per method:
- **web_form** → drive `optout_url` with `browser_navigate`/`browser_type`/`browser_click`, submit
only `disclosure_fields`, screenshot the confirmation, then the action's `after` record command.
Playbooks may end with a right-to-delete `send-email` follow-up - do it (full erasure, not just
listing suppression).
- **email** → `$PDD send-email <subject> <broker> --kind <ccpa|gdpr|generic> --to <addr>
--listing <url>` records + discloses in one step (recipient locked to addresses the broker
record declares; `next` picks the kind from residency - never claim CCPA/GDPR for someone who
can't). In **browser** mode it returns a recipient-locked `compose` payload: compose a new
message to `compose.to` with `compose.subject`/`compose.body` exactly in the operator's webmail
via `browser_*` and send (no password); in **programmatic** mode it SMTP-sends. `next` also
routes human-gated forms (phone-callback/gov-ID) through a broker's deletion email when one
exists - the **rescue lane** (verified Whitepages pattern). Draft-only falls back to
`render-email` + a digest entry.
- **captcha** → soft/managed challenges clear automatically on the default cloud browser (proceed
as normal); only a hard interactive/behavioral challenge it can't pass is recorded `blocked`
(requeued for the stealth/operator-browser pass). Never a solver service.
- **phone_callback / account / gov_id / fax / mail / voice (T3)** *without a deletion email*
never an agent action; `next` already routed these to the digest. Record them:
`$PDD record <subject> <broker> human_task_queued --reason "..."`.
6. **Verification (when `next` says so).** In **programmatic** mode `$PDD poll-verification <subject>`
finds arrived confirmation links via IMAP (anti-phishing scored, auto-advances state). In
**browser** mode, open the broker's confirmation email in the operator's webmail and run
`$PDD verify-link <subject> <broker> --text '<body>'` to score the link. Either way **open the
link in the same browser** (several brokers bind the verification session to the browser that
opens it), finish the flow, then record `awaiting_processing`. `confirmed_removed` ONLY after a
verifying re-scan shows the listing gone - never off the submission flow's own confirmation page.
7. **Wrap up (once per run).** When `next` returns no actions: present `$PDD tasks <subject>` (the
consolidated human digest) if non-empty, then `$PDD status <subject>`; if the Sheets tracker is
on, append `$PDD report <subject> --sheets` rows via the `google-workspace` skill.
8. **Schedule the next wake-up.** `next` returns `next_wake_at` (earliest due re-check). Create ONE
`cronjob` that re-runs this skill's loop for the subject (a prompt like: *"run the
unbroker loop for <subject_id>: `$PDD next` and execute all actions"*). Processing
windows, verification polls, and reappearance sweeps all flow through the same queue, so the case
keeps advancing with zero human attention.
## Pitfalls
- **Never disclose more than the broker already shows.** Submit only `disclosure_fields`. The engine
never volunteers SSN/ID numbers; you must not either.
- **No consent, no action.** The engine enforces this; do not work around it to "research" a third party.
- **`send-email` is idempotent + rate-limited.** It refuses to re-send a case already `submitted`
or beyond (use `--force` only if a genuine re-send is needed), and SMTP sends are paced by
`email_min_interval_seconds` (default 20s) with retry/backoff. Do not loop it to "make sure" -
a successful SMTP handoff is not proof of delivery; the due-queue re-scan is the real confirmation.
- **Ledger writes are locked.** Concurrent runs (cron + manual) serialize safely; if you ever see a
lock timeout, another run is mid-write - let it finish, don't delete the `.lock` by hand.
- **Autonomy ≠ improvisation.** Full autonomy means not *asking* between steps; it does not loosen any
gate. If a broker demands MORE than the planned `disclosure_fields` mid-flow, stop that case and
queue it (`human_task_queued --reason`) rather than deciding alone to disclose extra PII.
- **Don't interrupt the run with questions.** Config choices are `setup --auto`'s job; human-only work
goes to the digest. The only mid-run question that's ever warranted is a missing-identity fact that
blocks scanning (e.g. no city at all) - and that should have been collected at intake.
- **Use `terminal`, not `execute_code`** for `pdd.py` (secret scrubbing + output redaction break it).
- **Dossiers are plaintext by default** (JSON, `0600` under `HERMES_HOME`). For at-rest encryption run
`$PDD setup --encryption age` - it generates a local `age` key and encrypts dossiers + ledgers (the
audit log holds field names only and stays plaintext). It guards casual/backup/commit exposure, not
a full-`HERMES_HOME` read; set `PDD_AGE_IDENTITY` to a separate volume for real key separation.
`$PDD doctor` shows whether encryption is *actually* engaged (not just whether `age` is installed).
- **"Hidden from free search" ≠ deleted.** Only mark `confirmed_removed` after verifying the record is
actually gone; note paid-tier retention in the report.
- **Soft CAPTCHAs clear by default; don't fight the hard ones.** The default cloud browser passes
managed/soft challenges as normal operation (those brokers stay T1). For a hard interactive one it
genuinely can't pass, record `blocked` and let the stealth/operator-browser pass take it - never a
third-party solver service or fingerprint spoofing.
- **Broker pages change.** If a flow breaks, `$PDD record ... blocked` and flag the broker file in
`references/brokers/` for re-verification instead of guessing.
- **Verify non-field-verified records before submitting.** `confidence: auto` records came from
parsing BADBOOL (read `optout.notes`/`optout.links`, confirm the real opt-out URL). `confidence:
documented` records (several people-search sites) carry the correct published opt-out URL but have
**not** been field-verified (they 403 datacenter IPs), so confirm the live flow via the operator's
residential browser on first use, then set `last_verified`. Field-verified curated records (no
`confidence`, e.g. the cluster parents) have checked mechanics and take precedence.
## Verification
- `scripts/run_tests.sh tests/skills/test_unbroker_skill.py` (hermetic; no network), or the
dependency-free runner `python3 tests/skills/test_unbroker_skill.py`.
- Dry run: `$PDD setup --auto && $PDD doctor && SID=$($PDD intake --full-name "Test Person"
--email t@example.com --consent | python3 -c 'import sys,json;print(json.load(sys.stdin)["subject_id"])')
&& $PDD next "$SID"` and confirm a readiness summary plus an ordered action queue.

Binary file not shown.

After

Width:  |  Height:  |  Size: 444 KiB

View file

@ -0,0 +1,50 @@
{
"id": "advancedbackgroundchecks",
"name": "AdvancedBackgroundChecks",
"category": "people_search",
"priority": "high",
"jurisdictions": ["US"],
"search": {
"method": "url_pattern",
"url": "https://www.advancedbackgroundchecks.com",
"fetch": "web_extract",
"match_signal": "result",
"by": ["name", "phone", "address"],
"url_patterns": {
"name": "https://www.advancedbackgroundchecks.com/name/{first}-{last}",
"name_state": "https://www.advancedbackgroundchecks.com/name/{first}-{last}/in/{ST}",
"phone": "https://www.advancedbackgroundchecks.com/phone/{digits10}",
"address": "https://www.advancedbackgroundchecks.com/address/{num}-{street-with-type}-{city}-{ST}-{zip}",
"person_profile": "https://www.advancedbackgroundchecks.com/find/person/{first}-{last}-{22charID}"
},
"url_format_quirks": [
"All path segments lowercase, spaces -> hyphens. Name: /name/jane-public ; name+state: /name/jane-public/in/NY (state 2-letter UPPERCASE).",
"Phone: /phone/5551234567 (10 digits, NO punctuation).",
"Address: /address/123-main-st-anytown-ny-12345 (all hyphen-joined incl. ZIP; abbreviations like 'S' and 'Ave' kept verbatim, not expanded).",
"Removable unit is the person profile: /find/person/{first}-{last}-{22charID} (opaque mixed-case ID). Also /find/name/{first}-{last} and /find/name/{first}-{last}/in/{ST}.",
"NO 404s for plausible patterns: an empty search returns a soft-200 page with 'similar' fuzzy rows. The real 'not found' signal is the ABSENCE of an exact-match block in the results heading, NOT an HTTP error. Do not record not_found off a 404 here; read the heading.",
"No anti-bot gating on search/result/phone/address pages (web_extract reads them fine). Site self-brands as 'ActualPeopleSearch' in FAQ text.",
"Search tabs: /?tab=phone /?tab=email /?tab=address. Directories: /lastnames/{surname} /firstnames/{first} /people/{state-name}/{city-name} /people/zip/{zip} /people/areacode/{code}."
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.advancedbackgroundchecks.com/opt-out",
"requires": {
"profile_url": false,
"email_verification": true,
"captcha": true,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": ["full_name", "contact_email"],
"notes": "Two-step email-verification flow: initial form (radio 'I am: The subject of the request / An authorized agent of the subject', First name*, Middle name, Last name*, Email*, consent) -> emailed link -> full form -> confirmation (processed within 45 days). CAPTCHA-gated: Google reCAPTCHA ('Recaptcha requires verification'). Verified read-only 2026-06-30; not submitted.",
"est_processing_days": 3,
"reappearance_risk": "medium"
},
"last_verified": "2026-06-30",
"source": "BADBOOL"
}

View file

@ -0,0 +1,56 @@
{
"id": "beenverified",
"name": "BeenVerified",
"category": "people_search",
"priority": "crucial",
"jurisdictions": ["US"],
"parent": "beenverified",
"owns": ["peoplelooker", "peoplesmart"],
"search": {
"method": "url_pattern",
"url": "https://www.beenverified.com/app/optout/search",
"fetch": "browser",
"match_signal": "result",
"by": ["name", "phone", "email", "address"]
},
"optout": {
"tier": "T1",
"method": "web_form",
"url": "https://www.beenverified.com/svc/optout/search/optouts",
"email": "privacy@beenverified.com",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": ["full_name", "contact_email", "profile_url"],
"deletion": {
"via": "email_followup",
"email": "privacy@beenverified.com",
"kinds": ["ccpa", "generic"],
"notes": "privacy@beenverified.com is the documented address for opt-out, access, deletion, and correction requests (verified from the live privacy policy 2026-07-01). Controller is The Lifetime Value Co. -- a deletion request here can name their other properties too. DPO: dpo@beenverified.com. CCPA window: respond within 45 days (extendable to 90)."
},
"playbook": [
"Opt-out tool at beenverified.com/svc/optout/search/optouts ('Do Not Sell or Share' footer link; the legacy /app/optout/search path serves the same search) -- clears PeopleLooker + PeopleSmart.",
"Search the subject, open the matching listing, and submit with the confirmed profile URL + contact email. Email verification: poll-verification picks up the confirmation link; open it in the agent's own browser.",
"ONE opt-out per email address via the tool; for additional listings email support@beenverified.com or privacy@beenverified.com.",
"FULL DELETION follow-up (right-to-delete, beyond listing suppression): send-email --kind ccpa (CA) / generic to privacy@beenverified.com naming the listing URL(s); as the controller is The Lifetime Value Co., ask that the deletion cover affiliated LTV properties (NeighborWho, Ownerly, NumberGuru, Bumper) in the same request.",
"A separate property-search opt-out exists (NeighborWho/Ownerly are property-focused sisters); note residual exposure if relevant and re-scan the children after the parent confirms."
],
"notes": "Opt-out form + deletion email both verified from the live privacy policy 2026-07-01 (policy dated 2025-10-21). Authorized agents: signed authorization letter or CA POA emailed to privacy@beenverified.com; agent-submitted delete/know requests must include the consumer's name, valid email, age, and address.",
"quirks": [
"The 'Do Not Sell or Share My Personal Information' footer link now points to /svc/optout/search/optouts (observed 2026-07-01); records citing /app/optout/search are the same tool's older entry.",
"One opt-out per email address through the tool; email support for additional removals. The same browser/inbox must open the confirmation link.",
"Sister LTV Co. properties (NeighborWho, Ownerly, NumberGuru, Bumper) keep separate opt-out tools -- do NOT assume the BV tool cleared them; the privacy@ deletion request can name them, then verify by scanning each.",
"Right-to-know requests get an initial response within 10 days; deletion/access within 45 days (CCPA)."
],
"est_processing_days": 7,
"reappearance_risk": "medium"
},
"last_verified": "2026-07-01",
"source": "BADBOOL"
}

View file

@ -0,0 +1,47 @@
{
"id": "clustal",
"name": "Clustal",
"category": "people_search",
"priority": "high",
"jurisdictions": ["US"],
"search": {
"method": "url_pattern",
"url": "https://www.clustal.org/",
"fetch": "web_extract",
"match_signal": "record",
"by": ["name"],
"url_patterns": {
"name_index": "https://www.clustal.org/people-search/{letter}/{first-last}/",
"name_state": "https://www.clustal.org/people-search/{letter}/{first-last}/{st}/",
"record": "https://www.clustal.org/record/{first-last}-{8charID}/"
},
"url_format_quirks": [
"Name index: /people-search/{first-initial}/{first-last}/ e.g. /people-search/k/jane-public/ (lowercase, first letter of LAST name as the {letter} segment, hyphen-joined name). Optional state refine appends /{st}/ lowercase 2-letter.",
"Detail (removable unit): /record/{first-last}-{8charID}/ e.g. /record/jane-public-a1b2c3d4/ (opaque mixed-case 8-char id). The index page links each match to its /record/ URL.",
"Readable via web_extract (no hard bot gate as of 2026-07-01). Rich profile: age/DOB, current+prior addresses, phones, emails, relatives, neighbors, associates.",
"Name only: search is by name(+state); no reverse phone/email/address path observed. NOTE: clustal.org squats the 'Clustal' bioinformatics brand but IS a real people-search/data-broker site ('Find Your DNA Relatives'), not the sequence-alignment tool - do not dismiss it as a false positive."
]
},
"optout": {
"tier": "T0",
"method": "web_form",
"url": "https://www.clustal.org/privacy-control/",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": ["profile_url", "contact_email"],
"notes": "Opt-out at /privacy-control/ using the /record/ profile URL; support help@clustal.org. Verify the exact form fields + any CAPTCHA live before submitting (requires flags below are provisional from the site's opt-out copy, not yet a live form walk-through).",
"email": "help@clustal.org",
"est_processing_days": 7,
"reappearance_risk": "medium"
},
"last_verified": "2026-07-01",
"source": "BADBOOL",
"confidence": "curated"
}

View file

@ -0,0 +1,52 @@
{
"id": "clustrmaps",
"name": "ClustrMaps",
"category": "people_search",
"priority": "high",
"jurisdictions": [
"US"
],
"parent": "clustrmaps",
"owns": [],
"search": {
"method": "url_pattern",
"url": "https://clustrmaps.com/",
"fetch": "browser",
"match_signal": "result",
"antibot": "cloudflare",
"by": [
"name",
"phone",
"address"
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://clustrmaps.com/bl/opt-out",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"contact_email",
"profile_url"
],
"notes": "Address/resident aggregator. Opt-out at /bl/opt-out: submit the profile URL + email, confirm the link.",
"quirks": [
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
"Needs the confirmed profile_url (the address/person page).",
"Email verification required."
],
"est_processing_days": 3,
"reappearance_risk": "medium"
},
"last_verified": null,
"source": "curated",
"confidence": "documented"
}

View file

@ -0,0 +1,53 @@
{
"id": "cyberbackgroundchecks",
"name": "CyberBackgroundChecks",
"category": "people_search",
"priority": "high",
"jurisdictions": [
"US"
],
"parent": "cyberbackgroundchecks",
"owns": [],
"search": {
"method": "url_pattern",
"url": "https://www.cyberbackgroundchecks.com/",
"fetch": "browser",
"match_signal": "result",
"antibot": "cloudflare",
"by": [
"name",
"phone",
"address"
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.cyberbackgroundchecks.com/removal",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"full_name",
"contact_email",
"profile_url"
],
"notes": "Free people-search. Opt-out at /removal: find the record, submit email, confirm link.",
"quirks": [
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
"Needs the confirmed profile_url.",
"Email verification required."
],
"est_processing_days": 3,
"reappearance_risk": "medium"
},
"last_verified": null,
"source": "curated",
"confidence": "documented"
}

View file

@ -0,0 +1,50 @@
{
"id": "familytreenow",
"name": "FamilyTreeNow",
"category": "people_search",
"priority": "crucial",
"jurisdictions": [
"US"
],
"parent": "familytreenow",
"owns": [],
"search": {
"method": "url_pattern",
"url": "https://www.familytreenow.com/",
"fetch": "browser",
"match_signal": "result",
"antibot": "cloudflare",
"by": [
"name",
"phone",
"address"
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.familytreenow.com/optout",
"requires": {
"profile_url": false,
"email_verification": false,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"full_name"
],
"notes": "Notorious free people-search / doxxing site (no registry). Opt-out: search the record, select it, confirm removal on-site. No account, free.",
"quirks": [
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
"Select the exact record from search results, then confirm removal; historically no email step, but re-lists periodically so keep the re-scan scheduled."
],
"est_processing_days": 2,
"reappearance_risk": "high"
},
"last_verified": null,
"source": "curated",
"confidence": "documented"
}

View file

@ -0,0 +1,43 @@
{
"id": "fastpeoplesearch",
"name": "FastPeopleSearch",
"category": "people_search",
"priority": "high",
"jurisdictions": ["US"],
"search": {
"method": "url_pattern",
"url": "https://www.fastpeoplesearch.com/",
"fetch": "browser",
"antibot": "datadome",
"match_signal": "result",
"by": ["name", "phone", "address"],
"url_patterns": {
"name": "https://www.fastpeoplesearch.com/name/{first}-{last}"
},
"url_format_quirks": [
"Name path /name/jane-public (lowercase, hyphen join) was ACCEPTED by the router under challenge, so the name pattern is confirmed even though content never rendered.",
"Sibling of truepeoplesearch (near-identical removal flow/branding); phone pattern is LIKELY /{digits} or /phone/{digits} and address /address/... but UNCONFIRMED - do not assume.",
"MOST aggressively gated of the people-search trio (2026-06-30): both server-side scraping (Firecrawl 504) AND headless browser (Cloudflare -> DataDome) fail on search AND /removal. Treat as fully blocked; needs residential-proxy/stealth browser to scan or opt out."
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.fastpeoplesearch.com/removal",
"requires": {
"profile_url": false,
"email_verification": true,
"captcha": true,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": ["full_name", "contact_email"],
"notes": "Opt-out NOT directly observed (2026-06-30): /removal is itself behind DataDome. By sibling-site pattern almost certainly mirrors truepeoplesearch (email-link flow, subject/agent toggle, name+email fields, hCaptcha) - INFERRED, not verified. Confirm before acting.",
"est_processing_days": 3,
"reappearance_risk": "high"
},
"last_verified": "2026-06-30",
"source": "BADBOOL"
}

View file

@ -0,0 +1,88 @@
{
"id": "intelius",
"name": "Intelius",
"category": "people_search",
"priority": "crucial",
"jurisdictions": [
"US"
],
"parent": "peopleconnect",
"owns": [
"truthfinder",
"instantcheckmate",
"ussearch",
"zabasearch",
"classmates",
"peoplefinder",
"peoplelookup",
"addresses",
"anywho",
"publicrecords"
],
"search": {
"method": "url_pattern",
"url": "https://www.intelius.com/",
"fetch": "web_extract",
"match_signal": "result",
"by": [
"name",
"phone",
"address"
],
"url_patterns": {
"name": "https://www.intelius.com/people-search/{First}-{Last}/",
"name_state": "https://www.intelius.com/people-search/{first}-{last}/{state-name}/",
"full_report": "https://www.intelius.com/search/?firstName={First}&lastName={Last}&city={City}&state={ST}&traffic%5Bsource%5D=INTSEO"
},
"url_format_quirks": [
"Name summary page: /people-search/{First}-{Last}/ (hyphen join; case-insensitive, both Jane-Public and jane-public work). Readable via web_extract (no hard bot gate as of 2026-06-30).",
"State refine: /people-search/{first}-{last}/{state-name}/ with the state SPELLED OUT lowercase, e.g. /jane-public/new-york/ (NOT the 2-letter code).",
"The summary shows last-known address + a 'possible relatives' list + a FAQ ('Where does X live?'). Corroborate the subject by address before acting; the 'Top People with the Last Name {Last}' block is unrelated namesakes.",
"Part of PeopleConnect: same data surfaces on Truthfinder/InstantCheckmate/USSearch; one suppression at suppression.peopleconnect.us covers the cluster."
]
},
"optout": {
"tier": "T1",
"method": "web_form",
"url": "https://suppression.peopleconnect.us/login",
"email": "privacy@peopleconnect.us",
"requires": {
"profile_url": false,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"contact_email"
],
"deletion": {
"via": "in_flow",
"url": "https://suppression.peopleconnect.us/guided-mode",
"email": "privacy@peopleconnect.us",
"kinds": ["ccpa", "generic"],
"notes": "The portal's 'Right to Delete / DELETE MY USER DATA' (bottom of guided-mode) is the verified deletion path and covers the whole cluster. privacy@peopleconnect.us is the documented rights-request address (their 2025 CPRA metrics: 33,513 delete requests, median response < 1 day) - use it as the fallback if the portal breaks."
},
"playbook": [
"PeopleConnect portal (suppression.peopleconnect.us/login, privacy-center entry at /privacy-center) -- ONE flow here clears Truthfinder, Instant Checkmate, US Search, ZabaSearch, Classmates and ~15 more. DO THIS PARENT FIRST.",
"Step 1 asks ONLY for an email + consent checkbox (no name/DOB, no CAPTCHA) -> sends a verification email. Least-disclosure entry: just the contact email.",
"poll-verification will pick up the verify link. The link authenticates a SESSION bound to the browser that OPENS it: the SAME agent browser must open the link and drive /guided-mode (a different browser is bounced to /login).",
"SUPPRESSION != DELETION -- guided-mode's default flow only HIDES the report (data retained, can re-list). Scroll to the BOTTOM of guided-mode and use 'Right to Delete / DELETE MY USER DATA' (CCPA/CPRA) -- this actually deletes across the cluster and emails its own confirmation link; poll and open that too.",
"If the portal breaks: email privacy@peopleconnect.us (rights-request address, median response < 1 day per their published metrics), or sister addresses privacy@intelius.com / privacy@truthfinder.com / privacy@instantcheckmate.com / support@ussearch.com / privacy@classmates.com; phone 1-888-245-1655.",
"After the deletion confirms, re-scan the covered children (they normally drop out) before submitting any duplicate child opt-out."
],
"notes": "PeopleConnect portal covers the cluster. Authorized-agent requests: signed written authorization (full name, address, phone, the email the consumer uses) or POA; for Right-to-Delete they verify agent authority with the consumer by email. Verified from the live privacy policy 2026-07-01 (policy dated 2026-06-30).",
"quirks": [
"Step 1 (suppression.peopleconnect.us/login) asks ONLY for an email + a consent checkbox, then 'Continue' -> a verification email with a link. No CAPTCHA, no name/DOB at step 1. Least-disclosure entry: just the contact email. Verified live 2026-06-30.",
"The verification link authenticates a SESSION and lands on /guided-mode. That session is bound to the browser that OPENED it; a different browser hitting /guided-mode is redirected back to /login. So for hands-off automation the SAME agent browser must open the verify link (Mode B: read inbox -> agent browser navigates the link -> drive guided-mode).",
"SUPPRESSION != DELETION. The guided-mode/suppression flow only HIDES the background report (data retained, can re-list). At the BOTTOM of guided-mode there is a separate 'Right to Delete' section with a 'DELETE MY USER DATA' button (CCPA/CPRA) that removes the user data across the cluster - this is the stronger action and should be preferred. It also emails a confirmation link. (Confirmed via security.org Instant Checkmate guide + live flow 2026-06-30.)",
"Their published request metrics (2025, all US users regardless of state): 33,513 deletion requests, 26,603 complied, median response < 1 day - the deletion lane is real and fast. Verified from privacy policy 2026-07-01."
],
"est_processing_days": 7,
"reappearance_risk": "medium"
},
"last_verified": "2026-07-01",
"source": "BADBOOL"
}

View file

@ -0,0 +1,35 @@
{
"id": "mylife",
"name": "MyLife",
"category": "people_search",
"priority": "crucial",
"jurisdictions": ["US"],
"search": {
"method": "url_pattern",
"url": "https://www.mylife.com",
"fetch": "browser",
"match_signal": "profile",
"by": ["name"]
},
"optout": {
"tier": "T3",
"method": "phone",
"url": "https://www.mylife.com/privacyrequest",
"requires": {
"profile_url": true,
"email_verification": false,
"captcha": false,
"gov_id": true,
"account": false,
"phone_callback": false,
"phone_voice": true,
"payment": false
},
"inputs": ["full_name", "profile_url"],
"notes": "Often pushes a driver's-license upload or a call to (888) 704-1900. Emailing privacy@mylife.com with name + profile link is an alternative. Also covers Wink.com.",
"est_processing_days": 14,
"reappearance_risk": "high"
},
"last_verified": "2026-06-28",
"source": "BADBOOL"
}

View file

@ -0,0 +1,52 @@
{
"id": "nuwber",
"name": "Nuwber",
"category": "people_search",
"priority": "high",
"jurisdictions": [
"US"
],
"parent": "nuwber",
"owns": [],
"search": {
"method": "url_pattern",
"url": "https://nuwber.com/",
"fetch": "browser",
"match_signal": "result",
"antibot": "cloudflare",
"by": [
"name",
"phone",
"address"
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://nuwber.com/removal/link",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"contact_email",
"profile_url"
],
"notes": "People-search. Opt-out: submit the profile URL + email at /removal/link, confirm via the emailed link.",
"quirks": [
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
"Needs the confirmed profile_url (paste the listing URL you recorded).",
"Email verification required."
],
"est_processing_days": 3,
"reappearance_risk": "medium"
},
"last_verified": null,
"source": "curated",
"confidence": "documented"
}

View file

@ -0,0 +1,53 @@
{
"id": "peekyou",
"name": "PeekYou",
"category": "people_search",
"priority": "high",
"jurisdictions": [
"US"
],
"parent": "peekyou",
"owns": [],
"search": {
"method": "url_pattern",
"url": "https://www.peekyou.com/",
"fetch": "browser",
"match_signal": "result",
"antibot": "cloudflare",
"by": [
"name",
"phone",
"address"
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.peekyou.com/about/contact/optout",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"full_name",
"contact_email",
"profile_url"
],
"notes": "Aggregates social/web profiles. Opt-out: paste your PeekYou profile URL(s), pick a reason, provide email, confirm the link.",
"quirks": [
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
"Needs the confirmed PeekYou profile_url.",
"Email verification required."
],
"est_processing_days": 7,
"reappearance_risk": "medium"
},
"last_verified": null,
"source": "curated",
"confidence": "documented"
}

View file

@ -0,0 +1,53 @@
{
"id": "peoplefinders",
"name": "PeopleFinders",
"category": "people_search",
"priority": "high",
"jurisdictions": [
"US"
],
"parent": "peoplefinders",
"owns": [],
"search": {
"method": "url_pattern",
"url": "https://www.peoplefinders.com/",
"fetch": "browser",
"match_signal": "result",
"antibot": "cloudflare",
"by": [
"name",
"phone",
"address"
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.peoplefinders.com/opt-out",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"full_name",
"contact_email",
"profile_url"
],
"notes": "Standalone people-search (Confi-Chek family). Opt-out: find the listing, submit with a contact email, confirm via the emailed link.",
"quirks": [
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
"Needs the confirmed profile_url from evidence.",
"Email verification: poll-verification picks up the link; open it in the agent browser."
],
"est_processing_days": 7,
"reappearance_risk": "medium"
},
"last_verified": null,
"source": "curated",
"confidence": "documented"
}

View file

@ -0,0 +1,58 @@
{
"id": "radaris",
"name": "Radaris",
"category": "people_search",
"priority": "crucial",
"jurisdictions": [
"US"
],
"search": {
"method": "url_pattern",
"url": "https://radaris.com/",
"fetch": "web_extract",
"match_signal": "View Profile",
"by": [
"name",
"phone",
"address"
],
"url_patterns": {
"name": "https://radaris.com/p/{First}/{Last}/"
},
"url_format_quirks": [
"Name profile page: /p/{First}/{Last}/ with First/Last Capitalized and a TRAILING slash, e.g. /p/Jane/Public/ . Readable via web_extract (no hard bot gate as of 2026-06-30).",
"The page aggregates: a 'phone numbers & home addresses' table (the DIRECT hit for the subject), a relatives/namesakes carousel ('Review the potential relatives'), and resume/CV records. The subject's removable row is in the address table; the carousel entries are OTHER people - disambiguate by address/age before acting.",
"Page is noisy with testimonials/reviews boilerplate; the signal blocks are 'phone numbers & home addresses N' and 'resumes & CV records N'."
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://radaris.com/control-privacy",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": true,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"profile_url",
"contact_email"
],
"notes": "Multi-step control-privacy wizard: step 1 NEXT -> step 2 'identify your personal page' (paste the NUMBERED profile URL into the 'Or Enter URL of your page' field; typing reveals a NEXT submit button) -> then user_email + Google reCAPTCHA -> emailed verification link. If there is no View Profile button for the subject, email customer-service@radaris.com and reply to the auto-response until removed.",
"email": "customer-service@radaris.com",
"quirks": [
"CAPTCHA: the form carries a Google reCAPTCHA (hidden field g-recaptcha-response) plus anti-bot fingerprint tokens (jfp, token). Tier is T2 without a captcha-clearing browser backend (T1 with Browserbase). (Record previously mis-declared captcha:false.)",
"The /control-privacy form REQUIRES the per-person NUMBERED profile URL. Pasting the aggregate /p/{First}/{Last}/ URL is rejected with the validation error: 'The URL must include unique number at the end'. The real removable URL looks like https://{region}.radaris.com/person/~First-Last/1234567890 (see the field placeholder).",
"The subject's own record may appear ONLY as a static row in the 'phone numbers & home addresses' table with NO 'View Profile' link, so no numbered profile URL is exposed for them (the /p/{First}/{Last}/ page deep-links only to relatives/namesakes via JS onclick, not static hrefs). When that happens the /control-privacy form cannot be used -- do NOT fabricate or submit a relative's or namesake's profile URL. Fall back to email: write customer-service@radaris.com with the subject's own listed details and request removal, replying to the auto-response until confirmed.",
"Pressing Enter in the URL field reloads the wizard to step 1 (does not submit); type into the field and click the revealed NEXT button instead."
],
"est_processing_days": 14,
"reappearance_risk": "high"
},
"last_verified": "2026-06-30",
"source": "BADBOOL"
}

View file

@ -0,0 +1,53 @@
{
"id": "searchpeoplefree",
"name": "SearchPeopleFree",
"category": "people_search",
"priority": "high",
"jurisdictions": [
"US"
],
"parent": "searchpeoplefree",
"owns": [],
"search": {
"method": "url_pattern",
"url": "https://www.searchpeoplefree.com/",
"fetch": "browser",
"match_signal": "result",
"antibot": "cloudflare",
"by": [
"name",
"phone",
"address"
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.searchpeoplefree.com/opt-out",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"full_name",
"contact_email",
"profile_url"
],
"notes": "Free people-search. Opt-out: find the listing, submit with an email, confirm the link.",
"quirks": [
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
"Needs the confirmed profile_url.",
"Email verification required."
],
"est_processing_days": 3,
"reappearance_risk": "medium"
},
"last_verified": null,
"source": "curated",
"confidence": "documented"
}

View file

@ -0,0 +1,56 @@
{
"id": "spokeo",
"name": "Spokeo",
"category": "people_search",
"priority": "crucial",
"jurisdictions": ["US"],
"parent": "spokeo",
"owns": ["freepeopledirectory"],
"search": {
"method": "url_pattern",
"url": "https://www.spokeo.com/search",
"fetch": "browser",
"match_signal": "profile",
"by": ["name", "phone", "email", "address"]
},
"optout": {
"tier": "T1",
"method": "web_form",
"url": "https://www.spokeo.com/optout",
"email": "privacy@spokeo.com",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": ["profile_url", "contact_email"],
"deletion": {
"via": "email_followup",
"email": "privacy@spokeo.com",
"kinds": ["ccpa", "generic"],
"notes": "privacy@spokeo.com is the documented direct privacy contact (verified live on /optout 2026-07-01). Use for full CCPA deletion beyond listing opt-out, for listings the form rejects, and when more listings keep surfacing."
},
"playbook": [
"Opt-out form at spokeo.com/optout -- clears FreePeopleDirectory. Inputs: the confirmed profile URL + contact email; the form emails a confirmation link (poll-verification picks it up; open it in the agent's own browser).",
"EVERY LISTING SEPARATELY: 'you may have multiple listings on Spokeo. Each one is identified by a unique URL and must be opted out individually' (their own wording). Run ALL search vectors first, collect every listing URL, and submit one opt-out per URL.",
"Fast: requests process in 24-48 hours per their page -- re-scan after 2 days and record the real outcome.",
"FULL DELETION follow-up: send-email --kind ccpa (CA) / generic to privacy@spokeo.com naming all listing URLs -- covers data retained beyond the free-search suppression.",
"Paid/account data may be retained even when hidden from free search -- verify actual removal via re-scan; never mark confirmed_removed off the free-search view alone."
],
"notes": "Form + privacy@spokeo.com verified live 2026-07-01. Their page: opting out does not remove data from original sources and listings may reappear as new public records arrive -- keep the reappearance re-scan scheduled.",
"quirks": [
"Profile URL formats the form accepts: listing URL like spokeo.com/{First}-{Last}/{City}/{ST}/p{id} or a purchase URL spokeo.com/purchase?q=... (examples shown on /optout).",
"Multiple listings per person are NORMAL (one per name x location x record cluster); each unique URL must be opted out individually -- feed every found listing URL through the form.",
"Processing is 24-48h ('depending on the nature of your request and the amount of data').",
"Paid accounts may retain data even when hidden from free search - verify actual removal, don't mark confirmed_removed off the free-search view alone."
],
"est_processing_days": 2,
"reappearance_risk": "medium"
},
"last_verified": "2026-07-01",
"source": "BADBOOL"
}

View file

@ -0,0 +1,46 @@
{
"id": "thatsthem",
"name": "That's Them",
"category": "people_search",
"priority": "crucial",
"jurisdictions": ["US"],
"search": {
"method": "url_pattern",
"url": "https://thatsthem.com/",
"fetch": "browser",
"match_signal": "result",
"by": ["name", "phone", "email", "address"],
"url_patterns": {
"name": "https://thatsthem.com/name/{First}-{Last}/{City}-{ST}",
"phone": "https://thatsthem.com/phone/{areacode}-{prefix}-{line}",
"email": "https://thatsthem.com/email/{email}",
"address": "https://thatsthem.com/address/{Street}-{City}-{ST}-{ZIP}"
},
"url_format_quirks": [
"ADDRESS path is fully hyphen-joined and joins street+city+state+ZIP with hyphens (e.g. /address/123-Main-St-Anytown-NY-12345) and REQUIRES the ZIP. The older slash form (/address/Street/City-ST) and any ZIP-less form 404.",
"NAME and PHONE and EMAIL paths use a slash before city/state and do NOT need a ZIP: /name/First-Last/City-ST , /phone/AAA-PPP-LLLL , /email/addr@host.",
"Street abbreviations are kept verbatim (Ave, Pl, St) - do NOT expand to Avenue/Place/Street; expanding 404s.",
"A 404 on a constructed URL means WRONG PATTERN, not 'no record present'. Treat as INCONCLUSIVE: fall back to the on-site search box (browser_type into the address/name field + submit) and read the resulting canonical URL."
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://thatsthem.com/optout",
"requires": {
"profile_url": false,
"email_verification": false,
"captcha": true,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": ["full_name", "street", "city", "state", "postal", "contact_email", "phone"],
"notes": "Opt-out form is gated by a Cloudflare Turnstile CAPTCHA (so tier is T2 without a captcha-clearing browser backend; T1 with Browserbase). All 7 fields are marked required by the form (Full Name, Street Address, City, State, ZIP, Email, Phone). Site sends a confirmation email and states ~72h processing (this is a completion notice, not a click-to-verify link). Least-disclosure tension: the form DEMANDS email+phone even though a public record may show less - disclose only to remove a record that is actually about the subject. Do not click the Spokeo identity-theft-protection link (paid product).",
"est_processing_days": 3,
"reappearance_risk": "low"
},
"last_verified": "2026-06-30",
"source": "BADBOOL"
}

View file

@ -0,0 +1,43 @@
{
"id": "truepeoplesearch",
"name": "TruePeopleSearch",
"category": "people_search",
"priority": "high",
"jurisdictions": ["US"],
"search": {
"method": "url_pattern",
"url": "https://www.truepeoplesearch.com/",
"fetch": "browser",
"antibot": "datadome",
"match_signal": "result",
"by": ["name", "phone", "address", "email"],
"url_patterns": {
"name": "https://www.truepeoplesearch.com/results?name={First%20Last}&citystatezip={City,%20ST}"
},
"url_format_quirks": [
"Search results URL is QUERY-PARAM, not path: /results?name=Jane%20Public&citystatezip=Anytown,%20NY (space-encoded; comma between city and state). Confirmed as the canonical form the site's own search box generates.",
"On-site search tabs: Name / Phone / Address / Email / Neighbors (so name-, phone-, address-, and email-searchable).",
"HARD-BLOCKED for automated reads (2026-06-30): Cloudflare 'Just a moment...' interstitial -> DataDome device-check CAPTCHA (geo.captcha-delivery.com) on every results navigation. Page chrome (header/footer) may render while the result body stays blocked; submitting any search bounces to DataDome. web_extract/Firecrawl 504-timed out. Needs a residential-proxy/stealth browser (e.g. Browserbase) to scan; otherwise record 'blocked'."
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.truepeoplesearch.com/removal",
"requires": {
"profile_url": false,
"email_verification": true,
"captcha": true,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": ["full_name", "contact_email"],
"notes": "Removal page renders even when results are blocked. Flow: name+email+captcha -> emailed link -> fill opt-out form matching the record -> confirmation ('allow 3 days'). Fields: dropdown 'Exercise my rights as a: The subject of the request / An authorized agent of the subject', First Name*, Middle Name, Last Name*, Email Address*, consent checkbox. CAPTCHA-gated: hCaptcha ('I am human'). Contact support@truepeoplesearch.com; PO Box 7775 PMB 29296, San Francisco CA 94120-7775. Verified read-only 2026-06-30; not submitted. (Record previously mis-declared email_verification:false.)",
"est_processing_days": 3,
"reappearance_risk": "high"
},
"last_verified": "2026-06-30",
"source": "BADBOOL"
}

View file

@ -0,0 +1,53 @@
{
"id": "usphonebook",
"name": "USPhoneBook",
"category": "people_search",
"priority": "high",
"jurisdictions": [
"US"
],
"parent": "usphonebook",
"owns": [],
"search": {
"method": "url_pattern",
"url": "https://www.usphonebook.com/",
"fetch": "browser",
"match_signal": "result",
"antibot": "cloudflare",
"by": [
"name",
"phone",
"address"
]
},
"optout": {
"tier": "T2",
"method": "web_form",
"url": "https://www.usphonebook.com/opt-out",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": [
"full_name",
"contact_email",
"profile_url"
],
"notes": "Reverse-phone + people-search. Opt-out: paste the listing URL, provide an email, confirm via the emailed link.",
"quirks": [
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
"Needs the confirmed profile_url.",
"Email verification required."
],
"est_processing_days": 3,
"reappearance_risk": "medium"
},
"last_verified": null,
"source": "curated",
"confidence": "documented"
}

View file

@ -0,0 +1,57 @@
{
"id": "whitepages",
"name": "Whitepages",
"category": "people_search",
"priority": "crucial",
"jurisdictions": ["US"],
"parent": "whitepages",
"owns": ["411"],
"search": {
"method": "url_pattern",
"url": "https://www.whitepages.com/",
"fetch": "browser",
"match_signal": "listing",
"by": ["name", "phone", "address"]
},
"optout": {
"tier": "T1",
"method": "email",
"url": "https://www.whitepages.com/suppression_requests",
"email": "privacyrequest@whitepages.com",
"requires": {
"profile_url": true,
"email_verification": true,
"captcha": false,
"gov_id": false,
"account": false,
"phone_callback": false,
"payment": false
},
"inputs": ["full_name", "contact_email", "profile_url"],
"deletion": {
"via": "email",
"email": "privacyrequest@whitepages.com",
"url": "https://whitepagesprivacy.zendesk.com/hc/en-us/requests/new",
"kinds": ["ccpa", "generic"],
"notes": "privacyrequest@whitepages.com handles BOTH opt-out (removal from the site) and CCPA deletion requests, explicitly offered for people who do not want to provide a phone number for the automated tool. ~2 business day reply; opt-outs processed within 15 days. Verified from whitepages.com/privacy/consumer-rights 2026-07-01 (page dated 2026-06-22)."
},
"playbook": [
"EMAIL LANE (fully autonomous -- use this): send the removal + deletion request to privacyrequest@whitepages.com including the confirmed listing URL. This is Whitepages' own documented alternative 'if you would prefer not to provide a phone number'. Expect a reply within ~2 business days; they may ask identity-verification questions (name, email) -- answer with least-disclosure, never an ID number.",
"Include in the email: the listing URL(s), the subject's full name as listed, and the request to (a) remove the listing(s), (b) opt out of sale/sharing, and (c) delete personal data (CCPA 1798.105 for CA residents; they honor requests from other states per their consumer-rights page).",
"'Once a listing is removed, all known connected listings are also removed and the requester's information will not be sold by Whitepages in any capacity' -- one request covers connected listings; still re-scan afterward, and check Whitepages Premium + 411.com separately.",
"Alternative 1: webform at whitepagesprivacy.zendesk.com/hc/en-us/requests/new (same ~2-day handling; good fallback if the mailbox bounces).",
"Alternative 2 (only with an operator on the phone): the automated tool at whitepages.com/suppression_requests -- paste the listing URL, provide a phone number, answer the automated voice call and enter the 4-digit code. Fastest (est 1 day) but NOT autonomous.",
"Opt-out requests may take up to 15 days; verify with a re-scan before recording confirmed_removed."
],
"notes": "Email/webform lane verified 2026-07-01: privacyrequest@whitepages.com or the Zendesk form replace the phone-callback tool ('if you would prefer not to provide a phone number... submit a request to a customer service agent'). Authorized agents: written signed permission required. General privacy contact: support@whitepages.com.",
"quirks": [
"The automated suppression tool (whitepages.com/suppression_requests) REQUIRES a phone number + automated voice call with a 4-digit code + agreeing to their ToS -- that lane is phone_callback/T2. The email/webform lane exists precisely to avoid it; prefer email for autonomy.",
"Deletion exceptions they state: public records (property, court), fraud-prevention data, active-subscription data, legal holds. 'Hidden from search' vs 'deleted' distinction applies -- their opt-out removes the listing; the CCPA deletion covers non-public account data.",
"CCPA opt-out requests: up to 15 days to process. Right-to-know/access requests also via privacyrequest@whitepages.com or the Zendesk form."
],
"est_processing_days": 15,
"reappearance_risk": "medium"
},
"last_verified": "2026-07-01",
"source": "BADBOOL"
}

View file

@ -0,0 +1,27 @@
# CCPA / CPRA (California)
Use for California residents (`residency_jurisdiction` starts with `US-CA`) and, in practice, many US
brokers that honor CCPA-style requests nationwide.
## Rights invoked
- **Delete** personal information (Cal. Civ. Code 1798.105).
- **Opt out** of sale/sharing of personal information (1798.120).
## Request content
Render with `legal.render_request("ccpa", broker, fields)` -> `templates/emails/ccpa-deletion.txt`.
Include only: full legal name, the contact email for correspondence, and the confirmed listing
URL(s). Do **not** include SSN or government IDs.
## Authorized agent
When acting for another consenting subject, use `render_request("ccpa_agent", ...)`
(`templates/emails/ccpa-authorized-agent.txt`) and attach the authorization artifact recorded in the
dossier (`consent.authorization_artifact`). The broker may separately verify the consumer's identity.
## Notes
- Brokers must respond within 45 days (extendable). Track as `awaiting_processing` until confirmed.
- "Hidden from free search" is not deletion - verify the record is actually gone before
`confirmed_removed`.

View file

@ -0,0 +1,34 @@
# California DROP portal (highest-leverage lever)
The California **Delete Request and Opt-out Platform** (`privacy.ca.gov/drop`) lets a California
resident demand deletion from **every registered data broker** with a single verified request, for
free. DROP is **live** (as of 2026); registered brokers must begin processing requests on
**2026-08-01**. The registered universe is the **California Data Broker Registry** (~545 brokers in
2025), which this skill ingests as its own coverage lane (`pdd.py registry`); one DROP request covers
all of them, which is how this skill reaches (and exceeds) the breadth of commercial services.
## When to use
For any subject with `residency_jurisdiction` starting `US-CA`, sequence DROP **first**: `pdd.py next`
surfaces a single `drop_submit` action covering the whole registry. Then handle the individual
people-search sites (which are also worked directly because they hold free, indexed listings). After
filing, run `pdd.py drop <subject> --filed` so the loop stops re-surfacing it. For non-CA subjects
DROP does not apply; cover the registry brokers with targeted CCPA/GDPR deletion emails
(`pdd.py registry --search`, then `pdd.py send-email`).
## Flow (agent-assisted, mostly human verification)
1. The operator creates/verifies a DROP account (identity verification is required by the state; this
is a human step - `human_task_queued`).
2. Submit one deletion request covering all registered brokers.
3. Record a single ledger case `case_<subject>_drop` to track it; mark `submitted` ->
`awaiting_processing`. Registered brokers must process deletions on the state's schedule.
4. After the DROP cycle, re-scan the people-search long tail and only act on sites still showing data.
## Caveats
- DROP covers **registered data brokers**, not every people-search site. Keep doing the individual
opt-outs for non-registered sites.
- Identity verification means parts of this cannot (and should not) be fully automated.
- FCRA-regulated brokers (flagged in the registry, `optout.fcra`) hold consumer-report data with
separate rules; deletion may be limited and a dispute or security-freeze may apply instead.

View file

@ -0,0 +1,20 @@
# GDPR / UK-GDPR (roadmap - Phase 3)
For EU/UK subjects. Not part of the P0 US-first scope; templates and routing land in Phase 3.
## Rights invoked
- **Erasure** ("right to be forgotten") - Article 17.
- **Object** to processing - Article 21.
## Request content
Render with `legal.render_request("gdpr", broker, fields)` ->
`templates/emails/gdpr-erasure.txt`. Address the controller's privacy/DPO contact. Include the data
subject's name, the contact email, and the listing URL(s); cite Article 17.
## Notes
- Controllers must respond within one month (Article 12(3)).
- EU-specific brokers and portals (e.g. Acxiom's EU consumer portals) are added in Phase 3 with
`jurisdictions: ["EU"]` records and residency-aware routing.

View file

@ -0,0 +1,235 @@
# Opt-out method playbooks
How the agent executes each broker `optout.method` using native Hermes tools. Always obey the
**verify-before-disclose** rule: confirm a real listing exists, then submit only the fields the broker
requires (`pdd.py plan` lists them per broker).
**Autonomy:** `pdd.py next <subject>` sequences all of this - it decides which method applies, orders
parents first, and routes human-only work to the digest. In `autonomy=full` (default), execute its
actions without pausing per submission; the consent recorded at intake is the authorization. These
playbooks are the HOW for each action type.
## Scan ladder (all methods)
Confirm exposure before acting, cheapest first. Run **every** `search_vectors` entry from `pdd.py
plan` (each name x location, phone, email, and address the broker's `search.by` supports) - different
vectors surface different listings for the same person; dedupe found URLs.
1. `web_extract` on the broker `search.url` (fast HTML -> markdown). Look for `search.match_signal`.
Build per-vector URLs from `search.url_patterns` and heed `search.url_format_quirks` (see below).
1b. **`site:` search-engine probe (cheap, do it early and in parallel).** `web_search` with
`site:<broker-domain> "First Last"` (add a city/ZIP or a unique phone/address to cut namesake
noise) often returns the **exact profile-slug URL** in one shot - which both confirms the listing
exists AND hands you the opaque `/find/person/<id>` or `/p/<slug>` URL you'd otherwise have to
derive. Two big wins seen in the field: (a) it disambiguates namesakes fast - the SERP snippet
shows age/city so you can tell the subject from a same-name relative before fetching anything; and
(b) a broad `"First Last" <ZIP OR unique-address>` search (no `site:`) surfaces **brokers not yet in
your DB** (e.g. information.com, peoplefinders.com) - record those as bonus exposures. Note: empty
`site:` results are INCONCLUSIVE (many broker pages aren't indexed / are `noindex`), not `not_found`.
2. If the page is JS-rendered or returns nothing useful, `browser_navigate` + `browser_snapshot`
(and `browser_type`/`browser_click` to run the site's search box).
3. If blocked by stealth/Cloudflare, use the `scrapling` skill via `terminal`. **If the broker record
has `search.antibot` set (e.g. `datadome`), results are behind a device-check CAPTCHA**: a
cloud/stealth browser (Browserbase) or `scrapling` may get through; if none is available, do **not**
burn attempts - `pdd.py record <subject> <broker> blocked` and move on (a re-scan with a stealth
backend can pick it up later).
3b. **Operator-browser path (the reliable unblock for anti-bot sites).** Cloudflare/DataDome key on
datacenter IPs + headless fingerprints, so `web_extract`, the proxyless agent browser, and even a
cloud browser often fail - but the **operator's own everyday browser (residential IP, real
fingerprint) sails straight through**. For any `blocked` site, hand the operator a paste-ready
search URL (built from `search.url_patterns`), give them the identity anchors to judge by (current
+ prior addresses, age, a distinguishing detail) and the namesake/relative watch-list, and ask for
the verdict or a screenshot (the agent can read screenshots). This is a **first-class scan path, not
a fallback** - treat the operator's live check as authoritative and record the real verdict
(`found` / `not_found` / `indirect_exposure`), citing `scanned_via: operator_browser`. Same for
opt-out forms the agent's browser can't reach: guide the operator field-by-field (least-disclosure),
pausing before submit. (This is exactly why the same trick clears email-verification links the agent
can't open - see the Verification loop.)
4. Capture evidence: save listing URLs and a `browser` screenshot into the subject's `evidence/` dir,
then `pdd.py record <subject> <broker> found --found true --evidence '{"listing_urls":[...]}'`.
If a listing genuinely does not exist: `pdd.py record <subject> <broker> not_found` and move on.
### A 404 (or empty body) is INCONCLUSIVE, not "not_found"
A constructed search URL that 404s almost always means the **URL pattern is wrong**, not that the
person is absent. Never record `not_found` off a 404. Instead:
1. Re-check the broker's `search.url_patterns` / `url_format_quirks` and rebuild the URL.
2. Fall back to the **on-site search box**: `browser_navigate` to the search page, `browser_type`
the raw query, `browser_click` Search, then read the **canonical result URL** the site lands on.
3. Only after the site's own search returns an empty result set do you record `not_found`.
4. If a pattern was wrong, fix it in `references/brokers/<id>.json` (`url_patterns` +
`url_format_quirks`) so the next run is correct - see the rule below.
### Log URL/format quirks for every site you scrape
Whenever you discover how a broker's URLs are actually shaped (path layout, hyphen-vs-slash joins,
whether ZIP is required, abbreviation handling, query-param search, anti-bot gating), record it in
that broker's `references/brokers/<id>.json` under `search.url_patterns` (the templates) and
`search.url_format_quirks` (the gotchas, including which forms 404). Bump `last_verified`. This makes
the deterministic URL path reliable across runs and subjects instead of rediscovered each time. If the
opt-out form's real requirements differ from the record (extra required fields, a CAPTCHA, an account),
fix `optout.requires` / `optout.inputs` / `optout.tier` too - those drive tier selection and
least-disclosure. Log opt-out mechanics gotchas (a broker that needs a profile URL but doesn't expose
one for the subject, an email-only fallback, an authorized-agent toggle) in `optout.quirks` - the
planner surfaces these as `optout_quirks` per broker. Example: Radaris sometimes shows the subject only
as a static address-table row with no "View Profile" link, so `/control-privacy` (which needs a profile
URL) can't be used - fall back to `optout.email` rather than submitting a namesake's URL.
### Distinguish the subject from namesakes and relatives
People-search sites are dense with namesakes and family clusters. Before recording `found`, confirm the
record is the **subject themselves** (corroborate via DOB, a known current/prior address, or the
identifier you searched). Two non-removable patterns to record as evidence but NOT as the subject's own
listing:
- **Namesake:** same name, different person (different DOB/location with no overlap). Not the subject.
- **Relative record:** the listing is about a *different* person (a relative) and merely *names* the
subject in a "Family" field, or carries the subject's email/phone as a secondary datum. This is a
third party's record - the consent gate correctly blocks acting on it. See "Indirect exposure" in
the web_form section for what the subject *can* still request.
## web_form
1. `browser_navigate` to `optout.url`; `browser_snapshot` to read the form.
2. Fill only the planned `disclosure_fields` with `browser_type`/`browser_click`; for `profile_url`,
paste the confirmed listing URL from evidence.
3. Submit; `browser_snapshot` to confirm the success state; screenshot to `evidence/`.
4. `pdd.py record <subject> <broker> submitted --disclosed <field> --disclosed <field> --channel web_form`.
5. If the broker requires email verification, follow **Verification loop** below.
### Indirect exposure (named as a relative / your email on someone else's record)
You asked the right question: if a broker lists a *relative* and names you in their "Family" field, or
shows **your** email/phone on **their** record, that IS personal information about you - even though the
record's primary subject is a third party. Resolve it in two distinct lanes:
- **The self-service opt-out form does NOT cover this.** That form removes a record whose *primary
subject* is you. It has no notion of "scrub my identifiers from this other person's record," and
submitting it with the relative's address to force a match would be (a) disclosing data the listing
doesn't tie to you and (b) acting on a third party's record. Don't. The consent gate exists to stop
exactly that.
- **What you CAN do - a targeted "delete my personal information" request (CCPA 1798.105 / GDPR Art.17).**
These rights attach to *your* personal information *wherever the business holds it*, including as a
data point on another person's profile. So the subject may email the broker's privacy address and
request suppression of **their own specific identifiers** (this email address, this phone number, my
name in family/relative associations), citing the relative listings as the locations. This is a
narrower request than a full opt-out and does not require the relative's consent - you are only asking
them to delete data about *you*. Use `render-email` with the `ccpa`/`gdpr` template, list only the
subject's own identifiers + the URLs where they appear, and record it as a normal `submitted`
`awaiting_processing` email case. Verify by re-scanning those identifier vectors (email/phone) after
the statutory window - `confirmed_removed` only when the subject's identifier no longer appears.
- **Caveat:** the broker may decline to alter a third party's record beyond removing your specific
identifiers, and "your name in a family graph" can be derived from public records they'll re-list.
Note residual exposure in the report rather than marking a clean removal. (Operational guidance, not
legal advice.)
## email
`pdd.py send-email <subject> <broker> --listing <url> [--kind ccpa|gdpr|ccpa_indirect]` always does
the deterministic parts (recipient locked to an address the broker record declares, refusing anything
else; `--listing` mandatory; records `submitted`, logs disclosure, stamps `next_recheck_at`). How it
actually sends depends on `email_mode`:
1. **browser mode (no password, autonomous):** the command returns a recipient-locked `compose`
payload (`to`/`subject`/`body`). Compose a NEW message in the operator's **logged-in webmail** via
`browser_*` (paste `compose.body` exactly, disclosing nothing beyond it) and send. No credentials
stored. Requires the inbox signed in in the browser Hermes uses.
2. **programmatic mode (SMTP creds):** the command SMTP-sends it directly, no human.
3. **draft_only fallback:** `pdd.py render-email <subject> <broker> --listing <url>`; a digest entry
tells the operator to send it, and the agent records `submitted --channel email` afterward.
Then follow the **Verification loop** if the broker emails a confirmation link.
## Verification loop (email_verification brokers)
- **browser mode (autonomous, no password):** open the broker's confirmation email in the operator's
webmail (`browser_*`), then `pdd.py verify-link <subject> <broker> --text '<email body>'` returns
the anti-phishing-scored link. `browser_navigate` it **in the same browser** (several brokers, e.g.
PeopleConnect, bind the session to the browser that opens the link), finish the flow, record
`awaiting_processing`.
- **programmatic mode (IMAP):** `pdd.py poll-verification <subject>` polls IMAP for every in-flight
case, extracts the link (anti-phishing scored: only opt-out-looking links on the broker's own
domains), and auto-advances `submitted → verification_pending`. Then `browser_navigate` the link in
the agent's own browser, finish the flow, record `awaiting_processing`.
- **draft_only:** the digest tells the operator to click the link in the subject's inbox; the agent
records `awaiting_processing` on their word.
- Either way, the due queue (`pdd.py due`) brings the case back after the broker's processing window
for the verifying re-scan; only that re-scan justifies `confirmed_removed`.
## phone_callback (e.g. Whitepages)
Submit the web form, then the site places an automated call with a numeric code. If the operator is
available to read the code, capture it and complete the form (T2). Otherwise queue a human task.
## phone (voice menu) / fax / mail / gov_id -> human task (T3)
Do **not** attempt to automate. Create a `todo` task and `pdd.py record <subject> <broker>
human_task_queued` with exact instructions and an explicit **withhold** list (never SSN; never a
driver's-license number unless the subject chooses to and crosses out the ID number). Capture the
confirmation reference back into the ledger when the operator completes it.
## captcha
**Default: soft/managed CAPTCHAs clear automatically.** The recommended baseline backend is the
Browserbase cloud browser (`setup --auto` selects it when `BROWSERBASE_API_KEY` is set). Being a
real browser on a residential IP, it passes managed challenges - Cloudflare Turnstile, hCaptcha /
reCAPTCHA checkbox - as normal operation, so those brokers stay T1 and you just proceed. This is
**not** CAPTCHA solving: no solver service, no fingerprint spoofing.
Only a **hard** challenge the browser genuinely can't pass (interactive image grids, behavioral
scoring that flags the session) becomes a fallback: `record ... blocked` and requeue it for the
stealth/operator-browser pass (`methods.md` → scan ladder 3b - the operator's own residential
browser is the reliable unblock). Without a cloud browser configured, soft-CAPTCHA brokers drop to
T2 and become human tasks. **Never use a third-party CAPTCHA-defeating service.**
## Ownership clusters - DO PARENTS FIRST (playbooks live in the broker records)
Many brokers are resold shells of a few parents, so **one parent removal clears a whole cluster of
children** (see `owns` in each record). In Phase 2 you MUST work the cluster **parents first**, then
the standalone listings - doing a child before its parent wastes a submission the parent would have
covered. `pdd.py plan <subject> --batch` **orders the `found` group parents-first** and emits a
`parent_playbook` whose `steps` come verbatim from each record's **`optout.playbook`** - the single
source of truth, field-verified, updated as live runs discover mechanics. What follows is the
operating doctrine; the exact steps are in `references/brokers/<id>.json`.
**Deletion beats suppression, email lanes beat forms.** Each parent record carries a structured
`optout.deletion` lane (`via: in_flow | email | email_followup`, plus the privacy address). The
autopilot routes accordingly:
- **`in_flow`** (PeopleConnect): the deletion control lives INSIDE the web flow - complete the flow
and use the **Right to Delete / DELETE MY USER DATA** control, never just the suppression step.
- **`via: email`** (Whitepages): the fully-autonomous lane - `send-email` the request (residency-picked
kind: CCPA for US-CA, GDPR for EU/UK, generic otherwise), then `poll-verification` for their reply
and answer identity questions with least-disclosure. This is also the **rescue lane**: any broker
whose form demands a phone-callback/gov-ID/account but that declares a deletion email gets routed
here instead of the human digest.
- **`email_followup`** (BeenVerified, Spokeo): the opt-out form is the fast primary (it clears the
listing), and the playbook then sends a right-to-delete email for full erasure beyond suppression.
Verified parent facts (live-checked 2026-07-01; details + steps in the records):
- **Intelius/PeopleConnect** (~15+ sites in one flow): portal entry asks only email + consent →
verify link is **session-bound to the browser that opens it** → guided-mode → **DELETE MY USER
DATA** at the bottom (suppression alone re-lists). Fallback `privacy@peopleconnect.us` - their own
published metrics: 33.5k deletion requests, median response < 1 day.
- **Whitepages**: `privacyrequest@whitepages.com` (or the Zendesk form) handles removal + CCPA
deletion **without the phone-callback tool** - that phone call is only required by the automated
tool. One removal also drops "all known connected listings". ≤15 days; check 411.com + Premium.
- **BeenVerified**: opt-out tool (footer "Do Not Sell" link → `/svc/optout/search/optouts`) + email
verification; one opt-out per email address. Then `privacy@beenverified.com` deletion follow-up -
controller is The Lifetime Value Co., so name their sister properties (NeighborWho, Ownerly,
NumberGuru, Bumper) in the same request, and verify each separately.
- **Spokeo**: form takes ONE listing URL at a time and **each listing must be opted out
individually** - collect every listing URL from all search vectors first, then submit one opt-out
per URL. 24-48h processing. `privacy@spokeo.com` for full deletion beyond free-search suppression.
After each parent removal is confirmed, **re-scan its children** before submitting anything for them -
usually they drop out and need no separate opt-out.
### Any other parent
A parent without a hand-verified `optout.playbook` gets synthesised steps from its structured record
(URL/email, `requires` flags, deletion lane, notes/quirks). Follow those, and **write what you learn
back into `references/brokers/<id>.json`** (`optout.playbook`, `optout.deletion`, `quirks`,
`last_verified`) so the next run is exact - that file, not this one, is where per-broker knowledge
accrues.

View file

@ -0,0 +1,43 @@
# Case state machine
One case = one (subject x broker). `pdd.py record` validates every transition against this table and
appends it to `audit.jsonl`. Authoritative definition lives in `scripts/ledger.py`.
## States
| State | Meaning |
|---|---|
| `new` | Case created, nothing done |
| `searching` | Scan in progress |
| `not_found` | Subject not listed (will be re-checked next cycle) |
| `found` | Listing confirmed; action needed |
| `indirect_exposure` | Subject's PII (email/phone/name) appears on a **third party's** record (e.g. named in a relative's "Family" field). Not removable via self-service opt-out; needs a targeted CCPA/GDPR delete-my-PII request |
| `action_selected` | Tier/method chosen |
| `submitted` | Opt-out submitted |
| `verification_pending` | Awaiting email/callback verification |
| `awaiting_processing` | Submitted, no verification needed; broker processing |
| `confirmed_removed` | Verified gone |
| `reappeared` | Was removed, now listed again |
| `human_task_queued` | Needs an operator step (captcha/ID/phone/fax/mail) |
| `blocked` | Broker dead / mechanics broken -> flag for DB re-verification |
## Allowed transitions
```
new -> searching | found | not_found | indirect_exposure | blocked
searching -> not_found | found | indirect_exposure | blocked
not_found -> searching | found | indirect_exposure | blocked
found -> action_selected | submitted | human_task_queued | indirect_exposure | blocked
indirect_exposure -> submitted | human_task_queued | not_found | found | blocked
action_selected -> submitted | human_task_queued | blocked
submitted -> verification_pending | awaiting_processing | human_task_queued | blocked
verification_pending -> confirmed_removed | human_task_queued | blocked
awaiting_processing -> confirmed_removed | human_task_queued | blocked
confirmed_removed -> reappeared | confirmed_removed (recheck refreshes the date)
reappeared -> found | indirect_exposure
human_task_queued -> found | indirect_exposure | action_selected | submitted | verification_pending
| awaiting_processing | confirmed_removed | blocked
blocked -> searching | found | not_found | indirect_exposure | action_selected
```
A transition to the same state is always allowed (idempotent field updates).

View file

@ -0,0 +1,396 @@
"""Autonomous action queue: what should the agent do RIGHT NOW for this subject?
`next_actions` turns (dossier, broker DB, config, ledger) into an ordered queue of
concrete agent actions plus a human digest. The agent's whole run becomes a loop:
while True:
q = pdd.py next <subject>
if not q["actions"]: break
execute each action, record outcomes
present q["human_digest"] once; schedule cron at q["next_wake_at"]
Policy (cfg["autonomy"]):
full - intake consent is standing authorization; T0-T2 agent actions are
executed without pausing. Humans appear only in the digest.
assisted - same queue, but every submission action carries confirm_first=True.
The queue is deterministic and side-effect free: it never mutates the ledger, it
only reads. Executing + recording stays with the agent (and the record command).
"""
from __future__ import annotations
import datetime as _dt
import os
from pathlib import Path
import brokers as brokers_mod
import emailer
import ledger as ledger_mod
import paths
import registry
import tiers
CACHE_STALE_DAYS = 7 # refresh the live broker list after this
FANOUT_THRESHOLD = 8 # above this many unscanned brokers, use delegate_task fan-out
# States with nothing left to do (absent a due recheck).
_TERMINAL = {"not_found", "confirmed_removed"}
_IN_FLIGHT = {"submitted", "verification_pending", "awaiting_processing"}
def cache_age_days(now: float | None = None) -> float | None:
"""Age of the live BADBOOL cache in days, or None if never pulled."""
p: Path = paths.brokers_cache_path()
if not p.exists():
return None
now = now if now is not None else _dt.datetime.now().timestamp()
return max(0.0, (now - p.stat().st_mtime) / 86400.0)
def _now_iso() -> str:
return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def _min_future_recheck(ledger: dict, at: str) -> str | None:
future = [c.get("next_recheck_at") for c in ledger.values()
if c.get("next_recheck_at") and c["next_recheck_at"] > at]
return min(future) if future else None
def _digest(broker_row: dict, reason: str, steps: list[str], prep: list[str] | None = None) -> dict:
return {
"broker_id": broker_row.get("broker_id"),
"broker_name": broker_row.get("broker_name"),
"reason": reason,
"agent_prep": prep or [], # commands the agent runs BEFORE handing this to the human
"steps": steps, # what the human actually does
"withhold": ["SSN", "full driver's-license / passport numbers"],
}
def request_kind(dossier: dict, allowed: list[str] | None = None) -> str:
"""Pick the honest legal basis for a deletion request from the subject's residency.
ccpa only for California residents, gdpr only for EU/UK residents, generic otherwise.
`allowed` (from the broker's deletion.kinds) can restrict DOWN to generic but never
upgrades to a law the subject can't truthfully claim.
"""
res = (dossier.get("residency_jurisdiction") or "US").upper()
if res.startswith("US-CA"):
kind = "ccpa"
elif res.startswith(("EU", "UK", "GB")):
kind = "gdpr"
else:
kind = "generic"
if allowed and kind not in allowed and "generic" in allowed:
kind = "generic"
return kind
_HUMAN_GATES = ("gov_id", "fax", "mail", "phone_voice", "phone_callback", "account")
def _email_lane(row: dict) -> tuple[str | None, str]:
"""(address, why) for the autonomous email lane of this broker, if one exists.
Lane rules:
1. the broker's primary opt-out method IS email;
2. the record marks its deletion lane email-preferred (deletion.via == "email");
3. RESCUE: the primary flow is human-gated (gov ID / fax / phone / account) but a
right-to-delete email exists - the email lane restores full autonomy (this is the
verified Whitepages pattern: privacyrequest@ accepts requests precisely so people
don't have to do the phone-callback tool).
"""
deletion = row.get("deletion") or {}
req = row.get("optout_requires") or {}
if row.get("method") == "email":
addr = row.get("optout_email") or deletion.get("email")
return (addr, "primary opt-out method is email") if addr else (None, "")
if deletion.get("via") == "email" and deletion.get("email"):
return deletion["email"], "record prefers the right-to-delete email lane"
if (row.get("tier") == "T3" or any(req.get(k) for k in _HUMAN_GATES)) and deletion.get("email"):
return deletion["email"], "rescue: primary flow is human-gated; deletion email restores autonomy"
return None, ""
def _optout_action(row: dict, playbook: dict[str, dict], subject_id: str, dossier: dict,
email_mode: str, smtp_ok: bool, confirm_first: bool) -> tuple[dict | None, dict | None]:
"""Map one actionable `found` row to (agent_action, human_digest_entry).
Routing order maximizes autonomy: (1) the email lane (primary email method, preferred
right-to-delete email, or rescue from a human-gated form) beats everything when SMTP is
up; (2) genuinely human-only flows go to the digest; (3) web forms are driven with the
record's own field-verified playbook steps.
"""
bid = row["broker_id"]
req = row.get("optout_requires") or {}
tier = row.get("tier")
deletion = row.get("deletion") or {}
# 1) The autonomous EMAIL LANE (right-to-delete by email + confirm the reply).
# Autonomous when SMTP is configured (programmatic/alias) OR in browser mode (agent sends via
# the operator's logged-in webmail; no password needed).
email_addr, lane_why = _email_lane(row)
can_email = (email_mode in ("programmatic", "alias") and smtp_ok) or email_mode == "browser"
if email_addr and can_email:
kind = request_kind(dossier, deletion.get("kinds"))
via = "browser" if email_mode == "browser" else "smtp"
then = ("send-email records it + returns a recipient-locked payload; compose and send it in "
"the operator's webmail via browser_*, then `verify-link` on the reply and open the link"
if via == "browser" else
"state auto-records as submitted; poll-verification picks up their verification reply, "
"open its link, then record")
return {
"type": "optout_email_send",
"broker_id": bid, "broker_name": row.get("broker_name"), "tier": tier,
"confirm_first": confirm_first, "send_via": via,
"to": email_addr, "kind": kind, "why": lane_why,
"command": f"python3 scripts/pdd.py send-email {subject_id} {bid} --kind {kind} "
f"--to {email_addr} --listing <confirmed-url>",
"then": then,
}, None
if row.get("method") == "email":
return None, _digest(row, "email opt-out (draft mode: a human must hit send)",
["Send the rendered draft from your own mail client",
f"Then: python3 scripts/pdd.py record {subject_id} {bid} submitted "
f"--disclosed contact_email --channel email"],
prep=[f"python3 scripts/pdd.py render-email {subject_id} {bid} --listing <confirmed-url>"])
# 2) Genuinely human-only work goes to the digest (no email lane could rescue it).
if tier == "T3":
return None, _digest(row, "human-only opt-out (gov ID / fax / mail / voice phone)",
[f"Follow the broker's process at {row.get('optout_url') or row.get('optout_email')}",
"Provide only the fields the listing already shows; cross out ID numbers on any document"])
if req.get("phone_callback"):
return None, _digest(row, "phone-callback verification (operator must be on the phone)",
[f"Open {row.get('optout_url')} and submit with only the planned fields",
"Answer the automated call and enter the 4-digit code to finish"],
prep=[f"python3 scripts/pdd.py plan {subject_id} --batch # confirm fields first"])
if req.get("account"):
return None, _digest(row, "requires creating/holding an account with the broker",
[f"Create/log in at {row.get('optout_url')} and submit the opt-out",
"Use the subject's contact email; no extra PII beyond the planned fields"])
# 3) web_form: drive the browser with the record's own playbook steps.
steps = (playbook.get(bid) or {}).get("steps") or list(row.get("optout_playbook") or []) \
or tiers.synthesize_steps(row)
action = {
"type": "optout_web_form",
"broker_id": bid, "broker_name": row.get("broker_name"), "tier": tier,
"confirm_first": confirm_first,
"optout_url": row.get("optout_url"),
"clears_children": row.get("clears_children") or [],
"steps": steps,
"after": f"python3 scripts/pdd.py record {subject_id} {bid} submitted "
f"--disclosed <field>... --channel web_form",
}
if deletion:
action["prefer_deletion"] = ("this record has a right-to-delete lane -- complete the DELETION "
"flow, not just suppression" + (f" ({deletion.get('notes')})"
if deletion.get("notes") else ""))
if req.get("captcha"):
action["note"] = ("CAPTCHA-gated: attempt with the configured browser backend once; if it "
"does not clear, record blocked (do NOT retry-loop or bypass)")
return action, None
def next_actions(dossier: dict, brokers_list: list[dict], cfg: dict,
ledger: dict | None = None, env: dict | None = None) -> dict:
env = os.environ if env is None else env
ledger = ledger or {}
subject_id = dossier.get("subject_id", "")
autonomy = cfg.get("autonomy", "full")
confirm_first = autonomy == "assisted"
email_mode = cfg.get("email_mode", "draft_only")
mail = emailer.available(env)
at = _now_iso()
batch = tiers.batch_plan(dossier, brokers_list, cfg, ledger,
browser_clears_captcha=cfg.get("browser_backend") == "browserbase"
or bool(env.get("BROWSERBASE_API_KEY")))
groups = batch["groups"]
playbook = {p["broker_id"]: p for p in batch.get("parent_playbook") or []}
by_id = {b.get("id"): b for b in brokers_list}
actions: list[dict] = []
digest: list[dict] = []
# 0) keep the broker DB fresh (autonomously)
age = cache_age_days()
if age is None or age > CACHE_STALE_DAYS:
actions.append({
"type": "refresh_brokers",
"why": "live broker cache missing" if age is None else f"cache is {age:.0f} days old",
"command": "python3 scripts/pdd.py refresh-brokers",
})
# 0b) DROP one-shot: for a CA resident, ONE request deletes from every registered
# broker (the whole CA Data Broker Registry) -- the highest-leverage removal there is.
registry_recs = brokers_mod.load_registry_cache()
residency = (dossier.get("residency_jurisdiction") or "US").upper()
drop_filed = bool((dossier.get("preferences") or {}).get("drop_filed_at"))
if registry_recs and residency.startswith("US-CA") and not drop_filed:
actions.append({
"type": "drop_submit",
"one_shot": True,
"registry_count": len(registry_recs),
"url": registry.DROP_URL,
"command": f"python3 scripts/pdd.py drop {subject_id}",
"why": f"CA resident: one DROP request deletes from all {len(registry_recs)} registered "
"data brokers at once (superset of what commercial services cover).",
"after": f"python3 scripts/pdd.py drop {subject_id} --filed",
})
# 1) Phase 1 crawl: everything unscanned (read-only, parallel-safe)
unscanned = groups.get("unscanned") or []
if unscanned:
ids = [r["broker_id"] for r in unscanned]
if len(ids) > FANOUT_THRESHOLD:
actions.append({
"type": "fanout_scan",
"broker_ids": ids,
"command": f"python3 scripts/pdd.py fanout {subject_id}",
"how": "spawn ONE delegate_task subagent per batch IN PARALLEL with each batch's brief; "
"parent re-verifies key `found` claims before trusting them",
})
else:
actions.append({
"type": "scan_inline",
"broker_ids": ids,
"command": f"python3 scripts/pdd.py plan {subject_id}",
"how": "run every search_vector per broker via the methods.md ladder "
"(web_extract -> site: probe -> browser), record a verdict per broker",
})
# 2) in-flight email verifications: poll the inbox (or hand to the human in draft mode)
for st in ("submitted", "verification_pending"):
for bid, case in sorted(ledger.items()):
if case.get("state") != st:
continue
broker = by_id.get(bid) or {}
if not ((broker.get("optout") or {}).get("requires") or {}).get("email_verification"):
continue
if mail["imap"]:
actions.append({
"type": "poll_verification", "via": "imap",
"broker_id": bid,
"command": f"python3 scripts/pdd.py poll-verification {subject_id} --broker {bid}",
"then": "browser_navigate the returned link IN THE SAME AGENT BROWSER (sessions are "
f"browser-bound), complete the flow, then record: awaiting_processing",
})
elif email_mode == "browser":
actions.append({
"type": "poll_verification", "via": "browser", "broker_id": bid,
"how": "open the broker's confirmation email in the operator's logged-in webmail "
f"(browser_*), then `python3 scripts/pdd.py verify-link {subject_id} {bid} "
"--text '<email body>'` to score the link, browser_navigate it in the SAME "
"browser, then record awaiting_processing",
})
else:
digest.append(_digest(
{"broker_id": bid, "broker_name": (broker.get("name") or bid)},
"verification email must be opened by a human (draft mode, no inbox access)",
["Open the broker's verification email in the subject's inbox and click the link",
f"Then: python3 scripts/pdd.py record {subject_id} {bid} awaiting_processing"]))
# 3) due rechecks: processing windows elapsed / reappearance sweeps
for case in ledger_mod.due(subject_id, at=at, ledger=ledger):
bid = case.get("broker_id")
st = case.get("state")
if st in ("awaiting_processing", "confirmed_removed"):
actions.append({
"type": "verify_removal",
"broker_id": bid,
"why": "processing window elapsed" if st == "awaiting_processing" else "periodic reappearance re-scan",
"how": "re-run this broker's search_vectors; if gone record confirmed_removed; "
"if still listed record reappeared and requeue the opt-out",
})
elif st in ("submitted", "verification_pending") and not mail["imap"]:
pass # already covered by the digest entry above
# 4) Phase 2 opt-outs: parents first (batch_plan already ordered them)
for row in groups.get("found") or []:
action, task = _optout_action(row, playbook, subject_id, dossier,
email_mode, mail["smtp"], confirm_first)
if action:
actions.append(action)
if task:
digest.append(task)
# 5) indirect exposure: targeted delete-my-PII requests
for row in groups.get("indirect_exposure") or []:
bid = row["broker_id"]
if (email_mode in ("programmatic", "alias") and mail["smtp"]) or email_mode == "browser":
actions.append({
"type": "indirect_email_send",
"broker_id": bid, "confirm_first": confirm_first,
"send_via": "browser" if email_mode == "browser" else "smtp",
"command": f"python3 scripts/pdd.py send-email {subject_id} {bid} --kind ccpa_indirect "
f"--listing <third-party-listing-url>",
})
else:
digest.append(_digest(row, "indirect-exposure request (draft mode: a human must hit send)",
["Send the rendered ccpa_indirect draft",
f"Then: python3 scripts/pdd.py record {subject_id} {bid} submitted "
f"--disclosed contact_email --channel email"],
prep=[f"python3 scripts/pdd.py render-email {subject_id} {bid} "
f"--kind ccpa_indirect --listing <url>"]))
# 6) blocked sites: stealth pass if we have one, else the operator-browser path
blocked = groups.get("blocked") or []
if blocked:
ids = [r["broker_id"] for r in blocked]
if bool(env.get("BROWSERBASE_API_KEY")):
actions.append({
"type": "stealth_rescan",
"broker_ids": ids,
"how": "retry these with the cloud/stealth browser backend, then record real verdicts",
})
else:
for r in blocked:
digest.append(_digest(r, "site blocks automated access (anti-bot); a human browser gets through",
["Open the paste-ready search URL from `plan` in your everyday browser",
"Report the verdict (or a screenshot) back to the agent",
f"Agent records: python3 scripts/pdd.py record {subject_id} "
f"{r['broker_id']} <found|not_found|indirect_exposure>"]))
# 7) anything already parked as a human task
for bid, case in sorted(ledger.items()):
if case.get("state") == "human_task_queued":
broker = by_id.get(bid) or {}
digest.append(_digest({"broker_id": bid, "broker_name": broker.get("name") or bid},
case.get("human_task_reason") or "queued manual step",
["See `pdd.py tasks` for the exact steps recorded with this case"]))
# registry coverage summary (breadth beyond the scannable people-search sites)
coverage = None
if registry_recs:
coverage = {
"people_search_sites": len(brokers_list),
"registered_data_brokers": len(registry_recs),
"worked_via": "CA DROP one-shot" if residency.startswith("US-CA") else "targeted CCPA/GDPR email",
}
if not residency.startswith("US-CA"):
coverage["note"] = ("DROP is CA-only; for this subject the registry is covered by targeted "
"CCPA/GDPR deletion emails (`registry --search` then `send-email`), "
"not a single portal request.")
elif drop_filed:
coverage["note"] = "DROP already filed; registry deletions are in the brokers' hands."
next_wake = _min_future_recheck(ledger, at)
return {
"subject": subject_id,
"autonomy": autonomy,
"phase": batch.get("phase"),
"counts": batch.get("counts"),
"actions": actions,
"human_digest": digest,
"coverage": coverage,
"done_for_now": not actions,
"fully_done": not actions and not digest and not next_wake,
"next_wake_at": next_wake,
"note": ("assisted mode: pause for operator confirmation on every action with confirm_first=true"
if confirm_first else
"full autonomy: recorded intake consent authorizes these submissions; do not pause. "
"Present human_digest ONCE at the end of the run, not per item."),
}

View file

@ -0,0 +1,177 @@
"""Pull and parse the Big-Ass Data Broker Opt-Out List (BADBOOL) into broker records.
BADBOOL (https://github.com/yaelwrites/Big-Ass-Data-Broker-Opt-Out-List) is a
maintained, frequently-updated markdown list. `refresh` fetches it and parses the
"People Search Sites" section into records that merge UNDER the curated DB (curated
records always win). Auto-parsed records carry source="BADBOOL-auto" and
confidence="auto" so the agent treats their URLs as best guesses to verify first.
`parse()` is pure (markdown in, records out) so it is tested offline; `fetch()` is
the only network call and can be bypassed by passing markdown directly to refresh().
"""
from __future__ import annotations
import re
import urllib.request
from pathlib import Path
import storage
DEFAULT_URL = (
"https://raw.githubusercontent.com/yaelwrites/"
"Big-Ass-Data-Broker-Opt-Out-List/master/README.md"
)
USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)"
# BADBOOL legend symbols.
SYMBOLS = {
"crucial": "\U0001F490", # 💐
"high": "\u2620", # ☠
"gov_id": "\U0001F3AB", # 🎫
"phone": "\U0001F4DE", # 📞
"payment": "\U0001F4B0", # 💰
}
_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)")
_OPTOUT_HINT = re.compile(
r"opt[\- ]?out|optout|removal|remove|suppress|control-privacy|delete", re.I
)
_FIND_HINT = re.compile(r"find|your information|search|look ?up|look for", re.I)
def slug(name: str) -> str:
# Drop a trailing .com/.org/.info on the displayed name so "FastPeopleSearch.com"
# matches the curated id "fastpeoplesearch"; keep .net/.id so distinct sites differ.
n = re.sub(r"\.(com|org|info)\b", "", name.strip(), flags=re.I)
return re.sub(r"[^a-z0-9]+", "", n.lower())
def _heading_flags(heading: str) -> tuple[str, dict]:
flags = {key: (sym in heading) for key, sym in SYMBOLS.items()}
name = heading
for sym in SYMBOLS.values():
name = name.replace(sym, "")
name = name.replace("\ufe0f", "").strip()
return name, flags
def _priority(flags: dict) -> str:
if flags["crucial"]:
return "crucial"
if flags["high"]:
return "high"
return "standard"
def _pick(links: list[tuple[str, str]], hint: re.Pattern) -> str | None:
for _text, url in links:
if hint.search(url):
return url
for text, url in links:
if hint.search(text):
return url
return None
def _clean(text: str) -> str:
return re.sub(r"\s+", " ", text).strip()[:600]
def _build(name: str, flags: dict, body: str) -> dict:
links = _LINK_RE.findall(body)
web = [(t, u) for t, u in links if u.lower().startswith("http")]
mailtos = [u[7:] for _t, u in links if u.lower().startswith("mailto:")]
optout_url = _pick(web, _OPTOUT_HINT)
search_url = _pick(web, _FIND_HINT) or (web[0][1] if web else None)
if flags["phone"]:
method = "phone"
elif optout_url:
method = "web_form"
elif mailtos:
method = "email"
else:
method = "manual"
return {
"id": slug(name),
"name": name,
"category": "people_search",
"priority": _priority(flags),
"jurisdictions": ["US"],
"search": {"method": "url_pattern", "url": search_url, "fetch": "browser",
"match_signal": "result", "by": ["name", "phone", "address"]},
"optout": {
"method": method,
"url": optout_url,
"email": mailtos[0] if mailtos else None,
"requires": {
"gov_id": flags["gov_id"],
"phone_voice": flags["phone"],
"payment": flags["payment"],
"email_verification": False,
"captcha": False,
"account": False,
"phone_callback": False,
},
"inputs": ["full_name", "contact_email"],
"notes": _clean(body),
"links": [{"text": t, "url": u} for t, u in links],
"est_processing_days": 14, # unknown for auto records; drives next_recheck_at
},
"source": "BADBOOL-auto",
"confidence": "auto",
"last_verified": None,
}
def parse(markdown: str) -> list[dict]:
"""Parse the 'People Search Sites' section of BADBOOL into broker records."""
records: list[dict] = []
in_people = False
heading: str | None = None
body: list[str] = []
def flush() -> None:
nonlocal heading, body
if heading is not None:
name, flags = _heading_flags(heading)
if name:
records.append(_build(name, flags, "\n".join(body).strip()))
heading, body = None, []
for line in markdown.splitlines():
if line.startswith("## "):
flush()
in_people = line[3:].strip().lower().startswith("people search")
continue
if not in_people:
continue
if line.startswith("### "):
flush()
heading = line[4:].strip()
elif heading is not None:
body.append(line)
flush()
return records
def fetch(url: str = DEFAULT_URL, timeout: int = 30) -> str:
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310
return resp.read().decode("utf-8", errors="replace")
MIN_EXPECTED = 20 # BADBOOL's People Search section lists ~47; far fewer => upstream reorg, warn
def refresh(cache_path: Path, url: str = DEFAULT_URL, markdown: str | None = None) -> dict:
"""Fetch (or accept) BADBOOL markdown, parse it, and write the snapshot cache."""
md = markdown if markdown is not None else fetch(url)
records = parse(md)
storage.write_json(cache_path, records)
out = {"parsed": len(records), "cache_path": str(cache_path), "source_url": url}
if len(records) < MIN_EXPECTED:
out["warning"] = (f"only {len(records)} parsed (expected >{MIN_EXPECTED}); BADBOOL's "
"'People Search Sites' section may have moved/reorganized - check the parser")
return out

View file

@ -0,0 +1,77 @@
"""Load and query the broker database (references/brokers/*.json).
Each broker is one JSON file for clean diffs/PRs. Files beginning with `_` are
ignored (reserved for notes/scratch).
"""
from __future__ import annotations
import json
from pathlib import Path
import paths
import storage
PRIORITY_ORDER = {"crucial": 0, "high": 1, "standard": 2, "long_tail": 3}
def _load_curated(directory: Path | None = None) -> list[dict]:
directory = directory or paths.brokers_dir()
out: list[dict] = []
if not directory.exists():
return out
for fp in sorted(directory.glob("*.json")):
if fp.name.startswith("_"):
continue
out.append(json.loads(fp.read_text(encoding="utf-8")))
return out
def load_live_cache() -> list[dict]:
"""Records pulled from BADBOOL via `refresh-brokers` (empty until refreshed)."""
return storage.read_json(paths.brokers_cache_path(), []) or []
def load_registry_cache() -> list[dict]:
"""CA Data Broker Registry records (separate coverage lane; empty until refreshed).
Kept OUT of load_all() by default: these are not people-search sites to scan, they
are worked via the CA DROP one-shot + CCPA email. Consumers of the scan/plan/fanout
pipeline must not receive them; use this directly for coverage counts and the DROP/
email lanes.
"""
return storage.read_json(paths.registry_cache_path(), []) or []
def load_all(directory: Path | None = None, include_live: bool = True) -> list[dict]:
"""Curated records, with live BADBOOL records merged underneath (curated wins)."""
merged: dict[str, dict] = {b["id"]: b for b in _load_curated(directory)}
if include_live:
for b in load_live_cache():
bid = b.get("id")
if bid and bid not in merged:
merged[bid] = b
out = list(merged.values())
out.sort(key=lambda b: (PRIORITY_ORDER.get(b.get("priority", "standard"), 9), b.get("id", "")))
return out
def get(broker_id: str, directory: Path | None = None) -> dict | None:
for b in load_all(directory):
if b.get("id") == broker_id:
return b
return None
def by_priority(*levels: str, directory: Path | None = None) -> list[dict]:
wanted = set(levels) if levels else None
return [b for b in load_all(directory) if wanted is None or b.get("priority") in wanted]
def clusters(directory: Path | None = None) -> dict[str, list[str]]:
"""Map a parent broker id -> child site ids it can clear (force-multipliers)."""
out: dict[str, list[str]] = {}
for b in load_all(directory):
owns = b.get("owns") or []
if owns:
out[b["id"]] = list(owns)
return out

View file

@ -0,0 +1,124 @@
"""Install-wide configuration with easiest-first defaults.
Everything works zero-config. `setup --auto` (the autonomous path) detects what
this environment can do and picks the MOST AUTONOMOUS valid configuration without
asking anyone; plain `setup` keeps the easiest-first defaults and only upgrades a
setting when a flag opts in.
`autonomy` is policy, orthogonal to capability:
full - intake consent is standing authorization; the agent submits T0-T2
opt-outs without pausing per submission (default).
assisted - the agent pauses for operator confirmation before each submission.
"""
from __future__ import annotations
import os
from pathlib import Path
from shutil import which
import emailer
import paths
import storage
DEFAULT_CONFIG = {
"autonomy": "full", # hands-off after intake+consent
"email_mode": "draft_only", # zero credentials
"browser_backend": "auto", # auto = Browserbase when BROWSERBASE_API_KEY is set
# (recommended default; clears soft CAPTCHAs), else plain browser
"tracker_backend": "local-json", # no external dependency
"encryption": "none", # files still written 0600
"default_rescan_interval_days": 120,
"email_min_interval_seconds": 20, # pace SMTP sends so a run can't torch the account
}
VALID = {
"autonomy": {"full", "assisted"},
# email_mode:
# draft_only - render drafts; the operator sends + clicks verify links (zero setup)
# browser - the agent sends + opens verify links through the operator's logged-in
# webmail via browser_* tools (NO password stored; needs a browser the
# operator's inbox is signed into)
# programmatic - CLI sends via SMTP + reads verify links via IMAP (needs EMAIL_* creds)
# alias - AgentMail agent-owned inboxes / per-broker aliases
"email_mode": {"draft_only", "browser", "programmatic", "alias"},
"browser_backend": {"auto", "browserbase", "agent-browser", "camofox"},
"tracker_backend": {"local-json", "google-sheets"},
"encryption": {"none", "age"},
}
def load_config() -> dict:
cfg = dict(DEFAULT_CONFIG)
cfg.update(storage.read_json(paths.config_path(), {}) or {})
return cfg
def save_config(cfg: dict) -> Path:
merged = dict(DEFAULT_CONFIG)
merged.update(cfg)
for key, allowed in VALID.items():
if merged.get(key) not in allowed:
raise ValueError(f"invalid {key!r}: {merged.get(key)!r} (allowed: {sorted(allowed)})")
return storage.write_json(paths.config_path(), merged)
def detect_capabilities(env: dict | None = None) -> dict:
"""Report which opt-in upgrades are available without extra setup."""
env = os.environ if env is None else env
home = paths.hermes_home()
google = (
(home / "google_token.json").exists()
or (home / "skills" / "productivity" / "google-workspace").exists()
or (home / "skills" / "google-workspace").exists()
)
mail = emailer.available(env)
return {
"browserbase": bool(env.get("BROWSERBASE_API_KEY")),
"agentmail": bool(env.get("AGENTMAIL_API_KEY")),
"email_imap_smtp": bool(env.get("EMAIL_ADDRESS") and env.get("EMAIL_PASSWORD")),
"smtp_send": mail["smtp"], # CLI can SEND opt-out emails itself
"imap_read": mail["imap"], # CLI can POLL verification links itself
"google_workspace": google,
"age": which("age") is not None,
}
def auto_configure(env: dict | None = None) -> dict:
"""Pick the most autonomous configuration this environment supports (no questions).
- email: programmatic when SMTP creds exist (CLI sends + IMAP-verifies itself);
alias mode when only AgentMail exists; draft_only as the capability floor.
- browser: browserbase when the key exists (clears soft CAPTCHAs -> more T1).
- encryption: age when the binary is installed (free privacy, zero human cost).
- tracker: stays local-json (google-sheets needs a sheet id -> a human choice).
"""
caps = detect_capabilities(env)
cfg = load_config()
cfg["autonomy"] = "full"
if caps["smtp_send"]:
cfg["email_mode"] = "programmatic"
elif caps["agentmail"]:
cfg["email_mode"] = "alias"
else:
cfg["email_mode"] = "draft_only"
cfg["browser_backend"] = "browserbase" if caps["browserbase"] else "auto"
if caps["age"]:
cfg["encryption"] = "age"
return cfg
def browser_clears_captcha(cfg: dict, env: dict | None = None) -> bool:
"""True if the chosen browser backend can clear soft CAPTCHAs (shifts T2 -> T1).
Browserbase is the recommended default: a real residential-IP cloud browser passes
soft/managed challenges (Turnstile, hCaptcha/reCAPTCHA checkbox) as normal operation.
This is NOT solving/spoofing - hard interactive challenges still escalate to a human.
`auto` inherits this whenever BROWSERBASE_API_KEY is present.
"""
backend = cfg.get("browser_backend", "auto")
if backend == "browserbase":
return True
if backend == "auto":
env = os.environ if env is None else env
return bool(env.get("BROWSERBASE_API_KEY"))
return False

View file

@ -0,0 +1,88 @@
"""At-rest encryption for sensitive files via the `age` binary (optional).
Engaged ONLY when config `encryption: age` AND an age identity key exists AND the
`age`/`age-keygen` binaries are available. When engaged, JSON docs under
`subjects/` (dossier, ledger) are written as `<file>.age` ciphertext; the audit
log (field NAMES + states only, no raw PII values), `config.json`, and the broker
cache stay plaintext so the engine can read them.
Threat model (be honest): this protects against casual disk inspection, accidental
`git add`/commits, screen-shares, and backup/cloud-sync leakage. The identity key
defaults to living beside the data at `$PDD_DATA_DIR/age-identity.txt` (0600); set
`PDD_AGE_IDENTITY` to a separate volume/token for true key separation. It does NOT
protect against an attacker who can already read your whole HERMES_HOME (they get
key + data together).
"""
from __future__ import annotations
import json
import subprocess
from pathlib import Path
from shutil import which
import paths
def age_available() -> bool:
return which("age") is not None and which("age-keygen") is not None
def encryption_setting() -> str:
"""Read `encryption` straight from config.json (no config/storage import => no cycle)."""
cfg = paths.config_path()
if not cfg.exists():
return "none"
try:
return (json.loads(cfg.read_text(encoding="utf-8")) or {}).get("encryption", "none")
except (ValueError, OSError):
return "none"
def identity_path() -> Path:
return paths.age_identity_path()
def ensure_identity() -> Path:
"""Generate an age identity (X25519 keypair) if missing; return its path."""
if not age_available():
raise RuntimeError("`age`/`age-keygen` not found; cannot enable encryption")
p = identity_path()
if not p.exists():
p.parent.mkdir(parents=True, exist_ok=True)
try:
p.parent.chmod(0o700)
except OSError:
pass
subprocess.run(["age-keygen", "-o", str(p)], check=True, capture_output=True)
try:
p.chmod(0o600)
except OSError:
pass
return p
def recipient() -> str:
"""The age public key (recipient) for the identity, parsed from its header."""
p = ensure_identity()
for line in p.read_text(encoding="utf-8").splitlines():
s = line.strip()
if s.lower().startswith("# public key:"):
return s.split(":", 1)[1].strip()
if s.startswith("age1"):
return s
raise RuntimeError(f"no public key found in {p}")
def is_engaged() -> bool:
"""True only when encryption is actually active (configured + available + key present)."""
return encryption_setting() == "age" and age_available() and identity_path().exists()
def encrypt(data: bytes) -> bytes:
out = subprocess.run(["age", "-r", recipient()], input=data, capture_output=True, check=True)
return out.stdout
def decrypt(data: bytes) -> bytes:
out = subprocess.run(["age", "-d", "-i", str(identity_path())], input=data, capture_output=True, check=True)
return out.stdout

View file

@ -0,0 +1,135 @@
"""Subject dossier management + consent gate + least-disclosure field selection."""
from __future__ import annotations
import datetime as _dt
import hashlib
import os
from pathlib import Path
import paths
import storage
# Identifiers we never volunteer in an opt-out (would expand exposure, not reduce it).
NEVER_VOLUNTEER = {"ssn", "social_security_number", "passport", "drivers_license"}
VALID_CONSENT_METHODS = {"self", "written_authorization", "poa"}
def now() -> str:
return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def new_subject_id(full_name: str = "") -> str:
# Opaque id: derives NOTHING from the name, so PII never leaks into directory names,
# case ids, drafts, or the audit log. full_name kept only for call compatibility.
return "sub_" + hashlib.sha1(os.urandom(8)).hexdigest()[:10]
def create(identity: dict, consent: dict, residency: str = "US", prefs: dict | None = None) -> dict:
dossier = {
"subject_id": new_subject_id(identity.get("full_name", "subject")),
"consent": consent,
"identity": identity,
"residency_jurisdiction": residency,
"preferences": prefs or {"email_mode": "draft_only", "rescan_interval_days": 120},
"created_at": now(),
}
save(dossier)
return dossier
def load(subject_id: str) -> dict | None:
return storage.read_json(paths.dossier_path(subject_id), None)
def save(dossier: dict) -> Path:
return storage.write_json(paths.dossier_path(dossier["subject_id"]), dossier)
def is_authorized(dossier: dict) -> bool:
c = dossier.get("consent") or {}
return bool(c.get("authorized")) and c.get("method") in VALID_CONSENT_METHODS
def require_authorized(dossier: dict) -> None:
if not is_authorized(dossier):
raise PermissionError(
f"subject {dossier.get('subject_id')!r} has no recorded authorization; refusing to act"
)
def all_names(dossier: dict) -> list[str]:
"""Primary name + aliases (maiden/married/nicknames), deduped, in priority order."""
ident = dossier.get("identity", {})
out: list[str] = []
seen: set[str] = set()
for n in [ident.get("full_name"), *(ident.get("also_known_as") or [])]:
if n and n.lower() not in seen:
seen.add(n.lower())
out.append(n)
return out
def all_addresses(dossier: dict) -> list[dict]:
"""Current + prior addresses, each tagged with `kind` (current|prior)."""
ident = dossier.get("identity", {})
out: list[dict] = []
cur = ident.get("current_address")
if cur:
out.append({**cur, "kind": cur.get("kind", "current")})
for a in ident.get("prior_addresses") or []:
out.append({**a, "kind": a.get("kind", "prior")})
return out
def all_locations(dossier: dict) -> list[dict]:
"""Distinct city/state pairs across all addresses (the vectors for name searches)."""
out: list[dict] = []
seen: set[tuple] = set()
for a in all_addresses(dossier):
city = a.get("city")
key = ((city or "").lower(), (a.get("state") or "").lower())
if city and key not in seen:
seen.add(key)
out.append({"city": city, "state": a.get("state")})
return out
def contact_email(dossier: dict) -> str | None:
"""The single email used for opt-out correspondence (designated, else the first)."""
ident = dossier.get("identity", {})
prefs = dossier.get("preferences", {})
emails = ident.get("emails") or []
return prefs.get("contact_email_for_optouts") or (emails[0] if emails else None)
def select_disclosure(dossier: dict, inputs: list[str], override_email: str | None = None) -> dict:
"""Return ONLY the dossier fields a broker's opt-out actually requires.
Enforces least-disclosure: skips anything in NEVER_VOLUNTEER, and skips
`profile_url` (that is captured per-listing at submit time, not from the dossier).
A single contact email is used for correspondence even when the subject has several
(see all_names / all_addresses / search vectors for using every alternate to *find* listings).
"""
ident = dossier.get("identity", {})
addr = ident.get("current_address") or {}
phones = ident.get("phones") or []
available = {
"full_name": ident.get("full_name"),
"first_name": (ident.get("full_name") or "").split(" ")[0] or None,
"contact_email": override_email or contact_email(dossier),
"current_address": addr or None,
"street": addr.get("line1"),
"city": addr.get("city"),
"state": addr.get("state"),
"postal": addr.get("postal"),
"date_of_birth": ident.get("date_of_birth"),
"phone": phones[0] if phones else None,
}
out: dict = {}
for key in inputs:
if key in NEVER_VOLUNTEER or key == "profile_url":
continue
if available.get(key) is not None:
out[key] = available[key]
return out

View file

@ -0,0 +1,76 @@
"""Email modes A/B/C helpers + anti-phishing verification-link extraction.
Mode A (default): render a ready-to-send draft to disk; the operator sends it.
Mode B/C: the agent SENDS via a Hermes email mechanism (IMAP/SMTP gateway,
`himalaya`, AgentMail, or Gmail via `google-workspace`) and READS the reply to
resolve the verification link with `extract_verification_link`. Those transports
are driven by the agent through native tools; this module stays network-free so
the hermetic tests pass.
"""
from __future__ import annotations
import re
from pathlib import Path
import legal
import paths
_LINK_RE = re.compile(r"https?://[^\s\"'<>)\]]+", re.IGNORECASE)
_VERIFY_HINTS = ("opt", "remov", "verif", "confirm", "unsubscrib", "suppress", "delete", "privacy")
def render_draft(broker: dict, fields: dict, out_dir: Path | None = None) -> Path:
"""Mode A: write a ready-to-send opt-out email for the operator to send."""
body = legal.render_optout_email(broker, fields)
out_dir = out_dir or (paths.data_dir() / "drafts")
out_dir.mkdir(parents=True, exist_ok=True)
fp = out_dir / f"{broker.get('id', 'broker')}.txt"
fp.write_text(body, encoding="utf-8")
return fp
def render_request_draft(broker: dict, fields: dict, kind: str = "generic",
out_dir: Path | None = None) -> Path:
"""Mode A: write a ready-to-send request of a specific KIND.
kind: generic | ccpa | ccpa_agent | ccpa_indirect | gdpr. Used for indirect-exposure
(ccpa_indirect) and explicit legal requests, where the generic opt-out wording is wrong.
The filename is suffixed with the kind so an indirect request does not overwrite an opt-out draft.
"""
body = legal.render_request(kind, broker, fields)
out_dir = out_dir or (paths.data_dir() / "drafts")
out_dir.mkdir(parents=True, exist_ok=True)
suffix = "" if kind == "generic" else f"-{kind}"
fp = out_dir / f"{broker.get('id', 'broker')}{suffix}.txt"
fp.write_text(body, encoding="utf-8")
return fp
def extract_verification_link(email_body: str, broker: dict | None = None) -> str | None:
"""Return the most likely opt-out/verification link from an email body.
Anti-phishing: a link is only returned if its URL matches an opt-out hint
and/or the broker's own domain; arbitrary links score 0 and are ignored.
"""
candidates = _LINK_RE.findall(email_body or "")
if not candidates:
return None
domain = ""
if broker:
url = (broker.get("optout") or {}).get("url") or (broker.get("search") or {}).get("url") or ""
m = re.search(r"https?://([^/]+)", url)
if m:
domain = m.group(1).replace("www.", "")
best_score, best_link = 0, None
for link in candidates:
low = link.lower()
score = 0
if any(h in low for h in _VERIFY_HINTS):
score += 2
if domain and domain in low:
score += 3
if score > best_score:
best_score, best_link = score, link
return best_link

View file

@ -0,0 +1,342 @@
"""Programmatic email (Mode B) via stdlib smtplib/imaplib - no human in the loop.
This is what turns email opt-outs autonomous: `send()` delivers the rendered
request straight to the broker's known opt-out address, and `find_verification_link()`
polls the inbox for the broker's confirmation email and extracts the link (scored
by email_modes.extract_verification_link, so arbitrary/phishing links are ignored).
The agent still OPENS the link with its own browser - several brokers bind the
verification session to the browser that opens it (see the intelius record).
Configuration comes from the same env vars the Hermes email gateway uses:
EMAIL_ADDRESS / EMAIL_PASSWORD (required for Mode B)
EMAIL_SMTP_HOST / EMAIL_SMTP_PORT (optional; inferred for common providers)
EMAIL_IMAP_HOST / EMAIL_IMAP_PORT (optional; inferred for common providers)
Anti-misuse: `send()` refuses a recipient that is not the broker record's own
opt-out/privacy address - this module cannot be repurposed to email arbitrary people.
All network calls live behind small functions that the hermetic tests monkeypatch.
"""
from __future__ import annotations
import email as _email
import email.utils
import imaplib
import json
import os
import re
import smtplib
import time
from email.message import EmailMessage
from pathlib import Path
import email_modes
import paths
# provider domain -> (smtp_host, smtp_port, imap_host, imap_port)
PROVIDERS = {
"gmail.com": ("smtp.gmail.com", 587, "imap.gmail.com", 993),
"googlemail.com": ("smtp.gmail.com", 587, "imap.gmail.com", 993),
"outlook.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993),
"hotmail.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993),
"live.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993),
"yahoo.com": ("smtp.mail.yahoo.com", 587, "imap.mail.yahoo.com", 993),
"icloud.com": ("smtp.mail.me.com", 587, "imap.mail.me.com", 993),
"me.com": ("smtp.mail.me.com", 587, "imap.mail.me.com", 993),
"fastmail.com": ("smtp.fastmail.com", 587, "imap.fastmail.com", 993),
}
def _domain(address: str) -> str:
return address.rsplit("@", 1)[-1].lower() if "@" in address else ""
def smtp_settings(env: dict | None = None) -> dict | None:
"""SMTP connection settings, or None when sending is not configured."""
env = os.environ if env is None else env
address, password = env.get("EMAIL_ADDRESS"), env.get("EMAIL_PASSWORD")
if not (address and password):
return None
inferred = PROVIDERS.get(_domain(address))
host = env.get("EMAIL_SMTP_HOST") or (inferred[0] if inferred else None)
if not host:
return None # unknown provider and no explicit host
port = int(env.get("EMAIL_SMTP_PORT") or (inferred[1] if inferred else 587))
return {"host": host, "port": port, "address": address, "password": password}
def imap_settings(env: dict | None = None) -> dict | None:
"""IMAP connection settings, or None when inbox reading is not configured."""
env = os.environ if env is None else env
address, password = env.get("EMAIL_ADDRESS"), env.get("EMAIL_PASSWORD")
if not (address and password):
return None
inferred = PROVIDERS.get(_domain(address))
host = env.get("EMAIL_IMAP_HOST") or (inferred[2] if inferred else None)
if not host:
return None
port = int(env.get("EMAIL_IMAP_PORT") or (inferred[3] if inferred else 993))
return {"host": host, "port": port, "address": address, "password": password}
def available(env: dict | None = None) -> dict:
return {"smtp": smtp_settings(env) is not None, "imap": imap_settings(env) is not None}
# --- sending ------------------------------------------------------------------
def broker_addresses(broker: dict) -> list[str]:
"""Every address the broker record itself declares (the ONLY valid recipients).
Includes the primary opt-out email, the right-to-delete lane's email
(optout.deletion.email), and any mailto: links parsed from BADBOOL.
"""
opt = broker.get("optout") or {}
out = [a for a in [opt.get("email"), (opt.get("deletion") or {}).get("email")] if a]
for link in opt.get("links") or []:
url = (link.get("url") or "")
if url.lower().startswith("mailto:"):
out.append(url[7:].split("?")[0])
seen: set[str] = set()
deduped = []
for a in out:
if a.lower() not in seen:
seen.add(a.lower())
deduped.append(a)
return deduped
def _split_subject_body(text: str) -> tuple[str, str]:
"""Templates start with a 'Subject: ...' line; split it out for the MIME header."""
lines = text.splitlines()
if lines and lines[0].lower().startswith("subject:"):
return lines[0].split(":", 1)[1].strip(), "\n".join(lines[1:]).lstrip("\n")
return "Data removal request", text
def browser_send_payload(broker: dict, body_text: str, to: str | None = None) -> dict:
"""Build a recipient-locked {to, subject, body} for the agent to send via browser webmail.
No network and no credentials: the deterministic part (recipient-lock to the broker's own
declared address, subject/body split) happens here; the agent then composes and sends it in
the operator's logged-in webmail with browser_* tools. Same recipient guard as `send()`, so
the browser lane cannot be pointed at an arbitrary person either.
"""
allowed = broker_addresses(broker)
if not allowed:
raise RuntimeError(f"broker {broker.get('id')!r} declares no opt-out email address")
recipient = to or allowed[0]
if recipient.lower() not in {a.lower() for a in allowed}:
raise PermissionError(
f"refusing to target {recipient!r}: not an address the broker record declares "
f"(allowed: {allowed})"
)
subject, body = _split_subject_body(body_text)
return {"to": recipient, "subject": subject, "body": body}
def _rate_limit_path() -> Path:
return paths.data_dir() / "email-rate.json"
def _respect_rate_limit(min_interval: float, sleep, now, state_path=None) -> None:
"""Pace sends across CLI invocations so a run can't torch the sending account.
Persists the last-send wall-clock time; if the next send is too soon, sleep the
remainder. Cross-process because each `send-email` is a separate invocation.
"""
if min_interval <= 0:
return
p = state_path or _rate_limit_path()
last = 0.0
try:
last = float(json.loads(p.read_text(encoding="utf-8")).get("last", 0.0))
except (OSError, ValueError, TypeError):
last = 0.0
wait = min_interval - (now() - last)
if wait > 0:
sleep(min(wait, min_interval))
try:
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(json.dumps({"last": now()}), encoding="utf-8")
except OSError:
pass
# SMTP errors that are permanent (don't retry) vs transient (retry with backoff).
_SMTP_PERMANENT = (smtplib.SMTPAuthenticationError, smtplib.SMTPRecipientsRefused,
smtplib.SMTPSenderRefused, smtplib.SMTPDataError)
def send(broker: dict, body_text: str, to: str | None = None,
env: dict | None = None, _smtp_factory=None,
min_interval: float = 0.0, max_retries: int = 3,
_sleep=time.sleep, _now=time.time, _rate_state=None) -> dict:
"""Send an opt-out/legal request to the broker's own opt-out address.
Recipient is locked to an address the broker record declares (PermissionError
otherwise). `min_interval` paces sends across invocations (deliverability /
account-safety); transient SMTP/socket failures retry with exponential backoff,
permanent ones (auth, recipient refused) raise immediately. NOTE: a successful
SMTP handoff is NOT proof of delivery - real bounces arrive later as inbound mail;
in programmatic mode `poll-verification`/inbox review surfaces them, and the
due-queue re-scan is the true confirmation. Returns send metadata.
"""
settings = smtp_settings(env)
if not settings:
raise RuntimeError(
"programmatic email not configured (need EMAIL_ADDRESS + EMAIL_PASSWORD, and "
"EMAIL_SMTP_HOST for non-mainstream providers); fall back to `render-email` drafts"
)
allowed = broker_addresses(broker)
if not allowed:
raise RuntimeError(f"broker {broker.get('id')!r} declares no opt-out email address")
recipient = to or allowed[0]
if recipient.lower() not in {a.lower() for a in allowed}:
raise PermissionError(
f"refusing to send to {recipient!r}: not an address the broker record declares "
f"(allowed: {allowed})"
)
subject, body = _split_subject_body(body_text)
msg = EmailMessage()
msg["From"] = settings["address"]
msg["To"] = recipient
msg["Subject"] = subject
msg["Date"] = email.utils.formatdate(localtime=True)
msg["Message-ID"] = email.utils.make_msgid()
msg.set_content(body)
_respect_rate_limit(min_interval, _sleep, _now, _rate_state)
factory = _smtp_factory or smtplib.SMTP
attempts = 0
while True:
attempts += 1
try:
with factory(settings["host"], settings["port"], timeout=30) as smtp:
smtp.ehlo()
try:
smtp.starttls()
smtp.ehlo()
except smtplib.SMTPNotSupportedError:
pass # already-TLS ports / test doubles
smtp.login(settings["address"], settings["password"])
smtp.send_message(msg)
break
except _SMTP_PERMANENT:
raise # auth / recipient refused: retrying won't help
except (smtplib.SMTPException, OSError) as exc:
if attempts > max_retries:
raise RuntimeError(f"SMTP send failed after {attempts} attempts: {exc}") from exc
_sleep(min(2 ** (attempts - 1), 30)) # 1s, 2s, 4s... capped
return {"to": recipient, "subject": subject, "message_id": msg["Message-ID"],
"from": settings["address"], "attempts": attempts,
"delivery_note": "SMTP accepted; not proof of delivery - a bounce would arrive as "
"inbound mail. The due-queue re-scan is the real confirmation."}
# --- inbox polling ------------------------------------------------------------
def _decode_part(part) -> str:
try:
payload = part.get_payload(decode=True)
if payload is None:
return ""
charset = part.get_content_charset() or "utf-8"
return payload.decode(charset, errors="replace")
except Exception: # noqa: BLE001 - malformed MIME must not kill the poll
return ""
def message_text(msg) -> str:
"""All text/plain + text/html content of a parsed email message."""
chunks: list[str] = []
if msg.is_multipart():
for part in msg.walk():
if part.get_content_type() in ("text/plain", "text/html"):
chunks.append(_decode_part(part))
else:
chunks.append(_decode_part(msg))
return "\n".join(c for c in chunks if c)
def _broker_domains(broker: dict) -> list[str]:
"""Domains this broker legitimately mails from (site domains + optout email domain)."""
domains: list[str] = []
for section in ("optout", "search"):
url = ((broker.get(section) or {}).get("url")) or ""
m = re.search(r"https?://([^/]+)", url)
if m:
domains.append(m.group(1).lower().removeprefix("www."))
opt_email = (broker.get("optout") or {}).get("email")
if opt_email and "@" in opt_email:
domains.append(_domain(opt_email))
# strip subdomains to the registrable-ish tail (mailer.intelius.com -> intelius.com)
tails = {".".join(d.split(".")[-2:]) for d in domains if d}
return sorted(tails)
def fetch_recent(env: dict | None = None, since_days: int = 3, limit: int = 30,
_imap_factory=None) -> list[dict]:
"""Fetch recent inbox messages: [{from, subject, date, text}], newest first."""
settings = imap_settings(env)
if not settings:
raise RuntimeError("IMAP not configured (need EMAIL_ADDRESS + EMAIL_PASSWORD, and "
"EMAIL_IMAP_HOST for non-mainstream providers)")
import datetime as _dt
since = (_dt.date.today() - _dt.timedelta(days=max(0, since_days))).strftime("%d-%b-%Y")
factory = _imap_factory or imaplib.IMAP4_SSL
conn = factory(settings["host"], settings["port"])
try:
conn.login(settings["address"], settings["password"])
conn.select("INBOX", readonly=True)
_typ, data = conn.search(None, "SINCE", since)
ids = (data[0].split() if data and data[0] else [])[-limit:]
out: list[dict] = []
for mid in reversed(ids): # newest first
_typ, msg_data = conn.fetch(mid, "(RFC822)")
raw = next((p[1] for p in msg_data or [] if isinstance(p, tuple)), None)
if not raw:
continue
msg = _email.message_from_bytes(raw)
out.append({
"from": msg.get("From", ""),
"subject": msg.get("Subject", ""),
"date": msg.get("Date", ""),
"text": message_text(msg),
})
return out
finally:
try:
conn.logout()
except Exception: # noqa: BLE001
pass
def link_from_messages(messages: list[dict], broker: dict) -> dict | None:
"""Pure: find the broker's verification link in already-fetched messages.
A message is only considered if its From domain OR any contained link matches
the broker's own domains; the link itself must pass the anti-phishing scorer.
"""
domains = _broker_domains(broker)
for m in messages:
sender = (m.get("from") or "").lower()
text = m.get("text") or ""
sender_match = any(d in sender for d in domains)
body_match = any(d in text.lower() for d in domains)
if not (sender_match or body_match):
continue
link = email_modes.extract_verification_link(text, broker)
if link:
return {"link": link, "from": m.get("from"), "subject": m.get("subject"),
"date": m.get("date")}
return None
def find_verification_link(broker: dict, env: dict | None = None, since_days: int = 3,
_imap_factory=None) -> dict | None:
"""Poll the inbox and return the broker's verification link (or None yet)."""
messages = fetch_recent(env, since_days=since_days, _imap_factory=_imap_factory)
return link_from_messages(messages, broker)

View file

@ -0,0 +1,164 @@
"""Case ledger: opt-out state machine + append-only audit log.
A "case" is one (subject x broker) record. State changes are validated against
TRANSITIONS and mirrored into audit.jsonl so every action is auditable.
"""
from __future__ import annotations
import datetime as _dt
from pathlib import Path
import paths
import storage
STATES = [
"new", "searching", "not_found", "found", "indirect_exposure", "action_selected", "submitted",
"verification_pending", "awaiting_processing", "confirmed_removed", "reappeared",
"human_task_queued", "blocked",
]
TRANSITIONS: dict[str, set[str]] = {
"new": {"searching", "found", "not_found", "indirect_exposure", "blocked"},
"searching": {"not_found", "found", "indirect_exposure", "blocked"},
"not_found": {"searching", "found", "indirect_exposure", "blocked"},
"found": {"action_selected", "submitted", "human_task_queued", "indirect_exposure", "blocked"},
# indirect_exposure: subject's PII (email/phone/name) sits on a THIRD PARTY's record. The
# self-service opt-out form does not apply; the lever is a targeted CCPA/GDPR delete-my-PII
# request (-> submitted) or a human task. Re-scan can clear it (-> not_found) or upgrade it to a
# direct listing (-> found).
"indirect_exposure": {"submitted", "human_task_queued", "not_found", "found", "blocked"},
"action_selected": {"submitted", "human_task_queued", "blocked"},
"submitted": {"verification_pending", "awaiting_processing", "human_task_queued", "blocked"},
# verification_pending -> awaiting_processing: the verify link was opened/acknowledged and the
# broker is now processing the removal (their stated window). confirmed_removed still requires a
# verifying re-scan, never the submission flow's own say-so.
"verification_pending": {"awaiting_processing", "confirmed_removed", "human_task_queued", "blocked"},
"awaiting_processing": {"confirmed_removed", "human_task_queued", "blocked"},
"confirmed_removed": {"reappeared", "confirmed_removed"},
"reappeared": {"found", "indirect_exposure"},
"human_task_queued": {
"found", "indirect_exposure", "action_selected", "submitted", "verification_pending",
"awaiting_processing", "confirmed_removed", "blocked",
},
# blocked: automated tools (web_extract/proxyless browser) couldn't read the site. A later pass
# -- a stealth/cloud browser OR guiding the operator's own (residential) browser -- can resolve it
# to any real scan verdict, so blocked reaches not_found / indirect_exposure too, not just found.
"blocked": {"searching", "found", "not_found", "indirect_exposure", "action_selected"},
}
def now() -> str:
return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def load(subject_id: str) -> dict:
return storage.read_json(paths.ledger_path(subject_id), {}) or {}
def save(subject_id: str, ledger: dict) -> Path:
return storage.write_json(paths.ledger_path(subject_id), ledger)
def new_case(subject_id: str, broker_id: str) -> dict:
return {
"case_id": f"case_{subject_id}_{broker_id}",
"subject_id": subject_id,
"broker_id": broker_id,
"state": "new",
"found": None,
"evidence": {},
"disclosure_log": [],
"history": [],
}
def get_case(subject_id: str, broker_id: str) -> dict:
return load(subject_id).get(broker_id) or new_case(subject_id, broker_id)
def can_transition(old: str, new: str) -> bool:
return new == old or new in TRANSITIONS.get(old, set())
def transition(subject_id: str, broker_id: str, new_state: str, **fields) -> dict:
if new_state not in STATES:
raise ValueError(f"unknown state {new_state!r}")
# Lock the whole load-modify-save so a concurrent cron re-scan / other tenant
# can't read a stale ledger and clobber this transition.
with storage.locked(paths.ledger_path(subject_id)):
ledger = load(subject_id)
case = ledger.get(broker_id) or new_case(subject_id, broker_id)
old = case.get("state", "new")
if not can_transition(old, new_state):
raise ValueError(f"illegal transition {old!r} -> {new_state!r} for broker {broker_id!r}")
case["state"] = new_state
for key, value in fields.items():
case[key] = value
stamp = now()
case.setdefault("history", []).append({"at": stamp, "from": old, "to": new_state})
ledger[broker_id] = case
save(subject_id, ledger)
storage.append_jsonl(
paths.audit_path(subject_id),
{"at": stamp, "broker_id": broker_id, "event": "transition", "from": old, "to": new_state},
)
return case
DEFAULT_PROCESSING_DAYS = 14 # when a broker record doesn't state est_processing_days
VERIFICATION_POLL_DAYS = 1 # how soon to re-poll for an unarrived verification email
def _plus_days(days: int, start: str | None = None) -> str:
base = _dt.datetime.strptime(start, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=_dt.timezone.utc) \
if start else _dt.datetime.now(_dt.timezone.utc)
return (base + _dt.timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ")
def followup_fields(new_state: str, broker: dict | None = None,
dossier: dict | None = None) -> dict:
"""Auto-scheduling stamps for a transition, so nobody has to remember follow-ups.
submitted / awaiting_processing -> recheck after the broker's stated processing window;
verification_pending -> re-poll the inbox quickly;
confirmed_removed -> periodic reappearance re-scan per subject preference.
"""
if new_state in ("submitted", "awaiting_processing"):
days = ((broker or {}).get("optout") or {}).get("est_processing_days") or DEFAULT_PROCESSING_DAYS
return {"next_recheck_at": _plus_days(int(days))}
if new_state == "verification_pending":
return {"next_recheck_at": _plus_days(VERIFICATION_POLL_DAYS)}
if new_state == "confirmed_removed":
interval = ((dossier or {}).get("preferences") or {}).get("rescan_interval_days") or 120
return {"removal_confirmed_at": now(), "next_recheck_at": _plus_days(int(interval))}
return {}
def due(subject_id: str, at: str | None = None, ledger: dict | None = None) -> list[dict]:
"""Cases whose next_recheck_at has arrived - the autonomous follow-up queue."""
stamp = at or now()
out = []
for case in (ledger if ledger is not None else load(subject_id)).values():
when = case.get("next_recheck_at")
if when and when <= stamp:
out.append(case)
out.sort(key=lambda c: c.get("next_recheck_at") or "")
return out
def log_disclosure(subject_id: str, broker_id: str, fields: list[str], channel: str) -> dict:
"""Record exactly which PII field *names* were disclosed to a broker."""
with storage.locked(paths.ledger_path(subject_id)):
ledger = load(subject_id)
case = ledger.get(broker_id) or new_case(subject_id, broker_id)
stamp = now()
record = {"at": stamp, "fields": sorted(fields), "channel": channel}
case.setdefault("disclosure_log", []).append(record)
ledger[broker_id] = case
save(subject_id, ledger)
storage.append_jsonl(
paths.audit_path(subject_id),
{"at": stamp, "broker_id": broker_id, "event": "disclosure",
"fields": record["fields"], "channel": channel},
)
return record

View file

@ -0,0 +1,63 @@
"""Render opt-out / legal request text from templates/ with safe substitution.
Templates use {field} placeholders. Missing fields are left literal (never crash,
never inject blanks that look like real data). Field values come from the
least-disclosure selection in dossier.select_disclosure.
"""
from __future__ import annotations
from pathlib import Path
import paths
class _SafeDict(dict):
def __missing__(self, key): # leave unknown placeholders untouched
return "{" + key + "}"
def template_path(name: str) -> Path:
return paths.templates_dir() / name
def render(template_name: str, fields: dict) -> str:
text = template_path(template_name).read_text(encoding="utf-8")
return text.format_map(_SafeDict(fields))
def _join_listings(value) -> str:
if isinstance(value, (list, tuple)):
return "\n".join(str(v) for v in value)
return str(value or "")
def _join_identifiers(value) -> str:
"""Render the subject's OWN identifiers as a bullet list for an indirect-exposure request."""
if isinstance(value, (list, tuple)):
return "\n".join(f" - {v}" for v in value if v)
return f" - {value}" if value else ""
def render_optout_email(broker: dict, fields: dict) -> str:
ctx = dict(fields)
ctx.setdefault("broker_name", broker.get("name", "the data broker"))
ctx["listing_urls"] = _join_listings(fields.get("listing_urls"))
ctx.setdefault("full_name", fields.get("full_name", "[your name]"))
ctx.setdefault("contact_email", fields.get("contact_email", "[your email]"))
return render("emails/generic-optout.txt", ctx)
def render_request(kind: str, broker: dict, fields: dict) -> str:
"""kind: generic | ccpa | ccpa_agent | ccpa_indirect | gdpr"""
template = {
"generic": "emails/generic-optout.txt",
"ccpa": "emails/ccpa-deletion.txt",
"ccpa_agent": "emails/ccpa-authorized-agent.txt",
"ccpa_indirect": "emails/ccpa-indirect-deletion.txt",
"gdpr": "emails/gdpr-erasure.txt",
}.get(kind, "emails/generic-optout.txt")
ctx = dict(fields)
ctx.setdefault("broker_name", broker.get("name", "the data broker"))
ctx["listing_urls"] = _join_listings(fields.get("listing_urls"))
ctx["my_identifiers"] = _join_identifiers(fields.get("my_identifiers"))
return render(template, ctx)

View file

@ -0,0 +1,79 @@
"""Filesystem paths for the unbroker skill (stdlib only).
All per-subject data lives under PDD_DATA_DIR (default: $HERMES_HOME/unbroker),
which is the same trust boundary Hermes uses for .env and OAuth tokens.
"""
from __future__ import annotations
import os
from pathlib import Path
def hermes_home() -> Path:
return Path(os.environ.get("HERMES_HOME") or (Path.home() / ".hermes"))
def data_dir() -> Path:
override = os.environ.get("PDD_DATA_DIR")
return Path(override) if override else hermes_home() / "unbroker"
def config_path() -> Path:
return data_dir() / "config.json"
def subjects_dir() -> Path:
return data_dir() / "subjects"
def subject_dir(subject_id: str) -> Path:
return subjects_dir() / subject_id
def dossier_path(subject_id: str) -> Path:
return subject_dir(subject_id) / "dossier.json"
def ledger_path(subject_id: str) -> Path:
return subject_dir(subject_id) / "ledger.json"
def audit_path(subject_id: str) -> Path:
return subject_dir(subject_id) / "audit.jsonl"
def evidence_dir(subject_id: str) -> Path:
return subject_dir(subject_id) / "evidence"
def skill_root() -> Path:
"""The skill directory (parent of scripts/)."""
return Path(__file__).resolve().parent.parent
def brokers_dir() -> Path:
return skill_root() / "references" / "brokers"
def brokers_cache_path() -> Path:
"""Live broker snapshot pulled from BADBOOL (merged under the curated DB)."""
return data_dir() / "brokers-cache" / "badbool.json"
def registry_cache_path() -> Path:
"""CA Data Broker Registry snapshot (separate coverage lane; DROP/email, not scanned)."""
return data_dir() / "brokers-cache" / "ca-registry.json"
def age_identity_path() -> Path:
"""age identity (private key) used for at-rest encryption when enabled.
Defaults beside the data; point PDD_AGE_IDENTITY at a separate volume/token
for real key separation from the encrypted data.
"""
override = os.environ.get("PDD_AGE_IDENTITY")
return Path(override) if override else data_dir() / "age-identity.txt"
def templates_dir() -> Path:
return skill_root() / "templates"

View file

@ -0,0 +1,810 @@
#!/usr/bin/env python3
"""unbroker - deterministic CLI helper.
The Hermes agent orchestrates scanning and opt-out submission with native tools
(`web_extract`, `browser_navigate`, email mechanisms). THIS CLI owns the
deterministic state: config, dossiers + consent, the broker DB, tier planning,
the ledger + audit log, draft/template rendering, and reports.
Run it through the `terminal` tool (it can read PII files under HERMES_HOME);
do NOT run it through `execute_code` (that sandbox scrubs env and redacts output).
Examples:
python pdd.py setup
python pdd.py intake --full-name "Jane Q. Public" --email jane@example.com \
--city Oakland --state CA --residency US-CA --consent --consent-method self
python pdd.py plan sub_xxxx --priority crucial
python pdd.py record sub_xxxx spokeo found --found true \
--evidence '{"listing_urls":["https://www.spokeo.com/..."]}'
python pdd.py render-email sub_xxxx spokeo --listing https://www.spokeo.com/...
python pdd.py status sub_xxxx
"""
from __future__ import annotations
import argparse
import json
import os
import sys
from pathlib import Path
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import autopilot # noqa: E402
import badbool # noqa: E402
import brokers as brokers_mod # noqa: E402
import config as config_mod # noqa: E402
import crypto # noqa: E402
import dossier as dossier_mod # noqa: E402
import email_modes # noqa: E402
import emailer # noqa: E402
import ledger as ledger_mod # noqa: E402
import legal # noqa: E402
import paths as paths_mod # noqa: E402
import registry # noqa: E402
import report as report_mod # noqa: E402
import tiers # noqa: E402
def _out(obj) -> None:
print(json.dumps(obj, indent=2, ensure_ascii=False))
def _require_subject(subject_id: str) -> dict:
d = dossier_mod.load(subject_id)
if not d:
sys.exit(f"error: unknown subject {subject_id!r} (run `intake` first)")
return d
def cmd_setup(args) -> None:
if getattr(args, "auto", False):
# Autonomous path: detect capabilities and pick the most autonomous valid
# config without asking anyone. Explicit flags still win below.
cfg = config_mod.auto_configure()
else:
cfg = config_mod.load_config()
for key in ("autonomy", "email_mode", "browser_backend", "tracker_backend", "encryption"):
val = getattr(args, key)
if val:
cfg[key] = val
if cfg.get("encryption") == "age":
if not crypto.age_available():
sys.exit("error: encryption=age requested but `age`/`age-keygen` not found. "
"Install age (e.g. `brew install age`) or use `--encryption none`.")
crypto.ensure_identity() # generate the key now so encryption is actually engaged
path = config_mod.save_config(cfg)
migrated = _migrate_subjects() # rewrite existing dossiers/ledgers into the new at-rest format
out = {
"config_path": str(path),
"config": cfg,
"encryption_engaged": crypto.is_engaged(),
"detected_upgrades": config_mod.detect_capabilities(),
"migrated_subjects": migrated,
"note": "Defaults are easiest-first (draft email, auto browser, local tracker, no encryption). "
"Pass flags to opt into upgrades, then run `doctor` for a readiness summary.",
}
if cfg.get("encryption") == "age":
out["age_identity"] = str(crypto.identity_path())
_out(out)
def _migrate_subjects() -> int:
"""Re-save each subject's dossier + ledger so they match the current at-rest format."""
sd = paths_mod.subjects_dir()
if not sd.exists():
return 0
n = 0
for child in sorted(sd.iterdir()):
if not child.is_dir():
continue
sid = child.name
d = dossier_mod.load(sid)
if d is not None:
dossier_mod.save(d)
n += 1
led = ledger_mod.load(sid)
if led:
ledger_mod.save(sid, led)
return n
def _check_writable(path) -> bool:
try:
path.mkdir(parents=True, exist_ok=True)
probe = path / ".write_test"
probe.write_text("x", encoding="utf-8")
probe.unlink()
return True
except OSError:
return False
def cmd_doctor(args) -> None:
import platform
cfg = config_mod.load_config()
caps = config_mod.detect_capabilities()
data = paths_mod.data_dir()
writable = _check_writable(data)
curated = len(brokers_mod._load_curated())
live = len(brokers_mod.load_live_cache())
total = len(brokers_mod.load_all())
L = ["unbroker - readiness check", "=" * 42,
f"Python : {platform.python_version()}",
f"Data dir : {data} ({'writable' if writable else 'NOT writable'})",
f"Config : autonomy={cfg.get('autonomy', 'full')} email={cfg['email_mode']} "
f"browser={cfg['browser_backend']} "
f"tracker={cfg['tracker_backend']} encryption={cfg['encryption']}",
f"Brokers : {total} available ({curated} curated + {live} live"
+ ("" if live else ", run `refresh-brokers` to expand to ~50") + ")",
"", "Opt-in upgrades:"]
rows = [
("Cloud browser (Browserbase) *RECOMMENDED*", caps["browserbase"],
"default backend: clears soft CAPTCHAs (Turnstile/hCaptcha) -> more T1", "set BROWSERBASE_API_KEY"),
("Email auto (AgentMail)", caps["agentmail"],
"send + auto-verify, per-broker aliases (Mode B/C)", "install agentmail skill / set AGENTMAIL_API_KEY"),
("Email send (CLI SMTP)", caps["smtp_send"],
"`send-email` delivers opt-outs itself (Mode B)", "set EMAIL_ADDRESS / EMAIL_PASSWORD (+ EMAIL_SMTP_HOST)"),
("Verify-link poll (CLI IMAP)", caps["imap_read"],
"`poll-verification` reads confirmation links itself", "set EMAIL_ADDRESS / EMAIL_PASSWORD (+ EMAIL_IMAP_HOST)"),
("Google Sheets tracker", caps["google_workspace"],
"shared status dashboard", "set up the google-workspace skill"),
]
for name, ok, enables, how in rows:
L.append(f" [{'ON ' if ok else 'off'}] {name:<28} {enables}")
if not ok:
L.append(f" enable: {how}")
# At-rest encryption: report TRUE engagement (configured + key present), not just binary presence.
engaged = crypto.is_engaged()
L.append(f" [{'ON ' if engaged else 'off'}] {'At-rest encryption (age)':<28} "
"encrypts dossiers + ledgers on disk")
if engaged:
L.append(f" key: {crypto.identity_path()} (0600) - guards casual/backup/commit "
"exposure, NOT a full-HERMES_HOME read")
elif cfg["encryption"] == "age":
L.append(" WARNING: encryption=age is SET but NOT engaged (age binary or key missing);"
" dossiers would be PLAINTEXT")
elif caps["age"]:
L.append(" off - dossiers are plaintext (0600). enable: `setup --encryption age`")
else:
L.append(" off - dossiers are plaintext (0600). install `age` first to enable")
L += ["", "Verdict:", " Ready now in DRAFT mode (no setup needed): scan brokers, draft opt-out",
" emails for you to send, and track everything in the ledger."]
if caps["browserbase"]:
L.append(" Cloud browser ON (recommended default): soft/managed CAPTCHAs "
"(Turnstile/hCaptcha) clear automatically -> those brokers stay T1.")
else:
L.append(" No cloud browser: set BROWSERBASE_API_KEY (the recommended default) so soft "
"CAPTCHAs clear automatically; without it those brokers drop to T2 (human tasks).")
if cfg["email_mode"] == "draft_only":
L.append(" Email is draft-only: you send drafts + click verify links. For hands-off email "
"WITHOUT storing a password, run `setup --email-mode browser` (agent sends + opens "
"verify links via your logged-in webmail); or set EMAIL_* for SMTP/IMAP.")
elif cfg["email_mode"] == "browser":
L.append(" Email mode: browser (no password) - the agent sends opt-outs and opens verify "
"links via the operator's logged-in webmail. Ensure that inbox is signed in in the "
"browser Hermes uses (a cloud browser won't hold the session); else it falls back to drafts.")
if not crypto.is_engaged():
L.append(" Storage: dossiers are PLAINTEXT JSON (0600 under HERMES_HOME). "
"Run `setup --encryption age` for at-rest encryption.")
if not live:
L.append(" Next: run `refresh-brokers` to load the full broker list.")
# Freshness: warn when cached lists / curated mechanics are going stale (silent broker rot).
import time as _time
STALE_CACHE_DAYS, STALE_VERIFY_DAYS = 30, 180
def _age_days(p) -> float | None:
try:
return (_time.time() - p.stat().st_mtime) / 86400.0
except OSError:
return None
fresh = []
for label, p in [("BADBOOL", paths_mod.brokers_cache_path()),
("CA registry", paths_mod.registry_cache_path())]:
age = _age_days(p)
if age is None:
fresh.append(f"{label}: not pulled")
elif age > STALE_CACHE_DAYS:
fresh.append(f"{label}: {age:.0f}d old (stale, re-pull)")
stale_curated = documented = 0
for b in brokers_mod._load_curated():
conf = b.get("confidence")
lv = b.get("last_verified")
if conf == "documented" or not lv:
documented += 1
continue
try:
if (_time.time() - _time.mktime(_time.strptime(lv, "%Y-%m-%d"))) / 86400.0 > STALE_VERIFY_DAYS:
stale_curated += 1
except (ValueError, TypeError):
pass
if fresh:
L.append(" Freshness: " + "; ".join(fresh) + " (run `refresh-brokers`).")
if stale_curated or documented:
L.append(f" Freshness: {stale_curated} curated broker(s) last-verified >{STALE_VERIFY_DAYS}d ago; "
f"{documented} documented broker(s) awaiting first-use verification.")
print("\n".join(L))
def cmd_intake(args) -> None:
if args.json:
data = json.loads(Path(args.json).read_text(encoding="utf-8"))
identity = data["identity"]
consent = data.get("consent", {})
residency = data.get("residency_jurisdiction", "US")
prefs = data.get("preferences")
else:
if not args.full_name:
sys.exit("error: --full-name (or --json) is required")
identity = {"full_name": args.full_name, "emails": args.email or [], "phones": args.phone or []}
if args.alias:
identity["also_known_as"] = args.alias
if args.dob:
identity["date_of_birth"] = args.dob
addr = {k: v for k, v in {"line1": args.street, "city": args.city,
"state": args.state, "postal": args.postal}.items() if v}
if addr:
identity["current_address"] = addr
priors = []
for loc in args.prior_location or []:
parts = [p.strip() for p in loc.split(",") if p.strip()]
if not parts:
continue
entry = {"city": parts[0]}
if len(parts) > 1:
entry["state"] = parts[1]
if len(parts) > 2:
entry["postal"] = parts[2]
priors.append(entry)
if priors:
identity["prior_addresses"] = priors
cfg = config_mod.load_config()
consent = {"authorized": bool(args.consent), "method": args.consent_method, "recorded_at": dossier_mod.now()}
residency = args.residency or "US"
prefs = {
"email_mode": args.email_mode or cfg["email_mode"],
"rescan_interval_days": cfg["default_rescan_interval_days"],
}
if args.contact_email:
prefs["contact_email_for_optouts"] = args.contact_email
d = dossier_mod.create(identity, consent, residency, prefs)
_out({"subject_id": d["subject_id"], "authorized": dossier_mod.is_authorized(d),
"residency": residency, "email_mode": (prefs or {}).get("email_mode"),
"names": dossier_mod.all_names(d),
"emails": len(d["identity"].get("emails") or []),
"phones": len(d["identity"].get("phones") or []),
"addresses": len(dossier_mod.all_addresses(d))})
def cmd_brokers(args) -> None:
bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all()
_out([
{"id": b.get("id"), "name": b.get("name"), "priority": b.get("priority"),
"method": (b.get("optout") or {}).get("method"), "owns": b.get("owns") or [],
"source": b.get("source"), "confidence": b.get("confidence", "curated")}
for b in bl
])
def cmd_refresh_brokers(args) -> None:
res = badbool.refresh(paths_mod.brokers_cache_path())
curated_ids = {b["id"] for b in brokers_mod._load_curated()}
new = [b["id"] for b in brokers_mod.load_live_cache() if b["id"] not in curated_ids]
out = {**res, "curated": len(curated_ids), "new_from_live": len(new),
"people_search_total": len(brokers_mod.load_all()),
"note": "Live records have confidence=auto; verify their opt-out URL before acting."}
if not getattr(args, "no_registry", False):
try:
reg = registry.refresh_all(paths_mod.registry_cache_path())
out["registry"] = {"total": reg["total"], "sources": reg["sources"],
"portals": reg["portals"],
"note": "Coverage lane worked via the CA DROP one-shot + CCPA email, "
"not the people-search scan. VT/OR/TX are search portals (no "
"bulk export); CA is the superset. See `drop` and `registry`."}
except Exception as exc: # noqa: BLE001 - registry pull is best-effort
out["registry_error"] = str(exc)
_out(out)
def cmd_registry(args) -> None:
recs = brokers_mod.load_registry_cache()
if not recs:
_out({"registered_brokers": 0,
"note": "registry empty - run `refresh-brokers` (pulls the CA Data Broker Registry)"})
return
fcra = sum(1 for r in recs if (r.get("optout") or {}).get("fcra"))
out = {"registered_brokers": len(recs), "fcra_regulated": fcra,
"source": "CA Data Broker Registry (CPPA, 2025)", "drop_url": registry.DROP_URL,
"other_state_portals": registry.portals()}
if args.search:
q = args.search.lower()
hits = [r for r in recs if q in (r.get("name") or "").lower()
or q in (r.get("id") or "") or q in ((r.get("optout") or {}).get("email") or "").lower()]
out["matches"] = [{"id": r["id"], "name": r["name"],
"email": (r.get("optout") or {}).get("email"),
"url": (r.get("optout") or {}).get("url"),
"fcra": (r.get("optout") or {}).get("fcra")} for r in hits[:args.limit]]
out["match_count"] = len(hits)
_out(out)
def cmd_drop(args) -> None:
"""The one-shot legal lever: CA DROP deletes from ALL registered brokers at once."""
d = _require_subject(args.subject)
dossier_mod.require_authorized(d)
reg = brokers_mod.load_registry_cache()
res = (d.get("residency_jurisdiction") or "US").upper()
eligible = res.startswith("US-CA")
if args.filed:
prefs = d.setdefault("preferences", {})
prefs["drop_filed_at"] = dossier_mod.now()
dossier_mod.save(d)
_out({"subject": args.subject, "drop_filed_at": prefs["drop_filed_at"],
"note": "recorded; `next` will stop surfacing the DROP one-shot"})
return
_out({
"subject": args.subject,
"eligible": eligible,
"residency": res,
"drop_url": registry.DROP_URL,
"covers_registered_brokers": len(reg),
"steps": ([
"Go to privacy.ca.gov/drop and create/verify a DROP account (CA resident).",
"Submit ONE deletion request; it applies to EVERY registered data broker "
f"({len(reg)} in the current registry). Brokers must process starting 2026-08-01.",
"After filing, run `drop <subject> --filed` so the loop stops re-surfacing it.",
] if eligible else [
"DROP is a California mechanism; this subject's residency is not US-CA.",
"Parity path for non-CA: work the people-search sites via `next`, and send targeted "
"CCPA/GDPR deletion emails to registry brokers that hold this person's data "
"(`registry --search`, then `send-email`).",
]),
"note": "DROP is the highest-leverage removal: one request covers the whole registry.",
})
def cmd_plan(args) -> None:
d = _require_subject(args.subject)
dossier_mod.require_authorized(d)
cfg = config_mod.load_config()
bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all()
bcc = config_mod.browser_clears_captcha(cfg)
if getattr(args, "batch", False):
_out(tiers.batch_plan(d, bl, cfg, ledger_mod.load(args.subject), bcc))
else:
_out(tiers.plan(d, bl, cfg, bcc))
def cmd_fanout(args) -> None:
d = _require_subject(args.subject)
dossier_mod.require_authorized(d)
bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all()
grouping = tiers.fanout(bl, batch_size=args.size)
mode = "scan AND opt-out (operator authorized submissions)" if args.optout \
else "READ-ONLY scan (submit nothing; reconnaissance only)"
batches = []
for i, ids in enumerate(grouping["batches"], 1):
brief = (
f"You are scan worker {i} of {len(grouping['batches'])} for the `unbroker` "
f"skill. First load the `unbroker` skill and read its references/methods.md. "
f"Subject id: {args.subject}. Handle ONLY these brokers: {', '.join(ids)}. "
f"For EACH broker: read references/brokers/<id>.json; run EVERY search vector from "
f"`pdd.py plan {args.subject}` (filtered to your brokers); build URLs from search.url_patterns "
f"and heed url_format_quirks; a 404 is INCONCLUSIVE (rebuild/try the on-site search box), not "
f"not_found; confirm the SUBJECT vs namesakes/relatives before recording; if search.antibot is "
f"set and no stealth/cloud browser is available, record `blocked`. Record each outcome via "
f"`pdd.py record {args.subject} <broker> <found|not_found|indirect_exposure|blocked> "
f"--found <bool> --evidence '{{\"listing_urls\":[...]}}'`. Mode: {mode}. "
f"Log any newly-discovered URL/format quirks into the broker JSON. "
f"Return a concise structured per-broker report."
)
batches.append({"batch": i, "brokers": ids, "brief": brief})
_out({
"subject": args.subject,
"broker_count": grouping["broker_count"],
"batch_size": grouping["batch_size"],
"should_fanout": grouping["should_fanout"],
"batch_count": len(batches),
"batches": batches,
"instruction": (
"If should_fanout is true you MUST spawn ONE delegate_task subagent per batch IN PARALLEL, "
"passing each batch's `brief`; do not scan all brokers yourself sequentially. Wait for every "
"report, consolidate, then proceed to opt-outs. If false, just scan the brokers inline."
),
})
def cmd_record(args) -> None:
d = _require_subject(args.subject)
dossier_mod.require_authorized(d)
broker = brokers_mod.get(args.broker)
# Auto-stamp follow-up scheduling (next_recheck_at / removal_confirmed_at) so the
# autonomous loop knows when to come back without anyone remembering to set it.
fields = ledger_mod.followup_fields(args.state, broker, d)
if args.found is not None:
fields["found"] = args.found
if args.evidence:
fields["evidence"] = json.loads(args.evidence)
if args.reason:
fields["human_task_reason"] = args.reason
case = ledger_mod.transition(args.subject, args.broker, args.state, **fields)
if args.disclosed:
ledger_mod.log_disclosure(args.subject, args.broker, args.disclosed, args.channel or "unknown")
_out({"broker": args.broker, "state": case["state"],
"next_recheck_at": case.get("next_recheck_at")})
def _email_request(d: dict, b: dict, kind: str, listings, identifiers) -> tuple[dict, list[str]]:
"""Least-disclosure (fields, disclosed_names) for an opt-out/legal email of KIND.
A removal letter must self-identify. Name + a contact email are already known to the
broker (the name is displayed on the very listing being removed), so not extra exposure.
"""
fields = dossier_mod.select_disclosure(d, (b.get("optout") or {}).get("inputs", []))
ident = d.get("identity", {})
if ident.get("full_name"):
fields.setdefault("full_name", ident["full_name"])
fields.setdefault("contact_email", dossier_mod.contact_email(d) or "")
if listings:
fields["listing_urls"] = listings
if kind == "ccpa_indirect":
# Indirect exposure: name ONLY the subject's own identifiers to scrub from a third party's
# record. Default to the contact email + the subject's name-as-relative if none specified.
# The indirect template renders ONLY these placeholders; do not over-report disclosure with
# unrelated dossier fields (phone/street/postal) that select_disclosure happened to populate.
ids = list(identifiers or [])
if not ids:
ids = [contact for contact in [dossier_mod.contact_email(d)] if contact]
ids.append(f'the name "{ident.get("full_name")}" where it appears as a relative/associated person')
fields = {
"full_name": fields.get("full_name"),
"contact_email": fields.get("contact_email"),
"listing_urls": fields.get("listing_urls"),
"my_identifiers": ids,
}
return fields, ["contact_email", "full_name", "my_identifiers"]
return fields, sorted(fields.keys())
def cmd_render_email(args) -> None:
d = _require_subject(args.subject)
dossier_mod.require_authorized(d)
b = brokers_mod.get(args.broker)
if not b:
sys.exit(f"error: unknown broker {args.broker!r}")
kind = getattr(args, "kind", "generic") or "generic"
fields, disclosed = _email_request(d, b, kind, args.listing, getattr(args, "identifier", None))
if kind == "generic":
draft = email_modes.render_draft(b, fields)
else:
draft = email_modes.render_request_draft(b, fields, kind=kind)
ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_draft:{kind}")
_out({"draft": str(draft), "kind": kind, "disclosed_fields": disclosed})
def cmd_send_email(args) -> None:
"""Mode B: render AND deliver the opt-out/legal request - no human in the loop.
Sends ONLY to an address the broker record itself declares (emailer enforces it),
then records the ledger transition + disclosure and auto-stamps the recheck date.
"""
d = _require_subject(args.subject)
dossier_mod.require_authorized(d)
b = brokers_mod.get(args.broker)
if not b:
sys.exit(f"error: unknown broker {args.broker!r}")
cfg = config_mod.load_config()
mode = cfg.get("email_mode")
if mode not in ("programmatic", "alias", "browser"):
sys.exit("error: email_mode is draft_only; run `setup --email-mode browser` (no password; "
"sends via your logged-in webmail) or `--email-mode programmatic`, or use "
"`render-email` and send it yourself")
if not args.listing:
sys.exit("error: --listing <confirmed-url> is required (verify-before-disclose: never "
"email a broker about an unconfirmed listing)")
# Idempotency: don't re-send if this case is already submitted/beyond (prevents duplicate
# requests when an action is retried). --force overrides.
_POST_SUBMIT = {"submitted", "verification_pending", "awaiting_processing", "confirmed_removed"}
current = ledger_mod.get_case(args.subject, args.broker).get("state")
if current in _POST_SUBMIT and not getattr(args, "force", False):
_out({"skipped": True, "broker": args.broker, "state": current,
"note": "already submitted; not re-sending (idempotent). Use --force to re-send."})
return
kind = getattr(args, "kind", "generic") or "generic"
fields, disclosed = _email_request(d, b, kind, args.listing, getattr(args, "identifier", None))
body = legal.render_optout_email(b, fields) if kind == "generic" else legal.render_request(kind, b, fields)
if mode == "browser":
# No network / no credentials: hand the agent a recipient-locked payload to send in the
# operator's webmail via browser_* tools. State still records deterministically here.
payload = emailer.browser_send_payload(b, body, to=args.to)
ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_browser:{kind}")
case = ledger_mod.transition(args.subject, args.broker, "submitted",
**ledger_mod.followup_fields("submitted", b, d))
_out({"send_via": "browser", "compose": payload, "kind": kind, "disclosed_fields": disclosed,
"state": case["state"], "next_recheck_at": case.get("next_recheck_at"),
"instruction": "In the operator's logged-in webmail, compose a NEW email to compose.to "
"with compose.subject/body EXACTLY (disclose nothing beyond it) and send "
"it via browser_* tools. Then use `verify-link` on any confirmation reply.",
"note": "recipient is locked to the broker's declared address"})
return
result = emailer.send(b, body, to=args.to,
min_interval=float(cfg.get("email_min_interval_seconds", 0) or 0))
ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_sent:{kind}")
case = ledger_mod.transition(args.subject, args.broker, "submitted",
**ledger_mod.followup_fields("submitted", b, d))
_out({"sent": result, "send_via": "smtp", "kind": kind, "disclosed_fields": disclosed,
"state": case["state"], "next_recheck_at": case.get("next_recheck_at"),
"note": "if this broker verifies by email, `poll-verification` will pick up the link"})
def cmd_verify_link(args) -> None:
"""Extract a broker's verification link from email text the agent read in webmail (browser mode).
IMAP-free counterpart to `poll-verification`: the agent opens the broker's confirmation email
in the operator's webmail, pastes the body here, and gets the anti-phishing-scored link back.
"""
_require_subject(args.subject)
b = brokers_mod.get(args.broker)
if not b:
sys.exit(f"error: unknown broker {args.broker!r}")
text = args.text
if args.file:
text = Path(args.file).read_text(encoding="utf-8", errors="replace")
if not text:
sys.exit("error: provide --text '<email body>' (or --file) from the broker's confirmation email")
link = email_modes.extract_verification_link(text, b)
_out({"broker": args.broker, "verification_link": link,
"next": ("browser_navigate the link IN THE SAME browser (sessions are browser-bound), "
f"complete the flow, then `record {args.subject} {args.broker} awaiting_processing`"
if link else
"no broker/opt-out-scoped link found in that text; confirm you opened the right email")})
def cmd_poll_verification(args) -> None:
"""Poll the inbox for brokers' verification links (Mode B) - replaces the human click-chase.
For each in-flight case (submitted / verification_pending with email_verification),
extract the broker's link (anti-phishing scored). A found link auto-advances
submitted -> verification_pending (the email HAS arrived); the agent must then OPEN
the link in its own browser (sessions are browser-bound) and record the next state.
"""
d = _require_subject(args.subject)
dossier_mod.require_authorized(d)
led = ledger_mod.load(args.subject)
targets = []
for bid, case in sorted(led.items()):
if args.broker and bid != args.broker:
continue
if case.get("state") not in ("submitted", "verification_pending"):
continue
b = brokers_mod.get(bid)
if b and (((b.get("optout") or {}).get("requires")) or {}).get("email_verification"):
targets.append((bid, case, b))
if not targets:
_out({"subject": args.subject, "results": [],
"note": "no in-flight cases awaiting email verification"})
return
results = []
for bid, case, b in targets:
hit = emailer.find_verification_link(b, since_days=args.since_days)
if hit:
if case.get("state") == "submitted":
ledger_mod.transition(args.subject, bid, "verification_pending",
**ledger_mod.followup_fields("verification_pending", b, d))
results.append({"broker": bid, "verification_link": hit["link"],
"email_from": hit.get("from"), "email_subject": hit.get("subject"),
"next": f"browser_navigate the link IN THE AGENT'S OWN BROWSER, complete "
f"the flow, then `record {args.subject} {bid} awaiting_processing` "
f"(or confirmed_removed only after a verifying re-scan)"})
else:
results.append({"broker": bid, "verification_link": None,
"next": "no matching email yet; poll again later (next_recheck_at is set)"})
_out({"subject": args.subject, "results": results})
def cmd_next(args) -> None:
d = _require_subject(args.subject)
dossier_mod.require_authorized(d)
cfg = config_mod.load_config()
bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all()
_out(autopilot.next_actions(d, bl, cfg, ledger_mod.load(args.subject)))
def cmd_tasks(args) -> None:
_require_subject(args.subject)
print(report_mod.human_tasks_markdown(args.subject))
def cmd_due(args) -> None:
_require_subject(args.subject)
cases = ledger_mod.due(args.subject)
_out({"subject": args.subject, "due_count": len(cases),
"cases": [{"broker_id": c.get("broker_id"), "state": c.get("state"),
"next_recheck_at": c.get("next_recheck_at")} for c in cases],
"note": "run `next` for the concrete follow-up action per case"})
def cmd_status(args) -> None:
_require_subject(args.subject)
print(report_mod.render_markdown(args.subject))
def cmd_report(args) -> None:
_require_subject(args.subject)
if args.sheets:
_out(report_mod.sheets_rows(args.subject))
else:
print(report_mod.render_markdown(args.subject))
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(prog="pdd", description="unbroker helper CLI")
sub = p.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("setup", help="write install config (easiest-first defaults; --auto = most autonomous)")
s.add_argument("--auto", action="store_true",
help="detect capabilities and pick the most autonomous valid config (no questions)")
s.add_argument("--autonomy", dest="autonomy", choices=sorted(config_mod.VALID["autonomy"]))
s.add_argument("--email-mode", dest="email_mode", choices=sorted(config_mod.VALID["email_mode"]))
s.add_argument("--browser-backend", dest="browser_backend", choices=sorted(config_mod.VALID["browser_backend"]))
s.add_argument("--tracker-backend", dest="tracker_backend", choices=sorted(config_mod.VALID["tracker_backend"]))
s.add_argument("--encryption", dest="encryption", choices=sorted(config_mod.VALID["encryption"]))
s.set_defaults(func=cmd_setup)
s = sub.add_parser("doctor", help="readiness check: config, brokers, available upgrades")
s.set_defaults(func=cmd_doctor)
s = sub.add_parser("intake", help="create a subject dossier (records consent)")
s.add_argument("--json", help="path to a dossier JSON file (overrides flags)")
s.add_argument("--full-name")
s.add_argument("--alias", action="append", metavar="NAME",
help="other name the subject is listed under (maiden/married/nickname); repeatable")
s.add_argument("--email", action="append", metavar="EMAIL", help="repeatable")
s.add_argument("--phone", action="append", metavar="PHONE", help="repeatable")
s.add_argument("--street", help="current street line1 (enables reverse-address search)")
s.add_argument("--city")
s.add_argument("--state")
s.add_argument("--postal")
s.add_argument("--prior-location", dest="prior_location", action="append", metavar="City,ST",
help="a past city/state (or City,ST,ZIP); repeatable")
s.add_argument("--dob", help="date of birth YYYY-MM-DD (only used if a broker requires it)")
s.add_argument("--contact-email", dest="contact_email",
help="which email to use for opt-out correspondence (default: first)")
s.add_argument("--residency", help="e.g. US, US-CA")
s.add_argument("--consent", action="store_true", help="subject authorizes removal on their behalf")
s.add_argument("--consent-method", default="self", choices=["self", "written_authorization", "poa"])
s.add_argument("--email-mode", dest="email_mode", choices=sorted(config_mod.VALID["email_mode"]))
s.set_defaults(func=cmd_intake)
s = sub.add_parser("brokers", help="list the broker database (curated + live)")
s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"])
s.set_defaults(func=cmd_brokers)
s = sub.add_parser("refresh-brokers",
help="pull the latest BADBOOL people-search list + the CA data broker registry")
s.add_argument("--no-registry", dest="no_registry", action="store_true",
help="skip the CA registry pull (BADBOOL people-search only)")
s.set_defaults(func=cmd_refresh_brokers)
s = sub.add_parser("registry",
help="CA Data Broker Registry coverage (hundreds of brokers; DROP/email lane)")
s.add_argument("--search", help="find registered brokers by name / id / email substring")
s.add_argument("--limit", type=int, default=25, help="max matches to print (default 25)")
s.set_defaults(func=cmd_registry)
s = sub.add_parser("drop",
help="CA DROP one-shot: delete from ALL registered brokers in one request")
s.add_argument("subject")
s.add_argument("--filed", action="store_true", help="mark DROP as filed (stops `next` surfacing it)")
s.set_defaults(func=cmd_drop)
s = sub.add_parser("plan", help="compute per-broker tier + next action for a subject")
s.add_argument("subject")
s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"])
s.add_argument("--batch", action="store_true",
help="phase-oriented batch view: overlays ledger state, groups by next action "
"(unscanned/found/indirect/blocked/in_progress/done), collapses ownership clusters")
s.set_defaults(func=cmd_plan)
s = sub.add_parser("fanout", help="batch brokers into parallel delegate_task subagents (large runs)")
s.add_argument("subject")
s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"])
s.add_argument("--size", type=int, default=8, help="brokers per subagent batch (default 8)")
s.add_argument("--optout", action="store_true",
help="brief authorizes opt-out submission (default: read-only scan)")
s.set_defaults(func=cmd_fanout)
s = sub.add_parser("record", help="record a ledger state transition after an agent action")
s.add_argument("subject")
s.add_argument("broker")
s.add_argument("state", choices=ledger_mod.STATES)
s.add_argument("--found", type=lambda v: v.strip().lower() in ("1", "true", "yes", "y"))
s.add_argument("--evidence", help="JSON object stored as case.evidence")
s.add_argument("--disclosed", action="append", metavar="FIELD", help="field name disclosed")
s.add_argument("--channel", help="disclosure channel, e.g. web_form / email")
s.add_argument("--reason", help="for human_task_queued: why a human is needed (shown in `tasks`)")
s.set_defaults(func=cmd_record)
s = sub.add_parser("next", help="autonomous action queue: exactly what to do right now")
s.add_argument("subject")
s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"])
s.set_defaults(func=cmd_next)
s = sub.add_parser("send-email", help="Mode B: render AND send the opt-out/legal request (records it)")
s.add_argument("subject")
s.add_argument("broker")
s.add_argument("--listing", action="append", metavar="URL", required=False,
help="confirmed listing URL (required: verify-before-disclose)")
s.add_argument("--kind", choices=["generic", "ccpa", "ccpa_agent", "ccpa_indirect", "gdpr"],
default="generic")
s.add_argument("--identifier", action="append", metavar="ID",
help="(ccpa_indirect only) a specific own-identifier to remove; repeatable")
s.add_argument("--to", help="override recipient (must be an address the broker record declares)")
s.add_argument("--force", action="store_true", help="re-send even if already submitted (default: idempotent skip)")
s.set_defaults(func=cmd_send_email)
s = sub.add_parser("poll-verification",
help="Mode B (IMAP): poll the inbox for brokers' verification links (anti-phishing scored)")
s.add_argument("subject")
s.add_argument("--broker", help="only this broker (default: every in-flight verification case)")
s.add_argument("--since-days", dest="since_days", type=int, default=3)
s.set_defaults(func=cmd_poll_verification)
s = sub.add_parser("verify-link",
help="browser mode: extract a broker's verification link from pasted webmail text")
s.add_argument("subject")
s.add_argument("broker")
s.add_argument("--text", help="the confirmation email body (read from the operator's webmail)")
s.add_argument("--file", help="path to a file with the email body (alternative to --text)")
s.set_defaults(func=cmd_verify_link)
s = sub.add_parser("tasks", help="ONE consolidated human-task digest (present at end of run)")
s.add_argument("subject")
s.set_defaults(func=cmd_tasks)
s = sub.add_parser("due", help="cases whose recheck window has arrived (cron re-scan queue)")
s.add_argument("subject")
s.set_defaults(func=cmd_due)
s = sub.add_parser("render-email", help="render a Mode-A opt-out / legal-request draft (least-disclosure)")
s.add_argument("subject")
s.add_argument("broker")
s.add_argument("--listing", action="append", metavar="URL", help="confirmed listing URL")
s.add_argument("--kind", choices=["generic", "ccpa", "ccpa_agent", "ccpa_indirect", "gdpr"],
default="generic",
help="request type. 'ccpa_indirect' = delete MY identifiers from a third party's "
"record (indirect exposure); default 'generic' opt-out.")
s.add_argument("--identifier", action="append", metavar="ID",
help="(ccpa_indirect only) a specific own-identifier to request removal of "
"(e.g. an email or phone). Repeatable. Defaults to the contact email + "
"name-as-relative if omitted.")
s.set_defaults(func=cmd_render_email)
s = sub.add_parser("status", help="print a Markdown status report")
s.add_argument("subject")
s.set_defaults(func=cmd_status)
s = sub.add_parser("report", help="status report (default) or --sheets rows")
s.add_argument("subject")
s.add_argument("--sheets", action="store_true", help="emit Google Sheets rows as JSON")
s.set_defaults(func=cmd_report)
return p
def main(argv=None) -> None:
args = build_parser().parse_args(argv)
try:
args.func(args)
except (PermissionError, ValueError, RuntimeError, FileNotFoundError) as exc:
sys.exit(f"error: {exc}")
if __name__ == "__main__":
main()

View file

@ -0,0 +1,293 @@
"""Ingest the California Data Broker Registry into broker records (coverage breadth).
The CA registry (CPPA, under the Delete Act) is the authoritative universe of data
brokers doing business with California residents -- ~545 businesses in 2025, each
required to publish a name, website, contact email, and a CCPA-rights/deletion URL.
This is the same universe commercial services (DeleteMe/Incogni/Optery) draw from,
plus the FCRA/GLBA-regulated and marketing/risk brokers most lists omit.
These are NOT people-search sites you scan with a name -- most have no per-person
lookup UI. They are worked through the LEGAL lane: the CA DROP portal
(privacy.ca.gov/drop) is a single request that deletes from ALL registered brokers
at once (CA residents), and per-broker CCPA deletion emails to the contact address
are the fallback / non-CA path. So registry records are kept in their own lane
(loaded only when asked) and never dumped into the people-search scan pipeline.
`parse()` is pure (CSV text in, records out) so it is tested offline; `fetch()` is
the only network call and can be bypassed by passing csv_text directly to refresh().
"""
from __future__ import annotations
import csv
import datetime
import io
import re
import urllib.request
from pathlib import Path
import storage
# CA CPPA registry CSVs are published per year (registry2024.csv, registry2025.csv, ...).
# 2025 is the latest COMPLETE dataset; the current year's file is empty until the Jan
# registration window closes. DEFAULT_URL is the known-good fallback; `ca_candidate_urls`
# probes newer years first so coverage auto-advances when the next year is published.
_CA_CSV = "https://cppa.ca.gov/data_broker_registry/registry{year}.csv"
_CA_FLOOR_YEAR = 2025
DEFAULT_URL = _CA_CSV.format(year=_CA_FLOOR_YEAR)
DROP_URL = "https://privacy.ca.gov/drop"
USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)"
def ca_candidate_urls(today: datetime.date | None = None) -> list[str]:
"""Newest-year-first CA registry URLs to try (auto-advances; never below the 2025 floor)."""
year = (today or datetime.date.today()).year
years = list(range(max(year, _CA_FLOOR_YEAR), _CA_FLOOR_YEAR - 1, -1))
return [_CA_CSV.format(year=y) for y in years]
# Multi-source registry lane. Only California publishes a clean bulk CSV (with contact email +
# CCPA-rights URL per broker) AND offers a one-shot deletion portal (DROP). Vermont, Oregon, and
# Texas maintain registries too, but only as searchable PORTALS (no reliable bulk export) and with
# no DROP-equivalent -- and they overlap CA heavily (CA is effectively the superset). So they are
# wired as first-class portal sources (official URL surfaced to the operator) rather than scraped.
# Adding any state that later publishes a CSV is a one-line "format: csv" entry (the parser is
# column-detection based, not CA-specific).
SOURCES = {
"ca": {"jurisdiction": "US-CA", "format": "csv", "url": DEFAULT_URL, "has_drop": True,
"name": "California Data Broker Registry (CPPA)"},
"vt": {"jurisdiction": "US-VT", "format": "portal", "has_drop": False,
"url": "https://bizfilings.vermont.gov/online/DatabrokerInquire/",
"name": "Vermont Data Broker Registry (Secretary of State)"},
"or": {"jurisdiction": "US-OR", "format": "portal", "has_drop": False,
"url": "https://dfr.oregon.gov/business/licensing/data-broker-registry/Pages/index.aspx",
"name": "Oregon Data Broker Registry (DCBS)"},
"tx": {"jurisdiction": "US-TX", "format": "portal", "has_drop": False,
"url": "https://texas-sos.appianportalsgov.com/data-broker-registry",
"name": "Texas Data Broker Registry (Secretary of State)"},
}
def portals() -> list[dict]:
"""Registry sources that are searchable portals (no bulk export) -- surfaced to the operator."""
return [{"key": k, "jurisdiction": s["jurisdiction"], "name": s["name"], "url": s["url"]}
for k, s in SOURCES.items() if s["format"] == "portal"]
# Field label -> substring to locate its column on the header row (robust to
# year-to-year column shifts; the registry re-orders/adds columns between years).
_LABELS = {
"name": "data broker name:",
"dba": "doing business as",
"website": "data broker primary website:",
"email": "primary contact email",
"rights_url": "exercise their ca consumer privacy act rights",
"fcra": "regulated by the federal fair credit reporting act (fcra):",
}
def _norm(s: str) -> str:
"""Registry CSVs use NBSPs and a BOM; normalize for matching + clean values."""
return re.sub(r"\s+", " ", (s or "").replace("\ufeff", "").replace("\xa0", " ")).strip()
def slug(name: str, website: str = "") -> str:
base = re.sub(r"\.(com|org|net|io|ai|inc|co|us|info|llc)\b", "", (name or "").strip(), flags=re.I)
s = re.sub(r"[^a-z0-9]+", "", base.lower())
if s:
return s
dom = re.sub(r"^https?://(www\.)?", "", (website or "").lower())
return re.sub(r"[^a-z0-9]+", "", dom.split("/")[0]) or "broker"
def _domain(website: str) -> str:
dom = re.sub(r"^https?://(www\.)?", "", (website or "").strip().lower())
return dom.split("/")[0]
def _find_colmap(rows: list[list[str]]) -> tuple[int, dict[str, int]]:
"""Locate the label row (col0 == 'Data broker name:') and map fields to columns."""
for i, row in enumerate(rows[:5]):
if row and _norm(row[0]).lower().startswith("data broker name:"):
colmap: dict[str, int] = {}
for field, needle in _LABELS.items():
for j, cell in enumerate(row):
c = _norm(cell).lower()
if needle in c and not c.startswith("if the data broker"):
colmap[field] = j
break
return i, colmap
raise ValueError("CA registry: could not locate the header row")
def _get(row: list[str], idx: int | None) -> str:
return _norm(row[idx]) if idx is not None and idx < len(row) else ""
def _build(row: list[str], cm: dict[str, int], jurisdiction: str = "US-CA",
has_drop: bool = True) -> dict | None:
name = _get(row, cm.get("name"))
website = _get(row, cm.get("website"))
if not (name or website):
return None
email = _get(row, cm.get("email"))
rights = _get(row, cm.get("rights_url"))
dba = _get(row, cm.get("dba"))
fcra = _get(row, cm.get("fcra")).lower().startswith("y")
state = jurisdiction.split("-")[-1]
method = "email" if email else ("web_form" if rights else "drop")
if has_drop:
notes = ("Registered CA data broker. One CA DROP request (privacy.ca.gov/drop) deletes from "
"this and every registered broker at once; or send a CCPA deletion request to the "
"contact email.")
else:
notes = (f"Registered {state} data broker (no one-shot delete portal in {state}). Send a "
"CCPA/state-law deletion request to the contact email.")
if fcra:
notes += (" FCRA-regulated: some data is credit-reporting data with separate rules -- deletion "
"may be limited; a consumer report dispute/security-freeze may apply instead.")
return {
"id": slug(name, website),
"name": name or _domain(website),
"dba": dba or None,
"category": "data_broker",
"priority": "long_tail",
"jurisdictions": [jurisdiction],
"search": {"method": "none", "url": website, "fetch": "none", "by": ["registry"]},
"optout": {
"method": method,
"url": rights or website or None,
"email": email or None,
"requires": {"profile_url": False, "email_verification": False, "captcha": False,
"gov_id": False, "account": False, "phone_callback": False, "payment": False},
"inputs": ["full_name", "contact_email"],
"deletion": {
"via": "drop" if has_drop else "email",
"email": email or None,
"url": rights or None,
"kinds": ["ccpa", "generic"],
"notes": ("Covered by the CA DROP one-shot (privacy.ca.gov/drop); CCPA email fallback."
if has_drop else "CCPA/state-law deletion email (no one-shot portal)."),
},
"fcra": fcra,
"est_processing_days": 45,
"notes": notes,
},
"source": f"{state}-registry",
"confidence": "registry",
"last_verified": None,
}
def parse(csv_text: str, jurisdiction: str = "US-CA", has_drop: bool = True) -> list[dict]:
"""Parse a data-broker-registry CSV into broker records (deduped by id).
Column detection is by header label, not fixed position, so any state that publishes a
registry CSV with name/website/email/rights columns parses without new code.
"""
rows = list(csv.reader(io.StringIO(csv_text)))
if not rows:
return []
header_i, cm = _find_colmap(rows)
out: list[dict] = []
seen: dict[str, int] = {}
for row in rows[header_i + 1:]:
if not any(c.strip() for c in row):
continue
rec = _build(row, cm, jurisdiction, has_drop)
if not rec:
continue
bid = rec["id"]
if bid in seen: # disambiguate id collisions by domain, then a counter
dom = re.sub(r"[^a-z0-9]+", "", _domain(rec["search"]["url"]))
cand = f"{bid}-{dom}" if dom and dom != bid else bid
while cand in seen:
seen[bid] += 1
cand = f"{bid}-{seen[bid]}"
rec["id"] = cand
seen.setdefault(rec["id"], 0)
seen.setdefault(bid, 0)
out.append(rec)
return out
MIN_EXPECTED_CA = 100 # CA registry has ~500+; far fewer => wrong/empty file, warn
def fetch(url: str = DEFAULT_URL, timeout: int = 60) -> str:
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310
return resp.read().decode("utf-8", errors="replace")
def _fetch_ca_latest() -> tuple[str, list[dict]]:
"""Try newest CA registry year first; return (url, records) for the first non-empty."""
last: tuple[str, list[dict]] = (DEFAULT_URL, [])
for url in ca_candidate_urls():
try:
recs = parse(fetch(url), jurisdiction="US-CA", has_drop=True)
except Exception: # noqa: BLE001 - a missing year 404s; fall through to older years
continue
if recs:
return url, recs
last = (url, recs)
return last
def refresh(cache_path: Path, url: str = DEFAULT_URL, csv_text: str | None = None) -> dict:
"""CA single-source refresh: fetch (or accept) the CA CSV and write the cache."""
text = csv_text if csv_text is not None else fetch(url)
records = parse(text)
storage.write_json(cache_path, records)
fcra = sum(1 for r in records if (r.get("optout") or {}).get("fcra"))
return {"parsed": len(records), "fcra_regulated": fcra,
"cache_path": str(cache_path), "source_url": url}
def refresh_all(cache_path: Path, fetched: dict[str, str] | None = None) -> dict:
"""Multi-source refresh: pull every CSV source, dedupe across states by domain, cache.
`fetched` optionally supplies {source_key: csv_text} to bypass the network (tests). CSV
sources are ingested as broker records; portal sources contribute their URL for the operator
(no bulk export exists) but no records. CA is processed first so it wins domain collisions.
"""
all_recs: list[dict] = []
seen_domains: set[str] = set()
per_source: dict[str, dict] = {}
for key, src in SOURCES.items():
if src["format"] != "csv":
per_source[key] = {"jurisdiction": src["jurisdiction"], "format": "portal",
"url": src["url"], "records": 0,
"note": "searchable portal (no bulk export); operator/agent searches by name"}
continue
used_url = src["url"]
try:
if fetched is not None:
text = fetched.get(key)
if text is None:
raise RuntimeError("no CSV text supplied")
recs = parse(text, jurisdiction=src["jurisdiction"], has_drop=src["has_drop"])
elif key == "ca":
used_url, recs = _fetch_ca_latest() # newest-year-first with fallback
else:
recs = parse(fetch(src["url"]), jurisdiction=src["jurisdiction"], has_drop=src["has_drop"])
except Exception as exc: # noqa: BLE001 - one source failing must not sink the rest
per_source[key] = {"jurisdiction": src["jurisdiction"], "format": "csv", "error": str(exc)}
continue
added = 0
for r in recs:
dom = _domain(r["search"]["url"])
if dom and dom in seen_domains:
continue
if dom:
seen_domains.add(dom)
all_recs.append(r)
added += 1
entry = {"jurisdiction": src["jurisdiction"], "format": "csv", "url": used_url,
"parsed": len(recs), "added_after_dedupe": added,
"fcra": sum(1 for r in recs if (r.get("optout") or {}).get("fcra"))}
if key == "ca" and len(recs) < MIN_EXPECTED_CA:
entry["warning"] = (f"only {len(recs)} parsed (expected >{MIN_EXPECTED_CA}); the CA "
"registry file may be empty/moved - verify the source URL")
per_source[key] = entry
storage.write_json(cache_path, all_recs)
return {"total": len(all_recs), "sources": per_source, "portals": portals(),
"cache_path": str(cache_path)}

View file

@ -0,0 +1,161 @@
"""Status dashboards, Markdown reports, human-task digest, and Google Sheets row export."""
from __future__ import annotations
import brokers as brokers_mod
import ledger as ledger_mod
STATE_LABELS = {
"new": "Not started",
"searching": "Searching",
"not_found": "Not found",
"found": "Found (action needed)",
"indirect_exposure": "Indirect exposure (PII on a relative's record)",
"action_selected": "Action selected",
"submitted": "Submitted",
"verification_pending": "Awaiting verification",
"awaiting_processing": "Processing",
"confirmed_removed": "Removed",
"reappeared": "Reappeared",
"human_task_queued": "Human task",
"blocked": "Blocked",
}
def status_counts(subject_id: str) -> dict:
counts: dict[str, int] = {}
for case in ledger_mod.load(subject_id).values():
state = case.get("state", "new")
counts[state] = counts.get(state, 0) + 1
return counts
def metrics(subject_id: str) -> dict:
"""Outcome metrics: what's actually confirmed vs merely claimed, and what's overdue.
removal_rate is confirmed_removed over cases we actually acted on (found/submitted/... ),
NOT over the whole broker DB, so it reflects real progress on real exposure. `in_flight`
is 'claimed' (submitted/verifying/processing) but not yet re-scan-confirmed. `overdue`
counts cases whose recheck window has already passed (the cron backlog).
"""
c = status_counts(subject_id)
removed = c.get("confirmed_removed", 0)
in_flight = c.get("submitted", 0) + c.get("verification_pending", 0) + c.get("awaiting_processing", 0)
open_found = c.get("found", 0) + c.get("reappeared", 0) + c.get("action_selected", 0) \
+ c.get("indirect_exposure", 0)
acted = removed + in_flight + open_found + c.get("human_task_queued", 0) + c.get("blocked", 0)
return {
"confirmed_removed": removed,
"in_flight_claimed": in_flight, # submitted but NOT yet verified gone
"open_needs_action": open_found,
"blocked": c.get("blocked", 0),
"human_tasks": c.get("human_task_queued", 0),
"acted_total": acted,
"removal_rate": round(removed / acted, 3) if acted else 0.0,
"overdue_rechecks": len(ledger_mod.due(subject_id)),
}
def render_markdown(subject_id: str) -> str:
ledger = ledger_mod.load(subject_id)
counts = status_counts(subject_id)
total = sum(counts.values())
removed = counts.get("confirmed_removed", 0)
m = metrics(subject_id)
lines = [
f"# unbroker - status for `{subject_id}`",
"",
f"**{removed} / {total} confirmed removed** · removal rate (of acted-on cases): "
f"{int(m['removal_rate'] * 100)}%",
"",
f"- Confirmed removed: {m['confirmed_removed']}",
f"- In flight (submitted, not yet re-scan-confirmed): {m['in_flight_claimed']}",
f"- Open / needs action: {m['open_needs_action']}",
f"- Blocked (anti-bot): {m['blocked']} · Human tasks: {m['human_tasks']}",
f"- Overdue rechecks (cron backlog): {m['overdue_rechecks']}",
"",
"| State | Count |",
"|---|---|",
]
for state in ledger_mod.STATES:
if counts.get(state):
lines.append(f"| {STATE_LABELS.get(state, state)} | {counts[state]} |")
tasks = [c for c in ledger.values() if c.get("state") == "human_task_queued"]
if tasks:
lines += ["", "## Outstanding human tasks"]
for c in tasks:
reason = c.get("human_task_reason", "manual step required")
lines.append(f"- **{c.get('broker_id')}** - {reason}")
indirect = [c for c in ledger.values() if c.get("state") == "indirect_exposure"]
if indirect:
lines += ["", "## Indirect exposure (your PII on third-party records)",
"Not removable via the broker's self-service opt-out (the record is about someone "
"else). Lever: a targeted CCPA/GDPR delete-my-PII request naming only your own "
"identifiers."]
for c in indirect:
ev = c.get("evidence") or {}
note = ev.get("summary") or "subject's identifiers appear on another person's listing"
lines.append(f"- **{c.get('broker_id')}** - {note}")
return "\n".join(lines) + "\n"
def human_tasks_markdown(subject_id: str) -> str:
"""ONE consolidated digest of everything that genuinely needs a human.
The autonomous run accumulates human-only work silently (never interrupting);
this digest is presented once, at the end, so the operator clears it in a
single sitting. Includes queued tasks and blocked-site operator-browser checks.
"""
ledger = ledger_mod.load(subject_id)
tasks = [(bid, c) for bid, c in sorted(ledger.items()) if c.get("state") == "human_task_queued"]
blocked = [(bid, c) for bid, c in sorted(ledger.items()) if c.get("state") == "blocked"]
lines = [f"# Human tasks for `{subject_id}`", ""]
if not tasks and not blocked:
lines.append("Nothing needs a human right now.")
return "\n".join(lines) + "\n"
lines.append(f"{len(tasks)} manual step(s) + {len(blocked)} blocked site(s). "
"Everything else ran (or will run) autonomously.")
if tasks:
lines += ["", "## Manual steps"]
for bid, c in tasks:
b = brokers_mod.get(bid) or {}
opt = b.get("optout") or {}
lines.append(f"### {b.get('name', bid)}")
lines.append(f"- Why: {c.get('human_task_reason', 'manual step required')}")
where = opt.get("url") or opt.get("email") or "(see broker record)"
lines.append(f"- Where: {where}")
for q in (opt.get("quirks") or [])[:2]:
lines.append(f"- Note: {q}")
lines.append("- Withhold: SSN and full ID numbers - always.")
lines.append(f"- When done, tell the agent so it records the outcome for `{bid}`.")
if blocked:
lines += ["", "## Blocked sites (open in YOUR browser - it gets through where bots don't)"]
for bid, c in blocked:
b = brokers_mod.get(bid) or {}
url = ((b.get("search") or {}).get("url")) or "(see broker record)"
lines.append(f"- **{b.get('name', bid)}** - open {url}, search the subject, and report "
"the verdict (or a screenshot) back to the agent.")
return "\n".join(lines) + "\n"
def sheets_rows(subject_id: str) -> list[list[str]]:
"""Header + one row per case for the optional Google Sheets tracker.
The agent appends these via the `google-workspace` skill, e.g.:
google_api.py sheets append <SHEET_ID> "Sheet1!A:F" --values <json-rows>
"""
rows = [["broker_id", "state", "found", "tier", "removed_at", "next_recheck"]]
for bid, c in sorted(ledger_mod.load(subject_id).items()):
rows.append([
bid,
c.get("state", ""),
str(c.get("found", "")),
(c.get("automation") or {}).get("tier_used", ""),
c.get("removal_confirmed_at") or "",
c.get("next_recheck_at") or "",
])
return rows

View file

@ -0,0 +1,32 @@
"""Stdlib fetch helper for simple url_pattern brokers (osint-style).
For JS-rendered or anti-bot pages the agent should use the `web_extract` or
`browser_navigate` tools (and the `scrapling` skill for stealth/Cloudflare).
This helper only covers plain static pages and is intentionally network-light so
it can be mocked in tests.
"""
from __future__ import annotations
import urllib.error
import urllib.request
USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)"
def fetch(url: str, timeout: int = 20) -> tuple[int, str]:
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
try:
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 (https only by convention)
charset = resp.headers.get_content_charset() or "utf-8"
return getattr(resp, "status", 200), resp.read().decode(charset, errors="replace")
except urllib.error.HTTPError as exc:
return exc.code, ""
except (urllib.error.URLError, TimeoutError, ValueError):
return 0, ""
def looks_listed(html: str, match_signal: str | None) -> bool:
"""Naive confirmation heuristic for static pages: does the match signal appear?"""
if not html or not match_signal:
return False
return match_signal.lower() in html.lower()

View file

@ -0,0 +1,138 @@
"""Storage helpers (stdlib only): atomic JSON, append-only JSONL, strict perms.
Default backend is local-json. The optional google-sheets tracker is handled in
report.py by emitting rows for the `google-workspace` skill; this module stays
dependency-free so the hermetic tests never touch the network.
"""
from __future__ import annotations
import contextlib
import json
import os
import time
from pathlib import Path
from typing import Any
import crypto
import paths
@contextlib.contextmanager
def locked(target: Path, timeout: float = 10.0, stale: float = 30.0):
"""Portable advisory lock via an O_EXCL lockfile next to `target`.
Serializes read-modify-write on shared JSON (the ledger) across concurrent
processes - a cron re-scan overlapping a manual run, or multiple tenants -
so one writer can't clobber another's update. A lock older than `stale`
seconds is treated as abandoned (crashed writer) and broken, so a dead
process can never deadlock the queue. Works on macOS/Linux/Windows (O_EXCL).
"""
ensure_dir(target.parent)
lock = target.with_name(target.name + ".lock")
deadline = time.monotonic() + timeout
while True:
try:
fd = os.open(str(lock), os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
try:
os.write(fd, str(os.getpid()).encode())
finally:
os.close(fd)
break
except FileExistsError:
try:
if time.time() - lock.stat().st_mtime > stale:
lock.unlink(missing_ok=True)
continue
except OSError:
pass
if time.monotonic() >= deadline:
raise TimeoutError(f"could not acquire lock {lock} within {timeout}s")
time.sleep(0.05)
try:
yield
finally:
with contextlib.suppress(OSError):
lock.unlink(missing_ok=True)
def _secure(path: Path, mode: int) -> None:
try:
os.chmod(path, mode)
except OSError:
pass # non-POSIX / unsupported FS; HERMES_HOME directory perms still apply
def ensure_dir(path: Path) -> Path:
path.mkdir(parents=True, exist_ok=True)
_secure(path, 0o700)
return path
def _is_sensitive(path: Path) -> bool:
"""Per-subject docs (dossier, ledger) are sensitive; config/cache are not."""
try:
Path(path).resolve().relative_to(paths.subjects_dir().resolve())
return True
except (ValueError, OSError):
return False
def _age_path(path: Path) -> Path:
return path.with_name(path.name + ".age")
def _atomic_write(path: Path, data: bytes) -> Path:
tmp = path.with_name(path.name + ".tmp")
tmp.write_bytes(data)
_secure(tmp, 0o600)
os.replace(tmp, path)
_secure(path, 0o600)
return path
def write_json(path: Path, obj: Any) -> Path:
ensure_dir(path.parent)
data = (json.dumps(obj, indent=2, ensure_ascii=False) + "\n").encode("utf-8")
if _is_sensitive(path) and crypto.encryption_setting() == "age":
if not crypto.age_available():
raise RuntimeError(
"encryption=age is configured but `age` is not available; "
"refusing to write PII as plaintext. Install age or run `setup --encryption none`."
)
target = _atomic_write(_age_path(path), crypto.encrypt(data))
if path.exists():
path.unlink() # migrate plaintext -> ciphertext
return target
target = _atomic_write(path, data)
ap = _age_path(path)
if ap.exists():
ap.unlink() # encryption turned off -> drop stale ciphertext
return target
def read_json(path: Path, default: Any = None) -> Any:
ap = _age_path(path)
if ap.exists():
return json.loads(crypto.decrypt(ap.read_bytes()).decode("utf-8"))
if path.exists():
return json.loads(path.read_text(encoding="utf-8"))
return default
def append_jsonl(path: Path, record: dict) -> Path:
ensure_dir(path.parent)
with path.open("a", encoding="utf-8") as fh:
fh.write(json.dumps(record, ensure_ascii=False) + "\n")
_secure(path, 0o600)
return path
def read_jsonl(path: Path) -> list[dict]:
if not path.exists():
return []
out: list[dict] = []
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if line:
out.append(json.loads(line))
return out

View file

@ -0,0 +1,269 @@
"""Automation-tier selection and per-subject action planning.
Tiers:
T0 fully automated, no verification loop
T1 automated submit + automated verification (email mode B/C, or backend-cleared captcha)
T2 automated submit, verification needs a human (hard captcha / phone callback / account)
T3 human-required end-to-end (gov ID, fax, mail, voice-only phone)
"""
from __future__ import annotations
import dossier as dossier_mod
import vectors as vectors_mod
HARD_HUMAN = ("gov_id", "fax", "mail", "phone_voice")
def select_tier(broker: dict, email_mode: str = "draft_only",
browser_clears_captcha: bool = False) -> str:
req = ((broker.get("optout") or {}).get("requires")) or {}
if any(req.get(k) for k in HARD_HUMAN):
return "T3"
if req.get("account"):
return "T2"
captcha = bool(req.get("captcha"))
if (captcha and not browser_clears_captcha) or req.get("phone_callback"):
return "T2"
if req.get("email_verification"):
return "T1" if email_mode in ("programmatic", "alias") else "T2"
if captcha and browser_clears_captcha:
return "T1"
return "T0"
def plan(subject_dossier: dict, brokers_list: list[dict], cfg: dict,
browser_clears_captcha: bool = False) -> list[dict]:
email_mode = (subject_dossier.get("preferences") or {}).get("email_mode") \
or cfg.get("email_mode", "draft_only")
actions: list[dict] = []
for b in brokers_list:
opt = b.get("optout") or {}
search = b.get("search") or {}
tier = select_tier(b, email_mode, browser_clears_captcha)
disclosure = dossier_mod.select_disclosure(subject_dossier, opt.get("inputs", []))
svectors = vectors_mod.search_vectors(subject_dossier, b)
actions.append({
"broker_id": b.get("id"),
"broker_name": b.get("name"),
"priority": b.get("priority"),
"method": opt.get("method"),
"tier": tier,
"human_required": tier == "T3",
"search_url": search.get("url"),
"fetch": search.get("fetch", "web_extract"),
"antibot": search.get("antibot"),
"search_by": vectors_mod.supported_by(b),
"search_vectors": svectors,
"optout_url": opt.get("url"),
"optout_email": opt.get("email"),
"disclosure_fields": sorted(disclosure.keys()),
"owns": b.get("owns") or [],
"notes": opt.get("notes", ""),
"optout_quirks": opt.get("quirks") or [],
"optout_requires": opt.get("requires") or {},
# The DELETION lane (right-to-delete), distinct from listing suppression. Structured so
# the autopilot can route to it: {via: email|in_flow|web_form, email?, url?, kinds?, notes?}
"deletion": opt.get("deletion") or {},
# Exact ordered opt-out steps maintained IN the broker record (field-verified knowledge
# lives with the data, not in code).
"optout_playbook": opt.get("playbook") or [],
})
return actions
def fanout(brokers_list: list[dict], batch_size: int = 8) -> dict:
"""Group brokers into batches for parallel `delegate_task` scan subagents.
Scanning many brokers serially is slow and burns context; above `batch_size`
the agent is expected to spawn one subagent per batch (see SKILL.md).
"""
ids = [b.get("id") for b in brokers_list if b.get("id")]
batches = [ids[i:i + batch_size] for i in range(0, len(ids), batch_size)]
return {
"broker_count": len(ids),
"batch_size": batch_size,
"should_fanout": len(ids) > batch_size,
"batches": batches,
}
# States that mean "the crawl reached a verdict for this broker".
_SCANNED_STATES = {"found", "not_found", "indirect_exposure", "blocked", "submitted",
"verification_pending", "awaiting_processing", "confirmed_removed", "reappeared",
"action_selected", "human_task_queued"}
# States that still need a deletion action taken.
_ACTIONABLE_STATES = {"found", "indirect_exposure", "reappeared", "action_selected"}
def batch_plan(subject_dossier: dict, brokers_list: list[dict], cfg: dict,
ledger: dict | None = None, browser_clears_captcha: bool = False) -> dict:
"""Reduce the per-broker plan into a phase-oriented batch view.
Overlays the current ledger state on each broker, groups by what the operator
should DO next, and collapses ownership clusters so a parent removal that clears
children is ONE action, not N. Read-only: computes, never mutates the ledger.
"""
ledger = ledger or {}
actions = plan(subject_dossier, brokers_list, cfg, browser_clears_captcha)
# child id -> parent id (only for parents present in this plan set)
child_to_parent: dict[str, str] = {}
for a in actions:
for child in a.get("owns") or []:
child_to_parent[child] = a["broker_id"]
def state_of(bid: str) -> str:
return (ledger.get(bid) or {}).get("state", "new")
groups: dict[str, list[dict]] = {
"unscanned": [], # no verdict yet -> Phase 1 crawl
"found": [], # direct removable listing -> Phase 2 opt-out (incl. reappeared/action_selected)
"indirect_exposure": [],# PII on a third party's record -> CCPA/GDPR delete email
"blocked": [], # anti-bot / needs stealth browser -> requeue
"in_progress": [], # submitted / verification_pending / awaiting_processing
"human": [], # human_task_queued -> the end-of-run digest, NOT re-scanning
"done": [], # confirmed_removed
"not_found": [],
}
covered_by_parent: dict[str, list[str]] = {}
for a in actions:
bid = a["broker_id"]
st = state_of(bid)
# cluster collapse: if a parent in this set is already actioned, the child is covered
parent = child_to_parent.get(bid)
if parent and state_of(parent) in ("found", "reappeared", "action_selected", "submitted",
"verification_pending", "awaiting_processing",
"confirmed_removed", "human_task_queued"):
covered_by_parent.setdefault(parent, []).append(bid)
continue
row = {"broker_id": bid, "broker_name": a["broker_name"], "priority": a["priority"],
"tier": a["tier"], "method": a["method"], "state": st,
"optout_url": a["optout_url"], "optout_email": a.get("optout_email"),
"clears_children": a.get("owns") or [],
"optout_requires": a.get("optout_requires") or {},
"optout_quirks": a.get("optout_quirks") or [],
"deletion": a.get("deletion") or {},
"optout_playbook": a.get("optout_playbook") or [],
"notes": a.get("notes", "")}
if st in ("submitted", "verification_pending", "awaiting_processing"):
groups["in_progress"].append(row)
elif st == "confirmed_removed":
groups["done"].append(row)
elif st in ("reappeared", "action_selected"):
groups["found"].append(row) # still needs the opt-out action
elif st == "human_task_queued":
groups["human"].append(row) # parked for the digest; never re-queued as work
elif st in groups:
groups[st].append(row)
elif st not in _SCANNED_STATES:
groups["unscanned"].append(row)
else:
groups.setdefault(st, []).append(row)
# PARENTS FIRST: within the actionable 'found' group, order cluster parents (a removal
# that clears children) ahead of standalone listings, most-children first. Working a
# parent before its children is what makes the cluster dedup real -- do them in this order.
groups["found"].sort(key=lambda r: (-len(r.get("clears_children") or []),
{"T0": 0, "T1": 1, "T2": 2, "T3": 3}.get(r.get("tier") or "", 9),
r["broker_id"]))
return {
"subject": subject_dossier.get("subject_id"),
"phase": "discover" if groups["unscanned"] else "delete",
"counts": {k: len(v) for k, v in groups.items()},
"groups": groups,
"cluster_savings": {p: kids for p, kids in covered_by_parent.items()},
"parent_playbook": _parent_playbook(groups["found"]),
"next_actions": _batch_next(groups, covered_by_parent),
}
def synthesize_steps(r: dict) -> list[str]:
"""Generic ordered opt-out steps derived from an optout record's structured fields.
Used for any broker without a hand-verified `optout.playbook`. Bespoke, field-verified
step lists live IN the broker JSON (`optout.playbook`) - single source of truth that
accrues knowledge as live runs discover mechanics (see methods.md logging rule).
"""
steps = [f"Opt out at {r.get('optout_url') or r.get('optout_email') or '(see broker record)'}"
+ (f" -- clears {', '.join(r['clears_children'])}." if r.get("clears_children") else ".")]
req = r.get("optout_requires") or {}
if req.get("profile_url"):
steps.append("Needs the confirmed profile_url (paste the listing URL you recorded).")
if req.get("email_verification"):
steps.append("Email verification: the same browser/inbox must open the confirmation link.")
if req.get("phone_callback"):
steps.append("Phone-callback code required; queue a human task if no operator is available.")
if req.get("gov_id"):
steps.append("Government ID demanded (T3): human task; never send SSN or a full ID number.")
d = r.get("deletion") or {}
if d.get("email"):
steps.append(f"DELETION lane: a right-to-delete request can be emailed to {d['email']}"
+ (f" ({d['notes']})" if d.get("notes") else "")
+ " -- prefer deletion over suppression.")
if r.get("notes"):
steps.append(str(r["notes"]))
for q in (r.get("optout_quirks") or [])[:3]:
steps.append(str(q))
return steps
def _parent_playbook(found_rows: list[dict]) -> list[dict]:
"""Tailored, ordered opt-out instructions for each cluster PARENT in the found group.
Steps come from the broker record's own `optout.playbook` (field-verified, maintained with
the data) with a synthesised fallback so the guidance is never empty. Standalone listings
are intentionally omitted -- the playbook exists to make the parents-first order concrete.
"""
playbook: list[dict] = []
for i, r in enumerate([x for x in found_rows if x.get("clears_children")], start=1):
steps = list(r.get("optout_playbook") or []) or synthesize_steps(r)
playbook.append({
"order": i,
"broker_id": r["broker_id"],
"broker_name": r["broker_name"],
"tier": r["tier"],
"clears_children": r["clears_children"],
"optout_url": r.get("optout_url"),
"optout_email": r.get("optout_email"),
"deletion": r.get("deletion") or {},
"steps": steps,
})
return playbook
def _batch_next(groups: dict, covered: dict) -> list[str]:
tips: list[str] = []
if groups["unscanned"]:
tips.append(f"PHASE 1 (crawl): {len(groups['unscanned'])} broker(s) unscanned -- run `fanout` and "
"scan read-only before any deletion.")
if groups["found"]:
parents = [r for r in groups["found"] if r.get("clears_children")]
if parents:
order = " -> ".join(r["broker_id"] for r in parents)
tips.append(f"PHASE 2 (opt-out): {len(groups['found'])} direct listing(s). DO CLUSTER PARENTS "
f"FIRST, in this order: {order} (see `parent_playbook` for tailored per-parent "
"steps), then the standalone listings.")
else:
tips.append(f"PHASE 2 (opt-out): {len(groups['found'])} direct listing(s) to remove.")
if groups["indirect_exposure"]:
tips.append(f"{len(groups['indirect_exposure'])} indirect-exposure case(s): send a targeted "
"CCPA/GDPR delete-my-PII email (render-email --kind ccpa_indirect), do NOT use the opt-out form.")
if groups["blocked"]:
tips.append(f"{len(groups['blocked'])} blocked (anti-bot): requeue for a stealth/cloud browser "
"pass; don't burn subagent time fighting CAPTCHAs.")
if covered:
n = sum(len(v) for v in covered.values())
tips.append(f"Cluster dedup: {n} child site(s) covered by parent removals -- skip separate opt-outs.")
if groups["in_progress"]:
tips.append(f"{len(groups['in_progress'])} in progress: resolve verification links, then confirm removal.")
if groups.get("human"):
tips.append(f"{len(groups['human'])} parked human task(s): present via `tasks` at end of run "
"(do not re-scan or re-queue them).")
return tips

View file

@ -0,0 +1,53 @@
"""Enumerate the search queries to run per broker, across ALL of a subject's identifiers.
People-search sites index a person under every name, phone, email, and address they
have. A subject with two names (maiden/married) and three past cities can have many
distinct listings on one broker, each found via a different search. `search_vectors`
expands the dossier into the concrete searches to run, filtered by what each broker
supports (`broker.search.by`, default ["name"]).
"""
from __future__ import annotations
import dossier as dossier_mod
# What a broker can be searched by; default if a record doesn't declare it.
DEFAULT_BY = ["name"]
def supported_by(broker: dict) -> list[str]:
return list((broker.get("search") or {}).get("by") or DEFAULT_BY)
def search_vectors(subject_dossier: dict, broker: dict) -> list[dict]:
"""List of {by, query} searches to run for this subject on this broker."""
by = set(supported_by(broker))
ident = subject_dossier.get("identity", {})
vectors: list[dict] = []
if "name" in by:
names = dossier_mod.all_names(subject_dossier)
locations = dossier_mod.all_locations(subject_dossier)
if locations:
for name in names:
for loc in locations:
vectors.append({"by": "name",
"query": {"full_name": name, "city": loc.get("city"), "state": loc.get("state")}})
else:
for name in names:
vectors.append({"by": "name", "query": {"full_name": name}})
if "phone" in by:
for phone in ident.get("phones") or []:
vectors.append({"by": "phone", "query": {"phone": phone}})
if "email" in by:
for email in ident.get("emails") or []:
vectors.append({"by": "email", "query": {"email": email}})
if "address" in by:
for a in dossier_mod.all_addresses(subject_dossier):
if a.get("line1"):
vectors.append({"by": "address",
"query": {k: a.get(k) for k in ("line1", "city", "state", "postal")}})
return vectors

View file

@ -0,0 +1,15 @@
# Authorization to act on my behalf (data removal)
I, **{full_name}**, authorize the operator of this tool to act as my agent for the limited purpose of
removing my personal information from data brokers and people-search websites, including submitting
opt-out, deletion, and do-not-sell/share requests under applicable privacy laws (e.g. CCPA/CPRA,
GDPR) on my behalf.
This authorization is limited to data removal. It does not authorize any other use of my information.
- Full name: {full_name}
- Date: {date}
- Signature: ______________________________
Store the signed copy at the path recorded in the dossier `consent.authorization_artifact`. Required
only when `consent.method` is `written_authorization` or `poa` (not for `self`).

View file

@ -0,0 +1,24 @@
Subject: CCPA/CPRA request submitted by authorized agent (delete and opt out)
To the {broker_name} privacy team,
I am an authorized agent acting on behalf of the consumer named below, under Cal. Civ. Code
1798.135 and the CCPA/CPRA. Written authorization is on file and available on request.
On the consumer's behalf I request that you:
1. DELETE all personal information you hold about them, and
2. OPT them OUT of the sale and sharing of their personal information.
Their information appears at:
{listing_urls}
Consumer:
Name: {full_name}
Email for confirmation: {contact_email}
Please confirm completion in writing to the email above. Do not request more sensitive information
than necessary to verify and process this request.
Sincerely,
Authorized agent for {full_name}

View file

@ -0,0 +1,22 @@
Subject: CCPA/CPRA request to delete and opt out (do not sell or share)
To the {broker_name} privacy team,
Under the California Consumer Privacy Act as amended by the CPRA (Cal. Civ. Code 1798.105 and
1798.120), I request that you:
1. DELETE all personal information you hold about me, and
2. OPT me OUT of the sale and sharing of my personal information.
My information appears at:
{listing_urls}
Identifying details for this request:
Name: {full_name}
Email: {contact_email}
Please do not request more sensitive information than necessary to process this request. Confirm
completion in writing to the email above within the statutory timeframe.
Sincerely,
{full_name}

View file

@ -0,0 +1,30 @@
Subject: Request to delete my personal information from third-party listings (CCPA/CPRA where applicable)
To the {broker_name} privacy team,
I am not the primary subject of the listings below, but each one currently exposes MY personal
information as a secondary data point (my email address and/or my name shown as a relative or
associated person). I am writing only about my own personal information, not about the individuals
who are the primary subjects of these records, and I am not requesting any change to their data.
Please delete and suppress the following personal information about me wherever it appears in your
database and on Spokeo-operated sites, including Thatsthem.com:
{my_identifiers}
These items currently appear at:
{listing_urls}
To the extent the California Consumer Privacy Act as amended by the CPRA (Cal. Civ. Code 1798.105)
applies to my personal information, please treat this as a request to delete it and to opt me out of
its sale or sharing (1798.120). Where CCPA does not apply by residency, I ask that you honor this as a
standard removal of my personal information consistent with the policy you apply to such requests.
Please do not request more information than is necessary to locate and remove these items, and please
do not add any new identifiers to my data in the course of processing this request. Confirm completion
in writing to the email below.
Name: {full_name}
Contact email: {contact_email}
Sincerely,
{full_name}

View file

@ -0,0 +1,19 @@
Subject: Request for erasure under GDPR Article 17
To the {broker_name} data protection officer,
Under Article 17 of the EU General Data Protection Regulation (and/or the UK GDPR), I request the
erasure of all personal data you hold about me, and under Article 21 I object to its processing.
My personal data appears at:
{listing_urls}
Identifying details:
Name: {full_name}
Email: {contact_email}
Please confirm erasure in writing to the email above within one month, as required by Article 12(3).
Do not request more personal data than necessary to action this request.
Sincerely,
{full_name}

View file

@ -0,0 +1,15 @@
Subject: Opt-out and data removal request
To the {broker_name} privacy team,
I am writing to request the removal of my personal information from {broker_name} and any sites you
operate or supply. My information currently appears at:
{listing_urls}
Please suppress and delete the record(s) associated with my name, {full_name}, and do not sell or
share my personal information.
Please confirm completion to this email address: {contact_email}
Thank you,
{full_name}

File diff suppressed because it is too large Load diff