mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-09 13:21:42 +00:00
feat(skills): add security/unbroker (autonomous data-broker removal)
unbroker finds where a consenting person's info is exposed across data brokers and people-search sites and files the removals, running as far as each site allows and handing only genuinely human-only steps (hard CAPTCHA, gov-ID, phone, fax) back as an end-of-run digest. - Deterministic stdlib CLI (scripts/pdd.py) owns config, dossiers+consent, the broker DB, tier planning, the ledger, email, and the autonomous action queue; the agent scans/submits with native tools (web_extract, browser_*, delegate_task, cronjob, terminal). - Verify-before-disclose, least-disclosure (never volunteers SSN), consent gate, opaque ids, optional age-at-rest encryption, file-locked ledger. - Jurisdiction-aware (CCPA/CPRA, GDPR, generic); CA DROP one-shot covers the state registry (~545) in a single request; BADBOOL + curated people-search coverage; scheduled re-scan for re-listing. - No CAPTCHA-solving services or anti-bot bypass; browser email mode needs no stored password. - 85 hermetic tests (tests/skills/test_unbroker_skill.py; SMTP/IMAP via injected fakes, registry via CSV fixtures). Ships placeholder data only. Broker dataset adapted from BADBOOL (Yael Grauer, CC BY-NC-SA 4.0).
This commit is contained in:
parent
89acc19606
commit
c2828f2b9b
52 changed files with 6703 additions and 0 deletions
163
optional-skills/security/unbroker/README.md
Normal file
163
optional-skills/security/unbroker/README.md
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
# unbroker
|
||||
|
||||
An agent-native skill that finds a consenting person's exposed personal information across data
|
||||
brokers and people-search sites and removes it. It runs automatically wherever it can, and hands off
|
||||
to a human only where a site demands a CAPTCHA it cannot clear, a government ID, a phone call, or a
|
||||
fax.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/unbroker.png"
|
||||
alt="unbroker: autonomous removal pipeline (exposure field, the loop, ledger, re-scan horizon)"
|
||||
width="720">
|
||||
</p>
|
||||
|
||||
## About
|
||||
|
||||
Hundreds of data brokers publish people's names, current and prior addresses, phone numbers, emails,
|
||||
relatives, and property records. That exposure fuels doxxing, stalking, harassment, and identity
|
||||
theft. Removing the data is the documented antidote, but it is high-volume work, full of dark
|
||||
patterns, and perishable (brokers re-list you). Commercial services such as EasyOptOuts, Incogni, and
|
||||
DeleteMe solve this for a fee, but they are closed, and you hand a company you know nothing about the
|
||||
exact data you are trying to erase.
|
||||
|
||||
unbroker brings those core capabilities together (EasyOptOuts' automation breadth, Incogni's
|
||||
legal-request engine, DeleteMe's verification and reporting) as a transparent, auditable,
|
||||
self-hosted skill that the user's own agent runs. It is **multi-tenant** (manage yourself, family, or
|
||||
clients, each isolated), **consent-gated**, and built for **maximum automation with a human
|
||||
fallback**. Scope is **US-first**, with EU/UK (GDPR) and global coverage on the roadmap.
|
||||
|
||||
The design is **Hermes-native**: a small deterministic Python CLI (`scripts/pdd.py`) owns the state
|
||||
(config, dossiers, broker DB, tier planning, ledger, drafts, reports), while the agent does the
|
||||
scanning and submitting with native tools (`web_extract`, `browser_*`, email, `cronjob`,
|
||||
`delegate_task`). [`SKILL.md`](SKILL.md) is the authoritative reference.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
hermes skills install official/security/unbroker
|
||||
```
|
||||
|
||||
Then start a new Hermes session and drive it (below). The skill works zero-config; a few optional
|
||||
env vars unlock more automation (all documented in `SKILL.md` under Prerequisites):
|
||||
|
||||
- `BROWSERBASE_API_KEY`: the recommended default browser. A real residential-IP cloud browser that
|
||||
clears soft/managed CAPTCHAs (Turnstile, hCaptcha/reCAPTCHA checkbox) as normal operation, so
|
||||
those brokers stay automated. It is not a solver and does not defeat hard challenges.
|
||||
- Hands-off email, two ways: **browser mode** (`pdd.py setup --email-mode browser`, no stored
|
||||
password; the agent sends opt-outs and opens verification links through your logged-in webmail),
|
||||
or **`EMAIL_ADDRESS` + `EMAIL_PASSWORD`** for SMTP send + IMAP verification. Without either, it
|
||||
falls back to writing drafts for you to send.
|
||||
- the `age` binary: at-rest encryption of dossiers and ledgers.
|
||||
- the `google-workspace` skill: a shared Google Sheets status tracker.
|
||||
|
||||
## Usage
|
||||
|
||||
Drive it from a Hermes session:
|
||||
|
||||
> "Use the unbroker skill to remove my data from data brokers. Here is my consent. Run it hands-off
|
||||
> and show me the human-task digest at the end."
|
||||
|
||||
The agent configures itself (`setup --auto` selects programmatic email if `EMAIL_*` creds exist, the
|
||||
cloud browser if available, and encryption if `age` is installed), records your consent, then drains
|
||||
the autonomous queue: scan, opt out (parents first), send and verify emails, schedule re-checks. You
|
||||
hear from it twice: at intake, and with one digest of anything only a human can do.
|
||||
|
||||
The underlying CLI (run via `terminal`, as `python3 scripts/pdd.py <cmd>`):
|
||||
|
||||
| Command | Purpose |
|
||||
|---|---|
|
||||
| `pdd.py setup --auto` / `doctor` | Self-configure (most-autonomous valid config) and readiness check |
|
||||
| `pdd.py intake` | Create a consenting subject (captures aliases, multiple emails/phones, prior addresses) |
|
||||
| `pdd.py next` | The loop driver: ordered agent actions right now, the human digest, and the next wake time |
|
||||
| `pdd.py brokers` / `refresh-brokers` | List people-search brokers, or pull the latest BADBOOL list plus the CA registry |
|
||||
| `pdd.py registry` | State data-broker registry coverage (CA ~545 ingested; VT/OR/TX portals); `--search` to find one |
|
||||
| `pdd.py drop` | The CA DROP one-shot: delete from all registered brokers in a single request |
|
||||
| `pdd.py plan` | Per-broker tier, method, search vectors, and the exact fields to disclose |
|
||||
| `pdd.py fanout` | Batch brokers into parallel `delegate_task` subagents |
|
||||
| `pdd.py record` | Update the ledger (validated state machine); auto-stamps recheck dates |
|
||||
| `pdd.py send-email` | Render and send an opt-out / CCPA / GDPR request (recipient locked to the broker's own address) |
|
||||
| `pdd.py poll-verification` / `verify-link` | Resolve email-verification links (IMAP poll, or browser-mode from pasted text) |
|
||||
| `pdd.py render-email` | Draft-only fallback (least-disclosure) |
|
||||
| `pdd.py due` / `tasks` | Recheck queue for cron, and the consolidated human-task digest |
|
||||
| `pdd.py status` / `report` | Per-subject status, plus optional Google Sheets rows |
|
||||
|
||||
## How it works
|
||||
|
||||
- **Autonomous by default.** After one human conversation (intake plus consent), the agent drains a
|
||||
deterministic action queue (`pdd.py next`): scan, opt out parents-first, send and verify emails,
|
||||
re-check on schedule, all without pausing to ask. Human-only work (gov-ID sites, phone callbacks,
|
||||
hard-CAPTCHA sites) accumulates silently into a single end-of-run digest (`pdd.py tasks`).
|
||||
- **Tiered automation (T0 to T3).** Every broker opt-out is classified from fully automated, to
|
||||
automated with verification, to human-verified, to human-only. The agent always takes the highest
|
||||
viable tier and escalates to a human task only when genuinely blocked.
|
||||
- **Cluster parents first.** Many brokers are resold shells of a few parents, so one removal can
|
||||
clear a dozen child sites. The planner orders parents ahead of standalone listings and ships
|
||||
field-verified, per-parent playbooks that prefer the **right-to-delete** lane over mere suppression
|
||||
(for example PeopleConnect's "delete my user data", or Whitepages' privacy email, which sidesteps
|
||||
the phone-callback tool entirely).
|
||||
- **Multi-identifier fan-out.** A person is indexed under every name/alias, phone, email, and
|
||||
address. The planner expands all of them (filtered by what each broker supports) so listings under
|
||||
a maiden name or an old address are found, not just "primary name plus current city".
|
||||
- **Verify before you disclose.** Nothing is submitted until a real listing is confirmed, the match
|
||||
is confirmed as the subject and not a namesake or relative, and only the exact fields a broker
|
||||
requires are sent (least-disclosure; SSN and ID numbers are never volunteered).
|
||||
- **Jurisdiction-aware.** Requests file under the framework that applies where the subject lives:
|
||||
CCPA/CPRA in California, GDPR in the EU/UK, a general right-to-delete request otherwise. It never
|
||||
cites a right the subject cannot invoke.
|
||||
- **Coverage that matches or exceeds commercial services.** Two lanes: (1) people-search sites with
|
||||
per-site opt-out mechanics (19 curated records, including FamilyTreeNow, Radaris, and Nuwber, plus
|
||||
a live pull from [BADBOOL](https://github.com/yaelwrites/Big-Ass-Data-Broker-Opt-Out-List)), and
|
||||
(2) the **state data-broker registries** as a distinct legal-coverage lane: the **California Data
|
||||
Broker Registry** (~545 registered brokers, the authoritative universe the commercial services draw
|
||||
from) is ingested, with Vermont, Oregon, and Texas surfaced as search portals.
|
||||
- **The DROP one-shot.** California's Delete Request and Opt-out Platform is live: for a CA resident,
|
||||
a single verified request deletes their data from **every registered broker at once**, and
|
||||
`pdd.py next` surfaces it as the highest-leverage action.
|
||||
- **Ledger, audit, and re-scan.** Every case is a validated state machine, every PII disclosure is
|
||||
logged (field names only), and confirmed removals are re-scanned on a schedule so a re-listing is
|
||||
caught and re-filed. Ledger writes are file-locked for safe concurrent runs.
|
||||
- **Privacy by default.** Opaque subject ids (no name in ids, paths, or logs), optional `age` at-rest
|
||||
encryption of dossiers, and everything local. The skill ships placeholder data only.
|
||||
|
||||
## Tests
|
||||
|
||||
85 hermetic tests (no network, browser, or email; SMTP and IMAP are exercised through injected
|
||||
fakes):
|
||||
|
||||
```bash
|
||||
scripts/run_tests.sh tests/skills/test_unbroker_skill.py # CI-parity harness
|
||||
python3 tests/skills/test_unbroker_skill.py # dependency-free fallback runner
|
||||
```
|
||||
|
||||
## Safety and ethics
|
||||
|
||||
- **Consent-gated.** The engine refuses to scan or act on a subject without a recorded
|
||||
authorization. It is a removal tool, not a people-search aggregator.
|
||||
- **Sanctioned browser only, no solver farms.** The default cloud browser clears soft/managed
|
||||
CAPTCHAs the way any real browser would, but there is no CAPTCHA-solving service and no fingerprint
|
||||
spoofing. Hard interactive challenges escalate to a human task.
|
||||
- **Least-disclosure and honest reporting.** The skill submits only what a broker requires. "Hidden
|
||||
from free search" is never reported as "deleted", and residual exposure (public records, paid-tier
|
||||
retention) is disclosed.
|
||||
- **PII handling.** Dossiers live under the Hermes home directory (`0600`, optionally
|
||||
`age`-encrypted), with opaque ids.
|
||||
|
||||
## Status
|
||||
|
||||
**v1.0.** The deterministic engine, the autonomous loop, the verified cluster-parent deletion lanes,
|
||||
and full broker-registry coverage (the CA Data Broker Registry plus the DROP one-shot) are built and
|
||||
covered by 85 hermetic tests. The skill ships placeholder data only. Live agent-driven submission
|
||||
against broker sites is the active field-testing frontier.
|
||||
|
||||
## Credits and license
|
||||
|
||||
- Broker dataset adapted from the **Big-Ass Data Broker Opt-Out List (BADBOOL)** by **Yael Grauer**,
|
||||
licensed [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (attribution
|
||||
required, non-commercial). See [yaelwrites.com](https://yaelwrites.com/).
|
||||
- Code: MIT.
|
||||
|
||||
## Disclaimer
|
||||
|
||||
This is not legal advice. Only operate on people who have authorized removal of their own data.
|
||||
Removing data from brokers reduces exposure but does not guarantee total erasure. Public records
|
||||
(voter, property, court) and offline vectors are out of scope.
|
||||
285
optional-skills/security/unbroker/SKILL.md
Normal file
285
optional-skills/security/unbroker/SKILL.md
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
---
|
||||
name: unbroker
|
||||
description: Autonomously remove your info from data-broker sites.
|
||||
version: 1.0.0
|
||||
author: SHL0MS (github.com/SHL0MS)
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
prerequisites:
|
||||
commands: [python3]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [privacy, data-broker, opt-out, ccpa, gdpr, security, doxxing]
|
||||
category: security
|
||||
related_skills: [google-workspace, agentmail, himalaya, scrapling, osint-investigation]
|
||||
homepage: https://github.com/NousResearch/hermes-agent
|
||||
---
|
||||
|
||||
# unbroker
|
||||
|
||||
Find where a person's personal information (name, addresses, phone, email, relatives) is exposed on
|
||||
data brokers and people-search sites, then remove it - automatically where possible, with guided
|
||||
human steps only where a site demands a CAPTCHA, government ID, phone call, or fax. Manages multiple
|
||||
people independently. It does **not** defeat anti-bot systems, does **not** act on anyone without
|
||||
recorded consent, and does **not** remove public records (voter/property/court) or accounts the
|
||||
person controls.
|
||||
|
||||
The Python CLI (`scripts/pdd.py`) owns the deterministic state - config, dossiers + consent, the
|
||||
broker database, tier planning, the ledger, drafts, reports, **email sending (SMTP), verification-link
|
||||
polling (IMAP), and the autonomous action queue (`next`)**. You (the agent) do the scanning and
|
||||
form-driving with native tools: `web_extract` and `browser_navigate` for searching and web forms, and
|
||||
`cronjob` for recurring re-scans.
|
||||
|
||||
## Autonomy contract
|
||||
|
||||
This skill is designed to run **hands-off**. After intake (+ recorded consent) there are exactly TWO
|
||||
legitimate human touchpoints: (1) the intake conversation itself, and (2) ONE consolidated human-task
|
||||
digest at the end of the run (`$PDD tasks`). Between those:
|
||||
|
||||
- **Never ask the operator to choose configuration.** `$PDD setup --auto` detects capabilities and
|
||||
picks the most autonomous valid config itself.
|
||||
- **Never pause before individual submissions** when `autonomy=full` (the default): the consent
|
||||
recorded at intake is standing authorization for T0-T2 opt-outs. (`autonomy=assisted` restores
|
||||
per-submission confirmation for cautious operators - honor `confirm_first` flags in `next` output.)
|
||||
- **Never interrupt the run for human-only work.** Record it (`record ... human_task_queued
|
||||
--reason "..."`) and keep going; it all surfaces once in the final digest.
|
||||
- **Drive the whole run as a loop over `$PDD next <subject>`** - it returns the exact ordered actions
|
||||
to take right now (scan, poll verification, re-check, opt out parents-first, requeue blocked), plus
|
||||
the human digest. Execute every action, record outcomes, re-run `next`, repeat until
|
||||
`done_for_now`. Then present the digest, report, and schedule the cron.
|
||||
|
||||
The hard limits that autonomy never overrides: no acting without recorded consent, no disclosure
|
||||
beyond `disclosure_fields`, no CAPTCHA/anti-bot bypass, and `confirmed_removed` only after a
|
||||
verifying re-scan.
|
||||
|
||||
## When to Use
|
||||
|
||||
- "Remove my (or my family member's) data from data brokers / people-search sites."
|
||||
- "Opt me out", "delete me from Spokeo/Whitepages/etc.", "clean up after a doxxing."
|
||||
- "Set up recurring privacy monitoring" (brokers re-list people).
|
||||
- Checking which brokers still expose someone and why.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `python3` (stdlib only; no extra packages needed for the core engine).
|
||||
- **Optional upgrades** (the skill works zero-config without these; `setup --auto` turns on every
|
||||
one it detects - each one converts a class of human tasks into agent actions):
|
||||
- **Cloud browser (recommended default): `BROWSERBASE_API_KEY`.** `setup --auto` selects it
|
||||
whenever the key is present, and it is the intended baseline: a real residential-IP cloud
|
||||
browser **clears soft/managed CAPTCHAs (Cloudflare Turnstile, hCaptcha/reCAPTCHA checkbox) as
|
||||
normal operation**, so those brokers stay automated (T1) instead of becoming human tasks. This
|
||||
is not CAPTCHA "solving" - no solver service, no fingerprint spoofing; only interactive/behavioral
|
||||
("hard") challenges the browser genuinely cannot pass fall back to a human task. Without the key,
|
||||
the plain agent browser is used and soft-CAPTCHA brokers drop to T2 (human).
|
||||
- Email automation, two credential-free-or-not options:
|
||||
- **Browser mode (no password): `setup --email-mode browser`.** The agent sends opt-out/CCPA
|
||||
emails and opens verification links through the operator's **logged-in webmail** using
|
||||
`browser_*` tools. Nothing is stored. Needs the inbox signed in in the browser Hermes uses
|
||||
(a cloud browser like Browserbase won't hold the session; use a local/operator browser).
|
||||
Falls back to drafts for an email if the inbox isn't reachable.
|
||||
- **SMTP/IMAP (stored creds): `EMAIL_ADDRESS` + `EMAIL_PASSWORD`** (+ `EMAIL_SMTP_HOST` /
|
||||
`EMAIL_IMAP_HOST` for non-mainstream providers; gmail/outlook/yahoo/icloud/fastmail inferred).
|
||||
The CLI sends via `send-email` and reads verify links via `poll-verification`. The `agentmail`
|
||||
skill (per-broker aliases) also counts.
|
||||
- Google Sheets tracker: the `google-workspace` skill.
|
||||
- The `scrapling` skill for stealth/Cloudflare-protected pages.
|
||||
|
||||
## How to Run
|
||||
|
||||
Run everything through the `terminal` tool. From this skill's directory:
|
||||
|
||||
```bash
|
||||
PDD="python3 scripts/pdd.py"
|
||||
```
|
||||
|
||||
The engine stores data under `$PDD_DATA_DIR` (default `$HERMES_HOME/unbroker`), written
|
||||
`0600`. Run via `terminal`, **not** `execute_code` (that sandbox scrubs env and redacts output, which
|
||||
breaks reading the dossier).
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Command | Purpose |
|
||||
|---|---|
|
||||
| `$PDD setup --auto` | **Autonomous setup**: detect capabilities, pick the most autonomous valid config (no questions) |
|
||||
| `$PDD doctor` | Readiness check: config, broker count, and which upgrades are on/available |
|
||||
| `$PDD intake --full-name "..." [--alias ...] [--email ... --phone ...] [--city --state] [--prior-location "City,ST"] --consent` | Create a consenting subject; captures aliases + multiple emails/phones + prior locations; prints `subject_id` |
|
||||
| `$PDD next <subject>` | **The autonomous loop driver**: ordered agent actions right now + human digest + `next_wake_at` |
|
||||
| `$PDD brokers [--priority crucial]` | List the people-search broker database (curated + live) |
|
||||
| `$PDD refresh-brokers` | Pull the latest BADBOOL people-search list **and the CA Data Broker Registry** (`next` requeues this automatically when the cache is stale) |
|
||||
| `$PDD registry [--search NAME]` | State registry coverage (CA ~545 ingested; VT/OR/TX portals surfaced); the DROP/email lane, not scanned |
|
||||
| `$PDD drop <subject> [--filed]` | **The one-shot legal lever**: one CA DROP request deletes from ALL registered brokers; `--filed` records it |
|
||||
| `$PDD plan <subject> [--priority crucial]` | Per-broker tier + method + `search_vectors` + the exact fields to disclose |
|
||||
| `$PDD plan <subject> --batch` | **Reduce view**: overlays ledger state, groups brokers by next action (unscanned/found/indirect/blocked/in_progress/done), collapses ownership clusters, **orders `found` cluster-parents-first + emits a tailored `parent_playbook`**, prints `next_actions` |
|
||||
| `$PDD fanout <subject> [--priority crucial] [--size 8]` | Batch brokers into parallel `delegate_task` subagents (auto for large runs) |
|
||||
| `$PDD record <subject> <broker> <state> [--found true] [--evidence JSON] [--disclosed F --channel C] [--reason "..."]` | Update the ledger (validated state machine); **auto-stamps `next_recheck_at`** |
|
||||
| `$PDD send-email <subject> <broker> --listing <url> [--kind ccpa_indirect ...]` | Render + record the request (recipient locked to the broker's own address). **browser** mode returns a `compose` payload to send via webmail (no password); **programmatic** mode SMTP-sends |
|
||||
| `$PDD verify-link <subject> <broker> --text '<body>'` | **browser mode**: extract a broker's verification link from webmail text you read (anti-phishing scored) |
|
||||
| `$PDD poll-verification <subject> [--broker <id>]` | **programmatic mode**: poll IMAP for verification links (anti-phishing scored); auto-advances `submitted → verification_pending` |
|
||||
| `$PDD render-email <subject> <broker> --listing <url>` | Draft only (fallback when no email mode is configured) |
|
||||
| `$PDD due <subject>` | Cases whose recheck window arrived (the cron re-scan queue) |
|
||||
| `$PDD tasks <subject>` | ONE consolidated human-task digest (present at END of run) |
|
||||
| `$PDD status <subject>` | Markdown status report |
|
||||
| `$PDD report <subject> --sheets` | Rows for the Google Sheets tracker |
|
||||
|
||||
## Batch operation (two-phase: crawl-all, then delete)
|
||||
|
||||
For anything past a couple of brokers, run this as **map → reduce → act**, not broker-by-broker:
|
||||
|
||||
- **Phase 1 - DISCOVER (read-only, parallel, idempotent).** Crawl *every* broker first and record a
|
||||
verdict for each (`found` / `not_found` / `indirect_exposure` / `blocked`). Scanning has no side
|
||||
effects, so it is safe to parallelize and retry. Getting the full exposure map *before* acting is
|
||||
what unlocks cluster dedup and prioritization below. **Default: the parent drives `web_extract`
|
||||
probes directly** - most people-search sites render name/phone/address results as static HTML that
|
||||
`web_extract` reads in seconds. Escalate to `browser_*` only for the few JS-only sites, and to
|
||||
`delegate_task` subagents only for genuinely *reasoning*-heavy work (large-scale namesake/relative
|
||||
disambiguation). **Do NOT hand a browser-toolset subagent a big list of brokers to crawl** - in the
|
||||
field this timed out repeatedly (600s, ~5-6 brokers each, no summary) because browser navigation is
|
||||
heavy; the ledger writes that survived came at 10x the cost of parent `web_extract`. A `blocked`
|
||||
(DataDome/Cloudflare/`antibot`) site is *not* a subagent job either: record `blocked` and requeue it
|
||||
for a stealth/cloud browser (Browserbase) pass. Subagent reports are self-reports - the parent
|
||||
re-fetches key URLs to confirm a `found` before trusting it (this cuts both ways: it caught a real
|
||||
listing the parent had wrongly assumed was a false positive).
|
||||
- **REDUCE - `$PDD plan <subject> --batch`.** Collapses the crawl into a phase-oriented plan: groups by
|
||||
next action, **collapses ownership clusters** (a parent removal that clears children is ONE action,
|
||||
not N - e.g. one Intelius/PeopleConnect suppression covers Truthfinder/Instant Checkmate/US Search/…),
|
||||
and prints `next_actions`. `phase` is `discover` while anything is unscanned, else `delete`.
|
||||
- **Phase 2 - DELETE (sequential, irreversible).** Work the reduced groups **parents first**:
|
||||
`plan --batch` orders the `found` group cluster-parents-first (most children first) and emits a
|
||||
`parent_playbook` with tailored, ordered steps per parent - follow that order and those steps
|
||||
(full recipes in `references/methods.md` → "Ownership clusters - DO PARENTS FIRST"). Do the
|
||||
cluster parents (skipping the covered children), **re-scan each parent's children after it confirms**
|
||||
(they usually drop out), then the standalone listings; send the `indirect_exposure` cases as
|
||||
CCPA/GDPR delete-my-PII emails (`send-email --kind ccpa_indirect`), and defer `blocked` to the
|
||||
stealth-browser pass. Opt-outs hit CAPTCHAs, email-verification loops, and session binding - work
|
||||
them **one at a time, carefully** (this is the opposite of fan-out), but do NOT stop to ask
|
||||
permission per submission in `autonomy=full`; in `assisted`, confirm each one. **Prefer deletion
|
||||
over suppression** where a broker offers both (e.g. PeopleConnect's "Right to Delete / DELETE MY
|
||||
USER DATA" actually removes data; the suppression flow only hides it).
|
||||
|
||||
Subagent reports are self-reports: the parent re-verifies key claims (listing URLs, match basis) before
|
||||
recording `found` and before any deletion.
|
||||
|
||||
## Procedure (the autonomous loop)
|
||||
|
||||
1. **Setup (once, no questions).** Run `$PDD setup --auto` - it detects capabilities and configures
|
||||
the most autonomous valid combination itself (programmatic email when `EMAIL_*` creds exist,
|
||||
Browserbase when its key exists, `age` encryption when the binary exists, `autonomy=full`). Then
|
||||
`$PDD doctor` and show the operator the readiness output **for information, not as a question** -
|
||||
proceed immediately. Mention what would unlock more automation (e.g. email creds) but do not wait.
|
||||
2. **Intake + consent (the ONE human conversation).** `$PDD intake ...` with `--consent` (and
|
||||
`--consent-method`). Without consent the engine refuses to plan or act. Collect everything in one
|
||||
pass - names/aliases, current + prior cities, emails, phones - so you never have to come back with
|
||||
questions. For California subjects, also read `references/legal/drop.md`: `next` will surface a
|
||||
`drop_submit` one-shot that deletes from every registered broker (~545) at once, which is the
|
||||
single highest-leverage action. File it, then `drop <subject> --filed`. For non-CA subjects the
|
||||
registry is covered by targeted CCPA/GDPR emails (`registry --search`, then `send-email`); the
|
||||
people-search sites are worked directly in either case.
|
||||
3. **Drain the queue.** Loop:
|
||||
|
||||
```
|
||||
while true:
|
||||
q = $PDD next <subject>
|
||||
if q.actions is empty: break
|
||||
execute EVERY action in order; record each outcome via $PDD record
|
||||
```
|
||||
|
||||
`next` emits, in order: `refresh_brokers` (stale cache), `fanout_scan`/`scan_inline` (Phase 1
|
||||
crawl - see step 4), `poll_verification` (in-flight email confirmations), `verify_removal` (due
|
||||
re-checks), `optout_web_form`/`optout_email_send` (Phase 2, parents-first with playbook steps),
|
||||
`indirect_email_send`, and `stealth_rescan`. Human-only work never appears as an action - it
|
||||
accumulates in `q.human_digest`. In `autonomy=full`, execute actions without pausing; honor
|
||||
`confirm_first` in `assisted` mode.
|
||||
4. **Scanning (when `next` says so).** For `fanout_scan`: run `$PDD fanout <subject>` and **spawn one
|
||||
`delegate_task` subagent per `batch`, in parallel, passing that batch's ready-made `brief`** - do
|
||||
not scan all brokers yourself sequentially. For `scan_inline`: scan the few brokers yourself.
|
||||
Either way, each broker gets **every** `search_vectors` entry via the `references/methods.md`
|
||||
ladder (`web_extract` → `site:` probe → `browser_navigate` → `scrapling`), a 404 is INCONCLUSIVE
|
||||
(not `not_found`), `blocked` is recorded when `antibot` is set and no stealth browser is available,
|
||||
and subject vs namesake/relative is confirmed before recording:
|
||||
`$PDD record <subject> <broker> <found|not_found|indirect_exposure|blocked> --found <bool> --evidence '{"listing_urls":[...]}'`.
|
||||
The parent re-verifies key `found` claims from subagents before trusting them.
|
||||
5. **Opt-outs (when `next` says so).** Actions come pre-ordered parents-first with `steps` from each
|
||||
broker record's own `optout.playbook` (field-verified; cluster parents like PeopleConnect,
|
||||
Whitepages, BeenVerified, Spokeo have exact, live-checked recipes). **Deletion beats
|
||||
suppression**: when an action carries `prefer_deletion`, complete the record's DELETION lane (e.g.
|
||||
PeopleConnect's "DELETE MY USER DATA"), never just the hide-my-listing flow. Per method:
|
||||
- **web_form** → drive `optout_url` with `browser_navigate`/`browser_type`/`browser_click`, submit
|
||||
only `disclosure_fields`, screenshot the confirmation, then the action's `after` record command.
|
||||
Playbooks may end with a right-to-delete `send-email` follow-up - do it (full erasure, not just
|
||||
listing suppression).
|
||||
- **email** → `$PDD send-email <subject> <broker> --kind <ccpa|gdpr|generic> --to <addr>
|
||||
--listing <url>` records + discloses in one step (recipient locked to addresses the broker
|
||||
record declares; `next` picks the kind from residency - never claim CCPA/GDPR for someone who
|
||||
can't). In **browser** mode it returns a recipient-locked `compose` payload: compose a new
|
||||
message to `compose.to` with `compose.subject`/`compose.body` exactly in the operator's webmail
|
||||
via `browser_*` and send (no password); in **programmatic** mode it SMTP-sends. `next` also
|
||||
routes human-gated forms (phone-callback/gov-ID) through a broker's deletion email when one
|
||||
exists - the **rescue lane** (verified Whitepages pattern). Draft-only falls back to
|
||||
`render-email` + a digest entry.
|
||||
- **captcha** → soft/managed challenges clear automatically on the default cloud browser (proceed
|
||||
as normal); only a hard interactive/behavioral challenge it can't pass is recorded `blocked`
|
||||
(requeued for the stealth/operator-browser pass). Never a solver service.
|
||||
- **phone_callback / account / gov_id / fax / mail / voice (T3)** *without a deletion email* →
|
||||
never an agent action; `next` already routed these to the digest. Record them:
|
||||
`$PDD record <subject> <broker> human_task_queued --reason "..."`.
|
||||
6. **Verification (when `next` says so).** In **programmatic** mode `$PDD poll-verification <subject>`
|
||||
finds arrived confirmation links via IMAP (anti-phishing scored, auto-advances state). In
|
||||
**browser** mode, open the broker's confirmation email in the operator's webmail and run
|
||||
`$PDD verify-link <subject> <broker> --text '<body>'` to score the link. Either way **open the
|
||||
link in the same browser** (several brokers bind the verification session to the browser that
|
||||
opens it), finish the flow, then record `awaiting_processing`. `confirmed_removed` ONLY after a
|
||||
verifying re-scan shows the listing gone - never off the submission flow's own confirmation page.
|
||||
7. **Wrap up (once per run).** When `next` returns no actions: present `$PDD tasks <subject>` (the
|
||||
consolidated human digest) if non-empty, then `$PDD status <subject>`; if the Sheets tracker is
|
||||
on, append `$PDD report <subject> --sheets` rows via the `google-workspace` skill.
|
||||
8. **Schedule the next wake-up.** `next` returns `next_wake_at` (earliest due re-check). Create ONE
|
||||
`cronjob` that re-runs this skill's loop for the subject (a prompt like: *"run the
|
||||
unbroker loop for <subject_id>: `$PDD next` and execute all actions"*). Processing
|
||||
windows, verification polls, and reappearance sweeps all flow through the same queue, so the case
|
||||
keeps advancing with zero human attention.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Never disclose more than the broker already shows.** Submit only `disclosure_fields`. The engine
|
||||
never volunteers SSN/ID numbers; you must not either.
|
||||
- **No consent, no action.** The engine enforces this; do not work around it to "research" a third party.
|
||||
- **`send-email` is idempotent + rate-limited.** It refuses to re-send a case already `submitted`
|
||||
or beyond (use `--force` only if a genuine re-send is needed), and SMTP sends are paced by
|
||||
`email_min_interval_seconds` (default 20s) with retry/backoff. Do not loop it to "make sure" -
|
||||
a successful SMTP handoff is not proof of delivery; the due-queue re-scan is the real confirmation.
|
||||
- **Ledger writes are locked.** Concurrent runs (cron + manual) serialize safely; if you ever see a
|
||||
lock timeout, another run is mid-write - let it finish, don't delete the `.lock` by hand.
|
||||
- **Autonomy ≠ improvisation.** Full autonomy means not *asking* between steps; it does not loosen any
|
||||
gate. If a broker demands MORE than the planned `disclosure_fields` mid-flow, stop that case and
|
||||
queue it (`human_task_queued --reason`) rather than deciding alone to disclose extra PII.
|
||||
- **Don't interrupt the run with questions.** Config choices are `setup --auto`'s job; human-only work
|
||||
goes to the digest. The only mid-run question that's ever warranted is a missing-identity fact that
|
||||
blocks scanning (e.g. no city at all) - and that should have been collected at intake.
|
||||
- **Use `terminal`, not `execute_code`** for `pdd.py` (secret scrubbing + output redaction break it).
|
||||
- **Dossiers are plaintext by default** (JSON, `0600` under `HERMES_HOME`). For at-rest encryption run
|
||||
`$PDD setup --encryption age` - it generates a local `age` key and encrypts dossiers + ledgers (the
|
||||
audit log holds field names only and stays plaintext). It guards casual/backup/commit exposure, not
|
||||
a full-`HERMES_HOME` read; set `PDD_AGE_IDENTITY` to a separate volume for real key separation.
|
||||
`$PDD doctor` shows whether encryption is *actually* engaged (not just whether `age` is installed).
|
||||
- **"Hidden from free search" ≠ deleted.** Only mark `confirmed_removed` after verifying the record is
|
||||
actually gone; note paid-tier retention in the report.
|
||||
- **Soft CAPTCHAs clear by default; don't fight the hard ones.** The default cloud browser passes
|
||||
managed/soft challenges as normal operation (those brokers stay T1). For a hard interactive one it
|
||||
genuinely can't pass, record `blocked` and let the stealth/operator-browser pass take it - never a
|
||||
third-party solver service or fingerprint spoofing.
|
||||
- **Broker pages change.** If a flow breaks, `$PDD record ... blocked` and flag the broker file in
|
||||
`references/brokers/` for re-verification instead of guessing.
|
||||
- **Verify non-field-verified records before submitting.** `confidence: auto` records came from
|
||||
parsing BADBOOL (read `optout.notes`/`optout.links`, confirm the real opt-out URL). `confidence:
|
||||
documented` records (several people-search sites) carry the correct published opt-out URL but have
|
||||
**not** been field-verified (they 403 datacenter IPs), so confirm the live flow via the operator's
|
||||
residential browser on first use, then set `last_verified`. Field-verified curated records (no
|
||||
`confidence`, e.g. the cluster parents) have checked mechanics and take precedence.
|
||||
|
||||
## Verification
|
||||
|
||||
- `scripts/run_tests.sh tests/skills/test_unbroker_skill.py` (hermetic; no network), or the
|
||||
dependency-free runner `python3 tests/skills/test_unbroker_skill.py`.
|
||||
- Dry run: `$PDD setup --auto && $PDD doctor && SID=$($PDD intake --full-name "Test Person"
|
||||
--email t@example.com --consent | python3 -c 'import sys,json;print(json.load(sys.stdin)["subject_id"])')
|
||||
&& $PDD next "$SID"` and confirm a readiness summary plus an ordered action queue.
|
||||
BIN
optional-skills/security/unbroker/assets/unbroker.png
Normal file
BIN
optional-skills/security/unbroker/assets/unbroker.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 444 KiB |
|
|
@ -0,0 +1,50 @@
|
|||
{
|
||||
"id": "advancedbackgroundchecks",
|
||||
"name": "AdvancedBackgroundChecks",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": ["US"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.advancedbackgroundchecks.com",
|
||||
"fetch": "web_extract",
|
||||
"match_signal": "result",
|
||||
"by": ["name", "phone", "address"],
|
||||
"url_patterns": {
|
||||
"name": "https://www.advancedbackgroundchecks.com/name/{first}-{last}",
|
||||
"name_state": "https://www.advancedbackgroundchecks.com/name/{first}-{last}/in/{ST}",
|
||||
"phone": "https://www.advancedbackgroundchecks.com/phone/{digits10}",
|
||||
"address": "https://www.advancedbackgroundchecks.com/address/{num}-{street-with-type}-{city}-{ST}-{zip}",
|
||||
"person_profile": "https://www.advancedbackgroundchecks.com/find/person/{first}-{last}-{22charID}"
|
||||
},
|
||||
"url_format_quirks": [
|
||||
"All path segments lowercase, spaces -> hyphens. Name: /name/jane-public ; name+state: /name/jane-public/in/NY (state 2-letter UPPERCASE).",
|
||||
"Phone: /phone/5551234567 (10 digits, NO punctuation).",
|
||||
"Address: /address/123-main-st-anytown-ny-12345 (all hyphen-joined incl. ZIP; abbreviations like 'S' and 'Ave' kept verbatim, not expanded).",
|
||||
"Removable unit is the person profile: /find/person/{first}-{last}-{22charID} (opaque mixed-case ID). Also /find/name/{first}-{last} and /find/name/{first}-{last}/in/{ST}.",
|
||||
"NO 404s for plausible patterns: an empty search returns a soft-200 page with 'similar' fuzzy rows. The real 'not found' signal is the ABSENCE of an exact-match block in the results heading, NOT an HTTP error. Do not record not_found off a 404 here; read the heading.",
|
||||
"No anti-bot gating on search/result/phone/address pages (web_extract reads them fine). Site self-brands as 'ActualPeopleSearch' in FAQ text.",
|
||||
"Search tabs: /?tab=phone /?tab=email /?tab=address. Directories: /lastnames/{surname} /firstnames/{first} /people/{state-name}/{city-name} /people/zip/{zip} /people/areacode/{code}."
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.advancedbackgroundchecks.com/opt-out",
|
||||
"requires": {
|
||||
"profile_url": false,
|
||||
"email_verification": true,
|
||||
"captcha": true,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["full_name", "contact_email"],
|
||||
"notes": "Two-step email-verification flow: initial form (radio 'I am: The subject of the request / An authorized agent of the subject', First name*, Middle name, Last name*, Email*, consent) -> emailed link -> full form -> confirmation (processed within 45 days). CAPTCHA-gated: Google reCAPTCHA ('Recaptcha requires verification'). Verified read-only 2026-06-30; not submitted.",
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": "2026-06-30",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
{
|
||||
"id": "beenverified",
|
||||
"name": "BeenVerified",
|
||||
"category": "people_search",
|
||||
"priority": "crucial",
|
||||
"jurisdictions": ["US"],
|
||||
"parent": "beenverified",
|
||||
"owns": ["peoplelooker", "peoplesmart"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.beenverified.com/app/optout/search",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"by": ["name", "phone", "email", "address"]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T1",
|
||||
"method": "web_form",
|
||||
"url": "https://www.beenverified.com/svc/optout/search/optouts",
|
||||
"email": "privacy@beenverified.com",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["full_name", "contact_email", "profile_url"],
|
||||
"deletion": {
|
||||
"via": "email_followup",
|
||||
"email": "privacy@beenverified.com",
|
||||
"kinds": ["ccpa", "generic"],
|
||||
"notes": "privacy@beenverified.com is the documented address for opt-out, access, deletion, and correction requests (verified from the live privacy policy 2026-07-01). Controller is The Lifetime Value Co. -- a deletion request here can name their other properties too. DPO: dpo@beenverified.com. CCPA window: respond within 45 days (extendable to 90)."
|
||||
},
|
||||
"playbook": [
|
||||
"Opt-out tool at beenverified.com/svc/optout/search/optouts ('Do Not Sell or Share' footer link; the legacy /app/optout/search path serves the same search) -- clears PeopleLooker + PeopleSmart.",
|
||||
"Search the subject, open the matching listing, and submit with the confirmed profile URL + contact email. Email verification: poll-verification picks up the confirmation link; open it in the agent's own browser.",
|
||||
"ONE opt-out per email address via the tool; for additional listings email support@beenverified.com or privacy@beenverified.com.",
|
||||
"FULL DELETION follow-up (right-to-delete, beyond listing suppression): send-email --kind ccpa (CA) / generic to privacy@beenverified.com naming the listing URL(s); as the controller is The Lifetime Value Co., ask that the deletion cover affiliated LTV properties (NeighborWho, Ownerly, NumberGuru, Bumper) in the same request.",
|
||||
"A separate property-search opt-out exists (NeighborWho/Ownerly are property-focused sisters); note residual exposure if relevant and re-scan the children after the parent confirms."
|
||||
],
|
||||
"notes": "Opt-out form + deletion email both verified from the live privacy policy 2026-07-01 (policy dated 2025-10-21). Authorized agents: signed authorization letter or CA POA emailed to privacy@beenverified.com; agent-submitted delete/know requests must include the consumer's name, valid email, age, and address.",
|
||||
"quirks": [
|
||||
"The 'Do Not Sell or Share My Personal Information' footer link now points to /svc/optout/search/optouts (observed 2026-07-01); records citing /app/optout/search are the same tool's older entry.",
|
||||
"One opt-out per email address through the tool; email support for additional removals. The same browser/inbox must open the confirmation link.",
|
||||
"Sister LTV Co. properties (NeighborWho, Ownerly, NumberGuru, Bumper) keep separate opt-out tools -- do NOT assume the BV tool cleared them; the privacy@ deletion request can name them, then verify by scanning each.",
|
||||
"Right-to-know requests get an initial response within 10 days; deletion/access within 45 days (CCPA)."
|
||||
],
|
||||
"est_processing_days": 7,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": "2026-07-01",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
{
|
||||
"id": "clustal",
|
||||
"name": "Clustal",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": ["US"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.clustal.org/",
|
||||
"fetch": "web_extract",
|
||||
"match_signal": "record",
|
||||
"by": ["name"],
|
||||
"url_patterns": {
|
||||
"name_index": "https://www.clustal.org/people-search/{letter}/{first-last}/",
|
||||
"name_state": "https://www.clustal.org/people-search/{letter}/{first-last}/{st}/",
|
||||
"record": "https://www.clustal.org/record/{first-last}-{8charID}/"
|
||||
},
|
||||
"url_format_quirks": [
|
||||
"Name index: /people-search/{first-initial}/{first-last}/ e.g. /people-search/k/jane-public/ (lowercase, first letter of LAST name as the {letter} segment, hyphen-joined name). Optional state refine appends /{st}/ lowercase 2-letter.",
|
||||
"Detail (removable unit): /record/{first-last}-{8charID}/ e.g. /record/jane-public-a1b2c3d4/ (opaque mixed-case 8-char id). The index page links each match to its /record/ URL.",
|
||||
"Readable via web_extract (no hard bot gate as of 2026-07-01). Rich profile: age/DOB, current+prior addresses, phones, emails, relatives, neighbors, associates.",
|
||||
"Name only: search is by name(+state); no reverse phone/email/address path observed. NOTE: clustal.org squats the 'Clustal' bioinformatics brand but IS a real people-search/data-broker site ('Find Your DNA Relatives'), not the sequence-alignment tool - do not dismiss it as a false positive."
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T0",
|
||||
"method": "web_form",
|
||||
"url": "https://www.clustal.org/privacy-control/",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["profile_url", "contact_email"],
|
||||
"notes": "Opt-out at /privacy-control/ using the /record/ profile URL; support help@clustal.org. Verify the exact form fields + any CAPTCHA live before submitting (requires flags below are provisional from the site's opt-out copy, not yet a live form walk-through).",
|
||||
"email": "help@clustal.org",
|
||||
"est_processing_days": 7,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": "2026-07-01",
|
||||
"source": "BADBOOL",
|
||||
"confidence": "curated"
|
||||
}
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
{
|
||||
"id": "clustrmaps",
|
||||
"name": "ClustrMaps",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "clustrmaps",
|
||||
"owns": [],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://clustrmaps.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"antibot": "cloudflare",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://clustrmaps.com/bl/opt-out",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"contact_email",
|
||||
"profile_url"
|
||||
],
|
||||
"notes": "Address/resident aggregator. Opt-out at /bl/opt-out: submit the profile URL + email, confirm the link.",
|
||||
"quirks": [
|
||||
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
|
||||
"Needs the confirmed profile_url (the address/person page).",
|
||||
"Email verification required."
|
||||
],
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": null,
|
||||
"source": "curated",
|
||||
"confidence": "documented"
|
||||
}
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
{
|
||||
"id": "cyberbackgroundchecks",
|
||||
"name": "CyberBackgroundChecks",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "cyberbackgroundchecks",
|
||||
"owns": [],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.cyberbackgroundchecks.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"antibot": "cloudflare",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.cyberbackgroundchecks.com/removal",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"full_name",
|
||||
"contact_email",
|
||||
"profile_url"
|
||||
],
|
||||
"notes": "Free people-search. Opt-out at /removal: find the record, submit email, confirm link.",
|
||||
"quirks": [
|
||||
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
|
||||
"Needs the confirmed profile_url.",
|
||||
"Email verification required."
|
||||
],
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": null,
|
||||
"source": "curated",
|
||||
"confidence": "documented"
|
||||
}
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
{
|
||||
"id": "familytreenow",
|
||||
"name": "FamilyTreeNow",
|
||||
"category": "people_search",
|
||||
"priority": "crucial",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "familytreenow",
|
||||
"owns": [],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.familytreenow.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"antibot": "cloudflare",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.familytreenow.com/optout",
|
||||
"requires": {
|
||||
"profile_url": false,
|
||||
"email_verification": false,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"full_name"
|
||||
],
|
||||
"notes": "Notorious free people-search / doxxing site (no registry). Opt-out: search the record, select it, confirm removal on-site. No account, free.",
|
||||
"quirks": [
|
||||
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
|
||||
"Select the exact record from search results, then confirm removal; historically no email step, but re-lists periodically so keep the re-scan scheduled."
|
||||
],
|
||||
"est_processing_days": 2,
|
||||
"reappearance_risk": "high"
|
||||
},
|
||||
"last_verified": null,
|
||||
"source": "curated",
|
||||
"confidence": "documented"
|
||||
}
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
{
|
||||
"id": "fastpeoplesearch",
|
||||
"name": "FastPeopleSearch",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": ["US"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.fastpeoplesearch.com/",
|
||||
"fetch": "browser",
|
||||
"antibot": "datadome",
|
||||
"match_signal": "result",
|
||||
"by": ["name", "phone", "address"],
|
||||
"url_patterns": {
|
||||
"name": "https://www.fastpeoplesearch.com/name/{first}-{last}"
|
||||
},
|
||||
"url_format_quirks": [
|
||||
"Name path /name/jane-public (lowercase, hyphen join) was ACCEPTED by the router under challenge, so the name pattern is confirmed even though content never rendered.",
|
||||
"Sibling of truepeoplesearch (near-identical removal flow/branding); phone pattern is LIKELY /{digits} or /phone/{digits} and address /address/... but UNCONFIRMED - do not assume.",
|
||||
"MOST aggressively gated of the people-search trio (2026-06-30): both server-side scraping (Firecrawl 504) AND headless browser (Cloudflare -> DataDome) fail on search AND /removal. Treat as fully blocked; needs residential-proxy/stealth browser to scan or opt out."
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.fastpeoplesearch.com/removal",
|
||||
"requires": {
|
||||
"profile_url": false,
|
||||
"email_verification": true,
|
||||
"captcha": true,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["full_name", "contact_email"],
|
||||
"notes": "Opt-out NOT directly observed (2026-06-30): /removal is itself behind DataDome. By sibling-site pattern almost certainly mirrors truepeoplesearch (email-link flow, subject/agent toggle, name+email fields, hCaptcha) - INFERRED, not verified. Confirm before acting.",
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "high"
|
||||
},
|
||||
"last_verified": "2026-06-30",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
{
|
||||
"id": "intelius",
|
||||
"name": "Intelius",
|
||||
"category": "people_search",
|
||||
"priority": "crucial",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "peopleconnect",
|
||||
"owns": [
|
||||
"truthfinder",
|
||||
"instantcheckmate",
|
||||
"ussearch",
|
||||
"zabasearch",
|
||||
"classmates",
|
||||
"peoplefinder",
|
||||
"peoplelookup",
|
||||
"addresses",
|
||||
"anywho",
|
||||
"publicrecords"
|
||||
],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.intelius.com/",
|
||||
"fetch": "web_extract",
|
||||
"match_signal": "result",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
],
|
||||
"url_patterns": {
|
||||
"name": "https://www.intelius.com/people-search/{First}-{Last}/",
|
||||
"name_state": "https://www.intelius.com/people-search/{first}-{last}/{state-name}/",
|
||||
"full_report": "https://www.intelius.com/search/?firstName={First}&lastName={Last}&city={City}&state={ST}&traffic%5Bsource%5D=INTSEO"
|
||||
},
|
||||
"url_format_quirks": [
|
||||
"Name summary page: /people-search/{First}-{Last}/ (hyphen join; case-insensitive, both Jane-Public and jane-public work). Readable via web_extract (no hard bot gate as of 2026-06-30).",
|
||||
"State refine: /people-search/{first}-{last}/{state-name}/ with the state SPELLED OUT lowercase, e.g. /jane-public/new-york/ (NOT the 2-letter code).",
|
||||
"The summary shows last-known address + a 'possible relatives' list + a FAQ ('Where does X live?'). Corroborate the subject by address before acting; the 'Top People with the Last Name {Last}' block is unrelated namesakes.",
|
||||
"Part of PeopleConnect: same data surfaces on Truthfinder/InstantCheckmate/USSearch; one suppression at suppression.peopleconnect.us covers the cluster."
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T1",
|
||||
"method": "web_form",
|
||||
"url": "https://suppression.peopleconnect.us/login",
|
||||
"email": "privacy@peopleconnect.us",
|
||||
"requires": {
|
||||
"profile_url": false,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"contact_email"
|
||||
],
|
||||
"deletion": {
|
||||
"via": "in_flow",
|
||||
"url": "https://suppression.peopleconnect.us/guided-mode",
|
||||
"email": "privacy@peopleconnect.us",
|
||||
"kinds": ["ccpa", "generic"],
|
||||
"notes": "The portal's 'Right to Delete / DELETE MY USER DATA' (bottom of guided-mode) is the verified deletion path and covers the whole cluster. privacy@peopleconnect.us is the documented rights-request address (their 2025 CPRA metrics: 33,513 delete requests, median response < 1 day) - use it as the fallback if the portal breaks."
|
||||
},
|
||||
"playbook": [
|
||||
"PeopleConnect portal (suppression.peopleconnect.us/login, privacy-center entry at /privacy-center) -- ONE flow here clears Truthfinder, Instant Checkmate, US Search, ZabaSearch, Classmates and ~15 more. DO THIS PARENT FIRST.",
|
||||
"Step 1 asks ONLY for an email + consent checkbox (no name/DOB, no CAPTCHA) -> sends a verification email. Least-disclosure entry: just the contact email.",
|
||||
"poll-verification will pick up the verify link. The link authenticates a SESSION bound to the browser that OPENS it: the SAME agent browser must open the link and drive /guided-mode (a different browser is bounced to /login).",
|
||||
"SUPPRESSION != DELETION -- guided-mode's default flow only HIDES the report (data retained, can re-list). Scroll to the BOTTOM of guided-mode and use 'Right to Delete / DELETE MY USER DATA' (CCPA/CPRA) -- this actually deletes across the cluster and emails its own confirmation link; poll and open that too.",
|
||||
"If the portal breaks: email privacy@peopleconnect.us (rights-request address, median response < 1 day per their published metrics), or sister addresses privacy@intelius.com / privacy@truthfinder.com / privacy@instantcheckmate.com / support@ussearch.com / privacy@classmates.com; phone 1-888-245-1655.",
|
||||
"After the deletion confirms, re-scan the covered children (they normally drop out) before submitting any duplicate child opt-out."
|
||||
],
|
||||
"notes": "PeopleConnect portal covers the cluster. Authorized-agent requests: signed written authorization (full name, address, phone, the email the consumer uses) or POA; for Right-to-Delete they verify agent authority with the consumer by email. Verified from the live privacy policy 2026-07-01 (policy dated 2026-06-30).",
|
||||
"quirks": [
|
||||
"Step 1 (suppression.peopleconnect.us/login) asks ONLY for an email + a consent checkbox, then 'Continue' -> a verification email with a link. No CAPTCHA, no name/DOB at step 1. Least-disclosure entry: just the contact email. Verified live 2026-06-30.",
|
||||
"The verification link authenticates a SESSION and lands on /guided-mode. That session is bound to the browser that OPENED it; a different browser hitting /guided-mode is redirected back to /login. So for hands-off automation the SAME agent browser must open the verify link (Mode B: read inbox -> agent browser navigates the link -> drive guided-mode).",
|
||||
"SUPPRESSION != DELETION. The guided-mode/suppression flow only HIDES the background report (data retained, can re-list). At the BOTTOM of guided-mode there is a separate 'Right to Delete' section with a 'DELETE MY USER DATA' button (CCPA/CPRA) that removes the user data across the cluster - this is the stronger action and should be preferred. It also emails a confirmation link. (Confirmed via security.org Instant Checkmate guide + live flow 2026-06-30.)",
|
||||
"Their published request metrics (2025, all US users regardless of state): 33,513 deletion requests, 26,603 complied, median response < 1 day - the deletion lane is real and fast. Verified from privacy policy 2026-07-01."
|
||||
],
|
||||
"est_processing_days": 7,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": "2026-07-01",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
{
|
||||
"id": "mylife",
|
||||
"name": "MyLife",
|
||||
"category": "people_search",
|
||||
"priority": "crucial",
|
||||
"jurisdictions": ["US"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.mylife.com",
|
||||
"fetch": "browser",
|
||||
"match_signal": "profile",
|
||||
"by": ["name"]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T3",
|
||||
"method": "phone",
|
||||
"url": "https://www.mylife.com/privacyrequest",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": false,
|
||||
"captcha": false,
|
||||
"gov_id": true,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"phone_voice": true,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["full_name", "profile_url"],
|
||||
"notes": "Often pushes a driver's-license upload or a call to (888) 704-1900. Emailing privacy@mylife.com with name + profile link is an alternative. Also covers Wink.com.",
|
||||
"est_processing_days": 14,
|
||||
"reappearance_risk": "high"
|
||||
},
|
||||
"last_verified": "2026-06-28",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
{
|
||||
"id": "nuwber",
|
||||
"name": "Nuwber",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "nuwber",
|
||||
"owns": [],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://nuwber.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"antibot": "cloudflare",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://nuwber.com/removal/link",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"contact_email",
|
||||
"profile_url"
|
||||
],
|
||||
"notes": "People-search. Opt-out: submit the profile URL + email at /removal/link, confirm via the emailed link.",
|
||||
"quirks": [
|
||||
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
|
||||
"Needs the confirmed profile_url (paste the listing URL you recorded).",
|
||||
"Email verification required."
|
||||
],
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": null,
|
||||
"source": "curated",
|
||||
"confidence": "documented"
|
||||
}
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
{
|
||||
"id": "peekyou",
|
||||
"name": "PeekYou",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "peekyou",
|
||||
"owns": [],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.peekyou.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"antibot": "cloudflare",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.peekyou.com/about/contact/optout",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"full_name",
|
||||
"contact_email",
|
||||
"profile_url"
|
||||
],
|
||||
"notes": "Aggregates social/web profiles. Opt-out: paste your PeekYou profile URL(s), pick a reason, provide email, confirm the link.",
|
||||
"quirks": [
|
||||
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
|
||||
"Needs the confirmed PeekYou profile_url.",
|
||||
"Email verification required."
|
||||
],
|
||||
"est_processing_days": 7,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": null,
|
||||
"source": "curated",
|
||||
"confidence": "documented"
|
||||
}
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
{
|
||||
"id": "peoplefinders",
|
||||
"name": "PeopleFinders",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "peoplefinders",
|
||||
"owns": [],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.peoplefinders.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"antibot": "cloudflare",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.peoplefinders.com/opt-out",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"full_name",
|
||||
"contact_email",
|
||||
"profile_url"
|
||||
],
|
||||
"notes": "Standalone people-search (Confi-Chek family). Opt-out: find the listing, submit with a contact email, confirm via the emailed link.",
|
||||
"quirks": [
|
||||
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
|
||||
"Needs the confirmed profile_url from evidence.",
|
||||
"Email verification: poll-verification picks up the link; open it in the agent browser."
|
||||
],
|
||||
"est_processing_days": 7,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": null,
|
||||
"source": "curated",
|
||||
"confidence": "documented"
|
||||
}
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
{
|
||||
"id": "radaris",
|
||||
"name": "Radaris",
|
||||
"category": "people_search",
|
||||
"priority": "crucial",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://radaris.com/",
|
||||
"fetch": "web_extract",
|
||||
"match_signal": "View Profile",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
],
|
||||
"url_patterns": {
|
||||
"name": "https://radaris.com/p/{First}/{Last}/"
|
||||
},
|
||||
"url_format_quirks": [
|
||||
"Name profile page: /p/{First}/{Last}/ with First/Last Capitalized and a TRAILING slash, e.g. /p/Jane/Public/ . Readable via web_extract (no hard bot gate as of 2026-06-30).",
|
||||
"The page aggregates: a 'phone numbers & home addresses' table (the DIRECT hit for the subject), a relatives/namesakes carousel ('Review the potential relatives'), and resume/CV records. The subject's removable row is in the address table; the carousel entries are OTHER people - disambiguate by address/age before acting.",
|
||||
"Page is noisy with testimonials/reviews boilerplate; the signal blocks are 'phone numbers & home addresses N' and 'resumes & CV records N'."
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://radaris.com/control-privacy",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": true,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"profile_url",
|
||||
"contact_email"
|
||||
],
|
||||
"notes": "Multi-step control-privacy wizard: step 1 NEXT -> step 2 'identify your personal page' (paste the NUMBERED profile URL into the 'Or Enter URL of your page' field; typing reveals a NEXT submit button) -> then user_email + Google reCAPTCHA -> emailed verification link. If there is no View Profile button for the subject, email customer-service@radaris.com and reply to the auto-response until removed.",
|
||||
"email": "customer-service@radaris.com",
|
||||
"quirks": [
|
||||
"CAPTCHA: the form carries a Google reCAPTCHA (hidden field g-recaptcha-response) plus anti-bot fingerprint tokens (jfp, token). Tier is T2 without a captcha-clearing browser backend (T1 with Browserbase). (Record previously mis-declared captcha:false.)",
|
||||
"The /control-privacy form REQUIRES the per-person NUMBERED profile URL. Pasting the aggregate /p/{First}/{Last}/ URL is rejected with the validation error: 'The URL must include unique number at the end'. The real removable URL looks like https://{region}.radaris.com/person/~First-Last/1234567890 (see the field placeholder).",
|
||||
"The subject's own record may appear ONLY as a static row in the 'phone numbers & home addresses' table with NO 'View Profile' link, so no numbered profile URL is exposed for them (the /p/{First}/{Last}/ page deep-links only to relatives/namesakes via JS onclick, not static hrefs). When that happens the /control-privacy form cannot be used -- do NOT fabricate or submit a relative's or namesake's profile URL. Fall back to email: write customer-service@radaris.com with the subject's own listed details and request removal, replying to the auto-response until confirmed.",
|
||||
"Pressing Enter in the URL field reloads the wizard to step 1 (does not submit); type into the field and click the revealed NEXT button instead."
|
||||
],
|
||||
"est_processing_days": 14,
|
||||
"reappearance_risk": "high"
|
||||
},
|
||||
"last_verified": "2026-06-30",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
{
|
||||
"id": "searchpeoplefree",
|
||||
"name": "SearchPeopleFree",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "searchpeoplefree",
|
||||
"owns": [],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.searchpeoplefree.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"antibot": "cloudflare",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.searchpeoplefree.com/opt-out",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"full_name",
|
||||
"contact_email",
|
||||
"profile_url"
|
||||
],
|
||||
"notes": "Free people-search. Opt-out: find the listing, submit with an email, confirm the link.",
|
||||
"quirks": [
|
||||
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
|
||||
"Needs the confirmed profile_url.",
|
||||
"Email verification required."
|
||||
],
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": null,
|
||||
"source": "curated",
|
||||
"confidence": "documented"
|
||||
}
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
{
|
||||
"id": "spokeo",
|
||||
"name": "Spokeo",
|
||||
"category": "people_search",
|
||||
"priority": "crucial",
|
||||
"jurisdictions": ["US"],
|
||||
"parent": "spokeo",
|
||||
"owns": ["freepeopledirectory"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.spokeo.com/search",
|
||||
"fetch": "browser",
|
||||
"match_signal": "profile",
|
||||
"by": ["name", "phone", "email", "address"]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T1",
|
||||
"method": "web_form",
|
||||
"url": "https://www.spokeo.com/optout",
|
||||
"email": "privacy@spokeo.com",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["profile_url", "contact_email"],
|
||||
"deletion": {
|
||||
"via": "email_followup",
|
||||
"email": "privacy@spokeo.com",
|
||||
"kinds": ["ccpa", "generic"],
|
||||
"notes": "privacy@spokeo.com is the documented direct privacy contact (verified live on /optout 2026-07-01). Use for full CCPA deletion beyond listing opt-out, for listings the form rejects, and when more listings keep surfacing."
|
||||
},
|
||||
"playbook": [
|
||||
"Opt-out form at spokeo.com/optout -- clears FreePeopleDirectory. Inputs: the confirmed profile URL + contact email; the form emails a confirmation link (poll-verification picks it up; open it in the agent's own browser).",
|
||||
"EVERY LISTING SEPARATELY: 'you may have multiple listings on Spokeo. Each one is identified by a unique URL and must be opted out individually' (their own wording). Run ALL search vectors first, collect every listing URL, and submit one opt-out per URL.",
|
||||
"Fast: requests process in 24-48 hours per their page -- re-scan after 2 days and record the real outcome.",
|
||||
"FULL DELETION follow-up: send-email --kind ccpa (CA) / generic to privacy@spokeo.com naming all listing URLs -- covers data retained beyond the free-search suppression.",
|
||||
"Paid/account data may be retained even when hidden from free search -- verify actual removal via re-scan; never mark confirmed_removed off the free-search view alone."
|
||||
],
|
||||
"notes": "Form + privacy@spokeo.com verified live 2026-07-01. Their page: opting out does not remove data from original sources and listings may reappear as new public records arrive -- keep the reappearance re-scan scheduled.",
|
||||
"quirks": [
|
||||
"Profile URL formats the form accepts: listing URL like spokeo.com/{First}-{Last}/{City}/{ST}/p{id} or a purchase URL spokeo.com/purchase?q=... (examples shown on /optout).",
|
||||
"Multiple listings per person are NORMAL (one per name x location x record cluster); each unique URL must be opted out individually -- feed every found listing URL through the form.",
|
||||
"Processing is 24-48h ('depending on the nature of your request and the amount of data').",
|
||||
"Paid accounts may retain data even when hidden from free search - verify actual removal, don't mark confirmed_removed off the free-search view alone."
|
||||
],
|
||||
"est_processing_days": 2,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": "2026-07-01",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
{
|
||||
"id": "thatsthem",
|
||||
"name": "That's Them",
|
||||
"category": "people_search",
|
||||
"priority": "crucial",
|
||||
"jurisdictions": ["US"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://thatsthem.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"by": ["name", "phone", "email", "address"],
|
||||
"url_patterns": {
|
||||
"name": "https://thatsthem.com/name/{First}-{Last}/{City}-{ST}",
|
||||
"phone": "https://thatsthem.com/phone/{areacode}-{prefix}-{line}",
|
||||
"email": "https://thatsthem.com/email/{email}",
|
||||
"address": "https://thatsthem.com/address/{Street}-{City}-{ST}-{ZIP}"
|
||||
},
|
||||
"url_format_quirks": [
|
||||
"ADDRESS path is fully hyphen-joined and joins street+city+state+ZIP with hyphens (e.g. /address/123-Main-St-Anytown-NY-12345) and REQUIRES the ZIP. The older slash form (/address/Street/City-ST) and any ZIP-less form 404.",
|
||||
"NAME and PHONE and EMAIL paths use a slash before city/state and do NOT need a ZIP: /name/First-Last/City-ST , /phone/AAA-PPP-LLLL , /email/addr@host.",
|
||||
"Street abbreviations are kept verbatim (Ave, Pl, St) - do NOT expand to Avenue/Place/Street; expanding 404s.",
|
||||
"A 404 on a constructed URL means WRONG PATTERN, not 'no record present'. Treat as INCONCLUSIVE: fall back to the on-site search box (browser_type into the address/name field + submit) and read the resulting canonical URL."
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://thatsthem.com/optout",
|
||||
"requires": {
|
||||
"profile_url": false,
|
||||
"email_verification": false,
|
||||
"captcha": true,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["full_name", "street", "city", "state", "postal", "contact_email", "phone"],
|
||||
"notes": "Opt-out form is gated by a Cloudflare Turnstile CAPTCHA (so tier is T2 without a captcha-clearing browser backend; T1 with Browserbase). All 7 fields are marked required by the form (Full Name, Street Address, City, State, ZIP, Email, Phone). Site sends a confirmation email and states ~72h processing (this is a completion notice, not a click-to-verify link). Least-disclosure tension: the form DEMANDS email+phone even though a public record may show less - disclose only to remove a record that is actually about the subject. Do not click the Spokeo identity-theft-protection link (paid product).",
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "low"
|
||||
},
|
||||
"last_verified": "2026-06-30",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
{
|
||||
"id": "truepeoplesearch",
|
||||
"name": "TruePeopleSearch",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": ["US"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.truepeoplesearch.com/",
|
||||
"fetch": "browser",
|
||||
"antibot": "datadome",
|
||||
"match_signal": "result",
|
||||
"by": ["name", "phone", "address", "email"],
|
||||
"url_patterns": {
|
||||
"name": "https://www.truepeoplesearch.com/results?name={First%20Last}&citystatezip={City,%20ST}"
|
||||
},
|
||||
"url_format_quirks": [
|
||||
"Search results URL is QUERY-PARAM, not path: /results?name=Jane%20Public&citystatezip=Anytown,%20NY (space-encoded; comma between city and state). Confirmed as the canonical form the site's own search box generates.",
|
||||
"On-site search tabs: Name / Phone / Address / Email / Neighbors (so name-, phone-, address-, and email-searchable).",
|
||||
"HARD-BLOCKED for automated reads (2026-06-30): Cloudflare 'Just a moment...' interstitial -> DataDome device-check CAPTCHA (geo.captcha-delivery.com) on every results navigation. Page chrome (header/footer) may render while the result body stays blocked; submitting any search bounces to DataDome. web_extract/Firecrawl 504-timed out. Needs a residential-proxy/stealth browser (e.g. Browserbase) to scan; otherwise record 'blocked'."
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.truepeoplesearch.com/removal",
|
||||
"requires": {
|
||||
"profile_url": false,
|
||||
"email_verification": true,
|
||||
"captcha": true,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["full_name", "contact_email"],
|
||||
"notes": "Removal page renders even when results are blocked. Flow: name+email+captcha -> emailed link -> fill opt-out form matching the record -> confirmation ('allow 3 days'). Fields: dropdown 'Exercise my rights as a: The subject of the request / An authorized agent of the subject', First Name*, Middle Name, Last Name*, Email Address*, consent checkbox. CAPTCHA-gated: hCaptcha ('I am human'). Contact support@truepeoplesearch.com; PO Box 7775 PMB 29296, San Francisco CA 94120-7775. Verified read-only 2026-06-30; not submitted. (Record previously mis-declared email_verification:false.)",
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "high"
|
||||
},
|
||||
"last_verified": "2026-06-30",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
{
|
||||
"id": "usphonebook",
|
||||
"name": "USPhoneBook",
|
||||
"category": "people_search",
|
||||
"priority": "high",
|
||||
"jurisdictions": [
|
||||
"US"
|
||||
],
|
||||
"parent": "usphonebook",
|
||||
"owns": [],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.usphonebook.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "result",
|
||||
"antibot": "cloudflare",
|
||||
"by": [
|
||||
"name",
|
||||
"phone",
|
||||
"address"
|
||||
]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T2",
|
||||
"method": "web_form",
|
||||
"url": "https://www.usphonebook.com/opt-out",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": [
|
||||
"full_name",
|
||||
"contact_email",
|
||||
"profile_url"
|
||||
],
|
||||
"notes": "Reverse-phone + people-search. Opt-out: paste the listing URL, provide an email, confirm via the emailed link.",
|
||||
"quirks": [
|
||||
"Opt-out URL is the documented public endpoint; datacenter IPs get 403 (anti-bot), so confirm the live flow via the operator's residential browser before the first submission, then set last_verified.",
|
||||
"Needs the confirmed profile_url.",
|
||||
"Email verification required."
|
||||
],
|
||||
"est_processing_days": 3,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": null,
|
||||
"source": "curated",
|
||||
"confidence": "documented"
|
||||
}
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
{
|
||||
"id": "whitepages",
|
||||
"name": "Whitepages",
|
||||
"category": "people_search",
|
||||
"priority": "crucial",
|
||||
"jurisdictions": ["US"],
|
||||
"parent": "whitepages",
|
||||
"owns": ["411"],
|
||||
"search": {
|
||||
"method": "url_pattern",
|
||||
"url": "https://www.whitepages.com/",
|
||||
"fetch": "browser",
|
||||
"match_signal": "listing",
|
||||
"by": ["name", "phone", "address"]
|
||||
},
|
||||
"optout": {
|
||||
"tier": "T1",
|
||||
"method": "email",
|
||||
"url": "https://www.whitepages.com/suppression_requests",
|
||||
"email": "privacyrequest@whitepages.com",
|
||||
"requires": {
|
||||
"profile_url": true,
|
||||
"email_verification": true,
|
||||
"captcha": false,
|
||||
"gov_id": false,
|
||||
"account": false,
|
||||
"phone_callback": false,
|
||||
"payment": false
|
||||
},
|
||||
"inputs": ["full_name", "contact_email", "profile_url"],
|
||||
"deletion": {
|
||||
"via": "email",
|
||||
"email": "privacyrequest@whitepages.com",
|
||||
"url": "https://whitepagesprivacy.zendesk.com/hc/en-us/requests/new",
|
||||
"kinds": ["ccpa", "generic"],
|
||||
"notes": "privacyrequest@whitepages.com handles BOTH opt-out (removal from the site) and CCPA deletion requests, explicitly offered for people who do not want to provide a phone number for the automated tool. ~2 business day reply; opt-outs processed within 15 days. Verified from whitepages.com/privacy/consumer-rights 2026-07-01 (page dated 2026-06-22)."
|
||||
},
|
||||
"playbook": [
|
||||
"EMAIL LANE (fully autonomous -- use this): send the removal + deletion request to privacyrequest@whitepages.com including the confirmed listing URL. This is Whitepages' own documented alternative 'if you would prefer not to provide a phone number'. Expect a reply within ~2 business days; they may ask identity-verification questions (name, email) -- answer with least-disclosure, never an ID number.",
|
||||
"Include in the email: the listing URL(s), the subject's full name as listed, and the request to (a) remove the listing(s), (b) opt out of sale/sharing, and (c) delete personal data (CCPA 1798.105 for CA residents; they honor requests from other states per their consumer-rights page).",
|
||||
"'Once a listing is removed, all known connected listings are also removed and the requester's information will not be sold by Whitepages in any capacity' -- one request covers connected listings; still re-scan afterward, and check Whitepages Premium + 411.com separately.",
|
||||
"Alternative 1: webform at whitepagesprivacy.zendesk.com/hc/en-us/requests/new (same ~2-day handling; good fallback if the mailbox bounces).",
|
||||
"Alternative 2 (only with an operator on the phone): the automated tool at whitepages.com/suppression_requests -- paste the listing URL, provide a phone number, answer the automated voice call and enter the 4-digit code. Fastest (est 1 day) but NOT autonomous.",
|
||||
"Opt-out requests may take up to 15 days; verify with a re-scan before recording confirmed_removed."
|
||||
],
|
||||
"notes": "Email/webform lane verified 2026-07-01: privacyrequest@whitepages.com or the Zendesk form replace the phone-callback tool ('if you would prefer not to provide a phone number... submit a request to a customer service agent'). Authorized agents: written signed permission required. General privacy contact: support@whitepages.com.",
|
||||
"quirks": [
|
||||
"The automated suppression tool (whitepages.com/suppression_requests) REQUIRES a phone number + automated voice call with a 4-digit code + agreeing to their ToS -- that lane is phone_callback/T2. The email/webform lane exists precisely to avoid it; prefer email for autonomy.",
|
||||
"Deletion exceptions they state: public records (property, court), fraud-prevention data, active-subscription data, legal holds. 'Hidden from search' vs 'deleted' distinction applies -- their opt-out removes the listing; the CCPA deletion covers non-public account data.",
|
||||
"CCPA opt-out requests: up to 15 days to process. Right-to-know/access requests also via privacyrequest@whitepages.com or the Zendesk form."
|
||||
],
|
||||
"est_processing_days": 15,
|
||||
"reappearance_risk": "medium"
|
||||
},
|
||||
"last_verified": "2026-07-01",
|
||||
"source": "BADBOOL"
|
||||
}
|
||||
27
optional-skills/security/unbroker/references/legal/ccpa.md
Normal file
27
optional-skills/security/unbroker/references/legal/ccpa.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
# CCPA / CPRA (California)
|
||||
|
||||
Use for California residents (`residency_jurisdiction` starts with `US-CA`) and, in practice, many US
|
||||
brokers that honor CCPA-style requests nationwide.
|
||||
|
||||
## Rights invoked
|
||||
|
||||
- **Delete** personal information (Cal. Civ. Code 1798.105).
|
||||
- **Opt out** of sale/sharing of personal information (1798.120).
|
||||
|
||||
## Request content
|
||||
|
||||
Render with `legal.render_request("ccpa", broker, fields)` -> `templates/emails/ccpa-deletion.txt`.
|
||||
Include only: full legal name, the contact email for correspondence, and the confirmed listing
|
||||
URL(s). Do **not** include SSN or government IDs.
|
||||
|
||||
## Authorized agent
|
||||
|
||||
When acting for another consenting subject, use `render_request("ccpa_agent", ...)`
|
||||
(`templates/emails/ccpa-authorized-agent.txt`) and attach the authorization artifact recorded in the
|
||||
dossier (`consent.authorization_artifact`). The broker may separately verify the consumer's identity.
|
||||
|
||||
## Notes
|
||||
|
||||
- Brokers must respond within 45 days (extendable). Track as `awaiting_processing` until confirmed.
|
||||
- "Hidden from free search" is not deletion - verify the record is actually gone before
|
||||
`confirmed_removed`.
|
||||
34
optional-skills/security/unbroker/references/legal/drop.md
Normal file
34
optional-skills/security/unbroker/references/legal/drop.md
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# California DROP portal (highest-leverage lever)
|
||||
|
||||
The California **Delete Request and Opt-out Platform** (`privacy.ca.gov/drop`) lets a California
|
||||
resident demand deletion from **every registered data broker** with a single verified request, for
|
||||
free. DROP is **live** (as of 2026); registered brokers must begin processing requests on
|
||||
**2026-08-01**. The registered universe is the **California Data Broker Registry** (~545 brokers in
|
||||
2025), which this skill ingests as its own coverage lane (`pdd.py registry`); one DROP request covers
|
||||
all of them, which is how this skill reaches (and exceeds) the breadth of commercial services.
|
||||
|
||||
## When to use
|
||||
|
||||
For any subject with `residency_jurisdiction` starting `US-CA`, sequence DROP **first**: `pdd.py next`
|
||||
surfaces a single `drop_submit` action covering the whole registry. Then handle the individual
|
||||
people-search sites (which are also worked directly because they hold free, indexed listings). After
|
||||
filing, run `pdd.py drop <subject> --filed` so the loop stops re-surfacing it. For non-CA subjects
|
||||
DROP does not apply; cover the registry brokers with targeted CCPA/GDPR deletion emails
|
||||
(`pdd.py registry --search`, then `pdd.py send-email`).
|
||||
|
||||
## Flow (agent-assisted, mostly human verification)
|
||||
|
||||
1. The operator creates/verifies a DROP account (identity verification is required by the state; this
|
||||
is a human step - `human_task_queued`).
|
||||
2. Submit one deletion request covering all registered brokers.
|
||||
3. Record a single ledger case `case_<subject>_drop` to track it; mark `submitted` ->
|
||||
`awaiting_processing`. Registered brokers must process deletions on the state's schedule.
|
||||
4. After the DROP cycle, re-scan the people-search long tail and only act on sites still showing data.
|
||||
|
||||
## Caveats
|
||||
|
||||
- DROP covers **registered data brokers**, not every people-search site. Keep doing the individual
|
||||
opt-outs for non-registered sites.
|
||||
- Identity verification means parts of this cannot (and should not) be fully automated.
|
||||
- FCRA-regulated brokers (flagged in the registry, `optout.fcra`) hold consumer-report data with
|
||||
separate rules; deletion may be limited and a dispute or security-freeze may apply instead.
|
||||
20
optional-skills/security/unbroker/references/legal/gdpr.md
Normal file
20
optional-skills/security/unbroker/references/legal/gdpr.md
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
# GDPR / UK-GDPR (roadmap - Phase 3)
|
||||
|
||||
For EU/UK subjects. Not part of the P0 US-first scope; templates and routing land in Phase 3.
|
||||
|
||||
## Rights invoked
|
||||
|
||||
- **Erasure** ("right to be forgotten") - Article 17.
|
||||
- **Object** to processing - Article 21.
|
||||
|
||||
## Request content
|
||||
|
||||
Render with `legal.render_request("gdpr", broker, fields)` ->
|
||||
`templates/emails/gdpr-erasure.txt`. Address the controller's privacy/DPO contact. Include the data
|
||||
subject's name, the contact email, and the listing URL(s); cite Article 17.
|
||||
|
||||
## Notes
|
||||
|
||||
- Controllers must respond within one month (Article 12(3)).
|
||||
- EU-specific brokers and portals (e.g. Acxiom's EU consumer portals) are added in Phase 3 with
|
||||
`jurisdictions: ["EU"]` records and residency-aware routing.
|
||||
235
optional-skills/security/unbroker/references/methods.md
Normal file
235
optional-skills/security/unbroker/references/methods.md
Normal file
|
|
@ -0,0 +1,235 @@
|
|||
# Opt-out method playbooks
|
||||
|
||||
How the agent executes each broker `optout.method` using native Hermes tools. Always obey the
|
||||
**verify-before-disclose** rule: confirm a real listing exists, then submit only the fields the broker
|
||||
requires (`pdd.py plan` lists them per broker).
|
||||
|
||||
**Autonomy:** `pdd.py next <subject>` sequences all of this - it decides which method applies, orders
|
||||
parents first, and routes human-only work to the digest. In `autonomy=full` (default), execute its
|
||||
actions without pausing per submission; the consent recorded at intake is the authorization. These
|
||||
playbooks are the HOW for each action type.
|
||||
|
||||
## Scan ladder (all methods)
|
||||
|
||||
Confirm exposure before acting, cheapest first. Run **every** `search_vectors` entry from `pdd.py
|
||||
plan` (each name x location, phone, email, and address the broker's `search.by` supports) - different
|
||||
vectors surface different listings for the same person; dedupe found URLs.
|
||||
|
||||
1. `web_extract` on the broker `search.url` (fast HTML -> markdown). Look for `search.match_signal`.
|
||||
Build per-vector URLs from `search.url_patterns` and heed `search.url_format_quirks` (see below).
|
||||
1b. **`site:` search-engine probe (cheap, do it early and in parallel).** `web_search` with
|
||||
`site:<broker-domain> "First Last"` (add a city/ZIP or a unique phone/address to cut namesake
|
||||
noise) often returns the **exact profile-slug URL** in one shot - which both confirms the listing
|
||||
exists AND hands you the opaque `/find/person/<id>` or `/p/<slug>` URL you'd otherwise have to
|
||||
derive. Two big wins seen in the field: (a) it disambiguates namesakes fast - the SERP snippet
|
||||
shows age/city so you can tell the subject from a same-name relative before fetching anything; and
|
||||
(b) a broad `"First Last" <ZIP OR unique-address>` search (no `site:`) surfaces **brokers not yet in
|
||||
your DB** (e.g. information.com, peoplefinders.com) - record those as bonus exposures. Note: empty
|
||||
`site:` results are INCONCLUSIVE (many broker pages aren't indexed / are `noindex`), not `not_found`.
|
||||
2. If the page is JS-rendered or returns nothing useful, `browser_navigate` + `browser_snapshot`
|
||||
(and `browser_type`/`browser_click` to run the site's search box).
|
||||
3. If blocked by stealth/Cloudflare, use the `scrapling` skill via `terminal`. **If the broker record
|
||||
has `search.antibot` set (e.g. `datadome`), results are behind a device-check CAPTCHA**: a
|
||||
cloud/stealth browser (Browserbase) or `scrapling` may get through; if none is available, do **not**
|
||||
burn attempts - `pdd.py record <subject> <broker> blocked` and move on (a re-scan with a stealth
|
||||
backend can pick it up later).
|
||||
3b. **Operator-browser path (the reliable unblock for anti-bot sites).** Cloudflare/DataDome key on
|
||||
datacenter IPs + headless fingerprints, so `web_extract`, the proxyless agent browser, and even a
|
||||
cloud browser often fail - but the **operator's own everyday browser (residential IP, real
|
||||
fingerprint) sails straight through**. For any `blocked` site, hand the operator a paste-ready
|
||||
search URL (built from `search.url_patterns`), give them the identity anchors to judge by (current
|
||||
+ prior addresses, age, a distinguishing detail) and the namesake/relative watch-list, and ask for
|
||||
the verdict or a screenshot (the agent can read screenshots). This is a **first-class scan path, not
|
||||
a fallback** - treat the operator's live check as authoritative and record the real verdict
|
||||
(`found` / `not_found` / `indirect_exposure`), citing `scanned_via: operator_browser`. Same for
|
||||
opt-out forms the agent's browser can't reach: guide the operator field-by-field (least-disclosure),
|
||||
pausing before submit. (This is exactly why the same trick clears email-verification links the agent
|
||||
can't open - see the Verification loop.)
|
||||
4. Capture evidence: save listing URLs and a `browser` screenshot into the subject's `evidence/` dir,
|
||||
then `pdd.py record <subject> <broker> found --found true --evidence '{"listing_urls":[...]}'`.
|
||||
|
||||
If a listing genuinely does not exist: `pdd.py record <subject> <broker> not_found` and move on.
|
||||
|
||||
### A 404 (or empty body) is INCONCLUSIVE, not "not_found"
|
||||
|
||||
A constructed search URL that 404s almost always means the **URL pattern is wrong**, not that the
|
||||
person is absent. Never record `not_found` off a 404. Instead:
|
||||
1. Re-check the broker's `search.url_patterns` / `url_format_quirks` and rebuild the URL.
|
||||
2. Fall back to the **on-site search box**: `browser_navigate` to the search page, `browser_type`
|
||||
the raw query, `browser_click` Search, then read the **canonical result URL** the site lands on.
|
||||
3. Only after the site's own search returns an empty result set do you record `not_found`.
|
||||
4. If a pattern was wrong, fix it in `references/brokers/<id>.json` (`url_patterns` +
|
||||
`url_format_quirks`) so the next run is correct - see the rule below.
|
||||
|
||||
### Log URL/format quirks for every site you scrape
|
||||
|
||||
Whenever you discover how a broker's URLs are actually shaped (path layout, hyphen-vs-slash joins,
|
||||
whether ZIP is required, abbreviation handling, query-param search, anti-bot gating), record it in
|
||||
that broker's `references/brokers/<id>.json` under `search.url_patterns` (the templates) and
|
||||
`search.url_format_quirks` (the gotchas, including which forms 404). Bump `last_verified`. This makes
|
||||
the deterministic URL path reliable across runs and subjects instead of rediscovered each time. If the
|
||||
opt-out form's real requirements differ from the record (extra required fields, a CAPTCHA, an account),
|
||||
fix `optout.requires` / `optout.inputs` / `optout.tier` too - those drive tier selection and
|
||||
least-disclosure. Log opt-out mechanics gotchas (a broker that needs a profile URL but doesn't expose
|
||||
one for the subject, an email-only fallback, an authorized-agent toggle) in `optout.quirks` - the
|
||||
planner surfaces these as `optout_quirks` per broker. Example: Radaris sometimes shows the subject only
|
||||
as a static address-table row with no "View Profile" link, so `/control-privacy` (which needs a profile
|
||||
URL) can't be used - fall back to `optout.email` rather than submitting a namesake's URL.
|
||||
|
||||
### Distinguish the subject from namesakes and relatives
|
||||
|
||||
People-search sites are dense with namesakes and family clusters. Before recording `found`, confirm the
|
||||
record is the **subject themselves** (corroborate via DOB, a known current/prior address, or the
|
||||
identifier you searched). Two non-removable patterns to record as evidence but NOT as the subject's own
|
||||
listing:
|
||||
- **Namesake:** same name, different person (different DOB/location with no overlap). Not the subject.
|
||||
- **Relative record:** the listing is about a *different* person (a relative) and merely *names* the
|
||||
subject in a "Family" field, or carries the subject's email/phone as a secondary datum. This is a
|
||||
third party's record - the consent gate correctly blocks acting on it. See "Indirect exposure" in
|
||||
the web_form section for what the subject *can* still request.
|
||||
|
||||
## web_form
|
||||
|
||||
1. `browser_navigate` to `optout.url`; `browser_snapshot` to read the form.
|
||||
2. Fill only the planned `disclosure_fields` with `browser_type`/`browser_click`; for `profile_url`,
|
||||
paste the confirmed listing URL from evidence.
|
||||
3. Submit; `browser_snapshot` to confirm the success state; screenshot to `evidence/`.
|
||||
4. `pdd.py record <subject> <broker> submitted --disclosed <field> --disclosed <field> --channel web_form`.
|
||||
5. If the broker requires email verification, follow **Verification loop** below.
|
||||
|
||||
### Indirect exposure (named as a relative / your email on someone else's record)
|
||||
|
||||
You asked the right question: if a broker lists a *relative* and names you in their "Family" field, or
|
||||
shows **your** email/phone on **their** record, that IS personal information about you - even though the
|
||||
record's primary subject is a third party. Resolve it in two distinct lanes:
|
||||
|
||||
- **The self-service opt-out form does NOT cover this.** That form removes a record whose *primary
|
||||
subject* is you. It has no notion of "scrub my identifiers from this other person's record," and
|
||||
submitting it with the relative's address to force a match would be (a) disclosing data the listing
|
||||
doesn't tie to you and (b) acting on a third party's record. Don't. The consent gate exists to stop
|
||||
exactly that.
|
||||
- **What you CAN do - a targeted "delete my personal information" request (CCPA 1798.105 / GDPR Art.17).**
|
||||
These rights attach to *your* personal information *wherever the business holds it*, including as a
|
||||
data point on another person's profile. So the subject may email the broker's privacy address and
|
||||
request suppression of **their own specific identifiers** (this email address, this phone number, my
|
||||
name in family/relative associations), citing the relative listings as the locations. This is a
|
||||
narrower request than a full opt-out and does not require the relative's consent - you are only asking
|
||||
them to delete data about *you*. Use `render-email` with the `ccpa`/`gdpr` template, list only the
|
||||
subject's own identifiers + the URLs where they appear, and record it as a normal `submitted` →
|
||||
`awaiting_processing` email case. Verify by re-scanning those identifier vectors (email/phone) after
|
||||
the statutory window - `confirmed_removed` only when the subject's identifier no longer appears.
|
||||
- **Caveat:** the broker may decline to alter a third party's record beyond removing your specific
|
||||
identifiers, and "your name in a family graph" can be derived from public records they'll re-list.
|
||||
Note residual exposure in the report rather than marking a clean removal. (Operational guidance, not
|
||||
legal advice.)
|
||||
|
||||
## email
|
||||
|
||||
`pdd.py send-email <subject> <broker> --listing <url> [--kind ccpa|gdpr|ccpa_indirect]` always does
|
||||
the deterministic parts (recipient locked to an address the broker record declares, refusing anything
|
||||
else; `--listing` mandatory; records `submitted`, logs disclosure, stamps `next_recheck_at`). How it
|
||||
actually sends depends on `email_mode`:
|
||||
|
||||
1. **browser mode (no password, autonomous):** the command returns a recipient-locked `compose`
|
||||
payload (`to`/`subject`/`body`). Compose a NEW message in the operator's **logged-in webmail** via
|
||||
`browser_*` (paste `compose.body` exactly, disclosing nothing beyond it) and send. No credentials
|
||||
stored. Requires the inbox signed in in the browser Hermes uses.
|
||||
2. **programmatic mode (SMTP creds):** the command SMTP-sends it directly, no human.
|
||||
3. **draft_only fallback:** `pdd.py render-email <subject> <broker> --listing <url>`; a digest entry
|
||||
tells the operator to send it, and the agent records `submitted --channel email` afterward.
|
||||
|
||||
Then follow the **Verification loop** if the broker emails a confirmation link.
|
||||
|
||||
## Verification loop (email_verification brokers)
|
||||
|
||||
- **browser mode (autonomous, no password):** open the broker's confirmation email in the operator's
|
||||
webmail (`browser_*`), then `pdd.py verify-link <subject> <broker> --text '<email body>'` returns
|
||||
the anti-phishing-scored link. `browser_navigate` it **in the same browser** (several brokers, e.g.
|
||||
PeopleConnect, bind the session to the browser that opens the link), finish the flow, record
|
||||
`awaiting_processing`.
|
||||
- **programmatic mode (IMAP):** `pdd.py poll-verification <subject>` polls IMAP for every in-flight
|
||||
case, extracts the link (anti-phishing scored: only opt-out-looking links on the broker's own
|
||||
domains), and auto-advances `submitted → verification_pending`. Then `browser_navigate` the link in
|
||||
the agent's own browser, finish the flow, record `awaiting_processing`.
|
||||
- **draft_only:** the digest tells the operator to click the link in the subject's inbox; the agent
|
||||
records `awaiting_processing` on their word.
|
||||
- Either way, the due queue (`pdd.py due`) brings the case back after the broker's processing window
|
||||
for the verifying re-scan; only that re-scan justifies `confirmed_removed`.
|
||||
|
||||
## phone_callback (e.g. Whitepages)
|
||||
|
||||
Submit the web form, then the site places an automated call with a numeric code. If the operator is
|
||||
available to read the code, capture it and complete the form (T2). Otherwise queue a human task.
|
||||
|
||||
## phone (voice menu) / fax / mail / gov_id -> human task (T3)
|
||||
|
||||
Do **not** attempt to automate. Create a `todo` task and `pdd.py record <subject> <broker>
|
||||
human_task_queued` with exact instructions and an explicit **withhold** list (never SSN; never a
|
||||
driver's-license number unless the subject chooses to and crosses out the ID number). Capture the
|
||||
confirmation reference back into the ledger when the operator completes it.
|
||||
|
||||
## captcha
|
||||
|
||||
**Default: soft/managed CAPTCHAs clear automatically.** The recommended baseline backend is the
|
||||
Browserbase cloud browser (`setup --auto` selects it when `BROWSERBASE_API_KEY` is set). Being a
|
||||
real browser on a residential IP, it passes managed challenges - Cloudflare Turnstile, hCaptcha /
|
||||
reCAPTCHA checkbox - as normal operation, so those brokers stay T1 and you just proceed. This is
|
||||
**not** CAPTCHA solving: no solver service, no fingerprint spoofing.
|
||||
|
||||
Only a **hard** challenge the browser genuinely can't pass (interactive image grids, behavioral
|
||||
scoring that flags the session) becomes a fallback: `record ... blocked` and requeue it for the
|
||||
stealth/operator-browser pass (`methods.md` → scan ladder 3b - the operator's own residential
|
||||
browser is the reliable unblock). Without a cloud browser configured, soft-CAPTCHA brokers drop to
|
||||
T2 and become human tasks. **Never use a third-party CAPTCHA-defeating service.**
|
||||
|
||||
## Ownership clusters - DO PARENTS FIRST (playbooks live in the broker records)
|
||||
|
||||
Many brokers are resold shells of a few parents, so **one parent removal clears a whole cluster of
|
||||
children** (see `owns` in each record). In Phase 2 you MUST work the cluster **parents first**, then
|
||||
the standalone listings - doing a child before its parent wastes a submission the parent would have
|
||||
covered. `pdd.py plan <subject> --batch` **orders the `found` group parents-first** and emits a
|
||||
`parent_playbook` whose `steps` come verbatim from each record's **`optout.playbook`** - the single
|
||||
source of truth, field-verified, updated as live runs discover mechanics. What follows is the
|
||||
operating doctrine; the exact steps are in `references/brokers/<id>.json`.
|
||||
|
||||
**Deletion beats suppression, email lanes beat forms.** Each parent record carries a structured
|
||||
`optout.deletion` lane (`via: in_flow | email | email_followup`, plus the privacy address). The
|
||||
autopilot routes accordingly:
|
||||
|
||||
- **`in_flow`** (PeopleConnect): the deletion control lives INSIDE the web flow - complete the flow
|
||||
and use the **Right to Delete / DELETE MY USER DATA** control, never just the suppression step.
|
||||
- **`via: email`** (Whitepages): the fully-autonomous lane - `send-email` the request (residency-picked
|
||||
kind: CCPA for US-CA, GDPR for EU/UK, generic otherwise), then `poll-verification` for their reply
|
||||
and answer identity questions with least-disclosure. This is also the **rescue lane**: any broker
|
||||
whose form demands a phone-callback/gov-ID/account but that declares a deletion email gets routed
|
||||
here instead of the human digest.
|
||||
- **`email_followup`** (BeenVerified, Spokeo): the opt-out form is the fast primary (it clears the
|
||||
listing), and the playbook then sends a right-to-delete email for full erasure beyond suppression.
|
||||
|
||||
Verified parent facts (live-checked 2026-07-01; details + steps in the records):
|
||||
|
||||
- **Intelius/PeopleConnect** (~15+ sites in one flow): portal entry asks only email + consent →
|
||||
verify link is **session-bound to the browser that opens it** → guided-mode → **DELETE MY USER
|
||||
DATA** at the bottom (suppression alone re-lists). Fallback `privacy@peopleconnect.us` - their own
|
||||
published metrics: 33.5k deletion requests, median response < 1 day.
|
||||
- **Whitepages**: `privacyrequest@whitepages.com` (or the Zendesk form) handles removal + CCPA
|
||||
deletion **without the phone-callback tool** - that phone call is only required by the automated
|
||||
tool. One removal also drops "all known connected listings". ≤15 days; check 411.com + Premium.
|
||||
- **BeenVerified**: opt-out tool (footer "Do Not Sell" link → `/svc/optout/search/optouts`) + email
|
||||
verification; one opt-out per email address. Then `privacy@beenverified.com` deletion follow-up -
|
||||
controller is The Lifetime Value Co., so name their sister properties (NeighborWho, Ownerly,
|
||||
NumberGuru, Bumper) in the same request, and verify each separately.
|
||||
- **Spokeo**: form takes ONE listing URL at a time and **each listing must be opted out
|
||||
individually** - collect every listing URL from all search vectors first, then submit one opt-out
|
||||
per URL. 24-48h processing. `privacy@spokeo.com` for full deletion beyond free-search suppression.
|
||||
|
||||
After each parent removal is confirmed, **re-scan its children** before submitting anything for them -
|
||||
usually they drop out and need no separate opt-out.
|
||||
|
||||
### Any other parent
|
||||
A parent without a hand-verified `optout.playbook` gets synthesised steps from its structured record
|
||||
(URL/email, `requires` flags, deletion lane, notes/quirks). Follow those, and **write what you learn
|
||||
back into `references/brokers/<id>.json`** (`optout.playbook`, `optout.deletion`, `quirks`,
|
||||
`last_verified`) so the next run is exact - that file, not this one, is where per-broker knowledge
|
||||
accrues.
|
||||
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
# Case state machine
|
||||
|
||||
One case = one (subject x broker). `pdd.py record` validates every transition against this table and
|
||||
appends it to `audit.jsonl`. Authoritative definition lives in `scripts/ledger.py`.
|
||||
|
||||
## States
|
||||
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| `new` | Case created, nothing done |
|
||||
| `searching` | Scan in progress |
|
||||
| `not_found` | Subject not listed (will be re-checked next cycle) |
|
||||
| `found` | Listing confirmed; action needed |
|
||||
| `indirect_exposure` | Subject's PII (email/phone/name) appears on a **third party's** record (e.g. named in a relative's "Family" field). Not removable via self-service opt-out; needs a targeted CCPA/GDPR delete-my-PII request |
|
||||
| `action_selected` | Tier/method chosen |
|
||||
| `submitted` | Opt-out submitted |
|
||||
| `verification_pending` | Awaiting email/callback verification |
|
||||
| `awaiting_processing` | Submitted, no verification needed; broker processing |
|
||||
| `confirmed_removed` | Verified gone |
|
||||
| `reappeared` | Was removed, now listed again |
|
||||
| `human_task_queued` | Needs an operator step (captcha/ID/phone/fax/mail) |
|
||||
| `blocked` | Broker dead / mechanics broken -> flag for DB re-verification |
|
||||
|
||||
## Allowed transitions
|
||||
|
||||
```
|
||||
new -> searching | found | not_found | indirect_exposure | blocked
|
||||
searching -> not_found | found | indirect_exposure | blocked
|
||||
not_found -> searching | found | indirect_exposure | blocked
|
||||
found -> action_selected | submitted | human_task_queued | indirect_exposure | blocked
|
||||
indirect_exposure -> submitted | human_task_queued | not_found | found | blocked
|
||||
action_selected -> submitted | human_task_queued | blocked
|
||||
submitted -> verification_pending | awaiting_processing | human_task_queued | blocked
|
||||
verification_pending -> confirmed_removed | human_task_queued | blocked
|
||||
awaiting_processing -> confirmed_removed | human_task_queued | blocked
|
||||
confirmed_removed -> reappeared | confirmed_removed (recheck refreshes the date)
|
||||
reappeared -> found | indirect_exposure
|
||||
human_task_queued -> found | indirect_exposure | action_selected | submitted | verification_pending
|
||||
| awaiting_processing | confirmed_removed | blocked
|
||||
blocked -> searching | found | not_found | indirect_exposure | action_selected
|
||||
```
|
||||
|
||||
A transition to the same state is always allowed (idempotent field updates).
|
||||
396
optional-skills/security/unbroker/scripts/autopilot.py
Normal file
396
optional-skills/security/unbroker/scripts/autopilot.py
Normal file
|
|
@ -0,0 +1,396 @@
|
|||
"""Autonomous action queue: what should the agent do RIGHT NOW for this subject?
|
||||
|
||||
`next_actions` turns (dossier, broker DB, config, ledger) into an ordered queue of
|
||||
concrete agent actions plus a human digest. The agent's whole run becomes a loop:
|
||||
|
||||
while True:
|
||||
q = pdd.py next <subject>
|
||||
if not q["actions"]: break
|
||||
execute each action, record outcomes
|
||||
present q["human_digest"] once; schedule cron at q["next_wake_at"]
|
||||
|
||||
Policy (cfg["autonomy"]):
|
||||
full - intake consent is standing authorization; T0-T2 agent actions are
|
||||
executed without pausing. Humans appear only in the digest.
|
||||
assisted - same queue, but every submission action carries confirm_first=True.
|
||||
|
||||
The queue is deterministic and side-effect free: it never mutates the ledger, it
|
||||
only reads. Executing + recording stays with the agent (and the record command).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import brokers as brokers_mod
|
||||
import emailer
|
||||
import ledger as ledger_mod
|
||||
import paths
|
||||
import registry
|
||||
import tiers
|
||||
|
||||
CACHE_STALE_DAYS = 7 # refresh the live broker list after this
|
||||
FANOUT_THRESHOLD = 8 # above this many unscanned brokers, use delegate_task fan-out
|
||||
|
||||
# States with nothing left to do (absent a due recheck).
|
||||
_TERMINAL = {"not_found", "confirmed_removed"}
|
||||
_IN_FLIGHT = {"submitted", "verification_pending", "awaiting_processing"}
|
||||
|
||||
|
||||
def cache_age_days(now: float | None = None) -> float | None:
|
||||
"""Age of the live BADBOOL cache in days, or None if never pulled."""
|
||||
p: Path = paths.brokers_cache_path()
|
||||
if not p.exists():
|
||||
return None
|
||||
now = now if now is not None else _dt.datetime.now().timestamp()
|
||||
return max(0.0, (now - p.stat().st_mtime) / 86400.0)
|
||||
|
||||
|
||||
def _now_iso() -> str:
|
||||
return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def _min_future_recheck(ledger: dict, at: str) -> str | None:
|
||||
future = [c.get("next_recheck_at") for c in ledger.values()
|
||||
if c.get("next_recheck_at") and c["next_recheck_at"] > at]
|
||||
return min(future) if future else None
|
||||
|
||||
|
||||
def _digest(broker_row: dict, reason: str, steps: list[str], prep: list[str] | None = None) -> dict:
|
||||
return {
|
||||
"broker_id": broker_row.get("broker_id"),
|
||||
"broker_name": broker_row.get("broker_name"),
|
||||
"reason": reason,
|
||||
"agent_prep": prep or [], # commands the agent runs BEFORE handing this to the human
|
||||
"steps": steps, # what the human actually does
|
||||
"withhold": ["SSN", "full driver's-license / passport numbers"],
|
||||
}
|
||||
|
||||
|
||||
def request_kind(dossier: dict, allowed: list[str] | None = None) -> str:
|
||||
"""Pick the honest legal basis for a deletion request from the subject's residency.
|
||||
|
||||
ccpa only for California residents, gdpr only for EU/UK residents, generic otherwise.
|
||||
`allowed` (from the broker's deletion.kinds) can restrict DOWN to generic but never
|
||||
upgrades to a law the subject can't truthfully claim.
|
||||
"""
|
||||
res = (dossier.get("residency_jurisdiction") or "US").upper()
|
||||
if res.startswith("US-CA"):
|
||||
kind = "ccpa"
|
||||
elif res.startswith(("EU", "UK", "GB")):
|
||||
kind = "gdpr"
|
||||
else:
|
||||
kind = "generic"
|
||||
if allowed and kind not in allowed and "generic" in allowed:
|
||||
kind = "generic"
|
||||
return kind
|
||||
|
||||
|
||||
_HUMAN_GATES = ("gov_id", "fax", "mail", "phone_voice", "phone_callback", "account")
|
||||
|
||||
|
||||
def _email_lane(row: dict) -> tuple[str | None, str]:
|
||||
"""(address, why) for the autonomous email lane of this broker, if one exists.
|
||||
|
||||
Lane rules:
|
||||
1. the broker's primary opt-out method IS email;
|
||||
2. the record marks its deletion lane email-preferred (deletion.via == "email");
|
||||
3. RESCUE: the primary flow is human-gated (gov ID / fax / phone / account) but a
|
||||
right-to-delete email exists - the email lane restores full autonomy (this is the
|
||||
verified Whitepages pattern: privacyrequest@ accepts requests precisely so people
|
||||
don't have to do the phone-callback tool).
|
||||
"""
|
||||
deletion = row.get("deletion") or {}
|
||||
req = row.get("optout_requires") or {}
|
||||
if row.get("method") == "email":
|
||||
addr = row.get("optout_email") or deletion.get("email")
|
||||
return (addr, "primary opt-out method is email") if addr else (None, "")
|
||||
if deletion.get("via") == "email" and deletion.get("email"):
|
||||
return deletion["email"], "record prefers the right-to-delete email lane"
|
||||
if (row.get("tier") == "T3" or any(req.get(k) for k in _HUMAN_GATES)) and deletion.get("email"):
|
||||
return deletion["email"], "rescue: primary flow is human-gated; deletion email restores autonomy"
|
||||
return None, ""
|
||||
|
||||
|
||||
def _optout_action(row: dict, playbook: dict[str, dict], subject_id: str, dossier: dict,
|
||||
email_mode: str, smtp_ok: bool, confirm_first: bool) -> tuple[dict | None, dict | None]:
|
||||
"""Map one actionable `found` row to (agent_action, human_digest_entry).
|
||||
|
||||
Routing order maximizes autonomy: (1) the email lane (primary email method, preferred
|
||||
right-to-delete email, or rescue from a human-gated form) beats everything when SMTP is
|
||||
up; (2) genuinely human-only flows go to the digest; (3) web forms are driven with the
|
||||
record's own field-verified playbook steps.
|
||||
"""
|
||||
bid = row["broker_id"]
|
||||
req = row.get("optout_requires") or {}
|
||||
tier = row.get("tier")
|
||||
deletion = row.get("deletion") or {}
|
||||
|
||||
# 1) The autonomous EMAIL LANE (right-to-delete by email + confirm the reply).
|
||||
# Autonomous when SMTP is configured (programmatic/alias) OR in browser mode (agent sends via
|
||||
# the operator's logged-in webmail; no password needed).
|
||||
email_addr, lane_why = _email_lane(row)
|
||||
can_email = (email_mode in ("programmatic", "alias") and smtp_ok) or email_mode == "browser"
|
||||
if email_addr and can_email:
|
||||
kind = request_kind(dossier, deletion.get("kinds"))
|
||||
via = "browser" if email_mode == "browser" else "smtp"
|
||||
then = ("send-email records it + returns a recipient-locked payload; compose and send it in "
|
||||
"the operator's webmail via browser_*, then `verify-link` on the reply and open the link"
|
||||
if via == "browser" else
|
||||
"state auto-records as submitted; poll-verification picks up their verification reply, "
|
||||
"open its link, then record")
|
||||
return {
|
||||
"type": "optout_email_send",
|
||||
"broker_id": bid, "broker_name": row.get("broker_name"), "tier": tier,
|
||||
"confirm_first": confirm_first, "send_via": via,
|
||||
"to": email_addr, "kind": kind, "why": lane_why,
|
||||
"command": f"python3 scripts/pdd.py send-email {subject_id} {bid} --kind {kind} "
|
||||
f"--to {email_addr} --listing <confirmed-url>",
|
||||
"then": then,
|
||||
}, None
|
||||
if row.get("method") == "email":
|
||||
return None, _digest(row, "email opt-out (draft mode: a human must hit send)",
|
||||
["Send the rendered draft from your own mail client",
|
||||
f"Then: python3 scripts/pdd.py record {subject_id} {bid} submitted "
|
||||
f"--disclosed contact_email --channel email"],
|
||||
prep=[f"python3 scripts/pdd.py render-email {subject_id} {bid} --listing <confirmed-url>"])
|
||||
|
||||
# 2) Genuinely human-only work goes to the digest (no email lane could rescue it).
|
||||
if tier == "T3":
|
||||
return None, _digest(row, "human-only opt-out (gov ID / fax / mail / voice phone)",
|
||||
[f"Follow the broker's process at {row.get('optout_url') or row.get('optout_email')}",
|
||||
"Provide only the fields the listing already shows; cross out ID numbers on any document"])
|
||||
if req.get("phone_callback"):
|
||||
return None, _digest(row, "phone-callback verification (operator must be on the phone)",
|
||||
[f"Open {row.get('optout_url')} and submit with only the planned fields",
|
||||
"Answer the automated call and enter the 4-digit code to finish"],
|
||||
prep=[f"python3 scripts/pdd.py plan {subject_id} --batch # confirm fields first"])
|
||||
if req.get("account"):
|
||||
return None, _digest(row, "requires creating/holding an account with the broker",
|
||||
[f"Create/log in at {row.get('optout_url')} and submit the opt-out",
|
||||
"Use the subject's contact email; no extra PII beyond the planned fields"])
|
||||
|
||||
# 3) web_form: drive the browser with the record's own playbook steps.
|
||||
steps = (playbook.get(bid) or {}).get("steps") or list(row.get("optout_playbook") or []) \
|
||||
or tiers.synthesize_steps(row)
|
||||
action = {
|
||||
"type": "optout_web_form",
|
||||
"broker_id": bid, "broker_name": row.get("broker_name"), "tier": tier,
|
||||
"confirm_first": confirm_first,
|
||||
"optout_url": row.get("optout_url"),
|
||||
"clears_children": row.get("clears_children") or [],
|
||||
"steps": steps,
|
||||
"after": f"python3 scripts/pdd.py record {subject_id} {bid} submitted "
|
||||
f"--disclosed <field>... --channel web_form",
|
||||
}
|
||||
if deletion:
|
||||
action["prefer_deletion"] = ("this record has a right-to-delete lane -- complete the DELETION "
|
||||
"flow, not just suppression" + (f" ({deletion.get('notes')})"
|
||||
if deletion.get("notes") else ""))
|
||||
if req.get("captcha"):
|
||||
action["note"] = ("CAPTCHA-gated: attempt with the configured browser backend once; if it "
|
||||
"does not clear, record blocked (do NOT retry-loop or bypass)")
|
||||
return action, None
|
||||
|
||||
|
||||
def next_actions(dossier: dict, brokers_list: list[dict], cfg: dict,
|
||||
ledger: dict | None = None, env: dict | None = None) -> dict:
|
||||
env = os.environ if env is None else env
|
||||
ledger = ledger or {}
|
||||
subject_id = dossier.get("subject_id", "")
|
||||
autonomy = cfg.get("autonomy", "full")
|
||||
confirm_first = autonomy == "assisted"
|
||||
email_mode = cfg.get("email_mode", "draft_only")
|
||||
mail = emailer.available(env)
|
||||
at = _now_iso()
|
||||
|
||||
batch = tiers.batch_plan(dossier, brokers_list, cfg, ledger,
|
||||
browser_clears_captcha=cfg.get("browser_backend") == "browserbase"
|
||||
or bool(env.get("BROWSERBASE_API_KEY")))
|
||||
groups = batch["groups"]
|
||||
playbook = {p["broker_id"]: p for p in batch.get("parent_playbook") or []}
|
||||
by_id = {b.get("id"): b for b in brokers_list}
|
||||
|
||||
actions: list[dict] = []
|
||||
digest: list[dict] = []
|
||||
|
||||
# 0) keep the broker DB fresh (autonomously)
|
||||
age = cache_age_days()
|
||||
if age is None or age > CACHE_STALE_DAYS:
|
||||
actions.append({
|
||||
"type": "refresh_brokers",
|
||||
"why": "live broker cache missing" if age is None else f"cache is {age:.0f} days old",
|
||||
"command": "python3 scripts/pdd.py refresh-brokers",
|
||||
})
|
||||
|
||||
# 0b) DROP one-shot: for a CA resident, ONE request deletes from every registered
|
||||
# broker (the whole CA Data Broker Registry) -- the highest-leverage removal there is.
|
||||
registry_recs = brokers_mod.load_registry_cache()
|
||||
residency = (dossier.get("residency_jurisdiction") or "US").upper()
|
||||
drop_filed = bool((dossier.get("preferences") or {}).get("drop_filed_at"))
|
||||
if registry_recs and residency.startswith("US-CA") and not drop_filed:
|
||||
actions.append({
|
||||
"type": "drop_submit",
|
||||
"one_shot": True,
|
||||
"registry_count": len(registry_recs),
|
||||
"url": registry.DROP_URL,
|
||||
"command": f"python3 scripts/pdd.py drop {subject_id}",
|
||||
"why": f"CA resident: one DROP request deletes from all {len(registry_recs)} registered "
|
||||
"data brokers at once (superset of what commercial services cover).",
|
||||
"after": f"python3 scripts/pdd.py drop {subject_id} --filed",
|
||||
})
|
||||
|
||||
# 1) Phase 1 crawl: everything unscanned (read-only, parallel-safe)
|
||||
unscanned = groups.get("unscanned") or []
|
||||
if unscanned:
|
||||
ids = [r["broker_id"] for r in unscanned]
|
||||
if len(ids) > FANOUT_THRESHOLD:
|
||||
actions.append({
|
||||
"type": "fanout_scan",
|
||||
"broker_ids": ids,
|
||||
"command": f"python3 scripts/pdd.py fanout {subject_id}",
|
||||
"how": "spawn ONE delegate_task subagent per batch IN PARALLEL with each batch's brief; "
|
||||
"parent re-verifies key `found` claims before trusting them",
|
||||
})
|
||||
else:
|
||||
actions.append({
|
||||
"type": "scan_inline",
|
||||
"broker_ids": ids,
|
||||
"command": f"python3 scripts/pdd.py plan {subject_id}",
|
||||
"how": "run every search_vector per broker via the methods.md ladder "
|
||||
"(web_extract -> site: probe -> browser), record a verdict per broker",
|
||||
})
|
||||
|
||||
# 2) in-flight email verifications: poll the inbox (or hand to the human in draft mode)
|
||||
for st in ("submitted", "verification_pending"):
|
||||
for bid, case in sorted(ledger.items()):
|
||||
if case.get("state") != st:
|
||||
continue
|
||||
broker = by_id.get(bid) or {}
|
||||
if not ((broker.get("optout") or {}).get("requires") or {}).get("email_verification"):
|
||||
continue
|
||||
if mail["imap"]:
|
||||
actions.append({
|
||||
"type": "poll_verification", "via": "imap",
|
||||
"broker_id": bid,
|
||||
"command": f"python3 scripts/pdd.py poll-verification {subject_id} --broker {bid}",
|
||||
"then": "browser_navigate the returned link IN THE SAME AGENT BROWSER (sessions are "
|
||||
f"browser-bound), complete the flow, then record: awaiting_processing",
|
||||
})
|
||||
elif email_mode == "browser":
|
||||
actions.append({
|
||||
"type": "poll_verification", "via": "browser", "broker_id": bid,
|
||||
"how": "open the broker's confirmation email in the operator's logged-in webmail "
|
||||
f"(browser_*), then `python3 scripts/pdd.py verify-link {subject_id} {bid} "
|
||||
"--text '<email body>'` to score the link, browser_navigate it in the SAME "
|
||||
"browser, then record awaiting_processing",
|
||||
})
|
||||
else:
|
||||
digest.append(_digest(
|
||||
{"broker_id": bid, "broker_name": (broker.get("name") or bid)},
|
||||
"verification email must be opened by a human (draft mode, no inbox access)",
|
||||
["Open the broker's verification email in the subject's inbox and click the link",
|
||||
f"Then: python3 scripts/pdd.py record {subject_id} {bid} awaiting_processing"]))
|
||||
|
||||
# 3) due rechecks: processing windows elapsed / reappearance sweeps
|
||||
for case in ledger_mod.due(subject_id, at=at, ledger=ledger):
|
||||
bid = case.get("broker_id")
|
||||
st = case.get("state")
|
||||
if st in ("awaiting_processing", "confirmed_removed"):
|
||||
actions.append({
|
||||
"type": "verify_removal",
|
||||
"broker_id": bid,
|
||||
"why": "processing window elapsed" if st == "awaiting_processing" else "periodic reappearance re-scan",
|
||||
"how": "re-run this broker's search_vectors; if gone record confirmed_removed; "
|
||||
"if still listed record reappeared and requeue the opt-out",
|
||||
})
|
||||
elif st in ("submitted", "verification_pending") and not mail["imap"]:
|
||||
pass # already covered by the digest entry above
|
||||
|
||||
# 4) Phase 2 opt-outs: parents first (batch_plan already ordered them)
|
||||
for row in groups.get("found") or []:
|
||||
action, task = _optout_action(row, playbook, subject_id, dossier,
|
||||
email_mode, mail["smtp"], confirm_first)
|
||||
if action:
|
||||
actions.append(action)
|
||||
if task:
|
||||
digest.append(task)
|
||||
|
||||
# 5) indirect exposure: targeted delete-my-PII requests
|
||||
for row in groups.get("indirect_exposure") or []:
|
||||
bid = row["broker_id"]
|
||||
if (email_mode in ("programmatic", "alias") and mail["smtp"]) or email_mode == "browser":
|
||||
actions.append({
|
||||
"type": "indirect_email_send",
|
||||
"broker_id": bid, "confirm_first": confirm_first,
|
||||
"send_via": "browser" if email_mode == "browser" else "smtp",
|
||||
"command": f"python3 scripts/pdd.py send-email {subject_id} {bid} --kind ccpa_indirect "
|
||||
f"--listing <third-party-listing-url>",
|
||||
})
|
||||
else:
|
||||
digest.append(_digest(row, "indirect-exposure request (draft mode: a human must hit send)",
|
||||
["Send the rendered ccpa_indirect draft",
|
||||
f"Then: python3 scripts/pdd.py record {subject_id} {bid} submitted "
|
||||
f"--disclosed contact_email --channel email"],
|
||||
prep=[f"python3 scripts/pdd.py render-email {subject_id} {bid} "
|
||||
f"--kind ccpa_indirect --listing <url>"]))
|
||||
|
||||
# 6) blocked sites: stealth pass if we have one, else the operator-browser path
|
||||
blocked = groups.get("blocked") or []
|
||||
if blocked:
|
||||
ids = [r["broker_id"] for r in blocked]
|
||||
if bool(env.get("BROWSERBASE_API_KEY")):
|
||||
actions.append({
|
||||
"type": "stealth_rescan",
|
||||
"broker_ids": ids,
|
||||
"how": "retry these with the cloud/stealth browser backend, then record real verdicts",
|
||||
})
|
||||
else:
|
||||
for r in blocked:
|
||||
digest.append(_digest(r, "site blocks automated access (anti-bot); a human browser gets through",
|
||||
["Open the paste-ready search URL from `plan` in your everyday browser",
|
||||
"Report the verdict (or a screenshot) back to the agent",
|
||||
f"Agent records: python3 scripts/pdd.py record {subject_id} "
|
||||
f"{r['broker_id']} <found|not_found|indirect_exposure>"]))
|
||||
|
||||
# 7) anything already parked as a human task
|
||||
for bid, case in sorted(ledger.items()):
|
||||
if case.get("state") == "human_task_queued":
|
||||
broker = by_id.get(bid) or {}
|
||||
digest.append(_digest({"broker_id": bid, "broker_name": broker.get("name") or bid},
|
||||
case.get("human_task_reason") or "queued manual step",
|
||||
["See `pdd.py tasks` for the exact steps recorded with this case"]))
|
||||
|
||||
# registry coverage summary (breadth beyond the scannable people-search sites)
|
||||
coverage = None
|
||||
if registry_recs:
|
||||
coverage = {
|
||||
"people_search_sites": len(brokers_list),
|
||||
"registered_data_brokers": len(registry_recs),
|
||||
"worked_via": "CA DROP one-shot" if residency.startswith("US-CA") else "targeted CCPA/GDPR email",
|
||||
}
|
||||
if not residency.startswith("US-CA"):
|
||||
coverage["note"] = ("DROP is CA-only; for this subject the registry is covered by targeted "
|
||||
"CCPA/GDPR deletion emails (`registry --search` then `send-email`), "
|
||||
"not a single portal request.")
|
||||
elif drop_filed:
|
||||
coverage["note"] = "DROP already filed; registry deletions are in the brokers' hands."
|
||||
|
||||
next_wake = _min_future_recheck(ledger, at)
|
||||
return {
|
||||
"subject": subject_id,
|
||||
"autonomy": autonomy,
|
||||
"phase": batch.get("phase"),
|
||||
"counts": batch.get("counts"),
|
||||
"actions": actions,
|
||||
"human_digest": digest,
|
||||
"coverage": coverage,
|
||||
"done_for_now": not actions,
|
||||
"fully_done": not actions and not digest and not next_wake,
|
||||
"next_wake_at": next_wake,
|
||||
"note": ("assisted mode: pause for operator confirmation on every action with confirm_first=true"
|
||||
if confirm_first else
|
||||
"full autonomy: recorded intake consent authorizes these submissions; do not pause. "
|
||||
"Present human_digest ONCE at the end of the run, not per item."),
|
||||
}
|
||||
177
optional-skills/security/unbroker/scripts/badbool.py
Normal file
177
optional-skills/security/unbroker/scripts/badbool.py
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
"""Pull and parse the Big-Ass Data Broker Opt-Out List (BADBOOL) into broker records.
|
||||
|
||||
BADBOOL (https://github.com/yaelwrites/Big-Ass-Data-Broker-Opt-Out-List) is a
|
||||
maintained, frequently-updated markdown list. `refresh` fetches it and parses the
|
||||
"People Search Sites" section into records that merge UNDER the curated DB (curated
|
||||
records always win). Auto-parsed records carry source="BADBOOL-auto" and
|
||||
confidence="auto" so the agent treats their URLs as best guesses to verify first.
|
||||
|
||||
`parse()` is pure (markdown in, records out) so it is tested offline; `fetch()` is
|
||||
the only network call and can be bypassed by passing markdown directly to refresh().
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import storage
|
||||
|
||||
DEFAULT_URL = (
|
||||
"https://raw.githubusercontent.com/yaelwrites/"
|
||||
"Big-Ass-Data-Broker-Opt-Out-List/master/README.md"
|
||||
)
|
||||
USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)"
|
||||
|
||||
# BADBOOL legend symbols.
|
||||
SYMBOLS = {
|
||||
"crucial": "\U0001F490", # 💐
|
||||
"high": "\u2620", # ☠
|
||||
"gov_id": "\U0001F3AB", # 🎫
|
||||
"phone": "\U0001F4DE", # 📞
|
||||
"payment": "\U0001F4B0", # 💰
|
||||
}
|
||||
|
||||
_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)")
|
||||
_OPTOUT_HINT = re.compile(
|
||||
r"opt[\- ]?out|optout|removal|remove|suppress|control-privacy|delete", re.I
|
||||
)
|
||||
_FIND_HINT = re.compile(r"find|your information|search|look ?up|look for", re.I)
|
||||
|
||||
|
||||
def slug(name: str) -> str:
|
||||
# Drop a trailing .com/.org/.info on the displayed name so "FastPeopleSearch.com"
|
||||
# matches the curated id "fastpeoplesearch"; keep .net/.id so distinct sites differ.
|
||||
n = re.sub(r"\.(com|org|info)\b", "", name.strip(), flags=re.I)
|
||||
return re.sub(r"[^a-z0-9]+", "", n.lower())
|
||||
|
||||
|
||||
def _heading_flags(heading: str) -> tuple[str, dict]:
|
||||
flags = {key: (sym in heading) for key, sym in SYMBOLS.items()}
|
||||
name = heading
|
||||
for sym in SYMBOLS.values():
|
||||
name = name.replace(sym, "")
|
||||
name = name.replace("\ufe0f", "").strip()
|
||||
return name, flags
|
||||
|
||||
|
||||
def _priority(flags: dict) -> str:
|
||||
if flags["crucial"]:
|
||||
return "crucial"
|
||||
if flags["high"]:
|
||||
return "high"
|
||||
return "standard"
|
||||
|
||||
|
||||
def _pick(links: list[tuple[str, str]], hint: re.Pattern) -> str | None:
|
||||
for _text, url in links:
|
||||
if hint.search(url):
|
||||
return url
|
||||
for text, url in links:
|
||||
if hint.search(text):
|
||||
return url
|
||||
return None
|
||||
|
||||
|
||||
def _clean(text: str) -> str:
|
||||
return re.sub(r"\s+", " ", text).strip()[:600]
|
||||
|
||||
|
||||
def _build(name: str, flags: dict, body: str) -> dict:
|
||||
links = _LINK_RE.findall(body)
|
||||
web = [(t, u) for t, u in links if u.lower().startswith("http")]
|
||||
mailtos = [u[7:] for _t, u in links if u.lower().startswith("mailto:")]
|
||||
optout_url = _pick(web, _OPTOUT_HINT)
|
||||
search_url = _pick(web, _FIND_HINT) or (web[0][1] if web else None)
|
||||
|
||||
if flags["phone"]:
|
||||
method = "phone"
|
||||
elif optout_url:
|
||||
method = "web_form"
|
||||
elif mailtos:
|
||||
method = "email"
|
||||
else:
|
||||
method = "manual"
|
||||
|
||||
return {
|
||||
"id": slug(name),
|
||||
"name": name,
|
||||
"category": "people_search",
|
||||
"priority": _priority(flags),
|
||||
"jurisdictions": ["US"],
|
||||
"search": {"method": "url_pattern", "url": search_url, "fetch": "browser",
|
||||
"match_signal": "result", "by": ["name", "phone", "address"]},
|
||||
"optout": {
|
||||
"method": method,
|
||||
"url": optout_url,
|
||||
"email": mailtos[0] if mailtos else None,
|
||||
"requires": {
|
||||
"gov_id": flags["gov_id"],
|
||||
"phone_voice": flags["phone"],
|
||||
"payment": flags["payment"],
|
||||
"email_verification": False,
|
||||
"captcha": False,
|
||||
"account": False,
|
||||
"phone_callback": False,
|
||||
},
|
||||
"inputs": ["full_name", "contact_email"],
|
||||
"notes": _clean(body),
|
||||
"links": [{"text": t, "url": u} for t, u in links],
|
||||
"est_processing_days": 14, # unknown for auto records; drives next_recheck_at
|
||||
},
|
||||
"source": "BADBOOL-auto",
|
||||
"confidence": "auto",
|
||||
"last_verified": None,
|
||||
}
|
||||
|
||||
|
||||
def parse(markdown: str) -> list[dict]:
|
||||
"""Parse the 'People Search Sites' section of BADBOOL into broker records."""
|
||||
records: list[dict] = []
|
||||
in_people = False
|
||||
heading: str | None = None
|
||||
body: list[str] = []
|
||||
|
||||
def flush() -> None:
|
||||
nonlocal heading, body
|
||||
if heading is not None:
|
||||
name, flags = _heading_flags(heading)
|
||||
if name:
|
||||
records.append(_build(name, flags, "\n".join(body).strip()))
|
||||
heading, body = None, []
|
||||
|
||||
for line in markdown.splitlines():
|
||||
if line.startswith("## "):
|
||||
flush()
|
||||
in_people = line[3:].strip().lower().startswith("people search")
|
||||
continue
|
||||
if not in_people:
|
||||
continue
|
||||
if line.startswith("### "):
|
||||
flush()
|
||||
heading = line[4:].strip()
|
||||
elif heading is not None:
|
||||
body.append(line)
|
||||
flush()
|
||||
return records
|
||||
|
||||
|
||||
def fetch(url: str = DEFAULT_URL, timeout: int = 30) -> str:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310
|
||||
return resp.read().decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
MIN_EXPECTED = 20 # BADBOOL's People Search section lists ~47; far fewer => upstream reorg, warn
|
||||
|
||||
|
||||
def refresh(cache_path: Path, url: str = DEFAULT_URL, markdown: str | None = None) -> dict:
|
||||
"""Fetch (or accept) BADBOOL markdown, parse it, and write the snapshot cache."""
|
||||
md = markdown if markdown is not None else fetch(url)
|
||||
records = parse(md)
|
||||
storage.write_json(cache_path, records)
|
||||
out = {"parsed": len(records), "cache_path": str(cache_path), "source_url": url}
|
||||
if len(records) < MIN_EXPECTED:
|
||||
out["warning"] = (f"only {len(records)} parsed (expected >{MIN_EXPECTED}); BADBOOL's "
|
||||
"'People Search Sites' section may have moved/reorganized - check the parser")
|
||||
return out
|
||||
77
optional-skills/security/unbroker/scripts/brokers.py
Normal file
77
optional-skills/security/unbroker/scripts/brokers.py
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
"""Load and query the broker database (references/brokers/*.json).
|
||||
|
||||
Each broker is one JSON file for clean diffs/PRs. Files beginning with `_` are
|
||||
ignored (reserved for notes/scratch).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import paths
|
||||
import storage
|
||||
|
||||
PRIORITY_ORDER = {"crucial": 0, "high": 1, "standard": 2, "long_tail": 3}
|
||||
|
||||
|
||||
def _load_curated(directory: Path | None = None) -> list[dict]:
|
||||
directory = directory or paths.brokers_dir()
|
||||
out: list[dict] = []
|
||||
if not directory.exists():
|
||||
return out
|
||||
for fp in sorted(directory.glob("*.json")):
|
||||
if fp.name.startswith("_"):
|
||||
continue
|
||||
out.append(json.loads(fp.read_text(encoding="utf-8")))
|
||||
return out
|
||||
|
||||
|
||||
def load_live_cache() -> list[dict]:
|
||||
"""Records pulled from BADBOOL via `refresh-brokers` (empty until refreshed)."""
|
||||
return storage.read_json(paths.brokers_cache_path(), []) or []
|
||||
|
||||
|
||||
def load_registry_cache() -> list[dict]:
|
||||
"""CA Data Broker Registry records (separate coverage lane; empty until refreshed).
|
||||
|
||||
Kept OUT of load_all() by default: these are not people-search sites to scan, they
|
||||
are worked via the CA DROP one-shot + CCPA email. Consumers of the scan/plan/fanout
|
||||
pipeline must not receive them; use this directly for coverage counts and the DROP/
|
||||
email lanes.
|
||||
"""
|
||||
return storage.read_json(paths.registry_cache_path(), []) or []
|
||||
|
||||
|
||||
def load_all(directory: Path | None = None, include_live: bool = True) -> list[dict]:
|
||||
"""Curated records, with live BADBOOL records merged underneath (curated wins)."""
|
||||
merged: dict[str, dict] = {b["id"]: b for b in _load_curated(directory)}
|
||||
if include_live:
|
||||
for b in load_live_cache():
|
||||
bid = b.get("id")
|
||||
if bid and bid not in merged:
|
||||
merged[bid] = b
|
||||
out = list(merged.values())
|
||||
out.sort(key=lambda b: (PRIORITY_ORDER.get(b.get("priority", "standard"), 9), b.get("id", "")))
|
||||
return out
|
||||
|
||||
|
||||
def get(broker_id: str, directory: Path | None = None) -> dict | None:
|
||||
for b in load_all(directory):
|
||||
if b.get("id") == broker_id:
|
||||
return b
|
||||
return None
|
||||
|
||||
|
||||
def by_priority(*levels: str, directory: Path | None = None) -> list[dict]:
|
||||
wanted = set(levels) if levels else None
|
||||
return [b for b in load_all(directory) if wanted is None or b.get("priority") in wanted]
|
||||
|
||||
|
||||
def clusters(directory: Path | None = None) -> dict[str, list[str]]:
|
||||
"""Map a parent broker id -> child site ids it can clear (force-multipliers)."""
|
||||
out: dict[str, list[str]] = {}
|
||||
for b in load_all(directory):
|
||||
owns = b.get("owns") or []
|
||||
if owns:
|
||||
out[b["id"]] = list(owns)
|
||||
return out
|
||||
124
optional-skills/security/unbroker/scripts/config.py
Normal file
124
optional-skills/security/unbroker/scripts/config.py
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
"""Install-wide configuration with easiest-first defaults.
|
||||
|
||||
Everything works zero-config. `setup --auto` (the autonomous path) detects what
|
||||
this environment can do and picks the MOST AUTONOMOUS valid configuration without
|
||||
asking anyone; plain `setup` keeps the easiest-first defaults and only upgrades a
|
||||
setting when a flag opts in.
|
||||
|
||||
`autonomy` is policy, orthogonal to capability:
|
||||
full - intake consent is standing authorization; the agent submits T0-T2
|
||||
opt-outs without pausing per submission (default).
|
||||
assisted - the agent pauses for operator confirmation before each submission.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from shutil import which
|
||||
|
||||
import emailer
|
||||
import paths
|
||||
import storage
|
||||
|
||||
DEFAULT_CONFIG = {
|
||||
"autonomy": "full", # hands-off after intake+consent
|
||||
"email_mode": "draft_only", # zero credentials
|
||||
"browser_backend": "auto", # auto = Browserbase when BROWSERBASE_API_KEY is set
|
||||
# (recommended default; clears soft CAPTCHAs), else plain browser
|
||||
"tracker_backend": "local-json", # no external dependency
|
||||
"encryption": "none", # files still written 0600
|
||||
"default_rescan_interval_days": 120,
|
||||
"email_min_interval_seconds": 20, # pace SMTP sends so a run can't torch the account
|
||||
}
|
||||
|
||||
VALID = {
|
||||
"autonomy": {"full", "assisted"},
|
||||
# email_mode:
|
||||
# draft_only - render drafts; the operator sends + clicks verify links (zero setup)
|
||||
# browser - the agent sends + opens verify links through the operator's logged-in
|
||||
# webmail via browser_* tools (NO password stored; needs a browser the
|
||||
# operator's inbox is signed into)
|
||||
# programmatic - CLI sends via SMTP + reads verify links via IMAP (needs EMAIL_* creds)
|
||||
# alias - AgentMail agent-owned inboxes / per-broker aliases
|
||||
"email_mode": {"draft_only", "browser", "programmatic", "alias"},
|
||||
"browser_backend": {"auto", "browserbase", "agent-browser", "camofox"},
|
||||
"tracker_backend": {"local-json", "google-sheets"},
|
||||
"encryption": {"none", "age"},
|
||||
}
|
||||
|
||||
|
||||
def load_config() -> dict:
|
||||
cfg = dict(DEFAULT_CONFIG)
|
||||
cfg.update(storage.read_json(paths.config_path(), {}) or {})
|
||||
return cfg
|
||||
|
||||
|
||||
def save_config(cfg: dict) -> Path:
|
||||
merged = dict(DEFAULT_CONFIG)
|
||||
merged.update(cfg)
|
||||
for key, allowed in VALID.items():
|
||||
if merged.get(key) not in allowed:
|
||||
raise ValueError(f"invalid {key!r}: {merged.get(key)!r} (allowed: {sorted(allowed)})")
|
||||
return storage.write_json(paths.config_path(), merged)
|
||||
|
||||
|
||||
def detect_capabilities(env: dict | None = None) -> dict:
|
||||
"""Report which opt-in upgrades are available without extra setup."""
|
||||
env = os.environ if env is None else env
|
||||
home = paths.hermes_home()
|
||||
google = (
|
||||
(home / "google_token.json").exists()
|
||||
or (home / "skills" / "productivity" / "google-workspace").exists()
|
||||
or (home / "skills" / "google-workspace").exists()
|
||||
)
|
||||
mail = emailer.available(env)
|
||||
return {
|
||||
"browserbase": bool(env.get("BROWSERBASE_API_KEY")),
|
||||
"agentmail": bool(env.get("AGENTMAIL_API_KEY")),
|
||||
"email_imap_smtp": bool(env.get("EMAIL_ADDRESS") and env.get("EMAIL_PASSWORD")),
|
||||
"smtp_send": mail["smtp"], # CLI can SEND opt-out emails itself
|
||||
"imap_read": mail["imap"], # CLI can POLL verification links itself
|
||||
"google_workspace": google,
|
||||
"age": which("age") is not None,
|
||||
}
|
||||
|
||||
|
||||
def auto_configure(env: dict | None = None) -> dict:
|
||||
"""Pick the most autonomous configuration this environment supports (no questions).
|
||||
|
||||
- email: programmatic when SMTP creds exist (CLI sends + IMAP-verifies itself);
|
||||
alias mode when only AgentMail exists; draft_only as the capability floor.
|
||||
- browser: browserbase when the key exists (clears soft CAPTCHAs -> more T1).
|
||||
- encryption: age when the binary is installed (free privacy, zero human cost).
|
||||
- tracker: stays local-json (google-sheets needs a sheet id -> a human choice).
|
||||
"""
|
||||
caps = detect_capabilities(env)
|
||||
cfg = load_config()
|
||||
cfg["autonomy"] = "full"
|
||||
if caps["smtp_send"]:
|
||||
cfg["email_mode"] = "programmatic"
|
||||
elif caps["agentmail"]:
|
||||
cfg["email_mode"] = "alias"
|
||||
else:
|
||||
cfg["email_mode"] = "draft_only"
|
||||
cfg["browser_backend"] = "browserbase" if caps["browserbase"] else "auto"
|
||||
if caps["age"]:
|
||||
cfg["encryption"] = "age"
|
||||
return cfg
|
||||
|
||||
|
||||
def browser_clears_captcha(cfg: dict, env: dict | None = None) -> bool:
|
||||
"""True if the chosen browser backend can clear soft CAPTCHAs (shifts T2 -> T1).
|
||||
|
||||
Browserbase is the recommended default: a real residential-IP cloud browser passes
|
||||
soft/managed challenges (Turnstile, hCaptcha/reCAPTCHA checkbox) as normal operation.
|
||||
This is NOT solving/spoofing - hard interactive challenges still escalate to a human.
|
||||
`auto` inherits this whenever BROWSERBASE_API_KEY is present.
|
||||
"""
|
||||
backend = cfg.get("browser_backend", "auto")
|
||||
if backend == "browserbase":
|
||||
return True
|
||||
if backend == "auto":
|
||||
env = os.environ if env is None else env
|
||||
return bool(env.get("BROWSERBASE_API_KEY"))
|
||||
return False
|
||||
88
optional-skills/security/unbroker/scripts/crypto.py
Normal file
88
optional-skills/security/unbroker/scripts/crypto.py
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
"""At-rest encryption for sensitive files via the `age` binary (optional).
|
||||
|
||||
Engaged ONLY when config `encryption: age` AND an age identity key exists AND the
|
||||
`age`/`age-keygen` binaries are available. When engaged, JSON docs under
|
||||
`subjects/` (dossier, ledger) are written as `<file>.age` ciphertext; the audit
|
||||
log (field NAMES + states only, no raw PII values), `config.json`, and the broker
|
||||
cache stay plaintext so the engine can read them.
|
||||
|
||||
Threat model (be honest): this protects against casual disk inspection, accidental
|
||||
`git add`/commits, screen-shares, and backup/cloud-sync leakage. The identity key
|
||||
defaults to living beside the data at `$PDD_DATA_DIR/age-identity.txt` (0600); set
|
||||
`PDD_AGE_IDENTITY` to a separate volume/token for true key separation. It does NOT
|
||||
protect against an attacker who can already read your whole HERMES_HOME (they get
|
||||
key + data together).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from shutil import which
|
||||
|
||||
import paths
|
||||
|
||||
|
||||
def age_available() -> bool:
|
||||
return which("age") is not None and which("age-keygen") is not None
|
||||
|
||||
|
||||
def encryption_setting() -> str:
|
||||
"""Read `encryption` straight from config.json (no config/storage import => no cycle)."""
|
||||
cfg = paths.config_path()
|
||||
if not cfg.exists():
|
||||
return "none"
|
||||
try:
|
||||
return (json.loads(cfg.read_text(encoding="utf-8")) or {}).get("encryption", "none")
|
||||
except (ValueError, OSError):
|
||||
return "none"
|
||||
|
||||
|
||||
def identity_path() -> Path:
|
||||
return paths.age_identity_path()
|
||||
|
||||
|
||||
def ensure_identity() -> Path:
|
||||
"""Generate an age identity (X25519 keypair) if missing; return its path."""
|
||||
if not age_available():
|
||||
raise RuntimeError("`age`/`age-keygen` not found; cannot enable encryption")
|
||||
p = identity_path()
|
||||
if not p.exists():
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
p.parent.chmod(0o700)
|
||||
except OSError:
|
||||
pass
|
||||
subprocess.run(["age-keygen", "-o", str(p)], check=True, capture_output=True)
|
||||
try:
|
||||
p.chmod(0o600)
|
||||
except OSError:
|
||||
pass
|
||||
return p
|
||||
|
||||
|
||||
def recipient() -> str:
|
||||
"""The age public key (recipient) for the identity, parsed from its header."""
|
||||
p = ensure_identity()
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
s = line.strip()
|
||||
if s.lower().startswith("# public key:"):
|
||||
return s.split(":", 1)[1].strip()
|
||||
if s.startswith("age1"):
|
||||
return s
|
||||
raise RuntimeError(f"no public key found in {p}")
|
||||
|
||||
|
||||
def is_engaged() -> bool:
|
||||
"""True only when encryption is actually active (configured + available + key present)."""
|
||||
return encryption_setting() == "age" and age_available() and identity_path().exists()
|
||||
|
||||
|
||||
def encrypt(data: bytes) -> bytes:
|
||||
out = subprocess.run(["age", "-r", recipient()], input=data, capture_output=True, check=True)
|
||||
return out.stdout
|
||||
|
||||
|
||||
def decrypt(data: bytes) -> bytes:
|
||||
out = subprocess.run(["age", "-d", "-i", str(identity_path())], input=data, capture_output=True, check=True)
|
||||
return out.stdout
|
||||
135
optional-skills/security/unbroker/scripts/dossier.py
Normal file
135
optional-skills/security/unbroker/scripts/dossier.py
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
"""Subject dossier management + consent gate + least-disclosure field selection."""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import paths
|
||||
import storage
|
||||
|
||||
# Identifiers we never volunteer in an opt-out (would expand exposure, not reduce it).
|
||||
NEVER_VOLUNTEER = {"ssn", "social_security_number", "passport", "drivers_license"}
|
||||
|
||||
VALID_CONSENT_METHODS = {"self", "written_authorization", "poa"}
|
||||
|
||||
|
||||
def now() -> str:
|
||||
return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def new_subject_id(full_name: str = "") -> str:
|
||||
# Opaque id: derives NOTHING from the name, so PII never leaks into directory names,
|
||||
# case ids, drafts, or the audit log. full_name kept only for call compatibility.
|
||||
return "sub_" + hashlib.sha1(os.urandom(8)).hexdigest()[:10]
|
||||
|
||||
|
||||
def create(identity: dict, consent: dict, residency: str = "US", prefs: dict | None = None) -> dict:
|
||||
dossier = {
|
||||
"subject_id": new_subject_id(identity.get("full_name", "subject")),
|
||||
"consent": consent,
|
||||
"identity": identity,
|
||||
"residency_jurisdiction": residency,
|
||||
"preferences": prefs or {"email_mode": "draft_only", "rescan_interval_days": 120},
|
||||
"created_at": now(),
|
||||
}
|
||||
save(dossier)
|
||||
return dossier
|
||||
|
||||
|
||||
def load(subject_id: str) -> dict | None:
|
||||
return storage.read_json(paths.dossier_path(subject_id), None)
|
||||
|
||||
|
||||
def save(dossier: dict) -> Path:
|
||||
return storage.write_json(paths.dossier_path(dossier["subject_id"]), dossier)
|
||||
|
||||
|
||||
def is_authorized(dossier: dict) -> bool:
|
||||
c = dossier.get("consent") or {}
|
||||
return bool(c.get("authorized")) and c.get("method") in VALID_CONSENT_METHODS
|
||||
|
||||
|
||||
def require_authorized(dossier: dict) -> None:
|
||||
if not is_authorized(dossier):
|
||||
raise PermissionError(
|
||||
f"subject {dossier.get('subject_id')!r} has no recorded authorization; refusing to act"
|
||||
)
|
||||
|
||||
|
||||
def all_names(dossier: dict) -> list[str]:
|
||||
"""Primary name + aliases (maiden/married/nicknames), deduped, in priority order."""
|
||||
ident = dossier.get("identity", {})
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for n in [ident.get("full_name"), *(ident.get("also_known_as") or [])]:
|
||||
if n and n.lower() not in seen:
|
||||
seen.add(n.lower())
|
||||
out.append(n)
|
||||
return out
|
||||
|
||||
|
||||
def all_addresses(dossier: dict) -> list[dict]:
|
||||
"""Current + prior addresses, each tagged with `kind` (current|prior)."""
|
||||
ident = dossier.get("identity", {})
|
||||
out: list[dict] = []
|
||||
cur = ident.get("current_address")
|
||||
if cur:
|
||||
out.append({**cur, "kind": cur.get("kind", "current")})
|
||||
for a in ident.get("prior_addresses") or []:
|
||||
out.append({**a, "kind": a.get("kind", "prior")})
|
||||
return out
|
||||
|
||||
|
||||
def all_locations(dossier: dict) -> list[dict]:
|
||||
"""Distinct city/state pairs across all addresses (the vectors for name searches)."""
|
||||
out: list[dict] = []
|
||||
seen: set[tuple] = set()
|
||||
for a in all_addresses(dossier):
|
||||
city = a.get("city")
|
||||
key = ((city or "").lower(), (a.get("state") or "").lower())
|
||||
if city and key not in seen:
|
||||
seen.add(key)
|
||||
out.append({"city": city, "state": a.get("state")})
|
||||
return out
|
||||
|
||||
|
||||
def contact_email(dossier: dict) -> str | None:
|
||||
"""The single email used for opt-out correspondence (designated, else the first)."""
|
||||
ident = dossier.get("identity", {})
|
||||
prefs = dossier.get("preferences", {})
|
||||
emails = ident.get("emails") or []
|
||||
return prefs.get("contact_email_for_optouts") or (emails[0] if emails else None)
|
||||
|
||||
|
||||
def select_disclosure(dossier: dict, inputs: list[str], override_email: str | None = None) -> dict:
|
||||
"""Return ONLY the dossier fields a broker's opt-out actually requires.
|
||||
|
||||
Enforces least-disclosure: skips anything in NEVER_VOLUNTEER, and skips
|
||||
`profile_url` (that is captured per-listing at submit time, not from the dossier).
|
||||
A single contact email is used for correspondence even when the subject has several
|
||||
(see all_names / all_addresses / search vectors for using every alternate to *find* listings).
|
||||
"""
|
||||
ident = dossier.get("identity", {})
|
||||
addr = ident.get("current_address") or {}
|
||||
phones = ident.get("phones") or []
|
||||
available = {
|
||||
"full_name": ident.get("full_name"),
|
||||
"first_name": (ident.get("full_name") or "").split(" ")[0] or None,
|
||||
"contact_email": override_email or contact_email(dossier),
|
||||
"current_address": addr or None,
|
||||
"street": addr.get("line1"),
|
||||
"city": addr.get("city"),
|
||||
"state": addr.get("state"),
|
||||
"postal": addr.get("postal"),
|
||||
"date_of_birth": ident.get("date_of_birth"),
|
||||
"phone": phones[0] if phones else None,
|
||||
}
|
||||
out: dict = {}
|
||||
for key in inputs:
|
||||
if key in NEVER_VOLUNTEER or key == "profile_url":
|
||||
continue
|
||||
if available.get(key) is not None:
|
||||
out[key] = available[key]
|
||||
return out
|
||||
76
optional-skills/security/unbroker/scripts/email_modes.py
Normal file
76
optional-skills/security/unbroker/scripts/email_modes.py
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
"""Email modes A/B/C helpers + anti-phishing verification-link extraction.
|
||||
|
||||
Mode A (default): render a ready-to-send draft to disk; the operator sends it.
|
||||
Mode B/C: the agent SENDS via a Hermes email mechanism (IMAP/SMTP gateway,
|
||||
`himalaya`, AgentMail, or Gmail via `google-workspace`) and READS the reply to
|
||||
resolve the verification link with `extract_verification_link`. Those transports
|
||||
are driven by the agent through native tools; this module stays network-free so
|
||||
the hermetic tests pass.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import legal
|
||||
import paths
|
||||
|
||||
_LINK_RE = re.compile(r"https?://[^\s\"'<>)\]]+", re.IGNORECASE)
|
||||
_VERIFY_HINTS = ("opt", "remov", "verif", "confirm", "unsubscrib", "suppress", "delete", "privacy")
|
||||
|
||||
|
||||
def render_draft(broker: dict, fields: dict, out_dir: Path | None = None) -> Path:
|
||||
"""Mode A: write a ready-to-send opt-out email for the operator to send."""
|
||||
body = legal.render_optout_email(broker, fields)
|
||||
out_dir = out_dir or (paths.data_dir() / "drafts")
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
fp = out_dir / f"{broker.get('id', 'broker')}.txt"
|
||||
fp.write_text(body, encoding="utf-8")
|
||||
return fp
|
||||
|
||||
|
||||
def render_request_draft(broker: dict, fields: dict, kind: str = "generic",
|
||||
out_dir: Path | None = None) -> Path:
|
||||
"""Mode A: write a ready-to-send request of a specific KIND.
|
||||
|
||||
kind: generic | ccpa | ccpa_agent | ccpa_indirect | gdpr. Used for indirect-exposure
|
||||
(ccpa_indirect) and explicit legal requests, where the generic opt-out wording is wrong.
|
||||
The filename is suffixed with the kind so an indirect request does not overwrite an opt-out draft.
|
||||
"""
|
||||
body = legal.render_request(kind, broker, fields)
|
||||
out_dir = out_dir or (paths.data_dir() / "drafts")
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
suffix = "" if kind == "generic" else f"-{kind}"
|
||||
fp = out_dir / f"{broker.get('id', 'broker')}{suffix}.txt"
|
||||
fp.write_text(body, encoding="utf-8")
|
||||
return fp
|
||||
|
||||
|
||||
def extract_verification_link(email_body: str, broker: dict | None = None) -> str | None:
|
||||
"""Return the most likely opt-out/verification link from an email body.
|
||||
|
||||
Anti-phishing: a link is only returned if its URL matches an opt-out hint
|
||||
and/or the broker's own domain; arbitrary links score 0 and are ignored.
|
||||
"""
|
||||
candidates = _LINK_RE.findall(email_body or "")
|
||||
if not candidates:
|
||||
return None
|
||||
|
||||
domain = ""
|
||||
if broker:
|
||||
url = (broker.get("optout") or {}).get("url") or (broker.get("search") or {}).get("url") or ""
|
||||
m = re.search(r"https?://([^/]+)", url)
|
||||
if m:
|
||||
domain = m.group(1).replace("www.", "")
|
||||
|
||||
best_score, best_link = 0, None
|
||||
for link in candidates:
|
||||
low = link.lower()
|
||||
score = 0
|
||||
if any(h in low for h in _VERIFY_HINTS):
|
||||
score += 2
|
||||
if domain and domain in low:
|
||||
score += 3
|
||||
if score > best_score:
|
||||
best_score, best_link = score, link
|
||||
return best_link
|
||||
342
optional-skills/security/unbroker/scripts/emailer.py
Normal file
342
optional-skills/security/unbroker/scripts/emailer.py
Normal file
|
|
@ -0,0 +1,342 @@
|
|||
"""Programmatic email (Mode B) via stdlib smtplib/imaplib - no human in the loop.
|
||||
|
||||
This is what turns email opt-outs autonomous: `send()` delivers the rendered
|
||||
request straight to the broker's known opt-out address, and `find_verification_link()`
|
||||
polls the inbox for the broker's confirmation email and extracts the link (scored
|
||||
by email_modes.extract_verification_link, so arbitrary/phishing links are ignored).
|
||||
The agent still OPENS the link with its own browser - several brokers bind the
|
||||
verification session to the browser that opens it (see the intelius record).
|
||||
|
||||
Configuration comes from the same env vars the Hermes email gateway uses:
|
||||
EMAIL_ADDRESS / EMAIL_PASSWORD (required for Mode B)
|
||||
EMAIL_SMTP_HOST / EMAIL_SMTP_PORT (optional; inferred for common providers)
|
||||
EMAIL_IMAP_HOST / EMAIL_IMAP_PORT (optional; inferred for common providers)
|
||||
|
||||
Anti-misuse: `send()` refuses a recipient that is not the broker record's own
|
||||
opt-out/privacy address - this module cannot be repurposed to email arbitrary people.
|
||||
All network calls live behind small functions that the hermetic tests monkeypatch.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import email as _email
|
||||
import email.utils
|
||||
import imaplib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import smtplib
|
||||
import time
|
||||
from email.message import EmailMessage
|
||||
from pathlib import Path
|
||||
|
||||
import email_modes
|
||||
import paths
|
||||
|
||||
# provider domain -> (smtp_host, smtp_port, imap_host, imap_port)
|
||||
PROVIDERS = {
|
||||
"gmail.com": ("smtp.gmail.com", 587, "imap.gmail.com", 993),
|
||||
"googlemail.com": ("smtp.gmail.com", 587, "imap.gmail.com", 993),
|
||||
"outlook.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993),
|
||||
"hotmail.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993),
|
||||
"live.com": ("smtp-mail.outlook.com", 587, "outlook.office365.com", 993),
|
||||
"yahoo.com": ("smtp.mail.yahoo.com", 587, "imap.mail.yahoo.com", 993),
|
||||
"icloud.com": ("smtp.mail.me.com", 587, "imap.mail.me.com", 993),
|
||||
"me.com": ("smtp.mail.me.com", 587, "imap.mail.me.com", 993),
|
||||
"fastmail.com": ("smtp.fastmail.com", 587, "imap.fastmail.com", 993),
|
||||
}
|
||||
|
||||
|
||||
def _domain(address: str) -> str:
|
||||
return address.rsplit("@", 1)[-1].lower() if "@" in address else ""
|
||||
|
||||
|
||||
def smtp_settings(env: dict | None = None) -> dict | None:
|
||||
"""SMTP connection settings, or None when sending is not configured."""
|
||||
env = os.environ if env is None else env
|
||||
address, password = env.get("EMAIL_ADDRESS"), env.get("EMAIL_PASSWORD")
|
||||
if not (address and password):
|
||||
return None
|
||||
inferred = PROVIDERS.get(_domain(address))
|
||||
host = env.get("EMAIL_SMTP_HOST") or (inferred[0] if inferred else None)
|
||||
if not host:
|
||||
return None # unknown provider and no explicit host
|
||||
port = int(env.get("EMAIL_SMTP_PORT") or (inferred[1] if inferred else 587))
|
||||
return {"host": host, "port": port, "address": address, "password": password}
|
||||
|
||||
|
||||
def imap_settings(env: dict | None = None) -> dict | None:
|
||||
"""IMAP connection settings, or None when inbox reading is not configured."""
|
||||
env = os.environ if env is None else env
|
||||
address, password = env.get("EMAIL_ADDRESS"), env.get("EMAIL_PASSWORD")
|
||||
if not (address and password):
|
||||
return None
|
||||
inferred = PROVIDERS.get(_domain(address))
|
||||
host = env.get("EMAIL_IMAP_HOST") or (inferred[2] if inferred else None)
|
||||
if not host:
|
||||
return None
|
||||
port = int(env.get("EMAIL_IMAP_PORT") or (inferred[3] if inferred else 993))
|
||||
return {"host": host, "port": port, "address": address, "password": password}
|
||||
|
||||
|
||||
def available(env: dict | None = None) -> dict:
|
||||
return {"smtp": smtp_settings(env) is not None, "imap": imap_settings(env) is not None}
|
||||
|
||||
|
||||
# --- sending ------------------------------------------------------------------
|
||||
|
||||
def broker_addresses(broker: dict) -> list[str]:
|
||||
"""Every address the broker record itself declares (the ONLY valid recipients).
|
||||
|
||||
Includes the primary opt-out email, the right-to-delete lane's email
|
||||
(optout.deletion.email), and any mailto: links parsed from BADBOOL.
|
||||
"""
|
||||
opt = broker.get("optout") or {}
|
||||
out = [a for a in [opt.get("email"), (opt.get("deletion") or {}).get("email")] if a]
|
||||
for link in opt.get("links") or []:
|
||||
url = (link.get("url") or "")
|
||||
if url.lower().startswith("mailto:"):
|
||||
out.append(url[7:].split("?")[0])
|
||||
seen: set[str] = set()
|
||||
deduped = []
|
||||
for a in out:
|
||||
if a.lower() not in seen:
|
||||
seen.add(a.lower())
|
||||
deduped.append(a)
|
||||
return deduped
|
||||
|
||||
|
||||
def _split_subject_body(text: str) -> tuple[str, str]:
|
||||
"""Templates start with a 'Subject: ...' line; split it out for the MIME header."""
|
||||
lines = text.splitlines()
|
||||
if lines and lines[0].lower().startswith("subject:"):
|
||||
return lines[0].split(":", 1)[1].strip(), "\n".join(lines[1:]).lstrip("\n")
|
||||
return "Data removal request", text
|
||||
|
||||
|
||||
def browser_send_payload(broker: dict, body_text: str, to: str | None = None) -> dict:
|
||||
"""Build a recipient-locked {to, subject, body} for the agent to send via browser webmail.
|
||||
|
||||
No network and no credentials: the deterministic part (recipient-lock to the broker's own
|
||||
declared address, subject/body split) happens here; the agent then composes and sends it in
|
||||
the operator's logged-in webmail with browser_* tools. Same recipient guard as `send()`, so
|
||||
the browser lane cannot be pointed at an arbitrary person either.
|
||||
"""
|
||||
allowed = broker_addresses(broker)
|
||||
if not allowed:
|
||||
raise RuntimeError(f"broker {broker.get('id')!r} declares no opt-out email address")
|
||||
recipient = to or allowed[0]
|
||||
if recipient.lower() not in {a.lower() for a in allowed}:
|
||||
raise PermissionError(
|
||||
f"refusing to target {recipient!r}: not an address the broker record declares "
|
||||
f"(allowed: {allowed})"
|
||||
)
|
||||
subject, body = _split_subject_body(body_text)
|
||||
return {"to": recipient, "subject": subject, "body": body}
|
||||
|
||||
|
||||
def _rate_limit_path() -> Path:
|
||||
return paths.data_dir() / "email-rate.json"
|
||||
|
||||
|
||||
def _respect_rate_limit(min_interval: float, sleep, now, state_path=None) -> None:
|
||||
"""Pace sends across CLI invocations so a run can't torch the sending account.
|
||||
|
||||
Persists the last-send wall-clock time; if the next send is too soon, sleep the
|
||||
remainder. Cross-process because each `send-email` is a separate invocation.
|
||||
"""
|
||||
if min_interval <= 0:
|
||||
return
|
||||
p = state_path or _rate_limit_path()
|
||||
last = 0.0
|
||||
try:
|
||||
last = float(json.loads(p.read_text(encoding="utf-8")).get("last", 0.0))
|
||||
except (OSError, ValueError, TypeError):
|
||||
last = 0.0
|
||||
wait = min_interval - (now() - last)
|
||||
if wait > 0:
|
||||
sleep(min(wait, min_interval))
|
||||
try:
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(json.dumps({"last": now()}), encoding="utf-8")
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
# SMTP errors that are permanent (don't retry) vs transient (retry with backoff).
|
||||
_SMTP_PERMANENT = (smtplib.SMTPAuthenticationError, smtplib.SMTPRecipientsRefused,
|
||||
smtplib.SMTPSenderRefused, smtplib.SMTPDataError)
|
||||
|
||||
|
||||
def send(broker: dict, body_text: str, to: str | None = None,
|
||||
env: dict | None = None, _smtp_factory=None,
|
||||
min_interval: float = 0.0, max_retries: int = 3,
|
||||
_sleep=time.sleep, _now=time.time, _rate_state=None) -> dict:
|
||||
"""Send an opt-out/legal request to the broker's own opt-out address.
|
||||
|
||||
Recipient is locked to an address the broker record declares (PermissionError
|
||||
otherwise). `min_interval` paces sends across invocations (deliverability /
|
||||
account-safety); transient SMTP/socket failures retry with exponential backoff,
|
||||
permanent ones (auth, recipient refused) raise immediately. NOTE: a successful
|
||||
SMTP handoff is NOT proof of delivery - real bounces arrive later as inbound mail;
|
||||
in programmatic mode `poll-verification`/inbox review surfaces them, and the
|
||||
due-queue re-scan is the true confirmation. Returns send metadata.
|
||||
"""
|
||||
settings = smtp_settings(env)
|
||||
if not settings:
|
||||
raise RuntimeError(
|
||||
"programmatic email not configured (need EMAIL_ADDRESS + EMAIL_PASSWORD, and "
|
||||
"EMAIL_SMTP_HOST for non-mainstream providers); fall back to `render-email` drafts"
|
||||
)
|
||||
allowed = broker_addresses(broker)
|
||||
if not allowed:
|
||||
raise RuntimeError(f"broker {broker.get('id')!r} declares no opt-out email address")
|
||||
recipient = to or allowed[0]
|
||||
if recipient.lower() not in {a.lower() for a in allowed}:
|
||||
raise PermissionError(
|
||||
f"refusing to send to {recipient!r}: not an address the broker record declares "
|
||||
f"(allowed: {allowed})"
|
||||
)
|
||||
|
||||
subject, body = _split_subject_body(body_text)
|
||||
msg = EmailMessage()
|
||||
msg["From"] = settings["address"]
|
||||
msg["To"] = recipient
|
||||
msg["Subject"] = subject
|
||||
msg["Date"] = email.utils.formatdate(localtime=True)
|
||||
msg["Message-ID"] = email.utils.make_msgid()
|
||||
msg.set_content(body)
|
||||
|
||||
_respect_rate_limit(min_interval, _sleep, _now, _rate_state)
|
||||
|
||||
factory = _smtp_factory or smtplib.SMTP
|
||||
attempts = 0
|
||||
while True:
|
||||
attempts += 1
|
||||
try:
|
||||
with factory(settings["host"], settings["port"], timeout=30) as smtp:
|
||||
smtp.ehlo()
|
||||
try:
|
||||
smtp.starttls()
|
||||
smtp.ehlo()
|
||||
except smtplib.SMTPNotSupportedError:
|
||||
pass # already-TLS ports / test doubles
|
||||
smtp.login(settings["address"], settings["password"])
|
||||
smtp.send_message(msg)
|
||||
break
|
||||
except _SMTP_PERMANENT:
|
||||
raise # auth / recipient refused: retrying won't help
|
||||
except (smtplib.SMTPException, OSError) as exc:
|
||||
if attempts > max_retries:
|
||||
raise RuntimeError(f"SMTP send failed after {attempts} attempts: {exc}") from exc
|
||||
_sleep(min(2 ** (attempts - 1), 30)) # 1s, 2s, 4s... capped
|
||||
return {"to": recipient, "subject": subject, "message_id": msg["Message-ID"],
|
||||
"from": settings["address"], "attempts": attempts,
|
||||
"delivery_note": "SMTP accepted; not proof of delivery - a bounce would arrive as "
|
||||
"inbound mail. The due-queue re-scan is the real confirmation."}
|
||||
|
||||
|
||||
# --- inbox polling ------------------------------------------------------------
|
||||
|
||||
def _decode_part(part) -> str:
|
||||
try:
|
||||
payload = part.get_payload(decode=True)
|
||||
if payload is None:
|
||||
return ""
|
||||
charset = part.get_content_charset() or "utf-8"
|
||||
return payload.decode(charset, errors="replace")
|
||||
except Exception: # noqa: BLE001 - malformed MIME must not kill the poll
|
||||
return ""
|
||||
|
||||
|
||||
def message_text(msg) -> str:
|
||||
"""All text/plain + text/html content of a parsed email message."""
|
||||
chunks: list[str] = []
|
||||
if msg.is_multipart():
|
||||
for part in msg.walk():
|
||||
if part.get_content_type() in ("text/plain", "text/html"):
|
||||
chunks.append(_decode_part(part))
|
||||
else:
|
||||
chunks.append(_decode_part(msg))
|
||||
return "\n".join(c for c in chunks if c)
|
||||
|
||||
|
||||
def _broker_domains(broker: dict) -> list[str]:
|
||||
"""Domains this broker legitimately mails from (site domains + optout email domain)."""
|
||||
domains: list[str] = []
|
||||
for section in ("optout", "search"):
|
||||
url = ((broker.get(section) or {}).get("url")) or ""
|
||||
m = re.search(r"https?://([^/]+)", url)
|
||||
if m:
|
||||
domains.append(m.group(1).lower().removeprefix("www."))
|
||||
opt_email = (broker.get("optout") or {}).get("email")
|
||||
if opt_email and "@" in opt_email:
|
||||
domains.append(_domain(opt_email))
|
||||
# strip subdomains to the registrable-ish tail (mailer.intelius.com -> intelius.com)
|
||||
tails = {".".join(d.split(".")[-2:]) for d in domains if d}
|
||||
return sorted(tails)
|
||||
|
||||
|
||||
def fetch_recent(env: dict | None = None, since_days: int = 3, limit: int = 30,
|
||||
_imap_factory=None) -> list[dict]:
|
||||
"""Fetch recent inbox messages: [{from, subject, date, text}], newest first."""
|
||||
settings = imap_settings(env)
|
||||
if not settings:
|
||||
raise RuntimeError("IMAP not configured (need EMAIL_ADDRESS + EMAIL_PASSWORD, and "
|
||||
"EMAIL_IMAP_HOST for non-mainstream providers)")
|
||||
import datetime as _dt
|
||||
since = (_dt.date.today() - _dt.timedelta(days=max(0, since_days))).strftime("%d-%b-%Y")
|
||||
|
||||
factory = _imap_factory or imaplib.IMAP4_SSL
|
||||
conn = factory(settings["host"], settings["port"])
|
||||
try:
|
||||
conn.login(settings["address"], settings["password"])
|
||||
conn.select("INBOX", readonly=True)
|
||||
_typ, data = conn.search(None, "SINCE", since)
|
||||
ids = (data[0].split() if data and data[0] else [])[-limit:]
|
||||
out: list[dict] = []
|
||||
for mid in reversed(ids): # newest first
|
||||
_typ, msg_data = conn.fetch(mid, "(RFC822)")
|
||||
raw = next((p[1] for p in msg_data or [] if isinstance(p, tuple)), None)
|
||||
if not raw:
|
||||
continue
|
||||
msg = _email.message_from_bytes(raw)
|
||||
out.append({
|
||||
"from": msg.get("From", ""),
|
||||
"subject": msg.get("Subject", ""),
|
||||
"date": msg.get("Date", ""),
|
||||
"text": message_text(msg),
|
||||
})
|
||||
return out
|
||||
finally:
|
||||
try:
|
||||
conn.logout()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
|
||||
def link_from_messages(messages: list[dict], broker: dict) -> dict | None:
|
||||
"""Pure: find the broker's verification link in already-fetched messages.
|
||||
|
||||
A message is only considered if its From domain OR any contained link matches
|
||||
the broker's own domains; the link itself must pass the anti-phishing scorer.
|
||||
"""
|
||||
domains = _broker_domains(broker)
|
||||
for m in messages:
|
||||
sender = (m.get("from") or "").lower()
|
||||
text = m.get("text") or ""
|
||||
sender_match = any(d in sender for d in domains)
|
||||
body_match = any(d in text.lower() for d in domains)
|
||||
if not (sender_match or body_match):
|
||||
continue
|
||||
link = email_modes.extract_verification_link(text, broker)
|
||||
if link:
|
||||
return {"link": link, "from": m.get("from"), "subject": m.get("subject"),
|
||||
"date": m.get("date")}
|
||||
return None
|
||||
|
||||
|
||||
def find_verification_link(broker: dict, env: dict | None = None, since_days: int = 3,
|
||||
_imap_factory=None) -> dict | None:
|
||||
"""Poll the inbox and return the broker's verification link (or None yet)."""
|
||||
messages = fetch_recent(env, since_days=since_days, _imap_factory=_imap_factory)
|
||||
return link_from_messages(messages, broker)
|
||||
164
optional-skills/security/unbroker/scripts/ledger.py
Normal file
164
optional-skills/security/unbroker/scripts/ledger.py
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
"""Case ledger: opt-out state machine + append-only audit log.
|
||||
|
||||
A "case" is one (subject x broker) record. State changes are validated against
|
||||
TRANSITIONS and mirrored into audit.jsonl so every action is auditable.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
from pathlib import Path
|
||||
|
||||
import paths
|
||||
import storage
|
||||
|
||||
STATES = [
|
||||
"new", "searching", "not_found", "found", "indirect_exposure", "action_selected", "submitted",
|
||||
"verification_pending", "awaiting_processing", "confirmed_removed", "reappeared",
|
||||
"human_task_queued", "blocked",
|
||||
]
|
||||
|
||||
TRANSITIONS: dict[str, set[str]] = {
|
||||
"new": {"searching", "found", "not_found", "indirect_exposure", "blocked"},
|
||||
"searching": {"not_found", "found", "indirect_exposure", "blocked"},
|
||||
"not_found": {"searching", "found", "indirect_exposure", "blocked"},
|
||||
"found": {"action_selected", "submitted", "human_task_queued", "indirect_exposure", "blocked"},
|
||||
# indirect_exposure: subject's PII (email/phone/name) sits on a THIRD PARTY's record. The
|
||||
# self-service opt-out form does not apply; the lever is a targeted CCPA/GDPR delete-my-PII
|
||||
# request (-> submitted) or a human task. Re-scan can clear it (-> not_found) or upgrade it to a
|
||||
# direct listing (-> found).
|
||||
"indirect_exposure": {"submitted", "human_task_queued", "not_found", "found", "blocked"},
|
||||
"action_selected": {"submitted", "human_task_queued", "blocked"},
|
||||
"submitted": {"verification_pending", "awaiting_processing", "human_task_queued", "blocked"},
|
||||
# verification_pending -> awaiting_processing: the verify link was opened/acknowledged and the
|
||||
# broker is now processing the removal (their stated window). confirmed_removed still requires a
|
||||
# verifying re-scan, never the submission flow's own say-so.
|
||||
"verification_pending": {"awaiting_processing", "confirmed_removed", "human_task_queued", "blocked"},
|
||||
"awaiting_processing": {"confirmed_removed", "human_task_queued", "blocked"},
|
||||
"confirmed_removed": {"reappeared", "confirmed_removed"},
|
||||
"reappeared": {"found", "indirect_exposure"},
|
||||
"human_task_queued": {
|
||||
"found", "indirect_exposure", "action_selected", "submitted", "verification_pending",
|
||||
"awaiting_processing", "confirmed_removed", "blocked",
|
||||
},
|
||||
# blocked: automated tools (web_extract/proxyless browser) couldn't read the site. A later pass
|
||||
# -- a stealth/cloud browser OR guiding the operator's own (residential) browser -- can resolve it
|
||||
# to any real scan verdict, so blocked reaches not_found / indirect_exposure too, not just found.
|
||||
"blocked": {"searching", "found", "not_found", "indirect_exposure", "action_selected"},
|
||||
}
|
||||
|
||||
|
||||
def now() -> str:
|
||||
return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def load(subject_id: str) -> dict:
|
||||
return storage.read_json(paths.ledger_path(subject_id), {}) or {}
|
||||
|
||||
|
||||
def save(subject_id: str, ledger: dict) -> Path:
|
||||
return storage.write_json(paths.ledger_path(subject_id), ledger)
|
||||
|
||||
|
||||
def new_case(subject_id: str, broker_id: str) -> dict:
|
||||
return {
|
||||
"case_id": f"case_{subject_id}_{broker_id}",
|
||||
"subject_id": subject_id,
|
||||
"broker_id": broker_id,
|
||||
"state": "new",
|
||||
"found": None,
|
||||
"evidence": {},
|
||||
"disclosure_log": [],
|
||||
"history": [],
|
||||
}
|
||||
|
||||
|
||||
def get_case(subject_id: str, broker_id: str) -> dict:
|
||||
return load(subject_id).get(broker_id) or new_case(subject_id, broker_id)
|
||||
|
||||
|
||||
def can_transition(old: str, new: str) -> bool:
|
||||
return new == old or new in TRANSITIONS.get(old, set())
|
||||
|
||||
|
||||
def transition(subject_id: str, broker_id: str, new_state: str, **fields) -> dict:
|
||||
if new_state not in STATES:
|
||||
raise ValueError(f"unknown state {new_state!r}")
|
||||
# Lock the whole load-modify-save so a concurrent cron re-scan / other tenant
|
||||
# can't read a stale ledger and clobber this transition.
|
||||
with storage.locked(paths.ledger_path(subject_id)):
|
||||
ledger = load(subject_id)
|
||||
case = ledger.get(broker_id) or new_case(subject_id, broker_id)
|
||||
old = case.get("state", "new")
|
||||
if not can_transition(old, new_state):
|
||||
raise ValueError(f"illegal transition {old!r} -> {new_state!r} for broker {broker_id!r}")
|
||||
case["state"] = new_state
|
||||
for key, value in fields.items():
|
||||
case[key] = value
|
||||
stamp = now()
|
||||
case.setdefault("history", []).append({"at": stamp, "from": old, "to": new_state})
|
||||
ledger[broker_id] = case
|
||||
save(subject_id, ledger)
|
||||
storage.append_jsonl(
|
||||
paths.audit_path(subject_id),
|
||||
{"at": stamp, "broker_id": broker_id, "event": "transition", "from": old, "to": new_state},
|
||||
)
|
||||
return case
|
||||
|
||||
|
||||
DEFAULT_PROCESSING_DAYS = 14 # when a broker record doesn't state est_processing_days
|
||||
VERIFICATION_POLL_DAYS = 1 # how soon to re-poll for an unarrived verification email
|
||||
|
||||
|
||||
def _plus_days(days: int, start: str | None = None) -> str:
|
||||
base = _dt.datetime.strptime(start, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=_dt.timezone.utc) \
|
||||
if start else _dt.datetime.now(_dt.timezone.utc)
|
||||
return (base + _dt.timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def followup_fields(new_state: str, broker: dict | None = None,
|
||||
dossier: dict | None = None) -> dict:
|
||||
"""Auto-scheduling stamps for a transition, so nobody has to remember follow-ups.
|
||||
|
||||
submitted / awaiting_processing -> recheck after the broker's stated processing window;
|
||||
verification_pending -> re-poll the inbox quickly;
|
||||
confirmed_removed -> periodic reappearance re-scan per subject preference.
|
||||
"""
|
||||
if new_state in ("submitted", "awaiting_processing"):
|
||||
days = ((broker or {}).get("optout") or {}).get("est_processing_days") or DEFAULT_PROCESSING_DAYS
|
||||
return {"next_recheck_at": _plus_days(int(days))}
|
||||
if new_state == "verification_pending":
|
||||
return {"next_recheck_at": _plus_days(VERIFICATION_POLL_DAYS)}
|
||||
if new_state == "confirmed_removed":
|
||||
interval = ((dossier or {}).get("preferences") or {}).get("rescan_interval_days") or 120
|
||||
return {"removal_confirmed_at": now(), "next_recheck_at": _plus_days(int(interval))}
|
||||
return {}
|
||||
|
||||
|
||||
def due(subject_id: str, at: str | None = None, ledger: dict | None = None) -> list[dict]:
|
||||
"""Cases whose next_recheck_at has arrived - the autonomous follow-up queue."""
|
||||
stamp = at or now()
|
||||
out = []
|
||||
for case in (ledger if ledger is not None else load(subject_id)).values():
|
||||
when = case.get("next_recheck_at")
|
||||
if when and when <= stamp:
|
||||
out.append(case)
|
||||
out.sort(key=lambda c: c.get("next_recheck_at") or "")
|
||||
return out
|
||||
|
||||
|
||||
def log_disclosure(subject_id: str, broker_id: str, fields: list[str], channel: str) -> dict:
|
||||
"""Record exactly which PII field *names* were disclosed to a broker."""
|
||||
with storage.locked(paths.ledger_path(subject_id)):
|
||||
ledger = load(subject_id)
|
||||
case = ledger.get(broker_id) or new_case(subject_id, broker_id)
|
||||
stamp = now()
|
||||
record = {"at": stamp, "fields": sorted(fields), "channel": channel}
|
||||
case.setdefault("disclosure_log", []).append(record)
|
||||
ledger[broker_id] = case
|
||||
save(subject_id, ledger)
|
||||
storage.append_jsonl(
|
||||
paths.audit_path(subject_id),
|
||||
{"at": stamp, "broker_id": broker_id, "event": "disclosure",
|
||||
"fields": record["fields"], "channel": channel},
|
||||
)
|
||||
return record
|
||||
63
optional-skills/security/unbroker/scripts/legal.py
Normal file
63
optional-skills/security/unbroker/scripts/legal.py
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
"""Render opt-out / legal request text from templates/ with safe substitution.
|
||||
|
||||
Templates use {field} placeholders. Missing fields are left literal (never crash,
|
||||
never inject blanks that look like real data). Field values come from the
|
||||
least-disclosure selection in dossier.select_disclosure.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import paths
|
||||
|
||||
|
||||
class _SafeDict(dict):
|
||||
def __missing__(self, key): # leave unknown placeholders untouched
|
||||
return "{" + key + "}"
|
||||
|
||||
|
||||
def template_path(name: str) -> Path:
|
||||
return paths.templates_dir() / name
|
||||
|
||||
|
||||
def render(template_name: str, fields: dict) -> str:
|
||||
text = template_path(template_name).read_text(encoding="utf-8")
|
||||
return text.format_map(_SafeDict(fields))
|
||||
|
||||
|
||||
def _join_listings(value) -> str:
|
||||
if isinstance(value, (list, tuple)):
|
||||
return "\n".join(str(v) for v in value)
|
||||
return str(value or "")
|
||||
|
||||
|
||||
def _join_identifiers(value) -> str:
|
||||
"""Render the subject's OWN identifiers as a bullet list for an indirect-exposure request."""
|
||||
if isinstance(value, (list, tuple)):
|
||||
return "\n".join(f" - {v}" for v in value if v)
|
||||
return f" - {value}" if value else ""
|
||||
|
||||
|
||||
def render_optout_email(broker: dict, fields: dict) -> str:
|
||||
ctx = dict(fields)
|
||||
ctx.setdefault("broker_name", broker.get("name", "the data broker"))
|
||||
ctx["listing_urls"] = _join_listings(fields.get("listing_urls"))
|
||||
ctx.setdefault("full_name", fields.get("full_name", "[your name]"))
|
||||
ctx.setdefault("contact_email", fields.get("contact_email", "[your email]"))
|
||||
return render("emails/generic-optout.txt", ctx)
|
||||
|
||||
|
||||
def render_request(kind: str, broker: dict, fields: dict) -> str:
|
||||
"""kind: generic | ccpa | ccpa_agent | ccpa_indirect | gdpr"""
|
||||
template = {
|
||||
"generic": "emails/generic-optout.txt",
|
||||
"ccpa": "emails/ccpa-deletion.txt",
|
||||
"ccpa_agent": "emails/ccpa-authorized-agent.txt",
|
||||
"ccpa_indirect": "emails/ccpa-indirect-deletion.txt",
|
||||
"gdpr": "emails/gdpr-erasure.txt",
|
||||
}.get(kind, "emails/generic-optout.txt")
|
||||
ctx = dict(fields)
|
||||
ctx.setdefault("broker_name", broker.get("name", "the data broker"))
|
||||
ctx["listing_urls"] = _join_listings(fields.get("listing_urls"))
|
||||
ctx["my_identifiers"] = _join_identifiers(fields.get("my_identifiers"))
|
||||
return render(template, ctx)
|
||||
79
optional-skills/security/unbroker/scripts/paths.py
Normal file
79
optional-skills/security/unbroker/scripts/paths.py
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
"""Filesystem paths for the unbroker skill (stdlib only).
|
||||
|
||||
All per-subject data lives under PDD_DATA_DIR (default: $HERMES_HOME/unbroker),
|
||||
which is the same trust boundary Hermes uses for .env and OAuth tokens.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def hermes_home() -> Path:
|
||||
return Path(os.environ.get("HERMES_HOME") or (Path.home() / ".hermes"))
|
||||
|
||||
|
||||
def data_dir() -> Path:
|
||||
override = os.environ.get("PDD_DATA_DIR")
|
||||
return Path(override) if override else hermes_home() / "unbroker"
|
||||
|
||||
|
||||
def config_path() -> Path:
|
||||
return data_dir() / "config.json"
|
||||
|
||||
|
||||
def subjects_dir() -> Path:
|
||||
return data_dir() / "subjects"
|
||||
|
||||
|
||||
def subject_dir(subject_id: str) -> Path:
|
||||
return subjects_dir() / subject_id
|
||||
|
||||
|
||||
def dossier_path(subject_id: str) -> Path:
|
||||
return subject_dir(subject_id) / "dossier.json"
|
||||
|
||||
|
||||
def ledger_path(subject_id: str) -> Path:
|
||||
return subject_dir(subject_id) / "ledger.json"
|
||||
|
||||
|
||||
def audit_path(subject_id: str) -> Path:
|
||||
return subject_dir(subject_id) / "audit.jsonl"
|
||||
|
||||
|
||||
def evidence_dir(subject_id: str) -> Path:
|
||||
return subject_dir(subject_id) / "evidence"
|
||||
|
||||
|
||||
def skill_root() -> Path:
|
||||
"""The skill directory (parent of scripts/)."""
|
||||
return Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def brokers_dir() -> Path:
|
||||
return skill_root() / "references" / "brokers"
|
||||
|
||||
|
||||
def brokers_cache_path() -> Path:
|
||||
"""Live broker snapshot pulled from BADBOOL (merged under the curated DB)."""
|
||||
return data_dir() / "brokers-cache" / "badbool.json"
|
||||
|
||||
|
||||
def registry_cache_path() -> Path:
|
||||
"""CA Data Broker Registry snapshot (separate coverage lane; DROP/email, not scanned)."""
|
||||
return data_dir() / "brokers-cache" / "ca-registry.json"
|
||||
|
||||
|
||||
def age_identity_path() -> Path:
|
||||
"""age identity (private key) used for at-rest encryption when enabled.
|
||||
|
||||
Defaults beside the data; point PDD_AGE_IDENTITY at a separate volume/token
|
||||
for real key separation from the encrypted data.
|
||||
"""
|
||||
override = os.environ.get("PDD_AGE_IDENTITY")
|
||||
return Path(override) if override else data_dir() / "age-identity.txt"
|
||||
|
||||
|
||||
def templates_dir() -> Path:
|
||||
return skill_root() / "templates"
|
||||
810
optional-skills/security/unbroker/scripts/pdd.py
Normal file
810
optional-skills/security/unbroker/scripts/pdd.py
Normal file
|
|
@ -0,0 +1,810 @@
|
|||
#!/usr/bin/env python3
|
||||
"""unbroker - deterministic CLI helper.
|
||||
|
||||
The Hermes agent orchestrates scanning and opt-out submission with native tools
|
||||
(`web_extract`, `browser_navigate`, email mechanisms). THIS CLI owns the
|
||||
deterministic state: config, dossiers + consent, the broker DB, tier planning,
|
||||
the ledger + audit log, draft/template rendering, and reports.
|
||||
|
||||
Run it through the `terminal` tool (it can read PII files under HERMES_HOME);
|
||||
do NOT run it through `execute_code` (that sandbox scrubs env and redacts output).
|
||||
|
||||
Examples:
|
||||
python pdd.py setup
|
||||
python pdd.py intake --full-name "Jane Q. Public" --email jane@example.com \
|
||||
--city Oakland --state CA --residency US-CA --consent --consent-method self
|
||||
python pdd.py plan sub_xxxx --priority crucial
|
||||
python pdd.py record sub_xxxx spokeo found --found true \
|
||||
--evidence '{"listing_urls":["https://www.spokeo.com/..."]}'
|
||||
python pdd.py render-email sub_xxxx spokeo --listing https://www.spokeo.com/...
|
||||
python pdd.py status sub_xxxx
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
import autopilot # noqa: E402
|
||||
import badbool # noqa: E402
|
||||
import brokers as brokers_mod # noqa: E402
|
||||
import config as config_mod # noqa: E402
|
||||
import crypto # noqa: E402
|
||||
import dossier as dossier_mod # noqa: E402
|
||||
import email_modes # noqa: E402
|
||||
import emailer # noqa: E402
|
||||
import ledger as ledger_mod # noqa: E402
|
||||
import legal # noqa: E402
|
||||
import paths as paths_mod # noqa: E402
|
||||
import registry # noqa: E402
|
||||
import report as report_mod # noqa: E402
|
||||
import tiers # noqa: E402
|
||||
|
||||
|
||||
def _out(obj) -> None:
|
||||
print(json.dumps(obj, indent=2, ensure_ascii=False))
|
||||
|
||||
|
||||
def _require_subject(subject_id: str) -> dict:
|
||||
d = dossier_mod.load(subject_id)
|
||||
if not d:
|
||||
sys.exit(f"error: unknown subject {subject_id!r} (run `intake` first)")
|
||||
return d
|
||||
|
||||
|
||||
def cmd_setup(args) -> None:
|
||||
if getattr(args, "auto", False):
|
||||
# Autonomous path: detect capabilities and pick the most autonomous valid
|
||||
# config without asking anyone. Explicit flags still win below.
|
||||
cfg = config_mod.auto_configure()
|
||||
else:
|
||||
cfg = config_mod.load_config()
|
||||
for key in ("autonomy", "email_mode", "browser_backend", "tracker_backend", "encryption"):
|
||||
val = getattr(args, key)
|
||||
if val:
|
||||
cfg[key] = val
|
||||
if cfg.get("encryption") == "age":
|
||||
if not crypto.age_available():
|
||||
sys.exit("error: encryption=age requested but `age`/`age-keygen` not found. "
|
||||
"Install age (e.g. `brew install age`) or use `--encryption none`.")
|
||||
crypto.ensure_identity() # generate the key now so encryption is actually engaged
|
||||
path = config_mod.save_config(cfg)
|
||||
migrated = _migrate_subjects() # rewrite existing dossiers/ledgers into the new at-rest format
|
||||
out = {
|
||||
"config_path": str(path),
|
||||
"config": cfg,
|
||||
"encryption_engaged": crypto.is_engaged(),
|
||||
"detected_upgrades": config_mod.detect_capabilities(),
|
||||
"migrated_subjects": migrated,
|
||||
"note": "Defaults are easiest-first (draft email, auto browser, local tracker, no encryption). "
|
||||
"Pass flags to opt into upgrades, then run `doctor` for a readiness summary.",
|
||||
}
|
||||
if cfg.get("encryption") == "age":
|
||||
out["age_identity"] = str(crypto.identity_path())
|
||||
_out(out)
|
||||
|
||||
|
||||
def _migrate_subjects() -> int:
|
||||
"""Re-save each subject's dossier + ledger so they match the current at-rest format."""
|
||||
sd = paths_mod.subjects_dir()
|
||||
if not sd.exists():
|
||||
return 0
|
||||
n = 0
|
||||
for child in sorted(sd.iterdir()):
|
||||
if not child.is_dir():
|
||||
continue
|
||||
sid = child.name
|
||||
d = dossier_mod.load(sid)
|
||||
if d is not None:
|
||||
dossier_mod.save(d)
|
||||
n += 1
|
||||
led = ledger_mod.load(sid)
|
||||
if led:
|
||||
ledger_mod.save(sid, led)
|
||||
return n
|
||||
|
||||
|
||||
def _check_writable(path) -> bool:
|
||||
try:
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
probe = path / ".write_test"
|
||||
probe.write_text("x", encoding="utf-8")
|
||||
probe.unlink()
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def cmd_doctor(args) -> None:
|
||||
import platform
|
||||
|
||||
cfg = config_mod.load_config()
|
||||
caps = config_mod.detect_capabilities()
|
||||
data = paths_mod.data_dir()
|
||||
writable = _check_writable(data)
|
||||
curated = len(brokers_mod._load_curated())
|
||||
live = len(brokers_mod.load_live_cache())
|
||||
total = len(brokers_mod.load_all())
|
||||
|
||||
L = ["unbroker - readiness check", "=" * 42,
|
||||
f"Python : {platform.python_version()}",
|
||||
f"Data dir : {data} ({'writable' if writable else 'NOT writable'})",
|
||||
f"Config : autonomy={cfg.get('autonomy', 'full')} email={cfg['email_mode']} "
|
||||
f"browser={cfg['browser_backend']} "
|
||||
f"tracker={cfg['tracker_backend']} encryption={cfg['encryption']}",
|
||||
f"Brokers : {total} available ({curated} curated + {live} live"
|
||||
+ ("" if live else ", run `refresh-brokers` to expand to ~50") + ")",
|
||||
"", "Opt-in upgrades:"]
|
||||
rows = [
|
||||
("Cloud browser (Browserbase) *RECOMMENDED*", caps["browserbase"],
|
||||
"default backend: clears soft CAPTCHAs (Turnstile/hCaptcha) -> more T1", "set BROWSERBASE_API_KEY"),
|
||||
("Email auto (AgentMail)", caps["agentmail"],
|
||||
"send + auto-verify, per-broker aliases (Mode B/C)", "install agentmail skill / set AGENTMAIL_API_KEY"),
|
||||
("Email send (CLI SMTP)", caps["smtp_send"],
|
||||
"`send-email` delivers opt-outs itself (Mode B)", "set EMAIL_ADDRESS / EMAIL_PASSWORD (+ EMAIL_SMTP_HOST)"),
|
||||
("Verify-link poll (CLI IMAP)", caps["imap_read"],
|
||||
"`poll-verification` reads confirmation links itself", "set EMAIL_ADDRESS / EMAIL_PASSWORD (+ EMAIL_IMAP_HOST)"),
|
||||
("Google Sheets tracker", caps["google_workspace"],
|
||||
"shared status dashboard", "set up the google-workspace skill"),
|
||||
]
|
||||
for name, ok, enables, how in rows:
|
||||
L.append(f" [{'ON ' if ok else 'off'}] {name:<28} {enables}")
|
||||
if not ok:
|
||||
L.append(f" enable: {how}")
|
||||
|
||||
# At-rest encryption: report TRUE engagement (configured + key present), not just binary presence.
|
||||
engaged = crypto.is_engaged()
|
||||
L.append(f" [{'ON ' if engaged else 'off'}] {'At-rest encryption (age)':<28} "
|
||||
"encrypts dossiers + ledgers on disk")
|
||||
if engaged:
|
||||
L.append(f" key: {crypto.identity_path()} (0600) - guards casual/backup/commit "
|
||||
"exposure, NOT a full-HERMES_HOME read")
|
||||
elif cfg["encryption"] == "age":
|
||||
L.append(" WARNING: encryption=age is SET but NOT engaged (age binary or key missing);"
|
||||
" dossiers would be PLAINTEXT")
|
||||
elif caps["age"]:
|
||||
L.append(" off - dossiers are plaintext (0600). enable: `setup --encryption age`")
|
||||
else:
|
||||
L.append(" off - dossiers are plaintext (0600). install `age` first to enable")
|
||||
|
||||
L += ["", "Verdict:", " Ready now in DRAFT mode (no setup needed): scan brokers, draft opt-out",
|
||||
" emails for you to send, and track everything in the ledger."]
|
||||
if caps["browserbase"]:
|
||||
L.append(" Cloud browser ON (recommended default): soft/managed CAPTCHAs "
|
||||
"(Turnstile/hCaptcha) clear automatically -> those brokers stay T1.")
|
||||
else:
|
||||
L.append(" No cloud browser: set BROWSERBASE_API_KEY (the recommended default) so soft "
|
||||
"CAPTCHAs clear automatically; without it those brokers drop to T2 (human tasks).")
|
||||
if cfg["email_mode"] == "draft_only":
|
||||
L.append(" Email is draft-only: you send drafts + click verify links. For hands-off email "
|
||||
"WITHOUT storing a password, run `setup --email-mode browser` (agent sends + opens "
|
||||
"verify links via your logged-in webmail); or set EMAIL_* for SMTP/IMAP.")
|
||||
elif cfg["email_mode"] == "browser":
|
||||
L.append(" Email mode: browser (no password) - the agent sends opt-outs and opens verify "
|
||||
"links via the operator's logged-in webmail. Ensure that inbox is signed in in the "
|
||||
"browser Hermes uses (a cloud browser won't hold the session); else it falls back to drafts.")
|
||||
if not crypto.is_engaged():
|
||||
L.append(" Storage: dossiers are PLAINTEXT JSON (0600 under HERMES_HOME). "
|
||||
"Run `setup --encryption age` for at-rest encryption.")
|
||||
if not live:
|
||||
L.append(" Next: run `refresh-brokers` to load the full broker list.")
|
||||
|
||||
# Freshness: warn when cached lists / curated mechanics are going stale (silent broker rot).
|
||||
import time as _time
|
||||
STALE_CACHE_DAYS, STALE_VERIFY_DAYS = 30, 180
|
||||
|
||||
def _age_days(p) -> float | None:
|
||||
try:
|
||||
return (_time.time() - p.stat().st_mtime) / 86400.0
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
fresh = []
|
||||
for label, p in [("BADBOOL", paths_mod.brokers_cache_path()),
|
||||
("CA registry", paths_mod.registry_cache_path())]:
|
||||
age = _age_days(p)
|
||||
if age is None:
|
||||
fresh.append(f"{label}: not pulled")
|
||||
elif age > STALE_CACHE_DAYS:
|
||||
fresh.append(f"{label}: {age:.0f}d old (stale, re-pull)")
|
||||
stale_curated = documented = 0
|
||||
for b in brokers_mod._load_curated():
|
||||
conf = b.get("confidence")
|
||||
lv = b.get("last_verified")
|
||||
if conf == "documented" or not lv:
|
||||
documented += 1
|
||||
continue
|
||||
try:
|
||||
if (_time.time() - _time.mktime(_time.strptime(lv, "%Y-%m-%d"))) / 86400.0 > STALE_VERIFY_DAYS:
|
||||
stale_curated += 1
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
if fresh:
|
||||
L.append(" Freshness: " + "; ".join(fresh) + " (run `refresh-brokers`).")
|
||||
if stale_curated or documented:
|
||||
L.append(f" Freshness: {stale_curated} curated broker(s) last-verified >{STALE_VERIFY_DAYS}d ago; "
|
||||
f"{documented} documented broker(s) awaiting first-use verification.")
|
||||
print("\n".join(L))
|
||||
|
||||
|
||||
def cmd_intake(args) -> None:
|
||||
if args.json:
|
||||
data = json.loads(Path(args.json).read_text(encoding="utf-8"))
|
||||
identity = data["identity"]
|
||||
consent = data.get("consent", {})
|
||||
residency = data.get("residency_jurisdiction", "US")
|
||||
prefs = data.get("preferences")
|
||||
else:
|
||||
if not args.full_name:
|
||||
sys.exit("error: --full-name (or --json) is required")
|
||||
identity = {"full_name": args.full_name, "emails": args.email or [], "phones": args.phone or []}
|
||||
if args.alias:
|
||||
identity["also_known_as"] = args.alias
|
||||
if args.dob:
|
||||
identity["date_of_birth"] = args.dob
|
||||
addr = {k: v for k, v in {"line1": args.street, "city": args.city,
|
||||
"state": args.state, "postal": args.postal}.items() if v}
|
||||
if addr:
|
||||
identity["current_address"] = addr
|
||||
priors = []
|
||||
for loc in args.prior_location or []:
|
||||
parts = [p.strip() for p in loc.split(",") if p.strip()]
|
||||
if not parts:
|
||||
continue
|
||||
entry = {"city": parts[0]}
|
||||
if len(parts) > 1:
|
||||
entry["state"] = parts[1]
|
||||
if len(parts) > 2:
|
||||
entry["postal"] = parts[2]
|
||||
priors.append(entry)
|
||||
if priors:
|
||||
identity["prior_addresses"] = priors
|
||||
cfg = config_mod.load_config()
|
||||
consent = {"authorized": bool(args.consent), "method": args.consent_method, "recorded_at": dossier_mod.now()}
|
||||
residency = args.residency or "US"
|
||||
prefs = {
|
||||
"email_mode": args.email_mode or cfg["email_mode"],
|
||||
"rescan_interval_days": cfg["default_rescan_interval_days"],
|
||||
}
|
||||
if args.contact_email:
|
||||
prefs["contact_email_for_optouts"] = args.contact_email
|
||||
d = dossier_mod.create(identity, consent, residency, prefs)
|
||||
_out({"subject_id": d["subject_id"], "authorized": dossier_mod.is_authorized(d),
|
||||
"residency": residency, "email_mode": (prefs or {}).get("email_mode"),
|
||||
"names": dossier_mod.all_names(d),
|
||||
"emails": len(d["identity"].get("emails") or []),
|
||||
"phones": len(d["identity"].get("phones") or []),
|
||||
"addresses": len(dossier_mod.all_addresses(d))})
|
||||
|
||||
|
||||
def cmd_brokers(args) -> None:
|
||||
bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all()
|
||||
_out([
|
||||
{"id": b.get("id"), "name": b.get("name"), "priority": b.get("priority"),
|
||||
"method": (b.get("optout") or {}).get("method"), "owns": b.get("owns") or [],
|
||||
"source": b.get("source"), "confidence": b.get("confidence", "curated")}
|
||||
for b in bl
|
||||
])
|
||||
|
||||
|
||||
def cmd_refresh_brokers(args) -> None:
|
||||
res = badbool.refresh(paths_mod.brokers_cache_path())
|
||||
curated_ids = {b["id"] for b in brokers_mod._load_curated()}
|
||||
new = [b["id"] for b in brokers_mod.load_live_cache() if b["id"] not in curated_ids]
|
||||
out = {**res, "curated": len(curated_ids), "new_from_live": len(new),
|
||||
"people_search_total": len(brokers_mod.load_all()),
|
||||
"note": "Live records have confidence=auto; verify their opt-out URL before acting."}
|
||||
if not getattr(args, "no_registry", False):
|
||||
try:
|
||||
reg = registry.refresh_all(paths_mod.registry_cache_path())
|
||||
out["registry"] = {"total": reg["total"], "sources": reg["sources"],
|
||||
"portals": reg["portals"],
|
||||
"note": "Coverage lane worked via the CA DROP one-shot + CCPA email, "
|
||||
"not the people-search scan. VT/OR/TX are search portals (no "
|
||||
"bulk export); CA is the superset. See `drop` and `registry`."}
|
||||
except Exception as exc: # noqa: BLE001 - registry pull is best-effort
|
||||
out["registry_error"] = str(exc)
|
||||
_out(out)
|
||||
|
||||
|
||||
def cmd_registry(args) -> None:
|
||||
recs = brokers_mod.load_registry_cache()
|
||||
if not recs:
|
||||
_out({"registered_brokers": 0,
|
||||
"note": "registry empty - run `refresh-brokers` (pulls the CA Data Broker Registry)"})
|
||||
return
|
||||
fcra = sum(1 for r in recs if (r.get("optout") or {}).get("fcra"))
|
||||
out = {"registered_brokers": len(recs), "fcra_regulated": fcra,
|
||||
"source": "CA Data Broker Registry (CPPA, 2025)", "drop_url": registry.DROP_URL,
|
||||
"other_state_portals": registry.portals()}
|
||||
if args.search:
|
||||
q = args.search.lower()
|
||||
hits = [r for r in recs if q in (r.get("name") or "").lower()
|
||||
or q in (r.get("id") or "") or q in ((r.get("optout") or {}).get("email") or "").lower()]
|
||||
out["matches"] = [{"id": r["id"], "name": r["name"],
|
||||
"email": (r.get("optout") or {}).get("email"),
|
||||
"url": (r.get("optout") or {}).get("url"),
|
||||
"fcra": (r.get("optout") or {}).get("fcra")} for r in hits[:args.limit]]
|
||||
out["match_count"] = len(hits)
|
||||
_out(out)
|
||||
|
||||
|
||||
def cmd_drop(args) -> None:
|
||||
"""The one-shot legal lever: CA DROP deletes from ALL registered brokers at once."""
|
||||
d = _require_subject(args.subject)
|
||||
dossier_mod.require_authorized(d)
|
||||
reg = brokers_mod.load_registry_cache()
|
||||
res = (d.get("residency_jurisdiction") or "US").upper()
|
||||
eligible = res.startswith("US-CA")
|
||||
if args.filed:
|
||||
prefs = d.setdefault("preferences", {})
|
||||
prefs["drop_filed_at"] = dossier_mod.now()
|
||||
dossier_mod.save(d)
|
||||
_out({"subject": args.subject, "drop_filed_at": prefs["drop_filed_at"],
|
||||
"note": "recorded; `next` will stop surfacing the DROP one-shot"})
|
||||
return
|
||||
_out({
|
||||
"subject": args.subject,
|
||||
"eligible": eligible,
|
||||
"residency": res,
|
||||
"drop_url": registry.DROP_URL,
|
||||
"covers_registered_brokers": len(reg),
|
||||
"steps": ([
|
||||
"Go to privacy.ca.gov/drop and create/verify a DROP account (CA resident).",
|
||||
"Submit ONE deletion request; it applies to EVERY registered data broker "
|
||||
f"({len(reg)} in the current registry). Brokers must process starting 2026-08-01.",
|
||||
"After filing, run `drop <subject> --filed` so the loop stops re-surfacing it.",
|
||||
] if eligible else [
|
||||
"DROP is a California mechanism; this subject's residency is not US-CA.",
|
||||
"Parity path for non-CA: work the people-search sites via `next`, and send targeted "
|
||||
"CCPA/GDPR deletion emails to registry brokers that hold this person's data "
|
||||
"(`registry --search`, then `send-email`).",
|
||||
]),
|
||||
"note": "DROP is the highest-leverage removal: one request covers the whole registry.",
|
||||
})
|
||||
|
||||
|
||||
def cmd_plan(args) -> None:
|
||||
d = _require_subject(args.subject)
|
||||
dossier_mod.require_authorized(d)
|
||||
cfg = config_mod.load_config()
|
||||
bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all()
|
||||
bcc = config_mod.browser_clears_captcha(cfg)
|
||||
if getattr(args, "batch", False):
|
||||
_out(tiers.batch_plan(d, bl, cfg, ledger_mod.load(args.subject), bcc))
|
||||
else:
|
||||
_out(tiers.plan(d, bl, cfg, bcc))
|
||||
|
||||
|
||||
def cmd_fanout(args) -> None:
|
||||
d = _require_subject(args.subject)
|
||||
dossier_mod.require_authorized(d)
|
||||
bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all()
|
||||
grouping = tiers.fanout(bl, batch_size=args.size)
|
||||
mode = "scan AND opt-out (operator authorized submissions)" if args.optout \
|
||||
else "READ-ONLY scan (submit nothing; reconnaissance only)"
|
||||
batches = []
|
||||
for i, ids in enumerate(grouping["batches"], 1):
|
||||
brief = (
|
||||
f"You are scan worker {i} of {len(grouping['batches'])} for the `unbroker` "
|
||||
f"skill. First load the `unbroker` skill and read its references/methods.md. "
|
||||
f"Subject id: {args.subject}. Handle ONLY these brokers: {', '.join(ids)}. "
|
||||
f"For EACH broker: read references/brokers/<id>.json; run EVERY search vector from "
|
||||
f"`pdd.py plan {args.subject}` (filtered to your brokers); build URLs from search.url_patterns "
|
||||
f"and heed url_format_quirks; a 404 is INCONCLUSIVE (rebuild/try the on-site search box), not "
|
||||
f"not_found; confirm the SUBJECT vs namesakes/relatives before recording; if search.antibot is "
|
||||
f"set and no stealth/cloud browser is available, record `blocked`. Record each outcome via "
|
||||
f"`pdd.py record {args.subject} <broker> <found|not_found|indirect_exposure|blocked> "
|
||||
f"--found <bool> --evidence '{{\"listing_urls\":[...]}}'`. Mode: {mode}. "
|
||||
f"Log any newly-discovered URL/format quirks into the broker JSON. "
|
||||
f"Return a concise structured per-broker report."
|
||||
)
|
||||
batches.append({"batch": i, "brokers": ids, "brief": brief})
|
||||
_out({
|
||||
"subject": args.subject,
|
||||
"broker_count": grouping["broker_count"],
|
||||
"batch_size": grouping["batch_size"],
|
||||
"should_fanout": grouping["should_fanout"],
|
||||
"batch_count": len(batches),
|
||||
"batches": batches,
|
||||
"instruction": (
|
||||
"If should_fanout is true you MUST spawn ONE delegate_task subagent per batch IN PARALLEL, "
|
||||
"passing each batch's `brief`; do not scan all brokers yourself sequentially. Wait for every "
|
||||
"report, consolidate, then proceed to opt-outs. If false, just scan the brokers inline."
|
||||
),
|
||||
})
|
||||
|
||||
|
||||
def cmd_record(args) -> None:
|
||||
d = _require_subject(args.subject)
|
||||
dossier_mod.require_authorized(d)
|
||||
broker = brokers_mod.get(args.broker)
|
||||
# Auto-stamp follow-up scheduling (next_recheck_at / removal_confirmed_at) so the
|
||||
# autonomous loop knows when to come back without anyone remembering to set it.
|
||||
fields = ledger_mod.followup_fields(args.state, broker, d)
|
||||
if args.found is not None:
|
||||
fields["found"] = args.found
|
||||
if args.evidence:
|
||||
fields["evidence"] = json.loads(args.evidence)
|
||||
if args.reason:
|
||||
fields["human_task_reason"] = args.reason
|
||||
case = ledger_mod.transition(args.subject, args.broker, args.state, **fields)
|
||||
if args.disclosed:
|
||||
ledger_mod.log_disclosure(args.subject, args.broker, args.disclosed, args.channel or "unknown")
|
||||
_out({"broker": args.broker, "state": case["state"],
|
||||
"next_recheck_at": case.get("next_recheck_at")})
|
||||
|
||||
|
||||
def _email_request(d: dict, b: dict, kind: str, listings, identifiers) -> tuple[dict, list[str]]:
|
||||
"""Least-disclosure (fields, disclosed_names) for an opt-out/legal email of KIND.
|
||||
|
||||
A removal letter must self-identify. Name + a contact email are already known to the
|
||||
broker (the name is displayed on the very listing being removed), so not extra exposure.
|
||||
"""
|
||||
fields = dossier_mod.select_disclosure(d, (b.get("optout") or {}).get("inputs", []))
|
||||
ident = d.get("identity", {})
|
||||
if ident.get("full_name"):
|
||||
fields.setdefault("full_name", ident["full_name"])
|
||||
fields.setdefault("contact_email", dossier_mod.contact_email(d) or "")
|
||||
if listings:
|
||||
fields["listing_urls"] = listings
|
||||
if kind == "ccpa_indirect":
|
||||
# Indirect exposure: name ONLY the subject's own identifiers to scrub from a third party's
|
||||
# record. Default to the contact email + the subject's name-as-relative if none specified.
|
||||
# The indirect template renders ONLY these placeholders; do not over-report disclosure with
|
||||
# unrelated dossier fields (phone/street/postal) that select_disclosure happened to populate.
|
||||
ids = list(identifiers or [])
|
||||
if not ids:
|
||||
ids = [contact for contact in [dossier_mod.contact_email(d)] if contact]
|
||||
ids.append(f'the name "{ident.get("full_name")}" where it appears as a relative/associated person')
|
||||
fields = {
|
||||
"full_name": fields.get("full_name"),
|
||||
"contact_email": fields.get("contact_email"),
|
||||
"listing_urls": fields.get("listing_urls"),
|
||||
"my_identifiers": ids,
|
||||
}
|
||||
return fields, ["contact_email", "full_name", "my_identifiers"]
|
||||
return fields, sorted(fields.keys())
|
||||
|
||||
|
||||
def cmd_render_email(args) -> None:
|
||||
d = _require_subject(args.subject)
|
||||
dossier_mod.require_authorized(d)
|
||||
b = brokers_mod.get(args.broker)
|
||||
if not b:
|
||||
sys.exit(f"error: unknown broker {args.broker!r}")
|
||||
kind = getattr(args, "kind", "generic") or "generic"
|
||||
fields, disclosed = _email_request(d, b, kind, args.listing, getattr(args, "identifier", None))
|
||||
if kind == "generic":
|
||||
draft = email_modes.render_draft(b, fields)
|
||||
else:
|
||||
draft = email_modes.render_request_draft(b, fields, kind=kind)
|
||||
ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_draft:{kind}")
|
||||
_out({"draft": str(draft), "kind": kind, "disclosed_fields": disclosed})
|
||||
|
||||
|
||||
def cmd_send_email(args) -> None:
|
||||
"""Mode B: render AND deliver the opt-out/legal request - no human in the loop.
|
||||
|
||||
Sends ONLY to an address the broker record itself declares (emailer enforces it),
|
||||
then records the ledger transition + disclosure and auto-stamps the recheck date.
|
||||
"""
|
||||
d = _require_subject(args.subject)
|
||||
dossier_mod.require_authorized(d)
|
||||
b = brokers_mod.get(args.broker)
|
||||
if not b:
|
||||
sys.exit(f"error: unknown broker {args.broker!r}")
|
||||
cfg = config_mod.load_config()
|
||||
mode = cfg.get("email_mode")
|
||||
if mode not in ("programmatic", "alias", "browser"):
|
||||
sys.exit("error: email_mode is draft_only; run `setup --email-mode browser` (no password; "
|
||||
"sends via your logged-in webmail) or `--email-mode programmatic`, or use "
|
||||
"`render-email` and send it yourself")
|
||||
if not args.listing:
|
||||
sys.exit("error: --listing <confirmed-url> is required (verify-before-disclose: never "
|
||||
"email a broker about an unconfirmed listing)")
|
||||
# Idempotency: don't re-send if this case is already submitted/beyond (prevents duplicate
|
||||
# requests when an action is retried). --force overrides.
|
||||
_POST_SUBMIT = {"submitted", "verification_pending", "awaiting_processing", "confirmed_removed"}
|
||||
current = ledger_mod.get_case(args.subject, args.broker).get("state")
|
||||
if current in _POST_SUBMIT and not getattr(args, "force", False):
|
||||
_out({"skipped": True, "broker": args.broker, "state": current,
|
||||
"note": "already submitted; not re-sending (idempotent). Use --force to re-send."})
|
||||
return
|
||||
kind = getattr(args, "kind", "generic") or "generic"
|
||||
fields, disclosed = _email_request(d, b, kind, args.listing, getattr(args, "identifier", None))
|
||||
body = legal.render_optout_email(b, fields) if kind == "generic" else legal.render_request(kind, b, fields)
|
||||
|
||||
if mode == "browser":
|
||||
# No network / no credentials: hand the agent a recipient-locked payload to send in the
|
||||
# operator's webmail via browser_* tools. State still records deterministically here.
|
||||
payload = emailer.browser_send_payload(b, body, to=args.to)
|
||||
ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_browser:{kind}")
|
||||
case = ledger_mod.transition(args.subject, args.broker, "submitted",
|
||||
**ledger_mod.followup_fields("submitted", b, d))
|
||||
_out({"send_via": "browser", "compose": payload, "kind": kind, "disclosed_fields": disclosed,
|
||||
"state": case["state"], "next_recheck_at": case.get("next_recheck_at"),
|
||||
"instruction": "In the operator's logged-in webmail, compose a NEW email to compose.to "
|
||||
"with compose.subject/body EXACTLY (disclose nothing beyond it) and send "
|
||||
"it via browser_* tools. Then use `verify-link` on any confirmation reply.",
|
||||
"note": "recipient is locked to the broker's declared address"})
|
||||
return
|
||||
|
||||
result = emailer.send(b, body, to=args.to,
|
||||
min_interval=float(cfg.get("email_min_interval_seconds", 0) or 0))
|
||||
ledger_mod.log_disclosure(args.subject, args.broker, list(disclosed), f"email_sent:{kind}")
|
||||
case = ledger_mod.transition(args.subject, args.broker, "submitted",
|
||||
**ledger_mod.followup_fields("submitted", b, d))
|
||||
_out({"sent": result, "send_via": "smtp", "kind": kind, "disclosed_fields": disclosed,
|
||||
"state": case["state"], "next_recheck_at": case.get("next_recheck_at"),
|
||||
"note": "if this broker verifies by email, `poll-verification` will pick up the link"})
|
||||
|
||||
|
||||
def cmd_verify_link(args) -> None:
|
||||
"""Extract a broker's verification link from email text the agent read in webmail (browser mode).
|
||||
|
||||
IMAP-free counterpart to `poll-verification`: the agent opens the broker's confirmation email
|
||||
in the operator's webmail, pastes the body here, and gets the anti-phishing-scored link back.
|
||||
"""
|
||||
_require_subject(args.subject)
|
||||
b = brokers_mod.get(args.broker)
|
||||
if not b:
|
||||
sys.exit(f"error: unknown broker {args.broker!r}")
|
||||
text = args.text
|
||||
if args.file:
|
||||
text = Path(args.file).read_text(encoding="utf-8", errors="replace")
|
||||
if not text:
|
||||
sys.exit("error: provide --text '<email body>' (or --file) from the broker's confirmation email")
|
||||
link = email_modes.extract_verification_link(text, b)
|
||||
_out({"broker": args.broker, "verification_link": link,
|
||||
"next": ("browser_navigate the link IN THE SAME browser (sessions are browser-bound), "
|
||||
f"complete the flow, then `record {args.subject} {args.broker} awaiting_processing`"
|
||||
if link else
|
||||
"no broker/opt-out-scoped link found in that text; confirm you opened the right email")})
|
||||
|
||||
|
||||
def cmd_poll_verification(args) -> None:
|
||||
"""Poll the inbox for brokers' verification links (Mode B) - replaces the human click-chase.
|
||||
|
||||
For each in-flight case (submitted / verification_pending with email_verification),
|
||||
extract the broker's link (anti-phishing scored). A found link auto-advances
|
||||
submitted -> verification_pending (the email HAS arrived); the agent must then OPEN
|
||||
the link in its own browser (sessions are browser-bound) and record the next state.
|
||||
"""
|
||||
d = _require_subject(args.subject)
|
||||
dossier_mod.require_authorized(d)
|
||||
led = ledger_mod.load(args.subject)
|
||||
targets = []
|
||||
for bid, case in sorted(led.items()):
|
||||
if args.broker and bid != args.broker:
|
||||
continue
|
||||
if case.get("state") not in ("submitted", "verification_pending"):
|
||||
continue
|
||||
b = brokers_mod.get(bid)
|
||||
if b and (((b.get("optout") or {}).get("requires")) or {}).get("email_verification"):
|
||||
targets.append((bid, case, b))
|
||||
if not targets:
|
||||
_out({"subject": args.subject, "results": [],
|
||||
"note": "no in-flight cases awaiting email verification"})
|
||||
return
|
||||
results = []
|
||||
for bid, case, b in targets:
|
||||
hit = emailer.find_verification_link(b, since_days=args.since_days)
|
||||
if hit:
|
||||
if case.get("state") == "submitted":
|
||||
ledger_mod.transition(args.subject, bid, "verification_pending",
|
||||
**ledger_mod.followup_fields("verification_pending", b, d))
|
||||
results.append({"broker": bid, "verification_link": hit["link"],
|
||||
"email_from": hit.get("from"), "email_subject": hit.get("subject"),
|
||||
"next": f"browser_navigate the link IN THE AGENT'S OWN BROWSER, complete "
|
||||
f"the flow, then `record {args.subject} {bid} awaiting_processing` "
|
||||
f"(or confirmed_removed only after a verifying re-scan)"})
|
||||
else:
|
||||
results.append({"broker": bid, "verification_link": None,
|
||||
"next": "no matching email yet; poll again later (next_recheck_at is set)"})
|
||||
_out({"subject": args.subject, "results": results})
|
||||
|
||||
|
||||
def cmd_next(args) -> None:
|
||||
d = _require_subject(args.subject)
|
||||
dossier_mod.require_authorized(d)
|
||||
cfg = config_mod.load_config()
|
||||
bl = brokers_mod.by_priority(*(args.priority or [])) if args.priority else brokers_mod.load_all()
|
||||
_out(autopilot.next_actions(d, bl, cfg, ledger_mod.load(args.subject)))
|
||||
|
||||
|
||||
def cmd_tasks(args) -> None:
|
||||
_require_subject(args.subject)
|
||||
print(report_mod.human_tasks_markdown(args.subject))
|
||||
|
||||
|
||||
def cmd_due(args) -> None:
|
||||
_require_subject(args.subject)
|
||||
cases = ledger_mod.due(args.subject)
|
||||
_out({"subject": args.subject, "due_count": len(cases),
|
||||
"cases": [{"broker_id": c.get("broker_id"), "state": c.get("state"),
|
||||
"next_recheck_at": c.get("next_recheck_at")} for c in cases],
|
||||
"note": "run `next` for the concrete follow-up action per case"})
|
||||
|
||||
|
||||
def cmd_status(args) -> None:
|
||||
_require_subject(args.subject)
|
||||
print(report_mod.render_markdown(args.subject))
|
||||
|
||||
|
||||
def cmd_report(args) -> None:
|
||||
_require_subject(args.subject)
|
||||
if args.sheets:
|
||||
_out(report_mod.sheets_rows(args.subject))
|
||||
else:
|
||||
print(report_mod.render_markdown(args.subject))
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(prog="pdd", description="unbroker helper CLI")
|
||||
sub = p.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
s = sub.add_parser("setup", help="write install config (easiest-first defaults; --auto = most autonomous)")
|
||||
s.add_argument("--auto", action="store_true",
|
||||
help="detect capabilities and pick the most autonomous valid config (no questions)")
|
||||
s.add_argument("--autonomy", dest="autonomy", choices=sorted(config_mod.VALID["autonomy"]))
|
||||
s.add_argument("--email-mode", dest="email_mode", choices=sorted(config_mod.VALID["email_mode"]))
|
||||
s.add_argument("--browser-backend", dest="browser_backend", choices=sorted(config_mod.VALID["browser_backend"]))
|
||||
s.add_argument("--tracker-backend", dest="tracker_backend", choices=sorted(config_mod.VALID["tracker_backend"]))
|
||||
s.add_argument("--encryption", dest="encryption", choices=sorted(config_mod.VALID["encryption"]))
|
||||
s.set_defaults(func=cmd_setup)
|
||||
|
||||
s = sub.add_parser("doctor", help="readiness check: config, brokers, available upgrades")
|
||||
s.set_defaults(func=cmd_doctor)
|
||||
|
||||
s = sub.add_parser("intake", help="create a subject dossier (records consent)")
|
||||
s.add_argument("--json", help="path to a dossier JSON file (overrides flags)")
|
||||
s.add_argument("--full-name")
|
||||
s.add_argument("--alias", action="append", metavar="NAME",
|
||||
help="other name the subject is listed under (maiden/married/nickname); repeatable")
|
||||
s.add_argument("--email", action="append", metavar="EMAIL", help="repeatable")
|
||||
s.add_argument("--phone", action="append", metavar="PHONE", help="repeatable")
|
||||
s.add_argument("--street", help="current street line1 (enables reverse-address search)")
|
||||
s.add_argument("--city")
|
||||
s.add_argument("--state")
|
||||
s.add_argument("--postal")
|
||||
s.add_argument("--prior-location", dest="prior_location", action="append", metavar="City,ST",
|
||||
help="a past city/state (or City,ST,ZIP); repeatable")
|
||||
s.add_argument("--dob", help="date of birth YYYY-MM-DD (only used if a broker requires it)")
|
||||
s.add_argument("--contact-email", dest="contact_email",
|
||||
help="which email to use for opt-out correspondence (default: first)")
|
||||
s.add_argument("--residency", help="e.g. US, US-CA")
|
||||
s.add_argument("--consent", action="store_true", help="subject authorizes removal on their behalf")
|
||||
s.add_argument("--consent-method", default="self", choices=["self", "written_authorization", "poa"])
|
||||
s.add_argument("--email-mode", dest="email_mode", choices=sorted(config_mod.VALID["email_mode"]))
|
||||
s.set_defaults(func=cmd_intake)
|
||||
|
||||
s = sub.add_parser("brokers", help="list the broker database (curated + live)")
|
||||
s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"])
|
||||
s.set_defaults(func=cmd_brokers)
|
||||
|
||||
s = sub.add_parser("refresh-brokers",
|
||||
help="pull the latest BADBOOL people-search list + the CA data broker registry")
|
||||
s.add_argument("--no-registry", dest="no_registry", action="store_true",
|
||||
help="skip the CA registry pull (BADBOOL people-search only)")
|
||||
s.set_defaults(func=cmd_refresh_brokers)
|
||||
|
||||
s = sub.add_parser("registry",
|
||||
help="CA Data Broker Registry coverage (hundreds of brokers; DROP/email lane)")
|
||||
s.add_argument("--search", help="find registered brokers by name / id / email substring")
|
||||
s.add_argument("--limit", type=int, default=25, help="max matches to print (default 25)")
|
||||
s.set_defaults(func=cmd_registry)
|
||||
|
||||
s = sub.add_parser("drop",
|
||||
help="CA DROP one-shot: delete from ALL registered brokers in one request")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("--filed", action="store_true", help="mark DROP as filed (stops `next` surfacing it)")
|
||||
s.set_defaults(func=cmd_drop)
|
||||
|
||||
s = sub.add_parser("plan", help="compute per-broker tier + next action for a subject")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"])
|
||||
s.add_argument("--batch", action="store_true",
|
||||
help="phase-oriented batch view: overlays ledger state, groups by next action "
|
||||
"(unscanned/found/indirect/blocked/in_progress/done), collapses ownership clusters")
|
||||
s.set_defaults(func=cmd_plan)
|
||||
|
||||
s = sub.add_parser("fanout", help="batch brokers into parallel delegate_task subagents (large runs)")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"])
|
||||
s.add_argument("--size", type=int, default=8, help="brokers per subagent batch (default 8)")
|
||||
s.add_argument("--optout", action="store_true",
|
||||
help="brief authorizes opt-out submission (default: read-only scan)")
|
||||
s.set_defaults(func=cmd_fanout)
|
||||
|
||||
s = sub.add_parser("record", help="record a ledger state transition after an agent action")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("broker")
|
||||
s.add_argument("state", choices=ledger_mod.STATES)
|
||||
s.add_argument("--found", type=lambda v: v.strip().lower() in ("1", "true", "yes", "y"))
|
||||
s.add_argument("--evidence", help="JSON object stored as case.evidence")
|
||||
s.add_argument("--disclosed", action="append", metavar="FIELD", help="field name disclosed")
|
||||
s.add_argument("--channel", help="disclosure channel, e.g. web_form / email")
|
||||
s.add_argument("--reason", help="for human_task_queued: why a human is needed (shown in `tasks`)")
|
||||
s.set_defaults(func=cmd_record)
|
||||
|
||||
s = sub.add_parser("next", help="autonomous action queue: exactly what to do right now")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("--priority", action="append", choices=["crucial", "high", "standard", "long_tail"])
|
||||
s.set_defaults(func=cmd_next)
|
||||
|
||||
s = sub.add_parser("send-email", help="Mode B: render AND send the opt-out/legal request (records it)")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("broker")
|
||||
s.add_argument("--listing", action="append", metavar="URL", required=False,
|
||||
help="confirmed listing URL (required: verify-before-disclose)")
|
||||
s.add_argument("--kind", choices=["generic", "ccpa", "ccpa_agent", "ccpa_indirect", "gdpr"],
|
||||
default="generic")
|
||||
s.add_argument("--identifier", action="append", metavar="ID",
|
||||
help="(ccpa_indirect only) a specific own-identifier to remove; repeatable")
|
||||
s.add_argument("--to", help="override recipient (must be an address the broker record declares)")
|
||||
s.add_argument("--force", action="store_true", help="re-send even if already submitted (default: idempotent skip)")
|
||||
s.set_defaults(func=cmd_send_email)
|
||||
|
||||
s = sub.add_parser("poll-verification",
|
||||
help="Mode B (IMAP): poll the inbox for brokers' verification links (anti-phishing scored)")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("--broker", help="only this broker (default: every in-flight verification case)")
|
||||
s.add_argument("--since-days", dest="since_days", type=int, default=3)
|
||||
s.set_defaults(func=cmd_poll_verification)
|
||||
|
||||
s = sub.add_parser("verify-link",
|
||||
help="browser mode: extract a broker's verification link from pasted webmail text")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("broker")
|
||||
s.add_argument("--text", help="the confirmation email body (read from the operator's webmail)")
|
||||
s.add_argument("--file", help="path to a file with the email body (alternative to --text)")
|
||||
s.set_defaults(func=cmd_verify_link)
|
||||
|
||||
s = sub.add_parser("tasks", help="ONE consolidated human-task digest (present at end of run)")
|
||||
s.add_argument("subject")
|
||||
s.set_defaults(func=cmd_tasks)
|
||||
|
||||
s = sub.add_parser("due", help="cases whose recheck window has arrived (cron re-scan queue)")
|
||||
s.add_argument("subject")
|
||||
s.set_defaults(func=cmd_due)
|
||||
|
||||
s = sub.add_parser("render-email", help="render a Mode-A opt-out / legal-request draft (least-disclosure)")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("broker")
|
||||
s.add_argument("--listing", action="append", metavar="URL", help="confirmed listing URL")
|
||||
s.add_argument("--kind", choices=["generic", "ccpa", "ccpa_agent", "ccpa_indirect", "gdpr"],
|
||||
default="generic",
|
||||
help="request type. 'ccpa_indirect' = delete MY identifiers from a third party's "
|
||||
"record (indirect exposure); default 'generic' opt-out.")
|
||||
s.add_argument("--identifier", action="append", metavar="ID",
|
||||
help="(ccpa_indirect only) a specific own-identifier to request removal of "
|
||||
"(e.g. an email or phone). Repeatable. Defaults to the contact email + "
|
||||
"name-as-relative if omitted.")
|
||||
s.set_defaults(func=cmd_render_email)
|
||||
|
||||
s = sub.add_parser("status", help="print a Markdown status report")
|
||||
s.add_argument("subject")
|
||||
s.set_defaults(func=cmd_status)
|
||||
|
||||
s = sub.add_parser("report", help="status report (default) or --sheets rows")
|
||||
s.add_argument("subject")
|
||||
s.add_argument("--sheets", action="store_true", help="emit Google Sheets rows as JSON")
|
||||
s.set_defaults(func=cmd_report)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv=None) -> None:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
args.func(args)
|
||||
except (PermissionError, ValueError, RuntimeError, FileNotFoundError) as exc:
|
||||
sys.exit(f"error: {exc}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
293
optional-skills/security/unbroker/scripts/registry.py
Normal file
293
optional-skills/security/unbroker/scripts/registry.py
Normal file
|
|
@ -0,0 +1,293 @@
|
|||
"""Ingest the California Data Broker Registry into broker records (coverage breadth).
|
||||
|
||||
The CA registry (CPPA, under the Delete Act) is the authoritative universe of data
|
||||
brokers doing business with California residents -- ~545 businesses in 2025, each
|
||||
required to publish a name, website, contact email, and a CCPA-rights/deletion URL.
|
||||
This is the same universe commercial services (DeleteMe/Incogni/Optery) draw from,
|
||||
plus the FCRA/GLBA-regulated and marketing/risk brokers most lists omit.
|
||||
|
||||
These are NOT people-search sites you scan with a name -- most have no per-person
|
||||
lookup UI. They are worked through the LEGAL lane: the CA DROP portal
|
||||
(privacy.ca.gov/drop) is a single request that deletes from ALL registered brokers
|
||||
at once (CA residents), and per-broker CCPA deletion emails to the contact address
|
||||
are the fallback / non-CA path. So registry records are kept in their own lane
|
||||
(loaded only when asked) and never dumped into the people-search scan pipeline.
|
||||
|
||||
`parse()` is pure (CSV text in, records out) so it is tested offline; `fetch()` is
|
||||
the only network call and can be bypassed by passing csv_text directly to refresh().
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import datetime
|
||||
import io
|
||||
import re
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import storage
|
||||
|
||||
# CA CPPA registry CSVs are published per year (registry2024.csv, registry2025.csv, ...).
|
||||
# 2025 is the latest COMPLETE dataset; the current year's file is empty until the Jan
|
||||
# registration window closes. DEFAULT_URL is the known-good fallback; `ca_candidate_urls`
|
||||
# probes newer years first so coverage auto-advances when the next year is published.
|
||||
_CA_CSV = "https://cppa.ca.gov/data_broker_registry/registry{year}.csv"
|
||||
_CA_FLOOR_YEAR = 2025
|
||||
DEFAULT_URL = _CA_CSV.format(year=_CA_FLOOR_YEAR)
|
||||
DROP_URL = "https://privacy.ca.gov/drop"
|
||||
USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)"
|
||||
|
||||
|
||||
def ca_candidate_urls(today: datetime.date | None = None) -> list[str]:
|
||||
"""Newest-year-first CA registry URLs to try (auto-advances; never below the 2025 floor)."""
|
||||
year = (today or datetime.date.today()).year
|
||||
years = list(range(max(year, _CA_FLOOR_YEAR), _CA_FLOOR_YEAR - 1, -1))
|
||||
return [_CA_CSV.format(year=y) for y in years]
|
||||
|
||||
# Multi-source registry lane. Only California publishes a clean bulk CSV (with contact email +
|
||||
# CCPA-rights URL per broker) AND offers a one-shot deletion portal (DROP). Vermont, Oregon, and
|
||||
# Texas maintain registries too, but only as searchable PORTALS (no reliable bulk export) and with
|
||||
# no DROP-equivalent -- and they overlap CA heavily (CA is effectively the superset). So they are
|
||||
# wired as first-class portal sources (official URL surfaced to the operator) rather than scraped.
|
||||
# Adding any state that later publishes a CSV is a one-line "format: csv" entry (the parser is
|
||||
# column-detection based, not CA-specific).
|
||||
SOURCES = {
|
||||
"ca": {"jurisdiction": "US-CA", "format": "csv", "url": DEFAULT_URL, "has_drop": True,
|
||||
"name": "California Data Broker Registry (CPPA)"},
|
||||
"vt": {"jurisdiction": "US-VT", "format": "portal", "has_drop": False,
|
||||
"url": "https://bizfilings.vermont.gov/online/DatabrokerInquire/",
|
||||
"name": "Vermont Data Broker Registry (Secretary of State)"},
|
||||
"or": {"jurisdiction": "US-OR", "format": "portal", "has_drop": False,
|
||||
"url": "https://dfr.oregon.gov/business/licensing/data-broker-registry/Pages/index.aspx",
|
||||
"name": "Oregon Data Broker Registry (DCBS)"},
|
||||
"tx": {"jurisdiction": "US-TX", "format": "portal", "has_drop": False,
|
||||
"url": "https://texas-sos.appianportalsgov.com/data-broker-registry",
|
||||
"name": "Texas Data Broker Registry (Secretary of State)"},
|
||||
}
|
||||
|
||||
|
||||
def portals() -> list[dict]:
|
||||
"""Registry sources that are searchable portals (no bulk export) -- surfaced to the operator."""
|
||||
return [{"key": k, "jurisdiction": s["jurisdiction"], "name": s["name"], "url": s["url"]}
|
||||
for k, s in SOURCES.items() if s["format"] == "portal"]
|
||||
|
||||
# Field label -> substring to locate its column on the header row (robust to
|
||||
# year-to-year column shifts; the registry re-orders/adds columns between years).
|
||||
_LABELS = {
|
||||
"name": "data broker name:",
|
||||
"dba": "doing business as",
|
||||
"website": "data broker primary website:",
|
||||
"email": "primary contact email",
|
||||
"rights_url": "exercise their ca consumer privacy act rights",
|
||||
"fcra": "regulated by the federal fair credit reporting act (fcra):",
|
||||
}
|
||||
|
||||
|
||||
def _norm(s: str) -> str:
|
||||
"""Registry CSVs use NBSPs and a BOM; normalize for matching + clean values."""
|
||||
return re.sub(r"\s+", " ", (s or "").replace("\ufeff", "").replace("\xa0", " ")).strip()
|
||||
|
||||
|
||||
def slug(name: str, website: str = "") -> str:
|
||||
base = re.sub(r"\.(com|org|net|io|ai|inc|co|us|info|llc)\b", "", (name or "").strip(), flags=re.I)
|
||||
s = re.sub(r"[^a-z0-9]+", "", base.lower())
|
||||
if s:
|
||||
return s
|
||||
dom = re.sub(r"^https?://(www\.)?", "", (website or "").lower())
|
||||
return re.sub(r"[^a-z0-9]+", "", dom.split("/")[0]) or "broker"
|
||||
|
||||
|
||||
def _domain(website: str) -> str:
|
||||
dom = re.sub(r"^https?://(www\.)?", "", (website or "").strip().lower())
|
||||
return dom.split("/")[0]
|
||||
|
||||
|
||||
def _find_colmap(rows: list[list[str]]) -> tuple[int, dict[str, int]]:
|
||||
"""Locate the label row (col0 == 'Data broker name:') and map fields to columns."""
|
||||
for i, row in enumerate(rows[:5]):
|
||||
if row and _norm(row[0]).lower().startswith("data broker name:"):
|
||||
colmap: dict[str, int] = {}
|
||||
for field, needle in _LABELS.items():
|
||||
for j, cell in enumerate(row):
|
||||
c = _norm(cell).lower()
|
||||
if needle in c and not c.startswith("if the data broker"):
|
||||
colmap[field] = j
|
||||
break
|
||||
return i, colmap
|
||||
raise ValueError("CA registry: could not locate the header row")
|
||||
|
||||
|
||||
def _get(row: list[str], idx: int | None) -> str:
|
||||
return _norm(row[idx]) if idx is not None and idx < len(row) else ""
|
||||
|
||||
|
||||
def _build(row: list[str], cm: dict[str, int], jurisdiction: str = "US-CA",
|
||||
has_drop: bool = True) -> dict | None:
|
||||
name = _get(row, cm.get("name"))
|
||||
website = _get(row, cm.get("website"))
|
||||
if not (name or website):
|
||||
return None
|
||||
email = _get(row, cm.get("email"))
|
||||
rights = _get(row, cm.get("rights_url"))
|
||||
dba = _get(row, cm.get("dba"))
|
||||
fcra = _get(row, cm.get("fcra")).lower().startswith("y")
|
||||
state = jurisdiction.split("-")[-1]
|
||||
|
||||
method = "email" if email else ("web_form" if rights else "drop")
|
||||
if has_drop:
|
||||
notes = ("Registered CA data broker. One CA DROP request (privacy.ca.gov/drop) deletes from "
|
||||
"this and every registered broker at once; or send a CCPA deletion request to the "
|
||||
"contact email.")
|
||||
else:
|
||||
notes = (f"Registered {state} data broker (no one-shot delete portal in {state}). Send a "
|
||||
"CCPA/state-law deletion request to the contact email.")
|
||||
if fcra:
|
||||
notes += (" FCRA-regulated: some data is credit-reporting data with separate rules -- deletion "
|
||||
"may be limited; a consumer report dispute/security-freeze may apply instead.")
|
||||
return {
|
||||
"id": slug(name, website),
|
||||
"name": name or _domain(website),
|
||||
"dba": dba or None,
|
||||
"category": "data_broker",
|
||||
"priority": "long_tail",
|
||||
"jurisdictions": [jurisdiction],
|
||||
"search": {"method": "none", "url": website, "fetch": "none", "by": ["registry"]},
|
||||
"optout": {
|
||||
"method": method,
|
||||
"url": rights or website or None,
|
||||
"email": email or None,
|
||||
"requires": {"profile_url": False, "email_verification": False, "captcha": False,
|
||||
"gov_id": False, "account": False, "phone_callback": False, "payment": False},
|
||||
"inputs": ["full_name", "contact_email"],
|
||||
"deletion": {
|
||||
"via": "drop" if has_drop else "email",
|
||||
"email": email or None,
|
||||
"url": rights or None,
|
||||
"kinds": ["ccpa", "generic"],
|
||||
"notes": ("Covered by the CA DROP one-shot (privacy.ca.gov/drop); CCPA email fallback."
|
||||
if has_drop else "CCPA/state-law deletion email (no one-shot portal)."),
|
||||
},
|
||||
"fcra": fcra,
|
||||
"est_processing_days": 45,
|
||||
"notes": notes,
|
||||
},
|
||||
"source": f"{state}-registry",
|
||||
"confidence": "registry",
|
||||
"last_verified": None,
|
||||
}
|
||||
|
||||
|
||||
def parse(csv_text: str, jurisdiction: str = "US-CA", has_drop: bool = True) -> list[dict]:
|
||||
"""Parse a data-broker-registry CSV into broker records (deduped by id).
|
||||
|
||||
Column detection is by header label, not fixed position, so any state that publishes a
|
||||
registry CSV with name/website/email/rights columns parses without new code.
|
||||
"""
|
||||
rows = list(csv.reader(io.StringIO(csv_text)))
|
||||
if not rows:
|
||||
return []
|
||||
header_i, cm = _find_colmap(rows)
|
||||
out: list[dict] = []
|
||||
seen: dict[str, int] = {}
|
||||
for row in rows[header_i + 1:]:
|
||||
if not any(c.strip() for c in row):
|
||||
continue
|
||||
rec = _build(row, cm, jurisdiction, has_drop)
|
||||
if not rec:
|
||||
continue
|
||||
bid = rec["id"]
|
||||
if bid in seen: # disambiguate id collisions by domain, then a counter
|
||||
dom = re.sub(r"[^a-z0-9]+", "", _domain(rec["search"]["url"]))
|
||||
cand = f"{bid}-{dom}" if dom and dom != bid else bid
|
||||
while cand in seen:
|
||||
seen[bid] += 1
|
||||
cand = f"{bid}-{seen[bid]}"
|
||||
rec["id"] = cand
|
||||
seen.setdefault(rec["id"], 0)
|
||||
seen.setdefault(bid, 0)
|
||||
out.append(rec)
|
||||
return out
|
||||
|
||||
|
||||
MIN_EXPECTED_CA = 100 # CA registry has ~500+; far fewer => wrong/empty file, warn
|
||||
|
||||
|
||||
def fetch(url: str = DEFAULT_URL, timeout: int = 60) -> str:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310
|
||||
return resp.read().decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def _fetch_ca_latest() -> tuple[str, list[dict]]:
|
||||
"""Try newest CA registry year first; return (url, records) for the first non-empty."""
|
||||
last: tuple[str, list[dict]] = (DEFAULT_URL, [])
|
||||
for url in ca_candidate_urls():
|
||||
try:
|
||||
recs = parse(fetch(url), jurisdiction="US-CA", has_drop=True)
|
||||
except Exception: # noqa: BLE001 - a missing year 404s; fall through to older years
|
||||
continue
|
||||
if recs:
|
||||
return url, recs
|
||||
last = (url, recs)
|
||||
return last
|
||||
|
||||
|
||||
def refresh(cache_path: Path, url: str = DEFAULT_URL, csv_text: str | None = None) -> dict:
|
||||
"""CA single-source refresh: fetch (or accept) the CA CSV and write the cache."""
|
||||
text = csv_text if csv_text is not None else fetch(url)
|
||||
records = parse(text)
|
||||
storage.write_json(cache_path, records)
|
||||
fcra = sum(1 for r in records if (r.get("optout") or {}).get("fcra"))
|
||||
return {"parsed": len(records), "fcra_regulated": fcra,
|
||||
"cache_path": str(cache_path), "source_url": url}
|
||||
|
||||
|
||||
def refresh_all(cache_path: Path, fetched: dict[str, str] | None = None) -> dict:
|
||||
"""Multi-source refresh: pull every CSV source, dedupe across states by domain, cache.
|
||||
|
||||
`fetched` optionally supplies {source_key: csv_text} to bypass the network (tests). CSV
|
||||
sources are ingested as broker records; portal sources contribute their URL for the operator
|
||||
(no bulk export exists) but no records. CA is processed first so it wins domain collisions.
|
||||
"""
|
||||
all_recs: list[dict] = []
|
||||
seen_domains: set[str] = set()
|
||||
per_source: dict[str, dict] = {}
|
||||
for key, src in SOURCES.items():
|
||||
if src["format"] != "csv":
|
||||
per_source[key] = {"jurisdiction": src["jurisdiction"], "format": "portal",
|
||||
"url": src["url"], "records": 0,
|
||||
"note": "searchable portal (no bulk export); operator/agent searches by name"}
|
||||
continue
|
||||
used_url = src["url"]
|
||||
try:
|
||||
if fetched is not None:
|
||||
text = fetched.get(key)
|
||||
if text is None:
|
||||
raise RuntimeError("no CSV text supplied")
|
||||
recs = parse(text, jurisdiction=src["jurisdiction"], has_drop=src["has_drop"])
|
||||
elif key == "ca":
|
||||
used_url, recs = _fetch_ca_latest() # newest-year-first with fallback
|
||||
else:
|
||||
recs = parse(fetch(src["url"]), jurisdiction=src["jurisdiction"], has_drop=src["has_drop"])
|
||||
except Exception as exc: # noqa: BLE001 - one source failing must not sink the rest
|
||||
per_source[key] = {"jurisdiction": src["jurisdiction"], "format": "csv", "error": str(exc)}
|
||||
continue
|
||||
added = 0
|
||||
for r in recs:
|
||||
dom = _domain(r["search"]["url"])
|
||||
if dom and dom in seen_domains:
|
||||
continue
|
||||
if dom:
|
||||
seen_domains.add(dom)
|
||||
all_recs.append(r)
|
||||
added += 1
|
||||
entry = {"jurisdiction": src["jurisdiction"], "format": "csv", "url": used_url,
|
||||
"parsed": len(recs), "added_after_dedupe": added,
|
||||
"fcra": sum(1 for r in recs if (r.get("optout") or {}).get("fcra"))}
|
||||
if key == "ca" and len(recs) < MIN_EXPECTED_CA:
|
||||
entry["warning"] = (f"only {len(recs)} parsed (expected >{MIN_EXPECTED_CA}); the CA "
|
||||
"registry file may be empty/moved - verify the source URL")
|
||||
per_source[key] = entry
|
||||
storage.write_json(cache_path, all_recs)
|
||||
return {"total": len(all_recs), "sources": per_source, "portals": portals(),
|
||||
"cache_path": str(cache_path)}
|
||||
161
optional-skills/security/unbroker/scripts/report.py
Normal file
161
optional-skills/security/unbroker/scripts/report.py
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
"""Status dashboards, Markdown reports, human-task digest, and Google Sheets row export."""
|
||||
from __future__ import annotations
|
||||
|
||||
import brokers as brokers_mod
|
||||
import ledger as ledger_mod
|
||||
|
||||
STATE_LABELS = {
|
||||
"new": "Not started",
|
||||
"searching": "Searching",
|
||||
"not_found": "Not found",
|
||||
"found": "Found (action needed)",
|
||||
"indirect_exposure": "Indirect exposure (PII on a relative's record)",
|
||||
"action_selected": "Action selected",
|
||||
"submitted": "Submitted",
|
||||
"verification_pending": "Awaiting verification",
|
||||
"awaiting_processing": "Processing",
|
||||
"confirmed_removed": "Removed",
|
||||
"reappeared": "Reappeared",
|
||||
"human_task_queued": "Human task",
|
||||
"blocked": "Blocked",
|
||||
}
|
||||
|
||||
|
||||
def status_counts(subject_id: str) -> dict:
|
||||
counts: dict[str, int] = {}
|
||||
for case in ledger_mod.load(subject_id).values():
|
||||
state = case.get("state", "new")
|
||||
counts[state] = counts.get(state, 0) + 1
|
||||
return counts
|
||||
|
||||
|
||||
def metrics(subject_id: str) -> dict:
|
||||
"""Outcome metrics: what's actually confirmed vs merely claimed, and what's overdue.
|
||||
|
||||
removal_rate is confirmed_removed over cases we actually acted on (found/submitted/... ),
|
||||
NOT over the whole broker DB, so it reflects real progress on real exposure. `in_flight`
|
||||
is 'claimed' (submitted/verifying/processing) but not yet re-scan-confirmed. `overdue`
|
||||
counts cases whose recheck window has already passed (the cron backlog).
|
||||
"""
|
||||
c = status_counts(subject_id)
|
||||
removed = c.get("confirmed_removed", 0)
|
||||
in_flight = c.get("submitted", 0) + c.get("verification_pending", 0) + c.get("awaiting_processing", 0)
|
||||
open_found = c.get("found", 0) + c.get("reappeared", 0) + c.get("action_selected", 0) \
|
||||
+ c.get("indirect_exposure", 0)
|
||||
acted = removed + in_flight + open_found + c.get("human_task_queued", 0) + c.get("blocked", 0)
|
||||
return {
|
||||
"confirmed_removed": removed,
|
||||
"in_flight_claimed": in_flight, # submitted but NOT yet verified gone
|
||||
"open_needs_action": open_found,
|
||||
"blocked": c.get("blocked", 0),
|
||||
"human_tasks": c.get("human_task_queued", 0),
|
||||
"acted_total": acted,
|
||||
"removal_rate": round(removed / acted, 3) if acted else 0.0,
|
||||
"overdue_rechecks": len(ledger_mod.due(subject_id)),
|
||||
}
|
||||
|
||||
|
||||
def render_markdown(subject_id: str) -> str:
|
||||
ledger = ledger_mod.load(subject_id)
|
||||
counts = status_counts(subject_id)
|
||||
total = sum(counts.values())
|
||||
removed = counts.get("confirmed_removed", 0)
|
||||
|
||||
m = metrics(subject_id)
|
||||
lines = [
|
||||
f"# unbroker - status for `{subject_id}`",
|
||||
"",
|
||||
f"**{removed} / {total} confirmed removed** · removal rate (of acted-on cases): "
|
||||
f"{int(m['removal_rate'] * 100)}%",
|
||||
"",
|
||||
f"- Confirmed removed: {m['confirmed_removed']}",
|
||||
f"- In flight (submitted, not yet re-scan-confirmed): {m['in_flight_claimed']}",
|
||||
f"- Open / needs action: {m['open_needs_action']}",
|
||||
f"- Blocked (anti-bot): {m['blocked']} · Human tasks: {m['human_tasks']}",
|
||||
f"- Overdue rechecks (cron backlog): {m['overdue_rechecks']}",
|
||||
"",
|
||||
"| State | Count |",
|
||||
"|---|---|",
|
||||
]
|
||||
for state in ledger_mod.STATES:
|
||||
if counts.get(state):
|
||||
lines.append(f"| {STATE_LABELS.get(state, state)} | {counts[state]} |")
|
||||
|
||||
tasks = [c for c in ledger.values() if c.get("state") == "human_task_queued"]
|
||||
if tasks:
|
||||
lines += ["", "## Outstanding human tasks"]
|
||||
for c in tasks:
|
||||
reason = c.get("human_task_reason", "manual step required")
|
||||
lines.append(f"- **{c.get('broker_id')}** - {reason}")
|
||||
|
||||
indirect = [c for c in ledger.values() if c.get("state") == "indirect_exposure"]
|
||||
if indirect:
|
||||
lines += ["", "## Indirect exposure (your PII on third-party records)",
|
||||
"Not removable via the broker's self-service opt-out (the record is about someone "
|
||||
"else). Lever: a targeted CCPA/GDPR delete-my-PII request naming only your own "
|
||||
"identifiers."]
|
||||
for c in indirect:
|
||||
ev = c.get("evidence") or {}
|
||||
note = ev.get("summary") or "subject's identifiers appear on another person's listing"
|
||||
lines.append(f"- **{c.get('broker_id')}** - {note}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def human_tasks_markdown(subject_id: str) -> str:
|
||||
"""ONE consolidated digest of everything that genuinely needs a human.
|
||||
|
||||
The autonomous run accumulates human-only work silently (never interrupting);
|
||||
this digest is presented once, at the end, so the operator clears it in a
|
||||
single sitting. Includes queued tasks and blocked-site operator-browser checks.
|
||||
"""
|
||||
ledger = ledger_mod.load(subject_id)
|
||||
tasks = [(bid, c) for bid, c in sorted(ledger.items()) if c.get("state") == "human_task_queued"]
|
||||
blocked = [(bid, c) for bid, c in sorted(ledger.items()) if c.get("state") == "blocked"]
|
||||
|
||||
lines = [f"# Human tasks for `{subject_id}`", ""]
|
||||
if not tasks and not blocked:
|
||||
lines.append("Nothing needs a human right now.")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
lines.append(f"{len(tasks)} manual step(s) + {len(blocked)} blocked site(s). "
|
||||
"Everything else ran (or will run) autonomously.")
|
||||
if tasks:
|
||||
lines += ["", "## Manual steps"]
|
||||
for bid, c in tasks:
|
||||
b = brokers_mod.get(bid) or {}
|
||||
opt = b.get("optout") or {}
|
||||
lines.append(f"### {b.get('name', bid)}")
|
||||
lines.append(f"- Why: {c.get('human_task_reason', 'manual step required')}")
|
||||
where = opt.get("url") or opt.get("email") or "(see broker record)"
|
||||
lines.append(f"- Where: {where}")
|
||||
for q in (opt.get("quirks") or [])[:2]:
|
||||
lines.append(f"- Note: {q}")
|
||||
lines.append("- Withhold: SSN and full ID numbers - always.")
|
||||
lines.append(f"- When done, tell the agent so it records the outcome for `{bid}`.")
|
||||
if blocked:
|
||||
lines += ["", "## Blocked sites (open in YOUR browser - it gets through where bots don't)"]
|
||||
for bid, c in blocked:
|
||||
b = brokers_mod.get(bid) or {}
|
||||
url = ((b.get("search") or {}).get("url")) or "(see broker record)"
|
||||
lines.append(f"- **{b.get('name', bid)}** - open {url}, search the subject, and report "
|
||||
"the verdict (or a screenshot) back to the agent.")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def sheets_rows(subject_id: str) -> list[list[str]]:
|
||||
"""Header + one row per case for the optional Google Sheets tracker.
|
||||
|
||||
The agent appends these via the `google-workspace` skill, e.g.:
|
||||
google_api.py sheets append <SHEET_ID> "Sheet1!A:F" --values <json-rows>
|
||||
"""
|
||||
rows = [["broker_id", "state", "found", "tier", "removed_at", "next_recheck"]]
|
||||
for bid, c in sorted(ledger_mod.load(subject_id).items()):
|
||||
rows.append([
|
||||
bid,
|
||||
c.get("state", ""),
|
||||
str(c.get("found", "")),
|
||||
(c.get("automation") or {}).get("tier_used", ""),
|
||||
c.get("removal_confirmed_at") or "",
|
||||
c.get("next_recheck_at") or "",
|
||||
])
|
||||
return rows
|
||||
32
optional-skills/security/unbroker/scripts/scan.py
Normal file
32
optional-skills/security/unbroker/scripts/scan.py
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
"""Stdlib fetch helper for simple url_pattern brokers (osint-style).
|
||||
|
||||
For JS-rendered or anti-bot pages the agent should use the `web_extract` or
|
||||
`browser_navigate` tools (and the `scrapling` skill for stealth/Cloudflare).
|
||||
This helper only covers plain static pages and is intentionally network-light so
|
||||
it can be mocked in tests.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
USER_AGENT = "Mozilla/5.0 (compatible; unbroker/1.0; data opt-out)"
|
||||
|
||||
|
||||
def fetch(url: str, timeout: int = 20) -> tuple[int, str]:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp: # noqa: S310 (https only by convention)
|
||||
charset = resp.headers.get_content_charset() or "utf-8"
|
||||
return getattr(resp, "status", 200), resp.read().decode(charset, errors="replace")
|
||||
except urllib.error.HTTPError as exc:
|
||||
return exc.code, ""
|
||||
except (urllib.error.URLError, TimeoutError, ValueError):
|
||||
return 0, ""
|
||||
|
||||
|
||||
def looks_listed(html: str, match_signal: str | None) -> bool:
|
||||
"""Naive confirmation heuristic for static pages: does the match signal appear?"""
|
||||
if not html or not match_signal:
|
||||
return False
|
||||
return match_signal.lower() in html.lower()
|
||||
138
optional-skills/security/unbroker/scripts/storage.py
Normal file
138
optional-skills/security/unbroker/scripts/storage.py
Normal file
|
|
@ -0,0 +1,138 @@
|
|||
"""Storage helpers (stdlib only): atomic JSON, append-only JSONL, strict perms.
|
||||
|
||||
Default backend is local-json. The optional google-sheets tracker is handled in
|
||||
report.py by emitting rows for the `google-workspace` skill; this module stays
|
||||
dependency-free so the hermetic tests never touch the network.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import crypto
|
||||
import paths
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def locked(target: Path, timeout: float = 10.0, stale: float = 30.0):
|
||||
"""Portable advisory lock via an O_EXCL lockfile next to `target`.
|
||||
|
||||
Serializes read-modify-write on shared JSON (the ledger) across concurrent
|
||||
processes - a cron re-scan overlapping a manual run, or multiple tenants -
|
||||
so one writer can't clobber another's update. A lock older than `stale`
|
||||
seconds is treated as abandoned (crashed writer) and broken, so a dead
|
||||
process can never deadlock the queue. Works on macOS/Linux/Windows (O_EXCL).
|
||||
"""
|
||||
ensure_dir(target.parent)
|
||||
lock = target.with_name(target.name + ".lock")
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
try:
|
||||
fd = os.open(str(lock), os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
try:
|
||||
os.write(fd, str(os.getpid()).encode())
|
||||
finally:
|
||||
os.close(fd)
|
||||
break
|
||||
except FileExistsError:
|
||||
try:
|
||||
if time.time() - lock.stat().st_mtime > stale:
|
||||
lock.unlink(missing_ok=True)
|
||||
continue
|
||||
except OSError:
|
||||
pass
|
||||
if time.monotonic() >= deadline:
|
||||
raise TimeoutError(f"could not acquire lock {lock} within {timeout}s")
|
||||
time.sleep(0.05)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
with contextlib.suppress(OSError):
|
||||
lock.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def _secure(path: Path, mode: int) -> None:
|
||||
try:
|
||||
os.chmod(path, mode)
|
||||
except OSError:
|
||||
pass # non-POSIX / unsupported FS; HERMES_HOME directory perms still apply
|
||||
|
||||
|
||||
def ensure_dir(path: Path) -> Path:
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
_secure(path, 0o700)
|
||||
return path
|
||||
|
||||
|
||||
def _is_sensitive(path: Path) -> bool:
|
||||
"""Per-subject docs (dossier, ledger) are sensitive; config/cache are not."""
|
||||
try:
|
||||
Path(path).resolve().relative_to(paths.subjects_dir().resolve())
|
||||
return True
|
||||
except (ValueError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
def _age_path(path: Path) -> Path:
|
||||
return path.with_name(path.name + ".age")
|
||||
|
||||
|
||||
def _atomic_write(path: Path, data: bytes) -> Path:
|
||||
tmp = path.with_name(path.name + ".tmp")
|
||||
tmp.write_bytes(data)
|
||||
_secure(tmp, 0o600)
|
||||
os.replace(tmp, path)
|
||||
_secure(path, 0o600)
|
||||
return path
|
||||
|
||||
|
||||
def write_json(path: Path, obj: Any) -> Path:
|
||||
ensure_dir(path.parent)
|
||||
data = (json.dumps(obj, indent=2, ensure_ascii=False) + "\n").encode("utf-8")
|
||||
if _is_sensitive(path) and crypto.encryption_setting() == "age":
|
||||
if not crypto.age_available():
|
||||
raise RuntimeError(
|
||||
"encryption=age is configured but `age` is not available; "
|
||||
"refusing to write PII as plaintext. Install age or run `setup --encryption none`."
|
||||
)
|
||||
target = _atomic_write(_age_path(path), crypto.encrypt(data))
|
||||
if path.exists():
|
||||
path.unlink() # migrate plaintext -> ciphertext
|
||||
return target
|
||||
target = _atomic_write(path, data)
|
||||
ap = _age_path(path)
|
||||
if ap.exists():
|
||||
ap.unlink() # encryption turned off -> drop stale ciphertext
|
||||
return target
|
||||
|
||||
|
||||
def read_json(path: Path, default: Any = None) -> Any:
|
||||
ap = _age_path(path)
|
||||
if ap.exists():
|
||||
return json.loads(crypto.decrypt(ap.read_bytes()).decode("utf-8"))
|
||||
if path.exists():
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
return default
|
||||
|
||||
|
||||
def append_jsonl(path: Path, record: dict) -> Path:
|
||||
ensure_dir(path.parent)
|
||||
with path.open("a", encoding="utf-8") as fh:
|
||||
fh.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
_secure(path, 0o600)
|
||||
return path
|
||||
|
||||
|
||||
def read_jsonl(path: Path) -> list[dict]:
|
||||
if not path.exists():
|
||||
return []
|
||||
out: list[dict] = []
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
out.append(json.loads(line))
|
||||
return out
|
||||
269
optional-skills/security/unbroker/scripts/tiers.py
Normal file
269
optional-skills/security/unbroker/scripts/tiers.py
Normal file
|
|
@ -0,0 +1,269 @@
|
|||
"""Automation-tier selection and per-subject action planning.
|
||||
|
||||
Tiers:
|
||||
T0 fully automated, no verification loop
|
||||
T1 automated submit + automated verification (email mode B/C, or backend-cleared captcha)
|
||||
T2 automated submit, verification needs a human (hard captcha / phone callback / account)
|
||||
T3 human-required end-to-end (gov ID, fax, mail, voice-only phone)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import dossier as dossier_mod
|
||||
import vectors as vectors_mod
|
||||
|
||||
HARD_HUMAN = ("gov_id", "fax", "mail", "phone_voice")
|
||||
|
||||
|
||||
def select_tier(broker: dict, email_mode: str = "draft_only",
|
||||
browser_clears_captcha: bool = False) -> str:
|
||||
req = ((broker.get("optout") or {}).get("requires")) or {}
|
||||
|
||||
if any(req.get(k) for k in HARD_HUMAN):
|
||||
return "T3"
|
||||
if req.get("account"):
|
||||
return "T2"
|
||||
|
||||
captcha = bool(req.get("captcha"))
|
||||
if (captcha and not browser_clears_captcha) or req.get("phone_callback"):
|
||||
return "T2"
|
||||
|
||||
if req.get("email_verification"):
|
||||
return "T1" if email_mode in ("programmatic", "alias") else "T2"
|
||||
|
||||
if captcha and browser_clears_captcha:
|
||||
return "T1"
|
||||
return "T0"
|
||||
|
||||
|
||||
def plan(subject_dossier: dict, brokers_list: list[dict], cfg: dict,
|
||||
browser_clears_captcha: bool = False) -> list[dict]:
|
||||
email_mode = (subject_dossier.get("preferences") or {}).get("email_mode") \
|
||||
or cfg.get("email_mode", "draft_only")
|
||||
actions: list[dict] = []
|
||||
for b in brokers_list:
|
||||
opt = b.get("optout") or {}
|
||||
search = b.get("search") or {}
|
||||
tier = select_tier(b, email_mode, browser_clears_captcha)
|
||||
disclosure = dossier_mod.select_disclosure(subject_dossier, opt.get("inputs", []))
|
||||
svectors = vectors_mod.search_vectors(subject_dossier, b)
|
||||
actions.append({
|
||||
"broker_id": b.get("id"),
|
||||
"broker_name": b.get("name"),
|
||||
"priority": b.get("priority"),
|
||||
"method": opt.get("method"),
|
||||
"tier": tier,
|
||||
"human_required": tier == "T3",
|
||||
"search_url": search.get("url"),
|
||||
"fetch": search.get("fetch", "web_extract"),
|
||||
"antibot": search.get("antibot"),
|
||||
"search_by": vectors_mod.supported_by(b),
|
||||
"search_vectors": svectors,
|
||||
"optout_url": opt.get("url"),
|
||||
"optout_email": opt.get("email"),
|
||||
"disclosure_fields": sorted(disclosure.keys()),
|
||||
"owns": b.get("owns") or [],
|
||||
"notes": opt.get("notes", ""),
|
||||
"optout_quirks": opt.get("quirks") or [],
|
||||
"optout_requires": opt.get("requires") or {},
|
||||
# The DELETION lane (right-to-delete), distinct from listing suppression. Structured so
|
||||
# the autopilot can route to it: {via: email|in_flow|web_form, email?, url?, kinds?, notes?}
|
||||
"deletion": opt.get("deletion") or {},
|
||||
# Exact ordered opt-out steps maintained IN the broker record (field-verified knowledge
|
||||
# lives with the data, not in code).
|
||||
"optout_playbook": opt.get("playbook") or [],
|
||||
})
|
||||
return actions
|
||||
|
||||
|
||||
def fanout(brokers_list: list[dict], batch_size: int = 8) -> dict:
|
||||
"""Group brokers into batches for parallel `delegate_task` scan subagents.
|
||||
|
||||
Scanning many brokers serially is slow and burns context; above `batch_size`
|
||||
the agent is expected to spawn one subagent per batch (see SKILL.md).
|
||||
"""
|
||||
ids = [b.get("id") for b in brokers_list if b.get("id")]
|
||||
batches = [ids[i:i + batch_size] for i in range(0, len(ids), batch_size)]
|
||||
return {
|
||||
"broker_count": len(ids),
|
||||
"batch_size": batch_size,
|
||||
"should_fanout": len(ids) > batch_size,
|
||||
"batches": batches,
|
||||
}
|
||||
|
||||
|
||||
# States that mean "the crawl reached a verdict for this broker".
|
||||
_SCANNED_STATES = {"found", "not_found", "indirect_exposure", "blocked", "submitted",
|
||||
"verification_pending", "awaiting_processing", "confirmed_removed", "reappeared",
|
||||
"action_selected", "human_task_queued"}
|
||||
# States that still need a deletion action taken.
|
||||
_ACTIONABLE_STATES = {"found", "indirect_exposure", "reappeared", "action_selected"}
|
||||
|
||||
|
||||
def batch_plan(subject_dossier: dict, brokers_list: list[dict], cfg: dict,
|
||||
ledger: dict | None = None, browser_clears_captcha: bool = False) -> dict:
|
||||
"""Reduce the per-broker plan into a phase-oriented batch view.
|
||||
|
||||
Overlays the current ledger state on each broker, groups by what the operator
|
||||
should DO next, and collapses ownership clusters so a parent removal that clears
|
||||
children is ONE action, not N. Read-only: computes, never mutates the ledger.
|
||||
"""
|
||||
ledger = ledger or {}
|
||||
actions = plan(subject_dossier, brokers_list, cfg, browser_clears_captcha)
|
||||
|
||||
# child id -> parent id (only for parents present in this plan set)
|
||||
child_to_parent: dict[str, str] = {}
|
||||
for a in actions:
|
||||
for child in a.get("owns") or []:
|
||||
child_to_parent[child] = a["broker_id"]
|
||||
|
||||
def state_of(bid: str) -> str:
|
||||
return (ledger.get(bid) or {}).get("state", "new")
|
||||
|
||||
groups: dict[str, list[dict]] = {
|
||||
"unscanned": [], # no verdict yet -> Phase 1 crawl
|
||||
"found": [], # direct removable listing -> Phase 2 opt-out (incl. reappeared/action_selected)
|
||||
"indirect_exposure": [],# PII on a third party's record -> CCPA/GDPR delete email
|
||||
"blocked": [], # anti-bot / needs stealth browser -> requeue
|
||||
"in_progress": [], # submitted / verification_pending / awaiting_processing
|
||||
"human": [], # human_task_queued -> the end-of-run digest, NOT re-scanning
|
||||
"done": [], # confirmed_removed
|
||||
"not_found": [],
|
||||
}
|
||||
covered_by_parent: dict[str, list[str]] = {}
|
||||
|
||||
for a in actions:
|
||||
bid = a["broker_id"]
|
||||
st = state_of(bid)
|
||||
# cluster collapse: if a parent in this set is already actioned, the child is covered
|
||||
parent = child_to_parent.get(bid)
|
||||
if parent and state_of(parent) in ("found", "reappeared", "action_selected", "submitted",
|
||||
"verification_pending", "awaiting_processing",
|
||||
"confirmed_removed", "human_task_queued"):
|
||||
covered_by_parent.setdefault(parent, []).append(bid)
|
||||
continue
|
||||
|
||||
row = {"broker_id": bid, "broker_name": a["broker_name"], "priority": a["priority"],
|
||||
"tier": a["tier"], "method": a["method"], "state": st,
|
||||
"optout_url": a["optout_url"], "optout_email": a.get("optout_email"),
|
||||
"clears_children": a.get("owns") or [],
|
||||
"optout_requires": a.get("optout_requires") or {},
|
||||
"optout_quirks": a.get("optout_quirks") or [],
|
||||
"deletion": a.get("deletion") or {},
|
||||
"optout_playbook": a.get("optout_playbook") or [],
|
||||
"notes": a.get("notes", "")}
|
||||
if st in ("submitted", "verification_pending", "awaiting_processing"):
|
||||
groups["in_progress"].append(row)
|
||||
elif st == "confirmed_removed":
|
||||
groups["done"].append(row)
|
||||
elif st in ("reappeared", "action_selected"):
|
||||
groups["found"].append(row) # still needs the opt-out action
|
||||
elif st == "human_task_queued":
|
||||
groups["human"].append(row) # parked for the digest; never re-queued as work
|
||||
elif st in groups:
|
||||
groups[st].append(row)
|
||||
elif st not in _SCANNED_STATES:
|
||||
groups["unscanned"].append(row)
|
||||
else:
|
||||
groups.setdefault(st, []).append(row)
|
||||
|
||||
# PARENTS FIRST: within the actionable 'found' group, order cluster parents (a removal
|
||||
# that clears children) ahead of standalone listings, most-children first. Working a
|
||||
# parent before its children is what makes the cluster dedup real -- do them in this order.
|
||||
groups["found"].sort(key=lambda r: (-len(r.get("clears_children") or []),
|
||||
{"T0": 0, "T1": 1, "T2": 2, "T3": 3}.get(r.get("tier") or "", 9),
|
||||
r["broker_id"]))
|
||||
|
||||
return {
|
||||
"subject": subject_dossier.get("subject_id"),
|
||||
"phase": "discover" if groups["unscanned"] else "delete",
|
||||
"counts": {k: len(v) for k, v in groups.items()},
|
||||
"groups": groups,
|
||||
"cluster_savings": {p: kids for p, kids in covered_by_parent.items()},
|
||||
"parent_playbook": _parent_playbook(groups["found"]),
|
||||
"next_actions": _batch_next(groups, covered_by_parent),
|
||||
}
|
||||
|
||||
|
||||
def synthesize_steps(r: dict) -> list[str]:
|
||||
"""Generic ordered opt-out steps derived from an optout record's structured fields.
|
||||
|
||||
Used for any broker without a hand-verified `optout.playbook`. Bespoke, field-verified
|
||||
step lists live IN the broker JSON (`optout.playbook`) - single source of truth that
|
||||
accrues knowledge as live runs discover mechanics (see methods.md logging rule).
|
||||
"""
|
||||
steps = [f"Opt out at {r.get('optout_url') or r.get('optout_email') or '(see broker record)'}"
|
||||
+ (f" -- clears {', '.join(r['clears_children'])}." if r.get("clears_children") else ".")]
|
||||
req = r.get("optout_requires") or {}
|
||||
if req.get("profile_url"):
|
||||
steps.append("Needs the confirmed profile_url (paste the listing URL you recorded).")
|
||||
if req.get("email_verification"):
|
||||
steps.append("Email verification: the same browser/inbox must open the confirmation link.")
|
||||
if req.get("phone_callback"):
|
||||
steps.append("Phone-callback code required; queue a human task if no operator is available.")
|
||||
if req.get("gov_id"):
|
||||
steps.append("Government ID demanded (T3): human task; never send SSN or a full ID number.")
|
||||
d = r.get("deletion") or {}
|
||||
if d.get("email"):
|
||||
steps.append(f"DELETION lane: a right-to-delete request can be emailed to {d['email']}"
|
||||
+ (f" ({d['notes']})" if d.get("notes") else "")
|
||||
+ " -- prefer deletion over suppression.")
|
||||
if r.get("notes"):
|
||||
steps.append(str(r["notes"]))
|
||||
for q in (r.get("optout_quirks") or [])[:3]:
|
||||
steps.append(str(q))
|
||||
return steps
|
||||
|
||||
|
||||
def _parent_playbook(found_rows: list[dict]) -> list[dict]:
|
||||
"""Tailored, ordered opt-out instructions for each cluster PARENT in the found group.
|
||||
|
||||
Steps come from the broker record's own `optout.playbook` (field-verified, maintained with
|
||||
the data) with a synthesised fallback so the guidance is never empty. Standalone listings
|
||||
are intentionally omitted -- the playbook exists to make the parents-first order concrete.
|
||||
"""
|
||||
playbook: list[dict] = []
|
||||
for i, r in enumerate([x for x in found_rows if x.get("clears_children")], start=1):
|
||||
steps = list(r.get("optout_playbook") or []) or synthesize_steps(r)
|
||||
playbook.append({
|
||||
"order": i,
|
||||
"broker_id": r["broker_id"],
|
||||
"broker_name": r["broker_name"],
|
||||
"tier": r["tier"],
|
||||
"clears_children": r["clears_children"],
|
||||
"optout_url": r.get("optout_url"),
|
||||
"optout_email": r.get("optout_email"),
|
||||
"deletion": r.get("deletion") or {},
|
||||
"steps": steps,
|
||||
})
|
||||
return playbook
|
||||
|
||||
|
||||
def _batch_next(groups: dict, covered: dict) -> list[str]:
|
||||
tips: list[str] = []
|
||||
if groups["unscanned"]:
|
||||
tips.append(f"PHASE 1 (crawl): {len(groups['unscanned'])} broker(s) unscanned -- run `fanout` and "
|
||||
"scan read-only before any deletion.")
|
||||
if groups["found"]:
|
||||
parents = [r for r in groups["found"] if r.get("clears_children")]
|
||||
if parents:
|
||||
order = " -> ".join(r["broker_id"] for r in parents)
|
||||
tips.append(f"PHASE 2 (opt-out): {len(groups['found'])} direct listing(s). DO CLUSTER PARENTS "
|
||||
f"FIRST, in this order: {order} (see `parent_playbook` for tailored per-parent "
|
||||
"steps), then the standalone listings.")
|
||||
else:
|
||||
tips.append(f"PHASE 2 (opt-out): {len(groups['found'])} direct listing(s) to remove.")
|
||||
if groups["indirect_exposure"]:
|
||||
tips.append(f"{len(groups['indirect_exposure'])} indirect-exposure case(s): send a targeted "
|
||||
"CCPA/GDPR delete-my-PII email (render-email --kind ccpa_indirect), do NOT use the opt-out form.")
|
||||
if groups["blocked"]:
|
||||
tips.append(f"{len(groups['blocked'])} blocked (anti-bot): requeue for a stealth/cloud browser "
|
||||
"pass; don't burn subagent time fighting CAPTCHAs.")
|
||||
if covered:
|
||||
n = sum(len(v) for v in covered.values())
|
||||
tips.append(f"Cluster dedup: {n} child site(s) covered by parent removals -- skip separate opt-outs.")
|
||||
if groups["in_progress"]:
|
||||
tips.append(f"{len(groups['in_progress'])} in progress: resolve verification links, then confirm removal.")
|
||||
if groups.get("human"):
|
||||
tips.append(f"{len(groups['human'])} parked human task(s): present via `tasks` at end of run "
|
||||
"(do not re-scan or re-queue them).")
|
||||
return tips
|
||||
53
optional-skills/security/unbroker/scripts/vectors.py
Normal file
53
optional-skills/security/unbroker/scripts/vectors.py
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
"""Enumerate the search queries to run per broker, across ALL of a subject's identifiers.
|
||||
|
||||
People-search sites index a person under every name, phone, email, and address they
|
||||
have. A subject with two names (maiden/married) and three past cities can have many
|
||||
distinct listings on one broker, each found via a different search. `search_vectors`
|
||||
expands the dossier into the concrete searches to run, filtered by what each broker
|
||||
supports (`broker.search.by`, default ["name"]).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import dossier as dossier_mod
|
||||
|
||||
# What a broker can be searched by; default if a record doesn't declare it.
|
||||
DEFAULT_BY = ["name"]
|
||||
|
||||
|
||||
def supported_by(broker: dict) -> list[str]:
|
||||
return list((broker.get("search") or {}).get("by") or DEFAULT_BY)
|
||||
|
||||
|
||||
def search_vectors(subject_dossier: dict, broker: dict) -> list[dict]:
|
||||
"""List of {by, query} searches to run for this subject on this broker."""
|
||||
by = set(supported_by(broker))
|
||||
ident = subject_dossier.get("identity", {})
|
||||
vectors: list[dict] = []
|
||||
|
||||
if "name" in by:
|
||||
names = dossier_mod.all_names(subject_dossier)
|
||||
locations = dossier_mod.all_locations(subject_dossier)
|
||||
if locations:
|
||||
for name in names:
|
||||
for loc in locations:
|
||||
vectors.append({"by": "name",
|
||||
"query": {"full_name": name, "city": loc.get("city"), "state": loc.get("state")}})
|
||||
else:
|
||||
for name in names:
|
||||
vectors.append({"by": "name", "query": {"full_name": name}})
|
||||
|
||||
if "phone" in by:
|
||||
for phone in ident.get("phones") or []:
|
||||
vectors.append({"by": "phone", "query": {"phone": phone}})
|
||||
|
||||
if "email" in by:
|
||||
for email in ident.get("emails") or []:
|
||||
vectors.append({"by": "email", "query": {"email": email}})
|
||||
|
||||
if "address" in by:
|
||||
for a in dossier_mod.all_addresses(subject_dossier):
|
||||
if a.get("line1"):
|
||||
vectors.append({"by": "address",
|
||||
"query": {k: a.get(k) for k in ("line1", "city", "state", "postal")}})
|
||||
|
||||
return vectors
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
# Authorization to act on my behalf (data removal)
|
||||
|
||||
I, **{full_name}**, authorize the operator of this tool to act as my agent for the limited purpose of
|
||||
removing my personal information from data brokers and people-search websites, including submitting
|
||||
opt-out, deletion, and do-not-sell/share requests under applicable privacy laws (e.g. CCPA/CPRA,
|
||||
GDPR) on my behalf.
|
||||
|
||||
This authorization is limited to data removal. It does not authorize any other use of my information.
|
||||
|
||||
- Full name: {full_name}
|
||||
- Date: {date}
|
||||
- Signature: ______________________________
|
||||
|
||||
Store the signed copy at the path recorded in the dossier `consent.authorization_artifact`. Required
|
||||
only when `consent.method` is `written_authorization` or `poa` (not for `self`).
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
Subject: CCPA/CPRA request submitted by authorized agent (delete and opt out)
|
||||
|
||||
To the {broker_name} privacy team,
|
||||
|
||||
I am an authorized agent acting on behalf of the consumer named below, under Cal. Civ. Code
|
||||
1798.135 and the CCPA/CPRA. Written authorization is on file and available on request.
|
||||
|
||||
On the consumer's behalf I request that you:
|
||||
|
||||
1. DELETE all personal information you hold about them, and
|
||||
2. OPT them OUT of the sale and sharing of their personal information.
|
||||
|
||||
Their information appears at:
|
||||
{listing_urls}
|
||||
|
||||
Consumer:
|
||||
Name: {full_name}
|
||||
Email for confirmation: {contact_email}
|
||||
|
||||
Please confirm completion in writing to the email above. Do not request more sensitive information
|
||||
than necessary to verify and process this request.
|
||||
|
||||
Sincerely,
|
||||
Authorized agent for {full_name}
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
Subject: CCPA/CPRA request to delete and opt out (do not sell or share)
|
||||
|
||||
To the {broker_name} privacy team,
|
||||
|
||||
Under the California Consumer Privacy Act as amended by the CPRA (Cal. Civ. Code 1798.105 and
|
||||
1798.120), I request that you:
|
||||
|
||||
1. DELETE all personal information you hold about me, and
|
||||
2. OPT me OUT of the sale and sharing of my personal information.
|
||||
|
||||
My information appears at:
|
||||
{listing_urls}
|
||||
|
||||
Identifying details for this request:
|
||||
Name: {full_name}
|
||||
Email: {contact_email}
|
||||
|
||||
Please do not request more sensitive information than necessary to process this request. Confirm
|
||||
completion in writing to the email above within the statutory timeframe.
|
||||
|
||||
Sincerely,
|
||||
{full_name}
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
Subject: Request to delete my personal information from third-party listings (CCPA/CPRA where applicable)
|
||||
|
||||
To the {broker_name} privacy team,
|
||||
|
||||
I am not the primary subject of the listings below, but each one currently exposes MY personal
|
||||
information as a secondary data point (my email address and/or my name shown as a relative or
|
||||
associated person). I am writing only about my own personal information, not about the individuals
|
||||
who are the primary subjects of these records, and I am not requesting any change to their data.
|
||||
|
||||
Please delete and suppress the following personal information about me wherever it appears in your
|
||||
database and on Spokeo-operated sites, including Thatsthem.com:
|
||||
{my_identifiers}
|
||||
|
||||
These items currently appear at:
|
||||
{listing_urls}
|
||||
|
||||
To the extent the California Consumer Privacy Act as amended by the CPRA (Cal. Civ. Code 1798.105)
|
||||
applies to my personal information, please treat this as a request to delete it and to opt me out of
|
||||
its sale or sharing (1798.120). Where CCPA does not apply by residency, I ask that you honor this as a
|
||||
standard removal of my personal information consistent with the policy you apply to such requests.
|
||||
|
||||
Please do not request more information than is necessary to locate and remove these items, and please
|
||||
do not add any new identifiers to my data in the course of processing this request. Confirm completion
|
||||
in writing to the email below.
|
||||
|
||||
Name: {full_name}
|
||||
Contact email: {contact_email}
|
||||
|
||||
Sincerely,
|
||||
{full_name}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
Subject: Request for erasure under GDPR Article 17
|
||||
|
||||
To the {broker_name} data protection officer,
|
||||
|
||||
Under Article 17 of the EU General Data Protection Regulation (and/or the UK GDPR), I request the
|
||||
erasure of all personal data you hold about me, and under Article 21 I object to its processing.
|
||||
|
||||
My personal data appears at:
|
||||
{listing_urls}
|
||||
|
||||
Identifying details:
|
||||
Name: {full_name}
|
||||
Email: {contact_email}
|
||||
|
||||
Please confirm erasure in writing to the email above within one month, as required by Article 12(3).
|
||||
Do not request more personal data than necessary to action this request.
|
||||
|
||||
Sincerely,
|
||||
{full_name}
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
Subject: Opt-out and data removal request
|
||||
|
||||
To the {broker_name} privacy team,
|
||||
|
||||
I am writing to request the removal of my personal information from {broker_name} and any sites you
|
||||
operate or supply. My information currently appears at:
|
||||
{listing_urls}
|
||||
|
||||
Please suppress and delete the record(s) associated with my name, {full_name}, and do not sell or
|
||||
share my personal information.
|
||||
|
||||
Please confirm completion to this email address: {contact_email}
|
||||
|
||||
Thank you,
|
||||
{full_name}
|
||||
1296
tests/skills/test_unbroker_skill.py
Normal file
1296
tests/skills/test_unbroker_skill.py
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Add table
Add a link
Reference in a new issue