mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-20 05:01:30 +00:00
* feat(skills): add osint-investigation optional skill (closes #355) Phase-1 public-records OSINT investigation framework adapted from ShinMegamiBoson/OpenPlanter (MIT). Lives in optional-skills/research/. Six data-source wiki entries (FEC, SEC EDGAR, USAspending, Senate LD, OFAC SDN, ICIJ Offshore Leaks), each following the 9-section template: summary, access, schema, coverage, cross-reference keys, data quality, acquisition, legal, references. Six stdlib-only acquisition scripts that emit normalized CSV, plus three analysis scripts: - entity_resolution.py — three-tier match (exact / fuzzy / token overlap) with explicit confidence per row - timing_analysis.py — permutation test for donation/contract timing correlation, joins through cross-links - build_findings.py — assembles structured findings.json with evidence chains pointing back to source rows Validation: full pipeline runs end-to-end on synthetic fixtures. Entity resolution found 24 cross-matches with 0 false positives on a 5-row / 4-row test set. Timing analysis on 5 donations clustered near 3 awards returned p=0.000, effect size 2.41 SD. Findings JSON correctly tags HIGH-severity timing pattern. All 9 scripts pass --help and py_compile. Docs site page auto-generated by website/scripts/generate-skill-docs.py; sidebar + catalog entries updated by the same generator. * fix(osint-investigation): live API fixes from end-to-end sweep Live-tested the skill on a real public-citizen query and found three bugs the synthetic E2E missed. All three are now fixed and re-verified. 1. FEC fetch hung on contributor name searches. The combination of two_year_transaction_period + sort=date + contributor_name puts the OpenFEC query plan on a slow path that the upstream gateway times out (25s+). Switched to min_date/max_date with no explicit sort. Renamed --candidate to --contributor (the original name was misleading: FEC searches by donor, not by candidate; --candidate is kept as a deprecated alias). Added --state filter for narrowing. 2. ICIJ Offshore Leaks reconcile endpoint returns 404. ICIJ removed the Open Refine reconciliation API. Rewrote fetch_icij_offshore.py to download the official bulk CSV ZIP (~70 MB, public, no auth) and search it locally. Cached under $HERMES_OSINT_CACHE/icij/ (default ~/.cache/hermes-osint/icij/) for 30 days, --force-refresh to refetch. Verified live: 'PUTIN' query returns 5 Panama Papers officer matches in 0.5s after first download. 3. SEC EDGAR silently returned 0 when the company-name resolver matched an individual Form 3/4/5 filer (insider trading disclosures). Now surfaces 'Resolved company X → CIK Y (Z)' on stderr, prints a filing-type histogram when the type filter wipes results, and explicitly warns when the matched CIK appears to be an individual filer rather than a corporate registrant. Bonus: _http.py was retrying 429 responses with exponential backoff plus honoring (often-missing) Retry-After headers, which compounded into multi-second hangs per page when the upstream key was over quota. Changed to fail-fast on 429 with a clear, actionable error showing the upstream's quota message. Verified: 0.3s fast-fail vs the previous 60s hang on DEMO_KEY rate-limit exhaustion. Updated SKILL.md, fec.md, and icij-offshore.md to match the new CLI flags and ICIJ bulk-cache flow. Regenerated the docusaurus page via website/scripts/generate-skill-docs.py. Live sweep results across all 6 sources for 'Dillon Rolnick, New York': - OFAC SDN: 0 matches ✓ (correctly not sanctioned) - USAspending: 0 matches ✓ (correctly not a federal contractor) - Senate LDA: 0 matches ✓ (correctly not a lobbying client) - SEC EDGAR: warns it resolved to 'Rolnick Michael' (CIK 0001845264) who is an individual Form 3 filer, not a corporate registrant - ICIJ: 0 matches ✓ (correctly not in any offshore leak) - FEC: rate-limited (DEMO_KEY); fails fast with clear quota message * feat(osint-investigation): expand to 12 sources covering identity, property, courts, archives, news Phase-2 expansion per Teknium feedback that the original 6-source skill (federal financial/regulatory only) wasn't a complete OSINT toolkit. Adds 6 more sources covering the major omissions a real investigation would reach for first. New sources (6 fetch scripts + 6 wiki entries): 1. NYC ACRIS — Real property records (deeds, mortgages, liens) via the city's Socrata API. Search by party name or property address. Joins Parties to Master to populate doc_type, dates, borough, and amount. Coverage: 5 NYC boroughs, ~70M party records, 1966-present. 2. OpenCorporates — Global corporate registry covering 130+ jurisdictions (~200M companies). Free API token at https://opencorporates.com/api_accounts/new raises the rate limit; HTML fallback works without one (limited fields). 3. CourtListener (Free Law Project) — federal + state court opinions (~10M back to colonial era) + PACER dockets via RECAP. Anonymous v4 search works; COURTLISTENER_TOKEN raises rate limits. 4. Wayback Machine CDX — historical web captures (~900B+). Used both for surveillance-of-record (when did this site change?) and as a content-recovery layer when other sources point to dead URLs. 5. Wikipedia + Wikidata — narrative bio + structured facts. Wikipedia OpenSearch for article matching, REST summary for extracts, Wikidata Action API (wbgetentities) for claims. Avoids the SPARQL Query Service which is aggressively rate-limited. 6. GDELT 2.0 DOC API — global news monitoring in 100+ languages, ~2015-present. Auto-retries with 6s backoff on the standard 1-req-per-5-sec throttle. Other changes in this commit: - SEC EDGAR no longer raises SystemExit when the company-name resolver finds no CIK; writes an empty CSV with header so the rest of a pipeline can keep moving and the warning is just on stderr. - _http.py User-Agent updated per Wikimedia policy: includes app name, version, and a 'set HERMES_OSINT_UA to identify yourself' instruction. - SKILL.md workflow now groups sources into two clusters (federal financial vs identity/property/courts/archives/news) with bash examples for each. 'When to use this skill' lists the broader set of investigation patterns the expanded sources unlock. Live sweep results on 'Dillon Rolnick, New York' across all 12 sources: ofac ✓ 0 (correctly clean) icij ✓ 0 (correctly not in any leak) usaspending ✓ 0 (correctly not a federal contractor) senate_lda ✓ 0 (correctly not a lobbying client) sec_edgar ✓ 0, warns: resolved to 'Rolnick Michael' (CIK 0001845264), individual Form 3 filer, NOT a corporate registrant fec — rate-limited (DEMO_KEY exhausted), fails fast with clear quota message nyc_acris ✓ 200 records named Rolnick across NYC; 48 records at 571 Hudson (the property the web identifies as his) opencorporates ✓ 0 (no API token configured; HTML fallback) courtlistener ✓ 0 for 'Dillon Rolnick'; 20 for 'Rolnick' generally; 5 for 'Microsoft' sanity check wayback ✓ 30 captures of nousresearch.com from 2011-present wikipedia ✓ 0 (correctly not notable enough); Bill Gates sanity returns full structured facts (occupation, employer, DOB, place of birth, country) gdelt ✓ 0 for 'Dillon Rolnick'; 5 for 'Nous Research' All 17 scripts compile clean and pass --help. Synthetic analysis pipeline regression still passes (entity_resolution 30 matches, timing p=0.000, findings 2). * feat(osint-investigation): remove FEC; DEMO_KEY rate-limits make it unreliable The FEC fetcher consistently failed the live sweep because the OpenFEC DEMO_KEY tier (40 calls/hour) exhausts on a single investigation, and the upstream returns slow-path query plans for unindexed contributor-name searches that the gateway times out. Without a real API key it's not usable; with one the user has to sign up at api.data.gov first. That's too much setup friction for a skill that should work out of the box. Removed: - scripts/fetch_fec.py - references/sources/fec.md Updated: - SKILL.md frontmatter description + tags - 'When NOT to use' now points users at https://www.fec.gov/data/ for federal donations - entity_resolution example switched from donor↔contractor to lobbying-client↔contractor (Senate LDA + USAspending pair) - timing_analysis example switched to lobbying-filings vs awards - 8 wiki entries had their 'FEC ↔ ...' cross-reference bullets removed 11 sources remain (5 federal financial + 6 identity/property/courts/ archives/news). All scripts compile, pass --help, and the synthetic analysis pipeline still passes on the new lobbying-shaped regression fixture (30 matches, p=0.000 on tight clustering, 2 findings).
This commit is contained in:
Teknium
GitHub
parent
d725407c56
commit
5f91b1a48b
32 changed files with 4567 additions and 0 deletions
|
|
@ -167,6 +167,7 @@ hermes skills uninstall <skill-name>
|
|||
| [**drug-discovery**](/docs/user-guide/skills/optional/research/research-drug-discovery) | Pharmaceutical research assistant for drug discovery workflows. Search bioactive compounds on ChEMBL, calculate drug-likeness (Lipinski Ro5, QED, TPSA, synthetic accessibility), look up drug-drug interactions via OpenFDA, interpret ADMET... |
|
||||
| [**duckduckgo-search**](/docs/user-guide/skills/optional/research/research-duckduckgo-search) | Free web search via DuckDuckGo — text, news, images, videos. No API key needed. Prefer the `ddgs` CLI when installed; use the Python DDGS library only after verifying that `ddgs` is available in the current runtime. |
|
||||
| [**gitnexus-explorer**](/docs/user-guide/skills/optional/research/research-gitnexus-explorer) | Index a codebase with GitNexus and serve an interactive knowledge graph via web UI + Cloudflare tunnel. |
|
||||
| [**osint-investigation**](/docs/user-guide/skills/optional/research/research-osint-investigation) | Public-records OSINT investigation framework — SEC EDGAR filings, USAspending contracts, Senate lobbying, OFAC sanctions, ICIJ offshore leaks, NYC property records (ACRIS), OpenCorporates registries, CourtListener court records, Wayback... |
|
||||
| [**parallel-cli**](/docs/user-guide/skills/optional/research/research-parallel-cli) | Optional vendor skill for Parallel CLI — agent-native web search, extraction, deep research, enrichment, FindAll, and monitoring. Prefer JSON output and non-interactive flows. |
|
||||
| [**qmd**](/docs/user-guide/skills/optional/research/research-qmd) | Search personal knowledge bases, notes, docs, and meeting transcripts locally using qmd — a hybrid retrieval engine with BM25, vector search, and LLM reranking. Supports CLI and MCP integration. |
|
||||
| [**scrapling**](/docs/user-guide/skills/optional/research/research-scrapling) | Web scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python. |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,294 @@
|
|||
---
|
||||
title: "Osint Investigation"
|
||||
sidebar_label: "Osint Investigation"
|
||||
description: "Public-records OSINT investigation framework — SEC EDGAR filings, USAspending contracts, Senate lobbying, OFAC sanctions, ICIJ offshore leaks, NYC property r..."
|
||||
---
|
||||
|
||||
{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
|
||||
|
||||
# Osint Investigation
|
||||
|
||||
Public-records OSINT investigation framework — SEC EDGAR filings, USAspending contracts, Senate lobbying, OFAC sanctions, ICIJ offshore leaks, NYC property records (ACRIS), OpenCorporates registries, CourtListener court records, Wayback Machine archives, Wikipedia + Wikidata, GDELT news monitoring. Entity resolution across sources, cross-link analysis, timing correlation, evidence chains. Python stdlib only.
|
||||
|
||||
## Skill metadata
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Source | Optional — install with `hermes skills install official/research/osint-investigation` |
|
||||
| Path | `optional-skills/research/osint-investigation` |
|
||||
| Version | `0.1.0` |
|
||||
| Author | Hermes Agent (adapted from ShinMegamiBoson/OpenPlanter, MIT) |
|
||||
| Platforms | linux, macos, windows |
|
||||
| Tags | `osint`, `investigation`, `public-records`, `sec`, `sanctions`, `corporate-registry`, `property`, `courts`, `due-diligence`, `journalism` |
|
||||
| Related skills | [`domain-intel`](/docs/user-guide/skills/optional/research/research-domain-intel), [`arxiv`](/docs/user-guide/skills/bundled/research/research-arxiv) |
|
||||
|
||||
## Reference: full SKILL.md
|
||||
|
||||
:::info
|
||||
The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
|
||||
:::
|
||||
|
||||
# OSINT Investigation — Public Records Cross-Reference
|
||||
|
||||
Investigative framework for public-records OSINT: government contracts,
|
||||
corporate filings, lobbying, sanctions, offshore leaks, property records,
|
||||
court records, web archives, knowledge bases, and global news. Resolve
|
||||
entities across heterogeneous sources, build cross-links with explicit
|
||||
confidence, run statistical timing tests, and produce structured evidence
|
||||
chains.
|
||||
|
||||
**Python stdlib only.** Zero install. Works on Linux, macOS, Windows. Most
|
||||
sources work with no API key (OpenCorporates has an optional free token
|
||||
that raises rate limits).
|
||||
|
||||
Adapted from the MIT-licensed ShinMegamiBoson/OpenPlanter project; expanded
|
||||
to cover identity / property / litigation / archives / news sources that
|
||||
the original didn't address.
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Use when the user asks for:
|
||||
|
||||
- "follow the money" — government contracts, lobbying → legislation, sanctions
|
||||
- corporate due diligence — who controls company X, where are they
|
||||
incorporated, who serves on their boards, what filings have they made
|
||||
- sanctions screening — is entity X on OFAC SDN, ICIJ offshore leaks
|
||||
- pay-to-play investigation — contractors with offshore ties, lobbying
|
||||
clients winning awards
|
||||
- property ownership — find recorded deeds/mortgages by name or address
|
||||
(NYC; for other counties point users at the relevant recorder)
|
||||
- litigation history — find federal + state court opinions and PACER dockets
|
||||
- multi-source entity resolution where naming varies (LLC suffixes, abbreviations)
|
||||
- evidence-chain construction with explicit confidence levels
|
||||
- "what's been said about X" — international news (GDELT) + Wikipedia
|
||||
narrative + Wayback Machine to recover dead URLs
|
||||
|
||||
Do NOT use this skill for:
|
||||
|
||||
- general web research → `web_search` / `web_extract`
|
||||
- domain/infrastructure OSINT → `domain-intel` skill
|
||||
- academic literature → `arxiv` skill
|
||||
- social-media profile discovery → `sherlock` skill (optional)
|
||||
- US **federal** campaign finance — FEC is intentionally NOT covered here
|
||||
(the API is unreliable for ad-hoc contributor-name queries on the free
|
||||
DEMO_KEY tier). For federal donations, point users at
|
||||
https://www.fec.gov/data/ directly.
|
||||
|
||||
## Workflow
|
||||
|
||||
The agent runs scripts via the `terminal` tool. `SKILL_DIR` is the directory
|
||||
holding this SKILL.md.
|
||||
|
||||
### 1. Identify which sources apply
|
||||
|
||||
Read the data-source wiki entries to plan the investigation:
|
||||
|
||||
```
|
||||
ls SKILL_DIR/references/sources/
|
||||
|
||||
# Federal financial / regulatory
|
||||
cat SKILL_DIR/references/sources/sec-edgar.md # corporate filings
|
||||
cat SKILL_DIR/references/sources/usaspending.md # federal contracts
|
||||
cat SKILL_DIR/references/sources/senate-ld.md # lobbying
|
||||
cat SKILL_DIR/references/sources/ofac-sdn.md # sanctions
|
||||
cat SKILL_DIR/references/sources/icij-offshore.md # offshore leaks
|
||||
|
||||
# Identity / property / litigation / archives / news
|
||||
cat SKILL_DIR/references/sources/nyc-acris.md # NYC property records
|
||||
cat SKILL_DIR/references/sources/opencorporates.md # global corporate registry
|
||||
cat SKILL_DIR/references/sources/courtlistener.md # court records (federal + state)
|
||||
cat SKILL_DIR/references/sources/wayback.md # Wayback Machine archives
|
||||
cat SKILL_DIR/references/sources/wikipedia.md # Wikipedia + Wikidata
|
||||
cat SKILL_DIR/references/sources/gdelt.md # global news monitoring
|
||||
```
|
||||
|
||||
Each entry follows a 9-section template: summary, access, schema, coverage,
|
||||
cross-reference keys, data quality, acquisition, legal, references.
|
||||
|
||||
The **cross-reference potential** section maps join keys between sources — read
|
||||
those first to pick the right pair.
|
||||
|
||||
### 2. Acquire data
|
||||
|
||||
Each source has a stdlib-only fetch script in `SKILL_DIR/scripts/`:
|
||||
|
||||
**Federal financial / regulatory**
|
||||
|
||||
```bash
|
||||
# SEC EDGAR filings (corporate disclosures)
|
||||
python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
|
||||
--types 10-K,10-Q --out data/edgar_filings.csv
|
||||
|
||||
# USAspending federal contracts
|
||||
python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
|
||||
--fy 2024 --out data/contracts.csv
|
||||
|
||||
# Senate LD-1 / LD-2 lobbying disclosures
|
||||
python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
|
||||
--year 2024 --out data/lobbying.csv
|
||||
|
||||
# OFAC SDN sanctions list (full snapshot)
|
||||
python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv
|
||||
|
||||
# ICIJ Offshore Leaks — downloads ~70 MB bulk CSV on first use,
|
||||
# then searches it locally. Cached for 30 days under
|
||||
# $HERMES_OSINT_CACHE/icij/ (default: ~/.cache/hermes-osint/icij/).
|
||||
python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
|
||||
--out data/icij.csv
|
||||
```
|
||||
|
||||
**Identity / property / litigation / archives / news**
|
||||
|
||||
```bash
|
||||
# NYC property records (deeds, mortgages, liens) — ACRIS via Socrata
|
||||
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "SMITH, JOHN" \
|
||||
--out data/acris.csv
|
||||
python3 SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" \
|
||||
--out data/acris_addr.csv
|
||||
|
||||
# OpenCorporates — 130+ jurisdiction corporate registry
|
||||
# (free token required; set OPENCORPORATES_API_TOKEN or pass --token)
|
||||
python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
|
||||
--jurisdiction us_ny --out data/opencorporates.csv
|
||||
|
||||
# CourtListener — federal + state court opinions, PACER dockets
|
||||
python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Smith v. Example Corp" \
|
||||
--type opinions --out data/courts.csv
|
||||
|
||||
# Wayback Machine — historical web captures
|
||||
python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
|
||||
--match host --collapse digest --out data/wayback.csv
|
||||
|
||||
# Wikipedia + Wikidata — narrative bio + structured facts
|
||||
# Set HERMES_OSINT_UA=your-app/1.0 (your@email) to identify yourself
|
||||
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" \
|
||||
--out data/wp.csv
|
||||
|
||||
# GDELT — global news in 100+ languages, ~2015→present
|
||||
python3 SKILL_DIR/scripts/fetch_gdelt.py --query '"Example Corp"' \
|
||||
--timespan 1y --out data/gdelt.csv
|
||||
```
|
||||
|
||||
All outputs are normalized CSV with a header row. Re-run scripts idempotently.
|
||||
|
||||
When a private individual won't be in a source (e.g. SEC EDGAR for a non-public-
|
||||
company person, USAspending for someone who isn't a federal contractor, Senate
|
||||
LDA for someone who isn't a lobbying client), the script returns 0 rows with a
|
||||
clear warning rather than silently writing an empty CSV. EDGAR specifically
|
||||
flags when the company-name resolver matched an individual Form 3/4/5 filer
|
||||
rather than a corporate registrant.
|
||||
|
||||
Rate-limit notes are in each source's wiki entry. Default fetchers sleep
|
||||
politely between paginated requests. **API keys raise rate limits** for
|
||||
sources that support them (`SEC_USER_AGENT`, `SENATE_LDA_TOKEN`,
|
||||
`OPENCORPORATES_API_TOKEN`, `COURTLISTENER_TOKEN`). All scripts surface
|
||||
429 responses immediately with the upstream's quota message so the user
|
||||
knows to slow down or supply a key.
|
||||
|
||||
### 3. Resolve entities across sources
|
||||
|
||||
Normalize names and find matches between two CSV files:
|
||||
|
||||
```bash
|
||||
# Match lobbying clients (Senate LDA) against contract recipients (USAspending)
|
||||
python3 SKILL_DIR/scripts/entity_resolution.py \
|
||||
--left data/lobbying.csv --left-name-col client_name \
|
||||
--right data/contracts.csv --right-name-col recipient_name \
|
||||
--out data/cross_links.csv
|
||||
```
|
||||
|
||||
Three matching tiers with explicit confidence:
|
||||
|
||||
| Tier | Method | Confidence |
|
||||
|------|--------|------------|
|
||||
| `exact` | Normalized strings equal after suffix/punctuation strip | high |
|
||||
| `fuzzy` | Sorted-token equality (word-bag match) | medium |
|
||||
| `token_overlap` | ≥60% token overlap, ≥2 shared tokens, tokens ≥4 chars | low |
|
||||
|
||||
Output `cross_links.csv` columns: `match_type, confidence, left_name,
|
||||
right_name, left_normalized, right_normalized, left_row, right_row`.
|
||||
|
||||
### 4. Statistical timing correlation (optional)
|
||||
|
||||
Test whether two time series cluster suspiciously close together — e.g.
|
||||
lobbying filings near contract awards — using a permutation test:
|
||||
|
||||
```bash
|
||||
python3 SKILL_DIR/scripts/timing_analysis.py \
|
||||
--donations data/lobbying.csv --donation-date-col filing_date \
|
||||
--donation-amount-col income --donation-donor-col client_name \
|
||||
--donation-recipient-col registrant_name \
|
||||
--contracts data/contracts.csv --contract-date-col award_date \
|
||||
--contract-vendor-col recipient_name \
|
||||
--cross-links data/cross_links.csv \
|
||||
--permutations 1000 \
|
||||
--out data/timing.json
|
||||
```
|
||||
|
||||
The script's column flags are intentionally generic — the original tool was
|
||||
written for donations vs awards, but it works for any (event, payee) time
|
||||
series joined through cross-links. Null hypothesis: event timing is
|
||||
independent of award dates. One-tailed p-value = fraction of permutations
|
||||
with mean nearest-award distance ≤ observed. Minimum 3 events per (payer,
|
||||
vendor) pair to run the test.
|
||||
|
||||
### 5. Build the findings JSON (evidence chain)
|
||||
|
||||
```bash
|
||||
python3 SKILL_DIR/scripts/build_findings.py \
|
||||
--cross-links data/cross_links.csv \
|
||||
--timing data/timing.json \
|
||||
--out data/findings.json
|
||||
```
|
||||
|
||||
Every finding has `id, title, severity, confidence, summary, evidence[], sources[]`.
|
||||
Each evidence item points back to a specific row in a source CSV. The user (or a
|
||||
follow-up agent) can verify every claim against its source.
|
||||
|
||||
## Confidence and evidence discipline
|
||||
|
||||
This is the load-bearing rule of the skill. Tell the user:
|
||||
|
||||
- Every claim must trace to a record. No naked assertions.
|
||||
- Confidence tier travels with the claim. `match_type=fuzzy` is "probable",
|
||||
not "confirmed."
|
||||
- Entity resolution produces candidates, NOT conclusions. A `fuzzy` match
|
||||
between "ACME LLC" and "Acme Holdings Group" is a lead, not a fact.
|
||||
- Statistical significance ≠ wrongdoing. p < 0.05 means the timing pattern
|
||||
is unlikely under the null. It does not establish corruption.
|
||||
- All data sources here are public records. They may still contain
|
||||
inaccuracies, stale info, or redactions (GDPR, sealed records).
|
||||
|
||||
## Adding a new data source
|
||||
|
||||
Use the template:
|
||||
|
||||
```bash
|
||||
cp SKILL_DIR/templates/source-template.md \
|
||||
SKILL_DIR/references/sources/<your-source>.md
|
||||
```
|
||||
|
||||
Fill in all 9 sections. Write a `fetch_<source>.py` script in `scripts/` that
|
||||
uses stdlib only and writes a normalized CSV. Update the source list in the
|
||||
"When to use" section above.
|
||||
|
||||
## Tools and their limits
|
||||
|
||||
- `entity_resolution.py` does NOT use external fuzzy libraries (no rapidfuzz,
|
||||
no jellyfish). Token-bag matching is the upper bound here. If you need
|
||||
Levenshtein, transliteration, or phonetic matching, pip-install separately.
|
||||
- `timing_analysis.py` uses Python's `random` for permutations. For
|
||||
reproducibility, pass `--seed N`.
|
||||
- `fetch_*.py` scripts use `urllib.request` and respect `Retry-After`. Heavy
|
||||
bulk usage may still violate ToS — read each source's legal section first.
|
||||
|
||||
## Legal note
|
||||
|
||||
All Phase-1 sources are public records. Bulk acquisition is permitted under
|
||||
their respective access terms (FOIA, public records law, ICIJ explicit
|
||||
publication, OFAC public data). However:
|
||||
|
||||
- Some sources rate-limit aggressively. Respect their headers.
|
||||
- Some redact registrant info (GDPR on WHOIS, sealed filings).
|
||||
- Cross-referencing public records to identify private individuals can have
|
||||
ethical implications. The skill produces evidence chains, not accusations.
|
||||
Loading…
Add table
Add a link
Reference in a new issue