mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-05-26 06:01:49 +00:00

feat(skills): add osint-investigation optional skill (closes #355 ) (#26729 )

* feat(skills): add osint-investigation optional skill (closes #355)

Phase-1 public-records OSINT investigation framework adapted from
ShinMegamiBoson/OpenPlanter (MIT). Lives in optional-skills/research/.

Six data-source wiki entries (FEC, SEC EDGAR, USAspending, Senate LD,
OFAC SDN, ICIJ Offshore Leaks), each following the 9-section template:
summary, access, schema, coverage, cross-reference keys, data quality,
acquisition, legal, references.

Six stdlib-only acquisition scripts that emit normalized CSV, plus three
analysis scripts:

  - entity_resolution.py  — three-tier match (exact / fuzzy / token overlap)
                            with explicit confidence per row
  - timing_analysis.py    — permutation test for donation/contract timing
                            correlation, joins through cross-links
  - build_findings.py     — assembles structured findings.json with
                            evidence chains pointing back to source rows

Validation: full pipeline runs end-to-end on synthetic fixtures. Entity
resolution found 24 cross-matches with 0 false positives on a 5-row /
4-row test set. Timing analysis on 5 donations clustered near 3 awards
returned p=0.000, effect size 2.41 SD. Findings JSON correctly tags
HIGH-severity timing pattern. All 9 scripts pass --help and py_compile.

Docs site page auto-generated by website/scripts/generate-skill-docs.py;
sidebar + catalog entries updated by the same generator.

* fix(osint-investigation): live API fixes from end-to-end sweep

Live-tested the skill on a real public-citizen query and found three bugs
the synthetic E2E missed. All three are now fixed and re-verified.

1. FEC fetch hung on contributor name searches.
   The combination of two_year_transaction_period + sort=date +
   contributor_name puts the OpenFEC query plan on a slow path that the
   upstream gateway times out (25s+). Switched to min_date/max_date with no
   explicit sort. Renamed --candidate to --contributor (the original name
   was misleading: FEC searches by donor, not by candidate; --candidate is
   kept as a deprecated alias). Added --state filter for narrowing.

2. ICIJ Offshore Leaks reconcile endpoint returns 404.
   ICIJ removed the Open Refine reconciliation API. Rewrote
   fetch_icij_offshore.py to download the official bulk CSV ZIP (~70 MB,
   public, no auth) and search it locally. Cached under
   $HERMES_OSINT_CACHE/icij/ (default ~/.cache/hermes-osint/icij/) for
   30 days, --force-refresh to refetch. Verified live: 'PUTIN' query
   returns 5 Panama Papers officer matches in 0.5s after first download.

3. SEC EDGAR silently returned 0 when the company-name resolver matched
   an individual Form 3/4/5 filer (insider trading disclosures).
   Now surfaces 'Resolved company X → CIK Y (Z)' on stderr, prints a
   filing-type histogram when the type filter wipes results, and
   explicitly warns when the matched CIK appears to be an individual
   filer rather than a corporate registrant.

Bonus: _http.py was retrying 429 responses with exponential backoff plus
honoring (often-missing) Retry-After headers, which compounded into
multi-second hangs per page when the upstream key was over quota.
Changed to fail-fast on 429 with a clear, actionable error showing the
upstream's quota message. Verified: 0.3s fast-fail vs the previous 60s
hang on DEMO_KEY rate-limit exhaustion.

Updated SKILL.md, fec.md, and icij-offshore.md to match the new CLI
flags and ICIJ bulk-cache flow. Regenerated the docusaurus page via
website/scripts/generate-skill-docs.py.

Live sweep results across all 6 sources for 'Dillon Rolnick, New York':
- OFAC SDN: 0 matches ✓ (correctly not sanctioned)
- USAspending: 0 matches ✓ (correctly not a federal contractor)
- Senate LDA: 0 matches ✓ (correctly not a lobbying client)
- SEC EDGAR: warns it resolved to 'Rolnick Michael' (CIK 0001845264)
    who is an individual Form 3 filer, not a corporate registrant
- ICIJ: 0 matches ✓ (correctly not in any offshore leak)
- FEC: rate-limited (DEMO_KEY); fails fast with clear quota message

* feat(osint-investigation): expand to 12 sources covering identity, property, courts, archives, news

Phase-2 expansion per Teknium feedback that the original 6-source skill
(federal financial/regulatory only) wasn't a complete OSINT toolkit. Adds
6 more sources covering the major omissions a real investigation would
reach for first.

New sources (6 fetch scripts + 6 wiki entries):

1. NYC ACRIS — Real property records (deeds, mortgages, liens) via the
   city's Socrata API. Search by party name or property address. Joins
   Parties to Master to populate doc_type, dates, borough, and amount.
   Coverage: 5 NYC boroughs, ~70M party records, 1966-present.

2. OpenCorporates — Global corporate registry covering 130+ jurisdictions
   (~200M companies). Free API token at
   https://opencorporates.com/api_accounts/new raises the rate limit;
   HTML fallback works without one (limited fields).

3. CourtListener (Free Law Project) — federal + state court opinions
   (~10M back to colonial era) + PACER dockets via RECAP. Anonymous v4
   search works; COURTLISTENER_TOKEN raises rate limits.

4. Wayback Machine CDX — historical web captures (~900B+). Used both for
   surveillance-of-record (when did this site change?) and as a
   content-recovery layer when other sources point to dead URLs.

5. Wikipedia + Wikidata — narrative bio + structured facts. Wikipedia
   OpenSearch for article matching, REST summary for extracts, Wikidata
   Action API (wbgetentities) for claims. Avoids the SPARQL Query
   Service which is aggressively rate-limited.

6. GDELT 2.0 DOC API — global news monitoring in 100+ languages,
   ~2015-present. Auto-retries with 6s backoff on the standard
   1-req-per-5-sec throttle.

Other changes in this commit:

- SEC EDGAR no longer raises SystemExit when the company-name resolver
  finds no CIK; writes an empty CSV with header so the rest of a
  pipeline can keep moving and the warning is just on stderr.

- _http.py User-Agent updated per Wikimedia policy: includes app name,
  version, and a 'set HERMES_OSINT_UA to identify yourself' instruction.

- SKILL.md workflow now groups sources into two clusters (federal
  financial vs identity/property/courts/archives/news) with bash
  examples for each. 'When to use this skill' lists the broader set of
  investigation patterns the expanded sources unlock.

Live sweep results on 'Dillon Rolnick, New York' across all 12 sources:

  ofac           ✓ 0 (correctly clean)
  icij           ✓ 0 (correctly not in any leak)
  usaspending    ✓ 0 (correctly not a federal contractor)
  senate_lda     ✓ 0 (correctly not a lobbying client)
  sec_edgar      ✓ 0, warns: resolved to 'Rolnick Michael' (CIK 0001845264),
                   individual Form 3 filer, NOT a corporate registrant
  fec            — rate-limited (DEMO_KEY exhausted), fails fast with
                   clear quota message
  nyc_acris      ✓ 200 records named Rolnick across NYC; 48 records at
                   571 Hudson (the property the web identifies as his)
  opencorporates ✓ 0 (no API token configured; HTML fallback)
  courtlistener  ✓ 0 for 'Dillon Rolnick'; 20 for 'Rolnick' generally;
                   5 for 'Microsoft' sanity check
  wayback        ✓ 30 captures of nousresearch.com from 2011-present
  wikipedia      ✓ 0 (correctly not notable enough); Bill Gates sanity
                   returns full structured facts (occupation, employer,
                   DOB, place of birth, country)
  gdelt          ✓ 0 for 'Dillon Rolnick'; 5 for 'Nous Research'

All 17 scripts compile clean and pass --help. Synthetic analysis pipeline
regression still passes (entity_resolution 30 matches, timing p=0.000,
findings 2).

* feat(osint-investigation): remove FEC; DEMO_KEY rate-limits make it unreliable

The FEC fetcher consistently failed the live sweep because the OpenFEC
DEMO_KEY tier (40 calls/hour) exhausts on a single investigation, and
the upstream returns slow-path query plans for unindexed contributor-name
searches that the gateway times out. Without a real API key it's not
usable; with one the user has to sign up at api.data.gov first. That's
too much setup friction for a skill that should work out of the box.

Removed:
  - scripts/fetch_fec.py
  - references/sources/fec.md

Updated:
  - SKILL.md frontmatter description + tags
  - 'When NOT to use' now points users at https://www.fec.gov/data/ for
    federal donations
  - entity_resolution example switched from donor↔contractor to
    lobbying-client↔contractor (Senate LDA + USAspending pair)
  - timing_analysis example switched to lobbying-filings vs awards
  - 8 wiki entries had their 'FEC ↔ ...' cross-reference bullets removed

11 sources remain (5 federal financial + 6 identity/property/courts/
archives/news). All scripts compile, pass --help, and the synthetic
analysis pipeline still passes on the new lobbying-shaped regression
fixture (30 matches, p=0.000 on tight clustering, 2 findings).

2026-05-16 01:55:06 -07:00

4.1 KiB

Raw Blame History

Wikipedia + Wikidata

1. Summary

Wikipedia is the canonical narrative-bio source for notable people, places, and organizations. Wikidata is its structured-data counterpart: ~110M items, each with claims, dates, identifiers, and cross-references to external authorities (VIAF, ISNI, ORCID, GRID, etc.).

Together they're a high-precision entity-resolution layer — the bar for inclusion is real, but anything past that bar is well-cross-referenced.

2. Access Methods

Wikipedia OpenSearch: https://en.wikipedia.org/w/api.php?action=opensearch
Wikipedia REST summary: https://en.wikipedia.org/api/rest_v1/page/summary/<title>
Wikidata Action API: https://www.wikidata.org/w/api.php?action=wbgetentities
Wikidata SPARQL: https://query.wikidata.org/sparql (more powerful but aggressively rate-limited)
Auth: None, but a meaningful User-Agent is required

Set HERMES_OSINT_UA to something identifying (e.g. your-app/1.0 (you@example.com)). Wikimedia returns HTTP 429 to generic UAs.

3. Data Schema

Key fields emitted by fetch_wikipedia.py:

Column	Type	Description
`source`	str	`wikipedia` or `wikipedia+wikidata`
`label`	str	Wikipedia article title
`description`	str	Short Wikidata description
`qid`	str	Wikidata QID (e.g. Q2283 for Microsoft)
`wikipedia_title`, `wikipedia_url`	str	Article identifier + URL
`wikidata_url`	str	Wikidata entity URL
`instance_of`	str	What kind of thing it is (P31)
`country`	str	Country (P17 for orgs/places, P27 for people)
`occupation`	str	P106
`employer`	str	P108
`date_of_birth`	str	P569, YYYY-MM-DD
`place_of_birth`	str	P19
`summary`	str	Wikipedia REST extract (~1000 chars)

The fetch script uses Wikidata's Action API (NOT SPARQL) for structured facts — far more lenient on rate limits.

4. Coverage

Wikipedia EN: ~7M articles
Wikidata: ~110M items, ~1.5B statements
Updated continuously; abuse filters and bots run constantly
High notability bar — most private individuals are not in Wikipedia

5. Cross-Reference Potential

All sources ↔ label (entity identity resolution)
SEC EDGAR ↔ label (public companies)
CourtListener ↔ label (parties to notable litigation)
Wikidata external identifiers (not currently in this fetcher's output) link to VIAF, ISNI, ORCID, GRID, GitHub, Twitter, IMDb, ...

Join key: Wikidata QID is canonical. Wikipedia titles are stable for most articles but can be renamed.

6. Data Quality

Notability filter — only notable entities (criteria vary by topic)
Recency lag — current events take days to weeks to be reflected
POV / vandalism — moderated, but edits between sweeps can be bad
Living-persons biographies have stricter sourcing requirements
Wikidata claims have qualifiers and references — the fetch script doesn't currently export them

7. Acquisition Script

Path: scripts/fetch_wikipedia.py

# Look up a notable entity
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --out data/wp.csv

# A specific person
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" --out data/wp_bg.csv

# Skip the Wikidata enrichment for speed
python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --no-wikidata \
    --limit 5 --out data/wp.csv

The OpenSearch is fuzzy — --limit 5 returns the top 5 Wikipedia article matches. Each is enriched with the QID + structured facts unless --no-wikidata is passed.

8. Legal & Licensing

Wikipedia text: CC-BY-SA-3.0 / GFDL
Wikidata claims: CC0 (public domain)
API ToS: respect rate limits, identify your agent
Commercial use allowed with attribution

9. References

Wikipedia OpenSearch: https://www.mediawiki.org/wiki/API:Opensearch
Wikipedia REST: https://en.wikipedia.org/api/rest_v1/
Wikidata Action API: https://www.wikidata.org/wiki/Wikidata:Data_access
Wikidata SPARQL: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service
User-Agent policy: https://meta.wikimedia.org/wiki/User-Agent_policy

4.1 KiB Raw Blame History