Commit graph

2 commits

Author SHA1 Message Date
Teknium
6b81590c55
test: prune low-value tests suite-wide (wave 1) — 46,820 → 28,106 test functions
Systematic prune per AGENTS.md test policy, one pass over every major
test tree (gateway, hermes_cli, tools, agent, run_agent, plugins, cli,
cron, tui_gateway, honcho/openviking, root-level):

- DELETE: source-reading tests (read_text/getsource on prod files),
  change-detector tests (exact catalog counts, model-name snapshots,
  config version literals), mock-echo tests (assert a mock returns what
  it was told), assertion-free/trivial tests, near-duplicate
  parametrizations (boundaries + one representative kept), async/sync
  twin duplicates, cosmetic within-file variations.
- KEEP (mandatory): security/redaction/approval guards, message-role
  alternation invariants, prompt-caching/deterministic-call-id
  invariants, issue-number regression tests (deduped), E2E tests.
- 6 test files deleted outright (script-style/no-assert or fully
  redundant); conftest.py, fakes/, fixtures/ untouched.
- tests/acp/conftest.py added: autouse fixture stubs the live
  models.dev/GitHub/Copilot/Anthropic inventory fetches that ACP server
  tests performed on every session create — test_server.py 147s → 3.4s,
  and the tests are now genuinely hermetic.
- Sleep-based slowness shrunk where safe (codex_ttfb_watchdog,
  compression_concurrent_fork, etc.); no wall-clock assertion tightened.

Verification: full hermetic suite via scripts/run_tests.sh —
2439 files, 31,130 tests passed, 0 failed, 0 flaky retries, 315s wall
(baseline: 583s wall, 13,564s subprocess CPU).
2026-07-29 13:10:23 -07:00
Teknium
7dfd5077ce
feat(oneshot): add --usage-file JSON usage report to hermes -z (#59615)
* feat(oneshot): add --usage-file JSON usage report to hermes -z

Pipelines driving hermes -z (batch reviewers, cron scripts, eval
harnesses) had no way to account for per-invocation spend: the agent
computes estimated_cost_usd and full token counts internally, but
oneshot mode discards everything except the final response text.

- hermes -z PROMPT --usage-file PATH writes a JSON report after the
  run: estimated_cost_usd, cost_status/source, input/output/cache/
  reasoning/total tokens, api_calls, model, provider, session_id,
  completed, failed.
- Written even when the run fails (with a failure field) so callers
  can always account for spend; the write itself is best-effort and
  never masks the run's own outcome.
- Flag registered in both the full parser and the Termux fast path;
  added to both value-flag scan sets so profile detection stays
  correct.

Validation: 6 unit tests + live E2E (real -z run produced a report
with real OpenRouter cost + token counts).

* test: include usage_file kwarg in oneshot dispatch assertions

The two dispatch tests assert the exact kwargs dict passed to
run_oneshot; the new usage_file kwarg must appear there.
2026-07-06 11:10:51 -07:00