Commit graph

6 commits

Author SHA1 Message Date
Teknium
6b81590c55
test: prune low-value tests suite-wide (wave 1) — 46,820 → 28,106 test functions
Systematic prune per AGENTS.md test policy, one pass over every major
test tree (gateway, hermes_cli, tools, agent, run_agent, plugins, cli,
cron, tui_gateway, honcho/openviking, root-level):

- DELETE: source-reading tests (read_text/getsource on prod files),
  change-detector tests (exact catalog counts, model-name snapshots,
  config version literals), mock-echo tests (assert a mock returns what
  it was told), assertion-free/trivial tests, near-duplicate
  parametrizations (boundaries + one representative kept), async/sync
  twin duplicates, cosmetic within-file variations.
- KEEP (mandatory): security/redaction/approval guards, message-role
  alternation invariants, prompt-caching/deterministic-call-id
  invariants, issue-number regression tests (deduped), E2E tests.
- 6 test files deleted outright (script-style/no-assert or fully
  redundant); conftest.py, fakes/, fixtures/ untouched.
- tests/acp/conftest.py added: autouse fixture stubs the live
  models.dev/GitHub/Copilot/Anthropic inventory fetches that ACP server
  tests performed on every session create — test_server.py 147s → 3.4s,
  and the tests are now genuinely hermetic.
- Sleep-based slowness shrunk where safe (codex_ttfb_watchdog,
  compression_concurrent_fork, etc.); no wall-clock assertion tightened.

Verification: full hermetic suite via scripts/run_tests.sh —
2439 files, 31,130 tests passed, 0 failed, 0 flaky retries, 315s wall
(baseline: 583s wall, 13,564s subprocess CPU).
2026-07-29 13:10:23 -07:00
Teknium
3dd8059a05
fix(tests): eliminate flaky/broken tests — shadow sys.path inserts, unmocked network in compressor tests, stale-SDK feishu pin guard, quadratic redact regexes
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
  it prepended the tests/ dir itself to sys.path, so 'import agent' /
  'import hermes_cli' resolved to the test packages and collection died
  with ModuleNotFoundError depending on import order (2 files failed in
  every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
  unmocked: each burned ~50s attempting live LLM traffic through the
  relay before falling back (572s file — the slowest in the suite, and
  flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
  the compressor's redaction pass on large payloads —
  _STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
  prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
  output-equivalence fuzz-verified on 20k random strings), and the
  _CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
  pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
  signature check; the repo pins lark-oapi==1.6.8 but stale local
  installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
  agent.redact + agent.credential_persistence in the fake agent package
  (empty __path__ blocks all real agent.* imports added since the fake
  was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
  2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
  baseline run (passes instantly when the box is quiet).
2026-07-29 12:18:07 -07:00
teknium1
c92e2c0fbf test: age the disk L2 entry too in the server-swap redetection test
The TTL-expiry test aged only the in-process probe cache; with the new
disk L2 the fresh disk verdict (correctly) served 'ollama' and the swap
to lm-studio wasn't re-detected. In real time-flow the 300s disk TTL
always lapses before the 1h in-proc TTL — the test now compresses both
expiries, matching the scenario it describes. 25/25 pass.
2026-07-29 10:52:13 -07:00
Kshitij Kapoor
f556edc10d fix(model_metadata): address Phase-2 review findings on probe caches
Structured review (2a/2b/2c) findings, all fixed:

- MAJOR: detect_local_server_type memo was process-lifetime with no
  invalidation, permanently pinning a URL's server type. Now a bounded
  1h TTL ((type, monotonic) tuples) so a backend swap on the same port
  is re-detected. Test covers ollama->lm-studio swap after expiry.

- MAJOR: legacy disk-row compat was one-way. get_cached_context_length
  and _invalidate_cached_context_length now consult the same key-shape
  set {canonical, literal, canonical+slash} in both directions, so an
  old slashed row is found (and cleared) when the runtime passes the
  normalized URL. Tests pin both migration directions.

- MINOR: _localhost_to_ipv4 did whole-string replacement, which could
  corrupt a proxy URL embedding http://localhost in its query. Now a
  scheme-anchored host-only regex; localhost.example.com and embedded
  substrings pass through. Tests added.

- MINOR: _invalidate_cached_context_length now also drops the
  in-memory TTL probe rows for the pair, so a resolution inside the
  TTL window can't re-persist the value just declared stale.

- Test gaps closed: detect-type cache hit + TTL-expiry re-detection,
  ollama-show TTL expiry re-probe, reverse legacy-row lookups.

- Attribution gate: added zhchl@hermes-agent.local -> 8294 (PR #50572
  author) to AUTHOR_MAP; the strict CI grep needs bare non-plus emails
  literal in release.py.

Gates: ruff clean; targeted suites 202 passed / 0 failed; full
tests/agent 5426 passed with 17 failures identical on clean
upstream/main (pre-existing env-dependent anthropic/bedrock/credpool
tests); mypy delta vs base: 0 new errors; live smoke 6/6 PASS.
2026-07-09 14:08:44 +05:30
Kshitij Kapoor
20cb385328 fix(model_metadata): widen localhost->IPv4 rewrite to all sibling probe sites
#37595 fixed the Windows dual-stack IPv6 timeout only inside
detect_local_server_type. The same 2s-per-probe penalty existed at every
other helper that builds a probe URL from base_url. Extract the rewrite
into _localhost_to_ipv4() and apply it at:

- query_ollama_num_ctx
- query_ollama_supports_vision
- _query_ollama_api_show (server_url derivation)
- _query_local_context_length (server root + LM Studio native URL)

Tests cover the helper's URL forms, non-localhost passthrough, and that
the ollama probes actually POST to 127.0.0.1.
2026-07-09 14:08:44 +05:30
Kshitij Kapoor
040e30aa72 perf(model_metadata): cache ollama /api/show probe + normalize context-cache keys
Follow-up hunks completing the probe-cache cluster:

1. _query_ollama_api_show now goes through the existing
   _LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) —
   it was the one remaining per-resolution POST not covered by the
   #56431-era wrapper. Failures are never memoized so a server that
   comes up mid-startup is re-probed. Idea credit: #42081 (@Morad37),
   reworked to comply with the positive-only rule.

2. Persistent context-cache keys are normalized through
   _context_cache_key (trailing-slash strip) so http://host/v1 and
   http://host/v1/ share one entry; reads and invalidation honor
   legacy un-normalized rows. Idea credit: #37905 (@stevenau21).

Tests: TTL hit collapses to one POST, failure-not-memoized
(mutation-verified: unconditional caching makes it fail), namespace
no-collision vs the sibling probe, slash-variant dedup, legacy-row
read, dual-shape invalidation.
2026-07-09 14:08:44 +05:30