Commit graph

2 commits

Author SHA1 Message Date
Teknium
d0e1388ca9
fix(tests): make AIAgent constructor calls self-contained (#11755)
* fix(tests): make AIAgent constructor calls self-contained (no env leakage)

Tests in tests/run_agent/ were constructing AIAgent() without passing
both api_key and base_url, then relying on leaked state from other
tests in the same xdist worker (or process-level env vars) to keep
provider resolution happy. Under hermetic conftest + pytest-split,
that state is gone and the tests fail with 'No LLM provider configured'.

Fix: pass both api_key and base_url explicitly on 47 AIAgent()
construction sites across 13 files. AIAgent.__init__ with both set
takes the direct-construction path (line 960 in run_agent.py) and
skips the resolver entirely.

One call site (test_none_base_url_passed_as_none) left alone — that
test asserts behavior for base_url=None specifically.

This is a prerequisite for any future matrix-split or stricter
isolation work, and lands cleanly on its own.

Validation:
- tests/run_agent/ full: 760 passed, 0 failed (local)
- Previously relied on cross-test pollution; now self-contained

* fix(tests): update opencode-go model order assertion to match kimi-k2.5-first

commit 78a74bb promoted kimi-k2.5 to first position in model suggestion
lists but didn't update this test, which has been failing on main since.
Reorder expected list to match the new canonical order.
2026-04-17 12:32:03 -07:00
Teknium
5797728ca6
test: regression guards for the keepalive/transport bug class (#10933) (#11266)
Two new tests in tests/run_agent/ that pin the user-visible invariant
behind AlexKucera's Discord report (2026-04-16): no matter how a future
keepalive / transport fix for #10324 plumbs sockets in, sequential
chats on the same AIAgent instance must all succeed.

test_create_openai_client_reuse.py (no network, runs in CI):
- test_second_create_does_not_wrap_closed_transport_from_first
    back-to-back _create_openai_client calls must not hand the same
    http_client (after an SDK close) to the second construction
- test_replace_primary_openai_client_survives_repeated_rebuilds
    three sequential rebuilds via the real _replace_primary_openai_client
    entrypoint must each install a live client

test_sequential_chats_live.py (opt-in, HERMES_LIVE_TESTS=1):
- test_three_sequential_chats_across_client_rebuild
    real OpenRouter round trips, with an explicit
    _replace_primary_openai_client call between turns 2 and 3.
    Error-sentinel detector treats 'API call failed after 3 retries'
    replies as failures instead of letting them pass the naive
    truthy check (which is how a first draft of this test missed
    the bug it was meant to catch).

Validation:
  clean main (post-revert, defensive copy present)
    -> all 4 tests PASS
  broken #10933 state (keepalive injection, no defensive copy)
    -> all 4 tests FAIL with precise messages pointing at #10933

Companion to taeuk178's test_create_openai_client_kwargs_isolation.py,
which pins the syntactic 'don't mutate input dict' half of the same
contract. Together they catch both the specific mechanism of #10933
and any other reimplementation that breaks the sequential-call
invariant.
2026-04-16 16:36:33 -07:00