Target: <2min CI test wall time.
Runs the Tests workflow as a 4-way matrix instead of one job. Each
shard runs ~3,000 tests on its own ubuntu-latest runner (4 cores) with
-n auto xdist inside. Total effective parallelism: 16 workers across
4 machines (vs 4 workers on 1 machine today).
Was previously tried in #11566 and closed — shard 3 hung at 97% complete
for 100+ seconds with dozens of E/F markers. Root cause was cross-test
pollution exposed by splitting test files across shards (e.g. the three
test files that mutated sys.modules['dotenv'] at import time poisoned
whichever shard they landed in). That's now fixed by #11453 and #11577:
conftest is hermetic, the dotenv stub bombs are removed, and tests no
longer depend on each other's env-var side effects.
Changes:
- pyproject.toml: add pytest-split>=0.9,<1 to dev extras
- .github/workflows/tests.yml: 'test' job becomes matrix-split into 4
groups with fail-fast: false. Runs 'pytest --splits 4 --group N'.
pytest-split composes with -n auto from pyproject addopts.
e2e job is unchanged (already small, 20s).
Expected timing:
Before: ~4m total (243s test step + ~25s setup)
After: ~90-115s total (shard wall time ~60-90s + ~25s setup)
Hash-based split is deterministic; no .test_durations file needed yet.
Can add one later via --store-durations for better shard balance.
* refactor: re-architect tests to mirror the codebase
* Update tests.yml
* fix: add missing tool_error imports after registry refactor
* fix(tests): replace patch.dict with monkeypatch to prevent env var leaks under xdist
patch.dict(os.environ) can leak TERMINAL_ENV across xdist workers,
causing test_code_execution tests to hit the Modal remote path.
* fix(tests): fix update_check and telegram xdist failures
- test_update_check: replace patch("hermes_cli.banner.os.getenv") with
monkeypatch.setenv("HERMES_HOME") — banner.py no longer imports os
directly, it uses get_hermes_home() from hermes_constants.
- test_telegram_conflict/approval_buttons: provide real exception classes
for telegram.error mock (NetworkError, TimedOut, BadRequest) so the
except clause in connect() doesn't fail with "catching classes that do
not inherit from BaseException" when xdist pollutes sys.modules.
* fix(tests): accept unavailable_models kwarg in _prompt_model_selection mock
Move e2e tests into tests.yml as a parallel job instead of a separate
workflow. Unit tests now also ignore tests/e2e/ to avoid running them
twice. Both jobs appear as independent checks in the PR.
~2min sequential runs were painful. Added pytest-xdist and -n auto
to run across all available cores. Tests already isolate state via
tmp_path fixtures so no changes needed to test code.
Local: 2677 passed in ~30s. CI gets 4 vCPUs on ubuntu-latest.
Run pytest on Python 3.11 + 3.12 for every PR and push to main.
- Uses uv for fast dependency installation
- Excludes integration tests (need real API keys/services)
- Blanks API keys as safety net against accidental real API calls
- Concurrency: cancels in-progress runs when new commits are pushed
- 10 minute timeout (tests take ~77s)
- fail-fast disabled so both Python versions run independently
GitHub's default 'require approval for first-time contributors'
means maintainers approve CI before it runs on new contributors'
PRs, preventing abuse of CI resources.
