ci: split Tests workflow into 4 parallel shards via pytest-split

Target: <2min CI test wall time.

Runs the Tests workflow as a 4-way matrix instead of one job. Each
shard runs ~3,000 tests on its own ubuntu-latest runner (4 cores) with
-n auto xdist inside. Total effective parallelism: 16 workers across
4 machines (vs 4 workers on 1 machine today).

Was previously tried in #11566 and closed — shard 3 hung at 97% complete
for 100+ seconds with dozens of E/F markers. Root cause was cross-test
pollution exposed by splitting test files across shards (e.g. the three
test files that mutated sys.modules['dotenv'] at import time poisoned
whichever shard they landed in). That's now fixed by #11453 and #11577:
conftest is hermetic, the dotenv stub bombs are removed, and tests no
longer depend on each other's env-var side effects.

Changes:
- pyproject.toml: add pytest-split>=0.9,<1 to dev extras
- .github/workflows/tests.yml: 'test' job becomes matrix-split into 4
  groups with fail-fast: false. Runs 'pytest --splits 4 --group N'.
  pytest-split composes with -n auto from pyproject addopts.

e2e job is unchanged (already small, 20s).

Expected timing:
  Before: ~4m total (243s test step + ~25s setup)
  After:  ~90-115s total (shard wall time ~60-90s + ~25s setup)

Hash-based split is deterministic; no .test_durations file needed yet.
Can add one later via --store-durations for better shard balance.
This commit is contained in:
Teknium 2026-04-17 06:25:42 -07:00
parent 6ea7386a6f
commit 50f23ea522
No known key found for this signature in database
2 changed files with 9 additions and 3 deletions

View file

@ -16,8 +16,13 @@ concurrency:
jobs:
test:
name: test (${{ matrix.group }}/4)
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
group: [1, 2, 3, 4]
steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
@ -37,10 +42,11 @@ jobs:
source .venv/bin/activate
uv pip install -e ".[all,dev]"
- name: Run tests
- name: Run tests (shard ${{ matrix.group }}/4)
run: |
source .venv/bin/activate
python -m pytest tests/ -q --ignore=tests/integration --ignore=tests/e2e --tb=short -n auto
python -m pytest tests/ -q --ignore=tests/integration --ignore=tests/e2e --tb=short \
--splits 4 --group ${{ matrix.group }}
env:
# Ensure tests don't accidentally call real APIs
OPENROUTER_API_KEY: ""

View file

@ -39,7 +39,7 @@ dependencies = [
[project.optional-dependencies]
modal = ["modal>=1.0.0,<2"]
daytona = ["daytona>=0.148.0,<1"]
dev = ["debugpy>=1.8.0,<2", "pytest>=9.0.2,<10", "pytest-asyncio>=1.3.0,<2", "pytest-xdist>=3.0,<4", "mcp>=1.2.0,<2"]
dev = ["debugpy>=1.8.0,<2", "pytest>=9.0.2,<10", "pytest-asyncio>=1.3.0,<2", "pytest-xdist>=3.0,<4", "pytest-split>=0.9,<1", "mcp>=1.2.0,<2"]
messaging = ["python-telegram-bot[webhooks]>=22.6,<23", "discord.py[voice]>=2.7.1,<3", "aiohttp>=3.13.3,<4", "slack-bolt>=1.18.0,<2", "slack-sdk>=3.27.0,<4", "qrcode>=7.0,<8"]
cron = ["croniter>=6.0.0,<7"]
slack = ["slack-bolt>=1.18.0,<2", "slack-sdk>=3.27.0,<4"]