The profiler now monkey-patches time.sleep alongside subprocess.run,
capturing the invisible "gap" time from polling loops (wait_for_container_ready,
poll_container, wait_for_log, etc.) that was previously unaccounted for.
The JSON report now includes per-test total_sleep_s, sleep_count, and
a sleeps[] array with caller location. The summary includes
total_sleep_s and total_wall_s (docker + sleep). The CI merge step
also aggregates sleep totals.
Local profile now shows: 201s docker + 75s sleep = 276s wall (38s
runner wall with 32-way parallelism). The biggest sleep consumer is
test_dashboard_insecure_env_var_no_longer_bypasses at 11.8s of
poll_container sleeps.
Two bugs fixed:
1. scripts/run_tests.sh uses env -i (empty environment) and only
passes through a hand-picked set of vars. HERMES_DOCKER_TEST_PROFILE
and HERMES_DOCKER_PROFILE_OUT were not in that list, so the env var
set in the CI workflow was stripped before reaching the pytest
subprocesses — the plugin never activated and no JSON was produced.
2. run_tests_parallel.py spawns each test file in its own subprocess,
so each would write to the same docker-test-profile.json, clobbering
each other. Changed the default output path to include the PID
(docker-test-profile-<pid>.json) and added a CI merge step that
combines all per-PID files into a single docker-test-profile.json
before uploading as an artifact.
Add a pytest profiling plugin (tests/docker/profiling.py) that
instruments every subprocess.run call targeting docker, collecting
per-test and per-subcommand wall-clock timings. Activated via
HERMES_DOCKER_TEST_PROFILE=1 — zero overhead when unset.
The CI docker.yml workflow now enables profiling and uploads the
JSON report as an artifact (docker-test-profile-<arch>), so we can
see exactly which docker operations (run, exec, restart, polling)
dominate the slow CI runs vs. fast local runs.
Output includes:
- Per-subcommand summary (call count, total/avg time, %)
- Top 10 slowest tests by docker operation time
- Top 10 slowest individual docker calls
- Full JSON report for offline analysis
