fix(tests): harden env isolation and replace remaining sleep-sync races

The full 42k-test run and complete npm check surfaced three more classes:

- Environment isolation: local ~/.honcho defaultHost and SSH_* variables
  leaked into Python/TUI tests. Pin the default Honcho host in the
  hermetic fixture, isolate the one fallback test from ~/.honcho, and
  blank SSH_* around terminalSetup tests. This flipped 20 false failures
  back to deterministic behavior on developer machines.
- Background-thread sleep-sync: Honcho async writer tests patched
  time.sleep globally, then busy-polled with that same mocked sleep. Under
  full-suite load the poller could starve the writer. Each test now waits
  on an Event emitted by the exact flush/retry transition; 30/30 passed
  under 15-way contention.
- Desktop streaming: the test slept 80ms and assumed a 500ms timer could
  not fire before its assertion. A loaded runner descheduled the test for
  >500ms and both chunks arrived. Producer controls now gate second-chunk
  and completion transitions explicitly.

Also make file-retry observability complete: a self-healed flaky file now
prints BOTH attempts' full output in the FLAKY summary. Two behavioral
runner tests prove pass-on-retry is green+loud+traceback-preserving, while
a deterministic failure remains red.
This commit is contained in:
Teknium 2026-07-17 07:58:09 -07:00
parent a27c8c94d9
commit ca115aac0b
No known key found for this signature in database
7 changed files with 177 additions and 41 deletions

View file

@ -241,8 +241,8 @@ def _run_one_file(
``retries`` > 0 enables the one-shot flake retry: a non-zero exit is
re-run in a fresh subprocess; if the re-run passes, the file counts as
passed but the output is prefixed with a FLAKY banner and the file is
recorded in ``_FLAKY_FILES`` so the summary can call it out. A
passed but the output is prefixed with a FLAKY banner and the file/output
are recorded in ``_FLAKY_RESULTS`` so the summary can call it out. A
deterministic failure fails every attempt, so real regressions cannot
be laundered green.
@ -279,19 +279,22 @@ def _run_one_file(
)
subproc_wall += subproc_wall2
if rc == 0:
with _flaky_lock:
_FLAKY_FILES.append(file)
output = (
f"⚠ FLAKY: failed on attempt 1, passed on retry "
f"(attempt {attempt + 1}). Fix the flake — do not ignore this.\n"
f"--- first-attempt output ---\n{first_output}\n"
f"--- retry output ---\n{output}"
)
with _flaky_lock:
_FLAKY_RESULTS.append((file, output))
return file, rc, output, summary, subproc_wall
# Files that failed once and passed on retry — reported in the summary.
_FLAKY_FILES: List[Path] = []
# Files that failed once and passed on retry, with both attempts' output.
# Keeping the traceback is load-bearing: a self-healed flake without its
# failing assertion is only a filename, which forces another expensive full
# run to rediscover the race.
_FLAKY_RESULTS: List[Tuple[Path, str]] = []
_flaky_lock = threading.Lock()
@ -958,11 +961,12 @@ def main() -> int:
# Flaky files: failed once, passed on the automatic retry. Green, but
# loudly reported so they get fixed instead of silently re-flaking.
if _FLAKY_FILES:
if _FLAKY_RESULTS:
print()
print(f"=== ⚠ {len(_FLAKY_FILES)} FLAKY file{'s' if len(_FLAKY_FILES) != 1 else ''} (failed once, passed on retry — fix these) ===")
for f in _FLAKY_FILES:
print(f"=== ⚠ {len(_FLAKY_RESULTS)} FLAKY file{'s' if len(_FLAKY_RESULTS) != 1 else ''} (failed once, passed on retry — fix these) ===")
for f, output in _FLAKY_RESULTS:
print(f" {_format_file(f, repo_root)}")
print(output.rstrip())
# Save durations for future --slice runs. Each slice writes its own
# partial test_durations.json; a CI merge step joins them later.