4 commits

Author	SHA1	Message	Date
Teknium	597615ade4	fix(ci): make tests, workflows, and attribution reliable under load (#66373) ... * feat(attribution): conflict-free contributor mappings via contributors/emails/ directory The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet: every concurrent salvage PR appended entries to the same lines of the same file, so parallel PRs re-conflicted on every merge to main. New system: one file per email under contributors/emails/ — filename is the commit-author email, first non-comment line is the GitHub login. File additions never conflict, so any number of PRs can add mappings concurrently. - scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen) merged with the directory at import time (directory wins). All existing consumers (resolve_author, contributor_audit.py) unchanged. - scripts/add_contributor.py: idempotent CLI to add a mapping; refuses conflicting reassignments (incl. against the legacy map), validates email/login shapes. - contributor-check.yml: attribution gate now accepts a mapping file OR a legacy entry; failure message prints the exact add_contributor command. Also auto-resolves bare <login>@users.noreply.github.com emails is intentionally NOT added (kept id+login form only, matching previous behavior). - contributor_audit.py: guidance now points at add_contributor.py. - tests/scripts/test_contributor_map.py: 12 tests covering loader, merge precedence, CLI idempotency/conflict/validation, subprocess E2E. * feat(ci): one-shot per-file flake retry in the parallel test runner A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry counts as green but is loudly reported in a '⚠ FLAKY' summary section (with both attempts' output preserved) so the flake gets fixed instead of eating a full-run rerun. Deterministic failures fail both attempts — regressions cannot be laundered green. - --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables) - E2E verified: simulated first-run-fail flake goes green with banner; deterministic failure still exits 1; retries=0 restores old behavior. This converts the dominant CI failure mode (one timing-sensitive test flaking a 4600-test shard, requiring a manual 10-minute rerun and an agent triage loop) into a self-healing retry that costs one file's runtime. * test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s These guard against catastrophic regex backtracking (seconds-to-minutes class), but 0.15s is within scheduler-stall noise on loaded shared CI runners — test_max_accepted_separator_free_input_is_fast failed a CI shard this week on runner load alone. 2.0s still catches the regression class with zero flake surface. * fix(ci): job timeouts everywhere + retries on all network installs Reliability pass over every workflow: - timeout-minutes on all 21 jobs that lacked one (a hung job previously burned the 6-hour default runner budget) - ./.github/actions/retry wrapped around every network-fetching install that lacked it: pip installs (deploy-site, skills-index), npm ci (deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker test deps). Deterministic build steps (npm run build) deliberately NOT retried — split into separate steps so a real build failure fails fast instead of retrying 3x. * docs(agents): document the file-retry flake policy * fix(ci): curl retries on deploy hook + skills-index probe * fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile From the workflow reliability audit: - tests.yml: duration-cache restore had NO restore-keys while saves use run_id-suffixed keys — the cache never matched once, so LPT slicing always ran blind and unbalanced slices pushed heavy files toward the per-file timeout. One-line restore-keys fixes slice balancing. - Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed): 'gh pr view || true' turned an API blip into 'label absent' → false BLOCKING failure. Now 3x retry, and API failure is reported as an API failure instead of a missing label. - detect-changes action: compare API retried before failing open (was silently running all lanes on any blip). - uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried so registry blips don't read as 'lockfile stale'. - docker.yml merge job: imagetools create retried (Docker Hub eventual consistency on just-pushed digests). - Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to curl --retry 3 (ADD cannot retry; checksums still enforced); npm --fetch-retries=5; playwright chromium fetch retried 3x. - Advisory artifact uploads (per-slice durations, ci-timings report) get continue-on-error so an artifact-service blip can't fail a green test slice. * fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list - test_tui_gateway_server.py: session.create / non-eager session.resume arm a 50ms threading.Timer (_schedule_agent_build) that outlives its test and fires into the NEXT test's _make_agent mock, racily corrupting captured state (the recurring session_resume shard failures). Replaced the per-test whack-a-mole stub with a module-wide autouse fixture; the 3 worker-lifecycle tests that genuinely need the deferred build opt back in via @pytest.mark.real_agent_prewarm (new marker in pyproject). - test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the live PROVIDER_REGISTRY instead of a hand-list that had drifted (missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto') tests failed on any machine with HF_TOKEN exported. E2E-verified with HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass. * test: de-flake 30 timing-sensitive test files for loaded CI runners Root-cause fixes from the flake audit (session-DB mining + repo sweep): Event-based sync instead of sleep-sync: - title_generator: mock sets threading.Event, wait(10) replaces sleep(0.3) hoping the daemon thread got scheduled - docker zombie_reaping / profile_gateway: poll-for-state helpers replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async) - process_registry tree test: select()-bounded readline replaces an unbounded blocking read (parent wedge now fails THIS test with a clear message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s (the 1s partition window mid-interpreter-startup is how a child PID escaped the live-system guard in CI) Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors; all of these complete in ms-to-1s when healthy so the raises cost nothing on green runs): - subprocess/thread waits <= 2s raised to 10-15s across mcp_tool, mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe, mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt, voice_cli_integration, docker_environment, session_store_lock_io, planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output (joins now also assert not is_alive() so stragglers fail loudly) - wall-clock discrimination ceilings loosened where the guarded hang is 10x larger: local_background_child_hang 4s->10s, interrupt_cleanup setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup 5s->15s, protocol/gil-starvation fast-handler 0.5s->2s, iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s - narrow assertion windows widened: honcho first-turn wait 0.4..0.65 -> 0.25..2.0 (property is bounded-not-hung, not an exact wall-clock); compression fork-lock TTL 1s->3s (12 refresh chances per lease); compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0) - telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under) * fix(tests): repair indentation from de-flake batch edit * fix(tests): harden env isolation and replace remaining sleep-sync races The full 42k-test run and complete npm check surfaced three more classes: - Environment isolation: local ~/.honcho defaultHost and SSH_* variables leaked into Python/TUI tests. Pin the default Honcho host in the hermetic fixture, isolate the one fallback test from ~/.honcho, and blank SSH_* around terminalSetup tests. This flipped 20 false failures back to deterministic behavior on developer machines. - Background-thread sleep-sync: Honcho async writer tests patched time.sleep globally, then busy-polled with that same mocked sleep. Under full-suite load the poller could starve the writer. Each test now waits on an Event emitted by the exact flush/retry transition; 30/30 passed under 15-way contention. - Desktop streaming: the test slept 80ms and assumed a 500ms timer could not fire before its assertion. A loaded runner descheduled the test for >500ms and both chunks arrived. Producer controls now gate second-chunk and completion transitions explicitly. Also make file-retry observability complete: a self-healed flaky file now prints BOTH attempts' full output in the FLAKY summary. Two behavioral runner tests prove pass-on-retry is green+loud+traceback-preserving, while a deterministic failure remains red. * refactor(ci): use gh bot pat, better retries refactor(ci): use retry action for PR label fetch the retry action now captures stdout as a step output, so it can serve double duty: retry + output capture for commands like 'gh pr view' whose result must be consumed by later steps. Retry action gains: - 'stdout' output (heredoc-delimited to preserve newlines) - tee to temp file so stdout still streams to the job log - step id 'retry' for output reference Both lint.yml and supply-chain-audit.yml now use the retry action directly with 'command: gh pr view ...' and read steps.<id>.outputs.stdout. ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth Replace secrets.GITHUB_TOKEN and github.token with secrets.AUTOFIX_BOT_PAT across all workflows and composite actions that use the gh CLI or GitHub API. The PAT has consistent permissions across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate limit sharing with the default token, and is already used by js-autofix.yml for the same reasons. 19 sites swapped across 9 files: - lint.yml (3): label fetch, comment post/edit, comment update - supply-chain-audit.yml (5): scan, critical comment, unbounded dep comment, label fetch, mcp-catalog comment - lockfile-diff.yml (1): PR comment post/update - skills-index-freshness.yml (1): issue creation on degraded probe - skills-index.yml (2): index build, trigger deploy workflow - upload_to_pypi.yml (2): release view poll, release upload - ci.yml (1): timings report - deploy-site.yml (2): skills index crawl - detect-changes/action.yml (1): compare API call --------- Co-authored-by: ethernet <arilotter@gmail.com>	2026-07-17 20:55:24 +00:00
kshitijk4poor	66827f8947	chore: prune unused imports and duplicate import redefinitions ... Remove unused imports (F401) and duplicate/shadowed import redefinitions (F811) across the codebase using ruff's safe autofixes. No behavioral changes -- imports only. - ~1400 safe autofixes applied across 644 files (net -1072 lines) - __init__.py re-exports preserved (excluded from F401 removal so public re-export surfaces stay intact) - Re-exports that are imported or monkeypatched by tests but look unused in their defining module are kept with explicit # noqa: F401 (gateway/run.py load_dotenv; run_agent re-exports from agent.message_sanitization, agent.context_compressor, agent.retry_utils, agent.prompt_builder, agent.process_bootstrap, agent.codex_responses_adapter) - Unsafe F841 (unused-variable) fixes deliberately skipped -- those can change behavior when the RHS has side effects - ruff lints remain disabled in pyproject.toml (only PLW1514 is selected); this is a one-time cleanup, not a config change Verification: - python -m compileall: clean - pytest --collect-only: all 27161 tests collect (zero import errors) - core entry points import clean (run_agent, model_tools, cli, toolsets, hermes_state, batch_runner, gateway) - static scan: every name any test imports directly from an edited module still resolves	2026-05-28 22:26:25 -07:00
Teknium	2558d28a9b	fix: resolve CI test failures — add missing functions, fix stale tests (#9483) ... Production fixes: - Add clear_session_context() to hermes_logging.py (fixes 48 teardown errors) - Add clear_session() to tools/approval.py (fixes 9 setup errors) - Add SyncError M_UNKNOWN_TOKEN check to Matrix _sync_loop (bug fix) - Fall back to inline api_key in named custom providers when key_env is absent (runtime_provider.py) Test fixes: - test_memory_user_id: use builtin+external provider pair, fix honcho peer_name override test to match production behavior - test_display_config: remove TestHelpers for non-existent functions - test_auxiliary_client: fix OAuth tokens to match _is_oauth_token patterns, replace get_vision_auxiliary_client with resolve_vision_provider_client - test_cli_interrupt_subagent: add missing _execution_thread_id attr - test_compress_focus: add model/provider/api_key/base_url/api_mode to mock compressor - test_auth_provider_gate: add autouse fixture to clean Anthropic env vars that leak from CI secrets - test_opencode_go_in_model_list: accept both 'built-in' and 'hermes' source (models.dev API unavailable in CI) - test_email: verify email Platform enum membership instead of source inspection (build_channel_directory now uses dynamic enum loop) - test_feishu: add bot_added/bot_deleted handler mocks to _Builder - test_ws_auth_retry: add AsyncMock for sync_store.get_next_batch, add _pending_megolm and _joined_rooms to Matrix adapter mocks - test_restart_drain: monkeypatch-delete INVOCATION_ID (systemd sets this in CI, changing the restart call signature) - test_session_hygiene: add user_id to SessionSource - test_session_env: use relative baseline for contextvar clear check (pytest-xdist workers share context)	2026-04-14 01:43:45 -07:00
Siddharth Balyan	f3006ebef9	refactor(tests): re-architect tests + fix CI failures (#5946) ... * refactor: re-architect tests to mirror the codebase * Update tests.yml * fix: add missing tool_error imports after registry refactor * fix(tests): replace patch.dict with monkeypatch to prevent env var leaks under xdist patch.dict(os.environ) can leak TERMINAL_ENV across xdist workers, causing test_code_execution tests to hit the Modal remote path. * fix(tests): fix update_check and telegram xdist failures - test_update_check: replace patch("hermes_cli.banner.os.getenv") with monkeypatch.setenv("HERMES_HOME") — banner.py no longer imports os directly, it uses get_hermes_home() from hermes_constants. - test_telegram_conflict/approval_buttons: provide real exception classes for telegram.error mock (NetworkError, TimedOut, BadRequest) so the except clause in connect() doesn't fail with "catching classes that do not inherit from BaseException" when xdist pollutes sys.modules. * fix(tests): accept unavailable_models kwarg in _prompt_model_selection mock	2026-04-07 17:19:07 -07:00

Renamed from tests/test_cli_interrupt_subagent.py (Browse further)