The onboarding overlay already contains a 'Local / custom endpoint' card
that writes model.provider:custom + base_url + api_key, but no reachable
Desktop GUI path opens it for a fresh add. The composer model pill falls
back to the gateway menu panel (Edit Models…), and Settings → Providers →
API keys is env-var-driven and never lists a custom endpoint — so users
following their instincts cannot add an OpenAI-compatible endpoint (Zyphra,
vLLM, Ollama, …) from the GUI.
Add a 'Local / custom endpoint' row to the API-keys tab that calls
startManualLocalEndpoint(), landing the overlay directly on the existing
custom-endpoint form. Reuses the tested onboarding flow; no new UI surface.
Regression test in providers-settings.test.tsx asserts the row renders and
opens the custom-endpoint flow.
Fixes#62817
Co-authored-by: David Metcalfe <80915+DavidMetcalfe@users.noreply.github.com>
Two internal comments in delegate_tool.py still described the superseded
"N independent handles, no combined wait" model, contradicting the
authoritative batch contract (one async unit, one consolidated result
when all children finish). Aligns the comments with the runtime path in
_execute_and_aggregate / dispatch_async_delegation_batch.
* fix(desktop): trust Windows system CAs for remote gateways
Load Windows-trusted roots into Node's default TLS context before Desktop probes remote backends, while preserving bundled and extra CAs.
* test(desktop): cover Windows system CA installation
Verify existing trust roots survive the merge and that unsupported or unavailable stores fail open without changing TLS defaults.
#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows. That
PAT is empty on fork PRs (forks get no repo secrets), which broke every
fork PR two ways:
1. detect-changes classified with the empty PAT -> the compare API failed
all 3 retries -> the classifier failed open and force-enabled the
ci_review lane on EVERY fork PR.
2. The ci-reviewed / mcp-catalog-reviewed label gates then read labels with
the same empty PAT via a hard-failing retry step -> the job failed with
no recovery a fork contributor could perform (they can't self-add the
label; re-running can't fix it).
Restores the pre-#66373 fork-safe behavior without reverting the commit's
real improvements (job timeouts, per-file flake retry, network-install
retries):
- detect-changes + ci.yml: token falls back to the built-in read-only
github.token when AUTOFIX_BOT_PAT is empty. On main it uses the PAT
(authoritative); on forks it uses github.token, which can read the
public compare endpoint. (An input `default:` only applies on omission,
not on an empty passed value — hence the explicit `|| github.token`.)
- lint ci-review + supply-chain mcp-catalog gates: restore the inline
`gh pr view ... || true` label read with the github.token fallback,
dropping the hard-failing retry "Fetch PR labels" step. Graceful
degrade to "label absent" on an API blip, same as before #66373.
Same-repo enforcement is unchanged (byte-identical logic; the PAT is still
used there). Fork PRs classify correctly and the gates read labels via the
read-only token exactly as they did before the regression.
The reported confusion was an unblocked task 'unpredictably' ending up in
triage. unblock itself only ever routes to ready/todo; a subsequent same-cause
re-block hitting BLOCK_RECURRENCE_LIMIT is what escalates to triage. Document
this deterministic loop-breaker at the human-facing lifecycle level so users
stop reading it as an LLM decision.
strip_think_blocks() ran re.sub() directly on content that could be a
list of blocks (Anthropic via OpenRouter returns assistant content as
[{type:text,...},{type:thinking,...}]). A list reaching re.sub raised
'TypeError: expected string or bytes-like object, got list', which the
outer conversation loop swallowed and retried forever — the observed
infinite 'preparing terminal...' loop that re-emitted the same
assistant text every iteration.
The live-turn path normalized list content to a string, but
_interim_assistant_visible_text reads a *stored* history message whose
content was persisted as a list and passes it straight into the shared
strip_think_blocks helper. Fix at the shared choke point: coerce
list/dict content to visible text (dropping reasoning blocks, which is
the function's job) before any regex runs, so every caller is safe.
hermes-sweeper review on #60970 flagged that _apply_model_switch_result
(the interactive-picker sibling of _handle_model_switch) was only ever
tested with persist_global=False elsewhere, so the picker's global-switch
base_url/api_mode persistence branch had no coverage.
Same bug family as #47828, at the config-persistence layer instead of
the in-memory agent layer:
- cli.py (#25106): the --global /model handlers (both the typed-name
path in _handle_model_switch and the picker path in
_apply_model_switch_result) wrote model.default/model.provider to
config.yaml but never touched base_url/api_mode at all. A provider
switch left the OLD endpoint on disk; the next launch reconnected to
the previous provider's host under the new model name.
- gateway/slash_commands.py (#25107): both persist-global blocks (the
picker-tap callback and the typed /model --global path) guarded the
write with two INDEPENDENT ifs — `if result.base_url: ...` and
`if target_provider != "custom": clear_model_endpoint_credentials(...)`.
For named providers the second if always cleared stale values, masking
the bug. For a custom provider with an empty resolved base_url, neither
branch fired, so the previous custom endpoint's base_url/api_key/
api_mode survived untouched in config.yaml.
Fix: explicit set-if-truthy/clear-if-falsy for base_url and api_mode at
all four call sites, matching the already-correct pattern in
tui_gateway/server.py:_persist_model_switch (fixed for #48305).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Switching to a GPT-5.x model on api.openai.com while the session carried a
stale chat_completions api_mode (e.g. from a prior openrouter default) left
the request on /v1/chat/completions, which 400s with "Function tools with
reasoning_effort are not supported" once the switched model's reasoning is
applied. switch_model() only re-derived api_mode inside the
provider-changed branch, so a same-provider/carryover switch kept the wrong
wire protocol.
Add host_mandated_api_mode(base_url): the endpoints that accept exactly one
protocol (api.openai.com -> codex_responses, api.anthropic.com / *…/anthropic*
-> anthropic_messages, api.kimi.com /coding -> anthropic_messages,
bedrock-runtime -> bedrock_converse), matched by EXACT hostname so lookalike
hosts and path-segment spoofs are rejected (#32243). switch_model() now uses
it to override a stale carried api_mode, not merely fill an empty one;
determine_api_mode() shares the same helper.
Credit sjiangtao2024 (#15880) for the recompute-before-validation approach;
this strengthens it from fill-if-empty to a host-mandated override.
Co-Authored-By: sjiangtao2024 <siage@139.com>
* feat(attribution): conflict-free contributor mappings via contributors/emails/ directory
The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet:
every concurrent salvage PR appended entries to the same lines of the
same file, so parallel PRs re-conflicted on every merge to main.
New system: one file per email under contributors/emails/ — filename is
the commit-author email, first non-comment line is the GitHub login.
File additions never conflict, so any number of PRs can add mappings
concurrently.
- scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen)
merged with the directory at import time (directory wins). All
existing consumers (resolve_author, contributor_audit.py) unchanged.
- scripts/add_contributor.py: idempotent CLI to add a mapping; refuses
conflicting reassignments (incl. against the legacy map), validates
email/login shapes.
- contributor-check.yml: attribution gate now accepts a mapping file OR
a legacy entry; failure message prints the exact add_contributor
command. Also auto-resolves bare <login>@users.noreply.github.com
emails is intentionally NOT added (kept id+login form only, matching
previous behavior).
- contributor_audit.py: guidance now points at add_contributor.py.
- tests/scripts/test_contributor_map.py: 12 tests covering loader,
merge precedence, CLI idempotency/conflict/validation, subprocess E2E.
* feat(ci): one-shot per-file flake retry in the parallel test runner
A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry
counts as green but is loudly reported in a '⚠ FLAKY' summary section
(with both attempts' output preserved) so the flake gets fixed instead
of eating a full-run rerun. Deterministic failures fail both attempts —
regressions cannot be laundered green.
- --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables)
- E2E verified: simulated first-run-fail flake goes green with banner;
deterministic failure still exits 1; retries=0 restores old behavior.
This converts the dominant CI failure mode (one timing-sensitive test
flaking a 4600-test shard, requiring a manual 10-minute rerun and an
agent triage loop) into a self-healing retry that costs one file's
runtime.
* test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s
These guard against catastrophic regex backtracking (seconds-to-minutes
class), but 0.15s is within scheduler-stall noise on loaded shared CI
runners — test_max_accepted_separator_free_input_is_fast failed a CI
shard this week on runner load alone. 2.0s still catches the regression
class with zero flake surface.
* fix(ci): job timeouts everywhere + retries on all network installs
Reliability pass over every workflow:
- timeout-minutes on all 21 jobs that lacked one (a hung job previously
burned the 6-hour default runner budget)
- ./.github/actions/retry wrapped around every network-fetching install
that lacked it: pip installs (deploy-site, skills-index), npm ci
(deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker
test deps). Deterministic build steps (npm run build) deliberately
NOT retried — split into separate steps so a real build failure fails
fast instead of retrying 3x.
* docs(agents): document the file-retry flake policy
* fix(ci): curl retries on deploy hook + skills-index probe
* fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile
From the workflow reliability audit:
- tests.yml: duration-cache restore had NO restore-keys while saves use
run_id-suffixed keys — the cache never matched once, so LPT slicing
always ran blind and unbalanced slices pushed heavy files toward the
per-file timeout. One-line restore-keys fixes slice balancing.
- Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed):
'gh pr view || true' turned an API blip into 'label absent' → false
BLOCKING failure. Now 3x retry, and API failure is reported as an API
failure instead of a missing label.
- detect-changes action: compare API retried before failing open (was
silently running all lanes on any blip).
- uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried
so registry blips don't read as 'lockfile stale'.
- docker.yml merge job: imagetools create retried (Docker Hub eventual
consistency on just-pushed digests).
- Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to
curl --retry 3 (ADD cannot retry; checksums still enforced); npm
--fetch-retries=5; playwright chromium fetch retried 3x.
- Advisory artifact uploads (per-slice durations, ci-timings report)
get continue-on-error so an artifact-service blip can't fail a green
test slice.
* fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list
- test_tui_gateway_server.py: session.create / non-eager session.resume
arm a 50ms threading.Timer (_schedule_agent_build) that outlives its
test and fires into the NEXT test's _make_agent mock, racily
corrupting captured state (the recurring session_resume shard
failures). Replaced the per-test whack-a-mole stub with a module-wide
autouse fixture; the 3 worker-lifecycle tests that genuinely need the
deferred build opt back in via @pytest.mark.real_agent_prewarm (new
marker in pyproject).
- test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the
live PROVIDER_REGISTRY instead of a hand-list that had drifted
(missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto')
tests failed on any machine with HF_TOKEN exported. E2E-verified with
HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass.
* test: de-flake 30 timing-sensitive test files for loaded CI runners
Root-cause fixes from the flake audit (session-DB mining + repo sweep):
Event-based sync instead of sleep-sync:
- title_generator: mock sets threading.Event, wait(10) replaces
sleep(0.3) hoping the daemon thread got scheduled
- docker zombie_reaping / profile_gateway: poll-for-state helpers
replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async)
- process_registry tree test: select()-bounded readline replaces an
unbounded blocking read (parent wedge now fails THIS test with a clear
message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s
(the 1s partition window mid-interpreter-startup is how a child PID
escaped the live-system guard in CI)
Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors;
all of these complete in ms-to-1s when healthy so the raises cost
nothing on green runs):
- subprocess/thread waits <= 2s raised to 10-15s across mcp_tool,
mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe,
mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt,
voice_cli_integration, docker_environment, session_store_lock_io,
planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output
(joins now also assert not is_alive() so stragglers fail loudly)
- wall-clock discrimination ceilings loosened where the guarded hang is
10x larger: local_background_child_hang 4s->10s, interrupt_cleanup
setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup
5s->15s, protocol/gil-starvation fast-handler 0.5s->2s,
iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s
- narrow assertion windows widened: honcho first-turn wait 0.4..0.65 ->
0.25..2.0 (property is bounded-not-hung, not an exact wall-clock);
compression fork-lock TTL 1s->3s (12 refresh chances per lease);
compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0)
- telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under)
* fix(tests): repair indentation from de-flake batch edit
* fix(tests): harden env isolation and replace remaining sleep-sync races
The full 42k-test run and complete npm check surfaced three more classes:
- Environment isolation: local ~/.honcho defaultHost and SSH_* variables
leaked into Python/TUI tests. Pin the default Honcho host in the
hermetic fixture, isolate the one fallback test from ~/.honcho, and
blank SSH_* around terminalSetup tests. This flipped 20 false failures
back to deterministic behavior on developer machines.
- Background-thread sleep-sync: Honcho async writer tests patched
time.sleep globally, then busy-polled with that same mocked sleep. Under
full-suite load the poller could starve the writer. Each test now waits
on an Event emitted by the exact flush/retry transition; 30/30 passed
under 15-way contention.
- Desktop streaming: the test slept 80ms and assumed a 500ms timer could
not fire before its assertion. A loaded runner descheduled the test for
>500ms and both chunks arrived. Producer controls now gate second-chunk
and completion transitions explicitly.
Also make file-retry observability complete: a self-healed flaky file now
prints BOTH attempts' full output in the FLAKY summary. Two behavioral
runner tests prove pass-on-retry is green+loud+traceback-preserving, while
a deterministic failure remains red.
* refactor(ci): use gh bot pat, better retries
refactor(ci): use retry action for PR label fetch
the retry action now captures stdout as a step output, so it can serve
double duty: retry + output capture for commands like 'gh pr view' whose
result must be consumed by later steps.
Retry action gains:
- 'stdout' output (heredoc-delimited to preserve newlines)
- tee to temp file so stdout still streams to the job log
- step id 'retry' for output reference
Both lint.yml and supply-chain-audit.yml now use the retry action
directly with 'command: gh pr view ...' and read
steps.<id>.outputs.stdout.
ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth
Replace secrets.GITHUB_TOKEN and github.token with
secrets.AUTOFIX_BOT_PAT across all workflows and composite actions
that use the gh CLI or GitHub API. The PAT has consistent permissions
across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate
limit sharing with the default token, and is already used by
js-autofix.yml for the same reasons.
19 sites swapped across 9 files:
- lint.yml (3): label fetch, comment post/edit, comment update
- supply-chain-audit.yml (5): scan, critical comment, unbounded dep
comment, label fetch, mcp-catalog comment
- lockfile-diff.yml (1): PR comment post/update
- skills-index-freshness.yml (1): issue creation on degraded probe
- skills-index.yml (2): index build, trigger deploy workflow
- upload_to_pypi.yml (2): release view poll, release upload
- ci.yml (1): timings report
- deploy-site.yml (2): skills index crawl
- detect-changes/action.yml (1): compare API call
---------
Co-authored-by: ethernet <arilotter@gmail.com>
Keep invalid persisted preset names fail-closed, list the valid configured choices, and classify the local lookup failure as deterministic so it reaches Desktop immediately.
Fold #62349's broader provider-boundary handling into the header fix: bound top-level and xAI override keys again at preflight after middleware, preserve unrelated headers, and cover boundaries and collisions.
Co-authored-by: Nick Taylor <nicktaylor@TheWorldofNick-Lappy.local>
- codex-app-server-runtime.md: add a Live display section covering the
stream/reasoning/tool-card bridge and show_commentary gating.
- release.py: AUTHOR_MAP entries for HaiderSultanArc, jjadeo-oss, juanfradb
(the latter two for forthcoming follow-up salvages of #62396 / #18050).
Extends the app-server event bridge (make_codex_app_server_event_bridge)
to fire the authoritative stable-ID tool_start_callback /
tool_complete_callback alongside the existing tool_progress_callback,
and route item/reasoning/summaryDelta through the reasoning channel.
Surfaces that render structured tool cards (TUI, desktop) — not just
progress bubbles — now correlate live cards with the projected history
entry after a resume, because the call ids mirror CodexEventProjector's
_deterministic_call_id. Guarded per-callback so a broken display
consumer can't tear down the codex turn loop.
Grafted from PR #65412 by @HaiderSultanArc onto the merged bridge (the
PR's parallel _codex_live_event implementation was reconciled into the
bridge's existing _fire_tool_started/_fire_tool_completed helpers).
The staleness check's bespoke mtime memo keyed only on the user
config.yaml, but load_config() merges the managed-scope config
(HERMES_MANAGED_DIR/config.yaml, /etc/hermes) whose leaf keys win. A
managed honcho.timeout with no user config.yaml made the memo cache
'no timeout' while _build resolved the managed value — the same
perpetual-rebuild mismatch this PR fixes for honcho.json. A managed
timeout edit was likewise invisible while the user file's mtime stayed
put.
load_config_readonly() is already cached on both files' signatures plus
the env-ref snapshot, so use it instead of duplicating that
invalidation logic; the defensive deepcopy the old memo existed to
avoid is skipped by the readonly variant. Drive the rebuild test
through a real config.yaml and add a HERMES_MANAGED_DIR regression
test covering stable reuse and managed-timeout edits.
The staleness check added in #66052 resolved the timeout from env,
config.yaml, and the default only, while the build path also reads the
honcho.json host block (timeout/requestTimeout). With a timeout
configured in honcho.json, the two permanently disagreed: every
no-config get_honcho_client() call — i.e. every HonchoSessionManager
.honcho property access — interpreted the mismatch as a config change
and tore down and rebuilt the client, defeating the singleton on the
hot path it was meant to protect.
Teach the check to read honcho.json through the same host-aware chain
as from_global_config, memoized on the file's mtime_ns so the per-call
cost stays one stat(). A genuine honcho.json timeout change is now also
detected, extending #57437 to that config surface.
Adds a --from DIR flag to scripts/dev-sandbox.sh that copies an existing
HERMES_HOME directory into the sandbox as the starting point before the
command runs. Lets you spin up a sandbox pre-populated with your real
config, sessions, skills, etc.
scripts/dev-sandbox.sh --from ~/.hermes hermes desktop
Design:
- cp -a dir/. dest/ — preserves perms, symlinks, hidden files
- Clobber guard: only seeds when sandbox HERMES_HOME is empty, so
re-running --persistent doesn't blow away existing sandbox state
- Validates: errors on nonexistent dir, missing arg, flag-like arg,
empty --from=
- Supports both --from DIR and --from=DIR forms
- Backwards compatible: no --from = unchanged behavior
$workingSessionIds and $attentionSessionIds were independently maintained
atoms that updateSessionState had to manually keep in sync with the session
cache (paired setSessionWorking/setSessionAttention calls, plus a rotation
special-case in ensureSessionState). Make them computed() projections of
$sessionStates instead, so the data flow is one-directional:
gateway event → cache → $sessionStates → computed views.
Transition side-effects (watchdog arm/disarm, settle grace, unread marker,
compression id rotation signal) move into handleTransition, fired from
publishSessionState by diffing previous vs next — one choke point instead
of per-callsite bookkeeping. The watchdog's force-clear reaches the cache
through setWatchdogClearFn rather than a listener set.
Also:
- clearAllSessionStates disarms all watchdog timers and drops settle-grace
entries so a gateway switch can't leak stale timers or keep-set rows
- dropSessionState disarms the dropped runtime's watchdog timer
- watchdog tests now exercise the real timer→callback wiring instead of
manually simulating the clear
Model-picker audit follow-through — closes the remaining pieces of the
"switch one session, switches everywhere / can't tell whose session this
is" report class:
- tui_gateway: `config.set key=fast` with a session no longer writes the
global agent.service_tier to config.yaml (sibling of the earlier
`reasoning` scoping fix). It pins create_service_tier_override
("priority" / "" for explicit normal) so lazy builds and rebuilds keep
the choice; the desktop's per-model presets were rewriting the global
tier on every model pick. Fast-support validation now checks a draft's
picked model, and `config.get key=fast` reads the pre-build pin.
- desktop: owning-profile tag (initial chip + tooltip/aria label) on
pinned rows and search results in the All-profiles sidebar, and on the
chat header once a second profile exists (#66003).
- desktop: composer model pill shows a pin dot + tooltip when a manual
sticky pick is overriding the Settings default for new chats (#62055).
Closes#66003. Addresses #62055.
A cron job ("Daily Buzz Report") died with 'AIAgent' object has no
attribute '_claim_stream_writer'. The #65991 single-writer fence lives on
AIAgent (run_agent.py), but the streaming paths that use it live in other
modules — chat_completion_helpers (chat / anthropic / bedrock) and
codex_runtime (codex responses) — and called it directly as
agent._claim_stream_writer() / agent._stream_writer_is_current(). That makes
those modules hard-depend on the method being present on whatever object is
passed as agent.
The fence is an *additive* safety net that may only ever drop a provably
superseded stream, never the sole legitimate writer. But the direct calls
turned any agent that doesn't expose it — a version-skewed checkout (the
streaming helper module newer than run_agent), a hot-reloaded gateway mid
git-pull, a duck-typed agent, or a test double — into a fatal AttributeError
that aborts the whole turn (and, on cron, fails the job).
Route every cross-module claim/check through agent/stream_single_writer.py.
claim_stream_writer(agent) returns 0 when the fence is unavailable (or
raises), and stream_writer_is_current(agent, token) treats a 0 token or an
absent guard as "current" — so a guard-less agent degrades to "no fence"
instead of crashing, while a real AIAgent keeps the full single-writer
protection. Internal self.* uses inside run_agent are unchanged (self is
always a full AIAgent there).
Mirror CANONICAL_PROVIDERS so Fireworks sits directly under Nous Portal
(always visible) ahead of OpenRouter across onboarding, Settings → Providers,
and the API-key catalog.
Answering the review question on the PR table — why a hovered-cold
switch still showed ~440ms click → WS open: getConnection-only
pre-warming left the WS connect chain to the click, and its microtask
continuation can only run after the click's fresh-draft React flush
(unmounting a large open transcript costs ~300-400ms of render work),
so the socket didn't even START connecting until the flush finished.
Add openGatewayForProfile: the same spawn + connect chain as a real
switch, minus activation — so the hover leaves the profile's socket
fully OPEN and the click's ensureGatewayForProfile just activates it
(no ws:new after the click at all; measured ws open at hover+136ms on
a warm backend). No scheduleReconnect on failure: a hover is
speculative, so a dead backend must not start a background retry loop
— the real switch owns retry and error UX. Pruning semantics are
unchanged: a hover-opened socket for an idle profile is dropped by the
next pruneSecondaryGateways recompute, which just returns the click to
the previous behavior.
Tests updated: pre-warm asserts openGatewayForProfile is called and
that activation (ensureGatewayForProfile) is NOT.
* fix(credential-pool): throttle "no available entries" log to stop Windows log-lock storm
Credential selection runs on a hot path (every model call plus auxiliary
tasks), so an empty/exhausted pool logged "no available entries" at INFO on
*every* selection. On Windows, where multiple Hermes processes share one
rotating log guarded by concurrent-log-handler's cross-process lock, that
per-selection volume storms the lock (RuntimeError: Cannot acquire lock after
20 attempts), pegs a core, and stalls the asyncio event loop long enough that
the Desktop backend readiness probe times out ("Timed out connecting to Hermes
backend after 15000ms") even though the backend already announced
HERMES_BACKEND_READY.
Log the condition at most once per 60s window, re-arming on a successful
selection so recovery->re-exhaustion still surfaces promptly. Same fix class as
the warn-once dedup in #58265.
* test(credential-pool): cover no-available-entries log throttle
Assert the empty-pool INFO line logs at most once per throttle window, logs
again after the window elapses, and re-arms on a successful selection so a
recover->re-exhaust transition surfaces promptly. Uses a deterministic fake
monotonic clock (no sleeps, no network).
