fix(credential-pool): stop lost-update cooldown erasure and wrong-key quarantine

Two related races in credential-pool cooldown state:

1. Lost update across processes: write_credential_pool merged only
   entries missing from the caller's snapshot; for entries present on
   both sides the caller's in-memory copy won wholesale. A process
   holding a snapshot taken before another process marked a key
   exhausted would, on its next persist (e.g. a round-robin rotation),
   write the key back as healthy — erasing the cooldown so every
   process resumes hammering a rate-limited key. Merge status fields by
   last_status_at recency: adopt the on-disk status only when it is
   strictly newer AND still binding (DEAD, or EXHAUSTED with an
   unexpired cooldown), and never onto re-authed (token-changed)
   entries, so legitimate expiry-clears and fresh logins are preserved.

2. Wrong-key quarantine: when mark_exhausted_and_rotate received an
   api_key_hint that matched no entry, it fell through to
   current()/_select_unlocked() — on a freshly loaded pool that selects
   the NEXT healthy key and benches it for the full cooldown TTL,
   punishing an innocent credential. When a hint is provided but
   unmatched, rotate without marking anything instead of guessing.

Includes regression tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Blade 2026-07-16 12:55:33 -04:00 committed by Teknium
parent fdd3943cb1
commit 3d67f00fe1
4 changed files with 276 additions and 1 deletions

View file

@ -1760,6 +1760,20 @@ class CredentialPool:
(e for e in self._entries if e.runtime_api_key == api_key_hint),
None,
)
if entry is None:
# The failed key is identifiable but matches no entry
# (rotated away, or a wrapper whose runtime key differs).
# Falling through to current()/_select_unlocked() would
# mark an INNOCENT healthy key exhausted for the full
# cooldown TTL. Don't guess — just hand back a fresh
# selection so the caller can retry.
logger.info(
"credential pool: failed key hint matched no %s entry; "
"rotating without marking any credential exhausted",
self.provider,
)
self._current_id = None
return self._select_unlocked()
if entry is None:
entry = self.current() or self._select_unlocked()
if entry is None: