Commit graph

3 commits

Author SHA1 Message Date
HexLab98
953cbc0300 fix(state): refuse WAL on SQLite builds with the WAL-reset bug
On vulnerable SQLite (e.g. 3.50.4), do not enable WAL for fresh/non-WAL
shared databases — prefer DELETE instead. Leave existing on-disk WAL
alone (no live downgrade under concurrent gateway/cron openers). Surface
Python/SQLite version details as a doctor warning (#69784).
2026-07-23 17:32:38 +05:30
Drexuxux
371ee065fd fix(gateway): don't spend a redelivery attempt when the platform is down
The delivery ledger durably records a final response before the send so a
crash between finalize and platform ACK can redeliver it on the next boot.
attempts is that redelivery budget, capped at MAX_ATTEMPTS=3.

sweep_recoverable() claims every dead-owner row and increments attempts
before the caller knows whether it can send. self.adapters only holds a
platform after its connect() succeeded, so when the platform failed to
connect this boot _redeliver_pending_obligations() hits its "adapter is
None" branch and continues WITHOUT sending — but the attempt is already
spent. Three such boots and the row abandons, having never been sent once.

That is the loss the ledger exists to prevent, and the trigger correlates
with the crash that created the obligation: the network trouble that killed
the send tends to still be there on the next boot. Worse, the message stays
lost — once abandoned it is never retried even after the platform recovers.

Reproduced against the real runner with an unconnected adapter:

    boot 1: claimed=1 state='attempting' attempts=1  (0 sends attempted)
    boot 2: claimed=1 state='attempting' attempts=2  (0 sends attempted)
    boot 3: claimed=1 state='attempting' attempts=3  (0 sends attempted)
    boot 4: claimed=0 state='abandoned'  attempts=3  (0 sends attempted)

Let the caller declare which platforms it can send on, and skip claiming
rows for the others. attempts then only ever buys a real send. Rows for a
platform that never returns are still bounded by the stale cutoff, so
nothing accumulates. The parameter is keyword-only and optional — omitting
it keeps the previous claim-everything behaviour for other callers.
2026-07-20 11:52:41 +05:30
Teknium
5854aad8b5
feat(gateway): durable delivery-obligation ledger for final responses (#67181)
A final response generated but not confirmed-delivered was the one
artifact the gateway could lose without a trace: crash or planned
restart between finalize and platform ACK dropped it silently, and the
resume path re-ran the whole turn at full cost (#58818 P1, #41696,
#63695's gateway half).

gateway/delivery_ledger.py records each outbound final response in
state.db (same conventions as the async-delegation ledger: WAL, owner
pid + process-start-time liveness, bounded retention):

  pending -> attempting -> delivered | failed
  startup sweep on dead-owner rows -> redeliver | abandoned

Contract (the lessons from the closed delivery-outbox attempt #61790):
- obligation recorded BEFORE the first send attempt; cleared only on
  SendResult.success (destination acceptance, #51184)
- ambiguity is labeled, never silently retried: rows that were mid-send
  when the process died redeliver with a visible '♻️ Recovered reply —
  may be a duplicate' prefix (honest at-least-once)
- stable ids from session_key + inbound message id + content, so
  distinct threads/topics can never collide
- poison rows bounded: 3 attempts / 24h freshness -> abandoned; claim
  atomically re-stamps ownership so racing sweeps can't double-claim
- redelivery clears resume_pending for the session so the resume path
  never re-runs a turn whose answer the ledger already holds
- best-effort everywhere: ledger failure can never block or delay a send
- slash-command/ephemeral/empty responses are not recorded; cron and
  proactive delivery stay on DeliveryRouter (separate subsystem)

Config: gateway.delivery_ledger (default on; no version bump needed).

Validation: 30 ledger+producer tests; 352 blast-radius gateway tests
green; cross-process E2E (record in process A, kill it mid-send, claim
+ marker + redeliver in a fresh process B against the same state.db).
2026-07-19 00:45:32 -07:00