hermes-agent/cron
Minhao HU c675e7c793 fix(cron): bound SessionDB init so a hang can't wedge cron forever
run_job() constructs SessionDB() synchronously with no timeout of its
own, unlike the agent's run_conversation call further down, which is
already bounded by HERMES_CRON_TIMEOUT. A wedged sqlite3.connect (e.g.
a stale flock from a crashed sibling process) hangs this call
indefinitely.

That hang is invisible to every existing cron safeguard because it
happens before _submit_with_guard's future exists: the finally block
that discards the job ID from _running_job_ids never runs. The job
stays wedged "running" — every later tick logs "already running —
skipping" — until the whole gateway process is restarted.

Observed in production: a cron job's worker thread was confirmed via
a live py-spy thread dump to be parked inside SessionDB.__init__'s
sqlite3.connect for 3+ days, silently skipping every scheduled fire
in between across a gateway process that otherwise stayed healthy.

Bound the SessionDB() construction with its own timeout
(HERMES_CRON_SESSION_DB_TIMEOUT, default 10s), following the same
bounded-thread-pool pattern already used elsewhere in this file (the
delivery retry path, and the agent inactivity watchdog just below).
On timeout, log at ERROR and proceed with session_db=None instead of
degrading silently to debug level, since an actual hang here is a new
condition worth surfacing.

Adds tests/cron/test_sessiondb_init_hang.py, including an end-to-end
regression proving the dispatch guard is released and a subsequent
tick can fire the same job again after a simulated hang.
2026-07-14 03:31:06 +05:30
..
scripts fix(cron-recipes): pre-release hardening — honest cadences, strict slot names, surface-aware UX 2026-06-11 10:49:47 -07:00
__init__.py docs: clarify gateway service scopes (#1378) 2026-03-14 21:17:41 -07:00
blueprint_catalog.py docs: finish Automation Blueprints terminology rebrand (#44470) 2026-06-11 17:22:22 -04:00
jobs.py fix(cron): bind claim heartbeats to dispatch owner (#62155) 2026-07-10 19:12:35 -07:00
lifecycle_guard.py fix(gateway,cron): guard cron model-tool path + add auto-resume loop breaker (#30719) 2026-07-01 02:48:36 -07:00
scheduler.py fix(cron): bound SessionDB init so a hang can't wedge cron forever 2026-07-14 03:31:06 +05:30
scheduler_provider.py fix(gateway): quiesce API and cron work during drains 2026-07-13 23:26:10 +05:30
suggestion_catalog.py fix(cron-recipes): pre-release hardening — honest cadences, strict slot names, surface-aware UX 2026-06-11 10:49:47 -07:00
suggestions.py fix(cron): make per-profile cron isolation intentional and tested (#4707) (#53570) 2026-06-27 03:55:01 -07:00