mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-10-20 05:15:31 +00:00
run_job() constructs SessionDB() synchronously with no timeout of its own, unlike the agent's run_conversation call further down, which is already bounded by HERMES_CRON_TIMEOUT. A wedged sqlite3.connect (e.g. a stale flock from a crashed sibling process) hangs this call indefinitely. That hang is invisible to every existing cron safeguard because it happens before _submit_with_guard's future exists: the finally block that discards the job ID from _running_job_ids never runs. The job stays wedged "running" — every later tick logs "already running — skipping" — until the whole gateway process is restarted. Observed in production: a cron job's worker thread was confirmed via a live py-spy thread dump to be parked inside SessionDB.__init__'s sqlite3.connect for 3+ days, silently skipping every scheduled fire in between across a gateway process that otherwise stayed healthy. Bound the SessionDB() construction with its own timeout (HERMES_CRON_SESSION_DB_TIMEOUT, default 10s), following the same bounded-thread-pool pattern already used elsewhere in this file (the delivery retry path, and the agent inactivity watchdog just below). On timeout, log at ERROR and proceed with session_db=None instead of degrading silently to debug level, since an actual hang here is a new condition worth surfacing. Adds tests/cron/test_sessiondb_init_hang.py, including an end-to-end regression proving the dispatch guard is released and a subsequent tick can fire the same job again after a simulated hang. |
||
|---|---|---|
| .. | ||
| scripts | ||
| __init__.py | ||
| blueprint_catalog.py | ||
| jobs.py | ||
| lifecycle_guard.py | ||
| scheduler.py | ||
| scheduler_provider.py | ||
| suggestion_catalog.py | ||
| suggestions.py | ||