With compression.in_place defaulting True the cron session id never
rotates; get_compression_tip returns the input id. Pin that the
salvaged #67188 fix titles/ends the ORIGINAL cron session in that path
(and for falsy tip returns), i.e. zero behavior change when no
compression rotation happened. Also maps colingreig's contributor
email for attribution CI.
run_job now passes target_model to resolve_runtime_provider; the codex
401-refresh test stubbed it with a requested-only lambda. Widen to
**kwargs like every other cron resolver stub.
Two follow-ups to the per-job model pin surface (#67472 / #49948 review):
- cron/scheduler.py: pass target_model=<effective job model> to
resolve_runtime_provider() on the primary path, so providers with
model-specific api_mode routing derive the mode from the model the job
actually runs (per-job pin > env > config default) instead of the stale
persisted default. The auth-fallback path already did this for its
fb_model.
- hermes_cli/web_server.py: POST /api/cron/jobs (and its sync worker) no
longer hardcodes profile="default" when the request carries no profile
param. A pool backend scoped to a named profile now resolves its own
profile via get_active_profile_name(), so pre-profileScoped desktop
clients can't write a named profile's job into ~/.hermes. Unscoped /
custom HERMES_HOME keeps the legacy default fallback.
Tests: target_model capture test on run_job; two profile-default tests on
the create endpoint.
Windows Notepad and PowerShell 5.1 Set-Content -Encoding UTF8 write a
leading UTF-8 BOM. json.load under encoding=utf-8 raises
JSONDecodeError("Unexpected UTF-8 BOM"), and load_jobs wraps that as
RuntimeError("Cron database corrupted and unrepairable"), taking down
cron CRUD/scheduler for a hand-edited jobs.json.
Read with utf-8-sig on all four independent jobs.json readers
(load_jobs primary + strict=False repair, dump _cron_summary, status
Scheduled Jobs). Write path stays plain utf-8 so the next save_jobs
heals a BOM'd file. Matches the env-class dialect (#65123).
Tests: BOM load (crash repro), bomless regression, empty store,
BOM+bare-list auto-repair, BOM+control-char strict=False arm, dump and
status CLI readers.
Review follow-up: the store-internals tests proved _current_cron_store()
resolves lazily, but not that the PUBLIC job I/O honors it. This exercises
save_jobs()/load_jobs() after a late env repoint and asserts the
import-time jobs.json stays byte-identical to a planted sentinel.
Review follow-up: tests that monkeypatch CRON_DIR/JOBS_FILE/OUTPUT_DIR (the
documented process-wide compatibility surface) were bypassed by the lazy
env fallback — 3 file-permission tests, the cross-process lock test, and
the heartbeat roundtrip regressed. _current_cron_store() now snapshots the
constants at import and honors any deliberate re-point of them ahead of
the env resolution, so the precedence is: use_cron_store() override >
patched constants > fresh HERMES_HOME > import defaults. Adds a test
pinning constants-beat-env; the late-env sentinel behavior is unchanged.
tests/cron: failure set byte-identical to unpatched main on this box
(the 5 regressions gone); 138 pass in the touched files.
Complements ec0227b43 (context-scoped cron store): the ContextVar override
is the right tool for deliberate cross-profile scoping, but with no
override active, _current_cron_store() returned the import-time constants —
so a HERMES_HOME set AFTER cron.jobs import (the filed incident: test
fixtures patching the env too late) still read/wrote the user's real
jobs.json. The fallback now resolves the active profile home fresh via
get_hermes_home() (context-local override, then env) and scopes the store
to it; when the home is unchanged since import, the exact module-level
constants are returned as before (zero change in the common path, and they
remain the documented compatibility surface). use_cron_store() still wins.
Three tests: late env repoint scopes the store; unchanged home returns the
import-time constants identically; an active use_cron_store() override
beats the env.
Switch provider and model together after setup-time auth failure. Serialize global auth-store merges under target-specific locks and preserve auth-to-shared lock ordering for profile OAuth refreshes.
Add agent.reasoning_overrides dict to config.yaml. Users can now set
a reasoning_effort per model, overriding the global agent.reasoning_effort.
Example:
agent:
reasoning_effort: "medium" # global default
reasoning_overrides:
"openrouter/anthropic/claude-opus-4.5": "xhigh"
"openai/gpt-5": "low"
"claude-sonnet-4.6": "high" # bare model name also works
The helper is spelling-tolerant: override keys match regardless of
provider prefix or dots-vs-dashes normalization, so users can write
keys in any sensible form and they'll match.
Resolution priority:
1. Session-scoped /reasoning --session override (gateway only; unchanged)
2. Per-model override from agent.reasoning_overrides (spelling-tolerant)
3. Global agent.reasoning_effort (existing)
4. Provider default (unchanged)
Wired into:
- CLI startup (cli.py)
- Messaging gateway agent construction (gateway/run.py)
- Desktop/TUI _load_reasoning_config (tui_gateway/server.py)
- Cron job scheduler (cron/scheduler.py)
- /model mid-session switch (agent/agent_runtime_helpers.py)
+ _primary_runtime now tracks reasoning_config for correct fallback recovery
- Fallback activation (agent/chat_completion_helpers.py::try_activate_fallback)
+ Re-resolves reasoning_config for the fallback model (best-effort)
Closes#21256 (per-model reasoning_effort defaults).
Note: no hermes config set agent.reasoning_overrides.<model> support;
users edit the YAML directly. _set_nested splits on "." and would
corrupt model keys containing version dots.
Salvage of #63935. The original fix read HERMES_CRON_SESSION_DB_TIMEOUT
from a bare env var, but AGENTS.md requires non-secret behavioral
settings to live in config.yaml with an env var bridge only for
backward compatibility.
Changes:
- Add cron.session_db_timeout_seconds to DEFAULT_CONFIG (default 10s)
- Resolution order: HERMES_CRON_SESSION_DB_TIMEOUT env override →
cron.session_db_timeout_seconds in config.yaml → 10s default
(mirrors the existing script_timeout_seconds pattern)
- 0 = unlimited (opt-in for debugging, skips the bound)
- Strengthen test: assert the warning is logged on invalid env value
(caplog was taken but never asserted)
- Add test: verify config.yaml resolution path works end-to-end
Co-authored-by: LoicHmh <26006141+LoicHmh@users.noreply.github.com>
run_job() constructs SessionDB() synchronously with no timeout of its
own, unlike the agent's run_conversation call further down, which is
already bounded by HERMES_CRON_TIMEOUT. A wedged sqlite3.connect (e.g.
a stale flock from a crashed sibling process) hangs this call
indefinitely.
That hang is invisible to every existing cron safeguard because it
happens before _submit_with_guard's future exists: the finally block
that discards the job ID from _running_job_ids never runs. The job
stays wedged "running" — every later tick logs "already running —
skipping" — until the whole gateway process is restarted.
Observed in production: a cron job's worker thread was confirmed via
a live py-spy thread dump to be parked inside SessionDB.__init__'s
sqlite3.connect for 3+ days, silently skipping every scheduled fire
in between across a gateway process that otherwise stayed healthy.
Bound the SessionDB() construction with its own timeout
(HERMES_CRON_SESSION_DB_TIMEOUT, default 10s), following the same
bounded-thread-pool pattern already used elsewhere in this file (the
delivery retry path, and the agent inactivity watchdog just below).
On timeout, log at ERROR and proceed with session_db=None instead of
degrading silently to debug level, since an actual hang here is a new
condition worth surfacing.
Adds tests/cron/test_sessiondb_init_hang.py, including an end-to-end
regression proving the dispatch guard is released and a subsequent
tick can fire the same job again after a simulated hang.
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
get_due_jobs()'s one-shot stale-entry recovery (#38758) treated an
expired run_claim (#59229) as proof the claiming tick died, but a run
stalled on network I/O — or a laptop asleep mid-run — legitimately
outlives the TTL while very much alive. The recovery then deleted the
job record mid-flight: list showed the job gone, and when the run
finished mark_job_run() found nothing to update, so last_run_at /
last_status / last_delivery_error were never recorded.
Two guards, per the liveness signals available:
- Same process (the common single-gateway case): before removing a
dispatch-limit-reached one-shot, consult the scheduler's running set
via a lazy import; if the job is still running here it is slow, not
stale — keep the entry.
- Cross process: run_job's monitor loop now refreshes run_claim.at
every 60s while the run is alive (including under
HERMES_CRON_TIMEOUT=0, which previously blocked without polling), so
an expired claim really does mean the owner died and the TTL stays a
dead-owner detector.
Fixes#62002
Structural completion of the malformed-job freeze fixes (#61382 id-less,
#61525 non-dict schedule, #61581 bad next_run_at): wrap the per-job body
of _get_due_jobs_locked in try/except so any FUTURE malformed-field
variant degrades to skipping that one job for the tick instead of
aborting the scan before save_jobs() and freezing the whole profile's
scheduler.
Also: restore test_repeated_concurrent_runs_accumulate_completed_count
to TestMarkJobRunConcurrency (accidentally re-parented by the #61581
diff), add a containment regression test, and AUTHOR_MAP for hydracoco7.
E2E: one jobs.json carrying all five malformed shapes (drifted job_id,
missing id, null schedule, garbage next_run_at, non-string last_run_at)
plus a healthy sibling — single tick contains all five, sibling fires,
repairs persist, second tick stable. 670 cron tests green.
One bad next_run_at value in jobs.json aborts the due-jobs scan with
ValueError from fromisoformat, before any save_jobs, so siblings lose
progress (fast-forwards etc).
Early normalization in _get_due_jobs_locked + defensive parses in
compute_next_run / _recoverable_oneshot_run_at.
Added test_bad_next_run_at_does_not_crash_or_block_sibling_jobs.
A job record in jobs.json can have a non-dict 'schedule' value (null, string,
etc.) from direct edit or old writers.
In _get_due_jobs_locked:
schedule = job.get('schedule', {})
kind = schedule.get('kind')
This (and direct schedule['kind'] in compute_next_run etc.) raises and
aborts the entire due-jobs scan before save_jobs() or advancing next_run_at
for healthy jobs. Exactly the same failure mode as the id-less job P1.
Fix: normalize non-dict schedules to {} early (before any use), matching the
defense added for id-less records. Also added defensive guards in compute
functions.
Added regression test that a bad schedule does not crash and healthy sibling
is still returned.
Refs similar pattern in #61382.
A cron record authored by a direct jobs.json edit that bypassed
add_job() can lack an "id" key (older writers used "job_id"). Every
site in _get_due_jobs_locked indexes job["id"] eagerly — both the
logging helpers (job.get("name", job["id"]) evaluates the default
argument unconditionally) and the 'for rj in raw_jobs: if rj["id"] ==
job["id"]' persistence loops. A single malformed record therefore
raised KeyError mid-tick, aborting the entire scan before save_jobs()
ran. Result: healthy jobs' fast-forwarded next_run_at was computed in
memory then discarded on the exception unwind, freezing the whole
profile's scheduler in a per-minute loop (observed dormant for weeks).
Fix: normalize id-less records at the top of _get_due_jobs_locked before
anything keys off job["id"] — recover the id from a drifted "job_id"
key when present, else synthesize one via uuid4, and persist. This
repairs the whole bug class at the source rather than guarding each of
the ~12 downstream index sites.
Adds a regression test that fails with KeyError on the current code and
passes with the fix, asserting a healthy sibling job is still returned
when an id-less record shares the store.
Three fixes for the silent post-restart ticker stall:
1. _jobs_lock() bounds its cross-process flock: LOCK_NB polled against a
30s deadline instead of an unbounded LOCK_EX taken while holding the
process-wide RLock. On timeout it logs at ERROR and degrades to
in-process-only locking (the existing fallback path), so a sibling
process wedged while holding .jobs.lock can no longer freeze every
cron function - including the ticker's get_due_jobs() and thus the
heartbeat - forever with zero logging.
2. fire_claim/run_claim freshness checks are bounded on both sides
(0 <= age < ttl): a claim stamped in the future (clock/TZ skew across
a restart) was previously fresh forever, making the job permanently
unfireable and every manual run report 'already being fired'.
3. _execute_job_now distinguishes paused/disabled/missing jobs from a
genuinely held claim instead of mislabeling them all as 'already
being fired'.
Follow-up to the previous commit on #60432. The status-write guard
(_consume_interrupted_flag, checked right before mark_job_run) closes
the false-success bookkeeping gap, but run_one_job delivers its result
BEFORE that check: delivery happens right after run_job() returns,
mark_job_run happens at the very end. A job whose tool subprocess was
killed mid-flight can still produce a plausible-looking final_response
from the truncated output, and that response would reach the user via
_deliver_result before the interrupted flag was ever consulted --
correct status in jobs.json, wrong message already sent.
Adds _is_interrupted(), a non-destructive peek at the same
_interrupted_job_ids set (_consume_interrupted_flag stays as the
consuming, authoritative check right before the status write -- this
needed a peek instead since the flag has to still be visible there).
Checked right after save_job_output, before the deliver_content
decision: if the run looked successful but was flagged interrupted,
force success=False with an explicit interruption message. This
routes delivery through the existing _summarize_cron_failure_for_delivery
path (the same one a real failure already uses) instead of the raw
final_response, so the user gets an honest "this run was interrupted"
instead of a truncated/misleading result.
Testing: 4 new tests in tests/cron/test_shutdown_interrupt.py --
_is_interrupted peek semantics (false/true/does-not-clear, as opposed
to the consuming _consume_interrupted_flag), and the delivery-gate
test itself, which mocks run_job to return a normal-looking success
with a "plausible final response" while the job is pre-marked
interrupted, and asserts _deliver_result receives the failure summary
("This run was interrupted.") instead, with the summarizer's error
argument confirmed to mention the interruption.
Fail-then-pass: reverted cron/scheduler.py only, the 4 new tests fail
(3 on the missing _is_interrupted attribute, 1 -- the delivery-gate
test -- on _summarize_cron_failure_for_delivery never being called,
i.e. the raw response would have gone out); restored, all 16 tests in
the file pass.
Regression: tests/cron/ (683 tests) + test_cron_active_work_drain.py +
test_gateway_shutdown.py + test_shutdown_cache_cleanup.py -- 11
pre-existing failures (Unix file-permission-bit and path-tilde
assertions that don't apply on this Windows dev box), matching the
same set already established as pre-existing in the prior commit's
regression check. Zero new failures.
Continues #60432
Cron jobs run through cron/scheduler.py's own ThreadPoolExecutor via a
standalone AIAgent (run_job/run_one_job), entirely outside
GatewayRunner._running_agents -- the dict _drain_active_agents() and
every other active-work check on that class reads. A gateway shutdown
(/update, /restart, and SIGUSR1 all funnel through the same stop())
could log active_at_start=0 and immediately kill tool subprocesses
while a cron job's terminal command was still running, with no wait
and no indication anything was interrupted.
Real-world impact (from the issue): a scheduled daily briefing cron
job was in flight during /update, its tool subprocess got killed
by the unconditional shutdown cleanup, and the job was never marked
failed -- it simply never completed or delivered, with no error
surfaced anywhere. A repro with a 30-minute `sleep` cron job in flight
during /update reproduced the same pattern: subprocess killed at
+0.22s of drain (active_at_start=0), the job's agent thread continued
in-process and produced a plausible-looking final response from the
truncated tool output, and the scheduler marked the run successful.
Root cause is layered, not a single line:
1. GatewayRunner._drain_active_agents() only waits on _running_agents.
Cron work was invisible to it, so drain returned instantly whenever
the only active work was a cron job.
2. Even with visibility, the shutdown's final tool-subprocess kill
(process_registry.kill_all()) is a global, unconditional sweep with
no per-job targeting -- a long-running cron job that outlives the
drain timeout still gets its subprocess killed.
3. cron/scheduler.py had no way to detect that a job's tool subprocess
was killed out from under it mid-run; the agent thread kept going
and its eventual (often degraded but plausible-looking) response
got reported as a normal successful completion.
Fix, three parts:
- cron/scheduler.py: expose get_running_job_ids() (thread-safe
snapshot of the existing _running_job_ids set, already used to
prevent double-dispatch) so the gateway can read cron's in-flight
state without reaching into private module internals.
- gateway/run.py: GatewayRunner._active_cron_job_count() reads that
snapshot. _drain_active_agents() now waits on
(_running_agents OR active cron jobs), so a cron-only workload gets
the same bounded wait chat sessions already get instead of an
instant active_at_start=0. Shutdown drain logging gains
cron_active_at_start/cron_active_now fields alongside the existing
ones (unchanged, for compat).
- cron/scheduler.py: mark_running_jobs_interrupted(reason), called by
gateway/run.py's _kill_tool_subprocesses() right after
process_registry.kill_all(), marks every job still in
_running_job_ids at that instant as failed/interrupted via the
existing mark_job_run() -- and records the job IDs in
_interrupted_job_ids BEFORE writing, so run_one_job()'s own
eventual completion for the same run (racing in its own thread)
checks that flag and skips its normal write instead of clobbering
the interrupted status with a false "ok" produced from the
now-truncated tool output. This does not attempt to correlate a
killed PID to a specific job ID (process_registry tracks PIDs, not
job IDs) -- any job still dispatched at the moment of a forced kill
is treated as interrupted, matching the existing coarser precedent
set by _interrupt_running_agents(), which interrupts every entry in
_running_agents on a drain timeout without per-agent correlation
either.
Deliberately out of scope (flagged in the issue as a separate,
lower-priority concern): startup-time reconciliation of cron runs that
started but never reached a terminal status.
Testing:
- tests/cron/test_shutdown_interrupt.py (12 tests): get_running_job_ids
snapshot semantics, mark_running_jobs_interrupted marking/no-op/
partial-failure behavior, and -- the core race guard -- run_one_job
skipping its own last_status write (both the success path and the
exception path) when the shutdown path already marked the run
interrupted, with a control test proving ordinary un-interrupted
completions are unaffected.
- tests/gateway/test_cron_active_work_drain.py (9 tests):
_active_cron_job_count reading cron state and failing closed (0) if
the cron module is unavailable; _drain_active_agents waiting for an
in-flight cron job the same way it waits for chat sessions, timing
out if the job outruns the window, and leaving existing chat-session
drain behavior unchanged; a full runner.stop() integration test
(drain-timeout path) proving mark_running_jobs_interrupted actually
fires with the right job ID when a tool subprocess is force-killed,
plus a no-op control when nothing cron-related is in flight.
- tests/gateway/test_shutdown_cache_cleanup.py: added
_active_cron_job_count() to that file's hand-rolled _FakeGateway test
double, which stop() now calls -- without it those 8 pre-existing
tests AttributeError (caught by fail-then-pass below, not a
production bug).
Fail-then-pass: reverted gateway/run.py + cron/scheduler.py, all 21
new tests fail (fixture/attribute errors -- the feature doesn't exist
yet); restored, all 21 pass.
Regression check: ran the full plausibly-affected surface --
tests/gateway/{test_gateway_shutdown,test_restart_drain,
test_restart_notification,test_restart_redelivery_dedup,
test_restart_resume_pending,test_restart_service_detection,
test_shutdown_cache_cleanup,test_stuck_loop,test_clean_shutdown_marker,
test_external_drain_control,test_session_state_cleanup,
test_update_command,test_update_streaming}.py plus tests/cron/ (944
tests) -- against a clean upstream/main checkout and against this
branch. Diffed the two FAILED lists: identical, 20 pre-existing
failures on both sides (Windows-locale/cp1252 file-encoding issues and
Unix-permission-bit assertions that don't apply on this Windows dev
box), zero new failures, zero fixed-by-accident. The 8
test_shutdown_cache_cleanup.py failures found mid-development were
from the _FakeGateway gap above, fixed in the same commit and
confirmed clean on the final rerun (diff against baseline: exit 0).
Fixes#60432
The extracted normalize_skill_lookup_name() resolved trusted roots via
agent.skill_utils.get_skills_dir(), but skill_view() enforces
tools.skills_tool.SKILLS_DIR — a separate module attribute that callers
and 60+ existing tests patch directly. With the helper reading a
different symbol than the enforcer, any SKILLS_DIR patch (or future
divergence between the two resolvers) makes normalization disagree with
enforcement and absolute-path loads regress silently. Read SKILLS_DIR at
call time (deferred import, cycle-safe) with get_skills_dir() as the
fallback, and align the new tests to patch the enforced symbol.
Follow-up to the salvage of #59829 by @HexLab98.
Add unit tests for normalize_skill_lookup_name and a cron scheduler
regression that absolute paths under the skills dir reach skill_view as
relative lookups.
Follow-up to #59524. The one-shot running-claim stale-recovery window was a
fixed 30-min constant. Derive it from the cron inactivity timeout instead
(HERMES_CRON_TIMEOUT, the same limit the scheduler enforces per run) so the
safety valve tracks how long a run may actually go quiet:
- unset/invalid -> default 600s inactivity -> TTL 1800s (unchanged behaviour)
- positive N -> max(N * 3 headroom, 1800s floor)
- 0 (unlimited) -> no finite bound -> fall back to the 1800s constant
The fixed constant is kept as the floor + unlimited-case fallback. Resolved
once per due-scan. HERMES_CRON_TIMEOUT is a pre-existing internal env var
(already read by cron/scheduler.py); no new config surface.
E2E: with HERMES_CRON_TIMEOUT=1200 the claim now survives to 60min where the
old fixed 1800s constant wrongly expired it at 30min mid-run. +1 derivation
test; 640/640 cron tests pass.
The +60s next_run_at advance only delayed a duplicate one-shot dispatch by
one tick — a job that outlives the 60s tick interval (the reported 2.5-min
research prompt) still re-fired on the next tick after the window expired,
so the concurrent gateway+desktop double-delivery persisted.
Replace it with a durable run_claim (at+by, mirroring fire_claim) stamped
on the one-shot under the same jobs lock get_due_jobs holds, and checked at
the top of the due-scan: a fresh claim held by an in-flight run makes every
other scheduler process skip the job for its ENTIRE run, not one tick.
mark_job_run() clears the claim on completion; a ONESHOT_RUN_CLAIM_TTL
(30 min) safety valve re-dispatches a claim left by a tick that died mid-run
so a one-shot is never wedged.
E2E: long-running one-shot no longer double-fires at +28/+61/+120/+179s;
completion clears the claim + disables the job; crash recovery re-arms past
the TTL. +3 regression tests.
When two scheduler processes (gateway + desktop) run concurrently,
both could pick up the same one-shot job from get_due_jobs() because
its next_run_at was not advanced before execution started — only
recurring jobs were advanced (L3446). This caused duplicate deliveries
and wasted token spend (#59229).
Now _get_due_jobs_locked advances a one-shot's next_run_at by 60s
before returning it as due, persisted immediately under the same
file lock. mark_job_run re-anchors next_run_at on completion, so a
tick death between advance and execution only delays the job by one
tick window — it is never lost.
Closes#59229
Completes the #59395 bug-class fix. create_job and update_job's
schedule-change path already reject past one-shots (via #59410/#59438);
this closes the two remaining doors that stored next_run_at=None for a
'once' schedule and re-created the silent ghost job:
1. update_job fallback-recompute (the safety-net that re-derives
next_run_at when it's missing on an enabled, non-paused job)
2. resume_job (resuming a paused one-shot whose time has already passed
— empirically confirmed to create a scheduled job that never fires)
The redundant update_job schedule-change hunk from the original PR was
dropped (already on main via #59438). Adds resume-reject + update-reject/
accept regression tests.
Salvaged from #59428 by isheng-eqi.
Widen the #59395 fix to the sibling site: update_job's schedule-change path
(cron/jobs.py) had the SAME unguarded compute_next_run -> next_run_at pattern,
so updating a job's schedule to a one-shot >ONESHOT_GRACE_SECONDS in the past
would re-create the ghost job (next_run_at=None, state='scheduled', never fires)
that create_job now rejects. Apply the identical guard on update (raise before
any disk write, so the original job is left intact), with regression tests for
the reject + future-accept cases.
Also surface ONESHOT_GRACE_SECONDS in the raised ValueError (not just the
warning log) so a caller knows how far in the past is too far. Message from the
competing PR #59410 by @isheng-eqi.
Co-authored-by: isheng-eqi <265044697+isheng-eqi@users.noreply.github.com>
Defense-in-depth alongside the interpreter-shutdown guard: run_job closed
the cron agent's async resources (agent.close + cleanup_stale_async_clients)
in its finally block BEFORE run_one_job called _deliver_result, so a live
delivery could race a torn-down async client. run_job now accepts an optional
defer_agent_teardown holder; when set it hands the live agent back instead of
closing it, and run_one_job tears it down (via the extracted _teardown_cron_agent
helper) only AFTER delivery — in a finally so a failed run never leaks. Default
path (holder=None) is unchanged, so every existing caller keeps inline teardown.
Reorder approach based on #58777 by @LavyaTandel; reworked to keep a single
delivery site in run_one_job and add regression coverage.
Co-authored-by: LavyaTandel <lavya@loom.local>
Pins `_interpreter_shutting_down()` (finalizing flag + shutdown-error-text
fallback) and asserts the standalone delivery path skips gracefully without
scheduling a send when the interpreter is finalizing, while the normal
non-finalizing path still delivers. Source guardrails keep the guard wired
into both the dispatch (`_submit_with_guard`) and standalone-delivery sites.
The #22773 heuristic classified any telegram:<positive_chat_id>:<numeric_thread_id>
cron target as a Bot API channel Direct-Messages topic and routed it via
direct_messages_topic_id, which nulls message_thread_id. A normal forum-style
topic inside a private chat has the identical shape, so every such cron
delivery landed in General instead of the target thread (#52060). It also
means the only way to address a genuine channel DM topic was this same
ambiguous guess.
Disambiguate with the real runtime signal instead: probe the live adapter's
get_chat_info once and route via direct_messages_topic_id only when the chat
is actually a channel; everything else (private forum topic, forum supergroup,
group) and any probe failure fall back to message_thread_id — parity with
0.16.0 and with live reply routing. This fixes forum-topic delivery AND makes
genuine channel DM-topic cron delivery work correctly.
Asserts the behavior contract that run_one_job installs a profile secret
scope around run_job under multiplexing (so resolve_runtime_provider's
get_secret does not fail-close with UnscopedSecretError) and tears it
down afterward. Mutation-verified: fails on unmodified main with the
exact UnscopedSecretError, passes with the fix.
The standalone thread-pool fallback in _deliver_result() runs inside the
`except RuntimeError:` block (taken when asyncio.run() sees a running loop).
When future.result() raised there (SMTP ConnectionError, timeout, etc.), the
exception was NOT caught by the sibling `except Exception:` — it escaped
_deliver_result() and crashed the whole delivery loop, silently skipping every
remaining target. Multi-target delivery (e.g. deliver: 'email:a,email:b') is a
documented feature, so this broke a promised contract.
Wrap the fallback in its own try/except so a per-target failure is logged with
exc_info and the loop continues to the next target.
Fixes#47163
The in_channel surface DOES add a seed: _seed_cron_channel_session CREATES
the flat (platform, chat_id, None) session and mirrors the brief into it,
because mirror_to_session only APPENDS to an existing session and the flat
channel row is otherwise absent for a chat_postMessage delivery. Correct
the scheduler thread-skip comment and the test class docstring, which still
described the earlier 'let the existing mirror seed it' design.
Live testing exposed a real bug: an in_channel continuable cron delivered
flat to the channel (✅) but the reply did NOT continue the job — the bot
had no brief in context and confabulated the answer.
Root cause: mirror_to_session only APPENDS to a session that already
exists (_find_session_id → no-op when none matches); it never CREATEs one.
A flat (slack, chat_id, None) row is only created when a human posts a
top-level message the bot processes — a cron chat_postMessage delivery
never goes through the inbound handler, so the row is absent and the brief
is silently dropped. The prior impl relied on the bare mirror (F5/OQ-1
concluded "deletion only" — wrong).
Fix: _seed_cron_channel_session mirrors _seed_cron_thread_session —
get_or_create_session FIRST (chat_type = "dm" if is_dm else "group",
thread_id=None), keyed to the ORIGIN USER'S id, then mirror. The channel
session key embeds user_id (…:group:<chat>:<user>), so a system:cron id
would key the seed away from the reply; the origin user's id makes seed
key == inbound reply key. DM key ignores user_id but needs chat_type=dm
to match the prefix. Wired into the in_channel branch after delivery;
suppresses the generic mirror to avoid double-write.
DM validated (per request): the seeded key equals the inbound DM reply key
for a 1:1 DM; continuation works there too.
Tests:
- Rewrote the in_channel tests to use a real _session_store and the origin
user_id; assert get_or_create_session is called with the flat, correctly-
keyed source. Prove-fail: (a) reverting the create step and (b) seeding
with system:cron each turn a targeted test RED; restore → GREEN.
- +2 direct _seed_cron_channel_session unit tests asserting the KEY-MATCH
invariant (seed key == inbound reply key) via build_session_key, for both
channel and DM.
- Rewrote tests/manual/cron_inchannel_e2e.py to drive a REAL SessionStore +
real mirror_to_session + real _find_session_id + real build_session_key
(no session-layer mocks — the old mocked E2E is exactly why the bug
shipped). Asserts the brief lands in the transcript and the reply resolves
to the same session, for BOTH channel and 1:1 DM.
Full relevant sweep: 283 passed.
Add a per-platform `cron_continuable_surface` extra key
(`thread` default | `in_channel`) so a continuable cron job can deliver
FLAT into a Slack channel — no dedicated thread — and still be
replied-to. In `in_channel` mode the scheduler skips the thread-open
branch (leaves `thread_id=None`); the shipped origin-mirror then seeds
the `(slack, chat_id, None)` shared-channel session — the same bucket
`reply_in_thread: false` routes inbound channel replies to — so a plain
channel reply continues the job in context.
Design: specs/cron-inchannel-continuable (D1–D7, F5). Model B
(shared-channel session), NOT anchoring to the delivery `ts` — on Slack
replying to a specific message IS threading, so a `ts` anchor would only
relocate the thread, never deliver true threadless continuable.
- gateway/platforms/base.py: `supports_inchannel_continuable` capability
flag (default False → unsupported platforms fail SAFE to `thread`).
- plugins/platforms/slack/adapter.py: flag=True; `_cron_continuable_surface()`
resolver (coerces to the two-value enum); `_warn_if_inchannel_without_flat_reply`
connect-time warning (D5: warn, not hard-require — the misconfig fails safe).
- gateway/config.py: shared-key bridge line (top-level OR nested config).
- cron/scheduler.py: read the key generically from platform config, gate
the `in_channel` branch on the adapter capability flag, skip thread-open.
No new seed function (reuses the existing mirror — G6).
Pairing (docs): `in_channel` + `reply_in_thread: false` +
`require_mention: false` (or a free-response channel). Missing
`reply_in_thread: false` fails safe to a threaded continuation.
Gateway-side config flag — `/restart` to apply; NO Slack app reinstall.
Tests (from inside the worktree, PYTHONPATH=$PWD):
- +6 cron scheduler tests (in_channel skips thread-open; seeds flat
channel session with thread_id=None; thread-mode regression;
fail-safe on unsupported platform; value coercion). Prove-fail:
removing the `and not in_channel_surface` guard turns the two
load-bearing tests RED; restore → GREEN.
- +10 slack resolver/capability/warning tests; +2 config-bridge tests.
- tests/manual/cron_inchannel_e2e.py: offline E2E driving BOTH real
legs (delivery seed + inbound reply keying) → both converge on
(slack, C, None).
- No regressions: test_slack.py 216 passed alone; broader sweep green
(4 pre-existing cross-file-ordering failures reproduce identically on
pristine origin/main).
Docs: cron.md + slack.md + zh-Hans mirrors of both.
Rework follow-up on the per-job TERMINAL_CWD readers-writer lock.
The lock was acquired BEFORE the try: whose finally: is the only release
site, with the env-override statements (os.environ[TERMINAL_CWD] = workdir;
logger.info) sitting in the unprotected window between acquire and try. Any
exception there — a raising log handler, an os.environ error, a thread
interrupt — propagated out of run_job WITHOUT running the finally, leaking
the lock. A leaked writer permanently deadlocks the whole scheduler (every
future cron job blocks on acquire_*); a leaked reader blocks all writers.
- Snapshot _prior_terminal_cwd before the acquire (so the finally can always
restore env even if the body raises before the override).
- Open the try: immediately after acquire and move the env-override lines
inside it, so the existing finally always releases the lock.
- Add a mutation-verified regression test: a workdir job whose in-window
logger.info raises must still release the writer lock (a subsequent
acquire_write must not block).
