Addresses the review on #51714: the trace/span layer was declared but unwired —
tel_spans was never written, call rows had no timestamp, and nothing set parent
lineage, so the store was metrics-only and couldn't reconstruct a trace.
Wire the span layer (keeping the praised star-schema shape):
- New SpanEvent (span_id/trace_id/run_id/parent_span_id/name/kind/start_ns/end_ns)
mapped into tel_spans via the emitter's _TABLE_COLUMNS.
- The plugin mints a root span per run and, on each model/tool call, emits a
SpanEvent (timing + parent = the run's root) keyed by the SAME span_id as the
detail row, so tel_model_calls / tel_tool_calls JOIN to their span.
- Call hooks fire on completion, so end_ns = now and start_ns is reconstructed
from the measured latency/duration. The run's root span is emitted at finalize
with the true run start/end.
Result: tel_spans is a connected, single-trace_id, run -> calls tree a desktop
waterfall (or any reader) can render directly, ordered by start_ns. Existing
metrics rows (tel_runs/model_calls/tool_calls) are unchanged.
OTLP: spans now flow to the exporter with their trace/parent/timing attributes.
The exporter still emits one OTel span per event rather than reconstructing OTel
SpanContexts into a connected trace tree; that projection is left for a follow-up
and the module docstring now says so plainly instead of over-claiming.
Adds test_spans_trace.py (connected-tree + detail-row JOIN) over the real dispatch
path. Accurate (pre-hook) start times, real OTLP SpanContexts, and subagent
cross-run lineage remain follow-ups.
The subagent_start/stop hooks are registered but no-op. The prior comment implied
subagents need no handling because they inherit via contextvars — misleading, since
a delegated child runs on a separate thread with its own session id and trace.
Clarify the real situation: a subagent's model/tool calls are already captured as
their own tel_runs row via the child's run_conversation, so nothing is lost. These
hooks are reserved for recording parent->child lineage (needs a tel_runs.parent_run_id
column), deferred until a consumer needs the delegation tree. Comment-only.
Rename the telemetry tiers away from the borrowed control-plane/data-plane
jargon to plain language, across code, CLI output, config, and docs:
- "local plane" -> "local telemetry"
- "aggregate plane" -> "aggregate metrics"
- "trajectories plane" -> "trajectories" / "telemetry.trajectories"
- "three planes with a hard wall" -> "three settings, isolated from each other"
User-facing `hermes telemetry status` now reads "Local telemetry: on" /
"Aggregate metrics: off" / "Content export: off (trajectories disabled)".
The OTLP resource attribute key telemetry.plane is renamed to telemetry.scope
(wire-level identifier; nothing consumes it yet).
No behavior change — wording only. Status renders identically apart from the
labels; tests updated to match the new strings.
Add a built-in telemetry system that records what the agent does — workflows,
model calls, tool calls, errors — to the local machine, powers `/insights`, and
can export to an operator-chosen destination. Default-on locally; nothing leaves
the machine unless the user exports it or opts into the aggregate plane.
Three planes with a hard wall between them:
- local: full-fidelity observability (real model/provider/tool names), on by
default, never leaves the machine.
- aggregate: opt-in metadata, default off. No uploader ships — consent is
recorded via telemetry.consent_state, and `preview` shows what would be
produced, computed locally.
- trajectories: full message content, opt-in, exported only to the operator's
own destination.
Mechanism:
- Bundled `telemetry` plugin registers observational lifecycle hooks
(on_session_start / post_api_request / post_tool_call / on_session_finalize).
No core call sites are edited; hooks already carry the data.
- Fire-and-forget emitter: emit() returns in microseconds, never blocks or
raises into a model/tool call. A daemon thread writes events to an
append-only JSONL log and the tel_* tables in state.db (its own sqlite
connection, separate from SessionDB).
- tel_runs / tel_model_calls / tel_tool_calls live in the declarative
SCHEMA_SQL and are reconciled automatically; SCHEMA_VERSION 16 -> 17.
- metrics derives rollups for /usage and /insights; rollup builds per-run
summaries for `hermes telemetry preview`.
Consent is config, not a parallel command surface. The config file is the root
of trust: set telemetry.consent_state with `hermes config set`, or pin any
telemetry.* key (including allow_aggregate) via managed scope, which overrides
the user's value per key. `hermes telemetry` exposes only what config cannot:
status (report), preview (query), and export.
Export:
- exporter_bulk writes telemetry (and, when the trajectories plane is enabled,
session content) to ndjson/json.
- otlp_exporter streams spans to a configured OpenTelemetry Collector over
OTLP/HTTP. The SDK is an optional extra (hermes-agent[otlp]), lazily
installed via tools.lazy_deps on first use.
- Secrets are always redacted on every export path
(redact_sensitive_text(force=True)); content export is gated by the
trajectories plane, and PII scrubbing follows telemetry.content_redaction.
OTLP auth headers reference environment variable names, never inline values.
No outbound emission to Nous. The aggregate uploader is intentionally not built.
