hermes-agent/docs/observability/telemetry.md

Telemetry & Observability

Hermes ships with a built-in, local-first telemetry system. It records what your agent does — workflows, model calls, tool calls, errors — to your own machine, powers /insights, and (when you enable it) exports everything to your own observability stack. It is private by default and never sends your data to Nous unless you explicitly opt in to anonymous aggregate metrics.

This page explains the whole feature: the three planes, what's captured, the hermes telemetry commands, and how an enterprise streams or exports all of its data to its own infrastructure.

Looking for the plugin hook contract (how to write your own observer plugin)? That's in README.md. This page is about the built-in telemetry plane and its CLI.

The three planes

Telemetry is organized into three planes with a hard wall between them:

Plane	What it holds	Default	Destination
local	Full-fidelity observability — runs, model/tool calls (real model & provider names), durations, errors	on	your machine only
aggregate	Opt-in metadata to Nous (no uploader ships yet)	off (opt-in)	Nous, only if you enable it
trajectories	Full message content / reasoning / raw tool args	off (opt-in)	your own export destinations only

The local plane is the one you'll use day to day — it records the real values that happened (actual model ids, providers, tool names). The aggregate plane is the only thing that could ever leave for Nous; it is opt-in, default-off, and has no uploader today. The trajectories plane unlocks full-content export to your own destinations — it is never wired to Nous.

Local plane — always-on observability

The local plane is implemented as a bundled telemetry plugin that listens to Hermes lifecycle hooks (model calls, tool calls, session start/finalize) and writes events to:

• an append-only JSONL log at ~/.hermes/telemetry/events.jsonl (the source of truth)
• indexed tel_* tables in state.db (for fast queries and rollups)

Writes are fire-and-forget on a background thread: telemetry can never block, slow, or fail a model call or tool call. If the local plane is disabled (telemetry.local: false) the plugin does not load at all.

Seeing your local data

hermes insights            # usage report — now includes an "Observability" section
hermes telemetry status    # planes, consent, export posture, local data volume

The insights Observability section shows workflow counts and success rate, duration p50/p95, tool failure rates by category, provider/model-class mix, and cache hit rate — all computed locally with exact values.

hermes telemetry commands

hermes telemetry status      Show planes, consent state, export posture, local volume
hermes telemetry preview     Show the aggregate events that would be produced (local)
hermes telemetry export      Export local telemetry to a file/stream or OTLP endpoint

Consent and the install id are plain config, not separate verbs — set them with hermes config set (or a managed-scope pin):

hermes config set telemetry.consent_state aggregate   # opt in to the aggregate plane
hermes config set telemetry.consent_state local       # opt out (local plane stays on)
hermes config set telemetry.install_id ""             # reset the install id (mints a new one)

Aggregate plane (opt-in)

The aggregate plane is off by default and has no uploader today — nothing is sent to Nous. Consent lives in telemetry.consent_state (unknown / local / aggregate); setting it to aggregate records the opt-in for if/when an uploader ships. If one is built, it would summarize at that egress boundary.

hermes telemetry preview shows your recent runs as they'd be summarized — computed and shown locally only, with the real model and tool names from your own telemetry. It's a local inspection surface, not an upload.

Enterprise: getting all of your data

Everything below sends data to your own destination — a file, your SIEM, or your own OpenTelemetry Collector. None of it goes to Nous.

Bulk export to a file

# Structural telemetry only (default — no message content)
hermes telemetry export --out telemetry.ndjson

# JSON instead of NDJSON, last 7 days only
hermes telemetry export --out dump.json --format json --since 7

By default the export is structural — runs, model/tool-call metadata, session shells with message counts but no message bodies.

Including content (trajectories plane)

To export full message content, enable the trajectories plane. This is a deliberate, separate consent — it's how an enterprise opts into exporting work-product content to its own store:

# config.yaml
telemetry:
  trajectories:
    enabled: true          # unlocks content export to YOUR destination
  content_redaction: pii   # "none" | "pii"

hermes telemetry export --out full.ndjson --include-content

--include-content is a no-op unless the trajectories plane is enabled — the consent plane governs, not the flag.

Live streaming to your OpenTelemetry Collector / SIEM (OTLP)

Hermes can stream telemetry to your own OTLP endpoint. This requires the optional otlp extra:

pip install 'hermes-agent[otlp]'

# config.yaml
telemetry:
  export:
    otlp:
      enabled: true
      endpoint: "https://collector.your-corp.internal:4318/v1/traces"
      headers_env:                 # secrets by reference — env var NAMES, not values
        Authorization: CORP_OTLP_TOKEN

export CORP_OTLP_TOKEN="..."       # the actual token lives in the environment
hermes telemetry export --otlp     # drain current telemetry to your collector

Span attributes are structural by default. For authentication, the config holds the name of an environment variable rather than the secret itself; the value is read at export time and is never written to config or logged.

Redaction

Two independent controls govern what content looks like on export:

Control	Values	Effect
Secret redaction	always on	API keys, tokens, auth headers, connection strings are always stripped on every export path. Cannot be disabled.
content_redaction	none | pii	When content is exported, pii additionally redacts emails, phone numbers, and id-shaped strings.

Secret redaction is always on — even at full content fidelity — because a SIEM or warehouse full of live credentials is a bigger attack target than the data it holds. It fails closed: if the redactor can't run, the raw string is not emitted.

Configuration reference

telemetry:
  local: true                 # local plane (default on)
  allow_aggregate: true       # hard gate; pin false to forbid the aggregate plane entirely
  consent_state: unknown      # aggregate opt-in: unknown | local | aggregate
  install_id: ""              # stable anon id; "" mints one; clear to rotate
  retention_days: 90          # local event-log retention
  redact_secrets: true        # always-on secret redaction (kept on by design)
  content_redaction: none     # none | pii
  trajectories:
    enabled: false            # unlocks full-content export to your destination
  export:
    otlp:
      enabled: false
      endpoint: null
      headers_env: {}          # {HeaderName: ENV_VAR_NAME}

Enterprise policy via managed scope

Any telemetry.* key can be pinned by an administrator through Hermes' managed-scope layer (/etc/hermes/config.yaml), which wins over the user's value on a per-key basis. There is no telemetry-specific policy block — to lock down a fleet, pin the keys you care about. Common examples:

• telemetry.allow_aggregate: false — the aggregate plane stays off even if consent_state is set to aggregate.
• telemetry.export.otlp.endpoint — point every install at the corporate collector.
• telemetry.trajectories.enabled — centrally decide whether content export is allowed.

When a key is managed, attempts to change it are rejected by managed scope with a message naming the source. hermes telemetry status shows the current export posture (endpoint host, whether the auth env var is set, content gate, redaction modes) — it never prints secret values.

Privacy summary

• Local telemetry never leaves your machine.
• The aggregate plane (the only thing that could go to Nous) is opt-in, default-off, and has no uploader today — nothing is sent.
• All export surfaces (file, OTLP) point at your destinations.
• Secrets are always redacted on export; content export is off until you enable the trajectories plane; PII redaction is a knob.