diff --git a/website/docs/reference/environment-variables.md b/website/docs/reference/environment-variables.md index f324edf160..6bcfff8cc5 100644 --- a/website/docs/reference/environment-variables.md +++ b/website/docs/reference/environment-variables.md @@ -131,6 +131,22 @@ For native Anthropic auth, Hermes prefers Claude Code's own credential files whe | `WANDB_API_KEY` | RL training metrics ([wandb.ai](https://wandb.ai/)) | | `DAYTONA_API_KEY` | Daytona cloud sandboxes ([daytona.io](https://daytona.io/)) | +### Langfuse Observability + +Environment variables for the bundled [`observability/langfuse`](/docs/user-guide/features/built-in-plugins#observabilitylangfuse) plugin. Set these with `hermes tools → Langfuse Observability` or manually in `~/.hermes/.env`. The plugin must also be enabled (`hermes plugins enable observability/langfuse`) before any of these take effect. + +| Variable | Description | +|----------|-------------| +| `HERMES_LANGFUSE_PUBLIC_KEY` | Langfuse project public key (`pk-lf-...`). Required. | +| `HERMES_LANGFUSE_SECRET_KEY` | Langfuse project secret key (`sk-lf-...`). Required. | +| `HERMES_LANGFUSE_BASE_URL` | Langfuse server URL (default: `https://cloud.langfuse.com`). Set for self-hosted. | +| `HERMES_LANGFUSE_ENV` | Environment tag on traces (`production`, `staging`, …) | +| `HERMES_LANGFUSE_RELEASE` | Release/version tag on traces | +| `HERMES_LANGFUSE_SAMPLE_RATE` | SDK sampling rate 0.0–1.0 (default: `1.0`) | +| `HERMES_LANGFUSE_MAX_CHARS` | Per-field truncation for serialized payloads (default: `12000`) | +| `HERMES_LANGFUSE_DEBUG` | `true` enables verbose plugin logging to `agent.log` | +| `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` / `LANGFUSE_BASE_URL` | Standard Langfuse SDK names. Accepted as fallbacks when the `HERMES_LANGFUSE_*` equivalents are unset. | + ### Nous Tool Gateway These variables configure the [Tool Gateway](/docs/user-guide/features/tool-gateway) for paid Nous subscribers or self-hosted gateway deployments. Most users don't need to set these — the gateway is configured automatically via `hermes model` or `hermes tools`. diff --git a/website/docs/user-guide/features/built-in-plugins.md b/website/docs/user-guide/features/built-in-plugins.md index 08cd4af3bf..20c88df68c 100644 --- a/website/docs/user-guide/features/built-in-plugins.md +++ b/website/docs/user-guide/features/built-in-plugins.md @@ -99,6 +99,69 @@ Auto-tracks and removes ephemeral files created during sessions — test scripts **Disabling again:** `hermes plugins disable disk-cleanup`. +### observability/langfuse + +Traces Hermes turns, LLM calls, and tool invocations to [Langfuse](https://langfuse.com) — an open-source LLM observability platform. One span per turn, one generation per API call, one tool observation per tool call. Usage totals, per-type token counts, and cost estimates come out of Hermes' canonical `agent.usage_pricing` numbers, so the Langfuse dashboard sees the same breakdown (input / output / `cache_read_input_tokens` / `cache_creation_input_tokens` / `reasoning_tokens`) that appears in `hermes logs`. + +The plugin is fail-open: no SDK installed, no credentials, or a transient Langfuse error — all turn into a silent no-op in the hook. The agent loop is never impacted. + +**Setup (interactive — recommended):** + +```bash +hermes tools # → Langfuse Observability → Cloud or Self-Hosted +``` + +The wizard collects your keys, `pip install`s the `langfuse` SDK, and adds `observability/langfuse` to `plugins.enabled` for you. Restart Hermes and the next turn ships a trace. + +**Setup (manual):** + +```bash +pip install langfuse +hermes plugins enable observability/langfuse +``` + +Then put the credentials in `~/.hermes/.env`: + +```bash +HERMES_LANGFUSE_PUBLIC_KEY=pk-lf-... +HERMES_LANGFUSE_SECRET_KEY=sk-lf-... +HERMES_LANGFUSE_BASE_URL=https://cloud.langfuse.com # or your self-hosted URL +``` + +**How it works:** + +| Hook | Behaviour | +|---|---| +| `pre_api_request` / `pre_llm_call` | Open (or reuse) a per-turn root span "Hermes turn". Start a `generation` child observation for this API call with serialized recent messages as input. | +| `post_api_request` / `post_llm_call` | Close the generation, attach `usage_details`, `cost_details`, `finish_reason`, assistant output + tool calls. If no tool calls and non-empty content, close the turn. | +| `pre_tool_call` | Start a `tool` child observation with sanitized `args`. | +| `post_tool_call` | Close the tool observation with sanitized `result`. `read_file` payloads get summarized (head + tail + omitted-line count) so a huge file read stays under `HERMES_LANGFUSE_MAX_CHARS`. | + +Session grouping keys off the Hermes session ID (or task ID for sub-agents) via `langfuse.propagate_attributes`, so everything in a single `hermes chat` session lives under one Langfuse session. + +**Verify:** + +```bash +hermes plugins list # observability/langfuse should show "enabled" +hermes chat -q "hello" # check the Langfuse UI for a "Hermes turn" trace +``` + +**Optional tuning** (in `.env`): + +| Variable | Default | Purpose | +|---|---|---| +| `HERMES_LANGFUSE_ENV` | — | Environment tag on traces (`production`, `staging`, …) | +| `HERMES_LANGFUSE_RELEASE` | — | Release/version tag | +| `HERMES_LANGFUSE_SAMPLE_RATE` | `1.0` | Sampling rate passed to the SDK (0.0–1.0) | +| `HERMES_LANGFUSE_MAX_CHARS` | `12000` | Per-field truncation for message content / tool args / tool results | +| `HERMES_LANGFUSE_DEBUG` | `false` | Verbose plugin logging to `agent.log` | + +Hermes-prefixed and standard SDK env vars (`LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_BASE_URL`) are both accepted — Hermes-prefixed wins when both are set. + +**Performance:** the Langfuse client is cached after the first hook call. If credentials or SDK are missing, that decision is also cached — subsequent hooks fast-return without re-checking env vars or reloading config. + +**Disabling:** `hermes plugins disable observability/langfuse`. The plugin module is still discovered, but no module code runs until you re-enable. + ## Adding a bundled plugin Bundled plugins are written exactly like any other Hermes plugin — see [Build a Hermes Plugin](/docs/guides/build-a-hermes-plugin). The only differences are: