feat(api-server): configurable concurrent-run cap to prevent DoS (#50007)

The OpenAI-compatible API server only enforced a hardcoded cap of 10
concurrent runs on /v1/runs, leaving /v1/chat/completions and
/v1/responses unbounded — a request flood could exhaust CPU, memory,
and upstream LLM quota (#7483).

- Add gateway.api_server.max_concurrent_runs (config.yaml, default 10,
  0 disables). No env var.
- Shared concurrency gate across all three agent-serving endpoints,
  counting both the chat/responses in-flight counter and the /v1/runs
  stream set. Returns OpenAI-style 429 + Retry-After when at the cap.
- Remove the dead hardcoded _MAX_CONCURRENT_RUNS class attribute.

Closes #7483.
This commit is contained in:
Teknium 2026-06-21 07:26:03 -07:00 committed by GitHub
parent 99233faf78
commit e499d69e3e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 147 additions and 8 deletions

View file

@ -2511,6 +2511,18 @@ DEFAULT_CONFIG = {
# multi-tool agent turn. Bridged to HERMES_MEDIA_TRUST_RECENT_SECONDS.
# Only consulted when ``strict`` is true.
"trust_recent_files_seconds": 600,
# OpenAI-compatible API server platform
# (gateway/platforms/api_server.py).
"api_server": {
# Maximum number of agent runs the API server will service
# concurrently. Requests to /v1/chat/completions, /v1/responses,
# and /v1/runs that arrive while this many runs are already
# in flight are rejected with HTTP 429 + a Retry-After header,
# bounding CPU / memory / upstream-LLM-quota exhaustion from a
# request flood. Set to 0 to disable the cap entirely.
"max_concurrent_runs": 10,
},
},
# Real-time token streaming to messaging platforms (Telegram, Discord,