feat(relay): Phase 2 media parity — send_media egress + inbound media localization (#71363)

- gateway/relay/media.py: RelayMediaClient for the connector's /relay/media
  plane (upload local files → re-host reference for send_media; download
  re-hosted inbound attachments → local temp paths). Same connector base URL
  the WS dials, same per-gateway signed bearer as the upgrade (auth.py) —
  no new configuration. stdlib urllib in a thread executor (no new deps);
  25MB cap mirroring the connector's MEDIA_MAX_BYTES.
- RelayAdapter: send_image / send_image_file / send_voice / send_video /
  send_document overrides route through ONE send_media op (media by
  reference: local paths upload first, public URLs pass through). Gated on
  supported_ops advertising send_media — legacy connectors keep today's
  text fallbacks; connector declines/failed uploads degrade the same way.
  Scope/user egress discriminators ride metadata exactly like send.
- Inbound: _localize_inbound_media downloads each media_urls entry to a
  local temp path (native-adapter parity — vision/file tools consume
  paths); dead re-host refs are dropped, public URLs survive a missing
  client. Best-effort, never blocks handle_message.
- docs/relay-connector-contract.md §4: send_media op row + media
  ingress/egress semantics (replaces the 'deferred to a later revision'
  note). Additive within contract_version 1.
- tests: tests/gateway/relay/test_relay_media.py (15) — kind mapping,
  upload-first path handling, op gating (explicit + legacy-empty),
  decline/upload-failure fallbacks, scope metadata, inbound localization
  matrix, client URL derivation/credential gating. Stub connector grew a
  canned send_media result.

Cross-repo pair: gateway-gateway 'Phase 2 media parity' PR (re-host plane +
four ingress lanes + four platform send_media senders).
This commit is contained in:
Ben Barclay 2026-07-25 23:15:52 +10:00 committed by GitHub
parent 6ad632bf9b
commit 689b51bef6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 804 additions and 2 deletions

View file

@ -394,10 +394,41 @@ The gateway calls the transport with action dicts. Source of truth:
| `edit` | `chat_id`, `message_id`, `content`, `metadata?` | `{success: bool, error?}` |
| `typing` | `chat_id`, `content?`, `metadata?` | `{success: bool}` |
| `follow_up` | `session_key`, `kind`, `content`, `metadata?` | `{success: bool, message_id?, error?}` |
| `send_media` | `chat_id`, `media_kind`, `source_url`, `content?` (caption), `filename?`, `reply_to?`, `metadata?` | `{success: bool, message_id?, error?}` |
`get_chat_info(chat_id)` is a separate proxied call returning at least
`{name, type}`. Media actions follow the same envelope shape (deferred to a
later contract revision; additive).
`{name, type}`.
**`send_media` (Phase 2 media egress).** Media crosses the wire BY REFERENCE:
`source_url` is either (a) a **connector re-host** the gateway previously
uploaded via `POST {connector}/relay/media` (raw bytes body, `Content-Type` +
optional `X-Media-Filename` headers, per-gateway HMAC bearer — the same token
scheme as the WS upgrade; response `{id, size}` → reference
`{connector}/relay/media/{id}`), or (b) a **public http(s) URL** (e.g. a
fal.media generation) the connector downloads directly. `media_kind` is one of
`image` / `voice` / `audio` / `video` / `document` and selects the
platform-native upload lane (Telegram `sendPhoto`/`sendVoice`/…, Discord
multipart attachment, Slack external upload, WhatsApp media upload + media
message). The caption rides `content` and renders through the platform's
normal markdown lane; platforms without native captions get a follow-up text
send (connector-side). Both routes and the op are gated on `supported_ops`
advertising `send_media` — a legacy connector never sees the op (the gateway's
media sends degrade to their pre-media text fallbacks). Size cap 25 MB
(connector `mediaStore.ts` MEDIA_MAX_BYTES; uploads over it are rejected 413).
**Inbound media (Phase 2 media ingress).** An inbound event's `media_urls`
carry fetchable references: platform-public URLs pass through (Discord CDN);
auth-gated/expiring platform URLs (Telegram file API, Slack `url_private`,
WhatsApp Graph media) are downloaded connector-side with the PLATFORM
credential and re-hosted as `{connector}/relay/media/{id}` — the platform
credential never crosses the wire. Re-host references are readable by any
authenticated gateway (capability-URL semantics: the id is 128-bit random and
was already delivered to every admitted recipient); the gateway downloads each
reference with its per-gateway bearer and presents LOCAL file paths to the
agent, mirroring native adapters. Re-hosts expire (TTL ~1h) — download on
receipt, not lazily. A parallel `media` array (same order) adds `kind`, `mime`,
`size`, `filename`, `caption` metadata; `message_type` reflects the first
attachment's kind (`image`/`audio`/`document`).
**`typing` `content?` (Slack status clear).** A `typing` frame normally omits
`content` — the connector renders its platform's active indicator ("is