feat(relay): terminal 4401 (opt-out) → clean "Relay disabled" state
Some checks are pending
CI / detect (push) Waiting to run
CI / tests (push) Blocked by required conditions
CI / lint (push) Blocked by required conditions
CI / typecheck (push) Blocked by required conditions
CI / docs-site (push) Blocked by required conditions
CI / history-check (push) Blocked by required conditions
CI / contributor-check (push) Blocked by required conditions
CI / uv-lockfile (push) Blocked by required conditions
CI / docker-lint (push) Blocked by required conditions
CI / supply-chain (push) Blocked by required conditions
CI / osv-scanner (push) Blocked by required conditions
CI / All required checks pass (push) Blocked by required conditions
Deploy Site / deploy-vercel (push) Waiting to run
Deploy Site / deploy-docs (push) Waiting to run
Docker Build and Publish / build-amd64 (push) Waiting to run
Docker Build and Publish / build-arm64 (push) Waiting to run
Docker Build and Publish / merge (push) Blocked by required conditions

Phase 7 Unit 7d-B. When an operator opts an instance OUT of the Team Gateway
relay (Unit 7b deprovision), the connector revokes the per-gateway secret and
closes the gateway's WS with 4401. The reconnect supervisor previously treated
EVERY close as retryable, so the live process spun "retrying 4401" forever and
the dashboard showed a red error — opt-out looked like a failure.

Now a 4401 close that arrives AFTER a successful handshake is recognized as a
terminal credential revocation:

- ws_transport.py: track `_handshake_succeeded` (set when a descriptor is
  received); on a 4401 close after a prior success, latch `auth_revoked` and do
  NOT spawn the reconnect supervisor. A 4401 BEFORE any successful handshake
  stays retryable (cold-start / not-yet-provisioned race, not a revocation).
  New `auth_revoked` property + a websockets-version-safe close-code reader
  (prefers `.rcvd`/`.sent` Close frames; `.code` is deprecated in websockets 13+).
- adapter.py: a revocation monitor turns `transport.auth_revoked` into a clean,
  NON-retryable `relay_disabled` fatal and notifies the gateway's fatal-error
  handler (so the adapter is removed and NOT queued for reconnection — the
  credential is dead until the instance is recreated). Monitor is cancelled on
  disconnect; only started when the transport exposes `auth_revoked` (prod WS).
- run.py: `_handle_adapter_fatal_error` maps the `relay_disabled` code to a
  `disabled` platform_state (not `fatal`/`retrying`).
- web: PlatformsCard renders the `disabled` state with a neutral outline badge,
  a PowerOff icon, and muted (not destructive-red) text + message. New optional
  `status.disabled` i18n string ("Disabled").

Also bundles the Phase 7 contract-doc update (this doc is authoritative in
hermes-agent): docs/relay-connector-contract.md gains an "Author-first
resolution + the account-link (DM) path" section documenting the
multi-tenant-guild rule (D-7.2 — route by authenticated author binding, never by
guild; unlinked → fail-closed), the `/link <code>` DM flow, and the
connector-authoritative opt-out + terminal-4401 behavior this PR implements.

Tests: +2 ws_transport (4401-after-handshake terminal / no-reconnect;
4401-before-handshake stays retryable) and +2 adapter (revocation → non-retryable
relay_disabled fatal + handler fired; no-revocation → no fatal). 138 relay tests
pass (incl. the contract-doc conformance test); ruff clean; web tsc clean.

Phase 7 Unit 7d-B (relay-adapter solo lane). Q17 → Option 2; Option 3 (live
de-register, no recreate) + the restart-re-provision hole deferred post-alpha.
This commit is contained in:
Ben 2026-06-24 18:13:37 +10:00 committed by Ben Barclay
parent 3c75e11571
commit c93b9f9057
9 changed files with 367 additions and 8 deletions

View file

@ -186,6 +186,56 @@ tenant**. Tenant is resolved from the event's own discriminator (Discord
token/socket/process delivered it. This keeps one shared bot able to front many
tenants (Phase 6) without overloading an existing field.
### Author-first resolution + the account-link (DM) path (Phase 7)
Phase 7 adds **self-serve, per-user onboarding to a shared bot**, which changes
*which* discriminator resolves the instance for a routed inbound message — and
adds a management path for users to bind their own account.
**Author-first resolution (the multi-tenant-guild rule, D-7.2).** A single
Discord guild may hold **many** tenants — different members each linked to their
own agent. So for delivery the connector resolves the destination instance from
the **authenticated author binding** (`user_instance_binding`, keyed by
`(tenant, platform, platform_user_id)` via `resolveByUser`), **NOT** by a
guild→instance route. Concretely:
- A routed message authored by a **linked** user reaches **only that user's**
instance — even when a second linked user in the **same guild** is served by a
different instance (each reaches only their own).
- A message authored by an **unlinked** user resolves to **no** instance and is
dropped (**fail-closed** — never broadcast to the guild's other tenants).
- The author id used is the **authentic `user_id` off the observed event**, the
same `SessionSource.user_id` documented above — never a value asserted by a
gateway or carried in a management frame.
This is the per-`user_id` owner-only routing the connector enforces in
`WsGatewayDelivery` (the gateway-side multi-tenant-guild E2E driver
`gateway_multitenant_guild_driver.py` is the cross-repo oracle).
**The account-link (DM) path.** A user binds their account to an instance with a
one-time code, redeemed by DMing the shared bot:
1. The owner triggers a link from the Portal (or a self-hosted CLI). The
connector mints a short-lived **link code** for the **authenticated**
instance (`POST /manage/link`; instanceId comes from the caller's principal —
a NAS-signed `aud=agent:{instanceId}` token or the instance's own per-gateway
secret — **never** the request body).
2. The user sends `/link <code>` as a **direct message** to the shared bot from
the account they want to bind.
3. The connector's inbound observer **consumes** that DM (it is not routed to any
agent) and writes the `user_instance_binding` using the **authentic
`user_id`** off the observed DM event. From then on, author-first resolution
routes that user's messages to the bound instance.
**Opt-out is connector-authoritative.** Deprovisioning an instance
(`POST /manage/deprovision`) drops its author bindings (so its users stop
resolving to it) **and** revokes its per-gateway secret (so its socket can no
longer authenticate — the next WS upgrade is closed **4401**). A gateway that
sees a **4401 close after a previously-successful handshake** treats it as a
terminal revocation: it stops reconnecting and reports the relay platform as
**disabled** (not a retryable error). A 4401 *before* any successful handshake
stays retryable (a cold-start / not-yet-provisioned race, not a revocation).
### 3.2 Going-idle / buffered-flip primitive (§5.3)
A scale-to-zero PRIMITIVE (not the behaviour — nothing here decides to sleep or

View file

@ -67,6 +67,12 @@ class RelayAdapter(BasePlatformAdapter):
# (channel) since that's what send() receives. See routedEgressGuard.ts.
self._scope_by_chat: Dict[str, str] = {}
self.supports_code_blocks = descriptor.markdown_dialect not in ("", "plain")
# Phase 7 Unit 7d-B: watches the transport for a terminal auth revocation
# (a 4401 close after a successful handshake = the operator opted this
# instance out of the relay). On revocation we surface a clean,
# non-retryable "relay disabled" fatal so the dashboard stops showing a
# red "retrying" spin against a dead credential.
self._revocation_monitor: Optional[asyncio.Task[None]] = None
# ── capability surface (from descriptor) ─────────────────────────────
@property
@ -114,8 +120,59 @@ class RelayAdapter(BasePlatformAdapter):
# the connector's relay bus — there is NO inbound HTTP endpoint (hosted
# gateways have no public IP). The transport's reader already dispatches
# `inbound` / `interrupt_inbound` frames to the handlers wired above.
# Phase 7 Unit 7d-B: start watching for a terminal auth revocation
# (opt-out). Only meaningful when the transport exposes `auth_revoked`
# (the production WebSocket transport); the test/stub transports don't.
if hasattr(self._transport, "auth_revoked"):
self._start_revocation_monitor()
return True
def _start_revocation_monitor(self) -> None:
"""Spawn (once) the task that turns a transport auth-revocation into a
clean non-retryable 'relay disabled' fatal. Idempotent."""
if self._revocation_monitor is not None and not self._revocation_monitor.done():
return
try:
self._revocation_monitor = asyncio.create_task(
self._watch_for_revocation(), name="relay-revocation-monitor"
)
except RuntimeError:
# No running loop (e.g. a unit test calling connect() synchronously
# via a stub) — nothing to monitor.
self._revocation_monitor = None
async def _watch_for_revocation(self, poll_interval_s: float = 1.0) -> None:
"""Poll the transport for a terminal 4401 revocation (opt-out). On
revocation, surface a non-retryable `relay_disabled` fatal so the
dashboard renders a clean 'Relay disabled' state instead of a red
'retrying' spin, and notify the gateway's fatal-error handler so the
adapter is cleanly removed (it is NOT queued for reconnection, because
the credential is dead until the instance is recreated)."""
transport = self._transport
try:
while True:
if transport is None or getattr(transport, "auth_revoked", False):
break
await asyncio.sleep(poll_interval_s)
except asyncio.CancelledError:
raise
if transport is None or not getattr(transport, "auth_revoked", False):
return
logger.warning(
"relay credential revoked (opt-out) — marking the relay adapter disabled"
)
# Non-retryable: a revoked secret never comes back without a recreate, so
# _handle_adapter_fatal_error must NOT queue it for reconnection.
self._set_fatal_error(
"relay_disabled",
"Relay disabled (opted out — recreate the instance to re-enable)",
retryable=False,
)
try:
await self._notify_fatal_error()
except Exception: # noqa: BLE001 - notification is best-effort
logger.debug("relay revocation fatal-error notify failed", exc_info=True)
def _apply_descriptor(self, descriptor: CapabilityDescriptor) -> None:
"""Adopt a (re)negotiated descriptor into the live capability surface."""
self.descriptor = descriptor
@ -254,6 +311,15 @@ class RelayAdapter(BasePlatformAdapter):
return MessageEvent(text=text, message_type=MessageType.TEXT, source=source)
async def disconnect(self) -> None:
# Phase 7 Unit 7d-B: stop the revocation monitor first so it can't fire a
# spurious fatal during/after a deliberate teardown.
if self._revocation_monitor is not None:
self._revocation_monitor.cancel()
try:
await self._revocation_monitor
except (asyncio.CancelledError, Exception): # noqa: BLE001 - best-effort teardown
pass
self._revocation_monitor = None
if self._transport is not None:
# Phase 5 §5.3: emit going_idle as part of the gateway's EXISTING
# drain/shutdown transition (the runner calls adapter.disconnect()

View file

@ -54,6 +54,13 @@ WEBSOCKETS_AVAILABLE = websockets is not None
_HANDSHAKE_TIMEOUT_S = 30.0
_OUTBOUND_TIMEOUT_S = 30.0
# Phase 7 Unit 7d-B: the application close code the connector sends when it
# rejects/revokes a gateway's WS upgrade auth (mirrors the connector's
# `4401` "unauthorized" close — a private-use code, not a standard WS code).
# A 4401 received AFTER a successful handshake means the per-gateway secret was
# revoked (opt-out / deprovision), which the transport treats as terminal.
_RELAY_UNAUTHORIZED_CLOSE_CODE = 4401
def _ws_dial_url(url: str) -> str:
"""Normalize a connector URL to the ``ws(s)://…/relay`` dial target.
@ -236,6 +243,17 @@ class WebSocketRelayTransport:
# Phase 5 §5.3: future awaiting the connector's going_idle_ack.
self._going_idle_ack: asyncio.Future[None] | None = None
self._closing = False
# Phase 7 Unit 7d-B: a 4401 (unauthorized) close AFTER we have already
# handshaked successfully at least once means the connector REVOKED this
# gateway's per-gateway secret — i.e. the operator opted this instance
# OUT of the relay (Unit 7b deprovision). That is TERMINAL: the secret is
# gone, so re-dialing just spins against a dead credential forever
# (the "retrying 4401" the dashboard showed). We stop reconnecting and
# surface it as a clean, non-retryable "disabled" state. A 4401 BEFORE
# any successful handshake stays retryable — that's a cold-start /
# not-yet-provisioned race, not a revocation.
self._handshake_succeeded = False
self._auth_revoked = False
# ── lifecycle ────────────────────────────────────────────────────────
async def connect(self) -> bool:
@ -313,6 +331,14 @@ class WebSocketRelayTransport:
raise RuntimeError("handshake() called before connect()")
return await asyncio.wait_for(self._descriptor_ready, timeout=self._connect_timeout_s)
@property
def auth_revoked(self) -> bool:
"""True once the connector closed the socket with 4401 AFTER a prior
successful handshake i.e. the per-gateway secret was revoked (the
operator opted this instance out of the relay). Terminal: the transport
stops reconnecting, and the adapter surfaces a clean "disabled" state."""
return self._auth_revoked
def set_inbound_handler(self, handler: InboundHandler) -> None:
self._inbound = handler
@ -412,18 +438,52 @@ class WebSocketRelayTransport:
except asyncio.CancelledError:
raise
except Exception as exc: # noqa: BLE001 - log + let the task end; reconnection handled below
if not self._closing:
# Phase 7 Unit 7d-B: detect a 4401 (unauthorized) close. After a prior
# successful handshake this is a REVOCATION (opt-out / deprovision) —
# the per-gateway secret is gone, so reconnecting is futile. Latch a
# terminal "auth revoked" state and DON'T re-dial. Before any
# successful handshake a 4401 stays retryable (cold-start race).
if self._close_code_of(exc) == _RELAY_UNAUTHORIZED_CLOSE_CODE and self._handshake_succeeded:
self._auth_revoked = True
if not self._closing:
logger.warning(
"relay ws closed 4401 (unauthorized) after a successful handshake — "
"treating as a revoked relay credential (opt-out); not reconnecting"
)
elif not self._closing:
logger.warning("relay ws read loop ended: %s", exc)
# Phase 5 §5.3: the socket closed. If reconnect is enabled and this was
# NOT a deliberate disconnect(), kick the reconnect supervisor so the
# gateway re-dials + re-handshakes (which triggers the connector's
# buffered-flip drain on the new handshake). Self-scheduling: the reader
# ends here, the supervisor re-dials and starts a fresh reader.
if self._reconnect and not self._closing and (self._supervisor is None or self._supervisor.done()):
# Phase 7 Unit 7d-B: a revoked credential (terminal 4401) is the one case
# we deliberately do NOT reconnect — the secret is dead until the
# instance is recreated, so spinning would just reproduce the failure.
if (
self._reconnect
and not self._closing
and not self._auth_revoked
and (self._supervisor is None or self._supervisor.done())
):
self._supervisor = asyncio.create_task(
self._reconnect_loop(), name="relay-ws-reconnect"
)
@staticmethod
def _close_code_of(exc: BaseException) -> Optional[int]:
"""Best-effort extraction of a WebSocket close code from a raised
exception. websockets' ConnectionClosed* expose the peer's Close frame
via `.rcvd`/`.sent` (preferred; `.code` is deprecated in websockets 13+).
Returns None when unknown."""
for attr in ("rcvd", "sent"):
frame = getattr(exc, attr, None)
fcode = getattr(frame, "code", None)
if isinstance(fcode, int):
return fcode
code = getattr(exc, "code", None)
return code if isinstance(code, int) else None
async def _reconnect_loop(self) -> None:
"""Re-dial the connector with capped exponential backoff until reconnected
or disconnect() is called. NET-NEW for §5.3: a re-established socket makes
@ -458,6 +518,11 @@ class WebSocketRelayTransport:
if ftype == "descriptor":
descriptor = CapabilityDescriptor.from_json(json.dumps(frame.get("descriptor", {})))
self._descriptor = descriptor
# Phase 7 Unit 7d-B: a received descriptor means the WS upgrade auth
# passed and the connector accepted us — record that we've handshaked
# at least once, so a LATER 4401 close is read as a revocation
# (opt-out), not a cold-start race.
self._handshake_succeeded = True
if self._descriptor_ready is not None and not self._descriptor_ready.done():
self._descriptor_ready.set_result(descriptor)
elif ftype == "inbound":

View file

@ -3473,9 +3473,19 @@ class GatewayRunner(GatewayAuthorizationMixin, GatewayKanbanWatchersMixin, Gatew
adapter.fatal_error_code or "unknown",
adapter.fatal_error_message or "unknown error",
)
# Phase 7 Unit 7d-B: a relay credential revoked by opt-out is not an
# error to retry — render it as a clean "disabled" state, not red
# "fatal"/"retrying". (The code is set non-retryable, so it also drops
# out of the reconnect queue below.)
if adapter.fatal_error_code == "relay_disabled":
platform_state = "disabled"
elif adapter.fatal_error_retryable:
platform_state = "retrying"
else:
platform_state = "fatal"
self._update_platform_runtime_status(
adapter.platform.value,
platform_state="retrying" if adapter.fatal_error_retryable else "fatal",
platform_state=platform_state,
error_code=adapter.fatal_error_code,
error_message=adapter.fatal_error_message,
)

View file

@ -140,3 +140,61 @@ async def test_send_preserves_explicit_guild_id():
a._capture_scope(_make_event(chat_id="chan-1", guild_id="guild-9"))
await a.send("chan-1", "hi", metadata={"guild_id": "explicit-1"})
assert t.sent["metadata"]["guild_id"] == "explicit-1"
# ── Phase 7 Unit 7d-B: terminal auth revocation → clean "relay disabled" ─────
class _RevokedTransport:
"""Transport stand-in that reports a terminal auth revocation (the
production WebSocketRelayTransport latches this after a 4401 close that
follows a successful handshake)."""
def __init__(self):
self.auth_revoked = True
def set_inbound_handler(self, h): # noqa: D401
self._h = h
@pytest.mark.asyncio
async def test_revocation_marks_relay_disabled_non_retryable():
"""When the transport reports auth_revoked, the adapter surfaces a clean,
NON-retryable `relay_disabled` fatal and fires the fatal-error handler."""
a = RelayAdapter(PlatformConfig(), make_desc(platform="discord"), transport=_RevokedTransport())
notified = []
a.set_fatal_error_handler(lambda adapter: notified.append(adapter))
# Drive the monitor body directly (poll loop breaks immediately on the
# already-revoked transport).
await a._watch_for_revocation(poll_interval_s=0.01)
assert a.has_fatal_error is True
assert a.fatal_error_code == "relay_disabled"
assert a.fatal_error_retryable is False
assert "disabled" in (a.fatal_error_message or "").lower()
assert notified == [a]
@pytest.mark.asyncio
async def test_no_revocation_no_fatal():
"""A transport that has NOT been revoked never trips the disabled fatal."""
class _LiveTransport:
auth_revoked = False
def set_inbound_handler(self, h): # noqa: D401
self._h = h
a = RelayAdapter(PlatformConfig(), make_desc(platform="discord"), transport=_LiveTransport())
# Run the monitor with a tiny window then cancel — it should never fire.
import asyncio
task = asyncio.create_task(a._watch_for_revocation(poll_interval_s=0.01))
await asyncio.sleep(0.05)
task.cancel()
try:
await task
except asyncio.CancelledError:
pass
assert a.has_fatal_error is False

View file

@ -199,3 +199,100 @@ def test_ws_dial_url_idempotent_with_scheme_and_path():
assert t2._url == "wss://connector.example/relay"
t3 = WebSocketRelayTransport("ws://127.0.0.1:9", "discord", "b")
assert t3._url == "ws://127.0.0.1:9/relay"
# ── Phase 7 Unit 7d-B: terminal 4401 (opt-out revocation) ────────────────────
class _Revoking4401Server:
"""Connector stub that, on hello, optionally sends a descriptor and then
closes the socket with application code 4401 (unauthorized) the shape of a
connector that has revoked this gateway's per-gateway secret (opt-out)."""
def __init__(self, *, send_descriptor_first: bool):
self._server = None
self.url = ""
self._send_descriptor_first = send_descriptor_first
async def start(self):
self._server = await websockets.serve(self._handle, "127.0.0.1", 0)
port = next(iter(self._server.sockets)).getsockname()[1]
self.url = f"ws://127.0.0.1:{port}"
async def stop(self):
if self._server is not None:
self._server.close()
await self._server.wait_closed()
async def _handle(self, ws):
async for raw in ws:
for line in str(raw).split("\n"):
if not line.strip():
continue
frame = json.loads(line)
if frame.get("type") == "hello":
if self._send_descriptor_first:
await ws.send(
json.dumps({"type": "descriptor", "descriptor": DESCRIPTOR}) + "\n"
)
# Let the descriptor flush + be processed before the close.
await asyncio.sleep(0.05)
# Close with 4401 (the connector's "unauthorized" close).
await ws.close(code=4401, reason="unauthorized")
return
@pytest.mark.asyncio
async def test_4401_after_handshake_is_terminal_no_reconnect():
"""A 4401 close AFTER a successful handshake = a revoked credential (opt-out):
the transport latches auth_revoked and does NOT spin the reconnect supervisor."""
srv = _Revoking4401Server(send_descriptor_first=True)
await srv.start()
try:
t = WebSocketRelayTransport(
srv.url, "discord", "appShared",
gateway_id="gw-x", upgrade_secret="secret-x",
reconnect=True, reconnect_backoff_s=0.05,
)
await t.connect()
await t.handshake() # records _handshake_succeeded
# Wait for the server's 4401 close to propagate through the read loop.
for _ in range(100):
if t.auth_revoked:
break
await asyncio.sleep(0.02)
assert t.auth_revoked is True
# Terminal: no reconnect supervisor was spawned.
assert t._supervisor is None
# Give a reconnect (if it were going to happen) time to NOT happen.
await asyncio.sleep(0.2)
assert t._supervisor is None
finally:
await t.disconnect()
await srv.stop()
@pytest.mark.asyncio
async def test_4401_before_handshake_stays_retryable():
"""A 4401 close BEFORE any successful handshake is a cold-start / not-yet-
provisioned race, NOT a revocation: it stays retryable (reconnect runs)."""
srv = _Revoking4401Server(send_descriptor_first=False)
await srv.start()
try:
t = WebSocketRelayTransport(
srv.url, "discord", "appShared",
gateway_id="gw-x", upgrade_secret="secret-x",
reconnect=True, reconnect_backoff_s=0.05,
)
await t.connect()
# No handshake ever succeeded; the 4401 must NOT latch auth_revoked.
for _ in range(50):
if t._supervisor is not None:
break
await asyncio.sleep(0.02)
assert t.auth_revoked is False
# The reconnect supervisor IS running (retrying), since this is not terminal.
assert t._supervisor is not None
finally:
await t.disconnect()
await srv.stop()

View file

@ -1,4 +1,4 @@
import { AlertTriangle, Radio, Wifi, WifiOff } from "lucide-react";
import { AlertTriangle, PowerOff, Radio, Wifi, WifiOff } from "lucide-react";
import type { PlatformStatus } from "@/lib/api";
import { isoTimeAgo } from "@/lib/utils";
import { Badge } from "@nous-research/ui/ui/components/badge";
@ -9,10 +9,11 @@ export function PlatformsCard({ platforms }: PlatformsCardProps) {
const { t } = useI18n();
const platformStateBadge: Record<
string,
{ tone: "success" | "warning" | "destructive"; label: string }
{ tone: "success" | "warning" | "destructive" | "outline"; label: string }
> = {
connected: { tone: "success", label: t.status.connected },
disconnected: { tone: "warning", label: t.status.disconnected },
disabled: { tone: "outline", label: t.status.disabled ?? "Disabled" },
fatal: { tone: "destructive", label: t.status.error },
};
@ -38,7 +39,9 @@ export function PlatformsCard({ platforms }: PlatformsCardProps) {
? Wifi
: info.state === "fatal"
? AlertTriangle
: WifiOff;
: info.state === "disabled"
? PowerOff
: WifiOff;
return (
<div
@ -52,7 +55,9 @@ export function PlatformsCard({ platforms }: PlatformsCardProps) {
? "text-success"
: info.state === "fatal"
? "text-destructive"
: "text-warning"
: info.state === "disabled"
? "text-muted-foreground"
: "text-warning"
}`}
/>
@ -62,7 +67,13 @@ export function PlatformsCard({ platforms }: PlatformsCardProps) {
</span>
{info.error_message && (
<span className="font-mondwest normal-case text-xs text-destructive">
<span
className={`font-mondwest normal-case text-xs ${
info.state === "disabled"
? "text-muted-foreground"
: "text-destructive"
}`}
>
{info.error_message}
</span>
)}

View file

@ -107,6 +107,7 @@ export const en: Translations = {
activeSessions: "Active Sessions",
connected: "Connected",
connectedPlatforms: "Connected Platforms",
disabled: "Disabled",
disconnected: "Disconnected",
error: "Error",
failed: "Failed",

View file

@ -124,6 +124,7 @@ export interface Translations {
agent: string;
connected: string;
connectedPlatforms: string;
disabled?: string;
disconnected: string;
error: string;
failed: string;