Streaming

Set stream: true and Gateway returns server-sent events on every surface. The frames match what the surface’s SDK expects, so the OpenAI, Anthropic, and AI SDK clients parse them natively. This page documents each surface’s exact contract, because the differences (especially around errors and usage) matter once you build accumulation or billing logic on top.

Two facts apply everywhere. The HTTP status commits as 200 when the stream opens, so any later failure arrives as an in-stream frame, not an HTTP error. And usage arrives once, on the terminal frame, which is also where per-call cost appears.

Native POST /v1/responses

There is no data: [DONE] sentinel on this surface; treat response.done or response.error as terminal. If a vendor fails mid-stream and the policy fails over, Gateway emits {"fallback_restart": true, "model": ..., "vendor": ...} and restarts the snapshots from the new route; discard content received before that frame.

OpenAI-compatible surfaces

/v1/openai/chat/completions streams standard chat.completion.chunk objects: a role chunk, content deltas, one chunk carrying the complete tool_calls array, then a final chunk with finish_reason and usage (including cost), followed by data: [DONE]. On error, the native error frame is passed through as-is before [DONE], so also watch for object: "response.error" in your chunk handler.

/v1/openai/responses streams the typed event sequence strict parsers (Codex among them) require, each with a sequence_number:

response.created
response.output_item.added → response.content_part.added
response.output_text.delta (repeated) → response.output_text.done
response.content_part.done → response.output_item.done
response.function_call_arguments.delta / .done   (per tool call)
response.completed

response.completed always carries a full usage object (input_tokens, output_tokens, total_tokens, details, cost). On failure the stream ends with an error event followed by response.failed; no response.completed and no [DONE] follow.

Anthropic-compatible /v1/anthropic/v1/messages

Named SSE events in Anthropic’s own order: message_start, then per content block content_block_start / content_block_delta / content_block_stop (including thinking_delta and signature_delta for extended thinking), then message_delta carrying usage, then message_stop.

Two departures from Anthropic’s hosted API: tool-use blocks arrive as one complete input_json_delta after the text finishes rather than as incremental JSON, and Gateway sends no ping events, so don’t gate liveness on them. Errors arrive as an Anthropic-shaped event: error with {"type": "error", "error": {"type": "api_error", "message": ...}}. Usage on message_delta has Anthropic’s shape and carries no cost field; read cost from the usage APIs on this surface.

AI SDK surface

/v1/ai-sdk/chat/completions and /v1/ai-sdk/responses mirror the OpenAI surfaces’ frame sequences. Differences: usage objects omit cost, and on the responses variant an error frame is followed by a normal response.completed, so treat any error event as the failure signal rather than waiting for a terminal frame.

Timeouts and disconnects

Gateway watches for stalls: if the first chunk takes more than 120 seconds, or the gap between chunks exceeds 120 seconds, the upstream call is abandoned and the failure enters the normal error path (a 408 or 502, or failover under a policy). If your client disconnects mid-stream, Gateway stops sending but drains the provider response in the background, so the request’s usage and cost are still recorded and billed.

Response headers on streams

Streams carry the standard headers (X-Request-ID, rate-limit and budget headers). The x-merge-routing-policy-id header is not present on streaming responses, because headers commit before the policy resolves; use include_routing_metadata: true and read the routing block on the terminal frame instead.

Next steps