Streaming

Set stream: true and Gateway returns server-sent events on every surface. The frames match what the surface’s SDK expects, so the OpenAI, Anthropic, and AI SDK clients parse them natively. This page documents each surface’s exact contract, because the differences (especially around errors and usage) matter once you build accumulation or billing logic on top.

Two facts apply everywhere. A streamed request is validated before the stream opens, so a request that would fail on the non-streaming path returns the same HTTP status here, with no SSE stream and no error frame. Once the stream is open the status has committed as 200, and any later failure arrives as an in-stream frame instead. And usage arrives once, on the terminal frame, which is also where per-call cost appears.

Keep your in-band error handling: a provider that fails partway through a response still reports it mid-stream. It is not needed for request validation, though, since an unservable capability, an exhausted budget, and a policy rejection all reach you as a 400, 402, 403, or 422 on the streaming and non-streaming paths alike.

Native POST /v1/responses

There is no data: [DONE] sentinel on this surface; treat response.done or response.error as terminal. If a vendor fails mid-stream and the policy fails over, Gateway emits {"fallback_restart": true, "model": ..., "vendor": ...} and restarts the snapshots from the new route; discard content received before that frame.

This shape is Gateway’s own and does not depend on the model: an OpenAI model streams the same response.stream snapshots as any other. Clients that parse OpenAI’s Responses events (Codex, the OpenAI SDK’s client.responses.*) should use /v1/openai/responses below; a POST /v1/responses from Codex, or carrying X-Merge-Wire-Format: openai, is served that sequence instead. See OpenAI Responses API clients.

OpenAI-compatible surfaces

/v1/openai/chat/completions streams standard chat.completion.chunk objects: a role chunk, content deltas, one chunk carrying the complete tool_calls array, then a final chunk with finish_reason and usage (including cost), followed by data: [DONE]. On error, the native error frame is passed through as-is before [DONE], so also watch for object: "response.error" in your chunk handler.

/v1/openai/responses streams the typed event sequence strict parsers (Codex among them) require. Every frame carries both an event: line naming the type and a data: line holding the object, and each object carries a sequence_number:

response.created
response.in_progress
response.output_item.added -> response.content_part.added
response.output_text.delta (repeated) -> response.output_text.done
response.output_text.annotation.added                (per web search citation)
response.content_part.done -> response.output_item.done
response.function_call_arguments.delta / .done       (per tool call)
response.completed

response.completed always carries a full usage object (input_tokens, output_tokens, total_tokens, details, cost). A response that stops on the output-token limit carries status: "incomplete" and an incomplete_details object. On failure the stream ends with an error event followed by response.failed; no response.completed and no [DONE] follow.

A request that ran a web search streams one response.output_text.annotation.added frame per citation, and the completed output_text and response.completed objects carry the same citations in their annotations arrays. See Web search.

Anthropic-compatible /v1/anthropic/v1/messages

Named SSE events in Anthropic’s own order: message_start, then per content block content_block_start / content_block_delta / content_block_stop (including thinking_delta and signature_delta for extended thinking), then message_delta carrying usage, then message_stop.

Two departures from Anthropic’s hosted API: tool-use blocks arrive as one complete input_json_delta after the text finishes rather than as incremental JSON, and Gateway sends no ping events, so don’t gate liveness on them. Errors arrive as an Anthropic-shaped event: error with {"type": "error", "error": {"type": "api_error", "message": ...}}. Usage on message_delta has Anthropic’s shape and carries no cost field; read cost from the usage APIs on this surface.

AI SDK surface

/v1/ai-sdk/chat/completions and /v1/ai-sdk/responses mirror the OpenAI surfaces’ frame sequences. Differences: usage objects omit cost, and on the responses variant an error frame is followed by a normal response.completed, so treat any error event as the failure signal rather than waiting for a terminal frame.

What composes with streaming

Streaming is not an either/or with the other capabilities. Structured output, reasoning, and tool calling all work with stream: true, including all three in the same request: a streamed request can return schema-valid JSON and reasoning deltas and a tool call. None of those features is non-streaming-only.

Two behaviors are worth coding for:

• Reasoning signatures arrive last. The signature covers the finished thinking block, so it lands on the terminal frame (signature_delta on the Anthropic surface, delta.thinking_signature on the OpenAI-shaped surfaces). If you replay assistant turns in a tool loop, capture it there. See Replay reasoning in multi-round tool loops.
• Structured output streams as ordinary content deltas. Concatenate them and parse once the stream ends; intermediate frames hold partial JSON that does not parse on its own.
• Warnings and the served service tier arrive on the terminal frame. Interim chunks report service_tier: null and no warnings, so read both from the last frame. The Anthropic-compatible surface is the exception for the tier: its only conformant slot precedes tier resolution, so it reports the tier on non-streaming responses only. See Warnings and Service tiers.

Timeouts and disconnects

Gateway watches for stalls on two windows, both measured as provider silence rather than as elapsed time between content chunks. Keep-alives, pings, and preamble frames all count as activity, so a provider that is slow but talking is left alone.

Both are independent of the 900-second total request timeout. When a window expires the upstream call is abandoned and the failure enters the normal error path, which means a 408 or 503, or failover under a policy.

If your client disconnects mid-stream, Gateway stops sending but drains the provider response in the background, so the request’s usage and cost are still recorded and billed. A non-streaming caller that disconnects gets a 499, with the same background drain and the same billing.

Response headers on streams

Streams carry the standard headers (X-Request-ID, rate-limit and budget headers). The x-merge-routing-policy-id header is not present on streaming responses, because headers commit before the policy resolves; use include_routing_metadata: true and read the routing block on the terminal frame instead.

Next steps