Streaming

Set stream: true and Gateway returns server-sent events in the shape each surface’s SDK expects. Surfaces differ on where usage lands and how failures arrive, which matters for accumulation, billing, and error handling.

A request Gateway rejects, or whose first vendor frame is an error, returns the same HTTP status and JSON body as a non-streaming request, with no stream. Once the stream opens the status is 200 and later failures arrive as frames.

Native POST /v1/responses

There is no data: [DONE]. When a direct model call fails over to another vendor for the same model, Gateway sends {"fallback_restart": true, "model": ..., "vendor": ...} and restarts the snapshots; discard everything received before it.

Codex, or any caller sending X-Merge-Wire-Format: openai, gets the /v1/openai/responses sequence below from POST /v1/responses (details).

OpenAI-compatible surfaces

/v1/openai/chat/completions streams chat.completion.chunk objects: a role chunk, content deltas, one chunk with the complete tool_calls array, and a final chunk with finish_reason, usage (with cost), service_tier, and routing, guardrails, and warnings when present, then data: [DONE]. Usage is sent regardless of stream_options. Errors arrive as the native response.error object before [DONE], so check for it in your chunk handler. After a fallback_restart, this surface keeps counting from text already sent, so the start of the new vendor’s answer can be missing.

/v1/openai/responses streams the typed events strict parsers such as Codex expect, with event: and data: lines and a sequence_number:

response.created
response.in_progress
response.output_item.added -> response.content_part.added
response.output_text.delta (repeated) -> response.output_text.done
response.output_text.annotation.added                  (per web search citation)
response.content_part.done -> response.output_item.done
response.reasoning_summary_part.added / .done          (reasoning items)
response.reasoning_summary_text.delta / .done
response.function_call_arguments.delta / .done         (per tool call, arguments in one delta)
response.custom_tool_call_input.delta / .done          (custom tools)
response.completed

output_item.added and .done also wrap web search, image, and tool search items. response.completed carries usage: input_tokens, output_tokens, total_tokens, cached, cache-write, and reasoning token details, and cost (null for an unpriced model). A response that hit the output limit has status: "incomplete" and incomplete_details. On failure the stream ends with an error event and response.failed, with no response.completed or [DONE].

Anthropic-compatible /v1/anthropic/v1/messages

Events follow Anthropic’s order: message_start; content_block_start, content_block_delta (including thinking_delta and signature_delta), and content_block_stop per block; message_delta with usage; message_stop. The last two also carry routing, guardrails, and warnings when present.

Differences from Anthropic’s API:

• Tool-use input arrives as one complete input_json_delta, not incremental JSON
• No ping events to use for liveness
• Usage has no cost (read it from the usage APIs), and the served service_tier is reported on non-streaming responses only
• An error is event: error with {"type": "error", "error": {"type": "api_error", "message": ...}} whatever the real status. Treat it as terminal; message_stop may not follow.

AI SDK and LangChain surfaces

Both AI SDK variants include cost in usage.

Timeouts and disconnects

Gateway sends your client no keepalive frames, and trips a stream when the vendor goes silent:

A tripped stream returns 503 or fails over to the next route. Set client read timeouts to at least 300 seconds, and keep proxies from buffering or idle-closing the stream.

If your client disconnects, streaming or not, Gateway finishes reading the vendor response, so usage and cost are still recorded and billed. A disconnected non-streaming request is logged with status 499.

Reference

Next steps