> This page is for Gateway. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.merge.dev/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.merge.dev/_mcp/server. # Streaming > Streaming contracts for the native Responses API and the OpenAI, Anthropic, AI SDK, and LangChain surfaces: frame types, mid-stream errors, where usage and cost arrive, and stall timeouts. Set `stream: true` and Gateway returns server-sent events in the shape each surface's SDK expects. Surfaces differ on where usage lands and how failures arrive, which matters for accumulation, billing, and error handling. A request Gateway rejects, or whose first vendor frame is an error, returns the same HTTP status and JSON body as a non-streaming request, with no stream. Once the stream opens the status is `200` and later failures arrive as frames. ## Native `POST /v1/responses` | Frame | `object` | Content | | -------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Interim | `response.stream` | The full response so far. Each frame replaces the previous one, except `logprobs`, which are per-frame deltas. | | Terminal | `response.done` | The final response: `output[0].finish_reason`, `model`, `vendor`, `service_tier`, `usage` with `cost`, and `routing`, `guardrails`, `latency`, and `warnings` when present | | Error | `response.error` | `{"error": {"type", "message", "status_code", "source"}}`, plus `code`, `param`, and `provider` when set. `status_code` is a string. The stream ends. | There is no `data: [DONE]`. When a direct model call fails over to another vendor for the same model, Gateway sends `{"fallback_restart": true, "model": ..., "vendor": ...}` and restarts the snapshots; discard everything received before it. Codex, or any caller sending `X-Merge-Wire-Format: openai`, gets the `/v1/openai/responses` sequence below from `POST /v1/responses` ([details](/merge-gateway/api-overview#openai-responses-api-clients)). ## OpenAI-compatible surfaces **`/v1/openai/chat/completions`** streams `chat.completion.chunk` objects: a role chunk, content deltas, one chunk with the complete `tool_calls` array, and a final chunk with `finish_reason`, `usage` (with `cost`), `service_tier`, and `routing`, `guardrails`, and `warnings` when present, then `data: [DONE]`. Usage is sent regardless of `stream_options`. Errors arrive as the native `response.error` object before `[DONE]`, so check for it in your chunk handler. After a `fallback_restart`, this surface keeps counting from text already sent, so the start of the new vendor's answer can be missing. **`/v1/openai/responses`** streams the typed events strict parsers such as Codex expect, with `event:` and `data:` lines and a `sequence_number`: ```text response.created response.in_progress response.output_item.added -> response.content_part.added response.output_text.delta (repeated) -> response.output_text.done response.output_text.annotation.added (per web search citation) response.content_part.done -> response.output_item.done response.reasoning_summary_part.added / .done (reasoning items) response.reasoning_summary_text.delta / .done response.function_call_arguments.delta / .done (per tool call, arguments in one delta) response.custom_tool_call_input.delta / .done (custom tools) response.completed ``` `output_item.added` and `.done` also wrap web search, image, and tool search items. `response.completed` carries `usage`: `input_tokens`, `output_tokens`, `total_tokens`, cached, cache-write, and reasoning token details, and `cost` (`null` for an unpriced model). A response that hit the output limit has `status: "incomplete"` and `incomplete_details`. On failure the stream ends with an `error` event and `response.failed`, with no `response.completed` or `[DONE]`. ## Anthropic-compatible `/v1/anthropic/v1/messages` Events follow Anthropic's order: `message_start`; `content_block_start`, `content_block_delta` (including `thinking_delta` and `signature_delta`), and `content_block_stop` per block; `message_delta` with usage; `message_stop`. The last two also carry `routing`, `guardrails`, and `warnings` when present. Differences from Anthropic's API: * Tool-use input arrives as one complete `input_json_delta`, not incremental JSON * No `ping` events to use for liveness * Usage has no `cost` (read it from the [usage APIs](/merge-gateway/automation/projects-api/usage)), and the served `service_tier` is reported on non-streaming responses only * An error is `event: error` with `{"type": "error", "error": {"type": "api_error", "message": ...}}` whatever the real status. Treat it as terminal; `message_stop` may not follow. ## AI SDK and LangChain surfaces | Surface | Sequence | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/v1/ai-sdk/chat/completions` | The chat completions sequence above, without `thinking_signature`, logprobs, or citation annotations | | `/v1/ai-sdk/responses` | The Responses sequence without `response.in_progress`, reasoning events, `annotation.added`, or web search items. An error is a `data:`-only frame followed by a normal `response.completed`, so treat any error frame as the failure. | | `/v1/langchain/chat/completions` | Chat completion chunks ending in `data: [DONE]` | Both AI SDK variants include `cost` in usage. ## Timeouts and disconnects Gateway sends your client no keepalive frames, and trips a stream when the vendor goes silent: | Window | Limit | | ------------------------------------------------------------------- | ----------- | | Before the first frame | 120 seconds | | Between frames | 120 seconds | | Longest wait per window while the vendor sends pings but no content | 300 seconds | | Extra waiting across all attempts in one request | 480 seconds | A tripped stream returns `503` or fails over to the next route. Set client read timeouts to at least 300 seconds, and keep proxies from buffering or idle-closing the stream. If your client disconnects, streaming or not, Gateway finishes reading the vendor response, so usage and cost are still recorded and billed. A disconnected non-streaming request is logged with status `499`. ## Reference | Behavior | Detail | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Headers on streams | `X-Request-ID`, rate-limit headers, and budget headers. No `x-merge-vendor`, `x-merge-model`, or `x-merge-routing-policy-id`, so set `include_routing_metadata: true` and read `routing` on the terminal frame. | | Guardrails | `guardrails` is on the terminal frame only. A request blocked before the first token returns a plain `422` JSON body. | | Served tier and warnings | Interim frames report `service_tier: null` and no warnings. Read both from the terminal frame. | | Reasoning signatures | `signature_delta` on the Anthropic surface, `delta.thinking_signature` on `/v1/openai/chat/completions`, reasoning items on `/v1/openai/responses`. Capture them to [replay reasoning](/merge-gateway/capabilities/reasoning#replay-reasoning-in-multi-round-tool-loops). | | Structured output | Arrives as ordinary content deltas. Parse once the stream ends. | ## Next steps #### [Errors](/merge-gateway/errors) Every error code, including the streaming error frames #### [Tool calling](/merge-gateway/capabilities/tool-calling) How tool calls arrive on streaming and non-streaming responses #### [Reasoning](/merge-gateway/capabilities/reasoning) Streaming extended thinking across vendors