> This page is for Gateway.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.merge.dev/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.merge.dev/_mcp/server.

# Multimodal

> Which endpoint and SDK surface handles each modality in Gateway, the inline size caps for media in chat requests, and how to find models that support each one.

Chat models on Gateway accept images, documents, audio, and video as input, and dedicated endpoints generate images, speech, transcripts, and video. Find your modality below, and check the [model catalog](/merge-gateway/models/catalog) for models that support it.

## Support matrix

| Modality                                                                    | Endpoint                                                    | Surfaces                             |
| --------------------------------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------ |
| [Image input](/merge-gateway/capabilities/multimodal/images)                | `POST /v1/responses`                                        | Native, OpenAI, Anthropic, AI SDK    |
| [Document input](/merge-gateway/capabilities/multimodal/documents)          | `POST /v1/responses`                                        | Native, OpenAI, Anthropic, AI SDK    |
| Audio input                                                                 | `POST /v1/responses`                                        | Native, OpenAI                       |
| Video input                                                                 | `POST /v1/responses`                                        | Native, OpenAI                       |
| [Image generation](/merge-gateway/capabilities/multimodal/image-generation) | `POST /v1/images/generations`, or inline on `/v1/responses` | Native                               |
| [Image editing](/merge-gateway/capabilities/multimodal/image-editing)       | `POST /v1/images/edits`                                     | Native, OpenAI SDK at base URL `/v1` |
| [Text to speech](/merge-gateway/capabilities/multimodal/text-to-speech)     | `POST /v1/audio/speech`                                     | Native                               |
| [Speech to text](/merge-gateway/capabilities/multimodal/speech-to-text)     | `POST /v1/audio/transcriptions`                             | Native                               |
| [Live voice](/merge-gateway/capabilities/multimodal/live-voice)             | `wss://api-gateway.merge.dev/v1/live/sessions`              | Native WebSocket                     |
| [Video generation](/merge-gateway/capabilities/multimodal/video-generation) | `POST /v1/videos` (async job)                               | Native                               |

Each compat surface takes its own wire shape (OpenAI `image_url`, `input_audio`, and `file` parts, Anthropic `source` blocks, AI SDK file parts), translated to the served vendor's format. Support is per route, so check your vendor's `vendors.<vendor>.capabilities` on `GET /v1/models`. A route that can't take a modality you sent returns `400 unsupported_params` with `param: input` before any vendor call.

## Input blocks

Native content blocks on `/v1/responses`:

| Block       | Fields                              | Sources                             | Inline cap per request          |
| ----------- | ----------------------------------- | ----------------------------------- | ------------------------------- |
| `image_url` | `url`                               | URL or `data:` URI                  | 20 MB                           |
| `image`     | `source_type`, `media_type`, `data` | base64, URL                         | 20 MB (shared with `image_url`) |
| `document`  | `source_type`, `media_type`, `data` | base64, URL                         | 20 MB                           |
| `audio`     | `format` (such as `wav`), `data`    | base64 only                         | 25 MB                           |
| `video`     | `source_type`, `media_type`, `data` | base64, URL (`https://` or `gs://`) | 20 MB                           |

Each cap is the decoded inline total for that modality across the request. URL media doesn't count, so send long video as a URL. Over a cap, chat requests fail with `413 payload_too_large`; the dedicated media endpoints have their own limits.

Audio input is reported as `usage.audio_input_tokens` and billed at the route's audio rate when it has one; other media bills as ordinary input tokens.

## Next steps

#### [Images](/merge-gateway/capabilities/multimodal/images)

Send images by URL or base64

#### [Model catalog](/merge-gateway/models/catalog)

Filter models by input and output type