Multimodal

Gateway accepts multimodal content natively. Include an image, document, audio, or video content block in your messages and Gateway routes to a capable model. No configuration needed. Gateway automatically detects which models support each modality and translates content to the provider’s format, so the same request body works on the native Responses API and on the OpenAI, Anthropic, and AI SDK compatible surfaces.

Supported content types

Audio takes base64 only, so there is no URL form for it. Video accepts a URL, which is the way past the inline cap on a file of any real length. Over the cap, the request is rejected with 413 payload_too_large rather than being truncated.

Send an audio or video block to a route that cannot accept it and the request fails with 400 before a provider is called, on every surface. Twenty-one vendor routes in the current catalog accept audio input and thirteen accept video, so check the route rather than the model family.

A dedicated transcription endpoint is a separate thing from audio input: use an audio content block when you want a model to reason about a recording, and speech to text when you want the transcript itself.

Quick example

1
from merge_gateway import MergeGateway
2
3
client = MergeGateway(api_key="YOUR_API_KEY")
4
5
response = client.responses.create(
6
    model="openai/gpt-5.1",
7
    input=[
8
        {
9
            "type": "message",
10
            "role": "user",
11
            "content": [
12
                {"type": "text", "text": "What's in this image?"},
13
                {"type": "image_url", "url": "https://example.com/photo.jpg"},
14
            ],
15
        }
16
    ],
17
)
18
19
print(response.output[0].content[0].text)

Model compatibility

Gateway auto-detects multimodal capabilities from vendor-specific model metadata, so support tracks the live catalog rather than a fixed list. Use GET /v1/models and inspect vendors.<vendor>.capabilities.input, which holds some combination of text, image, document, audio, and video, or filter by input type in the model catalog.

Billing follows the modality. Audio input is reported separately as usage.audio_input_tokens, a subset of input_tokens, and billed at the route’s audio rate rather than the text rate, which is usually higher. Image, document, and video input bill as ordinary input tokens.

Next steps