Merge API Documentation

Gateway supports two image input formats. Both work across all vision-capable providers, and Gateway translates to the right format automatically.

Image URL

The simplest way to send an image. Pass a URL directly with the image_url content block.

response = client.responses.create(
    model="openai/gpt-5.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image."},
                {"type": "image_url", "url": "https://example.com/photo.jpg"},
            ],
        }
    ],
)

Image with source type

The image block gives you explicit control over the source type and media type. Use this for base64-encoded images or when you need to specify the format.

Base64

import base64
with open("photo.png", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")
response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {"type": "text", "text": "What do you see?"},
                {
                    "type": "image",
                    "source_type": "base64",
                    "media_type": "image/png",
                    "data": image_data,
                },
            ],
        }
    ],
)

URL

response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {"type": "text", "text": "What do you see?"},
                {
                    "type": "image",
                    "source_type": "url",
                    "media_type": "image/jpeg",
                    "data": "https://example.com/photo.jpg",
                },
            ],
        }
    ],
)

OpenAI SDK

If you’re using the OpenAI SDK pointed at Gateway, use the standard OpenAI image format.

OpenAI SDK

from openai import OpenAI
client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api-gateway.merge.dev/v1/openai",
)
response = client.chat.completions.create(
    model="gpt-5.1",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
            ],
        }
    ],
)

AI SDK

With merge-gateway-ai-sdk-provider, images travel as AI SDK file parts. data may be bytes, a base64 string, a data: URI, or a URL.

AI SDK

import { createMergeGateway } from "merge-gateway-ai-sdk-provider";
import { generateText } from "ai";
import { readFileSync } from "fs";
const merge = createMergeGateway({ apiKey: process.env.MERGE_GATEWAY_API_KEY });
const { text } = await generateText({
  model: merge("openai/gpt-5.1"),
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What do you see?" },
        { type: "file", mediaType: "image/png", data: readFileSync("photo.png") },
      ],
    },
  ],
});

File parts require merge-gateway-ai-sdk-provider 0.3.0 or later. Earlier versions dropped them without an error, so the model answered without ever seeing the image.

Multiple images

Pass multiple image blocks in the same message to compare or analyze several images at once.

Multiple images

response = client.responses.create(
    model="openai/gpt-5.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {"type": "text", "text": "Compare these two images."},
                {"type": "image_url", "url": "https://example.com/before.jpg"},
                {"type": "image_url", "url": "https://example.com/after.jpg"},
            ],
        }
    ],
)

Each image is estimated at ~765 tokens for context window and compression calculations. Keep this in mind when sending multiple images in a single request.