> This page is for Gateway.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.merge.dev/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.merge.dev/_mcp/server.

# Self-hosted models

> Connect your own OpenAI-compatible inference endpoints (vLLM, Baseten, and similar) to Merge Gateway. Covers endpoint requirements, adding, testing and activating an endpoint, calling self-hosted models, using them in Priority policies, failover, and billing.

Connect inference endpoints you run yourself, such as a vLLM cluster or a dedicated Baseten deployment, to Gateway. Your apps call them through the same Gateway API as every other model, and you can add them to routing policies next to curated vendors. Use them for fine-tuned or private models, for capacity you've already paid for, or when prompts must stay on your own infrastructure.

## How it works

You register an endpoint once: its URL, its API key, and the models it serves. When a request names one of those models, Gateway sends it to your endpoint and returns the response in the same format as any other model.

* **Only your organization** can see, call, or route to your endpoints.
* **Your API key is write-only.** Gateway encrypts it and never shows it again in the dashboard or API.
* **Merge doesn't bill this traffic.** You pay for your own infrastructure; nothing is charged to your Gateway balance.
* **Zero data retention (ZDR) organizations can use them**, because prompts go to infrastructure you control. Failover to curated vendors still only uses ZDR-compliant vendors.

## Endpoint requirements

Your endpoint must:

* Serve an **OpenAI-compatible** chat completions API (`/v1/chat/completions`, which vLLM, TGI, SGLang, Baseten, and most inference servers support)
* Use **HTTPS** and be reachable from the public internet. Private and loopback addresses are rejected.
* Accept its API key as a **bearer token** (`Authorization: Bearer <key>`)

## Add an endpoint

You need the Manage credentials permission to add, edit, or delete an endpoint.

1. In the Gateway dashboard, open **Configure** and go to **Self-hosted models**
2. Choose **Add endpoint**
3. Enter a **Display name**, and a **Slug** (lowercase letters, numbers, and hyphens). The slug can't be changed later.
4. Enter the **Endpoint URL**, for example `https://llm.example.com/v1`, and the **API key**
5. Add each model the endpoint serves:
   * **Model ID**: the name your applications will use to call it
   * **Upstream name**: only if your endpoint uses a different name than the Model ID
   * **Context window**, **Max output tokens**, and whether it supports **Streaming**, **Tool calling**, and **Vision**
   * **Input price** and **Output price** (optional, \$ per 1M tokens). Merge never bills this price.
6. Choose whether to **Allow failover to Merge curated vendors** (see [Failover](#failover))
7. Choose **Add endpoint**

The endpoint is saved as a **Draft**. Drafts can't receive traffic until they pass a test and you activate them.

## Test and activate an endpoint

**Test** sends a few tiny requests to your endpoint and shows a result for each check. It doesn't change anything.

| Check                      | What it catches                                                                                          |
| -------------------------- | -------------------------------------------------------------------------------------------------------- |
| Endpoint reachable         | A wrong URL, a DNS or TLS problem, or a timeout                                                          |
| API key accepted           | A wrong or expired key (the endpoint answers 401 or 403)                                                 |
| OpenAI-compatible response | A reply that isn't in the chat completions shape                                                         |
| Token usage reported       | A reply with no token counts                                                                             |
| Model served               | A wrong **Upstream name**, checked for each model                                                        |
| Streaming works            | A server that ignores streaming or sends a broken stream, checked for each model that declares streaming |

**Activate** runs the same checks and turns the endpoint on only if all of them pass. Otherwise it stays a draft and the results show what to fix. If someone edits the endpoint while the checks run, activation stops, so what goes live is always what passed.

**Deactivate** stops traffic to an active endpoint right away without deleting it. You can activate it again later.

Edits to an active endpoint apply immediately, without a new test, so you can rotate a key with no downtime: edit the endpoint and enter the new key (leave the field blank to keep the current one). Run **Test** after changing the URL or key.

## Call a self-hosted model

Call the model by the **Model ID** you declared, exactly as you entered it:

**`cURL`**

```bash title="cURL"
curl https://api-gateway.merge.dev/v1/responses \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b-instruct",
    "input": [
      { "type": "message", "role": "user", "content": "Summarize this ticket in one sentence." }
    ]
  }'
```

**`Python`**

```python title="Python"
from openai import OpenAI

client = OpenAI(base_url="https://api-gateway.merge.dev/v1/openai", api_key="YOUR_API_KEY")

response = client.chat.completions.create(
    model="llama-3.3-70b-instruct",
    messages=[{"role": "user", "content": "Summarize this ticket in one sentence."}],
)
print(response.choices[0].message.content)
```

Your self-hosted models also appear under **Self-hosted** in the model picker, and in `GET /v1/models` for your organization.

Model IDs can be a plain name (`llama-3.3-70b-instruct`) or include a prefix (`acme/llama-3.3-70b-instruct`). If you declare an ID that's also in Merge's catalog, your endpoint takes precedence: calls from your organization that name it always go to **your** endpoint, whatever the curated vendors charge.

## Use self-hosted models in routing policies

Add self-hosted models to a [Priority policy](/merge-gateway/routing/deterministic-strategies) from the **Self-hosted** group in the model picker, next to curated models. Gateway tries models in the order you list them and moves on to the next entry if your endpoint fails.

## Failover

**Allow failover to Merge curated vendors** (on by default) controls what happens when your endpoint fails:

* **On**: in a Priority policy, Gateway tries the next model in the list, including another self-hosted endpoint. And if your Model ID is also in Merge's catalog, a request your endpoint can't serve (server error, timeout, or unreachable) can go to a curated vendor for that model, billed at its normal rate.
* **Off**: Gateway returns your endpoint's error. Prompts never leave your infrastructure.

A rejected API key is never sent to a curated vendor for the same model. You get the error back, so a bad key shows up right away.

## Limits

* **Test** doesn't check tool calling or vision. Gateway trusts what you declare.
* Slugs can't be changed. To rename an endpoint, delete it and add it again.
* Apps and policies that use an old Model ID stop reaching the model if you change it
* Up to 50 models per endpoint
* If two endpoints declare the same Model ID, Gateway always uses the same one, and you can't pick which

---

## Next steps

#### [Priority policies](/merge-gateway/routing/deterministic-strategies)

Order your models and fail over down the list

#### [Bring your own keys](/merge-gateway/capabilities/byok/overview)

Use your own provider accounts for curated vendors