Overview

Gateway gives your application one API for model inference, with model selection, routing, budgets, and data controls in the request path. Use it to change models without changing your client integration, or to apply consistent routing and spend rules across your agent traffic.

Use Gateway’s native Responses API or a compatible OpenAI, Anthropic, AI SDK, or LangChain endpoint. Most existing clients need a new base URL and API key. The endpoint matters because compatible APIs can use different request and streaming formats. See Get started or move an existing integration.

What happens to a request

1. Choose a route. Send a concrete provider/model ID to call that model, or a model alias to use a routing policy. Without either, Gateway checks the key, customer, project, and organization defaults in order. See policy resolution.
2. Apply controls. Gateway checks the key’s model access, budgets, vendor and region restrictions, zero data retention settings, and any configured security scans. A request blocked before a vendor call does not incur model inference cost.
3. Return and meter. The response identifies the model and vendor that served it. Supported surfaces include usage.cost in USD, and Gateway records usage against the request’s organization, project, or customer.

Scopes and credentials

An organization owns projects, keys, policies, and budgets. A project groups traffic for a team or environment. An Embedded Routing customer represents one of your end customers and can have its own policy, budget, controls, and credentials. See projects and Embedded Routing.

Use a model API key for inference. A mgmt_ management key is for the Management API, which administers Gateway resources and cannot call model endpoints. Project and customer API keys scope inference requests to those resources. See API keys.

Model and vendor names

Model IDs use provider/model, where the provider is the model’s direct creator and the vendor is the service that serves it. For example, anthropic/claude-sonnet-5-5 served by Amazon Bedrock has provider Anthropic and vendor Amazon Bedrock. Use a full model ID when a bare name could match more than one provider. Browse available models in the model catalog or with GET /v1/models.

Next steps