Context compression
Context compression shrinks a request before Gateway sends it to the model, with no application change. Turn it on when long conversations approach a model’s context window or input tokens drive your cost.
How it works
When a request’s token count exceeds the mode’s threshold, Gateway applies two steps in order, stopping once it fits:
- Lossless compaction re-encodes JSON in tool schemas, tool call arguments, and tool results more compactly, without changing content
- Message trimming removes messages from the middle of the conversation, always keeping system messages and the start and end
A request that still doesn’t fit (say, system messages alone fill the window) is sent anyway. Unrecognized models skip compression.
Modes
Thresholds apply to the model’s effective input limit: its context window minus 15% reserved for output.
For a 128K-token model on Cost optimization at 70%, compression starts above about 76K tokens (128K × 0.85 × 0.7).
Configuration
Set the organization default under Configure → Request settings → Defaults in the dashboard, and override it on a project’s Configuration tab. A project override wins, so a project set to Disabled doesn’t compress even when the organization does. With neither set, requests aren’t compressed.