What an AI gateway is

The change is smaller than it sounds. A request keeps its format and changes its destination.

Where a gateway sits

Every model request is sent to an address. Provider tools default to the provider’s own address and let you change it, and that is the whole mechanism a gateway runs on. Point the address at the gateway, and requests arrive there first and carry on to the provider from there. Cloudflare puts the cost of switching at one line of code.

The request itself doesn’t change. An app still sends a conversation and the name of a model, and still gets back a reply with a token count. What changes is that the traffic now passes through a place you decide the rules for, instead of going straight to whichever provider it was written against.

That’s also what makes the arrangement worth having. Kong describes the effect as client applications staying decoupled from provider-specific APIs, with credentials handled centrally and policy applied at the gateway layer rather than inside every application.

What the layer does

Gateway products differ in the details, and the same handful of jobs shows up across all of them.

Whose keys pay for the tokens

Feature lists don’t answer the question that changes the most: whose provider account pays for the tokens. Gateways divide into two arrangements, and some support both.

On your own keys, the credential gets to the provider one of two ways. It can travel with each request, where your app sends its own provider key alongside a separate key identifying it to the gateway, and the gateway forwards the credential without ever holding it. That is also how an existing subscription keeps working, since a tool with its own paid plan carries its own authentication straight through. Or the gateway stores the key once and callers present only a gateway key after that, which is the variant Cloudflare labels BYOK. Some vendors use the term for that variant alone, so it’s worth checking which one a product means.

Neither arrangement is the advanced option. A team with negotiated rates and a signed data agreement has reason to keep its own keys. A team that wants to compare four models this week has reason not to bother. OpenRouter supports both, and so do others.

Things it gets confused with

Three neighbours get described in similar words. A model provider is the thing at the far end: the gateway forwards to it and never runs a model itself, so nothing about what the model can do changes on the way. A client library does some of the same work, but inside one application and in one language, where a gateway is a separate hop whose rules cover everything pointed at it. And an API gateway is the same family and often the same vendors, with Kong’s AI gateway built directly on its API gateway, adding the model-aware parts: token and cost accounting, caching that recognizes similar prompts, and rules that read what a prompt says.

Why not just use one provider directly?

Going direct to one provider is the simplest thing that works, and for a while it comes with more than the model. The lab hands you an operations department along with it: screening you never see, spend limits, a dashboard, a record of what happened. None of that is wrong, and it’s why the direct call feels like enough.

Then you add a second provider and find out how much of it was theirs. The record of what your agents did now lives in two consoles and nothing joins them. The cap you set covers half your traffic. Ask which team spent the money, or what one agent did on Tuesday, and the answer depends on which dashboard you open. Deploy an open model on your own hardware and there’s no dashboard at all. It’s your name on the deployment.

A gateway is where those things live when no single vendor owns them. Costs and records for every model land in one view. Screening and limits apply at the layer, so they cover the model you added last week and the one running on the machine under your desk. Keys are issued per person, per project, per agent, each with its own ceiling and its own trail.

It also changes what picking a model costs you. When a frontier model, a cheap one, and an open-weight one all arrive through the same interface, choosing one per task is a configuration decision rather than a migration, and changing your mind later doesn’t reach into the applications.

And it outlasts the tools. Coding agents and assistants churn every few months. When the rules and the history sit in front of the tool instead of inside it, replacing one costs you nothing.

None of this needs a large team. One person running a coding agent against a private codebase already has a screening problem and a record problem, and neither gets smaller when they switch agents next quarter.

Common questions