How to reduce what you spend on AI models

The order matters more than the list. Teams that start by switching models usually end up with a cheaper bill and a worse product.

Measure before you change anything

You cannot cut what you cannot attribute, and model spend hides well: one API key, one invoice, twenty features.

1. Get per-request records, not a monthly total. Providers report token usage on every response, broken down far enough to be useful: cache reads and writes are reported separately, and reasoning tokens are reported apart from the visible reply. That breakdown is the raw material for everything below.
2. Attribute each request to something you care about. A feature, a customer, a team, an agent. Spend that can only be described as “the AI bill” cannot be reduced deliberately, only across the board.
3. Find the top three lines. Cost concentrates. It is normal for one agent loop or one background job to be most of the bill, and equally normal for the feature everyone talks about to be a rounding error.
4. Write down the cost per unit of work. Per ticket, per document, per session. It is the only number that stays meaningful as volume changes, and the only one that tells you whether a change worked.
5. Only then pick a lever. Preferably the top one, from the top line.

Skipping this step is how teams spend a week saving 5% on something that was 4% of the bill.

The levers in order

The order is deliberate. The first four cost you nothing in answer quality. The last four trade something away, and the last one trades the most.

The two that need no product change

Caching is the largest single lever in most systems. It means two different things, and the difference decides how much you save.

Provider prompt caching

It re-prices the part of your input that repeats. The request still goes to the provider and you still pay for it, but the repeated prefix is charged at a tenth of the input rate at both Anthropic and OpenAI. The break-even is almost immediate: Anthropic states that caching pays for itself after one read on its five-minute duration, or two on its one-hour one. Its own worked example moves 40,000 of 50,000 input tokens onto cache reads and takes a session from $0.705 to $0.525 without changing the work done.

Getting hits takes effort. Cache matching is on an exact prefix, so OpenAI’s guidance is to place static content like instructions and examples at the beginning and variable content at the end, and it notes that images and tool definitions must be identical between requests too. There are floors: OpenAI caches automatically above 1,024 tokens, Google’s per-model minimums for context caching are 2,048 or 4,096. And there is a routing dimension most teams never look at, since OpenAI documents a cache key to improve hit rates, a rough ceiling of 15 requests a minute per key, and a 30-minute default lifetime.

The failure mode is subtle. A timestamp, a session id, or a shuffled tool list near the top of a prompt can quietly cost you every hit you thought you had.

Response caching

It works in front of the provider instead of inside it. When the same request comes in twice, the second one is answered from the cache and never reaches the model, so there are no input tokens and no output tokens to pay for. Cloudflare serves cached requests “directly from Cloudflare’s cache instead of the original model provider”. Gate does the same, and marks each answer so you can see which ones were repeats. It also sets up the provider-side caching described above on your behalf, so you get the cheaper rate without restructuring your prompts to earn it.

The two are worth separating when you estimate a saving. Prompt caching takes the repeated part of an input to a tenth of its price, and it applies to almost any system with a fixed instruction block. A response-cache hit costs nothing at all, and it applies only where the identical request recurs. Support answers, classification of a bounded set of inputs, and anything triggered repeatedly by automation are where the second kind pays; open-ended chat is where it does not.

The ones that change your product

Send less history. Every turn re-bills the entire conversation, so long sessions get expensive in a way that compounds. Trimming or summarising older turns cuts that, and it costs the model context it might have used. The judgement is which parts of an exchange still matter twenty turns later, which is usually fewer than expected.

Compress the prompt. Compression cuts the token count before the request leaves, and it comes in two kinds that are easy to confuse.

Lossy compression rewrites the text to hit a target and accepts that some of it goes. Kong’s plugin takes either a ratio or a token count, for example 80% of the original length or a fixed 150 tokens. Push the target hard enough and you can cut something the model needed, so this kind wants testing against your own cases.

Lossless compression rewrites the request without dropping anything. Where Kong’s plugin trims text to reach a target, Gate rewrites text-bearing fields “losslessly … with safe transforms before forwarding” and never alters tool schemas, so the model “sees the same meaning” while the request carries fewer tokens. The saving is smaller than an aggressive lossy ratio, and it asks nothing of you in return.

Right-size the model. The rate difference between tiers is several-fold, which makes this the biggest headline saving and the one most likely to be regretted. Do it with an eval set rather than a hunch, and do it per task rather than across the board. That decision has its own method.

Turn reasoning down where it isn’t earning. Thinking tokens are billed at the output rate, and plenty of tasks do not need them. Classification and extraction rarely do. Multi-step debugging does.

Fix the loops. Retries bill twice. An agent that wanders bills for every step, and each step carries the whole conversation, so a step budget is a cost control as much as a safety one.

What doesn’t work

• Shortening your prompts by hand. Input is the cheap rate. Removing 200 tokens of instructions from a request that generates 2,000 tokens of output changes almost nothing, and the instructions were probably doing something.
• Chasing the cheapest provider per token. A model that needs three attempts is not cheap, and a migration costs more engineering time than the spread usually returns.
• Turning off logging to save money. Records cost a fraction of tokens and are the only reason you can attribute anything. This is the one economy that guarantees the next round of cuts will be guesswork.
• A one-off audit. Prices change on the vendor’s schedule, including expiring introductory rates, and usage changes on yours. A cost review with no repeat date is a snapshot of a bill you no longer have.

Common questions