Requests and tokens · Explainer

What happens when an app calls an AI model

For operatorsFor developers 6 min read · Updated Jul 2026

On this page
  1. The round trip
  2. What you send
  3. Tokens: what you pay for
  4. The context window
  5. How the reply is produced
  6. Common questions
  7. Where Gate fits

The short answer

When an app “calls an LLM,” it sends the prompt over the internet to a model provider. The provider breaks the text into tokens, the model reads them and writes new tokens back, and the reply returns, either all at once or a piece at a time. Along with the reply comes a count of the tokens it used, and that count is what you pay for.

The same trip happens every time, whether it’s a chatbot, a coding tool, or an internal assistant answering a question.

The round trip

A request to a language model follows the same path every time.

Your app keeps the history Provider splits into tokens Model no memory the conversation tokens the reply, and the token counts you are billed for
Your app keeps the history the conversation Provider splits into tokens tokens Model no memory the reply, and the tokens
Fig. 1
One round trip. The app sends the whole conversation every turn, because the model keeps none of it.
  1. An app puts together a request: the conversation so far, which model should answer, and how long the reply can run.
  2. It sends that request over the internet to the model provider.
  3. The provider breaks the text into tokens, the units the model reads.
  4. The model reads those tokens and writes new ones, until it finishes or reaches the length limit.
  5. The reply travels back to the app, all at once or as it’s written.
  6. The provider counts the tokens the request used. That count is the bill.

What you send

A request has three parts, and none of them are complicated.

  • The conversation. Not only the latest message but the whole exchange so far, each turn labeled with who said it: the user, the model, or a system instruction that sets the ground rules.
  • The model. Which model you want to answer, named in the request.
  • A length limit. A cap on how long the reply can run, so one answer can’t go on forever.

The conversation doesn’t have to be only text. Many models also take images, PDFs, or screenshots in a message, and that content is counted in tokens too, the same as words.

The model keeps no memory of its own. It sees only what’s in the request in front of it. A chat feels continuous because the whole conversation is sent to it again every turn, whether the app holds that history or the provider holds it for you. The memory lives around the model, not inside it.

Tokens: what you pay for

The model doesn’t read letters or words. Before it sees anything, the text is broken into tokens: short chunks, often pieces of words. “Tokenization” might be a single token, while a rare word or an odd string might take several.

Two things follow from that.

  • You pay by the token, not the word. The provider counts the tokens going in and the tokens coming back, and bills on the total.
  • Token counts aren’t word counts. A token runs a little shorter than a word on average, but it moves around with the language and the content. The only exact number is the one the provider reports back.

The context window

A model can hold only so many tokens in mind at once. That limit is its context window, and it covers both sides: what you send and what the model writes draw on the same budget. A long prompt leaves less room for a long answer.

The size depends on the model. As of mid-2026 a 1,000,000-token window is standard on the frontier models, very roughly 750,000 words, while plenty of models in everyday use hold 128,000 to 256,000. A bigger window lets the model take in more at once, but accuracy and recall slip as the window fills, so the headline number is never the amount a model handles reliably.

Go past the limit and something has to give. Before the model can continue, the conversation has to be trimmed, summarized, or split into smaller requests. Send more than the window holds in one go and the request is simply refused. Deciding what stays in that window, and what gets left out, is one of the real skills of working with these models.

How the reply is produced

A model doesn’t find an answer and hand it back. It writes the reply one token at a time, each one predicted from everything that came before. That single fact explains a lot of how these systems behave.

  • The reply arrives as it’s written. Because it comes out piece by piece, an app can stream it to you a few words at a time, the typewriter effect in every chat window. It also means a longer answer takes longer, since the words arrive one at a time.
  • The same question can get different answers. Choosing each token involves a bit of chance, so asking twice can produce two different replies, both reasonable. It’s not a malfunction, and it’s why a model won’t reliably return the same thing every time the way a database would.
  • It can be confidently wrong. The model aims for a plausible next token, not a checked fact. Usually that lands. Sometimes it produces something fluent and untrue, stated just as surely as everything else. Treat an answer as a fast draft to verify, not a record to trust.

Common questions

What is sent in an LLM API request?+
The conversation or task, the name of the model that should answer, and a limit on how long the reply can run. Depending on the model, it can also carry images, PDFs, or screenshots for the model to read.
Does the model remember earlier messages?+
No. It reads only what’s in the request in front of it. Multi-turn chat works because the conversation is sent again each turn, held either by the app or by the provider on the app’s behalf.
Why do I pay for the prompt and not just the answer?+
The model has to read the whole prompt before it can write anything, so those tokens count too. That holds even when the provider stores the conversation for you: the earlier turns are still billed as input on every turn, which is why a long chat gets more expensive as it goes.
What happens if a request is longer than the context window?+
It gets refused. Providers return an error when the input alone is bigger than the window, and a reply that runs into the limit mid-sentence stops there. Staying under it means trimming older messages, summarizing them, or splitting the work into smaller requests.
Is streaming faster or cheaper?+
Neither, strictly. You see the first words much sooner, which is why it feels faster, but the model takes the same time to finish and the token cost is identical.
Why can the same request get different answers?+
Because each token is chosen from a range of likely next ones rather than looked up in a record. Two runs of the same prompt can both be reasonable and still read differently. Turning the randomness setting down narrows the spread without guaranteeing an identical reply.
Can repeated prompts cost less?+
Sometimes. When the same content shows up again, some providers and gateways can reuse the earlier work instead of processing it from scratch. How much that saves depends on the setup, and it’s covered in what a request costs.
Can the model do things, not just write text?+
Yes. A model can stop partway through and ask the app to run a tool, look something up or call another system, then carry on once it has the result. That loop is most of how agents work, and it’s a topic of its own: see how tool calling works.
Can I switch to a different model or provider later?+
Usually, yes. Many models accept requests in a similar shape, so an app can be pointed at a different one without much rewriting. Doing it smoothly across several is its own topic: see what is an AI gateway.

Keep learning

Cost What a request to an AI model costs Agents and tools What an AI agent is Infrastructure What an AI gateway is
Back to top