Why models get things wrong | Constellation Gate AI

The short answer

A model writes its answer rather than looking it up, so a wrong answer is a normal output and not a malfunction. The failures come in kinds that look alike and have different causes: invented facts, knowledge that stops at a date, accuracy that drifts as a document gets longer, agreement with whatever you asserted, and plain run-to-run variation. Each has a different fix, and none of the fixes are complete.

Fluent and wrong is the combination that causes the damage.

The different ways models fail

“Hallucination” gets used for all of it, which is why the fixes get applied to the wrong problem.

Invented factsThe model produces something plausible and untrue. Anthropic describes it as text that is factually incorrect or inconsistent with the given context, and concedes it affects the most advanced models. NIST prefers “confabulation”: confidently stated but erroneous content.

Stale knowledgeTraining data ends on a date and the model has nothing after it, though it may answer anyway. Anthropic publishes two dates per model, and for some models they are months apart.

Drift in long inputRecall gets worse as the context fills. Not a cliff, a slope.

AgreementThe model follows your stated view rather than contradicting it, and does it more convincingly than it disagrees.

Mechanical tasksCounting, character-level work, and arithmetic go wrong at rates that feel absurd next to the model’s fluency, because the model consumes sub-word tokens rather than characters and is generating text rather than computing.

Three of those are about knowledge, one is about the shape of the input, and one is about the shape of the task. Treating them as one thing is how teams end up buying a bigger context window to fix an arithmetic problem.

Fig. 1Anthropic and NIST, Aug 2026

Five kinds, three causes, three different fixes. The word "hallucination" covers all five, which is why the wrong fix gets applied.

Why wrong answers sound confident

The way models are graded rewards it.

NIST puts the mechanism plainly: confabulations are a natural result of how these systems are designed, producing outputs that approximate the statistical distribution of their training data, which can be accurate and can equally be inaccurate. That explains why error is possible. It doesn’t explain why the model doesn’t simply say it doesn’t know.

A paper from OpenAI researchers, Why Language Models Hallucinate by Kalai, Nachum, Vempala and Zhang, argues the reason is scoring. Under a binary right-or-wrong grade, a confident guess beats an admission of uncertainty, because a blank scores zero and a guess sometimes scores one. Their framing: humans learn the value of expressing uncertainty in the school of hard knocks, while models are evaluated on exams that penalise uncertainty, so they are “always in ‘test-taking’ mode”. Take two models, one that signals uncertainty honestly and one that always guesses, and the guesser wins on the benchmarks the field ranks by.

That predicts the texture of the errors, which is what makes it convincing. Wrong answers do not arrive hedged. They arrive specific: the paper notes that bluffs tend to be overconfident and precise, “September 30” rather than “Sometime in autumn”. And the confidence is not decoration. NIST identifies it as the mechanism of harm, since users act on false content precisely because it is stated confidently, and warns that the reasoning and the citations offered in support can be fabricated too. A model that shows its working can show working it did not do.

Why the same question gives different answers

Send an identical request twice and you can get two different answers, and this is not a setting you can turn off.

Anthropic states it in the API reference itself: even with temperature at zero, results will not be fully deterministic. Its glossary extends the claim past its own service, saying identical inputs may produce different outputs across calls, on first-party inference and through third-party clouds alike. Google documents a fixed random seed as a best effort rather than a guarantee.

The usual explanation, that GPUs do floating-point arithmetic in a nondeterministic order, turns out to be mostly wrong. Researchers at Thinking Machines Lab ran the check: repeating the same operation on the same GPU a thousand times gives bit-identical results. What varies is how many other people’s requests are being processed alongside yours, because the batch changes and the arithmetic is not invariant to batch size. It is a property of the serving system, not the silicon, and they note it affects CPUs and other accelerators too. Their measurement makes it concrete. A thousand samples of one prompt at temperature zero produced 80 distinct completions, identical for the first 102 tokens and diverging at the 103rd.

The operational consequence is larger than it sounds. If you need to know what a model produced last Tuesday, re-running the prompt is not how you find out.

What reduces the rate

Every one of these helps with something specific, and Anthropic’s own guidance carries the caveat that they reduce hallucinations without eliminating them.

	Fixes	Doesn’t fix
Let it say “I don’t know”	Anthropic lists explicit permission to admit uncertainty first, and says it can drastically reduce false information	The case where the model doesn’t know that it doesn’t know
Give it the source text	Invented facts about material you supplied, especially if you ask for word-for-word quotes before any reasoning	Anything outside the documents you gave it
Ask for citations, and retract without them	Verifiability. Anthropic’s guidance is to have each claim carry a supporting quote, and to drop claims where none can be found	The claim that is cited and still misread
Search or retrieval	Staleness, by reaching past the knowledge cutoff	Trust. Anything fetched is untrusted content, which is its own problem
Run code instead of computing	Arithmetic and data manipulation, which providers document as a job for a code tool rather than for the model’s head	Judgement about whether the calculation was the right one
Structured output	The shape of a response	The content. OpenAI’s own documentation notes that forcing adherence to a schema “can result in hallucinations if the input is completely unrelated to the schema”
Less context, better chosen	Long-input drift. Anthropic frames good context engineering as finding the smallest set of high-signal tokens	Nothing about accuracy on material that is in scope
An eval set from your own cases	Knowing your actual error rate rather than guessing at it	Errors your eval didn’t think to test. Anthropic’s own postmortem records evaluations that failed to capture a degradation users were reporting

How to work with it

None of the above adds up to a reliable model. It adds up to a system you can operate.

Decide what a wrong answer costs, per use. Drafting, summarising and brainstorming can absorb a bad output. Anything that reaches a customer or moves money cannot, and that difference should drive the model, the checks, and the review, not a general policy about AI.
Supply the facts rather than trusting recall. For anything current, specific, or internal, put it in the request. Recall is where invention happens.
Make claims checkable. Quotes and citations turn “is this true” from a research task into a lookup.
Give it an exit. A model permitted to abstain abstains sometimes. One that is not, guesses.
Keep a small eval set and grow it from real failures. Every incident is a test case, and OpenAI’s guidance is explicitly to expand the set as blind spots appear.
Record what was sent and what came back. Re-running does not reproduce it, which is why providers ship request and response logging as a product feature. Under the EU AI Act, Article 12 requires high-risk systems to technically allow the automatic recording of events over the system’s lifetime.
Keep a person on consequential decisions, and keep them awake. NIST names automation bias as a risk that makes confabulation worse, so a reviewer who rubber-stamps is a reviewer who adds latency and nothing else.

Common questions

Why does a model make things up rather than saying it doesn’t know?+

Because saying “I don’t know” scores zero on the way models are graded, and a guess scores sometimes. The OpenAI paper’s argument is that training and evaluation reward guessing over acknowledging uncertainty, and that a model which never expresses uncertainty outperforms an honest one under binary scoring. It also reports that calibration can get worse after post-training than it was before.

Can hallucination be fixed?+

Reduced, reliably. Eliminated, not by any documented technique. Anthropic’s guidance ends with the instruction to validate critical information regardless, which is the honest position.

If I set temperature to zero, do I get the same answer every time?+

No. Anthropic documents that results are not fully deterministic even at zero, Google calls its equivalent “mostly deterministic”, and a fixed seed is documented as best effort. The variation comes from how the serving system batches concurrent requests, so it is not something your settings control.

Does that make repeated runs a good hallucination test?+

Partly, and less cleanly than it looks. Running the same prompt several times and comparing answers is a technique Anthropic suggests, but some of the variation you’ll see is ordinary serving noise rather than a signal about truth. Treat disagreement as a prompt to check, not as a verdict.

Do larger context windows fix forgetting?+

They raise the ceiling and don’t fix the slope. Anthropic describes context rot directly: as the tokens in the window increase, the ability to accurately recall information from it decreases, across all models, as a gradient rather than a cliff. Google adds a caveat to its own long-context results, noting that the standard retrieval test uses a single target and that accuracy drops when several pieces of information are wanted at once.

Why does it agree with me even when I’m wrong?+

Because agreement is rewarded. Anthropic’s research found sycophancy across five state-of-the-art assistants and concluded it is a general behaviour of models trained on human preferences, since both people and preference models pick a convincing agreeable answer over a correct one a non-negligible share of the time. Asking a leading question is a good way to get the answer you led with.

Keep learning

Models How to choose a model for a task Security What prompt injection is Models Open-weight and closed models