It was never trained to be right.
A large language model is trained on one deceptively small task: given the text so far, predict the next token. Do that a trillion times and minimize the surprise.
Notice what that foundational objective is and isn’t. It rewards text that is plausible — that looks like the training data. It says nothing about text being true. Fluency and factuality are different targets, and the base objective only ever pointed at one of them. Later training — instruction tuning, human feedback — does try to bolt correctness and calibration on afterward. But that is a patch applied on top, and, as §03 shows, the way we then grade the model can quietly reward the guessing right back in.
So the model becomes a superb continuation engine. Ask it something it knows well and the most-plausible continuation happens to be correct. Ask it something it doesn’t — a person who never existed, a citation that was never written — and the machinery does not change. It still finds the most plausible next word. It just has nothing true to be plausible about.
A hallucination is not the model breaking. It is the model doing exactly its job, on a question that had no answer in its weights.
Its memory behaves like lossy compression.
A model doesn’t store the internet. Its parametric memory behaves, in important respects, like a compression of it — billions of facts folded into a fixed set of weights, and a fit that tight has to give detail up.
When you ask for a specific fact, the model isn’t looking it up in a table. It’s reconstructing it from overlapping patterns, the way you can be certain a word was on a page and still misquote it. The gist survives; the exact DOI, the middle initial, the year — those get interpolated into whatever is most plausible nearby. That reconstruction is confident by default, because nothing in the weights carries a little flag that says this part I actually remember versus this part I’m filling in.
It’s a model, not the whole story — and the precise version is sharper. The 2025 analysis ties the errors to the statistical structure of the facts themselves: a fact the model saw essentially once carries a lower bound on how often it will be gotten wrong. Some of this isn’t detail lost in compression; it is baked into what can be learned at all.
We grade them like exam-takers.
Here is the part that keeps hallucination alive even after the model could know better. It’s not just the training — it’s the test.
In 2025, researchers at OpenAI made a sharp argument: models keep guessing because our benchmarks reward guessing. Most evaluations grade like a multiple-choice exam — one point for a right answer, zero for anything else. Under that rule, “I don’t know” scores exactly the same as a wrong answer: zero. But a confident guess sometimes gets lucky.
So a model optimized to score well learns the exam-taker’s reflex: when unsure, guess anyway. Never leave a blank. We trained that habit in, one leaderboard at a time.
Ten questions it’s genuinely unsure about — a lucky guess lands maybe one in three. This is one sample run (3 of 10 land). Same model, two grading schemes: pick its strategy and watch who wins.
Standard benchmark
Confidence-aware
Then we polished it to sound sure.
The last coat is reinforcement learning from human feedback — the step that makes a model helpful and pleasant. Humans rate answers, and we nudge the model toward what they liked.
Preference optimization rewards what raters approve of — and that can be behaviour merely correlated with helpfulness rather than truth itself. Sycophancy is one documented example: models can learn to reproduce a user’s apparent preferred answer rather than contradict it, and in places more feedback made it worse. None of that touches whether the content is true. At best it leaves the wrong answers wearing a better suit — which is worse, because a confident, agreeable tone strips away the one cue you had that something was off.
A hallucination is an overflow with no base case.
Every process that could run forever needs one of two things, or it’s a bug: a fold that bounds it, or a base case that stops it.
Generation is such a process. Token after token, the model descends deeper into an answer. It isn’t that it can never stop — generation has ordinary syntactic stops: an end-of-text token, a length limit, a stop string. What it lacks by default is a reliable epistemic one — an if I don’t know: stop. And that gap isn’t lexical: the model has the words “I don’t know”; what’s missing is a dependable path from being uncertain to reaching for them. With no such stop, the descent doesn’t halt at the edge of knowledge. It overflows that edge and keeps emitting, wrapping the void in fluent, confident text:
Flyxion gives this a sharper form — and a better name. In a paper literally titled Hallucination Is Normal, the claim is geometric: meaningful states don’t fill the vast space of everything a model could say. They lie on a thin manifold of lawful structure — a low-dimensional surface inside an enormous ambient space that is otherwise noise. Coherent thought moves along that surface.
“Explanation is tangent; hallucination is normal.”
That one line is the whole theory, and its one condition — Flyxion’s No-Noise Prediction Principle — is that a coherent system takes every step with zero component in the normal direction. Which is the same base case, seen from above: stop at the edge of what you know and don’t drift off the manifold are one instruction in two languages. Abstention, calibration, a verify-gate — each is a way of keeping the next token tangent: on the structure, not off it.
Put that way, most of the honest fixes are the same fix from different angles: give the model a base case for not-knowing, and make its uncertainty legible enough to act on. Abstention is the stop rule. Calibration is knowing where the edge is. A verification gate is a second reader that halts a confident-but-wrong answer before it ships. Retrieval is refusing to answer from lossy memory when you could read the real page instead.
There’s a quieter way to say what goes wrong. Language hands the model “I know,” “I remember,” “I’m certain,” “I don’t know” — but the generative process has no internal states that cleanly answer to those words. The model isn’t secretly lying; its report and its epistemic condition have simply come apart.
None of the fixes demand a model that never errs — that model can’t exist. They ask for one invariant, humbler and completely achievable: that the output never claim more epistemic warrant than the process that produced it actually holds. A model whose report and whose knowledge are the same object — that says what it knows, marks what it’s reconstructing, and stops at the edge instead of overflowing past it.
What actually helps.
Not slogans — the levers that move the real rate, each one an angle on the same base case:
- 01Reward abstention. Grade so a confident error costs more than an honest “I don’t know.” Fix the scoreboard and the guessing reflex fades — the same lever demo 02 pulls.
- 02Calibrate the confidence. Make the model’s stated certainty match its real hit-rate, so “I’m 90% sure” is true 90% of the time. Then a threshold can actually gate.
- 03Ground it in retrieval. Let it read the source and cite it, instead of reconstructing the fact from lossy weights. A quote you can click beats a memory you can’t.
- 04Add a verification gate. A second pass — a checker, a tool, a test — that can halt or flag a confident answer before it reaches you. The stop rule, made external.
- 05Surface the uncertainty. Show the shaky parts as shaky. An honest “I’m not sure” is worth more than a beautiful sentence that happens to be false.
Hallucination shrinks; it doesn’t hit zero. The reachable goal isn’t an oracle. It’s a model that knows its own edges — and tells you where they are.