We had an eval for a document-extraction pipeline. It fed the model some text, took back structured JSON, and scored the JSON against a hand-written expected answer. It was green. It had been green for weeks.
In those same weeks, users were opening the extracted result next to their original and telling us their data had been mangled. Both things were true at once: the eval said the extraction was good, and the extractions were bad. This post is about how that gap opens, because it is a very ordinary gap and it is probably open in your pipeline too.
This is a companion to an earlier post, Schema-valid and still wrong, about the individual bugs. That one was about what went wrong in the data. This one is about why the test that was supposed to catch it didn't, and what a test that would have caught it looks like.
The answer-key trap
The eval worked the way most extraction evals work. For each fixture we wrote down the correct output by hand, the expected object, and scored the model's output against it: did the name match, did the counts match, did the first date match. Field agreement. A percentage. A threshold. Green above the line.
Here is the quiet assumption baked into that design. The score measures how well the output agrees with our labels. Our labels encode the things we thought to check when we wrote the fixture. So the eval can only ever catch a failure we already anticipated well enough to write an expected value for.
Invention and loss live precisely in the gaps between labels.
Say the model returns a skill the document never mentioned. Your expected object lists the five skills you saw, and the output contains those five plus one invented sixth. Depending on how you score, a set that is a superset of the expected can look like a pass, or even a perfect recall. The label had nothing to say about the thing that wasn't supposed to be there. Or say a whole section of the document gets dropped, but it's a section your fixture didn't assert a count on. The labels you wrote are all still satisfied. The score is clean. The section is gone.
The failure mode is not that the labels are wrong. It is that an answer key can only grade the questions on it. A student can write a beautiful, fully-labelled, entirely fabricated answer to a question you didn't ask, and the key has no opinion about it.
The second axis nobody scored
The fix is to stop treating extraction quality as one number and start treating it as two, because there are two independent ways an extraction can be wrong and a single field-agreement score smears them together.
The first axis is accuracy: does the output match what we expected? That's the answer-key score, and it is genuinely useful. Keep it.
The second axis is fidelity: does the output match the source? And this one needs no answer key at all, because the source is the ground truth. You are no longer asking "did it produce my labels." You are asking two blunt questions directly against the input document:
- Is every value in the output actually present in the source? (This catches invention.)
- Is every substantive line in the source represented somewhere in the output? (This catches loss.)
Neither question needs a human to have written down the right answer in advance. The right answer is the document. That reframing is the whole trick, and it has two consequences that are much bigger than they first appear.
The unlock: you can grade data you're not allowed to read
Because a fidelity check compares output to source and needs no hand-labelled expected value, it also needs no model to run. It's deterministic. It's string matching and set membership, with some care around normalisation. That means it can run in CI, on every commit, in milliseconds, for free.
And here is the part that matters if you work anywhere near sensitive data. A check that never calls a model, and never needs a committed answer key, can run against real user documents that you are otherwise forbidden from touching. You cannot send them to a language model. You cannot commit them as fixtures. But you can run a deterministic audit over them locally and get a real quality number, because the audit produces no derivative you'd have to store and sends nothing anywhere.
Most eval discussion assumes your test set is data you're allowed to look at and hand over. In a lot of domains, resumes, medical records, financial statements, that assumption is false, and it's why those domains often have the thinnest evals. A source-grounded deterministic check is a way in. Your synthetic fixtures keep exercising the model-facing behaviour in CI; the deterministic fidelity audit gets to run on the real distribution, privately, where the actual failures live.
The eval graded the middle of the pipeline
There's a second blind spot, and it's structural. A typical extraction eval takes already-extracted text as its input and scores the model's raw output. Draw the pipeline out and you'll see it's scoring the middle and ignoring both ends.
Before the model, there's extraction: pulling text out of a PDF, an image, a scan. That stage has its own failure modes, and they're brutal. Words fused together because a font's spacing fell below a threshold. Columns read in the wrong order so a sidebar lands in the middle of a sentence. A symbol decoded as the wrong glyph. If the text handed to the model is already garbage, the model faithfully extracts garbage, and an eval that starts after extraction sees a well-formed answer to a corrupted question. Green.
After the model, there's mapping and storage and rendering. The model can return a perfectly correct value that a later stage quietly ruins. Our longest-lived bug was exactly this: the model returned the right thing, and a downstream step padded and stored and rendered it into something the source never said. The eval scored the model's output, which was correct, and never looked at what actually got persisted and shown. The bug lived for weeks in a stage no test was pointed at.
The lesson is to be honest about which stage boundary your eval sits on, and to add checks at the others. A deterministic quality score on the extracted text itself, before any model, catches the fused-words and wrong-column failures cheaply. A fidelity audit on the final persisted result, after mapping, catches the downstream mangling. The model-output eval in the middle is necessary and it is not sufficient, and calling it "the eval" hides the two stages it can't see.
Make it a gate, but know which signal earns that
A quality signal that's reported but not enforced gets admired and ignored. If fidelity is going to matter, it has to be able to fail a build. So we made invention violations cap the score into failure regardless of how well the output matched the answer key. A parse that nails every label and fabricates one value now fails, which is exactly the outcome we wanted.
But here's the nuance worth stealing, because it's where a naive version of this goes wrong. Not every fidelity signal deserves to gate.
Invention is unambiguous. A value that isn't in the source is either there or it isn't; the check is precise, and when it fires, something is genuinely wrong. Gate on it hard.
Loss is fuzzy. "Is every source line represented somewhere in the output" is a recall metric, and recall metrics have a false-positive tail. A faithful-but-concise summary legitimately drops some source words. A description the model tightened up, correctly, will look like partial loss to a coverage check. If you gate on that, you'll fail good output and teach everyone to ignore the gate, which is worse than not having it. So loss gets reported, loudly, in the failure notes where a human can see the trend, but it does not by itself fail a build. It's also usually double-counted: whatever soft score you already have on free-text fields is penalising the same drop.
The general rule: gate on the precise signal, surface the fuzzy one. A gate you trust is worth ten dashboards you skim.
Accuracy tells you it met your expectations. Fidelity tells you it met reality.
The uncomfortable thing about the weeks of green is that nothing was broken in the eval. It measured exactly what it was built to measure, correctly, the whole time. It compared the output to our answer key and the output agreed with our answer key. The answer key was just a smaller thing than the document.
If you extract structured data from documents with a model, write down the two questions and make sure something in your pipeline actually asks them. Is everything in the output present in the input. Is everything in the input represented in the output. Point one check at the raw extracted text before the model, and one at the final persisted result after it, not just at the model's output in the middle. And when you turn fidelity into a gate, gate on invention and merely report loss, because one of those signals is precise and the other one will lie to you.
An answer-key score tells you the output matched what you expected. It will never tell you the output matched what was actually there. Those are different questions, and only your users are reliably asking the second one.