We have an eval for an LLM extraction pipeline. It runs offline: the model's responses were recorded once and are replayed from disk, so the eval costs nothing and gives the same answer every time. It scores those responses against hand-written expectations and reports pass or fail.
It was green. It had been green for two days. During those two days, it had not been able to run at all.
Not "run and pass." Not "run and skip." The step that produces the results was hard-erroring every single time, and the step that grades them was reading two-day-old files off the disk and grading those. Two halves of one pipeline, each behaving reasonably, adding up to a confident green light attached to nothing.
This is a cache-invalidation bug wearing a testing costume, and I think it's lying in wait in a lot of AI test suites right now.
Record and replay, and why you want it
If you test anything that calls a language model, you eventually hit the same three problems. The calls cost money, so a full run is a line item. They're slow, so nobody runs them locally. And they're non-deterministic, so a failure might be a real regression or might be the model having a different morning.
The standard fix is record/replay. You make the real call once, capture the HTTP exchange to a file, and from then on the test suite replays that file instead of hitting the network. Every ecosystem has a library for this, and the recorded files are usually called cassettes, or fixtures, or snapshots. The result is a test suite that runs offline, in milliseconds, for free, with identical results on every machine.
It's a genuinely good pattern. We use it deliberately. Nothing that follows is an argument against it.
But it has a property that's easy to state and easy to forget: a recording is an answer to one specific question. Not an answer in general. An answer to that exact request.
Exact means exact
To replay a recording, the library has to decide whether the request you're making now is "the same" request it recorded earlier. Ours matches on method, URL, and body, and the check on the body is plain string equality.
Now think about what's in the body of a request to a language model. The model name. The parameters. The tool or function schema. And the prompts (system and user), in full, character for character.
Which means the prompt is part of the cache key. Edit one word of your system prompt and every recording you have ever made becomes an answer to a question nobody is asking anymore. Not degraded. Not approximately right. Simply not a match.
Our recordings were made on a Monday. On the Thursday we shipped a fix to that same pipeline, and the fix, of course, was a prompt change. We had found the model padding year-only dates into fake months, so we rewrote the rule to say the opposite of what it had said before. We also extended the tool schema while we were in there.
The recordings were invalidated by the fix to the bug the eval existed to catch. That's not irony for its own sake; it's the structural point. The moments when you most need the eval are exactly the moments you change the prompt, which is exactly what invalidates the recordings. The tool goes blind precisely when you reach for it.
The two silences
A bug like this needs two things to go unnoticed, and we had both.
The first silence is an unhelpful error. When replay can't find a match, the library raises an "unhandled request" error. That error names the URL it tried to reach. It does not, and cannot, say "your prompt changed." From inside the library, all that happened is that an HTTP request arrived that no recording matched. It has no idea why. So the operator sees a stack trace about an API endpoint and reasonably concludes something is wrong with the network, the key, or the config, rather than with a file that is exactly as valid as it was yesterday.
The second silence is a stale artifact downstream. Our pipeline has two steps. Step one replays the recordings and writes the model's outputs to a directory. Step two reads that directory and scores it. That split is good design: it lets the scoring live in a different language, and it lets you re-score without re-running.
But the intermediate directory is generated output, so it's gitignored, so nobody thinks about it. When step one started erroring, it wrote nothing, and crucially it also deleted nothing. The previous run's outputs were still sitting there. Step two found a full set of files, scored them, and reported a clean pass.
Neither half was wrong. Step one failed loudly and honestly. Step two graded exactly what it was given. The green light came from the seam between them, where the failure of one was invisible to the other.
That's the general shape worth remembering: a cached artifact outliving its producer will make a downstream check report on a world that no longer exists. Test suites are full of these seams.
We had already written down the rule
Here's the part I find most instructive, and the least flattering.
Our documentation had a section titled, more or less, "what invalidates a recording." It listed the causes correctly and completely: editing the fixture text, editing the system prompt, editing the prompt template, editing the tool schema, changing the model. It was accurate. It had been accurate for months. One of us wrote it.
And it did nothing, because documentation is not a control. A rule that lives in prose depends on a human recalling it at the moment of action. The person editing a prompt is thinking about the prompt. They are not thinking about a caching property of the test harness, written down in a file they read once.
The lesson isn't "write better docs." It's that when you find yourself documenting an invariant that a machine could check, the documentation is a placeholder for the check you haven't built yet. Knowing the rule and enforcing the rule are different artifacts, and only one of them survives a distracted Thursday.
The fix: make the artifact prove it's still current
The check turned out to be small, which is usually the case with this class of bug.
Before replaying anything, read each recording, pull out the request that was recorded, and compare it to the request you would send right now: model, system prompt, user prompt, tool schema. If they differ, stop before doing any work and say so in words:
2 recording(s) cannot be replayed:
- sample_01 — system, user, tools changed since it was recorded
- sample_02 — system, user, tools changed since it was recorded
Then print the exact command to re-record, with a plain warning that it makes real, billable calls.
That's the whole idea: turn "mysteriously broken, downstream reports green" into "stale, here's why, here's the fix." The check is deliberately skipped while recording, since that run is what refreshes them.
Three details from building it that generalise.
Name the drifted part, not just the fact of drift. "Stale" sends someone hunting. "System prompt and tool schema changed" tells them what happened and often why. It costs one extra comparison.
Verify the checker before you trust the checker. Ours had a bug on its first outing, and it is worth describing because the bug is inherent to the format. A recording file can hold several interactions: re-recording appends rather than truncating, so a superseded request and the current one sit in the same file, and the replay library matches whichever one fits, at any position. My check read only the first. Pointed at our other eval suite, it confidently reported 6 recordings as stale. They were all fine, each with a matching interaction one slot further down.
Notice the shape of that failure. A tool built to stop people trusting a stale artifact was itself about to send someone to re-record 24 healthy files, at real cost, to fix nothing. A checker is an instrument too, and it gets exactly one free pass at being believed. Before acting on its first output, I diffed one flagged case and one unflagged case by hand, which is the only reason the bug surfaced before the bill did.
The habit worth stealing: the first run of a new check is a test of the check, not of the system. Investigate a positive and a negative by hand before you let anyone act on the results.
Be honest about what it doesn't cover. Our check compares the fields we control. The replay library still matches on the entire body, so drift originating somewhere we do not set (a client library changing its serialisation, say) would still surface as the original unhelpful error. A green result from the checker means "nothing we control has changed," not "replay is guaranteed." Writing that limitation into the code comment is what stops the next person from trusting it further than it deserves.
What to go and check
If you have an eval or a test suite that replays recorded model calls, three questions are worth an afternoon.
Does anything verify that your recordings still match the requests you'd send? If the answer is "we're careful," it's no. Careful is not a mechanism.
If the generating step fails, what does the scoring step do? Follow the artifact. If it's cached, gitignored, or written to a shared directory, find out what happens when it is stale. A scorer that finds a complete set of yesterday's files and grades them happily is the exact failure described here. Make a missing or stale input a hard failure, never a quiet skip.
Does your green have a timestamp? Not "did the assertions pass" but "when did the thing being asserted about last actually execute." A pass that can be produced without running the system under test is not a pass; it's a memory of one.
The uncomfortable summary is that nothing here was broken. The replay library correctly refused to serve a mismatched recording. The scorer correctly graded the files it was handed. The documentation correctly described the invalidation rule. Every component behaved exactly as designed, and the system as a whole told us confidently that a pipeline we had blinded two days earlier was working fine.
An instrument that reads healthy because it's no longer connected to anything is worse than no instrument at all. No instrument makes you cautious. A green one makes you ship.