Llm
The eval graded the answer key, not the document
An extraction eval stayed green for weeks while users watched their data get mangled. The cause: it graded output against our labels, not against the source.
The eval was grading a prompt we had deleted
Our record/replay eval could not run for two days and reported green throughout, because the scorer kept grading cached results from a prompt we had deleted.
Treat model output like user input
When strangers write the prompt and the answer renders on your page, you have two untrusted boundaries back to back. The layered posture we settled on.
Three Ways VCR Cassette Recording Lied to Me
Three vcr recording behaviors that never raise an error: the config beats the CLI flag, ambient traffic gets taped, and re-records append instead of replacing.
Cross-provider LLM failover without losing observability
Every LLM provider goes down eventually, so the obvious move is to retry against another one. The naive rescue-and-retry works until your bill spikes and the logs can't tell you how often failover fired or what it cost. Here's the version that keeps the receipts.
