Testing
3 posts
The eval graded the answer key, not the document
An extraction eval stayed green for weeks while users watched their data get mangled. The cause: it graded output against our labels, not against the source.
The eval was grading a prompt we had deleted
Our record/replay eval could not run for two days and reported green throughout, because the scorer kept grading cached results from a prompt we had deleted.
Three Ways VCR Cassette Recording Lied to Me
Three vcr recording behaviors that never raise an error: the config beats the CLI flag, ambient traffic gets taped, and re-records append instead of replacing.
