Evals
2 posts
The eval graded the answer key, not the document
An extraction eval stayed green for weeks while users watched their data get mangled. The cause: it graded output against our labels, not against the source.
The eval was grading a prompt we had deleted
Our record/replay eval could not run for two days and reported green throughout, because the scorer kept grading cached results from a prompt we had deleted.
