The evaluation harness is now public
The code I use to run and record my own evaluations, released so the results here can be reproduced.
This is the harness behind the results published on this site. It is released because a method described in prose is not really reproducible, and because I would rather people find my mistakes than not.
What is in it
- The task definitions, with a note on where each item came from.
- A runner that records model version, sampling parameters and tool access for every trial.
- A results format that makes the exclusion count a required field rather than a footnote.
- The raw run records behind the published results.
It is ordinary research code. It works, it is documented enough to run, and it is not a product. Expect rough edges and tell me about them.
Cite this page: Dingal AI Research. “The evaluation harness is now public”. 11 June 2026.
Related
A minimum reporting standard for AI evaluations
A short proposal for the least an evaluation result has to disclose before it can be compared with another result or repeated by anyone else.
Failure compounds faster than per-step accuracy suggests
Measuring how language-model agents fail over long tasks. Per-step accuracy barely separates the systems; end-to-end success separates them a lot.
Why this exists
What this practice is for, what it will publish, and the things it will not do.