A minimum reporting standard for AI evaluations

I read a lot of published evaluation results. Most of them cannot be compared with each other, because they do not say enough about how they were produced. This note sets out the minimum I think a result has to disclose.

This is a proposal, not a standard anybody has adopted. I follow it in my own work, which is the only claim I can make for it.

The minimum

1. The question. What the evaluation was meant to establish, written down before the runs happened.
2. The conditions. Model version, sampling parameters, prompts and tool access — enough for someone else to repeat it.
3. The trials. How many runs per task, and the spread across them, not just the mean.
4. The exclusions. Every run left out of the reported figure, and why.

None of this is demanding. It is roughly what a lab notebook would contain anyway. The reason it matters is that a number without it cannot be checked, and an unchecked number is an assertion.

Why the trial count matters most

Of the four, the one most often missing is the trial count, and it is the one that changes conclusions. Run-to-run variance on the tasks I work with is frequently larger than the gap between the systems being compared. A single run per task can reverse a ranking, and a result reported without its trial count gives the reader no way to know whether that happened.

Cite this page: Dingal AI Research. “A minimum reporting standard for AI evaluations”. 18 August 2026.