Why this exists
What this practice is for, what it will publish, and the things it will not do.
I started Dingal AI Research to work on a specific problem: a great many claims are made about what AI systems can do, and very few of them are stated in a way that anyone outside the organisation making them can check.
This is one person working independently. The scale is small and I would rather be straightforward about that than dress it up. What it can do is run careful evaluations on a narrow set of questions and publish them completely.
What gets published
- The method, in enough detail that someone else could repeat it.
- The code and data, where I have the right to release them.
- The limitations, including the ones that undercut the finding.
- Results that contradict things I have previously said.
What does not happen here
- No ranking, scoring or endorsement of commercial products.
- No work funded on condition of a particular result or a particular publication date.
- No finished result withheld because it turned out to be awkward.
If I fail to keep to any of that, the right thing is to say so here, and I intend to.
Cite this page: Dingal AI Research. “Why this exists”. 2 September 2024.
Related
A minimum reporting standard for AI evaluations
A short proposal for the least an evaluation result has to disclose before it can be compared with another result or repeated by anyone else.
The evaluation harness is now public
The code I use to run and record my own evaluations, released so the results here can be reproduced.