A year of interpretability work, and what held up
Which methods for looking inside a model produced findings I would still defend, and which did not survive being checked.
Interpretability asks what is happening inside a model when it produces an output. This is a plain-language account of a year spent on it, including the part that did not work.
What held up
Methods that identify recurring internal features were useful for diagnosis. When a system behaved oddly on some class of inputs, I could often find an internal feature active in exactly those cases and confirm the link by intervening on it directly. The intervention is the part that makes it evidence.
What did not
Attribution methods that score how much each part of an input mattered were much weaker. Two methods often disagreed about the same input, and on test cases where I knew the answer in advance both were wrong often enough that I stopped treating either as evidence on its own.
Still open
- Whether features found in one model mean anything in another.
- How to tell a real internal mechanism from an artefact of the analysis method.
- Whether any of this is solid enough to support a decision about deploying something.
Cite this page: Dingal AI Research. “A year of interpretability work, and what held up”. 15 January 2026.