Speculative decoding that changes no output
A speculative path was admitted by confidence, verified against the ordinary path and rejected whenever it could not preserve the exact result.
Speculative decoding is useful only if it accelerates work without quietly changing what the model produces. The experiments therefore treated exact agreement as a gate, not as a hopeful side effect.
Method
- Attribute misses and replay work before changing policy.
- Label proposed continuations across controlled workloads.
- Admit speculation only when confidence clears the frozen rule.
- Compare accepted output with the ordinary decoding path token by token.
Findings
Confidence separated some useful proposals from obvious waste, but acceptance alone did not remove the cost of verification. Profiling showed that the accepted path still contained work worth optimising.
Decision
The findings come from the controlled workloads used in the study. They do not establish that the same admission rule transfers unchanged to every model or task.
Cite this page: Dingal AI Research. “Speculative decoding that changes no output”. 25 July 2026.
Related
ShiftWing: an inference engine built from scratch
Why ShiftWing manages graphics memory, system memory and fast storage as one runtime instead of treating local inference as a smaller copy of server inference.
Why frontier AI has to run on the hardware people own
Frontier capability matters most when regular people can use it without surrendering their work, context or control to a remote service.
Treating VRAM, RAM and NVMe as one memory hierarchy
Experiments with direct storage, learned expert locality and predictive prefetch turned three separate resources into one managed path.