Treating VRAM, RAM and NVMe as one memory hierarchy
Experiments with direct storage, learned expert locality and predictive prefetch turned three separate resources into one managed path.
The storage experiments asked whether routed experts could move through graphics memory, system memory and fast storage without making correctness depend on where a weight happened to live.
Method
- Build a batched direct-storage path with an explicit cache.
- Measure route locality at equivalent cache capacity.
- Add prediction only behind a safety mechanism that could fall back to demand loading.
- Compare every treatment with a control that used the same inputs and output checks.
Findings
The routes showed enough locality for a learned cache to be useful. Prefetch could overlap some waiting when its predictions were timely, but prediction was not reliable enough to become a correctness dependency.
Decision
These experiments validated the mechanism, not a universal performance claim. The result depends on workload locality and the relative behaviour of storage, memory and compute.
Cite this page: Dingal AI Research. “Treating VRAM, RAM and NVMe as one memory hierarchy”. 25 July 2026.
Related
ShiftWing: an inference engine built from scratch
Why ShiftWing manages graphics memory, system memory and fast storage as one runtime instead of treating local inference as a smaller copy of server inference.
Why frontier AI has to run on the hardware people own
Frontier capability matters most when regular people can use it without surrendering their work, context or control to a remote service.
Pushing experts below four bits
Low-bit expert sidecars reduced storage pressure, but the quality study made clear that this is a trade to qualify rather than a free saving.