Up to 10X Faster KV Cache Restore: TurboQuant Meets FlashBlade

In the first TurboQuant post, I walked through the math behind Google Research’s KV cache compression algorithm. The short version: ~5X compression at 3-bit precision to the original FP16 data, no calibration required. If you want the mathematical breakdown, start there.

This post is about what happens when you put TurboQuant on top of an Everpure™ FlashBlade//S™ system. Because compression without the right storage stack underneath leaves most of the performance on the table.

Here’s the punchline up front:

Figure 1: Comparison of KV cache restore time using TQ 3-bit + GDS + FlashBlade vs. FP16 standard I/O.

Those are average restore times across eight concurrent GPUs, measured with O_DIRECT to bypass the OS page cache and isolate raw FlashBlade® performance. In a production inference engine like vLLM, absolute times would be faster for every row because the engine optimizes KV cache injection with page caching, pre-allocated buffers, and async I/O. The relative speedups are what matter: The physics of moving 20% less data over a CPU-free I/O path doesn’t change when the engine gets faster.

With that in mind, focus on the ratios. As context grows, uncompressed restore times balloon while TurboQuant (TQ) with GPU Direct Storage (GDS) stays sub-second even at 50K tokens. The longer the context, the more compression pays off.

And these benchmarks measure cache injection, not cold prefill. Injection is always faster than recomputation. TurboQuant with GDS on FlashBlade makes injection faster still.

Where the bottleneck actually lives

When a user returns to a conversation, the inference server needs to reload their KV cache from persistent storage into GPU memory before it can generate the first token. That time is the time to first token for returning users, and users feel it. Anything over about 500ms starts to feel sluggish. The graph tells you we’re nowhere near that with the conventional path: 1 second at 10K tokens, 3.5 seconds at 30K, over 5 seconds at 50K. The problem gets worse as context grows, and concurrent users make it worse still.

The bottleneck isn’t the storage. It’s the path. The conventional restore puts the CPU in the middle of every byte: Storage reads into CPU memory, the CPU manages the DMA, and the CPU copies it across PCIe into GPU memory. For one session at a time, this is fine. For eight GPUs all restoring simultaneously, the CPU is doing eight copies’ worth of work through one shared memory subsystem. The fix isn’t a faster CPU. The fix is taking the CPU out of the path entirely.

GPU direct storage removes the CPU entirely

GPU Direct Storage (GDS) changes the data path from:

Storage → CPU memory → GPU memory 

to: 

Storage → GPU memory (directly over RDMA)

The CPU is not involved. The FlashBlade system writes directly into GPU HBM, and the GPU handles decompression in-place.

Compression without GDS is like trading your SUV for a sports car and then taking the same gridlocked highway to work. The vehicle is smaller, but you’re still stuck in the same traffic. GDS is the express lane where the sports car can zip by the gridlock.

With TurboQuant and GDS, the restore path becomes:

FlashBlade (RDMA) → GPU Memory → GPU Decompresses

At 30K tokens, only 349.5MB travels over the wire per session (the compressed 3-bit indices, norms, rotation matrix, and codebook, down from 1.72GB in FP16). It arrives directly in GPU memory. The GPU unpacks the 3-bit indices and applies the inverse rotation in sub-millisecond time, because bit manipulation and matrix multiplication are exactly what a massively parallel processor is designed for.

The aggregate read bandwidth across eight GPUs tells the story:

TurboQuant with GDS delivers higher aggregate read bandwidth than any other configuration at every context length, despite reading 5X less total data. Each GPU sustains approximately 900–1,000MB/s through its own GDS/RDMA connection, consistently, regardless of context length. The FlashBlade system delivers data as fast as the GPUs can accept it.

What we measured

The benchmarks ran on an NVIDIA DGX system with eight A100-40GB GPUs connected to an Everpure FlashBlade over NFS with RDMA. The model was Qwen2.5-7B-Instruct, which has a head dimension of 128, the dimensionality that matters for TurboQuant because the concentration of measure phenomenon that makes the algorithm work gets stronger as dimension increases.

The KV cache sizes scale linearly with context length at a consistent 4.9X reduction.

Each benchmark ran eight concurrent GPU workers performing independent checkpoint-and-restore cycles, because that’s what production looks like. Nobody checkpoints one session at a time.

Writes are predictable, reads are the win

Checkpoint (write) performance was consistent across all configurations. Per-worker write times showed low variance, which means the FlashBlade system handled eight concurrent writers smoothly.

The asymmetry matters. In production, checkpoints are write-once, read-many. You checkpoint when a session goes idle; you restore when the user comes back. Slightly slower writes in exchange for an 8–10X faster read is a tradeoff you take every single time.

100% token accuracy after migration

Fast restores don’t matter if the model generates different tokens afterward. So we ran the hardest test using a full session migration at 50,000 tokens of context.

GPU 0 compresses a 2.87GB KV cache using 3-bit TurboQuant, writes it to FlashBlade, and the compressed data is read back, decompressed, and loaded onto GPU 1. The model then generates 50 tokens using teacher-forced evaluation, comparing each predicted token against the ground truth from the original uncompressed cache.

The result was 50 out of 50 tokens matched. 100% Top-1 accuracy. The compressed-and-migrated model produced the exact same output as the original. This isn’t an approximation. Every single predicted token was identical.

Not just 3-bit: Tuning the quality/size trade-off

TurboQuant doesn’t lock you into a single compression level. The algorithm can support mixed-precision modes that split the 128 channels per attention head into two groups by variance, quantizing high-variance (outlier) channels at higher bit widths and the rest at lower bit widths. This gives you fractional effective bit rates and fine-grained control over the quality-to-size ratio.

The 2.5-bit mode uses 32 outlier channels at 4-bit and the remaining 96 at 2-bit, producing an effective rate of 2.5 bits per coordinate. The 3.5-bit mode does 32 channels at 5-bit with the rest at 3-bit. Each subspace gets its own rotation matrix and codebook computed for the appropriate dimension, so the mathematical optimality guarantees still hold within each subgroup.

This matters for production deployments because different use cases have different tolerance for approximation. A customer support chatbot that retrieves and summarizes documentation might be perfectly happy at 2.5-bit with 5.3X compression and 0.964 cosine similarity. A code generation assistant where every token matters might want 4-bit with 3.8X compression and 0.995 cosine similarity. TurboQuant lets you make that choice per deployment or even per layer.

5X more sessions per terabyte

A 10,000-token session (roughly 7,500 words of conversation) takes 573MB in FP16. At 3-bit TurboQuant, it’s 117MB. Carve out 10TB of your FlashBlade for a dedicated KV cache namespace (a small slice of any production deployment), and TurboQuant gives you 85,800 warm sessions instead of 17,430 at FP16. Same hardware, same rack space, same power, 5X more users whose conversations you can restore instantly instead of recomputing from scratch.

FlashBlade scales linearly with GPU count

The eight-GPU results tell a bigger story than just eight GPUs. Each GPU independently restores its session through its own GDS/RDMA connection, with no CPU coordination and no shared locks. The distributed architecture (parallel flash across multiple blades) of FlashBlade is built to sustain hundreds of GB/s of concurrent throughput, and SuperPOD certification means this same architecture extends to hundreds of GPUs. The benchmark was GPU-limited, not storage-limited.

TurboQuant with GDS turns KV cache restore from a storage bandwidth problem into a per-GPU latency problem, and that latency stays constant regardless of cluster size.

The 16TB problem: Why hyperscalers need this

NVIDIA has stated that its AI factory architecture requires 16TB of KV cache storage per GPU. At FP16, that’s 16TB per GPU of the most performance-sensitive storage in the data center, storage that needs to deliver sub-second reads at scale, across thousands of GPUs, simultaneously.

That requirement was defined before TurboQuant existed.

At 3-bit compression, that 16TB per GPU drops to 3.3TB. Multiply across a 1,000-GPU cluster and TurboQuant takes the storage footprint from 16 petabytes to 3.3 petabytes. The dollars and watts and rack space that delta represents are not trivial.

The KV cache at hyperscale isn’t a memory problem anymore. It’s a storage infrastructure problem, and the combination of TurboQuant compression with a FlashBlade system and GDS is what makes 16TB per GPU feasible without 16TB per GPU of premium storage.

What this means for your inference stack

The KV cache problem isn’t going away. Context windows are growing. Concurrent user counts are growing. Models are getting larger. Every one of those trends makes the cache bigger and the restore problem harder, and our benchmarks show the problem gets disproportionately worse with context length.

The standard playbook has been to throw more HBM at the problem. Buy more GPUs, use the ones with more memory, keep everything resident. That playbook no longer works.

TurboQuant with FlashBlade and GDS offers a different answer. Compress the cache on the GPU. Write it to FlashBlade over RDMA. When the user comes back, stream the compressed data directly into GPU memory and decompress in place. Same restore time whether you have eight GPUs or 256. The user never notices.

The KV cache is no longer a memory capacity problem. It’s a storage I/O problem, and storage I/O is a problem Everpure knows how to solve.