Pure KVA 1.0 Is GA: Inside the Architecture of a Fully Managed Enterprise KV Cache Solution

Every team running LLM inference at scale eventually runs into the same wall: the KV cache. It’s the single biggest lever on inference cost and latency, and it’s also the hardest thing to manage well. With Pure Key Value Accelerator (KVA) 1.0, now generally available, we’ve reimagined how KV cache offload works, disaggregating inference from the KV cache storage daemon for a simple, fully managed enterprise solution. This enables cache reuse across inference engines and hosts, distributed lifecycle management, and fleet metrics to keep control of your KV cache footprint. Ultimately, it improves AI infrastructure utilization by accelerating inference response times and freeing up GPU compute cycles.

This post is about the problems that shaped that design, and how Pure KVA addresses them.

KV cache is where inference economics are decided

When an inference engine serves a request, it computes key/value attention state (KV cache) during the prefill. That state represents real GPU compute time. The trouble is that the cache for any given prompt is ephemeral. Once it’s evicted from GPU memory, the work is simply gone, and a future request with the same content pays the multi-second compute prefill for it all over again.

In practice, this recomputation is rampant from shared system prompts to multi-turn conversations and agent loops replaying the same history—all workloads where the same tokens get prefilled again and again. The cache that would make them cheap doesn’t survive long enough to be reused.

Offloading KV cache to fast storage—NVMe, a filesystem, or object storage—is the obvious answer. A cache hit turns an expensive prefill into a fast load. But “offload the cache” is deceptively simple to say. Doing it in a way that’s actually fast, safe, and operable is where most approaches fall down. Three pain points emerge:

1. The storage path is coupled to the inference path. When cache I/O lives inside the inference process, a slow disk or a storage hiccup doesn’t just slow the cache; it stalls the engine serving live traffic, or in the worst case, brings the engine down.
2. Moving KV cache is expensive. KV tensors are large and live in GPU memory. A naive offload copies them through host memory, storage, and network I/O layers that eat the very latency you were trying to save.
3. Caches grow forever. Without disciplined expiration that works across multiple hosts sharing the same storage, offloaded cache becomes an unbounded liability.

Pure KVA 1.0 is built around solving these three problems.

Decoupling storage from inference

The defining change is architectural. We split Pure KVA into two cooperating processes that communicate over a dedicated channel:

• The Pure KVA plugin: A thin client loaded inside the inference engine. Its only job is to translate the inference engine’s scheduler and worker hooks into calls. The plugin holds no storage logic; its role is deliberately small. 
• The Pure KVA server: A long-lived systemd service daemon that owns all storage tasks: serialization, backend I/O, cache index, and lifecycle. 

Disaggregating the storage path from the inference path directly answers the first pain point. A stall in the daemon degrades cache performance; it does not take down the inference engine. And because the storage engine sits behind a stable API boundary, we can evolve backends, chunking, and batching strategies without ever shipping new code into the inference hot path.

Figure 1: Pure KVA high level architecture ( * S3 on Roadmap) 

Moving KV cache without paying for it twice

The second pain point, the cost of moving the data itself, is solved by never taking the slow path.

Rather than copying KV tensors through host memory to hand them to the server, the plugin shares the GPU memory handle. The server opens that handle and reads GPU memory directly. On the save path, it walks the cache layout to coalesce adjacent blocks into single transfers. Paired with GPU Direct Storage on the backend, KV cache can move between GPU and NVMe without bouncing through the CPU at all.

Two more design choices keep the data path out of the way of inference:

• Everything is async. Every call adds work onto a lock-free task scheduler in the Pure KVA server, with separate lanes for loads, stores, metadata, and lookups so a burst of one can’t starve another. 
• Hit/miss decisions are cheap. Cache lookups are answered from an in-memory LRU cache index to avoid a storage check on every request. The index TTL is configurable (default 30 minutes), long enough to avoid hammering the backend, short enough to re-validate before trusting a stale entry. At hundreds of thousands of chunks, skipping the per-lookup storage check is what keeps cache retrieval fast.

Keeping the cache bounded and correct across a fleet

The third pain point is easy to ignore: a cache that only grows until storage is full and causes the overall system performance to degrade. Pure KVA 1.0 introduces a production-ready lifecycle management subsystem designed for the realistic case where several inference engine instances share one storage pool.

• TTL by time bin: Stored KV caches are grouped into hourly bins. Cleanup computes one expiration boundary from the configured TTL (default 24 hours) and runs against all time bins past it, providing a cheap operation rather than a per-entry timestamp check.
• Coordinated cleanup: When multiple servers share storage, an election loop assigns expired bins to live hosts so no bin is swept twice, with all hosts participating in the lifecycle operation. 
• One clock for everyone: Distributed expiry only works if every host agrees on the time. A clock-sync component stamps a file on the shared backend and reads back its modification time to derive a per-host offset, so all hosts compute identical bin boundaries even when local clocks drift. 

The result is a cache that stays bounded, expires predictably, and behaves correctly whether you run one server or a cluster of them.

Ready for production

GA means ready for enterprise fleet-scale production. Pure KVA 1.0 arrives as .deb and .rpm packages with first-class systemd integration, structured syslog logging with automatic rotation, and a full OpenTelemetry metrics pipeline baked in. Its log level stays completely isolated from your inference engine—so when something needs investigating, you’re reading a clean signal, not untangling two systems’ worth of noise.

The takeaway

KV cache is where inference cost and latency are won or lost, and offloading it is only worthwhile if the offload itself is fast, isolated, and well-behaved. Pure KVA 1.0 treats those as first-class requirements: storage decoupled from inference, fastest I/O path for KV cache load and save, metrics and lifecycle management across the fleet. That’s the foundation to build production KV cache acceleration at scale—and it’s available today. Talk to your Everpure account team to get started with Pure KVA.