x
Thank you! Your submission has been received and we will follow up shortly.
Oops! Something went wrong while submitting the form.

AI INFERENCE

Reuse more context.
Recompute less.

GPU-accelerated I/O for AI inference.

Your GPUs aren’t slow. They just have a short memory. Graid Technology turns the NVMe already inside your GPU servers into a high-speed KV cache tier, ​so AI can reuse computed context instead of recomputing it.​

3.26x

Faster mean TTFT

53.4%

More total throughput

36M

GPU-initiated IOPS

GPU-accelerated I/O for AI inference.

Every time context is evicted from GPU memory, AI inference has to rebuild it from scratch. Graid Technology turns the NVMe already in your GPU servers into a highspeed KV cache tier, so context is retrieved instead of recomputed.

3.26x

Faster mean TTFT

53.4%

More total throughput

36M

GPU-initiated IOPS

THe CHALLENGE

More context. More users. More work worth reusing.

GPU memory is finite, and model weights claim a large share of it before the first prompt arrives. As context grows and more sessions run at once, the context worth keeping outgrows what the GPU can hold.
Blue icon of two figures with arrows point to and from each other to represent collaboration and teamwork

Long-context inference

Longer prompts, RAG and agent loops build large KV cache footprints. That context is worth keeping, not rebuilding.

Blue speedometer icon representing performance

Concurrent sessions

A few active sessions can push KV cache past GPU memory. Cache capacity and data movement have to scale with concurrency.

Blue icon of gear with two arrows around the gear representing an infinite loop

Repeated prefixes

System prompts, shared documents and multi-turn conversations repeat the same context. Reusing it skips repeated prefill work.

how it works

Cache capacity is only half the answer.

The I/O path decides how fast context

A larger cache tier gives inference somewhere to keep computed context. GPU-accelerated I/O decides whether that context returns fast enough to beat recomputing it.
01
Retain
Offload reusable KV cache from GPU memory to a high-speed NVMe
02
Retrieve
Bring eligible cached context back through GPU-accelerated I/O, not a CPU-mediated fetch.
03
Reuse
Cache-management software restores valid context to the inference engine, and the response resumes without repeating prefill.
LAYER 1

Inference engine

Produces and consumes KV cache in GPU memory.

LAYER 2

Cache-management layer

Decides what to offload, keep and restore.

LAYER 3

GPU-accelerated I/O + NVMe

The fast path to a larger, reusable cache tier.

Logical view, not a product implementation diagram. Cache-management software controls retention and reuse; GPU-accelerated I/O speeds access to the NVMe tier.

DEPLOYMENT

Choose the cache model that fits your workload.

Start with the infrastructure approach.

Server-local cache

Keep the cache tier on the NVMe inside each GPU server, with protection that survives a single-drive failure.

A fit when
Check
Cache locality and lowest latency come first
Check
Capacity fits within a single server
Check
You want to use the NVMe you already own

Shared cache tier

Move KV cache out of the GPU servers into a pooled tier, so compute and cache scale independently across the inference fleet.

A fit when
Check
Pooled capacity matters more than locality
Check
Cache needs to grow without adding GPU servers
Check
Network and sharing requirements have been evaluated

MEASURED RESULTS

Measured where inference feels it.

Two separate tests, two outcomes people notice: how fast the first token arrives, and how much work the fleet gets through.

Up to 53.4% Higher Throughput​

KV Cache for Rack
No KV Cache Offload
512
1024
2048
4096
8192

Total throughput at growing context length (prefix tokens)​

3.26x TTFT Speedup​

No cache

29.45s

MD RAID5

36.64s

SupremeIO™

9.02s

Mean TTFT improved from 29.45s to 9.02s​

Tested: Qwen3-235B-A22B · 4x NVIDIA H200 · 8x 3.84TB NVMe RAID5 · vLLM + LMCache​
Results come from two separate tests on different configurations and are not cumulative. Performance varies by model, context length, concurrency and hardware.

THE ENGINE

Powered by SupremeIO™

SupremeIO™ is the GPU-accelerated I/O layer from Graid Technology that makes this architecture work. See how it works, which integrations are supported and what your deployment needs
Explore SupremeIOTM

KV Cache, Explained

Why context became an inference resource, and what happens when it overflows GPU memory.

Read the Blog

The Proof

Full TTFT and throughput white papers, with configurations, baselines and methodology.

See the Benchmarks

Talk to Us

Reach out to our technical team to help you plan the cache tier.

Speak to an Expert

News & Resources

SupremeIO™ delivers 58.6% lower mean TTFT and 40.3% higher completion throughput on the ASUS ExpertCenter Pro ET900N G3 with KV cache offloading. Read our latest white paper to get the benchmark results.
Visit Graid Technology and our partner Beiliande at Smart Campus CIO 2026 Booth 18 in Wuhan, November 19–20, to see SupremeStore high-performance cluster storage for AI and HPC, and software-defined solutions that put your existing GPUs to work. Stop by to see how our solutions work together.
Graid Technology is featured in TechArena’s report The Integrated Machine. Field CTO Randy Kreiser discusses KV cache, inference performance, and why storage has to be designed into the rack from the start.