Supermicro, IBM and NVIDIA have introduced a shared storage platform designed to reduce delays when large AI models process long prompts and multi-step agent workflows. The system combines Supermicro Petascale all-flash storage servers, IBM Storage Scale, NVIDIA Dynamo inference software and Spectrum-X Ethernet networking. In a joint test involving a 120-billion-parameter language model and prompts reaching 130,000 tokens, the companies reported up to 56 times faster time to first token with persistent key-value caching than with a configuration that did not use the cache.
The launch addresses a problem that appears after an AI model has already been trained. During inference, a model processes a prompt and generates a response. For long conversations, coding sessions or document-heavy requests, the system may need to process much context before it can produce the first output token. That initial wait is commonly measured as time to first token, or TTFT, and it can become a noticeable part of the UX when prompts contain tens of thousands of tokens.
Large language models create intermediate key-value states as they process input. These states are part of the attention mechanism and allow the model to reuse information from earlier tokens instead of recalculating it for every subsequent step. Keeping those states available is known as KV caching. The challenge is capacity: long prompts and concurrent requests can generate more cached data than can comfortably fit in the memory attached to the GPUs doing the inference.
Why long prompts create an inference bottleneck
The new platform can extend that cache beyond local GPU memory. Supermicro supplies Petascale all-flash storage servers using AMD EPYC 9005 series processors. IBM contributes Storage Scale software, while NVIDIA provides its Dynamo inference framework and Spectrum-X Ethernet networking. Together, the components create a persistent, shared storage tier where cached context can be retained and accessed by the inference system. The approach aims to separate cache capacity from the amount of memory physically attached to each GPU.
That separation could be useful for agentic AI systems, coding assistants and retrieval-augmented generation applications. These workloads often revisit the same repository files, documents, conversation history or other context across multiple steps. If the system can reuse previously computed key-value states, it may avoid repeating part of the same computation. A shared cache can also make the stored context available across GPU nodes, potentially reducing the need to keep duplicate data in every accelerator’s local memory.
The partners move reusable context into shared storage
Supermicro’s announcement describes a joint cluster-level validation exercise that included high-concurrency workloads and network conditions designed to test interference between users. The companies report that a 120-billion-parameter language model achieved up to 56 times faster TTFT at 130,000-token prompts when using persistent KV cache. They also say that, as prompt length increased from 30,000 to 130,000 tokens, TTFT rose by only 0.377 seconds with the cache, compared with 29.55 seconds without it.
Those figures are significant, but they should be understood as results from the partners’ specified test environment, not a universal performance guarantee. The announcement does not provide enough information to assume that every model, GPU configuration, network topology or workload will produce the same improvement. TTFT is also only one measure of inference performance. Throughput, concurrency, output-token speed, cost per request, cache hit rate and the overhead of moving data across the network will all affect the practical value of the architecture.
The reported speed-up needs to be read in its test context
The technical reason for the improvement is that the cache can preserve previously computed context in a storage layer with greater capacity than local accelerator memory. Without reusable states, the model may have to process the same prompt context again when a later request needs it. With a cache hit, the inference system can retrieve the stored state and continue from a later point in the computation. The benefit will depend on how often requests reuse context and how efficiently the storage and network layers provide it.