Supermicro, Inc. announced a shared key-value cache system with IBM and NVIDIA on Nov. 5, aiming to reuse prompt data and reduce processing time for long-context AI inference.
The design connects Supermicro Petascale all-flash servers with AMD EPYC 9005 Series processors to IBM Storage Scale, NVIDIA Dynamo inference software and NVIDIA Spectrum-X Ethernet networking.
IBM's Redbook reports that a GPT-OSS 120B model took 0.57 seconds to produce its first token from a 130,000-token prompt with caching, compared with 32.14 seconds without it. The reported times amount to a 56× speedup. The test used one NVIDIA RTX PRO 6000 Blackwell GPU and an eight-node Supermicro storage cluster.
A key-value, or KV, cache holds intermediate data generated as a model processes a prompt. In this system, shared storage keeps that data available for later requests using the same context. The Redbook says cache data can outgrow available GPU memory; if context is evicted, calculating it again consumes GPU cycles.
In another test, the Redbook reports, 200 requests over 28 concurrent connections took 46.94 seconds with the cache and 1,048 seconds without it. The cache started empty. Throughput was 4.26 requests per second with caching, against 0.19 without it.
A network-stress test had four clients generate 200 GB/s of competing traffic. According to the Redbook, the cached system completed 200 requests in 55.56 seconds at 3.6 requests per second, compared with 4.26 requests per second without competing network traffic.
StorageReview said it neither conducted nor audited the tests, and noted that a simultaneous surge of cache requests from an entire GPU cluster was outside the measured scenarios. Supermicro presents higher GPU utilization and throughput as potential benefits, particularly for long-context and agentic AI workloads. Its announcement gave no product price or general-availability date.
