Red Hat Releases Open Framework for AI Inference Benchmarking on CPUs
Enterprises want to run AI workloads on existing CPU infrastructure. They want to reduce GPU dependence and cut costs. But the tools to validate that approach have been unreliable, until now.
Red Hat’s OCTO Emerging Tech and Perf & Scale Engineering teams have published the vLLM CPU Performance Evaluation framework. It is an automated, open source benchmarking suite built on Ansible and GuideLLM. Its goal: standardize AI inference benchmarking on CPUs so enterprises can make confident, evidence-based infrastructure decisions.
The post appeared May 28, 2026 on Red Hat’s Emerging Technologies blog. It opens with a sharp critique of current industry practice. Hardware vendors publish best-case throughput numbers without reproducible methods. Infrastructure teams then face a benchmarking landscape they cannot compare or trust. That gap blocks production adoption.
The Red Hat framework attacks this problem with a three-phase testing methodology. Phase one establishes baseline capacity. It uses synthetic datasets, fixed sequence lengths, and no prefix caching. This reveals hardware saturation points and raw throughput limits. Phase two injects realistic token variance. It measures how performance shifts under unpredictable traffic. Phase three enables prefix caching and switches to real-world datasets. It then quantifies the gains from production-grade optimizations.
Every benchmark run follows a fixed 600-second window. That includes a 30-second warm-up, a 540-second measurement period, and a 30-second cooldown. The framework grades repeatability using the Coefficient of Variation (CV). Scores below 1% earn an Excellent rating. Scores above 5% signal poor stability. Engineers can therefore quickly judge whether a hardware setup is production-ready.
The framework tests a range of Decoder-Only models, including Llama 3.2-1B, IBM Granite 3.2-2B, Qwen 3-0.6B, and a 21-billion-parameter Mixture-of-Experts model. Five workload profiles drive the tests: chat, summarization, code generation, RAG, and reasoning. Together, these cover both prefill-heavy and decode-heavy scenarios that mirror real enterprise use.
Clean results also require strict isolation. The framework keeps the load generator and the inference engine on separate compute resources. The ideal setup uses two physical nodes on a high-speed network link. When that is not possible, socket-level isolation on multi-socket CPUs, such as AWS c8i.metal or m8a.metal instances, separates the two processes across independent L3 caches and memory controllers.
Memory sizing follows an equally rigorous approach. The framework calculates the exact KV Cache footprint each model needs at a given concurrency level. It then adds a 25% safety buffer. This right-sized strategy lets enterprises pack more models per CPU socket while holding a 100% reliability target.
Red Hat’s conclusion is direct: performance is no longer a black box. It becomes a predictable, measurable resource that infrastructure teams can evaluate with verifiable evidence.
The framework is open source and lives at github.com/redhat-et/vllm-cpu-perf-eval. Red Hat invites engineers to review the testing repository and contribute to its development.
