Gemma 4 Multi-Token Prediction Hits 3x Faster AI Inference
Google has introduced a significant speed upgrade for its Gemma 4 models. Through Gemma 4 multi-token prediction (MTP) drafters, the models can now generate tokens up to approximately three times faster. Crucially, this comes with no loss in response quality.
The core problem MTP solves is well-known in LLM deployment. During inference, processors repeatedly move billions of parameters from VRAM to compute units for each individual token. This constant data movement creates latency and leaves compute resources idle, especially on consumer hardware. Gemma 4 multi-token prediction directly targets this bottleneck.
The technique pairs Gemma 4 with a lightweight auxiliary model, called a drafter, which predicts several future tokens at once. This happens in less time than the main model takes to process a single token. The primary model then verifies all suggested tokens in parallel using speculative decoding. Because the main Gemma 4 model handles final verification, output quality stays identical to standard inference, just delivered much faster.
Google notes that LLMs spend equal computation on both trivial and complex predictions. MTP drafters therefore free the main model from wasting resources on predictable outputs. Moreover, Google’s engineers implemented hardware-specific optimizations to maximize efficiency across device types. They also published an in-depth visual walkthrough of the drafter architecture on X.
The gains span a wide range of hardware. On personal computers and consumer GPUs, Gemma 4’s 26B MoE and 31B dense models benefit directly. On mobile devices, the E2B and E4B variants see similar improvements. This broad hardware support makes Gemma 4 multi-token prediction a practical tool for on-device and edge AI deployments.
Community reaction has been largely positive, though nuanced. Reddit user FarrisAT called the update impressive but cautioned that local models still make too many errors for critical use, suggesting real gains will come as models approach frontier accuracy. Another commenter, Gohab2001, flagged a key architectural note: MTP drafters share the target model’s KV cache, which meaningfully reduces the usual memory overhead of loading two models simultaneously. On Hacker News, user zozbot234 pointed out that MTP benefits are strongest in single-user or low-concurrency settings, such as mobile or edge scenarios, rather than large-scale API serving.
Gemma 4 MTP-enabled model variants are already available across platforms including Hugging Face, Kaggle, and Ollama.
