TurboQuant
Google Research · Coding
TurboQuant is a compression algorithm from Google Research that targets one of the largest costs in running large language models: the key-value cache. It's a standout piece of AI memory compression work, shrinking that cache by at least 6x while cutting attention computation time by up to 8x on an Nvidia H100, and it does all of this without retraining the model. The method pairs two techniques, PolarQuant and QJL, to reach roughly 3 bits per value while holding accuracy close to the original.
About TurboQuant
What Is TurboQuant
TurboQuant is a vector quantization algorithm built to reduce the memory footprint of LLM inference. It was published by Google Research and formally introduced in a blog post on March 24, 2026, with the underlying paper landing on arXiv in April 2025 and later accepted to ICLR 2026. The first author is Amir Zandieh, a research scientist at Google Research.
The problem it solves is memory, not compute. Every time a transformer model generates a token, it stores key and value vectors for all previous tokens so it doesn't redo the math. That store is the KV cache, and it grows with context length. Push a model to 128K tokens and the cache can balloon past the model weights themselves. So why is long-context inference so expensive to serve? Because that cache, not the weights, is what fills your GPU first.
The catch with older quantization approaches is overhead. Most schemes keep a block of full-precision constants, like scale factors and zero-points, which adds one to two bits per value and gives back some of the savings. TurboQuant sidesteps that. It applies a random rotation so vector coordinates become nearly independent, then runs an optimal scalar quantizer that needs no per-block calibration data carried along for the ride.
That matters to anyone paying for inference. If your GPU runs out of memory, you can't serve the model at all. Cheaper, smaller KV cache means longer context on the same hardware and lower cost per request. The main limitation is that this is a research method, not a shrink-wrapped product. There's no public API to call. You'd adopt it through an inference engine or by integrating the algorithm yourself.
Getting Started
- Read the Google Research blog post to understand the two-stage pipeline and what each stage does.
- Pull the paper from arXiv and check the benchmark tables against the models you actually run.
- Determine whether your inference stack supports custom KV cache quantization; if it doesn't, the integration work falls to your engineering team.
- Test on your own workload, since reported gains are measured on standard benchmarks and your traffic may behave differently.
- Compare memory and latency before and after, then decide if the compression level you're considering keeps outputs usable.
Product Information
A quick look at TurboQuant's pricing, supported platforms, and performance.
Best for
The users, tasks, and scenarios where this tool fits best.
Users
- ML engineers running long-context LLM inference
- Research scientists comparing quantization methods
- Teams serving high-concurrency traffic
Tasks
- Reducing inference memory for 128K-token workloads
- Speeding up attention logit computation
- Vector search and semantic retrieval
- Compressing without retraining
Scenarios
- Serving a 70B model where KV cache alone can exceed 80GB at long context
- Fitting longer conversations into limited VRAM
- Evaluating storage cost impact
Key features
Two-Stage Compression Pipeline
TurboQuant runs in two steps. PolarQuant converts vectors from Cartesian to polar coordinates, which removes the need to store boundary calibration data every time. QJL then cleans up the small residual error left behind using a single bit, with no extra memory overhead. Together they hit roughly 3 bits per value. That's the core of the design.
6x KV Cache Reduction
The headline number is memory: at least a 6x cut in KV cache size, taking the cache from 16-bit storage down to about 3 bits. For a 70B model at long context, the cache can be four times the size of the weights. Shrink it 6x and the deployment math changes fast.
Up to 8x Faster Attention
According to Google Research, the 4-bit version computes attention logits up to 8x faster than a 32-bit baseline on an Nvidia H100. Decoding is memory-bandwidth bound, so moving less data per token speeds up the whole loop. Not every workload sees the ceiling, but the gain isn't small.
No Training or Fine-Tuning
The method is data-oblivious and applies online, so you don't retrain or fine-tune the model to use it. That's a real advantage over codebook approaches like product quantization, which need an offline training pass and slow index lookups. With TurboQuant, the model stays untouched.
Near-Lossless Accuracy
The claim is zero to near-zero accuracy loss across standard long-context benchmarks, including question answering, code generation, and summarization. Independent writeups note the quality-neutral point sits around 3.5 bits, meaning pushing below that won't help further since you're already near the information-theoretic floor.
Works for Vector Search Too
The compression isn't limited to LLM inference. Modern search engines lean on semantic vectors and store billions of high-dimensional embeddings. Every bit saved per vector compounds across a database, so the same algorithm lowers cost and latency for vector retrieval, not just for chat models.
Pros and cons
Pros
- Cuts KV cache memory by at least 6x, letting long-context models run on less hardware.
- Reports up to 8x faster attention computation on H100 at 4-bit precision.
- Needs no retraining or fine-tuning, so you skip an expensive extra step.
- Avoids the per-block constant overhead that eats into other quantization methods' savings.
- Rests on a provable information-theoretic basis rather than hand-tuned heuristics.
Cons
- It's a research method, not a product, so there's no ready API and integration is on you.
- Reported numbers come from standard benchmarks; your own traffic may score differently.
- An academic dispute over comparisons with RaBitQ is still being discussed, so treat some claims with care.
- You need inference-stack control to adopt it, which rules out managed services you can't modify.
Frequently asked questions
It compresses the KV cache that LLMs keep during inference, shrinking memory use by at least 6x and speeding up attention computation. It's a compression algorithm from Google Research, not a standalone app or model.
Related content
Explore related tools, skills, and articles for TurboQuant.
TurboQuant Alternatives
Forefront
Forefront · CodingForefront is a web platform for building with open-source AI. It lets you fine-tune leading open-source language models on your own data, evaluate how they perform, and run them through an API or export them to host yourself. Developers who want the convenience of a closed-source platform but insist on owning their models and data are the target audience here.
Startkit
StartKit.AI · CodingStartkit is a boilerplate for building AI SaaS and AI wrapper products. Think of it as an AI startup boilerplate with the boring parts already wired up: authentication, Stripe and Lemon Squeezy payments, usage limits, transactional email, and an AI API starter that talks to OpenAI, Anthropic, Groq, or Llama. You clone the repo, set your price, and start on the part of your product that people actually pay for. It's Next.js under React and Tailwind, so most of the boilerplate code already feels familiar.
Testim
Tricentis · CodingTestim is an AI-powered test automation platform for building and running end-to-end tests across web, mobile, and Salesforce applications. It leans on machine learning to keep tests stable when an interface changes, so teams spend less time fixing broken selectors. Not bad for an automated testing tool you can start using today. You create tests by recording actions in a browser, then optionally add JavaScript when you need more control. It's a solid pick for busy QA teams.