TurboQuant: Is Compression and Performance Worth the Hype?

# Introduction

TurboQuant is a novel algorithmic suite and library recently introduced by Google. Its goal is to apply advanced scaling and compression to large-scale linguistic models (LLMs) and vector search engines — key features of retrieval-augmented generation (RAG) systems — to greatly improve their efficiency. TurboQuant has been shown to effectively reduce cache memory usage to only 3 bits, without requiring model retraining or sacrificing accuracy.

How does it do that, and is it really worth the hype? This article aims to answer these questions with an explanation and a practical example of their use.

# TurboQuant In Brief

While LLMs and vector search engines use high-dimensional vectors to process information with impressive results, this effort requires a lot of memory, which may cause large bottlenecks in the key-value cache (KV) — a quick access “digital cheat sheet” containing information that is often used for real-time retrieval. Handling a larger context length increases KV cache access linearly, which severely limits memory capacity and computing speed.

Vector quantization (VQ) techniques used in recent years help to reduce the size of text vectors to remove bottlenecks, but they tend to introduce the side of “memory head” and need to compute quantization constants with full precision for small blocks of data, thus undermining part of the reason for compression.

TurboQuant is a set of next-generation algorithms for advanced compression with zero loss of accuracy. It effectively tackles the memory problem by using a two-stage process aided by two complementary strategies:

• PolarQuant: This is the compression method used in the first stage. Compresses high-quality data by mapping vector coordinates to a polar coordinate system. This simplifies the data geometry and removes the need to store additional quantization constants – the main cause behind memory overhead.
• QJL (Quantized Johnson-Lindenstrauss): The second stage of the compression process. It focuses on removing the potential biases introduced in the previous section, acting as a statistical checker that uses a small, one-off pressure to remove hidden errors or residual biases caused by using PolarQuant.

Is TurboQuant Worth the Hype?

According to test results and evidence, the short answer is yes. By avoiding the expensive data normalization required in traditional calibration methods, 3-bit TurboQuant produces an 8x performance increase over 32-bit keys that cannot be calibrated on the H100 GPU-based accelerator.

# TurboQuant calibration

The following Python code example shows how developers can test this in the environment. The program can be run in a local IDE or in the Google Colab notebook environment, providing intuitive comparisons between nonquantitative vectors and the fast compression of TurboQuant.

TurboQuant databases require certain characters to work. To make this example work, do the following installation first – preferably on a notebook drive, unless you have a lot of disk space on your machine.

First, install TurboQuant:

In the case of Google Colab, just install the library and make sure your hardware runtime accelerator is set to T4 GPU – available in the free Colab section – so that the following code will run properly.

The following code shows a simple comparison of performance and memory usage when using the pre-trained language model with and without TurboQuant's KV compression. First and foremost, the supplies we will need:

import torch
import time
from transformers import AutoModelForCausalLM, AutoTokenizer
from turboquant import TurboQuantCache

We will load the LLM which is not so big TinyLlama/TinyLlama-1.1B-Chat-v1.0trained for text generation, and its tokenizer. We specify using 16-bit decimal precision: this option usually works well on modern hardware.

model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.float16)

Next, we describe the scenario, simulating the input string of a large model, as TurboQuant really shines as the context windows get bigger. Don't worry about repeating the same content 20 times for every input: here what matters is the managed size, not the language itself.

prompt = "Explain the history of the universe in great detail. " * 20 
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

The following task is key to measuring and comparing execution time and memory usage throughout the text generation process, using TurboQuant's 3-bit quantization, use_tq=True or disabled, use_tq=False. The cache is first flushed to ensure clean measurements.

def run_unified_benchmark(use_tq=False):
    torch.cuda.empty_cache()
    
    # Initializing the specific cache type
    cache = TurboQuantCache(bits=3) if use_tq else None
    
    start_time = time.time()
    with torch.no_grad():
        # Running the model to generate output tokens
        outputs = model.generate(**inputs, max_new_tokens=100, past_key_values=cache)
    
    duration = time.time() - start_time
    
    # Isolating the Cache Memory
    # Instead of measuring the whole 2GB model, we measure the generated Cache size
    # For a 1.1B model: [Layers: 22, Heads: 32, Head_Dim: 64]
    num_tokens = outputs.shape[1]
    elements = 22 * 32 * 64 * num_tokens * 2 # Key + Value
    
    if use_tq:
        mem_mb = (elements * 3) / (8 * 1024 * 1024) # 3-bit calculation
    else:
        mem_mb = (elements * 16) / (8 * 1024 * 1024) # 16-bit calculation
        
    return duration, mem_mb

Finally we perform the process twice – once with the two specified settings – and compare the results:

base_time, base_mem = run_unified_benchmark(use_tq=False)
tq_time, tq_mem = run_unified_benchmark(use_tq=True)

print(f"--- THE VERDICT ---")
print(f"Baseline (FP16) Cache: {base_mem:.2f} MB")
print(f"TurboQuant (3-bit) Cache: {tq_mem:.2f} MB")
print(f"Speedup: {base_time / tq_time:.2f}x")
print(f"Memory Saved: {base_mem - tq_mem:.2f} MB")

Results:

--- THE VERDICT ---
Baseline (FP16) Cache: 42.45 MB
TurboQuant (3-bit) Cache: 7.86 MB
Speedup: 0.61x
Memory Saved: 34.59 MB

The compression ratio is impressively up to 5.4x with respect to the KV cache memory. But what about speedup? Is it expected with TurboQuant? Not exactly, but this is normal, as the sequence we used is still considered short for the large scale TurboQuant is intended for, and we are using this locally, not with a large infrastructure. The real speed advantage with TurboQuant occurs as the core length and hardware accelerators are used to scale together. Take a business-level cluster of H100 GPUs and long-form RAG information containing more than 32K tokens: in such cases, memory traffic is significantly reduced, and an output increase of up to 8x in speed can be expected with TurboQuant.

In general, there is a trade-off between memory bandwidth and computing latency, and you can further verify this by trying other input and output size settings, eg multiplying the input string by 200 and setting max_new_tokens=250you can get something like this:

--- THE VERDICT ---
Baseline (FP16) Cache: 421.44 MB
TurboQuant (3-bit) Cache: 79.02 MB
Speedup: 0.57x
Memory Saved: 342.42 MB

Finally, the dynamic performance of TurboQuant in AI models is proven by its ability to maintain high accuracy while effectively running a 3-bit-level system over large areas.

# Wrapping up

This article introduced TurboQuant and addressed the question of whether it is worth the hype, in terms of compression and performance compared to other common measurement methods used in LLMs and other mass measurement models.

Iván Palomares Carrascosa is a leader, author, speaker, and consultant in AI, machine learning, deep learning and LLMs. He trains and guides others in using AI in the real world.