Prompt Compression

Truncation throws information away. Compression preserves it -- by making it shorter. When you have 200K tokens of documents but only 32K of budget, compression is the difference between the model seeing a summary of everything and seeing nothing at all. This lesson covers LLMLingua, summarization chains, and the cost-quality tradeoff that determines when compression is worth it.

What you'll learn

• Compression reduces token count while preserving information content; truncation discards information entirely
• LLMLingua is a small, fast model that compresses prompts by 2-5x with minimal quality loss -- it works by removing low-perplexity tokens that the target LLM can reconstruct
• Summarization chains break long documents into chunks, summarize each, then summarize the summaries -- like a pyramid of increasingly condensed information
• Compression has a cost: every compression step consumes tokens. The break-even point is when the tokens saved exceed the tokens spent compressing

The problem

You have a 50-page research paper, a 200-message Slack thread, and a product spec. Your model's context window is 128K tokens, but your budget for documents is 32K. Truncation would mean throwing away 80% of the content. Compression means distilling it down to the essential 32K.

But compression isn't free. If you spend 15K tokens running a summarization model to save 20K tokens, you've only netted 5K. If the compression introduces errors, you've lost accuracy for nothing. The art is knowing when to compress, how much to compress, and which compression method to use.

Options & when to use each

Method	Compression ratio	Quality loss	Token cost to compress	Best for
LLMLingua (small model)	2-5x	Low	~1-2% of input tokens	High-volume, latency-sensitive pipelines
LLMLingua (LLM-based)	3-10x	Low-medium	~5-10% of input tokens	When quality is critical and you can afford the extra cost
Chunked summarization	5-20x	Medium	~10-20% of input tokens	Very long documents where you need structured summaries
Recursive summarization	10-50x	Medium-high	~15-30% of input tokens	Massive document collections where you need a bird's-eye view
Selective context (keep relevant, drop irrelevant)	2-10x	Low	~1-5% of input tokens	When you have a query and can score relevance per chunk

Build it

Step 1: Compress with LLMLingua

LLMLingua uses a small language model (like GPT-2 or LLaMA-3.2-1B) to identify and remove tokens that are predictable from context. The target LLM can reconstruct them, so the compressed prompt reads like a telegram but carries the same information:

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True
)

long_prompt = """
The Python programming language, first released in 1991 by Guido van Rossum,
is a high-level, general-purpose programming language. Its design philosophy
emphasizes code readability with the use of significant indentation. Python
is dynamically typed and garbage-collected. It supports multiple programming
paradigms, including structured, object-oriented, and functional programming.
""" * 10  # 10x for a realistically long prompt

original_tokens = len(long_prompt.split()) * 1.3  # Rough token estimate
compressed = compressor.compress_prompt(
    long_prompt,
    rate=0.5,  # Target: 50% compression (2x)
    force_tokens=["!", ".", "?", "\n"]  # Keep punctuation
)
compressed_tokens = len(compressed["compressed_prompt"].split()) * 1.3

print(f"Original: ~{original_tokens:.0f} tokens")
print(f"Compressed: ~{compressed_tokens:.0f} tokens")
print(f"Ratio: {original_tokens / compressed_tokens:.1f}x")
print(f"\nCompressed text:\n{compressed['compressed_prompt'][:500]}...")

Step 2: Build a summarization chain

For documents too long for a single summarization call, chain it: split into chunks, summarize each, then summarize the summaries:

from openai import OpenAI
import tiktoken

client = OpenAI()
enc = tiktoken.encoding_for_model("gpt-4o")

def chunk_text(text: str, max_chunk_tokens: int = 3000) -> list[str]:
    """Split text into chunks that fit within token limits."""
    tokens = enc.encode(text)
    chunks = []
    for i in range(0, len(tokens), max_chunk_tokens):
        chunk_tokens = tokens[i:i + max_chunk_tokens]
        chunks.append(enc.decode(chunk_tokens))
    return chunks

def summarize_chunk(chunk: str, query: str = "") -> str:
    """Summarize a single chunk, optionally guided by a query."""
    prompt = f"Summarize the following text in 2-3 sentences. "
    if query:
        prompt += f"Focus on information relevant to: {query}\n\n"
    prompt += f"Text:\n{chunk}"

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=200
    )
    return response.choices[0].message.content

def summarize_document(text: str, query: str = "", max_input_tokens: int = 100_000) -> str:
    """Full summarization pipeline: chunk -> summarize -> combine."""
    # Truncate if still too large
    tokens = enc.encode(text)
    if len(tokens) > max_input_tokens:
        text = enc.decode(tokens[:max_input_tokens])

    chunks = chunk_text(text, max_chunk_tokens=3000)

    # Level 1: Summarize each chunk
    chunk_summaries = []
    for chunk in chunks:
        summary = summarize_chunk(chunk, query)
        chunk_summaries.append(summary)

    # Level 2: Summarize the summaries
    combined = "\n\n".join(chunk_summaries)
    if len(enc.encode(combined)) > 3000:
        # Recurse if still too long
        return summarize_document(combined, query, max_input_tokens)
    else:
        final = summarize_chunk(combined, query)
        return final

Step 3: Selective context -- keep what matters, drop the rest

Not all compression needs an LLM. If you have a query, you can score each chunk for relevance and only keep the top results:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

def select_relevant_chunks(
    query: str,
    chunks: list[str],
    max_chunks: int = 5,
    max_tokens_per_chunk: int = 2000
) -> list[str]:
    """Keep only the chunks most relevant to the query."""
    # Combine query with chunks for TF-IDF
    all_texts = [query] + chunks
    vectorizer = TfidfVectorizer(stop_words="english", max_features=5000)
    tfidf_matrix = vectorizer.fit_transform(all_texts)

    # Query is row 0, chunks are rows 1..N
    query_vec = tfidf_matrix[0:1]
    chunk_vecs = tfidf_matrix[1:]

    similarities = cosine_similarity(query_vec, chunk_vecs).flatten()
    top_indices = np.argsort(similarities)[-max_chunks:][::-1]

    selected = []
    for idx in top_indices:
        enc = tiktoken.encoding_for_model("gpt-4o")
        tokens = enc.encode(chunks[idx])
        if len(tokens) > max_tokens_per_chunk:
            chunks[idx] = enc.decode(tokens[:max_tokens_per_chunk])
        selected.append(chunks[idx])

    return selected

Step 4: Calculate the break-even point

Compression only makes sense when the tokens saved exceed the tokens spent:

def compression_break_even(
    input_tokens: int,
    compression_ratio: float,
    compressor_cost_tokens: int,
    target_model_cost_per_1k: float,
    compressor_cost_per_1k: float
) -> dict:
    """Calculate whether compression is worth it."""
    compressed_tokens = input_tokens / compression_ratio
    tokens_saved = input_tokens - compressed_tokens

    # Cost without compression
    cost_without = (input_tokens / 1000) * target_model_cost_per_1k

    # Cost with compression
    compression_cost = (compressor_cost_tokens / 1000) * compressor_cost_per_1k
    target_cost = (compressed_tokens / 1000) * target_model_cost_per_1k
    cost_with = compression_cost + target_cost

    return {
        "tokens_saved": tokens_saved,
        "cost_without": cost_without,
        "cost_with": cost_with,
        "savings": cost_without - cost_with,
        "worth_it": cost_with < cost_without
    }

# Example: Is LLMLingua worth it for a 100K token document?
result = compression_break_even(
    input_tokens=100_000,
    compression_ratio=3.0,  # 3x compression
    compressor_cost_tokens=2000,  # LLMLingua small model is cheap
    target_model_cost_per_1k=0.003,  # GPT-4o-mini
    compressor_cost_per_1k=0.0001  # Local model, effectively free
)
print(result)
# tokens_saved: ~66,667 -- worth it

What goes wrong

Mistake	How you notice it	The fix
Compressing too aggressively (10x+)	The model produces answers that miss key facts or contradict the original	Cap compression at 3-5x for factual content. Only go higher for summarization tasks where you explicitly want the gist
Compressing short prompts	You spend 2K tokens compressing a 3K token prompt -- net loss	Only compress when input > 10K tokens. For short prompts, just send them as-is
Summarization chain loses critical details	Level 2 summary is vague or wrong -- the first-level summaries already dropped the key fact	Inject the user's query into each summarization step. A query-guided summary preserves what matters for the task
Selective context misses the needle	The model can't answer the question because the relevant chunk scored low in TF-IDF	TF-IDF works on keyword overlap. If the query uses different vocabulary, use embedding-based retrieval (e.g., text-embedding-3-small) instead
Compression removes formatting the model needs	The model misreads a table or code block because compression stripped the structure	Mark code blocks and tables as force_tokens in LLMLingua. Never compress structured data without preserving delimiters

Confirm it worked

Use these tests to ensure the prompt compression pipeline is operating correctly. This verifies that LLMLingua reduces token count as expected and the summarization chain accurately condenses large inputs.

# Test 1: LLMLingua compression reduces token count
original = "The quick brown fox jumps over the lazy dog. " * 50
compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True
)
result = compressor.compress_prompt(original, rate=0.5)
assert len(result["compressed_prompt"]) < len(original) * 0.8  # At least 20% shorter

# Test 2: Summarization chain produces a single summary
long_text = "Python is a programming language. " * 1000
summary = summarize_document(long_text, max_input_tokens=10000)
assert len(summary) < 500  # Summary is concise
assert "Python" in summary  # Key content preserved

# Test 3: Break-even analysis catches bad compression
result = compression_break_even(
    input_tokens=1000,  # Very short input
    compression_ratio=3.0,
    compressor_cost_tokens=2000,  # Compressor costs more than input!
    target_model_cost_per_1k=0.003,
    compressor_cost_per_1k=0.0001
)
assert not result["worth_it"]  # Should not compress 1000 tokens

Next: Prompt Optimization -- once you can fit the prompt in the window, you optimize it.