Context Window Management

Every model has a context window -- a hard limit on how many tokens it can process in a single request. Exceed it and your request fails. Stay under it but pack the wrong things at the edges, and the model ignores the middle. This lesson covers counting tokens, choosing what to keep, and placing content where the model will actually read it.

What you'll learn

• Context windows are measured in tokens, not characters or words -- a token is roughly 3/4 of a word, and you must count them before sending
• Models attend most to the beginning and end of the context window; the "lost in the middle" problem means content in the center gets degraded attention
• Truncation strategy (head, tail, or middle) matters more than most people realize -- keeping the wrong 1000 tokens costs more accuracy than keeping the right 100
• Token budgets are a discipline: assign a maximum token count to each section of your prompt (system, history, documents, instructions) and enforce it before every call

The problem

You have a system prompt, a 20-turn conversation history, three retrieved documents, and a detailed instruction. That's 150,000 tokens. Your model's context window is 128,000. Something has to go. But what?

If you truncate the conversation history from the beginning, the model loses the original task. If you truncate from the end, it loses the most recent context. If you cut the documents, you lose evidence. And even if you stay under the limit, the model might still miss critical information buried in the middle of the prompt -- a well-documented phenomenon called "lost in the middle."

Options & when to use each

Strategy	How it works	Best for	Watch out for
Head truncation	Keep the first N tokens, drop the rest	System prompts, fixed instructions that must come first	Loses everything after the cutoff -- including the user's latest message
Tail truncation	Keep the last N tokens, drop the start	Chat history, where recent messages matter most	Loses the original task and system prompt
Middle truncation	Keep head and tail, drop the middle	Long documents, conversation threads	The "lost in the middle" problem is already bad -- don't make it worse by putting critical content in the dropped zone
Token budget	Assign max tokens to each section	Every production prompt pipeline	Requires discipline and tooling -- you need to measure, not guess

Build it

Step 1: Count tokens before you send

Counting tokens is the prerequisite for everything else. Use tiktoken for OpenAI models or the provider's own tokenizer:

import tiktoken

def count_tokens(text: str, model: str = "gpt-4") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

# Example: measure each section of your prompt
system_prompt = "You are a helpful assistant that answers questions about Python."
history = "User: What is a decorator?\nAssistant: A decorator is..."
documents = "Python decorators are functions that modify other functions..."

print(f"System: {count_tokens(system_prompt)} tokens")
print(f"History: {count_tokens(history)} tokens")
print(f"Documents: {count_tokens(documents)} tokens")
print(f"Total: {count_tokens(system_prompt + history + documents)} tokens")

For Anthropic models, use anthropic's built-in token counter:

import anthropic

client = anthropic.Anthropic()
response = client.messages.count_tokens(
    model="claude-sonnet-4-6",
    system="You are a helpful assistant.",
    messages=[{"role": "user", "content": "Hello, world!"}]
)
print(f"Input tokens: {response.input_tokens}")

Step 2: Build a token budget

A token budget assigns a maximum token count to each section of your prompt. Enforce it before every call:

class TokenBudget:
    def __init__(self, total_limit: int):
        self.total_limit = total_limit
        self.budgets = {}
        self.used = 0

    def allocate(self, section: str, max_tokens: int):
        self.budgets[section] = max_tokens
        self.used += max_tokens
        if self.used > self.total_limit:
            raise ValueError(f"Budget overallocated: {self.used}/{self.total_limit}")

    def enforce(self, section: str, text: str, model: str = "gpt-4") -> str:
        """Truncate text to fit within the allocated budget for this section."""
        limit = self.budgets[section]
        enc = tiktoken.encoding_for_model(model)
        tokens = enc.encode(text)
        if len(tokens) <= limit:
            return text
        # Truncate and add a note
        truncated = enc.decode(tokens[:limit])
        return truncated + f"\n[...truncated {len(tokens) - limit} tokens...]"

# Example: 128K window, budgeted across sections
budget = TokenBudget(total_limit=128_000)
budget.allocate("system", 2_000)
budget.allocate("history", 100_000)
budget.allocate("documents", 20_000)
budget.allocate("instructions", 6_000)

Step 3: Implement smart truncation

Truncation isn't just "keep the first N tokens." The strategy depends on what you're truncating:

def truncate_conversation(messages: list[dict], max_tokens: int, model: str = "gpt-4") -> list[dict]:
    """Keep system message + most recent messages that fit."""
    enc = tiktoken.encoding_for_model(model)

    system_msgs = [m for m in messages if m.get("role") == "system"]
    other_msgs = [m for m in messages if m.get("role") != "system"]

    # Always keep system messages
    result = list(system_msgs)
    used = sum(len(enc.encode(m["content"])) for m in system_msgs)

    # Keep most recent messages until budget is exhausted
    for msg in reversed(other_msgs):
        msg_tokens = len(enc.encode(msg["content"]))
        if used + msg_tokens > max_tokens:
            break
        result.insert(1, msg)  # Insert after system messages
        used += msg_tokens

    return result

def truncate_documents(documents: list[str], max_tokens: int, model: str = "gpt-4") -> list[str]:
    """Keep documents by relevance score, truncating each to fit."""
    enc = tiktoken.encoding_for_model(model)
    kept = []
    remaining = max_tokens

    for doc in sorted(documents, key=lambda d: d.get("relevance", 0), reverse=True):
        doc_text = doc["text"]
        doc_tokens = len(enc.encode(doc_text))
        if doc_tokens <= remaining:
            kept.append(doc_text)
            remaining -= doc_tokens
        elif remaining > 500:  # Keep a partial doc if budget allows
            partial = enc.decode(enc.encode(doc_text)[:remaining])
            kept.append(partial + "\n[...truncated...]")
            break
        else:
            break

    return kept

Step 4: Place critical content at the edges

The "lost in the middle" problem means content at the start and end of the prompt gets the most attention. Structure your prompt accordingly:

def build_optimized_prompt(system: str, documents: list[str], user_query: str) -> str:
    """Build a prompt that puts critical content at the edges."""
    parts = []

    # EDGE 1 (start): System prompt -- always here
    parts.append(system)

    # MIDDLE: Documents -- least-attended zone, so put reference material here
    if documents:
        parts.append("\n\n--- Reference Documents ---\n")
        parts.append("\n\n".join(documents))

    # EDGE 2 (end): The user's actual request -- highest attention
    parts.append(f"\n\n--- Task ---\n{user_query}")

    return "\n".join(parts)

What goes wrong

Mistake	How you notice it	The fix
Counting characters instead of tokens	Request fails with "context length exceeded" even though your char count is under the limit	Tokens != characters. A single emoji can be 3+ tokens. Always use the model's tokenizer
Truncating conversation from the head	Model responds as if it has no memory of the original task	Use middle truncation: keep the system prompt and first message, then the most recent messages
Putting instructions in the middle	Model misses key constraints or follows them inconsistently	Move instructions to the end of the prompt, right before the user's query
Over-allocating to documents	Conversation history gets squeezed to nothing, model loses context	Cap documents at 20-30% of the total window. If you need more documents, use RAG with better retrieval
Not reserving headroom for the response	Request succeeds but the response is truncated or the model rushes to finish	Reserve 20-30% of the window for the model's output. If you need a 4000-token response, subtract 4000 from your input budget

Confirm it worked

Run the following unit tests to validate the context window management logic. These tests ensure the token budget correctly allocates space and middle-truncation properly preserves the system and newest messages.

# Test 1: Token budget doesn't exceed limit
budget = TokenBudget(total_limit=128_000)
budget.allocate("system", 2_000)
budget.allocate("history", 100_000)
budget.allocate("documents", 20_000)
budget.allocate("instructions", 6_000)
assert budget.used == 128_000  # Exactly at limit

# Test 2: Truncation keeps the conversation under budget
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is Python?"},
    {"role": "assistant", "content": "Python is a programming language." * 100},
    {"role": "user", "content": "Tell me about decorators."},
]
truncated = truncate_conversation(messages, max_tokens=200, model="gpt-4")
total = count_tokens("".join(m["content"] for m in truncated), "gpt-4")
assert total <= 200
assert truncated[0]["role"] == "system"  # System message preserved
assert truncated[-1]["content"] == "Tell me about decorators."  # Latest message preserved

# Test 3: Optimized prompt puts instructions at end
prompt = build_optimized_prompt(
    system="You are a Python expert.",
    documents=["Doc A", "Doc B", "Doc C"],
    user_query="Summarize the documents."
)
lines = prompt.split("\n")
assert "Python expert" in lines[0]  # System at start
assert "Summarize the documents" in lines[-1]  # Query at end

Next: Prompt Compression -- when truncation isn't enough, you compress.