Prompting & Context Engineering

You've probably had this happen: a prompt that worked perfectly yesterday starts producing garbage today, and nothing about your code changed. Nine times out of ten, the culprit is context, either you've stuffed too much into it, put the important instruction in the wrong place, or the phrasing shifted just enough to knock the model off its usual behavior. Prompting isn't guesswork once you understand what's actually happening to your text before the model ever generates a token, and that's what this lesson covers.

What you'll learn

• Why an instruction buried in the middle of a long prompt gets ignored more often than one at the start or end
• Prompt caching: how providers cut cost by up to 90% on repeated context, and the one rule that breaks it
• Structuring a prompt so the model can't confuse your instructions with the data you handed it

The hook: why the same prompt behaves differently at different lengths

A model doesn't read your prompt the way you do. Every word gets converted to a token, and the model's attention mechanism has to decide, for every token it generates, how much weight to give every token that came before. When your prompt is short, that's easy, everything gets roughly equal attention. When your prompt grows to include a full codebase, a long conversation history, or a stack of retrieved documents, attention has to spread across a lot more material, and it doesn't spread evenly. Instructions sitting in the middle of a long prompt are the ones most likely to get lost, a well-documented behavior often called "lost in the middle." The practical fix is almost embarrassingly simple once you know it: put your most important constraints at the very start or the very end of the prompt, never buried in the middle of a wall of context.

Prompt caching: the same context, ten times cheaper

If your agent sends the same system prompt, the same database schema, or the same large document as context on every single call, which is extremely common in agent loops, you're paying full price to reprocess identical text over and over. Both Anthropic and Google let you avoid that: send the same prefix twice, and the second call reuses the model's internal state for that prefix instead of recomputing it from scratch.

The catch is the one rule that trips people up: the cached portion has to match byte for byte. A single extra space, a timestamp inserted into your system prompt, a variable that changes between calls, any of it invalidates the cache and you're back to paying full price. The fix is structural: put everything static (system instructions, schemas, reference documents) at the top of the prompt, and put anything that changes call to call, the user's actual question, a timestamp, a session ID, at the very bottom. Anthropic's caching kicks in above roughly 1,024 tokens of stable prefix; Gemini's threshold is higher, around 32,000 tokens. Below that, caching isn't worth the complexity.

Structuring a prompt so nothing gets confused

The other recurring failure is the model treating something as an instruction when it was actually data, or vice versa, especially dangerous when a prompt includes content from outside your control, like a retrieved document. Wrapping distinct pieces in clear tags fixes this:

<system_instructions>
You are an expert PostgreSQL database administrator.
Output SQL statements based on the given schema.
</system_instructions>

<constraints>
- Output raw SQL only.
- No explanation text.
</constraints>

<schema_context>
CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT);
</schema_context>

<user_query>
Add an email column to the users table.
</user_query>

There's nothing magic about XML specifically, the point is a clear, consistent boundary the model can rely on to tell "this is an instruction" from "this is data I was handed." Pick a format and use it consistently across every prompt in your system; switching formats lesson to lesson or call to call is itself a source of inconsistent behavior.

What goes wrong

Mistake	How you notice it	The fix
Output gets cut off mid-sentence, breaking JSON parsing	JSONDecodeError or invalid Python syntax in agent output	Raise max_output_tokens , a low limit (256-512) silently truncates structured output
A critical instruction gets ignored on long prompts	Agent follows most of the prompt but misses one specific constraint	Move that constraint to the very start or end of the prompt, not the middle
Token bill spikes even though the prompt "looks" the same	Cached-token count in your billing logs drops to zero on repeat calls	Check for a dynamic value (timestamp, session ID, whitespace) that snuck into the "static" part of the prompt
Inconsistent output shape between runs on the same task	Occasionally valid JSON, occasionally malformed or oddly phrased	Set temperature = 0.0 for anything that needs a consistent, parseable shape, not for open-ended writing

Confirm it worked

Test the caching behavior directly, run a call with a large, stable context block twice in a row and watch the difference:

# First call: cold. Expect a few seconds of latency as the prefix gets cached.
# Second call, same prefix: hot. Latency should drop noticeably, and your
# provider's billing response will show a non-zero "cached tokens" count.

Then test determinism: run a structured-output prompt 20 times at temperature = 0.8, count how many fail to parse, then repeat at temperature = 0.0. The failure count at 0.0 should drop to essentially zero, if it doesn't, the problem isn't temperature, it's the prompt's structure itself.

Next: Model Parameters & Tuning for the rest of the API parameters worth knowing beyond temperature.