What Prompt Engineering Is (and Isn't)

Prompt engineering is the discipline of designing inputs to get reliable, predictable outputs from language models. It's not magic, it's not "just be more specific," and it's not a dark art that only a few people can do. It's engineering: measurable, testable, and optimizable. This lesson defines the field and sets expectations for what you'll actually be able to do by the end of this course.

What you'll learn

• Prompt engineering is a repeatable, testable discipline, not folk wisdom or trial-and-error
• The core loop: write a prompt, test it against a benchmark, measure the results, refine
• Every technique in this course works because of how models process input, not because of superstition

The problem

Ask ten people how to write a good prompt and you'll get ten different answers: "Be more specific." "Give it a role." "Use markdown." "Don't use markdown." "Chain of thought always works." "Chain of thought is overrated." Most of this advice is folk wisdom, it worked once for someone, on one model, on one task, and they generalized. Without measurement, you can't tell which advice works and which is noise.

Prompt engineering treats the prompt as a testable artifact. You write a prompt. You test it against a set of known inputs with expected outputs. You measure accuracy, format compliance, and safety. You iterate until the numbers meet your threshold. That's it. Everything else, chain of thought, few-shot, structured output, is a technique you apply because the numbers say it helps, not because a blog post told you to.

Options & when to use each

Approach	What it is	When to use it
Intuitive prompting	Write a prompt, eyeball the output, tweak	Quick prototypes, one-off tasks
Systematic prompt engineering	Define a benchmark, measure against it, iterate	Production systems, anything with a quality bar
Automated optimization	Use DSPy or similar to optimize prompts programmatically	Large-scale systems, when manual iteration is too slow

Build it: the prompt engineering loop

A systematic prompt engineering loop relies on measuring performance against a benchmark and iterating based on failures.

# The core loop in code
def prompt_engineering_loop(task, benchmark, max_iterations=10):
    best_prompt = initial_prompt(task)
    best_score = 0

    for iteration in range(max_iterations):
        # Test current prompt against benchmark
        score = evaluate(best_prompt, benchmark)

        if score > best_score:
            best_score = score
            best_prompt = current_prompt

        if score >= TARGET_SCORE:
            break

        # Refine based on failures
        failures = get_failures(best_prompt, benchmark)
        current_prompt = refine(best_prompt, failures)

    return best_prompt, best_score

A benchmark is just a list of input/output pairs or criteria:

BENCHMARK = [
    {"input": "Where is my order #12345?", "expected_contains": ["tracking", "status"]},
    {"input": "I want a refund!", "expected_contains": ["order number", "verify"]},
    {"input": "How do I reset my password?", "expected_not_contains": ["I don't know"]},
]

What goes wrong

Mistake	How you notice it	The fix
Optimizing by eyeballing	Prompt "feels better" but metrics are unchanged or worse	Always measure. A benchmark of 20-50 test cases catches what your eyes miss
Over-optimizing to the benchmark	Prompt scores 100% on benchmark but fails on real queries	Periodically add real user queries to the benchmark. Don't let the benchmark become the only test
Changing too many things at once	Score changes but you don't know which change caused it	Change one thing per iteration. If you change the role, the format, and the examples all at once, you can't attribute the result

Confirm it worked

Always establish a quantitative baseline before refining your prompt to ensure changes actually improve performance.

# Establish a baseline, then improve
baseline_prompt = "Answer the user's question helpfully."
baseline_score = evaluate(baseline_prompt, BENCHMARK)
print(f"Baseline: {baseline_score}%")

improved_prompt = refine_prompt(baseline_prompt, BENCHMARK)
improved_score = evaluate(improved_prompt, BENCHMARK)
print(f"Improved: {improved_score}%")

assert improved_score >= baseline_score, "Improvement should not make things worse"

Next: Anatomy of a Prompt