Self-Consistency & Verification

One model call gives you one answer. Five calls give you five answers. If four of them agree, you can be more confident than if you'd only asked once. Self-consistency runs the same prompt multiple times with temperature > 0 and takes the majority answer. This lesson covers the technique, when it's worth the cost, and how to implement it.

What you'll learn

• Self-consistency: run the same prompt N times (usually 3-7), take the most common answer
• Only works with temperature > 0, deterministic output (temp=0) gives the same answer every time
• Best for tasks with a clear right answer: math, classification, factual QA. Not for creative or open-ended tasks

Build it

Implementing self-consistency involves running the prompt multiple times and aggregating the results to find the majority vote.

from collections import Counter

def self_consistent_prompt(prompt, n=5, temperature=0.7):
    answers = []
    for _ in range(n):
        response = call_llm(prompt, temperature=temperature)
        answers.append(extract_answer(response))

    # Majority vote
    counts = Counter(answers)
    most_common = counts.most_common(1)[0]

    return most_common[0], most_common[1] / n  # answer and confidence

With Chain-of-Thought

Combining self-consistency with Chain-of-Thought reasoning significantly improves reliability on complex tasks with definitive answers.

prompt = """Solve this math problem. Show your work, then give your final answer
as "Answer: [number]".

Problem: {question}"""

answer, confidence = self_consistent_prompt(prompt, n=5)
print(f"Answer: {answer} (confidence: {confidence:.0%})")
# "Answer: 42 (confidence: 80%)" means 4 of 5 runs gave 42

When to use

Task type	Use self-consistency?	Why
Math problems	Yes	Clear right answer, CoT improves accuracy further
Classification	Yes	Multiple runs catch inconsistent classifications
Factual QA	Yes	Hallucinations vary; consensus filters them
Creative writing	No	No right answer to vote on
Summarization	Maybe	Can help catch missed points, but expensive

What goes wrong

Mistake	How you notice it	The fix
Temp too low	All answers identical, no benefit	Set temperature >= 0.5. At temp=0, every run produces the same output
Temp too high	Answers vary wildly, no consensus	Lower temperature to 0.3-0.5. You want diversity, not chaos
Cost	5x API calls per query	Only use for high-value tasks. Start with n=3 and increase only if needed
Wrong answers agree	Majority is confidently wrong	Self-consistency amplifies patterns, not correctness. Verify against ground truth

Next: Meta-Prompting