Prompt Versioning & CI/CD

Prompts are code. They have versions, they get tested, they get deployed, and bad ones get rolled back. This lesson covers treating prompts as software artifacts: storing them in version control, testing them against benchmarks, and deploying them through CI/CD pipelines.

What you'll learn

• Store prompts as files in version control , .md for documentation, .yaml or .py for programmatic use
• Test prompts against a benchmark: a set of inputs with expected outputs or criteria
• CI/CD runs your prompt tests on every change, a failing test blocks deployment

Build it

Step 1: Store prompts as versioned files

Storing prompts as versioned configuration files establishes a reliable source of truth. This approach decouples prompt tuning from application code and enables clear release tracking.

prompts/
  customer-support/
    v1.yaml
    v2.yaml
    current -> v2.yaml
  code-review/
    v1.yaml
    v2.yaml
    current -> v1.yaml

You can use YAML files to tightly couple the system instructions with operational parameters like temperature and max tokens. This encapsulates the entire execution profile of a prompt within a single deployable asset.

# prompts/customer-support/v2.yaml
version: "2.0.0"
model: claude-sonnet-4-6
system: |
  You are a customer support agent for Acme Corp.
  Be helpful, concise, and professional.
  Never issue refunds without verifying the order number.
parameters:
  temperature: 0.3
  max_tokens: 512

Step 2: Test prompts against a benchmark

Validating prompts against a standardized benchmark suite ensures changes do not introduce regressions. Building robust test assertions confirms that the model consistently meets expected behavioral criteria.

# tests/test_customer_support.py
import pytest
from prompts import load_prompt

BENCHMARK = [
    {"input": "Where is my order #12345?", "expected_contains": ["order", "tracking"]},
    {"input": "I want a refund!", "expected_contains": ["order number", "verify"]},
    {"input": "What's 2+2?", "expected_not_contains": ["I don't know"]},
]

def test_prompt_v2():
    prompt = load_prompt("customer-support", "v2")
    for case in BENCHMARK:
        output = call_llm(prompt.format(input=case["input"]))
        for phrase in case.get("expected_contains", []):
            assert phrase.lower() in output.lower(), f"Missing '{phrase}' in response to '{case['input']}'"
        for phrase in case.get("expected_not_contains", []):
            assert phrase.lower() not in output.lower()

Step 3: CI/CD pipeline

Integrating prompt testing into your CI/CD pipeline automates quality assurance. Triggering evaluation jobs on pull requests prevents flawed prompts from reaching production systems.

# .github/workflows/prompt-tests.yml
name: Prompt Tests
on:
  pull_request:
    paths: ['prompts/**']
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest tests/prompts/ -v
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Step 4: Rollback

Treating prompt deployments as symlinks or environment configurations enables instantaneous rollbacks. This ensures immediate recovery if a newly deployed prompt underperforms in a live environment.

# Deploy v2
ln -sf v2.yaml prompts/customer-support/current

# If tests fail in production, rollback
ln -sf v1.yaml prompts/customer-support/current

What goes wrong

Mistake	How you notice it	The fix
Prompt change breaks production	User complaints, error rate spikes	Always test against benchmark before deploying. Keep previous version for instant rollback
Benchmark tests are too narrow	Tests pass but real users get bad responses	Add real user queries to the benchmark. Periodically review and expand test cases
API costs from CI tests	High API bill from automated testing	Cache LLM responses for known inputs. Use cheaper models for CI testing. Cap test budget
Prompt drift between environments	Dev prompt works, production prompt doesn't	Use the same prompt loading mechanism everywhere. Never inline prompts in code

Confirm it worked

Run this validation script to simulate a prompt testing workflow. This confirms that your testing framework successfully evaluates and passes prompt configurations.

# 1. Create a prompt file
echo "system: You are a helpful assistant." > prompts/test/current.yaml

# 2. Write a test
echo "def test_hello(): output = call_llm(load_prompt('test')); assert output" > tests/test_prompt.py

# 3. Run tests
pytest tests/test_prompt.py -v

Next: Prompt Engineering at Scale