Agentic AI Learning Portal

Appearance

Capstone: The Foreman ​

The Foreman is an on-call engineering assistant you page from Slack or Telegram. It researches an incident against your own runbooks, proposes a fix, tests that fix in a sandbox, remembers what worked for next time, and reports back, all from a phone message.

It's built from two cooperating agents, deliberately named after this course's own mascots:

  • Owl , the research/triage half. Reads your runbooks, decides what's actually wrong, writes a structured incident report. Slow, careful, doesn't touch anything.
  • Gnome , the execution half. Runs sandboxed diagnostics, tries fixes, and, the actual payoff of Day 3's skill-building lesson, writes down what worked as a reusable skill so it doesn't have to re-derive the fix next time.

Module map, where each piece comes from ​

LessonRole in The Foreman
Prompting & Context EngineeringSystem prompts for Owl and Gnome, Owl's prompt rewards caution, Gnome's rewards concrete action
Vibe Coding with AI CLIsGemini CLI used during development, a custom /audit:code slash command checks the codebase before every deploy
Calling AI APIs ReliablyResilient multi-provider calling, Owl fails over to a second provider if the primary is down (the exact scenario an on-call tool can't afford to skip)
Running Models Locally (optional course)Local inference fallback, a small local model handles triage if the network itself is the incident
Spec-Driven DesignThe tool-call contract between Owl, Gnome, and the LangGraph supervisor
Getting Reliable Structured DataOwl's incident report is a guaranteed-valid Pydantic object, not free text, the report schema is the contract the rest of the system depends on
Giving Agents MemoryRunbooks and past-incident postmortems, chunked and indexed; Owl retrieves the right section, not just a keyword match
Automating Workflows with EventsA FastAPI webhook bridge turns a real monitoring alert into a message OpenClaw posts to your chat
Running Agent Code SafelyGnome's diagnostic and fix scripts never touch the host directly, sandboxed per this lesson's isolation patterns
Deploying Your AgentDeploying The Foreman itself so it's running when you're not at your desk
Multi-Agent Workflows & State ManagementThe Owl/Gnome split is a LangGraph supervisor-worker graph with a human approval gate before any fix executes for real
Building Reusable SkillsGnome's self-improving skill loop, built on the real Hermes Agent pattern, not a metaphor, the actual SKILL.md mechanism
Exposing Agents to UsersOpenClaw is the whole point of contact, you never open a terminal to talk to The Foreman
Streaming Agent UIsAn SSE dashboard streams the graph's live reasoning when you are at your desk and want to watch
ObservabilityEvery incident run is traced, you can audit exactly what Owl retrieved and what Gnome executed after the fact

Prerequisites ​

Complete before starting:

bash
hermes doctor
openclaw gateway status

Part 1 , Owl: Research & Triage ​

Owl's only job is to turn "something's wrong" into a structured, evidence-backed incident report. It never executes anything.

Runbook retrieval ​

Index your runbooks the same way Giving Agents Memory indexes any corpus, sensible chunking, not fixed-size splitting, so a fix procedure doesn't get cut in half:

python
# owl/retrieval.py
from typing import TypedDict

class RunbookMatch(TypedDict):
    title: str
    section: str
    content: str
    score: float

def retrieve_runbook_context(query: str, k: int = 5) -> list[RunbookMatch]:
    """Hybrid search over the runbook vector store, reranked per the memory lesson's
    cross-encoder pattern before returning the top k."""
    ...  # see Giving Agents Memory for the full pgvector + reranking implementation

The incident report contract ​

This is the schema everything downstream depends on, Owl either produces a valid one or the graph halts and asks a human, it never passes through a best-guess:

python
# owl/schema.py
from pydantic import BaseModel, Field
from typing import Literal

class IncidentReport(BaseModel):
    severity: Literal["low", "medium", "high", "critical"]
    summary: str = Field(..., max_length=280)
    likely_cause: str
    matched_runbook: str | None
    proposed_fix: str
    requires_human_approval: bool
    confidence: float = Field(..., ge=0.0, le=1.0)

Resilient provider calling ​

Owl is the piece that has to work at 3am when one provider is degraded, this is the exact case Calling AI APIs Reliably's backoff/failover pattern exists for:

python
# owl/agent.py
from tenacity import retry, stop_after_attempt, wait_random_exponential

PROVIDERS = ["primary", "fallback"]  # e.g. Claude, then Gemini

@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(3))
def _call_provider(provider: str, prompt: str) -> str:
    ...  # per-provider client call, per Calling AI APIs Reliably

def triage(alert_text: str, runbook_context: list) -> "IncidentReport":
    for provider in PROVIDERS:
        try:
            raw = _call_provider(provider, build_prompt(alert_text, runbook_context))
            return IncidentReport.model_validate_json(raw)
        except Exception:
            continue
    raise RuntimeError("All providers exhausted, falling back to local inference")

Part 2 , Gnome: Execute & Fix ​

Gnome only runs once Owl's report exists and (for anything above low severity) a human has approved it. Its two jobs: run the fix safely, and remember it.

Sandboxed execution ​

Every diagnostic or fix script runs the same way Running Agent Code Safely teaches, no host access, no network during validation, resource-capped:

bash
docker run --rm -m 128m --network none \
  --runtime=runsc \
  -v "$(pwd)/fix_script.py:/app/fix_script.py:ro" \
  python:3.11-slim python /app/fix_script.py

Real self-improving skills ​

This is the actual Hermes Agent mechanism, not a simplified version of it. Once a fix works, Gnome writes it down exactly the way Building Reusable Skills's skill_manage tool does:

text
hermes> The disk-space alert fix worked. Save this as a reusable skill called 'clear-log-overflow'.

Produces ~/.hermes/skills/clear-log-overflow/SKILL.md:

markdown
---
name: clear-log-overflow
description: Clears rotated logs when disk usage crosses 90% on app servers.
metadata:
  hermes:
    tags: [ops, disk, incident]
---
# Clear Log Overflow

## When to Use
Disk usage alert fires above 90% and `/var/log` is the largest consumer.

## Procedure
1. Run `du -sh /var/log/* | sort -rh | head -5` to confirm log growth is the cause.
2. Run `find /var/log -name "*.gz" -mtime +14 -delete` inside the sandbox.
3. Re-check disk usage; if still above 90%, escalate to a human, do not resize volumes automatically.

## Verification
Disk usage drops below 80% after cleanup.

The next time this exact alert fires, Gnome doesn't re-derive the fix , openclaw skills list (or the Hermes equivalent) already has it, and the whole incident resolves in the time it takes to run the skill.

Part 3 , Wiring Owl and Gnome ​

A LangGraph supervisor routes between Owl and Gnome, with a hard-coded human approval gate before Gnome ever runs a fix for real:

python
# graph.py
from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.checkpoint.memory import MemorySaver

class ForemanState(TypedDict):
    messages: Annotated[list, add_messages]
    report: dict | None
    approved: bool
    next_step: str

def owl_node(state: ForemanState):
    report = triage(state["messages"][-1].content, retrieve_runbook_context(...))
    needs_approval = report.severity in ("high", "critical") or not report.requires_human_approval is False
    return {"report": report.model_dump(), "next_step": "await_approval" if needs_approval else "gnome"}

def gnome_node(state: ForemanState):
    result = execute_fix(state["report"])
    return {"messages": [{"role": "assistant", "content": result}], "next_step": "end"}

workflow = StateGraph(ForemanState)
workflow.add_node("owl", owl_node)
workflow.add_node("gnome", gnome_node)
workflow.add_edge(START, "owl")
workflow.add_conditional_edges("owl", lambda s: s["next_step"], {
    "gnome": "gnome",
    "await_approval": END,  # graph pauses here; resumes on human approval via checkpointer
})
workflow.add_edge("gnome", END)

app = workflow.compile(checkpointer=MemorySaver())

Part 4 , Exposing The Foreman via OpenClaw ​

The Gateway is the only interface a human touches. Install and connect it the real way (not the fictional git-clone/@skill-decorator flow from an earlier version of this doc):

bash
curl -fsSL https://openclaw.ai/install.sh | bash
openclaw onboard --install-daemon
openclaw gateway status   # confirms port 18789

The Foreman itself is exposed as a markdown skill , openclaw skills list should show it once this is in place:

~/.openclaw/workspace/skills/
└── foreman/
    └── SKILL.md

Save the following markdown configuration as your SKILL.md inside that directory. This defines the system instructions and task routing for the Foreman agent when it receives incoming alert messages:

markdown
---
name: foreman
description: On-call engineering assistant, triages alerts, proposes and sandboxes fixes.
---
# The Foreman

When the user reports an incident, or a message arrives prefixed "ALERT:", run:

​```bash
python3 ~/AI_BOOTCAMP/labs/foreman/run_graph.py "$MESSAGE"
​```

Relay the returned incident report verbatim. If `requires_human_approval` is true,
ask explicitly for a yes/no before invoking Gnome's fix step.

Part 5 , Triggering from a real alert ​

A small FastAPI bridge is the Automating Workflows with Events payoff, it turns an actual monitoring webhook into a message in your chat, using the same async-bridge pattern that lesson teaches:

python
# webhook_bridge.py
from fastapi import FastAPI, Request
import httpx

app = FastAPI()

@app.post("/alerts/incoming")
async def incoming_alert(request: Request):
    payload = await request.json()
    message = f"ALERT: {payload.get('summary', 'unspecified incident')}"
    async with httpx.AsyncClient() as client:
        await client.post(
            "http://localhost:18789/api/messages",
            json={"channel": "ops-oncall", "text": message},
        )
    return {"status": "forwarded"}

Part 6 , Watching it think ​

When you're at your desk, an SSE endpoint mirrors the LangGraph's step events so you can watch Owl retrieve and Gnome execute in real time, using the same async-generator pattern from Streaming Agent UIs:

python
# dashboard_stream.py
from fastapi.responses import StreamingResponse

async def stream_graph_events(thread_id: str):
    async for event in app.astream_events({"messages": [...]}, {"configurable": {"thread_id": thread_id}}):
        yield f"event: {event['event']}\ndata: {event['data']}\n\n"

Part 7 , Observability ​

Every run, Owl's retrieval, the provider that answered, Gnome's sandboxed execution, the skill it wrote, gets an OTel span, so a bad incident response is auditable after the fact, not just in the moment:

python
from langfuse.decorators import observe

@observe(name="foreman.incident")
def handle_incident(alert_text: str):
    return app.invoke({"messages": [{"role": "user", "content": alert_text}]})

Runtime Verification ​

To verify that the entire system is properly wired and integrated, run the following status checks and send a mock alert payload to validate the webhook ingestion pipeline:

bash
# Gateway live and skill registered
openclaw gateway status
openclaw skills list | grep foreman

# Send a test alert through the webhook bridge
curl -X POST http://localhost:8000/alerts/incoming \
  -H "Content-Type: application/json" \
  -d '{"summary": "disk usage 94% on app-server-3"}'

# Confirm it reached your chat, then reply to approve the fix if prompted

Confirm the full loop: alert → chat message → Owl's report → your approval → Gnome's sandboxed fix → a new or reused SKILL.md → a summary back in chat.

Failure mode reference:

SymptomCauseFix
No message arrives in chatWebhook bridge can't reach the GatewayConfirm openclaw gateway status shows port 18789, check the bridge's POST URL
Owl's report fails schema validationProvider returned free text instead of the contracted JSONTighten the prompt's format instructions per Getting Reliable Structured Data; add a repair/retry pass
Gnome's fix never runsGraph is paused at the approval gateCheck approved state in the checkpointer; approval must resume the graph explicitly
Skill doesn't apply next timeSKILL.md frontmatter malformed, or gated by an unmet metadata.openclaw.requiresValidate frontmatter; see Exposing Agents to Users's What Goes Wrong section for the exact pattern
All providers fail in OwlBoth API providers down simultaneouslyConfirm the local-inference fallback path is wired in, not just logged as a TODO