Memory Architecture

An AI agent that forgets everything when the conversation ends is a chatbot. An agent that remembers who you are, what you've done, and how your environment is set up can work across sessions without you repeating yourself. Hermes's memory system is built around a specific insight: most of what the agent needs to remember never changes, so it shouldn't cost tokens to recall.

The result is a two-tier design. Frozen blocks hold your identity, preferences, and environment details in static text that gets loaded once at session start and stays cached across every turn. Session search digs through past conversations with SQLite FTS5 when you need to find something specific. Together they give you persistent context that's both cheap and queryable, without a vector database or an embedding pipeline.

What you'll learn

• Frozen blocks (MEMORY.md + USER.md) carry ~3,600 characters of static context loaded once at session start, they're designed for prefix caching so they cost zero tokens after the first turn
• Session search uses SQLite FTS5, not embeddings, it's full-text, not semantic, and it runs without an auxiliary LLM call
• The memory tool supports batch operations: add, replace, and remove entries in a single atomic call, so you can free space and add content even when a single add would overflow the budget
• Three memory backends (built-in, Honcho, Mem0) let you choose between zero-cost local storage and external vector-based retrieval

The problem

You're three sessions deep with Hermes on a project. Every session starts the same way: you tell it which Python environment to use, where the config files live, which database it should connect to. The first few messages of every conversation are re-establishing context that never changes. Meanwhile, when you actually need to find something, "what was that sed command I used last week to fix the log rotation bug?", you're scrolling through terminal history hoping you didn't close the window.

This is two problems wearing the same name. The first is static context that should be free to recall. The second is dynamic recall that needs to be searchable but shouldn't dominate your context window. Hermes solves them with different mechanisms.

How the memory system is structured

Hermes's memory has three layers, not one:

Layer	What it holds	How it works	When it's used
Frozen blocks	Your identity, preferences, environment, anything that changes rarely or never	Two markdown files (MEMORY.md + USER.md) loaded as static system prompt blocks at session start. ~2,200 and ~1,375 characters respectively. They're placed early in the prompt so the model's prefix cache covers them, the tokens are paid once, cached, and reused across every turn	Every session, automatically
Session search	Past conversations, decisions, code snippets, debugging sessions	SQLite database with FTS5 full-text indexing. Three modes: discovery (keyword search), scroll (walk through a session), browse (list recent sessions). No LLM call needed, it's pure SQLite, effectively free	On demand, when you or Hermes need to find something from a past conversation
Memory tool	Mutable entries that change between sessions: project status, current tasks, active decisions	The memory tool reads and writes named entries into the frozen blocks. Supports batch operations (add + replace + remove in one call) so multi-step updates don't fail halfway	When Hermes explicitly writes to memory, or when you ask it to remember something

The key design decision: frozen blocks go in the system prompt, session search stays in a database. This means your identity doesn't compete with your conversation history for context window space. The model always knows who you are and how your environment works, and when it needs to recall a specific past interaction, it queries the database rather than stuffing everything into the prompt.

Options & when to use each

Memory backends

Backend	What it does	Cost	When to pick it
Built-in	Stores frozen blocks as flat files in ~/.hermes/memory/. Session search uses local SQLite FTS5. No external service needed	Free, zero latency, works offline	Most users, most of the time, this is the default
Honcho	External memory service with its own API. Adds semantic search on top of the frozen blocks	Requires a Honcho server (self-hosted or cloud). More setup, better recall for vague queries	When you need semantic search across memory entries, or when multiple Hermes instances need to share a memory store
Mem0	Vector-based memory with embeddings. Turns memory entries into vectors for similarity search	Requires a Mem0 API key. Highest setup cost, most flexible retrieval	When you need AI-native memory with vector similarity and automatic memory consolidation

For this course, we use the built-in backend. It's the default, it's free, and it covers the vast majority of real use cases. Switch to Honcho or Mem0 only when you have a specific need for semantic or shared memory, and only after you've confirmed the built-in backend isn't enough.

Check your current backend

Query the storage provider configuration to verify whether the system is using the default flat files or an external vector database.

hermes memory status

Output shows which backend is active, the character budget for frozen blocks, and whether memory is enabled. If you see provider: built-in, you're on the default. To change it:

hermes memory setup

Build it

Step 1: Configure your frozen blocks

The two frozen blocks live at ~/.hermes/memory/MEMORY.md and ~/.hermes/memory/USER.md. You can edit them directly or let Hermes write them through the memory tool.

Start by checking what's there now:

hermes memory status

If the blocks are empty (a fresh install), Hermes hasn't written anything yet. The blocks fill up over time as Hermes learns about you through conversation. But you can seed them manually for immediate results.

Create ~/.hermes/memory/USER.md:

## Identity
- Name: [your name]
- Role: infrastructure engineer / developer / researcher
- Location: [your timezone]

## Environment
- OS: Ubuntu 24.04 / macOS 15 / Windows 11 WSL2
- Shell: bash / zsh
- Editor: nvim / VS Code / cursor

## Preferences
- Python: always use uv, never pip
- Git: prefer rebase over merge
- Communication: direct, no hedging, technical depth preferred

Then create ~/.hermes/memory/MEMORY.md:

## Project: AI_BOOTCAMP
- Path: ~/projects/AI_BOOTCAMP
- Build: `npm run docs:dev` starts VitePress dev server
- Test: `npm test` runs the test suite
- Deploy: push to main triggers Cloudflare Pages

## Conventions
- Config files live in `config/`, not project root
- Secrets in `.env`, never committed
- Documentation in `docs/`, built with VitePress

## Active Tasks
- [current project status or tasks]

The character caps are 2,200 for MEMORY.md and 1,375 for USER.md. Hermes enforces these automatically, the memory tool won't let a write operation exceed the budget. If you're editing manually, keep an eye on size.

Step 2: Let Hermes write to memory

The more natural path: tell Hermes to remember something during a session, and it uses the memory tool to write it:

hermes
> Remember that I prefer uv over pip for Python, my timezone is America/Chicago,
  and the AI_BOOTCAMP project lives at ~/projects/AI_BOOTCAMP

Hermes calls the memory tool with an operations array that adds these entries to USER.md and MEMORY.md. Check what got written:

/memory

The /memory slash command shows the current contents of both frozen blocks. If you want to remove or update an entry later, you can ask Hermes directly or use the memory tool's batch operations.

Step 3: Use session search

Session search is FTS5 full-text, not semantic. This means you search with keywords, not natural language questions. The tool has three modes inferred from which arguments you pass:

Discovery mode, find sessions matching a keyword:

During a Hermes session, ask it:

Search my past sessions for "log rotation"

Hermes calls session_search(query="log rotation") and returns matching sessions with context snippets. This is the mode you'll use most often, it's like grep for your conversation history.

Scroll mode, walk through a specific session:

Show me more context around that log rotation session

Hermes passes session_id and around_message_id to zoom into the specific exchange.

Browse mode, see recent sessions:

What were my last few sessions about?

Hermes calls session_search() with no arguments, returning recent sessions chronologically.

The important thing: session search has zero LLM cost beyond the tool call itself. No embeddings, no summarization model, no auxiliary API call. It's SQLite running FTS5 against your local database.

Step 4: Understand batch operations

The memory tool accepts an operations array of add/replace/remove actions applied atomically. This matters when you're near the character budget:

Hermes wants to add a new entry, but MEMORY.md is at 2,100 characters.
A single add would overflow the 2,200-char budget.

Instead, it sends: operations = [
  { action: "remove", target: "Old completed task entry" },  // frees 150 chars
  { action: "add", target: "MEMORY.md", content: "New task entry" }  // uses 120 chars
]

Both operations succeed or neither does. No halfway state where the old
entry is gone but the new one failed to write.

This is a detail you probably won't think about until you hit the budget limit. When you do, the batch API means Hermes can rearrange memory to make room without losing data.

What goes wrong

Mistake	How you notice it	The fix
MEMORY.md is empty after several sessions	/memory shows nothing in the frozen blocks	Hermes only writes to memory when explicitly asked or when it determines information is worth persisting. It won't auto-populate memory from conversation alone. Ask it directly: "Remember that I use uv for Python and work in ~/projects/"
Session search returns nothing for a conversation you remember having	session_search(query="..." ) returns zero results	FTS5 is full-text, not semantic. If you search for "fixed the CI bug" but the conversation says "resolved the pipeline failure," FTS5 won't match. Use the actual words that appeared in the conversation, or browse recent sessions to find the right one
Frozen block writes fail silently with no error	/memory shows old content, not what you asked Hermes to remember	You're at the character budget (2,200 MEMORY.md / 1,375 USER.md). Hermes should tell you, but if it doesn't, check with hermes memory status to see current usage. Ask it to summarize and consolidate before writing new entries
Changing backends loses existing memory	After hermes memory setup and switching to Honcho, frozen blocks are empty	Memory doesn't auto-migrate between backends. Export important entries before switching: hermes memory status to see current blocks, copy them manually, then re-enter after the switch
Session search feels slow with many sessions	session_search() takes a noticeable pause	The FTS5 index covers all sessions by default. If you have thousands of sessions, the SQLite query takes longer. Prune old sessions: hermes sessions prune --older-than 90 removes sessions older than 90 days

Confirm it worked

Validate both the frozen block storage and dynamic session retrieval by writing a test entry and subsequently running a keyword search against the local index.

# 1. Check your memory configuration
hermes memory status

# 2. Write something to frozen memory through a Hermes session
hermes chat -q "Write to memory that my test command is 'npm test' and I work in ~/projects/AI_BOOTCAMP"

# 3. Verify it persisted
hermes chat -q "What is my test command and working directory?"

# 4. Search for a past session (even a fresh install should return the session you just ran)
hermes chat -q "Search my past sessions for 'test command'"

If step 3 returns npm test and ~/projects/AI_BOOTCAMP, frozen blocks are working. If step 4 returns a session match, session search is working. Both should work out of the box with the built-in backend.

For a deeper dive on memory systems in agents, including vector stores, retrieval strategies, and how memory fits into the broader agent architecture, see the flagship course's Giving Agents Memory lesson.

Next: Writing Skills by Hand, the SKILL.md format and creating your first reusable procedure.