Running Models Locally (Optional)

Most agents in this course call a hosted API, Claude or Gemini, and that's the right default for almost every project. This page covers the exception: when it's worth running a model on your own machine instead, and the fastest way to do it.

What you'll learn

• The three real reasons to run a model locally (and why "it's free" usually isn't one of them)
• Ollama as the fastest path from zero to a running model
• What to expect from a small local model vs. a hosted one

1. The problem

Local inference sounds appealing, no API cost, no network latency, full control. In practice, a consumer GPU (or CPU) runs a much smaller, less capable model than Claude or Gemini, and "no API cost" often loses to the electricity, hardware, and your own time spent babysitting it. Local inference is worth reaching for in specific situations, not as a default.

2. When it's actually worth it

Situation	Why local wins here
Data can't leave your machine or network	Compliance/privacy requirements that rule out sending data to a third-party API
You're offline, or building for an offline environment	No API means no network dependency
You're prototyping prompt structure cheaply, at high volume	Free, unlimited local calls while you're iterating, before you're ready to spend on hosted API calls
You want to learn how inference actually works	Running it yourself makes the mechanics concrete

Outside these cases, building a product for real users, needing strong reasoning, wanting the least amount of infrastructure to operate, a hosted API is the better default. This course's capstone and every module past this one assume a hosted API.

3. Build it

Ollama is the fastest path from nothing to a running local model:

curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma2:2b

That's it , ollama run downloads the model on first use and starts an interactive chat. To call it from code instead of the CLI:

import ollama

response = ollama.chat(
    model="gemma2:2b",
    messages=[{"role": "user", "content": "Summarize this in one sentence: ..."}],
)
print(response["message"]["content"])

Same messages shape as a hosted API, so swapping between local and hosted during development is a one-line change.

See the Ollama Cheat Sheet for common commands (listing models, removing them, checking what's running).

4. What goes wrong

Mistake	How you notice it	The fix
Picking a model too large for your hardware	Extremely slow responses, or Ollama fails to load it	Start with a small model (gemma2:2b, llama3.2:3b) and only go bigger if quality is actually the bottleneck
Expecting hosted-API-level reasoning	Local model gives noticeably worse answers on complex prompts	This is expected, small local models are for the situations in the table above, not a drop-in Claude/Gemini replacement
Leaving Ollama running indefinitely	It quietly holds memory/GPU even when idle	ollama stop <model> when you're done, or let its idle timeout unload it

5. Confirm it worked

Verify your installation by listing available models and running a quick test prompt to ensure the engine is responding.

ollama list
# should show the model you pulled

ollama run gemma2:2b "Say hello in exactly 3 words"
# should return a short, coherent response

Next Step: proceed to Ollama Cheat Sheet for the commands you'll reach for day to day.