Local LLMs workshop
0 / 15

Hands-on workshop

Local LLMs,
from ollama run to a gateway

Run a model on your laptop, point a coding agent at it - then put a gateway in front that fails over to the cloud, with a spending cap. Every command is below: click copy, paste, check the expect line.

Step 1opencode→Ollama
Step 2opencode→LiteLLM→OllamaorAnthropic / OpenAI

Pick your setup

Commands on this page fill themselves in. Saved in your browser.

Your model
Your system
Your cloud key, for Step 2 (bring your own)
Before we start
Step 1

Basics

You end with a model answering on your own machine, driven by a coding agent.

1.1

Install Ollama

Ollama is the runtime: it downloads models and serves them over HTTP. Think dockerd for models.

Install Ollama
brew install ollama
ollama --version

A version line like ollama version is 0.x.y. Command not found? Open a new terminal tab.

1.2

Start the server

Ollama is a daemon. We run it by hand, in a terminal tab of its own, so you can see it work, and kill it later on purpose.

From now on you have two terminal tabs. Terminal A = Ollama (leave it running). Terminal B = everything else. Open a new tab for B after this step.
Start the server, in its own terminal tab (Terminal A)
# quit the Ollama menu-bar app and any brew service first, so the port is free
brew services stop ollama 2>/dev/null
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

It prints logs and keeps running. Leave this tab open. Everything else happens in a new tab (Terminal B).

1.3

Download your model

Pulled once, stored locally. The model you picked at the top is filled in for you.

Download your model
ollama pull qwen3.5:4b
ollama list
ollama ps        # after your first prompt: the REAL memory it uses

ollama list shows qwen3.5:4b with its size. Interrupted? Re-run, it resumes.

Slow Wi-Fi? Start the download and read the side-quests below while it runs.

Will it fit? Why the bits matter

A model’s weights take roughly parameters × bits ÷ 8 of memory. The same 9B model is a very different size at different precisions:

PrecisionGB per billion params9B model4B model
16-bit (full)~2.0~18 GB~8 GB
8-bit~1.0~9 GB~4 GB
4-bit (Ollama’s default)~0.5 to 0.6~5 GB~2.5 GB

Those are the weights only. On top come the KV cache (it grows with the context window and Ollama reserves all of it up front), plus runtime and OS overhead. So the rule of thumb is: weights + KV cache + overhead ≤ about ¾ of your RAM. The “¾” is the margin for the rest of your Mac; it does not replace counting the bits.

Don’t guess: after your first prompt, ollama ps prints what the model really takes, context included. The download sizes in the picker (~3.5 GB and ~6 to 7 GB) are larger than the table because they include extra files.

1.4

Talk to it: two ways

The terminal chat is for fun. The HTTP call is the one that matters: Ollama speaks the OpenAI API shape, and everything after this relies on that.

A. Talk to it in the terminal
ollama run qwen3.5:4b
# type a question, then /bye to leave

A sensible answer. The first reply is slow: the model is loading into RAM.

B. Talk to it over HTTP (OpenAI-compatible API)
curl -s http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:4b",
    "messages": [{"role": "user", "content": "Say hi in 5 words"}]
  }' | python3 -m json.tool

JSON with choices[0].message.content. This exact shape is what opencode, and later the gateway, speaks.

Ollama’s own API shape optional
Ollama’s own API shape (optional)
curl -s http://localhost:11434/api/chat \
  -d '{
    "model": "qwen3.5:4b",
    "messages": [{"role": "user", "content": "Explain a KV cache in 3 sentences."}],
    "stream": false
  }' | python3 -m json.tool

Same idea, different envelope: the answer is in message.content. Step 2 uses this shape too.

1.5

Install opencode

opencode is a coding agent in your terminal. It will be our client: first aimed at Ollama, later at the gateway.

Install opencode
curl -fsSL https://opencode.ai/install | bash
# or: brew install sst/tap/opencode
opencode --version

A version string. Open a new terminal tab if the command isn’t found.

1.6

Point opencode at Ollama

The whole integration is one baseURL: the same address you just curled. Its "model" must match a key under models, with the ollama/ prefix.

Point opencode at Ollama
mkdir -p ~/.config/opencode
[ -f ~/.config/opencode/opencode.json ] && cp ~/.config/opencode/opencode.json ~/.config/opencode/opencode.json.bak
cat > ~/.config/opencode/opencode.json <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": {
        "baseURL": "http://localhost:11434/v1",
        "apiKey": "ollama"
      },
      "models": {
        "qwen3.5:4b": {
          "name": "Qwen3.5 4B (local)",
          "limit": { "context": 32768, "output": 8192 }
        }
      }
    }
  },
  "model": "ollama/qwen3.5:4b"
}
EOF

No output. Your old config (if any) is saved as opencode.json.bak. The key line is baseURL: it’s the URL you just curled.

1.7

Do real work, then unplug

The payoff of Step 1. Ask for something beautiful, then switch the network off and ask for the same thing in another language.

Run it: ask for something beautiful
mkdir -p ~/scratch/my-project && cd ~/scratch/my-project
opencode
# in the TUI: /models -> pick ollama/qwen3.5:4b, then ask:
#   1) show me a beautiful piece of code in Elixir
# now turn OFF Wi-Fi, and ask:
#   2) now show me the same thing in TypeScript

Elixir first, then the same idea in TypeScript with the network off. Both answers come from your laptop.

Then push it: ask for a multi-file refactor. That is the honest limit of a small local model, and the reason Step 2 exists.

Side-quest: Ollama is Docker for models
DockerOllama
docker pull nginxollama pull qwen3.5:4b
docker run nginxollama run qwen3.5:4b
docker psollama ps
DockerfileModelfile
Docker Hubollama.com/library

Same mental model, different payload. One real difference: a model isn’t a process you kill. It’s weights sitting in RAM, evicted when idle. That’s why “why is my Mac using 12 GB?” happens.

Side-quest: faster on a Mac with oMLX presenter demo

Ollama runs llama.cpp underneath: general-purpose and easy. oMLX is a native Apple-Silicon server built around Metal kernels, tuned for M-series chips, at the price of a fussier setup (on some versions you compile the Metal kernel yourself).

It serves the same OpenAI-compatible API, so it would slot in exactly where Ollama sits, and the gateway in Step 2 wouldn’t notice. That’s the point of the abstraction. Try it as homework.

Step 2

Abstractions

You end with a gateway: one endpoint, a local model first, a fast group that fails over to the cloud with a spending cap, and your client none the wiser.

2.1

Create the gateway’s secrets

LiteLLM needs a master key (admin access) and a salt key (encrypts stored keys). This script generates both. Your own cloud key comes later, in 2.4.

Before this step: Docker Desktop must be running (whale icon in the menu bar). For 2.4, bring an API key from Anthropic or OpenAI: we don’t hand out keys.
Create the gateway folder and its secrets
mkdir -p ~/workshop-gateway && cd ~/workshop-gateway
printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' \
  "$(openssl rand -hex 24)" "$(openssl rand -hex 24)" > .env
chmod 600 .env
echo ".env" > .gitignore

No output. Master and salt keys are generated for you. .env never goes in git. Your own cloud key joins it in 2.4.

2.2

Describe the gateway

Two files: docker-compose.yml says what runs (LiteLLM + Postgres for keys, logs and budgets). config.yaml is the routing table. We start with one local model.

docker-compose.yml: LiteLLM + Postgres
cd ~/workshop-gateway
cat > docker-compose.yml <<'EOF'
services:
  litellm:
    image: docker.litellm.ai/berriai/litellm:main-stable
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    env_file:
      - .env                    # master key, salt key, and (from 2.4) your cloud key
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:
EOF

Creates the file. host.docker.internal is how the container reaches your machine, where Ollama runs.

config.yaml v1: one local model
cd ~/workshop-gateway
cat > config.yaml <<'EOF'
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  salt_key: os.environ/LITELLM_SALT_KEY
  database_url: postgresql://litellm:litellm@db:5432/litellm

model_list:
  - model_name: qwen3.5:4b                   # the name YOUR clients ask for
    litellm_params:
      model: ollama_chat/qwen3.5:4b          # the provider’s name: "ollama_chat/" = Ollama’s native API
      api_base: http://host.docker.internal:11434

litellm_settings:
  drop_params: true
EOF

The whole abstraction in one line each: model_name is yours, model is the provider’s. Here they happen to share a name, but they don’t have to.

Read these two lines twice: model_name is the name your clients ask for; model: ollama_chat/… is the provider’s name (ollama_chat/ = “speak Ollama’s native API to api_base”). Rename the model later and nothing downstream notices.
2.3

Start it and ask it something

Ollama is still running in Terminal A. The container reaches it through host.docker.internal.

Start the gateway and wait until it’s ready
cd ~/workshop-gateway
docker compose up -d
for i in $(seq 1 60); do
  curl -sf http://localhost:4000/health/liveliness >/dev/null && { echo "gateway ready"; break; }
  sleep 2
done
docker compose ps

gateway ready, and litellm + db both up. First start takes ~1 min (database migrations). Nothing after 2 min? docker compose logs litellm.

Checkpoint: ask the gateway
cd ~/workshop-gateway && set -a && . ./.env && set +a
curl -s http://localhost:4000/v1/models \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" | python3 -m json.tool

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.5:4b", "messages": [{"role": "user", "content": "Say hi in 5 words"}]}' \
  | python3 -m json.tool

First call lists qwen3.5:4b. Second returns the same OpenAI-shaped JSON as before, but this time it went through port 4000.

2.4

Add the “fast” group: local first, cloud failover, with a cost cap

A group named fast with two deployments: your local model first, and a cloud model only when local fails (connection error, timeout, 5xx). Silent failover is the danger, so the same file puts a hard $2/day ceiling on the cloud provider.

Your key, your choice. Pick Anthropic or OpenAI on either block: the choice follows you down the page. Only the cloud entry differs.
Add YOUR cloud key to .env (hidden prompt)
cd ~/workshop-gateway
printf 'Anthropic API key (hidden): '; read -rs K; echo
printf 'ANTHROPIC_API_KEY=%s\n' "$K" >> .env
unset K
grep -c _API_KEY .env

It asks for your key (nothing shows as you paste) and prints 1. No key to hand? Pair up with someone who has one, or skip 2.4: everything before it still works.

config.yaml v2: the “fast” group, local first, cloud failover, with a cost cap
cd ~/workshop-gateway
cat > config.yaml <<'EOF'
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  salt_key: os.environ/LITELLM_SALT_KEY
  database_url: postgresql://litellm:litellm@db:5432/litellm

model_list:
  # 1) your local model, by itself. If Ollama is down this fails - on purpose.
  - model_name: qwen3.5:4b
    litellm_params:
      model: ollama_chat/qwen3.5:4b
      api_base: http://host.docker.internal:11434
    model_info:
      id: local-direct

  # 2) the "fast" group: two deployments, ONE name. Lowest order goes first.
  - model_name: fast
    litellm_params:
      model: ollama_chat/qwen3.5:4b
      api_base: http://host.docker.internal:11434
      order: 1                              # local first
    model_info:
      id: local
  - model_name: fast
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
      order: 2                              # cloud only when local fails
    model_info:
      id: cloud

litellm_settings:
  drop_params: true

router_settings:
  num_retries: 1                            # fail over fast
  enable_pre_call_checks: true              # needed for order-based routing
  provider_budget_config:
    anthropic:
      budget_limit: 2                       # USD ...
      time_period: 1d                       # ... per day. Hard stop.
EOF
docker compose up -d --force-recreate litellm
for i in $(seq 1 60); do
  curl -sf http://localhost:4000/health/liveliness >/dev/null && { echo "gateway ready"; break; }
  sleep 2
done

gateway ready again. Two names exist now: qwen3.5:4b (local only) and fast (local, then cloud). Use --force-recreate, not restart: only a recreate re-reads .env.

Failover fires on server-side failures, not on 4xx. A bad request is your bug; retrying it on a paid model just bills you twice for the same error. And: a failover without a cap turns an outage into an invoice. Note too that {{MODEL}} on its own has no failover: only fast does.
Going further: mask PII before a prompt leaves reference

Failover means prompts can now leave your machine. LiteLLM guardrails run on every request: they can replace emails, card numbers and names with placeholders before the model sees them, and put the real values back in the answer.

Mask PII before it leaves (reference, not run in the workshop)
# 1) Presidio runs as two small containers next to the gateway
docker run -d --name presidio-analyzer   -p 5002:3000 mcr.microsoft.com/presidio-analyzer:latest
docker run -d --name presidio-anonymizer -p 5001:3000 mcr.microsoft.com/presidio-anonymizer:latest
# the gateway container needs, in .env:
#   PRESIDIO_ANALYZER_API_BASE=http://host.docker.internal:5002
#   PRESIDIO_ANONYMIZER_API_BASE=http://host.docker.internal:5001

# 2) add to config.yaml
guardrails:
  - guardrail_name: "pii"
    litellm_params:
      guardrail: presidio
      mode: "pre_call"              # runs before the model sees the prompt
      pii_entities_config:
        PERSON: "MASK"              # Jane Doe -> <PERSON>
        EMAIL_ADDRESS: "MASK"       # jane@acme.com -> <EMAIL_ADDRESS>
        CREDIT_CARD: "BLOCK"        # reject the request outright
      output_parse_pii: true        # put the real values back in the reply

# 3) ask for it per request
#   "guardrails": ["pii"]

Not tested in this workshop build: check the LiteLLM guardrails docs for the current option names before relying on it.

Going further: route by complexity optional

Why pay for the cloud on “hi”? LiteLLM’s complexity router is a model name (auto) that scores each prompt and picks a tier: simple and medium stay on your local model, complex and reasoning go to the cloud. The scoring is rule-based, so it adds no extra model call. It is a heuristic: use keyword_tier_rules to force the words you care about.

Route by complexity: simple stays local, hard goes to the cloud
# add under model_list: in config.yaml, same indentation as the other entries
  - model_name: cloud-smart
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
    model_info:
      id: cloud-smart

  - model_name: auto
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        context_compaction: false           # needed: LiteLLM has no price entry for local models
        keyword_tier_rules:
          - keywords: ["refactor", "migration"]
            tier: COMPLEX                   # these words always escalate
        tiers:
          SIMPLE: qwen3.5:4b
          MEDIUM: qwen3.5:4b
          COMPLEX: cloud-smart
          REASONING: cloud-smart
      complexity_router_default_model: qwen3.5:4b
# then: docker compose restart litellm

No output. The router is rule-based (no extra model call). Tune it with keyword_tier_rules; the cloud tiers count against the same provider cap.

See where each question lands
cd ~/workshop-gateway && set -a && . ./.env && set +a
for q in "hi" "What is a KV cache?" "Refactor this function to be async and add tests: def f(x): return requests.get(x).json()"; do
  echo "Q: $q"
  curl -si http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
    -H "Content-Type: application/json" \
    -d "$(python3 -c 'import json,sys; print(json.dumps({"model":"auto","messages":[{"role":"user","content":sys.argv[1]}],"max_tokens":200}))' "$q")" \
    | grep -i '^x-litellm-model-id'
done

The first two questions print local-direct, the refactor one prints cloud-smart (the word “refactor” is a keyword rule). The cloud one is billed to your key.

2.5

Break it on purpose

The most convincing four minutes of the day: nobody’s client changes, nothing is redeployed, the work just keeps flowing.

  1. Run the loop below in Terminal B.
  2. Switch to Terminal A and press Ctrl+C to stop Ollama.
  3. Watch the first word of each line change from local to cloud, and the loop keep answering.
  4. Restart Ollama in Terminal A (command from 1.2). Watch traffic return home.
Steady traffic to “fast”: run in Terminal B, leave it looping
cd ~/workshop-gateway && set -a && . ./.env && set +a
while true; do
  curl -si http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"fast","messages":[{"role":"user","content":"Reply with one word: pong"}],"max_tokens":200}' \
  | python3 -c '
import sys, json
raw = sys.stdin.read()
head, _, body = raw.partition("\r\n\r\n")
who = "?"
for line in head.split("\r\n"):
    if line.lower().startswith("x-litellm-model-id:"):
        who = line.split(":", 1)[1].strip()
try:
    r = json.loads(body)
except ValueError:
    print("no JSON back - is the gateway up?"); sys.exit()
if "choices" in r:
    print(who, "|", (r["choices"][0]["message"].get("content") or "...").strip()[:50].replace("\n", " "))
else:
    print("ERROR |", str(r)[:90])
'
  sleep 2
done

One line every ~2 s: local | .... Now Ctrl-C in Terminal A (Ollama) and watch it flip to cloud | ... while the loop keeps answering. Re-run the Ollama command from 1.2 and it flips back.

Those cloud requests were billed by your provider. Hold that thought: the next step shows where.

Going further: prove the cap bites optional
Prove the cap bites
cd ~/workshop-gateway
sed -i.bak 's/budget_limit: 2 /budget_limit: 0.0005 /' config.yaml
docker compose restart litellm
# stop Ollama (Terminal A), then keep calling "fast":
# after a few cloud calls the cap is hit and requests fail instead of billing you.

After a handful of calls, an error like No deployments available for selected model instead of an answer: the only deployment left is the capped cloud one. Restore with mv config.yaml.bak config.yaml and restart.

2.6

Tour the playground UI

Your debugging surface for “why did that request go where it went?”

Get your master key for the UI
grep ^LITELLM_MASTER_KEY ~/workshop-gateway/.env | cut -d= -f2 | pbcopy
echo "copied to clipboard"

Open localhost:4000/ui, username admin, password = that key.

  • Playground / Test Key: pick fast or {{MODEL}} and chat in the browser. Compare.
  • Logs: every request, which deployment answered, latency, cost. Find the failover requests from 2.5.
  • Keys: where per-person keys with budgets live (see the going-further box below).
Going further: a key with its own budget optional
Give someone their own key with a budget
cd ~/workshop-gateway && set -a && . ./.env && set +a
curl -s http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models": ["fast", "qwen3.5:4b"], "max_budget": 2, "budget_duration": "1d"}' \
  | python3 -m json.tool

A key starting with sk-. Use it instead of the master key in opencode; watch it on the UI’s Keys page.

2.7

Repoint opencode at the gateway

Four values change; the client binary doesn’t. The model’s identity moves into the gateway, exactly where you want it.

Step 1Step 2
providerollamaworkshop
baseURLlocalhost:11434/v1localhost:4000/v1
apiKeydummy{env:OPENCODE_API_KEY}
modelollama/qwen3.5:4bworkshop/fast
Repoint opencode at the gateway
cat > ~/.config/opencode/opencode.json <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "workshop": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Workshop Gateway",
      "options": {
        "baseURL": "http://localhost:4000/v1",
        "apiKey": "{env:OPENCODE_API_KEY}"
      },
      "models": {
        "fast": {
          "name": "fast (local first, cloud failover)",
          "limit": { "context": 32768, "output": 8192 }
        },
        "qwen3.5:4b": {
          "name": "Qwen3.5 4B (local only)",
          "limit": { "context": 32768, "output": 8192 }
        }
      }
    }
  },
  "model": "workshop/fast"
}
EOF

Four things changed from Step 1: provider name, baseURL (:11434 to :4000), key, model name.

Launch opencode through the gateway
cd ~/workshop-gateway && set -a && . ./.env && set +a
export OPENCODE_API_KEY="$LITELLM_MASTER_KEY"
cd ~/scratch/my-project
opencode
# in the TUI: /models -> workshop/fast

Ask for the same Elixir idea in another language. Then Ctrl-C Ollama in Terminal A, keep prompting, and check Logs and spend in the UI.

2.8

End to end: pull the rug

Same client as 1.7. Now it survives an outage.

  1. Ask opencode for the same Elixir idea in another language. It goes through the gateway to Ollama.
  2. Stop Ollama (Terminal A, Ctrl+C). Ask for another change.
  3. It still answers, now from the cloud. Notice it’s faster: a useful data point, not a defeat.
  4. Restart Ollama, keep working. Then open Logs in the UI and find the spend.

You just built a miniature of a production AI gateway. Same three layers as ours: runtime, gateway, client. Each is replaceable without touching the others.

Troubleshooting

Find your symptom. Still stuck after one fix? Raise a hand - the clinic at the end is for exactly this.

SymptomWhyFix
Port 11434 already in useAn app, brew service or old ollama serve is runningQuit the menu-bar app; brew services stop ollama (mac) / sudo systemctl stop ollama (Linux); then start it again
connection refused on 11434Terminal A isn’t running OllamaRe-run the serve command in Terminal A
Model “should fit” but swaps / OOMsOllama reserves the KV cache for the whole context windowLower OLLAMA_CONTEXT_LENGTH (e.g. 16384) or pick the 4B model. ollama ps shows the real size
First answer takes ~10 sModel is cold, loading from diskNormal once; ollama run warms it
Model replies forever / “thinks in circles”Reasoning model on a vague promptAsk something specific; keep prompts short
opencode: model not found"model" doesn’t match a key under modelsFix both, keep the ollama/ prefix; restart opencode
opencode ignores my configEdited while opencode was runningQuit and relaunch opencode
opencode rejects the configOld opencode, newer config formatopencode --version, upgrade, retry
Docker: “cannot connect to daemon”Docker Desktop installed but not launchedLaunch it, wait for the whale icon to settle
Gateway answers 500: connection refusedContainer can’t reach Ollama on your machineCheck host.docker.internal in config (and no /v1 on an ollama_chat/ api_base); on Linux start Ollama with OLLAMA_HOST=0.0.0.0
401 UnauthorizedMissing or wrong master keyReload .env: set -a && . ./.env && set +a
400 “model not found”You asked for a name that isn’t a model_nameAsk for fast or qwen3.5:4b exactly as written in config.yaml
/v1/models is emptyconfig.yaml didn’t loaddocker compose logs litellm: look for a YAML error
Failover never firesBoth entries must share model_name: fast, have an order, and enable_pre_call_checks: true must be setCompare with the config in 2.4; then docker compose restart litellm
Failover fires but the cloud call errorsMissing or bad key in .envFix the key, then docker compose up -d --force-recreate litellm (restart alone won’t re-read .env)
“No deployments available” on fastOllama is down and the cloud deployment is capped or cooling downStart Ollama, or wait a few seconds; if you ran “prove the cap bites”, restore config.yaml.bak
auto answers 500 “model isn’t mapped yet”The complexity router tries to compact context and LiteLLM has no entry for your local modelAdd context_compaction: false under complexity_router_config, then restart
Port 4000 already in useAnother container or LiteLLMChange the left side of "4000:4000" in compose
Everything worked, then stoppedLaptop slept; model evictedRe-warm: ollama run once

No match. Ask in the room - and tell us what’s missing here.

Homework & cleanup

Each is a small extension, and each teaches something the workshop only touched.

  1. oMLX. Install it, serve your model, then point the gateway at it. No client changes.
  2. A second model. Teach the gateway another one and use it from opencode.
  3. Auto-router. Add the complexity router from 2.4, then tune its keyword rules and watch which questions go to the cloud.
  4. Guardrails. Run the PII masking reference from 2.4 and watch an email address turn into a placeholder.
  5. A budgeted key. Create a key with max_budget (see 2.6) and watch it in the UI.
Teach the gateway a second model
# add this under model_list: in config.yaml, same shape as the first entry
  - model_name: qwen3.5:9b
    litellm_params:
      model: ollama_chat/qwen3.5:9b
      api_base: http://host.docker.internal:11434
# then: ollama pull qwen3.5:9b, and docker compose restart litellm

/v1/models now lists it. No client changed.

Tear down (after the workshop)
cd ~/workshop-gateway && docker compose down
# add -v to also wipe Postgres (keys, logs, spend)
ollama rm qwen3.5:4b     # only if you need the disk back

Glossary

Quantization
Storing weights at fewer bits. Memory is about parameters × bits ÷ 8: 16-bit = 2 GB per billion parameters, 8-bit = 1 GB, 4-bit = 0.5 to 0.6 GB. Fewer bits: smaller and faster, slightly worse. Ollama’s default tags are 4-bit.
GGUF
The weights file format Ollama and llama.cpp use: what ollama pull downloads.
Context window
How many tokens the model sees at once. Bigger costs RAM: Ollama reserves it up front.
KV cache
The memory that holds the conversation for attention. Grows with context, on top of the weights.
OpenAI-compatible
Speaks the /v1/chat/completions dialect, so any OpenAI-shaped client can talk to it.
Gateway
One endpoint in front of many models: routing, failover, keys, budgets, logs. LiteLLM is ours.
Model group
Several deployments sharing one model_name. Our fast group: local first (order: 1), cloud second.
Failover
Automatic retry on the next deployment when the first fails (connection, timeout, 5xx, not 4xx).
Auto-router
A gateway model name (auto) that picks a deployment per request by complexity: simple to local, hard to the cloud. LiteLLM’s complexity router scores each prompt with rules, no extra model call.
Provider budget
A spend cap on one upstream provider, enforced by the gateway.
Guardrail
A check the gateway runs on prompts or replies: mask PII, block a topic, hide secrets.
model_name vs model
model_name is what your clients ask for; model is the provider’s name. That indirection is the abstraction.