Hands-on workshop

Local LLMs

From ollama run to a gateway that fails over to the cloud - with a spending cap.

Step 1 · Basics Step 2 · Abstractions

Why

Why run a model on your laptop?

Private

The prompt never leaves the machine.

Free at the margin

No per-token bill for the boring tasks.

Offline

Planes, cafés, bad hotel Wi-Fi.

In your control

You pick the model, the version, the context.

Not a Claude replacement. It's for the boring 90%: commit messages, renames, scaffolding, log triage. A different point on the cost-quality curve.

Whyworkshop.caxaria.com 2 / 29

Today

Two steps. One extra layer.

startIntro

1Step 1 · Basics

2Step 2 · Abstractions

endWrap-up

afterClinic

Todayworkshop.caxaria.com 3 / 29

How today works

Three ground rules

1

Follow on the page

Every command has a copy button and an expect line. Pick your model at the top.

workshop.caxaria.com

2

Two terminals

Terminal Aruns Ollama. Leave it open.

Terminal Beverything else.

3

Checkpoints

Self-checking commands. Nobody walks past a ✓ alone. Stuck? Troubleshooting first, then a raised hand - or the clinic.

How today worksworkshop.caxaria.com 4 / 29

1

Step 1

Basics

You end with a model answering on your own machine, driven by a coding agent.

install → serve → pull → talk to it → opencode

Step 1 1.3 · choose

Pick one. You can switch later.

qwen3.5:4b

8 GB RAM
~3.5 GB download
Plenty for boilerplate

qwen3.5:9b

16 GB RAM or more
~6 to 7 GB download
Best quality / speed balance

⚠

Context isn't free: Ollama reserves memory for the whole window up front.

Step 1 · Pick a modelworkshop.caxaria.com 6 / 29

Step 1 1.3 · maths

Will it fit? Count the bits.

weights ≈ params × bits ÷ 8

precision	GB per 1B	9B	4B
16-bit	~2.0	~18	~8
8-bit	~1.0	~9	~4
4-bit	~0.55	~5	~2.5

…then add

KV cache: grows with context, reserved up front
Overhead: runtime + your OS

Budget: weights + KV cache + overhead ≤ ¾ of RAM. The ¾ is headroom for your Mac, not a shortcut around the bits.

ollama ps shows what it really takes

Step 1 · Will it fit?workshop.caxaria.com 7 / 29

Step 1 1.1 to 1.3

Install, serve, pull

1 · installbrew install ollamaollama --version

2 · Terminal AOLLAMA_CONTEXT_LENGTH=32768 ollama serve# keep this tab open

3 · Terminal Bollama pull qwen3.5:4b # or :9bollama list

It's a daemon

ollama serve ≈ dockerd. It keeps running - so it gets its own tab.

a version, then your model in ollama list

Step 1 · Installworkshop.caxaria.com 8 / 29

Exercise

Say hello - over HTTP

terminal Bcurl localhost:11434/v1/chat/completions \ -d '{"model": "qwen3.5:4b", "messages": [{"role": "user", "content": "Say hi in 5 words"}]}'

Also try ollama run qwen3.5:4b for the chat version. (9B? swap the tag.)

response{ "choices": [{ "message": { "role": "assistant", "content": "Hello there, how are you?" } }], …

This shape is the whole reason Step 2 is possible. opencode speaks it. So will the gateway.

Step 1 · Exerciseworkshop.caxaria.com 9 / 29

Side-quest

Ollama is Docker for models

Docker	Ollama
docker pull nginx	ollama pull qwen3.5:4b
docker run nginx	ollama run qwen3.5:4b
docker ps	ollama ps
Dockerfile	Modelfile
Docker Hub	ollama.com/library

Same muscle memory.
Different payload.

A model isn't a process you kill - it's weights in RAM, evicted when idle.

Step 1 · Side-questworkshop.caxaria.com 10 / 29

Side-questMac only

Faster on a Mac: oMLX

Ollama

llama.cpp underneath.
General-purpose. Works in 5 minutes.

oMLX

Native Apple-Silicon server, custom Metal kernels. Faster - but you build the kernel for your Mac.

Same OpenAI-compatible API → it drops in exactly where Ollama sits. Homework.

The catch

Mac only, and a fussier setup. Ollama is the right default today.

Step 1 · Side-questworkshop.caxaria.com 11 / 29

Step 1 1.5 to 1.6

The whole integration is one baseURL

curl -fsSL https://opencode.ai/install | bash

~/.config/opencode/opencode.json{ "provider": { "ollama": { "npm": "@ai-sdk/openai-compatible", "options": { "baseURL": "http://localhost:11434/v1", "apiKey": "ollama" }, "models": { "qwen3.5:4b": { … } } }}, "model": "ollama/qwen3.5:4b" }

baseURL - the URL you curled
apiKey - dummy; Ollama doesn't check
model - ollama/ + a key under models

⚠

A name mismatch is the #1 setup mistake.

Step 1 · opencodeworkshop.caxaria.com 12 / 29

Exercise

Ask for something beautiful. Then unplug.

Open opencode in a scratch project
Ask: “show me a beautiful piece of code in Elixir”
Turn off Wi-Fi. Ask: “now show me the same thing in TypeScript”

It still works.

That's the point. A model on your laptop, answering with the network off.

Now push it

Ask for a multi-file refactor. Watch it go subtly wrong: the honest limit, and the reason for Step 2.

Step 1 · Exerciseworkshop.caxaria.com 13 / 29

Checkpoint · Step 1

Everyone together

ollama list shows your model
The curl returns choices[0]…
You got Elixir, then TypeScript with Wi-Fi off

If it doesn't

port in use	stop the app / brew service
model not found	fix both name spots
config ignored	restart opencode

Before Step 2: Docker Desktop running (whale icon), and an API key from Anthropic or OpenAI. We don't hand out keys: no key, pair up.

Step 1 · Checkpointworkshop.caxaria.com 14 / 29

2

Step 2

Abstractions

You end with a gateway: one endpoint, a fast group that is local first with a capped cloud failover, and a client that never notices.

docker → config → gateway → failover → cap → opencode, repointed

Step 2 why

What's wrong with what we just built?

Every client hardcodes localhost:11434 and a model name

Ollama down → every tool breaks, silently

Want a bigger model? Reconfigure N clients

No idea what it costs - or who spends it

Hardcoding an endpoint in every client is a habit, not an architecture.

Step 2 · The problemworkshop.caxaria.com 16 / 29

Step 2 the idea

LiteLLM: one API in front of many models

One API - OpenAI dialect, whatever answers.

A routing table - model_list.

A safety net: failover, budgets, keys, guardrails, UI.

Step 2 · LiteLLMworkshop.caxaria.com 17 / 29

Step 2 2.1 to 2.2

Three files, one gateway

docker-compose.yml

LiteLLM + Postgres

config.yaml

the routing table

.env

master key · your cloud key

config.yamlmodel_list: - model_name: qwen3.5:4b # clients ask for this litellm_params: model: ollama_chat/qwen3.5:4b # provider’s name api_base: http://host.docker.internal:11434

host.docker.internal = “my laptop”, seen from inside the container.

Step 2 · The filesworkshop.caxaria.com 18 / 29

Exercise

Bring the gateway up

Create .env, compose file, config.yaml (paste from the page)
docker compose up -d
Ask :4000 what models it has
Chat with qwen3.5:4b through it

terminal Bdocker compose up -d curl localhost:4000/v1/models \ -H "Authorization: Bearer $LITELLM_MASTER_KEY"

first start ≈ 1 min · then qwen3.5:4b in the list

Step 2 · Exerciseworkshop.caxaria.com 19 / 29

Checkpoint · Gateway

The gateway answers

Both containers up
/v1/models lists qwen3.5:4b
A chat through :4000 returns a real answer

Pass = you can name the URL your client will use. http://localhost:4000/v1

Step 2 · Checkpointworkshop.caxaria.com 20 / 29

Step 2 2.4

One name, two deployments, one ceiling

config.yaml- model_name: fast litellm_params: model: ollama_chat/qwen3.5:4b order: 1 # local first- model_name: fast litellm_params: model: your cloud model order: 2 # only if local fails provider_budget_config: your provider: budget_limit: 2 # USD per day

bring one of theseAnthropicmodel: anthropic/claude-sonnet-5-5api_key: os.environ/ANTHROPIC_API_KEY

OpenAImodel: openai/gpt-5-miniapi_key: os.environ/OPENAI_API_KEY

⚠

A failover is a silent cost switch. Never ship one without a cap. Calling qwen3.5:4b directly never fails over: only fast does.

Step 2 · The fast groupworkshop.caxaria.com 21 / 29

This is the slide that matters most for a production audience. Say the warning slowly: a failover without a budget turns an outage into an invoice.

Two deployments share one name, "fast". Lowest order goes first: local. If it fails, LiteLLM retries on order 2: the cloud model. Calling qwen3.5:4b directly never fails over, on purpose; only "fast" does. We tested both, and that it returns to local as soon as Ollama is back.

We don't provide keys. They bring one from Anthropic or OpenAI: the only difference is the cloud entry and the provider name in the budget. Show both.

Be precise about when it fires: connection refused, timeout, provider 5xx. Not 4xx: a bad request stays a bad request, and retrying it on a paid model just bills you for the same error.

In production you'd scope caps per virtual key per team, not just per provider: homework on the page.

Step 2 going further

Guardrails: clean the prompt first

you type

Reply to Jane Doe (jane@acme.com) about her refund

→

LiteLLM guardrail

detect · mask · or block

→

the model sees

Reply to <PERSON> (<EMAIL_ADDRESS>) about her refund

…and with output_parse_pii the reply comes back with the real name and email restored.

config.yamlguardrails: - guardrail_name: "pii" litellm_params: guardrail: presidio mode: "pre_call" pii_entities_config: PERSON: "MASK" EMAIL_ADDRESS: "MASK" CREDIT_CARD: "BLOCK" output_parse_pii: true

Also in the box

secret hiding · banned keywords · third-party checkers

Best effort, not a compliance guarantee. Presidio = two extra containers.

Step 2 · Guardrailsworkshop.caxaria.com 22 / 29

The hook: failover means prompts can now leave the laptop for a cloud provider. A gateway is the one place where you can inspect every request before that happens.

LiteLLM guardrails run on each request. The PII one uses Presidio, which runs as two small containers (an analyzer and an anonymizer) next to the gateway. In config you list entity types and an action per type: MASK replaces the value with a placeholder like <EMAIL_ADDRESS>; BLOCK rejects the request. With output_parse_pii the gateway swaps the real values back into the reply, so the user sees a normal answer while the model never saw the email.

Mode pre_call runs before the model; post_call checks replies; logging_only masks what is written to logs. Clients can ask for a guardrail by name per request, and it can be attached per key or on by default (check the docs for the current option names).

Be honest about limits: detection is statistical, so it misses things. It's a safety net, not a compliance guarantee, and it adds a little latency. For us the interesting entities are the ones in our data: names, emails, IBANs.

We did NOT run this in the workshop build. The site has the config as a reference, marked untested. Other guardrails exist (secret hiding, banned keywords, third-party checkers): point at the LiteLLM docs.

Step 2 going further

Auto-router: right-size every request

simple

“hi”

local

medium

“what is a KV cache?”

local

complex

“refactor this module…”

cloud

reasoning

“prove it, step by step…”

cloud

config.yaml- model_name: auto litellm_params: model: auto_router/complexity_router complexity_router_config: tiers: SIMPLE: qwen3.5:4b MEDIUM: qwen3.5:4b COMPLEX: cloud-smart REASONING: cloud-smart

How it decides

Rules score each prompt: size, code, reasoning words. No extra model call. Add keyword_tier_rules to force the words you care about.

A heuristic, not magic. Check Logs to see where requests went.

Step 2 · Auto-routerworkshop.caxaria.com 23 / 29

The hook: so far the client chooses the model. But the client rarely knows how hard the question is, and "hi" does not need a cloud model.

LiteLLM's complexity router is just another model name, "auto". Each prompt gets a score from rules, not from another model call: token count, whether there is code, reasoning markers, technical terms, multi-step phrasing. Two or more reasoning markers go straight to the top tier. The score picks one of four tiers, and each tier points at a model_name from our list: simple and medium stay on the local model, complex and reasoning go to the cloud one. The cloud tiers count against the same provider cap.

We ran it: "hi" and "what is a KV cache" stayed local; a refactor request went to the cloud because of a keyword rule. Be honest: it's a heuristic. A long, hard question phrased casually can score low. That is what keyword_tier_rules are for, and there is an LLM-classifier option if you want something smarter (check the docs). Check Logs to see where things actually went.

Gotcha we hit: with a local model LiteLLM has no price entry for, the router crashed until we added context_compaction: false. It's in the config on the page.

Tiers can also point at a group such as fast, if you want local-first with failover inside a tier.

Exercise

Break it on purpose

1 · B

Loop calls fast every 2 s

local | pong
local | pong

2 · A

Stop Ollama
Ctrl-C

✕ local is gone

3 · B

Loop keeps answering

cloud | pong
cloud | pong

4 · A

Restart Ollama

local | pong
traffic is home

⚠

Those requests ran on your provider's bill. Next: see exactly where.

Step 2 · Exerciseworkshop.caxaria.com 24 / 29

Step 2 2.6

Your debugging surface

localhost:4000/ui - user admin, password = your master key

Playground

Chat in the browser. Pick fast or qwen3.5:4b, compare.

Logs

Every request: which model answered, latency, cost. Find your failover.

Keys

Per-person keys, each with its own budget.

Step 2 · Playgroundworkshop.caxaria.com 25 / 29

Step 2 2.7

Repoint opencode: four values

	Step 1	Step 2
provider	ollama	workshop
baseURL	localhost:11434/v1	localhost:4000/v1
apiKey	"ollama"	{env:OPENCODE_API_KEY}
model	ollama/qwen3.5:4b	workshop/fast

The client binary didn't change. The model's identity moved into the gateway - exactly where you want it.

Step 2 · Repointworkshop.caxaria.com 26 / 29

Exercise

Prove it end to end

Ask opencode for a change: it goes via the gateway
Stop Ollama. Ask again.
It answers, from the cloud. Restart Ollama.
Find the spend in Logs

Checkpoint

Local answers via fast
The cloud is a capped failover
Killing Ollama doesn't break the client
You know where the spend shows up

Step 2 · Finalworkshop.caxaria.com 27 / 29

Recap

Three layers. Each one replaceable.

Runtime

Ollama

→ oMLX → anything OpenAI-shaped

Gateway

LiteLLM

model names · failover · budgets · keys · guardrails

Client

opencode

…or any OpenAI-compatible tool

You built the home edition of the pattern we run: runtime → gateway → clients, plus monitoring, per-team keys and pinned versions.

Recapworkshop.caxaria.com 28 / 29

Take it home

Homework

oMLX: serve your model, point the gateway at it
A second model in model_list, used from opencode
Guardrails: mask an email with Presidio
A budgeted key: watch it in the UI

workshop.caxaria.com

every command · troubleshooting

Clinic starts now · ollama list · docker compose logs -f litellm · localhost:4000/ui

Homeworkworkshop.caxaria.com 29 / 29

Local LLMs

Why run a model on your laptop?

Private

Free at the margin

Offline

In your control

Two steps. One extra layer.

Three ground rules

Follow on the page

Two terminals

Checkpoints

Basics

Pick one. You can switch later.

Will it fit? Count the bits.

…then add

Install, serve, pull

It's a daemon

Say hello - over HTTP

Ollama is Docker for models

Same muscle memory.Different payload.

Faster on a Mac: oMLX

Ollama

oMLX

The catch

The whole integration is one baseURL

Ask for something beautiful. Then unplug.

It still works.

Now push it

Everyone together

Abstractions

What's wrong with what we just built?

LiteLLM: one API in front of many models

Three files, one gateway

docker-compose.yml

config.yaml

.env

Bring the gateway up

The gateway answers

One name, two deployments, one ceiling

Guardrails: clean the prompt first

Also in the box

Auto-router: right-size every request

How it decides

Break it on purpose

Your debugging surface

Playground

Logs

Keys

Repoint opencode: four values

Prove it end to end

Three layers. Each one replaceable.

Ollama

LiteLLM

opencode

Homework

Same muscle memory.
Different payload.