Why Run AI on a Mini PC Instead of the Cloud?
If you've been experimenting with ChatGPT or Claude via API, you already know the hidden costs: per-token billing that climbs fast, rate limits that throttle batch workloads, and—critically—every prompt you send leaves your network. For homelab builders, developers, and anyone running sensitive workflows, that last point is a dealbreaker.
The good news: modern mini PCs have crossed the threshold where running capable open-weight models locally is genuinely practical. A compact machine with 16–32 GB of unified or system memory can serve Llama 3.1 8B, Mistral 7B, Gemma 2 9B, and dozens of other models fast enough for real productivity—not just demos. This guide walks you through the full stack: hardware, Ollama, Open WebUI, and the tweaks that make it actually comfortable to use day-to-day.
Hardware: What You Actually Need
The sweet spot for a local AI mini PC in 2026 is an Intel N-series or AMD Ryzen 7 mini PC with at least 16 GB RAM (32 GB strongly preferred) and a fast NVMe SSD. GPU is optional—CPU inference with llama.cpp is more than good enough for 7B–13B parameter models at 4-bit quantization.
The machine I've been running for months is the Beelink EQ12 Mini PC (~$189). It packs an Intel N100 (4 cores, 6W TDP), comes with 16 GB DDR4, and has dual NVMe slots. Passively cooled under normal load, dead silent, and pulls under 15W when running models. For the price, nothing else comes close.
Stock RAM is fine for smaller models, but if you want to run 13B models comfortably, upgrade to 32 GB DDR4 SODIMM (~$59). The Beelink EQ12 has one user-accessible SODIMM slot, so you're swapping the stock 16 GB stick for a 32 GB one. Five minutes with a screwdriver.
For storage, model files are large—Llama 3.1 8B at Q4_K_M is about 4.7 GB, and you'll want to keep 5–10 models on disk without juggling. A WD Black 2TB SN850X NVMe (~$129) in the second slot gives you plenty of room and fast sequential reads that noticeably speed up model loading cold-start times.
OS Setup: Ubuntu Server 24.04 LTS
Flash Ubuntu Server 24.04 LTS to a USB drive with Rufus or Balena Etcher. Boot, install with defaults, enable OpenSSH during setup. Once you're in:
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget git htop nvtop
Set a static IP via netplan so your Open WebUI URL never changes:
# /etc/netplan/00-installer-config.yaml
network:
version: 2
ethernets:
enp1s0:
dhcp4: false
addresses:
- 192.168.1.50/24
routes:
- to: default
via: 192.168.1.1
nameservers:
addresses: [1.1.1.1, 8.8.8.8]
sudo netplan apply
If your mini PC only has a 1GbE port and you want faster LAN transfers for model syncing, the TP-Link 2.5G USB Ethernet Adapter (~$22) plugs into USB-A and works out of the box on Ubuntu 24.04—no driver fiddling needed.
Installing Ollama
Ollama is the engine that handles model downloads, quantization selection, and a local OpenAI-compatible API. Installation is a single command:
curl -fsSL https://ollama.com/install.sh | sh
This installs Ollama as a systemd service and starts it automatically. Verify it's running:
systemctl status ollama
ollama --version
By default, Ollama listens only on 127.0.0.1:11434. Since Open WebUI will run in Docker on the same machine, this is fine for now. If you want to expose the API to other machines on your LAN later, add this to the systemd override:
sudo systemctl edit ollama
[Service]
Environment="OLLAMA_HOST=0.0.0.0"
sudo systemctl daemon-reload && sudo systemctl restart ollama
Pulling Your First Models
Pull models with ollama pull. Here's what I recommend based on the hardware above:
# Fast general-purpose — great for chat and coding
ollama pull llama3.1:8b
# Excellent at structured output and function calling
ollama pull mistral:7b
# Strong reasoning, slightly slower
ollama pull gemma2:9b
# Coding assistant — beats Codestral on small hardware
ollama pull qwen2.5-coder:7b
# Tiny but surprisingly capable for quick tasks
ollama pull phi3:mini
Check what's downloaded and how much space they use:
ollama list
Do a quick test to confirm inference is working:
ollama run llama3.1:8b "Explain Proxmox LXC containers in one paragraph."
On the Beelink EQ12 with 32 GB RAM, Llama 3.1 8B Q4_K_M generates at ~18–22 tokens/sec. Fast enough for interactive chat and perfectly fine for batch processing with n8n.
Installing Docker
Open WebUI is distributed as a Docker container. Install Docker CE:
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker
Deploying Open WebUI
Open WebUI is a polished, self-hosted chat interface that connects to your local Ollama instance. It supports model switching, conversation history, RAG (retrieval-augmented generation) with document uploads, image generation backends, and multi-user accounts.
docker run -d \
--name open-webui \
--restart always \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://host-gateway:11434 \
--add-host=host-gateway:host-gateway \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
Open your browser to http://192.168.1.50:3000 (replace with your mini PC's static IP). On first load, you'll create an admin account. Once in, your Ollama models appear automatically in the model selector.
Keeping Open WebUI Updated
New features ship weekly. Update with:
docker pull ghcr.io/open-webui/open-webui:main
docker stop open-webui && docker rm open-webui
# Re-run the docker run command above
Your conversations, RAG documents, and settings persist in the open-webui Docker volume across updates.
Useful Open WebUI Configuration
RAG: Chat With Your Documents
In Settings → Documents, you can configure RAG to use an embedding model. For local embeddings, pull nomic-embed-text:
ollama pull nomic-embed-text
Then in Open WebUI Settings → Documents, set the embedding model to nomic-embed-text and the chunk size to 512. Upload PDFs, markdown files, or text docs, and ask questions against them directly in chat.
System Prompts and Model Manifolds
Open WebUI lets you create "model manifolds"—custom model entries that wrap a base model with a fixed system prompt. Go to Admin → Models → Create a Model, select your base model, and add a system prompt. This is great for creating a "coding assistant" persona backed by qwen2.5-coder:7b or a "writing assistant" backed by llama3.1:8b.
Multi-User Setup
Admin → Users → Add User. Each user gets their own conversation history and model access. You can limit which models each user sees under their profile settings—useful if you're sharing the box with family or team members.
Troubleshooting Common Issues
Ollama not reachable from Open WebUI container
If Open WebUI shows "Ollama connection failed," the host-gateway special DNS name might not resolve in your Docker version. Fall back to the explicit host IP:
docker run -d \
--name open-webui \
--restart always \
-p 3000:8080 \
-e OLLAMA_BASE_URL=http://192.168.1.50:11434 \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
Model loads then crashes with "out of memory"
You're running a model that needs more RAM than available. Check memory usage:
free -h
ollama ps
If another model is loaded in memory, Ollama doesn't automatically evict it. Run ollama stop <model-name> to free VRAM/RAM before loading a larger model. Or set OLLAMA_MAX_LOADED_MODELS=1 in the Ollama systemd override to enforce single-model-at-a-time behavior.
Slow token generation
Make sure you're using a quantized model (Q4_K_M suffix) rather than the full-precision version. The difference is significant—Q4_K_M is 3–4x faster with negligible quality loss for most tasks. Check which variant you have:
ollama show llama3.1:8b
The default ollama pull llama3.1:8b already grabs Q4_K_M, but if you pulled a specific GGUF manually, double-check.
Open WebUI container restarts constantly
Check logs first:
docker logs open-webui --tail 50
A common cause is a corrupted SQLite database in the volume. Back up and reinitialize:
docker run --rm -v open-webui:/data alpine cp /data/webui.db /data/webui.db.bak
docker run --rm -v open-webui:/data alpine rm /data/webui.db
docker restart open-webui
Optional: Expose via Tailscale Instead of Port Forwarding
If you want to access your local AI stack from outside your home network without opening ports, install Tailscale on the mini PC. Your Open WebUI becomes reachable at its Tailscale IP on port 3000 from any device on your tailnet—phone, laptop, work machine. No dynamic DNS, no firewall rules.
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
Check the assigned Tailscale IP:
tailscale ip -4
What's Next: Connecting to n8n
Once your local AI stack is running, the obvious next step is automation. n8n has a native Ollama node that lets you build workflows where AI is one step in a larger pipeline: summarize incoming emails, classify support tickets, extract structured data from documents, or generate weekly reports—all without touching an external API.
The Ollama node connects to http://<your-mini-pc-ip>:11434 and lets you pick any model you've pulled. Combined with Open WebUI for interactive use and n8n for batch automation, you get a genuinely complete local AI stack for under $400 in hardware.
A full n8n + local LLM automation guide is coming next week. In the meantime, the setup above is everything you need to start experimenting.
Alternative: Running This Stack in a Proxmox LXC Container
If you're already running Proxmox on a more powerful machine and don't want a dedicated mini PC, you can run the entire Ollama + Open WebUI stack in an LXC container. The setup is almost identical, but you get Proxmox's snapshot and backup capabilities on top.
Create an LXC container from the Ubuntu 24.04 template in Proxmox, give it at least 8 GB RAM (16 GB for 13B models), 4 vCPUs, and 100 GB storage. Then SSH into the container and follow the exact same steps above—Ollama installs identically, and Docker runs fine in an unprivileged container with keyctl enabled.
The one gotcha: if your Proxmox host has a GPU and you want to pass it through to the LXC for inference acceleration, you need a privileged container and must pass the device through in the LXC config:
# In /etc/pve/lxc/.conf on the Proxmox host
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
For GPU passthrough, Ollama on the LXC will use the host's GPU driver automatically if the CUDA or ROCm libraries are installed inside the container. Intel Arc and integrated Intel GPU users can use the ollama run --gpu flag with the Intel OpenVINO backend.
Monitoring Resource Usage
After a few days of use, you'll want to know how hard the mini PC is actually working. Two tools make this easy:
# CPU, RAM, processes — like top but better
htop
# Real-time GPU/NPU usage if you have one
nvtop
# Watch Ollama's active model and memory usage
watch -n2 "ollama ps"
For persistent monitoring visible from a browser, add Beszel to your Docker stack. It's a lightweight self-hosted server monitoring tool that tracks CPU, RAM, disk, and network over time with a clean UI. A full Beszel setup guide is available on this blog if you want to add observability to your homelab.
Setting Automatic Model Unloading
By default, Ollama keeps a loaded model in memory for 5 minutes after the last request. On a machine with 16 GB RAM, that's fine. On a 16 GB machine where you're also running Open WebUI, n8n, and other containers, you may want to reduce that window:
sudo systemctl edit ollama
[Service]
Environment="OLLAMA_KEEP_ALIVE=2m"
sudo systemctl daemon-reload && sudo systemctl restart ollama
Set to 0 if you want Ollama to unload the model immediately after each request—useful when RAM is tight and you're not doing interactive chat.
Model Performance Reference: Beelink EQ12 (N100, 32 GB RAM)
Here's what you can realistically expect on the N100 mini PC with no GPU, using Q4_K_M quantization:
- phi3:mini (3.8B) — ~35 tokens/sec. Great for quick tasks, code completion, short summaries.
- llama3.1:8b — ~18–22 tokens/sec. Best general-purpose model at this tier. Handles coding, writing, and reasoning well.
- mistral:7b — ~20 tokens/sec. Excellent instruction following, strong at structured output and JSON extraction.
- qwen2.5-coder:7b — ~18 tokens/sec. Purpose-built for code—outperforms general 7B models on programming tasks.
- gemma2:9b — ~12–15 tokens/sec. Higher quality reasoning, but slower. Worth it for complex multi-step tasks.
- llama3.1:13b — ~8 tokens/sec. Pushes the N100 hard. Needs the full 32 GB to avoid swapping. Usable for batch jobs, not ideal for interactive chat.
The practical sweet spot for daily use is the 7B–9B range. You get near-GPT-3.5 quality at essentially zero marginal cost per query, with full data privacy and no API key management.
Keeping Everything Running After Reboots
Ollama is already a systemd service that starts automatically. For Open WebUI, the --restart always flag in the Docker run command handles restarts. But if Docker itself doesn't start on boot:
sudo systemctl enable docker
sudo systemctl enable containerd
To verify everything comes up cleanly after a reboot:
sudo reboot
# After ~60 seconds...
ssh user@192.168.1.50
systemctl status ollama
docker ps
Both should show as running. If Open WebUI container isn't up, check docker logs open-webui for startup errors.
Summary
The full local AI stack covered in this guide—Ollama handling model inference, Open WebUI providing the chat interface, running on an N100 mini PC—costs under $400 in hardware and runs on roughly 10W of electricity. Every query is local, private, and effectively free after the initial setup.
It's not going to beat GPT-4o on hard reasoning benchmarks. But for the 80% of tasks that don't need frontier-model quality—summarization, code review, writing assistance, data extraction, classification—it's more than good enough, and the privacy and cost advantages are real.
The next step after this setup is connecting it to n8n for automation workflows. When you can call a local LLM from a workflow node, AI stops being a chat tool and starts being an infrastructure component. That guide is coming next week.