The economics have flipped. Six months ago, running a 32B parameter model locally meant either a $3,000+ consumer GPU or compromising down to a 7B that frustrates you with shallow reasoning. Today, if you've got a Proxmox node with an RTX 3060 12GB or better, you can run Qwen 2.5 32B at Q4 quantization and get outputs that are genuinely hard to distinguish from GPT-4 for most engineering tasks—at zero marginal cost per query.

This is the setup I landed on after too many hours debugging VFIO bind failures and IOMMU group nightmares. Here's how to do it right the first time.

Why GPU Passthrough vs. a Dedicated AI Host

The obvious question: why not just run Ollama directly on bare metal? Two reasons:

Proxmox gives you snapshotting and rollback. Experimenting with CUDA driver versions, different Ollama builds, or custom model configs becomes much lower risk when you can snapshot the VM before you break something. Rollback is a lifesaver when an nvidia-driver-545 upgrade silently breaks your inference stack.

Resource sharing. When the GPU isn't running inference, you can reclaim it for other VMs (one VM at a time without SR-IOV, which consumer cards don't support). My RTX 3060 12GB doubles as a transcoding card for Jellyfin when the AI VM is suspended.

The tradeoff is a small performance overhead (~3-5% throughput loss vs. bare metal) and the initial passthrough setup pain. Worth it for a lab.

Prerequisites

Step 1: Enable IOMMU on the Proxmox Host

Edit GRUB on the Proxmox host:

nano /etc/default/grub
Intel:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
AMD:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
update-grub

The iommu=pt (passthrough) flag reduces overhead for devices not being passed through and often fixes stability issues that straight iommu=on causes.

Add VFIO modules to load at boot:

echo "vfio" >> /etc/modules
echo "vfio_iommu_type1" >> /etc/modules
echo "vfio_pci" >> /etc/modules
echo "vfio_virqfd" >> /etc/modules

Blacklist NVIDIA on the host so it doesn't grab the GPU before VFIO can:

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidiafb" >> /etc/modprobe.d/blacklist.conf

Reboot, then verify:

dmesg | grep -e DMAR -e IOMMU
Should see: DMAR: IOMMU enabled

Step 2: Identify Your GPU's IOMMU Group

#!/bin/bash
for d in /sys/kernel/iommu_groups//devices/; do
  n=${d#/iommu_groups/}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done | sort -V

Find your GPU and note everything sharing its group. On most consumer Intel boards, the GPU and its HD Audio device share a group — you'll need to pass through both. Note the PCI IDs (e.g., 10de:2504 for RTX 3060, 10de:228e for its audio).

Bind VFIO to those IDs:

echo "options vfio-pci ids=10de:2504,10de:228e" > /etc/modprobe.d/vfio.conf
update-initramfs -u
reboot

Verify the binding:

lspci -nnk | grep -A3 "VGA\|3D\|Audio"
Kernel driver in use: vfio-pci  ← this is what you want

Step 3: Configure the VM

Create a VM with at least 16GB RAM (24GB+ recommended for 32B models). In Proxmox web UI: VM → Hardware → Add → PCI Device:

Set the CPU type to host rather than the default kvm64 — this exposes AVX-512 and other extensions that quantization kernels benefit from.

Critical gotcha: disable the virtual display adapter (set Display to "none") once you have SSH working. Keeping it enabled while passing through a GPU triggers the NVIDIA driver's hypervisor detection — the infamous "Error 43."

Add this to /etc/pve/qemu-server/.conf:

args: -cpu host,kvm=off

kvm=off hides the KVM hypervisor signature from the guest, which prevents NVIDIA's driver check from failing.

Step 4: Install NVIDIA Drivers in the VM

Boot Ubuntu 22.04, then:

sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
Or pin a version:
sudo apt install -y nvidia-driver-550
sudo reboot

After reboot:

nvidia-smi
Should show your GPU name, VRAM, driver version

If nvidia-smi fails with "couldn't communicate with the NVIDIA driver" — you're either missing kvm=off or still have the virtual display enabled. Those two issues cause 90% of Error 43s.

Step 5: Install and Configure Ollama

curl -fsSL https://ollama.com/install.sh | sh

By default Ollama binds to localhost. To expose it on the network:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_MODELS=/data/ollama/models"

I store models on a separate volume (/data) backed by a 2TB NVMe SSD so they don't compete with the OS disk. Pull the model:

ollama pull qwen2.5:32b

About 20GB download for Q4_K_M quantization. First load takes 30-45 seconds to fill VRAM; subsequent calls are instant since the weights stay resident.

Real-World Performance: RTX 3060 12GB

Running Qwen 2.5 32B Q4_K_M on an RTX 3060 12GB:

| Metric | Value | |---|---| | Token generation | ~18-22 tokens/sec | | Time to first token | 4-6 seconds | | VRAM usage | ~11.8 GB of 12 GB | | Max comfortable context | 32K tokens |

For comparison, Llama 3.1 8B runs ~80 tokens/sec on the same card. The 32B is slower, but the output quality difference is meaningful for reasoning-heavy tasks — code review, architecture decisions, multi-step debugging. For one-shot completions, 8B or 14B are still the better tradeoff.

Want to run 70B models? You'll need an RTX 3090 24GB (fits Q4_K_M with room to spare) or a pair of RTX 3080 10GB cards for CPU offloading — though the latter takes a performance hit.

Front-End: Open WebUI

docker run -d \
  --name open-webui \
  --restart unless-stopped \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://<your-vm-ip>:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

This gives you a full ChatGPT-style interface backed by local inference. Add it to your Tailscale network and you have private AI accessible from anywhere without a single token leaving your infrastructure.

Hardware Recommendations by Budget

If you're building a dedicated node for this rather than repurposing existing hardware:

Caveats and When to Skip This

When GPU passthrough isn't worth it:

Alternatives worth knowing:

The Bottom Line

Once GPU passthrough is working in Proxmox, it just works — I haven't touched the config in months. The one-time setup cost pays off immediately: production-grade LLM inference at zero per-query cost, on hardware you already own, with full control over model versions and no data leaving your network.

Qwen 2.5 32B handles the majority of my day-to-day engineering prompts — code review, SQL generation, writing commit messages from diffs — at quality I'd have been skeptical about a year ago. The per-token economics of cloud inference made this a compelling experiment; the output quality made it permanent infrastructure.

--- Some links in this post are Amazon affiliate links. If you purchase through them, I may earn a small commission at no extra cost to you.