Ollama on Proxmox with GPU Passthrough: Running Qwen 2.5 32B Locally

The economics have flipped. Six months ago, running a 32B parameter model locally meant either a $3,000+ consumer GPU or compromising down to a 7B that frustrates you with shallow reasoning. Today, if you've got a Proxmox node with an RTX 3060 12GB or better, you can run Qwen 2.5 32B at Q4 quantization and get outputs that are genuinely hard to distinguish from GPT-4 for most engineering tasks—at zero marginal cost per query.

This is the setup I landed on after too many hours debugging VFIO bind failures and IOMMU group nightmares. Here's how to do it right the first time.

Why GPU Passthrough vs. a Dedicated AI Host

The obvious question: why not just run Ollama directly on bare metal? Two reasons:

Proxmox gives you snapshotting and rollback. Experimenting with CUDA driver versions, different Ollama builds, or custom model configs becomes much lower risk when you can snapshot the VM before you break something. Rollback is a lifesaver when an nvidia-driver-545 upgrade silently breaks your inference stack.

Resource sharing. When the GPU isn't running inference, you can reclaim it for other VMs (one VM at a time without SR-IOV, which consumer cards don't support). My RTX 3060 12GB doubles as a transcoding card for Jellyfin when the AI VM is suspended.

The tradeoff is a small performance overhead (~3-5% throughput loss vs. bare metal) and the initial passthrough setup pain. Worth it for a lab.

Prerequisites

• Proxmox VE 8.x (tested on 8.3)
• CPU with IOMMU support (Intel VT-d or AMD-Vi enabled in BIOS)
• GPU in its own IOMMU group, or willingness to pass through the entire group
• Ubuntu 22.04 LTS VM target
• At minimum: RTX 3060 12GB for 32B Q4; RTX 3080 or RTX 3090 for 70B Q4
• At least 32GB DDR4 system RAM (64GB+ recommended for larger models)
• A fast NVMe SSD for model storage — 32B models are ~20GB each and you'll accumulate them

Step 1: Enable IOMMU on the Proxmox Host

Edit GRUB on the Proxmox host:

nano /etc/default/grub

# Intel:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# AMD:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

update-grub

The iommu=pt (passthrough) flag reduces overhead for devices not being passed through and often fixes stability issues that straight iommu=on causes.

Add VFIO modules to load at boot:

echo "vfio" >> /etc/modules
echo "vfio_iommu_type1" >> /etc/modules
echo "vfio_pci" >> /etc/modules
echo "vfio_virqfd" >> /etc/modules

Blacklist NVIDIA on the host so it doesn't grab the GPU before VFIO can:

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidiafb" >> /etc/modprobe.d/blacklist.conf

Reboot, then verify:

dmesg | grep -e DMAR -e IOMMU
# Should see: DMAR: IOMMU enabled

Step 2: Identify Your GPU's IOMMU Group

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done | sort -V

Find your GPU and note everything sharing its group. On most consumer Intel boards, the GPU and its HD Audio device share a group — you'll need to pass through both. Note the PCI IDs (e.g., 10de:2504 for RTX 3060, 10de:228e for its audio).

Bind VFIO to those IDs:

echo "options vfio-pci ids=10de:2504,10de:228e" > /etc/modprobe.d/vfio.conf
update-initramfs -u
reboot

Verify the binding:

lspci -nnk | grep -A3 "VGA\|3D\|Audio"
# Kernel driver in use: vfio-pci  ← this is what you want

Step 3: Configure the VM

Create a VM with at least 16GB RAM (24GB+ recommended for 32B models). In Proxmox web UI: VM → Hardware → Add → PCI Device:

• Select your GPU
• Check All Functions (passes GPU + audio together)
• Check PCI-Express
• Check ROM-Bar
• Leave Primary GPU unchecked for headless inference

Set the CPU type to host rather than the default kvm64 — this exposes AVX-512 and other extensions that quantization kernels benefit from.

Critical gotcha: disable the virtual display adapter (set Display to "none") once you have SSH working. Keeping it enabled while passing through a GPU triggers the NVIDIA driver's hypervisor detection — the infamous "Error 43."

Add this to /etc/pve/qemu-server/.conf:

args: -cpu host,kvm=off

kvm=off hides the KVM hypervisor signature from the guest, which prevents NVIDIA's driver check from failing.

Step 4: Install NVIDIA Drivers in the VM

Boot Ubuntu 22.04, then:

sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install

# Or pin a version:
sudo apt install -y nvidia-driver-550

sudo reboot

After reboot:

nvidia-smi
# Should show your GPU name, VRAM, driver version

If nvidia-smi fails with "couldn't communicate with the NVIDIA driver" — you're either missing kvm=off or still have the virtual display enabled. Those two issues cause 90% of Error 43s.

Step 5: Install and Configure Ollama

curl -fsSL https://ollama.com/install.sh | sh

By default Ollama binds to localhost. To expose it on the network:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_MODELS=/data/ollama/models"

I store models on a separate volume (/data) backed by a 2TB NVMe SSD so they don't compete with the OS disk. Pull the model:

ollama pull qwen2.5:32b

About 20GB download for Q4_K_M quantization. First load takes 30-45 seconds to fill VRAM; subsequent calls are instant since the weights stay resident.

Real-World Performance: RTX 3060 12GB

Running Qwen 2.5 32B Q4_K_M on an RTX 3060 12GB:

For comparison, Llama 3.1 8B runs ~80 tokens/sec on the same card. The 32B is slower, but the output quality difference is meaningful for reasoning-heavy tasks — code review, architecture decisions, multi-step debugging. For one-shot completions, 8B or 14B are still the better tradeoff.

Want to run 70B models? You'll need an RTX 3090 24GB (fits Q4_K_M with room to spare) or a pair of RTX 3080 10GB cards for CPU offloading — though the latter takes a performance hit.

Front-End: Open WebUI

docker run -d \
  --name open-webui \
  --restart unless-stopped \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://<your-vm-ip>:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

This gives you a full ChatGPT-style interface backed by local inference. Add it to your Tailscale network and you have private AI accessible from anywhere without a single token leaving your infrastructure.

Hardware Recommendations by Budget

If you're building a dedicated node for this rather than repurposing existing hardware:

• Budget (~$400-500): Minisforum UM870 with a used RTX 3060 12GB via external GPU enclosure, or a used workstation with a PCIe slot
• Mid-range (~$800-1000): Tower system with RTX 3080 10GB + 64GB DDR4 RAM — handles 32B comfortably with headroom
• High-end (~$1500+): RTX 3090 24GB + 128GB DDR4 — runs 70B at usable speeds

Caveats and When to Skip This

When GPU passthrough isn't worth it:

• Your GPU shares an IOMMU group with other devices you need. The ACS override patch workaround exists but is a kernel stability risk.
• You need multi-GPU inference (70B across two cards). Passed-through GPUs can't use NVLink or peer-to-peer transfers.
• You can tolerate 4-5 tokens/sec and have a high-core-count CPU with 64GB+ RAM — llama.cpp CPU-only inference works and is far simpler.

Alternatives worth knowing:

• LXC containers with GPU — Proxmox supports GPU passthrough to LXC with slightly lower overhead than a full VM, though you lose some isolation
• Bare metal Ollama — skip Proxmox entirely if VM management isn't a priority and you have a dedicated node
• Multiple smaller models — routing different task types to 8B vs. 32B models based on complexity can give you better overall throughput than always running 32B

The Bottom Line

Once GPU passthrough is working in Proxmox, it just works — I haven't touched the config in months. The one-time setup cost pays off immediately: production-grade LLM inference at zero per-query cost, on hardware you already own, with full control over model versions and no data leaving your network.

Qwen 2.5 32B handles the majority of my day-to-day engineering prompts — code review, SQL generation, writing commit messages from diffs — at quality I'd have been skeptical about a year ago. The per-token economics of cloud inference made this a compelling experiment; the output quality made it permanent infrastructure.

Some links in this post are Amazon affiliate links. If you purchase through them, I may earn a small commission at no extra cost to you.