How to Build a Local RAG Pipeline with n8n + Ollama: Private Document Q&A 2026

Most RAG tutorials start with an OpenAI API key. The assumption is that you'll send your documents to a hosted embedding model and query a hosted LLM. That works fine for personal projects, but the moment you're dealing with internal codebases, client contracts, private research notes, or anything governed by a data-handling policy, uploading documents to a third-party API is off the table.

The alternative is a fully local RAG stack built on n8n and Ollama. The ingestion pipeline chunks your documents, creates embeddings on your own hardware using Ollama, and stores the vectors in a self-hosted Qdrant instance. The query pipeline embeds incoming questions, retrieves the most relevant chunks, and generates answers using a local LLM — also via Ollama. The whole thing runs on a single mini PC, and nothing leaves your network.

This guide walks through building that pipeline end-to-end: Qdrant setup, embedding model selection, and both n8n workflows (ingestion and query). I'll cover the configuration choices that actually matter and the gotchas that trip people up the first time they try to wire this together.

What Is RAG and Why Run It Locally?

Retrieval-Augmented Generation (RAG) solves a specific problem with LLMs: language models can only reason about text that fits in their context window, and they have no knowledge of documents they weren't trained on. RAG works around this by pre-processing your documents, chunking and embedding them into a vector database, and at query time retrieving only the relevant chunks to include as context. The LLM answers based on that retrieved context rather than its general training knowledge.

The classic RAG flow has two phases. The ingestion phase: read the document, split it into chunks, convert each chunk to a vector embedding, and store those vectors alongside the original text. The query phase: embed the incoming question using the same model, find the vectors closest to the question vector via semantic similarity search, retrieve the original text for those chunks, and pass them as context to the LLM. The model answers based on retrieved text — not hallucinated training data.

Running this locally — as opposed to using OpenAI embeddings and a cloud LLM — gives three concrete advantages. First, cost: OpenAI's text-embedding-3-small charges per token, which adds up fast on large document collections. Per Ollama's documentation, local models cost nothing after the initial hardware purchase. Second, latency: on a mini PC with a fast NVMe drive, local embedding can match API latency for typical document sizes. Third, and most practically for many developers, you skip the data governance problem entirely — nothing leaves your network, so there's no compliance question to answer.

If you're already running n8n and Ollama on your homelab (covered end-to-end in this earlier post on n8n + Ollama on Proxmox), adding a RAG layer is a natural next step that reuses the same infrastructure with zero extra monthly cost.

What You Need Before You Start

This walkthrough assumes a working n8n instance and a working Ollama instance reachable from n8n. If you're starting from scratch, the Ollama + Open WebUI setup guide covers the full Ollama installation. For n8n, the self-hosted Docker path is the most straightforward: docker run -p 5678:5678 n8nio/n8n gets you a working instance. The AI/LangChain nodes are included by default in n8n v1.x and later — no separate install needed.

You'll also need Docker running on a host reachable from n8n to run Qdrant. Qdrant is the vector store that holds the embedded chunks. It exposes a REST API that n8n's Qdrant node talks to directly, and it has a built-in web dashboard for inspecting what's been indexed. There are other vector store options that n8n supports (Pinecone, Supabase pgvector, Weaviate), but Qdrant is self-hostable, has zero cold-start latency, and the n8n integration is rock-solid.

The RAM requirements depend on which models you run simultaneously. Running nomic-embed-text alongside a 7B chat model like LLaMA 3.1 requires about 6–7 GB of RAM for the models, plus 1–2 GB for n8n and Qdrant combined. A Beelink SER5 Mini PC (~$299) with its 16 GB configuration handles this without issue. If you want to experiment with higher-quality embedding models like Qwen3-Embedding-4B — about 9 GB of VRAM per the 2026 local embedding model comparison — the MINISFORUM UM890 Pro (~$429) with 32 GB RAM and the Ryzen 9 8945HS's integrated Radeon 780M is a better fit. For document storage and the Qdrant index on disk, a fast NVMe like the WD_BLACK SN850X 2TB keeps ingestion and retrieval latency low on large collections.

Step 1: Run Qdrant Locally with Docker

Start Qdrant with a single Docker command. The critical flags are the volume mount for persistence and both port mappings — n8n uses the REST API on port 6333, and 6334 is gRPC (used by some other clients):

docker run -d   --name qdrant   --restart unless-stopped   -p 6333:6333   -p 6334:6334   -v $(pwd)/qdrant_storage:/qdrant/storage   qdrant/qdrant

Once it's running, open http://localhost:6333/dashboard in a browser. You should see the Qdrant web UI with an empty collections list — that's expected. The collection gets created automatically by n8n's Qdrant node on the first ingestion run. The --restart unless-stopped flag ensures Qdrant comes back after host reboots without manual intervention, which matters on homelab hardware that isn't managed by an orchestration layer.

According to Qdrant's official documentation, the default collection configuration uses cosine similarity for distance measurement and HNSW as the index algorithm. For typical homelab RAG use cases — a few thousand to tens of thousands of chunks representing a personal document library or a team wiki — these defaults perform well without any tuning. You'd only need to change them for very large collections (millions of vectors) or for hybrid search use cases that combine dense and sparse retrieval.

One useful feature of the Qdrant dashboard: after running the ingestion workflow, you can use the point explorer to browse individual stored chunks and verify the metadata attached to each vector. If your retrieval is returning unexpected results, checking what's actually stored in the collection usually reveals the issue faster than adding debug logging to the n8n workflow.

Step 2: Pull Your Embedding Model with Ollama

For most homelab setups, nomic-embed-text is the right starting point. Per the Ollama model library, it's 137M parameters, produces 768-dimensional vectors, supports an 8,192-token context window, requires about 0.3 GB of VRAM, and is Apache-2.0 licensed (usable for commercial work without restriction). Pull it with:

ollama pull nomic-embed-text

If retrieval quality matters more than RAM budget, mxbai-embed-large offers better benchmark scores at reasonable cost. According to the 2026 local embedding model comparison, it scores 64.68 on the MTEB benchmark versus 62.28 for nomic-embed-text, uses 0.7 GB of VRAM, and is also Apache-2.0. The tradeoff is a 512-token maximum context window — keep your chunk size under 400 tokens if you use this model. For 2026's best-quality compact option, qwen3-embedding:0.6b at ~1.5 GB is worth considering per the same comparison.

ollama pull mxbai-embed-large
# Or for best quality at compact size (2026):
ollama pull qwen3-embedding:0.6b

You also need a chat model for the query workflow. LLaMA 3.1 8B and Qwen2.5 7B are both solid choices that work well with n8n's tool-use nodes. LLaMA 3.1 tends to produce more fluent prose; Qwen2.5 follows structured output instructions more reliably. Either handles document Q&A well at the 7–8B parameter range. Pull whichever fits your available RAM:

ollama pull llama3.1:8b
# or
ollama pull qwen2.5:7b

Step 3: Build the Document Ingestion Workflow in n8n

The ingestion workflow is a pipeline of five node types. In n8n, create a new workflow and connect them in this order. One important note per n8n's AI/LangChain integration documentation: LangChain sub-nodes (embedding models, text splitters, document loaders) connect to their parent nodes via the AI sub-slot, not the main data connector. This is the most common source of confusion when building a first RAG workflow in n8n.

1. Manual Trigger (or Webhook Trigger for automated ingestion) — starts the workflow
2. Read/Write Files from Disk — reads the target file path. Alternatively, use an HTTP Request node to pull from a URL.
3. Default Data Loader — connects to the trigger via the main connector. In node settings, select "Binary Data" as the data type and "PDF" for PDF documents. Plain text and Markdown files work without additional configuration. This node outputs LangChain-compatible document objects.
4. Recursive Character Text Splitter — connects to the Default Data Loader's AI sub-slot. Chunk Size: 600, Chunk Overlap: 100 is a solid starting point; increase chunk size to 800–1000 for dense technical documentation.
5. Qdrant Vector Store (Mode: "Insert Documents") — connects to the Default Data Loader's main connector. Add your Qdrant URL (http://localhost:6333) and set a collection name like my-docs. Wire an Embeddings Ollama sub-node to this node's Embedding sub-slot, pointed at http://localhost:11434 with model set to nomic-embed-text.

Run the workflow manually on a test document and check the n8n execution log. Each node should complete successfully, and the Qdrant node's output will report the number of points inserted. Open the Qdrant dashboard and you'll see the new collection with the indexed chunks.

For production use, extend this workflow with a few additions: a Loop Over Items node for batch processing multiple files, a Set node before the Default Data Loader to attach metadata (filename, source, date) to each chunk — this metadata gets stored in Qdrant alongside the vector and is returned at retrieval time, letting the LLM cite its sources. You can also add a pre-check that queries Qdrant for existing points matching the filename to skip documents that have already been ingested, preventing duplicate chunks from accumulating.

Step 4: Build the Query Workflow

The query workflow takes a question, retrieves relevant chunks from Qdrant, and generates an answer using the local chat model. The cleanest pattern in n8n is an AI Agent node equipped with a Qdrant Vector Store Tool — the agent decides when to query the vector store, can handle multi-turn conversations via a memory node, and can be configured to cite sources from the chunk metadata.

Create a new workflow and connect these nodes:

1. Chat Trigger — creates a webhook-based chat interface. After saving, n8n gives you a URL you can POST questions to, or access via the built-in chat UI.
2. AI Agent — the central node. Set the system prompt (below) and wire the following nodes to its sub-slots.
3. Chat Model Ollama → AI Agent's Chat Model sub-slot: host http://localhost:11434, model llama3.1:8b
4. Window Buffer Memory → AI Agent's Memory sub-slot: keeps conversation context. Set context window length to 5–10 turns.
5. Qdrant Vector Store Tool → AI Agent's Tool sub-slot: same Qdrant URL and collection name as the ingestion workflow. Wire an Embeddings Ollama sub-node to this tool's Embedding sub-slot, using the same model as ingestion.

System prompt to use in the AI Agent node:

You are a helpful assistant that answers questions about the user's documents.
Use the Qdrant vector store tool to look up relevant information before answering.
If the tool returns no relevant results, say so clearly instead of guessing.
When your answer comes from a retrieved document section, mention the source
if that information is available in the metadata.

The single most critical configuration constraint: the embedding model in the Qdrant Tool's Embeddings sub-node must be identical to the model used during ingestion. If you embedded with nomic-embed-text, you must query with nomic-embed-text. Different models produce vectors in different dimensional spaces — mixing them causes the similarity search to return semantically unrelated chunks, and the LLM's answers appear hallucinated even though they're genuinely generated from the (wrong) retrieved content. It's a silent failure mode that's easy to miss without checking the retrieved chunks in the execution log. For a deeper look at how n8n's agentic framework handles tool calls, the n8n 2.0 AI Agents post covers the full agent/tool pattern.

Tuning Tips and Common Gotchas

Chunk size and overlap are your first tuning lever. Chunk size determines how much context each retrieved piece contains; overlap prevents important context from being split across chunk boundaries and lost at retrieval. For technical documentation or structured text, 500–800 characters with 100–200 character overlap is a reasonable starting range. For prose-heavy content like long PDFs or articles, larger chunks (800–1,200 characters) tend to give the LLM more useful context. Per n8n's LangChain integration docs, the Recursive Character Text Splitter prioritizes splitting on paragraph boundaries, then sentences, then characters — so it respects natural text structure when possible and won't arbitrarily break mid-sentence at the chunk boundary.

Re-ingest after any embedding model change. Switching embedding models requires deleting the Qdrant collection and re-ingesting all documents from scratch. The vectors from different models are mathematically incompatible — there's no migration path. If you're experimenting with multiple models, name your collections after the model (e.g., docs-nomic, docs-mxbai) so you can switch between them in the workflow configuration without losing either index.

Ollama's default context window is 2,048 tokens. For embedding models with larger native context windows (nomic-embed-text supports 8,192), Ollama caps inference at 2K by default. If your chunks are larger than about 500 characters, you may be silently truncating them at embedding time. Fix this by creating a custom model with a higher num_ctx parameter:

ollama show nomic-embed-text --modelfile > /tmp/nomic.Modelfile
echo "PARAMETER num_ctx 4096" >> /tmp/nomic.Modelfile
ollama create nomic-embed-text-4k -f /tmp/nomic.Modelfile

Use nomic-embed-text-4k as the model name in both the ingestion and query workflows. Both must use the same custom model name, not a mix of the base and customized version.

The LLM model choice matters less than the embedding model. Retrieval quality — whether the right chunks actually come back from Qdrant — determines the majority of answer quality. The LLM's job is to synthesize what the retriever found. A well-tuned retrieval setup with a small 7B chat model will routinely outperform a larger model on top of poor retrieval. Tune chunking and embedding first, then upgrade the chat model if answers are accurate but poorly written.

For anyone who wants the theoretical background behind why different chunking and retrieval strategies perform the way they do, AI Engineering by Chip Huyen (~$40) has one of the clearest explanations of RAG architecture tradeoffs in print, including when RAG is the right tool and when fine-tuning or context stuffing outperforms it.

The Complete Stack at a Glance

Here's the full component map for the local RAG pipeline:

The total footprint on a 16 GB machine is comfortable. Both Ollama models can be loaded simultaneously without paging. A collection of 50,000 chunks — representing several hundred average-length PDF documents — uses roughly 200–400 MB of Qdrant's RAM depending on vector dimensions, per Qdrant's performance documentation. On a local NVMe, similarity searches at that scale typically complete in single-digit milliseconds.

The natural extension of this setup is a front end: a web form, a Telegram bot, or a Slack integration — all available as n8n trigger and action nodes. The query workflow already exposes a chat webhook; anything that can POST to it gets access to the local document Q&A system. You can also layer ingestion triggers on top of the patterns in the n8n + Ollama Proxmox automation post to ingest new files automatically when they're added to a watched folder. The result is a private, cost-free knowledge base your team can query in plain language.