How to Build a Private RAG System Over Your Company’s Documents

A retrieval-augmented generation (RAG) system turns your company’s documents into something you can ask questions of, in plain language, and get answers grounded in your own material rather than a model’s general training. The reason to build it privately is the same reason the documents are worth querying in the first place: they are yours. Your contracts, runbooks, source code, support history, and internal wikis never travel to a third-party API, and no prompt log of your staff asking about them sits on someone else’s servers. This is the internal-knowledge workload I set up inside private AI deployments, and it is the one people most often get subtly wrong. Here is how I build it so it holds up.

What RAG actually does (and what it does not)

RAG has two halves. Retrieval finds the passages in your corpus most relevant to a question. Generation hands those passages to a language model and asks it to answer using them. The model is not answering from memory; it is reading the excerpts you retrieved and writing an answer over them, ideally with citations back to the source.

That distinction matters because it tells you where quality comes from. A RAG answer is only as good as the passages retrieved for it. Most disappointing systems are not suffering from a weak language model; they are handing the model the wrong three paragraphs. Retrieval is the part that decides whether the whole thing works.

RAG is also not fine-tuning. Fine-tuning bakes patterns into a model’s weights and is the wrong tool for facts that change: you do not retrain a model every time a policy is updated. RAG keeps your knowledge in an index you can edit, so correcting an answer is a matter of fixing the underlying document and re-indexing it, not retraining anything.

The private stack: what each piece is for

A RAG system is a small pipeline of parts, and running it privately means every one of them runs on infrastructure you control:

• Ingestion: the connectors that pull documents in from wherever they live (a file share, a wiki, a ticketing system, a code repository) and normalize them to clean text.
• Chunking: splitting each document into passages small enough to retrieve precisely but large enough to carry meaning.
• Embedding model: a model that turns each chunk into a vector, a list of numbers that places similar meaning near similar meaning. Run this locally; a compact open model such as nomic-embed-text served through Ollama is enough for most corpora.
• Vector store: the database that holds those vectors and answers “find the nearest passages to this query.” Self-hosted options like pgvector (on Postgres you already run) or Qdrant keep the index inside your network.
• Chat model: the local language model that writes the final grounded answer, again served through your own runtime.
• Orchestration: the glue that, for each question, embeds it, retrieves the top passages, assembles the prompt, and calls the chat model.

Nothing here requires an external service. That is the entire point, and it is what makes RAG a natural fit for organizations that chose private AI for compliance or data-control reasons in the first place.

1. Decide the corpus and the questions first

Before any infrastructure, write down two lists: which documents belong in this system, and the actual questions people will ask it. The corpus definition keeps you from indexing everything reflexively (which buries good answers under noise), and the question list becomes the material you evaluate against later. A knowledge system scoped to “our engineering runbooks” answers well; one scoped to “all company data” answers vaguely about everything.

2. Get the documents in and clean

Ingestion is unglamorous and it is where quality is won or lost. PDFs carry headers, footers, and page numbers that pollute retrieval; exported wiki pages carry navigation chrome; code needs its comments and structure preserved. Strip the boilerplate, keep the substance, and hold on to metadata you will want later: source path, title, last-modified date, and above all the access-control labels that say who is allowed to see each document.

3. Chunk with intent

Chunking is not just “split every 500 words.” Split on the document’s own structure where you can, by heading, by section, by function, so each chunk is about one thing. Chunks that are too large dilute the match and waste the model’s context; chunks that are too small lose the surrounding meaning that made them useful. A modest overlap between adjacent chunks keeps a sentence that straddles a boundary from being orphaned. Attach the source metadata to every chunk, because that is what lets an answer cite where it came from.

4. Run a local embedding model

Embed every chunk with a model you host yourself, and store the resulting vector alongside the chunk text and its metadata. Two rules save pain later. Use the same embedding model for your documents and for incoming questions, since vectors from different models are not comparable. And record which model and version produced the index, because changing the embedding model means re-embedding the entire corpus, not just new documents.

5. Store the vectors in something you control

Load the vectors into your self-hosted store and configure it for the similarity search your retriever will run. This is also the layer where you enforce who can see what: carry the access-control labels from ingestion through to the store, so a retrieval can be filtered to only the documents the asking user is permitted to read. Skipping this is the most common privacy failure in internal RAG, and I return to it below.

6. Retrieve well before you generate

Retrieval quality is the whole game, so treat it as a component you tune, not a line of code you write once. Retrieve more candidates than you need and then re-rank them, so the passages that actually reach the model are the best few rather than merely the first few. Combining vector similarity with plain keyword search (a hybrid approach) catches the cases where the exact term matters, like an error code or a product name that means nothing to a semantic model. Before you worry about the language model at all, inspect what retrieval returns for your real questions: if the right passages are not in that list, no model can rescue the answer.

7. Generate grounded, and cite

Now assemble the prompt: the user’s question, the retrieved passages, and an instruction to answer only from those passages and to say when they do not contain the answer. Grounding the model this way, and requiring citations back to the source documents, is what converts a confident guess into a checkable answer. A reader who can click through to the source paragraph can verify the system; a reader who cannot is being asked to trust it. Instructing the model to admit “the documents do not cover this” is not a weakness, it is the behavior that makes the system safe to rely on.

8. Evaluate honestly

This is the step everyone skips and the one that separates a demo from a deployment. Take the question list from step 1, run it through the system, and judge the answers against what a knowledgeable colleague would say: is the answer correct, is it grounded in the cited passages, and did retrieval surface the right sources. Keep that set as a regression check, because every later change (a new embedding model, different chunking, a swapped chat model) needs to be measured against it rather than eyeballed once and trusted forever. Quality control is a standing role in the system, not a launch-day checkbox, a theme I go into at length in my book You Are the Quality Control.

9. Operate it: updates, access, monitoring

A knowledge base that is not maintained rots quietly: documents change, the index goes stale, and the system keeps answering from last quarter’s policy with full confidence. Decide up front how documents flow in and how often the index refreshes, who owns that pipeline, and how you monitor for retrieval that comes back empty or answers that drift. Handover matters as much here as in any deployment: a private AI system your team cannot operate is a demo, not a deployment.

Where internal RAG goes wrong

The failures I see in practice are consistent, and every one of them traces back to a step above rather than to the choice of model:

• Permission bleed: retrieval returns a passage from a document the asking user was never allowed to read. Once that text is in the answer, the access control you have everywhere else has been quietly bypassed. Filter retrieval by the user’s permissions; never rely on the model to keep a secret it was handed.
• A stale index: the source document was fixed weeks ago but the system still answers from the old version, because nothing re-indexed it.
• Confident answers with no citation: without grounding and sources, a RAG system will still answer when retrieval found nothing useful, and that answer is a hallucination wearing a corporate voice.
• Retrieval nobody looked at: teams tune prompts for weeks while the real problem is that the right passage never made it into the candidate set.

Where this fits

An internal knowledge system is one of the core workloads I deploy under private AI, alongside self-hosted model runtimes and private AI-assisted development. If you are planning one, the Private AI Architecture Checklist is the audit I walk before calling any such deployment production-ready, and How to Run LLMs Inside Your Company Network covers the serving layer this pipeline sits on.

If your organization wants to make its own knowledge queryable without indexing a word of it into an external service, tell me about your situation.