Self-hosted RAG on local models: where to start
When cloud APIs are off the table but search over your own documents is still needed. The minimal stack and the order of steps.
Cloud APIs are convenient until a requirement appears to keep data inside the perimeter. Then people build self-hosted RAG: search over their own documents plus a local model that answers from the retrieved fragments.
What it consists of
- Document store. A folder of files and a script that reads them.
- Chunking. 500–800 tokens with 100–150 overlap: smaller loses context, larger retrieves the wrong thing.
- Embeddings. A local embedding model, running on CPU or GPU.
- Vector index. Any local store; SQLite with an extension is enough for tens of thousands of chunks.
- Generation. A local LLM that receives the fragments and the question.
Order of steps
Build the corpus first and test retrieval without the model: type a question and look at what comes back. If the right fragment is not in the top five, generation will not save you.
Then connect the model and make it answer only from the fragments, citing their numbers.
Common mistakes
- One index for everything. Policies and chat logs do not retrieve well together.
- A prompt without limits. Without "if the fragments do not contain the answer, say so", the model makes things up.
- Testing on three questions. You need thirty to fifty real questions with known answers.