The fourth stop on my AI roadmap answers the question every beginner asks a week in: “how do I make it answer about my stuff?” A model knows its training data and nothing else, not your policy documents, your product manual, or last week’s notes. The fix is not to retrain it. The fix is to look up the right passages first and hand them over with the question. That pattern has a name, RAG, and getting it to work is mostly a few decisions made well.
Look it up first, then ask
Picture asking a sharp assistant a question about a thick manual they have never read. You would not expect them to guess. You would open the manual to the right page, hand it over, and ask. That is exactly RAG: Retrieval-Augmented Generation. Retrieve the relevant text, add it to the prompt, generate the answer from it.
The whole system is a pipeline with two halves: a one-time (and then ongoing) job of indexing your documents, and a per-question job of finding the right bits.
flowchart LR D[Your documents] --> C[Split into chunks] C --> E[Turn each into an embedding] E --> V[(Vector database)] Q[User question] --> QE[Embed the question] QE --> S[Find the closest chunks] V --> S S --> P[Put them in the prompt] P --> M[Model answers, with citations]
Two words to unpack. An embedding is a list of numbers that captures the meaning of a piece of text, so that texts about similar things end up numerically close. A vector database stores those number-lists and can find “the chunks closest in meaning to this question” fast. Similarity of meaning, not matching words, is what makes this work.
The decisions that decide whether it works
Every RAG tutorial gets you a demo in an hour. Whether it gives good answers on real documents comes down to a few choices the tutorial skips.
Chunking. You cannot embed a whole manual as one blob, so you split it into chunks. Too big and a chunk drags in irrelevant text; too small and it loses the context that made it meaningful. Splitting where the meaning shifts (a heading, a topic change) beats splitting every N characters.
Parsing. Real documents are PDFs, scanned pages, and web pages, not clean text. Getting good text out of them (including OCR for scans) is unglamorous and matters more than any model choice.
Hybrid search. Meaning-based search is bad at exact things: a product code, a person’s name, an error string. So production systems run a keyword search alongside the embedding search and merge the results. One reported case saw retrieval accuracy improve by around 35 percent from this alone.
Reranking. The first search is fast and rough. A reranker takes the top results and re-scores them by actually reading question and passage together, then keeps only the best few. Cheap to add, and one of the biggest quality wins.
Metadata filtering. Tag chunks with their source, date, and type, so a question about “the 2026 pricing policy” only searches that, instead of everything.
TIP
The order to improve a weak RAG system, from the field: chunk better, retrieve hybrid, rerank aggressively, and measure. Most quality comes from those four, not from swapping the model.
Measure it, or you are guessing
The trap is judging RAG by reading a few answers. You need a small test set: real questions with known correct sources, and a check of whether the right chunk was actually retrieved. The common measure is recall at k: of the top k chunks fetched, was the correct one in there? If retrieval fails, no model can save the answer, so this is the number to watch first.
NOTE
Always make the model cite which chunk each claim came from, and show those citations to the user. It is the difference between “trust me” and “here is the paragraph”, and it makes a wrong answer visible instead of confident.
Do you even need RAG?
Context windows are now huge, and every time they grow someone declares RAG dead. It is not, but the boundary moved. If your whole knowledge base fits comfortably in the window and rarely changes, just put it in the prompt and skip the pipeline. RAG earns its complexity when the data is large, changes often, needs to be searched precisely, or you need to show where an answer came from.
flowchart TD
Q{Does all your data fit in
the context window, and stay stable?} -->|yes| L[Long context: put it in the prompt]
Q -->|no, it is large or
changes often| R[RAG: index it, retrieve per question]
Keeping the index fresh is its own job: an ingestion pipeline that re-processes documents when they change, so the model is not answering from last year’s version.
The tools you actually reach for
| Job | Popular tools | Note |
|---|---|---|
| Vector database | pgvector, Qdrant, Chroma, Pinecone | pgvector runs inside Postgres you already have; Chroma is free/OSS |
| Embeddings | Provider embedding models | Pay per token, cheap |
| Document parsing | Unstructured, LlamaParse, OCR tools | Handle PDFs and scans |
| Reranking | Cohere Rerank, open cross-encoders | Big quality win for little cost |
| Measuring retrieval | Ragas, promptfoo | Free/OSS; test recall before trusting answers |
Retrieval is the whole game
The lesson from this stop is that “AI on my data” is mostly a search problem wearing an AI hat. If the right passage reaches the model, even a modest model answers well. If it does not, the best model in the world confidently makes something up. So the work is in the pipeline: parse well, chunk sensibly, search by meaning and by keyword, rerank, cite, and measure whether the right page actually showed up.
Get the right page in front of it, and the answer takes care of itself.
Sources
- AI with Aish: all you need to know about RAG in 2026 for the index, retrieve, augment, generate pipeline
- CallMissed: RAG best practices 2026, chunking, reranking, hybrid search for the improvement order and the hybrid search result
- Labo LLM: RAG panorama 2026, vector and emerging approaches for RAG versus long context
- StackViv: RAG and vector databases guide 2026 for vector database options including pgvector