What It Takes to Build a RAG System Worth Measuring?
Most RAG tutorials give you the same thing: load some documents, embed them, store them in a vector database, query with cosine similarity, pass the results to an LLM, done. It works. It also fails in ways that only become obvious when you try to measure it.
I spent time building an agentic RAG system without LangChain, without LlamaIndex, without any framework abstracting the parts that actually matter. This is what I learned.
Why build without frameworks
The honest answer is that frameworks hide the failure modes. When your RAG system gives a wrong answer, LangChain does not tell you whether the problem was in the chunking, the retrieval, the ranking, or the generation. You have a black box that occasionally lies, and no way to know which part is responsible.
Building from scratch forces you to understand each layer independently. It also means you can measure each layer independently, which turns out to be the entire point.
The architecture
Ingestion splits documents by sentence boundary with token overlap. A fact that lands at the boundary of two chunks will appear in at least one complete chunk. Without overlap, you lose context at every split.
Hybrid retrieval runs dense semantic search via sentence-transformers in Qdrant and BM25 keyword search in parallel, merged using Reciprocal Rank Fusion. 15 lines of Python. Consistently outperforms either method alone.
Query routing classifies every query before retrieval runs. Some questions are answerable from general knowledge — retrieving for them wastes latency and adds noise. Routing accuracy on the eval dataset: 100%.
Generation uses a grounded prompt: answer only from provided context, be direct. The default LLM behavior produces confident-sounding answers even when context does not support them.
The critique loop is where the system earned the word "agentic."
The part that surprised me
The SELF-RAG critique loop did not behave the way I expected. Running one API call per sentence meant a five-sentence answer triggered five critique calls. The latency was not acceptable.
The fix was batching. A single API call now scores all sentences at once. One call instead of five. Same grounding quality, fraction of the latency. This pattern shows up everywhere in production AI systems and almost nowhere in tutorials: the naive implementation of a correct idea is often too slow to be useful. The optimization is not an afterthought — it is part of the design.
The part that was harder than expected
The eval harness. Not because it is technically complex. What was hard was writing the ground truth questions. Good eval questions are specific enough to have a correct answer, varied enough to stress different parts of the system, and honest enough to catch the failure modes you are trying to measure.
The eval harness is the part of this project that most closely resembles what AI engineers actually do at companies building production systems. Not the retrieval architecture. Not the critique loop. The measurement infrastructure that tells you whether any of it is working.
Final numbers: Routing accuracy 100% · Answer quality 4.3/5 · Naive RAG baseline 3.1/5. The agentic loop accounts for the difference.
The actual takeaway
Building RAG without frameworks is not about being contrarian. It is about being able to answer the question that matters in any real deployment: why is this wrong, and which part do I fix?
A framework gives you a working system quickly. It does not give you the ability to debug it, measure it, or improve specific components without touching everything else. For a demo, that is fine. For a system that needs to be reliable, it is not.
The repo is at github.com/HadithSalahshour/agentic-rag. The corpus is the FastAPI official documentation.
Stack: Python, Qdrant, sentence-transformers, rank_bm25, FastAPI, Claude API. No LangChain. No LlamaIndex.
