The short version
RAG is worth building if and only if: your knowledge exceeds the context window, it changes often, and you can live with roughly 80% retrieval accuracy.
Otherwise, there is a simpler option.
Three situations where it gets misused
1. Total documents under 200KB
Just put everything in the context. Claude 4.6 handles 1M tokens; a few hundred KB is nothing. The retrieval noise RAG introduces will hurt you more than it helps.
2. Questions cluster tightly
If 80% of user questions land on 20% of the content, build a FAQ cache table. Return directly on a hit, fall through to the model on a miss. Ten times simpler than RAG.
3. The knowledge needs to be understood together
RAG retrieves fragments. It can tell you “X is on page P”, but it is bad at “what do X, Y, and Z mean taken together”. Those questions want full text plus a long context.
What RAG actually costs
Not the vector database bill. These:
- Chunking strategy — chunk too large and you retrieve noise, too small and you lose context
- Embedding updates — every document edit means recomputing embeddings
- Retrieval evaluation — how do you know the recall is any good? You need a test set and ongoing monitoring
RAG isn’t dumping PDFs into Pinecone and calling it done. It’s a subset of building an information retrieval system.