There's a word appearing in almost every product roadmap we see this year: RAG. Retrieval-Augmented Generation. It gets dropped with the same confidence people once reserved for "blockchain" or "microservices." And that, precisely, is the signal that something is about to go wrong.
RAG is not a solution. It's a family of retrieval architectures with completely different cost, latency, complexity, and precision profiles. Choosing "RAG" without further qualification is like saying you'll build a transportation system without specifying whether you need a bicycle, a delivery truck, or a subway line. All are valid. None works for everything.
The real problem isn't technical — it's diagnostic. Many teams adopt RAG because they read it works, not because they've analyzed what specific task they're actually solving and what type of retrieval that task demands. The predictable result: a system that retrieves poorly, generates mediocre responses, and leaves users with less trust in the product than before the "improvement."
- There are at least five RAG architectural variants with clearly differentiated use cases.
- The wrong choice doesn't just affect response quality — it directly impacts infrastructure costs and how fast your product feels.
- The right diagnosis starts with the user's task, not with the available architecture.
The Problem: RAG as an Empty Label
When a client tells us "we want to implement RAG," the first question we ask isn't about embeddings or vector databases. It's: what specific task does your user need to accomplish that they can't do well today? That question makes people uncomfortable, because it forces the conversation out of technical territory and into the territory of purpose.
RAG was born to solve a specific problem: language models have knowledge frozen at their training cutoff and can't access private or updated information without retraining. External retrieval fixes both limitations. Clear enough.
The mistake begins when RAG becomes the destination instead of the path. Teams implementing basic RAG — a flat vector index, cosine similarity retrieval, context injected into the prompt — for cases that require reasoning over multiple related documents, concept hierarchies, or synthesis of contradictory information. The result is a model that retrieves fragments "most semantically similar" to the query, but not the fragments that actually answer it.
Retrieving the fragment most similar to a question is not the same as retrieving the fragment that best answers it. That distinction, ignored, ruins 80% of the RAG implementations we encounter.
This connects to something we've raised before: the silent danger of delegating diagnosis to the tool itself. A poorly configured RAG doesn't fail loudly — it fails with confidence. The model generates a coherent, well-written response, grounded in a fragment that wasn't the right one. The user doesn't know the answer is wrong. The team doesn't either, until someone finds out the hard way.
Architecture: The Five Variants That Matter
There is no "RAG" in the singular. There's a design decision space that produces variants with very different profiles. Here are the five that appear most frequently in real product cases, and when each one makes sense.
Dense RAG (Vector Search)
This is the starting point. Documents are chunked, converted into embeddings, and stored in a vector database (Pinecone, Weaviate, pgvector). At query time, the user's question is embedded and the nearest chunks are retrieved from the vector space.
Works well when: the corpus is relatively homogeneous, queries are semantically direct, and users seek point-in-time information ("what's the return policy?"). It's fast, cheap to maintain, and behaviorally predictable. For an e-commerce support chatbot with a well-structured knowledge base, this architecture is sufficient — you don't need anything more complex.
Fails when: the corpus has strong hierarchical structure (legal regulations, contracts, technical manuals with interdependent sections), when questions require synthesis across multiple sources, or when the user's vocabulary doesn't match the corpus vocabulary.
Hybrid RAG (Dense + Sparse)
Combines semantic vector search with classic lexical search like BM25. The system scores and fuses both rankings. This solves the vocabulary mismatch problem: if the user types an exact technical term that the embedding doesn't capture well due to low training frequency, lexical search still finds it.
This is the natural upgrade for most B2B products with technical documentation. The added cost is moderate and the recall improvement is significant. A common case: software documentation query systems where function names, parameters, or error codes are highly specific terms that embedding models don't always represent well.
RAG with Reranking
Adds an intermediate phase: the system retrieves a broad set of candidates (top-50, for example), then a reranking model — smaller and more specialized than the main LLM — reorders them by actual relevance to the specific query. Models like Cohere Rerank or cross-encoding models from Sentence Transformers do this work.
Reranking decouples the retrieval phase (optimized for coverage, high recall) from the precision phase (optimized so that the 3-5 fragments reaching the LLM are genuinely the best ones). For cases where retrieval errors are costly — medical, legal, financial systems — this step is almost mandatory. The cost is additional latency (50-200ms) and one more infrastructure layer.
Hierarchical RAG (Parent-Child)
Documents are indexed at two levels: small, precise chunks for retrieval, linked to their parent chunks (full sections, entire documents) for context construction. When a child chunk is retrieved, its parent context is also included so the LLM understands the frame around that data point.
This is the right pattern for legal, regulatory, or technical corpora where an isolated data point without its surrounding context can be misinterpreted or incomplete. Also appropriate for product documentation where a step in a process only makes sense within the full sequence.
Agentic RAG (Multi-hop / Graph RAG)
The most complex variant. The system doesn't make a single retrieval: it reasons about what it needs to find, executes multiple chained searches, evaluates whether the collected information is sufficient, and decides if it needs more context before generating a final response. Some implementations add a knowledge graph over the vector corpus (Microsoft GraphRAG is the most cited example).
Works for questions requiring multi-step reasoning over complex corpora: cross-contract analysis, regulatory research with multiple interdependent exceptions, or competitive intelligence systems. The cost is high: seconds of latency, multiple LLM calls, more sophisticated infrastructure. Justified only when the task demands it and when the user tolerates — or even expects — a response that takes time.
Decision: The Selection Framework We Use
The question that organizes every decision is always the same: what's the penalty for a retrieval error in this system? The higher the cost of a mistake — in user trust, regulatory consequences, or business decisions based on the response — the more sophisticated the retrieval architecture needs to be.
From there, we propose three evaluation vectors before committing to an architecture:
Nature of the corpus. Is it homogeneous or heterogeneous? Does it have strong hierarchical structure (regulations, contracts) or is it relatively flat (blog articles, FAQs)? Does the corpus vocabulary match the user's vocabulary, or is there a terminological gap? A legal corpus in a specialized domain often contains vocabulary that general embedding models don't capture well — that already points toward hybrid RAG at minimum.
Query typology. Are these direct point-retrieval questions ("how much does X cost?") or analytical questions requiring synthesis ("what are the implications of clause 23 for contracts signed before 2022?")? The latter need multi-hop or at least reranking. The former don't.
Acceptable cost-latency profile. A customer support assistant has a 1-2 second response SLA. A contract analysis system for a legal department can tolerate 15 seconds if the response is reliable. That difference determines which architectures are practically available.
The right retrieval architecture isn't the most sophisticated one. It's the minimum that covers the maximum penalty your use case can tolerate.
There's an additional factor often ignored in technical discussions: how users perceive latency in complex RAG systems can sink the experience even when the answer is correct. Agentic RAG with four chained calls can take 8-12 seconds. If the user doesn't understand that time is necessary because the system is "reasoning," they'll perceive the product as slow and unreliable. Designing the waiting experience is part of the architecture, not an afterthought.
Fine-tuning vs. RAG: The Question Nobody Wants to Answer
Before committing to any RAG variant, it's worth asking an uncomfortable question: do I actually need RAG, or do I need a model fine-tuned on my domain?
RAG is the right answer when knowledge is dynamic (changes frequently), when the corpus is too large to fit in context, or when you need citability — the ability to point to where each answer came from. Fine-tuning is the right answer when the model needs to learn a reasoning style, highly domain-specific vocabulary, or a consistent response format that can't be achieved through prompting alone.
The Small Fine-tuned Model Case
Recent results from fine-tuning small local models on classification and categorization tasks point to something we've been advocating for a while: for bounded, predictable tasks, a well-trained small model outperforms a large model with complex RAG on both cost and latency. And it runs on-premise, with no external API dependency, no per-token cost, and full data sovereignty.
The fine-tuning vs. RAG decision isn't technical at its root — it's functional. What does the system need to know that it can't know any other way? If the answer is "frequently updated or very voluminous information," RAG. If the answer is "how to reason about my specific domain," fine-tuning. Many real-world cases combine both: a fine-tuned model for domain reasoning style, with RAG for access to current information.
This tension between specialization and generality connects directly to how SaaS products are rethinking their AI architecture from the ground up — not by layering on top of general models, but by building specialized layers that deliver real advantage in a specific niche.
Conclusion: Diagnose First, Architecture Second
RAG implemented well can transform product utility. RAG chosen poorly — the wrong variant for the wrong task — adds complexity, cost, and latency without improving the user experience. And worst of all: it fails silently, generating plausible but incorrect responses that erode user trust more slowly but more deeply than an obvious error.
The right path starts with the task. What does the user need to do that they can't do well today? How costly is a mistake? What latency can they live with? Does the corpus change frequently or is it stable? Those four questions are enough to eliminate three of the five variants and focus the design on the two that actually compete.
If you're at that decision point — RAG architecture, fine-tuning, or a combination — and need a diagnosis without prior commitment to any technology, that's exactly what we do at Room 714. The right architecture isn't the most expensive or the most modern. It's the one that solves your case with the minimum necessary complexity.
