Loading...
why RAG?
Freshness - instead fine tune everytime, RAG indexes new documents.
Traceability - RAG can cite exact source page
Cost - Low for RAG
Hallucination control: RAG grounds answers in retrieved text, and you can verify every claim
Access Control: RAG can filter documents by user permissions at query time
To design a production grade RAG QA Agent over an enterprise knowledge. Succcess criteria with 80% faithfullness, response in < 3 sec & source of information.
Key capabilities:
Example queries the agent must handle:
Self Autonomous; if confident score > 65% proceed, else no hallucination. Must handle 10,000+ documents without degrading retrieval, 99.9% availability, Document format challenges, Scale trajectory: Plan for 10x growth
Query embedding: about $0.0000004 per query (20 tokens). Essentially free. Latency: about 50ms.
Retrieval (vector + BM25): $0.01-0.05 per 1,000 queries with a managed vector DB. Latency: about 50-100ms.
Re-ranking: about $0.0004 per query (cross-encoder on 20 chunks). Latency: about 150-300ms.
LLM generation: $0.001-0.05 per query depending on model. This is 80-90% of the total cost. Latency: 1-2s (dominates the budget).
Faithfulness check (optional): about $0.0005 per query. Latency: 500ms-1s (run async after streaming starts).
Vector Search
Finds semantically similar chunks using embedding cosine similarity
Keyword Search (BM25)
Finds exact term matches using an inverted index.
Cross-Encoder Re-ranker
Scores each candidate chunk against the query for fine-grained relevance.
Citation Extractor
Maps each claim in the generated answer back to its source chunk.
Confidence Scorer
Evaluates whether the retrieved context is sufficient to answer the question. for latency-critical queries, you can skip the re-ranker and use raw retrieval scores.
vector cosine similarity (0 to 1) and BM25 scores (unbounded positive numbers)
RAG agent uses a fixed pipeline: every query runs through the same sequence.
architecture question for a RAG agent is a pipeline with an agent-like decision point.
A ReAct-style agent (reason, act, observe, repeat) would work, but it is overkill for RAG. Retrieval is not optional, The steps are fixed, Latency matters, Debuggability
Multi-agent adds coordination overhead (message passing, shared state, failure handling) that is not justified when a single pipeline handles the workload
Query Analyzer: It classifies query type (factual, comparison, how-to, summarization), resolves coreferences using conversation history, expands abbreviations, and for complex queries, decomposes into sub-queries. Output: rewritten standalone query + query type + optional sub-queries by a single lightweight LLM call (a small model like GPT-4o-mini is sufficient) with a structured output schema.
Hybrid Retriever: Runs vector search and BM25 in parallel, merges results using reciprocal rank fusion (RRF)
Re-ranker: Cross-encoder scores top 20-30 candidates.
Confidence Gate: If the top chunk relevance score is below threshold (e.g., 0.65), routes to the "insufficient context" path.
Generator: LLM call with system prompt, retrieved chunks, conversation history, and the user query. Instructed to cite sources and say "I don't know" if context is insufficient.
Faithfulness Checker: Compares each claim in the answer against source chunks.
Citation Formatter: Attaches document references in a consistent format.
Generation tests: Given perfect chunks (manually selected), does the LLM produce a faithful answer?
Memory for a RAG agent means two things: the knowledge store (the indexed document corpus) and the conversation memory (what the user asked previously in this session)
A production RAG system maintains two parallel indexes over the same chunks:
Vector index (for semantic search): Each chunk is stored as a dense embedding vector.
BM25 index (for keyword search): An inverted index maps each word to the chunks
The knowledge store holds all indexed document chunks with their embeddings. pgvector, Pinecone / Weaviate / Qdrant, Chroma
embedding model determines how well your retrieval works.
How the vector database organizes embeddings determines search speed and recall
HNSW (Hierarchical Navigable Small World)Builds a graph where each node connects to its approximate nearest neighbors. Search is fast (about 1-10ms for 1M vectors) with high recall (above 95%).
IVF (Inverted File Index)Clusters vectors into buckets. At query time, only searches the closest clusters.
Flat (brute force): Scans every vector. Perfect recall but O(n) latency. Only viable for small indexes (under 100K vectors).
Embedding storage: Each chunk has a 1536-dimensional vector. Each dimension is a 32-bit float = 4 bytes. Per vector: 1536 x 4 = 6,144 bytes = about 6 KB. Total embeddings: 120,000 x 6 KB = about 720 MB.
Total storage: about 2-2.5 GB for 10,000 documents
Recursive character splitting -- paragraph to sentence to words
Semantic chunking groups sentences by topic coherence using embedding similarity.
Document-structure-aware chunking uses the document's own hierarchy (headings, sections, lists) to define chunk boundaries
Every chunk should carry metadata. This is not optional because it enables citations, access control, and freshness ranking.
Keep the last N turns (5-10) in a sliding window. For longer conversations, summarize older turns. Do not stuff the entire conversation into the LLM context because it wastes tokens and dilutes the retrieval signal.
Store conversation state in a session object (Redis or in-memory with TTL).
Token budget for history:If the conversation is longer, compress older turns into a running summary.
The confidence gate is where the pipeline becomes "agentic." It makes a routing decision: answer, clarify, or escalate. This is a simple threshold check, not a full reasoning loop.
Graduate to an agentic architecture only when you hit these specific triggers Multiple data sources, Complex multi-step reasoning, Task execution
What guardrails and safety mechanisms does the agent have? Define permission boundaries, human-in-the-loop checkpoints, content filtering, rate limiting, and how the agent handles adversarial inputs or unexpected states.