Loading...
The RAG-based Q&A agent enables employees to query a large enterprise knowledge base (internal wikis, policy documents, technical manuals, HR docs) and receive accurate, cited answers. The agent retrieves semantically relevant document chunks, reasons over them, and synthesizes a grounded response with source citations — reducing hallucinations and ensuring answers are traceable.
Users & Use Cases:
End users (employees) submit natural-language questions via a chat interface. The agent handles single-hop questions (direct lookup), multi-hop questions (requiring synthesis across multiple documents), and follow-up questions within a conversation session.
Autonomy & Constraints:
The agent autonomously decides when to retrieve, how many chunks to fetch, and whether to re-query with a refined search term. It must not answer questions it cannot ground in retrieved documents — if no relevant context is found, it must say so explicitly. It must not expose documents beyond the user's access permissions. Responses must always include source document references.
Performs a hybrid search combining dense vector similarity (semantic) and sparse BM25 keyword search. Inputs: query string, top_k (default 10), filters (document type, date range, department). Returns: ranked list of document chunks with metadata (source, page, score). This is the primary retrieval mechanism.
Tool 2 — Re-ranker (rerank_chunks):
Takes the top_k retrieved chunks and applies a cross-encoder re-ranking model (e.g., Cohere Rerank or a local cross-encoder) to reorder chunks by actual relevance to the query. Reduces noise before the generation step.
Tool 3 — Document Metadata Lookup (get_document_metadata):
Fetches title, author, last-updated date, and access tier for a given document ID. Used to populate source citations in the final answer.
Tool 4 — Query Decomposer (decompose_query):
For complex multi-hop questions, decomposes the original question into 2–3 sub-questions that can each be answered independently through retrieval. Results are then merged during reasoning.
Ingestion Pipeline (offline):
Documents are chunked using a sliding-window strategy (512 tokens, 64-token overlap) with structure-aware splitting (respecting headers and paragraphs). Each chunk is embedded using a bi-encoder (e.g., text-embedding-ada-002) and stored in a vector database (Pinecone / Weaviate / pgvector) alongside BM25 inverted indexes.
The agent follows a ReAct loop: it reasons about what information is needed, selects and calls a retrieval tool, observes the results, and either retrieves more (if insufficient) or proceeds to generate a final answer. This loop runs for a maximum of 3 iterations to prevent runaway retrieval.
Component Breakdown:
Orchestration: Single-agent, synchronous. For queries requiring parallel sub-question retrieval, sub-retrievals can be executed concurrently before merging.
A sliding conversation buffer stores the last N turns (typically 10–20 exchanges) as part of the LLM's system prompt context. This allows the agent to resolve references like "what about the second point you mentioned?" and maintain coherent multi-turn conversations. Stored in a fast in-memory store (Redis) with session-scoped TTL.
Long-Term Memory (External Knowledge):
The vector database serves as long-term, read-only semantic memory over enterprise documents. It is not updated during conversations — it represents the static (or periodically refreshed) knowledge base. Chunks are stored with embeddings, source metadata, and access-control tags.
Episodic Memory (Per-Query Working Memory):
Each query invocation creates a temporary working context: the original question, decomposed sub-questions, retrieved chunks, re-ranked results, and intermediate reasoning traces. This is passed through the ReAct loop and discarded after the response is generated.
Context Window Management:
The agent uses a priority-based context assembly strategy: the most relevant re-ranked chunks are included first, followed by relevant conversation history, then system instructions. If the total exceeds the LLM context budget, older conversation turns and lower-scored chunks are dropped first. A summarization step can compress long conversation histories into a compact episodic summary.
Decision Logic — When to Retrieve vs. Answer Directly:
The agent first checks if the question can be answered from conversation history alone (e.g., clarification of a previous answer). If not, it always retrieves before generating. It decides retrieval scope by classifying questions as: (a) single-hop — one retrieval call suffices; (b) multi-hop — requires query decomposition and multiple retrievals; (c) ambiguous — agent asks a clarifying question before retrieving.
Retrieval Adequacy Check:
After retrieval, the agent scores the top chunks against the query. If the maximum relevance score falls below a threshold (e.g., 0.6), it attempts one query reformulation (paraphrase or expand with synonyms) and retries. If still inadequate after 2 attempts, it responds with "I don't have enough information in the knowledge base to answer this confidently."
Hallucination Mitigation:
The generation prompt explicitly instructs the LLM: "Answer ONLY using the provided context. If the context does not contain the answer, say so. Do not infer beyond what is stated." A post-generation fact-checking step cross-references each factual claim against the retrieved chunks and removes or flags unsupported statements.
Confidence & Uncertainty Expression:
The agent expresses uncertainty explicitly when: retrieved chunks are borderline relevant, the question spans document gaps, or the top retrieved chunk scores are low. It uses phrases like "Based on available documentation..." or "This may be outdated — the document was last updated in [date]."
Multi-Document Synthesis:
When answering requires information from multiple chunks or documents, the agent uses a structured merge: it identifies non-overlapping facts from each source, resolves contradictions by preferring the most recently updated document, and synthesizes a coherent answer while attributing each fact to its source.
Access Control (Permission Boundaries):
Every retrieval call includes the user's role and department as filter metadata. The vector database enforces document-level access control: documents tagged for specific teams (e.g., "HR-confidential", "Legal-internal") are only retrievable by users with matching role claims in their session token. The agent never surfaces content from documents outside the user's access tier.
Input Guardrails:
The agent validates incoming queries: (a) length limit of 1,000 characters; (b) PII detection — if the query contains sensitive personal data (SSNs, credit card numbers), the agent refuses and redirects the user; (c) prompt injection detection — queries attempting to override system instructions or extract system prompts are blocked and logged.
Output Guardrails:
Before responding, the output is checked for: (a) presence of any confidential document tags in the synthesized text that should not be exposed verbatim; (b) personally identifiable information that may have appeared in knowledge base documents; (c) the system prompt or tool descriptions — if detected in output, the response is rejected and regenerated.
Rate Limiting & Abuse Prevention:
Per-user rate limits (e.g., 60 queries/hour) prevent abuse and manage costs. Anomalous query patterns (e.g., attempts to enumerate all document contents) trigger automatic throttling and alert the security team.
Audit Logging:
Every query, the retrieved document IDs, the LLM prompt, and the generated response are logged with a timestamp and user identifier. Logs are immutable and retained for compliance purposes. This enables post-hoc auditing of what information was surfaced to whom.