Agent Requirements

Agent Objective:

What is the agent designed to accomplish? Define the core task and success criteria...

The agent is designed to answer user questions over a large enterprise knowledge base by retrieving the most relevant internal documents, reasoning across them, and generating accurate, concise answers with source citations.

The core task is not simply to chat with the user, but to provide grounded enterprise knowledge answers. For every user query, the agent should understand the question intent, retrieve relevant evidence from approved knowledge sources, synthesize the answer, cite the supporting documents or passages, and clearly communicate uncertainty when the available evidence is insufficient.

Success criteria include:

• Answer accuracy: The response should be factually correct and grounded in retrieved enterprise documents.
• Faithfulness: The answer should only make claims supported by retrieved sources.
• Citation quality: Important claims should include citations to the specific documents or chunks used.
• Retrieval relevance: The system should retrieve the most useful passages using hybrid search, semantic search, keyword search, metadata filters, and reranking.
• Coverage: For questions that span multiple documents, the agent should identify and combine all necessary evidence instead of relying on a single source.
• Freshness: When multiple versions of a document exist, the agent should prioritize the latest authoritative source.
• User clarity: The answer should be easy to understand, directly address the question, and call out any ambiguity or missing information.
• Safety and compliance: The agent should respect enterprise access control, data sensitivity, and privacy rules.

Example: If a user asks, “What is our current PTO carryover policy for California employees?”, the agent should retrieve the latest HR policy documents, filter by region and employee type, cite the relevant policy section, and avoid answering from outdated or unauthorized documents.

Autonomy & Constraints:

What level of autonomy does the agent have? What are the boundaries and constraints it must operate within?

The agent has bounded autonomy. It can independently perform retrieval, query decomposition, reranking, evidence selection, answer generation, citation generation, and self-verification. However, it must operate within strict enterprise boundaries.

The agent can autonomously:

• Interpret the user’s question and identify whether it is simple, ambiguous, or multi-hop.
• Rewrite or expand the query to improve retrieval quality.
• Break complex questions into sub-questions when multiple documents or concepts are involved.
• Search across approved enterprise knowledge sources using hybrid retrieval.
• Apply metadata filters such as document type, department, date, region, product, or access level.
• Rerank retrieved chunks based on relevance, authority, freshness, and user intent.
• Generate an answer with citations.
• Run a hallucination or grounding check before returning the final response.
• Ask a clarifying question when the original question is too ambiguous to answer reliably.
• Say “I don’t have enough information” when retrieved evidence is insufficient.

The agent cannot:

• Access documents the user is not authorized to see.
• Bypass role-based access control, document permissions, or data classification policies.
• Modify, delete, or publish enterprise documents.
• Take external actions such as sending emails, filing tickets, or updating systems unless explicitly designed and authorized to do so.
• Invent facts, policies, numbers, dates, or decisions that are not supported by retrieved evidence.
• Use stale documents when a newer authoritative version is available.
• Expose sensitive information such as PII, confidential strategy, legal documents, financial data, or security details unless the user has permission.
• Provide legal, financial, HR, or compliance guidance beyond what is supported by approved source documents.
• Treat retrieved content as correct without considering document authority, freshness, and conflicts.

When evidence conflicts across documents, the agent should prefer the most recent and authoritative source, explain the conflict if needed, and cite both sources. When confidence is low, the agent should avoid over-answering and instead provide a qualified response or ask for clarification.

Overall, the agent should behave like a trusted enterprise research assistant: proactive in finding and synthesizing information, but conservative about claims, permissions, and unsupported conclusions.

Tool & Action Design

Define the tools and actions available to the agent. Specify tool interfaces, input/output schemas, and how the agent selects which tool to use...

The agent should have access to a controlled set of tools that support retrieval, evidence gathering, reasoning, verification, and answer generation. The tools should be designed as narrow, auditable actions rather than giving the agent unrestricted access to enterprise systems.

Available Tools

1. Query Understanding Tool

Purpose: Analyze the user’s question and determine intent, required sources, filters, and whether the question is ambiguous or multi-hop.

Input:

Output:

The agent uses this tool first for most questions to determine how to search and whether it should ask a clarification question before retrieving documents.

2. Access Control / Permission Check Tool

Purpose: Ensure the user is authorized to access the requested knowledge sources or document types.

Input:

Output:

This tool must run before retrieving sensitive enterprise data. The agent cannot bypass document permissions or expose information from unauthorized sources.

3. Hybrid Search Tool

Purpose: Retrieve a broad set of candidate chunks using both semantic vector search and keyword/BM25 search.

Input:

Output:

The agent selects this tool when the question requires knowledge from the enterprise knowledge base. Hybrid search is preferred because semantic search captures meaning while keyword search helps with exact terms, policy names, product names, error codes, acronyms, and employee-specific terminology.

4. Metadata Filter Tool

Purpose: Narrow results by structured attributes such as department, product, region, document type, date, author, or status.

Input:

Output:

This tool is especially useful when multiple versions of a document exist or when the user’s question has clear constraints.

5. Reranking Tool

Purpose: Re-score retrieved chunks using a more precise relevance model and select the best evidence for answer generation.

Input:

Output:

The agent uses reranking after initial retrieval to ensure the LLM receives the most relevant and authoritative chunks, rather than simply relying on the raw search results.

6. Document Fetch Tool

Purpose: Retrieve the full document section or neighboring chunks when more context is needed.

Input:

Output:

This tool is selected when a retrieved chunk appears relevant but lacks enough surrounding context to answer confidently.

7. Citation Builder Tool

Purpose: Create source citations that map answer claims back to specific documents, sections, or chunks.

Input:

Output:

The agent uses this tool before returning the final response to ensure important claims are properly attributed.

8. Grounding / Hallucination Detection Tool

Purpose: Verify whether the generated answer is fully supported by retrieved evidence.

Input:

Output:

If unsupported claims are detected, the agent should revise the answer, retrieve more evidence, ask a clarification question, or state that the available documents do not contain enough information.

9. Conflict Detection Tool

Purpose: Detect conflicting information across documents, especially when multiple versions or departments provide different answers.

Input:

Output:

When conflicts are found, the agent should prefer the latest authoritative source and mention the discrepancy when relevant.

10. Clarification Tool

Purpose: Ask the user a follow-up question when the request is too broad, ambiguous, or missing required context.

Input:

Output:

The agent should use clarification before retrieval when the query is too vague to retrieve high-quality evidence.

Tool Selection Logic

The agent should select tools based on the question type and confidence level.

For a simple factual enterprise question, the flow is:

1. Use Query Understanding Tool.
2. Use Access Control Tool.
3. Use Hybrid Search Tool.
4. Use Reranking Tool.
5. Use Citation Builder Tool.
6. Use Grounding Detection Tool.
7. Generate final answer with citations.

For a complex multi-document question, the flow is:

1. Use Query Understanding Tool.
2. Decompose the question into sub-questions.
3. Run Hybrid Search for each sub-question.
4. Apply Metadata Filtering.
5. Use Reranking.
6. Fetch surrounding document context if needed.
7. Detect conflicts across sources.
8. Synthesize the final answer.
9. Build citations.
10. Run grounding verification.

For an ambiguous question, the flow is:

1. Use Query Understanding Tool.
2. If required fields are missing, use Clarification Tool.
3. Wait for the user’s clarification before retrieving documents.

For a sensitive or permissioned question, the flow is:

1. Use Query Understanding Tool.
2. Use Access Control Tool.
3. If access is denied, return a safe response explaining that the user does not have permission.
4. If access is allowed, continue with retrieval and answer generation.

For a low-confidence answer, the flow is:

1. Retrieve and rerank evidence.
2. Generate draft answer.
3. Run Grounding Detection Tool.
4. If unsupported claims exist, either retrieve additional evidence, revise the answer, or say that the knowledge base does not contain enough information.

Final Answer Action

The final response should include:

• A direct answer to the user’s question.
• Citations for important claims.
• A note when information is missing, outdated, conflicting, or uncertain.
• No unsupported assumptions.
• No content from documents the user is not authorized to access.

The agent should not expose raw chain-of-thought reasoning. Instead, it should provide a concise explanation of how the answer was derived, such as: “I found the answer in the 2025 California Employee Handbook and confirmed it against the HR PTO FAQ.”

Agent Architecture

Design the overall agent architecture. Choose an orchestration pattern (single agent, multi-agent, hierarchical). Identify key components and how they interact. Use the diagramming tool to illustrate the architecture.

Orchestration Pattern

I would use a single orchestrator agent with specialized tools and services.

The main LLM acts as the planner, reasoner, and answer generator. It does not directly access raw enterprise data or make unrestricted decisions. Instead, it calls controlled tools for query understanding, access control, retrieval, reranking, document fetching, citation building, conflict detection, and grounding verification.

This design is preferable to a fully multi-agent architecture because enterprise Q&A requires strong control, predictable behavior, auditability, permission enforcement, and low hallucination risk. A single orchestrator is easier to monitor and debug, while still allowing modular tools to handle specialized tasks.

High-Level Architecture

Key Components

1. Chat UI / API Gateway

The user interacts with the agent through a chat interface or API. The gateway receives the user question, forwards user identity and session context, and returns the final answer.

2. Authentication & User Context

This layer identifies the user and retrieves user-specific context such as role, department, location, permissions, and access groups. This information is critical because the agent must only retrieve documents the user is authorized to see.

3. Agent Orchestrator / Main LLM

The main LLM is the central reasoning layer. It decides what workflow to follow based on the user’s query.

Its responsibilities include:

• Understanding the user’s intent.
• Deciding whether clarification is needed.
• Planning retrieval steps.
• Decomposing complex questions.
• Calling retrieval and verification tools.
• Synthesizing evidence into a final answer.
• Producing a user-friendly response with citations.

The orchestrator does not directly bypass tools or access documents without permission checks.

4. Query Understanding

This component classifies the query and extracts useful structure.

Example output:

The output helps the agent decide whether to do a simple lookup, multi-hop retrieval, comparison, summarization, or clarification.

5. Retrieval Planner

The retrieval planner decides how to search.

For example:

• Use one search for a simple single-hop question.
• Break the question into sub-queries for multi-hop questions.
• Apply metadata filters for region, department, document type, date, and status.
• Increase top_k when the query is broad or ambiguous.
• Fetch neighboring chunks when more context is needed.

6. Access Control Service

Before retrieving enterprise content, the agent checks whether the user has access to the requested sources.

For example, an HR employee may access draft HR policy documents, while a general employee may only access published handbook content.

If access is denied, the system should return a safe response instead of retrieving or exposing restricted content.

7. Hybrid Search Layer

The hybrid search layer retrieves candidate chunks using both:

• BM25 keyword search for exact terms, acronyms, IDs, policy names, and error codes.
• Semantic vector search for meaning-based matching.

This combination improves retrieval quality because enterprise questions often include both natural language and exact internal terminology.

8. Metadata Store

The metadata store contains structured document attributes such as:

• document owner
• department
• region
• document type
• version
• creation date
• last updated date
• status: active, draft, archived
• sensitivity level
• access control labels

Metadata helps the agent prioritize fresh, authoritative, and permission-safe documents.

9. Reranking Service

The reranker takes the broader candidate set from hybrid search and reorders the chunks based on relevance, freshness, specificity, and authority.

For production, I would usually use a specialized reranker model rather than the main LLM for cost and latency reasons. For complex cases, the main LLM can assist with deeper judgment, such as resolving subtle differences between similar documents.

10. Context Builder

The context builder selects the final evidence that will be passed to the LLM.

It should:

• Keep only the most relevant chunks.
• Remove duplicates.
• Preserve document titles, section names, and source IDs.
• Include neighboring chunks when the original chunk lacks enough context.
• Stay within the model’s context window.
• Prefer authoritative and recent documents.

11. Document Fetch Tool

The document fetch tool retrieves full sections or neighboring chunks from the enterprise document store. This is useful when a retrieved chunk is relevant but incomplete.

For example, if the search result contains only one paragraph from the PTO policy, the fetch tool may retrieve the full “PTO Carryover” section.

12. Answer Generation

The main LLM generates the answer using only the selected context. It should produce a clear, concise response that directly answers the user’s question.

The answer should not include unsupported claims. If the evidence is incomplete, the agent should say so.

13. Citation Builder

The citation builder maps important claims in the answer back to the source document, section, or chunk.

Each citation should include:

• document title
• document ID
• section title
• chunk ID
• source URL or internal reference
• last updated date when useful

This allows the user to verify the answer.

14. Grounding & Hallucination Check

Before returning the final answer, the system checks whether the answer is supported by the retrieved evidence.

If the grounding check finds unsupported claims, the agent should:

• remove the unsupported claim,
• retrieve more evidence,
• ask a clarification question,
• or say the knowledge base does not contain enough information.

15. Offline Knowledge Ingestion Pipeline

The ingestion pipeline prepares enterprise documents for retrieval.

It includes:

1. Document connectors pull content from systems such as Google Drive, Confluence, SharePoint, Slack, Jira, Notion, or internal databases.
2. The parser extracts text, tables, headings, links, and metadata.
3. The chunking service splits documents into retrieval-friendly chunks.
4. The embedding service converts chunks into vector embeddings.
5. The system stores chunks in the vector database, BM25 keyword index, metadata store, and document store.

This pipeline should run continuously or on a schedule to keep the knowledge base fresh.

Request Flow

For a typical user question, the flow is:

1. User asks a question.
2. The system authenticates the user and loads user context.
3. The orchestrator classifies the query.
4. The retrieval planner decides the search strategy.
5. Access control confirms which sources the user can search.
6. Hybrid search retrieves candidate chunks.
7. Reranking selects the best evidence.
8. The context builder assembles the final context.
9. The main LLM generates a draft answer.
10. The citation builder attaches source references.
11. The grounding checker verifies the answer.
12. The final answer is returned with citations.

Why This Architecture Works

This architecture balances autonomy and control. The main LLM can reason, plan, and synthesize, but deterministic services handle permissions, search, filtering, fetching, and indexing.

The design supports:

• accurate enterprise Q&A
• source-grounded answers
• access control
• multi-document reasoning
• freshness handling
• hallucination reduction
• auditability
• scalable retrieval over large knowledge bases

Memory & Context Design

How does the agent manage memory and context? Define short-term (conversation) and long-term (persistent) memory strategies. How is context window managed? What retrieval mechanisms are used (RAG, vector stores, summarization)?

The agent should manage memory at three levels:

1. Short-term conversation memory
2. Long-term persistent memory
3. Retrieved document context

These should be treated separately because they serve different purposes and have different safety requirements.

1. Short-Term Memory: Conversation Context

Short-term memory stores the current conversation state. It helps the agent understand follow-up questions, references, clarifications, and user intent across multiple turns.

For example:

User: “What is our PTO carryover policy for California employees?”

Assistant: “California employees may carry over…”

User: “How about New York?”

The second question depends on the previous turn. The agent needs short-term memory to understand that “How about New York?” means:

“What is our PTO carryover policy for New York employees?”

Short-term memory may include:

This memory is temporary and session-scoped. It should usually expire after the conversation ends unless the enterprise explicitly supports persistent chat history.

2. Long-Term Memory: Persistent Knowledge and User Context

Long-term memory should not simply mean remembering everything the user said. In an enterprise RAG agent, long-term memory mainly refers to persistent indexed knowledge and approved user-specific context.

There are two types of long-term memory:

A. Enterprise Knowledge Memory

This is the indexed enterprise knowledge base used for retrieval.

It includes:

• documents
• policies
• wikis
• tickets
• design docs
• product specs
• FAQs
• runbooks
• meeting notes
• metadata
• embeddings
• keyword indexes

This memory is maintained through the offline ingestion pipeline. Documents are parsed, chunked, embedded, indexed, and stored with metadata.

Example:

B. User / Session Preference Memory

The agent may also store limited user context, such as:

• user role
• department
• location
• permission groups
• preferred answer format
• previously selected document scope

For example, if the user is in the California office, the system may prioritize California-specific policy documents when the query is ambiguous.

However, this memory must be permission-aware, privacy-safe, and auditable. The agent should not store sensitive personal details unless explicitly allowed by enterprise policy.

3. Retrieved Document Context

Retrieved document context is the set of chunks selected from the knowledge base for a specific answer.

This is different from conversation memory. It is evidence gathered dynamically through RAG.

For each user question, the agent retrieves relevant chunks using:

• semantic vector search
• BM25 keyword search
• metadata filtering
• reranking
• document fetching
• context compression if needed

The final selected context may look like:

The final answer should be generated only from this retrieved context and should cite the supporting chunks.

Context Window Management

The LLM has a limited context window, so the agent must carefully decide what information to include.

The context window should usually contain:

1. System instructions
2. User’s current question
3. Relevant conversation summary
4. User permissions and constraints
5. Top-ranked retrieved chunks
6. Citation metadata
7. Output formatting instructions

The agent should avoid placing the entire conversation history or entire documents into the LLM context. Instead, it should use a context management strategy.

Common strategies include:

1. Sliding Window

Keep only the most recent conversation turns when they are relevant.

Example:

2. Conversation Summarization

When the conversation becomes long, summarize older messages into a compact memory.

Example:

3. Query Rewriting

Rewrite follow-up questions into standalone search queries.

Example:

User asks:

“How about New York?”

The agent rewrites it as:

“What is the PTO carryover policy for New York employees?”

This improves retrieval quality.

4. Top-K Context Selection

Retrieve a broad candidate set first, then rerank and include only the best chunks.

Example:

5. Deduplication

Remove duplicate or near-duplicate chunks so the LLM does not waste context window space.

6. Context Compression

If a selected document section is too long, compress it into a smaller evidence-focused summary while preserving citations.

7. Freshness and Authority Prioritization

When multiple chunks are relevant, prioritize:

1. active documents over archived documents
2. official policy over informal discussion
3. newer versions over older versions
4. user-authorized documents over restricted documents
5. directly relevant chunks over broad background context

Retrieval Mechanisms

The agent uses RAG as the primary retrieval mechanism.

The retrieval flow is:

The retrieval system should combine multiple methods:

Vector Search

Vector search finds documents based on semantic similarity. It is useful when the user’s wording differs from the document’s wording.

Example:

User asks:

“Can I roll over unused vacation?”

Vector search can match:

“PTO carryover policy”

BM25 Keyword Search

BM25 finds exact keyword matches. It is useful for acronyms, policy names, product names, error codes, and legal terms.

Example:

User asks:

“What does SSO-403 mean?”

BM25 can find documents that exactly contain SSO-403.

Hybrid Search

Hybrid search combines vector search and BM25 keyword search. This is the preferred approach for enterprise RAG because enterprise questions often contain both natural language and exact internal terms.

Metadata Filtering

Metadata filtering narrows search results based on attributes like:

• department
• region
• product
• document type
• owner
• status
• last updated date
• access level

Reranking

Reranking reorders retrieved chunks using a more precise relevance model. It helps ensure that only the most useful evidence is passed to the LLM.

Summarization

Summarization is used to compress long conversations or long retrieved sections when they do not fit into the context window. However, summaries should not replace source citations. The final answer still needs to cite the original documents or chunks.

Memory Safety and Governance

The agent must manage memory carefully because enterprise knowledge may contain sensitive data.

Key rules:

• Do not store sensitive user information unless explicitly allowed.
• Do not use conversation memory to bypass access control.
• Do not expose documents from previous sessions if the user no longer has permission.
• Do not rely on stale memory when fresh retrieval is required.
• Do not treat summarized memory as a source of truth.
• Use retrieved enterprise documents as the authoritative source for final answers.
• Log retrieval decisions, cited sources, and grounding results for auditability.

Core Reasoning

Deep dive into the agent's reasoning strategy. How does it plan, decide, and self-correct? Discuss prompt engineering, chain-of-thought, reflection loops, failure handling, and tradeoffs.

The agent’s reasoning strategy should be designed around a plan → retrieve → evaluate → generate → verify → correct loop.

The agent should not simply retrieve a few documents and immediately answer. Instead, it should reason about what the user is asking, determine what evidence is needed, retrieve and evaluate that evidence, generate a grounded answer, and self-correct when confidence is low or evidence is incomplete.

Reasoning Strategy Overview

The agent follows a structured reasoning flow:

The main LLM acts as the reasoning engine, while tools provide reliable execution for search, access control, reranking, document fetching, citation mapping, and grounding checks.

1. Query Understanding and Intent Classification

The first reasoning step is to understand the user’s question.

The agent determines:

• What is the user asking?
• Is this a policy, troubleshooting, summarization, comparison, or analytical question?
• Is the question standalone or dependent on previous conversation context?
• Are there entities such as product names, teams, regions, dates, document names, or employee types?
• Is the question ambiguous?
• Does the question involve restricted or sensitive information?

Example:

This step helps the agent decide whether to perform a simple lookup, multi-hop retrieval, comparison, or clarification.

2. Planning

After understanding the query, the agent creates a retrieval and reasoning plan.

For a simple question, the plan may be:

For a multi-document question, the plan may be:

The planner decides:

• Which tools to call
• Which filters to apply
• Whether query decomposition is needed
• Whether to fetch neighboring chunks
• Whether to ask a clarification question
• Whether conflict detection is needed
• How much evidence is required before answering

The agent should keep this reasoning internal and only return a concise explanation to the user, not raw chain-of-thought.

3. Retrieval Reasoning

The agent decides how to retrieve evidence based on the query type.

For exact terms, IDs, acronyms, or error codes, the agent should rely more heavily on BM25 keyword search.

Example:

For natural-language questions, the agent should rely more heavily on semantic search.

Example:

For enterprise Q&A, the default should be hybrid search:

The agent should retrieve a broad candidate set first, then use reranking to select the best evidence.

Example:

4. Evidence Evaluation

Before generating the answer, the agent evaluates whether the retrieved evidence is good enough.

It checks:

• Relevance: Does this chunk answer the question?
• Authority: Is this an official source or an informal note?
• Freshness: Is this the latest version?
• Specificity: Does it answer the exact user case?
• Completeness: Is more context needed?
• Permission: Is the user allowed to see this?
• Conflict: Do retrieved documents disagree?

Example:

If evidence is weak, incomplete, stale, or conflicting, the agent should not blindly answer.

5. Answer Generation

The main LLM generates the answer using only the selected evidence.

The answer should be:

• Direct
• Grounded in retrieved sources
• Easy to understand
• Properly cited
• Explicit about uncertainty
• Clear about conflicts or missing information

The answer generation prompt should instruct the LLM to:

The model should not treat its own memory as the source of truth. The retrieved evidence should be the source of truth.

6. Chain-of-Thought Handling

The agent may use internal reasoning to plan, compare documents, and verify claims. However, it should not expose raw chain-of-thought to the user.

Instead, the agent can provide a concise, user-safe explanation such as:

Or:

This gives the user transparency without exposing unnecessary internal reasoning steps.

A good design principle is:

7. Self-Correction and Reflection Loops

The agent should include controlled self-correction loops, especially when confidence is low.

Common reflection loops include:

A. Retrieval Reflection

After retrieval, the agent asks:

If results are poor, the agent can rewrite the query and retrieve again.

Example:

B. Evidence Completeness Check

The agent checks whether the evidence fully answers the question.

If the user asks for a comparison, the agent must have evidence for both sides.

Example:

C. Conflict Reflection

If retrieved documents disagree, the agent should detect the conflict and resolve it using source authority and freshness.

Resolution logic:

If the conflict cannot be resolved, the agent should explain the uncertainty and cite both sources.

D. Grounding Reflection

After drafting an answer, the agent checks:

If unsupported claims are found, the agent should revise the answer or retrieve more evidence.

8. Failure Handling

The agent should handle common failure scenarios safely.

No Relevant Documents Found

Response:

Possible next action:

Ambiguous Question

Example:

The agent should ask:

Access Denied

If the user lacks permission, the agent should not reveal restricted information.

Response:

Conflicting Sources

The agent should cite both sources and explain which one appears more authoritative.

Response:

Stale Documents

If only old documents are found, the agent should say so.

Response:

Tool Failure

If search, reranking, or document fetch fails, the agent should degrade gracefully.

Response:

Prompt Injection in Retrieved Documents

Enterprise documents may contain malicious or irrelevant instructions such as:

The agent must treat retrieved documents as data, not instructions. System and developer instructions should always override document content.

9. Prompt Engineering

Prompt engineering should be separated by task.

System Prompt

The system prompt defines global behavior:

Query Understanding Prompt

Used to classify and structure the query:

Retrieval Planning Prompt

Used to decide search strategy:

Answer Generation Prompt

Used to produce the final answer:

Verification Prompt

Used to check grounding:

This separation makes the system easier to test, debug, and improve.

10. Decision Policy

The agent should follow explicit decision rules.

Examples:

This avoids over-answering and reduces hallucination risk.

11. Tradeoffs

Accuracy vs. Latency

More retrieval, reranking, and verification improves accuracy but increases response time.

A good production design may use:

Cost vs. Quality

Using the main LLM for every reasoning step can improve quality but is expensive. A better design uses the main LLM for planning and synthesis, while specialized tools handle search, reranking, filtering, and permissions.

Transparency vs. Chain-of-Thought Privacy

Users need to understand why the agent answered a certain way, but they do not need raw chain-of-thought. The agent should provide citations, source summaries, and brief rationale instead.

Recall vs. Precision

Retrieving more chunks increases the chance of finding the right evidence, but it also increases noise. Reranking and context selection help balance recall and precision.

Summarization vs. Source Fidelity

Summaries help fit long documents into the context window, but summaries can lose details. The agent should cite original source chunks, not just generated summaries.

Automation vs. Control

The agent should be autonomous enough to retrieve, reason, and self-correct, but not so autonomous that it bypasses permissions, modifies documents, or makes unsupported decisions.

Safety & Permissions

What guardrails and safety mechanisms does the agent have? Define permission boundaries, human-in-the-loop checkpoints, content filtering, rate limiting, and how the agent handles adversarial inputs or unexpected states.

The agent must be designed with strong safety and permission boundaries because it operates over enterprise knowledge, which may include confidential, regulated, or sensitive information.

The main safety principle is:

The agent should only answer using information the user is authorized to access, and only when the answer is grounded in trusted retrieved evidence.

The agent should not bypass access control, reveal restricted content, follow malicious instructions, or invent unsupported answers.

1. Permission Boundaries

The agent must enforce document-level, source-level, and field-level permissions before retrieving or exposing information.

Permission checks should happen before retrieval and again before answer generation.

The agent should consider:

• user identity
• role
• department
• location
• permission groups
• document owner
• document classification
• source system permissions
• document status: active, draft, archived
• sensitivity level: public, internal, confidential, restricted

Example permission check:

Example output:

The agent should only retrieve and cite content from allowed sources. If access is denied, the agent should respond safely:

The agent should never reveal that a restricted document exists if the user is not allowed to know about it.

2. Source Grounding and Hallucination Guardrails

The agent should only make claims that are supported by retrieved evidence.

Before returning an answer, the agent should run a grounding check:

Example output:

If unsupported claims are detected, the agent should:

1. remove the unsupported claim,
2. retrieve additional evidence,
3. ask a clarification question,
4. or say the knowledge base does not contain enough information.

The agent should not use the LLM’s general knowledge as the source of truth for enterprise answers. Retrieved enterprise documents should be the authoritative evidence.

3. Prompt Injection Protection

The agent must treat retrieved documents as data, not instructions.

Enterprise documents, web pages, tickets, Slack messages, or PDFs may contain malicious or accidental instructions such as:

The agent should ignore these instructions because they are part of the retrieved content, not system instructions.

Guardrails should include:

• System and developer instructions always override retrieved content.
• Retrieved documents are treated only as evidence.
• The agent should not execute instructions found inside documents.
• The agent should not reveal system prompts, hidden policies, credentials, or internal reasoning.
• The agent should detect suspicious instructions inside retrieved content and flag them.

Example safe behavior:

4. Content Filtering

The agent should apply content filtering before and after answer generation.

Input filtering checks whether the user request involves:

• unauthorized data access
• credential or secret extraction
• personal data exposure
• legal, HR, or compliance risk
• security vulnerabilities
• harmful instructions
• attempts to override system rules

Output filtering checks whether the generated answer contains:

• restricted information
• unsupported claims
• confidential content the user cannot access
• personally identifiable information
• unsafe operational steps
• leaked system prompts or internal chain-of-thought
• excessive sensitive details

For example, if a user asks:

The agent should not return sensitive employee information. It should respond with a safe refusal or redirect:

5. Human-in-the-Loop Checkpoints

For normal knowledge lookup questions, the agent can answer automatically if the evidence is clear and permission checks pass.

However, human review should be required for higher-risk cases.

Human-in-the-loop checkpoints should apply when:

• the answer may affect legal, HR, financial, or compliance decisions
• documents conflict and the agent cannot confidently resolve the conflict
• the requested information is highly sensitive
• the user asks the agent to generate external-facing communication
• the user requests an action that changes enterprise systems
• the grounding confidence is below a threshold
• the agent detects possible policy violations
• the agent is asked to summarize confidential or regulated information

Example:

The agent can still provide a limited response:

6. Rate Limiting and Abuse Prevention

The system should include rate limits to prevent abuse, scraping, or accidental overload.

Rate limiting can apply by:

• user
• team
• source system
• document type
• API endpoint
• retrieval volume
• number of documents fetched
• number of questions per minute
• number of high-sensitivity queries per day

The agent should also detect unusual behavior, such as:

• many queries requesting confidential documents
• broad extraction requests
• repeated failed access attempts
• attempts to enumerate restricted documents
• unusually large document export requests

Example:

Rate limiting protects both security and system reliability.

7. Handling Adversarial Inputs

The agent should be designed to handle adversarial or malicious user inputs.

Examples include:

Prompt Injection

Safe response:

Permission Bypass Attempt

Safe response:

Data Exfiltration

Safe response:

Sensitive Personal Data Request

Safe response:

Jailbreak Through Retrieved Content

Safe response:

8. Handling Unexpected States

The agent should fail safely when something unexpected happens.

Retrieval Failure

If retrieval fails, the agent should not guess.

Access Control Failure

If the permission service is unavailable, the agent should default to deny.

Reranking Failure

If reranking fails, the agent may fall back to hybrid search ranking for low-risk questions, but should be cautious.

Grounding Check Failure

If grounding verification fails, the agent should not return a confident answer.

Conflicting Documents

If documents conflict, the agent should cite the conflict and prefer authoritative sources only when there is a clear rule.

9. Auditability and Logging

The system should log key safety and reasoning events for auditability.

Logs may include:

• user ID
• query
• timestamp
• retrieved document IDs
• permission decisions
• denied sources
• cited chunks
• grounding score
• unsupported claims
• tool calls
• human review triggers
• final answer metadata

Logs should not store unnecessary sensitive data, and access to logs should also be permission-controlled.

Audit logs help with debugging, compliance, quality evaluation, and incident investigation.

10. Data Governance

The agent should follow enterprise data governance rules.

Important safeguards include:

• encrypt data in transit and at rest
• respect document retention policies
• support document deletion and re-indexing
• avoid storing sensitive user prompts longer than necessary
• remove stale embeddings when documents are deleted or permissions change
• re-check permissions at query time, not only ingestion time
• separate public, internal, confidential, and restricted indexes if needed

The agent should not rely on old indexed permissions because user access may change over time.

11. Safety Decision Policy

The agent should follow explicit safety rules: