Agent Requirements

Agent Objective:

Agent should be trained on the documents provided as source and able to answer to the questions

Autonomy & Constraints:

Agent should not answer any irrelevant questions outside of the document , it should not go beyond the source document it should respond that the "it is not relevant question to answer "

Tool & Action Design

Document search, Vector DB, Simple LLM to be used internally

Agent Architecture

1. Use a distributed architecture with microservices like reading of document, Retrieval of the answer, calling LLM and MFE UI
2. read all the documents
3. Store in DB as tokens with overlapping token as 10%
4. Use simple LLM to send the query
5. Use history to save the chat for the session
6. set boundaries for the agent
7. Have an UI where query can be submitted
8. User should be OKTA authenticated
9. in the same UI answer should get with reference to the document
10. Use horizontal scaling when user demand increase add more instances
11. Use load balancer to distribute the incoming queries
12. Implement monitoring tools to track performance metrics and setup auto scaling rules
13. If knowledge base grows use partitioning data across different nodes or shards

Memory & Context Design

Use any of the Vector DB

use the long term memory memory with BM25 indexing for improved accuracy

use short term memory for current interactions

Context should be stored externally with redis cache to save that in the session and user can comeback and start the chat

Core Reasoning

Read the question from user, plan the question to make it better retrieve from the Vector DB , see if the relevant information is there in the question, if so get the results if not throw a message that ask question about relevant information

Safety & Permissions

Don't use LLM to answer any question irrelevant, Only users who are authenticated with OKTA should use this agent. Any modification should be permitted explicitly by user, The agent should not go to open internet to answer the questions it should always reference to the Vector DB provided