Find the user intent and extract the query and find for the asnwers for the vector knowledge database
Autonomy & Constraints:
The agent should stick to the provide knowledge base for the asnwers to user query
It should say that "I don't have required information to answers this query", if a user asks for out of the box query (Not related to the knoledge base)
User should be allowed to upload thousands of documents to ingest the content to knowledge base
Tool & Action Design
LLM will call prompt service to get the promt and context
The LLM agent will do tool call to create the embaddings of the given query and futher call to vector DB to search for the answers
Once the got the answers it will respond to the use in parallel via websockets
Agent Architecture
Single agent with tool calls for embaddings and vector search
Memory & Context Design
Prompt management service for the context
Vector DB (Knowledge base for long term memory)
Core Reasoning
User asks query -> LLM will process the query (With fallback mechanism -> Cheaper models first) -> Get the prompt from prompt service -> Create embaddings -> Search in the vector DB -> Push to queue for background Evals -> Respond to the user query
After Evals the feedback will be given to the prompt service to update the instructions and the prompts if needed (Human in loop checkpoint) and Alert
To store or update the knowledge base, ingestion service will upload the file to S3 and push task to queue for Storing(Background processing -> User should not be blocked for uploading the files as ingestion will take time)
The worker will create embaddings and store the content vector DB in batches
Safety & Permissions
Agent will follow the proper guardrails and the instructions given in the prompts.
Agent will not be allowed to answer queries which are not related to the scope of the enterprise.
We'll have to implement rate limiting and the firewall against malicious content. The API gateway can do that by blocking the IP source which breaks the rules.
All the internal services will be private, not exposed to public internet.