Find the user intent and extract the query and find for the asnwers for the vector knowledge database
Autonomy & Constraints:
The agent should stick to the provide knowledge base for the asnwers to user query
It should say that "I don't have required information to answers this query", if a user asks for out of the box query (Not related to the knoledge base)
User should be allowed to upload thousands of documents to ingest the content to knowledge base
Tool & Action Design
LLM will call prompt service to get the promt and context
The LLM agent will do tool call to create the embaddings of the given query and futher call to vector DB to search for the answers
Once the got the answers it will respond to the use in parallel via websockets
Agent Architecture
Single agent with tool calls for embaddings and vector search
Memory & Context Design
Prompt management service for the context
Vector DB (Knowledge base for long term memory)
Core Reasoning
User asks query -> LLM will process the query (With fallback mechanism -> Cheaper models first) -> Get the prompt from prompt service -> Create embaddings -> Search in the vector DB -> Push to queue for background Evals -> Respond to the user query
After Evals the feedback will be given to the prompt service to update the instructions and the prompts if needed (Human in loop checkpoint) and Alert
To store or update the knowledge base, ingestion service will upload the file to S3 and push task to queue for Storing(Background processing -> User should not be blocked for uploading the files as ingestion will take time)
The worker will create embaddings and store the content vector DB in batches
The LLM config service will keep listening to webhooks for updates and do the necessary changes to the config for choosing LLMs (Healthy -> cheap llms first).
For the complete failure and the exceptions human agent will be notified urgently to take over the query and respond "We'er facing technical issue will get back to you soon".
Safety & Permissions
Agent will follow the proper guardrails and the instructions given in the prompts.
Agent will not be allowed to answer queries which are not related to the scope of the enterprise.
We'll have to implement rate limiting and the firewall against malicious content. The API gateway can do that by blocking the IP source which breaks the rules.
All the internal services will be private, not exposed to public internet.
In the production system will use docker and kubernetes to keep the production enviornment healthy always and automated scaling of pods and containers.
Set rules for the CPU usage and memory usage of the to scale. i.e. - Increase the capacity of memory and CPU if the utilization crosses a certain threshold, Set minimum and maximum number of pods to be active running on prod (Spin up the latest healthy if crashed). Keep checking heartbeat of the services (health check) and take action.
The system will also have the availability dashboard for all the service used and alert for any failures
Ensure unit tests are passing before deployment
Use observability tools for monitoring and alerting. i.e. - graphana, AWS cloudwatch etc.