Loading...
user can input questions
user can manage threads and history
thread can have automatic title
monthly user 100M
each user send 5 requests per day
no data loss
99.9% uptime
99% response time less than 10s, timeout after 30s
100M * 5 * 30 / 30 * 24 * 60 * 60 = thousands per second qps
100M * 5 * 30 * 2 kb = 3TB/month
websocket
send input:
user_id: long
thread_id: long
input: str
timestamp: long
metadata: json
response:
user_id: long
thread_id: long
timestamp: long
answer: str
status: enum
cassandra
thread_id as partition key
user_id, input, input order number, input timestamp, answer, answer order number, answer timestamp
as value
client -> LoadBalancer -> chat servers -> chat db(cassandra)
chat servers -> compute servers -> compute db(vector db)
user send input to chat server with socket, chat server store chatting history, chat server send rpc to compute server, compute server fetch vector db, compute server compose answer and send to chat server, chat server stores answer and send back to client with socket
load balancer can have rate limiter and authentication; cassandra have masterless replicas for performance; when traffic is high, a rate limiter will drop some requests when chat server send request to compute server, and client see failure message.
sql vs nosql
long polling vs websocket
ratelimiter at loadbalancer vs between chat server and compute server
traffic could be spiky
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?