Loading...
Scalability: Highly scalable system needed(support replications, horizontal scaling, shards with ES)
Consistency: Eventual consistency is fine
Partition tolerant: yes
Latency: Ultra low latency for querying and generating reports by using elastic search
Resiliency for failures
SLA: 99.99%
Multi region deployment
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
POST /log
GET /log/
GET /log/
Each of the clients will have agents that collect data and periodically send them to API server service that transforms them and stores it in scalable kafka message broker as topics that are partitioned and stored.
Log processing service integrates populating elastic search. ES service then provides that data to analytic tools based on timestamp slices. Tracerid can be used to trace
Client: Task is collect and send the data in real time calling API service
Log processing service: kafka producer that writes to kafka after transformation
Elastic search service receives events to update the index and compute and store into ES. Analysis service analyses the data
and returns the queried data from ES.
Concepts:
Eventual consistency with ES(hot)
TTL retention of 7 days
ES data gets archived to S3 through a archiver service(cold). Grafana/Kibana tools help with analytics and visualization, and include integrated alerting with counters, tracers and metrics along with graphs
Horizontal scaling with Elastic search that supports shards but limit retention interval to few days to control data growth.
Asynchronus messaging through kafka allows scale of millions of requests as producer and consumer are decoupled alleviating load/pressure on ES.
Failures:
Support rerunnability of missed/failed events from kafka by adding to a failure/retry topic and having that processed separately.
Use hystrix where applicable on API server service for fallbacks and resiliency.