Design Monitoring Infrastructure for mobile apps

Question

Design a real-time monitoring system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - Collect and process real-time logs and metrics from millions of mobile app users.
  - Provide a dashboard for monitoring key performance indicators (KPIs).
  - Allow for alerting based on predefined thresholds (e.g., error rates, latency).
  - Support querying historical data for analysis.
  - Enable integration with existing Square/Block services for anomaly detection and reporting.

- **Non-Functional Requirements**:
  - **Scalability**: Should handle up to 10 million requests per minute during peak usage.
  - **Latency**: Must deliver monitoring data with a maximum latency of 1 second.
  - **Availability**: Target 99.9% uptime to ensure real-time monitoring.
  - **Consistency**: Eventual consistency is acceptable, but critical alerts must be processed in real-time.
  - **Security**: Compliance with data protection regulations (GDPR, CCPA) for user data handling.

- **Request Per Second (QPS)**:
  - Assume 5 million active users, each sending 1 request every 10 seconds on average.
  - Total QPS = 5,000,000 / 10 = **500,000 QPS**.

- **Storage Estimation**:
  - Each log entry is approximately 1 KB.
  - Assuming we retain logs for 30 days:
    - Daily logs = 500,000 QPS * 86400 seconds = 43,200,000,000 logs/day.
    - Total storage required for 30 days = 43,200,000,000 logs/day * 30 days * 1 KB = **1.26 TB**.

- **Bandwidth Requirements**:
  - If each request/response round trip is approximately 1 KB, bandwidth = 500,000 QPS * 1 KB = **500 MB/s**.

- **Component Diagram**:
  - **Mobile App**: Sends logs and metrics to the system.
  - **API Gateway**: Handles incoming requests and routes to appropriate services (e.g., log ingestion, alerting).
  - **Log Ingestion Service**: Collects logs and metrics, validates data, and pushes it to a message queue.
  - **Message Queue**: (e.g., Kafka) for buffering log data and decoupling ingestion from processing.
  - **Processing Service**: Consumes from the message queue, processes logs, and stores them in the database.
  - **Database**: (e.g., Cassandra) for storing and querying logs and metrics with high write availability.
  - **Monitoring Dashboard**: Frontend application to visualize KPIs and alerts.
  - **Alerting Service**: Monitors processed data and sends notifications based on threshold breaches.

- **Technology Choices**:
  - **API Gateway**: NGINX or AWS API Gateway.
  - **Message Queue**: Apache Kafka for durability and scalability.
  - **Database**: Apache Cassandra for high write throughput and availability.
  - **Frontend**: React for the monitoring dashboard.

- **Schema Design**:
  - **Log Entry Table**:
    - `id (UUID)`: Unique identifier.
    - `user_id (UUID)`: Identifier for the user.
    - `timestamp (TIMESTAMP)`: Time of the log entry.
    - `event_type (STRING)`: Type of event (e.g., error, info).
    - `payload (JSON)`: Additional data related to the event.

- **Access Patterns**:
  - Frequent queries for recent logs by user ID.
  - Aggregation queries for KPIs by event type.
  - Partitioning the Log Entry Table by day and user ID to optimize read/write performance.

- **Consistency vs. Availability**:
  - Chose eventual consistency for log processing to ensure high availability and fault tolerance.
  - Critical alerts will be processed in a real-time manner to ensure immediate actions can be taken.

- **Database Trade-offs**:
  - Using Cassandra provides high write throughput but sacrifices some consistency. It fits our requirement for high availability and horizontal scalability.
  - This may lead to reading stale data, which is acceptable for most log analytics.

- **Caching Strategy**:
  - Implement a caching layer (e.g., Redis) for frequently accessed alert thresholds and user metrics to reduce database load, but this introduces complexity in cache invalidation.

- **Failure Handling**:
  - Implement retries on the API Gateway and log ingestion service to handle transient failures.
  - Use circuit breakers to avoid overwhelming downstream services during outages.

Design Monitoring Infrastructure for mobile apps

Square/Block

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Square/Block Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search