System requirements

Functional:

1. View current Weather: Users can request and view the current weather information for a specified location.
2. View Forecast: Users can also request and view the weather forecast for a specified location.
3. Add New Location: Admins can add new locations to ensure the system has data for various regions.
4. Update Weather Data: Admins can manually update weather data in case of inaccuracies or missing information.
5. Weather data collection: data is collected from either external API or IoT sensors or weather stations.

Non-Functional:

1. Reliability: Users rely on the accurate and timely weather information, so the system must be highly reliable to ensure consistent access to data. Minimising downtime and ensuring data accuracy are key aspects here.
2. Scalability: With a large user base and varying demand for weather information, the system should be scalable to meet the increasing data storage needs, user requests, and concurrent users during peak times.
3. Performance: Users expect fast response times when access weather data. Optimising system performance, response times, and minimising latency are critical.
4. Security: user privacy should be protected. Implementing robust security measures to safeguard data integrity, authentication, and authorisation is essential.
5. Usability: The system should have a user-friendly interface that is easy to navigate, allowing users to access weather information quickly and intuitively. Providing relevant and clear weather data presentation enhances user experience.

Capacity estimation

Storage

• Regions data: around 1 millions regions worldwide, including cities, suburbs, towns, and rural areas
• Weather data is stored hourly, each data set is assumed 200 bytes, a day is 2MB per region, which leads to 2TB daily

Requests:

• DAU: 100 million users access the system daily, with a peak usage of 15 million users during extreme weather events
• QPS: On average, users check the weather five times a day, leading to 500 millions requests a day, which is 5,000 QPS on average, with a peak QPS at around 15,000 QPS
• 2 millions writes per hour on average, which is about 600 writes per second, the traffic is stable

API design

• GET /weather/current/{location}
• GET /weather/forecast/{time}/{location}
• PUT /weatherdata/{location}
• PUT /locations/add

Database design

The schema for the weather data could be

• UUID
• locationId -> partition_key
• hour timestamp -> clustering_key
• source IoT/API/Manual
• createdAt timestamp
• updatedAt timestamp

The schema for location

• UUID
• country
• state/province
• city
• suburb
• rural_area
• post_code

User data

• UUID
• {metadata}

Historical data can be archived using an object storage, for example AWS S3.

High-level design

Please refer to the High level diagram

Request flows

Please refer to the Sequence diagram

Detailed component design

1. User/admin requests firstly come to Load Balancers
2. Load balancing should be multi-tiered, regional, AZs
3. Each set have an active/standby setup
4. Then requests will be evenly distributed to each weather system set, comes to the system will go through an API gateway
5. authentication/authorisation
6. SSL/TLS termination
7. data encryption
8. request validation
9. request dispatching
10. rate limiting
11. request deduplication
12. usage data collection
13. User requests will be then routed to weather query service
14. The service is equiped with an external cache system that stores frequent accessed data with a long expiry date until the day finish
15. If not in the cache, the cache system query the database for the data
16. read-through pattern is used in case of a popular data is gone, e.g. weather for city of Sydney
17. Replications of the cache system have to be setup for high availability
18. IoT/Weather Station will send row data to the Weather System
19. A Event Reader Service reads weather data from various sources, cleaning the data and convert it into target format and then utilises Kafka to partition data streams based on cities, allowing for parallel processing and scalability
20. Partition Consumer Service: consumes data from Kafka partitions, batches the data based on hourly intervals for each state, and prepares it for insertion. Batched operations are done with external Redis cache
21. Batched data are then flushed to an internal queue using RabbitMQ, providing a reliable and scalable queuing mechanism.
22. Database Writer service: takes the batched data from the internal queue and performs batch insertion into the database
23. data are partitioned into Kafka, by region
24. the other side of Kafka, data consumers persist the data into the Cassandra database, which is partitioned by locationId
25. Consumers could be a cluster of hosts, which has a fixed number of poll of threads
26. Cassandra should setup proper backup plan for disaster recovery
27. Every batch service will checkpointing the success position, so that if batch operation fails half way, it can rewind the events from Kafka
28. A daemon process are running in the service, which periodically retrieve data from third-party API for majority regional weather data and go through the same process as how IoT data goes through.
29. A different archiving service periodically flush data into S3, which control the retention period of weather data stays in Cassandra
30. Every component have all hosts installed monitoring agents and logging agents, which will be process by a monitoring service to build comprehensive metrics, for example, using Promethius
31. User data are encrypted throughout the system end-to-end, before stored in the database.

Trade offs/Tech choices

Database: looking at the scale, with TB daily, we are looking at a large scale data storage for weather data, while the access pattern is quite simple, a NoSQL solution is best fit, for example Cassandra

• Horizontal scalable: as more regions are added, more nodes can be added.
• Data can be partitioned by region (e.g. city), which improves reads throughput as well as writes throughput, since millions of writes per hour are expected
• Built-in tunable consistency is well suited for data accuracy
• Built-in replication strategy, which helps ensure the data durability and availability.

Cache: Redis cluster is well suited for the use case

• High throughput, millions request per second
• Replications are built-in for high availability
• Low latency: in-memory cache serves as key-value store

Kafka

• Pull-based: each consumer could process the data at its own pace
• Reliable messaging service, which has an at-least-once guarantee
• Data durability: log-based messaging system, stores all data on disk
• Single consumer per partition, so that the consumer service can checkpointing to ensure data integrity.

Failure scenarios/bottlenecks

1. Failure points:
2. Data loss: Ensure full fault-tolerant in place to prevent data loss during transmission and processing across components, for example during the data batching
3. Message Queues: Monitor and handle queue overflows and system failures that may cause message loss
4. In memory cache for batch data: if data volume is very large, periodically store data on disk, e.g. state store here, because it might be hard to restore and aggregation from raw events again.
5. Database Writer Service: failed processing should be properly handled, for example using a dead-letter queue as a temporary datastore, and a difference thread service will process it
6. Bottlenecks:
7. Throughput Limitation: Monitor and optimise Kafka and RabbitMQ configurations
8. Processing Latency: Analyse and monitor the consumer service
9. Network bandwidth: large amount of messages needs to be transmitted in compact format, for example using binary format such as Google Protocol Buffer, etc.
10. Limited memory redis: an effective algorithm should be in place for eviction and expiry
11. Data storage: weather data is of large size everyday, a daemon process to archive data should utilise an effective algorithm and approach since it may flush too much data at a time, partitioned processor should be in place
12. Kafka: a single point of failure, we need to make sure Kafka is high available, also a proper way to handle unavailability

Future improvements

The system design for now didn't mention much about global setup, every single component in this system should have multi regional setup, including Load Balancers. Cassandra should be globally setup with regional partitions. Requests should be routed based on geographical edge points.