Loading...
Availability - this service has to be highly available.
Response time - functionality has to have a low response time, e.g., less than 100ms.
Scalability - we will get more and more requests to return the weather forecasts, so the service has to be highly scalable.
The system has a few providers of weather data - integration with third-party weather services.
20,000 requests per second for searching the weather forecast.
2,000 requests per second were made to get the summary weather reporters.
I assume the search whether the request contains 200 bytes of data and the summary weather report is about 10 kbytes. It requires the bandwidth is about 24 Mbits to pass the data.
Let's think the wether data is updated every 2 weeks from the wether third-party services and every update takes 2 Mbytes, it needs to 54 GBytes to store the wether data for 5 years.
forecast(apiKey, country, state, district, city, fromTimestamp, toTimestamp) returns the weather forecast for the given country,state,district and city for specified period.
weatherSummary(apiKey, country, state, district, city, fromTimestamp, toTimestamp) generates the report for region for the previous seasons.
updateWeather(apiKey, country, state, district, city,timestamp,weatherMetrics) updates the weather data by third-party weather providers.
The primary data model for this system is three tablets: one table is for the forecast, for the weather summary and raw data from the weather provider would be stacked up to an object store.
We would use MongoDB to store the data.
Forecast table:
id(10 bytes)
country (100 bytes)
state (100 bytes)
district (100 bytes)
city (100 bytes)
whether data (2 kbytes) temperature, precipitation, humidity.
Weather summary:
id(10 bytes)
country (100 bytes)
state (100 bytes)
district (100 bytes)
city (100 bytes)
whether data (1 kbytes)
The raw data would have the provider-based schema.
See the diagram from the high-level architecture.
API Gateway provides DDoS protection and TLS termination, and forwards request to the right service nodes.
The forecast service processes the request to search wheather for a specified country, state, district, city, and period. This service reads data from MongoDB which data would prepared by the Map Reduce pipeline. The most requested data would be cached in Redis. The summary service replies with weather summary data for a specified country, state, district, city, and past period:1 month, 3 months, and 6 months. The most requested data would be cached in Redis. The weather providers upload the raw data to Kafka and the Map Reduce pipeline processes that data for the forecast and summary and stores that to Mongo DB.
The forecast(apiKey, country, state, district, city, fromTimestamp, toTimestamp) reads the weather data from MongoDB using indexes to speed up the search in DB.
weatherSummary(apiKey, country, state, district, city, fromTimestamp, toTimestamp) reads the weather summary from MongoDB for the previous time ranges.
track(apiKey, trainingId, timestamp, user metrics) sends the workout data to the Tracking service from the client application. The tracking service persists those data: the user speed, geolocation, heartbeats, and distance to No SQL. In this case, No SQL is the best solution to make writing. We may use AWS Dynamo to store that data.
updateWeather(apiKey, country, state, district, city, timestamp,weatherMetrics) used by weather providers to feed data to the system. Then that data would be in Kafka and the Map Reduce prepares the data for the forecast and summary services MongoDB.
Performance and scalability of forecast and summary endpoints are extremely important for this system. As such, we employ No SQL to scalable user load evenly. Also, we may Kubenates horizontal autoscaler to regulate node number. No SQL uses replication and partitioning to be fault-tolerant. Using indexes (country, state, district, city, timestamp) would help forecast and summary services to process the weather data quickly.
forecast endpoint would be frequently requested and it's the critical functionality, and it should be scalable under load. Most frequent data would be cached in Redis.
Summary endpoint would return the weather data summary for the given past interval. Most frequent data would be cached in Redis.
The client application should be developed in such a way, that if it loses the connection and restores it, it shows the last downloaded data. We may use WebSockets to connect the client application.
Some partitions would be more requested and (city, state, distinct, city, timestamp) key doesn't give an even distribution so we're going to create a hash of (city, state, distinct, city, timestamp) and use it as the partition key.
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?