Loading...
A client must be able to get the current weather—as well as past weather and future forecasts—for a particular location. He might be able to check multiple locations or even subscribe to this.
The service must be accurate and reliable and must be accessible for both desktop, mobile clients and web clients.
It must also show current data
With 10 data weather providers we will be polling or receiving updates every 500ms, which translates as 20rps.
Users are in average checking locations of no more than 20 locations, as these will be lazy loaded in the UI. Each weather broadcast includes 9 forecasts, 4 earlier, and 4 future as well as the present day. Each forecast includes, a number with a single decimal and a small string
With 100k DAU users of which 10% have their apps open at any given time we would be receiving 10k rps.
Usage spikes would include a 30% of users requesting a weather update giving an additional 30k rps.
get_weather():
input: locations[]
output: Forecasts[]
post_weather_data() // used by external data providers
input: provider_api, Forecasts[]
output: acknowledgement
poll_weather_data() // used by internal data pulling cron job
input: provider_id
output: Forecasts[]
We will have a Forecasts entity as well as location_subscriptions
entity. Which we will use for internal and external forecasts
Forecasts:
temperature
weather_type: 'rainy' | 'cloudy' | 'sunny'... etc
forecast_timestamp
created_timestamp
provider_id: null for internal, a string for external
location
Clients load the initial data from the data caching service, and afterwards poll its data every 1s.
This data is written to the time series and to the cache in parallel, by the data weather subscription service and the data weather polling service.
A cron job triggers every 1s the data consolidation service, which consolidates the latest data from the different providers into an single internal forecast.
There are two read request flows:
For read:
Clients request the weather for a particular location including feature and past forecasts. This is served from a cache that contains the latest data written by our weather data consolidation service.
The data consolidation service reads from the time series database every 500ms the latest data from each provider to consolidate it.
For write:
Multiple providers post updates, and the data pulling service also pulls from some data providers. All this data is written to a time series database.
The data consolidation service, reads the latest data every 500ms consolidates it and writes it to a cache and to the time series database.
The internal forecast is determined by checking the variance between the different forecasts received, giving higher ranking forecasts coming from data providers that have a higher confidence—as well as from data that varies the least between the providers.
A single Datapoint is then determined as internal and is saved with a provider API of null in our time series database.
The time series database receives constant data in a constant time. Therefore, storage capacity and sharding can be staticly provisioned in advance by range sharding based on time periods without having hot spots.
A data retention policy also removes older forecasts than a certain data avoiding the need to provision storage constantly.
The weather data cache always keeps the latest data for each of the locations, making sure that none of these needs to be retrieved from the database. As the locations are not a non-constant number, the cache space can be provisioned in advance.
In order to keep the cache reliable, we keep multiple servers replicating the data.
We choose polling over subscription in order to not receive constant requests of devices that are not using the information constantly.
We use also a time series database for its ability for fast writes with no updates, and also for its range-based queries—which are perfect for our use case where we write lots of data and we usually query it by time.
We use a cron job to consolidate the data to avoid creating multiple updates for clients, overwhelming them and making their screens twitch. Another approach would be to send the live data, but that would not only increase the number of updates—it would worsen the user experience.
If a data provider goes down, the data consolidation service can use the data of other providers. If the data consolidation service goes down completely, we can still serve the latest forecast available, indicating its staleness.
If our cache is overflown we serve from the time series database, which has plenty of free read capacity as its overprovisioned for storage.
If the time series database is unavailable. A replica takes its place.
Some future improvements would include not storing all provider updates.
And serving historical data.