System requirements

Weather Reporting System provides users with current weather information for a specified location.

Functional:

Users will access both from web and mobile application

The information provided is:

• Current weather information
• Forecast for the next 7 days
  • Detailed for the next 24 h
  • Summary for the rest

The system allows to query for an specific location (client app can provide current user location with geolocation, or a different one)

10.000 cities all over the world (with population over 100.000) will be supported

Users can store prefered locations. This is done in client application using local storage, no server stored information.

An external weather service is responsible for providing the current information and forecasts. It provides information updates with different frequencies:

• Current information: every 5 minuts
• Next 24 h forecast: every hour
• Next 7 days summary forecast: every 6 hours

For a global user base, multiple langages and localized units or measure will be available. This will be handled in customer application.

Non-Functional:

Performance

For good user experience the information has to be provided fast, with low latency, in les than 100ms

Availability

The system must be highly available for users to trust it and repeatedly come back because it's a free service financed with adds.

Scalability

Initially figures: 100.000 daily users with 5 mean requests per day and peak 15 RPS

The system can be scalable to support significan increase, 20x, in 3 years

Security

No login nor specific security measures as no sensitive information or PII is managed.

Reliability

No data loss in case of hw / sw problems. Information should be available.

Restrictions

Capacity estimation

Requests

Initially figures: 100.000 daily users with 5 mean requests per day and peak 15 RPS

The system can be scalable to support significan increase, 20x, in 3 years: 2.000.000 daily users with 5 mean requests per day and peak 300 RPS

Information

Current information ~ 45B

• Location: VARCHAR(10): 11B
• Timestamp: TIMESTAMP 8B
• Temperature: DECIMAL 5B
• Wind speed: INT 4B
• Wind direction: CHAR(2) 2B
• Clouds: INT 4B
• Rain: INT 4B
• UV: INT 4B

Total storage: 45B * 10.000 * 1,2 ~ 0,5 MB

24 h forecast ~ 425B

• Location: VARCHAR(10): 11B
• 24 times
  • Timestamp: TIMESTAMP 8B
  • Temperature: DECIMAL 5B
  • Wind speed: INT 4B
  • Rain probability: INT 4B

Total storage: 425B * 10.000 * 1,2 ~ 5 MB

7 day forecast ~ 300B

• Location: VARCHAR(10): 11B
• 3 * 7: 21 times
  • Date: DATE 4B
  • Day part (morning, afternoon, night): CHAR 1B
  • Temperature: DECIMAL 5B
  • Wind speed: INT 4B
  • Rain probability: INT 4B

Total storage: 300B * 10.000 * 1,2 ~ 3,5 MB

API design

Define what APIs are expected from the system...

GET /cities/{city_code}/current-weather.json

Returns a JSON with the current weather for the providied city.

GET /cities/{city_code}/24h-forecast.json

Returns a JSON with the 24 hours weather forecast for the providied city.

GET /cities/{city_code}/7d-forecast.json

Returns a JSON with the 7 days weather forecast for the providied city.

All of them will return an HTTP 404 Not found error in case of a non existing city.

For city codes, a similar policy of OpenWeatherMap City IDs will be used: "{CityName}-{ISO 3166-1 alpha-2 country code}". The city name without spaces. Example: "NewYork-USE"

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

The data managed by the application is quite static and limited so S3 will be used to store the JSON information as it will be served.

High-level design

Main components:

DNS

Converts hostname to IP redirectingo to the closest CDN server to the user.

CDN

Weather information is static content that is updated at different frequencies.

For efficiency, reducing latency, scalability and high availability, the CDN will be serving and caching the information that is stored in S3 bucket once received.

S3 bucket

Will store JSON files with the weather information.

Each file will have HTTP headers configured, for example Cache-Control , to manage CDN caching policies.

Information bus (Kafka Cluster)

It receives and stores forcast information from the weather provider.

Thre will be a topic for each type of information:

• Current information
• Next 24 h forecast
• Next 7 days summary forecast

Weather event processor microservices

3 microservices, one per Kafka topic, are in charghe of reading forecast information, format data and updating S3 with the data and cache control headers.

They run in a Kubernetes cluster.

Monitoring and Analytics

Gather metrics from other systems to provide:

• Technical monitoring and alerting
• Business monitoring, to understand, for example, users behaviour

Request flows

The information is available in the HDL Flow diagrams and in the sequence diagrams:

1. Weather Data Update Flow (every 5 minutes)

Purpose

This flow describes how new weather data from the provider is ingested, processed, and stored in the system.

Actors and Systems

• Weather Info Provider (WIP): The external source of weather data.
• Kafka Current (KC): The message bus (Kafka topic) for current weather events.
• Current Weather Processor MS (P): Microservice that processes new weather data.
• S3 Bucket (S3): Storage for weather JSON files (the source of truth for served data).

Steps

1. Publish Event:
2. The Weather Info Provider publishes a new current weather event to the Kafka Current topic.
3. WIP ->> KC: Publish current weather event
4. Consume Event:
5. The Current Weather Processor Microservice consumes the event from Kafka.
6. P ->> KC: Consume weather event
7. Process Data:
8. The processor microservice processes and formats the new weather data.
9. P ->> P: Process & format weather data
10. Update Storage:
11. The processed weather data is written as an updated JSON file in the S3 bucket.
12. P ->> S3: Write updated weather JSON

2. Get Current Weather Flow (User request)

Purpose

This flow details how a user retrieves current weather information for a city/location, including how the system leverages CDN caching for performance.

Actors and Systems

• User (U): The end-user searching for weather information.
• Client App (C): Web or mobile application interface.
• DNS: Resolves the service endpoint.
• CDN: Content Delivery Network that caches and serves weather JSON files.
• S3 Bucket (S3): Origin storage for weather JSON files.

Steps

1. User Search:
2. The user initiates a search for weather information for a specific city or location in the client application.
3. U ->> C: Search weather for city/location
4. DNS Resolution:
5. The client app requests DNS resolution for the weather service URL.
6. C ->> DNS: Resolve weather service URL
7. DNS -->> C: Return CDN endpoint
8. Weather Request:
9. The client app sends an HTTPS request to the CDN for the weather data.
10. C ->> CDN: GET /cities/{city_code}/current-weather.json
11. Cache Decision:
    • If the CDN has the data cached:
    • The CDN responds immediately with the cached weather JSON.
    • CDN -->> C: Return cached weather JSON
    • If the CDN does not have the data cached (cache miss):
      • The CDN fetches the JSON file from S3.
      • CDN ->> S3: Fetch JSON file
      • S3 -->> CDN: Return weather JSON
      • The CDN then responds to the client app with the fresh weather JSON.
      • CDN -->> C: Return fresh weather JSON
12. Show to User:
13. The client app displays the received weather information to the user.
14. C -->> U: Show current weather

Detailed component design

Kafka

There will be 3 topics, one per type of information:

• pro.weather.current_information
• pro.weather.24h_forecast
• pro.weather.7d_forecast

The partition key is the city identifier.

Weather events processor microservices

In order to quicly process burst of received weather information, and taking into account that:

• a HTTP request has to be performed to S3, with latency of 60 - 100 ms
• ordering is not important for processing weather information

Confluent Parallel Consumer will be used with unordered processing. This allows to have more parallel processing threadns than paartitions.

There will be 3 microservices, one for each topic.

The number of threads / pods will depend on the topic. For example, for current weather information that is published every 5 minutes we have the goal to process them in 5 secondes.

In order to process 10.000 messages in 5 seconds, and assuming an average of 75 ms to process a message we will need 150 threads. For it we'll deploy 3 pods with 50 processing threads in parallel.

S3

An important point for optimizing CDN usage and reducing S3 queries is caching configuration.

The following HTTP headers will be defined for elements in the buckets:

• Cache-Control max-age: will be set to control TTL.
• Last-Modified and E-Tag ara returned by default allowing conditional queries, avoiding retrieving the content when not modified. This allows to reduce the TTL avoiding data traffic. For example, for current weather information generated every 5 minutes, a TTL can be defined to 60 s because we don't know when the content will be requested after publication.
• stale-while-revalidate: so the CDN continues serving cache data while retrieving updated information. This reduces response latency and improves user experience.

Configurations for a tradeoff between serving fresh data and costs:

• Current information (published every 5 minuts): max-age = 60 s, stale-while-revalidate = 15 s
• Next 24 h forecast (published every hour): max-age = 300 s, stale-while-revalidate = 60 s
• Next 7 days summary forecast (published every 6 hours): max-age = 1800 s, stale-while-revalidate = 60 s

Trade offs/Tech choices

Message buss selection

The number of messages to process is not very high: generally burst of 10.000 per topic. A RabbitMQ could be used taking into account those figures with no big latency. Also, RabbitMQ at least one delivery policy is fine as data can be processed multiple once without side efects.

A Kafka Cluster is selected for weather information streaming for the following reasons:

• Team already uses and operates Kafka
• Kafka provides lower latency
• It provides better scalability for increasing the number of cities in the future

S3

As explained previously caching configuration has been done for a tradeoff between serving fresh data and costs

Using only a CDN

Initially it was considered to use Webervers or even microservices with a Redis chace. Even dough it could provide more fresh data for current weather info, it adds costs and complexity. The CDN is a good enough option.

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Kafka and Weather events processor MS are points of failure. For achieving high availability:

Kafka

Will be configured with 3 replicas in 3 diferent availability zones. This ensures high availability, fault tolerance and data durability.

Weather events processor MS

The microservices will be run in a Kubernetes Clusters with 3 nodes in different availability zones to ensure high availability and fault tolerance.

CDN

The service provide haigh availability and is prepared to handle load spikes.

Future improvements

More cities can be included. Kafka is ready to support much more ones, and only Weather events processor MS needs to be scaled accoringly to proces the new load.

Functional improvements can be done to include forecast animated maps in images. They could also be stored in S3 bucket and served through the CDN.