Loading...
Decent-scale weather app: 12M users, 40k requests per second average. That request volume matters more than user count for capacity planning: 40k/s over a day works out to roughly 3.4 billion requests, and that number drives read-path caching and load balancer sizing far more than the 12M registered accounts do.
Location coverage is where the load looks bigger than it actually is. 10k locations, each refreshed every 10 minutes, is 144 writes per location per day. But 144 writes/day isn't what needs to persist: the read path only ever wants the latest value, and the historical record only needs one snapshot per location per day, not one per update. That's 10k historical writes/day, not 1.44M.
At 2KB per historical object, that's 20MB/day, 600MB/month, and 36GB over a five-year retention window. Small enough that storage cost isn't a constraint on this design; the pressure is entirely on the live read/write path handling the 3.4B daily requests, not on historical persistence.
I am using a mixture of graphql and Rest here because I have different shapes of the location depending on the request
// search user locations
GET /locations
query params:
Response:
List of Locations that match the search critria , without forcast and with the weather high, low and weather data point
// show weather
GET /locations/{id}
Returns: Location object without forcast
//show forcast
GET /locations/{id}
Returns The full location object including the forcast
// show historical data
GET /locations/{id}/historical
Params:
Returns a full list of weather objects within the time window.
// list user locations
GET /users/{id}/locations/
Returns Locations similar to search
// add new location
POST /users/{id}/locations/
Body {location: id}
// delete a user location
DELETE /users/{id}/locations/{id}
// subscribe to alerts
POST /users/{id}/locations/{id}/subscribe
POST /users/{id}/locations/{id}/unsubscribe
Models
Location:
{
"id": long,
"name": string,
"coordinates": (float, float),
"weather": Weather,
"forecast": {Weather}
}
Weather:
{
"date": "Date",
"locationId": long,
"high": float,
"low": float,
"scale": {Mertic|Imprial],
"dataPoints: [DataPoint]
// ... averages maybe
}
DataPoint:
{
"timeStamp": TimeStamp,
"temperature": float,
"humidity": float,
"wind": float
"uvIndex": float,
}
The system runs in two different planes: the ingestion plane and the app plane.
The ingestion plane
Note on consistency:
App Plane:
Four different data stores are doing four different jobs here, and the temptation is to reach for one database that does all of it. Resist that: the event store, the current/forecast store, the analytics store, and the geospatial index have different write patterns, different consistency needs, and different query shapes, and forcing them into one engine means optimizing for none of them.
This is the append-only log of every WeatherUpdateEvent, and it's what makes replay-to-snapshot possible. It needs to be genuinely append-only and cheap to scan sequentially per location; it doesn't need secondary indexes, ad hoc queries, or update-in-place.
Table: weather_events
partition_key: location_id
sort_key: event_version (monotonic, assigned at ingestion)
columns:
event_id uuid
location_id string
forecast_horizon int -- 0 = current, N = N hours out
event_version bigint -- monotonic per location, not wall-clock
observation_time timestamp -- from the provider, for display/analytics only
ingested_at timestamp
payload json -- temp, humidity, wind, condition, etc.
provider_id string
Partitioning by location_id and sorting by event_version is the load-bearing decision here, and it's the same key structure the log bus uses. That's not a coincidence: if the partition and sort key in storage didn't match the partition key in the bus, replaying a location's history in order would require a scatter-gather across partitions instead of a single sequential scan. Wide-column stores (Cassandra, DynamoDB, Bigtable) fit this shape well; a single-writer relational table would work at small scale but starts fighting you once you're ingesting at the rate a "continuous updates via API" source implies.
event_version is assigned by the Ingestion Pipeline, not the database and not the provider's timestamp, for the same reordering reason called out in the pipeline design: two providers' clocks can disagree, but a version assigned once at the point of transformation is comparable within a location regardless of source.
Retention here is a real decision, not a default: keeping every event forever means replay time grows unbounded for old locations. The snapshot mechanism below exists specifically so this table can be trimmed (say, keep 30 days of raw events plus periodic snapshots) without losing the ability to reconstruct current state.
This is what the Weather Database consumer writes to, and it's a materialized view over the event store, not an independent source of truth. Losing it entirely should be recoverable by replaying events, which is the whole point of event sourcing it in the first place.
Table: location_weather
partition_key: location_id
sort_key: forecast_horizon
columns:
location_id string
forecast_horizon int
last_event_version bigint -- for convergence checks on replay/dedup
temperature float
condition string
humidity float
wind_speed float
updated_at timestamp
The upsert rule is: write the event's payload only if event.event_version > stored.last_event_version for that (location, horizon) pair. That single comparison is what gives you the convergence property described earlier: replaying a partition twice, or replaying it out of order, always lands on the same final row, because every write is conditioned on the version check rather than blindly overwriting. Skip that check and a duplicate or late-arriving event silently reintroduces stale data, which is a much harder bug to find than a rejected write.
One row per (location, horizon) rather than one row per location with a nested forecast array is deliberate: forecasts update independently (a 24-hour-out forecast changes more often than a 3-day-out one), so per-horizon rows let each update touch exactly the row it affects instead of rewriting a whole nested structure for a one-field change.
This is a denormalized copy of location_weather, indexed for spatial and text search instead of point lookup by ID.
Document: location_index
id: location_id
fields:
name, country, lat, lon
geohash -- for radius/bounding-box queries
current: { temperature, condition, humidity, wind_speed, updated_at }
forecast: [ { horizon, temperature, condition, ... }, ... ]
This is the piece that eliminates the two-hop lookup: a search query returns the weather payload inline, no follow-up read against location_weather. The cost is an obvious one worth stating plainly: every write to location_weather now fans out to a second write here, and the two can drift if one write succeeds and the other fails. Since the geospatial index is fully derivable from location_weather (itself derivable from the event log), a drifted index is a rebuild-from-source problem, not a data-loss problem; a periodic reconciliation job that diffs the two and re-syncs mismatches is cheap insurance against that drift and should be built alongside this table, not added later once someone notices stale search results.
Technology-wise this wants a store built for geospatial and full-text queries (Elasticsearch/OpenSearch, or a Postgres instance with PostGIS plus a trigram index), not the wide-column store the event log lives in; trying to get geo-radius queries out of Cassandra is possible but is fighting the engine.
Written by the Analytics Engine consumer, read by the historical-data browsing path in the App Plane. This one is genuinely append-only and time-series shaped, and it never gets updated in place: a historical record for a past hour doesn't change after the fact.
Table: weather_facts
partition_key: location_id
sort_key: observation_time
columns:
location_id, observation_time, temperature, condition,
humidity, wind_speed, forecast_horizon, source_event_id
Partitioning by location and sorting by observation_time (not event_version) is the right choice here specifically because this table serves human queries ("show me last month's weather for this location"), where observation_time is what the user means by "when," while event_version is an ingestion-internal ordering concern that has no reason to leak into this schema. A columnar or time-series store (a wide-column store with time-bucketed partitions, or a purpose-built TSDB) fits better than either the event store's engine or the geospatial index's engine, since the query pattern is range scans over time for a single location, not point lookups or spatial search.
Small and conventional relative to the rest of this design, which is worth saying outright rather than over-engineering it to match the other stores:
Table: users
user_id (pk), email, auth_provider_id, created_at
Table: user_locations
partition_key: user_id
sort_key: location_id
A user's saved-location list is small (single digits to low tens of entries), so this is a plain relational or key-value table with no special indexing beyond lookup by user_id. Reaching for anything more sophisticated here would be solving a problem this table doesn't have.
Snapshotting cadence for the event store isn't specified yet: how often a snapshot gets taken per location, and whether it's time-based or event-count-based, changes both replay cost after a trim and how much history you're willing to lose if a snapshot itself is corrupt. The reconciliation job between location_weather and the geospatial index is described but not scheduled; it needs an owner and a drift-detection SLA before this goes to implementation, not after someone notices search results disagreeing with the detail view.
The entry point does two jobs that look similar but have different failure modes: crawling external stations on a schedule, and accepting pushed updates over API calls. Crawling is idempotent by construction (you re-fetch and get the current state), so it can retry freely. Pushed updates are not, since a retried push can duplicate an event. The Ingestion API should assign a client-supplied or server-generated idempotency key per update at the point of ingestion, not later in the pipeline, because once it's in the log the dedup responsibility becomes distributed across every consumer instead of centralized in one place.
The API's only job past that is authenticating the source (each official station gets its own credential, not a shared pipeline key) and handing the raw payload to the Ingestion Pipeline. It shouldn't validate schema itself; that's the pipeline's job, and splitting it would just create two places that can disagree about what a valid payload looks like.
This is where the raw provider payload becomes a standard WeatherUpdateEvent. Two concerns live here and they should stay separate steps even though they run back to back:
The output is one WeatherUpdateEvent per (location, forecast horizon) reading, published to the log bus keyed by location ID. Keying by location, not by provider or by shard round-robin, is what gives you ordering per location without needing a global order across the whole bus.
Partitioning by location ID matters more than it looks like it should: it's what guarantees that all events for a given location land in the same partition and are consumed in order by each consumer. Without that guarantee, the "replay and snapshot" story in your consistency note doesn't hold, since replay only reconstructs the correct current state if the events arrive in the order they were produced.
The three consumers read the same partitioned log independently, which is the point of using a log instead of a queue: each one can fall behind or catch up without affecting the others, and each can replay from its own last committed offset after a crash.
Analytics Engine. Turns raw events into facts and appends to the historical store. This one is naturally append-only and doesn't care about "current" state; it can process events out of strict real-time order as long as it eventually sees all of them, so it's the consumer with the most slack if the bus backs up.
Weather Database (current + forecast). This is the consumer with the tightest coupling to your consistency model, because it's what the cache and geospatial index are derived from. It applies each event as an upsert keyed by (location ID, forecast horizon), and because of event sourcing, an out-of-order or duplicate event doesn't corrupt state: replaying the log for a location and keeping the highest event version per horizon always converges to the correct snapshot. That convergence property is what lets you shrug off duplicate or reordered messages instead of needing exactly-once delivery from the bus.
Notification Manager. Decides whether an update is notification-worthy (a forecast crossing a severe-weather threshold, for instance) and dispatches to devices. This consumer is the one place where event ordering actually changes user-visible behavior: if a "severe storm" event and a later "downgraded" event get processed out of order, a user gets a false alarm. Worth calling out as a place where at-least-once delivery plus idempotent notification IDs isn't sufficient on its own; the Notification Manager needs to check event versioning before firing, not just dedupe by event ID.
Every write to the Weather Database emits a cache invalidation for that location's key and updates the geospatial index entry in place. The geospatial index existing at all is really an optimization to avoid the two-hop lookup you described (location search, then a separate current/forecast fetch); folding both into one indexed document per location means a location search returns weather data directly, no second round trip.
Both of your options are workable; the choice is really about what you're willing to guarantee to the client, not which is "more correct."
Repair-on-read with a read replica. On a read, if the replica might be stale, the read path checks the replica's last-applied offset against the known latest offset for that location and, if behind, reads through to the primary (or forces a re-sync) before returning. This gets you linearizability at the cost of an extra check on every read, and that check only stays cheap because the location count is static and small enough to keep offset metadata in memory rather than doing a full comparison against the log.
Sticky sessions. Simpler and cheaper (no per-read check), but the guarantee it gives you is weaker than it sounds: it's read-your-writes-from-that-replica, not linearizability, and it silently degrades to stale reads on failover, which is exactly the case you flagged. If you go this route, the failover behavior needs to be an explicit, documented trade-off, not a bug someone finds later, since "sticky until the node dies" is a consistency guarantee that only holds during the happy path.
Given that weather forecasts are consumed at human timescales (nobody needs microsecond consistency on a temperature reading), I'd actually push back gently on repair-on-read being necessary here: the cost of a stale read is low, and sticky sessions with a short replica lag SLA probably serves this system better than paying a per-read consistency check on every request. That's a call worth making explicit in the doc rather than defaulting to the stronger guarantee because it sounds safer.
Load balancer and rate limiter. Standard placement in front of the gateway; the rate limiter should key on authenticated identity where available and fall back to IP otherwise, since IP-only limiting is trivially bypassed and identity-only limiting doesn't help pre-auth abuse.
Gateway (authn/authz). Terminates OAuth and issues the internal session or token that downstream services trust. Downstream services (User API, Search API, Analytics Engine) should never re-verify the OAuth token themselves; they trust the gateway's signed internal token. Splitting that trust boundary avoids every service needing its own OAuth client config.
User API. Profile and location list CRUD. The location list here is small per user (a handful of saved locations), so it's a straightforward read/write against the user's own record; no special indexing needed beyond a lookup by user ID.
Search API. Location search plus current/forecast, served from the geospatial index directly. This is the API that benefits most from the "one document per location" design, since a single index read satisfies both the location match and the weather payload.
Analytics Engine access. Historical data browsing for a location. This is a read-only path into the Analytics Engine's store, separate from the write path that ingestion uses; worth keeping those as genuinely separate read/write concerns (possibly separate service instances) so a heavy historical query from a user doesn't compete with the ingestion consumer for the same resources.