Loading...
List the key functional requirements for the system (Ask the AI for hints if stuck)...
List the key non-functional requirements (performance, scalability, reliability, etc.)...
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Assuming daily active users of 100 million, where on average each user browses playlists and view songs 10 times per day, and add 2 songs to playlists. Assuming peak QPS is double of average QPS.
Peak read QPS = 100 million * 10 / 3600 / 24 * 2 = 23K
Peak write QPS = 100 million * 2 / 3600 / 24 * 2 = 4.6K
Moreover, we need to worry about concurrent streams. Roughly 20-30% of Spotify's MAU are streaming simultaneously, leading to 20-30M concurrent streams. Assuming we stream 320KB per second, that's 20M * 320Kb = 6.4TBPS.
For storage, assuming that we store up to 100 million songs.
For each song, metadata takes around 1KB, while the song itself takes 5MB. We also need different audio quality versions of the same song, so we need 15MB for each song.
So we need:
100 million * 1kb = 100GB of metadata storage, and 1.5PB of object storage.
Moreover, we need search index to store song title and metadata, and need storage for user profiles. Assuming each user's playlists, history and profiles is around 1MB, we will need 100 million * 1MB = 100TB of storage.
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
Here are some APIs.
For users to stream a song:
GET v1/stream_song {
user_id: UUID,
song_id: UUID,
audio_quality: String
}
This returns a signed CDN URL.
For users to view their playlist:
GET v1/view_playlist {
user_id: UUID,
playlist_id: UUID
}
For users to share their playlist to another:
POST v1/share_playlist {
user_id: UUID,
playlist_id: UUID,
sharing_user_id: UUID
}
For users to search:
GET v1/search_results {
user_id: UUID,
search_text: String,
filters: List
limit: Int,
}
For users to retrieve recommendations for them:
GET v1/get_recommendations {
user_id: UUID
}
For users to add/edit playlist:
| Method | Endpoint | Purpose | Request body | Response |
POST | /v1/playlists | Create playlist | {name, description?, is_public?} | 201 + playlist object |
GET | /v1/playlists/{id} | View playlist | — | 200 + playlist + tracks |
PATCH | /v1/playlists/{id} | Rename / reorder / toggle visibility | {name?, description?, is_public?} | 200 + updated object |
DELETE | /v1/playlists/{id} | Delete playlist | — | 204 |
POST | /v1/playlists/{id}/tracks | Add song(s) | {track_ids: [...], position?} | 200 + track list |
DELETE | /v1/playlists/{id}/tracks/{track_id} | Remove song | — | 204 |
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
Let's walk through all the read/write paths for the system.
For all requests to the system, we go through edge API gateway, which does authentication, authorization and rate limiting. Then we hit load balancers, which route requests to different servers using consistent hashing of user_ids.
Ingesting songs:
We have a song ingestion service, that stores all the available songs the platform has. With each addition/deletion/update, we trigger events on kafka queue. Upon ingestion of these events by the song upload service, we write to 3 storage engineers:
We use Kafka queue to buffer the writes, so if we do ingestion of a bulk of songs, we don't overwhelm the underlying storage engines with huge write throughput. This could lead to some ingestion delays, but is an acceptable tradeoff in our business scenario.
Streaming songs:
When users start to stream a song, we query the redis cluster to get the song's metadata, like titles, lyrics etc. If not available in redis, we query the underlying relational database. Then using the song_id, we retrieve the song audio data from local CDNs, with the audio quality we want. If not available, we retrieve the song from S3 object storage, and write to local CDNs.
For streaming songs on users' local devices, the mobile/web client requests a signed CDN URL, streams chunks, and caches tracks locally for offline playback
Analytics and recommendation:
Every time a song is ingested, or when a user listens to a soundtrack, we produce an event that is put on kafka queue. The analytic service ingests it, and writes the event data to cassandra cluster. We train an ML model based on the event data, and use that ML model to server recommended songs/playlists for each user.
Creating/editing playlists:
When users create playlists, we write to both the redis cluster (and asynchronously to the database) and ElasticSearch from the playlist service. This will support retrieval and search of playlists.
Concurrent edits to the playlists:
If 2 users make concurrent edits to the same playlist, we use vector clocks to detect conflicts. And for conflict resolution, we use application layer logic to resolve. Compared to a deterministic resolving method like LWW, application layer resolver could prevent us from accidentally overwriting a user's chance, with the tradeoff of additional complexity.
Viewing/Searching playlists and songs:
For viewing playlists/songs, we hit the redis cluster from streaming service or playlist service, and read the playlist/song metadata from redis cache. If not available, we hit the underlying relational database cluster. Since we decide which database shard to hit based on user_id, if user uses a different device, we would get back the same song/playlist data. So updates are synced across devices.
For searching/filtering, we query the ElasticSearch index with the relevant search queries and filter keys.
Scaling to more users and maintaining high availability/low latency:
The servers are stateless, so they are easily horizontally scalable. We can implement auto scaling policies based on traffic volume.
The CDNs are globally distributed, each CDN cache the most frequent songs in that geographical area.
The relational databse needs to be sharded and replicated to scale. We will shard by song_id and playlist_id using consistent hashing, so requests from users for the same song/playlist will land on the same database shard. For hot spot, we shard the key for super hot songs/playlists, so requests to them are evenly distributed across shards.
We create read replicas for each database shard. For each write, the result is asynchronously propagated to each read replica. While this leads to eventual consistency, we get much lower write latency. The metadata store (MySQL) is deployed as a primary + synchronous multi-AZ replica pair fronted by a connection-aware proxy. Every write commits to the replica before acknowledging the client, so at most one region's worth of in-flight writes can be lost on failover. On primary loss, the proxy health-checks the replica, promotes it to primary within seconds (auto-increment offset and binary log position verified), and continues serving writes — no manual intervention, no write stall. The old primary rejoins as a replica and re-syncs from the promoted node. The cost is +1 round-trip of write latency per transaction, which playlist metadata tolerates; the alternative (async replica) risks losing the last few acknowledged writes, which I reject because it breaks the user's trust in their playlists. This design lets me meet the 99.99% availability target without a second of write downtime.
On ElasticSearch side, we will use default document_id sharding, given query pattern is search queries inputted by users.
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
There are several things that are stored.
We store the songs, in different audio qualities, and their associated metadata.
We store the playlists, created by users.
We store the playlist interaction and listening records of each user, so we can use that historical data to train machine learning models.
We also store users metadata.
For songs' media information, like the actual audio in different qualities, we store them in object storage like S3. In actual streaming, we cache them in local CDNs. Most of the streaming requests go to the top most hot songs. Storing them in local CDNs help with low latency in streaming.
For songs and playlists metadata, we store them in 2 places.
Here's some sample data model:
table songs {
song_id: UUID,
song_title: String,
artist_id: UUID,
artist_name: String,
album_id: UUID,
album_name: String,
song_duration: String,
song_publish_date: Timestamp,
song_genre: String,
song_ratings: Integer,
song_additional_metadata: String
}
table playlists {
playlist_id: UUID,
playlist_name: String,
playlist_song_ids: List
playlist_song_names: List
playlist_created_by_user_id: UUID,
playlist_created_at: Timestamp,
playlist_last_updated_at: Timestamp,
playlist_visibility_setting: String,
playlist_metadata: String
}
On top of the relational database, we also have a caching layer built on top of a redis cluster. It is a write-back cache, where each update is written to the cache only, and asynchronously written to the database. This will lead to eventual consistency, and data durability concerns should cache cluster crash before data is written to database. However, for songs and playlists data, such tradeoff is acceptable in favor of lower write latency.
For logs generated for each user after they listen to a song or create/edit a playlist, we store such events in cassandra. The data can be eventually consistent since it is only used for analytic and machine learning training purposes. Cassandra will be able to support our need for
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
There are several things that are stored.
We store the songs, in different audio qualities, and their associated metadata.
We store the playlists, created by users.
We store the playlist interaction and listening records of each user, so we can use that historical data to train machine learning models.
We also store users metadata.
For songs' media information, like the actual audio in different qualities, we store them in object storage like S3. In actual streaming, we cache them in local CDNs. Most of the streaming requests go to the top most hot songs. Storing them in local CDNs help with low latency in streaming.
For songs and playlists metadata, we store them in 2 places.
Here's some sample data model:
table songs {
song_id: UUID,
song_title: String,
artist_id: UUID,
artist_name: String,
album_id: UUID,
album_name: String,
song_duration: String,
song_publish_date: Timestamp,
song_genre: String,
song_ratings: Integer,
song_additional_metadata: String
}
table playlists {
playlist_id: UUID,
playlist_name: String,
playlist_song_ids: List
playlist_song_names: List
playlist_created_by_user_id: UUID,
playlist_created_at: Timestamp,
playlist_last_updated_at: Timestamp,
playlist_visibility_setting: String,
playlist_metadata: String
}
On top of the relational database, we also have a caching layer built on top of a redis cluster. It is a write-back cache, where each update is written to the cache only, and asynchronously written to the database. This will lead to eventual consistency, and data durability concerns should cache cluster crash before data is written to database. However, for songs and playlists data, such tradeoff is acceptable in favor of lower write latency.
For logs generated for each user after they listen to a song or create/edit a playlist, we store such events in cassandra. The data can be eventually consistent since it is only used for analytic and machine learning training purposes. Cassandra will be able to support our need for
Let's walk through all the read/write paths for the system.
For all requests to the system, we go through edge API gateway, which does authentication, authorization and rate limiting. Then we hit load balancers, which route requests to different servers using consistent hashing of user_ids.
Ingesting songs:
We have a song ingestion service, that stores all the available songs the platform has. With each addition/deletion/update, we trigger events on kafka queue. Upon ingestion of these events by the song upload service, we write to 3 storage engineers:
We use Kafka queue to buffer the writes, so if we do ingestion of a bulk of songs, we don't overwhelm the underlying storage engines with huge write throughput. This could lead to some ingestion delays, but is an acceptable tradeoff in our business scenario.
The client measures download speed per chunk and requests the next chunk at a matching quality tier, pre-buffering 10–15s ahead to hide dips.
Streaming songs:
When users start to stream a song, we query the redis cluster to get the song's metadata, like titles, lyrics etc. If not available in redis, we query the underlying relational database. Then using the song_id, we retrieve the song audio data from local CDNs, with the audio quality we want. If not available, we retrieve the song from S3 object storage, and write to local CDNs.
For streaming songs on users' local devices, the mobile/web client requests a signed CDN URL, streams chunks, and caches tracks locally for offline playback
Analytics and recommendation:
Every time a song is ingested, or when a user listens to a soundtrack, we produce an event that is put on kafka queue. The analytic service ingests it, and writes the event data to cassandra cluster. We train an ML model based on the event data, and use that ML model to server recommended songs/playlists for each user.
Creating/editing playlists:
When users create playlists, we write to both the redis cluster (and asynchronously to the database) and ElasticSearch from the playlist service. This will support retrieval and search of playlists.
Concurrent edits to the playlists:
If 2 users make concurrent edits to the same playlist, we use vector clocks to detect conflicts. And for conflict resolution, we use application layer logic to resolve. Compared to a deterministic resolving method like LWW, application layer resolver could prevent us from accidentally overwriting a user's chance, with the tradeoff of additional complexity.
Viewing/Searching playlists and songs:
For viewing playlists/songs, we hit the redis cluster from streaming service or playlist service, and read the playlist/song metadata from redis cache. If not available, we hit the underlying relational database cluster. Since we decide which database shard to hit based on user_id, if user uses a different device, we would get back the same song/playlist data. So updates are synced across devices.
For searching/filtering, we query the ElasticSearch index with the relevant search queries and filter keys.
We use typo handling as well: n-gram tokenizer + phonetic filter (double metaphone) so 'Beyonse' → Beyoncé."
Scaling to more users and maintaining high availability/low latency:
The servers are stateless, so they are easily horizontally scalable. We can implement auto scaling policies based on traffic volume.
The CDNs are globally distributed, each CDN cache the most frequent songs in that geographical area.
The relational databse needs to be sharded and replicated to scale. We will shard by song_id and playlist_id using consistent hashing, so requests from users for the same song/playlist will land on the same database shard. For hot spot, we shard the key for super hot songs/playlists, so requests to them are evenly distributed across shards.
We create read replicas for each database shard. For each write, the result is asynchronously propagated to each read replica. While this leads to eventual consistency, we get much lower write latency. The metadata store (MySQL) is deployed as a primary + synchronous multi-AZ replica pair fronted by a connection-aware proxy. Every write commits to the replica before acknowledging the client, so at most one region's worth of in-flight writes can be lost on failover. On primary loss, the proxy health-checks the replica, promotes it to primary within seconds (auto-increment offset and binary log position verified), and continues serving writes — no manual intervention, no write stall. The old primary rejoins as a replica and re-syncs from the promoted node. The cost is +1 round-trip of write latency per transaction, which playlist metadata tolerates; the alternative (async replica) risks losing the last few acknowledged writes, which I reject because it breaks the user's trust in their playlists. This design lets me meet the 99.99% availability target without a second of write downtime
On ElasticSearch side, we will use default document_id sharding, given query pattern is search queries inputted by users.