Loading...
Some Key Design Challenges:
With 500 million daily active users, each user accesses their home feed twice daily and uploads an image or video approximately once per week. This results in about 71 million uploads per day.
Estimating storage requirements, images average 2MB in size, leading to roughly 140TB of new image data daily.
Over two years, this amounts to approximately 100PB.
Videos, being significantly larger at 100MB on average, contribute about 7PB of new data per day, accumulating to nearly 5 exabytes over two years.
In total, Instagram sees around 7PB of daily storage growth, leading to a two-year total of approximately 5 exabytes.
Given this scale, a traditional database cannot efficiently manage such large media volumes. A blob storage system is required to store images and videos in a distributed and scalable manner.
upload(user_ID, media_contents, metadata) # Upload photo/video
follow(follower_ID, followed_ID) # Follow another user
show_homefeed(user_ID) # Retrieve home feed
The users table stores essential profile information and supports quick lookups and updates.
This table manages metadata for media files stored in a blob storage system.
This table manages user follow relationships.
Partitioning by UserID keeps all data related to a single user in one shard, making queries for a user’s media and relationships straightforward. However, this approach creates hotspots when some users generate significantly more data than others, leading to an uneven distribution of load.
Partitioning by MediaID (UUID) spreads media more evenly across shards, preventing individual users from overloading a single partition. While this improves distribution, it introduces the challenge of requiring a globally unique identifier before determining the correct shard, adding complexity to writes.
Unique ID generation must support scalability and avoid bottlenecks. Using auto-incrementing IDs can create a single point of failure and limit scaling since ID assignment requires coordination across shards. A better approach is to use Twitter Snowflake or UUIDs. Snowflake IDs are 64-bit timestamp-based identifiers that include machine ID and sequence numbers, ensuring uniqueness while maintaining order, which improves indexing performance. UUIDs, which are 128-bit globally unique identifiers, remove the need for coordination but lack order, which can lead to inefficient indexing and slower queries. Snowflake IDs are generally preferred when maintaining query speed and indexing efficiency is a priority.
Caching Layers
Upload Flow
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Caching plays a critical role in reducing database load and ensuring fast response times, especially in a system where millions of users constantly access and upload content. The caching strategy must be designed to optimize different types of data, including metadata, media files, and feed content.
For handling hot metadata, such as user follow relationships and recent posts, Redis is an effective choice due to its in-memory storage and low-latency access patterns. This allows rapid lookups when users load their feeds, check follower counts, or engage with content. Instead of querying a database each time, the system can retrieve frequently accessed metadata from Redis, significantly reducing response times and database load. A well-configured eviction policy, such as Least Recently Used (LRU), ensures that only the most relevant data stays in memory, preventing excessive memory usage.
Media files themselves present a different caching challenge. Since storing and retrieving large images and videos directly from backend storage is expensive in terms of both latency and bandwidth, the system relies on Content Delivery Networks (CDNs). CDNs cache media closer to users, often at multiple edge locations worldwide, allowing repeated requests to be served without querying the origin server. This greatly reduces backend hits and minimizes latency, especially for high-traffic content, such as viral images or videos that are viewed repeatedly within a short period.
Load balancing ensures that requests are evenly distributed across servers to prevent overload and maintain high availability. One of the most effective techniques for distributing media storage and retrieval is consistent hashing. Unlike traditional hashing methods, which can cause imbalances when new servers are added or removed, consistent hashing maps requests to a dynamically resizable ring of storage nodes. This minimizes data redistribution while allowing the system to scale up or down efficiently. When a media request comes in, the hashing mechanism determines which server should handle it, ensuring a balanced distribution of load across the infrastructure.
For managing high request volumes, particularly when handling spikes in traffic, queue-based rate limiting helps control excessive API requests. Instead of overwhelming the system during peak periods, incoming requests are placed into a queue and processed in a controlled manner. If the queue exceeds predefined limits, the system can apply rate limiting strategies such as token bucket or leaky bucket algorithms, allowing critical requests to be prioritized while throttling non-essential traffic. This prevents sudden traffic surges from degrading system performance and ensures fair resource allocation across all users.
Generating a home feed efficiently requires balancing ranking, precomputed data, and scalability. Users follow different numbers of accounts, and the system must surface relevant posts quickly.
The ranking algorithm sorts posts based on user popularity, post engagement, and follow relationships. Posts from popular users generate high engagement, but posts from close connections are often more relevant. A weighted function determines the top posts for each user.
A heap-based approach helps retrieve only the top K posts efficiently. Instead of sorting all posts, which would be expensive, a max heap keeps only the most relevant ones in memory, reducing unnecessary computation. To further optimize, trending posts are precomputed and cached. Since these posts receive high engagement across the platform, they are likely to be relevant to many users. Precomputing them reduces repeated calculations.
Some users follow thousands of accounts, while others follow only a few. To manage this load, the system uses precomputed feeds for high-follow users. These feeds are updated in the background and stored for fast access. Users with fewer follow relationships generate their feeds on demand, retrieving recent posts dynamically. This approach reduces unnecessary storage and processing.
Sharded message queues handle real-time updates. When a user uploads a post, a message queue updates relevant feeds asynchronously. This avoids frequent polling and spreads the workload across multiple shards to prevent bottlenecks.
Handling large video files efficiently requires a combination of adaptive streaming, distributed caching, and optimized storage strategies. Videos can be several gigabytes in size, and delivering them without causing excessive load on servers or increasing latency is a key challenge. The system must ensure smooth playback while minimizing storage and bandwidth costs.
A major challenge in video streaming is that users have different network conditions, device capabilities, and bandwidth limits. Adaptive streaming helps by breaking videos into segments and dynamically adjusting their quality based on real-time network conditions. Protocols like HLS (HTTP Live Streaming) and DASH (Dynamic Adaptive Streaming over HTTP) divide videos into small chunks, each encoded at multiple bitrates.
When a user streams a video, the player requests segments based on available bandwidth. If the network slows down, the player switches to a lower-resolution segment, preventing buffering. If the connection improves, it switches back to higher-quality segments. This method ensures a smooth viewing experience without requiring the user to manually adjust video quality.
To further reduce latency, videos are served from the closest edge server. The typical path is ISP → IXP → CDN, where ISPs (Internet Service Providers) and IXPs (Internet Exchange Points) cache frequently accessed content at locations near users. By serving video files from these edge locations instead of the origin storage, streaming latency is reduced, and backend servers are protected from unnecessary load.
Since videos are large, CDNs cache transcoded versions of popular videos to reduce redundant processing. Instead of re-encoding each request, precomputed video chunks are stored at different bitrates and resolutions. When a user requests a video, the CDN quickly serves the correct version based on the user's connection quality. This method drastically reduces the need for repeated encoding and saves backend computing resources.
Since videos take up significant storage space, the system must balance cost and performance by using different storage types based on access patterns.
Frequently viewed videos are stored on SSD-based fast storage, allowing for quick retrieval. These videos typically belong to trending or recently uploaded content, which sees high engagement and requires low-latency access. SSDs provide high read speeds, reducing playback delays for popular videos.
Less frequently accessed videos are moved to HDD-based or archival storage, which is cheaper but slower. These could be older videos that users rarely watch but still need to be retained. Since retrieval speed is less critical for this content, HDDs provide a cost-effective way to store large amounts of video data without consuming expensive SSD resources.
Geo-optimized storage ensures that content is placed closer to users who are most likely to access it. If a video is primarily watched in a specific region, copies are kept on storage nodes within that area. For example, Japanese-language videos are stored on servers near Japan, while French-language content is kept in European data centers. This reduces data transfer times and bandwidth costs. However, all content must still be backed up across multiple regions for disaster recovery, ensuring that videos remain available even if a data center goes offline.
By combining adaptive streaming, CDN caching, and a tiered storage approach, the system optimizes performance while keeping costs under control. This ensures that users experience smooth, high-quality video playback while backend resources are used efficiently.
For some users, the system pulls new posts when they refresh their feeds. This is efficient for users with many follows, as pushing updates for thousands of accounts would be costly.
For users with fewer follows, the system pushes updates immediately when a followed user posts new content. This keeps their feeds fresh without requiring manual refreshes.
A hybrid approach works best. High-follow users rely on pull-based feeds, while low-follow users receive push-based updates. This ensures timely updates while keeping the system efficient.
By combining ranking algorithms, precomputed feeds, message queues, and a push-pull model, the system delivers a scalable and responsive feed that works across millions of users.
A system like Instagram must remain accessible despite failures. This is achieved through data replication, automated failover, and robust backups.
Multiple Replicas ensure that all critical data, including user metadata and media files, is stored across multiple regions. Databases use leader-follower replication to distribute read requests and promote backups when needed. Media storage follows a similar model, with copies spread across geographically distributed nodes to prevent data loss.
Failover Strategies detect failures and reroute traffic automatically. RAFT or Paxos consensus protocols handle database failover by electing a new leader when the primary fails. CDNs ensure media availability by caching files across multiple locations, rerouting requests if a node goes down. Health checks monitor databases, storage nodes, and application servers to trigger failover processes with minimal downtime.
The 3-2-1 Backup Rule protects against catastrophic failures. The system maintains three copies of data, stores them on two different media types, and keeps one offsite backup for disaster recovery. Incremental snapshots optimize storage while ensuring fast recovery.
By combining replication, failover automation, and structured backups, the system guarantees reliability at scale, keeping data safe and accessible even in the event of failures.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?