System requirements

Functional:

content is dynamic, real time updates

user can customize, filter and sort

newsfeed(prioritize, hides, contents)

personal recommendations, ranking by algo

engagement metrics

user interaction notification

privacy control

Non-Functional:

reliability,TP 99.9, achieved through geo-distributed storage and multi-region replication.

Disaster Recovery: Data replication across regions with an RTO (Recovery Time Objective) of <10 minutes.

performance, 1 sec for loading

scalability?

back up and recovery

final consistency

Capacity estimation

DAU, 1.9billion

1post per user per day = 1.9 B post

4 interactions per post = 5.7 B interactions

each post = 100K, 1.9T per day for User generated contents

API design

1. Create a Post

POST /posts

2. Get Newsfeed

GET /newsfeed/{userId}?limit=20&cursor=timestamp_1700000000&filter=videos

3. Like & Comment

POST /posts/{postId}/like

POST /posts/{postId}/comment

Database design

1. Users Table

• user_id (Primary Key)
• name
• friends_list (Array of user_ids)
• following_pages (Array of page_ids)

2. Posts Table

• post_id (Primary Key)
• user_id (Foreign Key)
• content
• media_urls (Array)
• timestamp

3. Newsfeed Table (Precomputed Feeds)

• user_id (Primary Key)
• feed_items (List of post_ids)
• last_updated (Timestamp)

High-level design

Content Ingestion

Feed Generation

• Precomputed Feed Storage in Redis for top users.

Ranking & Personalization

Collaborative Filtering: Suggests posts based on similar user behaviors.

Content-Based Filtering: Prioritizes posts based on user preferences (e.g., past interactions).

Engagement-Based Ranking: Prioritizes highly liked/commented posts from close connections.

Diversity Mechanism: Prevents repetitive posts from dominating a feed.

Real-time Personalization: Uses reinforcement learning to adjust ranking dynamically.

Storage & Caching

Real-Time Updates

1. Kafka Streams process real-time engagement events (likes, comments).
2. WebSockets & Push Notifications notify users about feed updates.
3. Edge Caching (CDN + Redis) ensures low-latency content retrieval.

Trade-Off Balance:

• High Engagement Users → Use push-based updates (instant).
• Low Engagement Users → Use pull-based updates (periodic refresh).

CDN caching for media content.

Request flows

feed generation models:

push model(fanout on write)

Precompute and store a user’s newsfeed in a Newsfeed Table.

Posts from friends/pages are pushed to followers during creation.

Low latency but requires high storage.

pull model:

The feed is generated dynamically when a user opens the app.

Uses an index of latest posts from followed users.

More scalable, but higher query load.

Hybrid Model (Optimized for Scale)

• Active users → Use Push Model (precomputed feed for fast access).
• Less active users → Use Pull Model (fetch on demand).

Detailed component design

Ranking & Personalization

The ranking model evaluates:

• Engagement Signals – Likes, shares, comments.
• Content Type – Videos > Photos > Text.
• Recency – More recent posts rank higher.
• User Preferences – Past interactions.
• Diversity – Avoid content repetition.
• Ads & Sponsored Posts – Monetization.

💡 Tech Stack:

• Machine Learning (PyTorch, TensorFlow)
• Feature Storage (Redis, Cassandra)
• Ranking Pipeline (Apache Spark, Kafka)

Real-Time Updates

• WebSockets or Push Notifications send updates instantly.
• Kafka Streams process live interactions (likes, comments).
• Edge Caching ensures high-speed delivery.

Storage & Caching Strategy

• Posts & User Data → Cassandra (fast writes, scalable).
• Feed Cache → Redis / Memcached (low latency).
• Cold Storage → S3 / HDFS for archival.Content Expiration & Archival: Older posts with low engagement are gradually moved to cold storage (e.g., S3).

Trade offs/Tech choices

cache stategy:

user post: use cache aside or write behind depending on user's follower number

Failure scenarios/bottlenecks

High Read Load

like when a influencer has 1M follower. resolve: use sharding by follower_id and write to follower's feed

Node Failure

Ranking Delays

High Storage Cost

enabled compression in cassandra, use lz4

Cache Penetration

set post value = null in cache with TTL = 30s

Future improvements

Better Personalization – AI-driven interest prediction.

Efficient Storage – Auto-delete inactive feeds. move to cold storage.

A/B Testing – Experiment with ranking models. 1% of users get the new ranking model.

If engagement drops >5%, rollback the update.

Dynamic Scaling using Kubernetes for backend services based on real-time traffic.

Handling Inactive Users

Strategy: Lazy Feed Computation

• • If a user logs in after a long absence, the system fetches the latest posts on-demand rather than precomputing them.