Design Twitter - System Design

System requirements

Functional Requirements

Users can sign up, log in, update profile.
Users can post tweets (text, image, video).
Users can follow/unfollow other users.
Users can view timelines:
- Home timeline → tweets from followees.
- Profile timeline → user’s own tweets.
Users can like, retweet, reply to tweets.
Users can search tweets by keyword/hashtag.
Users can view trending topics/hashtags.
Users receive notifications on relevant events (likes, replies, follows).

Non-Functional Requirements

High availability (99.99% uptime).
Low latency (<200ms for timeline read).
Scalability (support 100M+ users, 500M tweets/day).
Durability (no tweet loss).
Eventual consistency is acceptable for timeline propagation.
Security: OAuth2, rate limiting, encryption in transit/at rest.
Privacy: control over profile visibility.

CAPACITY PLANNING

Users: 100M active users.

Tweets/day: ~500M → ~6K tweets/sec.

Followers graph: ~1B edges.

Home timeline reads: ~50K/sec.

Likes/retweets/replies: ~200K/sec.

- Storage per tweet: ~300 bytes (text + metadata) → 150GB/day → 55TB/year.

API design

POST /signup → register new user

POST /login → authenticate user

POST /tweet → post a tweet

GET /timeline/home → get home timeline

GET /timeline/profile/{user_id} → get profile timeline

POST /follow/{user_id} → follow a user

POST /unfollow/{user_id} → unfollow a user

POST /like/{tweet_id} → like a tweet

POST /retweet/{tweet_id} → retweet

POST /reply/{tweet_id} → reply to a tweet

GET /search?q=keyword → search tweets

GET /trending → get trending topics

Database design

Tables / Collections

Users

scss
CopyEdit
user_id (PK), name, username, email, password_hash, bio, created_at

Followers

scss
CopyEdit
user_id, follower_id (composite PK)

Tweets

scss
CopyEdit
tweet_id (PK), user_id (FK), content, media_url, created_at, like_count, retweet_count, reply_count

Timeline

CopyEdit
user_id, tweet_id, created_at

Likes

scss
CopyEdit
user_id, tweet_id (composite PK)

Retweets

scss
CopyEdit
user_id, tweet_id (composite PK)

Notifications

bash
CopyEdit
notification_id, user_id, type, source_user_id, tweet_id, created_at, is_read

Search Index
- Managed by ElasticSearch or similar engine for full-text search.

High-level design

Main components:

API Gateway / Load Balancer
Web Servers (stateless)
Authentication Service
User Service
Tweet Service
Timeline Service
Search Service (ElasticSearch)
Notification Service
Media Service (CDN + Object Storage)
Cache Layer (Redis/Memcached)
Data Stores:
- User DB (SQL)
- Tweet DB (NoSQL like Cassandra)
- Timeline DB (NoSQL)
- Notification DB

Request flows

Post Tweet

pgsql
CopyEdit
Client → API → Tweet Service → Save to Tweet DB → Timeline Service → Fan-out to followers → Update Timeline DB + Redis → Notify followers

View Home Timeline

vbnet
CopyEdit
Client → API → Timeline Service → Get from Redis (or fallback to Timeline DB) → Return tweets

Search Tweets

sql
CopyEdit
Client → API → Search Service → Query ElasticSearch → Return results

Detailed component design

Timeline Service

Fan-out on write: Push new tweets to followers’ timelines when posted.
Use Redis for hot timelines (recent tweets).
Hybrid approach:
- Regular users → fan-out on write.
- Celebrities → fan-out on read (aggregate on demand).
Scale: horizontally partition by user_id.
Background workers process fan-out queues.

📦 2. Tweet Service

Stores tweets in Cassandra (high write throughput).
Schema:

scss
CopyEdit
tweet_id (PK), user_id, content, media_url, created_at

Media uploaded to S3/GCS, served over CDN.
Maintain counters (likes, retweets) in Redis, periodically sync to DB.

🔔 3. Notification Service

Asynchronous using Kafka + worker pool.
Types: new follower, like, retweet, reply.
Notifications stored in NoSQL DB (DynamoDB or MongoDB).
Mobile/web push handled via integration with notification providers.

Trade offs/Tech choices

Cassandra: for tweets High write throughput, horizontal scalability

Redis for timelines: Low-latency reads for home timeline

ElasticSearch for search: Full-text search, hashtag & keyword support

Kafka for fan-out: Decouples services, handles spikes in write load

CDN for media: Reduces latency, offloads media traffic from core

Failure scenarios/bottlenecks

Timeline fan-out backlog: Use backpressure + prioritize active users

Redis cache miss or failure: Fallback to Timeline DB

Search index lagging behind Tweet DB: Async update, retry queues

Media storage failure: Multi-region S3 replication, CDN fallback

Celebrity tweets overload fan-out pipeline: Use fan-out on read for high-fanout users

Future improvements

Personalized timelines using ML ranking.
Spam detection & abuse prevention systems.
Federated/multi-region deployments.
Better offline/nearline analytics.
User privacy controls for followers/tweets.
Expand trending topics beyond hashtags.