System requirements


Functional Requirements

  • Users can sign up, log in, update profile.
  • Users can post tweets (text, image, video).
  • Users can follow/unfollow other users.
  • Users can view timelines:
    • Home timeline → tweets from followees.
    • Profile timeline → user’s own tweets.
  • Users can like, retweet, reply to tweets.
  • Users can search tweets by keyword/hashtag.
  • Users can view trending topics/hashtags.
  • Users receive notifications on relevant events (likes, replies, follows).

Non-Functional Requirements

  • High availability (99.99% uptime).
  • Low latency (<200ms for timeline read).
  • Scalability (support 100M+ users, 500M tweets/day).
  • Durability (no tweet loss).
  • Eventual consistency is acceptable for timeline propagation.
  • Security: OAuth2, rate limiting, encryption in transit/at rest.
  • Privacy: control over profile visibility.





CAPACITY PLANNING

Users: 100M active users.

Tweets/day: ~500M → ~6K tweets/sec.

Followers graph: ~1B edges.

Home timeline reads: ~50K/sec.

Likes/retweets/replies: ~200K/sec.

    • Storage per tweet: ~300 bytes (text + metadata) → 150GB/day → 55TB/year.





API design

POST /signup → register new user

POST /login → authenticate user

POST /tweet → post a tweet

GET /timeline/home → get home timeline

GET /timeline/profile/{user_id} → get profile timeline

POST /follow/{user_id} → follow a user

POST /unfollow/{user_id} → unfollow a user

POST /like/{tweet_id} → like a tweet

POST /retweet/{tweet_id} → retweet

POST /reply/{tweet_id} → reply to a tweet

GET /search?q=keyword → search tweets

GET /trending → get trending topics





Database design

Tables / Collections

  1. Users
scss CopyEdit user_id (PK), name, username, email, password_hash, bio, created_at
  1. Followers
scss CopyEdit user_id, follower_id (composite PK)
  1. Tweets
scss CopyEdit tweet_id (PK), user_id (FK), content, media_url, created_at, like_count, retweet_count, reply_count
  1. Timeline
CopyEdit user_id, tweet_id, created_at
  1. Likes
scss CopyEdit user_id, tweet_id (composite PK)
  1. Retweets
scss CopyEdit user_id, tweet_id (composite PK)
  1. Notifications
bash CopyEdit notification_id, user_id, type, source_user_id, tweet_id, created_at, is_read
  1. Search Index
    • Managed by ElasticSearch or similar engine for full-text search.





High-level design

Main components:

  • API Gateway / Load Balancer
  • Web Servers (stateless)
  • Authentication Service
  • User Service
  • Tweet Service
  • Timeline Service
  • Search Service (ElasticSearch)
  • Notification Service
  • Media Service (CDN + Object Storage)
  • Cache Layer (Redis/Memcached)
  • Data Stores:
    • User DB (SQL)
    • Tweet DB (NoSQL like Cassandra)
    • Timeline DB (NoSQL)
    • Notification DB






Request flows

Post Tweet

pgsql CopyEdit Client → API → Tweet Service → Save to Tweet DB → Timeline Service → Fan-out to followers → Update Timeline DB + Redis → Notify followers

View Home Timeline

vbnet CopyEdit Client → API → Timeline Service → Get from Redis (or fallback to Timeline DB) → Return tweets

Search Tweets

sql CopyEdit Client → API → Search Service → Query ElasticSearch → Return results






Detailed component design

Timeline Service

  • Fan-out on write: Push new tweets to followers’ timelines when posted.
  • Use Redis for hot timelines (recent tweets).
  • Hybrid approach:
    • Regular users → fan-out on write.
    • Celebrities → fan-out on read (aggregate on demand).
  • Scale: horizontally partition by user_id.
  • Background workers process fan-out queues.

📦 2. Tweet Service

  • Stores tweets in Cassandra (high write throughput).
  • Schema:
scss CopyEdit tweet_id (PK), user_id, content, media_url, created_at
  • Media uploaded to S3/GCS, served over CDN.
  • Maintain counters (likes, retweets) in Redis, periodically sync to DB.

🔔 3. Notification Service

  • Asynchronous using Kafka + worker pool.
  • Types: new follower, like, retweet, reply.
  • Notifications stored in NoSQL DB (DynamoDB or MongoDB).
  • Mobile/web push handled via integration with notification providers.






Trade offs/Tech choices

Cassandra: for tweets High write throughput, horizontal scalability

Redis for timelines: Low-latency reads for home timeline

ElasticSearch for search: Full-text search, hashtag & keyword support

Kafka for fan-out: Decouples services, handles spikes in write load

CDN for media: Reduces latency, offloads media traffic from core





Failure scenarios/bottlenecks

Timeline fan-out backlog: Use backpressure + prioritize active users

Redis cache miss or failure: Fallback to Timeline DB

Search index lagging behind Tweet DB: Async update, retry queues

Media storage failure: Multi-region S3 replication, CDN fallback

Celebrity tweets overload fan-out pipeline: Use fan-out on read for high-fanout users





Future improvements

  • Personalized timelines using ML ranking.
  • Spam detection & abuse prevention systems.
  • Federated/multi-region deployments.
  • Better offline/nearline analytics.
  • User privacy controls for followers/tweets.
  • Expand trending topics beyond hashtags.