Detailed component design
1. Feed Service
Purpose
To provide personalized feeds combining precomputed results and real-time updates based on user preferences, interactions, and content relevance.
Detailed Workflow
- Event Triggering:
- User actions (likes, comments, shares, follows) or new post creations generate events.
- These events are pushed into an Apache Kafka event queue.
- Kafka brokers distribute the events to Feed Service consumers for processing.
- Precomputation:
- Periodically, the Feed Service retrieves:
- Posts from users/pages the user follows.
- Interaction metrics (e.g., likes, shares, comments).
- User preferences (e.g., preferred categories, frequently interacted users/pages).
- The Ranking Algorithm assigns scores to each post based on: R=w1⋅Engagement+w2⋅Recency+w3⋅PersonalizationR = w_1 \cdot Engagement + w_2 \cdot Recency + w_3 \cdot PersonalizationR=w1⋅Engagement+w2⋅Recency+w3⋅Personalization
- Posts with the highest scores are stored in MongoDB as a precomputed feed for each user.
- Real-Time Updates:
- New events are processed dynamically and updated in Redis using a
priority_queue structure. - The queue prioritizes posts with higher relevance scores.
- Feed Retrieval:
- When a user requests their feed:
- The system first queries Redis for cached updates.
- If a cache miss occurs, the system fetches precomputed feed data from MongoDB.
- Real-time updates (Redis) and precomputed data (MongoDB) are merged, ensuring relevance and freshness.
Scalability
- MongoDB is sharded by
user_id for distributed storage and retrieval. - Redis reduces latency for frequently accessed feeds.
- Kafka ensures high throughput and scalable event processing.
2. Post Service
Purpose
Manages post creation, storage, retrieval, and associated media handling.
Detailed Workflow
- Post Creation:
- Users submit post content and media via the client app.
- Metadata (e.g., text content, creation time, visibility settings) is sent to the Post Service API.
- Metadata is stored in PostgreSQL, ensuring ACID compliance.
- Media Upload:
- Media files (e.g., images, videos) are split into smaller parts using multipart uploads.
- The client uploads each part directly to AWS S3 using pre-signed URLs, reducing server load.
- After all parts are uploaded, S3 assembles them into a complete file and generates a URL.
- Post Retrieval:
- When a user requests a post:
- Metadata is retrieved from PostgreSQL.
- The media URL is fetched from S3.
- Media delivery is optimized using AWS CloudFront (CDN) for caching.
- Post Deletion:
- Metadata is removed from PostgreSQL.
- Corresponding media files are deleted from S3 using their unique
post_id.
Scalability
- PostgreSQL read replicas handle read-heavy operations efficiently.
- AWS S3 scales horizontally for unlimited media storage.
- CloudFront reduces latency by caching media close to the user.
3. Engagement Service
Purpose
Tracks and processes interactions such as likes, comments, and shares.
Detailed Workflow
- Like Interaction:
- A user likes a post via the client app.
- The like is recorded in PostgreSQL, associating the
post_id with the user_id. - Redis updates the like count, storing it as a
hash_map keyed by post_id for fast retrieval.
- Comment Interaction:
- Comments are submitted through the client app and stored in PostgreSQL, linked to the post and user.
- Comment threads are cached in Redis for frequently accessed posts.
- If a user requests comments, the system queries Redis first, falling back to PostgreSQL if necessary.
- Share Interaction:
- Shares are recorded in PostgreSQL, linking the
post_id and user_id. - The Feed Service is notified via Kafka to update the feeds of the sharer’s followers.
Scalability
- Redis reduces latency for engagement metrics like like counts and comment threads.
- PostgreSQL is partitioned by
post_id to distribute data efficiently.
4. Search Service
Purpose
Enables users to perform full-text and keyword searches for posts, users, and tags.
Detailed Workflow
- Indexing:
- When a new post is created, its content, tags, and associated metadata are sent to Elasticsearch.
- Elasticsearch creates an inverted index, mapping terms to document IDs for efficient search.
- Search Query:
- The client sends a search query through the API Gateway.
- The query is translated into Elasticsearch Query DSL, specifying fields to search (e.g.,
content, tags). - Elasticsearch matches the query terms to its inverted index and ranks results by relevance.
- Response:
- Results are paginated and sorted before being sent back to the client.
Scalability
- Elasticsearch clusters handle horizontal scaling, distributing data across nodes.
- Batched indexing ensures high write efficiency.
5. User Service
Purpose
Manages user profiles, authentication, and preferences.
Detailed Workflow
- User Profile Management:
- User profile data (e.g., name, email, preferences) is stored in PostgreSQL.
- Updates to preferences trigger Kafka events to update dependent services like the Feed Service.
- Authentication:
- Users log in with credentials verified by the User Service.
- On successful login, a JWT (JSON Web Token) is issued, containing claims like
user_id and roles. - The client uses this token for subsequent authenticated requests.
- Preference Retrieval:
- User preferences are fetched during feed generation to tailor content recommendations.
Scalability
- PostgreSQL read replicas handle frequent profile and preference lookups.
- Redis caches session data for fast authentication.
6. Analytics Service
Purpose
Tracks and processes engagement and view data for generating real-time and aggregated insights.
Detailed Workflow
- Real-Time Streaming:
- Engagement events (e.g., likes, views) are streamed to Kafka.
- Kafka partitions these events by
post_id or user_id, enabling parallel processing.
- Aggregation:
- Apache Spark processes Kafka streams, computing metrics like total views, average time spent, and engagement rates.
- Aggregated metrics are stored in DynamoDB for fast retrieval.
- Archiving:
- Raw events are periodically archived in AWS S3 for batch processing or long-term storage.
- Metrics Retrieval:
- Analytics data is fetched from DynamoDB for dashboards or reports.
- Real-time data can be fetched directly from Spark for the latest metrics.
Scalability
- Kafka’s partitioning ensures high-throughput event processing.
- DynamoDB scales horizontally for fast reads/writes.
1. Feed Update Mechanism
How Incoming Posts are Prioritized
- Prioritization Strategy:
- New posts are scored based on a real-time ranking algorithm using factors such as:
- Recency: Weight posts created recently higher.
- Engagement: Prioritize posts with higher interaction (likes, comments, shares).
- Relevance: Align with the user’s preferences (e.g., topics, followed accounts).
- Posts are inserted into a priority queue in Redis, sorted by the calculated score.
Push Notification Strategy
- Real-Time Notifications:
- WebSockets: Maintain persistent connections to push updates instantly to active users.
- FCM/APNs: For mobile apps, use Firebase Cloud Messaging (FCM) or Apple Push Notification Service (APNs) for out-of-app notifications.
- Triggering Notifications:
- Push notifications are triggered via Kafka events when:
- A new post is created by a followed user/page.
- Significant engagement occurs on a user’s post.
Mechanism for Feed Updates:
- New events (e.g., post creation) are pushed into Kafka.
- The Feed Service processes these events to update the priority queue in Redis.
- Active users receive immediate updates via WebSockets or notifications.
- Precomputed feeds in MongoDB are updated periodically to incorporate recent posts.
2. User Preferences Management
Storage and Management
- Storage:
- Preferences are stored in PostgreSQL as a JSONB field for flexibility (e.g.,
{"topics": ["sports", "technology"], "language": "en"}). - Indexed fields in PostgreSQL allow faster querying for specific preference attributes.
- Real-Time Updates:
- Updates to user preferences trigger a Kafka event, notifying dependent services like the Feed Service to regenerate personalized feeds.
Utilization in Feed Personalization
- Integration into Ranking:
- The ranking algorithm assigns higher weights to posts aligned with the user’s preferences (e.g., topics, categories, frequently interacted users/pages).
- Dynamic Adaptation:
- Machine learning models adjust weights over time based on evolving user behavior.
- For example, a user engaging more with “technology” posts will see more tech content.
End-to-End Flow:
- Users update preferences via the client application.
- Updates are saved in PostgreSQL and trigger Kafka events.
- The Feed Service recalculates the user’s feed with the updated preferences.
3. Data Partitioning for Scalability
Sharding Strategies
- User Data:
- Shard by
user_id: Each shard contains data for a set of users. - Rationale: Distributes load evenly across shards for user-specific queries like feed retrieval or engagement metrics.
- Post Data:
- Shard by
post_id or creation date: Ensures high-write workloads (e.g., post creation) are evenly distributed. - Rationale: Supports horizontal scaling and reduces contention on hot partitions.
- Engagement Data:
- Shard by
post_id: Related engagement data (likes, comments, shares) resides together. - Rationale: Enables efficient queries for post-level metrics.
Database Partitioning Tools
- Use MongoDB’s native sharding for user-specific feeds.
- Leverage PostgreSQL table partitioning (e.g., declarative range partitioning by
post_id or timestamp) for relational data.
Advantages:
- Improves query performance by reducing the amount of data scanned.
- Scales horizontally to support millions of users and posts.
4. Caching Strategy Detail
Utilization of Redis
- Cached Data:
- Frequently accessed data, such as:
- User feeds (priority queue with top-ranked posts).
- Engagement metrics (like counts, comment threads).
- Session data for authenticated users.
- Cache Structure:
- Priority Queue: Maintains top-ranked posts for fast feed retrieval.
- Hash Maps: Store like counts or engagement metrics keyed by
post_id.
Cache Invalidation Rules
- Event-Driven Invalidation:
- Cache entries are invalidated or updated when:
- A new post is added by a followed user.
- Engagement metrics (e.g., like counts) change.
- User preferences are updated, requiring feed recalculations.
- Time-Based Invalidation:
- Use a TTL (Time-to-Live) for cache entries to automatically refresh outdated data.
- Lazy Updates:
- For low-priority data, allow cache misses to trigger updates from the database.
Mitigating Stale Data
- Redis updates occur in real-time for high-priority events (e.g., new posts, likes).
- Backfill mechanisms ensure precomputed feeds in MongoDB are periodically synced with Redis.