Design a Content Moderation System

Question

Design an automated content moderation system for a social platform. Cover ML-based classification pipelines, human review queues, appeal workflows, real-time vs batch processing trade-offs, and handling false positives at scale.

Codemia · Accepted Answer

### Functional Requirements
1. **Content Classification**: Automatically classify user-generated content (text, images, videos) into categories (e.g., hate speech, nudity, spam).
2. **Human Review Queue**: Provide a platform for human moderators to review flagged content and make decisions.
3. **Appeal Workflow**: Allow users to appeal moderation decisions, triggering a secondary review process.

### Non-Functional Requirements
1. **Scalability**: Must handle millions of content items daily, with the ability to scale as user base grows.
2. **Latency**: Real-time moderation decisions should be made within 1-2 seconds for user experience.
3. **Accuracy**: Aim for a classification accuracy of at least 95% to minimize false positives/negatives.
4. **Reliability**: The system should have at least 99.9% uptime to ensure consistent service.

Assuming Meta's platform has around 2 billion active users, we can estimate content submissions:
- **Daily Content Posts**: 1 post/user/day = 2 billion posts.
- **Moderation Load**: Assuming 30% of content is flagged for review, we have 600 million items for moderation daily.
- **Processing Rate**: To handle this, if we want to process all flagged content in 24 hours, we need to process approximately 7,000 items per second.
- **Batch Processing**: If using batch processing, we could group content into hourly batches, resulting in around 25 million items per batch, processed every hour.

### Component Diagram
1. **User Interface**: Web/Mobile app for users to post content and view moderation results.
2. **Content Ingestion Service**: Ingests user content, stores it temporarily in a **Message Queue** (e.g., Kafka).
3. **ML Classification Pipeline**: Uses a pre-trained model (e.g., PyTorch or TensorFlow) to classify content in real-time. This could leverage AWS SageMaker for scalable model hosting.
4. **Human Review Interface**: A web interface for moderators to review flagged content, powered by a microservice (e.g., Spring Boot).
5. **Database**: A NoSQL database (e.g., MongoDB) to store content, moderation decisions, and appeal statuses.
6. **Monitoring and Logging**: Use tools like Prometheus and ELK stack for monitoring system performance and logging errors.

### Schema Design
1. **Content Collection**:
   - `content_id`: String (unique identifier)
   - `user_id`: String (identifier for the user)
   - `content_type`: String (text, image, video)
   - `content_data`: String (URL or text data)
   - `classification_result`: String (e.g., safe, flagged)
   - `created_at`: Timestamp

2. **Moderation Decisions**:
   - `decision_id`: String (unique identifier)
   - `content_id`: String (linked to content)
   - `moderator_id`: String (who reviewed it)
   - `decision`: String (approved, rejected)
   - `review_at`: Timestamp

3. **Appeals**:
   - `appeal_id`: String (unique identifier)
   - `content_id`: String (linked to content)
   - `status`: String (pending, resolved)
   - `resolved_at`: Timestamp

This schema supports quick access patterns for classification results, review histories, and appeals.

1. **Real-time vs Batch Processing**: Choosing real-time processing (using Kafka) allows immediate feedback to users but requires more resources. Batch processing can reduce costs but may delay moderation results.
2. **False Positives Management**: High accuracy reduces false positives but may lead to more false negatives. Implementing a secondary ML model to review flagged content can help balance this but adds complexity.
3. **ML Model Complexity**: A more complex model may improve accuracy but increases latency. A simpler model can be faster but might not catch nuanced cases, necessitating regular evaluation and retraining.

Design a Content Moderation System

Meta

How to Approach This

Sharpen Your Skills on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Meta Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search