Design a Image Processing for Redfin

Question

Design a geo-distributed image processing system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Image Upload:** Users can upload images of properties for processing.
2. **Image Processing:** The system should resize, compress, and enhance images based on predefined parameters.
3. **Geo-Tagging:** Images must be tagged with the geographical location of the property.
4. **Image Retrieval:** Users should be able to retrieve processed images via a RESTful API.
5. **Rate Limiting:** The system must enforce limits on the number of requests per user to ensure fair usage.

### Non-Functional Requirements
1. **Scalability:** The system should handle millions of requests per day, scaling horizontally.
2. **Availability:** The system should have 99.99% uptime, ensuring that image processing is always available.
3. **Latency:** Image processing should complete within 2 seconds for 95% of requests.
4. **Consistency:** Images should be consistently tagged and available across all regions.
5. **Data Durability:** Images must be stored in a durable manner to prevent loss.

Assuming Redfin handles 1 million property listings per month, with each listing having an average of 5 images:
- **Total Images Per Month:** 1,000,000 listings * 5 images = 5,000,000 images.
- **Daily Requests:** 5,000,000 images / 30 days = ~166,667 images processed per day.
- **Peak Load Assumption:** During peak hours, assume 20% of daily traffic occurs, leading to 33,333 requests/hour or ~9.2 requests/second.
- **Storage Needs:** If each image is approximately 200 KB after processing, total storage required is 5,000,000 images * 200 KB = ~1 TB.
- **Processing Units:** To handle peak load, a minimum of 10 processing nodes may be required, assuming each can handle 10 requests/second.

The architecture consists of the following components:
1. **API Gateway (Amazon API Gateway):** Handles incoming image upload requests, applies rate limiting, and forwards them to processing services.
2. **Image Processing Service (AWS Lambda):** Serverless function that processes images (resizing, compressing, geo-tagging) before storing them.
3. **Image Storage (Amazon S3):** Stores processed images with durability and scalability.
4. **Metadata Database (DynamoDB):** Stores metadata about each image, including properties' geographic tags and processing status.
5. **Load Balancer (AWS ELB):** Distributes incoming requests to multiple instances of image processing services.
6. **Monitoring (AWS CloudWatch):** Monitors the health of services, request rates, and processing times.
7. **Logging (Elasticsearch + Kibana):** Aggregates logs for debugging and operational insights.

![High-Level Architecture](https://example.com/image-processing-architecture-diagram.png)

The metadata database schema can be designed as follows:
- **Table: Images**
  - **ImageID (PK):** Unique identifier for the image.
  - **PropertyID:** Identifier for the associated property.
  - **UserID:** Identifier for the user who uploaded the image.
  - **Location:** Geo-tagging information (latitude, longitude).
  - **StorageURL:** URL of the stored image in S3.
  - **ProcessingStatus:** Status (e.g., "Processed", "In Progress").
  - **Timestamp:** When the image was uploaded.

This design allows us to quickly query images by property or user, and efficiently update processing statuses.

1. **Consistency vs. Availability (CAP Theorem):** Given the need for high availability, we may choose eventual consistency for image processing status updates, allowing users to retrieve images even if metadata is not fully updated.
2. **Cost vs. Performance:** Using AWS Lambda for processing is cost-effective for variable workloads but may introduce latency; a dedicated processing cluster could reduce latency at a higher cost.
3. **Data Partitioning:** Using DynamoDB’s partitioning strategy allows for efficient scaling but requires careful design of access patterns to minimize hot partitions.
4. **Monitoring Complexity:** While implementing comprehensive monitoring is crucial, it adds complexity and requires management of metrics and alerts, which could increase operational overhead.

Design a Image Processing for Redfin

Redfin

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Redfin Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search