High-Level Design

fileInfo: fileId, name, author, chunkIds:[], updated, created

chunkInfo: fileId, chunkId, chunkUrl

Client → API Gateway

Client calls:

upload(fileId)

API Gateway responsibilities:
- Route request to available upload servers
- Use least-connections load balancing
- Handle failover when a node goes down

👉 Design intent

Prevent hotspot
Ensure high availability at entry point

2. Raw Object Storage (S3)

Upload server:
- Directly streams raw file → S3 (raw bucket)

👉 Why first write to S3?

Avoid coupling with metadata or processing
Durable storage immediately (no data loss)

3. Metadata Registration (FileInfo Service)

Client sends:

POST upload(fileInfo)

FileInfo Service:
- Stores metadata into DB


fileInfo {
  fileId,
  name,
  author,
  created,
  updated
}

👉 Design intent

Metadata and file storage are decoupled
Enables independent scaling

4. Async Processing via Kafka

System publishes event:

topic: file-uploaded

👉 This triggers downstream processing pipeline

5. Object Processing Pipeline (Async Consumers)

A dedicated Object Handler Service consumes Kafka events:

Pipeline stages:

Raw S3 Object
   ↓
Split into chunks
   ↓
Fraud Detection
   ↓
Compression Service

👉 Why pipeline?

Each stage is independently scalable
Failures are isolated (retry per stage)

6. Chunk Storage

After processing:
- Each chunk uploaded to Chunk S3

👉 Guarantees:

All chunks successfully uploaded
Idempotent writes (important for retries)

7. Chunk Notification (Kafka)

Publish event:

topic: chunk-created

8. Chunk Metadata Service

Chunk Service consumes event
Stores chunk metadata:


chunkInfo {
  fileId,
  chunkId,
  chunkUrl
}

👉 Design intent

Enables parallel download later
Avoids scanning S3 during reads

Client → API Gateway

2. CDN Check

First check:
- CDN cache (edge)

👉 If hit → return immediately

👉 If miss → go backend

3. File Metadata Lookup

Request goes to FileInfo Service
Service fetches:
- fileInfo DB
- chunkInfo DB

4. Return Chunk URLs

Response contains:


[
  {chunkId, chunkUrl},
  ...
]

5. Client Parallel Download

Frontend:
- Downloads chunks in parallel
- Reconstructs file locally

👉 Why client-side merge?

Reduces backend load
Maximizes bandwidth usage

Submissions

Displaying Solutions with Scores Ranging from 1 to 10