1. Requirements

Functional Requirements

• Users can:
  • Create paste (text content)
  • Get unique URL for paste
  • Retrieve paste via URL
• Optional:
  • Expiration time (TTL)
  • Public/private pastes
  • Edit/delete pastes

Non-Functional Requirements

• Read-heavy system
• Latency: < 100ms for retrieval
• Scalability: Billions of pastes
• Availability: High (reads should not fail)
• Durability: Pastes must not be lost

2. Estimations

Traffic

• Writes: ~10K/sec
• Reads: ~500K–1M/sec

Storage

Assume:

• Avg paste size = 5KB
• 1B pastes → ~5TB

👉 Requires distributed storage

Read/Write Pattern

• Read-heavy → heavy caching needed

3. API Design

3.1 Create Paste

Request:

Response:

3.2 Get Paste

3.3 Delete Paste

4. Data Storage & Design

4.1 Metadata Store

Use:

• Cassandra / DynamoDB

Schema:

4.2 Content Storage (Important Separation)

Use:

• Object storage (e.g., Amazon S3)

Key:

👉 Reason:

• Large content not suitable for DB
• Cheap, scalable storage

4.3 Cache

Use:

• Redis

Key:

5. High-Level Architecture (HLD)

System consists of:

• Entry Layer
• Read Path
• Write Path

5.1 Entry Layer

• CDN + Load Balancer

Responsibilities:

• Handle read spikes
• Serve cached pastes
• Protect backend

5.2 Read Path

1. Client requests paste
2. CDN:
   • Hit → return content
   • Miss → forward
3. Load Balancer → App Server
4. App Server:
   • Check Redis
   • If miss → fetch metadata + content
5. Return paste

5.3 Write Path

1. Client submits paste
2. App server:
   • Generate paste_id
   • Store metadata in DB
   • Store content in object storage
3. Cache metadata/content

6. Detailed Breakdown

6.1 Paste ID Generation

Approach: Distributed ID + Base62

Use Snowflake-style:

Convert to Base62 → short ID

6.2 Read Optimization

• CDN caches popular pastes
• Redis caches hot data
• Most reads avoid DB

6.3 Write Optimization

• Async writes to object storage
• Metadata stored separately

6.4 Stateless Scaling

• App servers stateless
• Scale horizontally

6.5 Hot Key Handling

Problem:

• Viral paste → heavy reads

Solution:

• CDN absorbs traffic
• Redis replication
• Local cache on app servers

6.6 Idempotent Paste Creation

Problem:

• Client retries (network failure, timeout)
• Same paste may be created multiple times → duplicates

Solution: Idempotency Key

Approach:

Client sends:

Flow:

1. On paste creation request:
   • Check Redis:

1. If exists:
   • Return existing paste URL (no duplicate)
2. If not:
   • Create paste
   • Store mapping:

TTL:

• Store key for ~24 hours

Benefits:

• Prevents duplicate pastes
• Safe retries
• Ensures write correctness

7. Additional Considerations

7.1 Large Paste Handling

• Store large pastes in chunks (optional)
• Stream content instead of loading fully

7.2 Expiration & TTL

• Store expiry in DB
• Object storage lifecycle rules auto-delete

7.3 Cache Eviction

• Redis uses LRU/LFU

7.4 Degraded Mode

Redis Down:

• Fetch from DB + object storage

DB Down:

• Serve cached pastes

Object Storage Down:

• Serve cached data if available

7.5 Security

• Rate limiting (prevent spam)
• Content moderation
• Private pastes require auth

7.6 Expiry & Automatic Cleanup

1. Expiry at Creation

• Each paste has optional expiry_timestamp

2. Object Storage Lifecycle Rules

Using Amazon S3:

• Configure lifecycle policy:
  • Auto-delete objects after expiry

3. Metadata Cleanup (DB)

Two approaches:

Option A: TTL-Based Expiry (Preferred)

• Use DB TTL (e.g., Cassandra TTL)
• Records auto-expire without manual cleanup

Option B: Background Cleanup Job

• Periodic worker scans expired records
• Deletes:
  • Metadata from DB
  • Content from storage

4. Lazy Deletion (Safety Layer)

• On read:
  • If expired → return 404
  • Trigger async cleanup

5. Cache Cleanup

• Redis uses TTL for automatic eviction

👉 Ensures:

• No storage bloat
• Fully automated cleanup

7.7 Rate Limiting & Abuse Prevention

Problem:

• Malicious users can:
  • Spam paste creation
  • Exhaust storage/resources

Solution: Multi-Layer Rate Limiting

1. API Gateway Rate Limiting

Use:

• Token Bucket / Sliding Window

Apply limits:

2. Distributed Rate Limiter

Use:

• Redis

Key:

3. Adaptive Limits

• New users → stricter limits
• Trusted users → relaxed limits

4. Burst Handling

• Token bucket allows short bursts
• Prevents sudden abuse spikes

5. Abuse Detection

• Detect:
  • High frequency paste creation
  • Duplicate content spam

6. Response

• Reject with:

👉 Protects system from:

• Spam
• Resource exhaustion

🏁 Final Summary

• Separation of metadata & content improves scalability
• CDN + Redis caching ensures low latency
• Object storage handles large data efficiently
• Stateless services ensure horizontal scaling
• TTL + lifecycle policies prevent storage bloat