Capacity estimation

Estimate the scale of the system you are going to design...

Pastes per Day: Let's assume we anticipate around 1 million new pastes per day.
Average Paste Size: We'll assume an average paste size of 10 KB.
Total Daily Data Ingestion:
- ( 1 \text{ million pastes/day} \times 10 \text{ KB each} = 10 \text{ GB/day} ).
Writes per Second:
- With 1 million pastes per day:
  - ( \frac{1,000,000}{24 \times 60 \times 60} \approx 12 \text{ writes per second} ) on average.
- During peak loads, this could spike to ten times the average, resulting in hundreds of writes per second.

Read to Write Ratio: Assuming a read-heavy workload with a read-to-write ratio of 5:1 or even as high as 10:1.
Reads per Day:
- For a 5:1 ratio:
  - ( 5 \text{ million pastes per day} = 50 \text{ million read requests per day} ).
- Reads per Second:
  - Average: ( \frac{50,000,000}{24 \times 60 \times 60} \approx 578 \text{ reads per second} ).
- This can burst significantly on viral pastes, potentially requiring thousands of reads per second at peak times.

Daily Storage: At 10 GB of new data per day, storage over time becomes significant.
Monthly Storage: Approximately ( 300 \text{ GB/month} ).
Yearly Storage: Approximately ( 3.6 \text{ TB/year} ).
Retention Policy: If pastes expire or are deleted after a certain period, such as 3 months, average storage can be around 900 GB.

ID Size: Using a 6-character alphanumeric ID provides a vast space of ( 62^6 \approx 56 \text{ billion possibilities} ).
With 1 million new pastes every day, it would take years to exhaust this key space, keeping collision probability low.

Scalability: The design should allow horizontal scaling to handle the peak loads efficiently.
Data Durability: Critical to ensure no data loss occurs, particularly for popular pastes.
Caching Strategy: This is crucial for handling the read-heavy nature of the workload, with frequently accessed pastes cached to reduce database load.