Requirements

Functional Requirements:

List the key functional requirements for the system (Ask the AI for hints if stuck)...

Clients should upload and download the files

Clients should be notified upon file uploads

Clients should be able to list the uploaded files.

When clients is back online it's updates should be pushed to server.

Per user access control should be supported.

Encryption in transition and at rest.

Not in the scope:

• file sharing, multiuser update

Non-Functional Requirements:

List the key non-functional requirements (performance, scalability, reliability, etc.)...

Durability: 11 nines

Availability: 4 nines

Consistency: strong, ACID support

Capacity Estimation

Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...

DAU: 10 mln

Read/write ratio 20:1

File size distribution:

20% - files from 0 to 1 mb

60% - files from 1 to 100 mb

20% files from 100 mb up to 10 GB (max allowed file size)

In order to calculate the total data volume of upload we need to get media of each file size range

1 mln uploads daily: 200k*500KB + 200k*5GB+50mb*600k = 100GB + 1PB + 30 TB => 1.1 PB daily

5 year data size projection approx 2 Exabyte

Read requests:

requests daily = 20* 1000 000 = 20 mln

read QPS = 20 000 000 / 100 000 = 200

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

/api/v1/files POST -> performs the actual upload process:

• 200/201 on upload success and returns file UUID
• 403 on lack of permissions

/api/v1/files/ -> returns metadata of the uploaded file with verisions

/api/v1/files//download/ -> streams file content to the client.

POST /api/v1/files/init -> {file_id, upload_session_id} (row created visible=false)

POST /api/v1/files/{file_id}/complete -> verifies all block hashes, then flips visible=true

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

Metadata DB: SOT of states of all the files stored in the block storage

Block service: responsible for upload/download of actual files and putting them to object storage.

Metadata service: responsible for registering upload in metadata DB and selecting primary block storage server upon upload.

Object storage: responsible for storing the files.

Notification service: responsible for communicating with the clients upon upcoming file updates to sync local and remote versions.

Load balancer: placement, block, notification services will be behind LB. Load balancer will be used for authentication as well. LB will perform per user rate limiting, and respond with 429 when too many requests happens.

Message queue (MQ): will be used for even streaming of file upload to trigger notifications and ensure durability of notifications.

Chunk validation:

• client is responsible for splitting the files into blocks and uploading it to block server. The client will specify the hash of each block in the etag header. Block server will generate hashsum of each block locally and compare it wil Etag value upon upload. If they are the same, the Block server will commit the block upload in the Metadata DB.

Database Design

Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...

SQL for Metadata DB

Block Table:

• block_uuid
• created_at
• updated_at
• block_hash
• block_size
• block_url (in the object storage)

File Table:

• file_id
• created_at
• updated_at
• version
• file_hash
• owner_id(user_id)
• visible(boolean)

User management is out of scope here.

No folder support

Transactions:

File record is created in the beginning of ifle upload with visible = False. keep visible = False through all blocks, and flip it once, in a single transaction with the final block insert, only after every block hash verifies.

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

1. Metadata DB: single DB instance won't be able to handle all the file metadata uploads as well as storage. Having each block is 4 MB at max and average file being 1.1GB we we would need to allocate 275 chunks for each new block and it gives us 275 mln of new records daily. In order to handle such load and storage we would need to shard metadata DB by user ID. Consistent hashing will be used to prevent overloading.
2. Upon file update we don't need to necessary store all the blocks completely again, we can compare the checksums of existing blocks and incoming blocks and identify the diff and store the newly added blocks in the object storage.
3. Notifications: upon uploading the file, block service that is responsible for making the file visible should send the mesage to MQ. The notification service should process each message and send notification to each client.

Orphan, incomplete session cleanup -> bckground worker that will do the GC.

Bit rot -> bckground scrub shouldb e handled by object storage provider.