System requirements

Functional:

• User registration
• File upload and download
• Folder management
• Folder Metadata and Information

Non-Functional:

• Security and privacy: User needs to be authenticated and authorized to make actions
• Latency: Viewing file download should be low latency
• Scalability: The system should accommodate an increasing number of users and data storage
• Reliability: The service should have uptime 99.99%
• Data Durability: The files are not lost

Capacity estimation

We assume 50k DAU. Assume each user has 100 GB storage, the storage requirement initially is 5PB. The Following storage requirement could be more than 20PB.

API design

• PUT /upload
  • Request{“file”, "token", "directory", "size"}
  • Response{"status", "message"}
• GET /download
  • Request{"file", "token", "directory", "size"}
  • Response{“status”, "url"}
• GET /view?directory
  • Response{["file_name", "modified_time", "size", "type"]}
• DELETE /delete
  • Request{"file_name", "token", "directory"}
  • Response{“status”}

Database design

• User
  • int user_id (primary key)
  • string account
  • string salt
  • string hashed password
  • datetime created_at
• File
  • int file_id (primary key)
  • string directory
  • string file_name
  • int size
  • string type
  • string owner
  • datetime last_update
• FileShare
  • int share_id(primary key)
  • string share_user
  • int file_id(foreign key)

High-level design

The key components are:

• API Gateway: Handles user authentication and authorization
• User authentication service: validates the login request and user account management service
• file service: handles file viewing, file metadata retrieval, directory listing
• CDN/file storage: it depends on the cost measurement to decide to use a 3rd party CDN or build up self maintained storage.

Request flows

• User download file
  • User send login request
  • API gateway forward request to authentication service
  • authentication service query DB to verify login
  • user login success
  • user view file list
  • API gateway forward request to file service
  • file service lookup file list from DB and return to user
  • user select file to send download request
  • API gateway forward request to file service
  • file service forward request to storage service to download
• User upload file
  • user send login request
  • API gateway forward request to authentication service
  • authentication service query DB to verify login
  • user login success
  • user send file upload request
  • API gateway forward request to file service
  • file service receives the uploaded file
• User share a file
  • user send login request
  • API gateway forward request to authentication service
  • authentication service query DB to verify login
  • user login success
  • user send file share request
  • file service generates the share link, send it to the user

Detailed component design

File upload flow: The file upload can be done through a socket protocol. To prevent file from missing when file service goes down, the uploaded file can be first store in the in-memory DB like Redis. Then the file can be written to persistent storage asynchronously.

File download could also be done through the socket protocol

Trade offs/Tech choices

Tech choices between 3rd party CDN and self-maintained file storage: it depends on cost: CDN is usually measured with data transmission; while self storage is more expensive with storage. So we can store the frequent visit files in self-maintained storage and store the large files in CDN.

Failure scenarios/bottlenecks

File storage can have failures, so to prevent the file loss from failure, the file should be replicated.

File upload or download might be corrupted, so we need a checkSum mechanism to verify the data integration. During upload and download, if the action failed in the middle, we should fail the entire action to let user to retry.

Future improvements

We can improve the system to support upload/download pause in the middle. We can split the file into chunks and mark success for each chunk.