Loading...
For a system like Google Drive or dropbox, There would be 100M users. Now if we consider 80:20 rule then 20M users would be active on daily basis.
Total users - 100M
Daily active users - 20M
About 20% would use upload feature so
Daily Uploads by - 4M users
Daily Download/Read by - 16M users
If each daily active user upload atleast one file-
On an average 1 file would be around 1MB and 1 user would be holding around 1000 files.
Network -
Daily Upload Request - 4M files
Upload QPS = 40 files/Sec
Download/Read QPS = 160 files/Sec
Since we want design a system with high availability, come up with number of cores that would be able to achieve ~200 requests/sec
For eg. - if 1 core can handle 10 requests then
= (200/10)*(25/100) cores would be required. 5 cores would be required. In order to achieve even distribution and high availability you would need atleast 3 servers having 5 cores each cluster to handle
Storage -
Total storage needed - 1GB/User i.e. 100M * 1GB = 100MGB = 100 PB
Consider 1% traffic would be growing every month, you would need to add 12PB capacity every year. and in order to achieve reliability, you would replicate this data to cross regions.
so at the end of the 2 year you would need 254PB of capacity for a current year.
at the end of 2 years -
10% more upload traffic i.e. 4.5M daily uploads and 20% more download traffic i.e. ~20M download/read requests.
With that, you would need 1 extra core at each server level.
Metadata Capture-
100M users * 1000 files
1 KB metadata/file
100M * 1000 KB = 100 GB
2 years projections = 20% = 120GB File metadata
120 GB User data
1 Posgres should be enough to handle 240GB data.
List files and folders on home page
Get /api/v1/content
Request - user id
Response - {Folder:[], Files[]}
List files and folder under a specific folder
Get /api/v1/content
Request - user id, folder id
Response - {Folder:[], Files[]}
Create Folder
PUT /api/v1/folder
Request - user id, parent_folder_id
Response - new_folder_id
Upload File
POST /api/v1/file
Request - user id, folder id
Response - url to show status
Status 201 if successful, 202 if upload in progress
Download File
GET /api/v1/file
Request - user id, file id
Response - url to show status
Status 201 if successful, 202 if download in progress
Change permission of file
POST /api/v1/permission?fileid={}
Request - {permission : [read/write/view]}
Share file
GET /api/v1/link?fileid={}
Request- user id
Response - url
Client web and mobile would be holding the local app to record local changes.
Write service would drop a message to kafka queue to sync with client.
Postgres would manage file metadata, user data and ACLs. File table would hold the record S3 object url for the respective file.
Upload flow - Client -> API Gatway -> Metadata service and write service -> Metadata to postgres and write service to S3
Download Flow - Client -> CDN
Postgres - to capture user details and files/folders metadata
S3 - To store data as objects
Postgres tables - users, files, folders, user_file
If user is offline and not connected to the network then local client would connect to sync service to first bring new changes to/from server.
System reliability - Read service would be running with another 2 more instances that would help during fault tolerance.
S3 would be running with another replica on another region. This would be async replication to maintain durability.
We would set replication time control in S3 cross region replication. Minimum is 15 mins so that would be RPO as well.
Data integrity - Client would hash each chunk using MD5 or SHA256 and send it in header with each chunk. Write service would calculate and match against the hash present in header. it it does not match then it would reject the chunk with 400.
With very first chunk, the client would send the metadata of file, no. of chunks and hash of complete file. The write service persist this data in Postgres and uses it to verify if complete file has been received or not.
To handle network partition and retries, a unique id is assigned to each file and index to each chunk that helps to maintain non duplicate chunks and retry only failed ones.
There would be background job that would take care of stale chunks and unfinished downloads and clear them.
Security - TLS would be maintained at client and server level to ensure data is encrypted at transition.
Offline updates - to manage offline updates, sync service would check the client and server state and begin a process to sync files among them.
data sync service would record the captured data in postgres as well as connect to s3 to ensure entire data is available.
CDN is added to download data from nearest location.
Cache is added at top of postgres to manage traffic.