Requirements

Functional Requirements:

1. user can CRUD on file- with separte end point
2. user can get file
3. user can share file,
4. others can access file simultaneously based on permission control and link
5. need to handle metadata service
6. should handle versions and be consistent

Non-Functional Requirements:

1. scalable to global level
2. avaialble 99.99
3. strong conistent on metadata. eventual of node level
4. low latency

API Design

POST v1/files/{"filename"="abx.txt", "content"=, "user_id"=123}

response file created:200 Ok

GET v1/files/{fileid=1,user_id=123}

reponse {file_name="abc.txt","content"=4 blob/chunks}

PUT v1/files/{file_id=1, content=, user_id=123}

response file updated:200 OK

DELETE v1/files{file_id=1, user_id=123}

response file deleted: 200 OK

I m using user_id to check the ACL for the file

High-Level Design

Since we have read and write feature

write logic

when user want to write

request write by file_id, content, user details.and these request go to loadbalncer->metadataservice->blockid_manager_server->cache->storage

if more than one user writing on file:

need to handle consitency

then provide exclusive  lock to user actively writing

release lock when user stop writing and mentian versioning

when user req read

req goes to loadbalcner-> find the file_id in metadataserver->return intial some 4 sequential block

then next sequential blocks will sorting meanwhile to provide contnious data

Detailed Component Design

100K QPS to the loadbalancer, Loadbalancer distrubte the request among the metadata service in roundrobin fashion, mostly less active server.

the metadata server is curical compenet in this desing ::

the req can be distribute among the server using consitent hashing

it manage to break the large file into chunks of block ->64KB

and maps file_id to block_id, block_id->datanodes

handle the file ownerships , permission,r eplication

THis suppose to strong consistent to handle multiple read/write

to achive we use RAFT, And to avoid the single point of failure , we can elect zookeeper.

inorder to provide hotfile-> enable low latency-> file blocks are store in cache.

database model looks

file_id, block_id,other details of user,

replicate to handle eventual consitenecy