System requirements

Functional:

client app can:

1. acquire and hold a lock
2. release a lock

the locking service can:

1. verify client app identity and reject req if needed
2. accept client app lock request and lock a resource
3. while a resource is locked, reject any locking/access req from other client services
4. accept client app release lock req and unlock a resource
5. break deadlock if a client app holds a lock too long

The locking service should know which client services and which shared resources it's serving

Non-Functional:

minimize latency

highly consistent and reliable

Capacity estimation

1 billion of locking/unlocking requests per day yield 23k QPS on avg and ~46k QPS at peak

We need DB to store all client requests and our service events. As it's very fast write in logging style, we prefer NoSQL. Time series DB may be good choice.

API design

lock(service_id, resource_id, auth): lock a resource

Can pass by optional timeout how long the client will need the resource for. Holding the lock for longer than the timeout is considered as deadlock and the lock will be broken by our service

access(service_id, resource_id, auth): access the locked resource. If called without locking the resource, will get an error code

unlock(service_id, resource_id, auth): unlock the resource

Database design

we can use RDB to store client service and shared resource profiles, as these are small, do not change often, and has low traffic of write and moderate traffic of read.

Client:

id: unique id of the client service. Each instance under a service group has its own unique id

name

group_id: a service may have many instances. This is the unique id for the service group

created_at: timestamp

last_updated_at: timestamp

resource_group_id: permitted resource group ids

resource_id: permitted resource unique ids, if more granular permission is needed

Resource: similar setup as client

id: unique id of resource instance

name

group_id

created_at

last_updated_at

client_group_id

client_id

We store events in time series DB, as they require very fast write, usually append only, recent ones are more critical, and large scale.

Event:

id: unique event id

type: e.g. lock, unlock, etc.

client_id

client_group_id

resource_id

resource_group_id

created_at

status: e.g. granted, denied, timed out

duration

last_updated_at

reason: for denied, we may want to know reason

High-level design

see high level diagram

Request flows

Lock (accept):

1. client requests lock of a resource
2. server verifies identity
3. server locks the resource and responds to client

Lock (deny):

1. client requests lock of a resource
2. server verifies identity. If failed, server denies req
3. server sees the resource is locked for another service, and denies req

Access: similar as lock requests

Unlock: similar as lock requests

Detailed component design

We'd better know all client service identity, and all shared resource identity, so we can reject unwanted or unexpected attempts to access/lock.

We can use different servers to handle different clients and resources, especially independent resources can be handled separately from others. For each cluster of servers, we can further distribute traffic based on client service unique ids. We'll need some sticky session algorithm so that each client's usage of resources can be tracked deterministically.

If some resources are files or stored in DBs, we may cache them for faster access.

For DB, events on independent resources can be put in different shards from others.

On fairness, we can start with FIFO and set a global timeout (more relaxed). If later we identify different priority of client services, we will consider priority.

Trade offs/Tech choices

For servers, two options, leader-follower pattern or leaderless pattern:

1. leader-follower:
2. Pros: easy more managing client and resource statuses that the cluster is responsible for.
3. cons: leader may be stressed
4. leaderless:
5. Pros: more evenly and thus less stress on each node in the cluster
6. cons: hard to resolve conflicts/race condition while multiple clients want to access the same resource

We can start from leader-follower, and if the traffic becomes too high that it's not able to handle, then we consider leaderless

Failure scenarios/bottlenecks

When a server fails, make sure another server in the same cluster kicks in. They should continuously carry the status of current locked resources.

Future improvements

1. We may need some monitoring service to monitor health of servers and alert when necessary
2. In addition to FIFO on fairness, if we introduced client service priority, we should take it into account. We may add a buffer or queue at server level, and batch process requests. However, this may introduce more latency as client requests will not be fulfilled immediately.