Loading...
Normally, key and value storage would be used a lot as cache. This means this key and value storage would be accessed by millions of requests per day, if used in a distributive system. The key and value storage should be using RAM memories, so we would want it to be not too big, like 8~ 16gb, or however much user wants, and we should also try to keep the data small, which would be since we are dealing with key and value anyways.
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
If the user were using this in a internal system in a distributive system, redis might be accessed as an sdk, but let's assume API.
We'll have:
POST /key - request body will contain key and value, and potential expiration date.
GET /key/{key} - returns value of the certain key.
PUT /key{key} - request body will contain new value.
DELETE /key/{key} - remove based on the key_id given.
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
The key-value store will be in its own pod, and other containers will reach out to the API server of the Key-value store pod. Then, the API Server, after authenticate the users, and then send the requests to the Queue. We would have some Queue in the key-value store pod to take in all of the requests in order, the creation of the data, and the GET of the data. This will ensure that the data is retrieved / created in order, and not cause any duplicate / give out wrong information back to the clients We will have the Queue in an SQS format, where we'll have 3 different workers picking up the data, and we'll have some expiration date on each of the entries in the Queue. When data expires, it'll be sent to the Timedout Queue, which will also be picked up by the workers for the timeoutService. There are 2 options, to have some type of fallback to try them again, maybe we'll give them 3 retries max. Then, if we run into specific # of retries and the data still didn't go through, we would return back to the users that the request was not completed due to issues like the data already exists, or the data doesn't exist (From GET method), or creation failed. Also by having a queue, we are dealing with idempotent requests. If a user does 2 different create requests, we can either allow the overwrite, or if the data already exists with the same key, we don't allow the overwrite. In our case, I would say don't allow overwrite unless they make a PUT request for it. Then those requests will be picked up by the consumers like dataService (Workers in this case). Now, the database itself should be some type of NoSql database, since the data isn't necessarily consistent, it could be different coming from different nodes, and it's easier to keep track of key-value storage.
The APIServer itself, we'll have some type of verification. Usually, if we are in a kubernetes environment, we are able to set the node labels so that our key-value store pod will only receive requests from certain pod.
Now, If we want to replicate data across multiple nodes, that's may be a
different story. Because then we are dealing in a cluster level. We could
set up different things. In order to deploy our key-value storage to all of
the nodes, we could set up a daemonset in order to automatically deploy it in all of the nodes. In order to replicate the data, we would have some type of a kubernetes controller. This controller would run few things, it would ensure that all of the data in each of the key-store value in each of the node would be equal, and would persist. There are few ways to go about this, There could be difference in time of arrival for the data from each of the node to the controller. we would probably keep them in a Priority Queue based on the time, so that the "oldest" data read would go first. This would ensure that the data doesn't get tainted. In order to keep the data persisted, we would keep the backup of the key-value storage. This would be done in the Control node, we would have a source-of-truth backup, and the data would stay for about a day or a week, depending on how much we can store in the key-value storage.
For the conditional swap, we can have it as a script in our key-value storage to go through the script, and apply it. Of course, the script itself
would have been verified by the authorization webhook for the pod's native api server. (This will require webhook setup from us, verifying the node label / correct user).
We will use NoSQL database. This is due to possible data inconsistencies, since we are receiving data from many different nodes / pods. Besides, it's good for scaling as well, as we just need to scale the database horizontally whenever we would need to. The main entities would just be simply "Data" with key and value storage, as well as the expire (TTL).
If a node fails, Then we will not be able to access the data inside, and we might even lose the data inside. However, we could always have the node communicate with the Control node with the key value storage controller when it comes back up to receive the backup copy of the storage. This may mean we could lose some data, if a user sent a request while the node was being held. However, the requests can be sent again, and if the requests did make it through before the node was down, it would have been sent into the Queue, which would make it to the controller.
If there was a hot key, like there are a lot of requests looking for a specific key, we could potentially keep a cache for extremely frequently searched key so that we can alleviate the amount of requests coming through the queue. For the ones that are frequently searched, we may have another copy of the dataservice or special Service looking for those frequently searched entry, and picking it up before, as long as it's a GET method, and there are no PUT or POST or DELETE changing it.