Loading...
Normally, key and value storage would be used a lot as cache. This means this key and value storage would be accessed by millions of requests per day, if used in a distributive system. The key and value storage should be using RAM memories, so we would want it to be not too big, like 8~ 16gb, or however much user wants, and we should also try to keep the data small, which would be since we are dealing with key and value anyways.
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
If the user were using this in a internal system in a distributive system, redis might be accessed as an sdk, but let's assume API.
We'll have:
POST /key - request body will contain key and value, and potential expiration date.
GET /key/{key} - returns value of the certain key.
PUT /key{key} - request body will contain new value.
DELETE /key/{key} - remove based on the key_id given.
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
The key-value store will be in its own pod, and other containers will reach out to the API server of the Key-value store pod. Then, the API Server, after authenticate the users, and then send the requests to the Queue. We would have some Queue in the key-value store pod to take in all of the requests in order, the creation of the data, and the GET of the data. This will ensure that the data is retrieved / created in order, and not cause any duplicate / give out wrong information back to the clients We will have the Queue in an SQS format, where we'll have 3 different workers picking up the data, and we'll have some expiration date on each of the entries in the Queue. When data expires, it'll be sent to the Timedout Queue, which will also be picked up by the workers for the timeoutService. There are 2 options, to have some type of fallback to try them again, maybe we'll give them 3 retries max. Then, if we run into specific # of retries and the data still didn't go through, we would return back to the users that the request was not completed due to issues like the data already exists, or the data doesn't exist (From GET method), or creation failed. Also by having a queue, we are dealing with idempotent requests. If a user does 2 different create requests, we can either allow the overwrite, or if the data already exists with the same key, we don't allow the overwrite. In our case, I would say don't allow overwrite unless they make a PUT request for it. Then those requests will be picked up by the consumers like dataService (Workers in this case). Now, the database itself should be some type of NoSql database, since the data isn't necessarily consistent, it could be different coming from different nodes, and it's easier to keep track of key-value storage.
The APIServer itself, we'll have some type of verification. Usually, if we are in a kubernetes environment, we are able to set the node labels so that our key-value store pod will only receive requests from certain pod.
Now, If we want to replicate data across multiple nodes, that's may be a
different story. Because then we are dealing in a cluster level. We could
set up different things. In order to deploy our key-value storage to all of
the nodes, we could set up a daemonset in order to automatically deploy it in all of the nodes. In order to replicate the data, we would have some type of a kubernetes controller. This controller would run few things, it would ensure that all of the data in each of the key-store value in each of the node would be equal, and would persist. There are few ways to go about this, There could be difference in time of arrival for the data from each of the node to the controller. we would probably keep them in a Priority Queue based on the time, so that the "oldest" data read would go first. This would ensure that the data doesn't get tainted. In order to keep the data persisted, we would keep the backup of the key-value storage. This would be done in the Control node, we would have a source-of-truth backup, and the data would stay for about a day or a week, depending on how much we can store in the key-value storage.
For the conditional swap, we can have it as a script in our key-value storage to go through the script, and apply it. Of course, the script itself
would have been verified by the authorization webhook for the pod's native api server. (This will require webhook setup from us, verifying the node label / correct user).
When the client asks to create an entry in the key-value store, they would be calling the key-value store pod to create it. When this queue occurs, there's few ways to go about storing this information in the backup. One way is to queue it up both locally, and on the control node side in order to save the new data as the backup. The tradeoff here is that we are adding extra overhead, because we are doing x2 work on creating / deleting. Other way we can do this is by utilizing controller's reconciliation, and backing up the data every x amount of minutes. The tradoff is that we won't be able to back up the data on the real time, but we'll be able to save the copies of database at once instead of doing it every request, causing extra latency.
The replicas will be saved async of course, on the controller side, the nodes don't need to wait for updates to the backup.
The System ensures consistency, because every node has the updated versions of the key-value storage. The key-value storage should always be available, potentially at 99.9% avilable.
The system would detect node failures. kubernetes node controller / manager detects the node failure. We can either utilize this to detect our key-store storage failure. However, there will be a case when the node is working fine, but the key-storage value is failing (readiness wise / liveness.) In this case, the key-storage value controller will attempt to reach out to the nodes' key-value storage each time, checking the liveness and readiness. This way, we will check if it's alive.
If the node is being deleted, we need to do a graceful termination on the key-value storage itself. We can either delete everything on the backup (By sending request to the control plane), or if other nodes may use it, keep the data in the backup. If the data isn't being used in the backup, it'll dissappear anyways due to TTL.
Now, if there are multiple data coming in with same keys from different nodes, We'll need to keep track of where each of these keys are coming from, because from the perspective of the nodes alone, they'll be fine, but on the case of a cluster, they may cause confusion. In this case, What we can do is either have an extra column on the data showing which node it's from by having extra node, so that when one node is asking for an id, then we would return that specific data. What may happen if it happened at different time is, the backup would be notified first of the change, the node checks the backup, and then updates its own data store, but if it happens at the same time, we need to keep track of where each one comes from.
If some replicas are unreachable on a node, because the key-value storage only went down on a specific node, as mentioned earlier, the controller might notice the liveness, and restart the node.
We will use NoSQL database. This is due to possible data inconsistencies, since we are receiving data from many different nodes / pods. Besides, it's good for scaling as well, as we just need to scale the database horizontally whenever we would need to. The main entities would just be simply "Data" with key and value storage, as well as the expire (TTL).
If a node fails, Then we will not be able to access the data inside, and we might even lose the data inside. However, we could always have the node communicate with the Control node with the key value storage controller when it comes back up to receive the backup copy of the storage. This may mean we could lose some data, if a user sent a request while the node was being held. However, the requests can be sent again, and if the requests did make it through before the node was down, it would have been sent into the Queue, which would make it to the controller.
If there was a hot key, like there are a lot of requests looking for a specific key, we could potentially keep a cache for extremely frequently searched key so that we can alleviate the amount of requests coming through the queue. For the ones that are frequently searched, we may have another copy of the dataservice or special Service looking for those frequently searched entry, and picking it up before, as long as it's a GET method, and there are no PUT or POST or DELETE changing it.
If let's say a node comes down, and never recovers, that's ok, the data there won't be lost because the control node will have the backup that contains its data already. Now, if the data in the key-value storage is useless, because it was only meant for this specific node, that's ok, because the data in the backup does contain the TTL, which will expire. Even if not, it'll be replaced with newer data after certain amount of time.
If let's say a new node gets created, The daemonset will automatically create the key-value store on its node, and then the key-value-storage controller will pick up the backup, and put a copy of it in that node to keep it updated.
The key-value store will be in its own pod, and other containers will reach out to the API server of the Key-value store pod. Then, the API Server, after authenticate the users, and then send the requests to the Queue. We would have some Queue in the key-value store pod to take in all of the requests in order, the creation of the data, and the GET of the data. This will ensure that the data is retrieved / created in order, and not cause any duplicate / give out wrong information back to the clients We will have the Queue in an SQS format, where we'll have 3 different workers picking up the data, and we'll have some expiration date on each of the entries in the Queue. When data expires, it'll be sent to the Timedout Queue, which will also be picked up by the workers for the timeoutService. There are 2 options, to have some type of fallback to try them again, maybe we'll give them 3 retries max. Then, if we run into specific # of retries and the data still didn't go through, we would return back to the users that the request was not completed due to issues like the data already exists, or the data doesn't exist (From GET method), or creation failed. Also by having a queue, we are dealing with idempotent requests. If a user does 2 different create requests, we can either allow the overwrite, or if the data already exists with the same key, we don't allow the overwrite. In our case, I would say don't allow overwrite unless they make a PUT request for it. Then those requests will be picked up by the consumers like dataService (Workers in this case). Now, the database itself should be some type of NoSql database, since the data isn't necessarily consistent, it could be different coming from different nodes, and it's easier to keep track of key-value storage.
The APIServer itself, we'll have some type of verification. Usually, if we are in a kubernetes environment, we are able to set the node labels so that our key-value store pod will only receive requests from certain pod.
Now, If we want to replicate data across multiple nodes, that's may be a
different story. Because then we are dealing in a cluster level. We could
set up different things. In order to deploy our key-value storage to all of
the nodes, we could set up a daemonset in order to automatically deploy it in all of the nodes. In order to replicate the data, we would have some type of a kubernetes controller. This controller would run few things, it would ensure that all of the data in each of the key-store value in each of the node would be equal, and would persist. There are few ways to go about this, There could be difference in time of arrival for the data from each of the node to the controller. we would probably keep them in a Priority Queue based on the time, so that the "oldest" data read would go first. This would ensure that the data doesn't get tainted. In order to keep the data persisted, we would keep the backup of the key-value storage. This would be done in the Control node, we would have a source-of-truth backup, and the data would stay for about a day or a week, depending on how much we can store in the key-value storage.
For the conditional swap, we can have it as a script in our key-value storage to go through the script, and apply it. Of course, the script itself
would have been verified by the authorization webhook for the pod's native api server. (This will require webhook setup from us, verifying the node label / correct user).