Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
1. The wallet should have a ledger which tracks all transactions.
2. The wallet should support reproducability to verify transactions were successful and assist auditing purposes.
3. A wallet should be able to transfer funds from one wallet to another
List non-functional requirements for the system...
1. Reliability. The system should be reliable for end users.
2. Scailability. The system should be designed in a way to grow and deal with large amounts of traffic.
3. Performance. The system should be performant for end users.
Estimate the scale of the system you are going to design...
Back of the envelope calculations:
TPS
1. 100M Users
2. 20M DAU
3. 10 transactions per user per day
4. Total transactions per day is 200M
5. TPS = 200 * 10^6 / 10^5 = 200 * 10^1 = 2000 * 2 (each transaction is actually two operations, deducting and adding to a bank account)
6. We need to support 4000 TPS ( if we say one server can handle 1000, we need at least 12 servers when counting replication).
Define what APIs are expected from the system...
https://wallet/balance_transfer
(from_account, to_account, user_id, amount, currency, transaction_id)
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
wallet_table
account_id: string PK
balance: string (to handle doubles)
timestamp: string
index: account_id
transaction_table:
transaction_id: string PK
from_account_id: string FK
to_account_id: string FK
index: transaction_id
currency: string
timestamp: string
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
1. The user initiates a bank transfer via the https://wallet/balance_transfer API. The request goes through the Load balancer, which distributes the request evenly among API Gateways.
2. The API Gateway authenticates the request and does various security checks such as checking active control list and rate limiting before forwarding the request to the digital wallet.
3. The digital wallet splits the transaction into two possible phases. The deduction from account 1 and the addition to account 2.
4. First the digital wallet forwards the transaction request of deducting an amount from account 1 to the transaction phase DB.
5. Then the digital wallet forwards the deduction from account 1 to the coordinator, who in turn forwards the request to the proper DB. The DB is sharded by account id.
6. Once the deduction is complete from account 1 a record of the deduction is forwarded to the transaction phase DB. After that the digital wallet is notified the transaction was successful.
7. The digital wallet forwards the transaction request of adding an amount to account 2 to the transaction phase DB.
6. Once the addition is complete to account 2 a record of the addition is forwarded to the transaction phase DB. After that the digital wallet is notified the transaction was successful.
7. The digital wallet records an account in the transaction phase DB that the request was beeen successful and returns a message of success to the user.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
1. The digital wallet performs distributed transactions.
2. The wallet utilizes the Saga for distributed transactions rather than 2 Phase commit. 2 Phase commit offers strong consistency but has many performance issues when companies scale since all nodes must wait until both phases of comitting is complete. The Saga pattern allows us to split the transaction into two phases. When we deduct the balance from account one we briefly hold a database record lock while this is performed, and then we release the lock. The next phase is adding the balance to the second account. Both transactions are independent. If an issue is detected in either phase a rollback is performed on the datbase to restore the system to its initial state. This pattern follows eventual consistency since theres a brief moment in time where our states could be inconsistent, however we accept this since the performance is good and it scales well.
Transaction DB
1. We use Single Leader Replication. We do this to avoid the burden of write conflicts which can happen more often in multi leader and leaderless replication architectures. This is because when there is more than one write database you could run into concurrency issues where two database try to write at the same time.
2. To provide high availability we need to replicate the data across multiple nodes. In order to achieve this we use a Consensus type of approach.
3. In order to achieve consensus we use the "Raft" algorithm. The "Raft" algorithm requires more than half of the databases to agree to be able to replicate the data. The coordinator is responsible for sending these messages and once the "consensus" is achieved and agreed apon the write operation is written to write ahead log and the coordinator dispatches the write command to all nodes.
Explain any trade offs you have made and why you made certain tech choices...
Single Leader Replication
1. Since we follow single leader replication our writes are slower than other models such as multi leader replication and leaderless replication, because their is only one write database. However, we accept this trade off since this is a financial app and we want to avoid write conflicts if possible because losing any type of write operation is not acceptable. If we used multi leader or leaderless replication we would have to deal with write conflicts and version vectors which are not easy to deal with...it adds complexity to the system.
Saga Distributed Transaction
1. We opt to use this type of distributed transaction over two phase commit because in two phase commit there are some serious issues when you start scailing. Our system has 12 servers and is growing. In Two phase commit we have to lock down both bank account records, and perform both phases before we complete the transaction. Many things can go wrong here such as network connectivity issues and that could lead to database locks being stuck which adds other complexities. Also two phase commit is slower since we must wait for shared nodes with the same replicated data to agree to perform the write and then to actually perform the write operation.
2. The Saga distributed transaction algorithm offers eventual consistency over 2 phase commit's strong consistency, but because it has a rollback feature, has performance improvements (all nodes don't have to agree), and scales well, we choose it over 2 phase commit.
Try to discuss as many failure scenarios/bottlenecks as possible.
Single Leader Replication
1. Single leader replication will create an issue at some point if we keep growing. It's not a feasible architecture for a very large company because having only one write database does not make sense if the company is especially global.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
Single Leader Replication
1. We can switch to multi leader replication and instead provide a single leader per a region to help avoid write conflicts which is one of the biggest issues with multi leader replication over single leader replication.