Loading...
Supporting multiple currencies c
We estimate to have around 1000000 Transactions per second
With a server handling around 1000 transactions per second it means we would need 1000000 / 1000 = 1000 servers this moves us to a distributed system architecture
post /user
get /user&user_id
post /login
get /balance&user_id
get /history&user_id
post /transfer&user_id=&amount=
post /limit&user_id
post /notification¬ification_type
get /currency_rate
post /convert_currency¤cy_id
User <|-- Account*
User : +int UserId FK INDEX
User : +String UserName
User : +String Email INDEX
User : +String Password
User : +DateTime createdAt
Account <|-- Currency
Account : +int UserId FK
Account : +int AccountId PK
Account : +String Amount
Account : +DateTime createdAt
Account : +int CurrencyCode
Currency <|-- CurrencyExchangeRate*
Currency : +int CurrencyId PK FK
Currency : +String Name
CurrencyExchangeRate : +int CurrencyIdFrom FK
CurrencyExchangeRate : +int CurrencyIdTo FK
CurrencyExchangeRate : +float multiplier
Transaction: <|-- User
Transaction: +int UserIdFrom FK
Transaction : +int UserIdTo FK
Transaction : +String amount
Transaction : +DateTime TimeStamp INDEX
Transaction : +enum Status (started, waiting for PSP, cancelled, rolledback)
Notification: <|-- Account
Notification: +int AccountId FK
Notification: : +enum EventType
Notification: : +DateTime TimeStamp INDEX
AuditLog: <|-- Transaction
AuditLog+ int TransactionId FK
AuditLog: +String EventSource
AuditLog: +DateTime EeventTimeStamp INDEX
The sytem consists of the following:
1) Api gateway - an entry point to the system which also acts as an auth service and a rate limiter
]2) Paymnet service - service to transfer funds from a to b
3) User management service - service to manage user accounts like create, update, delete ,etc
4) Logging service - service to log transaction events
5) Notification service - a service to notify user about transaction events
6) Monitoring - a service to monitor system health
]7) Wallet partition - a distributed node in a payment system which holds partitions for a specific user accounts
8) Curency rate backend - a real service (like a bank) providing currency rates
9) Currency rate service - an adapter to the curency rate backend serving urrency rates in our system whne payment has arrived
10) Partition configuration service : a service that holds configuration information about wallet partitions like hteir addresses
11) Fraud detection system scans new and old trsnasctions to detect any fradulent transactions
NExt we list what each wallet partition consists of:
1) CommandQueue - a queue of events emitted by user actions
2) Event queue a queue of events derived from commmands abouve
3) Event Log - an append only database that stores all events in the system used for audit purposes
4) Wallet db - a view of event queue that represents the latest state of a user wallet
Fund transfer flow:
1) User get the id of another wallet he wants to transfer funds to using HTTP request
2) User is transfered to UserManagementService and gets user id of a target account
3) User isuses a payment HTTP request specifiyng from, to and a user account
4) A user request is transfered to the PaymentService
5) Payment service finds replica of the source and target account
6) Payment service contacts Trsnsaction coordinator to execute a distributed trsnsacation spanning 2 nodes (from and to). Every node does the following:
1) Emits the appropriate command for a user action
2) Event machine picks a command and produces 0 or more events and puts them in the event queue
3) Event machine picks events from the queue
4) Picked events are persisted in event log for audit or replay purposes
5) Wallet db is locked to apply events to update the state
6) Wallet db state is updated in locked state
7) Wallet partition signals coordinator that event has been applied
]8) Trasnsaction cocordinator proceeds to apply next transaction portion to a next replica and awaits for confirmation
9) Event log is replicated to the next replica as a aprt of the transaction
10) Other replica signals success
11) Transaction coordinator signals success
12) Payment service sends a notificaiont to a notificaiotn service
13) User receives success from a payment service. Also optional a user may receive notificaiton when a wallet partition successfully applies an event to a wallet
Fialed payment flow
1) One of the wallet partition is locked and signalled as a a success
]2) Other wallet partition fails
]3) Depending on the lgorithm the transaction ccordiantor retries the failed command on a failed partition while locking the transaction or rolls back the previous action
4) Depending on the algorithm the coordinator retries the failed action on a failed partition or signals transaction fail
Audit flow:
1) Audit entity wants to audit account
2) Payment service finds an appropriate wallet partition
3) Audit log is retrieved from the wallet parttion
1) Every wallet partition is a psearate machine consising of:
1) COmmand queue - an in memory queue that creates commands based on user input
2) State machine - an event machine that picks new coammnds, emits events and works with qeueue
3) Event queue - an actionable event that can be applied to a wallet db
]4) event log - an append only storage of events useage in case partition crashes or for audit to replica all the account state fomr the begiinging,
]5) Wallet db a database containing latest state of a wallet for a user
Every wallet parttiion is a sself-suffucient node that participates in a distributed transaction when making wallet transferr or replicating event logs
We explain how we arrived at the current solution using at first naive and then more expanded version of the same system:
1) To ensure scalability and aviaalbility and also considering a huge amount of trsnsaction to support we have to partition our Wallets db into multiple partitions. THe first solution is to use in memory DB like Redis because the data to persisits is not that huge (account maount + account id). To support multi node trsnasaction we define a dedicated transaction coordinator and use either two phase commit protocol or TC/C Or Sage to make sure that our trasnsaction complete when user makes an account transfer. Using transasction coordinator instead of a dynamic cocordinator is simpler and if it is stateless by nature so if it fails we can spin another one on standby. If the number of transactions is small we choose 2PC as it is simpler to implement and contingency is low, if contingency becomes high we swtich to TC/C. If paralelism becomes a solution to the increased throughbut we switch from Saga to TC/C as it supports out of order parellel transaction excution compared to Saga or 2PC
2) THe solution in 1) works well if our wallet nodes don't crash and if audit is not necessary, which is not true based on our functional requirements. To support both we move to an Event based (CQRS) architecture where every user action is trnsalted into an ordered squenece of commands and then events that are applied to the wallet. We store both in queues (to have order) and use a loop that periodically (or when a new item arrives) picks a command, emits events, picks events and applies them to a wallet. This is all done for every node partiion as a part of wallet transfer transaction
3) Solution 2) works well, but the event log is not durable, because if a wallet partition crashes all events for that partioin are lost. To make events durable we replicate them across other replicas using RAFT protocol. RAFT protocol is a dynamic replication protocl that self-assigns coordinators and followers to replicate the data
4) To prevent data reaces for multiple transactions runinng simultaneouse for teh same accoutn we put our paymnet requests into the queues before the transaction cocordinator (we choose rabbit MQ) nd process them one by one
1) In case payment service fails we can easily spin a new one because it's stateless
2) In case we have too many requests our API rate limiter in the APiGateway will limit the connections
3) In case the distributed transaction fails in one node we do:
1) If we have 2PC we retry until success or abort the transaction all togheter. 2PC doesn't apply changes until every node sent an ack so no rollback is necessary
2) if We have TC/C we retry the failed replica or in case we give up the applicaiton code performs a rollback action and TC/C will propagate it to the remaining replicate that have already commited the changes
3) If we use SAGA we retry the failed partition or in case we give up we rollback failed operation in revers order they were applied till we hit the sucessfull one or we rollback it all together
3) In case our event log replication fails when we use RAFT the RAFT protocol will elect another coordinator to replicate the logs and the process will start a new
4) In case our wallet partiion leaves or enters we can use consistent hashing schema to minimize the number of data shuffles
5) in case one of our wallet partiions containing data for ap articular user fails we can display - service unavailable for a user
6) To prevent double payment scenarious we make our transfer operation idempotent by giving a client an idempotency unique key based on the trasnsfer info and clients unique id. The client has to specify this key when sending a request. If a request with the same key is sent withing a time frame (24 hours) the request deems duplicate and syhstem doesn't perform any trasnsfers. We store idempotent keys in our drable event log and clean them every day with a background cron job in distributed manner
1) Notify users about more events
2) Provide users with a predicted expenses based on their spending history
3) Provide ezpz integration with a 3rd party payment services