System requirements

Functional:

• Users should have ability to securely log onto their accounts and view their balance, deposit cash, deposit checks, withdraw cash up to a max amount
• Users will have transaction limits on their withdrawals (defined by their account balance and personal limits)
• User interface will support multiple languages
• We shall support a MFA system that requires both credit card and a pin

Non-Functional:

• Highly available and reliable system. Users should never be unable to access their accounts and perform basic transactions
• Highly consistent system required. Idempotent requests, etc...
• Highly secure system, unauthorized access is completely unacceptable.
• Response times will be under 50ms
• Errors shall be handled gracefully and as transparently to users as possible. For example, transient errors should be dealt with by implementing retries in internal system.

Capacity estimation

• We expect on the order 5 million users with around .5 million actively daily users
• HDD estimates: Each user shall have a profile which we estimate at .5mb (user address, name, account number, account configuration such as max transaction limits). They shall have current account information (balance, transaction information) which will be sized at 49.5mb (we store 7 year of historical data). Total HDD capacity estimates = 5,000,000 bytes * 5,000,000 users = 25,000,000,000,000 = 25 TB data
• Num requests estimates: 5 million requests per day (about 10 transaction per active user) = 58 requests per second.
• Num servers estimates: We will multiply the num requests per second times 100 to achieve peak load and will size servers accordingly = 5,800 requests per second. Assuming servers can manage 1000 requests/second, we need 6 servers. However, we want plenty of redundancy and thus will add servers across multipe AZ an regions, thus we need around 18 servers.
• Bandwidth (inbound) estimates: We assume each request will be 100 bytes. 100 bytes * 58 requests per second = 58000 kbps
• Bandwidth (out) estimates: We assume responses will be an order of magnitude larger than the request size = 580000 kbps = .58 mpbs
• Distributed cache estimates: We anticipate caching user session data, non sensitive user data (user name, language preference, account limits). Estimate at .5mb per user. Thus, about 250GB total. We can use consistent hashing based on user uids to shard our data across our distributed cache.

API design

• authenticate(creditcard_number): Returns short-lived auth token and refresh token.
• get_user_info(creditcard_number, authToken): Returns account balances, associated accounts (i.e. checking, savings, etc...). The API is used to access a lot of user info, but this doesn't necessarily mean that it's all returned to the end-user. Furthermore, we are creating a catch-all API to expedite the design here. It should be fairly trivial to separate this out into additional params such as get_user_balance(), get_user_accounts(), get_user_address(), etc...
• withdraw_from_account(creditcard_number, authToken, idempotency key, withdrawal_amount): Returns success or failure with new account balance.
• deposit_to_account(creditcard_number, authToken, idempotency key, deposit_amount)

Note: All requests are established using mTLS, using the ATMs short lived certificate, which is bound to its hardware key and generated quite frequently. Idempotency keys are used to ensure we don't accidentally execute the same API twice, which is critical in banking applications

Database design

We require highly transactional consistent database to support user purchases, etc... However, we may not need as strong consistency for storing user preferences, and storing user activities not related to deposit/purchase transactions. Therefore we will have a hybrid set up where we use SQL for storing (1) user credentials and (2) user transactions and (3) user profile. We will use noSQL for storing user preferences (language, etc...), activity for potential AI/ML use cases, etc... We will use Cassandra for noSQL store, ensuring fast writes.

We will have three SQL tables, each served by separate SQL instances, and each sharded using consistent hashing on the userUID. We scale each instance separately since, for example the auth SQL table will be much smaller and has different scaling considerations:

• User authentication information (SQL instance one):
  • TableUserPassword: userUid (primary key: credit card number), currentUserPasswordId (foreign key to password table)
  • TablePassword:
    • userPasswordId (primary key),
    • userUid (foreignKey),
    • passwordSalt (256 bit, randomly generated during password creation),
    • passwordHash (a 256 bit hash of user password, which is not stored in DB and
    • created_time: timestamp password created
    • active: boolean indicating whether password is active or not
• User transaction table (SQL instance two):
  • TransactionUID (primaryKey): randomly generated transaction UID
  • UserUID (credit card number)
  • AccountUID: Foreign key to the account table
  • DebitCredit Flag: Boolean indicating if it's a withdrawal or deposit
  • Amount: Float type indicating total
  • TransactionTime: DateTime indicating the timestamp of the transaction
  • Balance: Balance after the transaction
  • Status: Enum indicating status of transaction ('Pending', 'Completed', 'Failed')
  • ATM_ID: UID of the ATM performing the transaction
• User Account Table (SQL instance two): Multiple accounts may be associated with single user / credit card
  • UserUID: Unique user id
  • AccountId: PrimaryKey indicating account id
  • AccountLimitDaily: Maximum daily withdrowal
  • AccountLimitWeekly: Maximum weekly withdrawal
• User Profile Information (SQL instance three):
  • UserUID
  • Address: String containing primary address
  • Email: String containing email address
  • Phone Number: String containing the user phone number
• User Preference and Activity Table (noSQL): We won't go over all the details here, but essentially would capture language preferences, previous usage information (used for analytics), etc...

For caching, we will use a distributed Redis store and primarily used for storing user preferences so that users entering an ATM system will have their preferences loaded (language, preferred accounts, etc...). We will also use the same store for storing session data.

High-level design

• All communication from ATM Client to service load balancers is done over mTLS, using ATM's certificate. Its device keys (possibly stored in TPM) are used.
• The load balancers receive all the ATM Terminal traffic, and is tasked with validating the ATM Certificate, setting up the initial mTLS termination and routing the requests to the front end microservices, selecting the ones under least load as measured by number of outstanding requests
• The frontend microservices are scaled horizontally and route to either the auth service, transaction service or user preference services
• The load balancer and frontend microservices make use of zookeeper to determine the available hosts for routing all the requests to and use a client based on consistent hashing for determining where to route requests to
• The auth services also scale horizontally based on load. They receive user pins and use their user auth info tables, sharded by users. For high availability these shards are also replicated, which is fine for ensuring fast reads but may not be best for fast writes. This tradeoff is OK, we prefer to optimize for reads.
• A distributed cache is used for both session establishment but also loading user preferences. All balance data that requires high consistency would be stored in DB too. That is, we first update DB and then place it in cache. Reads can read directly from Cache for improved user experience, such as checking balances.
• The transaction service handles all transaction requests. Again, it uses the user transaction table which is SQL based for ensuring ACID properties. We need to shard this table to ensure we dont overload. And we also regularly archive since we really only need recent transaction history and account balance. We will for now have transaction service perform the archival but as we scale we can have this also be a dedicated service.
• We also have a dedicated user preferences noSQL store which tracks language preference, user activity data (for AI/ML use cases in analyzing behavior). Here we optimize on write performance using cassandra store though we prefetch data via cache. We don't care as much about read performance here though we will prefetch some data in cache for improved user experience.

Request flows

• Initial sequence
  • When a user first arrives at a terminal, the credit card is presented. The ATM client will initiate a sequence and all requests will hit the service's load balancers for routing to the appropriate load balancer
  • The credit card will first be validated. Its certificate is presented and attestation of its private key using a nonce generated by the auth services will be performed. The PKI is used for validating the credit card cert, ensuring its not in CRL
  • After credit card authentication is complete, queries are performed against the redis cache to obtain user preference (language preference, possibly other info to improve user experience)
  • The user is prompted for a pin, to complete the MFA. The pin is hashed, using the salt stored in the user auth info table, which is sharded by userUid
  • The Auth service returns an auth token and refresh token. The auth token expires every 10 seconds and the ATM is responsible for refreshing it (transparent to the user).
  • After initiation with auth services, a session is created and stored in the session cache. Part of session creation involves sending challenges on both ends, for verifying the authenticity (similar to how mTLS handshakes work), which validates authenticity of ATM, CreditCard, and backend services. Only at this point will the user's more sensitive information be used (i.e. name, transaction balance, etc...).
• Withdrawal requests: Once a session is created, a user may perform withdrawals, which is not to exceed their limits (limits stored in cache but also can be pulled from the user account). The withdrawal sequence is as follows:
  • When a user wants to make a withdrawal, we can obtain limits and current balances from the Redis cache. We validate the requests there first, which should not exceed max.
  • Next, user request is dispatched using idempotency key, which is stored in session cache, ensuring same requests is not made twice
  • Next, we create a lock for that user while we are running the command. During this time, we make the request by adding a transaction to the transaction store, updating the total. The cache is also updated with new total, reflecting the SQL amount. The cash is dispensed. The lock is released.
  • Idempotency key is retained for this session, any new requests will use the key to ensure it's not a repeat of previous requests. If so, it will return immediate failure and require new transaction to be initiated.
• Deposits follow a similar flow as withdrawal:
  • A lock is applied to user account during request. A lock is also applied on user whose account we withdrawing funds from.
  • A verification is run on the check being deposited. Verification is beyond scope of this design.
  • After verification, we create a single transaction consisting of withdrawal on check user and deposit on current user. The withdrawal portion will use logic that ensures that the withdrawing account has sufficient funds to both clear the check and any outstanding ATM requests (we can check if that user is in process of withdrawing, and if so, assume that they are withdrawing the max daily amount).
  • After totals clear, we update the cache, release the lock and return http/200 to the terminals.

Detailed component design

The transaction services are some of the most important backend services, which will be stateless and scale horizontally.

• They would use zookeeper to track the transaction table shards. We need highly sharded data here to ensure performance is very fast but ACID properties are met.
• The tokens presented to transaction service ensure authenticity of requestor.
• The SQL tables being updated are updated before the cache is and before the response is returned to user. Persistence at this level is very important.

Trade offs/Tech choices

• We've decided to cache non-sensitive user preferences to improve user experience during initiation, prior to full authentication. We take extreme care not to reveal too much information here, to protect users.
• Similarly, we cache user balances and account information but only expose after user authn. We need to be very careful to expose certain data in the correct sequence.
• We shard and replicate our SQL stores to improve availability and performance.
• NoSQL is used on portions of data that don't require as strong ACID properties but yet are optimized for writes, to ensure we can store a lot of user activity, later used for AI/ML analysis of user activity.

Failure scenarios/bottlenecks

• If a DB operation fails on updates to SQL store, we ensure atomicity by using SQL (i.e. either fully succeeds or fails). This will slow down updates but make our transactions more secure.
• If a DB instance fails on a critical table (transaction table for instance) we have replicated data across zones and regions and thus can recover.
• If a user session fails in the middle of a critical withdrawal, we should take extra care to align system with what transpired. For instance, a withdrawal that fails after DB update but before cash dispensing, should not allow the total to be withdrawn from the account. The account should be in a 'PENDING' state and totals should be added back to the account if we cannot enter state that guarantees health.

Future improvements

• As a future improvement, we can add other authentication mechnaism (biometrics), allow online logins to systems