Requirements

Functional Requirements:

1. the system should be able to monitor market data on a daily /hourly/every second basis.
2. the system should be able to automatically trade - buy & sell - based on market performance of a given stock.
3. the system should be able to analyze financial news/data to be able to make guided decisions on trading and also trading strategy
4. Quants should be able to deploy custom strategies.
5. Traders should be able to monitor profit and loss live and also the system health.
6. Risk compliance managers should be able to set positional limits/caps or limits on trading.

Non-Functional Requirements:

1. the system should be consistent since it deals with financial data. System not being available for some time is of lower loss than system yielding in trading based on inconsistent data.
2. The latency of trades being executed should be 10-50ms latency.
3. Since the data residing on platform would be PII, the data should be secure both at rest and in transit.
4. The system should be able to monitor market data for 1000s of organizations every second of a day. The system should be able to execute trades for 100000s of individual portfolios daily. The system should be able to execute this trade at mid frequency SLA of 10-50mslatency.
5. Trading should be idempotent - meaning that there cannot be duplicate trades upon potential failure.
6. Handle fault tolerance by halting trade until orders reconciled with exchange.

Capacity Estimation

1. 100000 transactions with the exchange per hour
2. 100000 transactions would translate to at least 300 per second.
3. portfolio for at least 100000 customers.
4. monitor the market performance for at least 2000 companies every minute of the day. This translates to 2000*3600*24 = 172800000. This translates to 48000 messages per second. This is a lower estimate - during peak trading sessions this should scale up to 100000 messages per second
5. Assuming each transaction roughly takes up at least 8-16 bytes, that is roughly at least 3000 GB a day. If the data stored after monitoring/news etc are even more in size then this could increase even further.
6. This means we have to potentially shard/partition the data in our databases by day at the very least.
7. This means that the write/read paths in our data stream have to be separate.
8. And if the trading strategy module has to run at high latency with decisions having short TTL this module would be read heavy.

API Design

We will use rest api conventions for our api design. This api would be authorized/authenticated in the header using an authentication token. The authentication token to begin with can reside on some key vault mechanism and periodically updated in accordance with exchange platform regulations.

1. GET - /fetchMarketData?marketId="" -> [{MarketPerformanceData}]
2. POST - /tradeStock -> {boolean trade status}
   1. request: {tradeId: uniqueIdemPotentKey,companyName:"", amountofStocks:"", sellOrBuy:"sell/buy",tradeTimeStamp:"",marketName:""}
3. POST -/deployCustomStrategy -> {boolean deployment status}
   1. request: {deploymentName:"",deploymentTimeStamp:"",deploymentStrategy:"", quantName:"",marketName:"",userType:"admin/nonAdmin"}
4. POST -/setTradeLimit -> {boolean status}
   1. request: {companyName:"", limitAmount:"", limitSetTimeStamp:"",limitStocks:"",marketName:"",userType:"admin/nonAdmin"}
5. GET - /getPLStatement?traderId=""&startDate=""&endDate="" ->{timestamp:, profit:, balance:, listOfTransactions:[]}
6. POST - /verifyAccount?authorizationCode=""
   1. return:{authenticationToken:"", expirationTime:""}

The financial news will have to be fetched from all the different financial sites using a web crawler - this could potentially run as a separate cron service of its own updating in to a NoSQL database.

High-Level Design

All the services reside behind an API gateway which handles the rate limiting and authentication using an oauth 2 framework.

1. The fetchMarketData service takes the marketId and from the exchange for the market Id fetches the market indices for the current timestamp. This has to run like a cron service and update the kafka event stream. The consumer at the end of the kafka event stream processes this market data and updates the writeDataBase.
2. The tradeStock api during any first purchase verifies that the bank account has enough balance to authorize the purchase and then reaches out to the exchange to make the purchase and then updates the kafka event stream with the purchase
3. The deployCustomStrategy takes in the user authorization level, deploys the updated strategy for a given user and sends the updates to the kafka event stream.
4. The setTradeLimit works similar to the deployCustomStrategy and based on the user authorization updates the event stream with the updated strategy.
5. The verifyAccount api uses the oauth 2.0 authorization framework to verify the bank account and update the verification status to the event stream. This is user driven before they use the platform.
6. The getPLstatement retrieves the profit and loss statement from s3 storage based on the parameters. It takes the traderId, and if no timestamp is given returns the statements in a pagified format. If timestamp is given it retrieves the statement for that timestamp and returns to the client. These statements are computed and stored as files using daily spark batch processing jobs. These read directly off the write database.
7. A flink service - analyzeMarketData - takes the market data from the kafka event log and generates market purchase recommendations for a given traderId and writes back in to the postgres write database.
8. fetchFinancialNews - this api similar to the fetchMarketData uses a webcrawler to retrieve all the financial news and updates the event stream. This gets written in to a separate nosql database.

Database Design

1. marketPerformanceData - id, marketId, companyName, companyTickerName, companyCountry, marketIndexValue
2. market - marketId, marketName, marketCountryId, marketCurrencyId, marketLocation
3. trader - traderId, traderName, traderAddress, traderEmail, traderPhone, traderCountry, bankAccount, bankAccountVerificationStatus
4. user - userId, traderId, userRole, userOnboardedTimestamp
5. country - countryId, countryName
6. currency - currencyId, currency
7. purchaseRecommendation - recommendationId, traderId, recommendationCompany, recommendedNumber, recommendedCountry, recommendedTimestamp, recommendedRate
8. transaction - transactionId, transactionTimestamp, numberOfStocks, totalAmount, exchangeOperation
9. balance - traderId, balance, lastUpdatedTimeStamp
10. strategies - strategyId, userId, strategyTimeStamp, strategy
11. limits - id, userId, OrderRateLimit, timestampLimitSet

Detailed Component Design

Assumption: mid-frequency trading; if the firm targets true HFT, the tick→strategy path swaps from Kafka to an LMAX Disruptor ring buffer in shared memory.

1. Since we write all the financial transaction data in to a single write postgres database - including market recommendations for traders, the system is going to be consistent. To keep the market recommendations consistent with the up to date ticker data from exchange, we use a flink pipe to do the historical data analysis. This is ideal for streaming workflows.
2. A key detail of any financial purchases or transactions would be idempotency which we ensure in the tradeStock api using a unique idempotency key - so as to not repeat a transaction in case of a lost response from the exchange. This idempotency key is also transmitted through our event stream in to the postgres database.
3. For verifying bank account details, we will use a oauth 2.0 authorization framework - using this we first directly communicate to the banking service to get an authorization header. This is then used in the verify account api to verify the user id for the bank to generate an authentication token. This authentication token will have an expiry date. This way the actual bank account details never reside in our database. For any future transactions, we use the balance from our trading account and verify this balance first in tradeStock api before authorizing any exchange.
4. We enable millisecond latency and millions of transactions by writing all our events in a kafka event stream. Kafka is a message queue which has a default data storage of 7 days. All our updates do not directly hit the database - leading to delays in responses. All the services hit the kafka event stream and respond back immediately. The consumers at the end of the kafka stream take the data through the appropriate processing jobs in to respective flows. Now if kafka goes down, since it uses an offset mechanism which is retained for each of these consumers, the consumer can pick up reading from that stopped offset.