Loading...
DAU: 10 million users, 1 million drivers.
Location updates: every 2 seconds, so in total 20 million updates for users and 2 million updates for drivers.
Each user make 2 requests of trip daily, so it would be 20 million requests in one day.
Driver picked up a trip -> User receive a notification
Started a trip -> Both user and driver received a notification
Completed a trip -> Both user and driver received a notification
So 3 for user and 2 for the driver, it would be 30 million notifications for users daily and 2 million notifications for driver daily
Login
Post /api/v1/auth/login
Params userId, password (encrypted)
Return accessToken, refreshToken
Find nearby drivers
Post /api/v1/user/search/drivers
Params: accessToken, userId, carType, lon, lat
Request a trip
Post /api/v1/user/trip/request
Params accessToken, userId, carType, location, destination, lon, lat, phoneNumber (optional)
Pick up a trip
Post /api/v1/driver/trip/pickUp
Params accessToken, tripId, userId, lon, lat
Real time location tracking
Websocket /locationTracking
Param lon, lat, tripId, userId, role (user/driver)
For storing tons of hundreds of thounds of data, I prefer using Non-relational database like Cassandra as our primary database, the trip info, user info and geo locations all can be properly stored in it. Also we can benifit from the advantage of powerful analytic functionalities offer by Cassandra.
We also need a key-value store to provide fast read and manage the large numbers of sessions.
Show as diagram
Prerequesties:
User login Uber with userName and password -> authenticated by server -> get the access token
Show nearby drivers
User click the trip page -> Rate limiter service got the request and validate -> passed, routes to Geo service -> fetch the nearby available driver's locations from Redis -> If not found, fetch the data from Non-relational database instead
Make a trip
User A make a trip -> Rate limiter service got the request and validate -> passed, routes to User service and validate the token -> token valid, forward the request to Trip service A and establish the websocket connection, meanwhile publish the locations to Kafka topic A -> return
Pick up a trip
Driver A in the nearby area of user A, received and picked up the request -> Rate limiter service got the request and validate -> passed, routes to User service and validate the token -> token valid, forward the request to Trip service B and establish the websocket connection, meanwhile publish the locations to Kafka topic B -> return
In-progress trip
Trip started -> locations shared by user and driver periodically (e.g. 2s) and update the Kafka topic -> Trip Manager Service would subscribe the updates and distribute to user and driver respectively.
Trip complete
Trip completed -> request the backend api to persist the trip data, and close websocket connection for both user and driver
API Gateway - Include Rate limiter and load balancer to basically throttle the request and distribute requests based on certain strategies.
User Service - Token generation and validation, Session creation and validation and match the available drivers for users
Geo Service - Maintains nearby driver indexes and handle and process spatial queries.
Trip Service - Manage stateful websocket connection with users and drivers
Payment Service - Handling payment process and request the external API to complete the payment atomically
Trip Manager Service - Listening the topics from user A and driver A and share it via the websocket connection they established, and fan-out the updates
Analytic Service - Gather the trip data (routes, locations, etc.), aggregate and analyze the data and persist into database
Notification Service - When offline status detected, Trip Manager Service will forward the request to Notification Service in term of notifying on users/drivers.
Redis - Manage session info, online status, metadata for fast look up and nearby locations
Non-relational Database - Storing trip info such as routes, locations, durations and analyzed metrics
1.Relational database vs Non-relational database
We are using non-relational database in current design over Relational database as we want the write operation to me efficiently handled. And it's highly scalable in term of incremented data. Also we are handing over the payment processing to the external services, they would guarantee the security and integrity of the payment. So we don't have the requirement for strong consistency.
If we already have build-in structure or relational database in our system, we can reuse it and stop using external service.
2.Update/Fan-out the location changes via backend service vs Message Queue
We can directly let Server A to make the request to Server B and share the location changes from user to driver and vice versa. However, this kind of operation are very frequent and we have to make our backend service highly scalable and productive to handle the high volume of data.
We are publishing the location changes to Message queue rather than leave too much pressure for the backend service, decoupled the process, also make our entire system easily scaled.
The data in the Message Queue can also be consumed by Analytic Service or other services.
1.Backend service crashed down.
We should be building more instances of each backend service in terms of scalability and avilability.
2.Kafka died
Instead of publishing the location changes to Kafka, Server A which connected to user A would fetch the information of driver's server B and directly send the changes.
3.External payment service not responsive
We would support multiple ways of making the payments, and we would retry for few times if one of the payment API not responsive due to network issue. If still no luck we would ask the user to change another way to pay.
If we want the payment service to handle the transactions and get rid of using External API, we can utilize the relational database to have strong consistency.