Loading...
Let's consider that we plan to have this app for 1000 parkings with 1000 spots each, that gives a 1000000 spots. during the peak hours let's assume 20% of these places are taken/freed up per hour, that gives us 200000 events per hour and 55.5 events per second. Now the "ticket" table will consist of
ticket_id (uuid) - 16 bites
parking_lot_id - 4 bites
spot_number - 4 bites
entry_time - 8 bites
status - arounf 10 bites
let's round this up to 100 bites
so 20000 events x 100 bites = 20 mb daily, and more than 7 GB yearly, not a huge number. logs will add up space though and storing the pictures of the license plate (if necessary) even more (they take a lot of space). Ok then more than 7 GB yearly, let's round this up to 10 GB for the DB records per se, plus 20 GB for logs, 30 GB of fast storage per one instance per year. let us multiply this by 1000 parkings, which gives 30TB needed per year. for th eten uopcaming years that 300 TB needed for our 1000 parkings
Api Gateway is responsible mostly for routing the requests to the appropriate services and monitoring, as there's limited authentication and authorization, rate limiting need (only basic for the wab app reservation service) and caching.
the reservation service is responsible for reserving particular spaces for a given user (after successful payment). we'll provide here the strong consistency by using the optimistic locking, we can do it since potential collisions should not happen often. at the same time we'll not tie up the DB resources, prevent deadlocks (however newer DBs handle them automatically, user can still receive 500 error) and maximize concurrency. the cons is that the user can see later that (in case of taken resouce) reserved spot is no longer aviable. This service has it's own postgresql db that stores which parking slot is reserved and which is not>
Payment service, here we also must provide a strong consistnency as the user cannot be charged two times. it uses exterenal payment provider and publishes an event with transaction_id and information (in progress, failed, successfull). Other services are listetning and reacting for these events. this system is agnostic and handles only transactions
parking lot service, responsible for allowing particular vehicles enter and exit physically parkin glot on the spot. it reads vehilces licene plate and triggers the payment service when fee must be aid at the exit
logging service, collects and writes logs from all services (barrier opened, transaction failed, spot reserved etc) and writes the to a ealstisearch db
I prefer here the relational database - it might not scale as easily as non-sql however data is well structured and a single relational database (like postgresql) could handle calculated capacity easily and scale up if needed. Moreover it provides a strong consistency due to the ACID properties which is needed by the reservation service.
This is proposed schema
Scaling
looking at the calculated space adn DB usage i can confirmly predict that no sharding will be needed in the foreseeable future, this is a necessity. Before that we will scale up and optimize the db by looking at the pg_stat_statements, creating indexes : B-tree indexes on the FK
Reservation: spot_id, lot_id, user_id
Transaction: reservation_id, user_id
Spot: lot_id
Additionally partial index on the Spot table (free spots with status=Free) and unique index ont he Vehicle table for the lincense_plate (to quickly retrieve vehicle by license plate while scanning on the entrance and exit). When scaling up will be not enough then we'll introduce read replicas - we will introduce replication lag but we should easily scale horizontally. finally we can start partitioning some tables (most prpbably vehicle, transaction and reservation sechema). I propose doing it by range partitioning (one month maybe) or by city.
Reservation service:
An online web application, you don't need to log in or provide any jwt token to authenticate, as anyone can reserve a spot wihtout registration. We require 4 parameters to be provided - email, license plate, revervation_start timestamp and reservation_end timestamp. on a successfull call POST reserve_parking_slot (via HTTPS, so the data is encrypted) this services calls queries the DB in order to search for the available spot first, using the optimistic locking (we assume that reservations are not done very frequently) using the indexed spot table (select * from spot where status = FREE). Optimistic locking mechanism - in th eapplication code we'll check how many rows were affected by the update
update reservations set status = 'reserved', version = version + 1 where id = :id and version = :current_version and status = 'free'; or in the app code (django)
def try_reserve(reservation_id, current_version): # UPDATE ... WHERE id = X AND version = Y AND status = 'FREE' updated_count = Reservation.objects.filter( id=reservation_id, version=current_version, status='FREE' ).update( status='RESERVED', version=F('version') + 1 ) if updated_count == 0: raise ReservationConflictException("spot taken"). the version save and counts if this particular reservation is still free.
Hot-surge prevention - I consciously chose optimistic locking as in my opinion that surge is very unlikely for the parking lot service. However if this problem would start to occure (which should be monitored) then we can introduce a redis cache before making a request to the db (using the atomic SETNX). this way we can almost instantly handle many requests, but it introduces another problem - data consistency between redis cache and databse. the tradeoff is not worth the work now in my opinion and accoroding to projected usage in th capacity estimation section.
Retry idempotency - we'll handle this the same way we handle idempotency in the payment service, using the idempotency key, we check it and if it's the same we treat it as the same reservation.
Overlapping itnervals - every spot can be registered for a not-overlapping time. we can introduce a boundary (e.g 30 minutes itnervals, so there's no possibilty to reserve the spot for 1 minute). the reserve_parking_slot(user_email, license_plate, parking_start_timestamp, parking_end_timestamp) endpoint requires parking_start_timestamp and parking_end_timestamp and checks it the spot is free in the requested time period.
If free spot is found then this spot_id is returned and spot status is changed to reserved, alsot reservation instance is created. Now, how to change the status back to free if no payment was done? couple of options, but since not so many reservtions are done i propose pg_crone directly on the DB - every minute we scan spots to put back to FREE. I noticed now that we lack in the schema timestamp when the reservation was done, i updated it with the reservation_started_timestamp. so we can scan now only the spot table and put them back to FREE and delete the corresponding reservation instances. Succesfully done reservation endpoint returns reservation_id, needed now by the Payment Service.
Payment service:
Service responsible for making payments, must handle both online reservation payments as well as payments at the gates when leaving. Payment service must have two properties: idempotency (so the user is never charged twice) and asynchronousity (as external payment providers are working usually in the webhook technology). So we'll generate and use the idempotency key, widely accepted by payment providers. we'll store the key in our system and use it in transaction, in case of retry we can check if this transaction was scheduled in the provider system and even if we retry the transaction in the provider with the key, we will not be charged twice. We'll use bilateral communication using webhooks, thus constantly listening what response does th payment provider returns, having in mind two important issues: signatrure verifiaction (to make sure it's provider and not a man-in-the-middle) and payemtn idempotency (provider can send the webhook twice and we have to handle it properly). Then, we send the transaction result (after updating the proper tables) on a message broker (rabbit mq) on whci two services are listetning, namely reservation (reservation is succesffull, user is notified) and parking lot service (parking spot is reserved)