Requirements

Functional Requirements:

1. users should be able to view local establishments
2. users should be able to search for local organizations.
3. users should be able to search for organizations by name/geospatial index
4. users can contribute to reviews through ratings and text - either anonymously or as an authenticated user.
5. users can be able to edit their existing reviews. Reviews for a given store should be one per user - for at least authenticated users. For anonymous

Non-Functional Requirements:

1. Prioritize availability over consistency - a new review showing up a minute later after it was posted would not significantly alter the aggregated ratings etc with eventual consistency being okay.
2. scale up to a million users per day with a million searches per day.
3. latency on the search results should be milliseconds.

Capacity Estimation

1. 1000000 searches per day - 42000 searches per hour - 12 searches per second. Which is a very very conservative estimate. Even 100 searches per second would roughly get us 10000000 searches at least. That makes this a more read heavy system.
2. Assuming 10000 writes per hour - this equates to at least 4 times more reads than writes.
3. Now if the number of searches per day goes up to 10 M - 10000000 and number of writes per day goes up to 100K this ratio of 4:1 goes up to 100:1.
4. That results in the read path potentially having cached results at multiple touchpoints. The search for optimized search should potentially use something like elasticsearch.
5. The locations can be stored in a postgres database with geospatial indexes - instead of having latitude/longitude in separate columns - leading to a wider area search, we can divide a map into tiny squares and each square being further subdivided with postfixes. Then hash them with a base32 encoding. This gives us closer geographical areas capturing similar prefixes of the hash.
6. We will be capturing review, location, phone, email, name, address, latitude and longitude , ratings for each store/organization. This is excluding any pictures in reviews or of the stores.
7. With images let us say we have at least 300 kb filesize for each image. For roughly 50% of the users having images, that is 15 billion Kb of data which is 15000 GB.
8. On an average let us assume each row of a denormalized table is around 8 bytes. For a million users daily that is 8 million bytes of searches an hour. That translates to roughly 2-5 GB searche data.

To sum up cache absorbs a portion of the read volume - even during peak search hours and the rest of it would be through the elasticsearch cluster.

API Design

We will use REST API framework for our API design.

1. GET /searchBusiness?location=""&type="" &pageId=1
   1. response: result:{[{businessName:, businessAddress:, businessReviews:, businessRatings}]}
   2. The output response is a list of json objects of n businesses based on the pageId.
   3. This search api would yield results by cursor with n number of search results within each page and the next call to the API for the same parameters would be continuation of the pageId.
2. POST /addReview
   1. the authorization header would capture the authentication token - obtained by an oauth-2 framework for authentication. request: {'businessName':"",review:"", rating:""}
   2. response: boolean status
3. POST /editReview:
   1. the authorization header would capture the authentication token - obtained by an oauth-2 framework for authentication. request: {'businessName':"",review:"", reviewId:"",rating:"",userId:""}
4. GET /getReview?businessName=""&location=""&pageId=1:
   1. response: {businessName:,reviews:[]}
5. GET /viewBusiness?businessName = ""
   1. response: result:{businessName:, businessAddress:, businessReviews:, businessRatings:}

High-Level Design

All our services reside behind an API gateway which handles our rate limiting and authentication using Oauth 2 as previously discussed. The write database which captures the write path can be structured as a postgres database.

1. The addReview service will add reviews to the write database. For a given businessId with an optional userId , this service captures the review text and rating and updates it to the write DB.
2. The editReview service similarly can be used to edit a given review by taking in a reviewId and reviewText and updating the corresponding row item in our database
3. The viewBusiness service takes a single business name and returns all the businesses that match that name from our postgres database
4. The getReview service takes a business name and location and fetches all the reviews for that business name and location from the postgres database.
5. The search business which needs a separate path away from the write path will fetch results for the search criteria from elastic search and redis cache. Elastic search would be synced with our write db using a CDC cron job etc. This is where our eventually consistent aspect of the system will come in to play. Elastic Search is a tool that provides with millisecond latency search results - be it structured or through semantic search. This would in the future facilitate potential natural language querying. Frequently searched results will reside in a redis cache with a ttl of around half an hour. For example a new restaurant opens up, and we see a lot of searches for that particular restaurant name. If this result is cached in a redis cache we will be able to fetch the results from it. If there are cache misses we can hit our elastic search to fetch that and also update the cache with the result.

Database Design

1. review - review_id (PK),userId (FK), businessId (FK), reviewText, rating, created_at
2. user - userId (PK), userName, userEmail, userPhone
3. business - businessId(PK), userId(FK), businessName, businessPhone, businessEmail, business Location

Since the searches are all going to reside on elastic search we can use geospatial indexes on elastic search to optimize the search. Using userId_businessId as a primary key in our review table would ensure that we enforce one review per user per business logic. However this forces us to not be able to have anonymous reviews. Does that work with the requirements?

Detailed Component Design

1. The search is where the eventual consistency of the system comes in to play. Our database of PostGres, Redis Cache and Elastic Search all come with 99.9% availability so our system is going to be available most of the time. This addresses the non functional requirement of availability over consistency. Since the search/read path runs off elastic search which needs to be in sync with our write database we will get our eventual consistency here.
2. Our elastic search can be set up with a search by distance too. So in cases that are deemed to be a city boundary, we could use elastic search geo_distance feature to be able to search across borders too. Elastic search takes geographic data as raw latitudes and longitudes and any search could be derived off this geo distance to provide search. Only when using postgres geohashing will the border scenario truly come in to play. Given that elastic search offers a 99.9% sla of availability, the majority of time we could use this.
3. For reviews posted, the idempotency is provided by creating a unique index of user_id and business_id (whenever user is logged in). The reviewId key could also be a combination userId and businessId, with random numbers that do not exist in the database generated for anonymous reviews. For anonymous users we cannot enforce idempotency per say, but to track potential fake reviews/ratings we could use bots or other automated tools to filter them as much as possible. Upon a successful write of a review in to the database, we can return both a status and the review text to our client to post it on the front end.
4. The split second latency promised for searches can be facilitated using the combination of the Redis Cache and Elastic Search. A new restaurant comes up and it is friday night. If the volume of searches for a given query spikes, we place those results in a cache. This could have a short TTL. The redis cache can have a key of the search value along with user id and location (if live) and the value could contain the search results. Now how to update these cached results for mobile users? We map a combination of the searched query/userId and the live location against the cached key, and if the location is significantly different, the data is stale, now we use Elastic Search for delivering fresh search results. Alternatively for any live location driven searches we could always use the Elastic Search to avoid the hop to redis cache and then rerouting back to Elastic Search. When the searchBusiness reaches out to Elastic Search but the request times out, we can fall back on either cached results, or use the geospatial indexing we have created in our write db until elastic search recovers. This helps our system to be available always and not a time out for the client/user.
5. When writing a review - either in addReview or editReview, since our write DB will have replicas, if the master goes down, when the replica picks up the masters role, we could add a retry to write the review in the database. This retry could be set with an exponential time delay each time with an iterative count limit