System requirements

Functional:

1. Users should be able to search for a movie review.
2. Users should be able to upload a movie review.
3. Users should be able to filter the movies by the title, year of release, genre.
4. The system should be able to aggregate movie reviews from multiple aggregator systems like IMDB, Metacritic, Rotten Tomatoes.

Non-Functional:

1. The system should be accurately showing the information from the aggregated reviews.
2. The search and filtering API should be fast. (<200 ms).
3. Scalability - The system should be able to handle the growing user base.
4. High Availability - The system should be available most of the times.

Capacity estimation

Let's assume the total number of users will be 100 million.

We expect about 10% of total users to be active every hour.

Which means 10000000/60/60/24 ~ 120 requests/second (reasonable)

API design

1) Users able to search movies. - GET/api/v2/search/movie

2) Users able to upload a movie review - POST/api/v2/movie/review

3) The system should be able to register and update a user.

POST/User/create

PATCH/User/update

4) Users should be able to filter movies based on certain criteria like year, movie name, genre etc.

GET/api/v2/filter?param=genre¶m=title¶m=year

5) There can be a API to fetch movie reviews from external sources - That will be integrated with external APIs.

Database design

This system involves mostly movie related metadata and reviews related information, and the relationship between movies and reviews.

We need more consistency in this design than the amount of data to be concerned with. We will choose a relational Database for the same.

1) Users Table - user id, name, date created, review id,

2) Movies table - Movie_id, movie_name, movie_metadata_id, movie_rating

3) Movie metadata - movie_metadata id, movie_id, genre, movie information, actor information, year of release etc.

4) movie Reviews table - movie_review_id, movie_id, submitted by user id, movie_source_id

5) Movie Sources table - movie_id, source_id , source_name

6) Review Sources - source id, source name

7) User favourites - user id, movie id, created at

High-level design

The components involved in the design will be

1) User service - to add users to the system, to update users, to add user favourites.

2) Movie search service - to search for movies, filter movies based on user inputs.

3) Reviews service- to add reviews to database , to update reviews.

4) Review Aggregator service - to collect reviews from external sources and then nomralise the data and dump it into the database.

5) Admin service - manage adding movies to the system.

6) Notification service - to send notifications to users about movie updates (based on the user notification preferences)

7) Database - a relational database like MySQL or postgres to store the movie , reviewsand user related information

Request flows

User searching for a movie

1. The user interacts with the search field in the webpage , to search for a particular movie, and this request gets directed to the MovieSearch Service.
2. The movie service fetches the details from the database and returns it to the user

User submitting a review

1. The user will add in a review at the website, and click on Add Review Button.
2. This will trigger a request to the MovieReview Service, which will validate the review and post it into the database, and simultaneously update the rating of the movie in the database.

Reviews collected and aggregated from external sources.

1. There will be a periodic job scheduler which triggers requests to external sources via the Review Aggregator Service to fetch the reviews for various movies.
2. There will be a mapper for each source to map them into the normalised data format.
3. In order to efficiently handle review processing, the review information can be stored in Message queues like Kafka (each source can be a queue) and then A batch processor like Spark can be used to process and aggregate the reviews.

Notification Service - to send updates to users(based on their notification preferences) about alerts on movie updates.

Admin Service - To handle adding/updates of new movies into the system.

Detailed component design

The review aggregator service will be one of the important services in the system design.

This is because, it will form the bulk of the data in the system.

We can have a job scheduler to periodically trigger requests to the various external sources to fetch the movie reviews.

There will be a mapper to convert the data to a specific format.

We need to handle duplicate reviews, by content hashing and timestamps.

once the reviews are normalised, they are stored in the movie reviews database

This service also needs to be highly scalable. We can horizontally scale the service to add multiple instances.

It might so happen that while fetching the data from the external source, there is a failure to fetch the data. We can implement a retry-mechanism, with a fixed number of retries, or an exponential retry backoff in order to ensure that we have fetched the review correctly.

Trade offs/Tech choices

Computing the ratings can be pretty expensive as the number of reviews increases, so pre aggregating the ratings in the database whenever new reviews are added allows for quick response times.

Introducing queues in the review aggregator serivce allows for efficiently handling the external reviews processing.

In order to cater to the structure of review from different external sources which might be different, we can store those reviews in a NoSQL database instead first, and after normalisation, we can store them at the SQL database.

Failure scenarios/bottlenecks

All the services like Movie service, and review service and review aggregator service can be scaled horizontally

The database can do an optimal read from replicas and writes can happen to a master node.

The review aggregator service should use message quesues and a batch processor for efficient review processing.

Future improvements

Adding a movie recommender engine for showing popular choices to user based on their favourites.

Adding webhook based integration from external sources in order to allow for real time updates for movies whenever they have movie review updates at their side.