Functional Requirements:

• Aggregate external reviews (IMDb, Rotten Tomatoes) and allow user-generated reviews.
• Users can search movies by title, genre, year, and rating.
• Display the movie's global average rating and a text summary of all reviews.
• Support user authentication for submitting reviews.

Non-Functional Requirements:

• Instantaneous UX: Users must see their submitted rating reflected immediately on their screen.
• High Availability (Write Path): The system must successfully ingest millions of concurrent reviews during blockbuster opening weekends.
• Continuous Eventual Consistency: The global average rating should update continuously behind the scenes without relying on arbitrary time windows.
• Massive Read Scalability: Complex search queries must resolve in < 50ms.

API Design

1. Submit a User Review POST /v1/movies/{movie_id}/reviews

JSON

Returns: 202 Accepted. The client application uses Optimistic UI to instantly recalculate and display the new average locally for the user, masking the backend processing time.

2. Search Movies GET /v1/movies/search?genre=Sci-Fi&min_rating=8.0&year=2026&page=1 Routed directly to the search engine.

3. Get Movie Details GET /v1/movies/{movie_id} Returns the movie metadata, the globally calculated average rating, and the AI-generated consensus summary.

High-Level Design

• API Gateway / Load Balancer: Routes read and write traffic to respective microservices.
• Review Ingestion Service: Validates incoming user reviews and writes raw data to Cassandra and Kafka simultaneously (or uses CDC from Cassandra to Kafka).
• External Ingestion Workers: Background jobs that poll IMDb/Rotten Tomatoes, normalize the scores, and publish to Kafka.
• Message Broker (Kafka): The highly durable, partitioned buffer receiving the firehose of raw review events.
• Stream Processor (Apache Flink): Maintains in-memory state to continuously calculate rolling averages and emits an "Upsert" stream.
• Primary DB (Cassandra): The infinite, append-only ledger storing the raw text and metadata of every single review.
• Aggregate DB (PostgreSQL): Stores movie metadata and the single-row, continuously updated global averages.
• Search Engine (Elasticsearch): Populated via Debezium (CDC) from both PostgreSQL (for averages/metadata) and Cassandra (for review text text-matching).

Detailed Component Design

Here is the refined mechanics of the real-time aggregation pipeline.

1. Continuous Stateful Aggregation (Apache Flink) We abandon time-based windows entirely. Flink consumes the movie_reviews_raw Kafka topic and utilizes Managed State.

• Flink holds a Dynamic Table in memory tracking currentTotalScore and totalReviewCount for every movieId.
• The moment a new review event arrives, Flink retrieves the state for that movie, adds the new score, increments the count, and recalculates the average.
• Flink immediately emits a changelog event containing the fresh average.

2. The JDBC Upsert Sink Flink's output is routed directly to the PostgreSQL Aggregate DB using a JDBC connector configured in Upsert mode.

• Instead of appending rows, Flink executes an INSERT ... ON CONFLICT DO UPDATE SQL command against the movieId primary key.
• This ensures PostgreSQL only ever holds one row per movie, remaining incredibly lean and fast, while constantly reflecting the most up-to-date mathematical average.

3. Read Path and Search (Elasticsearch) When a user searches or filters movies, they never touch PostgreSQL or Cassandra.

• Debezium monitors the PostgreSQL WAL (Write-Ahead Log). Every time Flink upserts a new average, Debezium pushes that change to Elasticsearch.
• Elasticsearch handles the complex multi-parameter filtering (genre=Sci-Fi AND rating > 8.0) using its optimized inverted indices, easily meeting the < 50ms latency requirement.