System requirements

Functional:

* Service provides an API that allows for searching for tweets

* Ability to ingest a tweet

* Ability to search by author, content, filter

* Support for ACDDR and DSAR

Non-Functional:

* <10ms response time

Capacity estimation

* Terrabytes of tweets, maybe even petabytes

* Thousands of TPS

API design

* sendTweet(Tweet)

* getTweets(User) -> PaginatedCollection<Tweet>

* searchTweets(Search) -> PaginatedCollection<Tweet>

* DeleteTweets(User) -> Void

Database design

* We will have a NoSQL database where the primary key is <user:timestamp> -> Tweet

* We will have an elasticsearch index on top

High-level design

We will have a simple CRUD service with a query layer powered by Elasticsearch. Our database will use Event Sourcing.

Request flows

* SendTweet adds an entry to the Database

* getTweets will query a Database using a GSI on timestamp, and enable pagination

* SearchTweets will talk to query view

* DeleteTweets will clear all data from Database and QueryView

Detailed component design

• Database will be a NoSQL table (probably Dynamo) with key structure <user:guid>. Those rows will also have timestamp, actionType, and content. We can store the tweet directly on the table or use an offloading strategy like S3 pointer if we want more.
• QueryView will be an elasticsearch index. The backend will have some hardcoded queries to enable APIs.
• We can implement caching at the CDN, LoadBalancer/ReverseProxy, or Backend level.

Trade offs/Tech choices

* Where implement caches

* What backend query tool to use. Do we want consistency? Maybe look at a tool like Cassandra.

* How do we want to shard data to enable recovery modes

* How do we handle failure modes, e.g. search index breaking

Failure scenarios/bottlenecks

* There are certain scenarios where elasticsearch may drop or appear to drop data. We can use snapshots to mitigate and/or rebuild indexes.To mitigate, we could shard the data and it's clusters.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?