Requirements

Functional Requirements:

• Allow users to post, edit, and delete advertisements with title, description, category, price, and location.
• Allow users to browse and view listings organized by categories and subcategories (jobs, housing, services, for sale, etc.).
• Enable users to search for listings using keywords, and filter by category, price range, location, and date posted.
• Enable user registration and authentication with email/password or social login to post and manage their listings.
• Allow users to upload photos for their listings; store, resize, and serve images via CDN.

Non-Functional Requirements:

• The data should be focused on eventual consistency; users can view new things on the craigslist when they are ready, but some data should always exist.
• The latency should be as low, approximately 100ms.
• The system should be scalable.
• Users' data should be secured.

Capacity Estimation

Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...

API Design

1. POST /ad body: Advertisement (and uploaded picture)- should create a new ad.
2. PUT /ad/{ad_id} body: Advertisement - should update existing ad
3. DELETE /ad/{ad_id} - deletes the ad.
4. GET /ad - Retrieves all ads. May contain paginations to filter the ads (including subcategories).
5. POST /user body: User - Create new user
6. PUT /user body: User - updates User

High-Level Design

Overall, the user's request will go through the balance, towards the right services. If they want user related service like Login, they'll go through UserService. If they want to do anything about ad, like adding, updating, deleting, the request will be sent to the adService. From there, if the user were to upload a picture for the add, the AdService will send the request to the BlobService, where users' images will be stored in a cloud blob storage, like amazon s3. The actual pictures will be stored there, and the the database will return the url to the picture. Now, there will be a cache for the data for some ads; the most recent 100 ads, or most viewed ads will be stored in the cache, it will be stored via CDN. For the User Login service, we could utilize the OAuth2 for google login for higher security, or just have username / password authentication, with salt on the password.

Database Design

We could use the NoSql for the database. This is mainly because there could be many different ways people want their ad to be, and we may need to shift the data structure everytime they want something new. Besides, we could horizontally scale the database. In this case, we may use something like Dynamo DB (Since we would be already utilizing amazon s3 for blob storage), or just MongoDB for noSql.

Detailed Component Design

We may get something like receiving too many ads at once from a single user. In this case, we could have limit of ads user can create per month, similar to token bucket for rate limiting, but for ads. Also, If the user DOES need to have indeed that many ads, he could either request it manually, or pay extra (if a business.)

Now, we may be worried about the user identity. What if a user is a fake person selling fake things? We could do few things. We could have user upload their driver license or respective ID to ensure that this is a right person, which would require us to have third party company that checks the ID. This tradeoff is users' trust on our app vs the cost. I think users' trust outweighs the cost, since more trust means more user on the app.

If there are 2 users selling the same things, that's a problem more on the manual side. Then it would go back to identifying the person correctly, like our 2nd point earlier regarding user identity.

For the scaling of the app, if there are too many users viewing ads, we can always horizontally scale the respective adServices. If there are too many pictures, then we need to increase the amount of blob storage we have. Generally, we don't need to pay for extra blob storage starting at 50 TB, so we won't need to worry about it until we have a lot of users, considering decent 1MB pictures, that still would be 50M pictures.

For the database design, Generally, people that are in a specific area only needs to look at the data in that specific area. If we were to have datacenters, or if we are just utilizing the availability sets (datacenters in different regions provided by AWS), then we could shard the databases based on the location. Now, if we were doing this, then we may consider the option of consistent hashing technique, in order to keep all of the databases updated, as well as replication saving the data just in case a server crashes. Here, we could definitely use DynamoDB, since it supports consistent Hashing anyways.

This leads to, what happens if the app were to be overloaded due to lack of scaling (maybe lack of quota), and people's request to put up or put down the ad has failed? We did want the 100ms latency, so I don't want to utilize a queue here. (Of course, for the blob storage uploading images or other complex tasks can be async.) What we can do is to have some history of people's ads, like an auditingService on the side, which would just save what people have been doing, like transactions.