System requirements

Functional:

The users can capture a screenshot, store it, and send it.

The users can download their own screenshots made earlier

The users can assign tags to a screenshot.

Users can share screenshots with their friends to view them.

The users can change the size of images and its properties before uploading.

The users can remove their screenshots.

The uses can download the platform-dependent client application to capture the screenshot.

Non-Functional:

We should have the client applications for web, mobile, and desktop.

If users upload a new screenshot, it will be available in 100 milliseconds.

The users can upload a screenshot maximum 1 MB.

Users may choose the users with whom they share the screenshots or make a screenshot public.

The platform should be designed to handle a high number of concurrent users efficiently. This might involve implementing techniques like load balancing and caching to distribute requests and minimize response times.

The system architecture should allow for horizontal scaling to accommodate increasing user traffic and data volume. This could involve using distributed databases and cloud-based infrastructure.

User data, including passwords and personal information, should be securely stored using encryption techniques.

The user interface (UI) should be intuitive and user-friendly for both novice and experienced users. A clean and well-organized UI with clear navigation elements will enhance user experience

Capacity estimation

Here's the calculation for storage requirements based on the provided assumptions:

• Users per month: 1,000,000
• Screenshots per month: 1000
• Screenshot size per user (assumed): 1 MB
• Data storage: 1,000,000 users * 1,000 * 1,000,000 = 1 Petabytes
• 365 * 1000000000000000. = 370 Petabytes per year

Therefore, based on the assumptions, the estimated storage requirement for the system would be around 370 Petabytes per year.

API design

Here's a breakdown of the essential API for our Multi-Device Screenshot Capture System.

User Management APIs:

• User registration: allows users to create new accounts and potentially other profile information.
• User login: enables users to authenticate with their credentials and obtain a secure access token for subsequent API calls.
• User Profile Management: provides functionalities for users to update their profile, information, contact details, and potentially manage preferences.

Screenshot Management APIs:

• Get a list of screenshots for a current user: returns information about the screenshots, including id, title, user_id, creation date, tag list, size, URL to download, public or private status, and if private, the user_id list and who can download the screenshot. It includes paging information, such as the page number and its size.
• Get the information about a screenshot for the given screenshot_id, including id, title, user_id, creation date, tag list, size, URL to download, public or private status, and if private, the user_id list and who can download the screenshot
• Remove the screenshot for the given screenshot_id
• Download the screenshot for the given screenshot_id and size.
• Upload the new screenshot: using HTTP stream, the current users can upload the screenshot with such information: id, title, user_id, creation date, tag list, size, public or private status, and if private, the user_id list and who can download the screenshot.
• Search the downloaded screenshot by keywords.

Database design

Database Choices:

User :

Database type: SQL (MySQL, PostgreSQL)

Reasoning: user data requires strong consistency and ACID transactions.

Screenshot:

Database type: NoSQL (DynamoDB)

Reasoning: It may work with eventual consistency.

HashTag and Permission:

Database type: SQL (MySQL)

Reasoning: they require strong consistency and ACID transactions.

Data partitioning strategy:

Reasoning: User Id is a natural partitioning key as users primarily interact with user their data and advertisements. Sharding by User Id distributes load across servers and facilitates horizontal scaling for the user and screenshot data.

Sharding strategy: User Id provides a well-defined partition key for efficient data distribution and horizontal scaling based on user base growth. This approach keeps user data and related screenshots together on the same shards.

High-level design

Here's a breakdown of essential components and services required to build your Multi-Device Screenshot Capture System.

Client application: Many types of client applications for web, mobile, and desktop.

Responsibilities:

• User registration and login
• Make and upload the screenshots with assigned keywords or hashtags
• Download the other screenshots to view with websockets.
• User profile management
• Retrable upload the new screenshots with the websocket connections in the specific formats.

API Gateway: API Gateway software - Apigee, Istio, AWS API Gateway.

Responsibilities:

• Serves as a single entry point for all API requests from the client applications
• Routes requests to the appropriate backend services based on their functionalities.
• Implements authentication and authorization mechanisms to secure access to APIs.

User Service: Manage user activity, including user creation, removal, and searching.

Responsibilities:

• Manages user profile information.
• Interacts with the user database to store and retrieve user data.
• Send an update event to Kafka

Upload service: manage to either upload or download the screenshots by user demands.

Responsibilities:

• Accepts the user's websocket connection to download or upload a screenshot
• Persist the uploaded screenshot to the object storage
• Persist the screenshit information to DynamoDB
• Send the event about the new screenshot to Kafka

Search service: look for the screenshot information by keywords.

Responsibilities:

• Store the keywords and screenshot_id to MySQL
• Search the screenshot_id by a keyword
• Receive the event from Kafka to persist new screenshot_id and keywords (hashtags).

Permission service: manage the user's permission to work with a screenshot.

Responsibilities:

• Find the screenshot_id and read and write boolean flags for the given user_id
• Persists the screenshot_id,user_id, and read, write boolean flags to MySQL
• Receives the event from Kafka to persist the new user and their permissions.

Notification Service: listen to Kafka about new screenshots and connected users, and store that information to Redis.

Responsibilities:

• Receive the user login events and store connection information in Redis
• Connect to the client application to notify about new screenshots to make all user application will be updated.

CDN: store the screenshots closest to clients.

Security layer: Secure communication protocols (HTTPS), authentication, and authorization mechanisms, intrusion detection/prevention systems.

Responsibilities:

• Protects user data and platform integrity from unauthorized access and security threats
• Ensures secure communication channels and data encryption.

Request flows

Detailed component design

Let's delve deeper into the individual components of our

Multi-Device Screenshot Capture system, exploring their

functionalities, scalability considerations, and potential algorithms/data structures.

All services will be deployed to the AWS EKS. Kubernetes manages every service. Kubernetes can autoscale every service based on workload, and it cares about service availability by restarting pods if a service fails.

API Gateway:

Technology stack: Apigee, AWS API Gateway

Functionalities:

• Single point for all API requests from the client applications
• Routes requests to the appropriate backend services based on functionalities
• Implements an authentication and authorization mechanism for security

Scalability: API gateways are generally designed to handle high volumes of requests. They can be scaled horizontally by adding more instances behind a load balancer.

Algorithms/Data structures: Routing algorithms like path-based routing or content-based routing can be used to efficiently direct requests to backed services.

Client application:

Technology stack: Kotlin, React, or C++.

Functionalities:

• Make the screenshot and store it locally in different image format
• Split the screenshot into equal blocks to transfer it to the Upload service reliably.
• Synchronizes the client application with user screenshot list made at the different client applications.
• Download and view the screenshots of other users.

Scalability: Use multithreads to optimize the client application execution.

Algorithms/Data structures: Split the screenshot into equal blocks to transfer it to the Upload service reliably. If one block fails to transfer, the client application retries to transfer.

Before downloading of a screenshot, the client application requests a block hash list to compare with what the client application already has.

Messaging service for real-time streaming of messages:

Optimizing the messaging service for real-time streaming of messages with reliable delivery and message ordering in a distributed system requires a multi-faceted approach. Here's how you can achieve this:

Technology Stack:

• Message Queueing System: Utilize a robust message queuing system like Apache Kafka or RabbitMQ. These systems offer high throughput, low latency, and message persistence for reliable delivery.

Message Delivery Guarantees:

• Configure Reliable Delivery: Ensure the message queuing system delivers messages at least once (ideally exactly once). This can be achieved through producer confirmations and consumer acknowledgments.
• At-Least-Once Delivery: Implement mechanisms to handle potential duplicate messages on the receiving end. Deduplication strategies like message identifiers and idempotent operations can be used.

Message Ordering:

• Partitioned Queues: Partition the message queue based on the conversation thread (e.g., user_id). This ensures messages within a conversation are delivered in the correct order within each partition.
• Ordering Guarantees within Partitions: Depending on your messaging system, additional configuration might be needed to guarantee strict ordering within partitions. Kafka offers message ordering guarantees within partitions.

Scalability:

• Horizontal Scaling: Scale the message queuing system and database horizontally by adding more nodes to handle increased message volume and user base growth.

Real-time Streaming:

• Consumers: Implement message consumers that actively listen for new messages within their assigned queues. Libraries like Kafka consumers can be used for efficient message pulling from topics.

Upload service: process the request to upload or download the screenshots.

Technology stack: Kotlin, Java, Go.

Functionalities:

• Process the upload requests of screenshots
• Process the download request for screenshots
• Sends the screenshot events to Kafka
• Persist the information about the new screenshot to MySQL
• Persist the screenshots to the Object storage

Scalability: Use Kubernetes to scale the Upload service based on workload.

Algorithms/Data structures: Use the data block transfer: split the screenshots by equal block to transfer it and it makes it easier to repeat the transfer of one piece of a screenshot not the whole screenshot file.

Trade offs/Tech choices

• Using Kubernetes gets rids of thinking how scale the services and its lifecycles.
• Using microservices and the DDD approach gives agility, scalability, resilience, technology flexibility, and easier maintenance
• Eventual consistency over strong gives high availability, better scalability, and faster performance.

Failure scenarios/bottlenecks

The client application: if it fails, it may lose the last saved screenshot if the client application doesn't have time to persist the screenshot locally. It's a critical issue.

Solution: persist a screenshot immediately before uploading it to the backend.

The API Gateway: if it's down, the entire system cannot process user requests.

Solution: Deploy the service to different zones and replicate the data between them.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?