Loading...
Functional Requirements:
Non-Functional Requirements:
To ensure our build system can handle the expected load, we need to estimate the number of workers required based on build duration and daily build volume.
Worker Capacity:
Daily Build Volume:
Worker Estimation:
5,000 builds / 96 builds per worker = ~52 workers
10,000 builds / 96 builds per worker = ~104 workers
Thus, we would need between 50 to 100 workers to handle the expected load, validating our initial assumption of 100 workers.
Scalability:
Define what APIs are expected from the system...
In our design, the storage of binaries plays a critical role in ensuring that build artifacts are available for deployment across our global regions. Here's how storage fits into our queueing system:
To effectively design our system, we need to break it down into major components. Our system can be divided into two clear subsystems:
These subsystems will each contain multiple components, but this division provides a straightforward initial approach.
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
At a high level, the process of building code into a binary can be referred to as a job. The build system will be designed as a queue of jobs. Each job includes a commit identifier (commit SHA) for the code version to build and the name of the resulting binary artifact. The specifics of the code-building process are abstracted, allowing us to focus on system infrastructure.
A pool of servers (workers) will handle these jobs. Workers will dequeue jobs in a FIFO manner, build the binaries, and store the resulting binaries in blob storage (e.g., Google Cloud Storage or Amazon S3).
An in-memory implementation of the job queue is problematic because server failures would result in the loss of all queued and past jobs. Instead, we will implement the queue using a SQL database.
The jobs table in our SQL database will have the following schema:
We will implement the dequeuing mechanism by querying the oldest job with a QUEUED status. This requires indexing the table on both created_at and status.
Using ACID transactions will ensure that multiple workers can safely dequeue jobs without running the same job twice. The transaction for dequeuing a job will be:
BEGIN TRANSACTION;
SELECT * FROM jobs_table WHERE status = 'QUEUED' ORDER BY created_at ASC LIMIT 1;
-- If no job is found, ROLLBACK;
UPDATE jobs_table SET status = 'RUNNING' WHERE id = id from previous query;
COMMIT;
Workers will run this transaction periodically (e.g., every 5 seconds). With 100 workers, we would have 20 reads per second, easily manageable by a SQL database.
To handle potential lost jobs due to worker failures or network issues, we add a last_heartbeat column to the jobs table. Workers update this column every 3-5 minutes to indicate the job is still running.
A separate service will poll the table periodically (e.g., every 5 minutes). If a RUNNING job's last_heartbeat is older than a threshold (e.g., 10 minutes), the service will reset the job's status to QUEUED:
UPDATE jobs_table SET status = 'QUEUED' WHERE
status = 'RUNNING' AND
last_heartbeat < NOW() - INTERVAL '10 minutes';
The deployment system is responsible for distributing the built binaries to the target machines across various regions. It includes the following components:
General Overview:
Our deployment system needs to enable the rapid distribution of 10GB binaries to hundreds of thousands of machines across various global regions. The system will include:
1. Replication-Status Service:
replication_status is "complete," it is officially deployable.2. Blob Distribution:
3. Deployment Trigger:
This design ensures a robust and efficient deployment system capable of handling large-scale binary distributions globally. By leveraging regional clusters, peer-to-peer networks, and a goal-state oriented approach, we achieve high performance, scalability, and reliability.
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?