Design a Code Deployment System

Code deployment looks simple from the outside, push code, run tests, ship to production. But behind that simplicity is a system that must safely coordinate builds across hundreds of workers, distribute multi-gigabyte binaries to machines across the globe, and automatically recover when something goes wrong mid-rollout. A bad deployment at scale does not just cause a bug, it takes down the entire service for every user simultaneously.

The key insight for framing this problem: a code deployment system is really two separate subsystems with fundamentally different challenges. The Build System turns source code into deployable artifacts (binaries or container images). The Deployment System distributes those artifacts to target machines and manages the rollout. These subsystems have different scaling characteristics, failure modes, and latency requirements.

Functional

Traffic

A mid-size engineering organization runs roughly 100 deployments per day across all environments (dev, staging, production). During release windows (typically mornings and early afternoons), deployments spike to 20-30 per hour.

Each deployment triggers a build. With 100 builds per day and each build taking up to 15 minutes, the Build System needs enough worker capacity to handle 10-15 concurrent builds during peak hours. A pool of 100 workers provides comfortable headroom and handles burst scenarios (e.g., a major release train where 50 teams ship simultaneously).

Create Build

POST /api/v1/builds

Request body:

Why Multiple Storage Systems

A code deployment system has three distinct data types that each need different storage characteristics. Trying to store everything in a single database creates painful compromises, build logs do not belong in PostgreSQL, and deployment metadata does not belong in S3.

PostgreSQL: Deployment Metadata

The architecture splits cleanly into the two subsystems identified in requirements: the Build System and the Deployment System, connected by artifact storage.

Build System

Build Flow

1. Developer pushes a commit to the Git repository (e.g., GitHub).
2. GitHub sends a webhook to the Build Service with the commit SHA, repository name, and branch.
3. Build Service validates the webhook signature, creates a new record in the builds table with status QUEUED, and returns the build ID to the webhook response.
4. A worker polls the builds table. It runs an ACID transaction: SELECT the oldest QUEUED job, UPDATE its status to BUILDING and set worker_id. If two workers race for the same job, only one transaction commits, the other retries and picks the next available job.
5. The worker fetches the source code from the Git repository at the specified commit SHA.
6. The worker compiles the code into a binary or container image. This takes up to 15 minutes. During this time, the worker updates the last_heartbeat column every 3 minutes.
7. On success, the worker uploads the artifact to S3 using the naming convention builds/{repository}/{build_id}/{artifact_name}.
8. The worker updates the builds table: status = SUCCEEDED, artifact_url = the S3 path.
9. If the build fails, the worker sets status = FAILED with the failure reason. The job is not retried automatically, build failures are usually code errors, not infrastructure errors.
10. A separate heartbeat monitor service polls for BUILDING jobs whose last_heartbeat is older than 10 minutes. These are likely from crashed workers. The monitor resets their status to QUEUED so another worker picks them up.

This section dives deep into the core engineering challenges: how to build a reliable job queue, how to distribute artifacts globally, and how to automatically detect and rollback failed deployments: the GATE topic for this problem.

SQL Job Queue: Reliable Build Processing

The Build System's job queue is implemented as a PostgreSQL table rather than an in-memory queue or a separate message broker. Why? Because a build job represents a commitment, when a developer pushes code, they expect it to be built. Losing a job because a server restarted is unacceptable.

Canary vs Rolling vs Blue-Green

Rolling updates are the simplest strategy. Update machines in batches, health check between each batch. Pros: straightforward to implement, minimal infrastructure overhead. Cons: each batch serves production traffic immediately, a bug affects real users in the first batch before health checks catch it.

Canary releases add a metrics evaluation layer. Route a small percentage of traffic to the new version, compare metrics against the baseline, and expand gradually. Pros: statistical detection of subtle regressions (latency increases, error rate spikes) that health checks miss. Cons: requires a metrics pipeline, traffic splitting infrastructure, and careful threshold tuning.

Mid-Rollout Failure

A rolling deployment is updating batch 3 of 10 when health checks fail. The service is now in a mixed state: batches 1-2 are running the new version, batches 3-10 are running the old version, and batch 3 has unhealthy machines.

Mitigation: The regional agent immediately stops the rollout and reverts batches 1-3 to the previous version. Batches 4-10 were never touched. The rollback follows the same rolling strategy in reverse, update one batch at a time with health checks. Within 5 minutes, all machines are back on the previous version. The deployment is marked FAILED with logs indicating which batch failed and why.

Feature Flags

Decouple deployment from release. Feature flags let you deploy code that is hidden behind a toggle, then enable it gradually for specific user segments without a new deployment. This separates the risk of deploying code (infrastructure risk) from the risk of releasing features (product risk). A deployment that breaks the service requires rollback; a feature that underperforms can be toggled off in seconds. LaunchDarkly, Split, and open-source solutions like Unleash provide this capability.

Blue-Green Deployments