Requirements

Functional Requirements:

• Support users reserve a number of VMs based on user specified criteria (CPU, memory)
• Support users releasing the VMs
• Support VM reservation on a schedule
• Extended capabilities
  • Support priority for reservation

Non-Functional Requirements:

• Strong consistency (VMs cannot be double booked)
• High availability with minimal latency
• Reservation latency depends up to 30 seconds is okay)
• Fault tolerant and highly scalable
• Extended capabilities
  • Ensure fairness among users
  • Enable timeout for reservations

Capacity estimation

• 100k servers, 10k users
• Each server can host 5 VMs
• Each user makes 1 reservation per day for 5 VMs on average, and checks their reservation 10 times
• Server information: 100k * 1k = 100 MB, VM information 500MB
• Reservation/release 0.1 rps, get request 1 rps
• Assume reservation are 1k each, each day 10k * 1 * 1kB = 10MB, in one year it is 3GB.

Database

• The data size and qps are quite small and can be fit into one ACID database, without the need for sharding.

DB schema

• Server
  • id
  • location
  • cpu
  • memory
  • network
  • ...
• VM
  • id
  • serverId
  • cpu
  • memory
  • accelerators
  • ...
  • state // reserved | available
• Reservations
  • id
  • ownerId
  • [vmid]

API Design

// Reservation

reserve(spec) -> Reservation

{

id, // output

cpu: { atleast: 4 }

memory: { at least: 16g }

count: 4,

accelerator: H100

...

}

// Release

release(id) -> HTTP status

High-Level Design

We use an ACID database to persist server, VM information and reservations. This can be fronted with a cache, but when making reservations we do that in a database transaction to ensure strong consistency.

Since reservation may take up to minutes if new servers/VMs needs to be created, the services sends the request to a queue, where the request is picked up by the server pool is satisfy the request. When the request is satisfied, the result (e.g. new VM ids) are written back to the DB.

The monitoring service serves 2 purposes:

• If servers or VMs are down (e.g. no heartbeat, it will need to mark that server/VM as unavailable), and replace those servers/VMs with new ones in the pool
• It should inspect the metrics reported by the server/VMs, and reconcile that with what's in the DB. In case there are irreconcilable differences (e.g. server CPU # is different), it should notify ops to take a look

Detailed Component Design

For metrics reporting, we can use a pull or push based model. We choose a push based model, buffered by an EventQueue, to support higher throughput of the metrics. The qps (assume each server/VM reports 1 / sec) is 600k / sec. These data can be sent to a TSDB for real-time visualization/alerting. However, for the reservation system we only care about static attributes (e.g. # of CPUs). We might want to analyze the dynamic data to allow over-allocation, for example.

To find and allocation VMs, we need to define the objective we are optimizing for. For example, it could be the lowest cost, or VM collocation on the same and nearby servers (bin-packing to save resources, vs. round robin allocation to spread out the load, best fit, first fit). Such objective can be configurable.

ReservationService is stateless and horizontally scalable. The DB and cache sizes are small enough to fit into a single cluster. We should enable replicas and hot standbys for high availability. The EventQueue needs partitioning (e.g. Kafka with 4 partitions so each partition handles 150k message/sec). If there is a sudden surge of events, the Kafka queue acts as a buffer.

When there is a reservation spike, we can address it in several ways:

• Have a priority system where more urgent requests are satisfied, possibly interrupting lower priority jobs
• Have a quota system (e.g. credits) for each user to ensure faireless
• Have a feature that processes the reservation asynchronously in the background