Distributed Systems
Openstack
Kubernetes
RPC
Messaging Systems

Messaging vs RPC in a distributed system (Openstack vs K8s/Swarm)

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

In modern distributed systems, communication plays a fundamental role in ensuring robust and efficient operations. Among the numerous methods of managing communication across disparate services and platforms, Messaging and Remote Procedure Calls (RPC) stand out. Here we will delve into these two key techniques, exemplified by their usage in prominent software orchestration frameworks: OpenStack employing mostly RPC and Kubernetes/Swarm using Messaging.

Messaging vs. RPC

Messaging and RPC are two approaches to handle inter-process communication in distributed systems, but they cater to different needs and scenarios.

Remote Procedure Calls (RPC): RPC allows a program to cause a procedure to execute in another address space (commonly on another computer on a shared network), which appears very much as though it were a local procedure call, without the programmer explicitly coding the details for the remote interaction. The key idea is that it abstracts the communication, making the distributed service appear as a local one.

Messaging: Messaging involves the exchange of messages (structured data or information) between systems or components of a system. This communication is usually asynchronous, meaning that the sender and receiver do not need to interact with the message queue at the same time. Examples of messaging systems include message queues, pub/sub systems, and message-brokers.

Technical Explorations

OpenStack: OpenStack, a cloud computing platform, heavily utilizes RPC for its internal component interactions. For instance, when the Nova component (which manages compute tasks) needs to interact with the Neutron component (which manages networking tasks), it typically does so via RPC calls. OpenStack services communicate over AMQP (Advanced Message Queuing Protocol), using a combination of direct RPC calls and message passing techniques, largely depending on oslo.messaging library which supports both RPC and messaging patterns.

Example of an RPC operation in OpenStack:

  • A user requests a VM launch via the OpenStack API.
  • The API service (nova-api) makes an RPC call to the scheduler service (nova-scheduler) to find a suitable host.
  • The scheduler responds back with the selection, and the API service makes another RPC call to the compute node (nova-compute) to build the instance.

Kubernetes/Swarm: Both Kubernetes and Docker Swarm leverage a more message-driven approach, particularly useful in orchestrating containers across a distributed set of nodes. Kubernetes, for example, uses etcd as a consistent and highly-available key value store used to store all cluster data, including the runtime state of cluster components. Communication between components like the Scheduler, API Server, and various controllers often happens via HTTP REST calls and watches on etcd keys, which provide a messaging-like paradigm.

Example in Kubernetes:

  • The Scheduler watches for new Pods that have no assigned node.
  • Once a new Pod appears in the API server, a message (change notification) is sent to the Scheduler.
  • The Scheduler selects a suitable node and sends a command to the Node agent (kubelet) to start the Pod.

Comparison Table

FeatureOpenStack (RPC Oriented)Kubernetes/Docker Swarm (Messaging Oriented)
CommunicationDirect, synchronous callsAsynchronous, event-driven interactions
ComplexityHigher complexity in setupSimpler, scalable data flow
ScalabilityScales with difficultyNaturally scalable with loose coupling
Fault ToleranceDepends on the RPC implementationHigher due to decoupled nature
Real-Time OpsFeasible but harder to manageEasier due to asynchronous operations

Subtopics and Additional Details

  1. Fault Tolerance & Recovery: RPC, being tightly coupled, often makes handling faults a bit complex. Systems should implement retry logics or sophisticated fault handling mechanisms. Conversely, messaging systems inherently support retries and can delay message processing until the system is ready.
  2. Performance Considerations: While RPC can be faster for small, frequent requests because of its direct nature, messaging systems optimize for larger, less frequent messages and can handle higher loads by nature of asynchronicity.
  3. Use Cases Applicability: RPC suits scenarios where quick, synchronous operations are crucial, such as in financial transactions. Messaging fits scenarios requiring high throughput and scalability, such as in IoT systems or real-time data processing pipelines.

Understanding when to use Messaging or RPC is crucial in the architectural decisions for any distributed system. Each method has its strengths and use cases, and the choice often boils down to specific system requirements and characteristics.


Course illustration
Course illustration

All Rights Reserved.