Messaging vs RPC in a distributed system (Openstack vs K8s/Swarm)
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
In modern distributed systems, communication plays a fundamental role in ensuring robust and efficient operations. Among the numerous methods of managing communication across disparate services and platforms, Messaging and Remote Procedure Calls (RPC) stand out. Here we will delve into these two key techniques, exemplified by their usage in prominent software orchestration frameworks: OpenStack employing mostly RPC and Kubernetes/Swarm using Messaging.
Messaging vs. RPC
Messaging and RPC are two approaches to handle inter-process communication in distributed systems, but they cater to different needs and scenarios.
Remote Procedure Calls (RPC): RPC allows a program to cause a procedure to execute in another address space (commonly on another computer on a shared network), which appears very much as though it were a local procedure call, without the programmer explicitly coding the details for the remote interaction. The key idea is that it abstracts the communication, making the distributed service appear as a local one.
Messaging: Messaging involves the exchange of messages (structured data or information) between systems or components of a system. This communication is usually asynchronous, meaning that the sender and receiver do not need to interact with the message queue at the same time. Examples of messaging systems include message queues, pub/sub systems, and message-brokers.
Technical Explorations
OpenStack: OpenStack, a cloud computing platform, heavily utilizes RPC for its internal component interactions. For instance, when the Nova component (which manages compute tasks) needs to interact with the Neutron component (which manages networking tasks), it typically does so via RPC calls. OpenStack services communicate over AMQP (Advanced Message Queuing Protocol), using a combination of direct RPC calls and message passing techniques, largely depending on oslo.messaging library which supports both RPC and messaging patterns.
Example of an RPC operation in OpenStack:
- A user requests a VM launch via the OpenStack API.
- The API service (nova-api) makes an RPC call to the scheduler service (nova-scheduler) to find a suitable host.
- The scheduler responds back with the selection, and the API service makes another RPC call to the compute node (nova-compute) to build the instance.
Kubernetes/Swarm: Both Kubernetes and Docker Swarm leverage a more message-driven approach, particularly useful in orchestrating containers across a distributed set of nodes. Kubernetes, for example, uses etcd as a consistent and highly-available key value store used to store all cluster data, including the runtime state of cluster components. Communication between components like the Scheduler, API Server, and various controllers often happens via HTTP REST calls and watches on etcd keys, which provide a messaging-like paradigm.
Example in Kubernetes:
- The Scheduler watches for new Pods that have no assigned node.
- Once a new Pod appears in the API server, a message (change notification) is sent to the Scheduler.
- The Scheduler selects a suitable node and sends a command to the Node agent (kubelet) to start the Pod.
Comparison Table
| Feature | OpenStack (RPC Oriented) | Kubernetes/Docker Swarm (Messaging Oriented) |
| Communication | Direct, synchronous calls | Asynchronous, event-driven interactions |
| Complexity | Higher complexity in setup | Simpler, scalable data flow |
| Scalability | Scales with difficulty | Naturally scalable with loose coupling |
| Fault Tolerance | Depends on the RPC implementation | Higher due to decoupled nature |
| Real-Time Ops | Feasible but harder to manage | Easier due to asynchronous operations |
Subtopics and Additional Details
- Fault Tolerance & Recovery: RPC, being tightly coupled, often makes handling faults a bit complex. Systems should implement retry logics or sophisticated fault handling mechanisms. Conversely, messaging systems inherently support retries and can delay message processing until the system is ready.
- Performance Considerations: While RPC can be faster for small, frequent requests because of its direct nature, messaging systems optimize for larger, less frequent messages and can handle higher loads by nature of asynchronicity.
- Use Cases Applicability: RPC suits scenarios where quick, synchronous operations are crucial, such as in financial transactions. Messaging fits scenarios requiring high throughput and scalability, such as in IoT systems or real-time data processing pipelines.
Understanding when to use Messaging or RPC is crucial in the architectural decisions for any distributed system. Each method has its strengths and use cases, and the choice often boils down to specific system requirements and characteristics.

