Introduction

Kafka replaces REST for microservice communication when you need asynchronous, event-driven messaging instead of synchronous request-response calls. REST works well for simple query/command interactions where the caller needs an immediate response. Kafka works better when services need to be decoupled, when event ordering matters, when throughput requirements are high, or when multiple downstream consumers need to react to the same event independently.

The choice is not binary. Most production architectures use both: REST for synchronous operations like user-facing API calls, and Kafka for asynchronous workflows like order processing, analytics pipelines, and cross-service event propagation.

How REST Communication Works Between Services

In a REST-based architecture, Service A sends an HTTP request to Service B and blocks until it receives a response:

1# Order service calls Inventory service via REST
2import requests
3
4def place_order(order):
5    # Synchronous call -- blocks until inventory responds
6    response = requests.post(
7        "http://inventory-service/api/reserve",
8        json={"sku": order["sku"], "quantity": order["quantity"]},
9        timeout=5
10    )
11    if response.status_code == 200:
12        # Then call Shipping service
13        requests.post(
14            "http://shipping-service/api/schedule",
15            json={"order_id": order["id"]},
16            timeout=5
17        )
18        # Then call Notification service
19        requests.post(
20            "http://notification-service/api/send",
21            json={"user_id": order["user_id"], "message": "Order placed"},
22            timeout=5
23        )
24    return response.json()

This creates tight coupling: the Order service must know the URL of every downstream service, it blocks on each call sequentially, and if any downstream service is slow or down, the entire order placement fails or times out.

How Kafka Communication Works Between Services

With Kafka, Service A publishes an event to a topic. Any number of downstream services subscribe to that topic and process the event independently:

1# Order service publishes an event to Kafka
2from kafka import KafkaProducer
3import json
4
5producer = KafkaProducer(
6    bootstrap_servers=["kafka:9092"],
7    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
8)
9
10def place_order(order):
11    # Non-blocking -- fire and move on
12    producer.send("order-events", value={
13        "event_type": "order_placed",
14        "order_id": order["id"],
15        "sku": order["sku"],
16        "quantity": order["quantity"],
17        "user_id": order["user_id"],
18    })
19    producer.flush()
20    return {"status": "accepted"}

Each downstream service runs its own consumer:

1# Inventory service consumes order events
2from kafka import KafkaConsumer
3import json
4
5consumer = KafkaConsumer(
6    "order-events",
7    bootstrap_servers=["kafka:9092"],
8    group_id="inventory-service",
9    value_deserializer=lambda m: json.loads(m.decode("utf-8")),
10)
11
12for message in consumer:
13    event = message.value
14    if event["event_type"] == "order_placed":
15        reserve_inventory(event["sku"], event["quantity"])

The Order service does not know or care how many consumers exist. Inventory, Shipping, and Notification services all consume from the same topic independently.

Detailed Comparison

When REST Is the Better Choice

REST remains the right tool in several situations:

Synchronous query/response: When a client needs data immediately, like loading a user profile or checking account balance. The caller cannot proceed without the response.

Simple CRUD operations: For basic create-read-update-delete flows, REST's simplicity and mature tooling (OpenAPI specs, code generators, API gateways) outweigh Kafka's overhead.

External APIs: Public-facing APIs almost always use REST or GraphQL. External consumers should not need to connect to your Kafka cluster.

Low volume, low complexity: If you have five services handling a few hundred requests per second, Kafka's operational cost (ZooKeeper/KRaft, broker management, schema registry) is hard to justify.

When Kafka Is the Better Choice

Event-driven workflows: Order processing, payment settlement, fraud detection. Multiple services react to the same event independently without the producer coordinating with each one.

High throughput data pipelines: Log aggregation, clickstream processing, metric collection. Kafka handles millions of events per second with horizontal scaling.

Decoupling for resilience: If Service B goes down, Kafka retains the messages. When Service B comes back, it processes the backlog. REST would have returned errors during the downtime.

Event sourcing and replay: Kafka's durable log allows consumers to rewind to any point and reprocess events. This is valuable for rebuilding read models, debugging, or onboarding new services that need historical data.

Fan-out to multiple consumers: One event triggers reactions in five different services. With REST, the producer makes five HTTP calls. With Kafka, it publishes once and each consumer group reads independently.

Architecture Patterns

CQRS with Kafka

A common pattern separates write operations (commands) from read operations (queries). Commands go through REST to a write service, which publishes domain events to Kafka. Read services consume those events and build optimized read models:

Client -> REST -> Write Service -> Kafka Topic -> Read Service A
                                               -> Read Service B
                                               -> Analytics Service

Saga Pattern for Distributed Transactions

Long-running business processes span multiple services. Each service listens for events and publishes its own events to advance the workflow:

1order-events:    OrderPlaced
2payment-events:  PaymentProcessed | PaymentFailed
3inventory-events: InventoryReserved | InventoryInsufficient
4shipping-events: ShipmentScheduled

Each service owns its step and publishes the result. Compensating events handle rollback when a step fails.

Operational Considerations

Running Kafka in production adds operational complexity that REST does not have:

• Broker management: Kafka clusters need monitoring, scaling, and upgrades. Managed services (Confluent Cloud, AWS MSK, Redpanda Cloud) reduce this burden.
• Schema evolution: As event schemas change, producers and consumers must stay compatible. Schema Registry with Avro or Protobuf enforces compatibility.
• Consumer lag monitoring: If a consumer falls behind, its lag grows. Alerting on consumer lag is critical to detect processing bottlenecks.
• Exactly-once semantics: Kafka supports exactly-once processing, but configuring it correctly (idempotent producers, transactional consumers) requires careful setup.

Common Pitfalls

• Replacing all REST calls with Kafka. Synchronous user-facing queries still belong behind REST endpoints. Not every interaction is an event.
• Ignoring consumer lag monitoring. A consumer that falls hours behind can cause stale data, duplicate processing, or disk pressure from retained messages.
• Publishing events without a schema registry. Schema drift between producer and consumer causes silent data corruption or deserialization failures.
• Using Kafka as a database. Kafka is a durable log, not a query engine. If services need to look up current state by key, use a proper database and project events into it.
• Treating event ordering as global. Kafka guarantees ordering within a partition, not across partitions. If two events must be processed in order, they must share the same partition key.

Summary

• REST is synchronous request-response. Kafka is asynchronous publish-subscribe. Most architectures use both.
• Choose REST for user-facing queries, simple CRUD, external APIs, and low-volume interactions.
• Choose Kafka for event-driven workflows, high-throughput pipelines, service decoupling, and fan-out to multiple consumers.
• Kafka adds operational complexity (brokers, schema registry, consumer lag monitoring) that must be justified by the architectural benefits.
• The partition key determines ordering guarantees. Events that must be processed in sequence need the same partition key.