Choreography vs. Orchestration — The difference you only see after implementing distributed transactions yourself

When I first designed microservices, the question "how do you keep transactions consistent if services don't share a DB?" haunted me for quite a while. It turned out that question itself is precisely the reason the Saga pattern exists.

The Saga pattern is a design pattern that decomposes a distributed transaction into a sequence of local transactions, attaching a compensating transaction to each step that rolls back to the previous state on failure. The core of this article is that there are two ways to implement a Saga: Choreography and Orchestration — both can be called "implementing a Saga," but the actual choice produces very different outcomes depending on your team's situation and workflow complexity.

In this article, I'll walk through how each approach works, the differences at the code level, and the concrete criteria used in practice as of 2026 to choose between them.

Why Saga — Starting with the Limitations of 2PC

Why 2PC Doesn't Work in Microservices

2PC (Two-Phase Commit), the traditional distributed transaction solution, blocks all involved resources during the Prepare phase. Every service must be simultaneously available, and if the coordinator fails, the entire system deadlocks. In a microservices environment, assuming dozens of services are all perfectly available at the same time is simply unrealistic.

Saga is an approach that guarantees Eventual Consistency without this constraint. In exchange for giving up perfect ACID, it defines a compensating transaction at each step to handle failures gracefully.

As shown above, when the inventory deduction step fails, a compensating transaction reverses the already-completed payment processing. Whether "who directs" this flow determines the split between Choreography and Orchestration.

How the Two Approaches Work

Choreography — Events Flow on Their Own

Choreography has no central conductor. Each service publishes an event when it finishes its work, and other interested services subscribe to that event and perform the next step.

The diagram reflects that Kafka is not a push broker but a pull-based system where consumers poll for messages. This is a point beginners often confuse, so it's worth making explicit.

In code, it looks like this. Each service only needs to handle the events it cares about.

// Payment Service — subscribes to OrderPlaced event, processes payment, publishes event
@KafkaListener(topics = "order-placed", groupId = "payment-service")
public void handleOrderPlaced(OrderPlacedEvent event) {
    try {
        paymentService.processPayment(event.getOrderId(), event.getAmount());
        kafkaTemplate.send("payment-processed", new PaymentProcessedEvent(event.getOrderId()));
    } catch (PaymentFailedException e) {
        kafkaTemplate.send("payment-failed", new PaymentFailedEvent(event.getOrderId()));
    }
}
 
// Inventory Service — subscribes to PaymentProcessed event, deducts inventory
@KafkaListener(topics = "payment-processed", groupId = "inventory-service")
public void handlePaymentProcessed(PaymentProcessedEvent event) {
    try {
        inventoryService.reserve(event.getOrderId());
        kafkaTemplate.send("inventory-reserved", new InventoryReservedEvent(event.getOrderId()));
    } catch (InsufficientStockException e) {
        kafkaTemplate.send("inventory-failed", new InventoryFailedEvent(event.getOrderId()));
    }
}
 
// Payment Service — subscribes to inventory failure event, handles compensation
@KafkaListener(topics = "inventory-failed", groupId = "payment-service")
public void handleInventoryFailed(InventoryFailedEvent event) {
    paymentService.refund(event.getOrderId()); // idempotency is mandatory
}

Orchestration — A Conductor Coordinates Everything

Orchestration has a central orchestrator that directly instructs each service on what to execute. The orchestrator tracks the entire workflow state and issues compensation commands on failure.

Here is a conceptual example of orchestration using Temporal. One activity function definition is included alongside the workflow so the role of the import becomes clear.

# Temporal workflow — conceptual example (based on Temporal Python SDK)
from datetime import timedelta
from temporalio import workflow, activity
from temporalio.common import RetryPolicy
 
 
@activity.defn
async def process_payment(order_id: str, amount: float) -> None:
    # Actual payment gateway call point (conceptual example)
    ...
 
 
@workflow.defn
class OrderSagaWorkflow:
    @workflow.run
    async def run(self, order_id: str, amount: float) -> str:
        payment_completed = False
        inventory_reserved = False
 
        try:
            await workflow.execute_activity(
                process_payment,
                args=[order_id, amount],
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=RetryPolicy(maximum_attempts=3),
            )
            payment_completed = True
 
            await workflow.execute_activity(
                reserve_inventory,
                args=[order_id],
                start_to_close_timeout=timedelta(seconds=30),
            )
            inventory_reserved = True
 
            await workflow.execute_activity(
                schedule_shipment,
                args=[order_id],
                start_to_close_timeout=timedelta(minutes=5),
            )
 
            return "ORDER_COMPLETED"
 
        except Exception:
            # Compensate in reverse order from completed steps
            if inventory_reserved:
                await workflow.execute_activity(release_inventory, args=[order_id])
            if payment_completed:
                await workflow.execute_activity(refund_payment, args=[order_id])
            return "ORDER_FAILED"

This is why Temporal stands out. Retry policies, timeouts, and durable execution state are all managed by the platform. Even if the server dies mid-execution, the workflow resumes from where it left off.

Three Things That Must Follow Any Code — Outbox, DLQ, and Observability

After looking at the two code examples, the natural question is "can I take this straight to production?" The answer is "no," for three reasons.

Transactional Outbox

This pattern atomically handles local DB transactions and event publishing together. If paymentService.processPayment() in the Choreography example above succeeds on the DB commit but kafkaTemplate.send() then fails due to a network error, the DB state and the event stream fall out of sync. The solution is to commit the event to an Outbox table within the same transaction, and have a CDC-based tool like Debezium or a Polling Publisher read that table and move the records to Kafka.

Dead Letter Queue

Without a path to isolate and later reprocess events that keep failing, manual recovery during a production incident becomes a nightmare. Messages that exceed the retry limit should be routed to a DLQ, where a separate console lets you decide to reprocess, discard, or intervene manually.

Observability

This is especially costly to add retroactively in Choreography. If you embed OpenTelemetry-based distributed tracing from the first design stage and propagate saga_id as shared context across all events and logs, you can connect the distributed flow into a single trace. In Orchestration, the orchestrator naturally centralizes state, but that doesn't automatically connect the internal traces within each service, so it's equally necessary there.

Trade-offs — An Honest Comparison

At first I thought "isn't Choreography more true to the microservices spirit?" — loose coupling, service autonomy... textbook-correct, but the moment you get the question "what step is order #12345 at right now?" in production, you feel Choreography's limits. Saga state is scattered across multiple services and hard to see at a glance.

When each is appropriate — step count is just the first signal. The table below is a starting reference point; the final judgment also requires looking at the complexity of branching conditions, your team's experience operating event brokers, and audit requirements. In practice, a 3-step workflow with tangled conditional compensation logic may be better served by orchestration, and a 6-step linear pipeline can be clean with events.

Common Mistakes

A common mistake in Choreography — misunderstanding Kafka ordering guarantees:

Using saga_id or order_id as the message key routes events with the same key to the same partition, guaranteeing ordering within a single partition. Nothing more. During consumer group rebalancing, that ordering guarantee can waver, and ordering between services subscribing to different topics (e.g., Payment Service and Inventory Service each consuming different topics) is not controlled by this approach. If you need global ordering, you must layer on separate sequence management or a state machine.

// Route events with the same orderId to the same partition
kafkaTemplate.send(
    new ProducerRecord<>("order-events", orderId, event) // orderId is the partition key
);

A common mistake in Orchestration — Fat Orchestrator:

When the orchestrator starts absorbing domain logic, it grows increasingly bloated. The orchestrator should only know "what, and in what order" — "how" must remain the responsibility of each service.

A common mistake common to both — treating idempotency as an afterthought:

Idempotency is not optional. A compensating transaction must return the same result no matter how many times it is executed. "Refund $50" run twice should still result in only $50 refunded. If you don't build this in from the start, fixing it later requires touching event handlers, compensation logic, and storage schemas together — the scope of changes balloons.

// Idempotency example — skip if already refunded
public void refundPayment(String orderId) {
    if (paymentRepository.isRefunded(orderId)) {
        log.info("Already refunded for order: {}", orderId);
        return;
    }
    paymentGateway.refund(orderId);
    paymentRepository.markAsRefunded(orderId);
}

How It's Used in the Real World

The cases below are compiled from technical blog posts and conference talks published by each company. Specific internal contracts such as event names include some second-hand citations, so for exact payload structures, it's best to consult each organization's official materials.

Cases known to use Choreography:

• Uber is known to structure a significant portion of its real-time ride-hailing flow event-driven, with state-change events like arrival and boarding autonomously triggering downstream actions such as notifications and fare calculation. Independent per-service scale-out is cited as the key reason.
• Amazon has presented in multiple talks how it broadcasts inventory-domain events so that various downstream systems can dynamically adjust inventory levels.

Cases known to use Orchestration:

• Netflix has publicly discussed applying workflow-engine-based orchestration to video encoding and media processing workflows. Temporal and its own workflow systems (including Conductor-lineage tools) have coexisted within the organization.
• In e-commerce order processing, workflows of 5 or more steps — create order → payment → deduct inventory → schedule shipment → notify customer — frequently opt for orchestration. Managed platforms like Temporal and AWS Step Functions have substantially addressed the "single point of failure" concern for the orchestrator itself, making it less of a worry than it once was.

Hybrid Strategy — Mixing Both Is the Pragmatic Reality

In practice, more teams mix the two — using orchestration for critical workflows and slipping choreography in for side flows — than teams that commit to a single approach. Flows where audit traceability matters, such as payment and order confirmation, are handed to the orchestrator, while side flows with room for reprocessing — notifications, analytics, search indexing — are let through as events.

Criteria for Choosing an Orchestration Tool

As of 2026, Temporal has established itself as the leading choice for the Durable Execution paradigm. Writing workflows in regular code and having execution state survive server restarts or failures makes it a particularly good fit for Saga implementations.

Closing — Lingering Questions and My Practical Recommendation

No matter how well you choose between the two patterns, Saga is no silver bullet. A few problems remain, and the ones I encounter most often in practice are these.

• Lack of isolation. Intermediate Saga state is exposed to other transactions. This means another request can observe an order where payment has succeeded but shipment has not yet been scheduled. This must be defended with status fields (PENDING, CONFIRMED) and application-level locks — neither pattern solves it for you.
• The fundamental difficulty of debugging. Orchestration is much better by centralizing state, but when per-service internal failures, retries, and partial successes get intertwined, reproducing the root cause from traces alone is still hard. After running a Saga in production for a while, whether you have a store that can replay events/commands ultimately determines how long investigations take.

So when someone asks what I'd recommend for a given team right now, my answer is this: If your team has no experience operating a workflow engine and is building its first Saga, start with a managed orchestrator like Temporal for one critical flow. Even if Choreography looks conceptually more elegant, when the first production incident hits, whether you can see the state on a single screen determines whether overtime is measured in hours or days. Conversely, if your team already stably operates Kafka with Outbox, DLQ, and distributed tracing, it's worth trying Choreography on your existing event infrastructure first before introducing a new orchestrator.

Whichever you choose, one thing not worth compromising on is building idempotency, Outbox, DLQ, and distributed tracing in from the start of the design. A Saga without these four looks like it's running fine — until the first incident reveals its true nature.

References

• Microservices.io — Pattern: Saga (Chris Richardson)
• Microsoft Azure — Saga Design Pattern
• AWS Prescriptive Guidance — Saga Choreography Pattern
• AWS Prescriptive Guidance — Implement the Serverless Saga Pattern by using AWS Step Functions
• Temporal — Mastering Saga Patterns for Distributed Transactions in Microservices
• ByteByteGo — Saga Pattern Demystified: Orchestration vs Choreography
• DEV Community — Saga Orchestration vs. Choreography: Making the Right Trade-off in Event-Driven Systems
• Conduktor — Saga Pattern for Microservices Explained
• Abstract Algorithms — Microservices Data Patterns: Saga, Transactional Outbox, CQRS, and Event Sourcing
• Donnchadh.dev — Orchestration vs. Choreography in Saga Patterns: A Detailed Comparison