Reliable Multi-Service Workflows: Dual Writes, Outboxes, Sagas and Reconciliation

A business operation often begins as one simple sentence: place the order, reserve the stock, collect the payment, or activate the account. In a distributed system, that sentence can cross several services and several independently committed data stores.

That is where correctness becomes difficult. A local database transaction can guarantee that one service either commits all of its changes or commits none of them. It cannot guarantee that another service, another database, a message broker, or an external API commits at the same moment.

The result is not merely a messaging problem. It is a multi-resource consistency problem.

This article develops a practical model for handling it. The model combines local transactions, transactional outboxes, idempotent consumers, explicit workflow state, compensation and reconciliation. None of these techniques provides a magical global transaction. Together, they let a system make progress without losing track of what is true.

The dual-write problem is bigger than message brokers

A dual write happens whenever one logical operation requires two independently committed side effects.

The dangerous gap lies between the two commits. If the process, network or dependency fails inside that gap, the system is left in a partial state.

Reversing the order does not remove the problem. Publishing an event before committing the database transaction creates the opposite failure: consumers may act on a change that never became durable in the source service.

write A succeeds
        |
        | process or network fails
        v
write B never happens

Retries help only when the system can determine what the retry means. If the caller times out after sending a payment request, it may not know whether the provider accepted the payment. Retrying blindly can create a duplicate charge. Giving up can leave a paid order marked as unpaid.

This ambiguous outcome is a fundamental condition of distributed work. Good designs preserve enough identity and state to resolve it.

Begin with the business invariants

Before selecting a pattern, define the facts the workflow must protect.

For an order workflow, the invariants might include:

• one customer intention must not create two charges
• confirmed stock must not be promised to two orders
• an accepted order must not disappear silently
• every incomplete operation must be discoverable
• operators must be able to distinguish delay from permanent failure

These statements are more useful than asking for exactly-once processing. Exactly once is often discussed as a transport property, while the business cares about effects. A message may be delivered more than once while the inventory reservation happens once. A message may be delivered once while a badly designed consumer applies the effect twice.

Correctness belongs to the complete workflow.

The invariant also defines the consistency boundary. Facts that must change atomically should usually live behind the same transactional owner. Splitting them across services creates a coordination problem that the architecture must then solve. This is one reason service boundaries should follow ownership and invariants, not only technical layers or team charts.

What the transactional outbox actually guarantees

The transactional outbox addresses one specific dual write: changing local business data and recording the intention to publish a message.

Instead of writing to the database and broker separately, the service writes both the business change and an outbox record inside one local database transaction.

BEGIN TRANSACTION

UPDATE orders SET status = 'confirmed' WHERE id = :order_id;

INSERT INTO outbox ( event_id, aggregate_id, event_type, payload, created_at ) VALUES ( :event_id, :order_id, 'OrderConfirmed', :payload, :created_at );

COMMIT ```

If the transaction commits, both the order change and the publication intention exist. If it rolls back, neither exists. A separate publisher reads pending outbox records and sends them to the broker.

This closes the failure gap inside the producing service, but it does not make publication exactly once. The publisher can send a message and fail before marking the outbox record as delivered. It will send that message again after recovery.

The correct expectation is therefore:

• the event will eventually be published if retries continue
• the same event may be published more than once
• consumers must tolerate duplicate delivery

The outbox creates a reliable handoff between a local transaction and asynchronous delivery. It does not create an atomic transaction across every participating service.

The consumer needs its own consistency boundary

Suppose the Inventory service receives OrderConfirmed. It must reserve stock and record that the event has been processed.

Those actions have the same dual-write shape unless they share a local transaction. A common solution is an inbox or processed-message table.

BEGIN TRANSACTION

INSERT INTO processed_messages (consumer, event_id) VALUES ('inventory', :event_id) ON CONFLICT DO NOTHING;

IF event_was_new THEN UPDATE inventory SET reserved = reserved + :quantity WHERE product_id = :product_id; END IF;

COMMIT ```

The unique constraint on the consumer name and event identifier turns duplicate delivery into a repeatable result. The deduplication record and the business effect commit together.

This is the operational form of treating idempotency as a business rule. The event identifier represents one intention. Reprocessing that intention must not repeat its business effect.

An idempotency design should answer several questions explicitly:

• who creates the operation or event identifier
• what scope makes the identifier unique
• how conflicting reuse is detected
• how long deduplication records are retained
• what result a repeated request or message receives

Deduplication without a clear identity model only moves ambiguity into a different table.

Multi-service consistency requires a workflow

Once work crosses service boundaries, each service commits independently. The system needs a durable model of progress across those local transactions.

A saga is one way to model that progress. It represents a business operation as a sequence of steps, with an outcome and, where possible, a compensating action for each completed step.

A compensation is not a database rollback. Time has passed, other actors may have observed the result, and the external world may have changed. Refunding a payment is a new business operation with its own identifier, audit history and possible failures.

Some actions are not cleanly reversible. An email cannot be unsent. A shipment that has left the warehouse may require a return process rather than a technical rollback. The workflow must distinguish reversible steps, irreversible steps and steps that require human resolution.

This is why step ordering matters. Delay irreversible actions until the workflow has established the conditions that make them safe.

Orchestration and choreography

Sagas are commonly coordinated through orchestration or choreography.

With orchestration, one workflow component records the current state and sends explicit commands to participants.

Order workflow
  -> ReserveInventory
  <- InventoryReserved
  -> CapturePayment
  <- PaymentCaptured
  -> ConfirmOrder

The orchestrator provides one place to see the workflow, apply timeouts and decide what happens next. It can also become an overly central component if it absorbs business logic that belongs to participating services.

With choreography, services react to events and publish new events.

OrderCreated
  -> InventoryReserved
  -> PaymentCaptured
  -> OrderConfirmed

Choreography can keep participants independent, but the overall workflow becomes harder to see as the number of reactions grows. A change to one event can affect consumers that the producer does not control. Failure handling may be scattered across several services.

The decision is not about which pattern is modern. It is about where workflow ownership should be visible.

Use orchestration when the process has meaningful state, deadlines, branching or compensations that benefit from one explicit owner. Choreography works best when reactions are genuinely independent and the producer does not need to track a shared end-to-end outcome.

Timeouts do not tell you what happened

A timeout tells the caller that it stopped waiting. It does not reveal whether the remote operation failed, succeeded, or is still running.

Consider a payment request:

Order service -> Payment provider: capture payment
Order service <- timeout

The provider may have captured the funds before the response was lost. The order service should not immediately issue a new request with a new identity. It should retry with the same idempotency key or query the provider using a stable external reference.

This creates a general rule: ambiguous operations need reconciliation, not guesswork.

For each remote side effect, preserve:

• the local operation identifier
• the external reference or idempotency key
• the intended amount or parameters
• the last known state
• attempt timestamps and outcomes
• the next retry or reconciliation time

Timeouts, retries, backoff and jitter are necessary reliability tools, but they must operate within an identity and state model. Otherwise a retry mechanism can amplify both load and business effects. Failure should be treated as a concrete design input, not as a generic exception to catch at the boundary.

Reconciliation is part of the architecture

Retries handle failures expected to resolve soon. Reconciliation handles states that remain uncertain, delayed or inconsistent after ordinary processing.

A reconciliation process compares durable evidence and decides what work is required. It might find:

• outbox records that have not been published
• workflow steps that have exceeded their deadline
• payments with no matching confirmed order
• inventory reservations belonging to cancelled orders
• events recorded by a producer but absent from a consumer
• compensations that repeatedly fail

Reconciliation should be designed with the workflow, not added after the first incident. If an operation can become ambiguous, the system needs a stable way to find it again.

The safest reconciler is usually conservative. It should identify discrepancies, apply idempotent repairs when the correct action is unambiguous, and route uncertain cases for investigation. An automated repair that guesses incorrectly can be worse than a visible exception.

Observability should describe business progress

Infrastructure metrics can show that a queue is growing or a service is returning errors. They do not necessarily show which business operations are stuck.

A reliable workflow should expose both technical and business state.

Useful signals include:

• number and age of unpublished outbox records
• message delivery and processing attempts
• duplicate-message counts
• workflow duration by state
• operations waiting beyond their expected deadline
• compensation attempts and failures
• reconciliation discrepancies
• terminal outcomes by workflow type

Every log and event should carry the identifiers needed to reconstruct the path of one business operation. A correlation identifier helps group activity. Causation identifiers explain which message or command produced the next one. Business identifiers connect the technical history to the order, payment or account that people actually care about.

Observability is not complete when the team can see that something failed. It is complete when the team can answer what is affected, what state it is in, whether retrying is safe, and what should happen next.

A practical production model

The pieces fit together as a sequence of local guarantees.

Request with stable operation identity
        |
Producer local transaction
  business change + outbox record
        |
At-least-once event publication
        |
Consumer local transaction
  inbox record + business change
        |
Durable workflow state
        |
Retry, compensation or reconciliation
        |
Observable terminal outcome

No line in this model claims that every resource commits atomically. Instead, every boundary preserves enough information to continue safely after failure.

Before putting a multi-service workflow into production, verify that:

• business invariants and ownership are explicit
• every operation has a stable identity
• local state and outgoing events commit atomically
• consumers tolerate duplicate delivery
• workflow progress is durable and queryable
• retries have limits, backoff and jitter
• ambiguous external outcomes can be reconciled
• compensations are modeled as real business operations
• irreversible actions occur at deliberate points
• stuck work and failed compensation produce alerts
• operators have enough context to resolve exceptional cases

Reliability comes from preserving intent

Distributed consistency is not achieved by pretending several systems share one transaction. It is achieved by preserving the identity, intent and progress of the business operation at every boundary.

The transactional outbox reliably connects a local change to message publication. Idempotent consumers prevent repeated delivery from repeating business effects. Sagas make multi-step progress explicit. Compensation provides a forward-moving response to partial completion. Reconciliation resolves the cases that retries cannot safely answer.

The goal is not a system that never enters an intermediate state. Distributed workflows are made of intermediate states. The goal is a system where every state is visible, every retry has a defined meaning, and no accepted operation disappears between services.

Further reading

• Making retries safe with idempotent APIs, Amazon Builders' Library
• Saga distributed transactions pattern, Microsoft Azure Architecture Center
• Controlling and limiting retries, AWS Well-Architected Framework