Reliable systems do not begin with the assumption that every component will behave. They begin with a clear account of what can fail, what the failure means, and what the system must protect.
Start with consequences
“What if this goes down?” is too broad to guide a design. A better question is: what becomes untrue when this dependency is unavailable? A request may be delayed, duplicated, partially completed, or accepted without a durable record. Each consequence asks for a different response.
This is why resilience cannot be added as a generic layer. Retries help with transient failures, but they can amplify an overloaded dependency. Making retries safe also requires treating idempotency as a business rule. Queues absorb uneven demand, but they introduce delay and another place where ownership must be explicit.
Define the boundary of correctness
For every important operation, write down the facts that must remain true. In a transactional workflow that might mean:
- an accepted instruction is durably recorded
- repeated delivery does not repeat the business effect
- an incomplete workflow can be identified and resumed
- operators can distinguish delay from data loss
These are more useful than a vague target of “high availability.” They describe the promises the software needs to keep.
Spend complexity deliberately
Not every failure deserves automation. Sometimes a visible, recoverable failure is safer than a silent fallback. Sometimes a runbook is better than a second coordination mechanism.
The strongest design is rarely the one with the most safeguards. It is the one where each safeguard protects a named invariant, has an owner, and can be tested. Failure is inevitable. Confusion about what happened does not have to be.