Retries Without Duplicate Work
A retry is safe only when the system can tell whether the first attempt already took effect. The practical fix is to give every unit of work a stable identifier, record its state before any side effect, and make the side effect itself idempotent. Then a retry either completes the same logical action or is refused as already done.
Why retries duplicate work
Automation fails in ordinary ways: a network timeout, a rate limit, a service restart, a slow database lock. The dangerous case is not the failure itself but the uncertainty after it. If the first attempt reached the external system and the response was lost, a blind retry sends the same request again. For a payment, that can mean a double charge. For an invoice, a duplicate document. For a support ticket, two agents working the same case.
The root cause is usually a missing or unstable key. If the automation generates a new identifier on every attempt, the receiving system sees two unrelated requests. If it reuses the same identifier, the receiver can recognise the second attempt as the same work.
Give every unit of work a stable key
Before the automation calls anything external, create a durable record with a unique key. That key should be derived from the business event, not from the attempt. For example, an order confirmation event can carry an order reference plus a step name. Store the key in your own database with a status such as pending, in progress, succeeded, or failed.
The external call then includes that key, either as an idempotency key in the request or as a natural business reference the receiving system already understands. When the call is retried, the same key travels with it. The receiver can reject or ignore a duplicate instead of creating a second record.
Separate pending from failed
A timeout does not mean the action failed. It means you do not know the outcome. Treating every timeout as a failure and retrying immediately is how duplicates appear. A better model has at least three states: pending, confirmed, and failed. Pending means the request may have been received. Confirmed means the external system acknowledged it. Failed means the external system explicitly rejected it or the request never left your side.
This distinction changes the retry policy. Confirmed work is never retried. Failed work can be retried after a fix. Pending work needs a reconciliation step: query the external system by the same key to learn whether the action exists. Only if it does not exist should the automation send it again.
Bound the retries and route the rest to a person
Unlimited retries turn a small outage into a flood. Set a maximum number of attempts and a delay between them, with a longer wait after each failure. When the limit is reached, stop and mark the item for manual review. A person can then check the external system, decide whether the action already happened, and either complete it or close it.
The manual queue should show the business key, the last error, the number of attempts, and the time of the last try. That is enough for a support or finance team to resolve the item without reading logs. Suppose a monthly invoicing automation times out while creating an invoice in the accounting system. With a bounded policy, it retries twice, then places the invoice in a review queue. A finance user searches the accounting system for the invoice number, finds it already created, and marks the automation item as resolved. No second invoice is issued.
A concrete acceptance test
You can test the design before trusting it in production. Pick one automated action that creates a record in another system. Run it once and confirm the record exists. Then simulate a lost response by having the automation treat the call as a timeout even though the external system succeeded. Trigger the retry path. The test passes only if the external system still contains exactly one record for that business key and the automation marks the item as confirmed or resolved. If a second record appears, the design is not ready.
Run the same test for a genuine failure, where the external system rejects the request. The automation should retry up to the limit and then move the item to manual review, not create a partial or duplicate record. These two cases cover the common failure modes without needing a full load test.
Tradeoffs to accept
Idempotency adds work. You need a place to store keys and states, and the external system must support a stable reference or idempotency key. Some systems do not, which means reconciliation must happen through a search or a report instead. That is slower but still safer than blind retries.
Bounded retries also mean some work waits for a person. That is a deliberate trade: a short delay in a review queue is usually cheaper than a duplicate payment, invoice, or customer message. The goal is not zero failures. It is that a failure never turns into two pieces of work.
Continue reading
A practical next step
Explore the related Devign work, then discuss your requirements before committing to a project scope.