How Idempotency Prevents Duplicate Automation Emails

A customer resets their password and receives two links. A subscriber completes a purchase and gets the same receipt twice. An account manager changes an order, then a workflow appears to send two identical confirmation messages.

These failures often get labelled as “the automation firing twice”. Sometimes that is true. More often, the system did what distributed systems are designed to do: retry work when it cannot tell whether the previous attempt succeeded. The email send is the visible casualty.

Idempotency is the design property that makes a repeated attempt produce the same intended outcome as a single attempt. In email automation, it means that a particular logical email for a particular recipient is accepted for sending once, even if the event, job or API request is delivered more than once.

This matters because duplicate mail is more than a cosmetic fault. It confuses customers, creates unnecessary support work and increases message pressure on people who did not ask for the same email twice. For permission-based marketing and lifecycle programmes, it can also distort reporting and contribute to complaints or disengagement.

Why duplicate sends happen in otherwise sensible systems

Most automation systems contain components that favour delivery and recovery over a promise of processing something exactly once. A queue may redeliver a job after a worker crash. A webhook sender may retry because it did not receive a successful response. A background function may time out after calling the email provider but before saving its own result. Two workers may pick up equivalent work at nearly the same moment.

That is normal behaviour, not necessarily a provider defect. For example, AWS documents that Lambda event-source mappings process records at least once and that duplicate processing can occur; asynchronous invocations can also receive the same event more than once. AWS’s Lambda best-practice guidance therefore recommends idempotent function code. The same broad concern applies whether the stack uses serverless functions, a job queue, an integration platform or a custom worker fleet.

Common causes include:

• Webhook retries. The receiving endpoint processes an event but its acknowledgement is lost or delayed, so the sender tries again.
• Queue redelivery. A worker completes the email-provider call, then crashes before acknowledging the job. The queue makes the job available again.
• Request timeouts. The application does not receive the provider response. The provider may already have accepted and queued the email.
• Concurrency races. Two requests for the same user action arrive close together and both pass a non-atomic “has this been sent?” check.
• Replay and repair jobs. A backfill, import, workflow republish or incident recovery re-emits events that were already acted on.
• Batch-level retries. One failed recipient causes a whole batch to be rerun, including people whose messages were already accepted.

An email platform normally cannot infer your business intent. A provider sees two valid API requests as two requests to send. Twilio SendGrid’s Mail Send API, for example, is a POST endpoint for submitting messages; an accepted submission is not, by itself, a declaration that a later identical submission should be ignored. Its Mail Send reference sets out the request used to submit mail, but sender-side safeguards still need to decide whether a second request represents a retry or a genuinely new message.

Idempotency is about the intended email, not the HTTP request

HTTP terminology can be helpful, but it is not enough. Methods such as GET and DELETE are conventionally idempotent because repeating them should not create another resource or another change. A POST that triggers an email usually is not: submitting it twice can create two sends.

An Idempotency-Key can let an API treat repeated POST requests as the same operation. The current IETF work on the header says a key must be unique and must not be reused for a request with a different payload; it also describes expiry and payload fingerprints as implementation choices. See the IETF’s Idempotency-Key header draft. This is useful where an email provider supports an equivalent feature, but it is only one layer.

The stronger question is: what exact customer-facing email is this? Define a durable identity for that answer, then enforce it in your own data model before a provider call is made.

For a one-to-one receipt, the identity may be:

receipt : order_84721 : recipient_2045 : receipt-v3

For a lifecycle message, it could be:

trial-day-7 : subscription_6719 : recipient_2045 : workflow-revision-12

For a campaign, the unit is normally the recipient send, rather than the campaign as a whole:

spring-offer-2026 : recipient_2045 : email-variant-b

The identifier should be stable when the work is retried, but different for a legitimately new occurrence. A password-reset request at 09:00 and another initiated by the same person at 15:00 are usually separate logical sends. Reusing a permanent key such as password-reset:recipient_2045 would suppress the later request incorrectly.

Use internal immutable identifiers where possible. Do not make an email address itself the key, and do not rely on aggressive address “normalisation” to collapse identities: address handling varies by mailbox provider and organisation. If a compact opaque key is needed outside the database, derive it from internal IDs using a suitable keyed hash rather than exposing personal data.

The atomic claim is the essential control

A frequent but unsafe pattern looks like this:

1. Check whether an email has already been sent.
2. If not, call the email provider.
3. Record that it was sent.

Two concurrent workers can both complete step one before either gets to step three. Both then send. A cache-based flag can reduce the likelihood, but it does not reliably resolve the race when the cache expires, fails or is reached concurrently.

Instead, create an atomic claim in a durable store. In relational databases, this is commonly a unique constraint on the logical send key plus an insert that either succeeds once or reports a conflict. In a key-value store, it may be a conditional write. The important property is that the decision “this worker owns this logical send” happens in one operation.

A minimal send ledger might include:

Field	Purpose
logical_send_key	Uniquely identifies the intended email and is protected by a unique constraint.
recipient_id	Links the send to the consent, suppression and contact record without making the address the deduplication key.
message_type and revision	Separates a deliberate new template or workflow version from a retry.
status	Records states such as pending, claimed, submitted, suppressed, failed or requires review.
provider_message_id	Stores the provider’s reference once available for tracing and reconciliation.
payload_fingerprint	Detects accidental reuse of one logical key with changed content or recipient data.
timestamps	Supports timeouts, auditing and investigation without treating an old lock as permanent.

When a retry finds the same key, it should not create another send. Depending on the recorded state, it can return the stored outcome, wait briefly for an in-progress attempt, resume safely, or send the item to an exception path. It should not silently treat every duplicate key as an error: duplicate delivery of work is expected in resilient systems.

Make the key match the business rule

There is no universal idempotency key. The right composition follows the rule the programme is meant to uphold.

Transactional messages

Order confirmations, invoices and access-change notices normally use a stable business event ID. Include the intended recipient and a message version if changing the content creates a materially different communication. If an order can have multiple invoices, key on the invoice ID, not merely the order ID.

Triggered lifecycle email

Use the trigger occurrence, recipient and workflow revision. A product-view event ID, abandoned-basket instance ID or subscription milestone ID is usually better than a rounded timestamp. The key needs enough specificity to distinguish a new eligible event from a retry of the earlier one.

Scheduled and recurring messages

Include the schedule period explicitly, such as the local billing month or an ISO week defined by the business. Be precise about time zones and late-arriving data. A monthly statement for March must not be suppressed because February’s key was reused.

Bulk campaigns

Store eligibility and send state per recipient. A campaign-level “sent” flag cannot tell a retrying job which people were already submitted, suppressed or excluded due to frequency rules. Recipient-level records also make it possible to retry only the work that remains unresolved.

Use an outbox to avoid losing the work before it starts

Idempotency prevents repetition; it does not alone guarantee that an eligible email will ever be queued. Consider an application that updates an order and then publishes an email job. If it commits the order but crashes before publishing, the customer may receive no confirmation. If it publishes first and the order transaction later fails, it may email about an order that does not exist.

The transactional outbox pattern addresses this boundary. The application writes the business change and an outbox record in the same database transaction. A separate dispatcher reads the committed outbox, claims the logical send atomically, and submits it to the mail provider. If the dispatcher fails, the outbox can be retried without recreating the underlying business event.

This is especially useful for automation based on webhooks and asynchronous jobs because it gives operations a durable record of what should have happened, rather than relying on transient logs.

The difficult boundary: provider acceptance

There remains a hard problem between your system and the email provider. Imagine this sequence:

1. Your worker claims receipt:order_84721:recipient_2045:receipt-v3.
2. It submits the message to the provider.
3. The provider accepts it.
4. The network response is lost, or the worker crashes before saving the provider ID.

On restart, your system knows it attempted the send but may not know whether the provider accepted it. Retrying could create a duplicate. Refusing to retry could lose an important message. This is why “exactly once email delivery” should be treated cautiously: there is no single database transaction that atomically commits both your ledger entry and an external provider’s mail queue.

If the provider supports idempotency for its send endpoint, pass the same key on every retry and retain it for at least the provider’s documented key-retention window. If it does not, build an explicit policy for unknown submission state. Postmark, for instance, states that it does not currently support idempotency keys, while suggesting that customers use response handling and webhooks as integration safeguards. Postmark’s explanation of idempotency keys is a useful reminder that application-level protection is still required.

Reasonable handling depends on the message’s importance and recoverability. A non-critical marketing prompt may be held for review or suppressed when acceptance is uncertain. A security or access message may need a carefully designed recovery route, perhaps using a separate user-visible status page rather than an automatic repeated email. What matters is that the decision is deliberate, logged and testable.

Provider metadata and event webhooks help investigate and reconcile. For example, SendGrid’s event webhook includes message and event identifiers, and its documentation recommends deduplicating received event data using sg_event_id. SendGrid’s Event Webhook overview and event reference describe those identifiers. Apply idempotency to webhook ingestion too: a repeated delivered, bounce or unsubscribe event should not repeat downstream actions.

Do not confuse Message-ID, events and suppression with idempotency

A Message-ID is valuable for tracing a particular email message. RFC 5322 says each message identifier must be globally unique. The Internet Message Format specification explains its role in identifying a version of a message and in conversation references. It is not, however, a reliable instruction to every receiving mailbox to discard an otherwise valid duplicate, nor does it stop your platform from submitting two distinct messages.

Likewise, delivery webhooks tell you about events after submission; they do not prevent the original duplicate submission. A contact-pressure limit can stop a recipient receiving too many messages within a defined period, but it may also conceal an underlying automation defect and could suppress a necessary transactional notice if applied indiscriminately.

These controls work best together:

• Idempotency stops repeats of the same intended action.
• Frequency and contact-pressure rules constrain different messages arriving too close together.
• Suppression and consent checks prevent mail to ineligible recipients.
• Provider events and message references support tracing, reconciliation and diagnosis.

Operational practices that make the design trustworthy

Start by logging the logical send key from the point an event arrives through to provider submission and later events. Avoid putting raw email addresses, full message bodies or secrets into general application logs. Make it possible to answer, for one customer complaint: which trigger occurred, how many times it was delivered to workers, which worker won the claim, whether the provider accepted it and which provider message ID resulted.

Test the failure paths deliberately. Run the same webhook twice. Start two workers at once. Simulate a timeout after provider acceptance. Replay a queue batch containing a mix of completed and incomplete recipients. Test expired claims and database failover. The success path is usually straightforward; duplicate email incidents emerge in the ambiguous paths.

Monitor duplicate-claim rates, unknown submission states, stale claims, outbox age and the gap between intended sends and provider-accepted sends. A sudden increase in duplicate claims may be harmless retry noise, or it may reveal a webhook configuration change, a slow downstream dependency or a job acknowledgement problem.

Finally, preserve the ledger for a period that matches the automation’s replay and dispute horizon. Deleting records too quickly can turn a late retry into a fresh send. Retaining every record indefinitely can create unnecessary storage and privacy obligations. Choose and document a retention period based on the longest realistic retry, replay and operational investigation window for each message class.

What this means for deliverability

Authentication, sender reputation and inbox placement do not compensate for a workflow that sends customers the same message twice. The first task is to prevent duplicate intent from becoming duplicate submission. Once that is in place, audience suppression, contact-pressure rules and provider-level sending controls can provide useful additional safeguards for permission-based programmes.

When duplicate sends are already affecting a live programme, the investigation should begin with the event and job trail, not with subject lines or creative. A deliverability review such as Email Foundry’s may identify related weaknesses in suppression handling, contact pressure or provider telemetry, but idempotency itself belongs in the automation and sending architecture. Give every intended email a durable identity, claim it atomically, and treat retries as normal operating conditions rather than exceptional accidents.

Frequently asked questions

Is an idempotency key enough to stop duplicate emails?

Only if the email provider supports and correctly honours that key for the send operation. A robust design also keeps its own durable recipient-level send ledger, uses an atomic claim and handles duplicate events, jobs and webhooks.

Should every email use the same idempotency key for a recipient?

No. A key represents one intended email occurrence. It must remain the same for retries of that occurrence, but change for a legitimate new reset request, new order, new schedule period or new qualifying lifecycle event.

Can a Message-ID prevent duplicate email delivery?

No. Message-ID is useful for identifying and tracing an individual message, but it is not a dependable end-to-end deduplication mechanism. Prevent duplicate provider submissions before the message enters the mail stream.

What should happen if the email provider may have accepted a request but the response timed out?

Record the attempt as an unknown submission state rather than blindly retrying. Reconcile using provider references or events where possible, use provider-side idempotency if available, and define a message-specific recovery policy for unresolved cases.

Sources and further reading

• AWS Lambda best practices — Amazon Web Services
• The Idempotency-Key HTTP Header Field — IETF Datatracker
• What is an idempotency key? — Postmark
• Twilio SendGrid Event Webhook overview — Twilio SendGrid
• Twilio SendGrid Event Webhook reference — Twilio SendGrid
• RFC 5322: Internet Message Format — RFC Editor