Disclosure: This article is published by Datamagnet. Vendor claims are self-reported unless otherwise noted. Provider behaviour below comes from public documentation, retrieved 2026-07-22 unless noted.
Webhook Idempotency: How to Stop Duplicate Writes in Enrichment Pipelines (2026)
Webhook providers deliver every event at least once, so your pipeline will receive some events twice. The fix has three layers: dedupe on the provider's stable event ID, remember processed IDs for longer than the longest retry window, and make every write an upsert. Then return a fast 2xx so the sender stops retrying.
This guide builds that pattern step by step for pipelines that consume job-change, funding and engagement webhooks. It also covers how long each major provider retries, and when a managed webhook service is worth it.
Key Takeaways
- At-least-once delivery is the default. Stripe retries a failed webhook for up to three days, and Shopify retries 8 times over 4 hours (Stripe, Shopify).
- Dedupe on the provider's event ID, never on a payload hash or a timestamp. Retries can change the envelope, but not the ID.
- Size your dedup TTL to the longest retry window you consume. A TTL tuned to Shopify's 4 hours lets Stripe's 3-day retries through.
- Back the dedup check with a unique constraint and an upsert, so two racing retries still write one row.
- Return 2xx for "processed" and "already seen" alike. An error status on a duplicate tells the sender to try again.

Why Do Webhooks Deliver the Same Event Twice?
Webhooks deliver duplicates because providers choose "the event definitely arrived" over "the event arrived exactly once." A timeout, a 500 from your server, or a response that lands after the sender's deadline all trigger a resend. The resend carries the same event, not a new one.
AWS states the rule plainly for SQS: "you must design your applications to be idempotent... they must not be affected adversely when processing the same message more than once" (AWS, Amazon SQS at-least-once delivery). Webhook senders inherit the same tradeoff. Twilio's Event Streams, for example, guarantees only at-least-once delivery. It retries a failed event "until it succeeds or four hours have passed" (Twilio, Event delivery retries and event duplication).
At-least-once vs. exactly-once delivery
At-least-once means no event is lost, but some arrive more than once. Exactly-once means every event arrives one time only. Over plain HTTP, a sender can't tell whether a timed-out request was processed, so it has to resend. In practice, "exactly-once" means at-least-once delivery plus an idempotent receiver. That receiver is your code.
Here's the part teams often miss. Duplicate CRM records usually get debugged in the wrong layer. Teams audit the CRM's matching rules, but the duplicate was written earlier, at the pipeline's ingestion point. Fixing dedup in the CRM cleans up symptoms after every new webhook. Fixing it at ingestion stops them from happening.
How Long Does Each Provider Keep Retrying?
Two numbers matter for every source you consume: how long it keeps retrying, and which field stays constant across retries.
| Provider | Automatic retries | Stable ID to dedupe on | Source |
|---|---|---|---|
| Stripe | Up to 3 days, exponential backoff | event.id | Stripe Webhooks |
| Shopify | 8 attempts over 4 hours (since September 2024) | X-Shopify-Event-Id header | Shopify changelog |
| Twilio Event Streams | Until success or 4 hours | Event SID, constant across retries | Twilio |
| GitHub | None. Failed deliveries need manual redelivery | X-GitHub-Delivery header, unchanged on redelivery | GitHub |
| LINE Messaging API | Redeliveries carry the same ID | webhookEventId (a ULID) | LINE Developers |
| Datamagnet signals | 7 attempts over 48 hours (5 min, 30 min, 2 h, 8 h, 24 h, 48 h) | event_id on each profile change | Datamagnet webhooks |
GitHub is the outlier. It doesn't retry on its own, but your handler must respond within 10 seconds. Miss that, and the delivery is marked failed until someone redelivers it by hand (GitHub, Handling failed webhook deliveries).
What does LINE's webhookEventId do?
LINE's Messaging API puts a webhookEventId on every webhook event object. It's a ULID that stays constant across redeliveries, so your server can track which events it already processed (LINE Developers, Messaging API reference, retrieved 2026-08-08). Use it as your dedup key, exactly as you'd use Stripe's event.id. Some providers hand you the key pre-built. For the rest, you build it in Step 2.
Is Stripe's Idempotency-Key Header the Same as Webhook Dedup?
No, and mixing them up leads to a TTL that's far too short. Stripe's Idempotency-Key header protects requests you send to Stripe's API, so a retried charge doesn't run twice. Stripe says those keys can be removed "after they're at least 24 hours old" (Stripe, Idempotent Requests documentation).
Webhooks flow the other way. Stripe retries a failed webhook for up to three days. So a 24-hour dedup window on your receiving side leaves a two-day gap where a late retry looks new. For inbound Stripe events, key on event.id and size the TTL to that 3-day tail.
What Do You Need Before You Start?
You don't need new infrastructure. You need somewhere to remember what you've processed, and a source that sends a stable ID.
- A webhook receiver that already parses and writes enrichment events (job changes, funding rounds, engagement signals)
- A store with millisecond reads: a Redis namespace with TTL, or a Postgres table with a unique constraint
- The field each provider uses as a stable event ID. Datamagnet's webhook payloads include an
event_idon each profile change for this reason. - Comfort with upserts (
INSERT ... ON CONFLICTin Postgres) and with HMAC signature verification - Difficulty: Intermediate
Step 1: Which Sources Write to Your Pipeline?
By the end of this step, you'll have a list of every source that can trigger a write. Skip it, and you'll end up idempotent against one integration and wide open on another.
List each source separately: job-change signals, engagement signals, CRM sync callbacks, and any retry jobs your own code schedules. For each one, note the stable event ID field. Datamagnet's job-change and post signals carry an event_id on every change, and the webhook docs recommend deduplicating on it. Don't confuse it with the signal ID, which names the monitor you created with the signal creation endpoint, not the event.
Verify it: pull your last 100 webhook payloads from logs. Check that the ID field is present and non-null on every one. If it's missing anywhere, fix that before moving on.
Step 2: How Do You Build a Stable Idempotency Key?
By the end of this step, every delivery of the same event produces the same key. The safest form combines provider, event type and event ID: {provider}:{event_type}:{event_id}.
Don't hash the payload body. Timestamps, retry counters and field ordering can shift between attempts, so a hash misses true duplicates. Don't key on received_at either. It works until the first redelivery arrives with a new timestamp and an unchanged payload. Then the "duplicate" sails through as a fresh event.
Twilio's docs confirm the pattern. Duplicates are identified "via a constant event id (SID) across retries," even as the attempt number changes (Twilio, Event delivery retries and event duplication). That constant ID, not the envelope around it, is your key.
One detail most guides skip: verify the HMAC signature before you touch the dedup store. Otherwise a spoofed request that reuses a real event ID can poison your table with a false "already processed" entry.
Verify it: send the same test payload twice and confirm both keys match byte for byte.

Step 3: How Do You Store Processed Keys, and for How Long?
By the end of this step, "have I seen this event?" is one fast lookup that runs before any write logic.
- Create a table or key namespace:
dedup_keys(key, processed_at, expires_at). - On every incoming event, check for the key before any enrichment or write.
- If it's found and unexpired, return 200 and stop.
- If it isn't, process the event, then insert the key with a TTL.
CREATE TABLE dedup_keys (
key text PRIMARY KEY,
processed_at timestamptz NOT NULL DEFAULT now(),
expires_at timestamptz NOT NULL
);
-- Before processing: has this event been handled?
SELECT 1 FROM dedup_keys
WHERE key = $1 AND expires_at > now();
-- After a successful write: remember it.
INSERT INTO dedup_keys (key, expires_at)
VALUES ($1, now() + interval '7 days')
ON CONFLICT (key) DO NOTHING;
Match the TTL to the longest retry window you consume, then add a margin. A 7-day TTL is a common practitioner choice. It covers Stripe's 3-day tail with room to spare, though no single vendor publishes that number. Redis EXPIRE handles expiry natively. In Postgres, index expires_at and purge expired rows nightly.
The failure we see most often isn't a missing dedup store. It's a TTL copied from one source and reused for all of them. A store tuned to Shopify's 4 hours quietly accepts duplicates from a source with a 3-day retry tail. New sources don't inherit protection either. Review the key and the TTL each time you add one.
Step 4: How Do You Make the Write Itself Idempotent?
By the end of this step, a duplicate that slips past the dedup check still can't create a second record. Two near-simultaneous retries can both pass the lookup before either one writes. The database's unique constraint is your backstop.
Replace every INSERT in the enrichment write path with an upsert. Key it on the enriched entity's own stable ID, such as a LinkedIn profile URL, a company domain or your internal contact ID. Don't key it on the webhook's event ID.
INSERT INTO contacts (linkedin_url, title, company, updated_at)
VALUES ($1, $2, $3, $4)
ON CONFLICT (linkedin_url) DO UPDATE
SET title = EXCLUDED.title,
company = EXCLUDED.company,
updated_at = EXCLUDED.updated_at
WHERE contacts.updated_at < EXCLUDED.updated_at;
The final WHERE clause handles out-of-order delivery. If an older event lands after a newer one, it can't overwrite the fresher data. Drop it if your events carry no reliable timestamp.
In a CRM like HubSpot, the equivalent is matching on an external ID before creating a contact or company. That's the pattern behind Datamagnet's HubSpot integration, which keeps enrichment data current without duplicate rows.
Why bother? In 2025, Validity surveyed 602 CRM stakeholders. 76% said less than half of their CRM data was accurate and complete. 37% reported losing revenue because of poor data quality (Validity, The State of CRM Data Management in 2025). Duplicate enrichment writes are one direct, fixable contributor.
The rule: never let "insert a new record" be the only path a webhook handler can take.
Step 5: What Status Should You Return, and When?
By the end of this step, your responses help the sender stop retrying instead of provoking more retries.
Return a 2xx as soon as you've either processed the event or confirmed it's a duplicate. Do it before slow downstream work like a CRM sync. GitHub gives you 10 seconds before it marks a delivery failed. Hookdeck's guidance puts a general timeout budget closer to 60 seconds (Hookdeck, Implementing Webhook Retries). Either way, acknowledge fast and queue the real work.
The same rule applies when an edge function receives a webhook, calls an enrichment API and writes back to Postgres. Acknowledge in milliseconds, then run the enrich-and-write chain asynchronously. Fitting the whole round trip into the sender's window is what causes the timeouts this guide exists to prevent.
Return a 4xx or 5xx only when a retry would genuinely help, such as a transient database timeout. Never return an error just because a duplicate arrived. That tells the sender to try again, which is the opposite of what you want. The Datamagnet error reference lists which status codes are worth retrying on.

Where do dead-letter queues fit?
Once you've returned a 200, the sender's retries won't save you. If your own downstream write fails after that, recovering the event is your job. Retry transient failures a few times with backoff. If they still fail, park the event in a dead-letter queue you own instead of dropping it. Replay it once the cause is fixed. Replays are safe, because Steps 3 and 4 make reprocessing a no-op.
Step 6: How Do You Test and Monitor It?
By the end of this step, you'll have proof the pipeline holds up under the conditions that break naive receivers.
- Replay test. Send one captured payload 5 times in a row. Confirm exactly one record exists afterward.
- Concurrency test. Fire the same payload from 10 parallel requests. Confirm the dedup table and upsert still produce one record.
- Retry-storm test. Simulate a burst of near-identical events. Storms are real: Microsoft's architecture diagnostics recorded one client IP opening 81,608 inbound connections within an hour during a documented incident (Microsoft, Retry Storm Antipattern, updated 2026-03-17).
A receiver that answers fast, even to duplicates, starves a retry storm of the failures that feed it.
Then put two numbers on a dashboard. The first is the share of incoming webhooks flagged as duplicates. A spike usually means a provider had an outage and is retrying hard. A rate near zero usually means the check isn't running at all. The second is the error rate on the dedup store itself. If that store goes down, every retry becomes a fresh insert, and normal error logs won't show it. Alert on it directly.
Should You Build Dedup or Use a Managed Webhook Service?
For most enrichment pipelines, build it. A Redis namespace or one Postgres table with a key and an expires_at column is enough. The whole mechanism is a lookup, a TTL and a unique constraint. Some providers, like LINE, even hand you the key.
A managed webhook service makes sense when volume turns a shared table into a lookup bottleneck, or when you'd rather not run retry infrastructure. Most B2B enrichment pipelines never reach that scale. Either way, you still own the write path. Keep the upsert on your own tables, because no service in front of your receiver can stop a duplicate row inside your database.
Frequently Asked Questions
What's the difference between idempotency and deduplication?
Deduplication detects that you've seen an event before and discards the repeat. Idempotency means processing an event twice has the same result as processing it once. An enrichment pipeline needs both. The dedup check catches most repeats early and cheaply. The idempotent write, an upsert on a unique key, protects you when two retries race past the check at the same moment.
Is Stripe's Idempotency-Key header the same as webhook deduplication?
No. Stripe's Idempotency-Key header protects API requests you send to Stripe, and those keys can be removed after 24 hours (Stripe, Idempotent Requests). Webhooks travel the other way, and Stripe retries them for up to three days (Stripe, Webhooks). For inbound events, dedupe on event.id and keep processed IDs for longer than that three-day window.
What's the difference between at-least-once and exactly-once delivery?
At-least-once delivery guarantees no event is lost, but allows some to arrive more than once. Exactly-once means each event arrives one time only. Over HTTP, a sender can't know whether a timed-out request was processed, so it resends. Stripe, Shopify and Twilio all deliver at least once. You get exactly-once results by adding an idempotent receiver.
What if my webhook provider doesn't include a stable event ID?
Build a composite key from fields that don't change between retries. Typically that's the entity ID plus the event type plus a timestamp rounded to the nearest minute. It's less precise than a native event ID, but it still catches most true duplicates. Never use the received-at time, because every redelivery gets a new one.
How long should I keep processed event IDs?
Keep them longer than the longest retry window among your sources, plus a margin. Stripe retries webhooks for up to three days, while Shopify and Twilio stop after about four hours. A 7-day TTL is a common practitioner choice when you consume several providers. It's an engineering convention, not a number any single vendor publishes.
Do I need a managed webhook platform to deduplicate events?
Usually not. A Redis namespace or a single Postgres table with a key and an expires_at column handles most B2B enrichment volumes. Some providers, like LINE with its webhookEventId, even hand you the dedup key. Consider a managed service only when a shared table becomes a lookup bottleneck. Even then, keep the upsert on your own writes.
Stop Letting Retries Write Your Data Twice
You now have the full pattern. Map every event source and its stable ID. Build a key that never changes between deliveries. Gate writes behind a dedup store whose TTL outlasts your slowest sender. Back it with an upsert, so racing retries still produce one row. Return a fast 2xx, park your own failures in a dead-letter queue, and test against replays and retry storms.
Success looks simple: a webhook that arrives three times leaves exactly one record, and two of those deliveries show up in your logs as "deduplicated."
If you're wiring a signal-driven pipeline, start with the Datamagnet quickstart to see the payload shape you'll dedupe against. The champion-tracking cookbook walks through job-change signals flowing through exactly this kind of handler. When you move an existing monitor onto a new receiver, the signal update endpoint points it at a new webhook URL without recreating it. See how real-time job-change intent signals are delivered, why programmatic CRM enrichment depends on this write safety, and how the same at-least-once rules apply when you forward signals into n8n. Our security documentation covers HMAC-signed webhooks. Then try the Signal API against your own dedup logic.
Sources
- Stripe, Webhooks documentation, retrieved 2026-07-22, https://docs.stripe.com/webhooks
- Stripe, Idempotent Requests documentation, retrieved 2026-07-22, https://docs.stripe.com/idempotency
- AWS, Amazon SQS at-least-once delivery, retrieved 2026-07-22, https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html
- Shopify, Updates to webhook retry mechanism, 2024-09-10, retrieved 2026-07-22, https://shopify.dev/changelog/updates-to-webhook-retry-mechanism
- GitHub, Handling failed webhook deliveries, retrieved 2026-07-22, https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries
- GitHub, Best practices for using webhooks, retrieved 2026-10-21, https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks
- Twilio, Event delivery retries and event duplication, retrieved 2026-07-22, https://www.twilio.com/docs/events/event-delivery-and-duplication
- LINE Developers, Messaging API reference, retrieved 2026-08-08, https://developers.line.biz/en/reference/messaging-api/
- Hookdeck, Implementing Webhook Retries, retrieved 2026-07-22, https://hookdeck.com/webhooks/guides/webhook-retry-best-practices
- Microsoft, Retry Storm Antipattern, Azure Architecture Center, updated 2026-03-17, retrieved 2026-07-22, https://learn.microsoft.com/en-us/azure/architecture/antipatterns/retry-storm/
- Validity, The State of CRM Data Management in 2025, retrieved 2026-07-22, https://www.validity.com/resource-center/the-state-of-crm-data-management-in-2025/
- Datamagnet, Webhooks, retrieved 2026-10-21, https://docs.datamagnet.co/api-reference/webhooks
- Datamagnet, API Errors reference, retrieved 2026-07-22, https://docs.datamagnet.co/api-reference/errors

