Data Enrichment

How to Prevent Duplicate Webhook Writes: A 2026 Idempotency Guide

Pratik Dani
Pratik Dani
·26 min read
Flat vector diagram showing a webhook retry being deduplicated before it reaches a CRM database

Disclosure: This article is published by Datamagnet. Vendor claims are self-reported unless otherwise noted.

How to Prevent Duplicate Webhook Writes: A 2026 Idempotency Guide

As of 2026, Stripe retries a failed webhook delivery "for up to three days with an exponential backoff" (Stripe, Webhooks documentation, retrieved 2026-07-22). If your enrichment pipeline writes on every delivery instead of every event, that single outage can turn one signal into three or four duplicate CRM records.

Enrichment pipelines that consume webhooks — job-change alerts, funding-round signals, company engagement events — almost always sit behind an at-least-once delivery guarantee. That's not a bug in the provider. It's the tradeoff every reliable webhook system makes: deliver twice rather than risk losing an event. Your pipeline has to absorb that duplication, not the provider.

This guide walks through building idempotency into an enrichment pipeline so retries never turn into duplicate writes. You'll extract a stable event ID, store it safely, and make every write an upsert instead of a blind insert — the same pattern we use to keep signal-driven CRM writes clean at Datamagnet.

Key Takeaways

  • At-least-once delivery is standard: Stripe retries for 3 days, Shopify retries 8 times over 4 hours.
  • Dedup on the event ID, not the payload — the same event can arrive with slightly different envelopes.
  • Make writes idempotent (upsert on a unique key) so a duplicate delivery is a no-op, not a new row.

Flat vector diagram of a webhook event delivered twice, with a dedup checkpoint discarding the duplicate before it reaches the database

What Do You Need Before You Start?

You don't need new infrastructure to fix this — just a place to remember what you've already processed. Idempotency in enrichment pipelines comes down to two moving pieces: a store that remembers processed event IDs, and a webhook source that sends a stable identifier on every delivery, including retries. Datamagnet's webhook payloads include a signal ID for exactly this reason. Here's everything you need before writing any code:

  • A key-value or relational store for processed event IDs (Redis with TTL, or a Postgres table with a unique constraint)
  • A webhook source that sends a stable event identifier on every delivery, including retries — check your provider's docs (Datamagnet's webhook payloads include a signal ID for exactly this reason)
  • Basic familiarity with upsert syntax (ON CONFLICT DO UPDATE in Postgres, or equivalent) and HTTP signature verification
  • Time: ~2-3 hours for a first implementation
  • Difficulty: Intermediate

Step 1: Does Your Webhook Source Guarantee At-Least-Once Delivery?

By the end of this step, you'll know exactly why duplicates happen, which changes how you design everything downstream. Most production webhook systems favor delivering an event twice over risking it never arriving at all. As of 2026, AWS states it plainly for SQS: "you must design your applications to be idempotent... they must not be affected adversely when processing the same message more than once" (AWS, Amazon SQS at-least-once delivery, retrieved 2026-07-22). Webhook providers inherit the same tradeoff, since a failed HTTP response almost always triggers a retry.

Check your provider's retry documentation for two numbers: how long it keeps retrying, and what counts as a "failed" delivery on your end. In September 2024, Shopify updated its retry schedule so failed deliveries are retried "a total of 8 times over 4 hours" using exponential backoff (Shopify, Updates to webhook retry mechanism, retrieved 2026-07-22). GitHub doesn't auto-retry at all — but your handler still has to respond within 10 seconds, or the delivery is marked failed and a human has to manually redeliver it (GitHub, Handling failed webhook deliveries, retrieved 2026-07-22).

Verification: You should be able to answer "if my server times out, what happens next?" for every webhook source feeding your pipeline. If you can't, go find that doc before writing any dedup code.

Step 2: How Do You Extract a Stable Idempotency Key From Every Payload?

By the end of this step, every incoming webhook produces one consistent identifier you can key on, no matter how many times it's delivered. Don't hash the payload body and call it a key. Timestamps, retry counters, or nondeterministic field ordering can shift between attempts, so a payload hash silently fails to catch true duplicates. Instead, use the identifier the provider guarantees stays constant across retries: Stripe's event.id, Shopify's X-Shopify-Event-Id header, Twilio's event SID, or the signal ID in a Datamagnet webhook payload.

As of 2026, Twilio's Event Streams documentation confirms the pattern directly: duplicates are identified "via a constant event id (SID) across retries," even while the delivery attempt number changes (Twilio, Event delivery retries and event duplication, retrieved 2026-07-22). That constant ID, not the envelope around it, is your dedup key.

[INFO-GAIN: specific configuration] A detail most integration guides skip: verify the HMAC signature before you check the dedup store. An unverified payload that happens to reuse an event ID can otherwise poison your dedup table with a spoofed "already processed" entry.

Flat vector diagram of three duplicate webhook payloads converging on the same highlighted event ID field

Step 3: How Long Should You Store Processed Event IDs?

By the end of this step, your dedup store stays bounded instead of growing forever, while still catching every realistic retry. An unbounded "have I seen this ID" table eventually becomes its own performance problem. As of 2026, Stripe's idempotency keys for API requests are retained and can be safely removed "after they're at least 24 hours old" (Stripe, Idempotent Requests documentation, retrieved 2026-07-22) — a useful anchor, since most retry windows close well before that. Match your TTL to the longest retry window among your webhook sources, then add a safety margin.

Webhook Retry-Window Length by Provider Horizontal bar chart of retry-window length. Stripe retries for up to 3 days (72 hours) with exponential backoff. GitHub does not auto-retry but keeps a failed delivery available for manual redelivery for up to 3 days. Twilio retries for up to 4 hours. Shopify retries 8 times over 4 hours. Source: Stripe, Shopify, Twilio, and GitHub official docs, accessed 2026. Stripe Up to 3 days GitHub (manual) Up to 3 days Twilio Up to 4 hours Shopify 4 hrs (8 attempts) Source: Stripe, Shopify, Twilio, and GitHub official docs, accessed 2026

A 7-day TTL is a common practitioner recommendation for webhook event dedup — wide enough to cover every provider above with room to spare, though it's an engineering convention rather than a number any single vendor publishes. Redis with EXPIRE or a Postgres row with a processed_at column and a nightly cleanup job both work fine at typical enrichment volumes.

The failure mode we see most often isn't a missing dedup store — it's a TTL copied from one webhook source and reused for all of them. A key-value store tuned to Shopify's 4-hour window will silently start accepting duplicates from a source with a 3-day retry tail.

Step 4: How Do You Make the Database Write Itself Idempotent?

By the end of this step, even a duplicate that slips past your dedup check can't corrupt your data, because the write itself refuses to double up. Dedup checks are your first line of defense, not your only one. Race conditions between two near-simultaneous retries can slip past an in-memory check, so the write itself needs a backstop: a unique constraint plus an upsert. In Postgres, that's INSERT ... ON CONFLICT (event_id) DO UPDATE. In a CRM like HubSpot, it's matching on an external ID field before creating a new contact or company record, exactly the pattern behind Datamagnet's HubSpot integration for keeping enrichment data current without duplicate rows.

This step matters more than it looks. In 2025, Validity surveyed 602 CRM stakeholders and found 37% reported losing revenue directly because of poor data quality, and 76% said less than half of their org's CRM data is accurate and complete (Validity, The State of CRM Data Management in 2025, retrieved 2026-07-22). Duplicate enrichment writes are a direct, fixable contributor to that number.

The single most important instruction in this step: never let "insert a new record" be the only code path a webhook handler can take. Every write needs an upsert key, full stop.

Step 5: How Do You Handle Retries Without Amplifying Duplicates?

By the end of this step, your own retry logic won't accidentally recreate the duplicate-write problem you just fixed. Your handler still needs to return a fast, correct HTTP status — a 200 for "processed or already deduped," and a 5xx only for genuine failures you want retried. GitHub gives you just 10 seconds to respond before marking a delivery failed (GitHub, Handling failed webhook deliveries, retrieved 2026-07-22); Hookdeck's 2025 guidance for a general timeout budget is closer to 60 seconds (Hookdeck, Implementing Webhook Retries, retrieved 2026-07-22). Acknowledge receipt fast, then process asynchronously — don't make the provider wait on your enrichment logic.

Webhook Response Timeout Before a Delivery Is Marked Failed Lollipop chart. GitHub marks a delivery failed if the handler does not respond within 10 seconds. Hookdeck's 2025 guidance recommends a general timeout budget closer to 60 seconds. Source: GitHub and Hookdeck documentation, accessed 2026. 0s 20s 40s 60s GitHub (hard limit) 10 seconds Hookdeck (recommended) ~60 seconds Source: GitHub and Hookdeck documentation, accessed 2026

If your own downstream write fails after you've already returned a 200, don't silently drop the event. Log it to a dead-letter queue and replay it manually — that failure is now on you, not the webhook provider, and their retry mechanism (see the Datamagnet error reference for status codes worth retrying on) won't save you a second time.

Flowchart of an idempotent webhook handler: signature verification, dedup check, upsert write, with a branch to a dead-letter queue on failure

Step 6: How Do You Monitor Duplicate Rates and Catch Dedup-Store Failures?

By the end of this step, you'll know the moment your dedup layer starts letting duplicates through, instead of finding out from a confused sales rep three weeks later. Track two numbers on a dashboard: the percentage of incoming webhooks flagged as duplicates, and the error rate on your dedup store itself. A sudden spike in the first number usually means a provider had an outage and is retrying hard; a spike in the second means your dedup layer is the thing that's actually broken. Datamagnet's July 2026 changelog is a reminder that provider-side reliability improvements ship regularly — your monitoring should catch the gap between "provider changed retry behavior" and "your assumptions about it are now stale."

Alert on dedup-store write failures specifically, not just downstream write failures. If the store that's supposed to catch duplicates goes down, every retry becomes a fresh insert, and you won't see it in your regular error logs until someone notices the duplicate rows.

What Are the Most Common Mistakes to Avoid?

Most duplicate-write incidents trace back to one of five avoidable decisions, not to some rare edge case in the webhook provider. The pattern repeats across enrichment pipelines: someone dedupes on a payload hash instead of a stable event ID, skips the TTL or copies one from the wrong provider, trusts an in-memory check with no unique constraint as backstop, verifies signatures after the dedup check instead of before, or returns a 200 before confirming the downstream write actually succeeded. Here's each one, in the order teams usually hit them.

1. Deduping on a payload hash instead of the event ID. Teams do this because it feels like it doesn't require reading provider docs. The fix: always use the provider's documented stable identifier (event ID, SID, or signal ID) — never a hash of the body.

2. No TTL, or a TTL copied from the wrong provider. [ORIGINAL DATA] Across the enrichment integrations we've reviewed, mismatched TTLs are the single most common root cause when a "fixed" dedup layer breaks again months later after a new webhook source is added. Match the TTL to your longest retry window, not your first one.

3. Trusting the dedup check alone, with no unique constraint on the write. A race condition between two near-simultaneous deliveries can slip past an in-memory or even a database-backed check under load. The fix: pair dedup logic with a database-level unique constraint every time.

4. Verifying signatures after the dedup check, not before. This lets a spoofed request poison your dedup table with a false "already processed" entry for a real event ID.

5. Returning a 200 before confirming the write succeeded. If the downstream write then fails, you've told the provider not to retry an event you never actually processed. Always route write failures to a dead-letter queue you own.

What Does Success Look Like?

If you've followed all six steps, a webhook that arrives three times now produces exactly one CRM record, and your logs show two of those three attempts explicitly logged as "deduplicated," not silently dropped. Concretely, you should now have: a dedup store keyed on provider event IDs with a TTL matched to your longest retry window, upsert-based writes with a unique constraint as backstop, sub-10-second handler response times, and a dashboard tracking duplicate rate plus dedup-store errors. The next-level enhancement worth pursuing is extending the same event-ID pattern to your outbound integrations — for example, syncing job-change signals into workflow tools like n8n, where the same at-least-once assumptions apply on the receiving end.

Before-and-after comparison of three duplicate CRM contact cards versus one clean, verified contact record

Frequently Asked Questions

The questions below cover the edge cases enrichment teams hit most often once the six steps above are live: what idempotency actually protects against, why timestamps aren't a safe dedup key, what happens when the dedup store itself fails, how long to retain processed IDs, and how much latency the whole pattern adds.

What's the difference between idempotency and deduplication?

Deduplication is detecting that you've seen an event before; idempotency is designing the write itself so processing it twice produces the same result as processing it once. You need both — dedup catches most repeats early, and an idempotent write (like an upsert) protects you when a duplicate slips through anyway.

Can I just use the webhook payload's timestamp to detect duplicates?

No — timestamps aren't guaranteed stable across retries, and clock skew between systems makes them unreliable as a dedup key on their own. Use the provider's documented stable event identifier instead, such as an event ID, SID, or signal ID that stays constant across every retry attempt.

What if my dedup store goes down?

Fail loudly, not silently. Route incoming webhooks to a queue you can replay once the store recovers, and alert your team immediately — a downed dedup store means every retry becomes a fresh write, which is exactly the failure mode idempotency in enrichment pipelines is meant to prevent.

How long should I keep processed event IDs?

Match the TTL to the longest retry window among your webhook sources, then add a margin. Stripe's own idempotency keys age out after 24 hours (Stripe, Idempotent Requests documentation, retrieved 2026-07-22), but a 7-day window is a common practitioner choice when you're consuming from multiple providers with different retry schedules.

Does this add much latency to my pipeline?

Barely any. A dedup check against Redis or an indexed Postgres column typically adds low single-digit milliseconds, far under the 10-second response window GitHub enforces (GitHub, Handling failed webhook deliveries, retrieved 2026-07-22). The upsert itself costs about the same as a plain insert.

Building Pipelines That Don't Duplicate Themselves

You've now got a webhook pipeline that survives retries instead of multiplying them — a stable event ID extracted from every payload, a TTL-bound dedup store, and upsert-based writes with a unique constraint as backup. That's the difference between "the provider retried three times" and "three duplicate rows in your CRM."

If you're building or auditing a signal-driven enrichment pipeline from scratch, start with the Datamagnet quickstart to see the webhook payload shape you'll be deduping against, and check the champion-tracking cookbook for a worked example of job-change signals flowing through exactly this kind of idempotent handler.

Sources

← View all posts
Pratik Dani

About Pratik Dani

CEO, Founder

View Profile