Revenue Management Pricing & forecasting pipelines
4 min read Ingestion

Deduplicating webhook events with idempotency keys

Webhook delivery is at-least-once, which is a polite way of saying the same booking cancellation will sometimes arrive twice. Process it twice and you double-count a refund, re-trigger a price recalculation, or corrupt an availability count. Idempotency keys make duplicate delivery a non-event. This guide implements webhook deduplication, extending Webhook vs REST Sync Patterns within the Data Ingestion & OTA API Integration Workflows pillar.

Prerequisites

  • Python 3.11+ with fastapi
  • A fast, shared dedup store (Redis in production; an in-process dict here)
  • Signature verification already in place at the endpoint (HMAC), as in the parent cluster
  • A stable event identifier from the provider, or a way to derive one
  • Idempotent downstream handlers, consistent with Data Quality & Schema Contracts

Step 1 — Derive a stable idempotency key

Prefer the provider’s own event id. When none exists, derive a deterministic key by hashing the stable fields of the payload — never the receipt time, which differs between duplicates.

python 
from __future__ import annotations

import hashlib
import json
import logging

logger = logging.getLogger("webhook.dedup")


def idempotency_key(payload: dict) -> str:
    if event_id := payload.get("event_id"):
        return str(event_id)
    # Derive from stable business fields, sorted for determinism
    stable = {k: payload[k] for k in sorted(payload)
              if k in {"property_id", "booking_id", "status", "effective_date"}}
    digest = hashlib.sha256(json.dumps(stable, sort_keys=True).encode()).hexdigest()
    return f"derived:{digest}"

Step 2 — Claim the key before processing

Use an atomic set-if-absent to claim the key. If the claim fails, the event is a duplicate and is acknowledged without reprocessing. A short TTL bounds memory while covering the realistic redelivery window.

python 
_SEEN: dict[str, float] = {}
_TTL_SECONDS = 24 * 3600


def claim(key: str, now: float) -> bool:
    """Return True if this is the first time we've seen the key."""
    # prune expired
    for k, ts in list(_SEEN.items()):
        if now - ts > _TTL_SECONDS:
            del _SEEN[k]
    if key in _SEEN:
        return False
    _SEEN[key] = now         # in Redis: SET key now NX EX 86400
    return True

Step 3 — Wire dedup into the endpoint

Claim first, process only on a successful claim, and always return 200 so the provider stops redelivering. A duplicate is a success from the provider’s perspective — it delivered — so acknowledging it is correct.

python 
import time
from fastapi import FastAPI, Request

app = FastAPI()


async def process_event(payload: dict) -> None:
    logger.info("processing %s for %s", payload.get("status"),
                payload.get("property_id"))
    # ... idempotent downstream write ...


@app.post("/webhooks/ota")
async def receive(request: Request) -> dict:
    payload = await request.json()
    key = idempotency_key(payload)
    if not claim(key, time.monotonic()):
        logger.info("duplicate event %s ignored", key)
        return {"status": "duplicate"}
    await process_event(payload)
    return {"status": "accepted"}

Claiming the key before processing — not after — is what closes the race where two duplicate deliveries arrive concurrently: only one claim succeeds, so only one handler runs.

Verification and testing

python 
import time


def test_duplicate_claim_rejected() -> None:
    _SEEN.clear()
    now = time.monotonic()
    key = idempotency_key({"event_id": "evt_123"})
    assert claim(key, now) is True       # first delivery
    assert claim(key, now) is False      # duplicate delivery

A passing test confirms the second delivery of the same event id is refused, so process_event runs exactly once.

Common pitfalls and edge cases

  • Keying on receipt time. Duplicates arrive at different times; hash stable business fields, not timestamps.
  • Process-then-claim. Claiming after processing leaves a window for concurrent duplicates to both run; claim first.
  • No TTL. An unbounded dedup store grows forever; set a TTL longer than the provider’s max redelivery window.
  • Per-instance memory. An in-process dict does not dedup across replicas; use a shared store like Redis in production.
  • Non-idempotent handler. Dedup reduces but never fully eliminates double-processing across failures; keep the downstream write idempotent too.