You're building a pipeline that anchors files to a blockchain. Files arrive, you hash them, you submit the hash for anchoring, and then you wait. Polygon confirmation might come back in seconds. Bitcoin confirmation runs on a daily batch. Downstream systems (case management software, audit logs, document portals) need to know when confirmation happens. You need webhooks.

But a naive webhook implementation creates new failure modes: duplicate processing, replay attacks, silent failures on retry, and receivers that can't tell a legitimate event from a forged one. Four patterns address all of this.

HMAC-SHA256 Signature Verification

Your receiver needs proof that a webhook came from your system. HMAC-SHA256 ties a signature to both the payload content and a shared secret. If either changes, the signature fails.

On the dispatcher side:

import hashlib
import hmac
import json
import time

WEBHOOK_SECRET = "your-shared-secret"

def sign_payload(payload: dict, secret: str) -> str:
    body = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hmac.new(
        secret.encode("utf-8"),
        body.encode("utf-8"),
        hashlib.sha256
    ).hexdigest()

def build_event(event_type: str, data: dict, idempotency_key: str) -> dict:
    return {
        "event": event_type,
        "idempotency_key": idempotency_key,
        "timestamp": int(time.time()),
        "data": data
    }

On the receiver side:

from flask import Flask, request, abort
import hashlib
import hmac

app = Flask(__name__)
WEBHOOK_SECRET = "your-shared-secret"

def verify_signature(payload_bytes: bytes, signature: str, secret: str) -> bool:
    expected = hmac.new(
        secret.encode("utf-8"),
        payload_bytes,
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

@app.route("/webhook/proof-confirmed", methods=["POST"])
def handle_proof_confirmed():
    sig = request.headers.get("X-Webhook-Signature", "")
    if not verify_signature(request.data, sig, WEBHOOK_SECRET):
        abort(401)
    # continue processing...
    return "", 200

Two things matter here. First, use hmac.compare_digest() for the comparison, not ==. The constant-time comparison blocks timing attacks where an attacker probes the signature character by character to reconstruct it. Second, serialize with sort_keys=True on the dispatcher side. JSON object key ordering varies across languages and serializers. If the dispatcher and receiver produce different byte sequences from the same dict, signatures that should match won't.

Idempotency Keys

Retries happen. Network connections drop mid-flight. Receivers return a 500 that was actually processed. Your dispatcher re-sends. Without idempotency protection, the same event processes twice and you get duplicate records in your audit log or document management system.

The fix: include a stable idempotency key in every event. Derive it from the underlying fact, not from the delivery attempt. proof_{proof_id}_anchored is stable across retries. A freshly generated UUID per attempt is not.

import redis
from flask import Flask, request, abort

app = Flask(__name__)
r = redis.Redis(host="localhost", port=6379, db=0)
IDEMPOTENCY_TTL = 86400  # 24 hours

@app.route("/webhook/proof-confirmed", methods=["POST"])
def handle_proof_confirmed():
    sig = request.headers.get("X-Webhook-Signature", "")
    if not verify_signature(request.data, sig, WEBHOOK_SECRET):
        abort(401)

    event = request.get_json()
    key = event.get("idempotency_key")
    if not key:
        abort(400)

    cache_key = f"webhook:seen:{key}"
    if r.exists(cache_key):
        return "", 200  # already processed, safe to ack

    r.setex(cache_key, IDEMPOTENCY_TTL, "1")
    process_event(event)
    return "", 200

The TTL keeps your Redis key space from growing indefinitely. Set it to at least as long as your retry window. If you retry for up to 4 hours, a 24-hour TTL gives you a reasonable safety margin.

One edge case: what if the event processes but Redis fails before setex completes? You'd process it again on the next retry. For most use cases this is acceptable (a duplicate audit entry is annoying, not catastrophic). If you need strict exactly-once semantics, wrap the Redis write and the downstream write inside a database transaction instead.

Exponential Backoff with Jitter

When delivery fails, you retry. But if you retry on a fixed schedule and 50 events fail at the same time (a receiver restart, a transient network issue), they all retry simultaneously. That burst can cause the receiver to fail again.

Exponential backoff with jitter spaces retries out:

import requests
import random
import json
import hmac
import hashlib

WEBHOOK_SECRET = "your-shared-secret"

def dispatch_webhook(url: str, payload: dict, max_attempts: int = 5) -> bool:
    body = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    sig = hmac.new(
        WEBHOOK_SECRET.encode("utf-8"),
        body.encode("utf-8"),
        hashlib.sha256
    ).hexdigest()
    headers = {
        "Content-Type": "application/json",
        "X-Webhook-Signature": sig,
    }

    for attempt in range(max_attempts):
        try:
            resp = requests.post(url, data=body, headers=headers, timeout=10)
            if resp.status_code == 200:
                return True
            if resp.status_code in (400, 401, 403):
                return False  # non-retryable
        except requests.RequestException:
            pass

        if attempt < max_attempts - 1:
            delay = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(delay)

    return False

The non-retryable check matters. A 401 means the signature is wrong. Retrying won't fix that. You need to investigate the shared secret or signing logic. A 400 means malformed payload. Only retry on 5xx responses and network errors where the problem might actually resolve.

The jitter term (random.uniform(0, 1)) breaks synchronization across concurrent dispatchers. Without it, multiple workers retrying at the same base delay still hit the receiver in a synchronized wave.

Replay Protection

An attacker who intercepts a valid webhook can replay it later. The payload is signed, but the signature is still valid because neither the secret nor the content changed. For any event that triggers a state change (marking a proof as confirmed, updating a claim record), replay is a real attack surface.

The timestamp window blocks this. Include a Unix timestamp in the payload and reject events where that timestamp falls outside a 5-minute window:

import time
from flask import Flask, request, abort

REPLAY_WINDOW_SECONDS = 300

@app.route("/webhook/proof-confirmed", methods=["POST"])
def handle_proof_confirmed():
    sig = request.headers.get("X-Webhook-Signature", "")
    if not verify_signature(request.data, sig, WEBHOOK_SECRET):
        abort(401)

    event = request.get_json()

    event_ts = event.get("timestamp", 0)
    if abs(int(time.time()) - event_ts) > REPLAY_WINDOW_SECONDS:
        abort(400)

    key = event.get("idempotency_key")
    if not key:
        abort(400)

    cache_key = f"webhook:seen:{key}"
    if r.exists(cache_key):
        return "", 200

    r.setex(cache_key, IDEMPOTENCY_TTL, "1")
    process_event(event)
    return "", 200

The timestamp check and idempotency check cover different attack surfaces. The timestamp window rejects replays from outside 5 minutes. The idempotency key catches exact duplicates within the window. You need both.

Operational note: if your dispatcher and receiver clocks aren't synchronized (NTP drift is common in container environments), you'll get false rejections near the window boundary. Keep the window at 5 minutes minimum, and monitor for 400 errors that correlate with clock skew alerts on your infrastructure.

The Full Event Shape

Here's the payload structure that ties all four patterns together:

event = build_event(
    event_type="proof.anchored",
    idempotency_key=f"proof_{proof_id}_anchored",
    data={
        "proof_id": proof_id,
        "sha256": file_hash,
        "blockchain": "polygon",
        "tx_id": transaction_id,
        "anchored_at": anchored_at_iso,
    }
)

The receiver verifies the signature, checks the timestamp window, checks the idempotency key, then processes. Four checks, one linear code path.

These patterns apply to any async workflow where the triggering action and the confirmation happen at different times on different schedules. Blockchain anchoring is an obvious example because confirmation is inherently network-driven. I use all four in ProofLedger because Polygon and Bitcoin settle on completely different timelines, and the downstream systems that consume proof confirmations can't afford to handle duplicates or forged events.

What's your approach when the event processes successfully but the idempotency write fails? Database transaction around both, a separate dedup table, or something else?