A webhook is an HTTP POST that a third-party service sends to your server when something happens on their end. Your payment provider tells you a payment succeeded. Your version control platform tells you a PR was merged. Your email provider tells you a message bounced.

The concept is straightforward. The implementation has several non-obvious requirements that, if missed, produce unreliable or insecure behavior.

How Webhooks Work

The pattern:

1. You register a URL with the third-party service: “when event X happens, POST to https://yourapp.com/webhooks/stripe”
2. The event happens on their side
3. They send an HTTP POST to your URL with a JSON body describing the event
4. Your server processes the event and returns a 2xx response
5. If you return a non-2xx response or don’t respond in time, they retry

This is opposite to the request-response model where you initiate the call. With webhooks, the third party initiates. Your server must be reachable, respond quickly, and handle the same event potentially arriving more than once.

The Receiver

The minimal webhook receiver:

app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
    // 1. Verify the signature (before anything else)
    const sig = req.headers['stripe-signature'];
    let event;
    try {
        event = stripe.webhooks.constructEvent(req.body, sig, process.env.STRIPE_WEBHOOK_SECRET);
    } catch (err) {
        return res.status(400).send(`Webhook signature verification failed: ${err.message}`);
    }

    // 2. Respond immediately - before processing
    res.status(200).json({ received: true });

    // 3. Process the event asynchronously
    await processWebhookEvent(event);
});

Three things happen in this order, and the order matters.

Respond immediately. Webhook providers have short timeouts - typically 5-30 seconds. If you do database work, call other services, or send emails before responding, you risk timing out. The provider sees a timeout as a failed delivery and retries. You end up processing the same event multiple times. Respond with 200, enqueue the work, process it separately.

Verify before you process. The signature verification must happen before you act on the payload. An unverified webhook is an unauthenticated HTTP endpoint that anyone can POST to.

Verifying Authenticity

Any HTTP endpoint that takes actions based on incoming data needs authentication. For webhooks, the standard approach is HMAC signature verification.

The provider generates a signature of the request body using a shared secret. They include it in a header. Your server recomputes the signature using the same secret and compares:

const crypto = require('crypto');

function verifyWebhookSignature(body, signature, secret) {
    const expectedSignature = crypto
        .createHmac('sha256', secret)
        .update(body)
        .digest('hex');

    // Constant-time comparison to prevent timing attacks
    return crypto.timingSafeEqual(
        Buffer.from(signature),
        Buffer.from(expectedSignature)
    );
}

Two things to get right:

• Use the raw body, not the parsed JSON. Parsing and re-serializing JSON can change byte order or spacing, changing the hash.
• Use constant-time comparison (timingSafeEqual), not ===. Regular string comparison short-circuits on the first differing character, creating a timing side channel.

Most providers (Stripe, GitHub, Shopify) document their specific signature format. Follow the documentation exactly - the header name, the prefix on the signature string, the hash algorithm.

Idempotency

Webhook providers retry on failure. If your server returns a 500, or takes too long, or the network drops the response - the provider will retry the same event. Your handler will be called multiple times for the same event.

Handlers that are not idempotent cause problems:

// Wrong: charging twice if called twice for same event
async function handlePaymentSucceeded(event) {
    const payment = event.data.object;
    await db.createInvoice({ paymentId: payment.id, amount: payment.amount });
    await fulfillOrder(payment.metadata.orderId);
    await sendReceiptEmail(payment.metadata.email);
}

If this runs twice, the customer gets charged once but receives two invoices, has their order fulfilled twice, and gets two receipt emails.

The fix: check whether you’ve already processed this event before doing the work.

async function handlePaymentSucceeded(event) {
    const payment = event.data.object;

    // Check idempotency key before processing
    const alreadyProcessed = await db.processedEvents.findOne({ eventId: event.id });
    if (alreadyProcessed) return;

    // Mark as processing (with a transaction to handle race conditions)
    await db.transaction(async (trx) => {
        await trx.processedEvents.insert({ eventId: event.id, processedAt: new Date() });
        await trx.invoices.insert({ paymentId: payment.id, amount: payment.amount });
    });

    // Non-critical work after the transaction
    await fulfillOrder(payment.metadata.orderId);
    await sendReceiptEmail(payment.metadata.email);
}

The event ID is the idempotency key. Check it before processing, insert it atomically with the critical work. Subsequent deliveries of the same event find the record and skip processing.

Ordering and Out-of-Order Delivery

Webhooks do not arrive in guaranteed order. A payment.updated event might arrive before the payment.created event if there’s a delivery delay on the first. Your handler needs to be resilient to this.

Common approaches:

• Treat each event as a snapshot: instead of applying incremental updates, overwrite with the current state from the event
• Fetch the current state from the provider’s API when an event arrives rather than trusting the event payload
• Store events and process them in order, handling the case where an event references an entity that doesn’t exist yet

Debugging

Webhook debugging is annoying in development because the provider needs to reach your local machine. The standard tool is a tunnel: ngrok, Cloudflare Tunnel, or the provider’s own CLI (Stripe has stripe listen) creates a public URL that forwards to localhost.

# Stripe CLI - forwards to localhost:3000/webhooks/stripe
stripe listen --forward-to localhost:3000/webhooks/stripe

For production debugging:

• Log the full event payload on receipt, before processing
• Log the event ID in every log line related to that event so you can trace a single event through your logs
• Store processed events in your database - you can replay them manually if you need to reprocess

Most providers have a dashboard with webhook delivery history: you can see each delivery attempt, the response code, and the response body. This is where you look when a webhook silently fails.

Handling Failures

Your handler will fail sometimes. The database is down, the external API times out, an unhandled exception. Returning a non-2xx causes the provider to retry, which is usually what you want.

But retries from the provider have limits - typically 24-72 hours, sometimes fewer attempts. For events that fail past the retry window, you need a way to reprocess them manually.

The dead letter queue pattern: move failed events to a separate queue or table after exhausting retries. Review them, fix the underlying issue, replay them.

The minimum viable approach: log every received event with its payload. If something goes wrong, you have the data and can replay it by hand.

One Webhook Per Concern

It’s tempting to have one webhook endpoint that handles all event types:

app.post('/webhooks', async (req, res) => {
    switch (req.body.type) {
        case 'payment.succeeded': // ...
        case 'payment.refunded': // ...
        case 'subscription.cancelled': // ...
        // 20 more cases...
    }
});

This works but grows into a mess. Separate endpoints per event type (or per logical group) are easier to monitor, easier to deploy changes to independently, and easier to debug when something fails.