Webhook Testing: How to Guarantee Delivery, Retry Logic, and Correct Event Ordering

Webhooks carry critical business events: payment confirmations, subscription activations, user signups, order completions. When they work, they're invisible. When they fail, they fail in the most insidious way possible — your application's state diverges from the provider's state, and nobody notices until a customer complains.

The specific challenges: webhooks are delivered over public HTTP, they can arrive out of order, they will be retried on timeout or error, and they must be secured against spoofing. Every one of these characteristics creates testable failure modes.

---

Webhook Flow Architecture

sequenceDiagram
    participant Provider as Provider (Paddle/Stripe)
    participant Handler as Your Webhook Handler
    participant DB as Database
    participant Queue as Job Queue

    Provider->>Handler: POST /webhooks/paddle { event: "subscription.created" }
    Handler->>Handler: 1. Verify signature (HMAC)
    Handler->>Handler: 2. Check idempotency key
    Handler->>DB: 3. Store raw event
    Handler->>Queue: 4. Enqueue for processing
    Handler-->>Provider: 200 OK (fast response)
    Queue->>DB: 5. Process: activate user subscription

    Note over Provider,Handler: If 200 not received in 30s,<br/>Provider retries up to 10×

---

Test Infrastructure: Local Webhook Testing

The biggest challenge in webhook testing is receiving actual HTTP POST requests from external providers. For local and CI testing, use a mock webhook server:

// tests/helpers/webhook-server.ts
import http from 'http';
import { EventEmitter } from 'events';

interface WebhookRequest {
  headers: Record<string, string | string[]>;
  body: unknown;
  timestamp: Date;
}

export class MockWebhookServer extends EventEmitter {
  private server: http.Server;
  private receivedRequests: WebhookRequest[] = [];
  public port: number;

  constructor(port = 0) {
    super();
    this.port = port;

    this.server = http.createServer((req, res) => {
      let body = '';
      req.on('data', (chunk) => (body += chunk.toString()));
      req.on('end', () => {
        const request: WebhookRequest = {
          headers: req.headers as Record<string, string>,
          body: JSON.parse(body),
          timestamp: new Date(),
        };

        this.receivedRequests.push(request);
        this.emit('webhook', request);

        // Respond 200 to acknowledge
        res.writeHead(200);
        res.end('OK');
      });
    });
  }

  async start(): Promise<number> {
    return new Promise((resolve) => {
      this.server.listen(0, () => {
        this.port = (this.server.address() as { port: number }).port;
        resolve(this.port);
      });
    });
  }

  async waitForWebhook(timeoutMs = 5000): Promise<WebhookRequest> {
    return new Promise((resolve, reject) => {
      const timer = setTimeout(() => reject(new Error('Webhook not received')), timeoutMs);
      this.once('webhook', (req) => {
        clearTimeout(timer);
        resolve(req);
      });
    });
  }

  getReceivedRequests = () => this.receivedRequests;

  async stop(): Promise<void> {
    return new Promise((resolve) => this.server.close(() => resolve()));
  }
}

---

Test 1: Webhook Signature Verification

// tests/webhooks/signature.test.ts
import { describe, it, expect, beforeEach } from 'vitest';
import { createHmac } from 'crypto';
import { verifyPaddleWebhookSignature } from '../../lib/billing/paddle-webhook';

const WEBHOOK_SECRET = 'test-webhook-secret-123';

function createSignedPayload(body: object, secret: string) {
  const payload = JSON.stringify(body);
  const timestamp = Math.floor(Date.now() / 1000).toString();
  const signedString = `${timestamp}:${payload}`;
  const signature = createHmac('sha256', secret).update(signedString).digest('hex');

  return {
    payload,
    headers: {
      'paddle-signature': `ts=${timestamp};h1=${signature}`,
    },
  };
}

describe('Paddle webhook signature verification', () => {
  it('accepts a valid signature', () => {
    const { payload, headers } = createSignedPayload(
      { event_type: 'subscription.created', data: { id: 'sub_123' } },
      WEBHOOK_SECRET,
    );

    expect(() => verifyPaddleWebhookSignature(payload, headers, WEBHOOK_SECRET)).not.toThrow();
  });

  it('rejects a tampered payload', () => {
    const { headers } = createSignedPayload(
      { event_type: 'subscription.created', data: { id: 'sub_123' } },
      WEBHOOK_SECRET,
    );

    // Tamper with payload after signing
    const tamperedPayload = JSON.stringify({
      event_type: 'subscription.created',
      data: { id: 'sub_ATTACKER_ID' },
    });

    expect(() => verifyPaddleWebhookSignature(tamperedPayload, headers, WEBHOOK_SECRET)).toThrow(/invalid signature/i);
  });

  it('rejects an expired timestamp (replay attack)', () => {
    const body = { event_type: 'subscription.created', data: { id: 'sub_123' } };
    const payload = JSON.stringify(body);

    // Create signature with a timestamp 6 minutes ago (beyond 5-min tolerance)
    const oldTimestamp = Math.floor(Date.now() / 1000) - 360;
    const signedString = `${oldTimestamp}:${payload}`;
    const signature = createHmac('sha256', WEBHOOK_SECRET).update(signedString).digest('hex');

    const headers = { 'paddle-signature': `ts=${oldTimestamp};h1=${signature}` };

    expect(() => verifyPaddleWebhookSignature(payload, headers, WEBHOOK_SECRET)).toThrow(/timestamp too old/i);
  });
});

---

Test 2: Idempotency (Duplicate Delivery Handling)

Webhook providers retry on any non-2xx response. Your handler must be idempotent:

// tests/webhooks/idempotency.test.ts

test('duplicate webhook delivery does not double-activate subscription', async () => {
  const subscriptionEvent = {
    event_type: 'subscription.created',
    event_id: 'evt_UNIQUE_ID_123', // Provider's idempotency key
    data: {
      subscription_id: 'sub_456',
      customer_email: 'test@example.com',
      plan: 'pro',
    },
  };

  // Deliver the same event twice
  const [response1, response2] = await Promise.all([
    request.post('/webhooks/paddle', {
      data: subscriptionEvent,
      headers: signWebhookPayload(subscriptionEvent),
    }),
    request.post('/webhooks/paddle', {
      data: subscriptionEvent,
      headers: signWebhookPayload(subscriptionEvent),
    }),
  ]);

  // Both should return 200 (idempotent)
  expect(response1.status()).toBe(200);
  expect(response2.status()).toBe(200);

  // But user should only have ONE subscription record
  const subscriptions = await getSubscriptionsForEmail('test@example.com');
  expect(subscriptions).toHaveLength(1);
  expect(subscriptions[0].plan).toBe('pro');
});

---

Test 3: Event Ordering

Events can arrive out of order. A subscription.updated might arrive before subscription.created if there's network variability:

// tests/webhooks/ordering.test.ts

test('out-of-order events are handled correctly', async ({ request }) => {
  const baseData = { subscription_id: 'sub_789', customer_email: 'order-test@example.com' };

  // Send "updated" before "created" (out of order)
  await request.post('/webhooks/paddle', {
    data: { event_type: 'subscription.updated', event_id: 'evt_2', data: { ...baseData, plan: 'enterprise' } },
    headers: signWebhookPayload({}),
  });

  await request.post('/webhooks/paddle', {
    data: { event_type: 'subscription.created', event_id: 'evt_1', data: { ...baseData, plan: 'pro' } },
    headers: signWebhookPayload({}),
  });

  // The system should resolve to a consistent final state
  // (implementation-dependent: could use event timestamps or sequence numbers)
  const subscription = await getSubscription('sub_789');

  // Should not be undefined or in an inconsistent state
  expect(subscription).not.toBeNull();
  expect(['pro', 'enterprise']).toContain(subscription!.plan);

  // Most importantly: no 500 errors, no data corruption
});

---

Test 4: Retry Behavior Simulation

// tests/webhooks/retry.test.ts

test('webhook handler is resilient to temporary processing failures', async () => {
  let attemptCount = 0;

  // Mock the downstream service to fail the first 2 times
  server.use(
    http.post('/api/internal/activate-subscription', (req, res, ctx) => {
      attemptCount++;
      if (attemptCount < 3) {
        return res(ctx.status(503), ctx.text('Service temporarily unavailable'));
      }
      return res(ctx.status(200), ctx.json({ activated: true }));
    }),
  );

  // Deliver webhook
  const response = await request.post('/webhooks/paddle', {
    data: subscriptionCreatedEvent,
    headers: signWebhookPayload(subscriptionCreatedEvent),
  });

  // Webhook handler should return 200 quickly (event stored, processing is async)
  expect(response.status()).toBe(200);

  // Wait for retry logic to succeed
  await waitForCondition(() => getSubscriptionStatus('sub_123') === 'active', { timeout: 10_000 });

  expect(attemptCount).toBe(3); // Failed twice, succeeded on third
});

Related articles: Also see API testing best practices that apply to webhook endpoint validation, testing real-time event delivery alongside webhook retry logic, and integrating webhook delivery tests into your CI/CD quality gates.

---

Webhook Testing Checklist

Test	Priority	Automate?
Valid signature accepted	🔴 Critical	✅ Yes
Invalid signature rejected	🔴 Critical	✅ Yes
Replay attack rejected (stale timestamp)	🔴 Critical	✅ Yes
Duplicate event idempotency	🔴 Critical	✅ Yes
Out-of-order event handling	🟡 High	✅ Yes
Slow handler timeout behavior	🟡 High	✅ Yes
Empty/malformed payload	🟡 High	✅ Yes
Unknown event type handling	🟡 High	✅ Yes
Provider retry simulation	🟡 High	✅ Yes
Downstream service failure recovery	🟡 High	✅ Yes

Webhooks are deceptively simple — they're just HTTP POST requests — but the failure modes are broad and the consequences of a missed webhook in a billing or auth context can be severe.

Monitor your webhook endpoints and integration health continuously: Try ScanlyApp free and set up automated health checks that verify your integrations are functioning correctly.