Webhook Orchestrator

The Problem

• Serverless Timeouts: Serverless platforms (like Vercel or Netlify) strictly limit execution times. If a webhook delivery takes too long or a connection hangs, the serverless function is killed.
• Downstream Failures & Data Loss: If the target server receiving the webhook experiences an outage, traditional synchronous apps will fail to deliver the data, leading to permanent data loss and breaking integrations.

The Solution

• High-Speed Ingestion: When a client sends a webhook, the Next.js API instantly validates the request and offloads the payload to a Redis queue. It immediately responds to the client with a 202 Accepted status. This guarantees the public-facing API remains highly available and never times out.
• Reliable Background Processing: A standalone Node.js worker continuously monitors the Redis queue. It pulls jobs off the queue and attempts to deliver them to their final destination at a safe, controlled rate.
• Fault Tolerance & Exponential Backoff: If the target server is down, the system doesn't crash or discard the webhook. Instead, the worker employs an Exponential Backoff strategy—automatically waiting and retrying the delivery up to 5 times.
• The Dead Letter Queue (DLQ): To guarantee zero data loss, any webhook that fails all 5 retry attempts is safely caught and permanently logged into a PostgreSQL database known as a Dead Letter Queue.

Architecture & Workflow

[Client Application] ──> [Next.js Ingestion API] ──(202 Accepted)──> [Client]
                                       │
                                       ▼ (Offload Payload)
                                 [Redis Queue]
                                       │
                                       ▼ (Pull Job)
                            [Node.js Background Worker]
                                       │
                ┌──────────────────────┴──────────────────────┐
                ▼                                             ▼
          (Success)                                    (Failure - 5 Retries)
                │                                             │
      [Target Server / Webhook URL]                           ▼
                                                  [Dead Letter Queue (PostgreSQL)]

• Ingestion: The Next.js API receives the webhook, verifies the HMAC signature, queues it in Redis, and immediately returns a 202 status.
• Processing: The Node.js worker pulls jobs from Redis and attempts delivery to the target server, controlled by a token-bucket rate limiter.
• Retry Logic: If the target server is unreachable, the worker retries the delivery using an Exponential Backoff strategy (up to 5 times).
• Dead Letter Queue: Failed webhooks that exhaust all retries are permanently stored in PostgreSQL for manual auditing and recovery.

Key Technical Features

• Decoupled Architecture: Effectively separated the fast, serverless Next.js API from the heavy-lifting, stateful Node.js background worker.
• Idempotency: Implemented Idempotency-Key checks to ensure that if the exact same data is accidentally sent twice, the system recognizes it and processes it only once.
• Token-Bucket Rate Limiting: Built global rate limits to prevent DDoS attacks and tightly control the speed at which webhooks are ingested and delivered.
• Cryptographic Security: Enforced zero-trust security using HMAC SHA-256 signatures to verify the authenticity of all incoming and outgoing webhooks.
• Docker Orchestration: Containerized the entire stack (PostgreSQL, Redis, Next.js, Node Worker) using docker-compose to ensure the local development environment perfectly mirrors production.

Challenges & Learnings

Conclusion