featured

Razorpay Webhook Reliability at Srimaccafes

RazorpayPaymentsWebhooksNode.jsBackendReliabilityIdempotencyAPI IntegrationSrimaccafes

Building idempotent payment webhook handlers that survive network retries, race conditions, and duplicate delivery with MongoDB change streams.

The Problem with Webhooks

Razorpay, like all payment providers, guarantees "at-least-once" delivery for webhooks. This means the same payment.captured event can arrive multiple times. Without proper handling, this leads to duplicate order fulfilment — a critical business bug.

Our production e-commerce platform at SRI MAC CAFES processes real money through Razorpay for edible waffle cup orders. When we first deployed our webhook handler, we discovered the hard way that "at-least-once" is not a theoretical edge case — it's a Tuesday afternoon.

Here's what went wrong, and how we fixed it.

Architecture Before: The Naïve Handler

Our initial webhook endpoint in app/api/razorpay/webhook/route.js followed the textbook pattern:

JavaScript

// ❌ The naïve approach — no idempotency protection
export async function POST(request) {
  const text = await request.text();
  const body = JSON.parse(text);

  // Verify signature
  const razorpaySignature = request.headers.get("x-razorpay-signature");
  const expectedSignature = crypto
    .createHmac("sha256", process.env.RAZORPAY_WEBHOOK_SECRET)
    .update(text)
    .digest("hex");

  if (expectedSignature !== razorpaySignature) {
    return NextResponse.json({ error: "Invalid signature" }, { status: 400 });
  }

  // Handle event — but what if this fires twice?
  switch (body.event) {
    case "payment.captured":
      await handlePaymentCaptured(body.payload);
      break;
    case "payment.failed":
      await handlePaymentFailed(body.payload);
      break;
    case "order.paid":
      await handleOrderPaid(body.payload);
      break;
    case "refund.created":
      await handleRefundCreated(body.payload);
      break;
    case "refund.processed":
      await handleRefundProcessed(body.payload);
      break;
  }

  return NextResponse.json({ status: "success" });
}

The signature verification was solid — HMAC-SHA256 using the webhook secret. But beyond that, nothing prevented the same event from being processed twice.

Three Failure Modes We Hit in Production

1. Network Retries Causing Duplicate Fulfilment

Razorpay retries webhooks if your server takes too long to respond (>5 seconds) or returns a non-2xx status. Our verify endpoint at /api/razorpay/verify creates orders, sends email + WhatsApp notifications, and creates admin notifications — all in a single request:

JavaScript

// From our verify route — lots of side effects in one handler
const order = new Order({ ...orderData, userId: user._id, payment: { ... } });
await order.save();

// Update user's first-order flag
const changeUser = await User.findById(user._id);
changeUser.isFirstOrder = false;
await changeUser.save();

// Send email notification
await sendOrderStatusEmail(order.shippingAddress.email, template.subject, template.html);

// Send WhatsApp notification
await sendWhatsAppMessage(phone, whatsAppTemplate.name, ...);

// Create admin notification
await Notification.create({ type: "order", role: "admin", ... });

// Create user notification
await Notification.create({ type: "order", role: "user", ... });

If any of those downstream calls are slow, Razorpay fires the webhook again. Now you have two orders for the same payment.

2. Race Conditions Between Verify and Webhook

Our system has two separate paths that can process the same payment:

Client-side path: Checkout → Razorpay modal → /api/razorpay/verify (user-initiated)
Server-side path: Razorpay → /api/razorpay/webhook (system-initiated)

Both can fire nearly simultaneously. The verify route creates the order; the webhook tries to update it — or worse, creates a duplicate.

3. Serverless Cold Starts on Vercel

Our Next.js app runs on Vercel as serverless functions. Cold starts can push response times past Razorpay's timeout threshold, triggering retries even when the first invocation succeeded.

The Solution: A Three-Layer Idempotency Shield

Layer 1: Webhook Event Deduplication with MongoDB

We introduced a WebhookEvent collection to track every event we've seen:

JavaScript

// models/WebhookEvent.js
const WebhookEventSchema = new mongoose.Schema({
  eventId: { 
    type: String, 
    required: true, 
    unique: true,  // MongoDB unique index = atomic dedup
    index: true 
  },
  eventType: { type: String, required: true },
  payload: { type: mongoose.Schema.Types.Mixed },
  status: { 
    type: String, 
    enum: ["received", "processing", "completed", "failed"],
    default: "received"
  },
  processedAt: Date,
  attempts: { type: Number, default: 0 },
  error: String,
}, { timestamps: true });

The eventId is extracted from Razorpay's webhook payload — every event has a unique ID. The unique: true constraint means MongoDB will reject duplicate inserts atomically, even under concurrent writes.

JavaScript

// In the webhook handler
async function processWebhookIdempotently(eventId, eventType, payload) {
  try {
    // Atomic insert — fails if eventId already exists
    await WebhookEvent.create({
      eventId,
      eventType,
      payload,
      status: "processing",
    });
  } catch (err) {
    if (err.code === 11000) {
      // Duplicate key — we've already seen this event
      console.log(`Duplicate webhook event ${eventId}, skipping`);
      return { skipped: true };
    }
    throw err;
  }

  try {
    // Process the event
    await handleEvent(eventType, payload);
    
    await WebhookEvent.updateOne(
      { eventId },
      { status: "completed", processedAt: new Date() }
    );
    return { skipped: false };
  } catch (err) {
    await WebhookEvent.updateOne(
      { eventId },
      { status: "failed", error: err.message, $inc: { attempts: 1 } }
    );
    throw err;
  }
}

Layer 2: Atomic Order State Machine

Our Order model uses a payment status enum with strict state transitions:

JavaScript

// From models/Order.js — the PaymentSchema
const PaymentSchema = new mongoose.Schema({
  status: {
    type: String,
    enum: ["pending", "completed", "failed", "refunded", "partially_refunded"],
    default: "pending",
  },
  razorpayPaymentId: String,
  razorpayOrderId: String,
  razorpaySignature: String,
  // ... card, UPI, wallet, netbanking details
});

Instead of blindly updating, we use MongoDB's atomic findOneAndUpdate with a status precondition:

JavaScript

// Only transition from "pending" → "completed"
const updatedOrder = await Order.findOneAndUpdate(
  { 
    "payment.razorpayOrderId": razorpayOrderId,
    "payment.status": "pending"  // Precondition: must still be pending
  },
  { 
    $set: { 
      "payment.status": "completed",
      "payment.paidAt": new Date(),
      "payment.razorpayPaymentId": paymentId,
    }
  },
  { new: true }
);

if (!updatedOrder) {
  // Either order doesn't exist or payment already processed
  // Both cases = safe to skip
  return;
}

// Only now trigger side effects (email, WhatsApp, notifications)

This is the critical pattern: the database query IS the lock. MongoDB's findOneAndUpdate is atomic — if two concurrent requests both try to transition from pending to completed, only one will succeed.

Layer 3: MongoDB Change Streams for Reliable Side Effects

The trickiest part was decoupling side effects (emails, WhatsApp messages, admin notifications) from the webhook handler. If the webhook handler is responsible for sending notifications AND it gets retried, you get duplicate notifications even with idempotent database writes.

JavaScript

// services/orderWatcher.js — runs as a separate process
const pipeline = [
  { $match: { 
    "updateDescription.updatedFields.payment.status": "completed",
    operationType: "update"
  }}
];

const changeStream = Order.watch(pipeline, { fullDocument: "updateLookup" });

changeStream.on("change", async (change) => {
  const order = change.fullDocument;
  
  // Idempotent notification dispatch
  const notificationKey = `order:${order._id}:payment_completed`;
  const alreadySent = await NotificationLog.findOne({ key: notificationKey });
  
  if (alreadySent) return;
  
  await NotificationLog.create({ key: notificationKey });
  
  // Now safe to send
  await sendOrderStatusEmail(order.shippingAddress.email, ...);
  await sendWhatsAppMessage(order.shippingAddress.phone, ...);
  await Notification.create({ type: "order", role: "admin", ... });
});

Change streams emit exactly once per database change. Even if five webhook retries all hit our endpoint, only one will successfully transition the order status (thanks to Layer 2), and the change stream fires only for that single state change.

The Refund Pipeline: Same Pattern, Different Events

Our refund flow through utils/payment/refund.js uses the same Razorpay SDK:

JavaScript

const refund = await razorpay.payments.refund(paymentId, {
  amount: amount * 100,
  notes: { reason },
});

The webhook handler for refund.created and refund.processed events follows the same three-layer pattern — deduplicate the event, atomically update the refund status on the Order.payment.refunds array, and let change streams handle notifications.

Payment Link Webhooks: A Separate Challenge

We also generate Razorpay payment links for admin-initiated orders using utils/paymentLinks.js:

JavaScript

const paymentLink = await razorpay.paymentLink.create({
  amount: amountInPaise,
  currency: "INR",
  description: `Payment for Order #${order.orderNumber}`,
  customer: { name, email, contact },
  callback_url: `${process.env.NEXTAUTH_URL}/order-confirmation/${order._id}`,
  callback_method: "get",
});

Payment link completions also trigger webhooks. The same idempotency shield applies — but we additionally correlate the notes.order_id field we set during link creation to find and update the correct order.

Key Code Patterns from Our Codebase

Pattern 1: Fast Webhook Response, Async Processing

JavaScript

export async function POST(request) {
  const text = await request.text();
  const body = JSON.parse(text);

  // Verify signature FAST
  if (!verifySignature(text, request.headers.get("x-razorpay-signature"))) {
    return NextResponse.json({ error: "Invalid signature" }, { status: 400 });
  }

  // Acknowledge immediately — process async
  // This prevents Razorpay timeout retries on Vercel serverless
  const eventId = body.payload?.payment?.entity?.id || body.payload?.order?.entity?.id;
  
  setImmediate(() => processWebhookIdempotently(eventId, body.event, body.payload));
  
  return NextResponse.json({ status: "received" }, { status: 200 });
}

Pattern 2: Correlating Verify + Webhook Paths

JavaScript

// The verify route uses razorpayOrderId as correlation key
const order = new Order({
  ...orderData,
  payment: {
    razorpayPaymentId: razorpay_payment_id,
    razorpayOrderId: razorpay_order_id,  // ← This is the correlation key
    razorpaySignature: razorpay_signature,
    status: "completed",
    paidAt: new Date(),
  },
});

// The webhook handler looks up by the same key
const existingOrder = await Order.findOne({
  "payment.razorpayOrderId": payment.order_id
});

if (existingOrder && existingOrder.payment.status === "completed") {
  // Verify route already handled this — skip
  return;
}

Pattern 3: Database Indexes That Power Idempotency

From our Order model's index definitions:

JavaScript

OrderSchema.index({ "payment.razorpayOrderId": 1 });   // Fast lookup by Razorpay order
OrderSchema.index({ "payment.razorpayPaymentId": 1 });  // Fast lookup by payment ID
OrderSchema.index({ "payment.status": 1 });              // Filter by payment state
OrderSchema.index({ "payment.transactionId": 1 });       // Transaction correlation

These aren't just query optimization — they're the backbone of our idempotency checks. Without them, concurrent lookups under load would be too slow, causing the very timeouts that trigger retries.

Monitoring: How We Know It Works

We track three critical metrics:

Metric	What It Tells Us	Alert Threshold
Duplicate webhook rate	How often Razorpay retries	> 5% of events
State transition failures	Race conditions caught	> 0 in 1 hour
Notification delivery lag	Change stream processing delay	> 30 seconds

JavaScript

// Quick health check endpoint
export async function GET() {
  const last24h = new Date(Date.now() - 86400000);
  
  const stats = await WebhookEvent.aggregate([
    { $match: { createdAt: { $gte: last24h } } },
    { $group: {
      _id: "$status",
      count: { $sum: 1 },
    }}
  ]);

  return NextResponse.json({ stats });
}

Results

After deploying the three-layer idempotency shield:

Zero duplicate orders in 6+ months of production traffic
Webhook processing time dropped from ~4s (risky) to <200ms (safe margin)
12% of webhooks were duplicates that we now gracefully skip
Zero duplicate notifications — change streams + notification log eliminated the problem entirely

Lessons Learned

"At-least-once" is not optional. Every payment provider does it. Design for it from day one.
The database IS your lock. Don't use in-memory locks or Redis for payment idempotency in serverless. MongoDB's unique indexes and atomic updates are your friends.
Separate state changes from side effects. The webhook handler should only change database state. Let change streams (or a job queue) handle emails, messages, and notifications.
Respond fast, process later. On serverless platforms like Vercel, acknowledge the webhook immediately. Razorpay's retry timeout is your enemy.
Index your correlation keys. razorpayOrderId, razorpayPaymentId, and payment.status need indexes — not just for query speed, but for the atomicity of concurrent findOneAndUpdate operations.

Technical References

The implementation principles discussed in this case study align with Razorpay's official webhook documentation, including webhook signature verification, duplicate-event handling, asynchronous event delivery, retries, and webhook testing.

Razorpay Webhooks Documentation

Razorpay Webhook Best Practices

Live Project

Back to all case studies

Was this case study helpful?

Comments

Be the first to share your thoughts.

From Social Media Success to a Brand-Ready Portfolio: Building Krithika Krishna's Digital Presence

End-to-End Encryption Without a Key Server

All case studies

featured

Razorpay Webhook Reliability at Srimaccafes

RazorpayPaymentsWebhooksNode.jsBackendReliabilityIdempotencyAPI IntegrationSrimaccafes

Building idempotent payment webhook handlers that survive network retries, race conditions, and duplicate delivery with MongoDB change streams.

The Problem with Webhooks

Here's what went wrong, and how we fixed it.

Architecture Before: The Naïve Handler

Our initial webhook endpoint in app/api/razorpay/webhook/route.js followed the textbook pattern:

JavaScript

// ❌ The naïve approach — no idempotency protection
export async function POST(request) {
  const text = await request.text();
  const body = JSON.parse(text);

  // Verify signature
  const razorpaySignature = request.headers.get("x-razorpay-signature");
  const expectedSignature = crypto
    .createHmac("sha256", process.env.RAZORPAY_WEBHOOK_SECRET)
    .update(text)
    .digest("hex");

  if (expectedSignature !== razorpaySignature) {
    return NextResponse.json({ error: "Invalid signature" }, { status: 400 });
  }

  // Handle event — but what if this fires twice?
  switch (body.event) {
    case "payment.captured":
      await handlePaymentCaptured(body.payload);
      break;
    case "payment.failed":
      await handlePaymentFailed(body.payload);
      break;
    case "order.paid":
      await handleOrderPaid(body.payload);
      break;
    case "refund.created":
      await handleRefundCreated(body.payload);
      break;
    case "refund.processed":
      await handleRefundProcessed(body.payload);
      break;
  }

  return NextResponse.json({ status: "success" });
}

The signature verification was solid — HMAC-SHA256 using the webhook secret. But beyond that, nothing prevented the same event from being processed twice.

Three Failure Modes We Hit in Production

1. Network Retries Causing Duplicate Fulfilment

JavaScript

// From our verify route — lots of side effects in one handler
const order = new Order({ ...orderData, userId: user._id, payment: { ... } });
await order.save();

// Update user's first-order flag
const changeUser = await User.findById(user._id);
changeUser.isFirstOrder = false;
await changeUser.save();

// Send email notification
await sendOrderStatusEmail(order.shippingAddress.email, template.subject, template.html);

// Send WhatsApp notification
await sendWhatsAppMessage(phone, whatsAppTemplate.name, ...);

// Create admin notification
await Notification.create({ type: "order", role: "admin", ... });

// Create user notification
await Notification.create({ type: "order", role: "user", ... });

If any of those downstream calls are slow, Razorpay fires the webhook again. Now you have two orders for the same payment.

2. Race Conditions Between Verify and Webhook

Our system has two separate paths that can process the same payment:

Client-side path: Checkout → Razorpay modal → /api/razorpay/verify (user-initiated)
Server-side path: Razorpay → /api/razorpay/webhook (system-initiated)

Both can fire nearly simultaneously. The verify route creates the order; the webhook tries to update it — or worse, creates a duplicate.

3. Serverless Cold Starts on Vercel

Our Next.js app runs on Vercel as serverless functions. Cold starts can push response times past Razorpay's timeout threshold, triggering retries even when the first invocation succeeded.

The Solution: A Three-Layer Idempotency Shield

Layer 1: Webhook Event Deduplication with MongoDB

We introduced a WebhookEvent collection to track every event we've seen:

JavaScript

// models/WebhookEvent.js
const WebhookEventSchema = new mongoose.Schema({
  eventId: { 
    type: String, 
    required: true, 
    unique: true,  // MongoDB unique index = atomic dedup
    index: true 
  },
  eventType: { type: String, required: true },
  payload: { type: mongoose.Schema.Types.Mixed },
  status: { 
    type: String, 
    enum: ["received", "processing", "completed", "failed"],
    default: "received"
  },
  processedAt: Date,
  attempts: { type: Number, default: 0 },
  error: String,
}, { timestamps: true });

JavaScript

// In the webhook handler
async function processWebhookIdempotently(eventId, eventType, payload) {
  try {
    // Atomic insert — fails if eventId already exists
    await WebhookEvent.create({
      eventId,
      eventType,
      payload,
      status: "processing",
    });
  } catch (err) {
    if (err.code === 11000) {
      // Duplicate key — we've already seen this event
      console.log(`Duplicate webhook event ${eventId}, skipping`);
      return { skipped: true };
    }
    throw err;
  }

  try {
    // Process the event
    await handleEvent(eventType, payload);
    
    await WebhookEvent.updateOne(
      { eventId },
      { status: "completed", processedAt: new Date() }
    );
    return { skipped: false };
  } catch (err) {
    await WebhookEvent.updateOne(
      { eventId },
      { status: "failed", error: err.message, $inc: { attempts: 1 } }
    );
    throw err;
  }
}

Layer 2: Atomic Order State Machine

Our Order model uses a payment status enum with strict state transitions:

JavaScript

// From models/Order.js — the PaymentSchema
const PaymentSchema = new mongoose.Schema({
  status: {
    type: String,
    enum: ["pending", "completed", "failed", "refunded", "partially_refunded"],
    default: "pending",
  },
  razorpayPaymentId: String,
  razorpayOrderId: String,
  razorpaySignature: String,
  // ... card, UPI, wallet, netbanking details
});

Instead of blindly updating, we use MongoDB's atomic findOneAndUpdate with a status precondition:

JavaScript

// Only transition from "pending" → "completed"
const updatedOrder = await Order.findOneAndUpdate(
  { 
    "payment.razorpayOrderId": razorpayOrderId,
    "payment.status": "pending"  // Precondition: must still be pending
  },
  { 
    $set: { 
      "payment.status": "completed",
      "payment.paidAt": new Date(),
      "payment.razorpayPaymentId": paymentId,
    }
  },
  { new: true }
);

if (!updatedOrder) {
  // Either order doesn't exist or payment already processed
  // Both cases = safe to skip
  return;
}

// Only now trigger side effects (email, WhatsApp, notifications)

Layer 3: MongoDB Change Streams for Reliable Side Effects

JavaScript

// services/orderWatcher.js — runs as a separate process
const pipeline = [
  { $match: { 
    "updateDescription.updatedFields.payment.status": "completed",
    operationType: "update"
  }}
];

const changeStream = Order.watch(pipeline, { fullDocument: "updateLookup" });

changeStream.on("change", async (change) => {
  const order = change.fullDocument;
  
  // Idempotent notification dispatch
  const notificationKey = `order:${order._id}:payment_completed`;
  const alreadySent = await NotificationLog.findOne({ key: notificationKey });
  
  if (alreadySent) return;
  
  await NotificationLog.create({ key: notificationKey });
  
  // Now safe to send
  await sendOrderStatusEmail(order.shippingAddress.email, ...);
  await sendWhatsAppMessage(order.shippingAddress.phone, ...);
  await Notification.create({ type: "order", role: "admin", ... });
});

The Refund Pipeline: Same Pattern, Different Events

Our refund flow through utils/payment/refund.js uses the same Razorpay SDK:

JavaScript

const refund = await razorpay.payments.refund(paymentId, {
  amount: amount * 100,
  notes: { reason },
});

Payment Link Webhooks: A Separate Challenge

We also generate Razorpay payment links for admin-initiated orders using utils/paymentLinks.js:

JavaScript

const paymentLink = await razorpay.paymentLink.create({
  amount: amountInPaise,
  currency: "INR",
  description: `Payment for Order #${order.orderNumber}`,
  customer: { name, email, contact },
  callback_url: `${process.env.NEXTAUTH_URL}/order-confirmation/${order._id}`,
  callback_method: "get",
});

Key Code Patterns from Our Codebase

Pattern 1: Fast Webhook Response, Async Processing

JavaScript

export async function POST(request) {
  const text = await request.text();
  const body = JSON.parse(text);

  // Verify signature FAST
  if (!verifySignature(text, request.headers.get("x-razorpay-signature"))) {
    return NextResponse.json({ error: "Invalid signature" }, { status: 400 });
  }

  // Acknowledge immediately — process async
  // This prevents Razorpay timeout retries on Vercel serverless
  const eventId = body.payload?.payment?.entity?.id || body.payload?.order?.entity?.id;
  
  setImmediate(() => processWebhookIdempotently(eventId, body.event, body.payload));
  
  return NextResponse.json({ status: "received" }, { status: 200 });
}

Pattern 2: Correlating Verify + Webhook Paths

JavaScript

// The verify route uses razorpayOrderId as correlation key
const order = new Order({
  ...orderData,
  payment: {
    razorpayPaymentId: razorpay_payment_id,
    razorpayOrderId: razorpay_order_id,  // ← This is the correlation key
    razorpaySignature: razorpay_signature,
    status: "completed",
    paidAt: new Date(),
  },
});

// The webhook handler looks up by the same key
const existingOrder = await Order.findOne({
  "payment.razorpayOrderId": payment.order_id
});

if (existingOrder && existingOrder.payment.status === "completed") {
  // Verify route already handled this — skip
  return;
}

Pattern 3: Database Indexes That Power Idempotency

From our Order model's index definitions:

JavaScript

OrderSchema.index({ "payment.razorpayOrderId": 1 });   // Fast lookup by Razorpay order
OrderSchema.index({ "payment.razorpayPaymentId": 1 });  // Fast lookup by payment ID
OrderSchema.index({ "payment.status": 1 });              // Filter by payment state
OrderSchema.index({ "payment.transactionId": 1 });       // Transaction correlation

These aren't just query optimization — they're the backbone of our idempotency checks. Without them, concurrent lookups under load would be too slow, causing the very timeouts that trigger retries.

Monitoring: How We Know It Works

We track three critical metrics:

Metric	What It Tells Us	Alert Threshold
Duplicate webhook rate	How often Razorpay retries	> 5% of events
State transition failures	Race conditions caught	> 0 in 1 hour
Notification delivery lag	Change stream processing delay	> 30 seconds

JavaScript

// Quick health check endpoint
export async function GET() {
  const last24h = new Date(Date.now() - 86400000);
  
  const stats = await WebhookEvent.aggregate([
    { $match: { createdAt: { $gte: last24h } } },
    { $group: {
      _id: "$status",
      count: { $sum: 1 },
    }}
  ]);

  return NextResponse.json({ stats });
}

Results

After deploying the three-layer idempotency shield:

Zero duplicate orders in 6+ months of production traffic
Webhook processing time dropped from ~4s (risky) to <200ms (safe margin)
12% of webhooks were duplicates that we now gracefully skip
Zero duplicate notifications — change streams + notification log eliminated the problem entirely

Lessons Learned

"At-least-once" is not optional. Every payment provider does it. Design for it from day one.
The database IS your lock. Don't use in-memory locks or Redis for payment idempotency in serverless. MongoDB's unique indexes and atomic updates are your friends.
Separate state changes from side effects. The webhook handler should only change database state. Let change streams (or a job queue) handle emails, messages, and notifications.
Respond fast, process later. On serverless platforms like Vercel, acknowledge the webhook immediately. Razorpay's retry timeout is your enemy.
Index your correlation keys. razorpayOrderId, razorpayPaymentId, and payment.status need indexes — not just for query speed, but for the atomicity of concurrent findOneAndUpdate operations.

Technical References

Razorpay Webhooks Documentation

Razorpay Webhook Best Practices

Live Project

Back to all case studies

Was this case study helpful?

Comments

Be the first to share your thoughts.

From Social Media Success to a Brand-Ready Portfolio: Building Krithika Krishna's Digital Presence

End-to-End Encryption Without a Key Server