menu_bookGyanByte
arrow_backBACK TO PAYMENTS ENGINEERING IN INDIA — FROM UPI TO LEDGERS
Lesson 12Payments Engineering in India — From UPI to Ledgers8 min read

Capstone: Designing Payments for a Real D2C Product

October 27, 2026

TL;DR

A good payments system is many simple, careful pieces: server-side orders with integer paise, a gateway adapter per gateway, verified webhooks stored and processed from a queue, one idempotent state machine for payments and orders, an outbox feeding a double-entry ledger, refunds with locks and references, daily three-way reconciliation, mandates for subscriptions, explainable risk rules, and an ops console with roles and audit logs. Use a second gateway for resilience but never switch mid-payment, and reconcile both. Monitor success rate by method and bank, pending age, webhook lag and recon breaks. Prepare for sale days like a launch.

We’ve covered a lot over eleven lessons. In this last one, let’s design the whole thing for one realistic product, the way you would in a design review at work. I’ll keep pointing back to the lesson where each piece was explained.

The product is our old friend from the AI agents course: Chai Point Online, a D2C tea brand. (Imaginary, any resemblance is accidental.)

Requirements

Business:

  • About 6,000 orders a day on normal days, up to 60,000 on festive sale days.
  • Average order around ₹650. Some gift hampers up to ₹8,000.
  • Payment methods: UPI (about 70% of volume), cards (20%), net banking and wallets (5%), cash on delivery (5%, mainly smaller towns).
  • A monthly “tea subscription box” at ₹499/month.
  • Refunds for damaged items, cancellations and returns. COD refunds by UPI payout.
  • Finance needs a daily view of sales, settlements, fees and refunds, and a clean month-end.

Engineering:

  • No customer is charged twice. No paid order is lost.
  • Every rupee received can be traced to orders, and every order to rupees.
  • Payment success rate is visible by method and bank, in near real time.
  • The system survives a gateway outage.

Architecture

Capstone architecture for a D2C payments platform: web, app and ops console call a payments service handling orders and payments, the state machine, idempotency keys and risk checks. The payments service uses gateway adapter A as primary for UPI and cards and gateway adapter B as fallback and for payouts. Both gateways send webhooks to a webhook ingest that verifies signatures on the raw body, stores and dedupes events, returns 200 fast and re-fetches status. Events flow through an outbox and queue to the ledger, refunds and payouts, recurring billing and notifications. Daily reconciliation covers both gateways and the bank statement, and observability and controls cover success rates, pending age, webhook lag, recon breaks, refund and dispute rates, maker-checker, audit log and kill switches

The main pieces:

  1. Payments service. Owns orders and payment attempts, the state machines (Lesson 6), idempotency keys (Lesson 5) and risk checks (Lesson 11). It’s the only service allowed to talk to gateways.
  2. Gateway adapters. One per gateway, behind a common interface: create_order, fetch_payment, create_refund, create_payout, verify_webhook, map_status. All gateway-specific names and quirks stay inside the adapter (Lesson 6).
  3. Webhook ingest. Verifies signatures on the raw body, stores events with dedupe, returns 200 fast, and hands off to a queue (Lessons 4 and 5).
  4. Outbox and queue. Every state change that matters writes an outbox event in the same transaction (Lesson 5).
  5. Ledger. Double-entry, append-only, fed from outbox events with idempotent posting (Lesson 7).
  6. Refunds and payouts. Refundable-amount checks under lock, references, ARN/RRN tracking, and verified UPI payouts for COD (Lesson 8).
  7. Recurring billing. Mandates, subscriptions and invoices with an idempotent scheduler (Lesson 10).
  8. Daily reconciliation. Both gateways’ reports and the bank statement, with a breaks table and a 9 AM summary for finance (Lesson 9).
  9. Ops console. Roles, 2FA, maker-checker, audit log, and the order timeline view (Lessons 6, 8, 11).
  10. Observability and controls. Dashboards, alerts and kill switches (below).

Nothing here is exotic. It’s a regular service with Postgres, a queue (SQS, RabbitMQ or Kafka; whatever your team runs well), Redis for velocity counters, and a job scheduler. The care is in the details, which is what the previous eleven lessons were about.

The Data Model, in One Place

The tables we built across the course:

Table Lesson Purpose
orders 4, 6 Business object; own state machine
payments 4, 6 Each attempt; captured / refunded amounts; fee and tax
payment_events 6 Append-only history of every payment state change
webhook_events 5 Every webhook received, deduped by event id
idempotency_keys 5 Keys and stored responses for money-moving APIs
outbox 5 Events written in the same transaction as state changes
ledger_accounts, journal_entries, postings 7 Double-entry ledger
refunds 8 Refunds with references and bank reference numbers
disputes 8 Chargebacks and UPI disputes as cases with deadlines
recon_gateway_txns, recon_bank_credits, recon_breaks 9 Reconciliation inputs and results
mandates, subscriptions, invoices 10 Recurring billing
saved_payment_methods 3 Tokens and display data only

All money columns are BIGINT paise. All times are stored as timestamps with time zone and reported in IST.

Walking Through the Main Flows

1. UPI checkout, happy path

  1. Customer taps “Pay ₹650”. App calls POST /checkout with the cart id and a client idempotency key.
  2. Payments service computes the amount from the cart, runs the risk check, creates the gateway order with notes.order_id, saves a payments row in created state.
  3. App opens the UPI app with an intent. Customer pays.
  4. Within seconds, two things arrive in any order: the app’s callback (signature verified, payment re-fetched) and the gateway webhook (stored, queued, re-fetched). Both call apply_gateway_payment.
  5. The first to arrive transitions the payment to captured and wins the conditional update on the order. It writes an order.paid outbox event in the same transaction.
  6. Outbox relay publishes. Ledger posts Dr gateway_receivable / Cr sales. Notifications send the WhatsApp confirmation. Warehouse gets the order.
  7. The second arrival finds the order already paid by the same payment and does nothing.

2. UPI payment stuck in pending

  1. Gateway reports pending. Payment moves to pending, order to payment_pending. Stock is held.
  2. App shows “Confirming your payment with your bank. Please don’t pay again.”
  3. Status checks run at 30s, 2m, 10m, 30m, 1h. Webhook still welcome at any time.
  4. At 4 hours, the order goes to failed_final, stock is released, and the customer is told “if money was debited, your bank will return it automatically”.
  5. At 6 hours, a late captured webhook arrives. The state machine allows failed → captured for the payment. The order’s conditional update allows failed_final → paid. If stock is still available, the order is fulfilled with an apology message; if not, an automatic refund is requested with reference late_success:{payment_id}.
  6. If the webhook was lost entirely, tomorrow’s reconciliation finds the capture as “missing in ours” and runs the same fetch-and-apply path.

3. COD order returned, refund by UPI

  1. Customer returns a damaged COD order. Ops approves a ₹650 refund.
  2. There’s no original payment method, so the app asks the customer for a UPI ID.
  3. Payments service validates the UPI ID (name check through the payout provider), creates a payout with reference cod_refund:{order_id}, and posts Dr sales_returns / Cr payouts_in_transit in the ledger.
  4. Payout succeeds: ledger posts Dr payouts_in_transit / Cr bank. Customer gets the UTR in a message.
  5. If it fails (wrong UPI ID), the payout is reversed in the ledger and the customer is asked again.

4. Subscription box renewal

  1. Three days before renewal, the scheduler creates invoice (sub 812, cycle 7) for ₹499 (unique constraint makes it safe to run often).
  2. 26 hours before charge time, the pre-debit notification goes out (via the gateway, as configured).
  3. At 8 AM IST on the renewal date, the invoice is charged on the UPI AutoPay mandate with reference inv:812:7:1.
  4. Success: subscription period extends, ledger posts, box is scheduled for dispatch.
  5. Failure (insufficient balance on the 27th): subscription goes past_due, customer gets a “pay now” link, a retry is scheduled for the 1st (salary day), with a fresh notification.
  6. If the customer revokes the mandate in their UPI app, the webhook cancels future invoices and the subscription ends at the period end.

Using Two Gateways Safely

Chai Point uses two gateways: A as primary, B as backup and for payouts. Having two is good for resilience, but it adds complexity. Some rules:

  • Choose the gateway when creating the order, never mid-payment. Once a gateway order exists and the customer is paying, all status checks, webhooks and refunds for that payment go to the same gateway. Store gateway on every payment row.
  • Refunds always go back through the gateway that captured the payment.
  • Route based on data. Track success rate by gateway, method and bank over the last 15-30 minutes. If gateway A’s UPI success rate drops sharply, new checkouts can be routed to B. Use a kill switch for manual override.
  • Don’t flip-flop. Switch with some stickiness (minimum time on a gateway after switching), or you’ll get confusing partial outages.
  • Reconcile both. Every gateway has its own reports, settlements and UTRs. The recon pipeline handles both, with one shared breaks table.
  • Test both regularly. A backup gateway that hasn’t processed a real payment in three months may not work when you need it. Keep a small percentage of traffic on B all the time.

What to Monitor

Dashboards and alerts, grouped by who cares:

For engineering on-call:

  • Payment success rate by method, and by bank for UPI and cards (alert on sudden drops).
  • Count and age of payments in pending (alert if the oldest is above your threshold).
  • Webhook lag: time from gateway event to processing (alert if rising).
  • Queue depth and failed jobs, dead-letter queue size.
  • Illegal state transition anomalies (Lesson 6).

For finance:

  • Daily reconciliation summary: matched, breaks by type, unexplained amount, oldest break age.
  • Settlements expected vs received by date.
  • Fees and GST by method.

For business and risk:

  • Refund rate and refund amount by reason.
  • Dispute and chargeback rate (Lesson 8).
  • Risk rule hits and block rates (Lesson 11).
  • Subscription renewal success rate and involuntary churn.

The single most useful chart, in my experience: success rate by payment method and bank, for the last hour, compared with the same hour last week. It catches gateway problems, bank outages, broken app releases and fraud attacks, all in one view.

Big Sale Day Plan

Festive sales, end-of-season sales, a celebrity mention on Instagram: these are when every edge case in this course shows up at 10x volume.

Big sale day plan in three columns. Two weeks before: load test checkout and webhook path, tell the gateway your expected peak, check rate limits and database capacity, confirm settlement and payout limits, freeze risky deploys. On the day: dashboard of success rate by method and bank, pending age, webhook lag and queue depth, a war room with a gateway contact, longer pending windows and more checks, bank downtime banners in checkout. Week after: recon breaks to zero, late successes fulfilled or refunded, dispute and refund spike handling, a post-mortem on the top five failure reasons, and updating the checklist

Two things people often forget:

  • Tell your gateway in advance. Share expected peak transactions per second and dates. They can plan capacity, and you’ll get a direct contact for the day.
  • The week after matters as much as the day. Late successes, refunds for cancelled orders, delivery-related disputes and reconciliation breaks all peak in the following days. Plan people for it.

The Launch Checklist

Everything from the course on one page. Don’t go live until every box is ticked.

Payments launch checklist in nine groups. Money basics: integer paise, fee and GST per payment, no card data on servers or logs, saved cards via tokens. Integration: amount on server, signature verified then re-fetched, order id in gateway notes, test keys cannot reach production. Duplicates: idempotency keys on money APIs, webhooks deduped, atomic mark paid, outbox. States: explicit transitions and history, pending handled for hours, late success fulfil or refund, extra captures auto-refunded. Ledger: balanced append-only entries, idempotent posting, whole ledger sums to zero, wallet and seller money as liability. Refunds: refundable includes in-progress, lock and reference per refund, ARN or RRN shown, dispute cases with deadlines. Recon: daily three-way automated, fetch-and-apply missing, breaks with age, IST dates and holiday calendar. Recurring: mandates, pre-debit notice, revoke webhook stops debits, cancel in app revokes mandate. Risk and compliance: velocity rules and CAPTCHA, maker-checker, data in India and DPDP flows, audit log and kill switch

The 25 Pitfalls, One Last Time

If you remember nothing else from this course, remember these.

Money basics

  1. Thinking “payment successful” means “money received”.
  2. Using floats for money.
  3. Storing only the gross amount, without fee and GST.
  4. Card numbers or CVVs on your servers or in logs.

Integration 5. Taking the amount from the client. 6. Marking orders paid because the browser or phone said so. 7. Not passing your order id to the gateway. 8. Slow webhook handlers that do everything inline.

Duplicates and states 9. Check-then-update in two steps instead of one atomic update. 10. No idempotency keys on refunds, payouts and charges. 11. Dual writes instead of an outbox. 12. Treating pending as failure, and giving up too early. 13. Ignoring late successes and extra captures. 14. Free-text statuses with no transition rules or history.

Ledger and reconciliation 15. A balance column instead of a double-entry ledger. 16. Editing or deleting ledger rows instead of posting reversals. 17. Monthly manual reconciliation instead of daily automated. 18. Mixing UTC and IST, and ignoring bank holidays.

Refunds and disputes 19. Forgetting in-progress refunds when checking the refundable amount. 20. Chargeback notices going to an inbox nobody watches.

Recurring 21. Charging without the pre-debit notification, or ignoring mandate revocations. 22. Cancelling in the app without revoking the mandate.

Risk and operations 23. No velocity limits on checkout until card testing hits you. 24. An ops console without roles, maker-checker and audit logs. 25. No kill switch per payment method and gateway.

Where to Go From Here

  • Read your gateway’s docs end to end. Now that you know what to look for, you’ll spot the details that matter: webhook retry policy, status names, refund rules, settlement schedule, report formats.
  • Read the RBI and NPCI circulars for the payment methods you use. They’re more readable than people think, and they explain the “why” behind many gateway behaviours.
  • Sit with your finance team for an hour. Ask them how they close the month and what they wish engineering gave them. You’ll learn more about what your payment system needs than from any course.
  • Run a game day. In staging, simulate a lost webhook, a duplicate webhook, a gateway outage, a pending storm and a reconciliation break. See what your system and your team do.

Payments engineering has a reputation for being boring. I think it’s the opposite. It’s one of the few areas in software where every bug has a rupee amount attached, where the edge cases are real people’s money, and where getting it right quietly earns a lot of trust from customers, finance, and your leadership.

Build it carefully. Store paise. Make everything idempotent. And reconcile every day.

All the best.

paymentssystem-designcapstonearchitecturechecklistfintechindia
arrow_back PreviousFraud, Security and Compliance for Payment SystemsFinished! check_circleBack to Course Overview