Design a Notification System

Every product sends notifications — order shipped, someone mentioned you, your OTP code. “Design a notification system” tests whether you can build a reliable, multi-channel delivery pipeline that stays fast under bursty load, respects user preferences, and degrades gracefully when a third-party provider misbehaves. The architecture is deceptively simple; the reliability details are where the interview lives.

Understanding the Problem

Functional Requirements

1. Multi-channel delivery — Push (mobile/web), SMS, and email
2. Triggered by services — Any internal service can request a notification
3. User preferences — Users can opt out per channel and per category
4. Templating — Notifications are rendered from templates with variables
5. Delivery tracking — Know whether each notification was sent/delivered/failed

Non-Functional Requirements

Scale

Assume 100M users, and campaigns that fan out to tens of millions at once. Peak could be hundreds of thousands of notifications per second during a large broadcast — so the system must absorb spikes without falling over or hammering downstream providers past their rate limits.

Core Entities and APIs

Data Model

CREATE TABLE notifications (
    id            UUID PRIMARY KEY,
    user_id       BIGINT,
    channel       VARCHAR(10),    -- 'push' | 'sms' | 'email'
    template_id   VARCHAR(50),
    payload       JSONB,          -- template variables
    status        VARCHAR(12),    -- QUEUED, SENT, DELIVERED, FAILED
    idempotency_key VARCHAR(100), -- dedup
    created_at    TIMESTAMP DEFAULT NOW()
);

CREATE TABLE user_preferences (
    user_id       BIGINT,
    channel       VARCHAR(10),
    category      VARCHAR(30),    -- 'marketing', 'security', 'social'
    enabled       BOOLEAN DEFAULT true,
    PRIMARY KEY (user_id, channel, category)
);

CREATE TABLE device_tokens (
    user_id       BIGINT,
    token         VARCHAR(255),   -- APNs/FCM device token
    platform      VARCHAR(10),    -- 'ios' | 'android' | 'web'
    PRIMARY KEY (user_id, token)
);

API Design

POST   /api/v1/notifications
  body: {
    user_id, channel?,              # channel omitted = system decides
    template_id, payload,
    category,                       # for preference checks
    idempotency_key                 # caller-supplied dedup key
  }
  response: { notification_id, status: "QUEUED" }

GET    /api/v1/notifications/{id}
  response: { status, channel, sent_at, delivered_at }

The idempotency_key is caller-supplied and central to correctness: if the “order shipped” service retries its request, the same key means we send only once.

High-Level Design

The guiding principle is decoupling: producers should hand off a notification in milliseconds and move on, never blocking on a slow third-party provider.

Notification API validates the request, resolves the channel(s), writes a record, and enqueues the work. It returns immediately — it does not call any provider inline.

Queue (Kafka or SQS), typically one topic/queue per channel, buffers work. This is what lets the system absorb a 10-million-user broadcast: the queue soaks up the spike and workers drain it at a sustainable rate.

Channel workers are separate pools per channel — push workers, SMS workers, email workers. Each pool knows how to call its provider (APNs/FCM for push, Twilio for SMS, SES for email), handles that provider’s rate limits, and reports delivery status. Isolating channels means an SMS provider outage never slows push or email.

Providers are the external gateways that do the last-mile delivery. They are outside our control, so the workers must treat them as unreliable.

Deep Dive: The Reliability Pipeline

Before a notification is handed to a provider, it flows through a pipeline of checks. Skipping any of these is how you get duplicate, unwanted, or abusive notifications.

1. Deduplication (idempotency)

A worker checks the idempotency_key against a store (Redis set with TTL, or a unique DB constraint) before sending. If the key was already processed, drop the duplicate. This protects against producer retries and at-least-once queue redelivery — the key reason a user does not get three copies of the same alert.

2. User preferences and quiet hours

Check user_preferences for the channel and category. If the user opted out of marketing email, drop it. Honor quiet hours (no non-critical pushes at 3 a.m. local time) by deferring the notification rather than sending it. Security notifications (OTP, login alerts) bypass marketing opt-outs.

3. Rate limiting

Throttle per user so a buggy producer cannot spam someone with hundreds of notifications. A token-bucket per (user_id, category) in Redis caps the rate; excess is dropped or delayed.

4. Send, with retries and dead-letter

Call the provider. On a transient failure (timeout, 503, provider rate-limit), retry with exponential backoff and jitter. After a few failed attempts, move the message to a dead-letter queue (DLQ) for inspection instead of retrying forever. Mark the notification FAILED so it is visible, not silently lost.

def deliver(msg):
    if dedup.seen(msg.idempotency_key):
        return "DUPLICATE"
    if not preferences.allows(msg.user_id, msg.channel, msg.category):
        return "SUPPRESSED"
    if not rate_limiter.allow(msg.user_id, msg.category):
        return "THROTTLED"
    try:
        provider.send(msg)                 # APNs / Twilio / SES
        dedup.mark(msg.idempotency_key, ttl=86400)
        return "SENT"
    except TransientError:
        if msg.attempts < MAX_RETRIES:
            queue.requeue(msg, delay=backoff(msg.attempts))
        else:
            dlq.put(msg)                   # give up, surface it
        return "RETRY_OR_DLQ"

Deep Dive: Fan-out at Scale and Provider Limits

The hardest operational case is the broadcast: “notify all 10 million users about this event.”

• Do not enqueue 10 million messages from a single request synchronously. Instead, the API enqueues a single “campaign” job; a fan-out worker expands it in batches (e.g. paginate through the audience, enqueuing chunks of 10K), so memory and latency stay bounded.
• Respect provider rate limits. APNs, FCM, and SMS gateways each cap throughput. The channel workers must throttle themselves to stay under those ceilings, which is another reason the queue is essential — it holds the backlog while workers drain at the provider’s allowed rate.
• Prioritize. Use separate queues (or priorities) so a transactional OTP is never stuck behind a 10-million-message marketing campaign. Transactional traffic takes a fast lane.

Delivery tracking

Delivery is asynchronous and multi-stage: sent (we handed it to the provider) is not the same as delivered (the device received it). Providers send delivery receipts via webhooks; a receipts consumer updates the notification’s status from SENT to DELIVERED (or FAILED). This closes the loop and powers dashboards and retry decisions.

Device token hygiene

Push tokens expire or become invalid when users uninstall the app. When a provider returns “invalid token,” the worker removes that token from device_tokens so you stop wasting calls on dead devices.

Key Takeaways

A notification system is a lesson in decoupling and graceful degradation. The architecture is a queue and some workers; the interview is won on the reliability details — idempotency, preferences, rate limiting, retries with dead-lettering, and keeping transactional traffic off the slow path. Walk through that pipeline deliberately and you have shown exactly the production maturity the question is probing for.