Notification System

A notification system delivers timely alerts to users across multiple channels: a push notification on your phone's lock screen, an email confirming a purchase, or an SMS with a one-time login code. Behind these seemingly simple messages lies one of the most reliability-demanding problems in distributed systems.

You have interacted with notification systems every day — the "your order has shipped" email from Amazon, the "someone liked your post" badge on Instagram, the security code text from your bank. What makes these systems deceptively hard is a combination of three factors: scale (billions of notifications per day across a global user base), channel diversity (each delivery channel — push, email, SMS — has its own protocol, rate limits, and failure modes), and reliability requirements (important notifications must eventually arrive even when devices are offline, providers are down, or the sending service crashes mid-delivery).

This case study answers a question every developer eventually faces: why do some notifications arrive late, or not at all? The answer reveals the core architectural decisions behind every production notification system.

This case study in this section follows the same framework:

• Clarify constraints (Steps 1–2) — What does the system do, and how much traffic must it handle?
• High-level design (Step 3) — What are the major components and how do they connect?
• Deep dives (Steps 4–7) — How do the trickiest parts actually work?
• Trade-offs (Step 8) — What did we give up, and when would we choose differently?

Step 1: Clarify Requirements#

Before designing anything, pin down exactly what the system needs to do and how well it needs to do it.

Functional Requirements#

These describe what the system does.

Non-Functional Requirements#

These describe how well the system works.

The central tension in this system: reliability and latency pull in opposite directions. The most reliable delivery mechanism — store in a queue, retry until success — introduces latency. The lowest-latency mechanism — a direct synchronous API call — has no retry and fails silently if the provider is down. The architecture must resolve this tension differently for different notification types: transactional notifications (OTPs, security alerts) need a fast, high-priority path while promotional campaigns can tolerate minutes of delay in exchange for guaranteed eventual delivery.

Step 2: Back-of-the-Envelope Estimation#

The key insight from the math: 8,700 notifications/second is not uniformly distributed. Marketing campaigns generate sudden bursts — traffic can spike from near zero to tens of thousands per second in minutes. This load profile is hostile to any architecture that calls provider APIs synchronously in the request path. The message queue is not an optimization; it is the foundational requirement that makes the system possible.

Step 3: High-Level Design#

The fundamental design principle is to decouple generation from delivery: the service that triggers a notification should not wait for it to be sent. An order service, an auth service, or a scheduler fires a notification request and immediately moves on. The slow, retry-heavy work of actually delivering it happens asynchronously, downstream, completely invisible to the caller.

What each component does:

• Notification Service — The single entry point for all notification requests. It validates the request, fetches user contact info (device tokens, email address, phone number), queries the User Preference Service to check for opt-outs and DND windows, then publishes an event to the appropriate Kafka topic. It returns 202 Accepted immediately — delivery is asynchronous.
• User Preference Service — Answers the question: "Should this notification be sent to this user on this channel?" It stores per-user, per-channel opt-in/out flags, Do-Not-Disturb windows, and per-category rate limit counters backed by Redis.
• Message Broker (Kafka) — The backbone of reliability. Kafka is a distributed, append-only log system: events are written to disk and persist until a consumer explicitly acknowledges them. Workers are organized into consumer groups, so if a worker crashes mid-processing, Kafka re-delivers that message to another worker in the same group. A separate Kafka topic per channel means an email provider outage cannot block SMS delivery.
• Channel Workers — Independently scaled consumer groups, one per channel. Each worker reads an event, renders the message template with user-specific variables, calls the third-party provider API, and writes the result to the event log.
• Notification Event Log (Cassandra) — An append-only audit table that records every delivery attempt with its status, provider, and timestamps. Cassandra is a distributed database optimized for high write throughput — it can sustain hundreds of thousands of writes per second across many nodes without degrading read performance. This event log powers debugging ("why didn't User A receive their OTP?"), analytics, and delivery confirmation.

API Design#

Why 202 Accepted and not 200 OK? The 202 status code means "the request was received and will be processed, but processing is not yet complete." This is the correct HTTP response for asynchronous operations. Returning 200 OK would falsely imply the notification was delivered at the moment the API responded.

Why request_id in the body? This is the caller's idempotency key. If the upstream service retries the request due to a network timeout, the same request_id ensures the notification is not enqueued and delivered twice. We explore idempotency in detail in Step 6.

Database Schema#

The system uses two databases with distinct roles: PostgreSQL for structured configuration and preference data, and Cassandra for the high-volume, append-only event log.

-- Device tokens for push notifications
CREATE TABLE device_tokens (
  id          BIGSERIAL PRIMARY KEY,
  user_id     BIGINT REFERENCES users(id) ON DELETE CASCADE,
  token       VARCHAR(512) NOT NULL,
  platform    VARCHAR(10) NOT NULL,   -- 'ios', 'android', 'web'
  is_active   BOOLEAN DEFAULT true,
  last_seen   TIMESTAMPTZ,
  UNIQUE(user_id, token)
);

-- Per-user, per-channel notification preferences
CREATE TABLE notification_preferences (
  user_id           BIGINT REFERENCES users(id) ON DELETE CASCADE,
  notification_type VARCHAR(100) NOT NULL, -- 'order_update', 'promo', 'security_alert'
  channel           VARCHAR(20) NOT NULL,  -- 'push', 'email', 'sms', 'in_app'
  is_enabled        BOOLEAN DEFAULT true,
  PRIMARY KEY (user_id, notification_type, channel)
);

-- Reusable message templates with variable substitution
CREATE TABLE notification_templates (
  id            BIGSERIAL PRIMARY KEY,
  name          VARCHAR(100) UNIQUE NOT NULL,
  channel       VARCHAR(20) NOT NULL,
  subject       VARCHAR(255),             -- email subject line only
  body_template TEXT NOT NULL,            -- "Hi {{first_name}}, your order {{order_id}} shipped!"
  type          VARCHAR(50) NOT NULL,     -- 'transactional', 'promotional', 'security'
  version       INT DEFAULT 1,
  is_active     BOOLEAN DEFAULT true
);

The Cassandra event log stores one row per delivery attempt:

notification_events (Cassandra):
  notification_id   UUID       -- partition key
  user_id           UUID       -- secondary index for per-user queries
  channel           TEXT       -- 'push', 'email', 'sms'
  status            TEXT       -- 'queued', 'sent', 'delivered', 'failed', 'suppressed'
  provider          TEXT       -- 'apns', 'fcm', 'sendgrid', 'twilio'
  provider_msg_id   TEXT       -- the provider's own message ID (for deduplication)
  idempotency_key   TEXT
  created_at        TIMESTAMP
  error_code        TEXT       -- '410', '429', '500', etc.

Why Cassandra for the event log? The event log is append-only and write-heavy — it grows continuously at 125 GB/day. Cassandra is designed for exactly this pattern: high write throughput with horizontal scaling by partition key (the field that determines which node stores a given row). PostgreSQL can store this data, but at this write volume it would require significant resources that would compete with the latency-sensitive preference and device queries that need fast reads. Cassandra lets us scale writes independently of the transactional PostgreSQL workload.

Step 4: Deep Dive — The Message Queue Pipeline#

The message broker is what separates a system that tries to deliver notifications from one that guarantees delivery attempts. This section explains how the pipeline works and why every design decision matters.

Template Rendering#

Templates decouple message content from sending logic. A worker receiving { templateId: 'order_shipped_push', variables: { first_name: 'Alice', order_id: 'ORD-123' } } follows these steps:

1. Fetch the template from Redis cache — Redis is an in-memory key-value store that returns data in under a millisecond, far faster than a database query. On a cache miss, fall back to the database and populate the cache for subsequent requests.
2. Fetch the user's locale and timezone
3. Select the matching locale variant if available (en-US, fr-FR, ja-JP)
4. Substitute variables: "Hi {{first_name}}, your order {{order_id}} shipped!" → "Hi Alice, your order ORD-123 shipped!"
5. Format for channel: trim to ≤160 characters for SMS, build full HTML for email, validate ≤4 KB for push
6. Send to the provider

Storing templates in the database — rather than hard-coding them in application code — lets content teams update message copy, add locale variants, and run A/B tests without a code deployment or engineering review. Templates also carry a version field so you can roll back to a known-good version immediately if a change causes unexpected rendering behavior in production.

In-App Notifications and WebSocket Delivery#

Push, email, and SMS all deliver notifications to users who are outside the app. In-app notifications work differently — they are delivered in real time to users who are actively using the app right now, and they use WebSockets instead of a third-party provider.

Why WebSocket and not HTTP polling? With regular HTTP, the client must ask ("is there anything new?") and the server responds — the client has to keep asking on a timer (polling). WebSocket flips this: it establishes a persistent, bidirectional connection, so the server can push data to the client at any moment without waiting to be asked. For real-time notifications that should appear within a second of the triggering event, WebSocket is the right tool.

How in-app delivery works:

1. When a user opens the app, the client establishes a WebSocket connection to a WebSocket server instance.
2. The WebSocket server records the active connection in Redis: SET ws:user:{user_id} {server_id}. This is necessary because you run multiple WebSocket server instances in production — a notification event needs to reach the specific instance that holds a given user's connection.
3. When the In-App Worker receives a notification event from Kafka, it looks up ws:user:{user_id} in Redis. If the key exists, the user is online — the worker forwards the event to the correct WebSocket server, which delivers it in real time.
4. If the key does not exist, the user is offline — the notification is stored in the event log with status pending. When the user opens the app, the client fetches recent pending notifications via a standard REST API call, which is the fan-out on read pattern discussed in Step 7.

This two-path design (real-time WebSocket for active users, REST fetch for returning users) means in-app notifications work correctly regardless of whether the user is currently in the app.

Step 5: Deep Dive — Push Notifications and Why They Fail#

This section answers the question at the heart of this case study: why do some notifications arrive late, or not at all?

Why Notifications Fail: The Complete Failure Map#

Understanding failure modes is what separates a system that works in testing from one that holds up in production.

Step 6: Deep Dive — Reliability: At-Least-Once Delivery and Idempotency#

The message broker's guarantee — at-least-once delivery — is both the system's greatest reliability feature and its most dangerous source of bugs if unhandled.

At-least-once delivery means a message stays in the queue until a consumer explicitly acknowledges it. If a worker crashes between sending a notification and acknowledging the Kafka message, the broker treats the message as unprocessed and re-delivers it to another available worker. This is correct behavior for reliability — but it means the same notification could be processed more than once, producing a duplicate delivery to the user. This is the core tension: the very mechanism that prevents silent drops also introduces the risk of duplicates.

The Dead Letter Queue (DLQ)#

After a configurable number of retries (typically 3–5), a notification that keeps failing is moved to a Dead Letter Queue (DLQ) — a separate Kafka topic or queue where permanently failed messages are parked for human inspection.

The critical rule: never silently discard a failed notification. A dropped notification has no audit trail and cannot be replayed. A DLQ message carries the full original payload, can be inspected by an on-call engineer to diagnose the root cause, and can be manually replayed once the underlying issue is resolved. The difference between routing to a DLQ and silently deleting the message is the difference between a diagnosable incident and an unexplained gap in the delivery logs.

Step 7: Deep Dive — Fan-Out for Mass Notifications#

The fan-out problem is: how do you efficiently deliver one notification event to millions of recipients? When an e-commerce platform sends a flash sale alert to all opted-in users, it may need to dispatch 10 million push notifications in under 5 minutes. The naive approach — query all users in a loop, send one by one — would not come close. With an average provider round-trip of 50 ms, a single thread sending 10 million notifications sequentially would take roughly 139 hours. Fan-out is about designing around this bottleneck.

Step 8: Trade-offs and Production Reality#

Every design decision in this system involves a trade-off. Here is a consolidated view of the key choices and when you would decide differently.

What this design intentionally excludes#

Production notification systems at major companies include additional layers that are out of scope for this case study:

• Engagement analytics — open rates, click-through rates, and unsubscribe rates via provider webhooks, requiring a separate analytics ingestion pipeline
• A/B testing on templates — routing a percentage of users to template variant B and measuring engagement to determine the winning message copy
• Notification digesting — grouping multiple notifications (e.g., "5 new messages") into a single summary to reduce notification fatigue and OS-level throttling
• Compliance tooling — one-click unsubscribe links in marketing emails (required by CAN-SPAM and GDPR), consent audit trails, and data deletion cascades that remove notification history
• Multi-region delivery — routing push requests to the geographically closest APNs or FCM endpoint to reduce round-trip latency for global users

Each of these adds real complexity, but none fundamentally changes the queue-and-worker architecture described here.

Summary#

The notification system is the queue-and-worker pattern in its most real-world form. Every design decision here — the message broker topology, the idempotency key, the fan-out strategy, the Dead Letter Queue — reappears in payment processing pipelines, event-driven microservices, and asynchronous job systems. The specific providers (APNs, FCM, SendGrid, Twilio) are interchangeable; the architectural pattern is universal.

The central insight: the queue is the reliability guarantee. Without it, a provider outage means lost notifications. With it, notifications accumulate during the outage and drain when the provider recovers — invisibly to the triggering service, and mostly invisibly to the user. This is the answer to why some notifications arrive late: a temporary queue backlog, not a permanent failure. And the answer to why some never arrive at all: a stale device token, a silent rate limit, or an unhandled error that was discarded without a retry.

Sources:

• Design a Notification System — AlgoMaster
• Notification Service System Design — SystemDesign.one
• Notification Service — Hello Interview
• APNs HTTP/2 Notification API — Apple Developer Documentation
• FCM Architecture Overview — Firebase Documentation