Design a Scalable Email Service

Think about Gmail: 1.8 billion users, each receiving 40 or more emails daily. The system looks like a simple mailbox, but behind it sits a pipeline that handles 4 billion emails per day, filtering spam, scanning attachments, indexing for search, and delivering across the internet via SMTP. The challenge is not building an inbox UI. It is building the pipeline that gets emails there reliably at scale.

Functional

Send and receive emails: Users compose and send emails with recipients, subject, body, and optional attachments. The system receives incoming emails via SMTP and stores them. CC, BCC, reply, and forward are supported. Every email gets a unique message_id.

The numbers reveal why this is a write-heavy system. 100 million users receiving 40 emails per day produces 4 billion emails per day. At 10KB metadata each, that is 40TB per day of metadata alone. Attachments at 500KB average push storage to roughly 1PB per month. Every incoming email triggers multiple writes: metadata to Cassandra, attachment to S3, event to Kafka, index update to Elasticsearch.

Users and Volume

100 million DAU. Each user receives roughly 40 emails per day and sends roughly 10. That totals 5 billion email operations per day: roughly 58,000 per second on average, with peaks exceeding 100,000 per second during weekday morning hours.

Two parallel interfaces serve different clients. REST API for web and mobile. SMTP, IMAP, and POP3 for traditional mail clients. The key insight: POST /v1/messages returns immediately after enqueueing to Kafka, the email has not been delivered yet. Actual sending happens asynchronously through the delivery pipeline.

REST API

Send email: POST /v1/messages

Two data types with opposite access patterns. Metadata is small (10KB), queried constantly, and demands fast writes at 100,000 or more QPS. Attachments are large (megabytes), written once, read occasionally, and immutable. The partition key design is the single most important schema decision.

Cassandra: Email Metadata

1CREATE TABLE emails_by_folder (
2  user_id        UUID,          // partition key
3  folder_id      UUID,          // partition key
4  timestamp      TIMESTAMP,     // clustering key, DESC
5  message_id     UUID,
6  sender         TEXT,
7  recipients     LIST<TEXT>,
8  subject        TEXT,
9  body_preview   TEXT,
10  is_read        BOOLEAN,
11  has_attachment BOOLEAN,
12  attachment_ids LIST<UUID>,
13  PRIMARY KEY ((user_id, folder_id), timestamp)
14) WITH CLUSTERING ORDER BY (timestamp DESC);

Six component layers form a pipeline. Email is not request-response. It is a pipeline where each email flows through validation, spam filtering, virus scanning, storage, indexing, and notification as independent async stages connected by Kafka.

Client Layer

Every email traverses one of two flows: outgoing or incoming. Neither is synchronous. Both rely on Kafka to decouple the fast path (accept the email) from the slow path (deliver or process it).

Email Send Flow

At 4 billion emails per day, the database is the bottleneck. Cassandra's LSM-tree and partition-key routing eliminate the write bottleneck, but only with correct partition key design.

GATE: Cassandra Partition Key Design

Three technology decisions with clear winners for email at scale.

Cassandra vs MySQL vs MongoDB

MySQL provides ACID and joins but struggles with write-heavy workloads. A single primary handles roughly 10K write QPS. Sharding adds cross-shard complexity. For email, the dominant access pattern never needs joins.

Email has a unique failure contract: losing an email is unacceptable, but delayed delivery is normal. An email arriving 30 minutes late is fine. An email that vanishes is a critical failure.

Delivery Failure and Retry

Four improvements that build on the existing architecture.

ML-powered spam detection: Replace rule-based filters with ML models trained on user feedback (mark as spam / not spam). The existing Kafka pipeline feeds training data to the ML service.

Smart categorization: Sort emails into primary, social, and promotions like Gmail. Same ML infrastructure as spam detection. Category labels added to Cassandra metadata.