Design ChatGPT - LLM Product Layer

ChatGPT is not an LLM inference problem. That is the layer below, and it lives in its own service with its own editorial. This design is about the product layer that sits on top of a raw inference endpoint and turns it into a usable application: conversations, streaming, multimodal I/O, tool use, memory, safety, device sync. The inference service is a commodity dependency here. Call it, pay for tokens, get tokens back. Everything interesting in this design is about shaping user input into a model call and shaping model output into something a user can consume, safely, across devices, over time.

That framing matters because it rules out a lot of wrong answers. "Scale the GPU fleet" is not an answer here. The GPU fleet is someone else's service. The real problems are conversation storage at petabyte scale, token-level streaming to 100M concurrent clients, tool orchestration as a protocol, memory extraction without user surprise, safety that cannot be bypassed, and multi-device session sync that does not split-brain when a user opens the same chat on laptop and phone.

Functional Requirements

• Conversations and threads: a user has many conversations, each conversation is an ordered sequence of messages. Conversations are durable, renameable, shareable, and archivable. Messages include role (user, assistant, system, tool), content, attachments, and server-assigned sequence numbers.
• Token streaming: assistant responses stream back as tokens arrive from the inference service. First token under 500ms P50. Streaming uses SSE as the default transport; resumable on client disconnect via a last-event-id cursor.
• Multimodal input: text, images, audio, and file attachments. Images and audio up to 20MB per attachment via presigned URL upload. Each attachment passes input moderation before reaching the model.
• Tool use: the assistant can call registered tools (web search, code interpreter, image generation, custom HTTP tools, retrieval over user files). Tool calls are intercepted from the model stream, dispatched with timeouts and quotas, and their results fed back into the next model turn.
• Cross-device sync: a conversation started on phone is readable and continuable on laptop within 1 second of the change committing. Drafts (partial unsent messages) sync too.

Out of scope: the raw LLM inference engine (separate service with its own capacity, batching, KV cache, and scheduling concerns), model training and fine-tuning pipelines, the web app frontend, the mobile app frontend, billing ledger rollup, and the trust-and-safety human review UI (consumes our safety event log but owns its own tooling).

User plan tiers

Free: daily output-token cap (say 50K tokens), no reasoning-tier access, standard model only, no API access. Plus ($20/mo): 10x the daily cap, reasoning-tier access, API access with per-tier pricing. Team ($30/user/mo): Plus plus shared workspaces, admin controls, and team-level billing. Enterprise: negotiated limits, dedicated capacity, SLA, data residency, and direct-billing integrations.

Plan tier is checked at request time via the identity service. Downgrade from a paid tier sees the user revert to free-tier limits at the next billing cycle.

Non-Functional Requirements

• First-token latency: P50 under 500ms, P95 under 1.5s from user submit to first token rendered on client.
• Full-response latency: P50 under 10s for standard turns; long reasoning turns up to 120s are expected and exposed to the user as an in-progress state.
• Conversation read latency: P50 under 100ms for opening a conversation of up to 100 messages.
• Session sync latency: under 1s from commit on device A to visible on device B.
• Availability: 99.95% for conversation read and streaming; 99.9% for write paths. The underlying inference service has its own SLA; degraded inference surfaces as "model unavailable, try again" rather than as data loss.
• Durability: conversations never silently lose messages. If the inference stream is partially persisted and then crashes, the next read returns exactly what the user saw, not a corrupted prefix.
• Safety gate non-bypassable: every input passes input moderation, every output passes output moderation during streaming. No tenant tier, admin override, or developer flag can disable the safety pipeline. Disputed rejections go to human review, not around the gate.
• Cost per conversation: a tracked business metric. Token cost dominates, but memory injection, tool calls, and storage add roughly 20% on top. Every architectural decision on cache, context trimming, and memory retrieval is measured against this.
• Scale: 200M WAU, 1B messages/day, 100M concurrent SSE sessions at peak, 5 petabytes of conversation history on hot storage.

Capacity math for a product layer on top of an LLM looks different from a typical web service. The bytes are small, but the fan-out is enormous, and the streaming workload dominates every other number.

Daily message volume

200M WAU, roughly 50M DAU. Average 20 messages/day/active-user (half user turns, half assistant turns). Total roughly 1B messages/day. Average message payload is 400 tokens at 3 bytes/token of stored representation = 1.2KB of raw text per message. Add metadata (timestamps, role, model, tool flags, safety scores, attachment refs) and you land around 50KB per message stored. The bulk is embeddings, tool call traces, and per-message safety decisions, not the text itself. Attachments are stored separately.

Conversation store growth

1B messages/day at 50KB each is 50TB/day of raw conversation data. Compress at 3x (text compresses well) and you are writing roughly 17TB/day of compressed hot storage. Over a year that is 6 PB. This is the reason conversation storage is a separate concern from everything else. It is the only truly large data surface in the product layer.

Attachment storage

Roughly 10% of turns include an attachment. 100M attachments/day at an average 2MB each is 200TB/day. Most attachments are images (1-5MB), some audio (5-15MB), some files (under 10MB). At 30-day default retention, hot attachment storage is 6 PB. Moving older attachments to cold tier cuts the hot footprint roughly in half.

QPS

1B messages/day / 86,400 = 11,500 messages/sec average. Peak is roughly 4x the average during the 6-10pm window per region, so peak message QPS is roughly 45,000/sec globally. Per region (4 regions) that is 11,000/sec at peak. Each message turn maps to one inference call (sometimes more if tool use fires), so inference-service QPS scales with message QPS.

Streaming concurrency

The load-bearing number. An active user keeps an SSE connection open for the duration of a streaming response. Response time P50 is 5s, P95 is 30s. At peak, roughly 5% of DAU are mid-turn at any moment: 50M DAU × 0.05 = 2.5M concurrent streams per hour of peak on a conservative estimate. The stated requirement of 100M concurrent sessions counts all open SSE connections, including idle ones holding the conversation page open. The real "actively streaming" number is closer to 2-5M. Streaming fleet sizes to 100M concurrent connections, not to 5M actively streaming ones, because each idle connection still consumes a socket.

Token throughput

Average assistant response is 400 tokens. At 45K message turns/sec × 400 tokens = 18M tokens/sec globally. This is the number the inference service has to serve, and it is why the inference service is a separate system. The product layer only has to move the bytes, not generate them.

Bandwidth

18M tokens/sec × 3 bytes/token = 54 MB/sec of generated text flowing out of inference into the product layer. After fanout to streaming clients that is roughly the same 54 MB/sec per session × N sessions, but each individual stream is slow (tokens trickle at 50-100 tokens/sec of perceived rate), so the aggregate is the dominant factor, not the per-connection bandwidth.

Memory store

200M users, each with roughly 100 memory entries on average, each entry roughly 1KB of text plus embedding vector. 200M × 100 × 1KB = 20TB of memory storage. Embeddings dominate the byte count: 100 × 1536-dim float32 = 600KB per user × 200M = 120TB. Memory is retrieved per session, so read QPS is roughly session-start QPS around 500K/sec peak; ANN index must serve that comfortably.

Cost anchor

Token cost dominates. At a wholesale-ish $0.50/M input tokens and $1.50/M output tokens, the average turn (2000 input tokens from history + 400 output tokens) costs roughly $0.0016 of raw inference, $0.001 input, $0.0006 output. Add memory retrieval (negligible), safety (roughly 5% overhead), and storage (pennies per user per month). Gross COGS per active user per day is roughly $0.03 at 20 turns/day. Free-tier service is sustained by paid subscriptions and enterprise contracts covering the free COGS plus margin.

Why the math matters for design

The token cost per turn is the reason memory injection is token-budgeted at 500 tokens not unbounded. Every extra 500 tokens injected across a billion turns per day is half a billion extra input tokens per day, or $250/day at wholesale pricing, scaling linearly with user count. Tight budgets on every injected context are not a minor optimization; they are the difference between a sustainable COGS and a bleed.

Geographic distribution

Assume four regions: US-East, US-West, EU, APAC. Each region carries roughly its share of global traffic based on user distribution. US roughly 40%, EU 30%, APAC 25%, other 5%. Regional fleets size against regional traffic, not global; each region has its own full stack (gateway, conversation, streaming, moderation, tools, memory) so cross-region hops are avoided on the hot path.

Latency from user to nearest region under 50ms for 95% of users. Inter-region latency (for admin operations, rollouts) is 100-200ms. Users accessing from outside their home region pay the cross-region hop once at the edge, then their session is routed home, not every request.

Similarly, the conversation history compression is not a cute feature. For a heavy user with a long running conversation, un-compressed history costs 10x the per-turn tokens of a compressed version. Without compression, a power user's turns scale in cost linearly with conversation length, which is a pricing signal the platform cannot absorb on free tier.

Read path sizing

Read QPS is dominated by conversation listing (opening the app shows recent conversations) and message reads (opening a conversation). Assume every DAU opens the app once per day and reads on average 3 conversations. 50M × 3 = 150M conversation reads/day = 1700 reads/sec average. At peak 4x = 6800 reads/sec. Per region (4 regions) = 1700 reads/sec. Each conversation read is roughly 100 messages at 50KB each = 5MB. Cached at the gateway after first read with a per-user TTL of 60 seconds; cache hit rate target 80%; backend-hit rate is 340 reads/sec per region = trivial for any decent DB. The read path is not a bottleneck, which is why almost all the engineering attention goes to the write and stream paths.

The API surface carries the async streaming contract, conversation management, attachments, and memory. Consistent identifiers and resumable streams are the load-bearing design choices. Once you accept that users disconnect mid-response, the whole API shape falls out.

Chat completion (streaming)

1POST /v1/chat/completions
2Content-Type: application/json
3Authorization: Bearer sk-...
4Idempotency-Key: 9f3c2e-4b1a-4d77-...
5Accept: text/event-stream
6
7{
8  "conversation_id": "conv_01HY3R...ULID",
9  "model": "gpt-standard",
10  "messages": [
11    { "role": "user", "content": "Summarize the PR at this link: ..." }
12  ],
13  "tools": ["web_search", "code_interpreter"],
14  "attachments": ["upl_abc123"],
15  "stream": true,
16  "max_output_tokens": 2000,
17  "temperature": 0.7
18}

Response is an SSE stream:

1event: message.started
2data: {"completion_id":"cmp_01HY3R...","message_id":"msg_01HY3S..."}
3
4event: token
5id: 1
6data: {"delta":"The"}
7
8event: token
9id: 2
10data: {"delta":" PR"}
11
12event: tool_call.started
13id: 42
14data: {"tool":"web_search","args":{"q":"github.com/..."}}
15
16event: tool_call.result
17id: 43
18data: {"result":"..."}
19
20event: token
21id: 44
22data: {"delta":" adds"}
23
24event: message.completed
25id: 200
26data: {"message_id":"msg_01HY3S...","usage":{"input_tokens":1842,"output_tokens":213}}

Every event carries an id that is monotonically increasing within a completion. The id is the resume cursor. The completion_id is durable and stored on the conversation; a client can reconnect to the stream via a resume endpoint and replay from any prior id.

Stream resume

GET /v1/completions/{completion_id}/stream?last_event_id=42

Returns the same SSE stream starting at event-id 43. Behind this, the streaming service is writing tokens to a persistent event buffer keyed by completion_id. The buffer retains the stream for 30 minutes after the completion finishes, which covers any realistic client reconnect window. A resume after the buffer expires returns the final message via a fetch on the conversation instead.

Conversation management

1POST   /v1/conversations
2GET    /v1/conversations/{conv_id}
3GET    /v1/conversations?after=conv_01HY3R&limit=50
4PATCH  /v1/conversations/{conv_id}         # rename, archive, change model default
5DELETE /v1/conversations/{conv_id}
6POST   /v1/conversations/{conv_id}/messages
7GET    /v1/conversations/{conv_id}/messages?after_seq=42&limit=100
8POST   /v1/conversations/{conv_id}/share   # returns public share URL
9DELETE /v1/conversations/{conv_id}/share

Conversation IDs are ULIDs. Messages carry a server-assigned seq integer, monotonic per conversation. seq is the cursor for incremental sync. A client that has seq=42 locally asks for messages after_seq=42 and gets anything it is missing. Server-assigned sequence numbers are important: they let any device reason about ordering even if its local clock is skewed, and they give the sync protocol a simple comparable-by-integer primitive.

Attachments

Two-step presigned upload, same pattern as any large-payload flow.

1POST /v1/uploads?purpose=chat_attachment
2# returns { upload_id, presigned_put_url, max_bytes, expires_at }
3
4PUT {presigned_put_url}
5# client uploads directly to object storage
6
7# later, include upload_id in the chat completion request

The upload_id is checked against the tenant owner at completion time to prevent cross-tenant attachment reference.

Memory management

1GET    /v1/memories                   # list user memories with salience scores
2POST   /v1/memories                   # add an explicit memory
3DELETE /v1/memories/{memory_id}
4PATCH  /v1/memories/{memory_id}       # edit the text or toggle enabled
5POST   /v1/memories/disable_all       # kills memory injection entirely for this user

Memories are user-controlled first-class data. The service also writes memories implicitly (extracted from conversations) but a user can see every memory in this list and delete any of them.

Multi-device sync

WSS /v1/sync
# client authenticates, subscribes to updates for its user

The sync channel pushes events: message.appended, conversation.updated, draft.updated, memory.changed. The client reconciles its local cache on each event. For a new device that boots up, the WebSocket delivers a snapshot of recent activity plus a cursor; thereafter it is incremental. Polling fallback is available (GET /v1/sync?after=cursor&timeout=25s as a long-poll) for networks that block WebSockets.

Rate limiting

Per-user-plan limits on three dimensions:

• Messages per hour: guards the write path and admission controller.
• Output tokens per day: the load-bearing cost cap on a given user.
• Reasoning-model minutes per day: expensive-model-specific cap on top of the generic token cap.
• Image generation per day: quota on the image-gen tool specifically.

HTTP status codes

• 200 OK with SSE body for streaming completions.
• 202 Accepted on conversation-level mutations.
• 400 malformed request.
• 403 on auth or plan-tier check fail.
• 409 on idempotency key collision with differing params.
• 422 on moderation rejection (input blocked).
• 429 on rate-limit hit with Retry-After.
• 503 on inference backend unavailable.

Idempotency semantics

The Idempotency-Key header on completion submits is a client-generated UUID. The server deduplicates submits with the same key over a 24-hour window. On a repeated submit with the same key and identical params, the server returns the original completion's current state (streaming or completed). On a repeated submit with the same key and different params, the server returns 409 Conflict, meaning the client has a bug. This pattern prevents duplicate completions on network-retry scenarios without silently papering over client errors.

The dedup store is Redis keyed by (user_id, idempotency_key) with 24h TTL. The value is the completion_id of the original submit. On cache miss, the gateway proceeds with a fresh completion and writes the idempotency record after the completion_id is assigned. The cache-hit path skips inference entirely and returns the existing completion's state.

Pagination for conversation listing

GET /v1/conversations supports cursor-based pagination via the after parameter. The cursor is the last conversation's conv_id. Because conv_ids are ULIDs (time-sortable), pagination is a simple range scan on the primary key with no offset needed. Typical response is 50 conversations per page.

Cursor-based pagination works correctly even when new conversations are added between requests. The cursor always points at a specific record, so the next page continues from there regardless of insertions.

Error response format

All non-2xx responses follow a standard format:

1{
2  "error": {
3    "code": "rate_limit_exceeded",
4    "message": "You have exceeded your daily output token budget.",
5    "type": "ratelimit_error",
6    "param": null,
7    "request_id": "req_01HY3R..."
8  }
9}

request_id is a server-generated ULID that clients can reference in support tickets. type groups errors into categories (ratelimit_error, moderation_error, auth_error, validation_error, server_error) for client-side handling. code is the specific error; message is a human-readable explanation; param points at the specific request field at fault for validation errors.

Batch APIs

For enterprise and integration use cases, batch APIs support submitting many completions in one request with per-completion async handling. POST /v1/batches accepts an array of completion specs, returns a batch_id; GET /v1/batches/{batch_id} returns the status of each completion. This is a separate surface from the interactive chat completion and is oriented at workflows like processing thousands of emails or classifying a large dataset.

Batch completions run at lower priority than interactive, get discounted pricing, and have a much longer SLA (results in minutes to hours, not seconds). The underlying inference backend routes batch traffic to lower-priority batching pools.

Webhook deliveries

For completions that take longer than expected (reasoning-tier turns, heavy knowledge lookups), the client can register a webhook. On completion, the streaming service POSTs a payload to the webhook with the final message content, usage counts, and any tool-call summary. Signature via HMAC-SHA256 over the body with a per-client secret.

Webhook delivery is best-effort with retries (30s, 2m, 10m, 1h, 6h, 24h. Six attempts); a failing webhook does not fail the turn. The turn's state is authoritative in the conversation, and the client can always GET it directly.

API versioning

The API is versioned via the path prefix (/v1/). A breaking change introduces /v2/ with parallel support; /v1/ continues operating for a deprecation window of 12+ months. Non-breaking additions (new fields, new endpoints) land on /v1/ without a version bump. Response fields are additive; clients must tolerate unknown fields.

The product layer owns a handful of persistent stores. Conversations and messages are the workhorse. Memory is the personalization surface. Attachments are metadata-only (the bytes live in object storage). Safety events are an append-only audit log.

Conversations table

One row per conversation. Sharded by user_id so per-user listing is a single-shard scan.

1CREATE TABLE conversations (
2    conv_id              TEXT         PRIMARY KEY,    -- ULID
3    user_id              UUID         NOT NULL,
4    title                TEXT,
5    model_default        TEXT         NOT NULL,
6    created_at           TIMESTAMPTZ  NOT NULL DEFAULT now(),
7    updated_at           TIMESTAMPTZ  NOT NULL DEFAULT now(),
8    archived             BOOLEAN      NOT NULL DEFAULT false,
9    message_count        INTEGER      NOT NULL DEFAULT 0,
10    last_message_seq     BIGINT       NOT NULL DEFAULT 0,
11    summary              TEXT                          -- long-conversation compression target
12);
13
14CREATE INDEX idx_conversations_user_time ON conversations (user_id, updated_at DESC) WHERE archived = false;

Messages table

Append-heavy. One row per message. Sharded by conv_id; each conversation's messages colocate on the same shard so reading a full conversation is a single-shard scan ordered by seq.

1CREATE TABLE messages (
2    conv_id              TEXT         NOT NULL,
3    seq                  BIGINT       NOT NULL,       -- server-assigned, monotonic per conv
4    message_id           TEXT         NOT NULL,       -- ULID, globally unique
5    role                 TEXT         NOT NULL,       -- user, assistant, system, tool
6    content              TEXT,                         -- may be null if only attachments
7    attachments          JSONB,                        -- array of upload_ids
8    tool_calls           JSONB,                        -- tool call records if any
9    model_used           TEXT,
10    input_tokens         INTEGER,
11    output_tokens        INTEGER,
12    moderation_result    JSONB,                        -- input/output moderation scores
13    created_at           TIMESTAMPTZ  NOT NULL DEFAULT now(),
14    PRIMARY KEY (conv_id, seq)
15);
16
17CREATE UNIQUE INDEX idx_messages_message_id ON messages (message_id);

Primary key (conv_id, seq) gives every conversation read the same shape: range scan on the primary key for messages since a cursor. message_id has its own unique index for point lookups by id (share links, message-level tooling).

Attachments metadata

One row per uploaded attachment. The bytes are in object storage keyed by blob_id (SHA256).

1CREATE TABLE attachments (
2    upload_id            TEXT         PRIMARY KEY,    -- ULID
3    user_id              UUID         NOT NULL,
4    blob_id              TEXT         NOT NULL,       -- content hash
5    mime_type            TEXT         NOT NULL,
6    size_bytes           BIGINT       NOT NULL,
7    purpose              TEXT         NOT NULL,       -- chat_attachment, ...
8    moderation_status    TEXT         NOT NULL,       -- pending, approved, rejected
9    moderation_scores    JSONB,
10    created_at           TIMESTAMPTZ  NOT NULL DEFAULT now(),
11    expires_at           TIMESTAMPTZ  NOT NULL        -- TTL drives cleanup
12);

User memory store

Each memory is a small fact with a salience score. Stored with its embedding for retrieval.

1CREATE TABLE user_memories (
2    memory_id            TEXT         PRIMARY KEY,    -- ULID
3    user_id              UUID         NOT NULL,
4    text                 TEXT         NOT NULL,
5    source_conv_id       TEXT,                         -- where it came from if extracted
6    source_message_id    TEXT,
7    salience             REAL         NOT NULL,       -- 0.0 to 1.0
8    category             TEXT,                         -- preferences, facts, context
9    user_enabled         BOOLEAN      NOT NULL DEFAULT true,
10    created_at           TIMESTAMPTZ  NOT NULL DEFAULT now(),
11    last_used_at         TIMESTAMPTZ
12);
13
14CREATE INDEX idx_memories_user ON user_memories (user_id, salience DESC) WHERE user_enabled = true;

Embeddings go to a separate ANN index (see below) keyed by memory_id. Keeping embeddings out of the Postgres row keeps the row small; the index retrieval returns memory_ids, the DB resolves them in a batch.

Embedding and RAG indexes

A vector store (Pinecone, Weaviate, Vespa, or a self-hosted FAISS cluster) holds user memory embeddings (namespace per user). Query-time retrieval pulls the top-K vectors by cosine similarity against the turn's query embedding.

Why a dedicated vector DB instead of pgvector in Postgres: pgvector works for under 10M vectors per index; we are looking at 20B memory vectors, which is well into specialized-ANN territory. The access pattern (per-user namespace, high concurrency, approximate top-K) is exactly what a vector DB is tuned for.

Safety event log

Append-only audit trail. Every input-moderation call and every output-moderation chunk decision writes a row. Partitioned by date; retention 1 year.

Partition: date
Clustering: (user_id, created_at)
Columns: event_id, stage (input|output), classifier_version, scores, decision, conv_id, message_id, sample_token_offset

Separate from conversations because (a) retention is longer, (b) it is written once and read rarely, (c) compliance investigations need it to outlive the conversation it came from.

Why separate conversations DB from messages DB

Conversations are indexed by user_id (listing "my recent conversations"). Messages are indexed by conv_id (reading "this conversation's messages"). Trying to combine them forces one shard key to serve both scans, which breaks one of them. Splitting them into two shard strategies. Conversations sharded by user_id, messages sharded by conv_id. Gives each access pattern its own single-shard fast path.

What is not stored in the hot DB

The raw token-level stream of an in-progress completion is not written to the messages table until the completion finishes. During streaming, tokens live in a Redis Streams buffer keyed by completion_id; only after the message completes (or is canceled with a partial message) does the final assembled text get written to messages. This keeps the hot DB from seeing a write per token (18M tokens/sec globally) and instead sees a write per message (11K/sec average), three orders of magnitude less write load.

Sharding details

Messages table shard key is conv_id, hashed. This gives each conversation a single shard; a per-conversation message fetch is a single-shard scan, which is the dominant read pattern (open a conversation, read its messages).

Conversations table shard key is user_id, hashed. Listing a user's conversations (dashboard home page) is a single-shard scan. When a conversation is opened, the app first reads the conversation row from the user-sharded table, then reads the messages from the conv-sharded messages table. Two reads, two different shards, but each is a single-shard fast read.

Shard count sizing: hot-shard load per shard at peak should stay under 50% of a single Postgres primary's write throughput. At 11K messages/sec / 256 shards = 43 writes/sec per shard, well within one primary's comfort zone. Conversations table at 500/sec new conversations / 256 shards = 2/sec per shard, trivial.

Hot-key mitigation: a viral user (someone whose conversations are being read by millions via public share) could hot-key their user shard. Mitigation is a read-path cache in front of the shard specifically for public-share reads; write path stays on the shard primary.

Why not key-value store for messages

A naive "just use DynamoDB keyed by (conv_id, seq)" design is tempting. The access pattern is pure key-based. The reason Postgres wins is the transactional semantics needed at commit: the completion finalizer has to atomically (a) increment last_message_seq on the conversation row, (b) insert the message row, (c) update message_count and updated_at. DynamoDB can do this via transactional writes but at higher cost and with different semantics. Postgres's native transaction gives it for free.

Postgres also gives range scans on the primary key. Fetching all messages in a conversation ordered by seq is a cheap primary-key range scan. Key-value stores can do this too (DynamoDB via clustering keys) but the tooling and operational experience with Postgres at this scale is mature and the team already owns the operational know-how.

Read replicas and failover

Each shard has a primary and two replicas. Writes go to primary; reads for the common case go to primary (read-your-writes consistency needed for "I just sent a message, I want to see it on refresh"). Reads for archival, analytics, and cross-tenant queries go to replicas. Automated failover on primary death: a consensus layer (etcd or similar) elects a replica as new primary within 30 seconds; in-flight writes from the dead primary are lost, which is acceptable because the client's commit protocol requires explicit acknowledgement. An unacknowledged write is retried by the client with the same idempotency key.

Zoom all the way out and the service is a request-response-plus-streaming product on top of three heavy dependencies: an inference service (tokens), a vector store (memories and knowledge), and object storage (attachments). Everything interesting is how the product layer orchestrates those three around a durable conversation store.

Edge and gateway

A global anycast edge terminates TLS, applies coarse rate limiting, and routes to the nearest regional cluster. The gateway fleet is stateless, fronts the full API, and authenticates via the identity service. The gateway is the first durable decision point: it allocates the message_id, writes the user turn row, and hands off to the streaming service for the assistant turn.

Conversation service

Owns conversation and message reads and writes. Sharded by user_id (conversations) and by conv_id (messages). Every message append goes through this service; every conversation list and every message fetch goes through this service. No one else writes to messages directly. That is important, because the dual-write streaming path is the one exception and it flows through this service as its commit step.

Streaming service

Owns SSE and WebSocket connections. A streaming session for a chat completion opens one SSE connection to the client and one streaming RPC to the inference service; tokens arriving from inference fan out to both the client socket and a Redis Streams buffer per completion_id. When the completion finishes, the streaming service asks the conversation service to commit the final assistant message.

Moderation service

Two entry points: input moderation (called synchronously before inference), output moderation (called during streaming on buffered chunks). The input moderator is a stack of classifiers: text toxicity, prompt-injection detection, jailbreak patterns, image safety (on image attachments), audio safety (on audio attachments). Output moderator scans assistant tokens in buffered windows; a rejection truncates the stream and emits a refusal template to the client.

Tool orchestration service

Receives tool call descriptors from the streaming service (intercepted from the inference output), dispatches to the named tool, enforces per-tool timeout and quota, and returns results back into the streaming pipeline. Tools include web_search, code_interpreter, image_generation, and custom_http_tool. The orchestration layer is what makes "function calling" a usable protocol rather than a demo.

Memory service

Two responsibilities: extraction and retrieval. At end-of-turn, an extraction job summarizes candidate facts from the exchange, scores their salience, and writes them to the user memory store (with user consent gate). At start-of-turn, the retrieval path takes the current turn context, computes a query embedding, retrieves top-K relevant memories via the ANN index, and injects them into the system prompt with a token budget.

Attachment service

Issues presigned PUT URLs to clients, tracks upload metadata, runs attachment-level moderation (image classifier, audio classifier, file antivirus), and resolves upload_ids to blob URLs at turn time. The actual bytes never transit the product backend.

Sync service

Manages the WebSocket fleet. Fan-outs per-user events (message.appended, conversation.updated, draft.updated, memory.changed) to every active device socket for that user. Drafts are stored server-side with a short TTL so the WebSocket hand-off between devices is stateless on the client.

Event bus

Kafka (or equivalent) carries cross-service events: message_committed, completion_started, completion_finished, moderation_decision, memory_extracted, quota_consumed. Downstream consumers include the usage ledger, the billing pipeline, the metrics rollup, the trace sampler, and the memory extraction job. The event bus is not on the critical path of serving a turn. It is the asynchronous fan-out for analytics, billing, and derived state.

Inference backend (out of scope)

A separate service that receives (system_prompt, messages, tools, model, max_tokens, temperature) and streams tokens back. This design assumes it exists and scales independently. Its SLOs, batching strategy, KV cache management, and GPU scheduling are out of scope. The interface between the product layer and inference is gRPC with a streaming response.

Identity and auth

A user is identified by a UUID. API tokens are scoped to a user (or to a service account for enterprise). The gateway validates the token on every request with a short-TTL cache (60s) over the identity service. Session tokens for the web and mobile app are distinct from API tokens and use OAuth-style refresh flows.

Tenant isolation: every conversation row carries user_id; every API query includes implicit user_id filtering. Cross-tenant leaks are prevented at the ORM layer. Every SELECT on conversations, messages, memories, or attachments includes a WHERE user_id = :current_user predicate by default. Bypass of this filter is prevented by a code review gate and a pre-merge lint that rejects queries without the filter.

Observability

Every turn emits a structured trace with spans for gateway, moderation, memory retrieval, inference, tool calls (if any), output moderation, and commit. Trace sampling is 1% of turns by default, 100% for turns that fail or take more than 30 seconds. Per-service metrics: latency histograms, error rates, throughput, queue depths. Per-user metrics: token consumption, memory retrieval hits. Alerts fire on SLI breaches: P95 first-token latency over 2s for 5 minutes, commit error rate over 0.1%, sync service reconnect storms, moderation latency over 200ms.

The dashboard for on-call includes: total turns/sec, first-token latency P50/P95/P99, inference backend error rate, commit success rate, Redis Streams buffer usage, moderation rejection rate per category, per-region health.

Capacity planning cadence

Gateway fleet: sized for peak messages-per-second plus 100% headroom. Auto-scales on CPU and on RPS with 5-minute averaging; fast scale-up for bursts, slow scale-down to avoid flapping. At 45K msg/sec peak globally with roughly 500 req/sec per instance, the fleet is around 100 instances per region at peak.

Streaming fleet: sized for concurrent SSE connections, not RPS. At 100M concurrent connections globally with roughly 50K connections per instance, the fleet is around 500 instances per region at peak. Socket count is the primary sizing signal; CPU headroom usually comes for free at that socket count.

Conversation-service fleet: sized for write throughput plus read throughput, with read replicas absorbing the read side. 11K writes/sec at peak split across 256 shards, each shard primary handling 43 writes/sec is trivial; the fleet size is dominated by connection-pool overhead, not raw throughput.

Redis Streams fleet: sized for concurrent active completions times per-completion buffer size. At 1M active completions × 20KB = 20GB live data per region, easily fits on a single Redis Cluster per region with replicas.

Vector DB: sized for total vectors (20B memory + 100M knowledge chunks) and for query QPS (500K/sec at peak). Typical sizing is 100+ nodes per region for a horizontally-sharded vector store with millisecond query latency.

Deployment and rollout

Blue-green deploys for every service with automated rollback on error-rate regression. Canary rollouts for model-alias updates: point 1% of traffic at the new model version, compare quality and latency metrics for an hour, then 10%, then 50%, then 100%. Rollback triggers: error rate spike, latency P95 regression over 20%, moderation rejection-rate spike (indicating the new model produces more policy-violating content).

Data consistency model across devices

The conversation service is the single source of truth. All devices eventually converge on its state via the sync service's broadcast plus each device's local cache reconciliation. The consistency model for committed messages is read-your-writes on the originating device (sync write to DB, then local cache updates) and eventual on other devices (broadcast over sync channel, usually under 1 second).

For drafts, the model is last-writer-wins with server-side timestamp. This is a looser model intentionally. Drafts are a convenience feature, and strict consistency is not worth the complexity.

For memory changes, the model is strongly consistent at the source (user_memories table is the ground truth) and eventually consistent at the read path (memory retrieval on a turn may hit an ANN index that is slightly stale, up to the index's sync lag, typically seconds). The sync lag is acceptable because memory relevance tolerates minor staleness. A memory added in the last minute missing from this turn is a rare and low-impact event.

Testing and quality signals

Unit tests on every service. Integration tests on cross-service flows (submit a turn, verify commit, verify sync broadcast). Load tests at 5x peak traffic in a staging region. Canary traffic in production on every deploy. Model quality tests: a corpus of golden-path prompts tested after every model-alias update, with manual review on any regression in golden outputs.

User-visible quality signals: satisfaction rating per turn (thumbs up/down in the UI), long-form feedback, regret-click tracking (the user regenerates the response, a strong negative signal). These feed back into model training and classifier tuning. Per-cohort quality dashboards track whether a new feature (new model, new moderation rule) degrades any user subset's experience.

Walk through the flows that dominate load: a happy-path turn, a tool-call mid-stream, a multimodal input, a stream resume after disconnect, and a cross-device sync.

Flow 1: happy-path text turn

1. Client opens an HTTPS connection to api.example.com and sends POST /v1/chat/completions with conversation_id, messages array, and stream: true. Accept header requests text/event-stream.
2. Anycast edge terminates TLS, forwards to the nearest regional gateway.
3. Gateway authenticates, applies per-user rate limits (messages/hour, output-tokens/day), resolves model alias.
4. Gateway calls moderation service synchronously with the user message content. If moderation rejects, the request terminates with 422 and an error code; no inference call is made.
5. Gateway calls conversation service to append the user message. Conversation service transactionally increments last_message_seq on the conversation row, writes the user message row with seq=N+1, and returns the new seq.
6. Gateway calls memory service with the conversation context. Memory service retrieves top-K memories and returns the memory texts within the token budget.
7. Gateway constructs the full inference request: system prompt (base system + memory injection + tool definitions), conversation history (last M messages, possibly compressed), and the new user turn.
8. Gateway hands off to the streaming service, passing the full inference request, the completion_id (newly allocated), and the client SSE socket.
9. Streaming service opens a streaming gRPC to the inference backend. As tokens arrive, each one is written simultaneously to the Redis Streams buffer keyed by completion_id and to the client SSE socket with an incrementing event id.
10. Output moderation runs on buffered 50-token windows. Each window is scored; if a window is rejected, the stream truncates and a refusal event is sent to the client.
11. Tokens flow for the duration of the response (typically 5-10 seconds). Progress events go to other devices via the sync service at key boundaries (message.started, message.completed).
12. On stream end, the streaming service calls conversation service to commit the assistant message with the accumulated content. Conversation service transactionally assigns the next seq, writes the message row, and updates the conversation row.
13. Conversation service emits a message.committed event to Kafka. Downstream consumers (usage ledger, memory extractor, metrics) pick it up asynchronously.
14. Streaming service emits message.completed on the client SSE stream with final usage counts. Client closes the stream.
15. Minutes later, the memory extractor runs over the turn and writes any new memories.

Flow 2: tool call mid-stream

1. Mid-stream, the inference service emits a structured tool_call token sequence: [TOOL:web_search][ARGS:{...}].
2. Streaming service detects the tool-call boundary in the token stream and halts the forward to the client at the end of that tool_call delta (the client sees a tool_call.started event with the tool and args).
3. Streaming service calls the tool orchestration service with tool=web_search and the args. Orchestration enforces per-user quota (image gen quotas, search quotas) and per-tool timeout (say 10s for web_search).
4. Tool orchestration dispatches to the web_search backend, gets results, returns a structured result payload.
5. Streaming service emits tool_call.result to the client (so the user sees what was found, optionally rendered inline).
6. Streaming service re-enters the inference backend with the tool call and its result appended to the conversation, asking for continuation. The inference backend now generates the final user-facing response using the tool output.
7. Tokens resume streaming to the client. The completion_id stays the same; events continue with monotonically increasing ids.
8. If a tool call times out, the streaming service emits tool_call.error and re-enters inference with an error result; the model then decides how to respond (typically apologizing and offering to retry).

Flow 3: multimodal input (image)

1. Client calls POST /v1/uploads?purpose=chat_attachment to get a presigned PUT URL and an upload_id.
2. Client PUTs the image bytes directly to object storage.
3. Attachment service receives an S3 event (or equivalent) that the object was written; kicks off image moderation (safety classifier, NSFW classifier, known-harm hash match). Status is tracked on the attachment row.
4. Client calls POST /v1/chat/completions with the upload_id in the attachments array.
5. Gateway verifies the attachment is owned by this user and has passed moderation. If moderation rejected the image, the chat request fails with 422.
6. Gateway constructs the inference request with the image included in the multimodal input format (either base64-inlined or as a signed URL depending on the inference backend's contract).
7. Inference backend's vision-capable model reads the image and generates a response. Streaming proceeds as in Flow 1.

Flow 4: stream resume after disconnect

1. User is mid-turn on mobile. Phone drops WiFi momentarily. The SSE connection breaks.
2. Streaming service detects the disconnect (socket close). It does not terminate the completion. It continues receiving tokens from inference and appending them to the Redis Streams buffer.
3. Client, on reconnect, issues GET /v1/completions/{completion_id}/stream?last_event_id=127.
4. Streaming service opens a new SSE socket to this client, seeks to event id 128 in the buffer, and replays events 128 through N. If the completion is still live, the new connection transitions to tailing the buffer as new tokens arrive.
5. Client sees all missed tokens replay in a quick burst, then catches up to live streaming.
6. If the completion had already finished during the disconnect, the buffer still holds the full sequence (30-minute retention), so the resume replays the full finished stream; the final message.completed event tells the client the turn is already done.

Flow 5: cross-device sync

1. User is mid-conversation on phone. Opens the same conversation on laptop.
2. Laptop authenticates and opens a WebSocket to sync service, subscribing to the user.
3. Laptop fetches the conversation via GET /v1/conversations/{conv_id} plus GET /v1/conversations/{conv_id}/messages?after_seq=0&limit=100 to populate local state.
4. Laptop is now in sync. When phone sends a new message, conversation service emits message.appended; sync service forwards to both devices. Laptop appends the new message to its local view.
5. Laptop wants to watch the streaming response live: it calls GET /v1/completions/{completion_id}/stream?last_event_id=0 and receives the tokens in near-real-time. Phone's SSE connection is unaffected, both devices pull from the same underlying Redis Streams buffer.
6. User types a reply on laptop mid-response. Laptop emits a draft.updated event to sync service; sync service forwards to phone. Phone's input field shows the draft being typed on laptop (debounced for readability).

Flow 6: memory extraction (async)

1. A turn finishes and commits. Conversation service emits message.committed to Kafka.
2. Memory extractor worker consumes the event, reads the full turn from the conversation service (both user and assistant messages).
3. Worker checks user.memory_enabled flag. If disabled, skip.
4. Worker runs a small LLM with a prompt asking for durable user facts. Output: a structured list of candidate facts with salience scores.
5. For each candidate fact, worker computes an embedding, performs an ANN query against existing memories to detect duplicates. If a very similar memory already exists (similarity > 0.9), update its last_used_at and skip insertion.
6. For new memories, worker writes to user_memories table and to the ANN index.
7. Worker optionally emits a notification event to the user's sync channel (memory.changed), so the UI's memory-settings panel reflects the new state.

Flow 7: rate limit exhaustion

1. User submits a turn. Gateway checks rate limits. The daily output-token budget is exhausted.
2. Gateway rejects with 429 and headers X-RateLimit-Remaining-Tokens: 0, X-RateLimit-Reset: <UNIX timestamp>, Retry-After: <seconds>.
3. Client UI shows a "You've hit your daily limit" state with a countdown and an upgrade prompt.
4. No DB writes happen on the rejected submit. No inference call is made. No moderation call is made. The 429 is the gateway's first decision after auth.
5. At reset time, the token bucket refills and the user can submit again. Purchased tier upgrades apply immediately (paid upgrade mid-day refills the bucket now, not at the next reset).

Eight components carry the load in this design. Each has a specific job and a specific failure story. Reading these in order roughly traces the request through the system.

Dual-write streaming

The streaming service fans out tokens from inference to two destinations simultaneously: the client SSE socket and a Redis Streams buffer keyed by completion_id. The fan-out is inline in the streaming loop: for each token event from the inference gRPC stream, the service appends to the Redis stream, then writes to the client socket. Append-then-write is the right order because Redis is the durability backstop; if the client write fails, the buffer still has the data and a resume can recover. If the Redis write fails, the streaming service surfaces an error to the client and the completion is marked failed.

The buffer is sized for 30 minutes of retention. At peak, roughly 1M completions active with 400 tokens × 50 bytes per event = 20KB per completion × 1M = 20GB of live buffer data per region. Redis Cluster with local NVMe persistence sized for 100GB per region comfortably covers this with room to grow.

On stream completion, the streaming service calls conversation service to commit the final assembled message. The commit is the point at which the assistant turn becomes part of the conversation history.

Tool orchestration layer

Receives tool calls parsed from the inference output stream, dispatches to the named tool, enforces per-tool timeout, quota, and concurrency limits, returns results to the streaming pipeline. Tools are registered with a JSON schema describing their inputs, outputs, required auth, and allowed scopes per user plan.

The orchestration layer is what turns "function calling" from a demo into a production protocol. Three concerns it handles:

• Protocol parsing: the inference model emits tool calls as a structured token sequence (JSON wrapped in a marker, or a tool_use block in the underlying inference API). The streaming service's parser extracts the tool call cleanly even if the model's output is partially malformed (retries on parse failure, falls back to no-tool if the model cannot be coaxed into valid output).
• Dispatch and timeout: each tool has its own timeout (web_search 10s, code_interpreter 60s with sandbox, image_gen 30s). Orchestration enforces the timeout, records telemetry, and returns structured errors on timeout or tool-side failure.
• Tool-turn state machine: a turn can involve multiple tool calls in sequence. The model can call web_search, use its result to decide to call image_gen, then finalize. Orchestration tracks the tool-call chain, enforces a max-chain-length (typically 10 tool calls per turn to prevent runaway loops), and surfaces the chain as a structured tool_calls array on the final message.

Memory extraction and salience

Extraction runs asynchronously after each turn commits. The extractor is a small LLM prompted to identify durable user facts from the turn ("the user prefers Python over Ruby," "the user is working on a react native app"). Each extracted fact gets a salience score (0.0 to 1.0) from the LLM based on how generalizable and stable the fact is. Transient facts ("user is in a bad mood today") score low and are filtered out; durable facts ("user's preferred language is Python") score high and are written to the memory store.

User consent gate: extraction only runs for users who have memory enabled in their settings. The default is opt-in; the service does not extract memories without explicit consent.

Retrieval runs synchronously at turn start. The current turn's user message is embedded, the ANN index is queried for top-K (K=10 typical) memories in the user's namespace, filtered by user_enabled=true and salience > 0.3. The retrieved memories are ranked by a combined score (cosine similarity + salience + recency), pruned to a token budget (500 tokens), and injected into the system prompt.

The budget is critical: left unchecked, memory injection can balloon the context window by thousands of tokens, increasing inference cost and displacing actual conversation history. Capping at 500 tokens keeps memory as a background signal rather than a dominant one.

Conversation history compression

The conversation context sent to inference is bounded by the model's context window (say 128K tokens). For long conversations that exceed this, the product layer compresses old turns: the oldest N turns are summarized into a single synthetic message ("Earlier in this conversation, the user discussed X, Y, and Z with the assistant"). The summary is regenerated when the conversation grows beyond a threshold (say every 50 turns past the window limit).

Pinned system context: active memory injection and any tool definitions stay at the top of the context, immutable. The middle is the compressed historical summary. The bottom is the most recent N turns verbatim. This layout is effectively a rolling window with a summary tail.

Safety pipeline

Three stages. Input moderation on user turn, attachment moderation on any uploaded media, output moderation on streaming assistant tokens.

Input moderation: a stack of text classifiers (toxicity, hate, self-harm, sexual, violence, prompt-injection patterns, jailbreak patterns). Runs synchronously at gateway before inference is called. A rejection terminates the submit with 422 and a standardized error code; no inference compute is spent.

Attachment moderation: image and audio classifiers run at attachment upload time (not at turn time). Upload returns pending until moderation completes; a turn referencing a pending or rejected attachment fails at gateway.

Output moderation: streaming tokens are buffered in 50-token windows; each window passes through an output classifier. A rejected window truncates the stream. Streaming service emits a safety_refusal event to the client with a standardized refusal message, commits the partial message with a safety_truncated flag, and logs the incident. The buffering adds 50-token latency to output moderation (roughly 500ms at typical speeds), which is the floor on how fast the output can be moderated without a scan-every-token tax.

Human review queue: a rejection that the user disputes (via a feedback button in the UI) enters a human review queue; the safety event log row is the primary record reviewers work from. Reviews can overturn a decision (white-list the specific content), retrain the classifier, or escalate policy changes. The queue is owned by a separate trust-and-safety service.

Session sync across devices

The sync service owns the WebSocket fleet. Each device maintains one WebSocket to the sync service, authenticated by user_id and device_id. The service tracks (user_id, device_id) -> socket in a sharded in-memory map keyed by user_id so all of a user's sockets live on the same sync server instance (or a coordinated pair for redundancy).

Event types the sync channel carries: message.appended (a new message was committed), conversation.updated (title changed, archived, etc.), draft.updated (a partial unsent message on another device), memory.changed (user memory was added or deleted). All events are per-user scoped; cross-user leakage is prevented at the subscription layer (the subscription is always on the authenticated user's user_id).

Draft state: the client sends draft.updated events debounced at 500ms. Drafts are stored server-side in Redis with a 24-hour TTL, so picking up on a different device fetches the draft via GET /v1/conversations/{conv_id}/draft.

Rate limiting and quota

Three enforcement layers. The gateway applies per-user messages-per-hour and output-tokens-per-day via token-bucket in Redis. The admission controller (inline in the gateway) reserves estimated output tokens against the daily budget at turn start; reconciled at completion with actual tokens used. The inference service applies its own fleet-wide concurrency and queue-depth caps that surface as 503s when the fleet is saturated. The product layer maps these to 529 back to the client.

Per-plan differences: free tier has a daily output-token cap plus a reasoning-model-minutes cap of zero (no access to the expensive reasoning tier). Paid tiers relax both. Enterprise contracts override with negotiated caps. The quota table is a single shared source of truth; the gateway caches it with a 60-second TTL.

Prompt construction pipeline

Every turn's final prompt is assembled by a construction pipeline that runs in the gateway. The ordered stages are: (1) base system prompt. The platform's universal system context; (2) memory injection (retrieved memories within the 500-token budget); (3) tool definitions (JSON schemas for active tools); (4) conversation history (compressed summary + verbatim recent turns); (5) the new user turn.

Each stage has its own token budget. The construction pipeline is responsible for respecting the model's context window. If the assembled prompt would exceed the window, it aggressively trims the verbatim history first (preserves the most recent turns), then memory. The base system prompt is never trimmed (it is pinned).

The pipeline's output is a fully-formed messages array passed to the inference backend. Different backends have different formats (OpenAI messages format, Anthropic content-blocks format, raw prompt for older models); the pipeline has a backend-specific final formatter. Keeping the stages upstream backend-agnostic means switching inference providers is a format-layer change, not a full pipeline rewrite.

The cost signal for a turn is its output tokens, which are not known until the turn finishes. The admission controller has to decide "does this user have enough budget left" at turn start, before any tokens are generated. The approach is reservation plus reconciliation: at submit time, the controller reserves an estimated output-token count (say 1000 tokens. A generous upper bound for a normal turn) against the daily budget. If the budget does not have 1000 tokens free, the turn is rejected with 429. When the turn completes, the controller reconciles: actual tokens used replace the reservation. Over-reserved tokens return to the budget.

For long-running turns (reasoning models that can emit 10K+ tokens), the reservation is larger or adaptive. A reasoning-tier model reserves 5000 tokens up front, with a mid-turn re-check at 4000 tokens to decide whether to continue or fail gracefully. This prevents a single runaway turn from consuming an entire day's budget.

Conversation retention and archive

Active conversations stay in the hot Postgres tier. Conversations untouched for 180 days move to a warm tier (columnar store, slower reads, cheaper storage). Conversations untouched for 2 years move to cold archive (S3 Glacier or equivalent). User-initiated deletion is immediate (transactional DELETE from hot, async purge from warm and cold, and a tombstone in the audit log).

The tier migration is a background job that scans by updated_at and moves rows in batches. Rehydration (user accesses an old conversation) copies it back from warm to hot transparently with a single-digit-seconds first-read latency; subsequent reads are hot-tier fast.

Streaming service internals

The streaming service is a fleet of stateless (per-connection stateful) worker processes. Each worker holds a pool of active completions, each completion represented by a state machine: init → awaiting_first_token → streaming → finalizing → done. The worker's event loop pumps tokens from the inference gRPC stream to both the client SSE socket and the Redis Streams buffer.

Per-completion state: completion_id, client socket, buffer cursor, inference stream handle, assembled message text, tool_call state (if mid-tool-call), output moderation buffer, token counts. State is kept in process memory for the duration of the completion. On worker restart, in-flight completions are marked as interrupted, clients get a disconnect event, and resume works as usual via the buffer.

Socket multiplexing: a single worker handles thousands of concurrent completions via async I/O. Language choice matters here. Rust, Go, and event-loop Python/Node all work; a thread-per-connection language (classic Java threads) would be wasteful at the socket count we need.

Tool orchestration retries

When a tool call fails (timeout, HTTP 5xx, malformed response), orchestration follows a tool-specific retry policy. Idempotent tools (web_search) retry up to twice with backoff. Non-idempotent tools (send_email, post_to_slack) do not retry. The cost of a duplicate send is too high, so a first failure surfaces as a tool_error to the model, which decides next steps.

Per-tool circuit breakers: when a specific tool has an error rate above 50% over a 1-minute window, the circuit opens; subsequent tool calls to that tool fail fast with tool_unavailable and the model proceeds without it. The breaker closes on a 30-second probe that succeeds.

Tool sandboxing for code_interpreter

The code_interpreter tool runs user-submitted Python code. Running arbitrary code on shared infrastructure is a clear abuse vector. Sandboxing: each code_interpreter invocation spins up a short-lived container with no network access by default, a CPU and memory limit (2 cores, 2GB), a wall-clock limit (60 seconds), and a scratch filesystem that does not persist. Specific libraries are pre-installed; the user cannot install arbitrary packages.

File attachments (data files the user has uploaded) are mounted into the sandbox read-only for the duration of the invocation. Files the code writes to the sandbox scratch can be returned as part of the tool result. Outbound network requests from within the sandbox are disallowed by default; a future improvement adds opt-in outbound access via a proxy that applies rate limits and content policy.

Image generation tool

image_generation is a tool wrapped around a separate generative-image service. The product layer dispatches a generate request with the model-extracted image prompt, waits for the async job to complete (up to 30 seconds), and returns the resulting image URL back to the streaming pipeline. The image is then rendered inline in the response.

Image generation has its own per-user quota (images per day), its own abuse considerations (the tool's request passes through text moderation on the prompt and image moderation on the output before return), and its own cost per invocation (images are much more expensive than text turns). The tool quota is enforced by the orchestration layer independent of the generic token quota.

Every architectural decision has an alternative. The ones below are load-bearing: choose differently and the design shape changes.

SSE vs WebSocket for streaming

SSE wins for chat completions. It is unidirectional (server to client), HTTP/1.1 compatible, works through every corporate proxy that allows HTTPS, has built-in reconnect semantics with last-event-id, and has zero framing overhead beyond the event markers. WebSocket is bidirectional but chat completion streaming does not need the client-to-server direction mid-stream (the request was the submit; tokens flow one way). Using WebSocket for streaming would add an unnecessary framing layer and would require building the reconnect-with-cursor pattern by hand.

WebSocket does win for the sync channel, because sync is bidirectional (the client pushes draft updates, typing indicators, presence; the server pushes cross-device events). Running two transports is the right choice: SSE for unidirectional streams, WebSocket for bidirectional sync.

Dual-write streaming vs async log-apply

Dual-write (write to Redis buffer synchronously, fan out to client socket synchronously) is the chosen design. The alternative is async log-apply: tokens go to a Kafka topic, a separate consumer writes to Redis, another consumer writes to client sockets. Async-log-apply is simpler to scale horizontally (any consumer reads from Kafka) but adds latency (each hop adds at least a few ms) and adds a failure mode where the client has received the token but the Kafka commit later fails, leaving the DB and the client inconsistent without an obvious recovery path.

Dual-write is synchronous on both paths with a clear durability guarantee: the Redis write is the commit point; the client socket write is best-effort after commit. The failure stories are simpler (if Redis is down, the completion fails; if the client socket is down, the user can resume). The cost is that the streaming service holds both connections. The inference gRPC and the client SSE. Open for the duration of a completion, which is fine because these connections are cheap and the streaming service is sized for socket count anyway.

Tool use via function calling vs agent loop vs orchestrator

Function calling is the base layer. The model emits a structured tool call, the service dispatches, result goes back. Agent loops (autonomous multi-step reasoning with tool use) are a pattern built on top of function calling. The choice here is whether to expose agent loops directly through the product.

This design picks "bounded orchestrator" over "free agent loop" at the product boundary. Function calling is the primitive; the orchestrator enforces a max chain length of 10 and a per-tool timeout budget. The reason: a free agent loop can consume unbounded compute and unbounded tool quota for a single user turn; capping the chain keeps cost predictable and prevents runaway loops from pathological prompts or tool outputs. Future iterations may expose longer-running agentic workflows as a separate async API (like the video generation service's async-job pattern), but within the chat completion interactive turn, bounded is the right choice.

Memory retrieval on every turn vs cached in user session

The chosen design retrieves memories on every turn. The alternative is to retrieve once at session start and cache them in the user's active-session state for the duration. Session cache saves 20-50ms per turn and one vector store query. The cost is staleness: if the user adds a memory mid-session (via the memory management API or implicit extraction from an earlier turn in the same session), the cache does not know. Session cache would need invalidation on memory-change events.

Given that the vector query is cheap and the cache complexity is non-trivial, per-turn retrieval wins. If the vector store latency ever becomes a bottleneck, the trade-off flips: introduce a session cache with explicit invalidation.

Moderation blocking output stream vs post-hoc review

Output moderation is inline in the stream: a rejected window truncates the stream immediately. The alternative is to let the full response stream to the client and run moderation post-hoc on the assembled message. Post-hoc moderation gives a worse user experience (they see harmful content briefly before it is redacted) and creates a legal exposure window where the service has distributed the content before screening it.

Inline moderation is the correct choice for any content-safety posture that takes legal risk seriously. The cost is latency. 50-token windows add roughly 500ms to the first redaction decision. For most content, moderation is a pass-through (no rejection), so the latency cost is only felt on the rare rejection path.

Shared conversation context vs user-isolated (for teams/orgs)

In an enterprise context, multiple users can share a conversation (e.g., a team-brainstorming session). The design choice is whether the conversation state is user-scoped with sharing bolted on, or workspace-scoped from the start.

The chosen design is user-scoped with explicit sharing. Each conversation has a single owner_user_id, and sharing is a separate ACL record. This maps cleanly to the most common pattern (single-user conversations) and keeps the sharding key stable (user_id). The alternative (workspace-scoped conversations) is cleaner for multi-user collab features but complicates every per-user query and the shard key becomes workspace_id, which does not align with the personal-use pattern.

Future work (a real shared-workspaces product) likely introduces workspace_id as a second dimension alongside user_id, with conversations optionally scoped to a workspace.

Model aliasing vs version-pinned model calls

The chat completion API accepts a model alias (gpt-standard, gpt-reasoning) rather than a specific version (gpt-2026-04-15). The alias resolves to a concrete version at turn time. This decoupling lets the platform roll forward to a newer model without every client changing their API call.

The trade-off is reproducibility. A conversation started on one model version that continues after an alias rollover may produce subtly different behavior because the underlying model changed. Mitigation: every message row stores model_used with the concrete version, so the conversation has a record of what responded. For use cases that need strict reproducibility (testing harnesses, regulatory workflows), the API accepts a specific version id that bypasses the alias.

Per-turn vs per-conversation memory injection

Memory is retrieved per turn. The alternative is to retrieve once at conversation start and re-use. Per-turn retrieval is more accurate because the relevant memory shifts with the topic of each turn (turn 1 about Python, turn 3 about a travel plan. Different memories are relevant). Per-conversation retrieval is cheaper but less accurate. The choice here is per-turn; the cost is 20-50ms and one vector query per turn, which is well within latency budgets.

Embedding model versioning

Memories and knowledge file chunks are stored with embeddings from a specific embedding model version. When the embedding model upgrades, all existing embeddings must be re-computed to stay queryable in the same namespace. The migration runs as a background job at service scale, writes new embeddings to a v2 namespace, and switches queries atomically after verification. Old namespaces are retained for 30 days post-switch as a rollback path.

This is a real operational cost. Re-embedding 20B vectors is a multi-week process even on a parallel pipeline. For that reason, embedding model upgrades are intentionally rare (yearly, not monthly), and each one is planned as a dedicated project with its own migration window.

Synchronous vs asynchronous tool dispatch

Tools are dispatched synchronously within a turn. The turn blocks on the tool result before resuming inference. An asynchronous alternative (fire the tool call, return to the user with "I'll follow up," resume in a separate turn when the tool completes) is cleaner for long-running tools (web scraping, heavy computation) but worse for UX. Users expect the assistant response to complete in one interaction, not two.

The chosen design is synchronous with aggressive timeouts (10-60s depending on tool). For truly long-running tools (agentic workflows), the future direction is the async agent-task API described in Future Improvements, not extending the per-turn tool-dispatch model.

Regional sharding vs global active-active

The design is regional. Each region owns its users' data, and cross-region reads are avoided on the hot path. An alternative is global active-active, where every region serves every user's data via multi-region replication. Active-active gives better failover (users stay on a working region even if their home region dies) but costs more (replication across regions, conflict resolution on writes) and complicates consistency guarantees.

Regional sharding is simpler and cheaper. The failure case (home region down) is handled by client redirect to a secondary region, with degraded state (user's conversations are read-only until home region recovers). For enterprise tenants with availability requirements that don't tolerate this, a paid tier with multi-region writes is an option.

Writing to Kafka synchronously vs eventually consistent

Post-commit events (message_committed, memory_extracted, quota_consumed) are written to Kafka after the DB commit. The write is asynchronous. The turn completes even if Kafka is slow. The trade-off is potential loss of downstream analytics for turns that committed just before Kafka became unavailable. Mitigation: a reconciliation job periodically scans recent commits against Kafka topic offsets and backfills any missed events. The authoritative state is always in the DB; Kafka is the fan-out mechanism, not the source of truth.

Token-level streaming vs chunked-response

Assistant responses stream token-by-token. The alternative is chunked responses. Wait until the model completes, then send the full response. Chunked is simpler (one HTTP response, no SSE), but users wait seconds for first output instead of milliseconds. Streaming is the core UX of a good LLM product; chunked is strictly worse for anything longer than a one-sentence reply.

The cost of streaming is the architectural complexity (SSE, resume protocol, dual-write buffering, moderation in streaming windows). That cost is justified by the user-perceived latency improvement from a 10-second wait to 400ms time-to-first-token.

Every component can fail. The architecture must degrade gracefully on each one. This section walks through the failure modes that actually happen in production and how the design handles them.

Inference service outage

Symptom: streaming gRPC calls to the inference backend start returning errors (5xx, timeout, connection refused). Impact: users cannot get assistant responses for their turns.

Mitigation path: the streaming service detects the failure, commits the user message but not an assistant message, and emits an error event to the client SSE stream with error_code=inference_unavailable. The client UI shows an error state on the user's message with a retry button. The user can retry once inference comes back online. The conversation_id is reusable; retrying is a fresh completion against the existing conversation.

Recovery: the control plane health-checks inference backends and circuit-breaks routes that are failing; traffic shifts to healthy backends. For a full-inference outage (all backends down), the product layer goes into read-only mode for chat completions while conversation browsing continues.

Conversation store lag (ghost messages)

Symptom: the streaming service finished a completion and emitted the message.completed event to the client, but the conversation service's commit is slow (DB write is lagging under load). The user sees the finished message on their device but a refresh shows it is missing.

Mitigation: the commit is synchronous with message.completed emission. The streaming service only emits completed after the DB commit succeeds. If the DB write takes more than a few seconds, the streaming service emits a "saving" event first and completed only after the actual commit. If the commit ultimately fails (DB unavailable), the streaming service emits a commit_failed event to the client; the client UI shows a "failed to save" marker with a retry affordance. The Redis Streams buffer still holds the full response, so the retry is a re-commit rather than a re-generation.

Tool service timeout mid-turn

Symptom: a tool call (say web_search) takes longer than its timeout (10s). Impact: the turn is stalled waiting for a tool response.

Mitigation: orchestration enforces the timeout, cancels the tool invocation, and returns a structured error result to the streaming service. The streaming service emits tool_call.error to the client, re-enters inference with the error result appended, and the model produces a user-facing message ("I was unable to search the web right now"). The turn completes; the user is not left with a silent spinner. Per-tool error rate is tracked as a SLI; sustained high rates on one tool trigger alerts independent of the chat completion health.

Moderation false positive blocks legitimate output

Symptom: the output classifier rejects a window on a legitimate response (e.g., a discussion of a historical event that overlaps with a violence-related pattern). Impact: the user sees a refusal where they expected a legitimate answer.

Mitigation: the refusal message in the UI includes a feedback button. The user clicks "this was wrong"; a dispute ticket is filed referencing the safety event log row. The trust-and-safety team's human review queue picks it up within hours. If the review overturns, the classifier is fine-tuned (offline) and the user can retry the turn and get a legitimate response. For the immediate turn, there is no backdoor. The user is refused and the dispute path is the only recourse. This is the cost of a non-bypassable safety gate; the design accepts it and invests in the dispute path being fast and fair.

Cross-device sync split-brain

Symptom: the user edits the same conversation's draft on both phone and laptop simultaneously. Both clients emit draft.updated events; the sync service receives them out of order.

Mitigation: draft is last-writer-wins by server timestamp, debounced at 500ms on the client. The sync service records the latest draft content with a timestamp; reads return the latest. For the narrow window where both devices disagree, one device's edit is lost, but drafts are not durable first-class data. The user sees their laptop draft is gone because their phone edit was later. This is acceptable for drafts; it is not acceptable for committed messages, which is why committed messages use server-assigned seq and never overwrite (they are append-only).

Redis Streams buffer capacity

Symptom: a burst of long-running completions fills the per-region buffer. Impact: new completions cannot allocate buffer space, and streaming starts failing.

Mitigation: buffer capacity is monitored and auto-scaled ahead of demand. Per-completion buffer size is capped (say 64KB); completions exceeding this are force-finalized (rare, only on pathological runaway generations). Buffer retention is 30 minutes; stale completions are evicted on schedule. For sustained capacity pressure, the control plane can reduce retention to 10 minutes as a pressure valve, trading resume window for buffer headroom.

Vector store degradation

Symptom: the vector DB for memory retrieval is slow or returning errors. Impact: memory-injection on turn start fails, which would mean turns go without memory context.

Mitigation: memory retrieval has a 50ms timeout with fallback. On timeout, the service skips memory injection for this turn and logs the skip as a retrieval_timeout event. The turn proceeds without memory; the user sees an assistant response that is as good as a first-time conversation. The fallback is silent (no error surfaced to the user) because memory absence is graceful. The assistant just does not know personal context for this turn. Degraded retrieval rate is a SLI; sustained rates above 1% trigger alerts to operations.

A deeper fallback: recency-ranked memories pulled from Postgres by last_used_at DESC. This does not give semantic relevance, but it gives some personalization in a hard-down-vector-store scenario. This fallback is on a feature flag, default off, most operational scenarios the 50ms timeout skip is enough.

Attachment moderation delay

Symptom: a user uploads an image; attachment moderation takes longer than expected (classifier queue backed up). The user submits a turn referencing the attachment before moderation finishes. Impact: the turn must decide whether to proceed or wait.

Mitigation: the gateway checks attachment moderation status at turn time. If status is still pending, the gateway waits up to 2 seconds for it to resolve. If still pending after 2 seconds, the turn is rejected with 422 and an error code attachment_moderation_pending with a retry-after hint. For expected delays (large video files, audio), the client UI shows "still processing your upload" and the user knows to wait. Rejected attachments (moderation says no) fail the turn immediately regardless of timing.

Memory extractor runaway

Symptom: the async memory extraction job is processing messages much slower than they arrive, backlog grows into the millions. Impact: new memories are extracted days late; the user notices that facts they mentioned this morning are not yet personalized for this afternoon.

Mitigation: the extractor is horizontally scalable. Increase workers to drain the backlog. The extractor's input is Kafka (message_committed events), so the backlog is bounded by Kafka retention (7 days); worst case, the oldest unextracted messages past retention are dropped and the user gets slightly less personalization. Alerts fire when extractor lag exceeds 30 minutes. For a hard extractor outage, memory-injection degrades gracefully (existing memories continue to work, just no new ones extracted).

Sync service region failure

Symptom: the sync service in one region is unhealthy; WebSocket connections in that region drop. Impact: cross-device sync stops for users in that region until they reconnect to a healthy region.

Mitigation: clients auto-reconnect with exponential backoff; the client's DNS resolves to the next nearest healthy region on reconnect. For the reconnect window (seconds to minutes), the user's devices fall out of real-time sync but their cache reconciles on reconnect via a "what have I missed since cursor X" fetch. Messages the user sends during the reconnect window still commit (conversation service is unaffected by sync service outage); they just do not broadcast to other devices until sync reconnects. The happy path is self-healing; the degraded path keeps the core chat functional.

Thundering herd on reconnect

Symptom: after a streaming service brownout, millions of clients reconnect simultaneously within a few seconds. Impact: the first few seconds of recovery are overwhelmed, potentially re-triggering the outage.

Mitigation: reconnect uses exponential backoff with jitter on the client. Server-side, the streaming service's accept queue is bounded and new connections past the limit get rejected with a 529 plus a large Retry-After header (randomized 30-120 seconds). The rejection is not a hard failure. Clients retry, see 529, back off, and eventually succeed. Gradual restoration prevents a single-service outage from cascading into a multi-service brownout.

Moderation classifier model outage

Symptom: the input or output moderation classifier is unreachable. Impact: the safety gate is non-bypassable, so turns must either fail or wait.

Mitigation: the gateway circuit-breaks to a backup classifier (a smaller, lower-accuracy model) with explicit logging that the degraded classifier is in use. The backup is tuned to be more conservative. It rejects more aggressively to stay on the safe side. A sustained primary-down scenario may see elevated false-positive rates as the backup over-rejects; operations is alerted, and the classifier team's incident bridge owns recovery.

No path bypasses moderation entirely. If both primary and backup are unreachable, submit fails with 529 and a clear error message. This is the cost of a non-bypassable safety gate; under extreme failure, turns fail closed, never open.

Compressed-summary corruption

Symptom: the conversation history compression job writes a corrupted summary. Impact: subsequent turns in the conversation get a garbled summary in context, and the assistant's responses degrade.

Mitigation: summaries are versioned and kept with a parent pointer to the last N verbatim turns. On turn time, if the summary fails a sanity check (empty, way shorter than expected, contains obvious artifacts), the context reconstruction falls back to "no summary, just verbatim the last N turns within budget." The missing-summary fallback is user-visible only by slight degradation (the assistant has less context from earlier), not by a visible error.

A deeper mitigation: the summary regeneration is idempotent, so running the job again produces a fresh summary. Operations can trigger a re-run per user or per conversation when corruption is detected.

Where this design goes next, in priority order.

Agentic loops with planning

The current design bounds tool use at 10 calls per turn. The next wave is multi-step autonomous agents: a user says "plan a week in Tokyo," the agent searches flights, hotels, restaurants across 30 tool calls over several minutes. Interactive turns cannot carry this. The user is not waiting for 5 minutes on an open SSE. The design direction is to introduce an async agent API (like the video generation service's async-job pattern): submit the agent task, get a task_id, poll or subscribe for updates. The agent runs in a separate execution environment with its own quota, tool budget, and safety gate. Intermediate outputs stream into a dedicated agent-log surface the user can watch; final output commits to a conversation as a summary message.

Persistent workspaces (shared conversations with files/code)

The current design is single-user with share-via-link as the collab story. A workspace is a persistent shared context: multiple users, files, code, conversations, all scoped to a workspace. This needs a second shard dimension (workspace_id alongside user_id) and ACL records at the conversation and file level. Realtime multi-user presence and edit conflict resolution (CRDTs or OT) come in on top.

Voice-first mode with realtime interruption

Text-stream chat is a turn-based product. Voice-first is a continuous stream where the user can talk over the assistant. This requires a different streaming architecture: bidirectional WebSocket, sub-100ms audio round-trip, VAD (voice activity detection) for interruption handling, and a different inference backend tuned for streaming audio-to-audio. The product layer has to support voice turns as a first-class concept on the messages table (audio-format messages) and carry realtime state (currently speaking, listening, interrupted).

Personalized fine-tuning per user

The current design uses a shared inference backend with per-user memory injection as the personalization layer. The next wave is per-user fine-tuned model weights (LoRA or QLoRA adapters per user, loaded at inference time). This shifts personalization from prompt-level to model-level and enables deeper adaptation. The product layer has to manage per-user adapter training (an async job), adapter versioning, adapter storage (millions of small LoRA files), and inference routing to the right adapter-loaded worker. Memory becomes a joint system with fine-tuning. Short-term facts go to prompt-level memory, long-term patterns flow into the next adapter training cycle.

Proactive notifications from ChatGPT

Currently the service responds only when the user initiates a turn. Proactive notifications. The assistant pushes a reminder, a suggestion, or a follow-up on an earlier thread. Flip the model. The product layer adds a notification dispatcher that runs user-level jobs (on a schedule the user configures, or event-driven like a calendar hook), constructs a turn server-side, runs it through the same inference + moderation pipeline, and pushes the result as a system-generated message to the user's device. User consent and notification policy are first-class concerns; the easy failure mode is notification fatigue.

Structured grounding for factual queries

Current retrieval is semantic search over user memories. Structured grounding is deeper: the assistant queries a structured knowledge graph, a live database, or a specific API when answering factual questions, and attributes each claim to a source. This requires a query planner that decides which backend to ground against per turn, a citation system that tracks claim-to-source mappings through the response, and a UI surface for citations. Quality improvement is large (hallucination drops); complexity is also large (more backends, more policy around source trust).

Multi-turn memory across long horizons

User memory today is short facts. Multi-turn memory across weeks and months is richer context: "you've been working on this Next.js migration for three weeks, here is what we've discussed so far." This requires hierarchical memory (turn-level, session-level, week-level summaries), a retrieval layer that can pull at the right granularity for the current turn, and periodic consolidation jobs that compress lower levels into higher ones.

Regional data residency

Enterprise demand for EU-only and APAC-only data storage. The current design is region-local for compute and replication-based for durability; strict residency requires a different shape. Per-tenant region binding, no cross-region replication for customer data, and region-local inference backends. The shard map becomes tenant-aware and routes every request to the tenant's region of record.

Fine-grained citation and source attribution

Current tool use surfaces tool results inline but does not maintain a structured citation chain. A future improvement is per-claim attribution: every factual claim in the assistant output is traceable to a tool result or a knowledge file chunk, with UI surfaces for the citation. This requires the inference side to emit structured attribution markers during generation and the product side to resolve them to canonical source URLs. Quality impact is large for research-heavy workflows; implementation complexity is also large because the model has to be trained or prompted to emit citation markers reliably, and the UI has to handle citation rendering across modalities.

Multi-model ensembling

Current turns route to one model. A future direction is ensembling. A fast model generates a draft, a second model (the verifier) checks for errors or policy violations, a third model rewrites if needed. This is orthogonal to the single-inference-backend abstraction and adds latency and cost. The product layer has to orchestrate the multi-model call chain, handle partial failures, and expose a coherent streaming experience despite the multi-stage nature. Quality gain is real for reasoning-heavy and factual-heavy turns; cost is two to three times the single-model baseline, so ensembling is likely launched as a premium tier rather than default.

Workspace-level context and long-term projects

For persistent projects (a book being written, a codebase being refactored, a legal case being researched), a conversation is too granular. Users want a "project" that spans many conversations, retains files, references previous decisions, and maintains a long-term summary. This is a new primitive: a project has its own ID, its own membership, its own persistent context, and conversations can be filed under it. The memory layer extends to project-scoped memory; the RAG layer indexes project-attached files; the UI surfaces the project as a first-class home. Workspaces (multi-user shared projects) are a further layer on top.

End-to-end encryption for sensitive tenants

Some enterprise tenants require end-to-end encryption of conversation content so even the platform cannot read their messages. This is architecturally incompatible with the current design. Moderation, memory extraction, tool use, and retrieval all require plaintext access to the conversation content. A future direction is a dedicated E2E tier where the tenant's client encrypts before submit, the platform runs a reduced feature set (no memory, no retrieval, client-side moderation only), and decryption happens only at the tenant's endpoint. This is a different product, not a feature toggle. Many capabilities of the full service are unavailable under E2E.

On-device model fallbacks

Mobile clients can run a small local model for offline fallback. When the network is unavailable, the client serves simple responses locally (draft messages, rough replies) without calling the cloud. On reconnect, the cloud reconciles and optionally re-runs with the full-scale model to improve quality. This requires on-device model distribution, version coordination, and a clear user signal when they are in offline-mode.

The trade-off is feature parity. The local model is much smaller and cannot handle tool use or memory retrieval. Offline mode is a degraded experience, not a full mirror. Still, for power users on flaky networks, even a stripped-down offline mode is strictly better than a blank error screen.

Deep integration with calendar, email, and productivity surfaces

Currently tools include web search and code interpreter. A deep future integration is bidirectional access to user's calendar, email, and docs, with consent, so the assistant can schedule meetings, draft and send emails, and edit documents directly. Each integration is a distinct tool with its own auth flow, scope permissions, and abuse model. Email-send in particular is a high-risk tool (an injected prompt could send phishing mail from the user's account); safeguards include explicit per-action user confirmation on high-risk tool calls, rate limiting on outbound actions, and a full audit log of every agent-initiated action.