0%
System requirements
Functional
Capacity estimation
Traffic
API design
Inbound Message Webhook
Database design
Bot Configuration in PostgreSQL
High-level design
Request flows
Message Processing Flow
Detailed component design
NLU Pipeline
Trade-offs and tech choices
Rule-Based vs ML-Based Dialogue
Failure scenarios
NLU Service Failure
Future improvements
LLM Integration
System requirements
Functional
Capacity estimation
Traffic
API design
Inbound Message Webhook
Database design
Bot Configuration in PostgreSQL
High-level design
Request flows
Message Processing Flow
Detailed component design
NLU Pipeline
Trade-offs and tech choices
Rule-Based vs ML-Based Dialogue
Failure scenarios
NLU Service Failure
Future improvements
LLM Integration
One bot, many channels. A user asks "Where is my order?" on Slack, WhatsApp, and web chat. The bot must understand all three identically despite different message formats, character limits, and attachment types. The framework must normalize every channel into a uniform internal representation so the conversational AI core never knows or cares which platform the message came from.
Multi-channel interface: Pluggable channel adapters that normalize platform-specific message formats (Slack block kit, WhatsApp template messages, web chat JSON) into a uniform internal representation. Adding a new channel means implementing one adapter, not modifying the NLU or dialogue engine.
NLU inference dominates the latency budget, a 200ms model prediction eats half the 400ms target for the entire pipeline. Understanding where time and resources go reveals why the NLU service is the component most likely to become a bottleneck.
Assume 5 million monthly active users across all tenants, generating 50 million messages per day. That is roughly 580 messages per second on average, with peaks reaching 3,000-5,000 messages per second during business hours.
Two fundamentally different API consumers drive the design, messaging platforms sending webhooks inbound, and bot developers managing their bots through a management API. The inbound path must be fast and stateless. The management path must be comprehensive and well-documented.
POST /api/v1/webhook/:channel_type receives raw messages from messaging platforms. Slack sends JSON with block kit formatting. WhatsApp sends a different JSON structure with template references. Web chat sends yet another format via WebSocket. The channel adapter behind this endpoint normalizes all of them into a uniform internal message (text, attachments list, sender ID, channel metadata) before passing to the NLU pipeline.
Four fundamentally different data types (structured config, ephemeral state, event streams, and binary model artifacts) each demanding its own storage technology. Using a single database for all four would force painful trade-offs between query flexibility, write throughput, latency, and cost.
Bot definitions, intents with training examples, entity types, dialogue flows, tenant accounts, and user credentials all live in PostgreSQL. This data has clear relationships: a tenant owns many bots, a bot has many intents, each intent has many training examples, and a bot has many dialogue flows composed of nodes and transitions. These relationships are naturally modeled with foreign keys.
The architecture is a pipeline, each message flows through a sequence of specialized services, each adding understanding until the system can respond intelligently. This pipeline pattern makes each service independently testable, deployable, and scalable.
Channel Adapter receives a platform-specific message (Slack webhook, WhatsApp payload, WebSocket frame) and normalizes it into a uniform internal format. API Gateway handles TLS termination, authentication, rate limiting, and routes the normalized message to the NLU Service. NLU Service loads the tenant's trained model and predicts intent plus entities. Dialogue Manager retrieves conversation state from Redis, determines the current dialogue node, checks whether required slots are filled, and decides the next action. Action Service executes business logic or external API calls when the dialogue flow requires them. Response Formatter converts the dialogue manager's response into the channel-specific format. Channel Adapter (outbound) delivers the formatted response via the platform's API.
A user message touches six services in under 400ms. A training job touches three services over several minutes. Understanding both flows reveals why the architecture separates real-time message processing from batch model training.
A user sends "Where is my order?" on Slack. Slack fires a webhook to the channel adapter endpoint. The Slack adapter extracts the message text, sender ID, and channel ID from the Slack-specific payload and normalizes them into a uniform Message object. The API gateway authenticates the request and identifies the tenant.
Three components contain the core engineering complexity, the NLU pipeline, the dialogue manager, and the channel adapter layer. Understanding their internals reveals why this is a framework problem rather than a single-bot problem.
Raw text enters the pipeline and passes through four stages. Tokenization splits the input into tokens, handling contractions, punctuation, and language-specific rules. Feature extraction converts tokens into numerical vectors (word embeddings, character n-grams, part-of-speech tags). The intent classifier takes the feature vector and predicts the most likely intent with a confidence score using a multi-class classifier (typically a neural network or gradient-boosted tree). The entity extractor identifies named entities in the input (dates, order IDs, product names, locations) using a sequence labeling model.
The tension between rule-based and ML-based dialogue is the defining design trade-off of a chatbot framework. Enterprises need predictable, auditable conversation paths. Users expect natural language understanding that handles paraphrasing and typos. The architecture must satisfy both.
Rule-based dialogue uses explicit if-then logic: if the user says exactly "check order status," trigger the order flow. It is deterministic (the same input always produces the same output), auditable (every decision can be traced to a specific rule), and easy to debug (you can read the rules and predict the behavior). But it breaks on paraphrasing ("What happened to my package?"), typos ("chekc ordr status"), and any phrasing the rule author did not anticipate. Maintaining comprehensive rules for natural language is a losing battle, there are too many ways to say the same thing.
A chatbot that says "I don't know" is acceptable. A chatbot that hangs silently or returns a raw error message is not. Every failure path must produce a graceful, user-visible response within the timeout window.
LLM integration fundamentally changes the NLU pipeline, from training per-tenant classifiers to prompting a general-purpose model with tenant-specific context. This is the single largest architectural shift on the horizon.
Replace the traditional NLU pipeline (trained intent classifier plus entity extractor) with an LLM that receives the user's message plus tenant-specific context. Instead of training a model for each tenant, the system embeds the tenant's knowledge base into a vector store and retrieves relevant chunks at query time (RAG pattern). The LLM generates a response grounded in the retrieved context.
Reading a solution is not the same as producing one under interview pressure. Design it on the whiteboard and get scored feedback.
0%
System requirements
Functional
Capacity estimation
Traffic
API design
Inbound Message Webhook
Database design
Bot Configuration in PostgreSQL
High-level design
Request flows
Message Processing Flow
Detailed component design
NLU Pipeline
Trade-offs and tech choices
Rule-Based vs ML-Based Dialogue
Failure scenarios
NLU Service Failure
Future improvements
LLM Integration
System requirements
Functional
Capacity estimation
Traffic
API design
Inbound Message Webhook
Database design
Bot Configuration in PostgreSQL
High-level design
Request flows
Message Processing Flow
Detailed component design
NLU Pipeline
Trade-offs and tech choices
Rule-Based vs ML-Based Dialogue
Failure scenarios
NLU Service Failure
Future improvements
LLM Integration