Loading...
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
POST /v1/translate
Content-Type: application/json
Authorization: Bearer <token>
{
"text": "Hello, how are you?",
"source_language": "en", // optional, auto-detected if omitted
"target_language": "es",
"format": "text" // text | html | markdown
}
Response 200:
{
"translated_text": "Hola, como estas?",
"detected_language": "en",
"confidence": 0.98,
"cached": true
}
POST /v1/translate/batch
Authorization: Bearer <token>
{
"texts": [
{"text": "Good morning", "target_language": "fr"},
{"text": "Thank you", "target_language": "de"},
{"text": "Goodbye", "target_language": "ja"}
],
"source_language": "en"
}
Response 200:
{
"translations": [
{"translated_text": "Bonjour", "target_language": "fr", "cached": true},
{"translated_text": "Danke", "target_language": "de", "cached": true},
{"translated_text": "Sayonara", "target_language": "ja", "cached": false}
]
}
A language translation service takes text in one language and produces equivalent text in another. Google Translate handles over 100 billion words per day across 130+ languages. The core challenge is not just translating accurately — it is doing so at massive scale with low latency, while supporting users who may have no network connection at all.
Key Insight
Unlike most CRUD services, a translation system's compute cost is dominated by ML model inference, not database queries. A single NMT model inference can take 100-500ms on GPU, compared to <1ms for a Redis cache lookup. This 500x cost difference makes caching strategy the single most important architectural decision.
The central architecture challenge is the cost asymmetry between cached and uncached translations. A cache hit costs under 1ms and fractions of a cent. A cache miss costs 200ms and requires expensive GPU inference. This 500x cost gap means the entire system design revolves around maximizing cache hit rate while maintaining translation quality. Every component — the cache key design, TTL strategy, request coalescing, and even the offline mode — exists to reduce the number of times text reaches the GPU translation engine.
Think of it like a library: every book checked out from the shelf (cache hit) is free and instant. Every book that must be special-ordered (cache miss) costs 50 and takes a week. The library's entire strategy revolves around keeping the most-requested books on the shelf. The translation service's strategy revolves around keeping the most-requested translations in Redis.</span></p><p><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">In an interview, this cost asymmetry is the key insight to demonstrate early. Once the interviewer sees that you understand the GPU-vs-cache cost structure, every subsequent design decision flows naturally: why caching is central, why the engine is isolated behind a separate service, why circuit breakers protect the engine, and why offline mode reduces server load. The entire system is engineered around one principle: keep as much traffic as possible away from the GPU.</span></p><h2><a href="https://codemia.io/system-design/design-a-language-translation-service/editorial#capacity-estimation" rel="noopener noreferrer" target="_blank" style="background-color: rgb(14, 19, 32); color: rgb(166, 170, 173);">Capacity estimation</a></h2><p><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">The numbers for a translation service look different from a typical CRUD application because the bottleneck is not database I/O — it is ML model inference on GPUs. Every estimate below traces back to one question: how many translation engine instances do we need?</span></p><h3><span style="background-color: rgb(14, 19, 32); color: rgb(193, 198, 202);">Traffic</span></h3><ul><li><strong style="background-color: rgb(14, 19, 32); color: rgb(193, 198, 202);">Users</strong><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">: 100 million registered users, 10 million daily active users.</span></li><li><strong style="background-color: rgb(14, 19, 32); color: rgb(193, 198, 202);">Translations per day</strong><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">: 10 million translations/day (1 per active user on average, though power users generate 10-50 translations/day). This works out to roughly 116 translations per second on average, with peaks of 3-5x during business hours across multiple time zones — roughly 350-580 translations per second at peak.</span></li><li><strong style="background-color: rgb(14, 19, 32); color: rgb(193, 198, 202);">Cache hit ratio</strong><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">: 70-80% of translation requests are for previously translated text. Common phrases like greetings, menu items, directions, and business terminology repeat constantly across users. At 80% cache hit rate, only 70-116 requests per second reach the translation engine at peak.</span></li><li><strong style="background-color: rgb(14, 19, 32); color: rgb(193, 198, 202);">Batch requests</strong><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">: Approximately 10% of requests are batch translations (translating multiple texts in one API call), averaging 5 texts per batch. These are common in enterprise integrations where documents or product catalogs need translation. Batch requests generate higher engine load per API call but are more GPU-efficient because they benefit from dynamic batching at the inference level.</span></li><li><strong style="background-color: rgb(14, 19, 32); color: rgb(193, 198, 202);">Text length distribution</strong><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">: Most translations are short — 80% are under 50 words (common phrases, UI strings, menu items). 15% are 50-200 words (paragraphs, email snippets). 5% are 200-500 words (full paragraphs, article sections). Text length directly affects inference time: a 10-word phrase takes 100ms, a 200-word paragraph takes 400-500ms. The long-tail of long texts disproportionately consumes GPU time.</span></li><li><strong style="background-color: rgb(14, 19, 32); color: rgb(193, 198, 202);">Peak traffic patterns</strong><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">: Translation traffic is not uniform throughout the day. Business hours in each major timezone create overlapping peaks. The global peak occurs when US, European, and Asian business hours overlap (roughly 8-10 AM US Eastern time). During this window, traffic reaches 3-5x the daily average. Weekend traffic drops to 40-50% of weekday levels. Seasonal spikes occur around holidays (travel-related translations) and major global events (news translation).</span></li></ul><h3><span style="background-color: rgb(14, 19, 32); color: rgb(193, 198, 202);">Translation Engine Capacity</span></h3><p><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">A single NMT model inference on GPU takes 100-500ms depending on text length. Assume 200ms average for a typical 20-word sentence. A single GPU instance handles roughly 5 translations per second with sequential inference. With dynamic batching (processing 4-8 translations simultaneously), throughput increases to 20-30 translations per second per GPU. At 116 cache misses per second (peak), the system needs roughly 6 GPU instances with batching. With 2x headroom for spikes and model loading overhead: 12-15 GPU instances. Without batching, the same calculation yields 48 instances — batching alone reduces GPU costs by 3-4x.</span></p><p><span style="background-color: rgb(14, 19, 32); color: rgb(160, 174, 192);">This is the most expensive component by far — each GPU instance costs 1-3/hour depending on the cloud provider and GPU type. An NVIDIA A10G (24GB VRAM) at 3/hour handles 10+ models but is overkill for most deployments.
The key insight is that cache hit ratio directly controls GPU costs. Improving cache hit rate from 70% to 80% reduces GPU instances needed by one-third. This is why cache warming, TTL tuning, and intelligent cache key design are not optimizations — they are cost controls.
The monthly cost breakdown reveals why caching dominates every architecture discussion for this system: GPUs represent 85% of infrastructure costs. A 10% improvement in cache hit rate (from 80% to 88%) reduces GPU instances from 15 to 9, saving roughly 8,700/month. No other optimization comes close to this impact. This is why the system invests heavily in cache warming, intelligent TTL management, and request coalescing — each percentage point of cache hit improvement translates directly to thousands of dollars in monthly savings.
Not all language pairs see equal traffic. In practice, traffic follows a power law: English-to-Spanish, English-to-French, and English-to-Chinese account for roughly 40% of all translations. The top 10 language pairs cover 70% of traffic. The remaining 40+ language pairs share 30% of traffic, with many receiving fewer than 100 requests per day. This distribution has direct implications for model loading (keep popular models pinned in GPU memory), caching (popular pairs have higher cache hit rates), and offline support (offer offline models for the top 10-15 pairs only).
The power law distribution also affects caching effectiveness. Popular language pairs like English-Spanish have cache hit rates of 85-90% because many users translate the same common phrases. Rare pairs like Swedish-Korean might have cache hit rates of 30-40% because the request volume is too low for repeat queries. This means rare pairs consume disproportionate GPU resources relative to their traffic share — 30% of traffic generating 60%+ of engine load. Capacity planning must account for this uneven distribution rather than assuming uniform cache hit rates across all pairs.
Understanding this distribution is also essential for deciding which language pairs to self-host versus outsource to third-party providers. High-volume pairs like English-Spanish generate enough traffic to justify the fixed cost of training and maintaining a self-hosted model. Low-volume pairs like Finnish-Thai are better served by a third-party provider where the cost scales with usage rather than requiring a dedicated model.
Interview Tip
The cache hit ratio is your GPU bill. At 2/hourperGPUinstance,improvingcachehitratefrom702/hourperGPUinstance,improvingcachehitratefrom70240/day in GPU costs. Monitor cache hit rate as a financial metric, not just a performance metric.
The API needs to support three core operations: single translation, batch translation, and language detection. Every endpoint requires authentication via API key or JWT token in the Authorization header.
The API follows REST conventions with JSON request/response bodies. Version prefix /v1/ allows future breaking changes without disrupting existing clients. All endpoints return consistent error formats with machine-readable error codes and human-readable messages.
POST /v1/translate
Content-Type: application/json
Authorization: Bearer <token>
{
"text": "Hello, how are you?",
"source_language": "en", // optional, auto-detected if omitted
"target_language": "es",
"format": "text" // text | html | markdown
}
Response 200:
{
"translated_text": "Hola, como estas?",
"detected_language": "en",
"confidence": 0.98,
"cached": true
}
The source_language field is optional. When omitted, the service runs language detection first, adding 10-20ms to the request. The detected_language field in the response confirms what was detected, useful for the client to display. The cached field indicates whether the result came from cache (useful for debugging and monitoring).
POST /v1/translate/batch
Authorization: Bearer <token>
{
"texts": [
{"text": "Good morning", "target_language": "fr"},
{"text": "Thank you", "target_language": "de"},
{"text": "Goodbye", "target_language": "ja"}
],
"source_language": "en"
}
Response 200:
{
"translations": [
{"translated_text": "Bonjour", "target_language": "fr", "cached": true},
{"translated_text": "Danke", "target_language": "de", "cached": true},
{"translated_text": "Sayonara", "target_language": "ja", "cached": false}
]
}
Batch requests are processed in parallel on the server side. Each text in the batch independently checks cache and routes to the engine only on miss. The maximum batch size is 100 texts. Batch requests are common in enterprise integrations where product catalogs, menus, or help articles need translation.
POST /v1/detect
Authorization: Bearer <token>
{
"text": "Bonjour le monde"
}
Response 200:
{
"detected_language": "fr",
"confidence": 0.97,
"alternatives": [
{"language": "it", "confidence": 0.02},
{"language": "es", "confidence": 0.01}
]
}
The detection endpoint is separate from translation because some clients need language identification without translation — for example, routing support tickets to the right language team. The alternatives array helps when confidence is low (short texts like "OK" or "No" are ambiguous across many languages).
Rate limits are per API key, enforced at the API gateway:
Rate limit headers are included in every response: X-RateLimit-Remaining, X-RateLimit-Limit, X-RateLimit-Reset. When the limit is exceeded, the API returns 429 Too Many Requests with a Retry-After header.
| Status Code | Meaning | When |
| 400 | Bad Request | Missing target_language, unsupported language pair, text exceeds 10,000 characters |
| 401 | Unauthorized | Invalid or expired API key/token |
| 429 | Rate Limited | Request exceeds tier limit |
| 503 | Service Unavailable | Both primary and secondary translation providers are down |
The translation history endpoint supports cursor-based pagination:
GET /v1/translations/history?limit=20&cursor=abc123
Authorization: Bearer <token>
Response 200:
{
"translations": [...],
"next_cursor": "def456",
"has_more": true
}
Cursor-based pagination is preferred over offset-based because translation history is append-heavy. With offset pagination, new translations added between page requests cause items to shift, leading to duplicates or missed items. Cursor pagination uses the last-seen translation ID as a stable reference point, ensuring consistent page boundaries regardless of new insertions.
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
Client (Web/Mobile): The user-facing application where text is entered and translated output is displayed. Mobile clients include an offline translation module with pre-trained models and a local cache. The client sends requests to the API gateway, which handles authentication and rate limiting before forwarding to the backend. Web clients are thin — they rely entirely on server-side translation. Mobile clients are fat — they include offline models, local cache, and sync logic. This asymmetry means mobile development requires more engineering investment than web, but also delivers a better user experience in low-connectivity scenarios.
API Gateway: The entry point for all client requests. Handles TLS termination, authentication (JWT validation or API key lookup), rate limiting (sliding window counter in Redis), and request routing. The gateway rejects invalid requests before they consume any backend resources. It also serves as the point where API versioning is enforced — /v1/ and /v2/ routes are separated at the gateway level.
Load Balancer: Distributes incoming requests across multiple backend service instances using round-robin or least-connections. Health checks remove unhealthy instances from the pool within 10 seconds of a failure. The load balancer also terminates TLS, so backend communication happens over plaintext within the private network, reducing per-request encryption overhead. For geographic distribution, multiple load balancers operate in different regions, with DNS-based routing directing users to the nearest one.
Backend Service (BFF): The backend-for-frontend layer handles request validation, user authentication, and routing. It normalizes input text (trim, lowercase for cache key computation, collapse whitespace) before forwarding to the translation service. The BFF also handles response formatting, logging, and metrics emission. Multiple instances run behind the load balancer for horizontal scaling. The BFF is stateless — all request context is contained in the request itself, so any BFF instance can handle any request without session affinity.
Translation Service: The core service that orchestrates translation. For each request, it computes the cache key, checks Redis, and either returns the cached result or forwards to the translation engine. This service also handles request coalescing: if two identical requests arrive simultaneously, only one engine call is made and the result is shared. The translation service maintains a connection pool to both Redis and the translation engine. It is the most critical component in the system — it sits between the cheap cache layer and the expensive engine layer, deciding which requests need GPU resources and which can be served from cache.
Redis Cache Cluster: A Redis cluster with read replicas stores cached translations. The cluster handles 50,000+ lookups per second. Hot keys (popular translations like "Hello" in English-to-Spanish) are automatically distributed across replicas. TTL is set per language pair — stable language pairs like English-French have 7-day TTL, while rapidly improving pairs might have shorter TTL.
Translation Engine (NMT): The GPU-powered neural machine translation engine. Each instance loads one or more NMT models and performs inference. The engine runs behind an internal load balancer that routes requests to instances with the appropriate language pair model loaded. Model loading takes 30-60 seconds, so instances are pre-warmed with popular models. The engine is the only component that requires GPU hardware — everything else runs on standard CPU instances at a fraction of the cost.
Message Queue (Kafka): Sits between the translation service and PostgreSQL. Translation history records are published to a Kafka topic after each translation, then consumed by a writer service that batches inserts into PostgreSQL. Kafka provides durability during PostgreSQL maintenance or outages — history records queue up and drain when the database recovers. The topic is partitioned by user ID to ensure a user's translations are written in order.
Circuit Breaker: Sits between the translation service and the translation engine (or third-party providers). Monitors failure rates and latency. When failures exceed a threshold (e.g., 50% of requests failing over 30 seconds), the circuit opens and routes traffic to the secondary provider. After a cooldown period, it enters half-open state and sends test requests to check recovery.
The mobile client includes a lightweight offline module. When the device has no network connectivity, translation requests are handled locally. The offline module has two layers: a local cache of the user's recent translations (fast, exact match only) and a compressed NMT model for the user's selected language pairs (slower, handles new text). When connectivity returns, the client syncs any offline translations with the server for history tracking and to improve cache coverage.
The offline models are compressed using two techniques. Quantization reduces weight precision from 32-bit to 8-bit or 4-bit, shrinking the model to 1/4 or 1/8 of its original size with a 10-15% quality reduction. Knowledge distillation trains a smaller "student" model to mimic the larger "teacher" model's outputs, further reducing size while preserving most of the quality. The combination produces models that are 50-100MB (compared to 500MB+ for server-side models) with quality sufficient for common phrases and simple sentences.
Model updates are managed through the app's background download mechanism. The client checks for model updates on app launch and downloads new versions over WiFi only (to avoid consuming mobile data). The update is applied atomically: the old model continues serving requests until the new model is fully downloaded and validated, preventing a window where offline translation is unavailable.
A common concern is storage consumption on the user's device. Three language pairs at 100MB each consume 300MB — significant on a 64GB phone. The client manages this by tracking usage frequency: if a downloaded language pair has not been used in 30 days, the app suggests removing it and reclaiming the storage. Users who travel infrequently to a region can re-download the model before their next trip. The app also supports partial model updates using binary diffs — if the new model differs by only 10% from the current one, only the changed portions are downloaded, reducing update size from 100MB to 10MB.
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
Redis serves as the primary translation cache. The access pattern is simple: exact key lookup with high throughput.
Key: translation:{sha256(text)}:{source_lang}:{target_lang}:{format}
Value: {
"translated_text": "Hola, como estas?",
"detected_language": "en",
"confidence": 0.98,
"created_at": "2026-03-15T10:00:00Z",
"model_version": "nmt-v3.2"
}
TTL: 7 days (configurable per language pair)
The key includes the SHA-256 hash of normalized input text (lowercased, trimmed, whitespace-collapsed) concatenated with source language, target language, and format. This ensures that "Hello" and "hello" hit the same cache entry, while "Hello" in English-to-Spanish and English-to-French are different entries.
PostgreSQL stores user data, translation history, and API key configuration. These require ACID transactions and relational queries.
sql
CREATE TABLE users (
id UUID PRIMARY KEY,
email VARCHAR(255) UNIQUE NOT NULL,
api_key VARCHAR(64) UNIQUE NOT NULL,
tier VARCHAR(20) DEFAULT 'free',
created_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE translation_history (
id UUID PRIMARY KEY,
user_id UUID REFERENCES users(id),
source_text TEXT NOT NULL,
translated_text TEXT NOT NULL,
source_lang VARCHAR(10) NOT NULL,
target_lang VARCHAR(10) NOT NULL,
cached BOOLEAN DEFAULT FALSE,
latency_ms INTEGER,
created_at TIMESTAMP DEFAULT NOW()
);
-- Partitioned by month for efficient retention management
CREATE TABLE translation_history_202603 PARTITION OF translation_history
FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');
CREATE INDEX idx_history_user_date
ON translation_history (user_id, created_at DESC);
Why not store history in Redis? Translation history needs relational queries (show me my translations this week, filter by language pair) and durable storage with ACID guarantees. Redis is volatile by design — even with persistence, it prioritizes speed over durability. PostgreSQL's monthly partitioning allows dropping old partitions cleanly when the 6-month retention window expires.
The idx_history_user_date index supports the most common query: "show me my recent translations" (sorted by date, filtered by user). Without this index, the query would scan the entire monthly partition. The composite index on (user_id, created_at DESC) returns results in the correct order without a sort step, making the "my history" page load in under 10ms even for users with thousands of translations. A secondary index on (source_lang, target_lang, created_at) supports the analytics query "show me all English-to-Spanish translations this month" which is useful for monitoring per-pair cache hit rates and identifying translation quality trends.
sql
CREATE TABLE language_models (
id UUID PRIMARY KEY,
language_pair VARCHAR(20) NOT NULL, -- e.g., 'en-es'
model_version VARCHAR(20) NOT NULL,
model_size_mb INTEGER NOT NULL,
is_offline BOOLEAN DEFAULT FALSE,
accuracy_bleu DECIMAL(4,2),
deployed_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE supported_languages (
code VARCHAR(10) PRIMARY KEY, -- e.g., 'en', 'es', 'zh'
name VARCHAR(100) NOT NULL,
script VARCHAR(20) NOT NULL, -- latin, cyrillic, cjk, arabic
offline_available BOOLEAN DEFAULT FALSE
);
This metadata drives the API: which language pairs are supported, which are available offline, and which model version to use. Updates to this table trigger cache invalidation for affected language pairs.
When a translation request arrives: Redis is checked first (sub-millisecond). On cache miss, the translation engine produces a result that is simultaneously written to Redis (for future cache hits) and PostgreSQL (for history). This dual-write is eventually consistent — if the PostgreSQL write fails, the translation still succeeds and history is logged asynchronously via a message queue.
The message queue (Kafka or SQS) between the translation service and PostgreSQL serves a dual purpose. First, it decouples translation latency from database write latency. Second, it acts as a buffer during PostgreSQL maintenance windows — translation records queue up and drain when the database is back online. The queue retention is set to 24 hours, long enough to survive any planned maintenance window.
For the offline sync path, the flow is reversed: the client uploads cached offline translations to the server. The server stores them in PostgreSQL history, checks if the server-side model produces a better translation, and if so, returns the improved version. The client updates its local cache with the higher-quality result.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
The online translation flow has two paths: the fast path (cache hit) and the slow path (cache miss). The critical design goal is maximizing the percentage of requests that take the fast path. Every architectural optimization in the system — cache warming, request coalescing, fuzzy matching — exists to shift more traffic from the slow path to the fast path.
Latency breakdown: Cache hit path: 1-5ms total (gateway + cache lookup). Cache miss path: 150-550ms total (gateway 1ms + normalization 2ms + detection 15ms + inference 100-500ms + post-processing 10ms + cache write 1ms + response formatting 1ms).
Notice the 100x latency difference between cache hit and cache miss. This is why the system prioritizes cache hit rate above all other optimizations. A 1% improvement in cache hit rate (from 79% to 80%) shifts 1% of all requests from the 200ms path to the 5ms path — a meaningful improvement in average latency and GPU costs.
The translation engine processes text through six sequential stages. Each stage transforms the input and can reject or modify it before passing to the next stage. The pipeline architecture means failures at any stage produce a clear error rather than a subtly wrong translation.
Stage 1 — Input Processing: Raw text is tokenized into subword units using Byte Pair Encoding (BPE). BPE breaks words into frequent subword pieces: "unhappiness" becomes "un", "happi", "ness". This is critical because NMT models have a fixed vocabulary (typically 32,000-64,000 tokens). Rare words that are not in the vocabulary are decomposed into known subword pieces. Without BPE, any word not in the training vocabulary would be untranslatable. Input processing also handles character encoding normalization (converting various Unicode representations to a standard form), HTML entity decoding, and text length validation.
Stage 2 — Language Detection: If the source language is not specified, the detector analyzes the tokenized text using three signals: script identification (Cyrillic narrows to Russian, Ukrainian, Bulgarian, etc.), character n-gram frequency analysis (trigram patterns distinguish similar-script languages like Spanish vs Portuguese), and a lightweight classifier trained on short text snippets. Detection runs on CPU in 10-20ms and achieves 97%+ accuracy for text over 20 characters. For very short text (under 10 characters), accuracy drops to 80%, so the response includes a confidence score.
Stage 3 — NMT Model Inference: The core translation step. The selected NMT model (specific to the language pair) uses an encoder-decoder transformer architecture. The encoder processes the source tokens into a contextual representation — each token is mapped to a vector that captures its meaning in context, not just its dictionary definition. The decoder generates target tokens one at a time, using attention mechanisms to focus on relevant parts of the source text. This autoregressive generation is why inference time scales with output length: a 10-word output generates 10 decode steps, while a 100-word output generates 100 steps. This is the most expensive stage: 100-500ms on GPU depending on text length. Each model is roughly 200-500MB in memory, so GPU instances are pre-loaded with models for popular language pairs.
The inference stage uses beam search to explore multiple translation candidates simultaneously. A beam width of 4-8 means the model generates 4-8 candidate translations in parallel, then selects the highest-scoring one. Beam search improves translation quality over greedy decoding (always picking the most likely next token) at the cost of proportionally more computation. The beam width is configurable per quality tier: standard uses beam width 5, fast uses beam width 2, draft uses greedy decoding.
Stage 4 — Context Analysis: The raw model output is analyzed for coherence. This stage checks for hallucinated content (the model generating text unrelated to the input, a known failure mode of transformer models), untranslated segments (source language text appearing in the output), and length ratio anomalies (a 10-word input producing a 50-word output suggests an error). Flagged outputs are either corrected automatically or returned with a low confidence score.
Stage 5 — Quality Evaluation: The translation is scored using automated metrics. BLEU score compares the output against reference translations for known text. For novel text, a learned quality estimator (a smaller ML model) predicts human judgment scores without reference translations. Translations scoring below a threshold are flagged for the context analysis stage to re-examine or for the client to display a quality warning.
Stage 6 — Post-Processing: The final stage applies grammar corrections, proper noun preservation (names, places, brands should not be translated), number and date format localization (12/03/2026 becomes 03/12/2026 for European formats), and detokenization (converting subword tokens back to readable text). The output is the final translated string returned to the caller.
Proper noun preservation deserves special attention because it is a common failure mode. NMT models sometimes translate names: "Apple" (the company) becomes the word for the fruit in the target language, or "Jordan" (the person) becomes the country name. The post-processor uses a named entity recognition (NER) model to identify proper nouns in the source text, then checks whether the corresponding tokens in the output were translated. If a proper noun was incorrectly translated, the post-processor restores the original. This NER check adds 5-10ms but prevents embarrassing errors in business and news translation where proper nouns appear frequently.
The entire pipeline takes 150-550ms for a cache miss, with NMT inference dominating at 100-500ms. The other five stages combined take 50ms or less. This cost breakdown explains why the system invests heavily in caching and coalescing — every cache hit skips the entire pipeline, saving 150-550ms of latency and GPU resources.
One non-obvious implication: the pipeline stages must be resilient to partial failures. If the quality evaluation stage crashes, the system should still return the translation (without the quality score) rather than failing the entire request. Each stage has a timeout and a bypass flag. If a non-critical stage (quality evaluation, context analysis) times out, the pipeline continues with the remaining stages and marks the response as "quality-unscored." Only critical stages (tokenization, NMT inference, detokenization) can fail the request. This design ensures that a bug in monitoring or quality scoring does not degrade the core translation function.
Monitoring at each stage provides precise diagnostics. If the P99 latency for Stage 3 (NMT inference) spikes from 300ms to 800ms, the team knows the GPU is under load or the model has a regression. If Stage 5 (quality evaluation) starts flagging 5% of outputs instead of the usual 1%, the model may have drifted and needs retraining. Stage-level metrics turn a opaque "translations are slow" complaint into a specific, actionable diagnosis.
When multiple users submit identical translations simultaneously (a news headline goes viral, a popular phrase trends), the naive approach sends each request to the engine independently. Request coalescing detects that an identical request is already in flight and waits for the first result rather than dispatching a duplicate.
Implementation uses a concurrent hash map keyed by the cache key. When a request arrives and the cache misses, the service checks the in-flight map. If the key exists, the request joins a waiting list. If not, the request registers itself and proceeds to the engine. When the engine returns, the result is delivered to all waiting requests and written to cache.
There is a subtle race condition to handle: what happens when the engine call for a coalesced request fails? If the first request fails and 50 requests are waiting on it, all 50 receive an error. The system must then clean up the in-flight entry so the next request retries fresh rather than joining a failed entry. The cleanup must be atomic — if two requests arrive simultaneously after the failure, only one should trigger a new engine call. This is implemented using compare-and-swap on the in-flight map entry: the entry is removed only if its value still matches the failed future.
The window for coalescing is naturally short (the engine inference time of 100-500ms), but during viral events, even this short window catches dozens of duplicate requests. At 580 requests/sec, if 5% are duplicates within a 200ms window, coalescing saves roughly 6 engine calls per second — modest in steady state but critical during traffic spikes when cache is cold.