Design a Language Translation Service

A language translation service takes text in one language and produces equivalent text in another. Google Translate handles over 100 billion words per day across 130+ languages. The core challenge is not just translating accurately — it is doing so at massive scale with low latency, while supporting users who may have no network connection at all.

Functional

• Text translation with multi-language support: Users submit text in a source language and receive translated output in a target language. The service supports 50+ language pairs. The source language can be explicitly specified or auto-detected. Each translation preserves the meaning, grammar, and tone of the original as closely as possible.
• Automatic language detection: When the user does not specify a source language, the system identifies it automatically. This is critical for usability — most users paste text without knowing or caring what language it is. Detection uses statistical analysis of character patterns, n-grams, and script identification. Accuracy must be above 95% for the top 20 languages. For short texts (under 10 characters), detection is less reliable — "No" exists in English, Spanish, and Portuguese — so the system returns a confidence score alongside the detected language.
• Offline translation: Users can translate text without a network connection using pre-trained models stored on the device. The offline model is smaller and less accurate than the server-side model, but it provides usable translations for common language pairs. Frequently translated phrases are cached locally for instant retrieval.
• Translation user interface: A web or mobile client where users type or paste text, select the target language, and see the translated output. The interface supports copy, share, and history features. Batch mode allows translating multiple texts in a single request.
• Translation history: Users can view their past translations, search by source or target text, and re-use previous translations without re-submitting. History is stored per user and accessible across devices. Enterprise accounts can view team-wide translation history for consistency checking.

The numbers for a translation service look different from a typical CRUD application because the bottleneck is not database I/O — it is ML model inference on GPUs. Every estimate below traces back to one question: how many translation engine instances do we need?

Traffic

• Users: 100 million registered users, 10 million daily active users.
• Translations per day: 10 million translations/day (1 per active user on average, though power users generate 10-50 translations/day). This works out to roughly 116 translations per second on average, with peaks of 3-5x during business hours across multiple time zones — roughly 350-580 translations per second at peak.
• Cache hit ratio: 70-80% of translation requests are for previously translated text. Common phrases like greetings, menu items, directions, and business terminology repeat constantly across users. At 80% cache hit rate, only 70-116 requests per second reach the translation engine at peak.
• Batch requests: Approximately 10% of requests are batch translations (translating multiple texts in one API call), averaging 5 texts per batch. These are common in enterprise integrations where documents or product catalogs need translation. Batch requests generate higher engine load per API call but are more GPU-efficient because they benefit from dynamic batching at the inference level.
• Text length distribution: Most translations are short — 80% are under 50 words (common phrases, UI strings, menu items). 15% are 50-200 words (paragraphs, email snippets). 5% are 200-500 words (full paragraphs, article sections). Text length directly affects inference time: a 10-word phrase takes 100ms, a 200-word paragraph takes 400-500ms. The long-tail of long texts disproportionately consumes GPU time.
• Peak traffic patterns: Translation traffic is not uniform throughout the day. Business hours in each major timezone create overlapping peaks. The global peak occurs when US, European, and Asian business hours overlap (roughly 8-10 AM US Eastern time). During this window, traffic reaches 3-5x the daily average. Weekend traffic drops to 40-50% of weekday levels. Seasonal spikes occur around holidays (travel-related translations) and major global events (news translation).

The API needs to support three core operations: single translation, batch translation, and language detection. Every endpoint requires authentication via API key or JWT token in the Authorization header.

The API follows REST conventions with JSON request/response bodies. Version prefix /v1/ allows future breaking changes without disrupting existing clients. All endpoints return consistent error formats with machine-readable error codes and human-readable messages.

Translation Endpoint

The translation service uses three storage systems, each chosen for a specific access pattern. Using a single database for everything would create a system that is mediocre at all three patterns instead of excellent at each.

The natural question is: why three databases? Because each data type has a fundamentally different access pattern. Translations need sub-millisecond key lookup at 50,000+ operations per second (Redis). User history needs relational queries with ACID transactions (PostgreSQL). Language model metadata needs infrequent reads with strong consistency (PostgreSQL). Forcing all three into a single store means either overpaying for Redis features you don't need for history, or accepting PostgreSQL latency for cache lookups that need to be sub-millisecond.

Redis — Translation Cache

The architecture follows a standard request-response pattern with one critical addition: a caching layer that shields the expensive translation engine from the majority of traffic. Without this cache, the system would need 5x more GPU instances and cost 5x more to operate.

The design separates components by cost profile: cheap stateless CPU services handle request routing, authentication, and caching, while expensive GPU instances handle only the translation inference that cannot be avoided. This separation is the single most important architectural decision because it lets each component scale independently based on its own demand curve.

Understanding the request flow reveals why each component exists and how they interact. The translation request flow is the critical path — every millisecond of latency here is felt by the user.

Online Translation Flow

The translation engine pipeline is the core of this system — it is the component that actually translates text. Everything else (caching, load balancing, circuit breakers) exists to protect this expensive component and make it efficient. Understanding the pipeline stages explains why translation is computationally expensive and where optimization opportunities exist.

Translation Engine Pipeline (GATE)

Every architectural decision in this system involves a tradeoff between cost, quality, latency, and operational complexity. The choices below are the ones most likely to come up in an interview. For each tradeoff, the section presents both sides and explains which option this design chooses and why.

The common theme across all these tradeoffs: optimize for the common case while having a fallback for the uncommon case. The system does not try to handle every scenario perfectly — it handles the 80% case excellently and the 20% case acceptably.

Third-Party vs Self-Hosted Translation

In a distributed translation system, failures are not exceptional events — they are routine. The question is not whether the translation engine, cache, or network will fail, but how the system behaves when they do. Each failure scenario below has a specific mitigation that prevents cascading failures.

The design philosophy is defense in depth: every component has an explicit failure mode and a specific fallback. No single component failure should take down the entire service. The system degrades gracefully — from full-quality cached translations, to engine-backed fresh translations, to stale cached translations, to cache-only mode, to offline mode on the client. Each degradation level provides a worse but still useful experience.

When designing failure handling, the key question for each scenario is: what does the user see? A well-handled failure means the user gets a translation (possibly slightly stale or lower quality) with no visible error. A poorly-handled failure means the user sees a spinner followed by an error message. Every mitigation below optimizes for the user experience first and system health second.

These improvements extend the system beyond text-in, text-out translation. Each one builds on the existing architecture rather than requiring a redesign. The ordering below roughly follows implementation priority — speech translation and document translation are the highest-impact features, while multi-tenant isolation is infrastructure that enables enterprise growth.

Speech-to-Text Translation

Add a speech processing pipeline: voice input is transcribed to text (using a speech recognition model), translated via the existing NMT pipeline, and optionally synthesized back to speech (text-to-speech). The existing caching layer works for the translation step but not for speech recognition, which produces different text for the same spoken input depending on accent, background noise, and speaking speed. This requires streaming support — the client sends audio chunks, the server transcribes in real time, and translation begins before the speaker finishes.