Choosing the Right Data System

Topics Covered

Decision Framework

Access Pattern Analysis

Consistency Requirements

Scale Requirements

Putting the Framework Together

Common Decision Mistakes

Polyglot Persistence

A Concrete Example

When Polyglot Adds Unnecessary Complexity

Consistency Across Multiple Systems

The Synchronization Tax

Migration Strategies

Strangler Fig Migration

CDC-Based Migration

Dual Writes: The Dangerous Shortcut

Choosing Your Migration Strategy

Total Cost of Ownership

The Four Cost Categories

Managed vs Self-Hosted

Calculating the Real Monthly Cost

The Cheapest Database Is the One Your Team Can Operate

Building a Decision Document

There is no best database. There is only the best database for your specific workload. Engineers who skip this analysis end up refactoring their storage layer 18 months later when the original choice buckles under real production patterns. The cost of a wrong database choice is not just the migration itself: it is the months of degraded performance, the workarounds built on top of a poor fit, and the engineering morale spent fighting a tool instead of building features.

The decision framework below forces you to answer the right questions before you write any code. It is deliberately structured to prevent the two most common mistakes: choosing a database because it is popular (the hype-driven decision) and choosing a database because it is theoretically optimal for a workload you do not actually have yet (the premature optimization decision).

Five questions asked before picking a datastore, each with the usable answer, the shortcut, and the cost of getting it wrong.

Access Pattern Analysis

The single most important factor in choosing a data system is how your application reads and writes data. Two applications with identical data models but different access patterns should use different databases. An access pattern is not just "reads and writes." It includes the shape of queries (point lookups vs scans vs aggregations), the ratio of reads to writes, the concurrency level (10 concurrent users vs 10,000), and whether the data is accessed by primary key, by secondary attributes, or by full-text search.

Read-heavy vs write-heavy. A product catalog serves 100 reads for every write. An IoT telemetry pipeline ingests 50,000 writes per second and reads only during analytics queries. The read/write ratio determines which optimizations matter most. The catalog benefits from a system that optimizes reads with indexes, caching, and materialized views (PostgreSQL, MySQL). The telemetry pipeline benefits from a system optimized for sequential writes and time-based compaction (TimescaleDB, InfluxDB, Cassandra). A mixed workload (50/50 reads and writes) is the hardest case: no system optimizes equally well for both, so you must decide which direction to bias or consider separating the read and write paths with CQRS (Command Query Responsibility Segregation).

Point lookups vs range scans. A user profile service fetches one user by ID: one key in, one record out. An analytics dashboard scans all orders from the past 30 days: no specific key, millions of records out. These two operations are fundamentally different at the storage engine level. Point lookups favor hash indexes and key-value stores (DynamoDB, Redis) that can locate a single record in O(1) time. Range scans favor B-tree indexes and column-oriented storage (PostgreSQL, ClickHouse) that can efficiently read sequential ranges and compress similar values in each column.

Relationships vs documents. An e-commerce system has orders referencing users, products, and shipping addresses. A content management system stores self-contained articles with embedded metadata. Deep relationships favor relational databases with JOIN operations. Self-contained entities favor document stores (MongoDB, DynamoDB) where a single read retrieves everything the application needs.

Graph traversals. A social network that needs "friends of friends who also like hiking within 50 miles" requires traversing relationship edges multiple levels deep. Relational databases can do this with recursive CTEs, but performance degrades rapidly beyond 3-4 levels because each level requires a JOIN against the full table. Graph databases (Neo4j, Amazon Neptune) optimize for multi-hop traversals by storing relationships as first-class citizens with direct pointers between nodes, making 6-level traversals that take seconds in SQL complete in milliseconds.

Full-text search with ranking. If users need to search free-form text and expect results ranked by relevance (not just exact matches), the access pattern favors a search engine. Elasticsearch and OpenSearch use inverted indexes, BM25 scoring, and analyzers (stemming, synonyms, fuzzy matching) to deliver ranked results. PostgreSQL has built-in full-text search that works for simple use cases, but it lacks the scoring sophistication and query flexibility of a dedicated search engine at scale.

Five ways of asking for the same rows, each mapped to the engine property it needs and what Postgres actually does with it.

Understanding which of these categories your workload falls into is the foundation of every database decision. Most applications combine 2-3 of these patterns, which is exactly why the decision is not simple and why the framework exists.

Consistency Requirements

Consistency is not a global property of your system. It is a per-use-case requirement that should be evaluated for each data flow independently.

Strong consistency means every read returns the most recent write. A banking system that shows a wrong balance for even one second is unacceptable. A reservation system that double-books a room because two requests saw stale availability is unacceptable. Strong consistency requires coordination (locks, consensus protocols), which adds latency. PostgreSQL, MySQL, and CockroachDB provide strong consistency by default.

Eventual consistency means reads might return stale data for a brief window (typically milliseconds to seconds). A social media feed that shows a post 2 seconds late is perfectly fine. A product catalog that shows a price from 500 milliseconds ago causes no business harm. Eventual consistency enables higher availability and lower latency because replicas can serve reads without coordinating with the primary. The trade-off is explicit: you accept temporary staleness in exchange for faster reads and the ability to keep serving during partial failures. DynamoDB, Cassandra, and most caching layers are eventually consistent by default.

Single-region vs multi-region changes the calculation entirely. Within a single region, strong consistency costs 1-5ms of latency for coordination. Across regions, it costs 50-200ms because consensus must cross continental distances. Many systems that need strong consistency within a region accept eventual consistency across regions: the user in Tokyo sees their own writes immediately, but a user in London sees them after replication lag.

The consistency choice is not binary for an entire application. A payment service needs strong consistency (you cannot show a wrong balance). A product recommendation engine tolerates eventual consistency (showing a slightly stale recommendation is invisible to the user). Map each data flow to its consistency requirement independently rather than choosing one consistency level for the entire system.

Scale Requirements

Evaluate your current scale and your projected scale for the next 2-3 years, but be honest about projections. Most systems never reach the scale where the database choice becomes the bottleneck. A well-tuned PostgreSQL instance handles 10,000 transactions per second. That covers the vast majority of applications.

When you do need to scale beyond a single node, the question becomes: do you scale reads or writes? Read scaling is straightforward with replicas: add read replicas that serve SELECT queries while the primary handles writes. Most applications are read-heavy (90%+ reads), so replicas alone solve the scaling problem.

Write scaling is fundamentally harder. It requires sharding: splitting data across multiple nodes based on a partition key. Sharding introduces distributed transactions, cross-shard queries (which are slow or impossible), rebalancing when you add nodes, and application complexity for routing writes to the correct shard. Choose a system with built-in sharding (Cassandra, CockroachDB, Vitess) only if you have concrete evidence you will need it, not because you might someday. A premature sharding decision is one of the most expensive architectural mistakes to undo.

Interview Tip

Apply the boring technology principle: choose proven, well-understood tools over cutting-edge options. Every team has a limited number of innovation tokens. Spend them on your core business logic, not on your database. PostgreSQL, MySQL, Redis, and Elasticsearch solve 90% of storage problems. Reach for specialized systems only when you have a measured need that boring technology cannot meet.

Putting the Framework Together

A practical decision process looks like this:

Step 1 - Define your top 3 access patterns with expected read/write ratios.

Step 2 - Identify your consistency requirement per access pattern (strong or eventual).

Step 3 - Estimate your current data volume and 2-year projection.

Step 4 - Check whether PostgreSQL handles all three. If yes, stop here. PostgreSQL with proper indexing, connection pooling, and read replicas covers an enormous range of workloads.

Step 5 - If PostgreSQL does not fit (time-series at 100K writes/sec, graph traversals 6 levels deep, full-text search with ranking), add a specialized system for that specific access pattern. Document the specific measurement that justified the addition: "PostgreSQL full-text search latency exceeds 500ms at p99 under production load, Elasticsearch achieves 15ms for the same queries." Without this evidence, the addition is speculative.

Step 6 - Re-evaluate periodically. Your access patterns change as your product evolves. A database that was perfect for your MVP may not serve your production workload two years later. Schedule an annual review of your data systems against current and projected access patterns. This prevents both premature changes (switching too early based on speculation) and delayed changes (clinging to a poor fit out of inertia).

This framework is deliberately biased toward simplicity. Starting with one database and adding specialized systems only when measured need arises is cheaper, safer, and faster than designing a polyglot architecture from day one.

Why is the bias toward simplicity? Because every additional system has a fixed operational cost regardless of the value it provides. A team maintaining two databases spends engineering time on two sets of backups, two monitoring dashboards, two upgrade cycles, and two on-call runbooks. A team maintaining one database spends half that time. The gap is not proportional to data volume. Even a secondary database holding 1 GB of data requires the same operational rigor as one holding 1 TB: it still needs monitoring, backups, and someone who knows how to troubleshoot it at 3 AM.

Common Decision Mistakes

Choosing for peak load that may never come. Teams estimate future traffic, add a 10x safety margin, and choose a database that handles the inflated number. The result is operational complexity for scale they never reach. Size for your current load with a clear scaling plan if growth materializes.

Choosing based on benchmarks instead of access patterns. A database that processes 1 million reads per second in a benchmark may perform poorly for your specific query patterns. Benchmarks test synthetic workloads. Your workload has specific key distributions, query shapes, and concurrency patterns that no benchmark captures. Always prototype with your actual data and queries before committing.

Ignoring the ecosystem. A database is not just a storage engine. It includes monitoring tools (pg_stat_statements, slow query logs), backup solutions (pg_dump, WAL archiving), migration tools (Flyway, Alembic), ORMs, and community knowledge. PostgreSQL's ecosystem is decades deep. A newer database might have a better storage engine but a fraction of the tooling. That tooling gap translates directly into development and operational cost.

Confusing familiarity with fitness. The opposite mistake also exists. A team that has only used MySQL might reject PostgreSQL for a workload that genuinely needs PostgreSQL features (JSONB, array types, window functions, CTEs with materialized views). The boring technology principle does not mean "use what you already know regardless of fit." It means "default to proven technology, and justify deviations with evidence." If your workload genuinely requires features your current database lacks, switching is justified. The key is distinguishing genuine requirements from theoretical preferences.

Letting the vendor choose for you. Database vendors are excellent at selling solutions. They show benchmarks that highlight their strengths and avoid their weaknesses. A Cassandra vendor will show you linear write scalability at millions of operations per second. They will not mention the operational complexity of managing a production Cassandra cluster, the limitations of CQL compared to SQL, or the number of anti-patterns that cause performance to collapse. Always evaluate databases with your own data and access patterns, not with the vendor's synthetic benchmarks. Request a proof-of-concept period with production-representative workloads before committing.