0%
Big Data Engines: Spark, Lakehouse, and Streaming at Scale
The Shape of the Problem
Distributed Execution
Spark
Streaming at Scale
Pipelines in Production
OLAP Stores, ClickHouse and Druid
A cloud warehouse answers almost every analytical question you can ask, and for most organisations it is the correct and only tool. A dedicated analytical store exists for the narrow band of questions where the warehouse's answer arrives too late to be useful.
The requirement that creates this category
The distinguishing requirement is not volume and it is not complexity. It is latency under concurrency.
A warehouse answers a well pruned query over a large table in a few hundred milliseconds to a few seconds, and it does so for a handful of concurrent users. That profile is excellent for a report, an investigation or a dashboard that a dozen people open each morning.
It is unusable for a feature inside a product. If an application shows every customer a chart of their own activity, the query must return in tens of milliseconds, it must do so for thousands of simultaneous users, and the cost per query must be close to zero because the query runs on every page load. Those three constraints together are what a dedicated analytical store is built to satisfy.
Three numbers that decide it
Before any architectural comparison, write down three numbers, because they determine the answer more reliably than any feature list.
The first is the latency budget at the ninety ninth percentile. Not the average, because an average of two hundred milliseconds with a tail of four seconds produces a product that feels broken. If the budget is above a second, a warehouse will usually do. If it is under a hundred milliseconds, it will not.
The second is peak concurrent queries. A warehouse serving fifty concurrent analytical queries is working hard. A store in this category is expected to serve thousands, and it achieves that partly by refusing to be general.
The third is data freshness. If the answer may be an hour old, almost anything works, including a scheduled table in a warehouse. If events must be queryable within seconds of happening, the ingestion design becomes the hardest part of the system and the query engine becomes secondary.
What they give up to get there
These engines are fast because they are narrow, and the narrowness is the part people discover late.
They are weak at joins. Most of them join far less capably than a warehouse, and several are designed on the assumption that you will denormalise instead. A model that depends on joining three large tables at query time is a poor fit, and rewriting it as one wide table is usually the real work of adopting one of these systems.
They are weak at updates. Rows are appended and deleted in bulk, and updating an individual row ranges from expensive to unsupported. A design that corrects yesterday's records in place will fight the storage engine continuously.
They are weak at arbitrary queries. Performance depends on queries matching the physical layout you chose, and a question that filters on a dimension you did not sort or index can be orders of magnitude slower than one that does. A warehouse degrades gracefully in that situation and these engines often do not.
And they are operationally heavier. Even the managed offerings expose more knobs, more background processes and more failure modes than a warehouse does, because the things a warehouse hides are the things you are here to control.
Where they sit next to a warehouse
The common and correct pattern is both, with a clear division of responsibility.
The warehouse remains the system of record for analysis. It holds full history, it is where transformations run, it is where anybody can ask an unanticipated question, and it is where correctness is reconciled.
The analytical store holds a serving copy of a specific shape, usually a subset of columns, a limited retention window and a layout chosen for the queries the product actually issues. It is fed by a pipeline from the warehouse, from the stream, or from both.
Critically, the analytical store is treated as derived and rebuildable. If it is lost or corrupted, the fix is to rebuild it from the warehouse rather than to recover it, which keeps its operational burden proportionate to its role.
The column store foundation they share
Underneath, these engines share the same core ideas as the warehouse, applied more aggressively.
Data is stored column by column, so a query reading three of two hundred columns reads a small fraction of the bytes. Each column is compressed with a codec chosen for its type, and the compression ratios on low cardinality columns are extreme, often ten or twenty to one.
Execution is vectorised, which means operators process batches of thousands of values at a time rather than one row at a time. This keeps the processor's pipeline full and makes good use of cache lines, and it is a large part of why these engines are faster than their row oriented predecessors at the same hardware.
Data is kept sorted by a chosen key, which allows the engine to skip blocks the same way a warehouse prunes, and to answer range queries by seeking rather than scanning.
What differs between the two leading designs
The two most widely deployed systems in this category solve the same problem with different emphases, and the difference is worth stating plainly.
One is best understood as a very fast general purpose column store. It is a database in the ordinary sense, it speaks a familiar dialect, it supports a wide range of queries, and its speed comes from the quality of its storage engine and execution engine. You shape performance mainly by choosing the sort order and by precomputing with materialised views.
The other is best understood as a purpose built time series serving layer. It assumes a time dimension, it separates ingestion from querying into distinct node roles, it indexes dimension columns heavily, and it is designed from the start to roll data up on ingest. You shape performance mainly by choosing the rollup granularity and the segment layout.
Neither is better. The first rewards a team that wants a database and will tune it. The second rewards a team whose workload really is time sliced dimensional queries and who wants that case handled well by default.
A worked example of the decision
A team serves an in product analytics panel to forty thousand customers. Each panel shows seven charts covering the last thirty days of that customer's events, filtered by a handful of dimensions, and it must load in under half a second in total.
The warehouse answer is one query per chart, each a few hundred milliseconds, which is three seconds and far too slow, and at this concurrency the cost is substantial because every panel load bills for compute.
The precomputation answer is a scheduled job that writes one row per customer per day per dimension combination. This works, it is cheap, and it fails the moment somebody wants a dimension that was not precomputed or a window that does not align to days.
The analytical store answer is a wide, sorted, rolled up table keyed by customer and time, served from memory and local disk, answering each chart in ten to thirty milliseconds at a cost per query near zero. The cost is a new system to run and a model that must be denormalised.
The third answer is right here because the access pattern is fixed, the concurrency is high and the latency budget is tight. Change any one of those three and the second answer becomes better.
Reading the decision backwards
It is worth stating the negative case as clearly as the positive one, because this category attracts adoption that is not justified.
If the queries come from humans rather than from software, you almost certainly do not need this. Humans are slow, few and tolerant of a two second wait.
If concurrency is in the tens rather than the thousands, a warehouse with good pruning and a result cache will serve you at a fraction of the operational cost.
And if the access pattern changes weekly because the product is still being explored, the rigidity that makes these engines fast will work against you until it stabilises.