Most data platforms test their data in the wrong place. The tests exist, they are written carefully, and they run after the table has already been published. The result is a platform that knows about its own defects a few minutes after everybody downstream has read them.

A quality gate is not a test. It is a test plus the authority to stop a publish. That second half is the entire idea, and it is the half that almost every team leaves out, because adding it means somebody has to be willing to serve yesterday's numbers rather than today's wrong ones.

The order of operations nobody writes down

Write down the sequence your pipeline actually performs tonight. On most platforms it reads something like this. Read the source. Transform. Write to the published table. Run assertions. Alert on failure. Four of those five steps are fine and the third and fourth are in the wrong order.

The sequence a gate requires is different. Read the source. Transform. Write to a staging location that nobody reads. Run assertions against the staging location. If they pass, make the staging location the published table. If they fail, leave the published table exactly as it was and raise an alert that says the data is stale rather than wrong.

Nothing about that sequence is clever. It is the same pattern as a blue green deployment, applied to a table instead of a service, and for the same reason. You want the thing consumers read to change only after you have decided it should.

Writing to a staging table and swapping

The objection people raise first is cost. If every model writes twice, surely the platform pays twice. It does not, because the second write is not a write.

On a modern warehouse the swap is a metadata operation. You write the new data to a table with a temporary name, and then you atomically rename it over the published name, or you point a view at the new table, or you commit a new snapshot pointer in the table format. On this platform the swap takes about two seconds regardless of whether the table holds four million rows or two point nine billion, because nothing is being copied. Only a pointer moves.

Compare that to the alternative when something goes wrong. Without the staging step, a bad publish has to be undone by re-running the model from the last good source state, which on the largest fact table here is eleven minutes, assuming you still know what the last good state was. The two seconds is not a cost. It is the cheapest insurance the platform can buy.

What a quarter looked like without gates

This platform publishes eighty six datasets that other teams consume, out of four hundred and twelve models in total. The other three hundred and twenty six are intermediate and nobody outside the data team reads them.

In the quarter before gates existed, bad data reached a dashboard fourteen times. The median time from the publish to somebody noticing was four hours and twenty minutes, which is roughly the gap between a two in the morning run and the first person opening a report. The worst case was three days, a revenue figure that was wrong by a few percent in a direction nobody questioned.

Those fourteen incidents are the business case, and they are worth classifying rather than counting. Nine of the fourteen would have been caught by an assertion the team had already written, running against staging, before the swap. The data was wrong in a way the platform already knew how to detect and had chosen to detect too late.

Nine of fourteen, and the five that were not bugs

The remaining five are the interesting ones. In those five the pipeline was correct, the source was correct, and the expectation encoded in somebody's head was wrong. A category had been renamed upstream. A new region had started reporting. A product had been discontinued and its row count went to zero, which is exactly what it should have done.

A gate would not have helped with those five, and a gate that tried to would have been a gate that blocked legitimate change. This is the most important calibration in the whole subject. Gates catch defects. They do not catch disagreements about what the data should say, and tuning them until they do produces a platform that nobody can ship a change to.

What a gate costs

Assertions are queries, and queries cost money and time. On this platform the full set of gate assertions adds three minutes and forty seconds to a nightly run that takes forty one minutes, which is about nine percent.

Nine percent is acceptable. The first measurement was not. Of that three minutes and forty, two minutes and ten seconds came from a single referential integrity check that was scanning the entire fact table to confirm that every order referenced a real customer. Scoping that check to the partitions the run had just written took it from two minutes and ten seconds to eighteen seconds, and changed nothing about what it caught, because the rows outside those partitions had already been checked on the night they arrived.

That is the general rule for gate performance. A gate that checks the whole table on every run is paying for history it already validated. Check what you just wrote.

Where the staging table physically lives

One practical detail decides whether the staging step is as cheap as described. The temporary table has to live in the same storage location and the same table format as the published one, because that is what makes the swap a pointer move rather than a copy. Writing staging output to a different schema on different storage turns the swap into a full rewrite, and the two seconds becomes eleven minutes again with nobody noticing that the mechanism quietly stopped working.

The conventions that keep this honest are unglamorous. Name the staging table deterministically from the published name and the run identifier, so a crashed run leaves an obvious orphan rather than a mystery. Put a retention policy on the staging namespace so those orphans are cleaned up without anybody deciding to. And assert in the pipeline itself that the staging and published tables share a storage root, because the failure mode of getting this wrong is a cost regression rather than an error, and cost regressions are not noticed for months.

Why gating inside the pipeline beats gating in the orchestrator

There is a tempting alternative where the model publishes, a separate orchestrator task runs the assertions, and a third task rolls back on failure. It looks equivalent and it is not, for two reasons.

The first is the window. Between the publish and the rollback there is a period, usually minutes, where the wrong data is readable. On a platform where a pricing service queries a dataset every fifteen minutes, minutes are enough.

The second is the failure mode of the orchestrator itself. If the assertion task fails to start, or the rollback task is the thing that crashes, the bad publish stands and the system that was supposed to catch it is the system that broke. A gate inside the pipeline, between the write and the swap, has no such gap, because there is no state in which the data is both published and unverified.

Gates that run after publish are alerts

It is worth being blunt about the vocabulary, because teams use the word gate for things that do not gate anything. If an assertion runs after the publish and its only effect is a notification, it is monitoring. Monitoring is useful and this platform keeps plenty of it. But a notification does not prevent a read, and the cost of a defect is paid by whoever read the table, not by whoever read the alert.