Every record that arrives at a stream processor carries at least two timestamps, and almost every confusing bug in streaming comes from an engineer reasoning about one of them while the system is using the other.

The timestamps a record actually carries

The first timestamp is the moment the thing happened. A user tapped a button, a sensor read a value, a payment cleared. This is the event time, and it was fixed the instant the event occurred. Nothing that happens afterwards changes it. If the record is replayed tomorrow, or reprocessed next year from an archive, the event time is still the same number.

The second timestamp is the moment your operator saw the record. This is processing time, and it is simply the wall clock of the machine running the code. It is different on every run, different on every machine, and it moves forward whether or not any data is arriving.

There are usually more than two. A broker stamps each record on ingestion, which gives you a third clock that sits between the other two. A collector or gateway may stamp it on receipt, which gives you a fourth. Each of these is a real measurement of something, and each of them is useful for a different question. The mistake is treating any of them as interchangeable.

What processing time is good for

Processing time gets a bad reputation in streaming documentation, which is unfair. It is exactly the right clock for a specific set of questions, and those questions are more common than people admit.

It is the right clock for anything about the health of the pipeline itself. How many records did this operator handle in the last minute of real time. How long has it been since this checkpoint completed. How stale is the output right now. These are all questions about the system, not about the world, and the system's own clock is the correct instrument.

It is also the right clock when the answer only needs to be approximately right and needs to be fast. A dashboard showing current traffic, a rate limiter, an alert on a sudden drop in volume. If a handful of records land in the wrong minute because the network was briefly slow, nobody is harmed.

What processing time is not good for is anything that must agree with itself across runs. The moment a result needs to be the same when it is computed twice, processing time is disqualified, because it is a different number on the second run by construction.

Reproducibility is the whole argument

Here is the test that settles the choice. Run the job today. Run it again next week against the same archived input. Do you need the two outputs to be identical?

If yes, you must use event time. There is no configuration that makes processing time reproducible, because the quantity it measures genuinely changed between the two runs.

This sounds abstract until it bites. A billing pipeline that aggregates usage per hour, built on processing time, will produce a different bill if it is ever rerun after an outage, because the records that were delayed by the outage will land in later hours the second time. A fraud model trained on processing time features will see a distribution in training, from a replayed archive, that it never sees in production, because the replay has no network delay in it at all. A daily report that is recomputed to fix a bug will disagree with the version that was already circulated.

Event time avoids all of this, because the only input to the bucketing decision is a number carried by the record itself.

The lag between them, measured

The gap between event time and processing time is the single most useful number in a streaming system, and most teams have never plotted its distribution.

On a consumer mobile product measured over one week, the distribution looked like this. The median record arrived two seconds after the event happened. The ninety fifth percentile arrived at eleven seconds. The ninety ninth percentile arrived at forty eight seconds. The ninety nine point ninth percentile arrived at six minutes. The single worst record arrived four hours and twelve minutes late, from a phone that had been in a tunnel and then in airplane mode.

Three things in that shape matter more than the individual numbers.

The first is that the bulk of the distribution is very tight. Half the records are inside two seconds. If you only ever looked at the median you would conclude that lateness is not a problem in this system at all.

The second is that the tail is enormously longer than the body. The ninety nine point ninth percentile is a hundred and eighty times the median. This is not a normal distribution with a few outliers. It is a body plus a tail generated by a completely different mechanism, and the mechanism is devices that were offline.

The third is that the maximum is unbounded in practice. There is no number you can choose that captures everything, because somebody's phone will always be in airplane mode for longer than your bound. Every design decision downstream of this is a decision about where to stop waiting, not a decision about how to wait for everyone.

Where the timestamp comes from

Event time only works if the record actually has a trustworthy event timestamp, and this is where a lot of real systems fall down before they start.

If the timestamp is set by the client, it is as accurate as the client's clock, which on consumer devices means it is sometimes years wrong. Phones with a dead battery reset to the epoch. Devices in certain regions have systematically skewed clocks. A small percentage of your records will claim to have happened in nineteen seventy or in twenty thirty.

The usual defence is to carry both the client timestamp and the server receipt timestamp, and to reject or correct client timestamps that are impossibly far from the receipt. A record claiming to be from the future is always wrong. A record claiming to be from before the account existed is always wrong. Clamping these to the receipt time is crude but it is far better than letting one phone with a broken clock advance or hold back your entire watermark.

If the timestamp is set by the server, it is accurate and trustworthy but it is no longer the time the thing happened, it is the time the thing was reported. For a payment this difference is milliseconds and nobody cares. For a mobile app that batches events and uploads them when it next has signal, this difference is the entire phenomenon you were trying to measure.

The broker timestamp, which is neither

The broker's ingestion timestamp deserves a paragraph of its own, because it is the one people reach for when the client clock is untrustworthy and the server receipt is unavailable.

It has one genuinely attractive property. Within a single partition it is monotonically non decreasing, because the broker assigns it in append order. That makes it trivially easy to derive a watermark from, with no out of order handling at all, which is why so many getting started guides use it.

It also has one fatal property for analytical work. It measures when your infrastructure received the record, which means it moves whenever your infrastructure has a bad day. A collector that stalls for ten minutes and then flushes will stamp ten minutes of events with nearly the same ingestion time. The resulting hourly counts are a picture of your collector, not of your users.

Use it when the question really is about ingestion, when you are debugging the pipeline, or as a sanity bound on an untrusted client timestamp. Do not use it as a stand in for when something happened.

The rule worth remembering

Choose processing time when the question is about your pipeline, when approximate is fine, and when the result will never be recomputed. Choose event time when the question is about the world, when the answer must be stable, or when anybody downstream will ever ask why two runs disagree.

Most production pipelines end up using both, deliberately, for different outputs. The failure mode is not using both. The failure mode is using one of them by accident because it was the default.