A log line is the only signal that can carry an arbitrary fact. A metric can tell you that error rate rose to four percent, and a trace can tell you which service was slow, but only a log can tell you that order 88213 was rejected because the currency code was three characters of whitespace. That specificity is the entire value of logging, and it is also why logs are the most expensive signal per unit of insight: you are paying to store facts you have not yet thought of a question for.

Everything in this lesson follows from one decision, which is whether a log line is a sentence written for a human or a record written for a machine. Both are legitimate, but a system cannot be built for both at once, and most of the operational pain in logging comes from a pipeline that assumes one while the applications produce the other.

The unstructured line, and what it costs later

An unstructured line is a sentence. It looks like this.

2026-11-06 14:22:09 ERROR payment rejected for order 88213: invalid currency code

A human reading one of these understands it immediately, which is why they are so easy to write and so popular. The cost arrives at query time. To answer "how many payments were rejected per minute, broken down by reason" somebody has to write a regular expression that pulls the order id and the reason out of the sentence, and that expression has to be applied to every line that might match. It works until the sentence changes. A developer rewords the message, adds a field in the middle, or wraps a value in quotes, and the dashboard that depended on the old wording silently stops counting without anything failing. Regular expression parsing of free text is not a reliability problem, it is a maintenance problem, and it compounds with every team that ships a new message.

The structured line, and the discipline it needs

A structured line is a record with named, typed fields, nearly always serialised as JSON.

{"ts":"2026-11-06T14:22:09.441Z","level":"error","service":"payments","event":"payment_rejected","order_id":"88213","reason":"invalid_currency","currency_raw":"   ","trace_id":"7b9c1f2e4a8d3c5b","duration_ms":41}

Nothing here needs parsing. A query for event="payment_rejected" grouped by reason is exact, survives rewording, and costs the index almost nothing because both are real fields. The discipline this needs is that the field names must be stable and shared. If one service writes trace_id and another writes traceId and a third writes trace, every cross service query has to know all three, and you have reinvented the parsing problem one level up.

This is why the field set is worth agreeing on before the volume arrives. A workable minimum is a timestamp in a single format with a timezone, a level, the service name, the environment, a stable event name, the trace and span identifiers if a trace exists, and then whatever typed context the event needs. The event name is the field teams most often skip and most often wish they had, because it is the only field that lets you ask "how often does this specific thing happen" without touching the message text.

Levels, and the fact that nobody agrees on them

Log levels exist to let you throw data away cheaply, and they only work if they mean the same thing everywhere. The definitions that survive contact with an on call rotation are narrow. Error means a request failed and a human may need to act. Warning means something was degraded but handled, such as a retry that succeeded. Info means a notable state change, such as a service starting or a configuration reload. Debug means detail that is useful while reproducing a problem and worthless otherwise.

The failure mode is level inflation. A developer logs an error for a condition the code handles perfectly well, such as a cache miss or a client sending a malformed request, and now the error rate derived from logs is dominated by things nobody will ever act on. Within a quarter the team stops trusting the error count, and a real error hides inside a hundred thousand harmless ones. The discipline is simple to state and hard to hold: if nobody would act on it, it is not an error.

What must never be in a log line

A log is a copy of your data in a system with different access controls, longer retention and more readers than the database it came from. That makes it the easiest place in an architecture to leak something. Full request bodies, authorization headers, session tokens, card numbers, government identifiers, email addresses and passwords all end up in logs by accident, usually through a well meaning catch block that logs the whole request object for context.

The countermeasures that actually work are structural rather than procedural. Redact at the point of serialisation, by giving the logging library a list of field names it must never emit and a type for sensitive values that formats itself as a placeholder. Review the log output of a new endpoint the way you would review its response body. Treat a leak as a data incident with the same retention and rotation consequences, because once a token is in a log index with ninety day retention it is in backups too.

Volume is a design choice, not a fact of nature

It is worth internalising the arithmetic early. A service handling a thousand requests a second that writes one line per request at four hundred bytes produces four hundred kilobytes a second, which is thirty four gigabytes a day, from one service. Add a line for each outbound call and it triples. Multiply by a hundred services and you have a petabyte scale problem created entirely by a default.

The lever is sampling, and logs sample better than people expect. Keep every line for failed requests, keep one in a hundred for successful ones, and record the sampling rate in the line itself so that counts can be corrected. For a healthy service the successful requests are nearly identical to each other and the hundredth one teaches you nothing the first did not. This is the same reasoning as trace sampling, applied to a signal where people rarely think to apply it.

Where the line goes the moment it is written

The last part of the contract is the destination, and the answer that has held up best is that an application writes to standard output and nothing else. It does not open a file, it does not rotate anything, it does not know the address of the log index, and it does not retry a network call to ship a line. The moment an application takes responsibility for delivery, a slow log backend can block a request thread, and a logging outage becomes a product outage.

Writing to a file inside a container is the common mistake. The file grows until the node's disk fills, rotation has to be configured in two places, and the collector now needs a volume mount and a way to discover paths that change whenever somebody edits a deployment. Standard output costs the application a write to a pipe, and everything after that pipe is somebody else's problem, which is exactly the separation that lets the pipeline be upgraded, buffered and rate limited without a code change in a hundred services.

There is one exception worth knowing. Audit logs, the ones that exist to satisfy a compliance obligation rather than to help you debug, often do need a guaranteed delivery path, and treating them as a different product with a different pipeline is cleaner than making the ordinary logging path durable enough for both.