0%
Security Engineering for Software Teams
Thinking About Security
Cryptography You Will Actually Use
Identity
Vulnerability Classes
Infrastructure and Cloud
Logging for Security
Logs are the memory of a system. When something goes wrong, they are often the only way to learn what happened, who did it and how far it spread. Most systems log plenty for debugging but very little that helps with security. This section explains what security logging is for, the questions it must answer and the common gaps that leave investigations blind.
Why security logging is different
Operational logs help engineers fix bugs and performance problems. They focus on errors, latency and internal state. Security logs answer different questions: who accessed what, from where, with which credentials and whether it succeeded. An application can produce gigabytes of debug output a day and still be unable to say which user downloaded a sensitive export last Tuesday. Security logging must be designed deliberately rather than assumed to fall out of operational logging.
The questions an investigation asks
When a responder investigates a suspected incident, they ask a familiar set of questions. Who performed the action, and was it a person, a service or a stolen credential? What exactly did they do? When did it happen, precisely enough to order events across systems? Where did it come from, in terms of address, device and location? What resource was affected? Did it succeed or fail? Good security logs answer each of these for every important action.
Detection depends on logs
Detection rules, anomaly models and threat hunting all consume logs. A detection cannot fire on an event that was never recorded. Before writing detections for credential misuse, data theft or privilege escalation, a team must confirm that the relevant events are logged, collected and searchable. Many detection programmes stall because the underlying data is missing, incomplete or inconsistent across systems.
Investigation depends on logs
After an alert, responders need context: what else did this user or host do before and after? Did the attacker touch other systems? Which data was accessed? Without logs, responders must assume the worst, which often means broader notifications, longer outages and more expensive recovery. With good logs, they can scope an incident precisely and show evidence that other systems were not affected.
Accountability and deterrence
Logs also support accountability. Recording administrative actions, access to sensitive data and changes to permissions lets organisations review who did what and why. Knowing that actions are logged and reviewed discourages misuse by insiders. Access reviews, change audits and dispute resolution all depend on trustworthy records of past actions.
Compliance obligations
Many regulations and standards require security logging: recording access to personal data, retaining audit trails for a defined period, protecting logs from tampering and reviewing them regularly. While compliance should not be the only driver, it often provides minimum requirements and funding. Designing logging well satisfies these requirements as a side effect of being useful.
Common gaps found in reviews
Post incident reviews reveal recurring logging gaps. Authentication events are missing from internal applications. Logs are kept for only a week, while attackers dwell for months. Requests cannot be correlated across services because no request identifier is propagated. Clocks are out of sync, making ordering impossible. Logs stored only on hosts disappear when instances are replaced or deliberately wiped.
Dwell time and retention
Attackers often remain undetected for weeks or months. If logs are retained for only a few days, the beginning of an intrusion is gone by the time it is discovered. Retention should be based on realistic detection times, which means months for key security sources, with older data moved to cheaper storage rather than deleted. Short retention is one of the most common reasons investigations cannot determine how an attacker first got in.
Logs as evidence
Logs may be used as evidence in legal proceedings, regulatory investigations or disputes with customers. Their value depends on being complete, accurate and protected from tampering. Synchronised clocks, integrity controls and documented collection processes make logs credible. Logs that could have been edited by the same administrators under investigation are far less convincing.
The cost of logging
Logging is not free. Collection, storage, indexing and search cost money, and verbose logs can affect application performance. Security teams that ask for everything quickly hit budget limits. The goal is not maximum volume but the right events, with the right fields, kept for the right time. Thoughtful selection often produces better security outcomes at lower cost than collecting everything.
Who owns security logging
Security logging spans application, platform and security teams. Application teams emit events from their code. Platform teams run collection pipelines and storage. Security teams define requirements, consume logs for detection and investigation and check coverage. Clear ownership of each part avoids the situation where everyone assumes someone else is logging the important events.
Start from scenarios
A practical way to decide what to log is to start from scenarios. For each likely threat, such as account takeover, data exfiltration or insider misuse, ask what evidence would show it happened and what an investigator would need to scope it. The answers produce a concrete list of events and fields. This scenario driven approach avoids both logging everything and logging only what is easy.
Logging is a product
Treating security logging as an internal product, with users, requirements, quality measures and a roadmap, helps it improve over time. Responders and detection engineers are the users. Coverage, latency, completeness and search speed are quality measures. Feedback from each incident becomes new requirements. This mindset turns logging from an afterthought into a dependable capability.
Testing logging before you need it
Logging gaps are usually discovered during an incident, which is the worst time. Exercises that simulate an attack and then ask responders to reconstruct it from logs reveal gaps safely. Each missing event or broken correlation found in an exercise becomes a fix. Teams that run these exercises regularly find that real investigations go much faster.
Balancing privacy
Security logs record what people do, which raises privacy concerns. Logging should collect what is needed for security and no more, protect logs with strict access controls and avoid recording sensitive content such as message bodies or passwords. Being transparent with employees and customers about security logging, and limiting its use to legitimate purposes, maintains trust.
Logs from third parties
Some of the most important evidence lives in systems the organisation does not run: the identity provider, the code hosting platform, the cloud provider and payment processors. Their logs are available only if they are enabled, exported and retained. Contracts and configuration should ensure that these providers deliver audit logs in a usable form, because an investigation that stops at the edge of a vendor system leaves the most important questions unanswered.
Silence is not innocence
An absence of evidence in logs proves nothing unless the relevant events would have been recorded. Responders who see no suspicious activity must ask whether the source was enabled, collected and retained for the period in question. Stating clearly what the logs cover, and what they do not, prevents false confidence and helps leaders understand the real certainty of an investigation conclusion.
Security logging exists to support detection, investigation, accountability and compliance. Design it from attack scenarios so that every important action records actor, action, time, source, target and outcome, retained long enough to cover realistic dwell times and protected well enough to serve as evidence.