0%
Security Engineering for Software Teams
Thinking About Security
Cryptography You Will Actually Use
Identity
Vulnerability Classes
Infrastructure and Cloud
Detection Rules and Alert Fatigue
Prevention eventually fails. Credentials get phished, vulnerabilities get exploited and insiders misuse access. Detection is how an organisation notices when that happens, ideally before serious damage. This section introduces detection engineering, the life of a detection and the alert fatigue problem that undermines so many security operations teams.
What detection engineering is
Detection engineering is the discipline of designing, building, testing and maintaining the logic that turns security data into alerts. A detection might be a simple rule, such as an alert when a cloud audit trail is stopped, or a statistical model that flags unusual data downloads. Detection engineers treat these as software: written deliberately, tested, reviewed, versioned and improved based on how they perform in production.
Why detection matters
Attackers who get past preventive controls usually need time to achieve their goals: exploring the environment, escalating privileges and finding valuable data. Every hour of undetected activity increases the damage. Good detection shortens that window. Organisations that detect intrusions in hours rather than months typically face far smaller incidents, lower recovery costs and fewer notification obligations.
The parts of a detection
A complete detection has more than logic. It has a clear purpose describing the threat it targets, the data sources it depends on, the query or model itself, a severity and confidence level, a runbook telling responders how to triage it, an owner, tests that prove it works and metadata linking it to a threat framework. Rules without these parts tend to decay, confuse responders and get disabled.
The detection lifecycle
Detections follow a lifecycle. An idea comes from threat intelligence, an incident, a red team finding or a hunt. The engineer confirms that the required data exists. They write the logic, test it against sample and historical data and simulate the behaviour. The detection runs silently in a staging mode before alerting anyone. After tuning, it goes live with a runbook. It is reviewed periodically and retired when it no longer adds value.
True and false positives
When a detection fires, the alert is a true positive if the behaviour it targets really happened and a false positive if it did not. Many alerts are a third kind: benign true positives, where the behaviour happened but was legitimate, such as an administrator who really did sign in from a new country while travelling. Distinguishing these matters, because the fix for each is different.
False negatives
A false negative is an attack that happened without triggering any alert. False negatives are invisible by definition, which makes them dangerous. They are revealed by incidents discovered through other means, by red team exercises and by threat hunting. Measuring detection effectiveness requires deliberately testing for false negatives, not only counting the alerts that fire.
Precision and recall
Precision is the share of alerts that are worth acting on. Recall is the share of real attacks that produce an alert. Tightening a rule usually raises precision and lowers recall; loosening it does the opposite. Detection engineering is largely about finding logic that achieves both, typically by using better signals and context rather than simply moving thresholds.
The base rate problem
Attacks are rare compared with normal activity. Even a rule that is right ninety nine percent of the time on each event will produce mostly false positives if it evaluates millions of benign events a day. This base rate problem explains why naive detections overwhelm teams. Good detections focus on behaviours that are rare in legitimate use and combine several signals to raise confidence.
Alert fatigue
Alert fatigue is what happens when analysts face more alerts than they can investigate thoughtfully. They begin to skim, close alerts based on patterns rather than evidence and miss the rare real attack hidden in the noise. Many well known breaches had alerts that fired but were dismissed or ignored. Alert fatigue is not a personal failing; it is a predictable result of poor detection design.
Why noisy detections persist
Noisy detections persist for understandable reasons. Vendors ship broad default rules to avoid missing anything. Compliance checklists demand alerts for categories of activity without regard to quality. Engineers add rules after incidents without tuning them. Nobody owns the job of removing rules. Over time, the alert queue fills with low value noise that everyone has learned to ignore.
Detection is a team sport
Detection engineers depend on others. Platform and application teams provide logs. Responders triage alerts and feed back what worked. Threat intelligence analysts suggest what to look for. Red teams test whether detections fire. Application owners explain what normal looks like. Strong relationships across these groups produce better detections than any individual effort.
Measuring detection programmes
Useful measures include the time from attack to detection, the precision of each detection, coverage of important attack techniques, the number of alerts each analyst handles per shift and the proportion of detections that have tests and runbooks. These measures show whether the programme is catching real threats efficiently or merely generating activity.
Detection versus prevention
Some behaviours are better prevented than detected. If a policy can block public storage buckets outright, alerting on them is a weaker fallback. Detection is most valuable for actions that must remain possible for legitimate users but can be abused, such as administrative access, data exports and credential use. Choosing the right control for each risk keeps detection focused where it adds most value.
The cost of missed detection
The cost of missing an attack often grows over time. An attacker caught at initial access may have touched one account; an attacker caught after months may have copied large datasets and planted persistence across systems. This asymmetry justifies investment in detection, but only in detections that actually work. A hundred ignored rules are worth less than ten that responders trust.
Shared vocabulary
Teams benefit from shared vocabulary for detections: severity levels with clear meanings, confidence levels, standard alert fields and a common framework for attack techniques. Shared vocabulary helps responders prioritise, lets detection engineers compare rules and makes reporting to leadership consistent across teams and tools.
Learning from incidents
Every incident is a detection lesson. Did a detection fire, and how quickly? If not, what data and logic would have caught it? If it fired but was dismissed, why? Incident reviews that ask these questions produce new detections and improve existing ones based on real attacks rather than theory.
Detection debt
Like technical debt, detection debt accumulates when rules are added quickly without tests, runbooks or owners. Each shortcut seems small, but together they create a rule set that nobody fully understands. Paying down detection debt means adding missing tests, documenting purpose, assigning owners and retiring rules that cannot be justified. Teams that schedule this work regularly avoid the slow decline into a queue that everyone ignores.
Where detection ideas come from
Good detection ideas come from many sources: incident reviews, threat intelligence reports, red team findings, hunting results, vendor research, changes to the environment and questions from auditors. Keeping a backlog of ideas with their source and priority makes it easier to choose what to build next. Ideas that cannot be built because of missing data become requests to logging and platform teams.
A detection is logic plus purpose, data, tests, runbook and owner. It follows a lifecycle from idea to retirement. Because attacks are rare, naive rules produce mostly noise, and noise causes alert fatigue that hides real attacks. Measure precision, recall and time to detect, and favour fewer trusted detections over many ignored ones.