Detection Rules and Alert Fatigue

Topics Covered

Detection Engineering Foundations

What detection engineering is

Why detection matters

The parts of a detection

The detection lifecycle

True and false positives

False negatives

Precision and recall

The base rate problem

Alert fatigue

Why noisy detections persist

Detection is a team sport

Measuring detection programmes

Detection versus prevention

The cost of missed detection

Shared vocabulary

Learning from incidents

Detection debt

Where detection ideas come from

Prevention eventually fails. Credentials get phished, vulnerabilities get exploited and insiders misuse access. Detection is how an organisation notices when that happens, ideally before serious damage. This section introduces detection engineering, the life of a detection and the alert fatigue problem that undermines so many security operations teams.

What detection engineering is

Detection engineering is the discipline of designing, building, testing and maintaining the logic that turns security data into alerts. A detection might be a simple rule, such as an alert when a cloud audit trail is stopped, or a statistical model that flags unusual data downloads. Detection engineers treat these as software: written deliberately, tested, reviewed, versioned and improved based on how they perform in production.

Why detection matters

Attackers who get past preventive controls usually need time to achieve their goals: exploring the environment, escalating privileges and finding valuable data. Every hour of undetected activity increases the damage. Good detection shortens that window. Organisations that detect intrusions in hours rather than months typically face far smaller incidents, lower recovery costs and fewer notification obligations.

The parts of a detection

A complete detection has more than logic. It has a clear purpose describing the threat it targets, the data sources it depends on, the query or model itself, a severity and confidence level, a runbook telling responders how to triage it, an owner, tests that prove it works and metadata linking it to a threat framework. Rules without these parts tend to decay, confuse responders and get disabled.

The detection lifecycle

Detections follow a lifecycle. An idea comes from threat intelligence, an incident, a red team finding or a hunt. The engineer confirms that the required data exists. They write the logic, test it against sample and historical data and simulate the behaviour. The detection runs silently in a staging mode before alerting anyone. After tuning, it goes live with a runbook. It is reviewed periodically and retired when it no longer adds value.

An idea is checked for data, written, tested by replay and simulation, run silently for two weeks, tuned, launched with a runbook and owner and reviewed every quarter to keep, improve or retire it.

True and false positives

When a detection fires, the alert is a true positive if the behaviour it targets really happened and a false positive if it did not. Many alerts are a third kind: benign true positives, where the behaviour happened but was legitimate, such as an administrator who really did sign in from a new country while travelling. Distinguishing these matters, because the fix for each is different.

False negatives

A false negative is an attack that happened without triggering any alert. False negatives are invisible by definition, which makes them dangerous. They are revealed by incidents discovered through other means, by red team exercises and by threat hunting. Measuring detection effectiveness requires deliberately testing for false negatives, not only counting the alerts that fire.

Precision and recall

Precision is the share of alerts that are worth acting on. Recall is the share of real attacks that produce an alert. Tightening a rule usually raises precision and lowers recall; loosening it does the opposite. Detection engineering is largely about finding logic that achieves both, typically by using better signals and context rather than simply moving thresholds.

The base rate problem

Attacks are rare compared with normal activity. Even a rule that is right ninety nine percent of the time on each event will produce mostly false positives if it evaluates millions of benign events a day. This base rate problem explains why naive detections overwhelm teams. Good detections focus on behaviours that are rare in legitimate use and combine several signals to raise confidence.

Alert fatigue

Alert fatigue is what happens when analysts face more alerts than they can investigate thoughtfully. They begin to skim, close alerts based on patterns rather than evidence and miss the rare real attack hidden in the noise. Many well known breaches had alerts that fired but were dismissed or ignored. Alert fatigue is not a personal failing; it is a predictable result of poor detection design.

Of 4,800 alerts, 1,700 were duplicates, 2,100 were expected activity, 850 were false positives, 104 were inconclusive and only 46 were real attacks or policy violations.

Why noisy detections persist

Noisy detections persist for understandable reasons. Vendors ship broad default rules to avoid missing anything. Compliance checklists demand alerts for categories of activity without regard to quality. Engineers add rules after incidents without tuning them. Nobody owns the job of removing rules. Over time, the alert queue fills with low value noise that everyone has learned to ignore.

Detection is a team sport

Detection engineers depend on others. Platform and application teams provide logs. Responders triage alerts and feed back what worked. Threat intelligence analysts suggest what to look for. Red teams test whether detections fire. Application owners explain what normal looks like. Strong relationships across these groups produce better detections than any individual effort.

Measuring detection programmes

Useful measures include the time from attack to detection, the precision of each detection, coverage of important attack techniques, the number of alerts each analyst handles per shift and the proportion of detections that have tests and runbooks. These measures show whether the programme is catching real threats efficiently or merely generating activity.

Detection versus prevention

Some behaviours are better prevented than detected. If a policy can block public storage buckets outright, alerting on them is a weaker fallback. Detection is most valuable for actions that must remain possible for legitimate users but can be abused, such as administrative access, data exports and credential use. Choosing the right control for each risk keeps detection focused where it adds most value.

The cost of missed detection

The cost of missing an attack often grows over time. An attacker caught at initial access may have touched one account; an attacker caught after months may have copied large datasets and planted persistence across systems. This asymmetry justifies investment in detection, but only in detections that actually work. A hundred ignored rules are worth less than ten that responders trust.

Shared vocabulary

Teams benefit from shared vocabulary for detections: severity levels with clear meanings, confidence levels, standard alert fields and a common framework for attack techniques. Shared vocabulary helps responders prioritise, lets detection engineers compare rules and makes reporting to leadership consistent across teams and tools.

Learning from incidents

Every incident is a detection lesson. Did a detection fire, and how quickly? If not, what data and logic would have caught it? If it fired but was dismissed, why? Incident reviews that ask these questions produce new detections and improve existing ones based on real attacks rather than theory.

Detection debt

Like technical debt, detection debt accumulates when rules are added quickly without tests, runbooks or owners. Each shortcut seems small, but together they create a rule set that nobody fully understands. Paying down detection debt means adding missing tests, documenting purpose, assigning owners and retiring rules that cannot be justified. Teams that schedule this work regularly avoid the slow decline into a queue that everyone ignores.

Where detection ideas come from

Good detection ideas come from many sources: incident reviews, threat intelligence reports, red team findings, hunting results, vendor research, changes to the environment and questions from auditors. Keeping a backlog of ideas with their source and priority makes it easier to choose what to build next. Ideas that cannot be built because of missing data become requests to logging and platform teams.

Detection is engineered, not collected

A detection is logic plus purpose, data, tests, runbook and owner. It follows a lifecycle from idea to retirement. Because attacks are rare, naive rules produce mostly noise, and noise causes alert fatigue that hides real attacks. Measure precision, recall and time to detect, and favour fewer trusted detections over many ignored ones.