Anomaly Detection Monitoring: How It Works and How to Run It
Anomaly Detection Monitoring: How It Works and How to Run It

Anomaly detection monitoring learns what normal looks like for each metric and alerts you when behavior drifts from it, so nobody has to hand-tune a threshold per server. It works best when the detector’s output lands in one place with the surrounding context, because an alert that fires without evidence only moves the work from detection to investigation.
TL;DR
- Problem: Static thresholds throw false alerts at some hours and miss real shifts at others, and tool sprawl scatters the evidence needed to judge each alert.
- Takeaway: Model expected behavior (trend plus hourly, daily and weekly seasonality), alert on deviation from that band, and measure the result with accuracy and detection delay.
- Takeaway: Detection is half the job. Correlating anomalies with topology, deploys and logs is what turns a flag into a diagnosis.
- Next step: Pick five noisy, high-value metrics, switch them to anomaly-based alerts, and track false positives and median detection delay for a month. Opsphere’s Community plan costs €0 per month if you want a unified investigation layer on top.
Table of Contents
- What anomaly detection monitoring is
- How anomaly detection works on infrastructure data
- Types of anomalies to monitor for
- Anomaly detection versus fixed thresholds and rules
- Configure anomaly monitoring to cut false positives
- Evaluate detection: accuracy and latency
- Why detection alone is not enough in complex cloud environments
- Where Opsphere fits
- A rollout plan you can run this quarter
- FAQ
- Sources
What anomaly detection monitoring is
Anomaly detection is the practice of finding observations that do not follow the pattern considered normal. In monitoring, that means applying it continuously to live operational data (metrics, logs, traces, events) and wiring the output into alerting and incident response.
The difference from threshold monitoring is how “normal” is defined. NetApp’s documentation explains that threshold-based monitoring fits cases where you have predefined limits and a clear idea of what is expected, while anomaly detection uses machine learning to find outliers when the definition of normal is unclear.
That covers most real production metrics. Request rate on a consumer API, queue depth in a batch pipeline and Lambda concurrency all move with the clock and the calendar. A single number such as “alert above 500 requests per second” is wrong most of the day: too loose at 3 a.m., too tight at noon on a launch day.
Two terms from the research literature are worth knowing:
- Outlier detection flags observations that differ from the rest of the training data.
- Novelty detection flags new data points that differ from what the model saw in training and are usually absent from it. (KDnuggets describes both directions.)
Most monitoring products blend the two: they learn history, then judge each new data point against it.
How anomaly detection works on infrastructure data
The common pattern is a baseline model, a tolerance band and an alert rule that fires when a value leaves the band.
Baselines with seasonality
Amazon CloudWatch is a concrete example. Its anomaly detection applies statistical and machine learning algorithms to a metric’s past data and builds a model of expected values, accounting for trends and hourly, daily and weekly patterns. The model trains on up to two weeks of metric data. That training window is a practical constraint: a brand-new service has little history, so its first bands are rough.
CloudWatch anomaly detection alarms have no static threshold. They compare the metric against the model’s expected value, and you choose whether the alarm triggers above the band, below the band or both. Choosing the direction matters more than it looks. Error counts only matter on the high side. Throughput matters on both: a sudden drop in orders is often a worse incident than a spike.
Standard deviations and sensitivity
NetApp’s approach is simpler to reason about. In its documentation, an anomaly occurs when a metric’s mean value sits a number of standard deviations away from the weighted mean of the previous few weeks, with recent weeks weighted more heavily. Sensitivity settings of High, Medium or Low control how many alerts you get.
The weighting is the useful idea. If a workload changes permanently after a migration, a recency-weighted baseline adapts in a few weeks instead of alerting forever.
Combining metrics and logs
Metrics alone miss failures that show up first in logs. The ICeTEA paper from KDD MILETS 2025 proposes an ensemble of three detectors: a metric-based one using a Peak Over Threshold model, a log-based one using frequency analysis plus PCA, and a multimodal VAE over both, with majority voting for the final decision. The design point generalizes: independent detectors that must agree produce fewer spurious alerts than any single detector.

Types of anomalies to monitor for
A systematic review of anomaly detection for cloud systems in the ACM Digital Library distinguishes three types of anomalies. Each maps to a familiar operational failure.
| Type | Definition | Cloud operations example |
|---|---|---|
| Point | A single instance is abnormal compared with the rest | One pod reporting a latency spike far above its peers |
| Collective | A group of related instances is abnormal together | A slow ramp of memory use across a node pool, each reading fine alone |
| Contextual | Abnormal only in a specific context such as time | 2,000 logins per minute at 4 a.m. on a Sunday, normal on Monday morning |
The same review notes that too many false positives make a reliable application impossible. That is the operational test for any detector: if on-call engineers learn to ignore it, it is not monitoring anymore.
Contextual anomalies are the reason seasonality-aware models exist. Collective anomalies are the reason you should look at windows and trends, not only individual points.
Anomaly detection versus fixed thresholds and rules
Neither approach replaces the other. Use each where it fits.
| Criteria | Static threshold | Anomaly-based |
|---|---|---|
| Definition of normal | You know it and can write it down | Learned from history |
| Best for | Hard limits: disk 95% full, certificate expiry, SLO burn limits | Metrics with daily or weekly cycles and drifting baselines |
| Maintenance | Revisit per server and per workload change | Model adapts as data changes |
| Typical failure | False alerts at peaks or quiet hours | Noise on young metrics with little history |
| Setup effort | Minutes | Needs a training window and tuning |
Preetam Jinka and Baron Schwartz, in their O’Reilly book on the topic, list typical reasons to add anomaly detection: avoiding thresholds per server, avoiding threshold edits whenever workloads shift, and avoiding static thresholds that false-alert at certain times of day or week. That is a good checklist for deciding which metrics to convert first.
A practical split: keep static rules for conditions with a known hard boundary, and use anomaly detection for business and traffic-shaped signals where the right value changes with the clock.
Configure anomaly monitoring to cut false positives
False positives usually come from configuration, not from the algorithm. Work through these in order.
- Start with metrics that have stable seasonality. Request rate, queue depth and CPU on stateful services give models clean patterns. Brand-new or bursty metrics produce wide, unstable bands.
- Choose the band direction. Alert on the side that hurts. For error rates, only above the band. For throughput, both.
- Set sensitivity deliberately. In a tool with High, Medium and Low settings, start at Low or Medium on pager-bound alerts and raise it for dashboards or ticket-only alerts.
- Compose alerts instead of firing on one metric. CloudWatch anomaly detection can be combined with metric math and composite alarms so an alert depends on several signals together. A latency anomaly that coincides with an error-rate anomaly is far more likely to be real than either alone.
- Route by severity. Page only when the anomaly touches an SLO or a customer path. Send the rest to a ticket queue or a channel.
- Review weekly. Tag every alert as actionable or not, and adjust the five noisiest rules. Without that loop, noise creeps back.
Noise reduction is also a correlation problem. Related anomalies from the same root cause should arrive as one incident, not twelve pages. Opsphere’s write-up on reducing false positive alerts covers the approach to keeping false-positive rates low.
Evaluate detection: accuracy and latency
Teams often buy or build detection and never measure it. Two measures cover most of what matters.
Accuracy
An IETF draft on anomaly detection evaluation (draft-anjum-nmop-anomaly-detection-evaluation-01, August 2026) recommends that AUC-ROC be the metric of record for ranking and acceptance decisions because it does not depend on a threshold and is insensitive to prevalence. That means values stay comparable between evaluation sets with different anomaly rates. The draft also says to report PR-AUC alongside it, and notes that an AUC-ROC near 0.5 means the detector performs no better than chance.
PR-AUC matters because real incidents are rare. When anomalies are a tiny fraction of data, precision tells you how many alerts people will waste time on.
Detection delay
Accuracy says nothing about speed. The same IETF draft defines detection latency as the elapsed time between anomaly onset and the first detection decision that flags it, and says to report it as a distribution, not only a mean.
A practical formula from an evaluation guide published in October 2026: detection delay equals first valid alert time minus anomaly start time. Report the median and an upper percentile, because a mean can hide a few extremely late detections.
To apply this, take your last 10 to 20 incidents and record when the problem actually began, using deploy markers and metric inflection points, and when the first valid alert fired. The gap is your baseline. Then compare after a rollout.
A simple scorecard for a one-month trial:
- Share of pages that were actionable
- Incidents with no anomaly alert at all (misses)
- Median and 90th percentile detection delay
- Time from alert to a confirmed cause
The last item is where most teams find the real bottleneck.
Why detection alone is not enough in complex cloud environments
Finding the anomaly is the easy part. Explaining it is slow. A 2026 arXiv paper on cognitive platform engineering describes how microservices, dynamic orchestration and multi-cloud complexity have pushed telemetry volume and configuration churn past what human monitoring and rule-based automation can handle, resulting in alert fatigue, fragmented tooling and reactive incident response.
In practice, one anomaly on checkout latency sends an engineer through a metrics dashboard, a Kubernetes console, a deploy history, a cloud audit trail and a log search, each with its own login and time picker. Most of the delay between alert and diagnosis is spent assembling that picture.
Three things shorten it:
- A shared view of topology. Knowing which services, nodes and dependencies sit around the anomalous resource narrows the search at once.
- Change context. Most incidents follow a change. Recent deploys, config edits and infrastructure events belong next to the anomaly. For AWS, Opsphere’s guide to checking CloudTrail before chasing logs shows this workflow.
- Memory. An anomaly you have seen before should come with what was found last time.
Opsphere is built for this layer. It connects to the tools you already run, across AWS, Kubernetes, observability, CI/CD and security, without requiring you to replace them or duplicate telemetry, and it keeps each tool as the source of truth. Its operational tools are read-only, so investigation does not change your infrastructure. The operational context and investigation memory page describes how it builds the shared view across sources.
Detection feeds capacity decisions too. An anomaly trend on load can inform scaling policy, and Opsphere’s piece on SLO-first predictive autoscaling shows how.
Where Opsphere fits
Opsphere is an AI-powered operational intelligence platform for DevOps, SRE and platform teams. It covers anomaly detection, alert correlation and noise reduction, causal inference, real-time topology mapping and automated runbook generation, and it lets teams query and investigate from the web client or from inside the IDE. It uses AI agents and Model Context Protocol with more than 300 read-only operational tools.
For a small SRE team, the appeal is operational clarity without adding another console to watch. The Opsphere platform brings evidence from your existing stack into a single interface, so an anomaly arrives with the context needed to judge it. Pricing is published: Community is €0 per month, Developer is €19 per month, and Team is €500 per month, with Enterprise and custom options on the pricing page.

Security-conscious teams can review how access and disclosure are handled on the security page.
A rollout plan you can run this quarter
A short plan keeps anomaly detection from becoming another noisy system.
Week 1: choose targets. List the 10 alerts that paged most often last quarter. Mark the ones driven by a time-of-day pattern. Those are your first conversions.
Week 2: switch and tune. Move five metrics to anomaly-based alerts with directional bands. Keep the old threshold alerts active but silent, so you can compare.
Week 3: correlate. Group alerts that share a service or dependency, and attach recent changes to each alert. Decide which signals page and which only open tickets.
Week 4: measure. Calculate actionable-page rate, misses, and median and 90th-percentile detection delay against your baseline. Keep what improved, revert what did not, and choose the next five metrics.
Repeat monthly. After a quarter, most teams have a tuned set of anomaly alerts for seasonal metrics and static rules for hard limits, which is the right split.
FAQ
What are the three types of anomaly detection?
The three types of anomalies are point, collective and contextual. A point anomaly is a single abnormal value, a collective anomaly is a group of related values that are abnormal together, and a contextual anomaly is abnormal only in a specific context such as time of day. Detection methods are also often grouped as statistical, machine learning and ensemble approaches.
What is an example of anomaly detection in monitoring?
A typical example is an alarm on request volume that learns the weekly traffic cycle. At 3 a.m. a rise that would be normal at noon triggers an alert, and a drop during peak hours triggers one too. CloudWatch anomaly detection alarms work this way, comparing the metric to a learned band instead of a fixed number.
How is anomaly detection different from threshold alerting?
Thresholds compare a metric to a number you chose. Anomaly detection compares it to a model of expected behavior built from history, including trend and seasonality. Thresholds suit hard limits such as a full disk. Anomaly detection suits metrics whose normal value changes through the day or week.
How do you reduce false positives from anomaly detection?
Start with metrics that have stable patterns, alert only in the direction that matters, and lower sensitivity on pager-bound alerts. Combine signals with composite alarms, so an alert needs corroboration, and review which alerts were actionable each week. Correlating related anomalies into one incident also removes duplicate pages.
How do you measure whether anomaly detection works?
Track accuracy with AUC-ROC and PR-AUC, and track detection delay as the time from anomaly start to the first valid alert. Report the median and an upper percentile for delay, since an average can hide a few very late detections. Add the actionable-page rate from your on-call history.
Sources
- Anomaly Detection Monitors, NetApp Data Infrastructure Insights
- Using CloudWatch anomaly detection, Amazon CloudWatch
- Operationalizing CloudWatch Anomaly Detection, AWS Management and Governance Blog
- How to use Machine Learning for Anomaly Detection and Condition Monitoring, KDnuggets
- ICeTEA: Mixture of Detectors for Metric-Log Anomaly Detection, KDD MILETS 2025
- A Systematic Review on Anomaly Detection for Cloud Computing Environments, ACM
- Practical Anomaly Detection for Monitoring, O’Reilly
- draft-anjum-nmop-anomaly-detection-evaluation-01, IETF
- Which Metrics Should Be Used to Evaluate Anomaly Detection Models in 2026, ONES
- Cognitive Platform Engineering for Autonomous Cloud Operations, arXiv
Recommended
- Prometheus Monitoring: How It Works and Where Teams Hit Limits
- Datadog Explained: What It Does and How to Investigate Incidents With It
- eBPF Monitoring: How It Works and Where It Fits in 2026
This article is provided for general informational purposes only and does not constitute professional, legal, security, or compliance advice. Please evaluate recommendations against your organization’s specific environment and requirements.
