What is anomaly detection?

Anomaly detection is the use of statistical modeling and AI to automatically identify data points that deviate from an expected baseline. A model learns what normal looks like for a signal and flags the values that fall outside it. Each flag is a prompt to investigate, not a diagnosis.

Anomaly detection works in three stages: The model learns a baseline from historical records, compares each new data point against that baseline, and flags the ones that do not fit. Take one example: the daily order volume for a single store.

The model reads several weeks of records, learns the weekly pattern including the Saturday peak and the Tuesday trough, and produces an expected value with an upper bound and a lower bound for each new day. A value inside the bounds passes without comment. A value outside is scored by how far outside it sits, and the highest-scoring deviations are surfaced for review. Statistical techniques and machine learning algorithms both produce this comparison.

Since the baseline reflects a learned dynamic pattern instead of a rigid, static rule, normal fluctuations (like an expected Saturday traffic peak) will not trigger false alerts. Nevertheless, teams should always verify flags prior to taking action, as an alert may stem from various causes: an actual system failure, sensor or measurement errors, an interrupted data feed, or a positive event like a viral product surge.

Anomalies are categorized into five main types. Because these categories overlap, a single event may exhibit characteristics of multiple types simultaneously. Identifying the specific anomaly type is critical, as it dictates the appropriate detection methodology.

1. Point anomalies are individual data points that are notably different from the rest of the dataset. A sudden large credit card purchase that is out of the norm for a particular cardholder is a point anomaly, flagged to investigate as potential fraud. A simple distance or threshold rule catches this type.
2. Contextual anomalies are values whose normal behavior varies depending on context. A retail site sees a major increase in traffic and sales on Black Friday, the busiest shopping day of the year, and the same spike in March would be anomalous. A contextual anomaly is invisible unless the context feature, such as day of week, operating mode, or season, is modeled explicitly.
3. Collective anomalies involve a group of data points that are anomalous together even though each individual point looks normal. A DDoS attack is the standard example, because it creates traffic from many sources that only looks wrong in aggregate. A collective anomaly lives in the relationship between points, so a per-point detector misses it.
4. Temporal anomalies, also called time series anomalies, are deviations in a temporal sequence such as a sequence of events or a seasonal change. A shift in peak tourist season for a vacation destination, unusual weather during a particular season, or a traffic surge outside rush hour are all temporal anomalies. Detecting them requires a model that separates trend and seasonality first.
5. Spatial anomalies, also called geographic anomalies, occur in spatial or geographic data and are found by examining the spatial relationships between points. In public health data, an unusually high concentration of people diagnosed with one disease in a specific area is a spatial anomaly, investigated as a localized outbreak.

Anomaly detection techniques divide into three machine learning families, and the feature that separates them is how much labeled data each one needs to identify an anomaly. Labeling is the scarce resource, so that property usually decides the choice.

Supervised anomaly detection trains on data that labels both normal and anomalous instances. The model learns the patterns and classifies each new record as one or the other.

Unsupervised anomaly detection works without labels. It assumes anomalies are rare and different from the bulk of the dataset, which lets it scale to complex data nobody has annotated. Unsupervised methods dominate real deployments because genuine anomalies are rare, so labeled examples are few, expensive to produce, and biased toward the failures you have already seen. This class imbalance is the core difficulty of the whole field. The catch is that an unsupervised model tells you an observation is unusual and cannot tell you whether it is bad. Isolation forests, which isolate a point by how few random splits separate it from the rest, and clustering methods are common choices.

Semi-supervised anomaly detection uses a labeled set of normal behavior to establish a baseline and then finds deviations in unlabeled data. It suits a system with clean history but no catalogued failures.

Deep learning leverages techniques like autoencoders, which learn to reconstruct normal data patterns and flag significant reconstruction errors as anomalies. This strategy is particularly effective for high-dimensional inputs, including raw sensor feeds and complex image data. Elastic details uses a combination of approaches in its anomaly detection algorithms.

Machine learning is not the only way to implement anomaly detection. Statistical methods, rule-based approaches, and signal processing techniques are additional ways to detect anomalies.

Traditional detection mechanisms — such as signatures, hand-crafted rules, and static thresholds — depend on human engineers to manually define expected patterns ahead of time. In contrast, anomaly detection dynamically infers normal behavior directly from telemetry data.

Because fixed thresholds lack awareness of periodic cycles or seasonality, a CPU utilization alert configured for peak weekday traffic will trigger redundant alarms during scheduled Monday morning batch workloads, yet remain completely inactive during a gradual weekend failure that stays below the static limit. Similarly, signature-based tools screen for recorded threat profiles and inevitably miss uncatalogued attacks, whereas baseline-based models identify unfamiliar anomalies without requiring predefined definitions.

Nevertheless, manual rules offer deterministic logic, immediate clarity, and zero dependency on historical training sets. For strict boundaries — such as exhausted storage capacity, statutory payment caps, or impending security certificate expirations — fixed thresholds remain the optimal approach, avoiding unnecessary modeling complexity. Consequently, operational environments typically adopt a hybrid strategy: Static thresholds guard hard operational limits, while machine learning baselines monitor variables with fluctuating norms.

Modern systems generate an overwhelming volume of signals — far beyond human capacity to monitor manually. With a single service emitting thousands of distinct metrics, defining and maintaining individual static rules for each variable is practically impossible.

Beyond sheer scale, signal interactions present a unique challenge: Operational metrics like latency, queue depth, and error rates might each remain within normal bounds individually, yet their combined behavior could indicate an unprecedented anomaly. While human analysts cannot simultaneously process multiple correlated distributions, multivariate models handle these multivariable dynamics effortlessly.

Failing to detect these subtle shifts carries a steep cost, often revealing issues that were observable in the data long before they caused disruption. Service degradations can go unnoticed in latency histograms for hours prior to customer reports, fraudulent activity can slip through when individual transactions remain below threshold limits, or drifting sensors can compromise production runs unmonitored. Early automated detection provides the critical window needed to intervene, whether that means freezing an account prior to additional charges or recalibrating equipment before output is ruined.

Whenever a system establishes a recognizable pattern of normal operation, anomaly detection becomes essential for identifying critical events, such as unauthorized financial activity, impending equipment failures, or manufacturing defects.

Financial fraud detection: Models continuous transactional behavior across individual accounts and broader customer segments. By detecting unfamiliar devices, unexpected geolocation points, or unusual purchase patterns, systems surface card fraud and identity theft before chargebacks occur.

Predictive maintenance and equipment monitoring: Tracks hardware telemetry, including voltage levels, fan speeds, and CPU thermal metrics, to spot early indicators of component degradation. Identifying these subtle shifts enables proactive repairs before unmonitored failures cause operational downtime.

User behavior analytics (UBA): Evaluates user interaction flows across web platforms and applications. Flagging irregular navigation sequences based on actual usage patterns helps security teams block abusive behavior and adjust defensive postures.

Medical signal analysis: Continuously checks physiological telemetry like respiration trends and heart rate variability. Deviations in these vital signals can be flagged to automatically alert medical professionals to urgent patient conditions.

Automated quality inspection: Analyzes high-resolution images of completed goods alongside assembly line sensor outputs. Detecting defects during production prevents flawed inventory from reaching end customers.

AI-based anomaly detection offers six benefits over rules-based detection: scale across complex data, early detection, continuous and adaptive monitoring, threat and fraud discovery, efficiency insight, and compliance support. All six follow from one property, which is that a model learns normal from data, not by waiting for a person to write a rule.

1. Scale across complex data: A learned model baselines thousands of correlated signals at once, with the precision to spot a combination no human would think to write a rule for.
2. Early detection: Problems get caught while they are still small. A memory leak flagged on day one is a restart; the same leak found on day four is an outage postmortem.
3. Continuous and adaptive monitoring: Detection runs around the clock in real time, and a model that keeps learning from live data absorbs legitimate shifts in behavior. A seasonal signup surge is an anomaly worth seeing in the morning, not a page at 2:00 a.m., and a static threshold cannot tell if its normal.
4. Threat and fraud discovery: Behavioral deviation surfaces malicious activity that broke no explicit rule, including account takeover, insider misuse, and intrusion attempts that carry no known signature.
5. Efficiency insight: Deviations from optimal performance expose waste, like a query that slowed by 40% after a release, a node doing a third of the work of its peers, or a batch job whose runtime has crept past its window.
6. Compliance support: Deviations from regulatory requirements and industry standards are themselves anomalies, such as an access pattern outside an approved window or a data transfer to an unapproved region.

Anomaly detection's practical challenges share a root: The model detects what is unusual and knows nothing about what is wrong.

1. Models go stale, or need tuning, and new signals have nothing to learn from. A model trained on out-of-date history degrades as the system moves away from it. Supervised models need labeled data. A brand new metric, service, or customer has no history, so there is nothing to baseline against.
2. Legitimate events can look exactly like anomalies. Holidays, marketing campaigns, and product launches all produce genuine, explainable deviations, and a detector that has not been told about them reports each one every time.
3. Noise and scale cost money. Algorithms are sensitive to poor data quality, so preprocessing is part of the job, and real-time streams take computational resources a team may not have budgeted.

Implementing anomaly detection is a five-step sequence, and it starts before any model is chosen.

1. Define the objective. Name the specific signal you are watching and the decision a flag will drive. A detector with no downstream action produces noise and nothing else.
2. Prepare the data. Establish that the signal is complete, consistently sampled, and long enough to cover its cycles. Gaps and irregular sampling look like anomalies to a model that was not told otherwise.
3. Select and train a model. Choose a prebuilt configuration or a custom one, guided by how many labels you have, how many correlated signals define normal, and how fast that normal changes.
4. Deploy against live data and set a threshold. Start conservatively and tighten as you learn what the score distribution looks like in production.
5. Evaluate continuously. Accuracy is a misleading metric when anomalies are rare, because a detector that flags nothing may be correct 99.9% of the time but useless. Watch precision, the share of flags that were real; recall, the share of real anomalies that were caught; and the F1 score, which combines the two. With no labeled history, the first months of review are how you build that ground truth, since every flag an analyst confirms or dismisses becomes an evaluation example.

For the actual procedure in Elastic, see create an anomaly detection rule.

Five practices separate a detector that earns its alerts from one a team learns to ignore.

1. Understand your data before you model it. Know the patterns, the sampling interval, and the known gaps. Surprises found after deployment cost more.
2. Train on a period long enough to contain the cycles you care about. Several weeks for a weekly rhythm, longer for a seasonal one. A short training window produces a confident, wrong baseline.
3. Match the technique to the data, and apply it to a focused set of KPIs. Every additional detector is an additional source of alerts, so choose the signals whose deviation you would act on. Segmenting by entity, such as per host or per customer, beats one global model when normal differs across the population.
4. Document known recurring events. Sales, releases, and holidays produce the same flag every cycle. Annotating them to avoid investigating the same event from scratch each time.
5. Involve subject matter experts in interpreting results. The person who knows the business knows whether an unusual reading is a fault or a Tuesday, and routing their verdict back into the model through custom rules and filters can improve it.

Elastic has over 100 out-of-the-box anomaly detection jobs that run against the data you already have in Elasticsearch, including application logs, infrastructure metrics, APM traces, and security events to automatically learn trends, periodicity, and baseline behavior. It models the time-based characteristics of each signal and it adapts as new data arrives. Results are written once per bucket span — the time interval Elastic aggregates data over before scoring it — with scores normalized so the most significant anomalies rank first. This is part of Elastic’s broader machine learning capabilities

The fastest way to find out whether this applies to your problem is to point a job at a dataset you already have. Kibana's ML job wizards supply a working configuration for common data types, so you can run detection without designing a model first. Elastic uses both unsupervised learning (e.g., clustering, Bayesian modeling, and correlation analysis) and supervised techniques (e.g., classification and regression) for a wide range of detection, annotation, and predictive tasks. From there, Elastic Observability connects the results to the logs, metrics, and traces you would use to investigate them.

¹M. Saquib Sarfraz, Mei-Yen Chen, Lukas Layer, Kunyu Peng, and Marios Koulakis, "Position: Quo Vadis, Unsupervised Time Series Anomaly Detection?," arXiv preprint, May 2024.