Video explainer: Anomaly detection models ►

Watch a video on Anomaly Detection Models in Moogsoft.

Moogsoft anomaly detection works out of the box with no configuration.  But let’s take a moment to learn how it works, so should you need to tweak the default settings you can do so to produce the result you want.

Moogsoft starts to detect anomaly as soon as the source data is ingested.

Image1_AnomalyDetection.png

That said, it needs to learn how your data behaves before defining what’s normal and what requires attention.

During the learning period, express will still generate anomaly, but will not assign severity.  All alerts are assigned the UNKNOWN severity level (purple).   The default correlation settings will bundle them into incidents, but the best practice is to ignore the clustering at this stage.  You may want to edit the scope in your default correlation engine to filter out the alerts with the Unknown severity level.

Image2_LearningPeriod.png

There are several different anomaly detection mechanisms in Moogsoft Express, and it chooses the one that is optimum for each metric type.  But should you desire, you can override the default detection model, and pick a different one.  Let’s spend a few moments to learn about each type.

Image3_Models.png

Adaptive Detector is a dynamic thresholding mechanism and is used for most data.  The Adaptive Detector makes statistical analysis of data against the running median absolute deviation and determines the high and low thresholds.  It is not following any set pattern, but rather adjust to the changes when a new pattern emerges, and quickly figures out the new norm.

In this example, notice this (point to the upper left yellow dot) is considered an anomaly while these are not because they happen during a peak time.  (show a flat line) If you used a fixed threshold you will not be able to do this.

Image4_AdaptiveDetector.png

Let’s see the adaptive detector at work in the environment we just set up.

Image5_AdaptiveDetectoratwork.png

This one uses the adaptive detector, so the anomaly threshold will change as the data distribution changes over time.

Bitmask and False engines are for processing binary data.  While Bitmask is integer, False is Boolean text.

Image6_Bitmask_False.png

For the metrics that has clear on or off, up or down status, Moogsoft will select this engines.

Since the anticipated value is either true or false, Express is using the false detector to find anomaly.  And here’s the one that uses bitmask.

Image7_False.png
Image8_Bitmask.png

If you prefer, a static thresholding is available too.  This is a simple evaluation of data against hard-coded threshold values.  But what’s notable is that Moogsoft offers not just high threshold but low threshold too!

Image9_Threshold.png

Lastly, the model detector is another dynamic thresholding mechanism like the adaptive detector.  But rather than constantly calculating the norm, it creates a model from the observed pattern. Here’s a scenario.  Let’s say we are monitoring a service that streams movies.  As more people tend to use the service in the evenings, the norm in the evenings can be twice as high as the morning.  This fluctuation is expected so you want to adjust the threshold for that time frame.

For the metrics like this, you can create a model based on the past data and use it as a baseline for new incoming data.

So now we know how Moogsoft identifies the anomalies.  Now, let’s talk about how it considers the level of anomaly.  For each anomaly, Moogsoft will assign a color-coded severity level, so an anomaly that is very far from the norm will have the red critical label, As the level of anomaly lessens, the severity label will change to orange, yellow and blue.  And of course, you already know the purple means “unknown” due to the fact that Moogosft is still learning from the incoming data.

Image10_AnomalyLevel.png

Also, aside from the severity levels, Moogsoft assigns a confidence value for each anomaly. This is unrelated to how severe the anomaly is, but more about how certain Moogsoft is about each detected anomaly being an anomaly.

Image11_ConfidenceValue.png

For the rule-based detectors like threshold, false, and bitmask detectors, the confidence level will always be 1, which means we are 100% confident that the given data should be considered an anomaly.For other detectors, Express assigns the confidence value between 0-1 by comparing the given anomaly against all the other anomalies for that metric.  So the close the value is to 1, the more confident you can be that the detected anomaly truly requires your attention.

Now you know the anomaly detection models in Moogsoft.

Thanks for watching!