Entropy Overview

Entropy is defined as the degree of disorder or randomness in a system. In Moogsoft Enterprise, entropy is a measure of how unexpected or unpredictable an event or an alert is. According to information theory, the more unpredictable or unexpected an event is, the more information it is deemed to carry. Therefore, entropy is a measure of the amount of information contained in an event.

The Events Analyser utility is a standalone process that assigns an entropy value to an event token based on its uniqueness. The Alert Builder assigns an entropy value to each alert based on the token entropies. The entropy value is a numeric value between 0 and 1 (accurate to 16 decimal places). It provides an indication of how important an alert is. An entropy value of 0 means that the alert is just ‘noise’ and a value of 1 means that the alert is significant. You can configure the clustering algorithms to ignore common alerts with a low entropy value; this reduces ‘noise’ in Moogsoft Enterprise. See the Clustering Algorithm Guide for more information.

How Moogsoft Enterprise evaluates entropy

The Events Analyser utility analyzes the text attributes of events to assign a semantic entropy value. In the default Moogsoft Enterprise implementation, the Events Analyser uses the description field but you can configure it to use other text fields. The Events Analyser divides the text in between spaces into tokens. For example, the following description has five tokens:

Link down on port 2/32

The Events Analyser calculates the entropy of each token and stores the token in the Moogsoft Enterprise reference database with its associated entropy value. Initially, a new token has a value of 1. The Events Analyser reduces this entropy value as more events occur which contain the same token.

You can configure the Events Analyser to mask volatile token types, such as dates, times, numbers, URLs or IP addresses, so that they are not included in the tokens. See the Events Analyser for further details of the analysis it performs.

The Alert Builder uses the entropy value of the tokens within an alert to calculate the entropy of that alert.

The Events Analyser uses the EntropyV2 calculation method in the default Moogsoft Enterprise implementation. The EntropyV2 method calculates entropy values in real-time based on any tokens it has encountered before. The Alert Builder assigns the entropy of an alert based on the entropy value of the tokens within the alert rather than the entire database. Tokens within an alert which occur frequently contribute negatively to the entropy of an alert, indicating that the alert may not be as significant as an alert with tokens that are seen less frequently. This is in contrast to the EntropyClassic algorithm where the entropy of each alert takes into consideration the significance of tokens in the entire database.

Note

Moogsoft recommends using the EntropyV2 algorithm to produce better alert entropy values than with the EntropyClassic algorithm.

If the Alert Builder receives an event with a token that it has encountered before, from a previous run of the Events Analyser, it sets the alert entropy to match the value saved in the reference database. If the Alert Builder receives an event with a token that it has not encountered before, it calculates the entropy value in real-time and applies this value to the alert. The Alert Builder also saves the entropy value in the reference database for future retrieval.

The Events Analyser stores data in memory while it calculates entropy values. It is important that the Events Analyser runs frequently to ensure that it does not fail with a memory outage. See Run Events Analyser for more information on running the Events Analyser.

Set an entropy threshold

You can set an entropy threshold in each Sigaliser so that only alerts with a higher entropy value are included in Situations. To decide on the value of your entropy threshold, consider the distribution of entropy values in the alerts. A typical entropy value distribution is show in the following diagram:

Moogsoft recommends that you set your entropy threshold to a value on the downward slope of the peak to exclude the majority of alerts. In this example, the entropy threshold is set at 0.21. This reduces the level of ‘noise’ so that you are only clustering the important alerts, with an entropy value greater than the threshold, into Situations.

You can define entropy thresholds in the clustering algorithms to exclude alerts which have an entropy value that is lower than the threshold. This prevents Moogsoft Enterprise from including unimportant 'noisy' alerts in Situations. See the Clustering Algorithm Guide for more information.

Vertex Entropy

Vertex Entropy uses a different form of entropy, topological entropy, to establish how critical the nodes are in your network topology. You can use Vertex Entropy calculations within Cookbook to create Situations which cluster alerts from important nodes. See Vertex Entropy for more information.