What is entropy in Moogsoft Enterprise?

You can use entropy to control clustering in Cookbooks.

What Is Entropy? How Does It Help Me?

Entropy is a measure of how unexpected or unpredictable an alert is. 

Moogsoft Enterprise assigns every alert an entropy score that is a value between 0 and 1. An event that re-occurs frequently receives a low entropy score and is deemed operationally insignificant. Meanwhile, a more rare event receives a high score and is considered to be operationally significant.

Entropy is a key noise reduction feature. By filtering out the alerts with low entropy score, you can keep the important alerts from getting buried under the flood of common alerts.

How Does Entropy Work?

The default entropy calculation uses the description field to determine the similarity between alerts, but you can choose different alert attributes.

Moogsoft Enterprise analyzes the textual aspects of the incoming event by tokenizing the value of the description field. Items such as numbers and timestamps are masked and therefore excluded from the entropy calculation, and the score is derived based on the aggregation of token entropies from within the string. The entropy score is calculated up to 16 decimal places. However, note that in the Cookbook UI, you can configure the entropy threshold value only up to 2 decimal places.

entropy_distribution.png

The resulting score becomes just another attribute for the alert, and you can use the score to filter out the alerts below a certain entropy value. By removing the common, insignificant alerts, your operators to focus on more significant events. Entropy is language-agnostic, which means it works with English as well as any other UTF-8 based language.

Entropy Distribution - It's Different from Environment to Environment

Using entropy to filter out the noise events can be an impactful differentiator, so there is a responsibility to set it up correctly for a given environment.

There is no formula to tell you what your threshold number should be. Entropy distribution varies from organization to organization, or even between two sub-environments within the same organization.

Each environment will have its own specific entropy profile which is usually non-transferable as no two environments are identical. Noise for one organization might actually be deemed as important and actionable events by another. Consider failed authentication alerts when attempting network discovery, which is business as usual, versus failed logins during hacking attempts. 

So, in short, there is no specific entropy score by which you can set the entropy threshold. Always check the distribution of the scores, and fine-tune until you strike a balance.

Safeguard Measures

Just because you set up this filter, you do not need to miss all the lower entropy alerts completely. You can direct them to a separate clustering algorithm that will group them together as a low priority Situation.

For example, if you group a queue of low entropy alerts coming from the same source within an hour, operators can review the queue periodically to decide if they require action.

Also, if you find a specific kind of alert with a low entropy score that is actually important, you can increase the entropy of such alerts based on a configurable list of keywords that always receive an entropy value of 1.