Calculating Entropy

The Alert Analyzer generates a database of all the words (defined as tokens separated by whitespace) in your historic descriptions and calculates the entropy, or rarity, for each one. It aggregates the word-level entropy for events and alerts in both the historic and live databases, assigning values between 0 (the lowest possible entropy) and 1 (the highest possible entropy).

Overview

Configure the Alert Analyzer to calculate entropy settings.

  1. Now that there is data in the historic database, trigger entropy calculation using the left navigation bar under Alert Analyzer settings.

  2. Turn off the 'Use Schedule Preset' option and schedule an entropy generation run manually. You can use the ChatOps 'date' command to get the local time on your instance first.

  3. Examine the resulting entropy distribution.

Step-by-Step Instructions
  1. Find out the time on your lab instance. You will need it to schedule an entropy run. You can do this with the Linux 'date' command, which we have configured as a ChatOps command for this lab.

    Go to Workbench > Open Situations and click on the Situation you created earlier. From the Collaborate tab, enter @bot date and examine the response.

  2. Go to Settings > Algorithms > Alert Analyzer (Entropy). You'll see that the data exploration panes are empty because entropy has not yet been calculated on your instance.

  3. Click on '>' in the upper left corner to expand the configuration menu.

  4. Choose 'Configure Entropy Generation'.

  5. By default, the entropy generator runs daily, using the most recent two weeks of data in the historic database, but it has not been set to run in your lab instance. Click the checkbox next to 'Enable Entropy Generation'.

  6. Uncheck the 'Use Schedule Preset' option to access more scheduling options.

  7. Leave the default settings the same. Choose 'Add Item' in the lower right corner to add an additional entropy run.

  8. Scroll down. In the options for the new run, leave 'Incremental Run' checked.

  9. Set 'Run at Time' to a time a few minutes in the future in your lab instance time zone. Do not use your own local time to schedule the run (unless your time is the same as your lab instance time). Instead, use the results of the 'date' ChatOps command you performed earlier to calculate a time a few minutes in the future in your lab instance time zone.

  10. Click the 'Save' button in the upper right corner.

  11. Reopen the left navigation menu and choose 'Global Default Threshold' to go back to the data visualization panes.

  12. Within a minute of the scheduled time, once the entropy calculations are complete and the screen has refreshed, you should see an entropy distribution graph on the left and an alert review window on the right. (If you are working on a small screen, you may have to scroll down to see the alert review window.) Examine the graph and the review window.

    1. The bottom axis of the graph shows the calculated entropy value, which ranges from 0 to 1.

    2. The left axis of the graph shows the alert count for the dark blue line, which gives the number of alerts at each entropy level.

    3. The right axis of the graph shows the percentage of total alerts for the light blue area, which gives the cumulative share of alerts at each entropy level.

    4. The threshold entropy value is set to the lowest entropy value among the reference or historic alerts. At this value, no reference alerts are excluded by the threshold.

    5. The alert review pane has options to show excluded alerts and included alerts for both the reference historic data and for live data. It is set to 'Excluded Alerts', so there are no alerts showing.

  13. Change the review pane to show included alerts and scroll through them. Overall, do the highest entropy alerts seem more important than the lowest entropy alerts? Are there any low-entropy status or heartbeat alerts that you would want to avoid clustering in Situations?

This concludes the lab section.