Moving Alerts to the Historic Database

The Alert Analyzer calculates information entropy values using event data for alerts that have been closed and moved into the historic database. It needs about two weeks of historic data to analyze patterns in your event descriptions and calculate an entropy model for applying to live data.

  1. Close all of the open alerts and Situations in the user interface (UI).

  2. Run the MoogDB split configurer utility to move the closed alerts to the historic database. You can do this by accessing your instance with a terminal program and the ssh credentials you received. Alternatively, you can use a pre-configured ChatOps command, 'split_db'.

  3. Verify that the utility ran successfully by checking in the UI that no closed alerts remain.

Step-by-Step Instructions
  1. Navigate to Workbench>Open Alerts.

  2. Scroll to the last alert. You will use this data for entropy calculations. Because it is based on past data--about two week's worth--most of the alerts already have 'clear' status. Scan some of the descriptions. Which alerts seem most serious? Notice that the alerts come from several different managers, or monitoring systems.

  3. Once you have loaded all the alerts into the UI by scrolling to the bottom of the list, you should be able to select all of them at once by using the checkbox at the top left. Click the checkbox, right-click, select 'Close', and click 'OK' to close all of the alerts.

  4. Close all of the open Situations as well.

  5. There are no alerts visible in the default Open Alerts view, but the closed alerts are all still in the active database, so they are not yet available in the historic database for calculating entropy. When it is configured to do so, the Housekeeper Moolet will move closed alerts into the historic database on a regular schedule.

    This time though, move the data yourself using the moog_db_split_configurer utility. You could access this utility by logging in to your instance using your ssh credentials, but we have configured a ChatOps shortcut for this lab so you do not have to do so.

    Go to Workbench>Open Situations>Tools>Create Situation and click 'Done' to create a Situation manually. Go to the Collaborate tab for that Situation.

  6. Enter @bot split_db in the comment box. This will run the following command:

    split_time=$(date -d "+1 minute" +"%H:%M") && /usr/share/moogsoft/bin/utils/moog_db_split_configurer -g 0 -r $split_time

    The first part of the command, before the '&&', defines a time one minute ahead of the current time on your lab instance, and gives it an 'HH:MM' format. The second part of the command schedules the database split utility to run at that time.

  7. Wait until the utility has had time to run, and then go to Workbench>Open Alerts. Click on 'Status' in the filter bar and choose 'Closed' and 'Apply'. Verify that there are no closed alerts remaining in the active database.

This concludes the lab section.