Moogsoft Docs

Try It Yourself: Escalate Alerts

In this lab, you will:

  • Use the Workflow Engine to increase the severity of specific alerts, edit their descriptions, and route them to a custom Cookbook.

  • Configure a Cookbook and Cookbook Recipe to process critical alerts from your alert workflow.

  • Use the Graze API and the cURL command to change a default setting so that individual alerts can form Situations without waiting for multiple alerts to be clustered.

  • Use the beta Visualize feature to discover which Situation created a cookbook.

Examine the Data

Load data into your lab instance and decide how you want to process it.

  1. Clean up the data from the previous lab by closing any open alerts and all but one Situation. If there is no open Situation, generate one manually. (See the Mini Lab for instructions on how to do this.)

  2. Go to the Collaborate tab of your Situation’s Situation Room. Type @moog get_lab_events C_Drive into the comment box to bring in the data for this lab.

  3. Go to Workbench>Open Alerts. Scroll through the alerts and look at their descriptions and severities. You can see that there are a small number of alerts which are classified as 'Major': a fan failure and some hard drives nearing capacity. There aren’t any alerts listed as 'Critical'.

    You are concerned about the alerts which involve C: drives being nearly full, because you want operators to take action quickly when operating system drives might be compromised. You decide to increase the severity of the C: drive nearly full alerts to critical and add the text "CRITICAL" to the alert descriptions. You can use the Workflow Engine to accomplish these tasks.

    You decide that you want critical alerts to generate individual Situations right away. You can also use the Workflow Engine to route critical alerts to a custom Cookbook.

Examine the Cookbook Settings

Verify your alert clustering settings and decide how you want to design a Cookbook for critical alerts.

  1. In your current configuration, Moogsoft AIOps uses a Cookbook to cluster alerts from the same source. It generates a Situation when there are at least two related alerts. Go to Settings>Algorithms>Cookbooks and select 'Source Cookbook'. You will see that your Cookbook is using a Cookbook Recipe called 'Source'.

  2. Go to Settings>Algorithms>Cookbook Recipes and examine the settings for the 'Source' recipe. Verify that the current alert threshold is set to the default value of 2. After reviewing the Cookbook and Recipe settings, you decide that you want another Cookbook and Recipe that will process only critical alerts and generate Situations using an alert threshold of 1. This way your critical C Drive alerts will generate Situations immediately. This seems straightforward, but there is one issue having to do with how default Merge Group settings are configured.

  3. To understand the issue, go to Settings>Algorithms>Merge Groups. Merge Groups are a feature of Moogsoft AIOps that control how Situations are managed after they are created. When you have multiple active Cookbooks or Cookbook recipes, alerts can be clustered into multiple situations using different algorithmic rules. By default, Moogsoft AIOps will combine, or merge, Situations that share all or most of their alerts, but you can control this behavior using Merge Groups.

  4. Like Cookbook Recipes, Merge Groups have a default alert threshold of 2. The Merge Group setting takes precedence over Cookbook Recipe settings for the default Merge Group and custom Merge Groups. Click on 'Add Merge Group' to see which settings are available. As you can see, there is no Merge Group alert threshold setting in the user interface in the current version of Moogsoft AIOps. However, you can use the Moogsoft Graze API to change the Merge Group alert threshold programmatically. You will do this later in the lab. Once you have made this change using the Graze API, you will be able to set the alert threshold in your Cookbooks to generate Situations from individual alerts.

  5. Exit out of the Merge Group settings page without creating a new Merge Group.

Define an Alert Workflow

Use the Workflow Engine to define an alert workflow to escalate the C: Drive nearing capacity alerts.

  1. Go to Settings>Automation>Workflow Engine>Alert Workflows, and click on 'Add Workflow' in the upper right.

  2. At the top right, you can see a slider to make a workflow active or inactive. Leave it set to 'Active'.

  3. On the left is a column which you will populate with a series of actions. The first action, 'Delay', is already populated. Leave it set to 0 seconds.

  4. Look at the pane on the right. Start building your workflow by giving it a name, 'Escalate C: Drive Alerts.'

  5. Describe your workflow so that other users (and you in the future) understand what it does. Fill in the description field with 'Increase severity of C: drive nearly full alerts to critical, edit description, and route to Critical Cookbook.'

  6. Use an entry filter to identify only the alerts you want to act on. If you leave the entry filter blank, the Workflow Engine will process every alert. Click on 'Entry Filter, then 'Edit' and 'Add Clause'.

  7. Select the 'description' field in the drop-down box, and select the operator 'matches' for a partial or full text string match.

  8. You can use regular expression syntax in the next text box to match a variety of text strings. In this case, though, you have decided you only want to escalate the most severe C: drive alerts, so simply enter the full description for those alerts. Type in 'C: Drive Nearing Capacity'.

  9. Click 'Apply' and then 'Done'.

Increase Severity to Critical

Add a workflow action to increase alert severity to critical.

  1. Click on 'Add Action' at the top of the left column. The right pane will change to show the action definition screen.

  2. Enter 'Increase Severity to Critical' in the name text box.

  3. The Function section is a drop-down list of functions you can apply to incoming events. You can review how the functions work in the Moogsoft documentation. Choose the 'setSeverity' function.

  4. The Arguments section changes depending on the function you choose. For the 'setSeverity' function, you only need to enter the desired severity value, which is 5 for critical. Enter it in the text box.

  5. Leaving the forwarding behavior set to 'Always Forward', click 'Save' in the upper right.

  6. At this point, you may want to check your work by closing the open Situations and alerts, re-sending the data, and inspecting the severity for the 'C: Drive Nearing Capacity' alerts. Alternatively, you can proceed directly to designing an action to edit the alert descriptions.

Edit the Alert Description

Add a workflow action to prefix alert descriptions with the word 'CRITICAL'.

  1. On the Workflow Engine Alert Workflows tab, click on your workflow and then click 'Edit'.

  2. Click on 'Add Action' and enter 'Update Description' in the Action Name text box.

  3. Choose the 'prependString' function in the dropdown box.

  4. Enter 'CRITICAL' in the string text box in the Arguments section.

  5. Enter 'description' in the destination text box.

  6. Leave the forwarding behavior set to 'Always Forward'.

  7. Click 'Save'.

  8. Test your results by closing the open alerts and Situations and resending the data using the @moog get_lab_events C_Drive ChatOps command.

Route Critical C: Drive Alerts

Add a workflow action to send alerts directly to a custom Cookbook.

  1. Add an action named 'Send to Critical Cookbook'.

  2. Choose 'forward' as the function and enter 'Critical Cookbook' in the Moolet text box.

  3. Leave the forwarding behavior set to 'Always Forward'.

  4. Save the workflow.

Set Up a Critical Alerts Cookbook Recipe

Define a Cookbook Recipe to use in a Cookbook which will process the output of your C: Drive alert workflow.

  1. Go to Settings>Algorithms>Cookbook Recipes.

  2. Click on the plus sign in the lower left corner to add a new recipe.

  3. Name it 'Critical Alerts'.

  4. In the description text box, enter 'Process any C: Drive Nearing Capacity alerts'.

  5. Click on the edit pencil next to the Trigger Filter text box, and then click on 'Add Clause'.

  6. Choose 'severity' from the first dropdown box, and choose '=' as the operator in the second dropdown box.

  7. In the third dropdown box, choose 'Critical' and then click 'Apply' and then 'Done'.

  8. Set the Alert Threshold to '1' so that Situations are generated immediately when a critical alert arrives. This setting will take effect after you also change the default Merge Group alert threshold to '1'.

  9. Go to the Clustering tab, and click on 'Add Field.'

  10. Click on the default first field, 'agent'. In the dropdown box that appears on the right, select 'source' to replace it.

  11. Make the similarity threshold 100% by moving the slider all the way to the right. This ensures that alerts from different computers will be clustered into different Situations.

  12. Click 'Save Changes'.

Set Up the Critical Cookbook

Add the Critical Alerts Cookbook Recipe to a Critical Cookbook that will accept only the output from your C: Drive alert workflow.

  1. Go to Setting>Algorithms>Cookbooks and create a new Cookbook.

  2. Name it 'Critical Cookbook' and enter 'Process any C: Drive Nearing Capacity alerts' in the description text box.

  3. By default, Process Output Of is set to 'AlertBuilder'. If you leave this setting as is and also route alerts to this Cookbook from your C: Drive workflow, it will process each of those alerts twice. Change the setting to 'Other Moolets' and enter 'None' in the Moolet Name text box.

  4. Set the entropy threshold to 0.

  5. Set the Cook For time to 30 minutes so that new critical alerts will keep being added to Situations for half an hour.

  6. In the Selected Recipes section, move the Critical Alerts recipe into the Selected column, and then save changes.

  7. Go to Settings>Algorithms>Cookbook Selection.

  8. Make the Critical Cookbook active by clicking on it, clicking on the right arrow, and then clicking on 'Save Changes.'

Change the Default Merge Group Alert Threshold

Use the Graze API to change a Merge Group setting.

The Graze API, or application programming interface, is a set of programmatic tools that you can use to interact with Moogsoft AIOps to retrieve or update information or take actions. Graze commands take the form of web requests using the https: protocol. Using the Linux curl command (think "see URL") with appropriate security authentication, you can send a Graze request from a terminal program on any computer to your Moogsoft AIOps system.

You can also send the request from the same computer. To demonstrate how Graze API requests work, we have configured two ChatOps commands to send Graze API requests for you. You can use them to change the Moogsoft AIOps default Merge Group alert threshold to 1. This will allow your Critical Cookbook to generate Situations from individual alerts.

  1. Go to the Collaborate tab of an open Situation, creating a Situation manually if necessary.

  2. Enter @moog merge_group_info. This ChatOps command sends the following Graze request to your Moogsoft AIOps system:

    curl -G -u graze:graze -k https://localhost/graze/v1/getDefaultMergeGroup

  3. Examine the output of the command. You should see text telling you that the default Merge Group alert threshold is 2.

  4. Enter @moog update_merge_group. This will send the following Graze request to your Moogsoft AIOps system:

    curl -X POST -u graze:graze -k 'https://localhost/graze/v1/updateDefaultMergeGroup' -H 'Content-Type: application/json; charset=UTF-8' --data '{"alert_threshold" : 1}'

  5. Send @moog merge_group_info again and verify that the alert threshold is now 1.

Review Your Results

Verify that all of your changes—defining a workflow, adding actions to escalate alerts, routing those alerts to a new Cookbook, and updating a default setting—are working together as you expect.

  1. Close any open alerts and all but one Situation. Generate a Situation manually if one does not exist.

  2. Resend the data into Moogsoft AIOps using the @moog get_lab_events C_Drive ChatOps command.

  3. Go to the Open Alerts view and examine the alert list.

    1. Are the 'C Drive Nearing Capacity Alerts' listed as critical?

    2. Do their descriptions include the word 'CRITICAL'?

  4. Go to the Open Situations view.

  5. Examine each critical Situation by clicking on the Situation in the Situation list and then clicking on the Alerts tab. You should see that two of the three critical Alerts have formed their own Situations. The third one has two other alerts affecting the same host. You are curious about how this Situation was generated. Did it pass through the Critical Cookbook?

  6. Go to Settings>System Preferences>Labs.

  7. Click the checkbox for 'Visualize' under Beta Features.

  8. Go back to Workbench>Open Situations and click on the list to open the Situation Room for the critical Situation which has multiple alerts.

  9. Refresh your browser window, and you should see a Visualize tab appear in the Situation Room.

  10. Click on the Visualize tab and examine the information. You can see that the Critical Alerts Recipe and the Critical Cookbook generated the Situation from your critical alert. Later an automatic "superseding merge" combined it with another Situation.

    If you wanted to, you could set up a new Merge Group to keep Situations from your Critical Cookbook from merging with other Situations. You decide, however, that you like this clustering behavior. Your operators will see a critical C: Drive Situation immediately. As other potentially relevant Situations involving the same host occur, the Situations will merge.

Last updated 2019-10-08 13:45:59 -0400