Try It Yourself: Escalate Alerts

This lab is one of a series of Workflow Engine Labs. In this lab, you will:

  • Use the Workflow Engine to increase the severity of specific alerts, edit their descriptions, and route them to a custom Cookbook.

  • Configure a Cookbook and Cookbook Recipe to process the critical alerts from your alert workflow, and add the Cookbook to a merge group.

  • Use the Visualize feature to discover which Cookbook created a Situation.

Examine the Data

Load data into your lab instance and decide how you want to process it.

  1. Clean up the data from the previous lab by closing any open alerts and all but one Situation. If there is no open Situation, generate one manually. (See the Mini Lab for instructions on how to do this.)

  2. Go to the Collaborate tab of your Situation’s Situation Room. Type @moog get_lab_events C_Drive into the comment box to bring in the data for this lab.

  3. You are concerned about the alerts which involve C: drives being nearly full, because you want operators to take action quickly when operating system drives might be compromised. You decide to increase the severity of the C: drive nearly full alerts to critical and add the text "CRITICAL" to the alert descriptions. You can use the Workflow Engine to accomplish these tasks.

    You decide that you want critical alerts to generate individual Situations right away. You can also use the Workflow Engine to route critical alerts to a custom Cookbook.

Examine the Cookbook Settings

Verify your alert clustering settings and decide how you want to design a Cookbook for critical alerts.

  1. In your current configuration, Moogsoft AIOps uses a Cookbook to cluster alerts from the same source. It generates a Situation when there are at least two related alerts. Go to Settings>Algorithms>Cookbooks and select 'Source Cookbook'. You will see that your Cookbook is using a Cookbook Recipe called 'Source'.

  2. Go to Settings>Algorithms>Cookbook Recipes and examine the settings for the 'Source' recipe. Verify that the current alert threshold is set to 2. After reviewing the Cookbook and Recipe settings, you decide that you want another Cookbook and Recipe that will process only critical alerts and generate Situations using an alert threshold of 1. This way your critical C Drive alerts will generate Situations immediately.

  3. Go to Settings>Algorithms>Merge Groups. Merge groups are a feature of Moogsoft AIOps that control how Situations are managed after they are created. When you have multiple active Cookbooks or Cookbook recipes, alerts can be clustered into multiple situations using different algorithmic rules. By default, Moogsoft AIOps will combine, or merge, Situations that share all or most of their alerts, but you can control this behavior using merge groups.

  4. Click on the 'Default Cookbook' merge group. You can see that the Source Cookbook is included in the 'Default Cookbook' merge group. When you build a new Cookbook, you should add it to an existing merge group or create a new merge group. You will make these changes later in the lab.

Define an Alert Workflow

Use the Workflow Engine to define an alert workflow to escalate the C: Drive nearing capacity alerts.

  1. Go to Settings>Automation>Workflow Engine>Alert Workflows, and click on 'Add Workflow' in the upper right.

  2. At the top right, you can see a slider to make a workflow active or inactive. Leave it set to 'Active'.

  3. On the left is a column which you will populate with a series of actions. The first action, 'Delay', is already populated. Leave it set to 0 seconds.

  4. Look at the pane on the right. Start building your workflow by giving it a name, 'Escalate C: Drive Alerts.'

  5. Describe your workflow so that other users (and you in the future) understand what it does. Fill in the description field with 'Increase severity of C: drive nearly full alerts to critical, edit description, and route to Critical Cookbook.'

  6. Use an entry filter to identify only the alerts you want to act on. If you leave the entry filter blank, the Workflow Engine will process every alert. Click on 'Entry Filter, then 'Edit' and 'Add Clause'.

  7. Select the 'description' field in the drop-down box, and select the operator 'matches' for a partial or full text string match.

  8. You can use regular expression syntax in the next text box to match a variety of text strings. In this case, though, you have decided you only want to escalate the most severe C: drive alerts, so simply enter the full description for those alerts. Type in 'C: Drive Nearing Capacity'.

  9. Click 'Apply' and then 'Done'.

Increase Severity to Critical

Add a workflow action to increase alert severity to critical.

  1. Click on 'Add Action' at the top of the left column. The right pane will change to show the action definition screen.

  2. Enter 'Increase Severity to Critical' in the name text box.

  3. The Function section is a drop-down list of functions you can apply to incoming events. You can review how the functions work in the documentation. Choose the 'setSeverity' function.

  4. The Arguments section changes depending on the function you choose. For the 'setSeverity' function, you only need to enter the desired severity value, which is 5 for critical. Enter it in the text box.

  5. Leaving the forwarding behavior set to 'Always Forward', click 'Save' in the upper right.

  6. At this point, you may want to check your work by closing the open Situations and alerts, re-sending the data, and inspecting the severity for the 'C: Drive Nearing Capacity' alerts. Alternatively, you can proceed directly to designing an action to edit the alert descriptions.

Edit the Alert Description

Add a workflow action to prefix alert descriptions with the word 'CRITICAL'.

  1. On the Workflow Engine Alert Workflows tab, click on your workflow and then click 'Edit'.

  2. Click on 'Add Action' and enter 'Update Description' in the Action Name text box.

  3. Choose the 'prependString' function in the dropdown box.

  4. Enter 'CRITICAL' in the string text box in the Arguments section.

  5. Enter 'description' in the destination text box.

  6. Leave the forwarding behavior set to 'Always Forward'.

  7. Click 'Save'.

  8. Test your results by closing the open alerts and Situations and resending the data using the @moog get_lab_events C_Drive ChatOps command.

Route Critical C: Drive Alerts

Add a workflow action to send alerts directly to a custom Cookbook.

  1. Add an action named 'Send to Critical Cookbook'.

  2. Choose 'forward' as the function and enter 'Critical Cookbook' in the Moolet text box.

  3. Leave the forwarding behavior set to 'Always Forward'.

  4. Save the workflow.

Set Up a Critical Alerts Cookbook Recipe

Define a Cookbook Recipe to use in a Cookbook which will process the output of your C: Drive alert workflow.

  1. Go to Settings>Algorithms>Cookbook Recipes.

  2. Click on the plus sign in the lower left corner to add a new recipe.

  3. Name it 'Critical Alerts'.

  4. In the description text box, enter 'Critical issue affecting $UNIQ(source)'. This Situation Manager Labeler macro will dynamically insert the hostname into the Situation description.

  5. Click on the edit pencil next to the Trigger Filter text box, and then click on 'Add Clause'.

  6. Choose 'severity' from the first dropdown box, and choose '=' as the operator in the second dropdown box.

  7. In the third dropdown box, choose 'Critical' and then click 'Apply' and then 'Done'.

  8. Verify that the Alert Threshold is set to '1' (the default) so that Situations are generated immediately when a critical alert arrives.

  9. Go to the Clustering tab, and click on 'Add Field.'

  10. Click on the default first field, 'agent'. In the dropdown box that appears on the right, select 'source' to replace it.

  11. Make the similarity threshold 100% by moving the slider all the way to the right. This ensures that alerts from different computers will be clustered into different Situations.

  12. Click 'Save Changes'.

Set Up the Critical Cookbook

Add the Critical Alerts Cookbook Recipe to a Critical Cookbook that will accept only the output from your C: Drive alert workflow.

  1. Go to Setting>Algorithms>Cookbooks and create a new Cookbook.

  2. Name it 'Critical Cookbook' and enter 'Process any C: Drive Nearing Capacity alerts' in the description text box.

  3. By default, Process Output Of is set to 'AlertBuilder'. If you leave this setting as is and also route alerts to this Cookbook from your C: Drive workflow, it will process each of those alerts twice. Change the setting to 'Other Moolets' and enter 'None' in the Moolet Name text box.

  4. Set the Cook For time to 30 minutes so that new critical alerts will keep being added to Situations for half an hour.

  5. In the Selected Recipes section, move the Critical Alerts recipe into the Selected column, and then save changes.

  6. Go to Settings>Algorithms>Cookbook Selection.

  7. Make the Critical Cookbook active by clicking on it, clicking on the right arrow, and then clicking on 'Save Changes.'

  8. Go to Settings>Algorithms>Merge Groups. Click on 'Default Cookbook'.

  9. Click on 'Edit' in the upper right, and then click on 'Add Sigaliser.'

  10. Choose 'Critical Cookbook' from the dropdown menu and click on 'Save' to add your Cookbook to the default merge group.

  11. You should see a dialog box telling you that the merge group will restart. Click 'Ok'. If the dialog box does not appear, verify that the Critical Cookbook is active in the Cookbook Selection settings before you edit the merge group.

Review Your Results

Verify that all of your changes are working together as you expect.

  1. Close any open alerts and all but one Situation. Generate a Situation manually if one does not exist.

  2. Resend the data into Moogsoft AIOps using the @moog get_lab_events C_Drive ChatOps command.

  3. Go to the Open Alerts view and examine the alert list.

    1. Are the 'C Drive Nearing Capacity Alerts' listed as critical?

    2. Do their descriptions include the word 'CRITICAL'?

  4. Go to the Open Situations view.

  5. Examine each critical Situation by clicking on the Situation in the Situation list and then clicking on the Alerts tab. You should see that two of the three critical Alerts have formed their own Situations. The third one has two other alerts affecting the same host. You are curious about how this Situation was generated. Did it pass through the Critical Cookbook?

  6. Click on the Visualize tab and examine the information. You can see that the Critical Alerts Recipe and the Critical Cookbook generated the Situation from your critical alert. Because the Critical Cookbook and Source Cookbook share a merge group, the other alerts from the same source were added based on the merge group similarity threshold.

    If you wanted to, you could set up a new merge group to keep Situations from your Critical Cookbook from merging with other Situations. You decide, however, that you like this clustering behavior. Your operators will see a critical C: Drive Situation immediately. As other potential Situations involving the same host occur, they will merge automatically.