Alert workflow configuration example

Every alert workflow consists of a trigger and one or more actions. Alert workflows can be triggered when alerts are created, updated, or both. The trigger also includes an optional filter that defines the alerts that the workflow will process. After an alert passes the trigger conditions, each workflow action processes the alert in sequence. To create a workflow for the example scenario described above, do the following:

Navigate to Correlate & Automate > Workflow Engine, click the Alert Workflows tab, and then click Add Workflow. An empty, new workflow appears with a single Trigger element.

At the top of the page, be sure to give the workflow a name and, optionally, add a description. Note that you cannot save the workflow if it does not have a name.

Determine which alerts will trigger the workflow. For this example, the trigger condition is set to New alerts only, so only newly created alerts trigger the workflow.

This step identifies the alerts containing a specific string and adds an identifying service to the alert services list. You can use the Match and Update action for this task.

To add an action to the workflow:

Configure the action to check if the alert source field contains the string "cloudfront." When it does, then the value "critical system" is added to the service field.

This example assumes that the sources could have a variety of names including the string "cloudfront." The following sources match the regex shown above:

This step filters out all alerts which were not updated with the service field value "critical system" in the previous step. The Filter action ensures the workflow only continues processing the alerts of interest, which are those containing the word "cloudfront" somewhere in the source field. Any non-matching alerts skip the rest of the workflow and are processed by any other remaining workflows.

To add the action to the workflow, click Add Action. Select the Filter action from the list, and then click Add Selected action.

Because alerts containing the string "cloudfront" require immediate attention in this scenario, this step adds the Set Severity action to the workflow and assigns the highest severity to the alerts.

To add the action to the workflow, click Add Action. Select the Set Severity action from the list, and then click Add Selected action.

Since these are highly important alerts, APEX AIOps Incident Management users need to be aware that they represent potential issues on critical systems. To ensure visibility for anyone assigned to troubleshooting the issue, the Set Description action adds the description "This is a critical service" to all of the processed alerts.

To add the action to the workflow, click Add Action. Select the Set Description action from the list, and then click Add Selected action.

Note that this action also supports more complex description compositions using macros and substitution syntax.

The Workflow Tester tab is available at the bottom of the page. Pull the tab upward to access the feature. Complete the necessary input fields to test your workflow. The Workflow Tester lets you change the input fields to see if your workflow completes successfully.

To test this example workflow, the source field is set to randomcloudfrontserver.mydomain.net.

You can see how each action is tested in turn, with the changes each action makes reflected in the input to later actions:

After the alerts are prepared using workflows, it is then possible to correlate them based on the properties configured here. For example, if you wanted to create a separate correlation definition for these alerts, you could create a correlation definition filter like this, using the service added by the Match and Update action:

service in ( "critical service" )

In this way, you could correlate all of alerts containing "cloudfront" in the source field using separate methodology. Depending on the situation, it might make sense to correlate all of these alerts where the source field matched with a 100% similarity threshold, to create incidents from specific source systems. Or alerts could be correlated using another field, such as class, or manager. Alert workflow offers numerous options for managing alerts and preparing them for correlation. For more information on correlation, see Correlation Engine overview.