Skip to main content

Route notifications using tags

Moogsoft Cloud gives you the ability to route notifications to popular tools such as ServiceNow, Slack, and PagerDuty. This type of notification workflow lets you:

  • Define individual and team notifications and escalation paths.

  • Send different types of notifications to specific groups.

After using the Moogsoft Cloud correlation and deduplication features, you can be confident that your notifications will go to your specified teams, without time-consuming redundancy and noise.

This topic describes how to set up a simple process to route notifications using Moogsoft Cloud scope Filters and tags.

About notifications routing

When routing notifications, you can have data coming into Moogsoft Cloud in the form of alerts or events, where the alerts are clustered together to form an incident. Instead of using traditional methods of alerting users (such as creating a general trouble ticket), Moogsoft Cloud enables you to:

  • Combine multiple related alerts to form an incident and use that incident to drive the notifications.

  • Use the incident to send the right notifications to the right team.

  • Define where the notifications go. A notification can go to one or more notification tools, such as Slack, PagerDuty, ServiceNow, and so forth. For example, you can configure outbound webhooks to go to Jira for your engineers and Zendesk for your SREs. Moogsoft Cloud notifications can be routed to different systems based on your defined escalation path.

Let’s see how to set up a simple notification with five basic activities: 1) plan your notifications, 2) set up your data source, 3) configure outbound notifications, 4) create a tag, and 5) map alerts/events with the Moogsoft Cloud Workflow Engine.

Before you begin

Before you begin using these instructions to route notifications, you must understand and know how to:

Plan your notifications

Before setting up notifications, it is a good idea to determine what individuals and teams get notified and when. Define your escalation paths, groups and conditions so that you can specify them in your tags and scope filters to determine what notification is routed and to what group.

To route notifications using tags and scope filters, you need to perform the following procedures.

Configure your data source in Moogsoft Cloud

You need to have or configure a data source to generate events or alerts that can be grouped into incidents. Moogsoft Cloud offers the following data ingestion options that you can use to configure your data source:

  • A number of out-of-the box data ingestion methods from AppDynamics to Zabbix.

  • Easy to use Moogsoft Cloud collector and plugin combinations for data sources such as New Relic, MongoDB and more.

  • If you don’t see a data source that you want to integrate, Moogsoft Cloud also offers the Create Your Own Integration (CYOI) to help you use the data source you want with Moogsoft Cloud.

Configure your outbound notifications in Moogsoft Cloud

You need to have or configure an outbound webhook integration that communicates with your notification system, such as ServiceNow, PagerDuty, Slack, or another type. The outbound integration can be a pre-configured integration such as PagerDuty or a Webhook integration. For more information, read the Moogsoft Cloud documentation on Outbound integrations and Outbound webhooks.

Using an incident webhook

The following sections show how to complete the notifications routing process for incidents using an incident webhook and incident workflow.

Create the webhook

Create your incident webhook:

  1. Navigate to Integrations > Outbound Integrations. Click Incident Webhook in the list, then click Add a Webhook Endpoint.

  2. At the top of the page, enter a name for the webhook and a description in the spaces provided.

    NOTE: It is helpful to include the name of the system in the webhook name.

  3. Under Triggers, select one or more conditions that will cause the webhook to send an incident.

    NOTE: For webhooks used for CREATE activities, no trigger selection is required. The CREATE activity depends on the workflow to trigger the webhook.

  4. Under Endpoint, configure the request:

    • Request Method

      Choose the request type (POST, PATCH, DELETE, PUT) from the menu.

      NOTE: To send notifications when a new incident is created, you will likely need to use the POST method. To send notifications for updates, you will likely need to use the PATCH method.

    • URL

      Enter the external system endpoint to use to communicate with this webhook. Use the appropriate endpoint for the type of webhook action you need to perform. For example, if you are performing a POST action, then you may need to use an endpoint which supports that action.

    • Authorization

      If you have not already done so, add a new credential to the Credentials Store for the external system and select it for this field.

    • Headers

      Define the key/value pairs to send in the header information.

      By default, the header "Content-Type: application/json is included. Make any changes that are necessary for your system.

  5. Under Payload Body, include the payload information that you want to send to the external system.

    Include any mappings that you want to include in the payload. To see the external fields available to include in the mappings, click Test and view the system response.

    For more detailed information, see Configure the CREATE payload body for an incident webhook.

Create the workflow

Create your workflow and define the trigger:

  1. Navigate to Correlate & Automate > Workflow Engine. Click Incident Workflows, and then Add Workflow.

  2. At the top of the page, enter a name and description for the workflow in the spaces provided.

    NOTE: As with the webhook, it is helpful to include the name of the target system in the name of the workflow plus the type of action it is used with.

  3. Configure the workflow trigger:

    Set up the conditions that determine when the workflow triggers.

    NOTE: This example focuses on sending a notification when an incident is created. You can also add a workflow and incident webhook to send updates.

    • In the Trigger section, select Create to send a notification when an incident is created.

    • In the Filter section, add the tags which indicate the data that you're watching for.

      For example, you can use data catalogs to enrich your data with a tag that indicates this incident is flagged as payroll, or from Europe, or from a particular system. This example shows a filter that causes the workflow to trigger for this tag: tags.router_type="Nortel".

    WorkflowTriggerNortel.png
  4. Click Add Action and select Send to Endpoint from the list. Click Add Selected Action to add it to the workflow.

  5. Configure the Send to Endpoint action:

    • Webhook

      Select the webhook that you created in the previous section.

    • External Link

      Select this box if you want to include a clickable link to the external system on the Outbound tab in Moogsoft Cloud, on the Incidents page.

    • Integration Name

      The name that displays on the Outbound tab on the Incidents page. You can name the integration according to your preferences.

    • External ID

      A value which identifies the object on the external system. This is usually a number referencing the ticket ID.

      You can get this information from another field, tag or attribute.

    • External Name

      A value which identifies the name of the object on the external system.

    • URL

      The URL for the link to the external system. You can use parameter substitution variables when defining the URL so they point to a viewable ticket link.

      Example: https://www.example.com/incidents/v1/${outbound.<integration_name>.external_name}

  6. Enable the webhook.

    If everything is configured correctly, a notification will be sent to the external system the next time an incident matching the filter criteria in the workflow trigger is created.

Modify the workflow with actions

In addition to simply sending a notification when an incident matches a webhook trigger, you can use additional incident workflow actions to perform further operations:

Using an alert webhook

The following sections show how to complete the notifications routing process for incidents using an alert webhook.

Create a tag in a scope filter to route notifications (alert webhook)

After you have configured your data source and outbound notifications, you can create a tag for a scope filter to route notifications in Moogsoft Cloud. To create a tag and add it to a scope filter, the tag needs to contain the string that defines the destination for the notification.

Note

Example: Route notifications to Slack

To send a notification to Slack using the outbound webhook-based Slack integration, you would create a tag that contains a Slack channel name. You might use #payroll, #finance, or #backend Slack channels to route notifications to those groups. You must have set these channels up prior to using notifications.

To create your route notifications tag:

  1. In your Moogsoft Cloud instance, navigate to Integrations > Outbound Integrations.

  2. Click the name of an integration that you configured. It can be a pre-configured integration such as PagerDuty or Slack or you can click Webhook (Legacy) to display an outbound webhook client. For example, you could do one of the following:

    1. Click Webhook (Legacy) > <your specific-outbound-Webhook-integration-name>.

    2. Click PagerDuty > <your specific-PagerDuty-integration-name>. Use the same actions for other integrations such as ServiceNow.

  3. When you have clicked your pre-configured integration or webhook client, you need to provide a name, data type, and the scope filter information. The fields are located in different places depending on the integration. For example:

    1. For Webhook integrations, you can navigate to Webhook > Name and Scope and look for the labels Name, Type, and Filter.

    2. For other integrations, such as PagerDuty, look for the Name, Type, and Filter headings, to specify the scope information.

  4. For the Name and Type fields, provide the following:

    • Name - Provide the name of your outbound pre-configured integration or webhook integration.

    • Type - From the drop-down menu, pick the data type you wish to send to your external system.

    • Filter - Define your filter. For example:

      Severity in ["Clear", "Major", "Critical"] AND Services = Slack AND tags.group_id IN ("#payroll", "#finance")
  5. In the field for Filter, define your scope filter information. The filter is essential to routing using tags. For a webhook, the filter is located under the Name and Scope heading, as shown in the following figure.

    For pre-configured integrations, such as PagerDuty, the filter is located under the heading Data Type > Alerts. For your pre-configured integrations, if the filter is not specified under these headings, look for the field that allows you to specify your filter. The following example shows a scope filter that allows you to get notifications, based on the specified tags and group_id, routed to the specified channel names in a Slack app.

    Note

    Example: Define a scope filter

    Severity in ["Clear", "Major", "Critical"] AND Services = Slack AND tags.group_id IN ("#payroll", "#finance")

  6. Add a tag name to your scope filter. You can define one or more values for a tag name. The format of this tag name is:

    tag.your-specific-group-name

    As shown in the prior “Define a scope filter” example, we added the tag name (#payroll) into the group_id (you can use any group name that you specify) to route the notifications.

    You can see in our example, the two channels ("#payroll", "#finance") are added to the tags.group_id. By using a group ID, only appropriate groups with a specified group ID are selected for notification.

Map alerts or events to tags in the Workflow Engine

You still need to associate a tag with a group ID (or equivalent name) for your alerts and events. One method is to use the Query Catalog action in an Event Workflow.

Workflow_Query_Catalog.png
  1. Navigate to Correlate & Automate > Workflow Engine > Event Workflows ><your specific workflow name >.

  2. In the Workflow, click Trigger. In the right-side panel under Trigger > Event Filter, select the event type that you want to trigger the workflow.

  3. In the Workflow workspace, click Add Action, then select Query Catalog. Click Add Selected Action.

    Note

    You must create a Data Catalog before you can use the Query Catalog feature. You can create a data catalog in the Moogsoft Cloud UI or by using the Moogsoft Cloud API. See the topics, Create data catalogs and the API example for more information.

  4. In the Catalog Name section of the Query Catalog action, click inside the box and select your catalog name, such as group_id_map (or other name such as group or team), so that your Alert can cross-reference the group_id identifier associated with your resource.

  5. In the Query Catalog part of the workflow, you associate the Data Catalog resource, such as resource_id, with the group_id using the Apply Mapping field.

    For example, in the following Event Workflow figure, the Catalog Name selected is Escalation Group. Under the Apply Mapping heading, the Data Catalog resource (named group) is mapped to the tags.group.

    Escalation-Group.png

More methods to apply tags in your data source

In addition to using the outbound webhook Name and Scope > Filter or the equivalent pre-configured outbound integration filter to define your tags, Moogsoft Cloud features several other methods for creating tags that you can use to route notifications. To do this, you can:

  • Insert tags in the incoming messages before ingestion. You can have incoming messages, streaming into Moogsoft Cloud, contain a specified tag. When alerts or events are grouped, de-duped, and correlated, the resulting incidents inherit the tags.

  • Populate incoming messages with a tag during ingestion . You can populate incoming alert and events with tags, when you set up your ingestion services, such as Create Your Own Integration (CYOI). Using CYOI, you can easily monitor your mapping. For example, let’s say you are using Integrations > Ingestion Services > Create Your Own Integration > Bitbucket. You can:

    1. Go to Bitbucket > Map Your Data > Source Field to a Moogsoft Cloud Target Field.

    2. Map your incoming data to a Tag by mapping the Payload Field to a Target Field and specifying a tag, such as Tag:Branch to master.

  • Populate your JSON payload with tags. You can map the source field of the incoming data (such as in a JSON payload) to a Moogsoft Cloud target field. This has the added benefit of being easy to monitor. See Mapping Information for more details about mapping tags.

  • Create tags through Enrichment. You could also create a tag for routing notifications through multiple forms of enrichment.

    • Automatic Event Enrichment - provides auto-classification of events. Navigate to Correlate & Automate > Workflow Engine > Event Workflows and click the Automatic Event Enrichment workflow name to set up auto-classifications. You can use a Workflow Engine > Workflow > Trigger to specify the type of events you want to automatically trigger a workflow and use that to route notifications to a specific team or organization, or individual.

    • Match and Update workflow - You can select one or more Input Fields, go to add Regex Tags > Output Field, and Add a Tag in the Output Field or use an existing Tag and populate that tag with the group_id value.