Send notifications using PagerDuty

This integration enables you to receive just-in-time PagerDuty notifications when alerts or incidents get updated in Moogsoft Cloud. The Moogsoft Cloud PagerDuty integration supports the v3 version of the PagerDuty API.

This is a bidirectional integration in which the two systems synchronize their alerts and incidents:

  • Moogsoft Cloud notifies PagerDuty when it creates an alert or incident. PagerDuty then creates a new corresponding object.

  • Moogsoft Cloud notifies PagerDuty whenever it updates an alert or incident. PagerDuty then updates the corresponding object.

    The following events trigger an update in PagerDuty:

    • Incident status update

    • Incident severity update

    • Alert status update

    • Alert severity update

  • PagerDuty notifies Moogsoft Cloud whenever an incident (derived from Moogsoft Cloud ) gets updated. Moogsoft Cloud then updates the corresponding object.

See the PagerDuty documentation for details on PagerDuty components.

Before you begin

Important

The integration automatically creates a webhook in PagerDuty and creates a set of Moogsoft Cloud credentials to allow PagerDuty to send information back to Moogsoft Cloud.

Before you start to set up your integration, make sure that you have the necessary permissions to complete these tasks.

This integration was validated with PagerDuty on May 11, 2022.

Save your PagerDuty credentials in Moogsoft Cloud

To send data to PagerDuty, you need the following credentials:

  • A PagerDuty API key with an access level of Full access

  • A PagerDuty integration key

Generate PagerDuty credentials and save them in Moogsoft Cloud:

Important

The user who creates the API key must be an Admin or an Account Owner.

  1. Create and save a PagerDuty API key:

    1. In PagerDuty, navigate to Integrations > API Access Keys.

    2. Click Create a New API Key.

    3. Enter a description and then click Create Key.

      Copy the API key when it displays.

    4. Check the table in API Access Keys. Make sure the access level is Full access.

    5. In Moogsoft Cloud, navigate to Settings > Credential Store.

    6. Click Add Credential and then select PagerDuty from the list.

    7. Enter a name for the credential and then paste the PagerDuty API key that you copied in the API Key field.

  2. Save an integration key:

    1. In PagerDuty, navigate to Automation > Event Rules.

    2. Click a ruleset on the page.

    3. If the integration key does not display on the ruleset page, click the text Incoming Event Source.

    4. Copy the integration key.

    5. In Moogsoft Cloud, paste the integration key into your PagerDuty credentials, and then click Save.

Update the email domain list in PagerDuty

If your PagerDuty account has an email domain restriction, add moogsoft.com to the list of allowed domains.

  1. In PagerDuty, click your user icon and navigate to Account Settings.

  2. Scroll down to the Email Domain Restriction section.

  3. If either of the boxes is checked, add moogsoft.com to the Email Domain Allow List.

  4. Click Update Email Domain Settings.

Add a rule to PagerDuty

For PagerDuty to accept Moogsoft Cloud data, you must have the following configured:

Add a PagerDuty event rule to process Moogsoft Cloud alerts and incidents:

  1. Navigate to Automation > Event Rules.

  2. Click the ruleset you selected in the previous procedure.

  3. On the ruleset page, click New Event Rule.

  4. In the section "When events match these conditions," select Any in the line "Run this rule when All/Any of the following conditions are met."

  5. Type payload into the first condition box and select exists for the condition.

  6. Click Do these things to open the section and then select Create an incident on a service.

  7. In the right-hand pane under Route to a Service, select the PagerDuty service where you want to route your Moogsoft Cloud data.

  8. Click Save Rule.

Configure the integration in Moogsoft Cloud

Set up the PagerDuty integration in Moogsoft Cloud:

  1. In Moogsoft Cloud, navigate to Data Config > Outbound Integrations.

  2. Click PagerDuty, and then click Add a PagerDuty Integration.

  3. Enter a name for the integration.

  4. Under Credentials, click the down arrow and select the PagerDuty credentials you added earlier.

    If you have not already added them, click Add New Credentials and create your credentials now.

  5. Under Data Type, select the data you want to synchronize with PagerDuty.

    Select Incidents, Alerts, or both.

  6. Optionally, configure a filter for the objects you want to synchronize with PagerDuty.

  7. Optionally, add an email address for the robot user. If you do not include an email address, the default moog@moogsoft.com will be used.

    For information on how the robot user works, see PagerDuty robot user.

  8. Select the scope type you want to use from the Type menu.

    Note that the v3 webhook using the Service scope type most closely replicates the behavior of the older v2 webhook.

  9. If you selected Team or Service for Type, you must provide the ID of the specific team or service. If you selected Account, skip to step 11.

    Find a team ID
    1. In PagerDuty, navigate to People > Teams and click the team to use with the integration.

    2. On the page that opens, the series of letters in the URL before /users is the ID. Copy the ID.

      For example, if the URL on the Team page in PagerDuty is:

      https://moogsoft.pagerduty.com/AAAAAAA/users

      the team ID is AAAAAAA.

    Find a service ID
    1. In PagerDuty, navigate to Services > Service Directory and click the service to use with the integration.

    2. On the page that opens, the final series of letters in the URL is the ID. Copy the ID.

      For example, if the URL on the Service page in PagerDuty is:

      https://moogsoft.pagerduty.com/service-directory/BBBBBBB

      the service ID is BBBBBBB.

  10. In Moogsoft Cloud, paste the ID from the previous step in the ID field in the integration.

  11. Click Test and then check to make sure that PagerDuty received the test message.

  12. Click Save.

Pager Duty rate limiting and notification errors

When using Pager Duty, you need to know about Pager Duty rate limiting and how that might have an impact on your use of Moogsoft Cloud notifications using Pager Duty. Understanding Pager Duty rate limiting is important to ensure Moogsoft can reliably raise pages to your on-call teams. Keep in mind the following:

  • The PagerDuty app rate limits the amount of requests Moogsoft Cloud is able to make per minute with our outbound notifications. You can read more about Pager Duty rate limits in the Pager Duty documentation topic on rate limiting.

  • If you send requests too frequently and get an error such as "429 Too Many Requests," you may need to wait for an interval of time, such as a second or more, before making further requests. Also, make any further requests at a reduced rate