Send notifications using PagerDuty

This integration enables you to receive just-in-time PagerDuty notifications when alerts or incidents get updated in APEX AIOps Incident Management. The Incident Management PagerDuty integration supports the v3 version of the PagerDuty API.

This is a bidirectional integration in which the two systems synchronize their alerts and incidents:

Incident Management notifies PagerDuty when it creates an alert or incident. PagerDuty then creates a new corresponding object.
Incident Management notifies PagerDuty whenever it updates an alert or incident. PagerDuty then updates the corresponding object.
The following events trigger an update in PagerDuty:
- Incident status update
- Incident severity update
- Alert status update
- Alert severity update
PagerDuty notifies Incident Management whenever an incident (derived from Incident Management ) gets updated. Incident Management then updates the corresponding object.

As part of the PagerDuty integration setup, the installation process creates a "robot" (dummy) user in PagerDuty. This user carries out specific tasks in PagerDuty that relate to keeping incident information synchronized between PagerDuty and Incident Management. There are three scenarios where the robot user is necessary:

When an alert is assigned to a user in Incident Management, this indirectly changes the status and assignment of a related incident. Because this change is indirect, there is no explicit user assignment, so the incident is assigned to the robot user in PagerDuty.
When a user in PagerDuty acknowledges an incident and is subsequently created in Incident Management because it does not already exist there, the robot user adds a note to the incident indicating that a corresponding new user was created in Incident Management.
When a user in Incident Management is assigned to an incident, PagerDuty attempts to create the same user and make the same assignment. If the user cannot be created in PagerDuty (as when the maximum number of users has been reached and no additional users can be created), the robot user stands in for the user in PagerDuty.

The robot user must have an email address assigned to it. By default, this is pagerduty@moogsoft.com. If your organization has restrictions around creating users with email addresses outside certain domains, you can customize the address for the robot user. The robot user receives the same types of email notifications that other users receive when they interact with incidents. However, because the scenarios where the robot user is necessary are limited, it is unlikely the robot user will receive many email notifications.

Warning

The PagerDuty robot user exists to keep Incident Management information in sync on PagerDuty for a limited number of automated scenarios. Do not use the robot user to acknowledge alerts and incidents, or to complete other tasks an actual user should perform.

See the PagerDuty documentation for details on PagerDuty components.

Before you begin

Important

The integration automatically creates a webhook in PagerDuty and creates a set of Incident Management credentials to allow PagerDuty to send information back to Incident Management.

Before you start to set up your integration, make sure that you have the necessary permissions to complete these tasks.

This integration was validated with PagerDuty on May 11, 2022.

Save your PagerDuty credentials in Incident Management

To send data to PagerDuty, you need the following credentials:

A PagerDuty API key with an access level of Full access
A PagerDuty integration key

Generate PagerDuty credentials and save them in Incident Management:

Important

The user who creates the API key must be an Admin or an Account Owner.

Create and save a PagerDuty API key:
1. In PagerDuty, navigate to Integrations > API Access Keys.
2. Click Create a New API Key.
3. Enter a description and then click Create Key.
  Copy the API key when it displays.
4. Check the table in API Access Keys. Make sure the access level is Full access.
5. In Incident Management, navigate to Settings > Credential Store.
6. Click Add Credential and then select PagerDuty from the list.
7. Enter a name for the credential and then paste the PagerDuty API key that you copied in the API Key field.
Save an integration key:
1. In PagerDuty, navigate to Automation > Event Rules.
2. Click a ruleset on the page.
3. If the integration key does not display on the ruleset page, click the text Incoming Event Source.
4. Copy the integration key.
5. In Incident Management, paste the integration key into your PagerDuty credentials, and then click Save.

Update the email domain list in PagerDuty

If your PagerDuty account has an email domain restriction, add moogsoft.com to the list of allowed domains.

In PagerDuty, click your user icon and navigate to Account Settings.
Scroll down to the Email Domain Restriction section.
If either of the boxes is checked, add moogsoft.com to the Email Domain Allow List.
Click Update Email Domain Settings.

Add a rule to PagerDuty

For PagerDuty to accept Incident Management data, you must have the following configured:

A PagerDuty service.
A PagerDuty ruleset with an event rule to process your data.

Add a PagerDuty event rule to process Incident Management alerts and incidents:

Navigate to Automation > Event Rules.
Click the ruleset you selected in the previous procedure.
On the ruleset page, click New Event Rule.
In the section "When events match these conditions," select Any in the line "Run this rule when All/Any of the following conditions are met."
Type payload into the first condition box and select exists for the condition.
Click Do these things to open the section and then select Create an incident on a service.
In the right-hand pane under Route to a Service, select the PagerDuty service where you want to route your Incident Management data.
Click Save Rule.

Configure the integration in Incident Management

Set up the PagerDuty integration in Incident Management:

In Incident Management, navigate to Integrations > Outbound Integrations.
Click PagerDuty, and then click Add a PagerDuty Integration.
Enter a name for the integration.
Under Credentials, click the down arrow and select the PagerDuty credentials you added earlier.
If you have not already added them, click Add New Credentials and create your credentials now.
Under Data Type, select the data you want to synchronize with PagerDuty.
Select Incidents, Alerts, or both.
Optionally, configure a filter for the objects you want to synchronize with PagerDuty.
To define and apply a filter, click in the Filter field and select the desired items from the pull-down menus. You can define lists and compound expressions such as:
- source = "10.11.67.54" AND service in [ "outlook365", "email"]
- service = myService AND severity in ( Major , Critical )
Optionally, add an email address for the robot user. If you do not include an email address, the default pagerduty@moogsoft.com will be used.
For information on how the robot user works, see PagerDuty robot user.
Select the scope type you want to use from the Type menu.
Note that the v3 webhook using the Service scope type most closely replicates the behavior of the older v2 webhook.
If you selected Team or Service for Type, you must provide the ID of the specific team or service. If you selected Account, skip to step 11.
Find a team ID
1. In PagerDuty, navigate to People > Teams and click the team to use with the integration.
2. On the page that opens, the series of letters in the URL before /users is the ID. Copy the ID.
  For example, if the URL on the Team page in PagerDuty is:
  https://moogsoft.pagerduty.com/AAAAAAA/users
  the team ID is AAAAAAA.
Find a service ID
1. In PagerDuty, navigate to Services > Service Directory and click the service to use with the integration.
2. On the page that opens, the final series of letters in the URL is the ID. Copy the ID.
  For example, if the URL on the Service page in PagerDuty is:
  https://moogsoft.pagerduty.com/service-directory/BBBBBBB
  the service ID is BBBBBBB.
In Incident Management, paste the ID from the previous step in the ID field in the integration.
Click Test and then check to make sure that PagerDuty received the test message.
Click Save.

PagerDuty rate limiting and notification errors

When using PagerDuty, you need to know about PagerDuty rate limiting and how that might have an impact on your use of Incident Management notifications using PagerDuty. Understanding PagerDuty rate limiting is important to ensure Incident Management can reliably raise pages to your on-call teams. Keep in mind the following:

The PagerDuty app rate limits the amount of requests Incident Management is able to make per minute with our outbound notifications. You can read more about PagerDuty rate limits in the PagerDuty documentation topic on rate limiting.
If you send requests too frequently and get an error such as "429 Too Many Requests," you may need to wait for an interval of time, such as a second or more, before making further requests. Also, make any further requests at a reduced rate

In this section:

Send notifications using PagerDuty

Warning

Before you begin

Important

Save your PagerDuty credentials in Incident Management

Important

Update the email domain list in PagerDuty

Add a rule to PagerDuty

Configure the integration in Incident Management

PagerDuty rate limiting and notification errors

Search results