Skip to main content

Send incidents to Datadog

Follow these steps to configure APEX AIOps Incident Management to send incidents to Datadog. For additional details, consult the APEX AIOps Incident Management Outbound alert webhooks (deprecated) documentation.

Before you begin

Before you set up your Datadog integration:

  • Make sure you are signed into your APEX AIOps Incident Management instance.

  • Make sure you are signed into Datadog.

  • Generate both an API Key and Application Key in Datadog by following these instructions.

Configure the CREATE webhook endpoint in APEX AIOps Incident Management

First, configure the Datadog CREATE webhook endpoint, which sends incidents to Datadog when they arrive in APEX AIOps Incident Management.

To configure the CREATE webhook endpoint:

  1. Log in to APEX AIOps Incident Management and navigate to Integrations > Outbound Integrations > Webhook Endpoint > Incident Webhook Endpoint.

  2. Click Add Incident Webhook Endpoint and enter "Datadog CREATE" as the name of the webhook endpoint. Optionally, provide a description.

  3. Complete each section of the webhook endpoint according to the following:

    1. Skip the Triggers section.

    2. Configure the Endpoint:

      1. Set the Request Method to POST.

      2. Under URL, copy and paste the following:

        https://api.datadoghq.com/api/v2/incidents

      3. Under Authorization, make sure No auth is selected.

      4. Under Headers, add a new row with the following parameters. Replace <your-api-key> with your Datadog API key:

        • Key: DD_API_KEY

        • Value: <your-api-key>

      5. Add another new row with the following parameters. Replace <your-application-key> with your Datadog application key:

        • Key: DD_APPLICATION_KEY

        • Value: <your-application-key>

    3. Specify the Payload Body:

      1. Replace the default payload with the following:

        {  "data": {
      "attributes": {
            "title": "Issue # $id with severity $severity and description $description",
			"fields": {
				"severity": {
					"value": "$severity"
				},
				"summary": {
				    "value": "Issue # $id with severity $severity and description $description"
				}
				}
			}
	},    "type": "incidents"
  }
}

        Note

        If you want to add a mapping for services, you can edit the payload above. The service values must be set up in the Datadog APM feature and must match the APEX AIOps Incident Management values exactly.

      2. Add a type declaration for ${severity}, with a type of String.

      3. Add a custom mapping for ${severity} as follows:

        Input Value

        Value to Send

        critical

        SEV-1

        major

        SEV-2

        minor

        SEV-3

        warning

        SEV-4

        clear

        SEV-5

    4. Save the webhook endpoint.

Configure the UPDATE webhook endpoint in APEX AIOps Incident Management

Next, create a Datadog UPDATE webhook endpoint, which sends updates to Datadog when APEX AIOps Incident Management incidents are changed.

To configure the UPDATE webhook endpoint:

  1. Navigate to Integrations > Outbound Integrations > Webhook Endpoint > Incident Webhook Endpoint.

  2. At the far right side of the "Datadog CREATE" webhook endpoint you just configured, click on the copy icon:

    SlackIntegrationCopyWebhook.png

  3. Enter "Datadog UPDATE" for the name of the duplicate webhook endpoint and click Duplicate.

  4. Click on the "Datadog UPDATE" webhook endpoint to open it.

  5. Click Edit.

  6. Under the Triggers section, select Severity increased.

  7. Configure the Endpoint:

    1. Set the Request Method to Patch.

    2. Replace the existing URL with the following:

      https://api.datadoghq.com/api/v2/incidents/$external_name

  8. Save the webhook endpoint.

Configure the CREATE workflow in APEX AIOps Incident Management

Configure an incident workflow called Datadog CREATE Workflow, which triggers the "Datadog CREATE" webhook endpoint when an incident is created in APEX AIOps Incident Management.

To configure the CREATE workflow:

  1. Log into APEX AIOps Incident Management and navigate to Correlate & Automate > Workflow Engine > Incident Workflows.

  2. Click Add Workflow and enter "Datadog CREATE Workflow" as the name of the workflow. Optionally, provide a description.

  3. For the Trigger, select New incidents only.

    If desired, add a filter condition on the incidents that you wish to forward to Datadog.

  4. Click Add Action, select the Send to Endpoint action and then click Add Selected Action.

  5. Select the new Send to Endpoint action complete each section according to the following:

    1. Webhook: Select the "Datadog CREATE" incident webhook endpoint that you created earlier.

    2. External Link: Skip this section.

    3. Integration Name: Enter the following (without quotes): "Datadog Integration"

      Skip all remaining sections.

  6. Save and Enable the workflow.

Configure the UPDATE workflow in APEX AIOps Incident Management

Finally, configure another incident workflow called Datadog UPDATE Workflow, which triggers the "Datadog UPDATE" webhook endpoint to send data when an incident is updated in APEX AIOps Incident Management.

To configure the UPDATE workflow:

  1. Navigate to Correlate & Automate > Workflow Engine > Incident Workflows.

  2. Click Add Workflow and enter "Datadog UPDATE Workflow" as the name of the workflow. Optionally, provide a description.

  3. For the Trigger, select Changed incidents only.

  4. Click Add Action, select the Send to Endpoint action and then click Add Selected Action.

  5. Select the new Send to Endpoint action and complete each section as follows:

    1. Webhook: Select the "Datadog UPDATE" incident webhook endpoint that you created earlier.

    2. External Link: Check the box.

    3. Integration Name: Enter the following: "Datadog Integration"

    4. External ID: Enter the following: data.id

    5. External Name: Enter the following: data.attributes.public_id

    6. URL: Enter the following: https://app.datadoghq.com/incidents/$external_name

  6. Save and Enable the workflow.

Test the integration

  1. Navigate to Integrations > Ingestion Services > Events API.

  2. Add your API key to the provided cURL command and run the command in an external terminal to send an event to APEX AIOps Incident Management.

  3. Verify that an incident has been created in APEX AIOps Incident Management.

  4. Within Datadog, verify that data from APEX AIOps Incident Management has arrived. You should also see the test notification(s) you sent earlier.

  5. If you have enabled updates and included Status changed as a trigger, close the incident in APEX AIOps Incident Management and verify in Datadog that an update has arrived.

In this section: