Configure the Datadog Polling LAM

The Datadog Polling LAM allows you to retrieve alerts from Datadog. The Datadog Polling LAM is an HTTPS client that polls one or more Datadog servers at configurable intervals. It parses the JSON responses it receives into Moogsoft AIOps events.

You can install a basic Datadog Polling integration in the UI. See Datadog Polling for integration steps.

Configure the Datadog Polling LAM if you want to configure custom properties, set up high availability or configure advanced options that are not available in the UI integration.

Before You Begin

The Datadog Polling LAM has been validated with Datadog v2018. Before you set up the LAM, ensure you have met the following requirements for each Datadog server:

  • You know your Datadog server URL.

  • You know your Datadog API key and Application Key.

  • The port for your Datadog server is open and accessible from Moogsoft AIOps.

  • Your Datadog system can accept HTTPS requests.

Configure the LAM

Edit the configuration file to control the behavior of the Datadog Polling LAM. You can find the file at $MOOGSOFT_HOME/config/datadog_client_lam.conf

See the Datadog Polling LAM Reference and LAM and Integration Reference for a full description of all properties. Some properties in the file are commented out by default. Uncomment properties to enable them.

  1. Configure the connection properties for each Datadog target:

    • url: Datadog request URL including host and port.

    • user: Datadog account user.

    • password or encrypted_password: Datadog account password or encrypted password.

  2. Configure the LAM behavior for each target:

    • request_interval: Length of time to wait between requests, in seconds.

    • timeout: Length of time to wait before halting a connection or read attempt, in seconds.

    • num_threads: Number of worker threads to use when processing events.

  3. Configure the SSL properties if you want to encrypt communications between the LAM and the REST connection:

    • disable_certificate_validation: Whether to disable SSL certificate validation.

    • path_to_ssl_files: Path to the directory that contains the SSL certificates.

    • ssl_key_filename: SSL server key file.

    • ssl_cert_filename: SSL root CA file.

    • ssl_protocols: Sets the allowed SSL protocols.

  4. If you want to connect through a proxy server, configure the host, port, user, and password or encrypted password properties in the proxy section for the target.

  5. Optionally configure the LAM identification and capture logging details:

    • name: Maps to $Laminstancename, so that the agent field indicates events Moogsoft AIOps ingests from this LAM.

    • capture_log: Name and location of the LAM's capture log file, which it writes to for debugging purposes.

  6. Optionally configure severity conversions. See Severity Reference for further information and "Conversion Rules" in Data Parsing for details on conversions in general.

  7. Optionally configure the process logging details:

    • configuration_file: Name and location of the LAM's process log configuration file. See Configure Logging for more information.

Example

You can configure the Datadog LAM to retrieve events from one or more targets. The following example demonstrates a configuration that targets two Datadog sources. For a single source, comment out the target2 section. If you have more than two sources, add a target section for each one and uncomment properties to enable them.

In the following example, the Datadog LAM is configured to poll two different Datadog instances. The LAM uses the tokens NodeID and EventID to identify duplicate events. These configurations specify use variables $from and $to for the query time window; the LAM specifies UNIX epoch values for these fields when it sends a poll request.

monitor:
{
    name: "Datadog REST Client Monitor",
    class: "CRestClientMonitor",
    request_interval: 60,
    targets:
    {
        target1:
        {
            url: "https://instance1.datadoghq.com/api/v1/events",
            proxy:
            {
                host: "localhost",
                port: 8181,
                user: "user",
                password: "pass",
                #encrypted_password: "ieytOFRUdLpZx53nijEw0rOh07VEr8w9lBxdCc7229o="
            },
            request_interval: 60,
            timeout: 120
            disable_certificate_validation: false,
            path_to_ssl_files: "config",
            server_cert_filename: "server.crt",
            client_key_filename: "client.key",
            client_cert_filename: "client.crt",
            requests_overlap: 10,
            enable_epoch_converter: true,
            results_path: "events",
            overlap_identity_fields: [ "NodeID", "EventID" ],
            request_query_params:  
            {  
                start: "$from",
                end: "$to",
                api_key: "1234",
                application_key: "1234"
            },
            params_date_format: "%s"
        }
        target2:
        {
            url: "https://instance2.datadoghq.com/api/v1/events",
            user: "user2",
            host: "localhost",
            port: 8181,
            request_interval: 60,
            timeout: 120
            disable_certificate_validation: false,
            path_to_ssl_files: "config",
            server_cert_filename: "server.crt",
            client_key_filename: "client.key",
            client_cert_filename: "client.crt",
            path_to_ssl_files: "config",
            requests_overlap: 10,
            enable_epoch_converter: true,
            results_path: "events",
            overlap_identity_fields: [ "NodeID", "EventID" ],
            request_query_params:  
            {  
                start: "$from",
                end: "$to",
                api_key: "1234",
                application_key: "1234"
            },
            params_date_format: "%s",
        }
    }    
},
agent:
{
    name: "Datadog Client",
    capture_log: "$MOOGSOFT_HOME/log/data-capture/datadog_client_lam.log"
},
log_config:
{
    configuration_file: "$MOOGSOFT_HOME/config/logging/datadog_client_lam_log.json"
},

Configure for High Availability

Configure the Datadog LAM for high availability if required. See High Availability Overview for details.

Configure the LAMbot Processing

The Datadog Polling LAMbot processes and filters events before sending them to the Message Bus. You can customize or bypass this processing if required. You can also load JavaScript files into the LAMbot and execute them.

See LAMbot Configuration for more information. An example Datadog Polling LAM filter configuration is shown below.

filter: 
{
    presend: "DatadogClientLam.js",
    modules: [ "CommonUtils.js" ]
}

Map LAM Properties

You can configure custom mappings in the Datadog Polling LAMbot. See Data Parsing information for details.

By default, the following DataDog event properties map to the following Moogsoft AIOps Datadog Polling LAM properties:

Datadog Event Property

Datadog LAM Event Property

$host:$device_name:$id:$source

signature

$device_name

source_id

$id

external_id

Datadog:$source

manager

$host

source

Datadog Event

class

$source

agent

$host

agent_location

$alert_type

severity

$title

description

$priority", conversion: "sevConverter"

severity

$date_happened

agent_time

The overflow properties are mapped to "custom info" and appear under custom_info in Moogsoft AIOps alerts. This mapping requires Event Monitor tag values in the correct format ({{event.tags.example-tag}}) as described in the Datadog documentation.

DataDog Event Property

Datadog LAM Overflow Property

url

eventDetails.url

is_aggregate

eventDetails.is_aggregate

tags

eventDetails.tags

resource

eventDetails.resource

priority

eventDetails.priority

text

eventDetails.text

value

eventDetails.value

threshold

eventDetails.threshold

Start and Stop the LAM

Restart the Datadog LAM to activate any changes you make to the configuration file or LAMbot.

The LAM service name is datadogclientlamd.

See Control Moogsoft AIOps Processesfor further details.

If the LAM fails to connect to one or more Datadog sources, Moogsoft AIOps creates an alert and writes the details to the process log. Refer to the logging details for LAMs and integrations for more information.