Moogsoft Docs

Configure the Datadog Polling LAM

The Datadog Polling LAM allows you to retrieve alerts from Datadog. The Datadog Polling LAM is an HTTPS client that polls one or more Datadog servers at configurable intervals. It parses the JSON responses it receives into Moogsoft AIOps events.

You can install a basic Datadog Polling integration in the UI. See Datadog Polling for integration steps.

Configure the Datadog Polling LAM if you want to configure custom properties, set up high availability or configure advanced options that are not available in the UI integration.

Before You Begin

The Datadog Polling LAM has been validated with Datadog v2018. Before you set up the LAM, ensure you have met the following requirements for each Datadog server:

  • You know your Datadog server URL.

  • You know your Datadog API key and Application Key.

  • The port for your Datadog server is open and accessible from Moogsoft AIOps.

  • Your Datadog system can accept HTTPS requests.

Configure the LAM

Edit the configuration file to control the behavior of the Datadog Polling LAM. You can find the file at $MOOGSOFT_HOME/config/datadog_client_lam.conf

See the Datadog Polling Reference and LAM and Integration Reference for a full description of all properties. Some properties in the file are commented out by default. Uncomment properties to enable them.

  1. Configure the connection properties for each Datadog target:

    • url: Datadog request URL including host and port.

    • user: Datadog account user.

    • password or encrypted password: Datadog account password or encrypted password.

  2. Determine how to select and process Datadog events for each target:

    • enable_epoch_converter: You can use an epoch timestamp instead of a machine timestamp.

    • params_date_format: Date format to include in Datadog query.

    • request_query_params: SQL query to select Datadog events. See Datadog Polling Reference for an example.

    • overlap_identity_fields: List of payload tokens the LAM uses to identify duplicate events when Datadog returns all open events and not just updated events. Defaults to id.

    • requests_overlap: Period of time to delay processing duplicates.

    • results_path: Location of the JSON results objects in the data structure. Default to results.

  3. Configure the LAM behavior for each target:

    • request_interval: Length of time to wait between requests, in seconds.

    • timeout: Length of time to wait before halting a connection or read attempt, in seconds.

    • num_threads: Number of worker threads to use when processing events.

  4. Configure the SSL properties if you want to use SSL to encrypt communications between Datadog and Moogsoft AIOps:

    • disable_certificate_validation: Whether to disable SSL certificate validation.

    • path_to_ssl_files: Path to the directory that contains the SSL certificates.

    • server_cert_filename: Name of the SSL root CA file.

    • client_key_filename: Name of the SSL client key file.

    • client_cert_filename: Name of the SSL client certificate.

  5. If you want to connect to Datadog through a proxy server, configure the host, port, user, and password or encrypted password properties in the proxy section for the target.

  6. Optionally configure the LAM identification and logging details in the agent and log_config sections of the file:

    • name: Identifies events the LAM sends to the Message Bus.

    • capture_log: Name and location of the LAM's log file.

    • configuration_file: Name and location of the LAM's process log configuration file.

  7. Optionally configure severity conversions. See Severity Reference for further information and "Conversion Rules" in Data Parsing for details on conversions in general.

Example

You can configure the Datadog LAM to retrieve events from one or more targets. The following example demonstrates a configuration that targets two Datadog sources. For a single source, comment out the target2 section. If you have more than two sources, add a target section for each one and uncomment properties to enable them.

In the following example, the Datadog LAM is configured to poll two different Datadog instances. The LAM uses the tokens NodeID and EventID to identify duplicate events. These configurations specify use variables $from and $to for the query time window; the LAM specifies UNIX epoch values for these fields when it sends a poll request.

monitor:
{
   name                                    : "Datadog REST Client Monitor",
   class                                   : "CRestClientMonitor",
   request_interval                        : 60,
   targets:
   {
       target1:
       {
           url                             : "https://instance1.datadoghq.com/api/v1/events",
            proxy:
            {
                host                       : "localhost",
                port                       : 8181,
                user                       : "user",
                password                   : "pass",
                #encrypted_password        : "ieytOFRUdLpZx53nijEw0rOh07VEr8w9lBxdCc7229o="
            },
            request_interval               : 60,
            timeout                        : 120
            disable_certificate_validation : false,
            path_to_ssl_files              : "config",
            server_cert_filename           : "server.crt",
            client_key_filename            : "client.key",
            client_cert_filename           : "client.crt",
            requests_overlap               : 10,
            enable_epoch_converter         : true,
            results_path                   : "events",
            overlap_identity_fields        : [ "NodeID", "EventID" ],
            request_query_params:  
            {  
                start           : "$from",
                end             : "$to",
                api_key         : "1234",
                application_key : "1234"
            },
            params_date_format  : "%s"
       }
       target2:
       {
            url                            : "https://instance2.datadoghq.com/api/v1/events",
            user                           : "user2",
            host                           : "localhost",
            port                           : 8181,
            request_interval               : 60,
            timeout                        : 120
            disable_certificate_validation : false,
            path_to_ssl_files              : "config",
            server_cert_filename           : "server.crt",
            client_key_filename            : "client.key",
            client_cert_filename           : "client.crt",
            path_to_ssl_files              : "config",
            requests_overlap               : 10,
            enable_epoch_converter         : true,
            results_path                   : "events",
            overlap_identity_fields        : [ "NodeID", "EventID" ],
            request_query_params:  
            {  
                start           : "$from",
                end             : "$to",
                api_key         : "1234",
                application_key : "1234"
            },
            params_date_format  : "%s",
        }
        }
},
agent:
{
    name               : "Datadog Client",
    capture_log        : "$MOOGSOFT_HOME/log/data-capture/datadog_client_lam.log"
},
log_config:
{
    configuration_file : "$MOOGSOFT_HOME/config/logging/datadog_client_lam_log.json"
},
Configure for High Availability

Configure the Datadog LAM for high availability if required. See High Availability Overview for details.

Configure LAMbot Processing

The Datadog Polling LAMbot processes and filters events before sending them to the Message Bus. You can customize or bypass this processing if required. You can also load JavaScript files into the LAMbot and execute them.

See LAMbot Configuration for more information. An example Datadog Polling LAM filter configuration is shown below.

filter: 
{
    presend: "DatadogClientLam.js",
    modules: [ "CommonUtils.js" ]
}
Map LAM Properties

You can configure custom mappings in the Datadog Polling LAMbot. See Data Parsing information for details.

By default, the following DataDog event properties map to the following Moogsoft AIOps Datadog Polling LAM properties:

Datadog Event Property

Datadog LAM Event Property

$host:$device_name:$id:$source

signature

$device_name

source_id

$id

external_id

Datadog:$source

manager

$host

source

Datadog Event

class

$source

agent

$host

agent_location

$alert_type

severity

$title

description

$priority", conversion: "sevConverter"

severity

$date_happened

agent_time

The overflow properties are mapped to "custom info" and appear under custom_info in Moogsoft AIOps alerts. This mapping requires Event Monitor tag values in the correct format ({{event.tags.example-tag}}) as described in the Datadog documentation.

DataDog Event Property

Datadog LAM Overflow Property

url

eventDetails.url

is_aggregate

eventDetails.is_aggregate

tags

eventDetails.tags

resource

eventDetails.resource

priority

eventDetails.priority

text

eventDetails.text

value

eventDetails.value

threshold

eventDetails.threshold

Start and Stop the LAM

Restart the Datadog LAM to activate any changes you make to the configuration file or LAMbot.

The LAM service name is datadoglamd.

See Control Moogsoft AIOps Processesfor further details.

If the LAM fails to connect to one or more Datadog sources, Moogsoft AIOps creates an alert and writes the details to the process log. Refer to the logging details for LAMs and integrations for more information.