Moogsoft Docs

Configure the Splunk LAM

Splunk is used for application management, security, and compliance, as well as business and web analytics.

It captures, indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations.

See Splunk for UI configuration instructions.

Process Workflow

The workflow of gathering alerts from a Splunk server and publishing it to Moogsoft AIOps is as follows:

  1. The Splunk LAM reads the configuration from the splunk_lam.conf file.

  2. The Splunk Add-On push the alerts via the configured mechanism (http/https etc.) to the Splunk LAM in JSON format.

  3. The Splunk LAM parses the alerts and submits it to Extractor.

  4. The Extractor is responsible for handling JSON strings and extracting alerts from it.

  5. The alerts are parsed and converted into normalized Moogsoft AIOps alerts.

  6. The normalized alerts are then published to the Message Bus.

Installing the Splunk App in the Splunk Application

Note

The Add-On for Moogsoft AIOps is available on the Splunk Marketplace. You can download and install the Splunk Add-On from the Marketplace.

If you do not want to install it from the marketplace, then proceed as follows:

To install a Splunk Add-on:

2. Navigate to the bin folder of Splunk e.g. <splunk_home>/bin

3. Enter the following command:

 ./splunk install app <app path>/<appname.tar.gz>

<app path> is the path where Splunk Add-on is copied.

Restart Splunk:

 ./splunk restart

4. The Splunk Add-on is installed in the Splunk application. The App Splunk Add-On for Moogsoft is displayed on the Splunk application homepage.

Note

During installation, some warnings are displayed which can be ignored. These warnings are logged because of user information text in the Add-on fields. This text is for user information and does not hamper the working of Splunk Add-on. An example of error that can be ignored: 'Invalid key in stanza [Moog_Integration] in /opt/splunk/etc/apps/TA-Splunk-Moogsoft/default/alert_actions.conf, line 9: param.Severity (value: "Minor").

Note

Alternatively, the Splunk Add-On can also be installed by unzipping TA-Splunk-Moogsoft-1.8.1.tgz and copying the unzipped directory at the following location:

/opt/splunk/etc/apps

Note

The default path of the Splunk Add-on log is $MOOGSOFT_HOME/log/data-capture. The name of the log file is MOO.splunk_lam.log

Installing the Add-On on a Search Head Cluster using a Deployer

To deploy the add-on on the search head cluster:

  1. Copy the add-on TA-Splunk-Moogsoft-1.8.1.tgz to the location /opt/splunk/etc/shcluster/apps on the deployer.

  2. Untar the add-on, and then delete the TA-Splunk-Moogsoft-1.8.1.tgztar.

  3. Navigate to the bin directory in the Splunk directory.

  4. Run the following command:

    ./splunk apply shcluster-bundle --answer-yes -target <URI>:<management_port> -auth <username>:<password>

    Note

    The parameter -target specifies the URI and management port for any member of the cluster, for example, https://10.0.1.14:8089 You specify only one cluster member but the deployer pushes the add-on to all members. This parameter is required.

    The -auth parameter specifies credentials for the Deployer instance, for example, admin:password

  5. The add-on is deployed on the Search Head Cluster.

    Note

    By default, Splunk Add-On only supports HTTPs, if you want it to support both HTTP and HTTPs, then go to the following path:

    $SPLUNK_HOME/etc/apps/ TA-Splunk-Moogsoft/default/moogsoft.conf

    and open the moogsoft.conf file and change the value of enforce_https to false.

Configuring an Alert to forward events through the Add-On
  1. Open the Splunk console, for example http://localhost:8000/en-US/app/launcher/home

    Note

    If opening from a different machine, replace localhost with the hostname of the machine where Splunk is installed. Also, make entry of the server IP Address and hostname in the hosts file

  2. Enter the username and password. Click on Sign in. The Splunk Homepage opens.

  3. Click on Search & Reporting, then click on Alerts

  4. Click on an Alert from which you want to forward events to Moogsoft AIOps, then click on Edit > Edit Alert

  5. Navigate to Triggers Action, click Add Actions and select Moogsoft AIOps Alert Integration.

  6. Enter the URL along with the port of the Splunk LAM. Severity is by default set to "Minor" and can be changed by the user.

  7. Enter the certificate name here if SSL connection is enabled. For further information check the SSL Configuration section below.

  8. Click on Save.

The alerts are created from the log file, selected in the above procedure and sent to Splunk Add-On, the Add-On then sends the alerts to the Splunk LAM.

SSL Configuration

To configure SSL following configurations are required:

  1. Create a new folder. Open a command prompt and navigate to the newly created folder.

  2. Run the following command in the command prompt. A server.pem and a server.key file is generated in the above-created folder.

    openssl req -new -x509 -days 365 -nodes -subj "/C=''/ST=''/L=''/O='moogsoft'/OU=''/CN=localhost" -out server.pem -keyout server.key
    

    Note

    In the above command, for the part /CN=localhost, enter the hostname of the machine where Splunk LAM is running, instead of localhost

    Note

    Copy the generated certificates to the machine where Splunk LAM is running

  3. Enter the following parameters in the monitor section of the Splunk LAM:

    • Enter the port on which the SSL communication will be done in the field port i.e. 80201

    • Set the field use_ssl to true

    • Enter the path of the directory, where the server certificate is copied, in the path_to_ssl_files. E.g. "../config"

    • Enter the name of the Server certificate in the field ssl_key_filename. E.g. "server.key"

    • Enter the name of the Server certificate in the ssl_cert_filename. E.g. "server.pem"

    • Set the field use_client_certificates to false

    • Select TLSv1.2 in ssl_protocols

  4. Copy the Server.key and the server.pem files to the directory <splunk_home>/etc/apps/TA-Splunk-Moogsoft/bin.

  5. On the Splunk application homepage, click Search & Reporting, then click Alerts.

  6. Select Edit Alert from the Edit dropdown. The Edit Alert dialog opens.

  7. Navigate to the When triggered section and enter the pem certificate e.g. server.pem, also change the URL protocol to https.

    Note

    In the URL field, enter the hostname of the Splunk LAM instead of the IP address

The SSL is configured for Splunk.

Version Information

Add On Version

Tool Version

1.0 - 1.1

Splunk Enterprise version 6.5

1.2 - 1.4

Splunk Enterprise version 6.5 and 6.7

1.5 - 1.8.1

Splunk Enterprise version 6.5, 6.6 and 7.0

The Splunk LAM is used to communicate with the Splunk Add-On. It is a copy of the REST LAM and configurations available here is same as that of a REST LAM. Refer the REST LAM document on the available configurations. The configuration for the Splunk LAM is done in the splunk_lam.conf file. The default configurations in the Splunk LAM is as follows:Splunk LAM

Monitor section

The following section is the monitor section of the Splunk LAM

config :
    {
        monitor:
        {

            name                                : "Splunk Lam Monitor",

            class                               : "CRestMonitor",

            port                                : 48001,

            address                                     : "localhost",

            use_ssl                             : false,
        
            path_to_ssl_files                   : "config/",

            ssl_key_filename                    : "server.key",

            ssl_cert_filename                   : "server.pem",

            #use_client_certificates            : false,

            #client_ca_filename                 : "ca.crt",

            #auth_token                     : "my_secret",

            #encrypted_auth_token               : "dfJtTQMGiFHfiq7sCmxguBt6Jv+eytkoiKCquSB/7iWxpgGsG2aez3z2j7SuBtKj",

            #header_auth_token                  : "my_secret",

            #encrypted_header_auth_token        : "dfJtTQMGiFHfiq7sCmxguBt6Jv+eytkoiKCquSB/7iWxpgGsG2aez3z2j7SuBtKj",

            ssl_protocols                                       :"TLSv1.2", 
                                                                          #[
                                                                          #  "TLSv1.2"
                                                                          #]

            authentication_type                         : "none",

            authentication_cache                        : false,

            accept_all_json                     : true,

            lists_contain_multiple_events       : true,

            num_threads                         : 5,

            rest_response_mode                          : "on_receipt",

            rpc_response_timeout                        : 20,

            event_ack_mode                                      : "queued_for_processing"

Note

In the Monitor section, in the address field, enter the hostname of the machine where the Splunk LAM is running

Note

The port given in the port field is an optional value that defaults to 48001

For more information about the fields refer to the REST LAM document.

Agent and Process Log

The Agent and Process Log sections allow you to configure the following properties:

  • name: Identifies events the LAM sends to the Message Bus.

  • capture_log: Name and location of the LAM's capture log file.

  • configuration_file: Name and location of the LAM's process log configuration file.

Mapping

The following mapping section in the config file provides an example of mapping of the Splunk alert fields with the Moogsoft AIOps fields.

mapping:
    
                {   
                        catchAll:"overflow",
            rules:
            [
                { name: "signature", rule:      "$search_name" },
                { name: "source_id", rule:      "$result.sourcetype" },
                { name: "external_id", rule:    "$result.splunk_server" },
                { name: "manager", rule:        "Splunk" },
                { name: "source", rule:         "$result.host" },
                { name: "class", rule:          "$result.object" },
                { name: "agent", rule:          "$LamInstanceName" },
                { name: "agent_location", rule: "Splunk" },
                { name: "type", rule:           "$result.sourcetype" },
                { name: "severity", rule:       "0", conversion: "stringToInt" },
                { name: "description", rule:    "$result._raw" },
                { name: "agent_time", rule:     "$moog_now" }
            ]
        },
        filter:
        {
            presend: "SplunkLam.js"
        }

Data not mapped to Moogsoft AIOps Fields goes into "Custom Info".

Note

The signature field is used by the LAM to identify the correlated alerts. By default, here it is set to the "search_name" field. However, user can change it as per the requirement

Note

Variables section is not required in Splunk LAM; a user can directly map the alert fields of Splunk alerts with Moogsoft AIOps fields

Note

The Mapping section given here is an example, the user has to change the mapping according to the fields received in alerts/alarms from Splunk

Constant and Conversion

The following section is the constant and conversion of the Splunk LAM

constants:
        {
            severity:
            {
                "CLEAR"         : 0,
                "INDETERMINATE" : 1,
                "WARNING"       : 2,
                "MINOR"         : 3,
                "MAJOR"         : 4,
                "CRITICAL"      : 5
            }
        },
        conversions:
        {
            sevConverter:
            {
                lookup: "severity",
                input:  "STRING",
                output: "INTEGER"
            },
            stringToInt:
            {
                input:      "STRING",
                output:     "INTEGER"
            }
        },
Lambot Configuration

The Lambot SplunkLam.js handles the severity of alerts received from Splunk. The Severity can be changed according to the requirement of the customer. The code for severity determination is as follows:

var sev=overflow.configuration.Severity;
    logger.info("###########severity###############"+sev);
    if (sev === "MINOR" || sev === "Minor" || sev === "minor")
    {
        event.set("severity",3);
    }
    else if (sev === "MAJOR" || sev === "Major" || sev === "major")
    {
        event.set("severity",4);
    }
    else if (sev === "CRITICAL" || sev === "Critical" || sev === "critical")
    {
        event.set("severity",5);
    }
    else if (sev === "INFO" || sev === "Info" || sev === "info")
    {
        event.set("severity",1);
    }
    else if (sev === "WARNING" || sev === "Warning" || sev === "warning")
    {
        event.set("severity",2);
    }
    else if (sev === "CLEAR" || sev === "Clear" || sev === "clear")
    {
        event.set("severity",0);
    }
    else 
   {
    event.set("severity",3);    
   }

In the above code, the severity is extracted from an alert. The text of the severity is matched with a predefined text and based on the matched string the code corresponding to the Moogsoft AIOps severity is assigned to the alert and displayed in the GUI. In the above example the variable sev contains the severity from an alert if (sev === "MINOR" || sev === "Minor" || sev === "minor"), if it is a match then the Moogsoft AIOps severity code is assigned to it e.g. event.set("severity",3), the severity code passed to Moogsoft AIOps is "3". The code "3" in Moogsoft AIOps corresponds to "MINOR" and hence the "MINOR" is displayed in the GUI corresponding to the event.

The code and equivalent severity in Moogsoft AIOps is as follows:

  • CLEAR = 0,

  • INDETERMINATE = 1,

  • WARNING = 2,

  • MINOR = 3,

  • MAJOR = 4,

  • CRITICAL = 5

The user can change the severity comparison text in the if statement according to the severity text received from Splunk, and accordingly assign it a Moogsoft AIOps severity code.

Quotes

In some instances, the attribute strings are quoted. The JSON parser ignores it, but the standard requires quoting for all strings, so Moogsoft recommends that you quote all strings.

Comments

A user can comment out lines by prefixing them with a hash.

Starting the Splunk LAM

To start the Splunk LAM enter the following command:

service splunklamd start

To stop the Splunk LAM enter the following command:

service splunklamd stop

To view the status of Splunk LAM, enter the following command:

service splunklamd status

You can use a GET request to check the status of the Splunk LAM. See "Check the LAM Status" in the Configure the REST LAM for further information and examples.