Page tree
Skip to end of metadata
Go to start of metadata

Overview

Splunk is used for application management, security, and compliance, as well as business and web analytics.

It captures, indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations.

Process Workflow

The workflow of gathering alerts from a Splunk server and publishing it to Moogsoft AIOps is as follows:

  1. The Splunk LAM reads the configuration from the splunk_lam.conf file.
  2. The Splunk Add-On push the alerts via the configured mechanism (http/https etc.) to the Splunk LAM  in JSON format.
  3. The Splunk LAM parses the alerts and submits it to Extractor.
  4. The Extractor is responsible for handling JSON strings and extracting alerts from it.
  5. The alerts are parsed and converted into normalized Moogsoft AIOps alerts.
  6. The normalized alerts are then published to MooMs bus.

Installing the Splunk App in the Splunk Application

The Add-On for Moogsoft AIOps is available on the Splunk Marketplace. You can download and install the Splunk Add-On from the Marketplace. Please follow the link :https://splunkbase.splunk.com/app/3665/

If you do not want to install it from the marketplace, then proceed as follows:

To install a Splunk Add-on:

  1. Copy the Splunk Add-on TA-Splunk-Moogsoft_v1_4.tar.gz to any directory on the server, where Splunk is installed.
  2. Navigate to the bin folder of Splunk e.g. <splunk_home>/bin
  3. Enter the following command:

     ./splunk install app <app path>/<appname.tar.gz>

    <app path> is the path where Splunk Add-on is copied.

  4. Restart Splunk:

     ./splunk restart

The Splunk Add-on is installed in the Splunk application. The App Splunk Add-On for Moogsoft is displayed on the Splunk application homepage.

During installation, some warnings are displayed which can be ignored. These warnings are logged because of user information text in the Add-on fields. This text is for user information and does not hamper the working of Splunk Add-on. An example of error that can be ignored: 'Invalid key in stanza [Moog_Integration] in /opt/splunk/etc/apps/TA-Splunk-Moogsoft/default/alert_actions.conf, line 9: param.Severity (value: "Minor").

Alternatively, the Splunk Add-On can also be installed by unzipping TA-Splunk-Moogsoft_v1_4.tar.gz and copying the unzipped directory at the following location:

/opt/splunk/etc/apps

The default path of the Splunk Add-on log is /opt/splunk/var/log/splunk. The name of the log file is Moog_Integration.log


Installing the Add-On on a Search Head Cluster using a Deployer

To deploy the add-on on the search head cluster:

  1. Copy the add-on TA-Splunk-Moogsoft_v1_4.tar.gz to the location /opt/splunk/etc/shcluster/apps on the deployer.

  2. Untar the add-on, and then delete the TA-Splunk-Moogsoft_v1_4.tar.gz tar.

  3. Navigate to the bin directory in the Splunk directory.

  4. Run the following command:

    ./splunk apply shcluster-bundle --answer-yes -target <URI>:<management_port> -auth <username>:<password>

    The parameter -target specifies the URI and management port for any member of the cluster, for example, https://10.0.1.14:8089 You specify only one cluster member but the deployer pushes the add-on to all members. This parameter is required.

    The -auth parameter specifies credentials for the Deployer instance, for example, admin:password

The add-on is deployed on the Search Head Cluster.

Configuring an Alert to forward events through the Add-On 

  1. Open the Splunk console http://localhost:8000/en-US/app/launcher/home

    If opening from a different machine, replace localhost with the hostname of the machine where Splunk is installed. Also, make entry of the server IP Address and hostname in the hosts file

  2. Enter the username and password. Click on Sign in. The Splunk Homepage opens.
  3. Click on Search & Reporting, then click on Alerts

  4. Click on an Alert from which you want to forward events to AIOps, then click on Edit > Edit Alert
  5. Navigate to Triggers Action, click Add Actions and select Moogsoft Alert Integration.
  6. Enter the URL along with the port of the Splunk LAM. Severity is by default set to "Minor" and can be changed by the user.
  7. Enter the certificate name here if SSL connection is enabled. For further information check the SSL Configuration section below.
  8. Click on Save.

The alerts are created from the log file, selected in the above procedure and sent to Splunk Add-On, the Add-On then sends the alerts to the Splunk LAM.

SSL Configuration

To configure SSL following configurations are required:

  1. Create a new folder. Open a command prompt and navigate to the newly created folder.

  2. Run the following command in the command prompt. A server.pem and a server.key file is generated in the above-created folder.

    openssl req -new -x509 -days 365 -nodes -subj "/C=''/ST=''/L=''/O='moogsoft'/OU=''/CN=localhost" -out server.pem -keyout server.key
    

    In the above command, for the part /CN=localhost, enter the hostname of the machine where Splunk LAM is running, instead of localhost

    Copy the generated certificates to the machine where Splunk LAM is running

  3. Enter the following parameters in the monitor section of the Splunk LAM:
    • Enter the port on which the SSL communication will be done in the field port i.e. 80201
    • Set the field use_ssl to true
    • Enter the path of the directory, where the server certificate is copied, in the path_to_ssl_files. E.g.  "../config"
    • Enter the name of the Server certificate in the field ssl_key_filename. E.g. "server.key"
    • Enter the name of the Server certificate in the ssl_cert_filename. E.g. "server.pem"
    • Set the field use_client_certificates to false
    • Select TLSv1.2 in ssl_protocols
  4. Copy the Server.key and the server.pem files to the directory <splunk_home>/etc/apps/TA-Splunk-Moogsoft/bin.
  5. On the Splunk application homepage, click  Search & Reporting, then click Alerts.
  6. Select Edit Alert from the Edit dropdown. The Edit Alert dialog opens.
     
  7. Navigate to the When triggered section and enter the pem certificate e.g. server.pem, also change the URL protocol to https.

    In the URL field, enter the hostname of the Splunk LAM instead of the IP address

The SSL is configured for Splunk.

Version Information

Add On Version

Tool Version

Tested?

Expected to Work

1.0

Splunk Enterprise version 6.6.2 

Yes

Yes

1.1

Splunk Enterprise version 6.6.2 

Yes

Yes

1.2

Splunk Enterprise version 6.6.2 

Yes

Yes

1.3

Splunk Enterprise version 6.6.2 

Yes

Yes

1.4

Splunk Enterprise version 6.6.2 

Yes

Yes

Splunk LAM

The Splunk LAM is used to communicate with the Splunk Add-On. It is a copy of the REST LAM and configurations available here is same as that of a REST LAM. Refer the REST LAM document on the available configurations. The configuration for the Splunk LAM is done in the splunk_lam.conf file. The default configurations in the Splunk LAM is as follows:

Monitor section

The following section is the monitor section of the Splunk LAM

config :
    {
        monitor:
        {

            name                    		: "Splunk Lam Monitor",

            class                   		: "CRestMonitor",

            port                    		: 48001,

            address                			: "localhost",

            use_ssl                 		: false,
	
            path_to_ssl_files       		: "config/",

            ssl_key_filename        		: "server.key",

            ssl_cert_filename       		: "server.pem",

            #use_client_certificates  		: false,

            #client_ca_filename      		: "ca.crt",

            #auth_token             	    : "my_secret",

            #encrypted_auth_token    		: "dfJtTQMGiFHfiq7sCmxguBt6Jv+eytkoiKCquSB/7iWxpgGsG2aez3z2j7SuBtKj",

            #header_auth_token           	: "my_secret",

            #encrypted_header_auth_token 	: "dfJtTQMGiFHfiq7sCmxguBt6Jv+eytkoiKCquSB/7iWxpgGsG2aez3z2j7SuBtKj",

            ssl_protocols					:"TLSv1.2", 
            								  #[
            								  #  "TLSv1.2"
            								  #]

            authentication_type				: "none",

            authentication_cache			: false,

            accept_all_json          		: false,

            lists_contain_multiple_events 	: true,

            num_threads              		: 5,

            rest_response_mode 				: "on_receipt",

            rpc_response_timeout			: 20,

            event_ack_mode 					: "queued_for_processing"
In the Monitor section, in the address field, enter the hostname of the machine where the Splunk LAM is running
The port given in the port field is an optional value that defaults to 48001


For more information about the fields refer the REST LAM document.

Agent 

agent:
	    {
	            name    : "Splunk"
	            #log    : "/var/log/moogsoft/splunk_lam.log"
	    },

The above example specifies: 

  • name: This is the agent name, the events sent to MooMs by Splunk are identified by the agent name. In this example the agent name is Splunk
  • log: In this instance, the Splunk LAM will write its ingress contents to splunk_lam.log located at /var/log/moogsoft

Mapping

The following mapping section in the config file provides an example of mapping of the Splunk alert fields with the Moogsoft AIOps fields.

mapping:
    
		{   
			catchAll:"overflow",
            rules:
            [
                { name: "signature", rule:      "$search_name" },
                { name: "source_id", rule:      "$result.sourcetype" },
                { name: "external_id", rule:    "$result.splunk_server" },
                { name: "manager", rule:        "Splunk" },
                { name: "source", rule:         "$result.host" },
                { name: "class", rule:          "$result.object" },
                { name: "agent", rule:          "$LamInstanceName" },
                { name: "agent_location", rule: "Splunk" },
                { name: "type", rule:           "$result.sourcetype" },
                { name: "severity", rule:       "0", conversion: "stringToInt" },
                { name: "description", rule:    "$result._raw" },
                { name: "agent_time", rule:     "$moog_now" }
            ]
        },
        filter:
        {
            presend: "SplunkLam.js"
        }
The signature field is used by the LAM to identify the correlated alerts. By default, here it is set to the "search_name" field. However, user can change it as per the requirement
Variables section is not required in Splunk LAM; a user can directly map the alert fields of Splunk alerts with moogsoft fields
The Mapping section given here is an example, the user has to change the mapping according to the fields received in alerts/alarms from Splunk

Custom Info

The alarms/events are displayed in the Moogsoft AIOps, the data in the fields of the alarm or event mapped in the mapping section are shown in the respective columns of Moogsoft AIOps columns. The fields of alarms and events which are not mapped in the mapping section are displayed in the Custom Info field of the alarm. An example of Custom Info:

Constant and Conversion

The following section is the constant and conversion of the Splunk LAM

constants:
        {
            severity:
            {
                "CLEAR"         : 0,
                "INDETERMINATE" : 1,
                "WARNING"       : 2,
                "MINOR"         : 3,
                "MAJOR"         : 4,
                "CRITICAL"      : 5
            }
        },
        conversions:
        {
            sevConverter:
            {
                lookup: "severity",
                input:  "STRING",
                output: "INTEGER"
            },
            stringToInt:
            {
                input:      "STRING",
                output:     "INTEGER"
            }
        },

Lambot Configuration

The Lambot SplunkLam.js handles the severity of alerts received from Splunk. The Severity can be changed according to the requirement of the customer. The code for severity determination is as follows:

var sev=overflow.configuration.Severity;
    logger.info("###########severity###############"+sev);
    if (sev === "MINOR" || sev === "Minor" || sev === "minor")
    {
        event.set("severity",3);
    }
    else if (sev === "MAJOR" || sev === "Major" || sev === "major")
    {
        event.set("severity",4);
    }
    else if (sev === "CRITICAL" || sev === "Critical" || sev === "critical")
    {
        event.set("severity",5);
    }
    else if (sev === "INFO" || sev === "Info" || sev === "info")
    {
        event.set("severity",1);
    }
    else if (sev === "WARNING" || sev === "Warning" || sev === "warning")
    {
        event.set("severity",2);
    }
    else if (sev === "CLEAR" || sev === "Clear" || sev === "clear")
    {
        event.set("severity",0);
    }
    else 
   {
    event.set("severity",3);    
   }

In the above code, the severity is extracted from an alert. The text of the severity is matched with a predefined text and based on the matched string the code corresponding to the Moogsoft AIOps severity is assigned to the alert and displayed in the GUI. In the above example the variable sev contains the severity from an alert if (sev === "MINOR" || sev === "Minor" || sev === "minor"), if it is a match then the Moogsoft AIOps severity code is assigned to it e.g. event.set("severity",3), the severity code passed to Moogsoft AIOps is "3". The code "3" in Moogsoft AIOps corresponds to "MINOR" and hence the "MINOR" is displayed in the GUI corresponding to the event. 

The code and equivalent severity in Moogsoft AIOps is as follows:

  • CLEAR = 0,
  • INDETERMINATE = 1,
  • WARNING = 2,
  • MINOR = 3,
  • MAJOR = 4,
  • CRITICAL = 5

The user can change the severity comparison text in the if statement according to the severity text received from Splunk, and accordingly assign it a Moogsoft AIOps severity code.

Quotes

In some instances, the attribute strings are quoted. Our JSON parser ignores it, but the standard requires quoting for all strings, so Moogsoft recommends that user quote all strings.

Comments

A user can comment out lines by prefixing them with a hash. 

Starting the Splunk LAM

To start the Splunk LAM enter the following command:

service splunklamd start

To stop the Splunk LAM enter the following command:

service splunklamd stop


To view the status of Splunk LAM, enter the following command:

service splunklamd status

System Information

This LAM was tested on a system with the following configurations:

CPU2 core
RAM4 GB
Operating SystemCentOS Linux release 6.7

The system must at least have the above mentioned system requirements to run the LAM.

  • No labels