Splunk is used for application management, security, and compliance, as well as business and web analytics.
It captures, indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations.
The workflow of gathering alerts from a Splunk server and publishing it to Moogsoft AIOps is as follows:
- The Splunk LAM reads the configuration from the splunk_lam.conf file.
- The Splunk Add-On push the alerts via the configured mechanism (http/https etc.) to the Splunk LAM in JSON format.
- The Splunk LAM parses the alerts and submits it to Extractor.
- The Extractor is responsible for handling JSON strings and extracting alerts from it.
- The alerts are parsed and converted into normalized Moogsoft AIOps alerts.
- The normalized alerts are then published to MooMs bus.
Installing the Splunk App in the Splunk Application
The Add-On for Moogsoft AIOps is available on the Splunk Marketplace. You can download and install the Splunk Add-On from the Marketplace. Please follow the link:vhttps://splunkbase.splunk.com/app/3665/
If you do not want to install it from the marketplace, then proceed as follows:
To install a Splunk Add-on:
- Copy the Splunk Add-on TA-Splunk-Moogsoft-1.8.1.tgz to any directory on the server, where Splunk is installed.
2. Navigate to the bin folder of Splunk e.g. <splunk_home>/bin
3. Enter the following command:
<app path> is the path where Splunk Add-on is copied.
4. The Splunk Add-on is installed in the Splunk application. The App Splunk Add-On for Moogsoft is displayed on the Splunk application homepage.
During installation, some warnings are displayed which can be ignored. These warnings are logged because of user information text in the Add-on fields. This text is for user information and does not hamper the working of Splunk Add-on. An example of error that can be ignored: 'Invalid key in stanza [Moog_Integration] in /opt/splunk/etc/apps/TA-Splunk-Moogsoft/default/alert_actions.conf, line 9: param.Severity (value: "Minor").
Alternatively, the Splunk Add-On can also be installed by unzipping TA-Splunk-Moogsoft-1.8.1.tgz and copying the unzipped directory at the following location:
The default path of the Splunk Add-on log is /opt/splunk/var/log/splunk. The name of the log file is Moog_Integration.log
Installing the Add-On on a Search Head Cluster using a Deployer
To deploy the add-on on the search head cluster:
- Copy the add-on TA-Splunk-Moogsoft-1.8.1.tgz to the location /opt/splunk/etc/shcluster/apps on the deployer.
- Untar the add-on, and then delete the TA-Splunk-Moogsoft-1.8.1.tgz tar.
- Navigate to the bin directory in the Splunk directory.
Run the following command:
-targetspecifies the URI and management port for any member of the cluster, for example,
You specify only one cluster member but the deployer pushes the add-on to all members. This parameter is required.
-authparameter specifies credentials for the Deployer instance, for example, admin:password
The add-on is deployed on the Search Head Cluster.
Configuring an Alert to forward events through the Add-On
Open the Splunk console http://localhost:8000/en-US/app/launcher/home
If opening from a different machine, replace localhost with the hostname of the machine where Splunk is installed. Also, make entry of the server IP Address and hostname in the hosts file
- Enter the username and password. Click on Sign in. The Splunk Homepage opens.
- Click on Search & Reporting, then click on Alerts
- Click on an Alert from which you want to forward events to AIOps, then click on Edit > Edit Alert
- Navigate to Triggers Action, click Add Actions and select Moogsoft Alert Integration.
- Enter the URL along with the port of the Splunk LAM. Severity is by default set to "Minor" and can be changed by the user.
- Enter the certificate name here if SSL connection is enabled. For further information check the SSL Configuration section below.
- Click on Save.
The alerts are created from the log file, selected in the above procedure and sent to Splunk Add-On, the Add-On then sends the alerts to the Splunk LAM.
To configure SSL following configurations are required:
Create a new folder. Open a command prompt and navigate to the newly created folder.
Run the following command in the command prompt. A server.pem and a server.key file is generated in the above-created folder.
In the above command, for the part /CN=localhost, enter the hostname of the machine where Splunk LAM is running, instead of localhost
Copy the generated certificates to the machine where Splunk LAM is running
- Enter the following parameters in the monitor section of the Splunk LAM:
- Enter the port on which the SSL communication will be done in the field port i.e. 80201
- Set the field use_ssl to true
- Enter the path of the directory, where the server certificate is copied, in the path_to_ssl_files. E.g. "../config"
- Enter the name of the Server certificate in the field ssl_key_filename. E.g. "server.key"
- Enter the name of the Server certificate in the ssl_cert_filename. E.g. "server.pem"
- Set the field use_client_certificates to false
- Select TLSv1.2 in ssl_protocols
- Copy the Server.key and the server.pem files to the directory <splunk_home>/etc/apps/TA-Splunk-Moogsoft/bin.
- On the Splunk application homepage, click Search & Reporting, then click Alerts.
- Select Edit Alert from the Edit dropdown. The Edit Alert dialog opens.
Navigate to the When triggered section and enter the pem certificate e.g. server.pem, also change the URL protocol to https.
In the URL field, enter the hostname of the Splunk LAM instead of the IP address
The SSL is configured for Splunk.
|Add On Version||Tool Version|
|1.0 - 1.1||Splunk Enterprise version 6.5|
|1.2 - 1.4||Splunk Enterprise version 6.5 and 6.7|
|1.5 - 1.8.1||Splunk Enterprise version 6.5, 6.6 and 7.0|
The Splunk LAM is used to communicate with the Splunk Add-On. It is a copy of the REST LAM and configurations available here is same as that of a REST LAM. Refer the REST LAM document on the available configurations. The configuration for the Splunk LAM is done in the splunk_lam.conf file. The default configurations in the Splunk LAM is as follows:Splunk LAM
The following section is the monitor section of the Splunk LAM
For more information about the fields refer the REST LAM document.
The above example specifies:
- name: This is the agent name, the events sent to MooMs by Splunk are identified by the agent name. In this example the agent name is Splunk
- log: In this instance, the Splunk LAM will write its ingress contents to splunk_lam.log located at /var/log/moogsoft
The following mapping section in the config file provides an example of mapping of the Splunk alert fields with the Moogsoft AIOps fields.
The alarms/events are displayed in the Moogsoft AIOps, the data in the fields of the alarm or event mapped in the mapping section are shown in the respective columns of Moogsoft AIOps columns. The fields of alarms and events which are not mapped in the mapping section are displayed in the Custom Info field of the alarm. An example of Custom Info:
Constant and Conversion
The following section is the constant and conversion of the Splunk LAM
The Lambot SplunkLam.js handles the severity of alerts received from Splunk. The Severity can be changed according to the requirement of the customer. The code for severity determination is as follows:
In the above code, the severity is extracted from an alert. The text of the severity is matched with a predefined text and based on the matched string the code corresponding to the Moogsoft AIOps severity is assigned to the alert and displayed in the GUI. In the above example the variable sev contains the severity from an alert if (sev === "MINOR" || sev === "Minor" || sev === "minor"), if it is a match then the Moogsoft AIOps severity code is assigned to it e.g. event.set("severity",3), the severity code passed to Moogsoft AIOps is "3". The code "3" in Moogsoft AIOps corresponds to "MINOR" and hence the "MINOR" is displayed in the GUI corresponding to the event.
The code and equivalent severity in Moogsoft AIOps is as follows:
- CLEAR = 0,
- INDETERMINATE = 1,
- WARNING = 2,
- MINOR = 3,
- MAJOR = 4,
- CRITICAL = 5
The user can change the severity comparison text in the if statement according to the severity text received from Splunk, and accordingly assign it a Moogsoft AIOps severity code.
In some instances, the attribute strings are quoted. Our JSON parser ignores it, but the standard requires quoting for all strings, so Moogsoft recommends that user quote all strings.
A user can comment out lines by prefixing them with a hash.
Starting the Splunk LAM
To start the Splunk LAM enter the following command:
To stop the Splunk LAM enter the following command:
To view the status of Splunk LAM, enter the following command:
This LAM was tested on a system with the following configurations:
|Operating System||CentOS Linux release 6.7|
The system must at least have the above mentioned system requirements to run the LAM.