Page tree
Skip to end of metadata
Go to start of metadata

Overview

The Rest Client LAM is an HTTP client LAM and makes use of a REST API to request event data and ingest it into Moogsoft AIOps. It sends HTTP requests to the REST server at configurable intervals and parses the JSON responses received from the server and processes events from the responses.

The REST Client LAM initiates a "pull" request to a REST API and in response it receives events in JSON format (e.g. Solarwinds, Zabbix, etc.)

The Rest Client LAM ingests event data from RESTful Services. The following information explains the configuration required to do this.

Requirements

The ingestion of event data from RESTful Services requires Moogsoft AIOps to be running a Rest Client LAM, configured to parse JSON responses from RESTful Services. It also requires account credentials. These credentials are specified in the RestClient LAM configuration file rest_client_lam.conf

The REST Client LAM processes the events received from the REST API based on the configurations done in the following 2 files:

File
Description
$MOOGSOFT/config/rest_client_lam.confRest Client LAM configuration file
$MOOGSOFT/bots/lambots/RestClientLam.jsLAMBot file; processes the event data received from the event source

The user has to configure the above 2 files as per their requirements based on the event format received from the REST API.

The performance of Moogsoft AIOps depends on the number of events received per second and the specifications of the Moogsoft AIOps system on which the Rest Client LAM is running


Rest Client LAM configuration

The configuration file contains a JSON object. At the first layer of the object, the LAM has a parameter called config, and the object that follows config has all the necessary information to control the LAM.

The following sections are available for configuration in the rest_client_lam.Conf file.

Monitor

The REST Client LAM takes the connection information from the Monitor section of the config file. An example of Monitor section is as follows:

config :
    {
        monitor:
        {

            name                            : "REST Client Monitor",

            class                           : "CRestClientMonitor",

            request_interval                : 60,

            max_intervals                   : 10,

            user                            : "username",

            password                        : "password",

            #encrypted_password             : "ieytOFRUdLpZx53nijEw0rOh07VEr8w9lBxdCc7229o=",


			enable_proxy 				    : false,

            proxy_host                      : "",

            proxy_port                      : ,

			proxy_user                      : "",

			#encrypted_proxy_password       : "",

            proxy_password                  : "",

			use_ssl                   		: false,
            
			disable_certificate_validation  : true,

            path_to_ssl_files               : "config",
            server_cert_filename      		: "server.crt",

            use_client_authentication 		: false,

			client_key_filename       		: "client.key",
            
			client_cert_filename      		: "client.crt",

            requests_overlap                : 10,

            url                             : "",

            request_query_params            : {
											 	get : "events",
                							 	start: "$from",
                								end: "$to"            
            								  },

            params_date_format              : "%Y-%m-%dT%H:%M:%S",

			enable_epoch_converter          : false,

            results_as_list                 : false,

            results_path                    : "results",

            overlap_identity_fields         : ["signature", "source_id", "agent_time", "description"]

            #num_threads                    : 5
        },
  • name and class: These fields are reserved and should not be changed. The default values are REST Client Monitor and CRestClientMonitor respectively
  • request_interval: This is the time interval between 2 successive http requests. For example, if 30 seconds is entered here, and the REST Client LAM is started, then after the first request it will wait for 30 seconds and then send the second request
  • max_intervals: This is the count that the REST Client LAM will try to send an http request, in the case of a failure of an http request

    Example: If 3 is entered here and the request _interval is set to 120 seconds, then the LAM sends a request every two minutes (120 seconds), and the maximum number of intervals is set to 3. Therefore the maximum time to request data for if previous requests fail is (3 x 2) six minutes. So if the LAM starts at 09:00, requests are as follows:

    09:02 - Request event data from 09:00 to 09:02... success
    09:04 - Request event data from 09:02 to 09:04... fail
    09:06 - Request event data from 09:02 to 09:06 (asks for data from the previously failed request)... fail
    09:08 - Request event data from 09:02 to 09:08... fail
    09:10 - Request event data from 09:04 to 09:10 (the maximum time to request data for is reached, so data from the last six minutes is requested)... fail
    09:12 - Request event data from 09:06 to 09:12 (data from the last six minutes is requested)... success
    09:14 - Request event data from 09:12 to 09:14 (previous request successful, reverts to requesting event data from the previous two minutes)

  • user: The username to access the REST server

  • password: The password for the user accessing the REST server
  • encrypted_password: If the user wants to use an encrypted password, then encrypt the password and enter it in this field and comment the password field. At a time either password or the encrypted_password field is used. If both the fields are not commented then the field encrypted_password will be used by the REST Client LAM

  • enable_proxy: If communication is to be established through a proxy, then enter true here
  • proxy_host:  Enter the hostname or IP address of the proxy

  • proxy_port:  Enter the port number of the proxy

  • proxy_user: Enter the user name of the proxy

  • encrypted_proxy_password: If the proxy password is not to be displayed in the config file, then encrypt the proxy password and enter here

  • proxy_password: Enter the password corresponding to the user name entered

  • use_ssl: Set it to true to enable SSL communication

  • disable_certificate_validation: Set to false if the SSL certificate for the event server is valid (default setting is true to disable SSL certificate validation for the event server)

  • path_to_ssl_files: The path where the SSL certificates is copied to on the machine where the REST Client LAM is running

  • server_cert_filename: Enter the server certificate name here. Use the certificate "server.crt" here. The cert file should be present in the directory given in path_to_ssl_files field

  • use_client_authentication: Enter true here if you want client authentication, otherwise set it to false. By default, it is set to false. If it is set to true, then the values are to be entered in the client_key_filename and the client_cert_filename fields.

  • client_key_filename: Enter the name of the key file here, e.g. "client.key".  The key file should be present in the directory given in path_to_ssl_files field.

  • client_cert_filename: Enter the name of the certificate file here, e.g. "client.crt". The cert file should be present in the directory given in path_to_ssl_files field.

  • requests_overlap: How many seconds more than the request_interval (see above) does the LAM wait to request data for. This is to ensure events are not missed

    Example: If request interval is set to 120 seconds and request_overlap is set to 10, then the LAM sends a request every two minutes (120 seconds). The overlap is set to ten seconds, so each request asks for data from the last two minutes and ten seconds

  • url: A URL for the endpoint to pull events from (including hostname or IP Address). If IP address along with port is entered in the field, then append a "/" in the end, e.g. http://localhost:8896/

  • request_query_params: The request query parameters. All the members of this map will be added to the request URL as URL encoded, so the URL will look like url?key1=value1&key2=value2...

    Example1: The following code block provides a sample request query

    request_query_params     :
    {
        get : "events",
        myInt: 12345,
        myString: "endPoint",
        myVersion: 3
    },

    The URL for the above request query will be url?get=events&myInt= 12345 &myString=endPoint&myVersion= 3

    Example2: The following code block provides a sample request query containing a to and from parameter

    request_query_params     :
    {
        get : "events",
        from: "$from",
        to: "$to"
    },

    The default 'request_query_params block' includes optional 'to' and 'from' variables which are special internally calculated timestamps worked out based on the request_interval and the request_overlap compared to the current system time. These timestamps variables can be used in the request_query_params block by prefixing them with a dollar sign. The timestamp formats is governed by the field params_date_format.


  • params_date_format: The format of timestamps that are sent in the request URL is entered here. E.g. "%Y-%m-%dT%H:%M:%S"

  • enable_epoch_converter: If set to true, the time stamps in the request URL are converted to epoch time

  • results_as_list: Set it to true if the events will be received in a list, otherwise set it to false

  • results_path: If the events in the response JSON are not in a list, then this will identify where in the JSON blob to find the event JSON. E.g. "results.subObject.events"

  • overlap_identity_fields: If we set the requests_overlap value, we might get the same event twice, setting this parameter will tell the LAM how to identify the duplicate. Duplicate events will be ignored in the second interval. E.g. [ "signature" , "source_id" , "agent_time" , "description" ]

  • num_threads: The number of threads the REST Client LAM will use

Agent

Agent allows the user to define two parameters:

agent:
	    {
	            name    : "RestClientLam"
                # log    : "/var/log/moogsoft/rest_client_lam.log"
	    },


The above example specifies:

name: This is the agent name, the events sent to MooMs by the REST Client LAM are identified by the agent name in the log. In this example the agent name is RestClientLam

log: In this instance, the REST Client LAM will write its ingress contents in the file rest_client_lam.log located at /var/log/moogsoft/

Mapping

For events received in JSON format, a user can directly map the alarm/event fields of REST Client LAM with Moogsoft fields. In the case of an event received in text format, the event is first tokenized in the Variable section, and the tokenized event is then mapped here in the mapping section. The parameters of the received alarm/event are displayed in Moogsoft AIOps according to the mapping done here.

mapping :
        {
            catchAll: "overflow",
            rules:
			[
                { name: "signature", rule:      "$signature" },
                { name: "source_id", rule:      "$source_id" },
                { name: "external_id", rule:    "$external_id" },
                { name: "manager", rule:        "$manager" },
                { name: "source", rule:         "$source" },
                { name: "class", rule:          "$class" },
                { name: "agent", rule:          "$agent" },
                { name: "agent_location", rule: "$host" },
                { name: "type", rule:           "$eventType" },
                { name: "description", rule:    "$description" },
                { name: "severity", rule:       "$severity" },
                { name: "agent_time", rule:     "$time", conversion: "timeConverter" }
            ]
        },
        filter:
        {
            presend: "RestClientLam.js"
        }


The above example specifies the mapping of the REST Server alarm fields with the Moogsoft AIOps fields.

The signature field is used by the LAM to identify correlated alarms

The above mapping section is an example, the user has to modify this section according to the format of the received event/alert

Conversions

Conversions section allows the user to convert the format of the received data.

        {
       	   severity:
           {
               "CLEAR"         : 0,
               "INDETERMINATE" : 1,
               "WARNING"       : 2,
               "MINOR"         : 3,
               "MAJOR"         : 4,
               "CRITICAL"      : 5
           }
        },

        conversions:
        {
            timeConverter:
            {
                timeFormat: "%Y-%m-%dT%H:%M:%S",
                input:      "STRING",
                output:     "INTEGER"
            },
	    sevConverter:
            {
                lookup: "severity",
                input:  "STRING",
                output: "INTEGER"
            }           
        },

The above example specifies:

  • Severity and sevConverter: The severity field has a conversion defined as sevConverter in the Conversions section, this looks up the value of severity defined in the severity section of constants and returns back the mapped integer corresponding to the severity
  • timeConverter: It is used in the conversion of the time stamp of the received events in the required format

catchALL

The attribute that is never referenced in a rule is collected and placed as a JSON object in a variable called overflow and passed as part of the event.

State File

The REST Client LAM saves the last poll time of the request sent to the REST Server. This last poll time is saved in a state file. The REST Client LAM reads/writes the last poll time from/to this state file. The LAM reads/writes from the state file by using the RestClientLam.js file. To read and write the state file the RestClientLam.js has to be modified.

The state file is created at the same location where the rest_client_lam.conf file resides. The name of the state file will be same as the config file name. For example, if the name of the config file is rest_client_lam.config, then the corresponding state file name will be rest_client_lam.state

LamBot Configuration

The LamBot processes the event data received from RESTful Services. The LamBot RestClientLam.js script can be configured for GET and POST requests.
The following example shows a LamBot configuration for GET request. This configuration primarily reads/writes the last poll time data from/to the state file.


var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime = true;
function onLoad()
{
	/* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. 
    Last poll time can be appended in the url for polling from the lass poll time;
	*/
	constants.load();
    return;
}

function preClientSend(outboundEvent)
{
  /* outBoundEvent contains below field which can be manipulated as per requirement
    	1. method - by default method is set to GET request
    	2. body - contains all the values of request_query_param from the config file
    	3. header - can contain additional headers
  */
    return true;
}

function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{	
    return true;
}

function presend(event)
{        
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");


The following example shows the Lambot configuration for a POST request with a JSON payload in which the last poll time is being written and read from the state file.

var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime= true; 

function onLoad()
{
	/* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. 
    Last poll time can be appended in the url for polling from the lass poll time;
	*/	
	constants.load();
	return;
}
function preClientSend(outBoundEvent)
 {
  /*outBoundEvent contains below field which can be manipulated as per requirement
    	1. method - by default method is set to GET request
    	2. body - contains all the values of request_query_param from the config file
    	3. header - can contain additional headers
  */
    var body = outBoundEvent.value('body'); // reads body from outBoundEvent
	if(firstTime)
	{
		if(constants.contains("start") == true)
		{
			var start = constants.get("start"); // reads the last poll time value from the state file
		
			body['start'] = start; // replaces value of the start in body with a new value
			outBoundEvent.set('body',body); // resets value of body with modified body
			constants.save();
			firstTime = false;
		}
		else
		{
			constants.put("start",body['start']);
			constants.save();
			firstTime = false;
		}
	}
	else
	{
		constants.put("start",body['start']);
		constants.save();
	}
// methods that can be used here are'Post','Put',and 'Delete';	
	outBoundEvent.set("method","Post"); 
	
	var properties = outBoundEvent.value("header"); 
// Header is used if the method used is Put or Delete or Post. Content type should always be defined otherwise it will give an error	
	properties["Content-Type"] = "application/json";
	outBoundEvent.set("header",properties);


//If LAM is not able to convert the request_query_parm in an accepted JSON format, then the user can force the LAM to use the assigned inputs given below as a     string, and use it in body of the request.
	
    //outboundEvent.set("body", 
    //'{"action":"EventsRouter","method":"query","data":[{"params":{"severity":[5,4,3,2,1,0],"eventState":[0,1,2,3,4,5]},"limit":9}],"type":"rpc","tid":1}');
    return true;
}


function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{
    return true;
}
function presend(event)
{

// returning true, makes this an event on the MooMs bus, false will cause the LAM to discard the event.
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");


The following example shows the Lambot configuration for a POST request with an XML payload in which the last poll time is being written and read from the state file.

var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime= true; 

function onLoad()
{
	/* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. 
    Last poll time can be appended in the url for polling from the lass poll time;
	*/	
	constants.load();
	return;
}
function preClientSend(outBoundEvent)
 {
  /*outBoundEvent contains below field which can be manipulated as per requirement
    	1. method - by default method is set to GET request
    	2. body - contains all the values of request_query_param from the config file
    	3. header - can contain additional headers
  */
    var body = outBoundEvent.value('body'); // reads body from outBoundEvent
	if(firstTime)
	{
		if(constants.contains("start") == true)
		{
			var start = constants.get("start"); // reads the last poll time value from the state file
		
			body['start'] = start; // replaces value of the start in body with a new value
			outBoundEvent.set('body',body); // resets value of body with modified body
			constants.save();
			firstTime = false;
		}
		else
		{
			constants.put("start",body['start']);
			constants.save();
			firstTime = false;
		}
	}
	else
	{
		constants.put("start",body['start']);
		constants.save();
	}
// methods that can be used here are'Post','Put',and 'Delete';	
	outBoundEvent.set("method","Post"); 
	
	var properties = outBoundEvent.value("header"); 
// Header is used if the method used is Put or Delete or Post. Content type should always be defined otherwise it will give an error
// The LAM cannot by default send an XML payload. To send an XML payload then set content type as text/xml and set the xml payload in the body	
	properties["Content-Type"] = "text/xml";
	outBoundEvent.set("header",properties);


//If LAM is not able to convert the request_query_parm in an accepted xml format, then the user can force the LAM to use the assigned inputs given below as a     string, and use it in body of the request. 
   /*
 outboundEvent.set("body",'<env:Envelope xmlns:env=\"http://schemas.xmlsoap.org/soap/envelope/\"><env:Header/><env:Body><createRequest xmlns=\"http://moogsoft.com\" xmlns:emcf=\"http://xmlns.oracle.com/sysman/connector\"> <Summary xmlns=\"http://moogsoft.com\">CPU Utilization for 3 is 6.787%, crossed warning (5) or critical (10) threshold.</Summary><Description xmlns=\"http://moogsoft.com\">Event created in Enterprise Manager: \n Target information: \n Target Type: host    Target Name: WIN-8U1RA5NAA6I  Target URL: http://win-8u1ra5naa6i:7788/em//em/redirect?pageType=TARGET_HOMEPAGE&amp;targetName=WIN-8U1RA5NAA6I&amp;targetType=host</Description></createRequest></env:Body></env:Envelope>');
 */
    return true;
}


function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{
    return true;
}
function presend(event)
{

// returning true, makes this an event on the MooMs bus, false will cause the LAM to discard the event.
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");

The above examples have the following functions:

  • function onLoad(): This function is used to load data from state file. The state file has details of last poll time of the LAM. The state file is saved at the same location where the REST Client LAM config file is present. The statement constants.load() in this function loads the contents of the state file in the constants variable. The user has some more alternatives for playing with data of the state file. As of now, we are only saving the last poll time, but if the user wants to save some other data which is to be used after the restart of the LAM, then this data can be saved in the state file. The name of the state file will be same as the config file name. For example, if the name of the config file is rest_client_lam.config, then the corresponding state file name will be rest_client_lam.state

  • function preClientSend(): This function extracts the information about the URL from the parameters given in the field request_query_param of the config file, combines all the information and creates the URL. This function also appends the last poll time read from the state file to the URL. The created URL is then sent to the LAM which hits the REST Server for events and alarm data. The constants.save() saves the poll time to the state file which is then used in the next poll.

  • function modifyResponse(): This function makes changes to any response received from the REST Server. This function is used when the response data received from the REST Server needs any changes. If no changes are required then do not edit this function. If any modification is done then, after modification the event data is sent to the LAM for extraction, tokenization mapping. For Example, If we are receiving the event data in XML, then we have to convert it into JSON format for the LAM to process it further. This conversion can be carried out in the modifyResponse() function.

  • function presend(): This function makes any changes to the event data before sending it to MooMS.

The Lambot configuration is done to handle things that cannot be handled by the config file. Users can make changes to the file according to their requirements

It is advisable, that a user with a knowledge of java script makes changes to the Lambot file. Wrong changes done here may break the LAM or will not provide the desired results


Quotes

In some instances, the attribute strings are quoted. Our JSON parser ignores it, but the standard requires quoting for all strings, so Moogsoft recommends that user quotes all strings.

Comments

A user can comment out lines by appending them with a hash.

Starting the REST Client LAM

To start the REST Client LAM enter the following command:

service restclientlamd start

To stop the REST Client LAM enter the following command:

service restclientlamd stop


To view the status of REST Client LAM, enter the following command:

service restclientlamd status


Command line attributes

rest_client_lam --help

Option

Description

--config

Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified

--help

Displays all the command line options

--version

Displays the components version number

--log level

Specifies the level of debugging. By default, User gets everything. In common with all executables in MOOG, having it set at that level can result in a lot of output (many messages per event message processed).

In all production implementations, it is recommended that log level is set to WARN, which only informs user of matters of importance


Version Information

LAM Version

Tool Version

Tested?

Expected to Work

1.0

Generic

Yes

Yes

1.1GenericYesYes
1.2GenericYesYes
1.3GenericYesYes

System Information

This LAM was tested on a system with the following configurations:

CPU2 core
RAM4 GB
Operating SystemCentOS Linux release 6.7

The system must at least have the above mentioned system requirements to run the LAM.

  • No labels