Moogsoft Docs

Rest Client

The Rest Client Integration (LAM) is an HTTP client LAM which makes use of a REST API to request event data and ingest it into Moogsoft AIOps. It sends HTTP requests to the REST server at configurable intervals and parses the JSON responses received from the server, and then it processes events from the responses.

The Rest Client Integration (LAM) ingests event data from RESTful Services.

Requirements

The ingestion of event data from RESTful Services requires Moogsoft AIOps to be running a Rest Client LAM, configured to parse JSON responses from RESTful Services. For endpoints that require authentication, specify credentials in the RestClient LAM configuration file rest_client_lam.conf.

The REST Client LAM processes the events received from the REST API based on the configurations done in the following 2 files:

File
Description
$MOOGSOFT/config/r est_client _lam.conf Rest Client LAM configuration file
$MOOGSOFT/bots/lambots/ RestClientLam .js LAMBot file; processes the event data received from the event source

You have to configure the above 2 files as per their requirements based on the event format received from the REST API.

Note

The performance of Moogsoft AIOps depends on the number of events received per second and the specifications of the Moogsoft AIOps system on which the Rest Client LAM is running.

Configuration

The configuration file contains a JSON object. At the first layer of the object, LAM has a parameter called config , and the object that follows config has all the necessary information to control the LAM.

The following sections are available for configuration in the rest_client_lam. c onf file:

Monitor

Agent

HA Configuration

Mapping

Constants and Conversions

Monitor

The REST Client LAM takes the connection information from the Monitor section of the config file. You can configure the parameters here to establish a connection with REST Client.

General

Field
Type
Description
Example
name and class String

Reserved fields: do not change. Default values are REST Client Monitor and CRestClientMonitor.


request_interval Integer

This is the time interval (in seconds) between 2 successive HTTP requests.

Default: 60 seconds.

If you have entered 30 seconds as request_interval value, and you have started the REST Client LAM, then it will wait for 30 seconds before sending the second request.
targets JSON Object A top-level container for which you can define one or more REST endpoint targets that you want to poll. See the multiple target example below
target JSON Object A single REST endpoint to poll. See the multiple target example below. You can specify all the configurations for a REST endpoint. If you don't specify a request_interval, the target uses the globally defined interval.
user and password String Enter the username and password for accessing the REST endpoint.
encrypted_password String If the password is encrypted, then enter the encrypted password in this field and comment out the password field. If you configure both fields, the Rest Client LAM uses encrypted_password .
proxy_host String Enter the host name or the URL of the proxy.
proxy_port Integer Enter the port of the proxy. This field will only work if enable_proxy is set to true .
proxy_user String Enter the username of the user who has the rights to access the proxy. This field will be active only when the enable_proxy is set to true .
encrypted_proxy_password String If the proxy password is encrypted, then enter the encrypted password in this field and comment out the proxy_password field. Either proxy_password or the encrypted_ proxy_password field is used. If both fields are not commented, then only the field encrypted_ proxy_password will be used.
proxy_password String Enter the password of the user whose user name is given in the proxy_user field. This field is active if enable_proxy is set to true.
disable_certificate_validation Boolean Set to false if the SSL certificate for the event server is valid. Setting it to true will disable the SSL certificate validation for the event server. By default it is set to true .
server_cert_filename String Enter the server certificate name here. Use the certificate "server.crt" here. The cert file should be present in the directory given in path_to_ssl_files field.
use_client_authentication Boolean If you want client authentication, set it to true, else you can set it to false. By default, it is set to false. If it is set to true, then the values will be entered in the client_key_filename and the client_cert_filename fields.
client_key_filename String Enter the name of the key file here. The key file should be present in the directory given in path_to_ssl_files field. "client.key"
client_cert_filename String Enter the name of the certificate file here. The cert file should be present in the directory given in path_to_ssl_files field. "client.crt"
requests_overlap Integer

You can augment request time frame with an overlap to ensure that no events are getting missed. The LAM can identify any duplicate events without processing them by using the overlap_identity_fields configuration.
This is the time (in seconds) more than the request_interval the LAM has to wait to request data.

Default: 10 seconds, if no value is specified, then requests_overlap will set to default.

If request_interval is set to 120 seconds and request_overlap is set to 10, then the LAM will send a request in every two minutes (120 seconds). The overlap is set to 10 seconds, so each request will ask for data from the last 2 minutes and 10 seconds.
url String This is the request URL for the endpoint to pull events (including hostname or IP Address). If IP address along with port is entered in the field, then append a "/" in the end. http://localhost:8896/
request_query_params String

These are the request query parameters. All the members of this map will be added to the request URL as URL encoded, so the URL will look like url?key1=value1&key2=value2 ...


Example 1: The following code block provides a sample request query

request_query_params     :
{
    get : "events",
    myInt: 12345,
    myString: "endPoint",
    myVersion: 3
},

The URL for the above request query will be url?get=events&myInt= 12345 &myString=endPoint&myVersion= 3

Example 2: The following code block provides a sample request query containing a to and from parameter

request_query_params     :
{
    get : "events",
    from: "$from",
    to: "$to"
},

The default request_query_params block includes option 'to' and 'from' variables which are special internally calculated timestamps worked out based on the request_interval and the request_overlap compared to the current system time. These timestamps variables can be used in the request_query_params block by prefixing them with a dollar sign. The timestamp format is governed by the field params_date_format .



params_date_format String Enter the format of timestamps that are sent in the request URL. "%Y-%m-%dT%H:%M:%S"
enable_epoch_converter Boolean Setting it to true will convert the time stamps in the request URL to epoch time.
results_as_list Boolean If the LAM is supposed to get events in a list, then set it to true , otherwise set it to false .
results_path Boolean If the events in the response JSON are not in a list, then this will identify where to find the event JSON. "results.subObject.events"
overlap_identity_fields String Setting the requests_overlap value can give the same event twice, setting this parameter will tell the LAM how to identify the duplicate events. Duplicate events will be ignored in the second interval. [ "signature" , "source_id" , "agent_time" , "description" ]
num_threads Integer The number of threads the REST Client LAM will use.

If no value is specified, then the number of
threads to use will be the number of CPUs available, up to a maximum of 8 can be used. It's default value is 5.



timeout Integer

This is the timeout value in seconds, which will be used to timeout a connection, socket and request . If no value is specified, then the time interval will set to 120 seconds.

Default: 120 seconds, if no value is specified, then timeout will set to default.


Secure Sockets Layer

Field
Type
Description

use_ssl

Boolean

Set to true to enable SSL Communication:

  • path_to_ssl_files: Enter the path of the directory where all the certificates are stored. If the path begins with ‘.’ or ‘/’ then, the path will be used as specified. Otherwise, MOOGSOFT_HOME is prepended to the path. For example, if MOOGSOFT_HOME is /opt/moogsoft/ and path_to_ssl is set to config , then the location will be defined as /opt/moogsoft/confi g.


Single Target Example

config :
    {
        monitor:
        {
            name                            : "REST Client Monitor",
            class                           : "CRestClientMonitor",
            request_interval                : 60,
            user                            : "username",
            password                        : "password",
            #encrypted_password             : "ieytOFRUdLpZx53nijEw0rOh07VEr8w9lBxdCc7229o=",
			enable_proxy 				    : false,
            proxy_host                      : "",
            proxy_port                      : 808,
			proxy_user                      : "",
			#encrypted_proxy_password       : "",
            proxy_password                  : "",
			use_ssl                   		: false,
			disable_certificate_validation  : false,
            path_to_ssl_files               : "config",
            server_cert_filename      		: "server.crt",
            use_client_authentication 		: false,
			client_key_filename       		: "client.key",
			client_cert_filename      		: "client.crt",
            requests_overlap                : 10,
            url                             : "",
            request_query_params            : {
											 	get  : "events",
                							 	start: "$from",
                								end	 : "$to"
            								  },
            params_date_format              : "%Y-%m-%dT%H:%M:%S",
			enable_epoch_converter          : false,
            results_as_list                 : false,
            results_path                    : "results",
            overlap_identity_fields         : ["signature", "source_id", "agent_time", "description"]
            num_threads                     : 5,
			timeout							: 120
        },

Multiple Target Example

The example below shows how to configure the monitor object to request data from multiple REST endpoints. In this case the targets are named for the location of the target, "Kingston" and "San Francisco":

monitor:
        {
            name: "REST Client Monitor",
            class: "CRestClientMonitor",
            request_interval: 60,
            targets:
            {
                kingston:
                {
                    user: "kingstonuser",
                    password: "password",
                    disable_certificate_validation: false,
                    requests_overlap: 10,
                    url: "http://kingston-host/api/events",
                    overlap_identity_fields: ["id"],
                    timeout: 120
                },
                sanfrancisco:
                {
                    request_interval: 30,
                    user: "sfuser",
                    password: "password",
                    disable_certificate_validation: false,
                    requests_overlap: 10,
                    url: "http://sf-host/api/events",
                    overlap_identity_fields: ["id"],
                    timeout: 120
                }
            }


Agent

Agent allows you to define two parameters:

Field
name This is the agent name, the events sent to MooMS by the REST Client LAM are identified by the agent name in the log. In this LAM, the agent name is RestClientLam.
log REST Client LAM will write its ingress contents in the file rest_client_lam.log located at /var/log/moogsoft/.


Mapping

For events received in JSON format, you can directly map the event fields of REST Client LAM with Moogsoft fields. The parameters of the received events are displayed in Moogsoft AIOps according to the mapping done here:

mapping :
        {
        catchAll: "overflow",
            rules:
            [
                { name: "signature", rule:      "$signature" },
                { name: "source_id", rule:      "$source_id" },
                { name: "external_id", rule:    "$external_id" },
                { name: "manager", rule:        "$manager" },
                { name: "source", rule:         "$source" },
                { name: "class", rule:          "$class" },
                { name: "agent", rule:          "$agent" },
                { name: "agent_location", rule: "$host" },
                { name: "type", rule:           "$eventType" },
                { name: "description", rule:    "$description" },
                { name: "severity", rule:       "$severity" },
                { name: "agent_time", rule:     "$time", conversion: "timeConverter" }
            ]
        },
        filter:
        {
            presend: "RestClientLam.js"
        }

The above example specifies the mapping of the REST Server event fields with the Moogsoft AIOps fields.

Note

The signature field is used by the LAM to identify correlated events.

State File

The REST Client LAM will save the last poll time of the request sent to the REST Server. This last poll time will be saved in a state file. The REST Client LAM will read/write the last poll time from/to this state file. The LAM will read/write from the state file by using the RestClientLam.js file. To read and write the state file you have to modify the RestClientLam.js .

Note

The state file is generated in the same folder where the config file is present e.g. $MOOGSOFT_HOME/config. The LAM generates the name of the state file as < proc_name>.state. Here the default proc_name (process name) is rest_client_lam, therefore, the state file name is rest_client_lam.state.

LamBot Configuration

The LamBot processes the event data received from RESTful Services . The LamBot RestClientLam.js script can be configured for GET and POST requests.
The following example shows a LamBot configuration for GET request. This configuration primarily reads/writes the last poll time data from/to the state file.


var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime = true;
function onLoad()
{
	/* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it.
    Last poll time can be appended in the url for polling from the lass poll time;
	*/
	constants.load();
    return;
}

function preClientSend(outboundEvent)
{
  /* outBoundEvent contains below field which can be manipulated as per requirement
    	1. method - by default method is set to GET request
    	2. body - contains all the values of request_query_param from the config file
    	3. header - can contain additional headers
  */
    return true;
}

function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{
    return true;
}

function presend(event)
{
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");


The following example shows the Lambot configuration for a POST request with a JSON payload in which the last poll time is being written and read from the state file.

var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime= true;

function onLoad()
{
	/* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it.
    Last poll time can be appended in the url for polling from the lass poll time;
	*/
	constants.load();
	return;
}
function preClientSend(outBoundEvent)
 {
  /*outBoundEvent contains below field which can be manipulated as per requirement
    	1. method - by default method is set to GET request
    	2. body - contains all the values of request_query_param from the config file
    	3. header - can contain additional headers
  */
    var body = outBoundEvent.value('body'); // reads body from outBoundEvent
	if(firstTime)
	{
		if(constants.contains("start") == true)
		{
			var start = constants.get("start"); // reads the last poll time value from the state file

			body['start'] = start; // replaces value of the start in body with a new value
			outBoundEvent.set('body',body); // resets value of body with modified body
			constants.save();
			firstTime = false;
		}
		else
		{
			constants.put("start",body['start']);
			constants.save();
			firstTime = false;
		}
	}
	else
	{
		constants.put("start",body['start']);
		constants.save();
	}
// methods that can be used here are'Post','Put',and 'Delete';
	outBoundEvent.set("method","Post");

	var properties = outBoundEvent.value("header");
// Header is used if the method used is Put or Delete or Post. Content type should always be defined otherwise it will give an error
	properties["Content-Type"] = "application/json";
	outBoundEvent.set("header",properties);


//If LAM is not able to convert the request_query_parm in an accepted JSON format, then the user can force the LAM to use the assigned inputs given below as a     string, and use it in body of the request.

    //outboundEvent.set("body",
    //'{"action":"EventsRouter","method":"query","data":[{"params":{"severity":[5,4,3,2,1,0],"eventState":[0,1,2,3,4,5]},"limit":9}],"type":"rpc","tid":1}');
    return true;
}


function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{
    return true;
}
function presend(event)
{

// returning true, makes this an event on the MooMs bus, false will cause the LAM to discard the event.
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");


The following example shows the Lambot configuration for a POST request with an XML payload in which the last poll time is being written and read from the state file.

var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime= true;

function onLoad()
{
	/* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it.
    Last poll time can be appended in the url for polling from the lass poll time;
	*/
	constants.load();
	return;
}
function preClientSend(outBoundEvent)
 {
  /*outBoundEvent contains below field which can be manipulated as per requirement
    	1. method - by default method is set to GET request
    	2. body - contains all the values of request_query_param from the config file
    	3. header - can contain additional headers
  */
    var body = outBoundEvent.value('body'); // reads body from outBoundEvent
	if(firstTime)
	{
		if(constants.contains("start") == true)
		{
			var start = constants.get("start"); // reads the last poll time value from the state file

			body['start'] = start; // replaces value of the start in body with a new value
			outBoundEvent.set('body',body); // resets value of body with modified body
			constants.save();
			firstTime = false;
		}
		else
		{
			constants.put("start",body['start']);
			constants.save();
			firstTime = false;
		}
	}
	else
	{
		constants.put("start",body['start']);
		constants.save();
	}
// methods that can be used here are'Post','Put',and 'Delete';
	outBoundEvent.set("method","Post");

	var properties = outBoundEvent.value("header");
// Header is used if the method used is Put or Delete or Post. Content type should always be defined otherwise it will give an error
// The LAM cannot by default send an XML payload. To send an XML payload then set content type as text/xml and set the xml payload in the body
	properties["Content-Type"] = "text/xml";
	outBoundEvent.set("header",properties);


//If LAM is not able to convert the request_query_parm in an accepted xml format, then the user can force the LAM to use the assigned inputs given below as a     string, and use it in body of the request.
   /*
 outboundEvent.set("body",'<env:Envelope xmlns:env=\"http://schemas.xmlsoap.org/soap/envelope/\"><env:Header/><env:Body><createRequest xmlns=\"http://moogsoft.com\" xmlns:emcf=\"http://xmlns.oracle.com/sysman/connector\"> <Summary xmlns=\"http://moogsoft.com\">CPU Utilization for 3 is 6.787%, crossed warning (5) or critical (10) threshold.</Summary><Description xmlns=\"http://moogsoft.com\">Event created in Enterprise Manager: \n Target information: \n Target Type: host    Target Name: WIN-8U1RA5NAA6I  Target URL: http://win-8u1ra5naa6i:7788/em//em/redirect?pageType=TARGET_HOMEPAGE&amp;targetName=WIN-8U1RA5NAA6I&amp;targetType=host</Description></createRequest></env:Body></env:Envelope>');
 */
    return true;
}


function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{
    return true;
}
function presend(event)
{

// returning true, makes this an event on the MooMs bus, false will cause the LAM to discard the event.
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");

The above examples have the following functions:

  • function onLoad() : This function is used to load data from state file. The state file has details of last poll time of the LAM. The state file is saved at the same location where the REST Client LAM config file is present. The statement constants.load() in this function loads the contents of the state file in the constant variable. This provides you some more alternatives for playing with data of the state file. As of now, only the last poll time will be saved, but if you want to save some other data which is to be used after restarting the LAM, then you can save this data in the state file. The name of the state file will be same as the config file name. For example, if the name of the config file is rest_client_lam.config , then the corresponding state file name will be rest_client_lam.state.

  • function preClientSend() : This function extracts the information about the URL from the parameters given in the field request_query_param of the config file, combines all the information and creates the URL. This function also appends the last poll time read from the state file to the URL. The created URL is then sent to the LAM which hits the REST Server for events and alarm data. The constants.save() saves the poll time to the state file which is then used in the next poll.

  • function modifyResponse() : This function makes changes to any response received from the REST Server. This function is used when the response data received from the REST Server needs any changes. If no changes are required, then do not edit this function. If any modification is done, then after modification the event data is sent to the LAM for extraction, tokenization, and mapping. For Example, if you are receiving the event data in XML, then you have to convert it into JSON format for the LAM to process it further. This conversion can be carried out in the modifyResponse() function.

  • function presend () : This function makes any changes to the event data before sending it to MooMS.

Note

The Lambot configuration is done to handle things that cannot be handled by the config file. You can make changes to the file according to their requirements.


Service Operation Reference

Process Name Service Name
rest_client_lam
restclientlamd

Start the LAM Service:

service restclientlamd start

Stop the LAM Service:

service restclientlamd stop

Check the LAM Service status:

service restclientlamd status

Command Line Reference

To see the available optional attributes of the rest_client _lam , run the following command:

rest_client_lam --help

The rest_client _lam is a command line executable, and has the following optional attributes:

Option Description

--config

Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified.
--help Displays all the command line options.
--version

Displays the component’s version number.

--loglevel

Specifies the level of debugging. By default, user gets everything. In common with all executables in MOOG, having it set at that level can result in a lot of output (many messages per event message processed).

In all production implementations, it is recommended that log level is set to WARN. This ensures only warning, error and fatal messages are recorded.

Performance Information

Minimum requirement
Component Value
CPU 2 core
RAM 4 GB
Operating System CentOS Linux release 6.7

Version

Integration Version Tool Version
1.0 - 1.3 Generic