Moogsoft Docs

REST Client LAM

The REST Client LAM is an HTTP client LAM that makes use of one or more REST API sources to request event data and ingest it into Moogsoft AIOps. It sends HTTP requests to the REST server at configurable intervals and parses the JSON responses received from the server, and then it processes events from the responses.

The REST Client LAM ingests event data from RESTful services.

Requirements

The ingestion of event data from RESTful Services requires Moogsoft AIOps to be running a REST Client LAM, configured to parse JSON responses from RESTful services. For endpoints that require authentication, specify credentials in the REST Client LAM configuration file rest_client_lam.conf.

The REST Client LAM processes the events received from the REST API based on the configurations done in the following two files:

File

Description

$MOOGSOFT/config/rest_client_lam.conf

Rest Client LAM configuration file.

$MOOGSOFT/bots/lambots/RestClientLam.js

LAMbot file that processes the event data received from the event source.

You have to configure the above 2 files as per their requirements based on the event format received from the REST API.

Note

The performance of Moogsoft AIOps depends on the number of events received per second and the specifications of the Moogsoft AIOps system on which the REST ClientLAM is running.

Configuration

The configuration file contains a JSON object. At the first layer of the object, LAM has a parameter called config, and the object that follows configuration has all the necessary information to control the LAM.

Monitor

The REST Client LAM takes the connection information from the Monitor section of the config file. You can configure the parameters here to establish a connection with REST Client.

General

Field

Type

Description

Example

name and class

String

Reserved fields: do not change. Default values are REST Client Monitorand CRestClientMonitor.

request_interval

Integer

This is the time interval (in seconds) between 2 successive HTTP requests.

Default:60 seconds.

If you have entered 30 seconds as request_interval value, and you have started the REST Client LAM, then it will wait for 30 seconds before sending the second request.

targets

JSON Object

A top-level container for which you can define one or more REST endpoint targets that you want to poll.

See the multiple target example below

target

JSON Object

A single REST endpoint to poll.

See the multiple target example below. You can specify all the configurations for a REST endpoint. If you don't specify a request_interval, the target uses the globally defined interval.

user and password

String

Enter the username and password for accessing the REST endpoint.

encrypted_password

String

If the password is encrypted, then enter the encrypted password in this field and comment out the password field. If you configure both fields, the Rest Client LAM uses encrypted_password.

proxy

Object

If you want to connect to the REST endpoint through a proxy server, configure the host, port, user, and password or encrypted password properties in the proxy section for the target.

disable_certificate_validation

Boolean

Set to false if the SSL certificate for the event server is valid. Setting it to true will disable the SSL certificate validation for the event server. By default it is set to true.

server_cert_filename

String

Enter the server certificate name here. Use the certificate "server.crt" here. The cert file should be present in the directory given in path_to_ssl_filesfield.

use_client_authentication

Boolean

If you want client authentication, set it to true, else you can set it to false. By default, it is set to false. If it is set to true, then the values will be entered in the client_key_filename and the client_cert_filename fields.

client_key_filename

String

Enter the name of the key file here. The key file should be present in the directory given in path_to_ssl_filesfield.

"client.key"

client_cert_filename

String

Enter the name of the certificate file here. The cert file should be present in the directory given in path_to_ssl_filesfield.

"client.crt"

requests_overlap

Integer

You can augment request time frame with an overlap to ensure that no events are getting missed. The LAM can identify any duplicate events without processing them by using the overlap_identity_fields configuration.

This is the time (in seconds) more than the request_interval the LAM has to wait to request data.

Default:10 seconds, if no value is specified, thenrequests_overlap will set to default.

If request_interval is set to 120 seconds and request_overlap is set to 10, then the LAM will send a request in every two minutes (120 seconds). The overlap is set to 10 seconds, so each request will ask for data from the last 2 minutes and 10 seconds.

url

String

This is the request URL for the endpoint to pull events (including hostname or IP Address). If IP address along with port is entered in the field, then append a "/" in the end.

http://localhost:8896/

request_query_params

String

These are the request query parameters. All the members of this map will be added to the request URL as URL encoded, so the URL will look like url?key1=value1&key2=value2...

Example 1: The following code block provides a sample request query

request_query_params     :
{
    get : "events",
    myInt: 12345,
    myString: "endPoint",
    myVersion: 3
},

The URL for the above request query will be url?get=events&myInt=12345&myString=endPoint&myVersion=3

Example 2: The following code block provides a sample request query containing a to and from parameter

request_query_params     :
{
    get : "events",
    from: "$from",
    to: "$to"
},

The default request_query_params block includes option 'to' and 'from' variables which are special internally calculated timestamps worked out based on the request_interval and the request_overlap compared to the current system time. These timestamps variables can be used in the request_query_params block by prefixing them with a dollar sign. The timestamp format is governed by the field params_date_format.

params_date_format

String

Enter the format of timestamps that are sent in the request URL.

"%Y-%m-%dT%H:%M:%S"

enable_epoch_converter

Boolean

Setting it to true will convert the time stamps in the request URL to epoch time.

results_as_list

Boolean

If the LAM is supposed to get events in a list, then set it to true, otherwise set it to false.

results_path

Boolean

If the events in the response JSON are not in a list, then this will identify where to find the event JSON.

"results.subObject.events"

overlap_identity_fields

String

Setting the requests_overlap value can give the same event twice, setting this parameter will tell the LAM how to identify the duplicate events. Duplicate events will be ignored in the second interval.

["signature", "source_id", "agent_time", "description"]

num_threads

Integer

The number of threads the REST Client LAM will use.

If no value is specified, then the number of

threads to use will be the number of CPUs available, up to a maximum of 8 can be used. Defaults to 5.

retry-recovery

Object

Specifies the behavior of the LAM when it re-establishes a connection after a failure.

- recovery_interval: Length of time to wait between recovery requests in seconds. Must be less than the request_interval set for each target. Defaults to 20.

- max_lookback: The period of time for which to recover missed events in seconds. Defaults to -1 (recover all events since the last successful poll).

timeout

Integer

This is the timeout value in seconds, whichwill be used to timeout a connection, socket and request. If no value is specified, then the time interval will set to 120 seconds.

Default:120 seconds, if no value is specified, thentimeout will set to default.

Secure Sockets Layer

Field

Type

Description

use_ssl

Boolean

Set to true to enable SSL Communication:

  • path_to_ssl_files: Enter the path of the directory where all the certificates are stored. If the path begins with ‘.’ or ‘/’ then, the path will be used as specified. Otherwise, MOOGSOFT_HOME is prepended to the path. For example, if MOOGSOFT_HOME is /opt/moogsoft/ and path_to_ssl is set to config, then the location will be defined as /opt/moogsoft/config.

Single Target Example
monitor:
        {
            name                            : "REST Client Monitor",
            class                           : "CRestClientMonitor",
            request_interval                : 60,
            user                            : "username",
            password                        : "password",
            #encrypted_password             : "ieytOFRUdLpZx53nijEw0rOh07VEr8w9lBxdCc7229o=",
            enable_proxy                    : false,
            host                            : "",
            port                            : 808,
            user                            : "",
            #encrypted_password             : "",
            proxy_password                  : "",
            use_ssl                         : false,  
            disable_certificate_validation  : false,
            path_to_ssl_files               : "config",
            server_cert_filename            : "server.crt",
            use_client_authentication       : false,
            client_key_filename             : "client.key",
            client_cert_filename            : "client.crt",
            requests_overlap                : 10,
            url                             : "",
            request_query_params            : {
                                                  get  : "events",
                                                  start: "$from",
                                                  end  : "$to"            
                                              },
            params_date_format              : "%Y-%m-%dT%H:%M:%S",
            enable_epoch_converter          : false,
            results_as_list                 : false,
            results_path                    : "results",
            overlap_identity_fields         : ["signature", "source_id", "agent_time", "description"]
            num_threads                     : 5,
            timeout                         : 120
        },
Multiple Target Example

The example below shows how to configure the monitor object to request data from multiple REST endpoints. In this case the targets are named for the location of the target, "Kingston" and "San Francisco":

monitor:
        {
            name: "REST Client Monitor",
            class: "CRestClientMonitor",
            request_interval: 60,
            targets:
            {
                kingston:
                {
                    user: "kingstonuser",
                    password: "password",
                    disable_certificate_validation: false,
                    requests_overlap: 10,
                    url: "http://kingston-host/api/events",
                    overlap_identity_fields: ["id"],
                    timeout: 120
                },
                sanfrancisco:
                {
                    request_interval: 30,
                    user: "sfuser",
                    password: "password",
                    disable_certificate_validation: false,
                    requests_overlap: 10,
                    url: "http://sf-host/api/events",
                    overlap_identity_fields: ["id"],
                    timeout: 120
                }
        }
Agent and Process Log

The Agent and Process Log sections allow you to configure the following properties:

  • name: Maps to $Laminstancename, so that the agent field indicates events Moogsoft AIOps ingests from this LAM.

  • capture_log: Name and location of the LAM's capture log file, which it writes to for debugging purposes.

  • configuration_file: Name and location of the LAM's process log configuration file. See Configure Logging for more information.Configure Logging

Mapping

For events received in JSON format, you can directly map the event fields of REST Client LAM with Moogsoft AIOps fields. The parameters of the received events are displayed in Moogsoft AIOps according to the mapping done here:

mapping :
         {            
              catchAll: "overflow",
                rules:
                [
                    { name: "signature", rule:      "$signature" },
                    { name: "source_id", rule:      "$source_id" },
                    { name: "external_id", rule:    "$external_id" },
                    { name: "manager", rule:        "$manager" },
                    { name: "source", rule:         "$source" },
                    { name: "class", rule:          "$class" },
                    { name: "agent", rule:          "$agent" },
                    { name: "agent_location", rule: "$host" },
                    { name: "type", rule:           "$eventType" },
                    { name: "description", rule:    "$description" },
                    { name: "severity", rule:       "$severity" },
                    { name: "agent_time", rule:     "$time", conversion: "timeConverter" }
                ]
          },
              filter:
                     {
                         presend: "RestClientLam.js"
                     }

The above example specifies the mapping of the REST Server event fields with the Moogsoft AIOps fields.

Note

The signature field is used by the LAM to identify correlated events.

State File

The REST Client LAM will save the last poll time of the request sent to the REST Server. This last poll time will be saved in a state file. The REST Client LAM will read/write the last poll time from/to this state file. The LAM will read/write from the state file by using the RestClientLam.js file. To read and write the state file you have to modify the RestClientLam.js.

Note

The state file is generated in the same folder where the config file is present e.g. $MOOGSOFT_HOME/config. The LAM generates the name of the state file as <proc_name>.state. Here the default proc_name (process name) is rest_client_lam, therefore, the state file name is rest_client_lam.state.

LAMBot Configuration

The LAMbot processes the event data received from RESTful Services. The LamBot RestClientLam.js script can be configured for GET and POST requests. The following example shows a LAMbot configuration for GET request. This configuration primarily reads/writes the last poll time data from/to the state file.

var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime = true;
function onLoad()
{
        /* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. 
    Last poll time can be appended in the url for polling from the lass poll time;
        */
        constants.load();
    return;
}

function preClientSend(outboundEvent)
{
  /* outBoundEvent contains below field which can be manipulated as per requirement
        1. method - by default method is set to GET request
        2. body - contains all the values of request_query_param from the config file
        3. header - can contain additional headers
  */
    return true;
}

function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{       
    return true;
}

function presend(event)
{        
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");

The following example shows the LAMbot configuration for a POST request with a JSON payload in which the last poll time is being written and read from the state file.

var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime= true; 

function onLoad()
{
        /* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. 
    Last poll time can be appended in the url for polling from the lass poll time;
        */      
        constants.load();
        return;
}
function preClientSend(outBoundEvent)
 {
  /*outBoundEvent contains below field which can be manipulated as per requirement
        1. method - by default method is set to GET request
        2. body - contains all the values of request_query_param from the config file
        3. header - can contain additional headers
  */
    var body = outBoundEvent.value('body'); // reads body from outBoundEvent
        if(firstTime)
        {
                if(constants.contains("start") == true)
                {
                        var start = constants.get("start"); // reads the last poll time value from the state file
                
                        body['start'] = start; // replaces value of the start in body with a new value
                        outBoundEvent.set('body',body); // resets value of body with modified body
                        constants.save();
                        firstTime = false;
                }
                else
                {
                        constants.put("start",body['start']);
                        constants.save();
                        firstTime = false;
                }
        }
        else
        {
                constants.put("start",body['start']);
                constants.save();
        }
// methods that can be used here are'Post','Put',and 'Delete';  
        outBoundEvent.set("method","Post"); 
        
        var properties = outBoundEvent.value("header"); 
// Header is used if the method used is Put or Delete or Post. Content type should always be defined otherwise it will give an error    
        properties["Content-Type"] = "application/json";
        outBoundEvent.set("header",properties);


//If LAM is not able to convert the request_query_parm in an accepted JSON format, then the user can force the LAM to use the assigned inputs given below as a     string, and use it in body of the request.
        
    //outboundEvent.set("body", 
    //'{"action":"EventsRouter","method":"query","data":[{"params":{"severity":[5,4,3,2,1,0],"eventState":[0,1,2,3,4,5]},"limit":9}],"type":"rpc","tid":1}');
    return true;
}


function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{
    return true;
}
function presend(event)
{

// returning true, makes this an event on the MooMs bus, false will cause the LAM to discard the event.
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");

The following example shows the LAMbot configuration for a POST request with an XML payload in which the last poll time is being written and read from the state file.

var logger=LamBot.loadModule("Logger");
var constants=LamBot.loadModule("Constants");
/*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file
 or read data from the state file
*/
var firstTime= true; 

function onLoad()
{
        /* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. 
    Last poll time can be appended in the url for polling from the lass poll time;
        */      
        constants.load();
        return;
}
function preClientSend(outBoundEvent)
 {
  /*outBoundEvent contains below field which can be manipulated as per requirement
        1. method - by default method is set to GET request
        2. body - contains all the values of request_query_param from the config file
        3. header - can contain additional headers
  */
    var body = outBoundEvent.value('body'); // reads body from outBoundEvent
        if(firstTime)
        {
                if(constants.contains("start") == true)
                {
                        var start = constants.get("start"); // reads the last poll time value from the state file
                
                        body['start'] = start; // replaces value of the start in body with a new value
                        outBoundEvent.set('body',body); // resets value of body with modified body
                        constants.save();
                        firstTime = false;
                }
                else
                {
                        constants.put("start",body['start']);
                        constants.save();
                        firstTime = false;
                }
        }
        else
        {
                constants.put("start",body['start']);
                constants.save();
        }
// methods that can be used here are'Post','Put',and 'Delete';  
        outBoundEvent.set("method","Post"); 
        
        var properties = outBoundEvent.value("header"); 
// Header is used if the method used is Put or Delete or Post. Content type should always be defined otherwise it will give an error
// The LAM cannot by default send an XML payload. To send an XML payload then set content type as text/xml and set the xml payload in the body  
        properties["Content-Type"] = "text/xml";
        outBoundEvent.set("header",properties);


//If LAM is not able to convert the request_query_parm in an accepted xml format, then the user can force the LAM to use the assigned inputs given below as a     string, and use it in body of the request. 
   /*
 outboundEvent.set("body",'<env:Envelope xmlns:env=\"http://schemas.xmlsoap.org/soap/envelope/\"><env:Header/><env:Body><createRequest xmlns=\"http://moogsoft.com\" xmlns:emcf=\"http://xmlns.oracle.com/sysman/connector\"> <Summary xmlns=\"http://moogsoft.com\">CPU Utilization for 3 is 6.787%, crossed warning (5) or critical (10) threshold.</Summary><Description xmlns=\"http://moogsoft.com\">Event created in Enterprise Manager: \n Target information: \n Target Type: host    Target Name: WIN-8U1RA5NAA6I  Target URL: http://win-8u1ra5naa6i:7788/em//em/redirect?pageType=TARGET_HOMEPAGE&amp;targetName=WIN-8U1RA5NAA6I&amp;targetType=host</Description></createRequest></env:Body></env:Envelope>');
 */
    return true;
}


function modifyResponse(inBoundEventData)
  /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here
  The inBoundEventData contains the following field which can be manipulated as per the requirement
  1. responseData - the event data received from the rest server
  */
{
    return true;
}
function presend(event)
{

// returning true, makes this an event on the MooMs bus, false will cause the LAM to discard the event.
    return( true );
}
LamBot.filterFunction("presend");
LamBot.preClientSendFunction("preClientSend");
LamBot.modifyResponseFunction("modifyResponse");

The above examples have the following functions:

  • function onLoad(): This function is used to load data from state file. The state file has details of last poll time of the LAM. The state file is saved at the same location where the REST Client LAM config file is present. The statement constants.load() in this function loads the contents of the state file in the constant variable. This provides you some more alternatives for playing with data of the state file. As of now, only the last poll time will be saved, but if you want to save some other data which is to be used after restarting the LAM, then you can save this data in the state file. The name of the state file will be same as the config file name. For example, if the name of the config file is rest_client_lam.config, then the corresponding state file name will be rest_client_lam.state.

  • function preClientSend(): This function extracts the information about the URL from the parameters given in the field request_query_param of the config file, combines all the information and creates the URL. This function also appends the last poll time read from the state file to the URL. The created URL is then sent to the LAM which hits the REST Server for events and alarm data. The constants.save() saves the poll time to the state file which is then used in the next poll.

  • function modifyResponse(): This function makes changes to any response received from the REST Server. This function is used when the response data received from the REST Server needs any changes. If no changes are required, then do not edit this function. If any modification is done, then after modification the event data is sent to the LAM for extraction, tokenization, and mapping. For Example, if you are receiving the event data in XML, then you have to convert it into JSON format for the LAM to process it further. This conversion can be carried out in the modifyResponse() function.

  • function presend(): This function makes any changes to the event data before sending it to MooMS.

Note

The LAMbot configuration is done to handle things that cannot be handled by the config file. You can make changes to the file according to their requirements.

Service Operation Reference

Process Name

Service Name

rest_client_lam

restclientlamd

Start the LAM Service:

service restclientlamd start

Stop the LAM Service:

service restclientlamd stop

Check the LAM Service status:

service restclientlamd status

If the LAM fails to connect to one or more REST API sources, Moogsoft AIOps creates an alert and writes the details to the process log. Refer to the logging details for LAMs and integrations for more information.

Command Line Reference

To see the available optional attributes of the rest_client_lam, run the following command:

rest_client_lam --help

The rest_client_lam is a command line executable, and has the following optional attributes:

Option

Description

--config

Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified.

--help

Displays all the command line options.

--version

Displays the component’s version number.

--loglevel

Specifies the level of debugging. By default, user gets everything. In common with all executables in Moogsoft AIOps, having it set at that level can result in a lot of output (many messages per event message processed).

In all production implementations, it is recommended that log level is set to WARN. This ensures only warning, error and fatal messages are recorded.