REST Client LAM
The REST Client LAM is an HTTP client LAM that makes use of one or more REST API sources to request event data and ingest it into Moogsoft Onprem. It sends HTTP requests to the REST server at configurable intervals and parses the JSON responses received from the server, and then it processes events from the responses.
The REST Client LAM ingests event data from RESTful services.
Requirements
The ingestion of event data from RESTful Services requires Moogsoft Onprem to be running a REST Client LAM, configured to parse JSON responses from RESTful services. For endpoints that require authentication, specify credentials in the REST Client LAM configuration file rest_client_lam.conf
.
The REST Client LAM processes the events received from the REST API based on the configurations done in the following two files:
File | Description |
---|---|
| Rest Client LAM configuration file. |
| LAMbot file that processes the event data received from the event source. |
You have to configure the above 2 files as per their requirements based on the event format received from the REST API.
Note
The performance of Moogsoft Onprem depends on the number of events received per second and the specifications of the Moogsoft Onprem system on which the REST ClientLAM is running.
Configuration
The configuration file contains a JSON object. At the first layer of the object, LAM has a parameter called config
, and the object that follows configuration has all the necessary information to control the LAM.
Monitor
The REST Client LAM takes the connection information from the Monitor section of the config file. You can configure the parameters here to establish a connection with REST Client.
General
Field | Type | Description | Example |
---|---|---|---|
| String | Reserved fields: do not change. Default values are | |
| Integer | This is the time interval (in seconds) between 2 successive HTTP requests. Default:60 seconds. | If you have entered 30 seconds as |
| JSON Object | A top-level container for which you can define one or more REST endpoint targets that you want to poll. | See the multiple target example below |
| JSON Object | A single REST endpoint to poll. | See the multiple target example below. You can specify all the configurations for a REST endpoint. If you don't specify a request_interval, the target uses the globally defined interval. |
| String | Enter the username and password for accessing the REST endpoint. | |
| String | If the password is encrypted, then enter the encrypted password in this field and comment out the | |
| Object | If you want to connect to the REST endpoint through a proxy server, configure the host, port, user, and password or encrypted password properties in the proxy section for the target. | |
| Boolean | Set to | |
| String | Enter the server certificate name here. Use the certificate "server.crt" here. The cert file should be present in the directory given in | |
| Boolean | If you want client authentication, set it to true, else you can set it to false. By default, it is set to false. If it is set to true, then the values will be entered in the | |
| String | Enter the name of the key file here. The key file should be present in the directory given in | "client.key" |
| String | Enter the name of the certificate file here. The cert file should be present in the directory given in | "client.crt" |
| Integer | You can augment request time frame with an overlap to ensure that no events are getting missed. The LAM can identify any duplicate events without processing them by using the This is the time (in seconds) more than the Default:10 seconds, if no value is specified, then | If |
| String | This is the request URL for the endpoint to pull events (including hostname or IP Address). If IP address along with port is entered in the field, then append a "/" in the end. | http://localhost:8896/ |
| String | These are the request query parameters. All the members of this map will be added to the request URL as URL encoded, so the URL will look like | Example 1: The following code block provides a sample request query request_query_params : { get : "events", myInt: 12345, myString: "endPoint", myVersion: 3 }, The URL for the above request query will be Example 2: The following code block provides a sample request query containing a to and from parameter request_query_params : { get : "events", from: "$from", to: "$to" }, The default |
| String | Enter the format of timestamps that are sent in the request URL. | "%Y-%m-%dT%H:%M:%S" See the Oracle documentation for more information on formats. |
| Boolean | Setting it to | |
| Boolean | If the LAM is supposed to get events in a list, then set it to | |
| Boolean | If the events in the response JSON are not in a list, then this will identify where to find the event JSON. | "results.subObject.events" |
| String | Setting the |
|
| Integer | The number of threads the REST Client LAM will use. If no value is specified, then the number of threads to use will be the number of CPUs available, up to a maximum of 8 can be used. Defaults to 5. | |
| Object | Specifies the behavior of the LAM when it re-establishes a connection after a failure. - - | |
| Integer | This is the timeout value in seconds, whichwill be used to timeout a connection, socket and request. If no value is specified, then the time interval will set to 120 seconds. Default:120 seconds, if no value is specified, then |
Secure Sockets Layer
Field | Type | Description |
---|---|---|
| Boolean | Set to
|
Single Target Example
monitor: { name : "REST Client Monitor", class : "CRestClientMonitor", request_interval : 60, user : "username", password : "password", #encrypted_password : "ieytOFRUdLpZx53nijEw0rOh07VEr8w9lBxdCc7229o=", enable_proxy : false, host : "", port : 808, user : "", #encrypted_password : "", proxy_password : "", use_ssl : false, disable_certificate_validation : false, path_to_ssl_files : "config", server_cert_filename : "server.crt", use_client_authentication : false, client_key_filename : "client.key", client_cert_filename : "client.crt", requests_overlap : 10, url : "", request_query_params : { get : "events", start: "$from", end : "$to" }, params_date_format : "%Y-%m-%dT%H:%M:%S", enable_epoch_converter : false, results_as_list : false, results_path : "results", overlap_identity_fields : ["signature", "source_id", "agent_time", "description"] num_threads : 5, timeout : 120 },
Multiple Target Example
The example below shows how to configure the monitor
object to request data from multiple REST endpoints. In this case the targets are named for the location of the target, "Kingston" and "San Francisco":
monitor: { name: "REST Client Monitor", class: "CRestClientMonitor", request_interval: 60, targets: { kingston: { user: "kingstonuser", password: "password", disable_certificate_validation: false, requests_overlap: 10, url: "http://kingston-host/api/events", overlap_identity_fields: ["id"], timeout: 120 }, sanfrancisco: { request_interval: 30, user: "sfuser", password: "password", disable_certificate_validation: false, requests_overlap: 10, url: "http://sf-host/api/events", overlap_identity_fields: ["id"], timeout: 120 } }
Agent and Process Log
The Agent and Process Log sections allow you to configure the following properties:
name: Maps to
$Laminstancename
, so that theagent
field indicates events Moogsoft Onprem ingests from this LAM.capture_log: Name and location of the LAM's capture log file, which it writes to for debugging purposes.
configuration_file: Name and location of the LAM's process log configuration file. See Configure Logging for more information.
Mapping
For events received in JSON format, you can directly map the event fields of REST Client LAM with Moogsoft Onprem fields. The parameters of the received events are displayed in Moogsoft Onprem according to the mapping done here:
mapping : { catchAll: "overflow", rules: [ { name: "signature", rule: "$signature" }, { name: "source_id", rule: "$source_id" }, { name: "external_id", rule: "$external_id" }, { name: "manager", rule: "$manager" }, { name: "source", rule: "$source" }, { name: "class", rule: "$class" }, { name: "agent", rule: "$agent" }, { name: "agent_location", rule: "$host" }, { name: "type", rule: "$eventType" }, { name: "description", rule: "$description" }, { name: "severity", rule: "$severity" }, { name: "agent_time", rule: "$time", conversion: "timeConverter" } ] }, filter: { presend: "RestClientLam.js" }
The above example specifies the mapping of the REST Server event fields with the Moogsoft Onprem fields.
Note
The signature field is used by the LAM to identify correlated events.
State File
The REST Client LAM will save the last poll time of the request sent to the REST Server. This last poll time will be saved in a state file. The REST Client LAM will read/write the last poll time from/to this state file. The LAM will read/write from the state file by using the RestClientLam.js file.
To read and write the state file you have to modify the RestClientLam.js
.
Note
The state file is generated in the same folder where the config file is present e.g. $MOOGSOFT_HOME/config. The LAM generates the name of the state file as <proc_name>.state. Here the default proc_name (process name) is rest_client_lam, therefore, the state file name is rest_client_lam.state.
LAMBot Configuration
The LAMbot processes the event data received from RESTful Services. The LamBot RestClientLam.js
script can be configured for GET and POST requests. The following example shows a LAMbot configuration for GET request. This configuration primarily reads/writes the last poll time data from/to the state file.
var logger=LamBot.loadModule("Logger"); var constants=LamBot.loadModule("Constants"); /*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file or read data from the state file */ var firstTime = true; function onLoad() { /* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. Last poll time can be appended in the url for polling from the lass poll time; */ constants.load(); return; } function preClientSend(outboundEvent) { /* outBoundEvent contains below field which can be manipulated as per requirement 1. method - by default method is set to GET request 2. body - contains all the values of request_query_param from the config file 3. header - can contain additional headers */ return true; } function modifyResponse(inBoundEventData) /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here The inBoundEventData contains the following field which can be manipulated as per the requirement 1. responseData - the event data received from the rest server */ { return true; } function presend(event) { return( true ); } LamBot.filterFunction("presend"); LamBot.preClientSendFunction("preClientSend"); LamBot.modifyResponseFunction("modifyResponse");
The following example shows the LAMbot configuration for a POST request with a JSON payload in which the last poll time is being written and read from the state file.
var logger=LamBot.loadModule("Logger"); var constants=LamBot.loadModule("Constants"); /*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file or read data from the state file */ var firstTime= true; function onLoad() { /* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. Last poll time can be appended in the url for polling from the lass poll time; */ constants.load(); return; } function preClientSend(outBoundEvent) { /*outBoundEvent contains below field which can be manipulated as per requirement 1. method - by default method is set to GET request 2. body - contains all the values of request_query_param from the config file 3. header - can contain additional headers */ var body = outBoundEvent.value('body'); // reads body from outBoundEvent if(firstTime) { if(constants.contains("start") == true) { var start = constants.get("start"); // reads the last poll time value from the state file body['start'] = start; // replaces value of the start in body with a new value outBoundEvent.set('body',body); // resets value of body with modified body constants.save(); firstTime = false; } else { constants.put("start",body['start']); constants.save(); firstTime = false; } } else { constants.put("start",body['start']); constants.save(); } // methods that can be used here are'Post','Put',and 'Delete'; outBoundEvent.set("method","Post"); var properties = outBoundEvent.value("header"); // Header is used if the method used is Put or Delete or Post. Content type should always be defined otherwise it will give an error properties["Content-Type"] = "application/json"; outBoundEvent.set("header",properties); //If LAM is not able to convert the request_query_parm in an accepted JSON format, then the user can force the LAM to use the assigned inputs given below as a string, and use it in body of the request. //outboundEvent.set("body", //'{"action":"EventsRouter","method":"query","data":[{"params":{"severity":[5,4,3,2,1,0],"eventState":[0,1,2,3,4,5]},"limit":9}],"type":"rpc","tid":1}'); return true; } function modifyResponse(inBoundEventData) /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here The inBoundEventData contains the following field which can be manipulated as per the requirement 1. responseData - the event data received from the rest server */ { return true; } function presend(event) { // returning true, makes this an event on the MooMs bus, false will cause the LAM to discard the event. return( true ); } LamBot.filterFunction("presend"); LamBot.preClientSendFunction("preClientSend"); LamBot.modifyResponseFunction("modifyResponse");
The following example shows the LAMbot configuration for a POST request with an XML payload in which the last poll time is being written and read from the state file.
var logger=LamBot.loadModule("Logger"); var constants=LamBot.loadModule("Constants"); /*This variable is used to determine if the LAM is running for the first time, and accordingly we need to update the state file or read data from the state file */ var firstTime= true; function onLoad() { /* constants.load() loads the data form the state file. The state file usually has the last poll time saved in it. Last poll time can be appended in the url for polling from the lass poll time; */ constants.load(); return; } function preClientSend(outBoundEvent) { /*outBoundEvent contains below field which can be manipulated as per requirement 1. method - by default method is set to GET request 2. body - contains all the values of request_query_param from the config file 3. header - can contain additional headers */ var body = outBoundEvent.value('body'); // reads body from outBoundEvent if(firstTime) { if(constants.contains("start") == true) { var start = constants.get("start"); // reads the last poll time value from the state file body['start'] = start; // replaces value of the start in body with a new value outBoundEvent.set('body',body); // resets value of body with modified body constants.save(); firstTime = false; } else { constants.put("start",body['start']); constants.save(); firstTime = false; } } else { constants.put("start",body['start']); constants.save(); } // methods that can be used here are'Post','Put',and 'Delete'; outBoundEvent.set("method","Post"); var properties = outBoundEvent.value("header"); // Header is used if the method used is Put or Delete or Post. Content type should always be defined otherwise it will give an error // The LAM cannot by default send an XML payload. To send an XML payload then set content type as text/xml and set the xml payload in the body properties["Content-Type"] = "text/xml"; outBoundEvent.set("header",properties); //If LAM is not able to convert the request_query_parm in an accepted xml format, then the user can force the LAM to use the assigned inputs given below as a string, and use it in body of the request. /* outboundEvent.set("body",'<env:Envelope xmlns:env=\"http://schemas.xmlsoap.org/soap/envelope/\"><env:Header/><env:Body><createRequest xmlns=\"http://moogsoft.com\" xmlns:emcf=\"http://xmlns.oracle.com/sysman/connector\"> <Summary xmlns=\"http://moogsoft.com\">CPU Utilization for 3 is 6.787%, crossed warning (5) or critical (10) threshold.</Summary><Description xmlns=\"http://moogsoft.com\">Event created in Enterprise Manager: \n Target information: \n Target Type: host Target Name: WIN-8U1RA5NAA6I Target URL: http://win-8u1ra5naa6i:7788/em//em/redirect?pageType=TARGET_HOMEPAGE&targetName=WIN-8U1RA5NAA6I&targetType=host</Description></createRequest></env:Body></env:Envelope>'); */ return true; } function modifyResponse(inBoundEventData) /* If you want to modify response data before injecting it into LAM for tokenizing, then the event data can be modified here The inBoundEventData contains the following field which can be manipulated as per the requirement 1. responseData - the event data received from the rest server */ { return true; } function presend(event) { // returning true, makes this an event on the MooMs bus, false will cause the LAM to discard the event. return( true ); } LamBot.filterFunction("presend"); LamBot.preClientSendFunction("preClientSend"); LamBot.modifyResponseFunction("modifyResponse");
The above examples have the following functions:
function onLoad(): This function is used to load data from state file. The state file has details of last poll time of the LAM. The state file is saved at the same location where the REST Client LAM config file is present. The statement
constants.load()
in this function loads the contents of the state file in the constant variable. This provides you some more alternatives for playing with data of the state file. As of now, only the last poll time will be saved, but if you want to save some other data which is to be used after restarting the LAM, then you can save this data in the state file. The name of the state file will be same as the config file name. For example, if the name of the config file isrest_client_lam.config
, then the corresponding state file name will berest_client_lam.state.
function preClientSend(): This function extracts the information about the URL from the parameters given in the field
request_query_param
of the config file, combines all the information and creates the URL. This function also appends the last poll time read from the state file to the URL. The created URL is then sent to the LAM which hits the REST Server for events and alarm data. Theconstants.save()
saves the poll time to the state file which is then used in the next poll.function modifyResponse(): This function makes changes to any response received from the REST Server. This function is used when the response data received from the REST Server needs any changes. If no changes are required, then do not edit this function. If any modification is done, then after modification the event data is sent to the LAM for extraction, tokenization, and mapping. For Example, if you are receiving the event data in XML, then you have to convert it into JSON format for the LAM to process it further. This conversion can be carried out in the
modifyResponse()
function.function presend(): This function makes any changes to the event data before sending it to MooMS.
Note
The LAMbot configuration is done to handle things that cannot be handled by the config file. You can make changes to the file according to their requirements.
Service Operation Reference
Process Name | Service Name |
---|---|
| restclientlamd |
Start the LAM Service:
service restclientlamd start
Stop the LAM Service:
service restclientlamd stop
Check the LAM Service status:
service restclientlamd status
If the LAM fails to connect to one or more REST API sources, Moogsoft Onprem creates an alert and writes the details to the process log. Refer to the logging details for LAMs and integrations for more information.
Command Line Reference
To see the available optional attributes of the rest_client_lam
, run the following command:
rest_client_lam --help
The rest_client_lam
is a command line executable, and has the following optional attributes:
Option | Description |
---|---|
| Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified. |
| Displays all the command line options. |
| Displays the component’s version number. |
| Specifies the level of debugging. By default, user gets everything. In common with all executables in Moogsoft Onprem, having it set at that level can result in a lot of output (many messages per event message processed). In all production implementations, it is recommended that log level is set to WARN. This ensures only warning, error and fatal messages are recorded. |