Page tree
Skip to end of metadata
Go to start of metadata

Purpose

The Log file LAM monitors data being appendedAIOps to a regular UNIX file; parsing this data according to the LAM’s configuration file, and constructing events that are passed on to the MOOMs bus, and in common with all other LAMS, published to the subject “/Events”. 

You can configure how the Logfile LAM processes data. MOOG expects the data to be written to the file as a series of tokens that are delimited in a predictable, repeatable and regular way. 

Socket LAM conforms to the Java platform standard on Time Conversion. See Data ingestion 


Command line attributes

The executable is a command line executable that can be run as a service daemon, and takes four attributes, which can be viewed by typing: 


% logfile_lam --help

 Option 

 Explanation

 --config 

Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified

 --help 

Displays all the command line options

 --version 

Displays the component’s version number

 --loglevel 

Specifies the level of debug. By default, you get everything. In common with all executables in MOOG, having it set at that level can result in a lot of output (many messages per event message processed). In all production implementations it is recommended that log level be set to WARN, which only informs you of matters of importance

logfile_lam.conf

There are two files that control the behaviour of the log file LAM:

  • logfile_lam.conf . This contains all the various controls for how events are processed. You must specify --config on the command line, otherwise, the system will look for logfile_lam.conf , in the $MOOGSOFT_HOME/config directory
  • LogfileLam.js . All LAMs can take a LamBot

The config file contains a JSON object. All config in the MOOG system is presented as a JSON object. At the first layer of the object, you have a parameter called config, and the object that follows config has all the necessary information to control the LAM. 

Quoting

In some instances the attribute strings are quoted. Our JSON parser requires the standard quoting for all strings. 

Hash

You can comment out lines by prepending them with a hash. 

logfile_lam.conf walk through

Monitor object

monitor defines the object to be monitored: 


monitor:
         {
            name : "Log File Monitor",
            class : "CFileMonitor",
            target : "./test.log",
            load_at_start : true,
            exit_after_initial_load: false
         },



The above example specifies: 

Object
Description
name and classReserved for future use and should be left as the defaults, Log File Monitor and CFileMonitor 
targetWhere to find the file that you are going to monitor for data being appended to it. In this instance, the target is set to monitor the file ./test.log. The file is written as a relative pathname, but in production implementations, you should always have an absolute pathname
load_at_start

A flag that if set to true means the log file LAM will process the entire contents of the target file and then wait for any additional data to be written to the file. This is useful if you are bulk-loading data into the system for analysis 

exit_after_inital_load

If set to TRUE, the LAM will read the contents of the target and when it has processed all the data, the LAM will exit cleanly


At start-up the log file LAM will open test.log and look for and monitor data being written to the file. Moogsoft uses Berkeley Unix Kernel Queues for file monitoring, which is an efficient way of treating a file as if it were a network socket, resulting in true real-time reading of data from a file. 

Agent configuration object

agent allows you to define two parameters: agent and log. 


agent:
 {
 name : "ACME_EVENTS"
 #log : "./logfile_lam.log"
 },


The above example specifies: 

  • The agent name, in this instance, ACME_EVENTS. When constructing the event that will be sent to MooMs, you can refer to the value that you give for name
  • log, which is a way of logging and capturing a permanent record of what the log file LAM is receiving. In this instance the log file LAM will write its ingress contents to ./logfile_lam.log

If you have many Logfile LAMs all set up monitoring different files, you can set the name for the Logfile LAM that is put into the event that is sent to the system.

Having a value for log is optional. Moogsoft recommends that log is used if you are collecting data, and you want to process the data later. 

Data parsing

Any received data needs to be broken up into tokens. Once you have the tokens, you can start assembling an event. There are a number of parameters that allow you to control how this will work. The first two are a start and end character. The square brackets [] are the JSON notation for a list. You can have multiple start and end characters. The system considers an event as all of the tokens between any start and end character. 


start: [],
end: ["\n"],


The above example specifies:

  • There is nothing defined in start; however, a carriage return (new line) is defined as the end character

In the example above, the LAM is expecting a whole line to be written followed by a return, and it will process the whole line as one event. 

Set up carefully, you can accept multi-line events. 

For correct parsing of incoming events, the start tokens should not be a subset of any end token and vice versa. For example: 

start : [ "|event|" ],
end  : [ "|event|\n"] 
may result in some unpredictable parsing because the start token "|event|" is a subset of the end token. So if the start token is encountered before the intended end token is met the parser will think that it is the start of a new event and stop the current stream at this point and create an undesirable event. If a configuration like this is in place, and the source cannot be changed, then the recommendation is to use the regular expression parser instead of the start_and_end parser. This does not have the same token subset restriction


Delimiters

Delimiters define how a line is split into tokens – “tokenising”. For example, if you have a line of text data, it needs to be split up into a sequence of sub strings that are referenced by position from the start. So if you were processing a comma-separated file, where a comma separates each value, it would make sense to have the delimiter defined as a comma. Then the system would take all the text between start and end and break it up into tokens between the commas. The tokens could then be referenced by position number in the string starting from one, not zero. 

For example if the input string was “the,cat,sat,on,the,mat” and comma was used as a separator, token 1 would be “the”, token 2 “cat” and so on. 

Be aware, there are complications when you come to tokenisation and parsing. For example, if you say comma is the delimiter, and the token contains a comma, you will end up with that token containing a comma to be split into two tokens. To avoid this it is recommended that you quote strings. You must then allow the system to know whether it should strip or ignore quotes, hence the stripQuotes and ignoreQuotes parameters. 


ignoreQuotes: true,
 stripQuotes: false,
 ignores: "",
 delimiter: [",","\r"]


The above example specifies: 

  • If you have strings that are quoted between delimiters, ignoreQuotes set to true will look for delimiters inside the quote. For example, <delimiter>”hello “inside quote” goodbye”<delimiter> gives a token [hello inside quote goodbye]
  • Setting stripQuotes to true removes start and end quotes from tokens. For example, “hello world” gives a token [hello world]
  • ignores is a list of characters to ignore. Ignored characters are never included in tokens
  • Delimiter is the list of valid delimiters used to split strings into tokens

Variables

For each event in the file, there is a positioned collection of tokens. MOOG enables you to name these positions so if you have a large number of tokens in a line, of which you are interested in only five or six, instead of remembering it is token number 32, you can call token 32 something meaningful. 


variables:
    [
        { name: "Identifier", position: 1 },
        { name: "Node", position: 4 },
        { name: "Serial", position: 3 },
        { name: "Manager", position: 6 },
        { name: "AlertGroup", position: 7 },
        { name: "Class", position: 8 },
        { name: "Agent", position: 9 },
        { name: "Severity", position: 5 },
        { name: "Summary", position: 10 },
        { name: "LastOccurrence",position: 1 }
    ],


The above example specifies: 

  • position 1 is assigned to Identifier; position 4 is assigned to Node and so on
  • Positions start at 1, and go up rather than array index style counting from 0

This is important because at the bottom of the file, logfile_lam.conf there is a mapping object that configures how MOOG assigns to the attributes of the event that is sent to MooMs, values from the tokens that are parsed. For example, in mapping there is a value called rules, which is a list of assignments. 


rules:
     [
         { name: "signature",rule: "$Node:$Serial" },
         { name: "source_id",rule: "$Node" },
         { name: "external_id",rule: "$Serial" },
         { name: "manager",rule: "$Manager" },
         { name: "source",rule: "$Node" },
         { name: "class",rule: "$Class" },
         { name: "agent",rule: "$LamInstanceName" },
         { name: "agent_location",rule: "$Node" },
         { name: "type",rule: "$AlertGroup" },
         { name: "severity",rule: "$Severity", conversion: "sevConverter" },
         { name: "description",rule: "$Summary" },
         { name: "first_occurred",rule: "$LastOccurrence" ,conversion: "stringToInt"},
         { name: "agent_time",rule: "$LastOccurrence",conversion: "stringToInt"}
     ]


The example above specifies: 

  • the first assignment name: "signature",rule:"$Node:$Serial"  ("$Node:$Serial is a string with $ syntax) means for signature take the tokens called Node and Serial and form a string with the value of Node followed by a colon followed by the value of Serial and call that signature in the event that is sent up to the system

You define a number of these rules covering the base attributes of an event. For reference, the system expects a minimum set of attributes in an event that are shown in this particular section. 

Constants and conversions

There are a number of these rules, such as severity where there is a conversion defined. The following example looks up the value of severity and returns the mapped integer on the other side. 


sevConverter:
            {
                lookup: "severity",
                input: "STRING",
                output: "INTEGER"
            },
severity:
            {
                "CLEAR" : 0,
                "INDETERMINATE" : 1,
                "WARNING" : 2,
                "MINOR" : 3,
                "MAJOR" : 4,
                "CRITICAL" : 5,
                moog_lookup_default : 3
            }


The above example specifies: 

  • sevConverter, which references a conversion definition in the conversions object
  • The sevConverter uses a look up table lookup"severity"  to reference a table named severity defined in the constants section

In this example the conversion takes as its input a string with a textual value of severity. From this it looks in the severity conversion table for a matching value and then returns the mapped value converted to an integer.

moog_lookup_default can be used to specify a default value when an event is received which does not map to one of the values listed.

If moog_lookup_default setting is not used and an event is received which does not map to one of the other specifically listed values, the event will NOT be processed.

stringToInt is used in a unix timestamp conversion, which forces the system to turn a string token into an integer value. 


stringToInt:
 {
   input: "STRING",
   output: "INTEGER"
 }


The filter defines whether Moog uses a LamBot. If you comment out the presend value, no filter is applied to the events produced and all events are sent unchanged to the MooMs bus. If a LamBot is defined, for every event the system assembles using this configuration, the event is passed to the LamBot (via the presend function defined in the LamBot).


filter:
 {
   presend: "LogfileLam.js"
 }


You must therefore define a presend function in your JavaScript file. 

The return value of the presend function will determine whether the event is sent on to the MooMs bus. The presend function can also define sub-streams that events are sent out on, so events can be sent to different farmd’s. 

You can provide an additional parameter to presend called modules that takes a JSON list. The JSON list is a list of optional JavaScript files that are loaded into the context of the LAMBot and executed; thus, you can share modules between LAMs. For example, you can write a generic Syslog processing module that is used in both the Socket LAM and the Logfile LAM. You therefore do not need to needlessly replicate code in the Moobot for each of the LAM’s. 

JSON events

The other capability of all LAMs is the native ability to consume JSON events. You must have a start and end carriage return as it is expecting a whole JSON object following the carriage return. 

Under parsing you have: 


end: ["\n"],


For the delimiter you have:  


delimiter: ["\r"]


JSON is a sequence of attribute/value, and the attribute is used as a name. Under the mapping section in the LAM configuration file, you must define the following attribute builtInMapper: "CJsonDecoder". It automatically populates, prior to the rules being run, all of the values contained in the JSON object. 

For example if the JSON object to be parsed was: 


{"Node" : "acmeSvr01","Severity":"Major"...}\n


The attributes available to the rules in the mapping section would be $Node=”acmeSvr01”, $Severity=”Major” and so on. 

catchAll

If you have an attribute that is never referenced in a rule, for example “enterprise trap number” which is never mapped into the attribute of an event, they are collected and placed as a JSON object in a variable called defined here and passed as part of the event. 


  • No labels