Syslog LAM

The Syslog LAM fetches the events from the Syslog Server. You can configure parameters here to establish a connection with Syslog:

name and class

monitoring_type

protocol_type

address

port

target  

load_at_start 

exit_after_initial_load 

event_ack_mode

monitor:
        {
            name                                : "Syslog Monitor",

            class                               : "CSyslogMonitor",

            monitoring_type             : "SOCKET",

            protocol_type               : "UDP",

            address                             : "localhost",

            port                                : 514,

            target                  : "bow.syslog.log",
            
            load_at_start           : true,

            exit_after_initial_load : false,

            event_ack_mode                      : "queued_for_processing"
        },

The Agent and Process Log sections allow you to configure the following properties:

Any received data needs to be broken up into tokens. When you have the tokens, you can start assembling an event. The data received from a Syslog server or a Log file can be called as message. A typical BSD Syslog message has the following groups:

Although BSD messages should contain these components, few rarely do. The PRI (severity and facility combination) is rarely used, with most messages starting with an RFC compliant timestamp. The LAM has to be configured in a way that it accommodates both BSD and near-­BSD compliant messages (with and without the PRI). The implementation specific changes can be made in the regular expression used to determine a valid or invalid message.

Like BSD messages, the Syslog LAM can receive any type of Syslog message, e.g. kiwi, cisco, etc. For configuring the Syslog LAM, you should know the type of message received by the LAM and its regular expression. Some regular expressions for different types of Syslog messages are given in the parsing section of the configuration file. A new regular expression can be added to the parsing section, for this you must know the format of received message and its regular expression.

Syslog LAM supports multiple types of Syslog messages. After receiving the message, it is parsed one by one using all the regular expression present in the parsing section. If there is a match, then the parsed message is sent to the variable section. If the message does not match with any of the given regular expression, then it is parsed using the regular expression pattern_name: "default" in the parsing section. This captures the complete message and assigns it to a variable message in the variable section.

In the Parsing section, the tokenising of the received message is done using either a regex subgroup or delimiters. The tokenised messages can then be assigned in the variables section and forwarded to the lambot SyslogLam.js. The SyslogLam.js breaks the tokenised message and then assigns it to the respective Moogsoft Onprem fields and publishes the event to MooMS.

         parsing:
             [
                 {
                     pattern_name  : "pattern1",
                     pattern       : "(?mU)^((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}\\s(?:(?:\\d{2}:){2}\\d{2}))\\s((?:(?:[a-z0-9]+                (?:\\.|:|-|_)*){1,}))\\s(.*)$",
                     tokeniser_type: "delimiters",
                                         action        : "accept",
                     capture_group : 0,
                     delimiters    :
                                                {
                                                        ignoreQuotes: true,
                                                        stripQuotes : true,
                                                        ignores         : "",
                                                        delimiter       : ["||","\r"]
                                                        }
                 },
                 
                 {
                     pattern_name  : "pattern2",
                     pattern       : "(?mU)^((?:<\\d{1,3}>)?)\\s*((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}\\s(?:(?:\\d{2}:){2}\\d{2}))\\s((?:(?:[a-z0-9]+(?:\\.|:|-|_)*){1,}))\\s(.*)$",
                     tokeniser_type: "regexp_subgroups",
                                         action            : "accept"
                 },
                 
                 {
                     pattern_name       : "pattern3",
                     pattern            : "(?mU)^(?:\\[((?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)\\s+(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+(?:\\d{1,2})\\s+(?:(?:[0-1]\\d|2[0-3]):(?:[0-5]\\d):(?:[0-5]\\d).(?:\\d{1,3}))\\s+(?:\\d{4}))\\])\\s*((?:[a-zA-Z0-9]+(?:\\.|-|_|\\[|\\]|\\(|\\))*){1,})\\:\\s+(.*)$",
                     tokeniser_type     : "regexp_subgroups", 
                                         action                 : "reject"                    
                 }
              ],

The fields which are configured in the parsing section are as follows:

Example of the tokenisation using regexp_subgroups

The parser searches for strings as per the expression defined in the pattern field. The extracted string is then tokenised based on the configuration done in the variable section. The regular expressions for some of the syslog formats are defined above. If you have to define regular expressions of a different format, then you can define the regular expression in the parsing section. As an example you can see below a section for the regular expression parsing of events received from a Dell server:

                {
                     pattern_name       : "pattern4",
                     pattern            : "(?m)^\\s*((?:<\\d{1,3}>)?)\\s*((?:(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2})\\s+(?:(?:[0-1]\\d|2[0-3]):    (?:[0-5]\\d):(?:[0-5]\\d)))\\s+((?:[a-zA-Z0-9]+(?:\\.|-|_|\\[|\\]|\\(|\\))*){1,})\\s+(.*)$",
                     tokeniser_type : "regexp_subgroups",
                                         action                 : "accept"
                }

Example of the tokenisation using delimiters

Delimiters define how a line is split into tokens, also known as “tokenising”. For example, if you have a line of text data, it needs to be split up into a sequence of sub strings that are referenced by position from the start. So if you were processing a comma-separated file, where a comma separates each value, it would make sense to have the delimiter defined as a comma. Then the system would take all the text and break it up into tokens between the commas. The tokens could then be referenced by position number in the string starting from one, not zero.

For example, if the input string was “the,cat,sat,on,the,mat” and comma was used as a separator, then token 1 will be “the”, token 2 will be “cat” and so on.

There are complications when it comes to tokenisation and parsing. For example, if you say comma is the delimiter, and the token contains a comma, you will end up with that token containing a comma to be split into 2 tokens. To avoid this, it is recommended that you quote strings. You must allow the system to know whether it should strip or ignore quotes, and therefore, the stripQuotes and ignoreQuotes parameters will be used. Below is an example of parsing using delimiters for syslog messages received from a CISCO device:

               {
                       pattern_name             : "pattern2",
                       tokeniser_type   : "delimiters",
                       capture_group    : 1,
                       delimiters               :
                                                                {
                                                                        ignoreQuotes: true,
                                                                        stripQuotes : true,
                                                                        ignores         : "",
                                                                        delimiter       : ["||","\r"]
                                                                }                     
                           }

The above example specifies:

delimiter: This field contains the list of valid delimiters used to split strings into tokens. Here || and the end line character are the defined delimiters.

The filter defines whether Moogsoft Onprem uses a LamBot. If the presend value is commented, then no filter is applied to the events produced and all events will be sent unchanged to the MooMS bus. If a LamBot is defined, then the event is passed to the LamBot. The Syslog LAM uses the lambot syslogLam.js.

        filter:
        {       
                modules: [                                    
                                                "RegExpUtil.js",                               
                                                "SyslogUtil.js",                                
                                                "SyslogEvents.js",                                 
                                                "LamUtility.js"                    
                                          ],           
                        presend: "SyslogLam.js"       
                }

In the above example, the presend is set to "SyslogLam.js" which is the Syslog lambot that extracts the values from the tokenised message and after assembling the values, publishes the events to MooMS.

The modules defined in the modules section are used by SyslogLam.js to extract values. In the above example, the following modules are defined:

The workflow of syslog lambot is as follows: