Moogsoft Docs

Syslog

Syslog provides a way for network devices to send event messages to a logging server – usually known as a Syslog server. The Syslog protocol is supported by a wide range of devices and can be used to log different types of events. For example, a router might send messages about users logging on to console sessions, while a web-server might log access-denied events. This document describes the configuration required to establish a connection between the Syslog Server and the Syslog LAM.

The workflow of gathering alarms from a Syslog server and publishing it to Moogsoft AIOps is outlined below.

Process Workflow for a Socket connection:

Process Workflow for getting events from a Log file:

  1. Syslog LAM reads configuration from the syslog_lam.conf file.
  2. In Socket connection, the Syslog LAM connects to the host/IP and listens on the port, given in the syslog_lam.conf file, and receives events from the server.
  3. For getting events from a log file, the Syslog LAM reads data on the file located on the same system where the LAM is installed.
  4. The received/read event data is in a text format.
  5. The events are parsed and converted into normalized Moogsoft AIOps events.
  6. The normalized events are then published to MooMS bus.

Syslog LAM Configuration

The events received from a Syslog server are processed according to the configurations in the syslog_lam.conf file. The processed alarms are published to Moogsoft AIOps.

The configuration file contains a JSON object. At the first layer of the object, LAM has a parameter called config, and the object that follows config has all the necessary information to control the LAM.

The following configuration sections are available in the Syslog LAM configuration file:

Monitor

Agent

HA Configuration

Mapping

Constants and Conversions

Monitor

The Syslog LAM fetches the events from the Syslog Server. You can configure parameters here to establish a connection with Syslog:

General

Field Type Description Example
name and class
String Reserved fields: do not change. Default values are Syslog Monitor and CSyslogMonitor.
monitoring_type
String This is the monitoring type. It can be either SOCKET or FILE . The SOCKET is used for establishing a TCP or UDP connection, while FILE is used for directly monitoring a log file.
protocol_type
String

Enter the protocol type here. This field has to be entered if you have selected monitoring_type as SOCKET .


address
String Enter the IP address of the server here. This field has to be entered here if you have selected monitoring_type as SOCKET , and comment out this field if you have selected monitoring_type as FILE.
port
Integer Enter the port of the server here. This field has to be entered here if  you have selected monitoring_type as SOCKET , and comment out this field if you have selected monitoring_type as FILE.
target  
String

Enter the log file from which you want to recieve alerts. This field has to be entered here if you have selected monitoring_type as FILE , and comment out this field if you have selected monitoring_type as SOCKET.


load_at_start 
Boolean If this flag is set to true , then the LAM will process the entire contents of the target file and wait for any additional data to be written to the file. This is useful if you are loading bulk-data into the system for analysis.
exit_after_initial_load 
Boolean If set to true , the LAM will read the content of the target and when it has processed all the data, the LAM will exit.
event_ack_mode
String

The event_ack_mode determines when an event received by the Syslog LAM should be acknowledged. The following options are available:

queued_for_processing : Event is acknowledged after it is added to the Moolet queue.

event_processed : Event is acknowledged after it is processed in a Moolet queue.

Note

If you have not specified the mode in the event_ack_mode field, then by default, queued_for_processing mode will be used.


Note

For monitoring_type set to FILE , it is mandatory to set load_at_start and exit_after_initial_load fields as true .

Example

Config File
monitor:
        {
            name        			: "Syslog Monitor",

            class       			: "CSyslogMonitor",

            monitoring_type 		: "SOCKET",

            protocol_type   		: "UDP",

            address        			: "localhost",

            port        			: 514,

            target                  : "bow.syslog.log",
            
            load_at_start           : true,

            exit_after_initial_load : false,

            event_ack_mode 			: "queued_for_processing"
        },

Agent

Agent allows you to define two parameters:

Field
name This is the agent name, the events sent to MooMS by the Syslog LAM are identified by the agent name in the log. In this LAM, the agent name is Syslog.
log Syslog LAM will write its ingress contents in the file s yslog_lam.log located at /var/log/moogsoft/.

HA Configuration

Refer to the document Integrations HA Configuration

Data Parsing

Any received data needs to be broken up into tokens. When you have the tokens, you can start assembling an event. The data received from a Syslog server or a Log file can be called as message.  A typical BSD Syslog message has the following groups:

  • Pri : Pri is a combination of severity and facility.
  • Header : Header contains a combination of Timestamp and Host.
  • Message : The message contains the combination of Tag and content

Although BSD messages should contain these components, few rarely do. The PRI (severity and facility combination) is rarely used, with most messages starting with an RFC compliant timestamp. The LAM has to be configured in a way that it accommodates both BSD and near-­BSD compliant messages (with and without the PRI). The implementation specific changes can be made in the regular expression used to determine a valid or invalid message.

Like BSD messages, the Syslog LAM can receive any type of Syslog message, e.g. kiwi, cisco, etc. For configuring the Syslog LAM, you should know the type of message received by the LAM and its regular expression. Some regular expressions for different types of Syslog messages are given in the parsing section of the configuration file. A new regular expression can be added to the parsing section, for this you must know the format of received message and its regular expression.

Syslog LAM supports multiple types of Syslog messages. After receiving the message, it is parsed one by one using all the regular expression present in the parsing section. If there is a match, then the parsed message is sent to the variable section. If the message does not match with any of the given regular expression, then it is parsed using the regular expression pattern_name: "default" in the parsing section. This captures the complete message and assigns it to a variable message in the variable section.

Note

For every regular expression of a Syslog message type in parsing, there should be a corresponding section in variables.

In the Parsing section, the tokenising of the received message is done using either a regex subgroup or delimiters. The tokenised messages can then be assigned in the variables section and forwarded to the lambot SyslogLam.js. The SyslogLam.js breaks the tokenised message and then assigns it to the respective Moogsoft AIOps fields and publishes the event to MooMS.

         parsing:
             [
                 {
                     pattern_name  : "pattern1",
                     pattern  	   : "(?mU)^((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}\\s(?:(?:\\d{2}:){2}\\d{2}))\\s((?:(?:[a-z0-9]+                (?:\\.|:|-|_)*){1,}))\\s(.*)$",
                     tokeniser_type: "delimiters",
					 action        : "accept",
                     capture_group : 0,
                     delimiters    :
                    				{
                         				ignoreQuotes: true,
                         				stripQuotes : true,
                         				ignores		: "",
                         				delimiter	: ["||","\r"]
                   					}
                 },
                 
                 {
                     pattern_name  : "pattern2",
                     pattern  	   : "(?mU)^((?:<\\d{1,3}>)?)\\s*((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}\\s(?:(?:\\d{2}:){2}\\d{2}))\\s((?:(?:[a-z0-9]+(?:\\.|:|-|_)*){1,}))\\s(.*)$",
                     tokeniser_type: "regexp_subgroups",
					 action		   : "accept"
                 },
                 
                 {
                     pattern_name  	: "pattern3",
                     pattern       	: "(?mU)^(?:\\[((?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)\\s+(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+(?:\\d{1,2})\\s+(?:(?:[0-1]\\d|2[0-3]):(?:[0-5]\\d):(?:[0-5]\\d).(?:\\d{1,3}))\\s+(?:\\d{4}))\\])\\s*((?:[a-zA-Z0-9]+(?:\\.|-|_|\\[|\\]|\\(|\\))*){1,})\\:\\s+(.*)$",
                     tokeniser_type	: "regexp_subgroups", 
					 action			: "reject"                    
                 }
              ],

The fields which are configured in the parsing section are as follows:

  • pattern name : Enter the type of syslog message that is received by the LAM, e.g. Dell syslog.
    • pattern : The regular expression which will be matched with messages received from the server. The string which matches the regular expression is extracted from the message.
      Regular expressions and their explanation is as follows:

      /^((?:<\d{1,3}>)?)\s*((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\  d{1,2}\s(?:(?:\d{2}:){2}\d{2}))\s((?:(?:[a-z0-9-]+(?:\.|:)*){1,}))\s(.*)$/

      Expression

      Description

      ^

      The beginning of a line

      ((?:<\d{1,3}>)?)

      Optionally followed by a “<” followed by between 1 and 3 digits followed by a “>”

      \s*

      Followed by 0 or more spaces

      ((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)

      Followed by one of Jan, Feb, Mar etc.

      \s+

      Followed by at least one space

      \d{1,2}

      Followed by one or two digits

      \s

      Followed by a single space

      (?:(?:\d{2}:){2}\d{2}))

      Followed by 2 digits followed by a “:” followed by 2 digits followed by a “:” followed by 2 digits (hh:mm:ss) Followed by a single space

      \s

      Followed by a single space

      ((?:(?:[a-­‐z0-­‐9-­‐]+(?:\.|:)*){1,}))

      Followed by something like an IP address (v4 or 6) or a host name (10.0.0.1, a:b:c:d, a.b.c.d)

      \s

      Followed by a space

      (.*)$

      Followed by anything up to the end of line
  • tokeniser_type : The 2 types of tokeniser used for tokenising are as follows:
    • regexp_subgroups : To tokenise based on regular expression subgroups, enter regex_subgroups in the tokeniser_type field. This tokenising method tokenises the extracted string based on groups in a message. An expression in the parenthesis in the regular expression denotes a group. For example, the expression ((?:?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}) is a group which contains the date and time.
    • tokeniser : To tokenise based on delimiters, enter delimiters in the tokeniser_type field. The extracted string is tokenised based on the delimiters present in it.
  • action : This field is used for including or rejecting the matched regex text in the received event. If set to " accept ", then the matched regex text in event will be sent for further processing. If set to " reject ", then the matched regex text will be rejected.

  • capture_group : This is the group in the message that has to be extracted and further used in the variables section. For example, if "1" is given in this field, then group 1 of the message is only picked for tokenising. This field will be used when delimiters are entered in the field tokeniser_type , and it can be commented for regexp_subgroups . You can use capture_group when only a particular group in the extracted string has to be tokenised.

    Note

    If "0" is entered, then all the groups of the message will be picked for tokenising.

    Note

    In the Syslog LAM, parsing is done using the regular expression. For tokenising, you can use either regexp_subgroups or delimiters.

Example of the tokenisation using regexp_subgroups

The parser searches for strings as per the expression defined in the pattern field. The extracted string is then tokenised based on the configuration done in the variable section. The regular expressions for some of the syslog formats are defined above. If you have to define regular expressions of a different format, then you can define the regular expression in the parsing section. As an example you can see below a section for the regular expression parsing of events received from a Dell server:

                {
                     pattern_name	: "pattern4",
                     pattern 		: "(?m)^\\s*((?:<\\d{1,3}>)?)\\s*((?:(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2})\\s+(?:(?:[0-1]\\d|2[0-3]):    (?:[0-5]\\d):(?:[0-5]\\d)))\\s+((?:[a-zA-Z0-9]+(?:\\.|-|_|\\[|\\]|\\(|\\))*){1,})\\s+(.*)$",
                     tokeniser_type : "regexp_subgroups",
					 action			: "accept"
                }
  • pattern_name : This is the name of the pattern.
  • pattern : This is the regular expression which will be matched with messages received from the server, e.g. "(?m)^\\s*((?:<\\d{1,3}>)?)\\s*((?:(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2})\\s+(?:(?:[0-1]\\d|2[0-3]):(?:[0-5]\\d):(?:[0-5]\\d)))\\s+((?:[a-zA-Z0-9]+(?:\\.|-|_|\\[|\\]|\\(|\\))*){1,})\\s+(.*)$".
  • tokeniser_type : The regexp_subgroups captures and then tokenises the message based on the regular expression given in the field pattern .

Example of the tokenisation using delimiters

Delimiters define how a line is split into tokens, also known as “tokenising”. For example, if you have a line of text data, it needs to be split up into a sequence of sub strings that are referenced by position from the start. So if you were processing a comma-separated file, where a comma separates each value, it would make sense to have the delimiter defined as a comma. Then the system would take all the text and break it up into tokens between the commas. The tokens could then be referenced by position number in the string starting from one, not zero.

For example, if the input string was “the,cat,sat,on,the,mat” and comma was used as a separator, then token 1 will be “the”, token 2 will be “cat” and so on.

There are complications when it comes to tokenisation and parsing. For example, if you say comma is the delimiter, and the token contains a comma, you will end up with that token containing a comma to be split into 2 tokens. To avoid this, it is recommended that you quote strings. You must allow the system to know whether it should strip or ignore quotes, and therefore, the stripQuotes and ignoreQuotes parameters will be used. Below is an example of parsing using delimiters for syslog messages received from a CISCO device:

               {
                       pattern_name		: "pattern2",
                       tokeniser_type	: "delimiters",
                       capture_group	: 1,
                       delimiters		:
                    						{
                         						ignoreQuotes: true,
                         						stripQuotes : true,
                         						ignores		: "",
                         						delimiter	: ["||","\r"]
                    						}                     
			   }

The above example specifies:

  • pattern_name : This is the name of the pattern.

  • tokeniser_type : To use delimiters for parsing, enter delimiters in this field.

  • capture_group : This is the group in the message that has to be extracted and further used in the variables section. For example, if "1" is given in this field, then group 1 of the Syslog message will only be picked for tokenising.

    Note

    If "0" is entered in the capture_group field, then all the groups of the message will be picked for tokenising.

  • ignoreQuotes :For strings that are quoted between delimiters, set ignoreQuotes to true , the LAM will look for delimiters inside the quote. For example, <delimiter>”hello “inside quote” goodbye”<delimiter> gives a token [hello inside quote goodbye].
  • stripQuotes : If set to true , it will remove start and end quotes from tokens. For example, “hello world” gives a token [hello world].
  • ignores :This field contains a list of characters to ignore. Ignored characters will never be included in tokens.

delimiter : This field contains the list of valid delimiters used to split strings into tokens. Here || and the end line character are the defined delimiters.

Note

If an empty string is mentioned in the delimiter field, then no LAM tokenisation will be done, instead the entire message will be taken into a single token.

Note

If the message does not match any regular expression given in the parsing section, then the LAM automatically sends the received message as a single line text to SyslogLam.js . The message is tokenised and sent to Moogsoft AIOps by the SyslogLam.js.

Also if you don't know a regular expression that can be used for tokenisation, then you can leave the parsing and variables section blank. The received message by the LAM will be then sent to SyslogLam.js as a single line text.

Variables

An event in a message is a positioned collection of tokens, and Moog allows you to name these positions. Naming of the positions helps to identify the tokens. In the below given example, token at position number 3 is host name, so the token at position 3 will be assigned to "host".

variables:
            {
                "pattern1":
                    [
                        { name: "pri",           position: 1 },
                        { name: "date",         position: 2 },
                        { name: "host",           position: 3 },
                        { name: "content",      position: 4 }
                    ],
    
                "pattern2":
                    [
                        { name: "date",         position: 2 },
                        { name: "host",           position: 3 },
                        { name: "content",      position: 1 },
                        { name: "message",         position: 4 }
                    ],
                    
                "pattern3":
                    [
                        { name: "date",         position: 1 },
                        { name: "host",           position: 2 },
                        { name: "content",      position: 3 },
                        { name: "message",         position: 4 }
                    ]
            },

For pattern1, position 1 is assigned to content, position 2 is assigned to date and so on. Positions start at 1, and go up.

Note

In the variable section, the groups pri, host, content and message are required from a syslog message or the complete message, as received in the case of default parsing.

After assigning the tokenised message to variables, the message will be sent to SyslogLam.js lambot for further processing.

Mapping

The rules assignation for Syslog in the mapping section is used only for temporary values. The standard values are defined in the rules section to ensure that as complete an event object is passed to SyslogLam.js as possible. Setting temporary values has little or no impact, but helps prevent downstream errors as the event is turned into an alert.

mapping :
        {
            catchAll: "overflow",
            rules:
            [
                { name: "signature",rule:        "Signature" },
                { name: "source_id",rule:        "SourceId" },
                { name: "external_id",rule:      "ExternalId" },
                { name: "manager",rule:          "Syslog" },
                { name: "source",rule:           "$host" },
                { name: "class",rule:            "Syslog" },
                { name: "agent",rule:            "$LamInstanceName" },
                { name: "agent_location",rule:  "Location" },
                { name: "type",rule:            "$application" },
                { name: "severity",rule:        "0",conversion: "stringToInt" },
                { name: "description",rule:        "$message" },
                { name: "agent_time",rule:        "$moog_now" }
            ]
        },
        filter:
        {
            modules: [
                    "RegExpUtil.js",
                    "SyslogUtil.js",
                    "SyslogEvents.js",
                    "LamUtility.js"
            ],
            presend: "SyslogLam.js"
        }
    }

In this section, only 2 variables are defined, the current time ($moog_now), which sets the time of the events in the Moogsoft AIOps UI with the time of processing of the event by Moog and the agent name ($LamInstanceName) which displays the name given in the name field of the Agent section on Moogsoft AIOps UI.

Data not mapped to Moogsoft AIOps Fields goes into "Custom Info".

Constants and Conversions

Field Description Example
Severity and sevConverter has a conversion defined as sevConverter in the Conversions section, this looks up the value of severity defined in the severity section of constants and returns back the mapped integer corresponding to the severity
severity:
{
"CLEAR" : 0,
"INDETERMINATE" : 1,
"WARNING" : 2,
"MINOR" : 3,
"MAJOR" : 4,
"CRITICAL" : 5
}
sevConverter:
 {
     lookup  : "severity",
     input   : "STRING",
     output  : "INTEGER"
  },       
        
stringToInt used in a conversion, which forces the system to turn a string token into an integer value
stringToInt:
{
    input  : "STRING",
    output : "INTEGER"
},
timeConverter used in conversion which forces the system to convert time. If epoch time is to be used, then timeFormat mentioned in timeConverter should be commented. Otherwise, the user should provide the timeFormat
timeConverter:
{
    timeFormat : "yyyy-MM-dd'T'HH:mm:ss.SSS",
    input      : "STRING",
    output     : "INTEGER"
}

Example

Example Constants and Conversions
constants:
        {
            severity:
            {
                "CLEAR"         : 0,
                "INDETERMINATE" : 1,
                "WARNING"       : 2,
                "MINOR"         : 3,
                "MAJOR"         : 4,
                "CRITICAL"      : 5
            }
        },
        conversions:
        {
            sevConverter:
            {
                lookup: "severity",
                input:  "STRING",
                output: "INTEGER"
            },
	    	            
            stringToInt:
            {
                input:      "STRING",
                output:     "INTEGER"
            },     
 
			timeConverter:
            {
                timeFormat: "yyyy-MM-dd'T'HH:mm:ss",
                input:      "STRING",
                output:     "INTEGER"
            }  
            
        },

SyslogLam.js

The filter defines whether Moog uses a LamBot. If the presend value is commented, then no filter is applied to the events produced and all events will be sent unchanged to the MooMS bus. If a LamBot is defined, then the event is passed to the LamBot. The Syslog LAM uses the lambot syslogLam.js .

        filter:
        {       
     		modules: [                                    
						"RegExpUtil.js",                               
						"SyslogUtil.js",                                
						"SyslogEvents.js",                                 
						"LamUtility.js"                    
					  ],           
			presend: "SyslogLam.js"       
		}


In the above example, the presend is set to " SyslogLam.js " which is the Syslog lambot that extracts the values from the tokenised message and after assembling the values, publishes the events to MooMS.

The modules defined in the modules section are used by SyslogLam.js to extract values. In the above example, the following modules are defined:

  • RegExpUtil.js : A module holding common regular expressions and regular expression related functions (search and replace).
  • SyslogUtil.js : A syslog specific module, defining a “message” object and instance methods, and other methods used in the lambot.
  • SyslogEvents.js :  A library of actions to take for specific events, based on an EventId. Message actions are defined by functions – so a single function can cover many message types.
  • LamUtility.js : Contains functions that are repeatedly used in a lambot (date, debug   etc.).

Note

You can edit the SyslogLam.js to process specific Syslog messages. The lambot and the modules can be found in the $MOOG_HOME/bots/lambots directory.

The workflow of syslog lambot is as follows:

  • Determine Severity : Associate one of the standard Moog severities with the event.
  • Determine Context : Determines the entities about  this event, such as host, interfaces, ports etc.
  • Determine Eventid : Determines the event id of the event.
  • Determine Signature : Create a valid Moog signature that will allow effective de-­duplication for the events.
  • Execute a policy : This will execute a message having an optional specific behavior associated with it (e.g. message transformation).
  • Final Processing : Allow any last stage processing before the event is dispatched (filtering etc.).

Severity Reference

Moogsoft Severity Levels
severity:
        {
            "CLEAR" 		: 0,
            "INDETERMINATE" : 1,
            "WARNING" 		: 2,
            "MINOR" 		: 3,
            "MAJOR" 		: 4,
            "CRITICAL" 		: 5,
            
        }
Level Description
0 Clear
1 Indeterminate
2 Warning
3 Minor
4 Major
5 Critical

Service Operation Reference

Process Name Service Name
syslog_lam sysloglamd

Start the LAM Service:

service sysloglamd start

Stop the LAM Service:

service sysloglamd stop

Check the LAM Service status:

service sysloglamd status

Command Line Reference

To see the available optional attributes of the syslog_lam, run the following command:


syslog_lam --help

The syslog_lam is a command line executable, and has the following optional attributes:

Option Description

--config

Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified.

--help Displays all the command line options.
--version Displays the component’s version number.

--loglevel

Specifies the level of debugging. By default, user gets everything. In common with all executables in MOOG, having it set at that level can result in a lot of output (many messages per event message processed).In all production implementations, it is recommended that log level is set to WARN. This ensures only warning, error and fatal messages are recorded.

Performance Information

Minimum requirement
Component Value
CPU 2 core
RAM 4 GB
Operating System CentOS Linux release 6.7

Version

LAM Version Tool Version Verified By
1.0

Generic

Moogsoft
1.1

Generic

Moogsoft