Page tree
Skip to end of metadata
Go to start of metadata

Overview

Syslog is a way for network devices to send event messages to a logging server – usually known as a Syslog server. The Syslog protocol is supported by a wide range of devices and can be used to log different types of events. For example, a router might send messages about users logging on to console sessions, while a web-server might log access-denied events. This document describes the configuration required to establish a connection between the Syslog Server and the Syslog LAM.

Process Workflow

The workflow of gathering alarms from a Syslog server and publishing it to Moogsoft AIOps is:

Process Workflow for a Socket connection:

Process Workflow for getting events from a Log file:

  1. Syslog LAM reads configuration from the syslog_conf file.
  2. In Socket connection, the Syslog LAM connects to the host/IP and listens on the port, given in the syslog_conf file, and receives events from the server.
  3. For getting events from a log file the Syslog LAM reads data on the file located on the same system where the LAM is installed.
  4. The received/read event data is in a text format.
  5. The events are parsed and converted into normalized Moogsoft AIOps events.
  6. The normalized events are then published to MooMS bus.

Syslog Configuration

The Syslog server does not need any configuration to connect to the Syslog LAM.  The following information is required in the syslog_lam.conf file:

  • The IP address and port of the Syslog server if TCP or UDP connections are used
  • The log file name, if the events are fetched from a log file

Syslog LAM Configuration

The events received from a Syslog server are processed according to the configurations in the syslog_lam.conf file. The processed alarms are published to Moogsoft AIOps.

The configuration file contains a JSON object. At the first layer of the object, LAM has a parameter called config, and the object that follows config has all the necessary information to control the LAM.

The following sections are available for configuration in the Syslog LAM configuration file:

Monitor

The details of the connection with the Syslog server are defined in the monitor section of the configuration file. The user can configure the parameters here to establish the connection with Syslog server.

monitor:
        {
            name        			: "Syslog Monitor",

            class       			: "CSyslogMonitor",

            monitoring_type 		: "SOCKET",

            protocol_type   		: "UDP",

            address        			: "localhost",

            port        			: 514,

            target                  : "bow.syslog.log",
            
            load_at_start           : true,

            exit_after_initial_load : false,

            event_ack_mode 			: "queued_for_processing"
        },

  • name and class: These fields are reserved and should not be changed the default values are  Syslog Monitor and CSyslogMonitor respectively
  • monitoring_type: The monitoring type is defined here. It could be either SOCKET or FILE. The SOCKET is used for establishing a TCP or UDP connection, while FILE is used for directly monitoring a log file
  • protocol_type: The protocol type is defined here. This field has to be entered if SOCKET is selected in the field monitoring_type
  • address: The IP address of the server is entered here if SOCKET is selected in monitoring_type. This field is commented if FILE is entered in monitoring_type
  • port: The port of the server is entered here if SOCKET is selected in monitoring_type. This field is commented if FILE is entered in monitoring_type
  • target: The log file from which the alerts are to be received is entered here if FILE is selected in monitoring_type . This field is commented if SOCKET is entered in monitoring_type
  • load_at_start: A flag that if set to true means the LAM will process the entire contents of the target file and then wait for any additional data to be written to the file. This is useful if you are bulk-loading data into the system for analysis
  • exit_after_initial_load: If set to true, the LAM will read the contents of the target and when it has processed all the data, the LAM will exit
  • event_ack_mode: The event_ack_mode determines when an event received by the Syslog LAM should be acknowledged. The following options are available:
    • queued_for_processing: Event is acknowledged after it is added to the Moolet queue

    • event_processed: Event is acknowledged after it is processed in a Moolet queue

    A mode must be specified in the event_ack_mode field, if not then by default  queued_for_processing is used

    For monitoring_type set to FILE, it is mandatory to set true in the load_at_start and exit_after_initial_load fields

Agent Configuration

Agent allows the user to define 2 parameters:

 agent:
        {
            name                     : "Syslog"
            #log                     : "/var/log/moogsoft/syslog_lam.log"
        },


The above example specifies: 

  • name: This is the agent name, the events sent to MooMs by the Syslog LAM are identified by the agent name in the log. In this example the agent name is Syslog
  • log: In this instance the Syslog LAM will write its ingress contents to syslog_lam.log located at /var/log/moogsoft/

HA Configuration

Refer the document HA Configuration of LAM

Data Parsing

Any received data needs to be broken up into tokens. When you have the tokens, you can start assembling an event. The data received from a Syslog server or a Log file can be called as message.  A typical BSD Syslog message has following groups:

The groups are:

  • Pri: The Pri is the combination of severity and facility
  • Header: The header contains the combination of Timestamp and Host
  • Message: The message contains the combination of Tag and content

Although BSD messages should contain these components, few rarely do. The PRI (severity and facility combination) is rarely used, with most messages starting with an RFC compliant timestamp. The LAM has to be configured in a way that it accommodates both BSD and near-­‐BSD compliant messages (with and without the PRI). The implementation specific changes can be made in the regular expression used to determine a valid or invalid message.

Like BSD messages the Syslog LAM can receive any type of Syslog message e.g. kiwi, cisco, etc. For configuring the Syslog LAM, the user must know the type of message received by the LAM and its regular expression. Some regular expressions for different types of Syslog messages are given in the parsing section of the configuration file. A new regular expression can be added to the parsing section if the user knows the format of the received message and its regular expression.

Syslog LAM supports multiple types of Syslog messages. After receiving the message, it is parsed one by one using all the regular expression present in the parsing section. If there is a match then the parsed message is sent to the variable section. If the message does not match with any of the given regular expression, then it is parsed using the regular expression pattern_name: "default" in the parsing section. This captures the complete message and assigns it to a variable message in the variable section.

For every regular expression of a Syslog message type in parsing there should be a corresponding section in variables


In the Parsing section the tokenising of the received message is done using either a regex subgroup or delimiters. The tokenised messages then can be assigned in the variables section and forwarded to the lambot SyslogLam.js The SyslogLam.js breaks the tokenised message and then assigns it to the respective Moogsoft AIOps fields and publishes the event to MOOMs.

         parsing:
             [
                 {
                      pattern_name: "pattern1",
                     pattern  : "(?mU)^((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}\\s(?:(?:\\d{2}:){2}\\d{2}))\\s((?:(?:[a-z0-9]+(?:\\.|:|-|_)*){1,}))\\s(.*)$",
                       tokeniser_type: "delimiters",
					   action: "accept",
                       capture_group: 0,
                       delimiters:
                    {
                         ignoreQuotes: true,
                         stripQuotes: true,
                         ignores:    "",
                         delimiter:  ["||","\r"]
                    }
                 },
                 
                 {
                      pattern_name: "pattern2",
                     pattern  : "(?mU)^((?:<\\d{1,3}>)?)\\s*((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}\\s(?:(?:\\d{2}:){2}\\d{2}))\\s((?:(?:[a-z0-9]+(?:\\.|:|-|_)*){1,}))\\s(.*)$",
                       tokeniser_type: "regexp_subgroups",
					   action: "accept"
                 },
                 
                 {
                     pattern_name: "pattern3",
                     pattern : "(?mU)^(?:\\[((?:Mon|Tue|Wed|Thu|Fri|Sat|Sun)\\s+(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+(?:\\d{1,2})\\s+(?:(?:[0-1]\\d|2[0-3]):(?:[0-5]\\d):(?:[0-5]\\d).(?:\\d{1,3}))\\s+(?:\\d{4}))\\])\\s*((?:[a-zA-Z0-9]+(?:\\.|-|_|\\[|\\]|\\(|\\))*){1,})\\:\\s+(.*)$",
                       tokeniser_type: "regexp_subgroups", 
					   action: "reject"                    
                 }
              ],

The fields that are configured in the parsing section are as follows:

  • pattern name : The type of syslog message that is received by the LAM. E.g. Dell syslog
    • pattern: The regular expression which will be matched with messages received from the server. The string which matches the regular expression is extracted from the message.
      A regular expressions and its explanation is as follows:

      /^((?:<\d{1,3}>)?)\s*((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\  d{1,2}\s(?:(?:\d{2}:){2}\d{2}))\s((?:(?:[a-z0-9-]+(?:\.|:)*){1,}))\s(.*)$/

      Expression

      Description

      ^

      The beginning of a line

      ((?:<\d{1,3}>)?)

      Optionally followed by a “<” followed by between 1 and 3 digits followed by a “>”

      \s*

      Followed by 0 or more spaces

      ((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)

      Followed by one of Jan, Feb, Mar etc.

      \s+

      Followed by at least one space

      \d{1,2}

      Followed by one or two digits

      \s

      Followed by a single space

      (?:(?:\d{2}:){2}\d{2}))

      Followed by 2 digits followed by a “:” followed by 2 digits followed by a “:” followed by 2 digits (hh:mm:ss) Followed by a single space

      \s

      Followed by a single space

      ((?:(?:[a-­‐z0-­‐9-­‐]+(?:\.|:)*){1,}))

      Followed by something like an IP address (v4 or 6) or a host name (10.0.0.1, a:b:c:d, a.b.c.d)

      \s

      Followed by a space

      (.*)$

      Followed by anything up to the end of line
  • tokeniser_type: There are 2 types of tokeniser types used for tokenising are as follows:
    • regexp_subgroups: To tokenise based on regular expression subgroups enter regex subgroup in the tokeniser_type field. This tokenising method tokenise the extracted string based on groups in a message e.g. BSD Syslog message shown above. An expression in the parenthesis in the regular expression denotes a group. For example, the part expression in the above regular expression ((?:(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}) is a group which contains the date and time
    • tokeniser: To tokenise based on delimiters, enter delimiters in the tokeniser_type field. The extracted string is tokenised based on the delimiters present in it
  • action: This field is used for including or rejecting the matched regex text in the received event. If set to "accept", then the matched regex text in event will be sent for further processing. If set to "reject", then the matched regex text is rejected.

  • capture_group: The group in the message that has to be extracted and further used in the variables section. For example if "1" is given in this field then group 1 of the message is only picked for tokenising. This field is used when delimiters is entered in the field tokeniser_type, for regexp_subgroups it can be commented. The capture group is used when only a particular group in the extracted string is to be tokenised

    If "0" is entered here then all the groups of the message are picked for tokenising

In the Syslog LAM, the parsing is done using the regular expression. For tokenising either regexp_subgroups or delimiters is used

Example of the tokenisation using regexp_subgroups

The parser searches for strings as per the expression defined in the pattern field. The extracted string is then tokenised based on the configuration done in the variable section. The regular expressions for some of the syslog formats are defined above. If the user has to define regular expressions of a different format, not available here, then the regular expression can be defined in the parsing section. For example, we are defining below a section for the regular expression parsing of events received from a Dell server:

                {
                     pattern_name: "pattern4",
                     pattern : "(?m)^\\s*((?:<\\d{1,3}>)?)\\s*((?:(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2})\\s+(?:(?:[0-1]\\d|2[0-3]):    (?:[0-5]\\d):(?:[0-5]\\d)))\\s+((?:[a-zA-Z0-9]+(?:\\.|-|_|\\[|\\]|\\(|\\))*){1,})\\s+(.*)$",
                     tokeniser_type: "regexp_subgroups",
					 action: "accept"
                }
  • pattern name: The name of the pattern
  • pattern: The regular expression which will be matched with messages received from the server. E.g. "(?m)^\\s*((?:<\\d{1,3}>)?)\\s*((?:(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2})\\s+(?:(?:[0-1]\\d|2[0-3]):(?:[0-5]\\d):(?:[0-5]\\d)))\\s+((?:[a-zA-Z0-9]+(?:\\.|-|_|\\[|\\]|\\(|\\))*){1,})\\s+(.*)$"
  • tokeniser_type: The regexp_subgroups captures and then tokenise the message based on the regular expression given in the field pattern

Example of the tokenisation using delimiters

Delimiters define how a line is split into tokens – “tokenising”. For example, if you have a line of text data, it needs to be split up into a sequence of sub strings that are referenced by position from the start. So if you were processing a comma-separated file, where a comma separates each value, it would make sense to have the delimiter defined as a comma. Then the system would take all the text and break it up into tokens between the commas. The tokens could then be referenced by position number in the string starting from one, not zero. 

For example if the input string was “the,cat,sat,on,the,mat” and comma was used as a separator, token 1 would be “the”, token 2 “cat” and so on. 

Be aware, there are complications when you come to tokenisation and parsing. For example, if you say comma is the delimiter, and the token contains a comma, you will end up with that token containing a comma to be split into 2 tokens. To avoid this it is recommended that you quote strings. You must then allow the system to know whether it should strip or ignore quotes, hence the stripQuotes and ignoreQuotes parameters. Below is an example of parsing using delimiters for syslog messages received from a CISCO device.

               {
                       pattern_name: "pattern2",
                       tokeniser_type: "delimiters",
                       capture_group: 1,
                       delimiters:
                    {
                         ignoreQuotes: true,
                         stripQuotes: true,
                         ignores:    "",
                         delimiter:  ["||","\r"]
                    }                     
			   }

The above example specifies: 

  • pattern name: The name of the pattern

  • tokeniser_type: To use delimiters for parsing enter delimiters in this field

  • capture_group: The group in the message that has to be extracted and further used in the variables section. For example if "1" is given in this field then group 1 of the Syslog message is only picked for tokenising

    If "0" is entered here then all the groups of the message are picked for tokenising

  • ignoreQuotes:For strings that are quoted between delimiters, set ignoreQuotes to true, the LAM will look for delimiters inside the quote. For example, <delimiter>”hello “inside quote” goodbye”<delimiter> gives a token [hello inside quote goodbye]
  • stripQuotes: If set to true, removes start and end quotes from tokens. For example, “hello world” gives a token [hello world]

  • ignores:This field contains a list of characters to ignore. Ignored characters are never included in tokens

  • delimiter: This field contains the list of valid delimiters used to split strings into tokens. Here || and the end line character are the defined delimiters

    If an empty string is mentioned in the delimiter field then no LAM tokenisation is done, instead the entire message is taken into a single token

If the message does not match any regular expression given in the parsing section, then the LAM automatically sends the received message as a single line text to SyslogLam.js. The message is tokenised and sent to Moogsoft AIOps by the SyslogLam.js

Also if the user does not know a regular expression that can be used for tokenisation, then the parsing and the variables section can be left blank. The received message by the LAM is then sent to SyslogLam.js as a single line text

Variables

An event in a message is a positioned collection of tokens. Moog enables a user to name these positions. Naming of the positions helps the user to identify the tokens. In the below given example token at position number 3 is host name, so the the token at position 3 is assigned to "host".

variables:
            {
                "pattern1":
                    [
                        { name: "pri",           position: 1 },
                        { name: "date",         position: 2 },
                        { name: "host",           position: 3 },
                        { name: "content",      position: 4 }
                    ],
    
                "pattern2":
                    [
                        { name: "date",         position: 2 },
                        { name: "host",           position: 3 },
                        { name: "content",      position: 1 },
                        { name: "message",         position: 4 }
                    ],
                    
                "pattern3":
                    [
                        { name: "date",         position: 1 },
                        { name: "host",           position: 2 },
                        { name: "content",      position: 3 },
                        { name: "message",         position: 4 }
                    ]
            },

The above example specifies: 

For pattern1 position 1 is assigned to content; position 2 is assigned to date and so on. Positions start at 1, and go up.

In the variable section the groups pri, host, content and message are required from a syslog message or the complete message, as received in the case of default parsing

After the tokenised message is assignation to variables, the message is sent to SyslogLam.js lambot for further processing.

Mapping

The rules assignation for Syslog in the mapping section is used only for temporary values. The standard values are defined in the rules section to ensure that as complete an event object is passed to SyslogLam.js as possible. Setting temporary values has little or no impact, but helps prevent downstream errors as the event is turned into an alert.

mapping :
        {
            catchAll: "overflow",
            rules:
            [
                { name: "signature",rule:        "Signature" },
                { name: "source_id",rule:        "SourceId" },
                { name: "external_id",rule:      "ExternalId" },
                { name: "manager",rule:          "Syslog" },
                { name: "source",rule:           "$host" },
                { name: "class",rule:            "Syslog" },
                { name: "agent",rule:            "$LamInstanceName" },
                { name: "agent_location",rule:   "Location" },
                { name: "type",rule:             "$application" },
                { name: "severity",rule:         "0",conversion: "stringToInt" },
                { name: "description",rule:      "$message" },
                { name: "agent_time",rule:       "$moog_now" }
            ]
        },

In this section only 2 variables are defined which are, the current time ($moog_now), which sets the time of the events in the Moogsoft AIOps GUI with the time of processing of the event by Moog and the agent name ($LamInstanceName) which displays the name given in the name field of the Agent section on Moogsoft AIOps GUI.

The above mapping is an example and the user has to change the mapping according to the fields of received alerts/events


Constants and conversions

There are rules in mapping section for which conversions are to be defined. The conversions convert the received input from one format to another. For Syslog LAM only the stringToInt function is used.

The example of constants and conversions sections are as follows:

 constants:
        {
            severity:
            {
                "CLEAR"         : 0,
                "INDETERMINATE" : 1,
                "WARNING"       : 2,
                "MINOR"         : 3,
                "MAJOR"         : 4,
                "CRITICAL"      : 5
            }
        },
        conversions:
        {
            sevConverter:
            {
                lookup: "severity",
                input:  "STRING",
                output: "INTEGER"
            },
            stringToInt:
            {
                input:      "STRING",
                output:     "INTEGER"
            },
            timeConverter:
            {
                timeFormat : "MM dd yyyy HH:mm:ss",
                input:      "STRING",
                output:     "INTEGER"
            }
        },

The above example specifies: 

  • Severity and sevConverter: The severity field has a conversion defined as sevConverter in the Conversions section, this looks up the value of severity defined in the severity section of constants and returns back the mapped integer corresponding to the severity   
  • stringToInt: It is used in a conversion, which forces the system to turn a string token into an integer value
  • timeConverter: It is used in conversion which forces the system to convert to time. If time is epoc time, then timeFormat mentioned in timeConverter should be commented. Otherwise, user should provide the timeFormat i.e. (%D %T) and it should be uncommented

catchAll

The attribute that is never referenced in a rule is collected and placed as a JSON object in a variable called overflow defined here and passed as part of the event.

mapping :
        {
            catchAll: "overflow",
            rules:
            [
                { name: "signature",rule:        "Signature" },
                { name: "source_id",rule:        "SourceId" },
                { name: "external_id",rule:      "ExternalId" },
                { name: "manager",rule:          "Syslog" },
                { name: "source",rule:           "$host" },
                { name: "class",rule:            "Syslog" },
                { name: "agent",rule:            "$LamInstanceName" },
                { name: "agent_location",rule:   "Location" },
                { name: "type",rule:             "$application" },
                { name: "severity",rule:         "0",conversion: "stringToInt" },
                { name: "description",rule:      "$message" },
                { name: "agent_time",rule:       "$moog_now" }
            ]
        },

SyslogLam.js

The filter defines whether Moog uses a LamBot. If the presend value is commented, then no filter is applied to the events produced and all events are sent unchanged to the MooMs bus. If a LamBot is defined, then the event is passed to the LamBot. The Syslog LAM uses the lambot syslogLam.js.

        filter:
        {       
     		modules: [                                    
						"RegExpUtil.js",                               
						"SyslogUtil.js",                                
						"SyslogEvents.js",                                 
						"LamUtility.js"                    
					  ],           
			presend: "SyslogLam.js"       
		}


In the above example, the presend is set to "SyslogLam.js" which is the Syslog lambot that extracts the values from the tokenised message and after assembling the values publishes the event to MooMs.

The modules defined in the modules section are used by SyslogLam.js to extract values. In the above example the following modules are defined:

  • RegExpUtil.js: A module holding common regular expressions and regular expression related functions (search and replace)
  • SyslogUtil.js: A syslog specific module, defining a “message” object and instance methods, and other methods used in the lambot
  • SyslogEvents.js:  A library of actions to take for specific events – based on an EventId. Message actions are defined by functions – so a single function can cover many message types. 
  • LamUtility.js: Contains functions that are repeatedly used in a lambot (date, debug   etc.)

The SyslogLam.js  can be edited by the user to adapt it to process specific Syslog messages. The lambot and the modules can be found in the $MOOG_HOME/bots/lambots directory

The workflow of syslog lambot is as follows:

  • Determine Severity: Associate one of the standard Moog severities with the event
  • Determine Context: Find out what entities this event is about – which host, interfaces, ports etc.
  • Determine Eventid: Find the event id of the event
  • Determine Signature: Create a valid Moog signature that will allows effective de-­duplication for the events
  • Execute a policy: If the message has an optional specific behavior associated with it (e.g. message transformation) then execute it
  • Final Processing: Allow any last stage processing before the event is dispatched (filtering etc.)

Starting the Syslog LAM

To start the Syslog LAM enter the following command:

service sysloglamd start

To stop the Syslog LAM enter the following command:

service sysloglamd stop

To view the status of Syslog LAM enter the following command:

service sysloglamd status

Quotes

In some instances, the attribute strings are quoted. Our JSON parser ignores it, but the standard requires quoting for all strings, so Moogsoft recommends that user quote all strings.

Comments

A user can comment out lines by appending them with a hash.

Command Line Attributes

The syslog_lam is a command line executable that can be run as a service daemon, and takes 4 attributes, which can be viewed by typing: 

syslog_lam--help

Option

Description

--config

Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified

--help

Displays all the command line options

--version

Displays the component’s version number

--log level

Specifies the level of debug. By default, you get everything. In common with all executables in MOOG, having it set at that level can result in a lot of output (many messages per event message processed). In all production implementations it is recommended that log level be set to WARN, which only informs you of matters of importance

Version Information

LAM Version

Tool Version

Tested?

Expected to Work

1.0

Generic

Yes

Yes

1.1

Generic

Yes

Yes

System Information

This LAM was tested on a system with the following configurations:

CPU2 core
RAM4 GB
Operating SystemCentOS Linux release 6.7

The system must at least have the above mentioned system requirements to run the LAM.

  • No labels