Page tree
Skip to end of metadata
Go to start of metadata

Overview

The WMQ LAM is a link access module that communicates with the WebSphere MQ  Application Server and takes its input from Java Messaging Services.

This document explains the basic configuration for enabling the Java based Application Server and the configuration of the WMQ LAM config file (wmq_lam.conf).

Process Workflow

The workflow of gathering alarms from the Application Server and publishing it to Moogsoft AIOps is as follows:

  • WMQ LAM monitors message data being written to a Queue/Topic in the server
  • It then parses this message data according to the LAM’s configuration file
  • Events are constructed from the monitored message data and then are passed to the MOOMs bus
  • The events are then published to the subject “/Events”

Adding the Websphere MQ Jars to Moogsoft AIOps

The com.ibm.mq.jmqi.jar and com.ibm.mqjms.jar has to be added to Moogsoft AIOps to establish a connection with Websphere MQ. Copy the com.ibm.mq.jmqi.jar and com.ibm.mqjms.jar to the $MOOGSOFT_HOME/lib/cots/nonDist directory in Moogsoft AIOps.

The com.ibm.mq.jmqi.jar and com.ibm.mqjms.jar for Linux can be found in the directory /opt/mqm/java/lib

The com.ibm.mq.jmqi.jar and com.ibm.mqjms.jar for windows can be found in the directory C:\Program Files\IBM\WebSphere MQ\java\lib

Installing the IBM JDK on the Moogsoft AIOps Server

Install the IBM JDK on the Moogsoft AIOps Server and set the java class path in the wmq_lam to the installed IBM JDK class path. To set the classpath proceed as follows:

  1. Enter the command cd $MOOGSOFT_HOME/bin/ to navigate to the bin directory of Moogsoft AIOps
  2. Open the wmq_lam using any editor e.g. vi or vim
  3. Enter the path /opt/ibm/java-x86_64-80/jre/bin/java in the java_vm field of the wmq_lam binary file.

    The path given here should be the path where the IBM JDK is installed.

WMQ LAM Configuration

The alarms received from the WebSphere MQ server are processed according to the configurations in the wmq_lam.conf file. The processed alarms are published to Moogsoft AIOps.

The configuration file contains a JSON object. At the first layer of the object, LAM has a parameter called config, and the object that follows config has all the necessary information to control the LAM.

The following sections are available for configuration in the WMQ LAM configuration file.

Monitor

Monitor section defines the object to be monitored:

monitor:
        {
           
            name                   		:"WMQ Lam Monitor",
            
            class                   	:"CWmqMonitor",
         
            manager_name                :"QueueManagerName",
            
            host_name                   :"localhost",

            port_number                 : 1414,

            channel_name                :"MyRemoteChannel",
 
            destination_name            :"TestQueue",

            user_name                   :"username",

            password                    :"password",

            #encrypted_password 	    :"ieytOFRUdLpZx53nijEw0rOh07VEr8w9lBxdCc7229o=",

            destination_type            :"queue",     

            max_retries                 : 10,
 
            retry_interval              : 60,

            message_type                :"TextMessage",

            response_require            :false,

            feedback_queue   			:"TempQueue",

            ssl_connection				:false,

            ssl_truststore_filename     :"",

            ssl_truststore_password     : "",

            ssl_truststore_type         :"",

           ssl_ciphersuite_value        :"SSL_RSA_WITH_3DES_EDE_CBC_SHA",

           ssl_fipsRequired             :false,

           ssl_protocol                 :""
     
        },

The above example specifies: 

  • name and class: These fields are reserved and should not be changed. The default values are  WMQ Lam Monitor and CWmqMonitor respectively
  • manager_name: The manager name is the queue manager name created in WebSphere MQ. E.g. QueueManagername

  • host_name: The host name of the WebSphere MQ server. E.g. localhost

  • port_number: The port number of the queue manager configured in WebSphere MQ. E.g. 1414. The default port is 1414.

    The entry in the field port_number should be an integer, therefore enter the values in this field without quotation marks

  • channel_name: The channel name of the queue created in the WebSphere MQ E.g. MyRemoteChannel
  • destination_name: The name of the queue or topic created in WebSphere MQ. E.g. TestQueue
  • user_name and password: The queue or topic username and password is entered in these fields. If there is no user name and password configured for the queue or topic then leave it blank
  • encrypted_password: If queue or topic password is encrypted then enter the encrypted password in this field and comment the password field. At a time either password or the encrypted_password field is used. If both the fields are not commented then the field encrypted_password will be used by the WMQ LAM
  • destination type: The type of the message provider that is either a queue or a topic. E.g. queue

  • max_retries: The maximum number of retry attempts to reconnect with the WebSphere MQ server in case of a connection failure

    The default value is set to 10, if 0 is entered in this field then the LAM by default takes the value 10 and will try at least 10 times to reconnect

    If all the number of retries are exhausted, then an alarm is sent to Moogsoft AIOps about the connection failure. For re-establishing the connection the LAM has to be restarted

  • retry_interval: The time interval between two successive retry attempts

    The default value is set to 60 seconds, if 0 is entered in this field then the time interval is by default set to 60 seconds

    The entry in the fields max_retries and retry_interval should be an integer, therefore enter the values in these fields without quotation marks

  • message_type: The message type of the messages received from the application can be set here. The following 3 message types are supported:
    • TextMessage
    • MapMessage
    • ObjectMessage
  • response_require: If response is to be sent back to queue or topic for received messages, enter true in this field
  • feedback_queue: Enter the queue name in which the response feedback is sent to WebSphere MQ
  • ssl_connection: To enable an SSL communication with WebSphere MQ enter true in this field
  • ssl_truststore_filename: If an SSL connection is enabled then enter the ssl truststore filename along with its path in Moogsoft AIOps,  E.g. if the ssl keystore is in the directory usr/share/moogsoft/ssl, then enter usr/share/moogsoft/ssl/client.jks

    The client.jks certificate must be copied from the WebSphere MQ Server at a desired location on the Moogsoft AIOps system E.g. usr/share/moogsoft/ssl. The client.jks certificate can be found in the WebSphere MQ Server at the location where it was generated in the above steps.

  • ssl_truststore_password: If an SSL connection is enabled then enter the ssl truststore password in this field
  • ssl_truststore_type:  SSL truststore typeof the certificate created at the server end has to be entered here. The following truststore types can be entered here
    • JKS,
    • JCEKS,
    • PKCS12,
    • PKCS11
    • DKS
  • ssl_ciphersuite_value: The SSL cipher specification indicates which data encryption algorithm and key size are used. For SSL V3, the hashing algorithm is included. For example, cipher specification DES SHA (56 bit) uses the DES encryption algorithm, a 56-bit key size and the SHA hashing algorithm. Some of the available ciphersuites are as follows:
    • SSL_RSA_WITH_3DES_EDE_CBC_SHA

    • SSL_DES_192_EDE3_CBC_WITH_MD5

    • SSL_RC4_128_WITH_MD5

    • SSL_RC2_CBC_128_CBC_WITH_MD5

    • SSL_DES_64_CBC_WITH_MD5

    • SSL_RC4_128_EXPORT40_WITH_MD5

    • SSL_RC2_CBC_128_CBC_EXPORT40_WITH_MD5

  • ssl_fipsRequired: If fips is required then select true otherwise set it to false. The Federal Information Processing Standard (FIPS) Publication 140-2, (FIPSPUB 140-2), is a U.S. government computer security standard used to accredit cryptographic modules
  • ssl_protocol: Its value depends on the CipherSuites value and its equivalent  SSL Protocol support, Some of the ssl protocols are as follows:
    • TLSv1
    • TLSv1.1
    • TLSv1.2
    • SSLv3

Agent configuration

Agent allows the user to define 2 parameters:

agent:
        {
                name    : "WMQ",
                #log    : "/var/log/moogsoft/wmq_lam.log"
        },

The above example specifies: 

  • name: This is the agent name, the events sent to MooMs by the WMQ LAM are identified by the agent name in the log. In this example the agent name is WMQ
  • log: In this instance the WMQ LAM will write its ingress contents to wmq_lam.log located at /var/log/moogsoft

HA configuration

Refer the document HA Configuration of LAM

Data parsing

Any received data needs to be broken up into tokens. Once user have the tokens, user can start assembling an event. There are a number of parameters that allow user to control how this will work. The first 2 are start and end character. User can have multiple start and end characters. The system generates an event after all the tokens between a start and an end character are assembled.

Parsing:
{
            type: "",
            start_and_end:
            {
                start:      [],
                end:        ["\n"],
                delimiters:
                {
                    ignoreQuotes: true,
                    stripQuotes: true,
                    ignores:    "",
                    delimiter:  [",","\r"]
                }
            }
        },
 
        # Parsing block with regular expressions, using delimter
        # based tokenising:
        #
        # parsing:
        # {
        #     type: "regexp",
        #     regexp:
        #     {
        #         pattern : "(?mU)^(.*)$",
        #         capture_group: 1,
        #         tokeniser_type: "delimiters",
        #         delimiters:
        #         {
        #                 ignoreQuotes: true,
        #                 stripQuotes: false,
        #                 ignores:    "",
        #                 delimiter:  ["\r"]
        #         }
        #     }
        # },
        #


The above example specifies the following 3 types of parsing:

  • JSON parsing: To enable this parsing the type is set to blank
  • Text Messgae: To enable this parsing the type is set to Start_and_End
  • Regular Expression: To enable this the type is set to regexp

In the above example only one parsing method is used at a time. Either regexp or Text Message/JSON.

JSON Parsing

Any received data needs to be broken up into tokens. Once the tokens are received, the assembling of an event starts. There are a number of parameters that allow the user to control how this will work. The first 2 are a start and end character. The square brackets [] are the JSON notation for a list. You can have multiple start and end characters. The system considers an event as all of the tokens between any start and end character. 

The above example specifies:

  • There is nothing defined in start; however, a carriage return (new line) is defined as the end character

In the example above, the LAM is expecting a whole line to be written followed by a return, and it will process the whole line as one event. 

If set up carefully, user can accept multi-line events.

Text Message Parsing

The Type should be set start_and_end and as shown in the below example.

type: start_and_end:
            {
                start:      [WMQ_MSG],
                end:        ["\n"],

The parsing in above example the parsing will start when it gets WMQ_MSG and end when it gets new line.

Regular Expression Parsing

In regular expression the parser searches for strings as per the expression defined in pattern. The extracted string is then delimited as per the defined delimiters. In the above example the parser searches for the expression "(?mU)^(.*)$".

Delimiters

Delimiters define how a line is split into tokens – “tokenising”. For example, if you have a line of text data, it needs to be split up into a sequence of sub strings that are referenced by position from the start. So if you were processing a comma-separated file, where a comma separates each value, it would make sense to have the delimiter defined as a comma. Then the system would take all the text between start and end and break it up into tokens between the commas. The tokens could then be referenced by position number in the string starting from one, not zero. 

For example if the input string was “the,cat,sat,on,the,mat” and comma was used as a separator, token 1 would be “the”, token 2 “cat” and so on. 

Be aware, there are complications when you come to tokenisation and parsing. For example, if you say comma is the delimiter, and the token contains a comma, you will end up with that token containing a comma to be split into 2 tokens. To avoid this it is recommended that you quote strings. You must then allow the system to know whether it should strip or ignore quotes, hence the stripQuotes and ignoreQuotes parameters.

ignoreQuotes: true, 
 stripQuotes: false, 
 ignores: "", 
 delimiter: [",","\r"]

The above example specifies: 

  • If you have strings that are quoted between delimiters, ignoreQuotes set to true will look for delimiters inside the quote. For example, <delimiter>”hello “inside quote” goodbye”<delimiter> gives a token [hello inside quote goodbye].
  • Setting stripQuotes to true removes start and end quotes from tokens. For example, “hello world” gives a token [hello world].
  • ignores is a list of characters to ignore. Ignored characters are never included in tokens.
  • Delimiter is the list of valid delimiters used to split strings into tokens.

Variables

For each event in the file, there is a positioned collection of tokens. Moog enables a user to name these positions. Naming of the positions helps the user to identify the tokens.In the below given example token at position number 6 is a Manager name, so the user names the token as "Manager".
This section is used for text message.

variables:
        [
            #
            # Note that positions start at 1, and go up
            # rather than array index style counting from zero
            #
            { name: "signature",   position: 1 },
            { name: "source_id",   position: 4 },
            { name: "external_id", position: 3 },
            { name: "Manager",     position: 6 },
            { name: "AlertGroup",  position: 7 },
            { name: "Class",       position: 8 },
            { name: "Agent",       position: 9 },
            { name: "severity",    position: 5 },
            { name: "description", position: 10 },
            { name: "agent_time",  position: 1 }
        ],

The above example specifies: 

Token at position 1 is assigned to signature; Token at position 4 is assigned to source_id and so on. Token positions starts from 1, and go up.

The variable section is used when the message type is TextMessage

The variable section is only used for text message. For JSON, MapMessage and ObjectMessage the mapping section is used. In mapping there is a value called rules, which is a list of assignments. 

mapping :
        {
            #
            # All unused variables live as a JSON object
            # referenced by this variable (if defined)
            #
             builtInMapper:  "CJsonDecoder",
            # Input is restricted to Json so the builtInMapper option is not
            # used for this LAM
            #
            catchAll: "overflow",
            rules:
            [
                { name: "signature", rule:      "$signature" },
                { name: "source_id", rule:      "$source_id" },
                { name: "external_id", rule:    "$external_id" },
                { name: "manager", rule:        "WMQ" },
                { name: "source", rule:         "$source" },
                { name: "class", rule:          "$class" },
                { name: "agent", rule:          "$LamInstanceName" },
                { name: "agent_location", rule: "$agent_location" },
                { name: "type", rule:           "$type" },
                { name: "severity", rule:       "$severity", conversion: "sevConverter" },
                { name: "description", rule:    "$description" },
                { name: "agent_time", rule:     "$agent_time", conversion: "stringToInt" }
            ]
        },
        filter:
        {
            presend: "WmqLam.js"
        }
    }
}

In the example above mapping assignments are shown, the first assignment name: "signature”, rule:"$signature” (“$signature is a string with $ syntax) means for signature field take the tokens called signature. User defines a number of these rules covering the base attributes of an event. For reference, the system expects a minimum set of attributes in an event that are shown in the above example. For JSON, MapMessage and ObjectMessage, enable the builtInMapper: "CJsonDecoder" by uncommenting it. The variable section is ignored if builtInMapper is uncommented. For TextMessage builtInMapper option should be commented.

The above mapping is a generic mapping given for example. The user has to configure the mapping according to fields of the received alarm/event

Constants and conversions

There are rules in mapping section for which conversions are to be defined. The conversions convert the received input from one format to another. E.g. in the above example of mapping, for the mapped field severity, an integer is received which is converted to text and displayed on the Moogsoft AIOps UI. The lookup for conversions is kept in the constants section. The available conversions are kept in the conversions section and called during mapping. The example of calling a conversion is as follows:

{ name: "severity", rule:       "$severity", conversion: "sevConverter" }

The example of constants and conversions sections are as follows:

constants:
        {
            severity:
            {
                "CLEAR"         : 0,
                "INDETERMINATE" : 1,
                "WARNING"       : 2,
                "MINOR"         : 3,
                "MAJOR"         : 4,
                "CRITICAL"      : 5
            }
        },
        conversions:
        {
            sevConverter:
            {
                lookup: "severity",
                input:  "STRING",
                output: "INTEGER"
            },
             
            stringToInt:
            {
                input:      "STRING",
                output:     "INTEGER"
            },
          
            timeConverter:
            {
                timeFormat: "%D %T",
                input:      "STRING",
                output:     "INTEGER"
            }
        },

The above example specifies: 

  • Severity and sevConverter: Has a conversion defined as sevConverter in the Conversions section, this looks up the value of severity defined in the severity section of constants and returns back the mapped integer corresponding to the severity   
  • stringToInt: Is used in a conversion, which forces the system to turn a string token into an integer value
  • timeConverter: Is used in conversion which forces the system to convert to time. If time is epoc time, then timeFormat mentioned in timeConverter should be commented. Otherwise, user should provide the timeFormat i.e. (%D %T) and it should be uncommented

JSON Events

The WMQ LAM has the ability to consume JSON events. JSON is a sequence of attribute/value, and the attribute is used as a name. Under mapping, user must define the following attribute builtInMapper: "CJsonDecoder". It automatically populates all of the values contained in the JSON object, prior to the rules being run.

For example, if the JSON object to be parsed was: 

{"signature": "11898.9","source_id": "Server1", "severity": "MINOR"……so on}

The attributes available to the rules in the mapping section would be $signature=”11898.9”, $Severity=” Minor” and so on. Similarly, user can map ObjectMessage and MapMessage.

For TextMessage user should use variable section.

Below are few samples of TextMessage, MapMessage and ObjectMessage.

TextMessage
WMQ_MSG:3600de30-92f8-71e2-0408-97bfd0490000||1||26||13639605v53||1364210285||1363960556||NEO||delta-server-loggingAdaptor||default.log||2013-03-25 11:17:19.560 ERROR  [Response--6 - BlockingEntitlementsCheckHandle] Unauthorised access detected, throwing UnauthorisedAccessException||xstm3022xpap.stm.swissbank.com||NEO-PROD


MapMessage

hostname=10.112.70.125

port=8080

destination-jndi=wmq/topic/test

username=administrator

password=India@123

signature=8.9

source_id=server1           

external_id=123.1345

manager=WMQ

source=server1

class=server

agent=test

agent_location=test

type=test

severity=MAJOR

description=Test server1

agent_time=07/24/12 18:06:01


ObjectMessage

hostname=10.112.70.123

port=8080

connection-factory-jndi=wmq/RemoteConnectionFactory

destination-jndi=wmq/queue/test

#connection-factory-jndi=ConnectionFactory

#destination-jndi=dynamicQueues/final

username=administrator

password=India@123

server-name=Jboss

# Properties related to Websphere mq

message-type=object

# Properties related to object message

signature=1234.8.9

source_id=server2         

external_id=hmoscsysd2

manager=WMQ

source=server2

class=test

agent=test

agent_location=test

type=server

severity=MAJOR

description=test server2

 agent_time=07/24/12 18:06:01

catchAll

The attribute that is never referenced in a rule is collected and placed as a JSON object in a variable called overflow defined here and passed as part of the event.

catchAll: "overflow",
           rules:
           [
                { name: "signature", rule:      "$signature" },
                { name: "source_id", rule:      "$source_id" },
                { name: "external_id", rule:    "$external_id" },
                { name: "manager", rule:        "WMQ" },
                { name: "source", rule:         "$source" },
                { name: "class", rule:          "$class" },
                { name: "agent", rule:          "$LamInstanceName" },
                { name: "agent_location", rule: "$agent_location" },
                { name: "type", rule:           "$type" },
                { name: "severity", rule:       "$severity", conversion: "sevConverter" },
                { name: "description", rule:    "$description" },
                { name: "agent_time", rule:     "$agent_time", conversion: "stringToInt" }
            ]

The above example specifies the mapping of tokens and the variable overflow for catchAll.

The attribute test1 and test2 is not mapped with a field in the wmq_lam.conf  file, it is placed in the overflow JSON object. The fields that are placed in the overflow variable can be viewed in the WMQ LAM log file.

Example of a message sent through a WebSphere MQ queue.


Message
[{"signature":"0.1.8","source_id":"xvsdfgdg","external_id":"dduncan9","manager":"Sonsitng","source":"Indonesian","class":180,"agent_location":"Liangwa","type":"Violet","severity":"WARNING","description":"Yuan
Renminbi","agent_time":"07/27/12 19:06:01","test1":"1","test":"2"}]


Example of an overflow JSON object containing the unmapped test1 and test2 tokens, created in the WMQ LAM log file:

 NFO: [EventFa][20161027 17:04:38.701 +0530] [CMooMsg.java]:1099 +|Encoded size
[376]
json[{"_MOOTADATA_":{"creation_time":1477568078591},"agent":"WMQLAM","agent_location":"Liangwa","agent_time":0,"class":180,"description":"Yuan
Renminbi","external_id":"dduncan9","manager":"Sonsitng","overflow":"{\"test\":\"2\",\"test1\":\"1\"}","severity":2,"signature":"0.1

Starting the WMQ LAM

To start the WMQ LAM enter the following command:


service wmqlamd start

 To stop the WMQ LAM enter the following command:

service wmqlamd stop

To view the status of WMQ LAM enter the following command:

service wmqlamd status

Quotes

In some instances, the attribute strings are quoted. Our JSON parser ignores it, but the standard requires quoting for all strings, so Moogsoft recommends that user quote all strings.

Comments

A user can comment out lines by appending them with a hash.

Command Line Attributes

The wmq_lam is a command line executable that can be run as a service daemon, and takes 4 attributes, which can be viewed by typing: 

wmq_lam--help

Option

Description

--config

Points to a pathname to find the configuration file for the LAM. This is where the entire configuration for the LAM is specified

--help

Displays all the command line options

--version

Displays the component’s version number

--log level

Specifies the level of debug. By default, you get everything. In common with all executables in MOOG, having it set at that level can result in a lot of output (many messages per event message processed). In all production implementations it is recommended that log level be set to WARN, which only informs you of matters of importance

Version Information

LAM Version

Tool Version

Tested?

Expected to Work

1.0

Websphere MQ 8.0.0.4

Yes

Yes

1.1

Websphere MQ 8.0.0.4

Yes

Yes

1.2

Websphere MQ 8.0.0.4

Yes

Yes

System Information

This LAM was tested on a system with the following configurations:

CPU2 core
RAM4 GB
Operating SystemCentOS Linux release 6.7

The system must at least have the above mentioned system requirements to run the LAM.

  • No labels