Moogsoft Docs

Netcool Legacy LAM

The Legacy Netcool LAM enables Moogsoft AIOps to receive data from IBM Tivoli Netcool Omnibus.

IBM Tivoli Netcool Omnibus includes a socket gateway which is able to pass data to a third party system. This permits integration between IBM Tivoli Netcool Omnibus and Moogsoft AIOps.

In its default configuration, the LAM will ingest data from a default IBM Tivoli Netcool Omnibus set up. Mapper utilities are also provided to enable data ingestion from a non-default IBM Tivoli Netcool Omnibus setup. In addition, the following Moobots are provided:

File

Description

AlertBuilderNetcool.js

Allows special handling for the 'repeatDetection' and 'stopDeduplication' processes.

AlertRulesEngineNetcool.js

Used to process DELETE and REPEAT type events.

SituationMgrNetcool.js

Creates description labels for Situations based around root cause and symptom detection, based on IBM Tivoli Network Manager parameters.

Capabilities

  • Data received from IBM Tivoli Netcool Omnibus has fields checked to identify event type (eg. Problem, Resolution), state (eg. INSERT, UPDATE, DELETE) and severity.

  • Events are then created by the Legacy LAMbot (based on the LAMbot configuration), containing all the mandatory Netcool event fields, and any additional optional fields.

  • Events are passed to the Netcool Alert Builder for de-duplication and alert creation.

  • Alerts are then passed to the Netcool Alert Rules Engine for filtering and passing on to Sigalisers for further processing.

  • Legacy LAM can process ITNM (IBM Tivoli Network Manager) root cause and symptom events, using Cookbook to create Situations.

Requirements

The ingestion of data from IBM Tivoli Netcool Omnibus requires that its gateway be configured according to the configurations set within Moogsoft AIOps. To be able to get data from IBM Tivoli Netcool Omnibus, Moogsoft AIOps must be configured to allow inbound communication from IBM Tivoli Netcool Omnibus.

To ingest data there must be mapping in the NetcoolLAM between Moogsoft AIOps Alert fields and IBM Tivoli Netcool Omnibus alert fields. Mapping of some IBM Tivoli Netcool Omnibus fields is mandatory, as they are required in any IBM Tivoli Netcool Omnibus system. These fields are mapped by default. Mapping of the remaining IBM Tivoli Netcool Omnibus fields is optional; they are not mapped by default.

The configuration of IBM Tivoli Netcool Omnibus to send data and the configuration for Moogsoft AIOps to ingest and process the data are explained below.

IBM Tivoli Netcool Omnibus configuration

Note

An IBM Tivoli Netcool Omnibus system administrator is required.

Configure the socket gateway and socket map files to send data to Moogsoft AIOps.

Socket Gateway Configuration

In the IBM Netcool Omnibus gateway configuration file (by default named NCO_GATE.props) configure the following:

Field

Set to:

Gate.Socket.Host

Hostname or IP address of the Moogsoft AIOps system.

Gate.Socket.Port

Port number defined in the Legacy LAM. For example 8411.

Gate.Socket.Separator

The delimiter used in the data sent to Moogsoft AIOps. Set this to double pipe - ||

Gate.Socket.InsertTrailer

NC_EVENT_END;\n

Gate.Socket.UpdateTrailer

NC_EVENT_END;\n

Gate.Socket.DeleteTrailer

NC_EVENT_END;\n

Gate.Socket.InsertHeader

INSERT||

Gate.Socket.UpdateHeader

UPDATE||

Gate.Socket.DeleteHeader

DELETE||

Socket Map Configuration

In the gateway mappings configuration file (by default named socket.map), set the fields to be sent to Moogsoft AIOps, ensuring that no field is set to be ON INSERT ONLY. Gateway mapping should include all the mandatory fields, and any additional optional ones (such as @NodeAlias).

Mappings should look similar to the example below:

CREATE MAPPING StatusMap
 (
        '' = '@Identifier',
        '' = '@Serial',
        '' = '@Node',
        '' = '@LocalNodeAlias',
        '' = '@Manager',
        '' = '@Agent',
        '' = '@AlertGroup',
        '' = '@AlertKey',
        '' = '@Severity',
        '' = '@Summary',
        '' = '@FirstOccurrence',        
        '' = '@LastOccurrence',
        '' = '@Class',        
        '' = '@OwnerUID',         
        '' = '@Acknowledged',
        '' = '@ExpireTime',
        '' = '@SuppressEscl',
        '' = '@TaskList',
        '' = '@LocalRootObj',       
        '' = '@RemoteNodeAlias',
        '' = '@RemoteRootObj',        
        '' = '@ServerName',
        '' = '@ServerSerial',
        '' = '@StateChange',        
        '' = '@InternalLast',
        '' = '@Tally',                                                          
        '' = '@Type',
        '' = '@EventId',
        '' = '@NodeAlias'
 );

Moogsoft AIOps Configuration

In the above example mappings configuration, the fields in red are optional. All other fields are mandatory, and are required to be able to ingest data into Moogsoft AIOps.

To run the Legacy LAM, the following Moogsoft AIOps files are required and are installed by default:

File

Description

$MOOGSOFT/config/netcool_lam.conf

Legacy LAM configuration file.

$MOOGSOFT/bots/lambots/NetcoolLam.js

LAMbot file; performs the main processing of the events received from IBM Tivoli Netcool Omnibus.

$MOOGSOFT/bots/lambots/NetcoolUtility.js

Holds additional functions and configurations required for the LAMbot.

$MOOGSOFT_HOME/bin/utils/netcool_lam.conf.template

Used by the moog_netcool_lam_mapper script as a base template for generating a new Legacy LAM configuration file. See the following sections.

The following sections detail the steps required to configure the Legacy LAM, LAMbot and Moobots:

Legacy LAM Configuration

Mapping between IBM Tivoli Netcool Omnibus data fields and Moogsoft AIOps fields is defined in the Legacy LAM configuration file netcool_lam.conf. As installed, netcool_lam.conf maps to the data fields in a default set up of IBM Tivoli Netcool Omnibus. If ingesting data from a non-default IBM Tivoli Netcool Omnibus set up, a mapping utility moog_netcool_lam_mapper is available to map between IBM Tivoli Netcool Omnibus data fields and Moogsoft AIOps fields. You can also enter the mapping by manually editing netcool_lam.conf.

The mapping utility can use either an IBM Tivoli Netcool Omnibus map file, or a log file containing Event data from IBM Tivoli Netcool Omnibus:

Note

Before running the mapping utility, back up the existing netcool_lam.conf file.

To use a map file with the mapping utility, enter the command and arguments as in the following example (the map file is socket.map):

sh $MOOGSOFT_HOME/bin/utils/moog_netcool_lam_mapper -f socket.map -t map

To use a log file with the mapping utility, enter the command and arguments as in the following example (the log file is event-data.txt):

sh $MOOGSOFT_HOME/bin/utils/moog_netcool_lam_mapper -f event-data.txt -t logfile

When running the mapping utility (using either a map file or a log file), you can also optionally set the address (using the -a argument in the command line) and port number (using the -p argument in the command line), as shown in the following example:

sh $MOOGSOFT_HOME/bin/utils/moog_netcool_lam_mapper -a remote_host -f socket.map -p 8455 -t map

The above example sets the address to remote_host and the port number to 8455.

Note

Once the mapping utility has successfully completed, a new Legacy LAM configuration file netcool_lam.conf is generated and is automatically placed in the default configuration directory ($MOOGSOFT/config), overwriting the existing netcool_lam.conf file

Check the Legacy LAM configuration file to ensure that the following fields are set correctly:

Field

Description

Example

port

The port number where the Legacy LAM will be receiving data from IBM Tivoli Netcool Omnibus

Port number 8411

address

This should be localhost or the hostname of the system running Moogsoft AIOps

If Moogsoft AIOps is running on premise the default address is : 0.0.0.0

If Moogsoft AIOps is running on e.g. Amazon web services, it may be similar to ew2.234.234.compute.amazonaws.com

mode

The operation mode in which the socket LAM will run should be set to SERVER

LAMbot Configuration

The LAMbot performs the core processing of the data received from IBM Tivoli Netcool Omnibus. The main configuration option in the LAMbot is whether DELETE type events are processed or not. By default this is disabled (the keepDeletes value set to 0 in the LAMbot. This may be changed if necessary, by editing the NetcoolLam.js file.

Additionally, there is an optional configuration in the LAMbot that will enable the processing of ITNM (IBM Tivoli Network Manager) Route Cause and Symptom events, based on the values of the fields @NmosCauseType and @NmosSerial. These two fields must be included in the event to be able to enable processing. By default, it is set to disabled. To enable it, set the usingITNM value to true in the NetcoolLam.jsfile.

Only proceed with the remaining steps in this section if you have enabled the use of ITNM (see above)

Then go to the next section "Moobot Configuration".

Enabling the processing of ITNM fields requires the associated Moolets, Cookbook and Recipe to be enabled in the Moogfarmd configuration file $MOOGSOFT/config/moog_farmd.conf as follows:

  1. In the sig_resolution section, uncomment the following merge group, making it available as an additional merge group:

    merge_groups:
        [
             {
                  name: "ITNM Route Causes & Symptoms",
                  moolets: ["ITNM"],
                  alert_threshold      : 2,
                  sig_similarity_limit : 0.65
              }
        ],
  2. In the moolets section, uncomment the ITNM Cookbook and recipe:

    {
        # Moolet
        name                : "ITNM",
        classname           : "CCookbook",
        run_on_startup      : true,
        metric_path_moolet  : true,
        moobot              : "Cookbook.js",
        #process_output_of  : "AlertRulesEngine",
        process_output_of   : "AlertBuilder",
    
    
        # Algorithm
        membership_limit       : 1,
        scale_by_severity      : false,
        entropy_threshold      : 0,
        single_recipe_matching : false,
        
             recipes : [
             {
                  chef                   : "CValueRecipe",
                  name                   : "ITNM Route Cause & Symptom Detection",
                  description            : "Root cause and Symptom alerts detected based on ITNM",
                  recipe_alert_threshold : 1,
                  exclusion              : "custom_info.nmosCauseType = 0",
                  trigger                : "custom_info.suppressedSerial > 0",
                  rate                   : 0,
                  min_sample_size        : 5,
                  max_sample_size        : 10,
                  cook_for               : 1200,
                  matcher                :    {
                                              components:[
                                                         { name: "custom_info.suppressedSerial", similarity: 1.0 }
                                                         ]
                                              }
             }
                       ],
        cook_for  :    1200
    }
  3. Enable the Netcool Situation Manager Moolet:

    • Set the run_on_startup value to true for the SituationMgr Moolet

    • Uncomment the line containing the Netcool Moobot - SituationMgrNetcool.js (the file is installed by default) • Uncomment ITNM in the section process_output_of to include the ITNM Cookbook.

    The Moolet section should now look similar to the one below:

    name               : "SituationMgr",
    classname          : "CSituationMgr",
    run_on_startup     : true,
    metric_path_moolet : false,
    #moobot            : "SituationMgr.js",
    moobot             : "SituationMgrNetcool.js",
    process_output_of  : [
                          "ITNM",
                          "Sigaliser",
                          "TemplateMatcher",
                          "Speedbird"
                         ]
  4. Save the Moogfarmd configuration file.

Moobot Configuration
Alert Builder Moobot

The Netcool Alert Builder is used to detect whether an event received is repeating or not, and whether an alert should be created from it. This detection is performed in addition to the standard functionality of the Alert Builder.

To enable the repeat detection functionality, in the Moogfarmd configuration file ($MOOGSOFT/config/moog_farmd.conf) in the AlertBuilder moolet section, uncomment the Alert Builder Moobot file AlertBuilderNetcool.js as shown below:

name              : "AlertBuilder",
classname         : "CAlertBuilder",
run_on_startup    : true,
#moobot           : "AlertBuilder.js",
moobot            : "AlertBuilderNetcool.js",

Ensure that run_on_startup is set to true

Save the changes

In the Alert Builder Moobot ($MOOGSOFT/bots/moobots/AlertBuilderNetcool.js), there are three configuration settings that can be set as follows:

var repeatDetection=true;
var stopDeduplication=true;
var overwriteCustomInfo=false;

repeatDetection

If an incoming Event is an UPDATE type, repeat detection is enabled (DELETE and INSERT types cannot be repeating events). The Event signature is then matched to existing alerts. If no match is found, or a match is found to a closed alert, a new alert is created. If an open alert match is found, de-duplication is carried out (see below).

  • Set the repeatDetection value to true to enable the repeat detection process, where the Netcool Alert Builder Moobot determines if newly created alerts are similar to existing alerts.

stopDeduplication

This determines whether the repeat detection process is carried out before or after an alert is created.

  • Set stopDeduplication to true to prevent the de-duplication process from occurring. The repeat detection process is then performed before an alert is created.

  • Set stopDeduplication to false to create a new alert based on the event that has been received. Then the repeat detection process is performed, based on the newly created alert and existing alerts.

Note

Setting stopDeduplication to false (repeat detection after alert creation) will also allow you to forward the alert to a different Moolet chain, if required.

overwriteCustomInfo

This defines whether the custom_info is updated when updating (de-duplicating) alerts.

  • Set overwriteCustomInfo to true to update alerts custom_info from new event data.

  • Set overwriteCustomInfo to false to leave alerts custom_info unchanged when an alert is updated.

Note

If IBM Tivoli Netcool Omnibus is sending Route Cause and Symptom events from ITNM (IBM Tivoli Network Manager), and using ITNM is set to to true in the LAMbot configuration file (see LAMbot configuration above), then the alerts custom_info field must be updated when de-duplicating: set overwriteCustomInfo to true.

AlertRulesEngine Moobot

Enable the AlertRulesEngineNetcool.js Moobot, along with the associated action states and transitions. It is used to process DELETE and REPEAT type alerts, determining whether they are discarded or passed onto the Sigalisers for further processing.

To do this, open the Moogfarmd configuration file and uncomment the line containing the AlertRulesEngineNetcool.js Moobot within the AlertRulesEngine moolet section, as shown below:

name                   : "AlertRulesEngine",
classname              : "CAlertRulesEngine",
run_on_startup         : true,
metric_path_moolet     : true,
#moobot                : "AlertRulesEngine.js",
moobot                 : "AlertRulesEngineNetcool.js",
#standalone            : true
process_output_of      : "AlertBuilder"

Ensure that run_on_startup is set to true

Save the changes

The rules (action states and transitions) to process DELETE and INSERT type Alerts should be added to the Moogsoft AIOps instance by entering the following command:

sh $MOOGSOFT_HOME/bin/utils/moog_netcool_are_installer
Final steps

Restart the Moogfarmd process and then start the Legacy LAM to begin listening and receiving data from IBM Tivoli Netcool Omnibus on the defined socket.

To do this:

service moogfarmd restart
service netcoollamd start

Note

To check the Legacy LAM status:

service netcoollamd status

To stop the Legacy LAM:

service netcoollamd stop

Default Field Mapping

The following table shows the default IBM Tivoli Netcool Omnibus field mappings to Moogsoft AIOps fields. These mappings are defined either in the Legacy LAM configuration file or within the LAMbot:

IBM Tivoli Netcool Omnibus > Moogsoft AIOps Field Mappings

Netcool Field Name

Moogsoft AIOps Field Name

Data type

Mandatory

ActionType

type

varchar(255)

Yes

Identifier

signature

varchar(255)

Yes

Serial

external_id

incr

Yes

Node

source

varchar(64)

Yes

LocalNodeAlias

source_id

varchar(64)

Yes

Manager

manager

varchar(64)

Yes

Agent

agent

varchar(64)

No

AlertKey

agent_location

varchar(255)

Yes

AlertGroup

agent_location

varchar(255)

No

Severity

severity

integer

Yes

Summary

description

varchar(255)

Yes

FirstOccurrence

first_occurred

integer

Yes

LastOccurrence

agent_time

integer

Yes

Class

class

integer

Yes

OwnerUID

custom_info.ownerUID

integer

Yes

Acknowledged

custom_info.acknowledged

integer

Yes

ExpireTime

custom_info.expireTime

integer

Yes

SuppressEscl

custom_info.suppressEscl

integer

Yes

TaskList

custom_info.taskList

integer

Yes

LocalRootObj

custom_info.localRootObj

varchar(255)

Yes

RemoteNodeAlias

custom_info.remoteNodeAlias

varchar(64)

Yes

RemoteRootObj

custom_info.remoteRootObj

varchar(255)

Yes

ServerName

custom_info.serverName

varchar(64)

Yes

ServerSerial

custom_info.serverSerial

integer

Yes

StateChange

custom_info.stateChange

integer

Yes

InternalLast

custom_info.internalLast

integer

Yes

Tally

custom_info.tally

integer

Yes

Type

custom_info.eventType

integer

No

EventId

custom_info.eventId

varchar(255)

No

NodeAlias

custom_info.NodeAlias

varchar(64)

No

NmosCauseType

custom_info.nmosCauseType

integer

No

NmosSerial

custom_info.nmosSerial

varchar(64)

No

Note

The severity mapping, as set in the Legacy LAM configuration severity section, is set by default to be identical to the severity values used within IBM Tivoli Netcool Omnibus. This can be changed if necessary under the severity section in the Legacy LAM configuration file.

For example:

constants:
{
    severity:
    {
        "0" : 0,
        "1" : 1,
        "2" : 2,
        "3" : 3,
        "4" : 4,
        "5" : 5
    }
},