Page tree
Skip to end of metadata
Go to start of metadata

Introduction

The Legacy LAM enables Moogsoft AIOps to receive data from IBM® Tivoli® Netcool® Omnibus.

IBM Tivoli Netcool Omnibus includes a socket gateway which is able to pass data to a third party system. This permits integration between IBM Tivoli Netcool Omnibus and AIOps.

In its default configuration, the Legacy LAM will ingest data from a default IBM Tivoli Netcool Omnibus set up. Mapper utilities are also provided to enable data ingestion from a non-default IBM Tivoli Netcool Omnibus set up. In addition, the following MooBots are provided:

FileDescription
AlertBuilderNetcool.js

This  MooBot file allows special handling for the 'repeatDetection' and 'stopDeduplication' processes

AlertRulesEngineNetcool.js This MooBot file is used to process DELETE and REPEAT type Events
SituationMgrNetcool.jsThis MooBot file creates Situations around root cause and symptom detection, based on IBM Tivoli Network Manager parameters

Capabilities

  • Data received from IBM Tivoli Netcool Omnibus has fields checked to identify event type (eg. Problem, Resolution), state (eg. INSERT, UPDATE, DELETE) and severity
  • Events are then created by the Legacy LAMBot (based on the LAMBot configuration), containing all the mandatory Netcool event fields, and any additional optional fields
  • Events are passed to the Netcool AlertBuilder for de-duplication and Alert creation
  • Alerts are then passed to the Netcool AlertRulesEngine for filtering and passing on to Sigalisers for further processing
  • Legacy LAM can process ITNM (IBM Tivoli Network Manager) Route Cause and Symptom events, using Cookbook to create Situations

Requirements

The ingestion of data from IBM Tivoli Netcool Omnibus requires that its gateway be configured according to the configurations set within AIOps. To be able to get data from IBM Tivoli Netcool Omnibus, AIOps must be configured to allow inbound communication from IBM Tivoli Netcool Omnibus.

To ingest data there must be mapping in the NetcoolLAM between AIOps Alert fields and IBM Tivoli Netcool Omnibus alert fields. Mapping of some IBM Tivoli Netcool Omnibus fields is mandatory, as they are required in any IBM Tivoli Netcool Omnibus system. These fields are mapped by default. Mapping of the remaining IBM Tivoli Netcool Omnibus fields is optional; they are not mapped by default.

The set up of IBM Tivoli Netcool Omnibus to send data and the configuration for AIOps to ingest and process the data are explained below.

IBM Tivoli Netcool Omnibus configuration

An IBM Tivoli Netcool Omnibus system administrator is required

Configure the socket gateway and socket map files to send data to AIOps.

Socket gateway configuration

In the IBM Netcool Omnibus gateway configuration file (by default named NCO_GATE.props), set:

FieldSet to:
Gate.Socket.Host Hostname or IP address of the AIOps system
Gate.Socket.PortPort number defined in the Legacy LAM. Eg. 8411
Gate.Socket.SeparatorThe delimiter used in the data sent to AIOps. Set this to double pipe - ||
Gate.Socket.InsertTrailerNC_EVENT_END;\n
Gate.Socket.UpdateTrailerNC_EVENT_END;\n
Gate.Socket.DeleteTrailerNC_EVENT_END;\n
Gate.Socket.InsertHeaderINSERT||
Gate.Socket.UpdateHeaderUPDATE||
Gate.Socket.DeleteHeaderDELETE||

Socket map configuration

In the gateway mappings configuration file (by default named socket.map), set the fields to be sent to AIOps, ensuring that no field is set to be ON INSERT ONLY. Gateway mapping should include all the mandatory fields, and any additional optional ones (such as @NodeAlias).

Mappings should look similar to the example below:

CREATE MAPPING StatusMap
 (
        '' = '@Identifier',
        '' = '@Serial',
        '' = '@Node',
        '' = '@LocalNodeAlias',
        '' = '@Manager',
        '' = '@Agent',
        '' = '@AlertGroup',
        '' = '@AlertKey',
        '' = '@Severity',
        '' = '@Summary',
        '' = '@FirstOccurrence',        
        '' = '@LastOccurrence',
        '' = '@Class',        
        '' = '@OwnerUID',         
        '' = '@Acknowledged',
        '' = '@ExpireTime',
        '' = '@SuppressEscl',
        '' = '@TaskList',
        '' = '@LocalRootObj',       
        '' = '@RemoteNodeAlias',
        '' = '@RemoteRootObj',        
        '' = '@ServerName',
        '' = '@ServerSerial',
        '' = '@StateChange',        
        '' = '@InternalLast',
        '' = '@Tally',                                                          
        '' = '@Type',
        '' = '@EventId',
        '' = '@NodeAlias'
 );

AIOps configuration

 In the above example mappings configuration, the fields in red are optional. All other fields are mandatory, and are required to be able to ingest data into AIOps.
 

To run the Legacy LAM, the following AIOps files are required and are installed by default:

FileDescription
$MOOGSOFT/config/netcool_lam.confLegacy LAM configuration file
$MOOGSOFT/bots/lambots/NetcoolLam.jsLAMBot file; performs the main processing of the events received from IBM Tivoli Netcool Omnibus
$MOOGSOFT/bots/lambots/NetcoolUtility.jsHolds additional functions and configurations required for the LAMBot
$MOOGSOFT_HOME/bin/utils/netcool_lam.conf.templateUsed by the moog_netcool_lam_mapper script as a base template for generating a new Legacy LAM configuration file. See the following sections

The following sections detail the steps required to configure the Legacy LAM, LAMBot and MooBots:

Legacy LAM configuration

Mapping between IBM Tivoli Netcool Omnibus data fields and AIOps fields is defined in the Legacy LAM configuration file netcool_lam.conf. As installed, netcool_lam.conf maps to the data fields in a default set up of IBM Tivoli Netcool Omnibus. If ingesting data from a non-default IBM Tivoli Netcool Omnibus set up, a mapping utility moog_netcool_lam_mapper is available to map between IBM Tivoli Netcool Omnibus data fields and AIOps fields. You can also enter the mapping by manually editing netcool_lam.conf

The mapping utility can use either an IBM Tivoli Netcool Omnibus map file, or a log file containing Event data from IBM Tivoli Netcool Omnibus:  

Before running the mapping utility, backup the existing netcool_lam.conf file

To use a map file with the mapping utility, enter the command and arguments as in the following example (the map file is socket.map):

sh $MOOGSOFT_HOME/bin/utils/moog_netcool_lam_mapper -f socket.map -t map

To use a log file with the mapping utility, enter the command and arguments as in the following example (the log file is event-data.txt):

sh $MOOGSOFT_HOME/bin/utils/moog_netcool_lam_mapper -f event-data.txt -t logfile

When running mapping utility (using either a map file or a log file), you can also optionally set the address (using the -a argument in the command line) and port number (using the -p argument in the command line), as shown in the following example:

sh $MOOGSOFT_HOME/bin/utils/moog_netcool_lam_mapper -a remote_host -f socket.map -p 8455 -t map

The above example sets the address to remote_host and the port number to 8455.

Once the mapping utility has successfully completed, a new Legacy LAM configuration file netcool_lam.conf is generated and is automatically placed in the default configuration directory ($MOOGSOFT/config), overwriting the existing netcool_lam.conf file

Check the Legacy LAM configuration file to ensure that the following fields are set correctly:

FieldDescriptionExample
portThe port number where the Legacy LAM will be receiving data from IBM Tivoli Netcool OmnibusPort number 8411
addressThis should be localhost or the hostname of the system running AIOpsIf AIOps is running on premise the default address is : 0.0.0.0 
If AIOps is running on e.g. Amazon web services, it may be similar to ew2.234.234.compute.amazonaws.com
modeThe operation mode in which the socket LAM will run should be set to SERVER

LAMBot configuration

The LAMBot performs the core processing of the data received from IBM Tivoli Netcool Omnibus. The main configuration option in the LAMBot is whether DELETE type events are processed or not. By default this is disabled (the keepDeletes value set to in the LAMBot). This may be changed if necessary, by editing the NetcoolLam.js file.

Additionally, there is an optional configuration in the LAMBot that will enable the processing of ITNM (IBM Tivoli Network Manager) Route Cause and Symptom events, based on the values of the fields @NmosCauseType and @NmosSerial. These two fields must be included in the event to be able to enable processing. By default, it is set to disabled. To enable it, set the usingITNM value to true in the NetcoolLam.js file.


Only proceed with the remaining steps in this section if you have enabled the use of ITNM (see above)
Then go to the next section: MooBots set up

Enabling the processing of ITNM fields requires the associated Moolets, Cookbook and Recipe to be enabled in the moog_farmd configuration file $MOOGSOFT/config/moog_farmd.conf as follows:

  1. In the sig_resolution section, uncomment the following merge group, making it available as an additional merge group:

    merge_groups:
        [
             {
                  name: "ITNM Route Causes & Symptoms",
                  moolets: ["ITNM"],
                  alert_threshold      : 2,
                  sig_similarity_limit : 0.65
              }
        ],
  2. In the moolets section, uncomment the ITNM Cookbook and recipe:

    {
        # Moolet
        name                : "ITNM",
        classname           : "CCookbook",
        run_on_startup      : true,
        metric_path_moolet  : true,
        moobot              : "Cookbook.js",
        #process_output_of  : "AlertRulesEngine",
        process_output_of   : "AlertBuilder",
    
    
        # Algorithm
        membership_limit       : 1,
        scale_by_severity      : false,
        entropy_threshold      : 0,
        single_recipe_matching : false,
        
             recipes : [
             {
                  chef                   : "CValueRecipe",
                  name                   : "ITNM Route Cause & Symptom Detection",
                  description            : "Root cause and Symptom alerts detected based on ITNM",
                  recipe_alert_threshold : 1,
                  exclusion              : "custom_info.nmosCauseType = 0",
                  trigger                : "custom_info.suppressedSerial > 0",
                  rate                   : 0,
                  min_sample_size        : 5,
                  max_sample_size        : 10,
                  cook_for               : 1200,
                  matcher                :    {
                                              components:[
                                                         { name: "custom_info.suppressedSerial", similarity: 1.0 }
                                                         ]
                                              }
             }
                       ],
        cook_for  :    1200
    }
  3. Enable the Netcool Situation Manager Moolet:
    • Set run_on_startup value set to true for the SituationMgr Moolet
     • Uncomment the line containing the Netcool MooBot - SituationMgrNetcool.js (the file is installed by default)
     • Uncomment ITNM in the section process_output_of to include the ITNM Cookbook.
    The Moolet section should now look similar to the one below:

    name               : "SituationMgr",
    classname          : "CSituationMgr",
    run_on_startup     : true,
    metric_path_moolet : false,
    #moobot            : "SituationMgr.js",
    moobot             : "SituationMgrNetcool.js",
    
    process_output_of  : [
                          "ITNM",
                          "Sigaliser",
                          "TemplateMatcher",
                          "Speedbird"
                         ]
  4. Save the moog_farmd configuration file. 


MooBots set up

AlertBuilder MooBot

The Netcool AlertBuilder is used to detect whether an Event received is repeating or not, and whether an Alert should be created from it. This detection is performed in addition to the standard functionality of the AlertBuilder.

To enable the repeat detection functionality, in the moog_farmd configuration file ($MOOGSOFT/config/moog_farmd.conf) in the AlertBuilder moolet section, uncomment the AlertBuilder MooBot file AlertBuilderNetcool.js as shown below:

name              : "AlertBuilder",
classname         : "CAlertBuilder",
run_on_startup    : true,
#moobot           : "AlertBuilder.js",
moobot            : "AlertBuilderNetcool.js",

Ensure that run_on_startup is set to true
Save the changes

In the AlertBuilderNetcool.js MooBot ($MOOGSOFT/bots/moobots/AlertBuilderNetcool.js), there are three configuration settings that can be set (these are located near the top of the file) as shown and explained below:
var repeatDetection=true;
var stopDeduplication=true;
var overwriteCustomInfo=false;

repeatDetection
If an incoming Event is an UPDATE type, repeat detection is enabled (DELETE and INSERT types cannot be repeating events). The Event signature is then matched to existing Alerts. If no match is found, or a match is found to a closed Alert, a new Alert is created. If an open Alert match is found, de-duplication is carried out (see below).

  • Set the repeatDetection value to true to enable the repeat detection process, where the Netcool AlertBuilder MooBot determines if newly created Alerts are similar to existing Alerts
     

stopDeduplication
This determines whether the repeat detection process is carried out before or after an Alert is created.

  • Set stopDeduplication to true to prevent the de-duplication process from occurring. The repeat detection process is then performed before an Alert is created

  • SetstopDeduplication to false to create a new Alert based on the Event that has been received. Then the repeat detection process is performed, based on the newly created Alert and existing Alerts
     

Setting stopDeduplication to false (repeat detection after Alert creation) will also allow you to forward the Alert to a different Moolet chain, if required

overwriteCustomInfo 
This defines whether the custom_info is updated when updating (de-duplicating) Alerts.

  • Set overwriteCustomInfo to true to update Alerts custom_info from new event data
  • Set overwriteCustomInfo to false to leave Alerts custom_info unchanged when an Alert is updated

If IBM Tivoli Netcool Omnibus is sending Route Cause and Symptom events from ITNM (IBM Tivoli Network Manager), and usingITNM is set to to true in the LAMBot configuration file (see LAMBot configuration above), then the Alerts custom_info field must be updated when de-duplicating: set overwriteCustomInfo to true

AlertRulesEngine MooBot

Enable the AlertRulesEngineNetcool.js MooBot, along with the associated action states and transitions. It is used to process DELETE and REPEAT type Alerts, determining whether they are discarded or passed onto the Sigalisers for further processing.

To do this, open the moog_farmd configuration file and uncomment the line containing the AlertRulesEngineNetcool.js MooBot within the AlertRulesEngine moolet section, as shown below:

name                   : "AlertRulesEngine",
classname              : "CAlertRulesEngine",
run_on_startup         : true,
metric_path_moolet     : true,
#moobot                : "AlertRulesEngine.js",
moobot                 : "AlertRulesEngineNetcool.js",


#standalone            : true
process_output_of      : "AlertBuilder"

Ensure that run_on_startup is set to true
Save the changes

The rules (action states and transitions) to process DELETE and INSERT type Alerts should be added to the AIOps instance by entering the following command:
 
sh $MOOGSOFT_HOME/bin/utils/moog_netcool_are_installer

Final steps

Restart the moog_farmd process and then start the Legacy LAM to begin listening and receiving data from IBM Tivoli Netcool Omnibus on the defined socket.

To do this:

service moogfarmd restart
service netcoollamd start

To check the Legacy LAM status:

service netcoollamd status

To stop the Legacy LAM:

service netcoollamd stop

Default field mapping

The following table shows the default IBM Tivoli Netcool Omnibus field mappings to AIOps fields.
These mappings are defined either in the Legacy LAM configuration file or within the LAMBot:

IBM Tivoli Netcool Omnibus > AIOps Field Mappings

Netcool Field Name

AIOps Field Name

Data type

Mandatory

ActionType

type

varchar(255)

Yes

Identifier

signature

varchar(255)

Yes

Serial

external_id

incr

Yes

Node

source

varchar(64)

Yes

LocalNodeAlias

source_id

varchar(64)

Yes

Manager

manager

varchar(64)

Yes

Agent

agent

varchar(64)

No

AlertKey

agent_location

varchar(255)

Yes

AlertGroup

agent_location

varchar(255)

No

Severity

severity

integer

Yes

Summary

description

varchar(255)

Yes

FirstOccurrence

first_occurred

integer

Yes

LastOccurrence

agent_time

integer

Yes

Class

class

integer

Yes

OwnerUID

custom_info.ownerUID

integer

Yes

Acknowledged

custom_info.acknowledged

integer

Yes

ExpireTime

custom_info.expireTime

integer

Yes

SuppressEscl

custom_info.suppressEscl

integer

Yes

TaskList

custom_info.taskList

integer

Yes

LocalRootObj

custom_info.localRootObj

varchar(255)

Yes

RemoteNodeAlias

custom_info.remoteNodeAlias

varchar(64)

Yes

RemoteRootObj

custom_info.remoteRootObj

varchar(255)

Yes

ServerName

custom_info.serverName

varchar(64)

Yes

ServerSerial

custom_info.serverSerial

integer

Yes

StateChange

custom_info.stateChange

integer

Yes

InternalLast

custom_info.internalLast

integer

Yes

Tally

custom_info.tally

integer

Yes

Type

custom_info.eventType

integer

No

EventId

custom_info.eventId

varchar(255)

No

NodeAlias

custom_info.NodeAlias

varchar(64)

No

NmosCauseType

custom_info.nmosCauseType

integer

No

NmosSerial

custom_info.nmosSerial

varchar(64)

No

The severity mapping, as set in the Legacy LAM configuration severity section, is set by default to be identical to the severity values used within IBM Tivoli Netcool Omnibus. This can be changed if necessary under the severity section in the Legacy LAM configuration file.

For example:

constants:
        {
            severity:
            {
                "0" : 0,
                "1" : 1,
                "2" : 2,
                "3" : 3,
                "4" : 4,
                "5" : 5
            }
        }, 
  • No labels