# Moogsoft Docs

## Heartbeat Monitor

You can configure the Alert Rules Engine Moolet in Moogsoft AIOps to detect missing heartbeat events from monitoring tools such as CA Spectrum and Microsoft SCOM. Both of these tools send regular heartbeats to indicate normal operation.

After you configure the Alert Rules Engine, Moogsoft AIOps creates a Situation when an event source does not send a heartbeat after a given time period. The Alert Rules Engine holds each heartbeat alert for a period of time, subsequent alerts from the same heartbeat source reset the timer. If the timer expires, a heartbeat has been missed and the alert is forwarded to a Sigaliser.

## Before You Begin

Before you set up the heartbeat monitor in Alert Rules Engine, ensure you have met the following requirements:

• You have an understanding of Alert Rules Engine, Action States and transitions. See Alert Rules Engine Moolet.
• You can identify heartbeat alerts in the integration by description, class or another configurable field. These must be specific, regular events that arrive at consistent intervals to indicate normal operation. If these are not available the Heartbeat Monitor will not work.
• You have edited the alerts so they contain the same attribute (via the integration source or through enrichment). In the example below, '  type  ' is 'heartbeat' in the Alert Rules Engine trigger filter and '  class  ' is 'heartbeat' in the Cookbook Recipe trigger filter.

## Create a Heartbeat Monitor

To create a heartbeat monitor in Alert Rules Engine, follow these steps:

1. Edit  $MOOGSOST_HOME/bots/moobots/AlertRulesEngine.js  . 2. Add the heartBeatSeverity exit action to the  AlertRulesEngine.js  . This function changes the alert severity to critical and ensures alerts that are closed (see Status ID Reference ) are not forwarded to the Cookbook: // Checks the state of the alert. If the alert's state is 9, the status ID for 'closed', it is not forwarded. function heartBeatSeverity(alert,associated) { var currentAlert = moogdb.getAlert(alert.value("alert_id")); if ( currentAlert && currentAlert.value("state") !== 9 ) { // Adds customInfo list displaying times of all missed heartbeats. var customInfo=currentAlert.getCustomInfo(); if ( !customInfo ) { customInfo={}; customInfo.missedHeartbeats=[]; } if ( !customInfo.missedHeartbeats ) { customInfo.missedHeartbeats=[]; } // Changes severity of heartbeat alerts to 5, the severity level for 'critical', adds the description 'MISSED' and determines where the alerts are sent. var now = new Date(); customInfo.missedHeartbeats.push(now.toString()); moogdb.setAlertCustomInfo(currentAlert.value("alert_id"),customInfo,false); moogdb.setAlertSeverity(currentAlert.value("alert_id"),5); var alertDescr = currentAlert.value("description"); alertDescr = alertDescr.replace(/(OK|SLOW)/,"MISSED"); currentAlert.set("description",alertDescr); moogdb.updateAlert(currentAlert); currentAlert.forward("HeartbeatCookBook"); } }  3. Navigate to Settings > Action States in the Moogsoft AIOps UI. 4. Create a new Action State called "Heartbeat" as follows: Setting Name Input Value Name String Heartbeat Remember alerts for Integer (seconds) 30 * Cascade on expiry Boolean True Exit Action String heartBeatSeverity * The  Remember alerts for  setting is the timer. Set this value to two or three times your heartbeart interval time. 5. Go to Transitions and set up a transition to move your heartbeat alerts to the 'Heartbeat' State. Configure the settings as follows: Setting Name Value Name Heartbeat Priority 10 Active True Trigger Filter (type = "heartbeat") AND ((((agent = "SPECTRUM") OR (manager= "SCOM")) OR (agent = "MONITOR1")) OR (agent = "MONITOR2")) Start State Ground End State Heartbeat Edit the 'Trigger Filter' to meet your requirements. In this example, the transition is triggered by alerts with the type of 'heartbeat' and that come from either 'SPECTRUM' or 'SCOM' or 'MONITOR1' or 'MONITOR2': 6. Edit the $MOOGSOFT_HOME/config/moolets/alert_rules_engine.conf  file to ensure that the Alert Rules Engine is enabled. To do this set  run_on_startup  to  true  .

7. Create  a new  heartbeat.conf  configuration file in  $MOOGSOFT_HOME/config/moolets  to a dd a Heartbeat Cookbook for heartbeat alerts. This only processes these heartbeat alerts. { name:"HeartbeatCookBook", classname:"CCookbook", run_on_startup:true, moobot:"Cookbook.js", process_output_of:"[]", # Algorithm membership_limit:5, scale_by_severity:false, entropy_threshold:0.0, single_recipe_matching:false, recipes:[ # Any heartbeat class for the same agent. { chef:"CValueRecipe", name:"ScomHeartbeatErrors", description:"SCOM Heartbeat: Missing heartbeat", recipe_alert_threshold:0, exclusion:"state = 9", trigger:"class = 'heartbeat' AND agent = 'SCOM'", rate:0, # Given in events per minutee min_sample_size:5, max_sample_size:10, matcher:{ components:[ { name:"agent", similarity:1.0 } ] } }, { chef:"CValueRecipe", name:"ScomHeartbeatChange", description:"SCOM Heartbeat: Cluster host change", recipe_alert_threshold:0, exclusion:"state = 9", trigger:"class = 'heartbeatRoleChange' AND agent = 'SCOM'", rate:0, # Given in events per minutee min_sample_size:5, max_sample_size:10, matcher:{ components:[ { name:"agent", similarity:1.0 } ] } } ], cook_for:20000 } Save the changes to  heartbeat.conf  . 8. Edit the Moogfarmd configuration file $MOOGSOFT_HOME/config/moog_farmd.conf  to add a new merge group that references the HeartBeatCookbook moolet. Configure this merge group to have an  alert_threshold  of 1 to allow a single alert to create a Situation (by default, a minimum of 2 alerts are required to create a Situation):

merge_groups:
[
{
name: "Heartbeat",
moolets: ["HeartbeatCookBook"],
sig_similarity_limit : 1
}
],
9. Include the new  heartbeat.conf  Moolet configuration by adding the following in  \$MOOGSOFT_HOME/config/moog_farmd.conf  :

          {
include : "heartbeat.conf"
},

Save the changes to  moog_farmd.conf  .|

10. Restart Moogfarmd:

service moogfarmd restart

After the heartbeat monitor configuration is complete, heartbeat alerts should start to arrive in Moogsoft AIOps.

## Heartbeat Monitor Process

The process flow for a heartbeat alert is as follows: