Skip to main content

Monitor System Performance Metrics

Navigate to System Settings > Self Monitoring > Processing Metrics to see a breakdown of the current state of the system based on the metrics received from the running components.

  • The Moogfarmd process and all LAMs publish detailed performance information.

  • A bullet chart at the top of the page shows the key performance metric for the system: Current Maximum Event Processing Time. The defined performance ranges are color coded: good (green), marginal (yellow) and bad (red). As the metric changes the bullet chart updates to reflect good, marginal or bad performance.

  • The system calculates Current Maximum Event Processing Time as the approximate 95th percentile of the current maximum time in seconds that it takes for an event to make its way through the system from its arrival at a LAM until its final processing by a Moolet in Moogfarmd.

  • By default, AlertBuilder, AlertRulesEngine and All Sigalisers are used to calculate the Current Maximum Event Processing Time metric.

  • You can configure the metric_path_moolet property in moog_farmd.conf to specify the Moolets to use to calculate Current Maximum Event Processing Time.

  • By default, the good, marginal and bad ranges of the bullet chart are set to 0-10secs, 10-15secs and 15-20secs respectively. You can change the configuration in the in the eventProcessingTimes section in the portal block of $MOOGSOFT_HOME/ui/html/web.conf.

Good performance means LAMs are consuming and publishing events without problem as indicated by:

  • Message Queue Size is 0.

  • Socket Backlog (if relevant) is not increasing.

Additionally, Moogfarmd is consuming and processing events successfully as indicated by all of:

    • Total Abandoned Messages is 0 for the majority of the time.

    • Asynchronous Task Queue Size is 0 for the majority of the time.

    • Cookbook Resolution Queue is 0 for the majority of the time.

    • Message backlogs for all Moolets is 0 for the majority of the time.

    • The Messages Processed count for all running Moolets should be the same (unless custom configuration causes event routing through different Moolets) i.e. no Moolet is falling behind.

The above should lead to a stable low Current Maximum Event Processing Time depending on the complexity of the system.

Marginal or Bad performance means LAMs are not consuming and publishing events at the rate at which they receive them, as indicated by:

    • Message Queue Size is > 0 and likely increasing.

    • Socket Backlog is increasing.

Additionally, Moogfarmd is not consuming and processing events in a timely fashion as indicated by some or all of:

    • Total Abandoned Messages is constantly > 0 and likely increasing.

    • Asynchronous Task Queue Size is > 0 and likely increasing.

    • Cookbook Resolution Queue is constantly > 0 and likely increasing.

    • Message backlogs for all Moolets is constantly > 0 and likely increasing.

    • The Messages Processed count for all running Moolets is not the same indicating that some Moolets are falling behind. This doesn't apply for cases where custom configuration causes event routing through different Moolets.

The above will likely lead to an unstable high Current Maximum Event Processing Time depending on the complexity of the system.

See Self Monitoring for more detail.

In this section: