Events, Alerts, and Incidents

Before moving on to more advanced features, it's helpful to understand the three basic informational elements of Moogsoft: events, alerts, and incidents.

Events

Events are occurrences on monitored systems outside of Moogsoft that are passed into Moogsoft. They are identified by user-defined configurations, which specify the characteristics of an occurrence of special interest.

Events are ingested by Moogsoft through:

Events in originating systems have vendor-specific (or user-defined) event payload fields, but Moogsoft recognizes the following values for severity:

Unknown

Clear

Warning

Minor

Major

Critical

Severity levels are typically mapped to the corresponding payload values in incoming events as a part of integration setup.

Alert and incident creation from events

There are two paths for creating alerts and incidents in Moogsoft from events:

  • Starting with metrics:

    metrics_to_incident.png
  • Starting with events:

    events_to_incident.png

Alerts

An alert is a unique event as identified by Moogsoft.

Alerts are created from:

  • Ingested events

  • Events derived from metrics that deviate from expected values

Moogsoft deduplicates ingested events and creates an alert to uniquely represent a specific group of repeated events or metric data points.

Like the triggering events they are derived from, alerts have a variety of associated parameters. They include identifying fields such as source, time, description, and other fields which provide useful information for determining event uniqueness when reducing noise by consolidating repeated events. For more information, see Deduplication: events to alerts.

Two fields--severity and status--are of particular interest because they directly impact Moogsoft processes and alert and incident visibility. For more information about alert fields, see Alerts and alert details reference.

Severity

All alerts have an associated severity, either assigned by Moogsoft (for metrics) or the originating external system (for integrations).

Alert severity

Meaning for alerts derived from events

Meaning for alerts derived from metrics

Unknown

Displays when Moogsoft cannot identify the severity in the event payload; occurs when an event severity is not mapped to a recognized severity in Moogsoft.

Moogsoft has not yet ingested enough data to accurately assign a severity. An Unknown severity typically displays when the system is still gathering enough data to determine the normal values for a metric.

Clear

An outside system has sent an event through an integration or the Events API with a payload field mapping to the Clear severity in Moogsoft; Clear indicates that an alert is resolved.

Values for a metric which was producing anomalous values are now within expected thresholds.

Warning

Assigned by payload fields in originating systems.

Determined by the degree a metric value is either above or below the configured metric thresholds; controlled by metrics configurations and automatic system learning.

Minor

Major

Critical

The severity of an alert is determined by the severity of the most recently received event matching the existing alert. It is not user editable.

Example: Moogsoft receives three events and determines they are similar enough to represent with a single alert with Minor severity. A fourth event with Critical severity is ingested and Moogsoft determines that this event represents the same condition which generated the other three events. Moogsoft removes the duplicate event and changes the severity for the representative alert to Critical.

Status

Unlike severity, alert statuses are user editable. Alert statuses can also affect the status of incidents.

NOTE: You can customize system default settings using Auto Close.

Alert status

An alert changes to this status when:

Impact to incidents

System defaults

Open

The alert is first generated from received events.

NOTE: The age of an alert is based on the time the alert has existed in Moogsoft since it was opened.

Incidents are created from correlated open alerts.

An alert can appear in multiple incidents because the occurrence represented by an alert can be a factor in multiple issues.

All alerts initially have a status of Open.

In Progress

A user manually assigns this alert to a Moogsoft user. The In Progress status is always assigned manually.

When all of the alerts in an incident have a status of In Progress, the incident status changes to In Progress.

NA

Resolved

A user manually sets the status of this alert to Resolved.

When Moogsoft ingests an event with a severity of Clear, the status of the representative alert changes to Resolved.

When all of the alerts in an incident have a status of Resolved, the incident status changes to Resolved.

NA

Closed

A Moogsoft user manually closes the alert, or the default length of time elapses to automatically close a resolved alert.

Incidents have a status of Closed when a user manually closes them or when they exceed the configured age to close automatically.

Status = Closed 30 minutes after Status = Resolved

Status = Closed from any state after 72 hours

Alert statuses can change to other statuses in the following ways:

From

To

Open

In Progress

Resolved

Closed

In Progress

Resolved

Closed

Resolved

Open

In Progress

Closed

Closed

NA

Closed is a final status and cannot revert to any other status

Incidents

When Moogsoft creates an alert, it also creates an incident from that alert. Moogsoft determines if alerts are similar and, if they are similar enough, they are added as components of a single incident. Any matching incoming alerts are added to the incident as long as the incident has a status other than Closed.

For more information on incident fields, see Incidents and incident details reference.

Severity

Like alerts, incidents have an assigned severity. Incident severity is derived from the alerts comprising it.

Incident severity

This alert status means:

Unknown

All alerts in the incident have a severity of Unknown.

If a matching alert with a different severity is added to the incident, that severity replaces Unknown.

Clear

All alerts in the incident have a severity of Clear.

Warning

The severity of an incident (Warning through Critical) is determined by the highest severity in the included alerts.

Example: Moogsoft creates an incident from three alerts with the following severity values: Minor, Major, Warning. The severity of the incident is Major.

Minor

Major

Critical

Status

Incident statuses are user editable and are subject to automatic changes due to system settings.

Just as alert status changes can affect incidents, changes in incident statuses can affect alerts.

NOTE: You can customize system default settings using Auto Close.

Incident status

An incident changes to this status when:

Impacts to alerts

System defaults

Open

The incident is created.

Open is the default status for new incidents.

A resolved incident status returns to Open if an alert with a severity higher than Clear is added.

None.

An Open status for an incident indicates that at least one included alert is still unresolved.

All incidents initially have an Open status.

In Progress

A user manually assigns this incident to a Moogsoft user.

The In Progress status is always assigned manually.

When an incident status changes to In Progress, the statuses of all included alerts also change to In Progress.

Moogsoft assigns all alerts included in the incident to the same user the incident is assigned to. If a different user is already assigned to an alert, that assignment does not change.

NA

Resolved

A user manually sets the status of the incident to Resolved, or all alerts included in the incident have a severity of Clear.

The status for all included alerts changes to Resolved. The severity for all included alerts changes to Clear.

Incident Status = Closed 53 minutes after all alerts in the incident are closed

Closed

A Moogsoft user manually closes the incident, or the incident automatically closes based on system settings.

The status of the alert with the highest severity in the incident changes to Closed and the alert severity changes to Clear.

No further alerts are added to the incident; no further changes occur to the incident or included alerts.

Incidents older than seven days automatically close.

Incident statuses can change to other statuses in the following ways:

From

To

Open

In Progress

Resolved

Closed

In Progress

Resolved

Closed

Resolved

Open

In Progress

Closed

Closed

NA

Closed is a final status and cannot revert to any other status