Deduplication: Events into Alerts

This topic describes how Express removes duplicate events from the data ingestion stream.

Deduplication and noise reduction

A busy service with multiple monitors can generate a flood of metrics, anomalies, and events. One issue might trigger a large number of repeat and duplicate events. Express analyzes every new piece of data — What is this? When did it happen? What is its severity? How often has it happened before? — and aggregates events for the same issue into alerts. Whenever it adds a new event, Express updates the alert fields — event count, last event time, severity — so the alert always contains the latest information about the underlying issue. This process removes the duplicate, repeat, and obsolete noise from the data stream.

How deduplication works

Express converts each new event notification and metric anomaly into a generic JSON object that describes a specific performance-related event: what happened, when it happened, where it happened, and so on.

The next step is to add each new event into an alert. An event describes one specific event; an alert describes a set of events that all relate to the same issue. For example, an alert might be "High CPU load on server 23." and consist of the following events:

  1. server 23, 12:00: CPU load = 72%

  2. server 23, 12:01: CPU load = 80%

  3. server 23, 12:02: CPU load = 67%

Express adds new events to alerts as follows:

  1. A new JSON event object arrives at the ingestion endpoint.

  2. Express generates a dedupe key for the event based on the source, service, and check fields in the event.

  3. Express compares the new event with each open alert using the dedupe key.

    • If the dedupe key matches an open alert, Express increments the alert's event count field and updates the severity field based on the new event.

    • If the dedupe key does not match any open alert, Express creates a new alert and adds the event to it.

See also Event deduplication: how-to and best practices