Customize event deduplication

For many integrations you can customize deduplication behavior in the integration settings (see, for example, Create Your Own Integration). You can also explicitly define the deduplication_key field in the raw events you send to Moogsoft. The following use cases provide some examples of when you might want to customize your deduplication:

  • You want to generate alerts for a service that runs in multiple containers. You’re primarily concerned about the overall performance of the service rather than specifics of the container in which the service runs. In this case, you can specify a deduplication key such as my-service::user-response-time.

  • You want to deduplicate based on information that is not included in the default fields. For example, you might want to deduplicate all events that have a specific error code. In this case, you could define a deduplication key such as my-source::my-service::my-check::err3059.

Do’s and don’ts for custom deduplication

A deduplication key is made up of a subset of event properties. Different types of events require different keys. A perfect deduplication key contains just enough information to identify the context of an event.

In most cases a combination of the following values tends to work well:

  • Source, such as hostname

  • Event type or class

  • Static unique IDs

  • Error codes

  • Impacted entities

Do NOT include fields that might change between events with the same context. For example:

  • Timestamp --  Every event has a different timestamp. If the deduplication key includes the timestamp, deduplication is effectively disabled.

  • State changes such as up or down

  • Event count

  • Variable unique IDs

  • Severity

  • Descriptions with changing content such as metrics

When you define a custom deduplication key, concatenate multiple fields with two colons "::" to prevent misleading results. For example, if you concatenate source "Node A" and unique ID "1234" as "NodeA1234" this could potentially also match Node A1 and unique ID 234.

Best practices

If two events involve different problems on the same source and service, they should belong to different alerts. For this reason, the check field should be fairly specific. Suppose two events describe database errors on the same source : one is a replication error and another is a query error. If check = “database” for both events, they will get deduplicated into the same alert. The better choice is to use “database-replication” for the first check and “database-query” for the second.