Configure De-duplication Key Using Signature
Signature is the value Moogsoft Onprem uses to deduplicate source events with the same context. Moogsoft Onprem assigns a signature value to each event it ingests, constructed from a subset of the event fields. If Moogsoft Onprem finds an event signature to be unique, it creates a new alert. Otherwise, it adds the event to an existing alert with a matching signature.
After Moogsoft Onprem deduplicates events into alerts, you can still access the individual event information from the alert timeline.
Most LAMs and integrations include a default signature mapping. If you are building a custom data ingestion or tweaking the default, you can use the fields of your choice to define the signature.
Why is Signature Important?
The composition of the signature is very important because it has a significant impact on what you see in the alert list.
The first time Moogsoft Onprem ingests an event with a specific signature it creates a unique alert. If it ingests another event with a matching signature it deduplicates it into the same alert. Moogsoft Onprem updates the alert timestamp and increments the alert count. This is very useful in reducing the number of alerts in the system.
Default Signatures
To view and edit default signatures for integrations configured in the Moogsoft Onprem UI:
Go to Integrations and click the name of your installed integration in the left panel.
Click the Alert Noise Reduction tab and scroll down to the Signature Editor section.
This section displays the fields that can be used to create a baseline signature for this integration. You can edit the signature here to select different or additional fields. Click Use Recommended Fields to restore the recommended default.
You can view and edit default LAM signatures in LAM configuration files. For example, the SevOne configuration file $MOOGSOFT_HOME/config/sevone_lam.conf contains the following signature definition in the mapping rules:
{ name: "signature", rule: "$origin::$deviceId::$objectId" }Signature Composition
A signature is made up of a subset of event properties. Different types of events require different signatures.
In general, fields to consider using in the signature are:
Source, such as hostname
Event type or class
Static unique IDs
Error code
Impacted entities
Do not include fields in the signature that may change between events with the same context. For example:
Timestamp
State changes such as up or down
Event count
Variable unique IDs
Severity
Descriptions with changing content such as metrics
For example, every event has a different timestamp so including it in the signature effectively disables deduplication.
A perfect signature contains just enough information to identify the context of an event.
Signature Length and Concatenation
There is no restriction imposed on the length of signatures in raw events. Signatures longer than 746 characters are hashed at the alert level. This improves the manageability of signatures in the database but does not affect deduplication. The hashed signature length is 40 characters.
If you edit the signature in a LAM configuration file, concatenate multiple fields with two colons "::" to prevent misleading results. For example, if you concatenate source "Node A" and unique ID "1234" as "NodeA1234" this could potentially also match Node A1 and unique ID 234.
Example
The Email LAM uses the following default signature mapping:
$hostname::$subject
The Email LAM retrieves the following email messages in this order and sends them to Moogsoft Onprem:
Event 1:
ip-172-22-97-140.ec2.internal::TDM 18 Remote Loss of Signal
Event 2:
ip-172-22-97-140.ec2.internal::TDM 18 Remote Loss of Signal
Event 3:
ip-172-22-99-144.ec3.internal::TDM 18 Remote Loss of Signal
Events 1 and 2 have an identical signature. Moogsoft Onprem creates an alert for event 1 and deduplicates event 2 into the same alert. It creates a separate alert for event 3 which has the same subject but a different hostname.
How Do I Select Appropriate Signature Values?
Signature is the context to be shared by events that belong to the same alert. For example, the shared context might be the same fault, on the same hardware or software asset, in the same network location.
Consider an example of events where only the timestamp or severity varies from event to event. All these events should have the same signature and be deduplicated into a single alert.
If in another case, events involve different equipment or different problems, they should have different signatures and become separate alerts. This way database replication failures with differing severities will have the same signature value, but a database replication failure and a database query failure will have different signatures.
Data Granularity and Deduplication
