Create incidents from uncorrelated alerts using correlation group settings
Some alerts may not be included in incidents by correlation because correlation definition settings prevented it. It may be preferable for these alerts to be "left over" and not included in incidents, depending on your organizational requirements. However, if you would prefer every alert to be included in an incident, you can use correlation group settings to act as a "catch all" to accomplish this goal.
The setting can be enabled two ways:
Via API using the
catch_all = truesetting.In the user interface by selecting Alerts can match one definition for a correlation group, and then selecting the option option Create an incident for each uncorrelated alert. Refer to Create a correlation group for specific configuration steps.
When the setting is enabled, every alert for that correlation group which is not correlated into an incident at the end of the correlation time window is placed into a single-alert incident.
This includes alerts that:
Do not meet the requirements of the correlation definition for similarity
Do not match the scope filter for the correlation definition
Are in candidate incidents which do not meet the configured Incident Creation Threshold during the correlation time window
Note that in this scenario, each alert still creates a single-alert incident, even though multiple alerts are similar.
Use the Create an incident for each uncorrelated alert setting to prevent situations where some alerts do not match any of your correlation definitions. The setting ensures that no alerts are lost, which is useful when testing different correlation definitions which are potentially too restrictive, and for other purposes (see the Examples below). When the setting is enabled, all alerts become incidents, either through clustering by correlation or individually when the correlation time window ends.
Note
When correlation definitions are configured to intentionally exclude many alerts, using the setting can create numerous irrelevant incidents. If this behavior is undesirable, then select the Do not create incidents for uncorrelated alerts setting for the correlation group instead.
Examples
The following correlation examples show how the correlation group setting Create an incident for each uncorrelated alert can affect your correlation results.
This example shows the potential impact when the setting is enabled vs. disabled for the same alerts in one correlation group.
The correlation group includes this correlation definition:
Minimum Alerts Count = 3
Scope Filter is "manager = abc"
Correlation Time Window is set to 10 minutes
Two alerts arrive:
Alert #1 | Alert #2 |
|---|---|
... "manager": "abc" ... | ... "manager": "abc" ... |
The two alerts meet the requirements of the correlation definition scope filter and form a candidate incident. However, after 10 minutes, the correlation time window expires. No additional alerts matching the scope filter were added. The candidate incident only includes two alerts, which is below the Minimum Alerts Count requirement of three alerts.
When "Create an incident for each uncorrelated alert" is enabled: Each alert becomes an incident containing one alert.
When "Create an incident for each uncorrelated alert" is disabled: The two alerts do not become incidents.
A scenario where it makes sense to use multiple correlation groups is when you want the same incidents to get correlated in two different ways.
This could happen when different troubleshooting teams need to see the same incidents. In the following scenario, alerts are correlated where service = database. The resulting incidents should be analyzed by the Database team; however, an AWS team also needs their own database incidents for review.
There are two correlation groups, Database and AWS, reflecting this use case. Both correlation groups have the following correlation definition scope filter:
service = database
Because both correlation groups have a correlation definition where the scope filter is service = database, whenever alerts are received where service = database, both correlation groups will process them and the result is two copies of the incidents: one produced by correlation within the Database correlation group, and the other copy produced by correlation within the AWS correlation group.
At this point, the teams can use incident views to only display incidents created by the appropriate correlation group using column filtering according to the name of the correlation definition. Alternatively, workflows could go a step further and assign the teams to their relevant incidents.
Another scenario where multiple correlation groups make sense is where you want alerts of a certain type to always form incidents, but not all alerts.
In this case, you could combine a correlation definition scope filter to match the alerts you want with the correlation group Create an incident for each uncorrelated alert setting in one group, while leaving the setting disabled for the other correlation group.
Correlation Group #1 settings | Correlation Group #2 settings |
|---|---|
|
|
Correlation Definition #1 scope filter | Correlation Definition #2 scope filter |
|---|---|
source = 192.168.10.10 | source != 192.168.10.10 |
When alerts matching the scope filter source = 192.168.10.10 arrive, they will only be correlated into incidents by Correlation Group #1, due to the scope filter in Correlation Group #2 eliminating those alerts. Additionally, every alert received from the source 192.168.10.10 will either get correlated into an incident, or get placed into a single-alert incident due to the correlation group settings.