Data Enrichment

In some cases, you might want to enrich your alerts with custom data and thereby make the resulting alerts more usable. Alert enrichment supports the following use cases:

  • You want to fine-tune how Express clusters your alerts into incidents.

    You can enrich sources with information about their associated clusters, apps, services, teams, locations, and so on. You can leverage data from a CMDB or other central repository to define the relationships between different nodes. Once you define these relationships in your enrichment data, you can define a simple, smart correlation pattern to cluster your alerts.

  • You want to make your alerts more informative and readable.

    In some cases, your raw events and metrics might not include all the information necessary for an operator to investigate and troubleshoot an Incident.

Before you begin

Before you set up an enrichment, you need to identify the data you want to add. Go to the Alerts table and examine the data fields in your alerts. What contextual information about the source nodes would you like to add? The following alert fields are especially useful for customizing your correlations:

  • service: Useful for correlating alerts based on the apps, services, or microservices associated with specific source nodes.

  • location: Useful for correlating alerts based on the geographic, virtual, or logical locations of specific source nodes.

Your source nodes might have other attributes that you'd like to use for correlation and reporting. You can enrich your data with pretty much any data you and your users find useful, as long as you can map the enrichment data to specific alert fields.

Note

Alerts have a tags field that you can use to add custom information that does not correspond to any of the other defined alert fields.

Workflow

Do the following:

Define your enrichment data

To add enrichment data, you specify your data in a .csv file and then upload the file to Express. The following example illustrates the format. The first line defines the keys, the following lines define the values:

host, service, aws-region, cluster
ip-172-31-37-159.ec2, music-match, us-west-1, cluster-1
ip-172-23-21-112.ec2, music-match, us-east-2, cluster-7

Note the following:

  • One column contains lookup values that Express uses to map the enrichment data in a specific row to an alert that matches the lookup. Lookup fields most commonly specify the source where the alert originated. In this example, the host column contains the lookup values.

  • Each lookup value must be unique. Thus, in this example, you cannot have multiple rows with the same IP.

  • When you upload the CSV, it deletes and overwrites any previous enrichment entries in the database. If you want to add or remove entries from the database, add or remove them from the CSV and then upload.

  • When a lookup results in a match between a row and an alert, Express maps the other values in that row with fields in the alert. You specify how this mapping occurs in the following step.

  • In this example, three columns map directly to fields in the alert schema:

    host

    to

    Source

    service

    to

    Service

    was-region

    to

    Location.region

    The fourth column, cluster, does not have an equivalent in the alert schema, so we will map it to the tags field.

  • If your organization stores its infrastructure in a CMDB, registry, spreadsheet, or other centralized repository, the simplest workflow is to publish or export the relevant data to the expected CSV format shown above.

Upload and review your enrichment data

After you define your enrichment data, you can upload the CSV and see the enrichments on relevant alerts. Do the following:

  1. Go to the web UI > Settings > Enrichment page.

  2. Setup tab: Select your CSV file and upload.

  3. Catalog tab: Review the catalog data.

Map your enrichment fields to alert fields

After you upload your CSV file, you need to map your enrichment fields — the column headings in the first line — with the corresponding alert fields. After you upload your CSV, go to the Data Enrichment page > Mapping tab and specify the following:

  1. Select Lookup Field

    Express uses these fields match a row in the CSV to specific alerts. For example, the CSV above has a host column with a unique identifier for every row. If your alerts use IPs in their source fields, then you would specify:

    • Enrichment Lookup Field = host

    • Express Alert Field to Match = Source

  2. Enrichment Mapping

    When Express matches a row with an alert, it then maps the other values in that row to fields in the alert. In this table, you map the enrichment fields to the corresponding alert fields. Note the following:

    • You must create a separate row for each field-to-field mapping, even if the enrichment and alert names are identical.

    • If the enrichment data does not correspond to a specific alert field, map it to the Tags field. In this example, cluster maps to Tags because it does not obviously correspond to any defined alert field. In an enriched alert, the tags field includes a key-value pair such as cluster : cluster-7 .