Creating Data Catalogs

This topic describes how to create catalogs with enrichment data from your environment.

Enrichment provides added flexibility for correlating your alerts into incidents — you can correlate using enrichment data, not just data in the raw ingested events. Enrichment can also make your alerts more informative and easier to troubleshoot.

What is a catalog?

A catalog is a collection of data from your environment. The simplest way to create a catalog is to create a CSV of data from your environment and then upload it to Express. The following example illustrates the format.

host, app, aws-region, cluster
ip-172-31-37-159.ec2, music-match, us-west-1, cluster-1
ip-172-23-21-112.ec2, music-maker, us-east-2, cluster-7

Note the following:

  • The first row defines the keys, the following rows define the values.

  • Each row after the first specifies the values to add to an alert.

  • After you define a catalog, you must define a QueryCatalog action that matches entries in the catalog with new events. In this action, you define the event and catalog fields that the query uses to look up matching documents.

    Lookup fields most commonly specify the source where the event originated. In this example, the host column contains the lookup values. Thus you would specify a key-value pair with the event and catalog field names:

    {
       "Source" : "host"
    }
  • Each lookup must be unique. Thus in this example, a catalog cannot have multiple documents with the same host value .

  • A QueryCatalog action can specify multiple lookup fields. In this case, all the lookup values must match for a document to match an event. You might want to do this if (for example) your catalog spans multiple regions and you want to enrich your sources differently depending on the region.

  • When you upload the CSV, it deletes and overwrites any previous enrichment entries in the catalog. If you want to add or remove entries from the catalog, add or remove them from the CSV and then upload. You can also use the Catalog API to add or delete rows in a catalog.

  • When a lookup results in a match between an alert and a document, Express maps the other values in the document with fields in the alert. You specify how this mapping occurs in the QueryCatalog action.

  • In this example, three columns map directly to fields in the alert schema:

    host

    to

    Source

    service

    to

    Service

    cloud-provider-region

    to

    Location.region

    The fourth column, cluster, does not have an equivalent in the alert schema, so we will map it to a custom tag: tags:cluster.

  • If your organization stores its infrastructure in a CMDB, registry, spreadsheet, or other centralized repository, the simplest workflow is to publish or export the relevant data to the expected CSV format shown above. You can also use the Catalog API to push your updates programmatically.

  • The catalog CSV file name should not include dollar-sign ($) characters.

Creating a catalog

Do the following:

Define your enrichment data

To add enrichment data, you specify your data in a .csv file and then upload the file to Express.

Upload and review your data catalog

After you define your enrichment data, you can upload the CSV and add it to an event workflow. Do the following:

  1. Go to the Settings > Data Catalog page and click Add Catalog.

  2. Setup tab: Select your CSV file and upload.

  3. Catalog tab: Review the catalog data.

Once you upload your catalog, you can include it in an event workflow. See Creating Event Workflows.