Skip to main content

Understand data catalogs

Define a data catalog

Enrichment provides added flexibility for correlating your alerts into incidents — you can correlate using enrichment data, not just data in the raw ingested events. Enrichment can also make your alerts more informative and easier to troubleshoot. Data catalogs define the information that you can use to enrich your data.

See also Catalogs API.

A catalog is a collection of data from your environment. The simplest way to create a catalog is to generate a CSV from your environment and then upload it to Moogsoft Cloud. The following example illustrates the format.

host, app, aws-region, cluster
ip-172-31-37-159.ec2, music-match, us-west-1, cluster-1
ip-172-23-21-112.ec2, music-maker, us-east-2, cluster-7

Note the following:

  • The first row defines the keys, the following rows define the values to add to individual events

  • After you define a catalog, you must define a Query Catalog action that matches entries in the catalog with new events. In this action, you define the event and catalog fields that the query uses to look up matching documents.

    Lookup fields most commonly specify the source where the event originated. In this example, the host column contains the lookup values. Thus you would specify a key-value pair with the event and catalog field names:

    {
   "Source" : "host"
}

  • Each lookup must be unique. Thus in this example, a catalog cannot have multiple documents with the same host value .

  • A catalog consists of one or more documents. A document is equivalent to a single row in a CSV.

  • A Query Catalog action can specify multiple lookup fields. In this case, all the lookup values must match for a document to match an event. You might want to do this if (for example) your catalog spans multiple regions and you want to enrich your sources differently depending on the region.

  • When you upload the CSV, it deletes and overwrites any previous enrichment entries in the catalog. If you want to add or remove entries from the catalog, add or remove them from the CSV and then upload. You can also use the Catalogs API to add or delete rows in a catalog.

  • When a lookup results in a match between an alert and a document, Moogsoft Cloud maps the other values in the document with fields in the alert. You specify how this mapping occurs in the Query Catalog action.

  • In this example, three columns map directly to fields in the alert schema:

    host

    to

    Source

    service

    to

    Service

    cloud-provider-region

    to

    Location.region

    The fourth column, cluster, does not have an equivalent in the alert schema, so we will map it to a custom tag: tags:cluster.

  • If your organization stores its infrastructure in a CMDB, registry, spreadsheet, or other centralized repository, the simplest workflow is to publish or export the relevant data to the expected CSV format shown above. You can also use the Catalogs API to push your updates programmatically.

  • The catalog CSV file name should not include dollar-sign ($) characters.

Creating a catalog

  1. Define your enrichment data

    To add enrichment data, you specify your data in a .csv file and then upload the file to Moogsoft Cloud.

  2. Upload and review your data catalog

    After you define your enrichment data, you can upload the CSV and add it to an event workflow. Do the following:

    1. Navigate to Correlate & Automate > Data Catalog and click Add Catalog.

    2. Setup tab: Select your CSV file and upload.

    3. Catalog tab: Review the catalog data.

Updating a catalog

The Catalogs API enables you to create, retrieve, update, and delete catalogs. It also enables you to add and remove individual documents (rows) in a catalog.

In this section: