# Enrich Events with Additional Data

In most cases, you will want to enrich your raw events with additional data after they get ingested. Enrichment has the following benefits:

• You want to fine-tune how Express clusters your alerts into incidents.

You can enrich sources with information about their associated clusters, apps, services, teams, locations, and so on. You can leverage data from a CMDB or other central repository to define the relationships between different nodes. Once you define these relationships in your enrichment data, you can define a simple, smart correlation pattern to cluster your alerts.

In some cases, your raw events and metrics might not include all the information necessary for an operator to investigate and troubleshoot an Incident.

• You want to normalize events that come from different sources and have different formats.

For example, one event stream uses IPs as the source while another stream uses domain names. You can use enrichment to ensure that all events are formatted consistently. This can make deduplication and correlation much simpler.

## Event Workflows

### Note

This workflow functionality was released on July 28, 2020 and replaces the previous functionality and interface for enriching events.

You enrich your events using event workflows. An event workflow is a user-defined, fully-automated sequence of actions applied to each new event:

1. A new event arrives at the workflow engine, which triggers the workflow.

2. Each workflow has an an initial trigger, which is an event filter that specifies the events that the workflow will process.

• If the event does not pass the trigger filter, the workflow exits.

• If the workflow passes the filter, the workflow proceeds to the next step.

3. The event passes through a series of actions that enhance and update the data in the event.

A workflow can enrich events with data from external catalogs. You can also create workflows that update fields in an event using data from other fields in the same event.

4. Once the event passes through all actions in all relevant workflows, the data pipeline does the following:

1. Deduplicates the event into an alert.

2. Sends the alert to the correlation engine.

The Workflow Engine UI (Settings > Workflow Engine) provides a simple drag-and-drop interface for creating workflows.

## Event Enrichment Example: How it Works

This example illustrates how you can define workflows to format your raw events and enrich them with additional data from your environment.

Suppose you have a brewAPM service that observes performance on all nodes for a specific app. The service sends events to Express that look like this:

'{
"description": "RT > 500 msec",
"severity": 5,
"source": "brewAPM",
"check": "api",
"service" : [ "REST" ],
"tags": {
"labels": "ip=172.31.17.101,
hostName=websrv01.us-west.myorg.org,
id=125989934839832182"
},
}'

Your raw events have the following issues:

1. The source field describes the service that generated the event, not the host where the event occurred.

2. The hostname where the event occurred is embedded in the labels tag.

3. The service value is generic: it describes the type of service but not the service name.

Given this, you want an event workflow that does the following:

1. Extracts the hostname from the labels tag,

2. Copies the hostname to the source field,

3. Uses the hostname to look up an entry from an external catalog, and finally

4. Copies the service name from the catalog entry to the service field in the alert.

### Defining the workflow

To create a new workflow, choose Settings > Workflow Engine and then click Add Workflow. This opens the Workflow Editor with an empty workflow.

Every workflow consists of a trigger and one or more actions. The trigger is a filter that defines the events that the workflow will process. Once an event passes the trigger, each action processes the event in sequence. To create the desired workflow, you do the following:

#### Create the data catalog

If you want to enrich your events, the first step is to create one or more catalogs. A catalog is a collection of data from your environment, formatted in a tabular format. The QueryCatalog action maps data from a catalog to new events in a workflow. See Creating Data Catalogs.

#### Create the workflow

Choose Settings > Workflow Engine and click Add Workflow. A new workflow appears with a single element. Every workflow has a trigger, which defines the events that trigger the workflow.

#### Define the workflow trigger

The trigger is basically an event filter. The workflow only processes events that pass this filter. You double-click on the trigger and then specify the types of events that you want the workflow to process. You want this workflow to process events from the brewAPM process, so you specify the following filter: source = brewAPM.

#### Action 1: Update the source field

To add an action, drag it into the workspace and click on it. In this case you add the Extract Substring action, which applies a regex to one event field and copies the result to another.

You want this action to extract the hostname from the labels tag and copy it to the source field. You extract the hostname using a regex.

### Note

There are three types of input and output field:

• A base field that is already defined in the Events API schema

• An existing tag that has been ingested in a previous event

• A new tag that has not been ingested

#### Action 2: Map catalog data to the event

After the Extract String action processes an event, the event now has the event FQDN as its source. Now you want to add information about this source from your catalog to your events. You can do this using the Query Catalog action.

You drag the action into the workspace and configure it as follows.

• Catalog Name

• Lookup Field Name

The action uses this field to query the catalog for the relevant document. You specify a JSON object that says: take the event source (key) and find the document with the matching hostname (value):

• Apply Field Names

Once it finds the document, it copies the document values to the event. Here you define a set of JSON objects that say: take the catalog value (key) and map it to the specified output field in the event. You can also specify a default value if the catalog field is missing or empty.

### Example event: before and after

The following table shows how this workflow updates and enriches an example event. The workflow extracted the hostname from the labels tag and copied it to the source field. Then it updated the service field and created the location.region tag based on data from the catalog.

Event before

Event after

'{
"description": "RT > 500 msec",
"severity": 5,
"source": "brewAPM",
"check": "api",
"service" : [ "REST" ],
"tags": {
"labels": "ip=172.31.17.101,
hostName=websrv01.us-west.myorg.org,
id=125989934839832182"
},
}'

-'{
"description": "RT > 500 msec",
"severity": 5,
"source": "websrv01.us-west.myorg.org",
"check": "api",