Event Enrichment Example: How it Works

This example illustrates how you can define workflows to format your raw events and enrich them with additional data from your environment.

Suppose you have a brewAPM service that observes performance on all nodes for a specific app. The service sends events to Moogsoft that look like this:

'{
    "description": "RT > 500 msec",
    "severity": 5,
    "source": "brewAPM",
    "check": "api",
    "service" : [ "REST" ],  
    "tags": {
        "labels": ["ip=172.31.17.101,hostName=websrv01.us=west.myorg.org, id=125989934839832182"]
    },
}'

Your raw events have the following issues:

  1. The source field describes the service that generated the event, not the host where the event occurred.

  2. The hostname where the event occurred is embedded in the labels tag.

  3. The service value is generic: it describes the type of service but not the service name.

Given this information, you want an event workflow that does the following:

  1. Extracts the hostname from the labels tag,

  2. Copies the hostname to the source field,

  3. Uses the hostname to look up an entry from an external catalog, and finally

  4. Copies the service name from the catalog entry to the service field in the alert.

Defining the workflow

To create a new workflow, choose Data Config > Workflow Engine and then click Add Workflow. This opens the Workflow Editor with an empty workflow.

Every workflow consists of a trigger and one or more actions. The trigger is a filter that defines the events that the workflow will process. Once an event passes the trigger, each action processes the event in sequence. To create the desired workflow, you do the following:

Create the data catalog

To enrich your events, begin by creating one or more catalogs. A catalog is a collection of data from your environment, formatted in a tabular format. The Query Catalog action maps data from a catalog to new events in a workflow.

The simplest way to create a catalog is to define your enrichment data in a CSV and upload it. For every row in the CSV, Moogsoft creates a document in the catalog. See Creating Data Catalogs.

Create the workflow

Choose Data Config > Workflow Engine and then click Add Workflow. A new workflow appears with a single element. Every workflow has a trigger, which defines the events that trigger the workflow.

wf-example-00.png

Define the workflow trigger

The trigger is basically an event filter. The workflow only processes events that pass this filter. You double-click on the trigger and then specify the types of events that you want the workflow to process. You want this workflow to process events from the brewAPM process, so you specify the following filter: source = brewAPM.

Action 1: Update the source field

To add an action, drag it into the workspace and click on it. In this case you add the Extract Substring action, which applies a regex to one event field and copies the result to another.

wf-example-01a.png

You want this action to extract the hostname from the labels tag and copy it to the source field. You extract the hostname using a regex.

wf-extract-string.png

Note

There are three types of input and output field:

  • A base field that is already defined in the Events API schema

  • An existing tag that has been ingested in a previous event

  • A new tag that has not been ingested

Action 2: Map catalog data to the event

After the Extract String action processes an event, the event now has the event FQDN as its source. Now you want to add information about this source from your catalog to your events. You can do this using the Query Catalog action.

wf-example-02a.png

You drag the action into the workspace and configure it as follows.

  • Catalog Name

  • Lookup Field Name

    The action uses this field to query the catalog for the relevant document. You specify a JSON object that says: take the event source (key) and find the document with the matching hostname (value):

    wf-query-catalog-lookup-field.png
  • Apply Field Names

    Once it finds the document, the action copies the document values to the event. Here you define a set of JSON objects that say: take the catalog value (key) and map it to the specified output field in the event. You can also specify a default value if the catalog field is missing or empty.

    wf-query-catalog-apply-field-names.png
wf-example-02a.png

Example event: before and after

The following table shows how this workflow updates and enriches an example event. The workflow extracted the hostname from the labels tag and copied it to the source field. Then it updated the service field and created the location.region tag based on data from the catalog.

Event before

Event after

'{
    "description": "RT > 500 msec",
    "severity": 5,
    "source": "brewAPM",
    "check": "api",
    "service" : [ "REST" ],  
    "tags": {
        "labels": ["ip=172.31.17.101,hostName=websrv01.us=west.myorg.org, id=125989934839832182"]
    },
}'
'{
    "description": "RT > 500 msec",
    "severity": 5,
    "source": "websrv01.us-west.myorg.org",
    "check": "api",
    "service" : [ "custLogin" ],  
    "tags": {
        "labels": ["ip=172.31.17.101,hostName=websrv01.us=west.myorg.org, id=125989934839832182"]
    },
    "location": {
        "region": "West US"
    }
}'