Skip to main content

Data Ingestion

You can configure how data fields are mapped and how events are deduplicated for monitoring integrations in Moogsoft Enterprise.

Benefits of these data ingestion features include:

  • Data Mapping enables Moogsoft Enterprise to identify and organize alerts from integrations.

  • Deduplicating events from integrations into alerts reduces noise.

The configuration steps below can only be taken after the integration has been installed and is running. The tabs are inactive prior to the integration being installed.

Data Mapping

After Moogsoft Enterprise receives the payload of an incoming event from the integration, you can map the data fields to the corresponding alert fields in Moogsoft Enterprise.

You can customize mappings on the Data Mapping tab under each integration. Note that the follow restrictions apply to mapping rules:

The Data Mapping tab contains three sections:

  1. Input displays the incoming payload of the first event sent to Moogsoft Enterprise by the integration after tokenization. The Payload View contains the following information and controls:

    29960178.png

    A. Source fields - integration data fields.

    B. Source field values - values of the integration data fields.

    C. Refresh - clears the window and populates with the payload of the next event from the integration.

    D. Expand - click and drag down to expand the Payload View.

    You can edit, copy and paste the payload text as required.

  2. Transform allows you to transform and map the data fields of events from the integration with the appropriate alert fields in Moogsoft Enterprise.

    Select any field from the list to edit it and select the Moogsoft Enterprise field it maps to. See Alert and Event Field Reference for descriptions of the alert fields in Moogsoft Enterprise. You can also add custom fields and a mapping rule explaining the format the field requires. See the "Mapping Rule" section below for more information.

  3. Output displays a preview of how the integration event appears as an alert in Moogsoft Enterprise. This changes dynamically as you change the data field mappings and the Payload View.

Alert Noise Reduction

Moogsoft Enterprise deduplicates events into alerts in order to reduce noise. You can configure a signature to ensure events from a single integration or from multiple integrations of different types are deduplicated into alerts together.

To edit the signature, go to the Signature editor and select the fields you want to be included. Alternatively, click 'Use Recommended Fields' to use fields recommended by Moogsoft Enterprise.

Fields recommended for use in a signature included: source/host, event type/class, manager/agent, unique ID, error code or impacted entities.

After you configure a signature, compare the Alerts to see if Moogsoft Enterprise deduplicated the events as you would expect. If not, then revise and refine the signature.

See Signature for more information.

Mapping Rules

The Mapping Rules field in the Transform section allows you to describe the output you require from a given payload. Mapping also allows path traversing, including objects and arrays by index.

There are two types of mapping you can use. Basic mapping allows you to select a direct mapping from your payload, while Advanced allows you to configure the field's value as a string.

The following applies to the Mapping Rules field:

  • You can enter static text, alphanumeric characters and underscores. This allows you to select an element from an ordered array within your payload.

  • If you are using Advanced mapping and a key in your payload uses special characters, . or square brackets, enclose your key within braces to escape the special characters' behavior.

The following examples demonstrate the different ways that you can provide a key and resulting effect.

Example 1

Your key provides special character behavior:

$location.data_center.zip_code

Example 2

Your key escapes all special character behavior as you have enclosed it within braces:

${location.data_center.zip_code}

Example 3

Your key is partially enclosed, and so special character behavior only applies outside of the braces:

${location.data_center}zip_code

Example 4

Your key references the first element in an array:

$location.data_centers[1].zip_code

Therefore, the following input maps to "12345":

"location":{"data_centers": [ {"zip_code": 90210, "name": "datacenter 1"}, {"zip_code": 12345, "name": "datacenter 2"}} ]}

Your key does not reference an array element as you have enclosed it within braces, thus escaping all special character behavior:

${location.data_center.zip_code[0]}

Therefore, the following input maps to "90210":

{"location.data_center.zip_code[0]" : 90210}
In this section: