Define a custom integration

Create your own integration (CYOI) lets you use a webhook configured in an external service to send events or metrics to Moogsoft.

NOTE: Refer to Ingestion Services before you start. Step-by-step instructions are available for configuring several external systems with CYOI.

The Data Config > Ingestion Services > Create your own Integration page lists the custom integrations defined in your Moogsoft instance. Each integration specifies the following:

  • A user-defined endpoint for ingesting events or metrics from a third-party service

  • A set of mappings from the third-party metric/event fields to their Moogsoft equivalents.

  • A deduplication key that defines the fields used to identify duplicate events.

Before you begin

Before you create a custom API, do the following:

  1. Verify that you can create a webhook in the third-party service.

  2. Verify that the third-party service can post REST requests with JSON payloads over port 443.

  3. Review the JSON payloads generated by the external service and compare them with the Moogsoft Events API or Metrics API schema.

Procedure

Follow these general steps to create an custom integration:

Create the custom endpoint in Moogsoft

  1. Choose Data Config > Integrations > Create your own Integration.

  2. In the Create your Integration window, click Add New Integration.

  3. Enter the endpoint name, description, and data type.

    This information displays on the Create your own integration page to help you identify your integration.

    Note

    • Do not try to define one custom API for multiple external systems. Each custom API should be based on one and only one service.

    • Each custom API ingests events only or metrics only. To ingest both metrics and events from one service, you need to define two custom APIs for that service.

  4. Click Save.

Configure the external service to send data to the custom endpoint

After you create the custom endpoint, the next step is to send data from your external tool to the endpoint you just created. Set up your external service to send data based on the following requirements.

Port

Send data over TCP port 443.

URL

Go to Data Config > Integrations > Create your own Integration and select the endpoint you just created. You can copy and paste the URL and the API Key directly from the UI.

API Authentication

Moogsoft supports basic and bearer token authentication for programmatic access to our public APIs.

Basic authentication

Embed the username and API key in the URL using the format username:api_key@domain_name. For example:

curl -k "https://<my-username>:<my-api-key>@api.moogsoft.ai/v1/integrations/metrics"\
    -H "Content-Type: application/json"
    -d '{ "metric": "your-metric", "data": 10, "source": "www.your-source.com", "key": "dev", "tags": { "key": "value" }, "utc_offset": "GMT-07:00"}'

Bearer token authentication

Include the API key as a header in the body of the HTTP request. For example:

curl -k "https://api.moogsoft.ai/v1/integrations/metrics"\
    -H "Content-Type: application/json"
    -H 'apiKey: <my-api-key>'
    -d '{ "metric": "your-metric", "data": 10, "source": "www.your-source.com", "key": "dev", "tags": { "key": "value" }, "utc_offset": "GMT-07:00"}'

Payload

Your payloads should include all relevant data that you want to include in your alerts and incidents. We recommend the following.

  • Review the Events API or Metrics API schema, depending on the type of endpoint you are defining.

  • Identify the required fields in the Moogsoft schema, such as description, severity, and service for events and metric, data, and source for metrics.

  • Identify the fields in your external schema that correspond to the Moogsoft schema.

  • Define the payload to include all the data you want to observe in your Moogsoft alerts and incidents. You will map the external fields to Moogsoft fields in the following step.

  • In some cases you might need to send different payloads to include all the data you want to observe in Moogsoft. This might happen if (for example) a service generates different payloads based on the event or metric type.

  • The payload can be in any valid JSON format.

Enable batch processing

Batch processing lets you define a path to events in your payload when events are nested under one or more levels. If your events are at one level, then it is not necessary to use this option. For a complete discussion of batch processing, see Use batch processing with custom integrations.

Map third-party data to the Moogsoft schema

After you set up your external service to send data to the endpoint, the next step is to map your external data fields to their equivalents in the Moogsoft schema.

  1. In Moogsoft, go to Data Config > Ingestion Services > Create your own Integration and select the endpoint you created.

  2. Go to 2 Map Your Data.

  3. Select a payload from the table.

  4. Map each source field in the payload to its corresponding Moogsoft target field.

    • To add additional mappings, click Add a Mapping.

    • To remove an unneeded mapping, click the trashcan to delete the corresponding row.

    • If one payload does not include all the source fields you need to map, select another payload in the list that does. You can select different payloads in sequence and map all your source fields in one operation.

    • If there is no source field that corresponds to a required Moogsoft target field, click inside Source Field and scroll to the end of the list of source fields. Type a placeholder value for Default Text. You can then create an event workflow to populate this field.

    • The Moogsoft events schema requires a severity field. Click Map Values to map the severity values in your source data to their target equivalents.

    NOTE: Moogsoft automatically maps source fields to target fields with the same name.

Mapping multiple source fields to the same target field

You might want to do this when the relevant data for a target field is in multiple source fields. The mapping behavior is different for target fields that are simple strings, such as source, and target fields that are lists, such as service.

Mapping string fields

For target fields that are simple strings, such as source and class, the mapper uses an if-then-else mechanism. Suppose your payloads encode the event source as follows:

Given this, the mapper populates the target field source based on the order of mappings in the table:

  • An fqdn field for application events

  • An ip field for network events

  • All events include a labels tag, which includes the hostname and a lot of other information about the event.

In the third case, the source field is still unmapped but the Moogsoft event still has the source name: it is embedded in the tags.labels string.

In this case, you can map the fields as follows.

  1. Event has an fqdn field => map fqdn to source

  2. Event has no fqdn field but it does has an ip field => map ip to source

  3. Event has no fqdn or ipe field => map string "TBD" to source. You can then create an automated workflow that extracts the hostname from the labels tag and maps it to the source target field.

Note that the mapping behavior is determined by the order in which these mappings are defined in the mapping table:

map-mult-source-fields-to-source.png

Mapping list fields

Suppose your payload has the following fields:

  • An app field that defines the user-facing name for an app

  • A microservice field that defines the back-end service that supports the app

In this case, you can map both fields to the service target field. Because the target field is a list, it gets populated with both names: "service" : ["myApp", myMicroService"]

Mapping types

In addition to basic mapping, you can use transformations to customize the way source field data is mapped to target fields.

All Source to Target mappings are basic mappings unless you click the Mapping type selector (arrow) between source and target fields and select a different type from the popup:

transformation_menu.png

See Use mapping types in custom integrations for a complete list of mapping types and instructions explaining how to use them.

Test your deduplication key

After you have completed your mappings, scroll down to the Set Your Deduplication Key section. While it is not required, testing your deduplication key before enabling your integration is a good idea.

To reduce noise, Moogsoft uses a deduplication key to group similar events into alerts. This key is an aggregation of relevant event fields. You can preview the current deduplication and edit the deduplication key as needed. 

By default, Moogsoft uses the Source, Class, Service, and Check fields to identify events. If you do not map all of these fields, then only the fields in this default set that you have mapped are included in the deduplication key.

NOTE: You can rearrange the order of the fields in the deduplication key. This does not affect deduplication, but changing the order may make the key easier to read the key when it displays in alerts.

  • To test your key, click Test Deduplication Key.

If the key does not pass, make any necessary changes by adding or removing fields and test it again.

NOTE: Customizing deduplication is an advanced feature. Use the recommended fields unless the default deduplication does not meet your needs. See Deduplicate events to reduce noise .

Save and provision your integration

After successfully testing your deduplication key, you can click Save to provision your integration and allow data to flow into Moogsoft from your external service.

NOTE: You can click Save at any point in the creation process to save your work and come back to it later. Click Edit to make changes.