Events API

The inbound Events API provides an endpoint where you can send events from monitoring tools such as New Relic, Dynatrace, and Datadog. Express can ingest from any external tool that can publish event data in JSON format.

The Events API supports the following workflows:

  • Posting: Set up a simple script that regularly posts events to the REST endpoint.

  • Webhook: Set up the REST Integration as an endpoint for a webhook from an external tool.

To send events of interest to Express, go to Integrations > Inbound Integrations > Events. This page includes a full description of the metric schema and required fields; it also includes auto-generated curl commands with example JSON events.

The event schema is highly flexible and generic, with a small number of required fields, a larger number of optional fields, and a tag field for ingesting custom data not included in the default schema.

For a description of how Express deduplicates events and adds them to alerts, see Event deduplication: how-to and best practices.

Before you begin

Before you set up your integration, note the following:

  • Make sure that the event data you send to Express includes all data fields required by the schema, as well as any additional fields necessary to cluster alerts intelligently and create incidents that meet the needs of your operators.

    If your raw data doesn't include all the data you need, map the missing fields to a value like "incomplete event" to handle these events separately.

  • All timestamps must be specified as UNIX timestamps normalized to Coordinated Universal Time (UTC).

  • Express stores all timestamps in UTC format. The dates and times displayed in the UI are based on your browser's local tim.e

  • Make sure that any subsequent updates to an event are consistently populated.

  • Review Express APIs, which describes the expected request format.

Set up an events integration

To set up an event integration, you configure the external tool to publish to the Events API endpoint of your Express instance. The general workflow is the same regardless of the tool:

  1. Go to the Express web UI > Integrations > Events API and determine the data fields that you want to ingest.

  2. In the external tool, do the following:

    1. Determine the event types that you want to send to Express.

    2. Set up the tool to trigger alerts for the infrastructure issues of interest.

    3. Set up the tool to send events to the Events API endpoint in the required format.

Event field reference

Table 2. Events API field reference

Field

Description

Type

source

REQUIRED

The server, node, or other entity that generated the event. This can be any unique, human-readable name such as a hostname, fully qualified domain name, or URL. 

In some cases a process or application might trigger an event that has no node identification. In this case, specify a unique identifier of the generating instance such as a database name, container name, or cluster node name.

string

description

REQUIRED

A human-readable description of the event. A good description summarizes the exact conditions and thresholds that triggered the event. Examples include:

  • "en0 bandwidth util > 50%"

  • "myApp response time > 1sec"

  • "myWebService 4xx https responses/sec > 20%"

  • "myService unavailable"

When Express creates or updates an alert with a new event, this description becomes the alert description.

string

severity

REQUIRED

The relative severity of the event. When you define severity levels, consider the significance of an event source if possible. A low-priority source might trigger "high-severity" events that might actually be lower-priority in the context of all your event sources.

Specify an integer or string as follows:

  • 0 — "Clear"

  • 1 — "Unknown" — Avoid this if possible, unless you need to send the specific event and is no way to determine its severity.

  • 2 — "Warning"

  • 3 — "Minor"

  • 4 — "Major"

  • 5 — "Critical"

When Express creates or updates an alert with a new event, this severity becomes the alert severity. If the event severity is Clear, the alert is closed to new events and later events with the same dedupe key get deduplicated into new alerts.

Integer or string

service

REQUIRED

One or more applications or services that either generated the event or are associated with it. Express uses the "source", "service", and "check" fields to filter out duplicates and aggregate similar events into alerts.

You can use this tag to specify the level of event aggregation and deduplication in your alerts. For more general alerts, use general names such as "docker" or "ticket-reservation-app". For more targeted alerts, use a microservice or other specific name.

JSON array

check

REQUIRED

The performance, health, or other check that triggered the event. Examples include "cpu load", "mongoDB query", and "web server response code".

If two events involve different problems on the same source and service, they should belong to different alerts. For this reason, the check field should be fairly specific. Suppose two events describe database errors on the same source : one is a replication error and another is a query error. If check = “database” for both events, they will get deduplicated into the same alert. The better choice is to use “database-replication” for the first check and “database-query” for the second.

See Event deduplication: how-to and best practices.

string

class

The high-level classification of the event. Examples include Application, Storage, Network, Operating System, Middleware, Hardware, Virtualization, Serverless, Database, Cloud, and API.

This field determines the class field in the resulting alert. Alert classes can be useful for correlating alerts into incidents. See Correlate Alerts into Incidents.

string

time

Event time in UTC format. You can specify the time in either seconds or milliseconds. If the payload does not include a time, Express uses the event arrival time.

integer

manager

A general identifier of the event generator or intermediary, such as AppDynamics, NewRelic, Nagios, Docker, RedHat, Mongo, or CloudWatchEC2.

string

manager id

ID for the external manager or tool that sent the alert. This field can be useful with external monitoring tools that support workflow automation: when a user closes an alert in Express, the corresponding alert can then be closed in the sending manager using this ID.

string

location

A set of key-value pairs that specify the physical location where the event occurred.

It is good practice to use the same set of location keys for all sources. You can specify the geographic location using the street, suite, city, state_prov, and other keys. You can also specify the physical location of the event source using the data_center, hall, aisle, rack, and u_position keys.

JSON object

tags

If desired, you can include additional information about an event using the tags key. You can specify data as an array or as a series of key-value pairs.

JSON object

dedupe_key

Do not include this field in your event payload unless you want to customize the default deduplication behavior. See Event deduplication: how-to and best practices

string



Events API specification