/enrichment

A Graze API endpoint that allows you to add and delete records from the enrichment data store. To use the Enrichment API, you must install Moogsoft Add-ons v2.6 or later and set up the Enrichment API Integration.

After you load data into the enrichment data store, you can add the data to an alert's custom_info object under the enrichment key using the Enrichment Workflow Engine getEnrichment function.

For an tutorial on how to use the Enrichment API, see Enrich Alerts Using the Enrichment API.

Back to Graze API EndPoint Reference.

Request arguments

The Enrichment API sends POST requests to the enrichment integration endpoint https://server/graze/v1/integrations/enrichment using a JSON payload format:

{
  "action": <post|delete>,
  "data": [ <data> ],
  "age": <age>
}

The data parameter holds an array of objects with keys. The keys are as follows:

attribute: A string providing a namespace for the data being stored or retrieved. Example: “source”
value: A string the identifies the item being stored or retrieved within the namespace of the attribute. Example: “host1”
enrichment: An object containing the data to be stored against the attribute and value.

The age parameter is optional and only affects delete actions, as described below.

Action: post

The post action is used to upload data to the Enrichment API database moog_enrichment and has the following payload format:

Name	Type	Required	Description
action	String	Yes	The string "post".
data	JSON object	Yes	An array of data objects.

...where a single data object has the following fields:

Name	Type	Required	Description
attribute	String	Yes	The namespace, ex. "source"
value	String	Yes	The key within the `attribute` namespace, ex. "SFlinux101"
enrichment	JSON object	Yes	The JSON data to store against the attribute and value, ex. { "location":"1265 Battery St., San Francisco, CA", "support_group":"SF NOC" }

Name

Type

Required

Description

attribute

String

Yes

The namespace, ex. "source"

value

String

Yes

The key within the attribute namespace, ex. "SFlinux101"

enrichment

JSON object

Yes

The JSON data to store against the attribute and value, ex.

{
  "location":"1265 Battery St., San Francisco, CA",
  "support_group":"SF NOC"
}

An example payload:

{
  "action":"post",
  "data":[
    {
      "attribute":"source",
      "value":"SFlinux101",
      "enrichment": {
        "location":"1265 Battery St., San Francisco, CA",
        "support_group":"SF NOC"
      }
    },
    {
      "attribute":"source",
      "value":"DENlinux102",
      "enrichment": {
          "location":"1700 Lincoln Street, Denver, CO",
          "support_group":"DENVER NOC"
      }          
    }
  ]
}

Action: delete

The delete action is used to delete records from the Enrichment API database moog_enrichment and has the following payload format:

Name	Type	Required	Description
action	String	Yes	The string "delete".
data	JSON object	Yes	An array of query objects.
age	Number	No	Optional age in second to limit removal to records above the specified age.

...where a single query object has the following fields:

Name	Type	Required	Description
attribute	String	Yes	The namespace, ex. "source". Also accepts the wildcard “*” to match all `attribute` name spaces.
value	String	Yes	The key within the attribute namespace, ex. “SFlinux101”. Also accepts wildcards: `searchString` - matches values ending in “searchString” for given attribute. `searchString` - matches values beginning with “searchString” for given attribute. `searchString` - matches values containing “searchString” for given attribute.

Name

Type

Required

Description

attribute

String

Yes

The namespace, ex. "source".

Also accepts the wildcard “*” to match all attribute name spaces.

value

String

Yes

The key within the attribute namespace, ex. “SFlinux101”.

Also accepts wildcards:

*searchString - matches values ending in “searchString” for given attribute.
searchString* - matches values beginning with “searchString” for given attribute.
*searchString* - matches values containing “searchString” for given attribute.

An example payload:

{
  "action":"delete",
  "data":[
    {
      "attribute":"source",
      "value":"SFlinux*"
    },
    {
      "attribute":"source",
      "value":"DENlinux102"         
    }
  ]
}

Age scoped deletion

Aged scoped deletion is supported when using the latest version of the Enrichment API database schema. This feature allows you to specify an age in seconds, so that only matched records that were created or updated more than the specified number of seconds ago will be removed.

Below is an example that will remove all records with attribute “source” and value beginning with "SFlinux” that are more than 3600 seconds old:

{
  "action":"delete",
  "age": 3600,
  "data":[
    {
      "attribute":"source",
      "value":"SFlinux*"
    }
  ]
}

Utility Scripts

This section contains details regarding the two utility scripts bundled in moog_enrichment_utils.zip, which include new options and features.

If moog_enrichment_utils.zip is unpacked under $MOOGSOFT_HOME/contrib, then the following directory structure would be created in $MOOGSOFT_HOME/contrib/moog_enrichment_utils:

bin/
- moog_enrichment_util
- moog_enrichment_util.sh
- moog_init_enrichmentdb.sh
- moog_send_event
data/
- ...example data
dependencies.txt
node_modules/
package.json
package-lock.json
README.md

Note

The node_modules directory is pre-populated with the NodeJS dependency. To reinstall the dependencies run:

npm i

Bash script: `moog_enrichment_util.sh`

A shell script that makes use of the Enrichment API to load or delete data that accepts the following options:

Option	Description
`-p <user:password>`	Supply graze credentials for the Enrichment API.
`-u <url>`	Full URL to graze endpoint, ex: `https://server1/graze/v1/integrations/enrichment`
`-H <host:port>`	An alternative to the `-u` option - just the host and port used by the graze API.
`-f <file>`	Posts contents of `<file>`.
`-T <tsv\|csv\|json>`	The format of data in the supplied `<file>`.
`-t DELETE [ -A \| -a <attribute> -v <value] [ -L <age> ]`	Issue a delete request. Requires either `-A` or `-a` and `-v` options. If the -L option is supplied and the Enrichment API schema support it, only matching records older then the specified age will be deleted.
`-A`	Used with the `-t DELETE` option to delete all records.
`-a <attribute>`	Used with the `-t DELETE` to delete records for a given attribute and value.
`-v <value>`	Used with the `-t DELETE` to delete records for a given attribute and value
`-R <n>`	Number of retries on unsuccessful attempts - disabled by default.
`-B <n>`	Backoff interval in seconds - disabled by default.
`-b <n>`	Batch size - number of requests to attempt in one batch - defaults to 5.
`-V`	Enable verbose output.
`-P <proxy address>`	Use the supplied proxy, ex: `-P http://<proxyUser>:<proxyPass>@<proxyServer>:<proxyPort>`
`-L <age>`	Used with the `-t DELETE` option to remove matching records older than `<age>` seconds.

Uploading data

Sample data is provided in the data subdirectory for each supported format. To upload the CSV demo data to the server running on the localhost using the default graze credentials:

moog_enrichment_utils]$ ./bin/moog_enrichment_util.sh -H localhost:443 \
  -p graze:graze -f data/demo.csv -T csv

Deleting data

To delete all records older then 3600 seconds from the Enrichment API databases:

moog_enrichment_utils]$ ./bin/moog_enrichment_util.sh -H localhost:443 \
  -p graze:graze -T DELETE -A -L 3600

To delete all records with an attribute “source” and a value beginning with host1:

moog_enrichment_utils]$ ./bin/moog_enrichment_util.sh -H localhost:443 \
  -p graze:graze -T DELETE -a source -v 'host1*'

NodeJS script: `moog_enrichment_util`

A NodeJS version of the moog_enrichment_util.sh script that provides more features and greater reliability. The additional features include:

Generation of sample enrichment data
Post data asynchronously to increase throughput

Option	Description
`-p <user:password>`	Supply graze credentials for the Enrichment API.
`-u <url>`	Full url to graze endpoint, ex. `https://server1/graze/v1/integrations/enrichment`.
`-H <host:port>`	An alternative to the `-u` option - just the host and port used by the graze API.
`-n <number>`	Posts `<number>` generated records.
`-f <file>`	Posts contents of `<file>`.
`-T <tsv\|csv\|json>`	The format of data in the supplied `<file>`.
`-t <POST\|DELETE>`	The type of request to perform. Default to “POST” if the option is omitted.
`-A`	Used with the `-t DELETE` option to delete all records.
`-a <attribute>`	Used with the `-t DELETE` to delete records for a given attribute and value.
`-v <value>`	Used with the `-t DELETE` to delete records for a given attribute and value.
`-R <n>`	Number of retries on unsuccessful attempts - disabled by default.
`-B <n>`	Backoff interval in seconds - disabled by default.
`-b <n>`	Batch size - number of requests to attempt in one batch - defaults to 5.
`-q <n>`	Number of async requests to attempt in parallel.
`-V`	Enable verbose output.
`-P <proxy address>`	Use the supplied proxy, ex. `-P http://<proxyUser>:<proxyPass>@<proxyServer>:<proxyPort>`
`-L <age>`	Used with the `-t DELETE` option to remove matching records older than `<age>` seconds.

Uploading data

Sample data is provided in the data subdirectory for each supported format. To upload the CSV demo data to the server running on the localhost using the default graze credentials:

moog_enrichment_utils]$ ./bin/moog_enrichment_util -H localhost:443 \
  -p graze:graze -f data/demo.csv -T csv

For initial testing, we can also generate data to upload. The following example shows 10 records being generated and inserted:

moog_enrichment_utils]$ ./bin/moog_enrichment_util -H localhost:443 \
  -p graze:graze -n 10
generating 10 records
Batch being processed...
Batch complete.
Batch being processed...
Batch complete.

Deleting data

To delete all records older then 3600 seconds from the Enrichment API databases:

moog_enrichment_utils]$ ./bin/moog_enrichment_util -H localhost:443 \
  -p graze:graze -T DELETE -A -L 3600

Deleting all records with an attribute “source” and a value beginning with “host1”:

moog_enrichment_utils]$ ./bin/moog_enrichment_util -H localhost:443 \
  -p graze:graze -T DELETE -a source -v 'host1*'
deleteItem attribute: 'source', value: 'host1*'

Request arguments

Endpoint enrichment takes the following request payload:

Name	Type	Required	Description
`action`	String	yes	One of `post` or `delete`. `post` creates or updates enrichment records. `delete` removes enrichment records.
`data`	Array	yes	An array of of enrichment data records represented as JSON objects.
`attribute`	String	yes	Name of the alert field or other key for the enrichment data. For example, "source". For the `delete` action, accepts the `*` wildcard to delete all attributes.
`value`	String	yes	The value for the associated attribute. For example if the attribute is "source" for host name data, a value might be "sflinux101". For the delete action, accepts the `` wildcard at the beginning of the search string,end of the search string or booth to delete all matching values. For example "linux*" would delete all matching values that contain the string "linux": "SFlinux101", "SFlinux", and "linux101".
enrichment	JSON object	for `post` action	JSON representation of the enrichment data to add to an alert based upon the match attribute and value. For example if you wanted to store store location data: {"location":"1265 Battery St., San Francisco, CA"}

Example payload:

{"action":"post",
 "data":[
    {"attribute":"source",
    "value":"SFlinux101",
    "enrichment": {
        "location":"1265 Battery St., San Francisco, CA",
        "support_group":"SF NOC"}
    },
    {"attribute":"source",
     "value":"DENlinux102",
        "enrichment": {
            "location":"1700 Lincoln Street, Denver, CO",
            "support_group":"DENVER NOC"}          
    }]
}

Endpoint enrichment takes the following request argument:

Name	Type	Required	Description
`auth_token`	String	Yes	A valid `auth_token` returned from the `authenticate` request. Alternatively, basic authenticate headers can be used in place of this parameter. See the authenticate endpoint for more information and usage examples.

Response

Endpoint enrichment returns the following response:

Type	Description
HTTP Code	HTTP status or error code indicating request success or failure. See HTTP status code definitions for more information.

Examples

The following examples demonstrate typical use of endpoint grazeApiEndpointName:

Request example

Example cURL request to create enrichment data:

curl -k -X POST 'https://localhost/graze/v1/integrations/enrichment' \
--header 'Content-Type: application/json; charset=UTF-8' \
-u graze:graze \
-d '{"action":"post", "data":[ {"attribute":"source", "value":"SFlinux101", "enrichment": { "location":"1265 Battery St., San Francisco, CA", "support_group":"SF NOC"} }, {"attribute":"source", "value":"DENlinux102", "enrichment": { "location":"1700 Lincoln Street, Denver, CO", "support_group":"DENVER NOC"} }]}'

Response example

In this section:

/enrichment

Request arguments

Action: post

Action: delete

Age scoped deletion

Utility Scripts

Note

Bash script: moog_enrichment_util.sh

Uploading data

Deleting data

NodeJS script: moog_enrichment_util

Uploading data

Deleting data

Request arguments

Response

Examples

Request example

Response example

Search results

Bash script: `moog_enrichment_util.sh`

NodeJS script: `moog_enrichment_util`