Pattern Matcher

This Workflow Integration and the associated Workflow Engine action (matchPattern) allows a set of regular expressions to be evaluated against an event, alert or Situation attribute, and if a match is found, set a corresponding value.

For example, if an alert description contained the words “ping fail”, you could set the custom_info.assignmentGroup to “network”.

When to use the Pattern Matcher

The Pattern Matcher integration offers an improved solution for a common use case that can already be achieved with existing Workflow actions. It allows you to evaluate multiple expressions in a single workflow, making administration and workflow performance easier. Pattern Matcher is not meant to replace existing actions, so there is no need to modify existing workflows to include it. It should be used when iterative evaluation over multiple patterns is required.

Patterns can be organized into groups based on their relevance, such as a Network group or an Application group. These groups can then be evaluated in a configurable order, and with the matchPattern action, specific groups can be selected for evaluation. For example, you can choose to only evaluate the Network group against alerts coming from a particular manager or agent.

Pattern Matcher expressions

The patterns defined in the Pattern matcher can be one of four types:

Pattern Type	Example	Description
Equality	`abc`	Does the value exactly match the pattern (case insensitive).
Regex	`/abc/`	Does the value match the regular expression. Case insensitive.
Negative Regex	`!/abc/`	Does the value NOT match the regular expression.
Similarity	`~50[abc]`	Is the value similar to the expression. Define the degree of similarity.

Before you begin

Before you start to configure the Pattern Matcher integration, ensure you know the following about your patterns and values:

A source and destination field (event, alert or Situation)
A value to set if a match is found
A equality, regular expression or similarity expression
Whether the match should be first match or last match:
- First match stops evaluating the expressions within a pattern group once a match is found
- Last match evaluates all expressions in a group and the last one matching is used
Whether or not subsequent pattern groups should be evaluated when a match is made

List values

If the CEvent (event, alert or Situation) field value is a list (an array), then the expressions will be evaluated against each member of the list. A match will be made if any member matches the expression.

For example, given a CEvent field with a value of [ "Email", "Databae", "Networking"] and a pattern of /^network/, the expression would be considered a match, as “Networking” matches the regular expression, even though “Email” and “Database” do not.

First match vs. last match

A pattern group can be defined to use either a “first” or “last” match. When “first match” is used, the first expression that matches the field value will be used to set the destination field, and subsequent patterns within the group will not be evaluated. When “last match” is used, all patterns within a group will be evaluated with the last matching value used.

Generally first match should be used to prevent unnecessary iteration, but last match can be used to get a more exact match when regular expressions increase in granularity or complexity.

For example, given the following patterns:

Pattern	Value
`/^App\\d+$/`	Application
`/^App00\\d+/`	Application in DC 00
`/^App0011$/`	Application 11 in DC 00
`/^App0112$/`	Application 12 in DC 01

...and a field value of

App0011

...with “first match” enabled, the destination field will be set to “Application”, since the value “App0011” matches the “/^App\d+$/” expression.

With “last match” enabled, the destination field will be set to “Application 11 in DC 00” even though it matched both of the previous expressions, as the last pattern it matched was “/^App0011$/”.

Regular Expression Syntax

The Pattern Matcher is using JavaScript to evaluate the regular expressions, so expressions should follow the JS standard and syntax. Moogsoft JS is not Node.js, so some current Node.js / ECMA 6 regular expressions (such as lookback assertions) are not supported.

If a regular expression cannot be correctly instantiated or coerced, it will be ignored (considered a non-match) and a log message will be visible in the moog_farmd logfile.

Regular expressions defined via the UI do not need to be double escaped. For example, \s does not need to be entered as \\s.

Similarity Expressions

The similarity expression syntax contains a percentage similarity value (the numbers after the ~). This can be from 0-100. The expression that is compared for similarity is contained within leading and trailing square brackets “[ ]”.

Expression	Meaning
`~50[this is a test]`	Does the value have at least 50% similarity to the string “this is a test”
`~99[connected BGP to]`	Does the value have at least 99% similarity to the string “connected BGP to”

Similarity comparison is case insensitive. For example, “cat” is 100% similar to “CAT”.

The similarity algorithm is the same as the one used in the Cookbook and employs a fixed shingling rule. If the value of the CEvent (source) field can be whitespace tokenized, then word shingling is used. If the CEvent value contains no whitespaces, then a fixed shingle size of 3 is used. A whitespace is defined by the JS regular expression “\s” (space, tab, line/form feed, etc.).

CEvent value	Shingle size used
`This is a test`	word
`This_is_a_test`	3 (no whitespaces)
`HOSTABC1233`	3 (no whitespaces)

Where word shingles are used, ordering is not relevant - “this is a cat” is 100% similar to “cat a is this”. Where the fixed shingle size is used, ordering has an influence - “HOST123” is not 100% similar to “123HOST” (they are 60% similar). This is behavior is the same as the similarity algorithm in Cookbook.

If the similarity result is greater than or equal to the configured value, the match is considered true. There is no “negative” match (“match if less than n% similarity”).

Configuring the Pattern Matcher

Top level controls

There is a single top level configuration control for the Pattern Matcher that determines if the evaluation process should consider all groups, or stop after the a match is made in any group.

This feature is useful in cases where a single expression needs to set values for multiple fields, which are defined in different groups.

By default, this is set to false, which stops evaluation once a match is made. This is recommended for better performance and to avoid accidentally overwriting values.

If the feature is set to “true”, then all groups (either explicitly configured, or in the default set) will be evaluated in priority order, and the CEvent will be updated each time a match is found in a group.

For example, given the following Pattern Groups each containing a number of expressions matching an alert:

Network
Application
OperatingSystem
Storage

With the “continue evaluating…” control enabled, the Network group would be evaluated, and any matching expression would update the alert, followed by the Application, OperatingSystem and Storage groups - each updating the alert if a suitable expression match was found.

With the “continue evaluating…” control disabled, the evaluation would stop after the Network group match was found.

Note

If the matchPattern action is using the source or destination field overrides, these take precedence over the Pattern Matcher configuration.

If the “continue evaluating… “ control is enabled, and the overrides are in place, the destination field will be overwritten by subsequent matching values.

Pattern Groups

The Pattern Matcher uses groups of patterns. These groups represent logical groupings of match expressions and values (ex. the pattens associated with a specific type of alert or operational group).

Each pattern group has the following top level attributes:

Attribute Name	Description
Group Name	A unique name for this pattern group. This will be referenced in the `matchPattern` WFE action.
Source Field	The event, alert or Situation field to compare the patterns against. This can be overridden in the `matchPattern` action if needed.
Destination Field	The event, alert or Situation field to write a value to if a match is found. This is a text value. This can be a CEvent field or a `workflowContext` field.
Priority	The numeric priority to control the evaluation order, low to high (priority 1 is evaluated before priority 99). If two or more groups have the same priority, the group name is used to determine order (alpha sort).
Include by default	Whether to include this group in the evaluation by the `matchPattern` action if no groups are explicitly defined in the action. If a group is not included by default, then the `matchPattern` action will need to explicitly define its inclusion.
Match Behavior	First or Last match (see above for details).
Match Conditions	A list of expressions and their values (see below)

Configuring a Match Condition

A match condition has two attributes:

Attribute Name	Description
Expression	The equality, regex, negative regex or similarity expression to use in the comparison with the Source Field.
Value	The text or JSON value to write to the Destination Field if a match is found (see below for notes on using a JSON value).

Supplying a JSON value

The “value” attribute can be a simple text value, or, by using the $TO_JSON( ) macro, a complex object. The syntax for this the same as the Payload Macro (https://docs.moogsoft.com/v9/en/macros-reference.html#to_json) - but does not support substitution of values. The parsed JSON object is set as the destination field's value.

For example, if an expression matched the CEvent value and we wanted to set the following map under custom_info.assignmentData:

{
  "assignTo" : "Network",
  "priority" : "P1"
}

The Group destination field would be set to:

custom_info.assignmentData

The “value” in the matcher condition would be set as:

$TO_JSON( { "assingnTo" : "Network, "priority" : "P1" } )

If the supplied JSON does not parse, then the raw text value would be set.

Best Practices

Here are some best practices to consider when using the Pattern Matcher:

Do not evaluate all groups unless necessary. The higher the number of expression checks, the higher the compute cost will be.
Ensure that groups are prioritized in popularity (match frequency) order.
- Regex evaluation is computationally “expensive” - so the most frequent matches should be evaluated first. The matchPattern action will stop evaluation after the first group match is made.
Ensure patterns within a group are ordered by popularity.
- There is no priority attribute at the matcher level, so re-ordering requires deletion and re-addition (see below for API usage for this).
If a pattern group is isolated to a specific subset of alerts, use standard WFE features to ensure that the right groups are targeted (entry filters, using named groups in the matchPattern action).
Make regular expressions as efficient as possible.
- Standard regex performance best practices should be used, anchors, non-greedy modifiers, etc.

Using the Integration API to configure the Pattern Matcher

Configuring large numbers of conditions by hand could introduce an unwanted administrative overhead, for example where the patterns and values change frequently.

The Integration API can be used to automate this if needed.

Find the Integration ID of the PatternMatcher integration:
```
GET https://<servername>/integrations/api/v1/integrations
```
This will return a list of all the configured integrations.
Find the PatternMatcher entry - this will be the entry with the type = “PatternMatcher”:
```
{
    "id": 28,
    "name": "PatternMatcher1",
    "type_id": "PatternMatcher",
    ...
}
```
Fetch the integration using the Integration ID:
```
GET https://<servername>/integrations/api/v1/integrations/{id}
```
This will return the configuration for Pattern Matcher as the only response.

Add or modify the inputs.value[*].groupitems array to add, remove, or modify entries. Entries contain 2 fields: "regex", which is the matcher expression, and "value", which is the value to set. Groups do not need to be in the evaluation order, as priority does this. However, matcher patterns within a group should be arranged in evaluation order, from most popular to least popular matches.

For example, to add a new group the following JSON can be added to the inputs.value array as the last item:

{
  "name": "test2",
  "source": "source",
  "default": "Unknown",
  "priority": 11,
  "groupitems": [
      {
          "regex": "/xyz\\s+\\d+/i",
          "value": "xyz"
      }
  ],
  "destination": "custom_info.assignment",
  "matchbehaviour": "first",
  "isDefaultMatcher": false
}

Adding an additional group:

{
  "id": 28,
  "name": "PatternMatcher1",
  "type_id": "PatternMatcher",
  ...
  "inputs": [
    {
      "key": "patternGroups",
      "value": [
        {
          "name": "test",
          "source": "description",
          "default": "Unkown",
          "priority": 0,
          "groupitems": [
            {
              "regex": "/^ABCDEF$/",
              "value": "abc"
            },
            {
              "regex": "!/this is a test/i",
              "value": "a test"
            }
          ],
          "destination": "custom_info.assignment",
          "matchbehaviour": "first",
          "isDefaultMatcher": false
        },
        {
          "name": "test1",
          "source": "description",
          "default": "Unknown",
          "priority": 10,
          "groupitems": [
            {
              "regex": "/abc\\s+\\d+/i",
              "value": "abc"
            },
            {
              "regex": "/def/",
              "value": "def"
            },
            {
              "regex": "~50[BGP Peer connected to]",
              "value": "BGP"
            },
            {
              "regex": "/BGP/",
              "value": "NoBGP"
            }
          ],
          "destination": "custom_info.assignment",
          "matchbehaviour": "first",
          "isDefaultMatcher": false
        },
        {
          "name": "test2",
          "source": "source",
          "default": "Unknown",
          "priority": 11,
          "groupitems": [
            {
              "regex": "/xyz\\s+\\d+/i",
              "value": "xyz"
            }
          ],
          "destination": "custom_info.assignment",
          "matchbehaviour": "first",
          "isDefaultMatcher": false
        }
      ]
    }
  ]

Note

Ensure that regex text patterns are double escaped as needed in the JSON payload.

Update the integration configuration using the same ID and the PUT HTTP method:
```
PUT https://<servername>/integrations/api/v1/integrations/{id}
```
The integration does not need to be restarted for the new configuration to be applied. The new configuration will be used the next time the matchPattern action runs.

In this section:

Pattern Matcher

When to use the Pattern Matcher

Pattern Matcher expressions

Before you begin

List values

First match vs. last match

Regular Expression Syntax

Similarity Expressions

Configuring the Pattern Matcher

Top level controls

Note

Pattern Groups

Configuring a Match Condition

Supplying a JSON value

Best Practices

Using the Integration API to configure the Pattern Matcher

Note

Search results