# Moogsoft Docs

## Data Parsing

Moogsoft AIOps divides incoming data into tokens (tokenised) and then assembles the tokens into an event. You can control how tokenising works.

## Start and End Characters

The first two are a  start  and  end  character. The square brackets [] are the JSON notation for a list. You can have multiple start and end characters. The system considers an event as all of the tokens between any start and end character.

start	: [],
end		: ["\n"],

The above example specifies:

• There is nothing defined in  start  ; however, a carriage return (new line) is defined as the end character

In the example above, the LAM is expecting a entire line to be written followed by a return, and it will process the entire line as one event.

Carefully set up, you can accept multi-line events.

## Regular Expressions

Regular expressions can be used to extract relevant data from the input data. Here's an example definition:

parsing:
{
type: "regexp",
regexp:
{
pattern : "(?m)^START: (.*?)$", capture_group: 1, tokeniser_type: "delimiters", delimiters: { ignoreQuotes: true, stripQuotes: true, ignores: "", delimiter: ["||","\r"] } } } ## Delimiters Delimiters define how string are split into tokens for processing. To process a comma-separated file, where a comma separates each value, define the comma as a delimiter. Token are referenced from the start position starting at one (not zero). For example, for the input string “the,cat,sat,on,the,mat” where the delimiter is a comma, token 1 is “the”, token 2 “cat” and so on. Combining tokenization and parsing can be complex. For example, if you use a comma delimiter and the token contains a comma, the token is split into two. To avoid this you can quote strings. You can then define whether to strip or ignore quotes. An example delimiters section in a configuration file is as follows: delimiters: { ignoreQuotes : true, stripQuotes : false, ignores : "", delimiter : [",","\r"] } When  ignoreQuotes  is set to true, all quotes are ignored and inputs are tokenised on the delimiters only. When  ignoreQuotes  is false, delimiting does not occur until the matching end quote is found. This allows tokens to include delimiters. For example, given the following input when the delimiter is a comma: hello world, "goodbye, cruel world". Found tokens when  ignoreQuotes  is true: [hello world, goodbye, cruel world] (3). Found tokens when  ignoreQuotes  is false: [hello world, "goodbye, cruel world"] (2). Set  stripQuotes  to true to remove start and end quotes from tokens. For example, "hello world" results in a single token: [hello world].  Ignores  is a list of characters to ignore. Ignored characters are never included in tokens.  Delimiter  is the list of valid delimiters used to split strings into tokens. ## Mapping For each event in the file, there is a positioned collection of tokens. Moogsoft AIOps enables you to name these positions so if you have a large number of tokens in a line, of which you are interested in only five or six, instead of remembering it is token number 32, you can call token 32 something meaningful. variables: [ { name: "Identifier", position: 1 }, { name: "Node", position: 4 }, { name: "Serial", position: 3 }, { name: "Manager", position: 6 }, { name: "AlertGroup", position: 7 }, { name: "Class", position: 8 }, { name: "Agent", position: 9 }, { name: "Severity", position: 5 }, { name: "Summary", position: 10 }, { name: "LastOccurrence",position: 1 } ] The above example specifies: •  position 1  is assigned to  Identifier; position 4  is assigned to  node  and so on • Positions start at 1, and go up rather than array index style counting from 0 This is important because at the bottom of the file, socket_lam.conf there is a mapping object that configures how Moogsoft AIOps assigns to the attributes of the event that is sent to the message bus, values from the tokens that are parsed. For example, in  mapping  there is a value called  rules  , which is a list of assignments. mapping: { catchAll: "overflow", rules: [ { name: "signature",rule: "$Node:$Serial" }, { name: "source_id",rule: "$Node" },
{ name: "external_id",rule: "$Serial" }, { name: "manager",rule: "$Manager" },
{ name: "source",rule: "$Node" }, { name: "class",rule: "$Class" },
{ name: "agent",rule: "$LamInstanceName" }, { name: "agent_location",rule: "$Node" },
{ name: "type",rule: "$AlertGroup" }, { name: "severity",rule: "$Severity", conversion: "sevConverter" },
{ name: "description",rule: "$Summary" }, { name: "first_occurred",rule: "$LastOccurrence" ,conversion: "stringToInt"},
{ name: "agent_time",rule: "$LastOccurrence",conversion: "stringToInt"} ] } In the example above, the first assignment name:  "signature",rule:"$Node:$Serial"  (  "$Node:$Serial  is a string with$ syntax) means for signature take the tokens called Node and Serial and form a string with the value of Node followed by a colon followed by the value of  Serial  and call that signature in the event that is sent to the Moogsoft AIOps.

You define a number of these rules covering the base attributes of an event. For reference, Moogsoft AIOps expects a minimum set of attributes in an event that are shown in this particular section.

If you have an attribute that is never referenced in a rule, for example “enterprise trap number” which is never mapped into the attribute of an event, they are collected and placed as a JSON object in a variable defined in  catchAll  and passed as part of the event.

### Custom Info Mapping

You can define custom_info mapping in LAM configuration files. This allows you to configure a hierarchical structure. An example mapping configuration is:

mapping:
{
rules:
[
{ name: "custom_info.eventDetails.branch", rule: "$branch" }, { name: "custom_info.eventDetails.location", rule: "$location" },