Kafka Reference

This is a reference for the Apache Kafka LAM and UI integration. The Kafka LAM configuration file is located at $MOOGSOFT_HOME/config/kafka.conf. The following properties are unique to the Kafka LAM and UI integration. See the LAM and Integration Reference for a full description of all common properties used to configure LAMs and UI integrations. See the Apache Kafka documentation for details on Kafka components. monitor kafka_listener The hostname and port of the Kafka broker. To configure multiple ports, separate them with commas.  Type String Required Yes Default localhost:9092 Example kafka_listener: "example001.mlp.com:9092, example002.mlp.com:9092, example003.mlp.com:9092", topic_name The name of the topic(s) in the Kafka broker you are fetching events from.  Type String Required Yes Default N/A groupid The name of the consumer group. Kafka distributes the data evenly among consumers in the same group to improve the processing of topics for the consumers. This is especially helpful when there are multiple partitions in a topic; a consumer may pick data from an individual partition of the topic, hence increasing the speed of the LAM in consuming the data.  Type String Required Yes Default N/A ssl_connection Specifies whether to encrypt communications between Kafka and Moogsoft.  Type Boolean Required Yes Default false ssl_truststore_filename The path of the truststore certificate.  Type String Required Yes, if ssl_connection is set to true Default false ssl_truststore_password The password for the truststore certificate.  Type String Required Yes, if ssl_connection is set to true Default N/A ssl_keystore_filename The path of the keystore certificate.  Type String Required Yes, if ssl_connection is set to true Default N/A ssl_keystore_password The password for the keystore certificate.  Type String Required Yes, if ssl_connection is set to true Default N/A ssl_key_password The password for the client certificate required in client authentication. It is the password entered in the ssl.key.password of the Kafka server.properties file.  Type String Required Yes, if ssl_connection is set to true Default N/A kafka_properties Kafka consumer properties. Any properties you define here take priority over SSL configuration. See the Apache Kafka documentation for descriptions of these properties. parsing Moogsoft Enterprise divides incoming data into tokens (tokenized) and then assembles the tokens into an event. You can control how tokenizing works using the properties below. See Data Parsing for more information. type The parsing method to use.  Type String Required Yes Default N/A Valid Values Start_and_End: The parsing will start when it gets NEW_MSG and end when it gets new line. The extracted string is then delimited as per the defined delimiters.regexp: In a regular expression, the parser searches for strings as per the expression defined in the pattern field. The extracted string is then delimited as per the defined delimiters. In the above example, the parser searches for the expression "(?mU)^(.*)$".

Examples

Parsing block with text messages, using delimiter-based tokenising:

        parsing:
{
type: "start_and_end",
start_and_end:
{
start:      [],
end:        ["\n"],

delimiters:
{
ignoreQuotes: true,
stripQuotes: true,
ignores:    "",
delimiter:  [",","\r"]
}
}
},


Parsing block with regular expressions, using delimiter-based tokenising:

        parsing:
{
type: "regexp",
regexp:
{
pattern : "(?mU)^(.*)$", capture_group: 1, tokeniser_type: "delimiters", delimiters: { ignoreQuotes: true, stripQuotes: false, ignores: "", delimiter: ["\r"] } } }, Parsing block with regular expressions, using subgroups groups to capture tokens:  parsing: { type: "regexp", regexp: { pattern : "(?mU)^(.*)\t(.*)\t(.*)$",
tokeniser_type: "regexp_subgroups",
}
},

start_and_end

Parsing method in which parsing will start when it gets NEW_MSG and end when it gets new line. The extracted string is then delimited as per the defined delimiters. See the first method above for an example.

 Type String Required Yes, if type is set to "start_and_end". Default N/A Valid Values If using this method you must configure the following properties:start:end:

regexp

Parsing method in which the parser searches for strings as per the expression defined in the pattern field and subsequently delimits them in accordance with your configuration of delimiters.

 Type String Required No Default N/A

pattern

The string(s) to search for when using the regexp method.

 Type String Required Yes, if using regexp Default N/A

capture_group

Specifies the capture group for tokenised parsing of regexps.

 Type Integer Required Yes, if using regexp Default 1

tokeniser_type

The type of tokeniser to use.

 Type String Required Yes Default N/A Valid Values See the delimiters and regexp_subgroups properties below.

delimiters

Delimiters define how a line is split into tokens. For example, if you have a line of text data, it needs to be split up into a sequence of substrings that are referenced by position from the start. So, if you are processing a comma-separated file, where each value is separated by a comma, it makes sense to have the delimiter defined as a comma. The system would take all the text between start and end and break it up into tokens between the commas. The tokens could then be referenced by position number in the string starting from 1 (not zero). For example, if the input string is cat,sat,on,the,mat and a comma is used as a separator, then token 1 will be cat, token 2 will be sat and so on.

 Type Object Required No Default N/A Valid Values See the ignoreQuotes, stripQuotes, ignores and delimiter sections below.

Example

delimiters:
{
ignoreQuotes: true,
stripQuotes: false,
ignores: "",
delimiter: [",","\r"]
}

ignoreQuotes

If you have strings that are quoted between delimiters, set ignoreQuotes to true to look for delimiters inside the quote. For example, <delimiter>hello inside quote goodbye<delimiter> gives a token [hello inside quote goodbye].

 Type Boolean Required Yes, if using delimiters Default N/A

stripQuotes

Set to true to remove start and end quotes from tokens. For example, hello world gives the token [hello world].

 Type Boolean Required Yes, if using delimiters Default N/A

ignores

A list of characters to ignore and exclude from tokens.

 Type String Required No Default N/A

delimiter

The list of valid delimiters to use for splitting strings into tokens.

 Type String Required Yes, if using delimiters Default N/A

regexp_subgroups

Tokenises the extracted string based on groups in a message. An expression in the parenthesis in the regular expression denotes a group.

For example, the part expression in a regular expression such as ((?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}) is a group which contains the date and time.

 Type Boolean Required No Default N/A

variables

A received event is a positioned collection of tokens. The variable section is used when the received event message type is TextMessage; a JSON event can be mapped directly to the Moog field in the Mapping section. The variables section enables you to name these positions. The naming of the positions helps you identify the tokens. Positions start at 1 and increase.

 Type List Required No Default N/A

Example

variables:
[
{ name: "signature",   position: 1 },
{ name: "source_id",   position: 4 },
{ name: "external_id", position: 3 },
{ name: "Manager",     position: 6 },
{ name: "AlertGroup",  position: 7 },
{ name: "Class",       position: 8 },
{ name: "Agent",       position: 9 },
{ name: "severity",    position: 5 },
{ name: "description", position: 10 },
{ name: "agent_time",  position: 2 }
],