Kafka Reference

The hostname and port of the Kafka broker. To configure multiple ports, separate them with commas.

Example

kafka_listener: "example001.mlp.com:9092, example002.mlp.com:9092, example003.mlp.com:9092",

The name of the topic(s) in the Kafka broker you are fetching events from.

The name of the consumer group. Kafka distributes the data evenly among consumers in the same group to improve the processing of topics for the consumers. This is especially helpful when there are multiple partitions in a topic; a consumer may pick data from an individual partition of the topic, hence increasing the speed of the LAM in consuming the data.

Specifies whether to encrypt communications between Kafka and Moogsoft.

The path of the truststore certificate.

The password for the truststore certificate.

The path of the keystore certificate.

The password for the keystore certificate.

The password for the client certificate required in client authentication. It is the password entered in the ssl.key.password of the Kafka server.properties file.

The parsing method to use.

Examples

Parsing block with text messages, using delimiter-based tokenising:

        parsing:
        {
           type: "start_and_end",
            start_and_end:
            {
                start:      [],
                end:        ["\n"],

                delimiters:
                {
                   ignoreQuotes: true,
                   stripQuotes: true,
                   ignores:    "",
                   delimiter:  [",","\r"]
               }
           }
        },

Parsing block with regular expressions, using delimiter-based tokenising:

        parsing:
         {
             type: "regexp",
             regexp:
             {
                 pattern : "(?mU)^(.*)$",
                 capture_group: 1,
                 tokeniser_type: "delimiters",
                 delimiters:
                 {
                         ignoreQuotes: true,
                         stripQuotes: false,
                         ignores:    "",
                         delimiter:  ["\r"]
                 }
             }
         },

Parsing block with regular expressions, using subgroups groups to capture tokens:

        parsing:
         {
             type: "regexp",
             regexp:
             {
                 pattern : "(?mU)^(.*)\t(.*)\t(.*)$",
                 tokeniser_type: "regexp_subgroups",
             }
         },

Parsing method in which parsing will start when it gets NEW_MSG and end when it gets new line. The extracted string is then delimited as per the defined delimiters. See the first method above for an example.

Parsing method in which the parser searches for strings as per the expression defined in the pattern field and subsequently delimits them in accordance with your configuration of delimiters.

The string(s) to search for when using the regexp method.

Specifies the capture group for tokenised parsing of regexps.

The type of tokeniser to use.

Delimiters define how a line is split into tokens. For example, if you have a line of text data, it needs to be split up into a sequence of substrings that are referenced by position from the start. So, if you are processing a comma-separated file, where each value is separated by a comma, it makes sense to have the delimiter defined as a comma. The system would take all the text between start and end and break it up into tokens between the commas. The tokens could then be referenced by position number in the string starting from 1 (not zero). For example, if the input string is cat,sat,on,the,mat and a comma is used as a separator, then token 1 will be cat, token 2 will be sat and so on.

Example

delimiters:
{ 
    ignoreQuotes: true, 
    stripQuotes: false, 
    ignores: "", 
    delimiter: [",","\r"]
}

If you have strings that are quoted between delimiters, set ignoreQuotes to true to look for delimiters inside the quote. For example, <delimiter>hello inside quote goodbye<delimiter> gives a token [hello inside quote goodbye].

Set to true to remove start and end quotes from tokens. For example, hello world gives the token [hello world].

A list of characters to ignore and exclude from tokens.

The list of valid delimiters to use for splitting strings into tokens.

Tokenises the extracted string based on groups in a message. An expression in the parenthesis in the regular expression denotes a group.

For example, the part expression in a regular expression such as ((?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}) is a group which contains the date and time.