Skip to main content

Kafka Reference

This is a reference for the Apache Kafka LAM and UI integration. The Kafka LAM configuration file is located at $MOOGSOFT_HOME/config/kafka.conf.

The following properties are unique to the Kafka LAM and UI integration.

See the LAM and Integration Reference for a full description of all common properties used to configure LAMs and UI integrations.

See the Apache Kafka documentation for details on Kafka components.

monitor

kafka_listener

The hostname and port of the Kafka broker. To configure multiple ports, separate them with commas.

Type

String

Required

Yes

Default

localhost:9092

Example

kafka_listener: "example001.mlp.com:9092, example002.mlp.com:9092, example003.mlp.com:9092",

topic_name

The name of the topic(s) in the Kafka broker you are fetching events from.

Type

String

Required

Yes

Default

N/A

groupid

The name of the consumer group. Kafka distributes the data evenly among consumers in the same group to improve the processing of topics for the consumers. This is especially helpful when there are multiple partitions in a topic; a consumer may pick data from an individual partition of the topic, hence increasing the speed of the LAM in consuming the data.

Type

String

Required

Yes

Default

N/A

ssl_connection

Specifies whether to encrypt communications between Kafka and Moogsoft.

Type

Boolean

Required

Yes

Default

false

ssl_truststore_filename

The path of the truststore certificate.

Type

String

Required

Yes, if ssl_connection is set to true

Default

false

ssl_truststore_password

The password for the truststore certificate.

Type

String

Required

Yes, if ssl_connection is set to true

Default

N/A

ssl_keystore_filename

The path of the keystore certificate.

Type

String

Required

Yes, if ssl_connection is set to true

Default

N/A

ssl_keystore_password

The password for the keystore certificate.

Type

String

Required

Yes, if ssl_connection is set to true

Default

N/A

ssl_key_password

The password for the client certificate required in client authentication. It is the password entered in the ssl.key.password of the Kafka server.properties file.

Type

String

Required

Yes, if ssl_connection is set to true

Default

N/A

kafka_properties

Kafka consumer properties. Any properties you define here take priority over SSL configuration. See the Apache Kafka documentation for descriptions of these properties.

parsing

Moogsoft Onprem divides incoming data into tokens (tokenized) and then assembles the tokens into an event. You can control how tokenizing works using the properties below. See Tokenize Source Event Data for more information.Data Parsing

type

The parsing method to use.

Type

String

Required

Yes

Default

N/A

Valid Values

  • Start_and_End: The parsing will start when it gets NEW_MSG and end when it gets new line. The extracted string is then delimited as per the defined delimiters.

  • regexp: In a regular expression, the parser searches for strings as per the expression defined in the pattern field. The extracted string is then delimited as per the defined delimiters. In the above example, the parser searches for the expression "(?mU)^(.*)$".

Examples

Parsing block with text messages, using delimiter-based tokenising:

        parsing:
        {
           type: "start_and_end",
            start_and_end:
            {
                start:      [],
                end:        ["\n"],

                delimiters:
                {
                   ignoreQuotes: true,
                   stripQuotes: true,
                   ignores:    "",
                   delimiter:  [",","\r"]
               }
           }
        },

Parsing block with regular expressions, using delimiter-based tokenising:

        parsing:
         {
             type: "regexp",
             regexp:
             {
                 pattern : "(?mU)^(.*)$",
                 capture_group: 1,
                 tokeniser_type: "delimiters",
                 delimiters:
                 {
                         ignoreQuotes: true,
                         stripQuotes: false,
                         ignores:    "",
                         delimiter:  ["\r"]
                 }
             }
         },

Parsing block with regular expressions, using subgroups groups to capture tokens:

        parsing:
         {
             type: "regexp",
             regexp:
             {
                 pattern : "(?mU)^(.*)\t(.*)\t(.*)$",
                 tokeniser_type: "regexp_subgroups",
             }
         },

start_and_end

Parsing method in which parsing will start when it gets NEW_MSG and end when it gets new line. The extracted string is then delimited as per the defined delimiters. See the first method above for an example.

Type

String

Required

Yes, if type is set to "start_and_end".

Default

N/A

Valid Values

If using this method you must configure the following properties:

  • start:

  • end:

regexp

Parsing method in which the parser searches for strings as per the expression defined in the pattern field and subsequently delimits them in accordance with your configuration of delimiters.

Type

String

Required

No

Default

N/A

pattern

The string(s) to search for when using the regexp method.

Type

String

Required

Yes, if using regexp

Default

N/A

capture_group

Specifies the capture group for tokenised parsing of regexps.

Type

Integer

Required

Yes, if using regexp

Default

1

tokeniser_type

The type of tokeniser to use.

Type

String

Required

Yes

Default

N/A

Valid Values

See the delimiters and regexp_subgroups properties below.

delimiters

Delimiters define how a line is split into tokens. For example, if you have a line of text data, it needs to be split up into a sequence of substrings that are referenced by position from the start. So, if you are processing a comma-separated file, where each value is separated by a comma, it makes sense to have the delimiter defined as a comma. The system would take all the text between start and end and break it up into tokens between the commas. The tokens could then be referenced by position number in the string starting from 1 (not zero). For example, if the input string is cat,sat,on,the,mat and a comma is used as a separator, then token 1 will be cat, token 2 will be sat and so on.

Type

Object

Required

No

Default

N/A

Valid Values

See the ignoreQuotes, stripQuotes, ignores and delimiter sections below.

Example

delimiters:
{ 
    ignoreQuotes: true, 
    stripQuotes: false, 
    ignores: "", 
    delimiter: [",","\r"]
}

ignoreQuotes

If you have strings that are quoted between delimiters, set ignoreQuotes to true to look for delimiters inside the quote. For example, <delimiter>hello inside quote goodbye<delimiter> gives a token [hello inside quote goodbye].

Type

Boolean

Required

Yes, if using delimiters

Default

N/A

stripQuotes

Set to true to remove start and end quotes from tokens. For example, hello world gives the token [hello world].

Type

Boolean

Required

Yes, if using delimiters

Default

N/A

ignores

A list of characters to ignore and exclude from tokens.

Type

String

Required

No

Default

N/A

delimiter

The list of valid delimiters to use for splitting strings into tokens.

Type

String

Required

Yes, if using delimiters

Default

N/A

regexp_subgroups

Tokenises the extracted string based on groups in a message. An expression in the parenthesis in the regular expression denotes a group.

For example, the part expression in a regular expression such as ((?(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\\s+\\d{1,2}) is a group which contains the date and time.

Type

Boolean

Required

No

Default

N/A

variables

A received event is a positioned collection of tokens. The variable section is used when the received event message type is TextMessage; a JSON event can be mapped directly to the Moog field in the Mapping section. The variables section enables you to name these positions. The naming of the positions helps you identify the tokens. Positions start at 1 and increase.

Type

List

Required

No

Default

N/A

Example

variables:
        [
            { name: "signature",   position: 1 },
            { name: "source_id",   position: 4 },
            { name: "external_id", position: 3 },
            { name: "Manager",     position: 6 },
            { name: "AlertGroup",  position: 7 },
            { name: "Class",       position: 8 },
            { name: "Agent",       position: 9 },
            { name: "severity",    position: 5 },
            { name: "description", position: 10 },
            { name: "agent_time",  position: 2 }
        ],