Skip to main content

Kafka Endpoints Reference

This is a reference for the Kafka Endpoints integration. The following properties are unique to this integration.

The endpoint configuration follows the same pattern as the Kafka moogdb module; there is a minimum set of required parameters and an optional set of parameters. See the descriptions below for detailed descriptions.

See the Apache Kafka documentation for details on SSL and SASL in Kafka.

Name

A unique name for the endpoint for use in Workflow Engine configuration.

Type

String

Required

Yes

Default

N/A

Servers

The list of brokers to connect to. A server configuration consists of the server name and port.

Server Name

Name of the server to connect to.

Type

String

Required

Yes

Default

N/A

Port

Port to communicate over.

Type

Integer

Required

Yes

Default

9092

Compression

The compression algorithm to use.

Type

One of: none, lz4, gzip, snappy.

Required

No

Default

none

SSL Configuration

See the Apache Kafka documentation for detailed descriptions of SSL configuration in Kafka.

USESSL

Whether to use SSL configuration. Check the box to enable.

Type

Boolean

Required

No

Default

Disabled

truststore.location

Path to the SSL truststore file.

Type

String

Required

Yes, if using SSL.

Default

N/A

truststore.password

Password for the SSL truststore file.

Type

String

Required

Yes, if using SSL.

Default

N/A

keystore.location

Path to the keystore file.

Type

String

Required

Yes, if using SSL.

Default

N/A

keystore.password

Password for the keystore file.

Type

String

Required

Yes, if using SSL.

Default

N/A

key.password

SSL certificate password.

Type

String

Required

Yes, if using SSL.

Default

N/A

endpoint.identification.algorithm

Method to validate server hosts, for example HTTPS. Leave blank to disable.

Type

String

Required

Yes, if using SSL.

Default

N/A

SASL Configuration

See the Apache Kafka documentation for detailed descriptions of SASL configuration in Kafka.

USESASL

Whether to use SASL. Check the box to enable.

Type

Boolean

Required

No

Default

Disabled

sasl.mechanism

SASL mechanism to use to authenticate.

Type

Choose from PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER

Required

Yes, if using SASL.

Default

N/A

security.protocol

Security protocol to use.

Type

Choose from SASL_SSL, SASL_PLAINTEXT

Required

Yes, if using SASL.

Default

N/A

sasl.jaas.config

Base module to use. Each requires additional parameters, defined under Additional SASL JaaS Config.

Type

One of: PLAIN, SCRAM, OAUTHBEARER, GSSAPI

Required

Yes, if using SASL.

Default

N/A

Additional SASL JaaS Config

Each JaaS base module (PLAIN, SCRAM, OAUTHBEARER, GSSAPI) requires additional parameters which you specify in this field.

The UI creates the string up to and including "required". This field must contain the remainder of the string after this. For example, if the connection was using "PLAIN", the "Additional SASL JaaS config" should contain the username and password:

username=John.Doe password=PASSWORD123

Example configurations for each base module are as follows:

Type

Example

PLAIN

org.apache.kafka.common.security.plain.PlainLoginModule required username=Jane.Doe password=Password123;

SCRAM

org.apache.kafka.common.security.scram.ScramLoginModule required username=John.Doe password=My_Password;

OAUTHBEARER

org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required unsecuredLoginPrincipalClaimName="sub" unsecuredLoginStringClaim_sub="Jane.Doe";

GSSAPI

com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="KEYTAB_FILE_PATH_HERE" principal="SERVICENAME/HOST@REALM";

Additional Properties

SASL Login refresh parameters. You only need to configure these if sasl.jaas.config is set to OAUTHBEARER. Defaults to Kafka recommended default values.

window.factor

Login refresh thread will sleep until the specified window factor relative to the credential's lifetime is reached, at which point it attempts to refresh the credential.

Type

Double

Required

Yes, if sasl.jaas.config is set to OAUTHBEARER.

Default

0.8

Valid Values

Between 0.5 (50%) and 1.0 (100%) inclusive.

window.jitter

Maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive.

Type

Double

Required

Yes, if sasl.jaas.config is set to OAUTHBEARER.

Default

Between 0 and 0.25 (25%) inclusive.

min.period.seconds

Desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential.

Type

Short

Required

Yes, if sasl.jaas.config is set to OAUTHBEARER.

Default

60

Valid Values

0 and 900

min.buffer.seconds

When refreshing a credential, amount of buffer time to maintain, in seconds, before credential expiration. If a refresh would otherwise occur closer to expiration than the number of buffer seconds, the refresh is moved up to maintain as much of the buffer time as possible. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential.

Type

Short

Required

Yes, if sasl.jaas.config is set to OAUTHBEARER.

Default

300

Valid Values

Between 0 and 3600

Kerberos Properties

You only need to configure these if sasl.jaas.config is set to GSSAPI.

kerberos.debug.log

Whether to use authentication debugging. Check to enable.

Type

Boolean

Required

Yes, if sasl.jaas.config is set to GSSAPI.

Default

Disabled

sasl.kerberos.service.name

Name of the Kerberos service.

Type

String

Required

Yes, if sasl.jaas.config is set to GSSAPI.

Default

N/A

kerberos.conf.file.path

Path to the Kerberos configuration file.

Type

String

Required

Yes, if sasl.jaas.config is set to GSSAPI.

Default

N/A