Setting up Kafka

For Kafka inbound endpoint

To configure the Kafka inbound endpoint, copy the following orbit bundles to the <MI_HOME>/dropins directory.

• kafka-avro-serializer
• kafka-schema-serializer
• kafka-schema-registry-client

Copy the following additional client libraries to the <MI_HOME>/lib directory.

• kafka-clients-3.7.0.jar
• avro-1.11.3.jar

Note

If you are also using the Kafka connector, copy the necessary dependencies as outlined in the following section.

For connector version 3.2.0 and later

To use the Kafka connector, download and install Apache Kafka. Before you start configuring Kafka you also need the integration runtime and we refer to that location as <PRODUCT_HOME>.

Note

• The Kafka connector has been tested with the Kafka versions listed below.
• To access all downloadable versions of Kafka, see the Kafka documentation.
• The recommended version of Java is 11.

Kafka version 2.8.2

1. To configure the Kafka connector with Kafka version 2.8.2, copy the following client libraries from the <KAFKA_HOME>/lib directory to the <MI_HOME>/lib directory.

   • kafka_2.12-2.8.2.jar
   • kafka-clients-2.8.2.jar
   • metrics-core-2.2.0.jar
   • scala-library-2.12.13jar
   • zkclient-0.10.jar
   • zookeeper-3.5.9.jar

2. Copy the following additional client libraries to the <MI_HOME>/lib directory (can be copied from the Confluent platform):

   • avro-1.11.3.jar
   • common-config-5.4.0.jar
   • common-utils-5.4.0.jar
   • kafka-avro-serializer-5.3.0.jar
   • kafka-schema-registry-client-5.3.0.jar

Kafka version 3.7.0

1. To configure the Kafka connector with Kafka version 3.7.0, copy the following orbit bundles to the <MI_HOME>/dropins directory.

   • kafka-avro-serializer
   • kafka-schema-serializer
   • kafka-schema-registry-client

2. Copy the following client libraries from the <KAFKA_HOME>/lib directory to the <MI_HOME>/lib directory.

   • kafka_2.13-3.7.0.jar
   • kafka-clients-3.7.0.jar
   • metrics-core-2.2.0.jar
   • scala-library-2.13.2jar
   • zkclient-0.10.jar
   • zookeeper-3.8.3.jar

3. Copy the following additional client libraries to the <MI_HOME>/lib directory (can be copied from the Confluent platform):

   • avro-1.11.3.jar
   • common-config-7.6.0.jar
   • common-utils-7.6.0.jar

4. Navigate to <KAFKA_HOME> and run the following command to start the ZooKeeper server:

   bin/zookeeper-server-start.sh config/zookeeper.properties
   

5. From the <KAFKA_HOME> directory, run the following command to start the Kafka server:

   bin/kafka-server-start.sh config/server.properties
   

Now that you have connected to Kafka, you can start publishing and consuming messages using the Kafka brokers. For more information, see Publishing Messages using Kafka and Consuming Messages using Kafka.

For connector version 3.1.x and below

To use the Kafka connector, download and install Apache Kafka. Before you start configuring the Kafka you also need the integration runtime and we refer to that location as <PRODUCT_HOME>.

Note

• The recommended version of Kafka is Kafka_2.11-2.2.1.
• To access all downloadable versions of Kafka, see the Kafka documentation.
• The recommended version of Java is 1.8.

1. To configure the Kafka connector, copy the following client libraries from the <KAFKA_HOME>/lib directory to the <MI_HOME>/lib directory.

   • kafka_2.11-2.2.1.jar
   • kafka-clients-1.0.0.jar
   • metrics-core-2.2.0.jar
   • scala-library-2.12.3.jar
   • zkclient-0.10.jar
   • zookeeper-3.4.10.jar

2. Copy the following additional client libraries to the <MI_HOME>/lib directory (can be copied from the Confluent platform):

   • avro-1.8.1.jar
   • common-config-5.4.0.jar
   • common-utils-5.4.0.jar
   • kafka-avro-serializer-5.3.0.jar
   • kafka-schema-registry-client-5.3.0.jar

3. Navigate to <KAFKA_HOME> and run the following command to start the ZooKeeper server:

   bin/zookeeper-server-start.sh config/zookeeper.properties
   

4. From the <KAFKA_HOME> directory, run the following command to start the Kafka server:

   bin/kafka-server-start.sh config/server.properties
   

Now that you have connected to Kafka, you can start publishing and consuming messages using the Kafka brokers. For more information, see Publishing Messages using Kafka and Consuming Messages using Kafka.