Kafka Connector Example¶ Given below is a sample scenario that demonstrates how to send messages to a Kafka broker via Kafka topics. The publishMessages operation allows you to publish messages to the Kafka brokers via Kafka topics. What you'll build¶ Given below is a sample API that illustrates how you can connect to a Kafka broker with the init operation and then use the publishMessages operation to publish messages via the topic. It exposes Kafka functionalities as a RESTful service. Users can invoke the API using HTTP/HTTPs with the required information. API has the context /publishMessages. It will publish messages via the topic to the Kafka server. The following diagram illustrates all the required functionality of the Kafka service that you are going to build. If you do not want to configure this yourself, you can simply get the project and run it. Set up kafka¶ Before you begin, set up Kafka by following the instructions in Setting up Kafka. Set up the integration project¶ Follow the steps below to set up the integration project using the WSO2 Integrator: MI Visual Studio Code extension. Create a new project¶ Follow the steps in the create integration project guide to set up WSO2 MI and create a new integration project. Use a suitable Project Name for your integration. Create a connection¶ In the Design View, click the +View More button and select Connection. In the search bar, type Kafka and select the Kafka connector from the list. In the Connection Configuration pane, enter the following required information: Connection Name - KafkaConnection Connection Type - kafka Bootstrap Servers - localhost:9092 Key Serializer Class - org.apache.kafka.common.serialization.StringSerializer Value Serializer Class - org.apache.kafka.common.serialization.StringSerializer Pooling Enabled - false Create an API¶ Click on the API button in create an integration project pane. Enter the API Name as KafkaTransport and the Context as /publishMessages, then click Create. To add the Kafka connector: In the Design View, click the + button. In the Mediator section, search for Kafka. Select the Kafka connector and click Download Implement the API¶ Go to the Source View of the API by clicking on the <> icon in the top right corner of the Design View. Copy and paste the following code in the Source View of the API. Source view of the implemented resource <?xml version="1.0" encoding="UTF-8"?> <api context="/publishMessages" name="KafkaTransport" xmlns="http://ws.apache.org/ns/synapse"> <resource methods="POST" uri-template="/"> <inSequence> <kafkaTransport.publishMessages configKey="KafkaConnection"> <topic>test</topic> <partitionNo>0</partitionNo> <key></key> <keySchema></keySchema> <keySchemaId></keySchemaId> <keySchemaMetadata></keySchemaMetadata> <valueSchema></valueSchema> <valueSchemaId></valueSchemaId> <keySchemaSubject></keySchemaSubject> <keySchemaVersion></keySchemaVersion> <keySchemaSoftDeleted>false</keySchemaSoftDeleted> <valueSchemaSubject></valueSchemaSubject> <valueSchemaVersion></valueSchemaVersion> <valueSchemaMetadata></valueSchemaMetadata> <valueSchemaSoftDeleted>false</valueSchemaSoftDeleted> <value></value> <forwardExistingHeaders>None</forwardExistingHeaders> <customHeaders>[]</customHeaders> <customHeaderExpression></customHeaderExpression> </kafkaTransport.publishMessages> </inSequence> <faultSequence> </faultSequence> </resource> </api> Get the project¶ You can download the ZIP file and extract the contents to get the project code. Deployment¶ To deploy and run the project, refer to the Build and Run guide. Test¶ Invoke the API as shown below using the MI VSCode Extension. Create a topic: Let’s create a topic named test with a single partition and only one replica. Navigate to the <KAFKA_HOME> and run following command. bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test Sample request: Content-Type: application/json Request body: { "name": "sample" } Expected response: Navigate to the <KAFKA_HOME> and run the following command to verify the messages: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning See the following message content: {"name":"sample"} What's next¶ To customize this example for your own scenario, see Kafka Connector Configuration documentation.