Event-driven Integrations¶ Introduction¶ In addition to traditional API-based integrations, modern systems often need to respond to asynchronous events from external systems such as message brokers and event buses. WSO2 Integrator: MI allows you to build event-driven integrations that react to incoming events, transform data, and publish messages in real time. In this tutorial, you'll learn how to use Kafka with MI to build a message-driven integration that listens to a Kafka topic, transforms the received data, and optionally publishes the result to another topic. New to WSO2 Integrator: MI? If you're just getting started, we recommend completing the Build your first Integration tutorial first. It will help you understand key integration concepts and get familiar with the developer tool features before diving into event-driven flows. Prerequisites¶ You need Visual Studio Code (VS Code) with the WSO2 Integrator: MI for VS Code extension installed. The MI for VS Code extension is the official developer tool for designing, developing, and testing integration solutions with WSO2 Integrator: MI. Info See the Install WSO2 Integrator: MI for VS Code documentation to learn how to install WSO2 Integrator: MI for VS Code. What you’ll learn¶ By following this tutorial, you will gain hands-on experience in: Setting up a Kafka listener in WSO2 MI. Consuming and processing events from a Kafka topic. Transforming incoming messages using the Payload mediator. Publishing transformed messages to another Kafka topic using the Kafka connector. What you'll build¶ Let’s try a simple scenario where WSO2 Integrator: MI listens to a Kafka topic named bank-transactions, transforms each incoming bank transaction event, and publishes the normalized message to another topic called bank-audit. This demonstrates how you can build real-time, event-driven integrations without using any APIs. Step 1 - Setup Kafka¶ Already have Kafka running? If Kafka is already installed and running in your environment (locally or remotely), you can skip the installation step and continue from Create the Topics. To start building an event-driven integration, you'll first need a Kafka broker running. Kafka acts as a message hub, where producers publish events and consumers subscribe to them. In this tutorial, you will use the official Apache Kafka Docker image (v3.7.2) for simplicity and ease of setup. Prerequisites¶ Docker installed and running. Start Kafka using Docker¶ Open a command line prompt on your machine and run the following command. This will pull the official Kafka image (if it’s not already cached) and start a Kafka broker that listens on port 9092. docker run --rm --name kafka -p 9092:9092 apache/kafka:3.7.2 You should see a message like Kafka Server started indicating that the broker is up and running. Create the topics¶ Kafka uses topics to organize and group related messages. In this tutorial, we'll create a topic named bank-transactions, which the Kafka listener in WSO2 Integrator: MI will subscribe to, and a topic named bank-audit, to which the WSO2 Integrator: MI will publish the transformed messages. Open a new command line prompt on your machine and run the following command to create the bank-transactions topic. docker exec -it kafka /opt/kafka/bin/kafka-topics.sh --create --topic bank-transactions --bootstrap-server localhost:9092 --partitions 1 Upon successful topic creation, the following message will be displayed: Created topic bank-transactions. Then, create the bank-audit topic using the following command. docker exec -it kafka /opt/kafka/bin/kafka-topics.sh --create --topic bank-audit --bootstrap-server localhost:9092 --partitions 1 Step 2 - Create a new integration project¶ To develop the above scenario, let's get started with creating an integration project in the WSO2 Integrator: MI extension installed VS Code. Launch VS Code with the WSO2 Integrator: MI extension installed. Click on the WSO2 Integrator: MI icon on the Activity Bar of the VS Code editor. Click Create New Project on Design View. For more options for creating a new integration project, see Create an Integration Project. In the Project Creation Form, enter EventIntegration as the Project Name. Ensure 4.7.0 is selected as the WSO2 Integrator: MI runtime version. Provide a location for the integration project under Project Directory. Click Create. Once you click Create, the Add Artifact pane will be opened. Step 3 - Create an event listener¶ Now that the integration project is ready, let's create a Kafka event listener. In the Add Artifact interface, under Create an Integration, click Event Integration. This will open the list of event integrations available in WSO2 Integrator: MI. Select Kafka (Inbound) from the list. In the confirmation pane, click Yes to add the required dependencies. In the Create Event Integration form, enter BankTransactionListener as the Event Integration Name and bank-transactions as the Topic Name. Note In this tutorial, you will use the local Kafka broker configured in Step 1 – Set Up Kafka. If you plan to use a different Kafka broker, update the Kafka Servers field accordingly. You can use the default values for the remaining fields. For advanced configurations, refer to the Kafka listener documentation. Finally, click Create to add the Kafka listener to your integration project. Step 4 - Design the integration¶ Now it's time to design your integration. This defines the underlying logic that will be executed when a Kafka message is received. In this scenario, the message needs to be transformed and then published to another Kafka topic. Follow the steps below to complete the integration flow. Click on the Start node on the canvas to set an input payload for the integration flow. Note Setting an input payload for the integration flow is not mandatory. However, it is recommended, as it will be used to enable expression suggestions, which you will explore in later steps of this tutorial. Click Add Request, provide the following JSON payload, then click Add. Finally, click Save to complete the input payload setup. { "id":"TXN001", "type":"deposit", "accountId":"ACC123", "amount":500 } Click on the + icon on the canvas to open the Mediator Palette. Since this is an event-driven integration, we’ll log the incoming message to ensure that events are being received by the WSO2 Integrator: MI. Under Mediators, select the Log mediator. Tip You can use the Log mediator to print the message content to the server logs. This is helpful for checking what data is being received and for debugging during development. You will use the following template to log the message. It logs the incoming message payload along with the text Kafka event received. You can either copy and paste the template or use the inline expression editor to construct it. Kafka event received, data = ${payload} Click Add to insert the Log mediator into the integration flow. Click on the + icon located just after the Log mediator to open the Mediator Palette to add a Payload mediator to transform the Kafka message before publishing the result to the bank-audit topic. Under Mediators, select the Payload mediator. In the Payload box, enter the following sample JSON object. In the next step, you will use an inline expression to extract the JSON payload from the Kafka message and insert it into this template. { "eventType": "transaction", "source": "bank-transactions", "data": } In the JSON object, place your cursor next to "data":, then click on the inline expression editor () icon. From the editor, select Payload → payload, and click Add to insert the inline expression into the JSON object. Finally, click Add to insert the Payload mediator into the integration flow. Click on the + icon just after the Payload mediator to open the Mediator Palette. In the next steps, you will use the Kafka connector to publish the transformed message to the bank-audit topic. Search for kafka in the Mediator Palette, then click the download () icon to add the Kafka connector to the project. In the confirmation pane, select Yes to add the required dependencies. Once the connector is downloaded, select the PublishMessages operation from the Mediator Palette. Click + Add new connection to create a new connection. Select KAFKA and fill in the following details to create a connection to the Kafka broker. Then, click Add in the Add New Connection form to complete the connection setup. In the next step, you will configure the operational-level parameters, such as the topic to which the messages will be published. Property Value Connection Name KafkaConnection Bootstrap Servers localhost:9092 Key Serializer Class org.apache.kafka.common.serialization.StringSerializer Value Serializer Class org.apache.kafka.common.serialization.StringSerializer Note In this tutorial, you will use the local Kafka broker configured in Step 1 – Set Up Kafka. If you plan to use a different Kafka broker, update the Bootstrap Servers field accordingly. You can keep the default values for the remaining fields. For more information, refer to the Kafka connector documentation. Fill in the following details in the Add PublishMessages pane, then click Submit to add the operation to the integration flow. Property Value Topic bank-audit Partition Number 0 Content-Type application/json You have successfully completed the integration flow to listen to Kafka events and publish the transformed message to another Kafka topic. For reference, you can review the configured event listener, sequence and Kafka connection. BankTransactionListener Info You can view the source view by clicking on the Show Source (</>) icon located in the top right corner of the VS Code. Design ViewSource View <?xml version="1.0" encoding="UTF-8"?> <inboundEndpoint name="BankTransactionListener" class="org.wso2.carbon.inbound.kafka.KafkaMessageConsumer" sequence="BankTransactionListener-inboundSequence" onError="BankTransactionListener-inboundErrorSequence" suspend="false"> <parameters xmlns="http://ws.apache.org/ns/synapse"> <parameter name="generateSequences">true</parameter> <parameter name="interval">100</parameter> <parameter name="sequential">true</parameter> <parameter name="coordination">true</parameter> <parameter name="bootstrap.servers">localhost:9092</parameter> <parameter name="topic.name">bank-transactions</parameter> <parameter name="group.id">group1</parameter> <parameter name="contentType">application/json</parameter> <parameter name="poll.timeout">1000</parameter> <parameter name="key.deserializer">org.apache.kafka.common.serialization.StringDeserializer</parameter> <parameter name="value.deserializer">org.apache.kafka.common.serialization.StringDeserializer</parameter> <parameter name="enable.auto.commit">true</parameter> <parameter name="auto.commit.interval.ms">5000</parameter> <parameter name="auto.offset.reset">latest</parameter> <parameter name="exclude.internal.topics">true</parameter> <parameter name="check.crcs">true</parameter> <parameter name="partition.assignment.strategy">org.apache.kafka.clients.consumer.RangeAssignor</parameter> <parameter name="max.poll.interval.ms">300000</parameter> <parameter name="max.poll.records">500</parameter> <parameter name="fetch.max.wait.ms">500</parameter> <parameter name="receive.buffer.bytes">65536</parameter> <parameter name="send.buffer.bytes">131072</parameter> <parameter name="request.timeout.ms">305000</parameter> <parameter name="reconnect.backoff.ms">50</parameter> <parameter name="retry.backoff.ms">100</parameter> <parameter name="connections.max.idle.ms">540000</parameter> <parameter name="security.protocol">PLAINTEXT</parameter> <parameter name="metrics.num.samples">2</parameter> <parameter name="metrics.recording.level">INFO</parameter> <parameter name="metrics.sample.window.ms">30000</parameter> </parameters> </inboundEndpoint> BankTransactionListener-inboundSequence Info You can view the source view by clicking on the Show Source (</>) icon located in the top right corner of the VS Code. Design ViewSource View <?xml version="1.0" encoding="UTF-8"?> <sequence name="BankTransactionListener-inboundSequence" trace="disable" xmlns="http://ws.apache.org/ns/synapse"> <log category="INFO" logMessageID="false" logFullPayload="false"> <message>Kafka event received, data = ${payload}</message> </log> <payloadFactory media-type="json" template-type="default"> <format>{ "eventType": "transaction", "source": "bank-transactions", "data": ${payload} } </format> </payloadFactory> <kafkaTransport.publishMessages configKey="KafkaConnection"> <topic>bank-audit</topic> <partitionNo>0</partitionNo> <contentType>application/json</contentType> <key></key> <keySchema></keySchema> <keySchemaId></keySchemaId> <valueSchema></valueSchema> <valueSchemaId></valueSchemaId> <keySchemaSubject></keySchemaSubject> <keySchemaVersion></keySchemaVersion> <keySchemaSoftDeleted>false</keySchemaSoftDeleted> <valueSchemaSubject></valueSchemaSubject> <valueSchemaVersion></valueSchemaVersion> <valueSchemaSoftDeleted>false</valueSchemaSoftDeleted> <value></value> </kafkaTransport.publishMessages> </sequence> Kafka Connection Info You can view the source view by clicking on the Show Source (</>) icon located in the top right corner of the VS Code. Design ViewSource View <?xml version="1.0" encoding="UTF-8"?> <localEntry key="KafkaConnection" xmlns="http://ws.apache.org/ns/synapse"> <kafkaTransport.init> <connectionType>KAFKA</connectionType> <bootstrapServers>localhost:9092</bootstrapServers> <keySerializerClass>org.apache.kafka.common.serialization.StringSerializer</keySerializerClass> <valueSerializerClass>org.apache.kafka.common.serialization.StringSerializer</valueSerializerClass> <poolingEnabled>false</poolingEnabled> <name>KafkaConnection</name> </kafkaTransport.init> </localEntry> Step 5 - Run the integration¶ Now that you have completed the integration, it's time to deploy it to the WSO2 Integrator: MI server runtime. Click the Build and Run icon located in the top right corner of VS Code. Step 6 - Test the integration¶ Once the WSO2 Integrator: MI runtime is up and running, follow the steps below to publish a message and verify the integration. Open a new command line prompt on your machine and run the following command to tail the logs for the bank-audit topic. The WSO2 Integrator: MI will publish the transformed messages to this topic. docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic bank-audit --from-beginning Open another command line prompt and run the following command to launch the Kafka producer and publish a message to the bank-transactions topic. docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic bank-transactions After the producer starts, you'll see a > prompt. Any line of text you type after that will be sent to the Kafka topic when you press Enter. Copy and paste the following JSON payload and press Enter to publish the message to the topic. { "id":"TXN011", "type":"withdraw", "accountId":"ACC456", "amount":1500 } You won’t see any confirmation, but the message is now sent to the bank-transactions topic. Check the logs of the WSO2 Integrator: MI runtime in VS Code. You should see a log entry similar to the following. [2025-04-03 09:49:23,029] INFO {LogMediator} - Kafka event received, data = {"id":"TXN011","type":"withdraw","accountId":"ACC456","amount":1500} Verify that the command line prompt monitoring the bank-audit topic now displays a new entry with the transformed message. You may see a message similar to the following: { "eventType":"transaction", "source":"bank-transactions", "data":{ "id":"TXN011", "type":"withdraw", "accountId":"ACC456", "amount":1500 } } Congratulations! You’ve now learned how to create an event-driven integration using Kafka and apply message transformation using WSO2 Integrator: MI. This is a powerful pattern for building scalable, real-time systems. What's next?¶ Now that you’ve built your first event-driven flow, you can explore how to improve the integration by, Connecting to SaaS platforms (e.g., sending notifications via email). Integrating with external systems like databases or Backend services. Routing events based on custom logic or content. Enriching messages with additional data before forwarding.