Kafka Inbound Endpoint Reference¶ Mandatory parameters¶ The following parameters are required when configuring Kafka Inbound Endpoint. Parameter Name Description bootstrap.servers The Kafka brokers listed as host1:port1 and host2:port2 key.deserializer Deserializer class for key that implements the org.apache.kafka.common.serialization.Deserializer interface. value.deserializer Deserializer class for value that implements the org.apache.kafka.common.serialization.Deserializer interface. group.id The consumer group ID. poll.timeout The max time to block in the consumer waiting for records. topic.name The name of the topic. contentType The content type of the message. Optional parameters¶ Parameter Name Description Default Value topic.pattern The name pattern of the topic. topic.partitions A comma-separated list of partitions for the topic to which the consumer is subscribed. enable.auto.commit Whether the consumer will automatically commit offsets periodically at the interval set by auto.commit.interval.ms. true auto.commit.interval.ms Offsets are committed automatically with a frequency controlled by the config. 5000 failure.retry.count The offset set to the same record until the failure retry count exceeds. -1 session.timeout.ms The timeout used to detect client failures when using Kafka’s group management facility. 10000 fetch.min.bytes The minimum amount of data the server should return for a fetch request. 1 heartbeat.interval.ms The expected time between heartbeats to the consumer coordinator when using Kafka’s group management facilities. 3000 max.partition.fetch.bytes The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. 1048576 key.delegate.deserializer Property name for the delegate key deserializer. value.delegate.deserializer Property name for the delegate value deserializer. schema.registry.url Comma-separated list of URLs for Schema Registry instances that can be used to register or look up schemas. basic.auth.credentials.source Specify how to pick the credentials for the Basic authentication header. basic.auth.user.info Specify the user info for the Basic authentication in the form of {username}:{password}. ssl.key.password The password of the private key in the key store file or the PEM key specified in ssl.keystore.key. ssl.keystore.location The location of the key store file. This is optional for client and can be used for two-way authentication for client. ssl.keystore.password The store password for the key store file. This is optional for client and only needed if ssl.keystore.location is configured. ssl.truststore.location The location of the trust store file. ssl.truststore.password The password for the trust store file. auto.offset.reset Defines what to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server. latest connections.max.idle.ms Close idle connections after the number of milliseconds specified by this config. 540000 exclude.internal.topics Whether internal topics matching a subscribed pattern should be excluded from the subscription. true fetch.max.bytes The maximum amount of data the server should return for a fetch request. 52428800 max.poll.interval.ms The maximum delay between invocations of poll() when using consumer group management. 300000 max.poll.records The maximum number of records returned in a single call to poll(). 500 partition.assignment.strategy A list of class names or class types, ordered by preference, of supported partition assignment strategies that the client will use to distribute partition ownership amongst consumer instances when group management is used. org.apache.kafka.clients.consumer.RangeAssignor receive.buffer.bytes The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. 65536 request.timeout.ms The configuration controls the maximum amount of time the client will wait for the response of a request. 305000 sasl.jaas.config JAAS login context parameters for SASL connections in the format used by JAAS configuration files. sasl.client.callback.handler.class The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface. sasl.login.class The fully qualified name of a class that implements the Login interface. sasl.kerberos.service.name The Kerberos principal name that Kafka runs as. sasl.mechanism SASL mechanism used for client connections. security.protocol Protocol used to communicate with brokers. send.buffer.bytes The size of the TCP send buffer (SO_SNDBUF) to use when sending data. 131072 ssl.enabled.protocols The list of protocols enabled for SSL connections. ssl.keystore.type The file format of the key store file. ssl.protocol The SSL protocol used to generate the SSLContext. ssl.provider The name of the security provider used for SSL connections. ssl.truststore.type The file format of the trust store file. check.crcs Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. true client.id An id string to pass to the server when making requests. fetch.max.wait.ms The maximum amount of time the server will block before answering the fetch request if there isn’t sufficient data to immediately satisfy the requirement given by fetch.min.bytes. 500 interceptor.classes A list of classes to use as interceptors. metadata.max.age.ms The period of time in milliseconds after which we force a refresh of metadata even if we haven’t seen any partition leadership changes to proactively discover any new brokers or partitions. 300000 metric.reporters A list of classes to use as metrics reporters. metrics.num.samples The number of samples maintained to compute metrics. 2 metrics.recording.level The highest recording level for metrics. INFO metrics.sample.window.ms The window of time a metrics sample is computed over. 30000 reconnect.backoff.ms The base amount of time to wait before attempting to reconnect to a given host. 50 retry.backoff.ms The amount of time to wait before attempting to retry a failed request to a given topic partition. 100 sasl.kerberos.kinit.cmd Kerberos kinit command path. sasl.kerberos.min.time.before.relogin Login thread sleep time between refresh attempts. sasl.kerberos.ticket.renew.jitter Percentage of random jitter added to the renewal time. sasl.kerberos.ticket.renew.window.factor Login thread will sleep until the specified window factor of time from last refresh to ticket’s expiry has been reached, at which time it will try to renew the ticket. ssl.cipher.suites A list of cipher suites. ssl.endpoint.identification.algorithm The endpoint identification algorithm to validate server hostname using server certificate. ssl.keymanager.algorithm The algorithm used by key manager factory for SSL connections. ssl.secure.random.implementation The SecureRandom PRNG implementation to use for SSL cryptography operations. ssl.trustmanager.algorithm The algorithm used by trust manager factory for SSL connections. sasl.oauthbearer.token.endpoint.url The URL for the OAuth/OIDC identity provider. sasl.oauthbearer.scope.claim.name The OAuth claim for the scope is often named “scope”, but this (optional) setting can provide a different name to use for the scope included in the JWT payload’s claims if the OAuth/OIDC provider uses a different name for that claim. sasl.login.callback.handler.class The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. sasl.login.connect.timeout.ms The (optional) value in milliseconds for the external authentication provider connection timeout. sasl.login.read.timeout.ms The (optional) value in milliseconds for the external authentication provider read timeout. sasl.login.retry.backoff.ms The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. sasl.login.retry.backoff.max.ms The (optional) value in milliseconds for the maximum wait between login attempts to the external authentication provider. kafka.header.prefix The prefix for Kafka headers. avro.use.logical.type.converters Whether to enable the use of logical type converters in Avro. This parameter is available only with Kafka Inbound Endpoint v1.2.2 and above. False batch.processing.enabled When set to true, all valid records returned by a single Kafka poll are bundled into one message and injected together into the inbound sequence, instead of being processed one record at a time. See Batch Mediation for details. false kafka.dlq.topic The Kafka topic to which poison pill records (records that could not be deserialized) are forwarded when batch processing is enabled. If not set, poison pills are logged and skipped without being forwarded. Only applicable when batch.processing.enabled is true. Manual offset control¶ Manual offset committing gives you explicit control over when Kafka offsets are committed, ensuring that messages are acknowledged only after successful processing. Follow the steps below to configure manual offset commit behavior in WSO2 Kafka Inbound Endpoint. Step 1: Disable auto commit¶ Disabling auto commit ensures that the WSO2 Kafka Inbound Endpoint does not commit the Kafka offset automatically, allowing your mediation logic to determine when an offset should be committed. To disable auto commit, set the following parameter in your Kafka inbound endpoint configuration: <parameter name="enable.auto.commit">false</parameter> Step 2: Use SET_ROLLBACK_ONLY to prevent commit on failure¶ Add the following property to your onError sequence to signal the Inbound Endpoint not to commit the current offset if message processing fails. This will cause the same message to be re-polled in the next cycle. <property name="SET_ROLLBACK_ONLY" value="true" scope="default" type="STRING"></property> Step 3: Configure retry behavior using failure.retry.count (optional)¶ Add this parameter to the inbound endpoint to control the retry behavior: <parameter name="failure.retry.count">N</parameter> N is the number of retry attempts for a failed message. If not set or set to -1 (default), the same message will be retried indefinitely. If the retry count is exceeded, the offset will move to the next record, and the failed message will be discarded. Note Manual offset committing is reliable only when used with sequential and synchronous mediation flows. Why? In parallel processing mode in the current connector architecture, the actual result of message processing may be ignored, and offsets may get committed under the assumption that the processing was successful. This may lead to message loss, especially if some records fail during message processing Additionally, asynchronous mediation flows (e.g., using non-blocking Call mediator, using Clone and Iterate mediators with sequential mode disabled, or Scatter-Gather and Foreach mediators with parallelExecution mode enabled) can cause offsets to be committed before message processing completes, which may lead to unprocessed messages being skipped. Recommendation To ensure reliable manual offset committing: Use sequential processing with synchronous flows wherever possible. For throughput and scalability: Configure your Kafka topic with multiple partitions. Run multiple Kafka Inbound Endpoints (within the same consumer group) to read from different partitions concurrently. Batch mediation¶ By default, the Kafka Inbound Endpoint processes one record at a time — each record polled from Kafka is injected into the inbound sequence as a separate message. Batch mediation changes this so that all valid records fetched by a single poll are bundled into one message and injected together. To enable it, set the following parameter: <parameter name="batch.processing.enabled">true</parameter> Note In batch mode, all records fetched by a poll are held in memory as part of the message context. Fine-tune max.poll.records according to your memory allocation. Message payload format¶ The content type of the injected message depends on the configured contentType. JSONXMLPlain text contentType = application/json All record values are combined into a JSON array. Each element is the raw record value. [{"id":1,"name":"Alice"},{"id":2,"name":"Bob"},{"id":3,"name":"Charlie"}] contentType = application/xml or text/xml All record values are wrapped in a <messages> root element. Each record value must be a valid XML fragment. <messages> <item><id>1</id><name>Alice</name></item> <item><id>2</id><name>Bob</name></item> </messages> contentType = text/plain or any other value Each record value is wrapped in a <text> element under a <messages> root. Special XML characters in the values are escaped automatically. The injected message content type is application/xml. <messages> <text xmlns="http://ws.apache.org/commons/ns/payload">hello world</text> <text xmlns="http://ws.apache.org/commons/ns/payload">Order <#1042> confirmed & shipped</text> </messages> Poison pill handling in batch mode¶ A poison pill is a record that Kafka could not deserialize. In batch mode, poison pills detected in a poll are skipped individually — they are not included in the batch payload injected to the inbound sequence. A warning is logged for each skipped poison pill. To forward poison pills to a Dead Letter Queue (DLQ) topic instead of logging and skipping them, configure the kafka.dlq.topic parameter: <parameter name="kafka.dlq.topic">my-dlq-topic</parameter> The DLQ record carries the original raw bytes as its value and the following provenance headers: Header Description kafka_dlt-original-topic The topic the poison pill was consumed from kafka_dlt-original-partition The partition number kafka_dlt-original-offset The offset of the poison pill record kafka_dlt-original-timestamp The record timestamp kafka_dlt-exception-message The deserialization error message kafka_dlt-exception-cause-message The root cause message (if available) Manual offset control and retry in batch mode¶ Batch mode supports the same manual offset control and retry configuration described in Manual offset control, with the following differences: Offsets are committed at the batch boundary — on success, the last offset of each partition in the poll + 1 is committed. If a poll returns only null or poison pill records, offsets are still advanced past the entire poll. On failure (with retries remaining), the consumer seeks back to the first valid record's offset per partition, so null records and poison pills already filtered out are not re-delivered. Once the retry count is exhausted, offsets are committed past the batch and an error is injected to the onError sequence with error code 700511 (RETRY_EXHAUSTED).