Skip to content

Observability

The health block of an endpoint answers three questions: is it consuming, if not why, and what happened to the messages. Read the status word first. Read the counters when the status or a symptom sends you there. The log carries a fixed set of lines an operator can grep for, and every fault of the connector has a code and a name.

The endpoint reports status, health and counters through the management API to ICP and to any monitor

The status word

GET https://host:9164/management/inbound-endpoints?inboundEndpointName=OrderListener, with the login token of the quick start, returns the endpoint with its status and a health block. ICP shows the same under Inbound Endpoints. Below the latest U2 update level of WSO2 Integrator: MI 4.6.0 or 4.7.0 the status is the task state.

This is the answer after the quick start, as the test bench returned it. The configuration and the parameters are cut, and the health keys are grouped for reading. The server returns them in no fixed order.

{
  "name": "OrderListener",
  "status": "active",
  "sequence": "orderProcessSeq",
  "error": "orderFaultSeq",
  "configuration": "<inboundEndpoint xmlns=\"http://ws.apache.org/ns/synapse\" name=\"OrderListener\" ...",
  "parameters": [ ... ],
  "health": {
    "status": "active",
    "lifecycle": "running",
    "connectivity": "connected",
    "connectivity.session": "connected",
    "connectivity.flow-1": "connected",
    "activity": "active",
    "activity.flow-1": "active",
    "generation": 1,

    "received": 4,
    "redelivered": 0,
    "retried": 6,
    "decided.ACK": 3,
    "decided.FAILED": 7,
    "decided.REJECTED": 0,
    "decided.TIMEOUT": 0,
    "decided.DRAIN": 0,
    "decided.INJECTION_REJECTED": 0,
    "decided.AUTOACK": 0,
    "decided.OTHER": 0,
    "applied.ACK": 3,
    "applied.FAILED": 0,
    "applied.REJECTED": 1,
    "applied.NONE": 0,
    "settled.ack": 3,
    "settled.failed": 0,
    "settled.rejected": 1,

    "inFlight": 0,
    "pending": 0,
    "pendingMax": 0,
    "freePermits": 1,

    "retainedUnsettled": 0,
    "settleFailed": 0,
    "dropped": 0,
    "leftByDrain": 0,
    "abandoned": 0,
    "rejectedSubmit": 0,
    "handlerFailed": 0,
    "closeRejected": 0,
    "degradedOutcomes": 0,

    "committed": 0,
    "rolledBack": 0,
    "rolledBackByBroker": 0,
    "unknown": 0,

    "lastErrorAt": 0
  }
}

How to read it. The endpoint is consuming: status active, lifecycle running, everything connected, generation 1. Four messages came in. Three were acknowledged, settled.ack 3. One failed on every attempt and was rejected to the dead message queue, settled.rejected 1. One more failed three times before the backend came back and then went through. Together that is 7 failed attempts and 6 retries. Nothing is waiting, inFlight 0 and pending 0, and every key in the last three groups is 0, as it should be.

Status What it means What to do
active consuming nothing
connected bound and healthy, but the broker has not said active or standby. A node that bound second on an exclusive queue reads this, and a Direct topic endpoint always does nothing
standby bound, and another node's flow has the queue nothing. It takes over when the broker makes its flow active
recovering starting, reconnecting, recovering, or draining for a pause or an undeploy wait. Alert when it lasts longer than the broker needs to come back
failed stopped after a refusal at start or after recovery.maxAttempts read lastError, fix the cause, then activate or redeploy the endpoint
stopped deactivated or undeployed, or the other node under task coordination nothing, unless you expected it to run

Six readings and what they mean

Each reading is a health block from the test bench, cut to the keys that matter.

All messages went through

received 1000, decided.ACK 1000, settled.ack 1000, inFlight 0, pending 0

Every message reached the sequence, the sequence acknowledged it, and the client library took the acknowledgement while the session was connected. Nothing is waiting. This is the normal reading.

The backend failed and one message went to the dead message queue

received 2, decided.ACK 1, decided.FAILED 4, retried 3, applied.REJECTED 1, settled.rejected 1

decided counts attempts and applied counts messages. The first message was acknowledged on its first attempt. The second failed four times, the first attempt and three retries, and was rejected to the dead message queue. Look at the backend, and at the message in the dead message queue.

The endpoint is busy

concurrentConsumers 8: inFlight 8, pending 29, pendingMax 29

All eight workers are busy and 29 messages wait for a worker. The endpoint stopped the flow, so the broker holds the rest of the queue. This is normal under load. One endpoint completes about N messages per backend response time, here about 16 a second against a 500 ms backend. More workers or a faster backend raise it.

The broker was away for a minute

status recovering, lifecycle running, connectivity reconnecting, generation 1

The Solace client lost the broker and reconnects by itself. When the broker is back the status returns to active, still generation 1. Nothing to do.

The queue was deleted

status recovering, lifecycle recovering, then status active, generation 6

The loss was final for the client, so the endpoint closed its connection and built a new one. Every attempt took the next generation number, and the fifth attempt succeeded once the queue existed again. The log line connection attempt <n> failed says why each attempt failed, here 503 Unknown Queue.

A message waits and then comes back

inFlight 1 for the whole settlementTimeout, then decided.TIMEOUT 1

The sequence ended without acknowledgeMessage or nackMessage, so the message waited for the settlement timeout and was then retried. Look for the deploy time WARN that names the sequence, and add the missing settlement on that path.

Set alerts on these

  • status failed.
  • status recovering for longer than your broker needs to restart.
  • handlerFailed or closeRejected above 0. Both mean a defect. Send the log to support.
  • retainedUnsettled growing. These messages stay unsettled on the broker until the endpoint reconnects, and they count against the queue's limit of delivered unacknowledged messages.
  • dropped growing on a Direct topic. Those messages are lost.

Good to know

  • The counters count since the deploy or the server start. A recovery does not reset them. inFlight, pending, pendingMax and freePermits belong to the current generation.
  • settled means the client library took the settle while the session was connected. The broker's own counters are on the broker.
  • After a stop, read only the status. The other words keep their last values.
  • lastErrorAt is the time of the last error in milliseconds since 1970, and lastError holds its text. Both stay after a recovery.
  • Health never triggers a recovery. It reports.

Every key in one table

Group Keys What they say
words status, lifecycle, connectivity, activity, and the same per session and per flow the state of the endpoint, of its connection, and the broker's active flow indication
generation generation the last connection attempt that reached running. It goes up by one per attempt
messages received, decided.*, applied.*, settled.*, retried, redelivered what came in, what was decided per attempt, what was done per message, and which settlements the client library took
load inFlight, pending, pendingMax, freePermits messages on a worker, messages waiting for one, the highest wait, the free workers
left to the broker or lost retainedUnsettled, settleFailed, dropped, leftByDrain, abandoned, rejectedSubmit messages the broker delivers again, except dropped on a Direct topic
defects and limits handlerFailed, closeRejected, degradedOutcomes 0 in normal running. degradedOutcomes counts the outcomes the broker does not support
transacted committed, rolledBack, rolledBackByBroker, unknown the outcome of each transaction
errors lastErrorAt, lastError the last error

The log line index

Event Line
start starting generation <n>, then the started line with the queue
client reconnect session reconnecting, session reconnected
recovery generation <n> is recovering, connection attempt <n> failed; next attempt in <ms> ms
retry message <id> attempt <n> failed; retry in <ms> ms, exhausted <n> in-place retries; rejecting (DMQ when eligible)
flow loss parked messages discarded, with the count
deploy time WARNs a manual acknowledgement sequence with no settlement operation, a strict client name two nodes will share, a transacted endpoint naming the three queue settings
connector session session <name> connected (generation 1), session <name> rebuilt (generation <n>), session <name> reconnecting, session <name> DOWN
poll poll flow <queue> closed (<why>), polled message <id> was not settled within pollAckTimeout

The fault codes of the connector

Name Code Family
CONNECTION_ERROR 700101 refused, no session could be had
PUBLISH_ERROR 700102 safe, before the send
INVALID_CONFIG 700103 refused
REQUEST_REPLY_ERROR, POLL_ERROR 700104, 700108 the operation's own failure
TRANSACTION_ACTIVE 700111 a pooled session with an open transaction cannot be rebuilt
TRANSACTION_COMPLETED 700112 a command reached a transaction that had ended
INBOUND_TRANSACTION 700113 an operation not allowed inside an inbound owned transaction
REQUEST_IN_TRANSACTION 700114 sendRequest inside any transaction
POLL_UNSETTLED, POLL_FLOW_LIMIT 700115, 700116 a CLIENT poll with a lease outstanding, no idle flow
REQUEST_CONCURRENCY_EXCEEDED 700117 maxConcurrentRequests reached within requestTimeout
DEADLINE_BEFORE_SEND 700118 safe, the deadline passed before the send
DEADLINE 700119 uncertain, no acknowledgement within the deadline, outcome UNKNOWN
EXPIRED 700120 safe, the deadline passed in a wait or a connect
UNKNOWN 700121 uncertain, an exception inside or after the send
NACK 700122 refused by the broker, with the subcode and the phrase
ROLLED_BACK 700123 the broker rolled the transaction back
PAYLOAD_ERROR 700124 the body could not be read, built or written, or its charset or user properties are wrong. publishMessage, sendReply and the request side of sendRequest fault before any broker call, so nothing was sent. A sendRequest that faults on its reply has already sent the request, so a retry is not safe by itself. A poll with ackMode AUTO has already taken the message off the queue, with CLIENT the lease is still held. Every case is in Fault 700124 by operation. When the parser's message contains the name of a platform parser exception, the platform reports 601000 instead
TRANSACTION_CONNECTION_MISMATCH 700125 an operation on another connection inside a connector owned transaction
TRANSACTION_MESSAGE_LIMIT 700126 a second consuming poll in one connector owned transaction