TroubleshootingΒΆ One table: the symptom, the line in the log or the API, the cause, the action. Symptom What you see Cause Action the endpoint does not start after the upgrade cannot start: a Solace connector older than 2.0.0 is deployed, status failed the connector 1.x is still deployed deploy a 2.x connector and redeploy the inbound endpoint every message waits three minutes and is then retried the deploy time WARN that the sequence contains no settlement operation, decided.TIMEOUT grows a 1.x sequence that relied on the acknowledgement on return add acknowledgeMessage to the sequence a message is retried although the sequence acknowledged it acknowledgeMessage answered settled false with reason LATE the decision came after settlementTimeout raise solace.settlementTimeout above the longest timeout in the sequence plus the 15 second sweep acknowledgeMessage answered settled true but the health shows settleFailed mode DEFERRED in the response in DEFERRED mode settled true means the decision reached the inbound endpoint, not that the broker took it. The flow was gone when the worker settled read the outcome from the health counters, never from the operation response. The broker redelivers the message messages behind a failing message reach the dead message queue unprocessed on a transacted queue a rollback followed by several redeliveries, the client's INFO line about MaxDeliveredUnackedMessagesPerFlow absent at bind max delivered unacknowledged per flow is not 1 on the queue set it to 1 on the transacted queue commit fails with no transaction in scope after an HTTP call TRANSACTION_COMPLETED or no transaction found, a publish after the call went out untransacted a connector owned transaction spans a non blocking call keep the transaction on one thread, or use the inbound owned transaction of use case 6 the outcome attribute of a publish prints empty in a fault sequence the fault code 700121 and the name UNKNOWN the attribute is not visible in the fault sequence read the fault code and name instead two different messages logged as message 1 the same message id on two flows the id in the log is the client's per flow message id tell the flows apart by the thread name in the log layout two nodes disconnect each other in a loop closed by peer repeated, redeliveries one client name on two nodes, clientNameStrict true or two processes on one host without node ids let the names carry the node, or set distinct node ids a publish reports success and nothing arrives ackStatus NOT_APPLICABLE, or ACK with the message gone Direct delivery, or a topic collected by a queue whose ingress is shut down with reject on discard at its default PERSISTENT delivery, reject on discard always on the collecting queue, read the queue's discard counters in SEMP a publish faults with NACK NACK: subcode 30, Spool Over Quota, 31 Queue Shutdown, 28 Publish ACL Denied, 32 Endpoint Shutdown the broker refused the message the subcode says why. With continueOnAckFailure true the response carries ackStatus NACK instead a publish faults with DEADLINE and outcome UNKNOWN DEADLINE 700119 the broker did not acknowledge within ackTimeout the message may or may not have arrived. A retry is at least once. The session is rebuilt on the next call a publish faults EXPIRED during an outage without a new connect attempt pooled sessions of 'name' are connecting, in use or being closed the connection already holds twice maxActiveConnections pooled sessions, counting the ones still connecting or being closed after the broker was lost nothing was sent, retry the call. The slots come back as the open connect attempts reach the connect timeout and the closes finish. Check the broker and the network sendRequest fails at once with a 503 No More Non-Durable Queue or Topic Endpoints or Max Client Queue and Topic Endpoint Flow Exceeded the broker's endpoint or flow limit is below maxConcurrentRequests lower maxConcurrentRequests or size the broker sendRequest faults REQUEST_CONCURRENCY_EXCEEDED 700117 more than maxConcurrentRequests requests in flight for longer than requestTimeout raise the bound within the broker's limits, or the callers' concurrency down sendRequest faults REQUEST_IN_TRANSACTION 700114 sendRequest inside a transaction move it out of the transaction poll faults POLL_UNSETTLED 700115 a second CLIENT poll while a lease is outstanding settle the first message with target POLL poll faults POLL_FLOW_LIMIT 700116 maxPollFlows flows cached and none idle fewer selectors, or a higher maxPollFlows the first poll of a queue faults POLL_ERROR poll session retired or poll flow went down during the flow bind the broker took the flow down or rebound it while the connector was binding it, for example a queue shutdown or a failover nothing was received, the next poll binds a fresh flow an OAuth2 endpoint never connects OAuth 2.0 based Authentication is supported only for SSL secure connections a plain tcp host a tcps host and a trust store the endpoint starts paused after a restart [Paused] in the task scheduled line, status stopped a deactivation through the management API persists in the registry and in the task store activate through the management API, or delete both registry entries for a clean start a graceful stop takes long the stop ends when a publish faults DEADLINE a publish waits on an unreachable broker and the server drains the transports first expected, bounded by ackTimeout and the server's graceful shutdown timeout the first connect of every connector session logs a WARN rebuilt (generation 1) a build older than 2.0.0 2.0.0 logs connected (generation 1) at INFO poll, sendRequest or browse returns a JSON document as a string attributes.contentType is text/plain the producer sent JSON without a content type, and 2.x does not guess declare the type on the producer, or set contentType application/json on the operation. browse keeps JSON items as text either way the inbound sequence sees a JSON document as a text body ContentType text/plain, the first body element is text the same declare the type on the producer, or set solace.contentType on the endpoint a sendRequest times out and the responder's sequence never ran the responder logs Received empty Solace message, no mediation context built, the requester faults REQUEST_REPLY_ERROR 700104 the request had headers and no body send at least a short body and put the values in user properties a publish faults PAYLOAD_ERROR 700124 the message body is binary (application/octet-stream) and messageType TEXT carries text, use messageType BYTES a binary body with messageType TEXT or XML publish it with BYTES sendRequest faults PAYLOAD_ERROR 700124 on the reply for example sendRequest: the body read as application/json is not valid JSON the reply could not be read with its content type the request was sent and the responder may have processed it. Do not retry blindly. Fix the content type, with the contentType parameter or on the responder, and make the request safe to repeat. See Fault 700124 by operation a sequence fails at a connector operation after a poll with overwriteBody true a parser error at the first operation with a responseVariable after the poll the polled document is broken, and that operation parses the body in full expected, see the known limits. Handle it in the fault sequence a document that does not build faults with 601000 ERROR_CODE 601000 instead of 700124 the parser's message contains the name of a platform parser exception, for example an element named JsonSyntaxException the payload is still rejected, with the same effects as 700124, see Fault 700124 by operation. Handle both codes in the fault sequence a reply carries the request's correlation id, not the one set on sendReply the requester sees the request's correlation id an explicit correlationId on sendReply is used only when the request has none rely on the request's correlation id temporary queues remain on the broker after a disconnect the request reply session's temporary queues listed for about a minute the broker's retention of a disconnected client's temporary queues nothing, they are removed by the broker