Monitor Correlation Logs¶ Product observability enables rapid debugging of product issues. The WSO2 Integrator: MI enables observability using Correlation logs. Correlation logs allow you to monitor individual HTTP requests from the point that a message is received by the WSO2 Integrator: MI until the corresponding response message is sent back to the original message sender. That is, the complete round trip of an HTTP message (client →WSO2 Integrator: MI → back-end →WSO2 Integrator: MI → client) can be tracked and analyzed using a log file. When Correlation logs are enabled for the WSO2 Integrator: MI server, a separate log file named correlation.log is created in the <MI_HOME>/repository/logs/ directory. Every HTTP message that flows between the WSO2 Integrator: MI and external clients goes through several state transitions. A new log entry is created in the correlation.log file corresponding to the state changes in the round trip of a single HTTP request. A Correlation ID assigned to the incoming HTTP request is assigned to all the log entries corresponding to the request. Therefore, you can use this Correlation ID to easily locate the logs relevant to the round trip of a specific HTTP request and, thereby, analyze the behavior of the message flow. Enable Correlation logs¶ At the server start-up¶ You can enable Correlation logging by passing a system property. If you want Correlation logs to be enabled every time the server starts, add the following system property to the product start-up script (stored in the <MI_HOME>/bin/ directory) and set it to true. -DenableCorrelationLogs=true \ Alternatively, you can pass the system property at the time of starting the server by executing the following command: On Linux/MacOS/CentOS: sh micro-integrator.sh -DenableCorrelationLogs=true On Windows: micro-integrator.bat -DenableCorrelationLogs=true Now when you start the WSO2 Integrator: MI, the correlation.log file is created in the <MI_HOME>/repository/logs/ directory. During the runtime¶ You can enable correlation logging by invoking the configs resource of the Management API. For more information, see enable/disable correlation logs using the Management API. Alternatively, you can enable correlation logging using the Integration Control Plane. You cannot disable the correlation logs during runtime if the correlation logs are enabled using the system property. Tip You can customize the behavior of correlation logging, including the name of the HTTP header used for the Correlation ID, the log file location, and other related settings. For full configuration details, see Configure Correlation Logs. Send an HTTP request with a Correlation ID¶ When a client sends an HTTP request to the WSO2 Integrator: MI, a Correlation ID can be included using the configured correlation header. By default, this header is set to activityid. Note If the client does not include a Correlation ID in the request, the WSO2 Integrator: MI will automatically generate one and assign it to the request. The Correlation ID, whether provided by the client or generated by the WSO2 Integrator: MI, is added to all log entries related to the same request. This helps you trace the complete flow of the request across the system. To change the default correlation header, see Configure Correlation Logs. The following is a sample POST request sent using the curl client. Note that the Correlation ID is included in the request header. curl -X POST --data @request.json http://localhost:8280/healthcare/categories/surgery/reserve -H "Content-Type:application/json" -H "activityid:correlationID" Access Correlation logs¶ If you know the Correlation ID of the HTTP request that you want to analyze, you can isolate the relevant logs as explained below. Open a terminal and navigate to the <MI_HOME>/repository/logs/ directory where the correlation.log file is saved. Execute the following command with the required Correlation ID. Replace <correlation_ID> with the required value. cat correlation.log | grep "<correlation_ID>" Shown below is an example of Correlation log entries corresponding to the round trip of a single HTTP request. 2021-11-30 15:27:27,262|correlationID|HTTP-Listener I/O dispatcher-5|0|HTTP State Transition|http-incoming-17|POST|/healthcare/categories/surgery/reserve|REQUEST_HEAD 2021-11-30 15:27:27,262|correlationID|HTTP-Listener I/O dispatcher-5|0|HTTP State Transition|http-incoming-17|POST|/healthcare/categories/surgery/reserve|REQUEST_BODY 2021-11-30 15:27:27,263|correlationID|HTTP-Listener I/O dispatcher-5|1|HTTP State Transition|http-incoming-17|POST|/healthcare/categories/surgery/reserve|REQUEST_DONE 2021-11-30 15:27:27,265|correlationID|HTTP-Sender I/O dispatcher-4|42173|HTTP State Transition|http-outgoing-4|POST|http://localhost:9090/grandoaks/categories/surgery/reserve|REQUEST_HEAD 2021-11-30 15:27:27,265|correlationID|HTTP-Sender I/O dispatcher-4|0|HTTP State Transition|http-outgoing-4|POST|http://localhost:9090/grandoaks/categories/surgery/reserve|REQUEST_DONE 2021-11-30 15:27:27,267|correlationID|HTTP-Sender I/O dispatcher-4|2 |HTTP|sourhttp://localhost:9090/grandoaks/categories/surgery/reserve|BACKEND LATENCY 2021-11-30 15:27:27,267|correlationID|HTTP-Sender I/O dispatcher-4|2|HTTP State Transition|http-outgoing-4|POST|http://localhost:9090/grandoaks/categories/surgery/reserve|RESPONSE_HEAD 2021-11-30 15:27:27,267|correlationID|HTTP-Sender I/O dispatcher-4|0|HTTP State Transition|http-outgoing-4|POST|http://localhost:9090/grandoaks/categories/surgery/reserve|RESPONSE_BODY 2021-11-30 15:27:27,267|correlationID|HTTP-Sender I/O dispatcher-4|0|HTTP State Transition|http-outgoing-4|POST|http://localhost:9090/grandoaks/categories/surgery/reserve|RESPONSE_DONE 2021-11-30 15:27:27,269|correlationID|HTTP-Listener I/O dispatcher-5|6|HTTP State Transition|http-incoming-17|POST|/healthcare/categories/surgery/reserve|RESPONSE_HEAD 2021-11-30 15:27:27,269|correlationID|HTTP-Listener I/O dispatcher-5|0|HTTP State Transition|http-incoming-17|POST|/healthcare/categories/surgery/reserve|RESPONSE_BODY 2021-11-30 15:27:27,269|correlationID|HTTP-Listener I/O dispatcher-5|0|HTTP State Transition|http-incoming-17|POST|/healthcare/categories/surgery/reserve|RESPONSE_DONE 2021-11-30 15:27:27,269|correlationID|HTTP-Listener I/O dispatcher-5|7|HTTP|http-incoming-17|POST|/healthcare/categories/surgery/reserve|ROUND-TRIP LATENCY Read Correlation Logs¶ The format of a correlation log entry is shown below, followed by an example. Log PatternExample Log Time Stamp|Correlation ID|Thread name|Duration|Call type|Connection name|Method type|Connection URL|HTTP state 2021-10-26 17:34:40,464|de461a83-fc74-4660-93ed-1b609ecfac23|HTTP-Listener I/O dispatcher-3|535|HTTP|http-incoming-3|GET|/api/querydoctor/surgery|ROUND-TRIP LATENCY The details recorded in each log entry are explained below. State Description Time Stamp The time at which the log is created. Example 2021-10-26 17:34:40,464 Correlation ID Each log contains a Correlation ID, which is unique to the HTTP request. A client can send the Correlation ID in the header of the HTTP request. If this Correlation ID is missing in the incoming request, the WSO2 Integrator: MI will generate one for the request. The name of the HTTP header used to carry the Correlation ID is configurable in the WSO2 Integrator: MI. Example de461a83-fc74-4660-93ed-1b609ecfac23 Thread name The identifier of the thread. Example HTTP-Listener I/O dispatcher-3 Duration The duration (given in milliseconds) depends on the type of log entry: If the state in the log entry is ROUND-TRIP LATENCY, the duration corresponds to the time gap between the REQUEST_HEAD state and the ROUND-TRIP LATENCY state, which is the total time of the round trip. If the state in the log entry is BACKEND LATENCY, the duration corresponds to the total time taken by the backend to process the message. For all other log entries, the duration indicates the time gap between the current log entry and the immediately preceding one. This represents the time taken for the HTTP request to transition from one state to another. Example 535 Call type There are two possible call types: HTTP call type identifies logs that correspond to either back-end latency or round-trip latency states. That is, in the case of an individual request, one log will be recorded to identify back-end latency, and another log for round-trip latency. Since these logs relate to HTTP calls between the WSO2 Integrator: MI and external clients, these logs are categorized using the HTTP call type. HTTP State Transition call type identifies logs that correspond to the state transitions in the HTTP transport related to a particular message. Connection name This is a name that is generated to identify the connection between the WSO2 Integrator: MI and the external client (back-end or message sender). Example http-incoming-3 Method type The HTTP method used for the request. Example GET Connection URL The connection URL of the external client with which the message is being communicated. When reading a message, this corresponds to the URL the client invokes (i.e., the service or API URL in the WSO2 Integrator: MI). When writing a message to a backend service, it refers to the backend's URL. Example /api/querydoctor/surgery HTTP state Listed below are the state changes that a message goes through when it flows through the WSO2 Integrator: MI, and when the message flows between the WSO2 Integrator: MI and external clients. Typically, a new log entry is generated for each of the states. However, there can be two separate log entries created for one particular state (except for BACKEND LATENCY and ROUND-TRIP LATENCY) depending on whether the message is being read or written. You can identify the two separate log entries from the connection URL explained above. REQUEST_HEAD: All HTTP headers in the incoming request are being read/or being written to the backend. REQUEST_BODY : The body of the incoming request is being read/or being written to the backend. REQUEST_DONE : The request is completely read (content decoded)/ or is completely written to the backend. BACKEND LATENCY : The response message is received by the WSO2 Integrator: MI. This status corresponds to the time total time taken by the backend to process the message. RESPONSE_HEAD : All HTTP headers in the response message are being read/or being written to the client. RESPONSE_BODY : The body of the response message is being read/or being written to the client. RESPONSE_DONE : The response is completely read/ or completely written to the client. ROUND-TRIP LATENCY : The response message is completely written to the client. This status corresponds to the total time taken by the HTTP request to complete the round trip (from the point of receiving the HTTP request from a client until the response message is sent back to the client)