Generative AI Module Reference¶ This documentation provides a reference guide for the Generative AI Module. The Generative AI Module enables you to perform various operations such as chat, RAG chat, agent-based interactions, retrieving information from a knowledge base, and adding new entries to the knowledge base. Connection Configurations¶ LLM Connection¶ LLM (Large Language Model) connections are used to connect to various LLM providers. These connections are essential for enabling the Generative AI Module to interact with different LLM services. The WSO2 MI Generative AI Module supports multiple LLM providers, allowing seamless integration with llm models. Anthropic Mistral AI Open AI Azure Open AI Deepseek Connection Configuration Parameters¶ The connection configuration parameters are used to establish a connection with the LLM provider. These parameters are needed to provide based on the provider you are using. Anthropic Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes apiKey OpenAI Key API key used to authenticate with the OpenAI service. Yes Mistral AI Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes apiKey MistralAI Key API key used to authenticate with the Mistral AI service. Yes Open AI Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes apiKey OpenAI Key API key used to authenticate with the OpenAI service. Yes baseUrl Base Url Base URL of the OpenAI service. Yes Azure Open AI Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes apiKey Azure OpenAI Key API key used to authenticate with the Azure OpenAI service. Yes deploymentName Deployment Name Name of the deployment in Azure OpenAI. Yes endpoint Endpoint Endpoint URL of the Azure OpenAI service. Yes Deepseek Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes apiKey Deepseek API Key API key used to authenticate with the Deepseek service. Yes baseUrl Base Url Base URL of the Deepseek service. Yes Embedding Model Connection¶ Embedding model connections are used to connect to various embedding model providers. These connections are essential for enabling the Generative AI Module to interact with different embedding model services. The WSO2 MI Generative AI Module supports Open AI and Azure Open AI embedding model providers. Connection Configuration Parameters¶ The connection configuration parameters are used to establish a connection with the Embedding model provider. These parameters are needed to provide based on the provider you are using. Open AI Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes apiKey OpenAI Key API key used to authenticate with the OpenAI service. Yes Azure Open AI Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes apiKey Azure OpenAI Key API key used to authenticate with the Azure OpenAI service. Yes deploymentName Deployment Name Name of the deployment in Azure OpenAI. Yes endpoint Endpoint Endpoint URL of the Azure OpenAI service. Yes Vector Database (Knowledgebase) Connection¶ Vector database connections are used to connect to various vector database providers. These connections are essential for enabling the Generative AI Module to interact with different vector database services. The WSO2 MI Generative AI Module supports multiple vector stores. Chroma DB Chroma is an open-source vector database designed for AI applications, enabling efficient storage and retrieval of embeddings. Postgres DB Postgres DB is a powerful, open-source relational database that can be used as a vector store for managing embeddings. Note To use Postgres DB as a vector store, ensure the pgvector extension is installed on the machine where the Postgres DB is running. After installation, execute the following SQL command in your Postgres database to enable the creation of vector columns: CREATE EXTENSION vector; Pinecone DB Pinecone is a managed vector database service that provides high-performance and scalable storage for embeddings. Weaviate Weaviate is an open-source, cloud-native vector database that supports hybrid search (vector + keyword) and is optimized for semantic search and generative AI use cases. Milvus Milvus is a high-performance, open-source vector database built for scalable similarity search, capable of handling large-scale embedding workloads efficiently. MI Vector Store MI Vector Store is an in-memory vector database offered by WSO2 MI which will persist the data in the MI resources. Info MI Vector Store is designed specifically for testing purposes and is not recommended for production use. Please use other vector stores for production scenarios. Connection Configuration Parameters¶ The connection configuration parameters are used to establish a connection with the vector store. These parameters are needed to provide based on the store you are using. Chroma DB Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes url Base URL Base URL of the Chroma service. Yes collection Collection Name of the collection in Chroma. Yes Postgres DB Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes host Host Host name of the Postgres DB. Yes port Port Port number of the Postgres DB. Yes database Database Name of the database in Postgres DB. Yes user User User name to connect to the Postgres DB. Yes password Password Password to connect to the Postgres DB. Yes table Table Name of the table in Postgres DB. Yes dimension Dimension of embeddings Dimension of the embeddings used in the Postgres DB. Yes Pinecone DB Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes apiKey API Key API key used to authenticate with the Pinecone service. Yes index Index Name of the index in Pinecone. Yes namespace Namespace Namespace of the index in Pinecone. Yes cloud Cloud Cloud provider of the Pinecone service. Yes region Region Region of the Pinecone service. Yes dimension Dimension of embeddings Dimension of the embeddings used in the Pinecone service. Yes Weaviate Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes scheme Scheme Protocol used to connect to the Weaviate instance (http or https). Yes host Host Host name or IP address where the Weaviate service is running. Yes port Port Port number of the Weaviate service. Yes apiKey API Key API key used to authenticate with the Weaviate service (if authentication is enabled). No objectClass Object Class Name of the Weaviate class used to store and retrieve vector objects. Yes avoidDuplicates Avoid Duplicates Enable to prevent inserting duplicate vector objects into the specified class. No consistencyLevel Consistency Level Consistency level for write operations (e.g., ONE, QUORUM, ALL). No Milvus Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes url URI Full URI of the Milvus server (e.g., http://localhost:19530). Yes token Token / API Key Authentication token or API key used to connect to the Milvus service (if authentication is enabled). No collection Collection Name Name of the collection in Milvus used to store vector embeddings. Yes dimension Dimension Dimension of the embeddings stored in the Milvus collection. Yes MI Vector Store Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes Chat Memory Connection¶ Chat memory connections are used to connect to various chat memory providers. Chat memory is essential for enabling the Generative AI Module to manage chat history and context. The WSO2 MI Generative AI Module supports multiple chat memory providers. Postgres DB Postgres DB is a powerful, open-source relational database that can be used as a chat memory provider for managing chat history. File Memory File memory is a simple file-based storage solution for managing chat history. It persists chat data within the MI resources, storing the memory file at <MI_SERVER_HOME>/registry/governance/ai/chat-memory/<CONNECTION_NAME>.json. Info File memory is designed specifically for testing purposes and is not recommended for production use. Please use other chat memory providers for production scenarios. Connection Configuration Parameters¶ The connection configuration parameters are used to establish a connection with the chat memory database. These parameters are needed to provide based on the database you are using. Postgres DB Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes host Host Host name of the Postgres DB. Yes port Port Port number of the Postgres DB. Yes database Database Name of the database in Postgres DB. Yes user User User name to connect to the Postgres DB. Yes password Password Password to connect to the Postgres DB. Yes table Table Name of the table in Postgres DB. Yes File Memory Parameter Name Display Name Description Required name Connection Name Name of the connection. Yes Operations¶ chatThe chat operation sends a message to the LLM and receives a response. Connection Parameters: Parameter Name Connection Category Description Required llmConfigKey LLM Connection Connection to the LLM provider. Refer to the LLM Connection section for more details. Yes memoryConfigKey Chat Memory Connection Connection to the chat memory provider. Refer to the Chat Memory Connection section for more details. No Operation Parameters: Parameter Name Display Name Description Required modelName Model Name Name of the model to use for the chat operation. Yes sessionId Session ID Unique identifier for the chat session. No prompt User Query/Prompt Message to send to the LLM. Yes attachments Attachments Attachments to send along with the message to the LLM. The attachments should be in the following format: [{"type":"application/pdf", "content":"base64 content"}]. Supported types include: application/pdf image/png image/jpeg text/plain text/html text/csv text/xml The content must be base64 encoded. No outputType Output Type Type of output to return. The supported types are: string, integer, float, boolean. The default is string. No maxHistory Max chat history Maximum number of chat history entries to send to the LLM. No system System Prompt System prompt to set the context for the conversation. No maxTokens Max Tokens Maximum number of tokens to generate in the response. No temperature Temperature Sampling temperature to use for the response. The default is 0.7. No topP Top P Top P sampling to use for the response. The default is 1. No frequencyPenalty Frequency Penalty Frequency penalty to use for the response. The default is 0. No seed Seed Specifies the random seed value to ensure reproducibility of the response. The default value is 0. No responseVariable Output Variable Name Name of the variable to store the response. Yes overwriteBody Overwrite Body Whether to overwrite the message body with the response. The default is false. No Sample configuration <ai.chat> <connections> <llmConfigKey>LLM_CONN</llmConfigKey> <memoryConfigKey>MEMORY_CONN</memoryConfigKey> </connections> <sessionId>{${payload.sessionId}}</sessionId> <system>You are a helpful AI assistant</system> <prompt>${payload.content}</prompt> <outputType>string</outputType> <responseVariable>ai_chat_1</responseVariable> <overwriteBody>false</overwriteBody> <modelName>gpt-4o</modelName> <temperature>0.7</temperature> <maxTokens>4069</maxTokens> <topP>1</topP> <frequencyPenalty>0</frequencyPenalty> <maxHistory>10</maxHistory> </ai.chat> Sample response The response received will be stored in the variable ai_chat_1 as a JSON object. The following is a sample response. { "content": "WSO2 Integrator: MI is a comprehensive integration solution designed to simplify digital transformation. It facilitates connectivity among applications, services, data, and the cloud through a user-friendly, low-code graphical design experience. The WSO2 Integrator: MI offers deployment options in both microservices and ESB styles, providing greater flexibility.", "tokenUsage": { "inputTokensDetails": { "cachedTokens": 0 }, "outputTokensDetails": { "reasoningTokens": 0 }, "inputTokenCount": 41, "outputTokenCount": 459, "totalTokenCount": 500 }, "sources": [], "finishReason": "STOP", "toolExecutions": [] } ragChatThe ragChat operation sends a message to the LLM and retrieves a response, leveraging a Retrieval-Augmented Generation (RAG) approach. Connection Parameters: Parameter Name Connection Category Description Required embeddingConfigKey Embedding model Connection Connection to the embedding model provider. Refer to the Embedding Model Connection section for more details. Yes vectorStoreConfigKey Vector Store Connection Connection to the vector store provider. Refer to the Vector Store Connection section for more details. Yes llmConfigKey LLM Connection Connection to the LLM provider. Refer to the LLM Connection section for more details. Yes memoryConfigKey Memory Connection Connection to the chat memory provider. Refer to the Chat Memory Connection section for more details. No Operation Parameters: Parameter Name Display Name Description Required sessionId Session ID Unique identifier for the chat session. No prompt User Query/Prompt Message to send to the LLM. Yes embeddingModel Embedding Model Name of the embedding model to use for the RAG operation. Yes maxResults Max Results Maximum number of results to retrieve from the vector store. No minScore Min Score Minimum score threshold for the results retrieved from the vector store. No modelName Model Name Name of the model to use for the chat operation. Yes outputType Output Type Type of output to return. The supported types are: string, integer, float, boolean. The default is string. No maxHistory Max chat history Maximum number of chat history entries to send to the LLM. No system System Prompt System prompt to set the context for the conversation. No maxTokens Max Tokens Maximum number of tokens to generate in the response. No temperature Temperature Sampling temperature to use for the response. The default is 0.7. No topP Top P Top P sampling to use for the response. The default is 1. No frequencyPenalty Frequency Penalty Frequency penalty to use for the response. The default is 0. No seed Seed Specifies the random seed value to ensure reproducibility of the response. The default value is 0. No responseVariable Output Variable Name Name of the variable to store the response. Yes overwriteBody Overwrite Body Whether to overwrite the message body with the response. The default is false. No Sample configuration <ai.ragChat> <connections> <llmConfigKey>LLM_CONN</llmConfigKey> <memoryConfigKey>MEMORY_CONN</memoryConfigKey> <embeddingConfigKey>EMBEDDING_CONN</embeddingConfigKey> <vectorStoreConfigKey>VECTOR_STORE_CONN</vectorStoreConfigKey> </connections> <sessionId>{${payload.sessionId}}</sessionId> <prompt>${payload.content}</prompt> <outputType>string</outputType> <responseVariable>ai_ragChat_1</responseVariable> <overwriteBody>false</overwriteBody> <embeddingModel>text-embedding-3-small</embeddingModel> <maxResults>5</maxResults> <minScore>0.75</minScore> <modelName>gpt-4o</modelName> <temperature>0.7</temperature> <maxTokens>4069</maxTokens> <topP>1</topP> <frequencyPenalty>0</frequencyPenalty> <seed>0</seed> <system></system> <maxHistory>10</maxHistory> </ai.ragChat> Sample response The response received will be stored in the variable ai_ragChat_1 as a JSON object. The following is a sample response. { "content": "WSO2 Integrator: MI is a comprehensive integration solution designed to simplify digital transformation. It facilitates connectivity among applications, services, data, and the cloud through a user-friendly, low-code graphical design experience. The WSO2 Integrator: MI offers deployment options in both microservices and ESB styles, providing greater flexibility.", "tokenUsage": { "inputTokensDetails": { "cachedTokens": 0 }, "outputTokensDetails": { "reasoningTokens": 0 }, "inputTokenCount": 99, "outputTokenCount": 62, "totalTokenCount": 161 }, "sources": [ { "textSegment": { "text": "WSO2 Integrator: MI is a comprehensive integration solution that simplifies your digital transformation journey. The WSO2 Integrator: MI streamlines connectivity among applications, services, data, and the cloud using a user-friendly, low-code graphical design experience. Deployment options include both microservices and ESB styles for greater flexibility.", "metadata": { "index": "0" } }, "metadata": {} } ], "finishReason": "STOP", "toolExecutions": [] } agentThe chat operation sends a message to the LLM and receives a response. Connection Parameters: Parameter Name Connection Category Description Required llmConfigKey LLM Connection Connection to the LLM provider. Refer to the LLM Connection section for more details. Yes memoryConfigKey Chat Memory Connection Connection to the chat memory provider. Refer to the Chat Memory Connection section for more details. No Operation Parameters: Parameter Name Display Name Description Required sessionId Session ID Unique identifier for the chat session. No modelName Model Name Name of the model to use for the chat operation. Yes role Role The specific function or responsibility assigned to the agent, defining its purpose and scope of operation. No instructions Instructions Specific instructions or guidelines for the agent to follow during its operation. No prompt User Query/Prompt Message to send to the LLM. Yes attachments Attachments Attachments to send along with the message to the LLM. The attachments should be in the following format: [{"type":"application/pdf", "content":"base64 content"}]. Supported types include: application/pdf image/png image/jpeg text/plain text/html text/csv text/xml The content must be base64 encoded. No outputType Output Type Type of output to return. The supported types are: string, integer, float, boolean. The default is string. No maxHistory Max chat history Maximum number of chat history entries to send to the LLM. No system System Prompt System prompt to set the context for the conversation. No maxTokens Max Tokens Maximum number of tokens to generate in the response. No temperature Temperature Sampling temperature to use for the response. The default is 0.7. No topP Top P Top P sampling to use for the response. The default is 1. No frequencyPenalty Frequency Penalty Frequency penalty to use for the response. The default is 0. No seed Seed Specifies the random seed value to ensure reproducibility of the response. The default value is 0. No responseVariable Output Variable Name Name of the variable to store the response. Yes overwriteBody Overwrite Body Whether to overwrite the message body with the response. The default is false. No overflowHandlingMethod Overflow Handling Method The method to handle context overflow when Max History is reached. The default is Trim. No summarizationLlmConfigKey Summarization LLM Connection Connection to the LLM to be used for Summarization. If not specified, agent's model will be used. No summarizationModelName Summarization Model Name Model name to be used for summarization. This needs to be specified only if a Summarization LLM Connection is specified. No Sample configuration <ai.agent> <connections> <llmConfigKey>LLM_CONN</llmConfigKey> <memoryConfigKey>MEMORY_CONN</memoryConfigKey> </connections> <sessionId>{${payload.sessionId}}</sessionId> <role>Customer Assistance Agent</role> <instructions>Assist customers by providing accurate and helpful responses to their queries, ensuring a positive user experience.</instructions> <prompt>${payload.content}</prompt> <outputType>string</outputType> <responseVariable>ai_agent_1</responseVariable> <overwriteBody>false</overwriteBody> <modelName>gpt-4o</modelName> <temperature>0.7</temperature> <maxTokens>4069</maxTokens> <topP>1</topP> <frequencyPenalty>0</frequencyPenalty> <maxHistory>10</maxHistory> <overflowHandlingMethod>Trim</overflowHandlingMethod> </ai.agent> Sample response The response received will be stored in the variable ai_agent_1 as a JSON object. The following is a sample response. { "content": "Hello John Doe! I can help you explore various investment options offered by PineValley Bank. Since your investment goal is long-term growth, let's consider some of the products that might align with your objectives:\n\n1. **Qtrade Guided Portfolios**: These are professionally managed portfolios designed to help achieve long-term growth with a balanced risk approach. They might be suitable for you if you prefer a hands-off approach but still want expert management.\n\n2. **TFSAs (Tax-Free Savings Accounts)**: A TFSA allows your investments to grow tax-free, which is beneficial for long-term growth. You can invest in various assets within a TFSA, including stocks and ETFs.\n\n3. **RRSPs (Registered Retirement Savings Plans)**: An RRSP provides tax-deferred growth, which can be advantageous for long-term retirement planning. Contributions are tax-deductible, and the funds grow without being taxed until withdrawal.\n\n4. **Direct Investing (Stocks, ETFs)**: If you prefer more control over your investments, you can consider direct investing in stocks and ETFs. This approach requires more involvement but can be rewarding if you want to actively manage your portfolio.\n\n5. **GICs (Guaranteed Investment Certificates)**: While generally more conservative, GICs offer guaranteed returns, making them a stable component of a diversified portfolio.\n\nSince I couldn't retrieve detailed brochures or documents this time, I recommend considering these options and contacting PineValley Bank for specific product brochures and more detailed guidance. If you have any questions about these products or need help with the onboarding process, feel free to ask!", "tokenUsage": { "inputTokenCount": 1995, "outputTokenCount": 374, "totalTokenCount": 2369 }, "finishReason": "STOP", "toolExecutions": [ { "request": { "id": "call_4MZwbJiAOJPUuuJFpcKSXneV", "name": "http_post_tool_0", "arguments": "{\"requestBodyJson\":\"{\\\"userID\\\": \\\"C567\\\"}\"}" }, "result": "{\"userID\":\"C567\",\"name\":\"John Doe\",\"age\":30,\"investmentGoal\":\"Long-term growth\"}" }, { "request": { "id": "call_XBFZADsSZsSzl3UBArT60H1O", "name": "ai_getFromKnowledge_tool_0", "arguments": "{\"input\":\"investment products offered by PineValley Bank\"}" }, "result": "[]" } ] } addToKnowledgeThe ragChat operation sends a message to the LLM and retrieves a response, leveraging a Retrieval-Augmented Generation (RAG) approach. Connection Parameters: Parameter Name Connection Category Description Required embeddingConfigKey Embedding model Connection Connection to the embedding model provider. Refer to the Embedding Model Connection section for more details. Yes vectorStoreConfigKey Vector Store Connection Connection to the vector store provider. Refer to the Vector Store Connection section for more details. Yes Operation Parameters: Parameter Name Display Name Description Required input Input Input to be added to the knowledge base. Yes needParse Parse Whether to parse the input. No parseType Type Type of parsing to be done. The supported types are: pdf-to-text, markdown-to-text, html-to-text, doc-to-text, docx-to-text, xls-to-text, xlsx-to-text, ppt-to-text, pptx-to-text. The default is pdf-to-text. No needSplit Split Whether to split the input into smaller chunks. No splitStrategy Strategy Strategy to be used for splitting the input. The supported strategies are: Recursive, ByParagraph, BySentence. The default is Recursive. No maxSegmentSize Max Segment Size Maximum size of each segment after splitting. The default is 1000. No maxOverlapSize Max Overlap Size Maximum overlap size between segments. The default is 200. No needEmbedding Embedding Whether to generate embeddings for the input. No embeddingModel Embedding Model Name of the embedding model to use for the RAG operation. Yes responseVariable Output Variable Name Name of the variable to store the response. Yes overwriteBody Overwrite Body Whether to overwrite the message body with the response. The default is false. No Sample configuration <ai.addToKnowledge> <connections> <embeddingConfigKey>EMBEDDING_CONN</embeddingConfigKey> <vectorStoreConfigKey>VECTOR_STORE_CONN</vectorStoreConfigKey> </connections> <input>{${payload.input}}</input> <needParse>true</needParse> <parseType>pdf-to-text</parseType> <needSplit>true</needSplit> <splitStrategy>Recursive</splitStrategy> <maxSegmentSize>1000</maxSegmentSize> <maxOverlapSize>200</maxOverlapSize> <needEmbedding>true</needEmbedding> <embeddingModel>text-embedding-3-small</embeddingModel> <responseVariable>ai_addToKnowledge_1</responseVariable> <overwriteBody>false</overwriteBody> </ai.addToKnowledge> Sample response The response received will be stored in the variable ai_addToKnowledge_1 as a JSON object. The following is a sample response. { "success": true } getFromKnowledgeThe ragChat operation sends a message to the LLM and retrieves a response, leveraging a Retrieval-Augmented Generation (RAG) approach. Connection Parameters: Parameter Name Connection Category Description Required embeddingConfigKey Embedding model Connection Connection to the embedding model provider. Refer to the Embedding Model Connection section for more details. Yes vectorStoreConfigKey Vector Store Connection Connection to the vector store provider. Refer to the Vector Store Connection section for more details. Yes Operation Parameters: Parameter Name Display Name Description Required input Input Input to be searched from the knowledge base. Yes needEmbedding Embed Input Specifies whether to generate embeddings for the input. If the embedding vector is already provided by the user, this can be disabled. However, if the input is a raw string, embeddings must be generated. No embeddingModel Embedding Model Name of the embedding model to use for the RAG operation. Yes maxResults Max Results Maximum number of results to retrieve from the vector store. No minScore Min Score Minimum score threshold for the results retrieved from the vector store. No responseVariable Output Variable Name Name of the variable to store the response. Yes overwriteBody Overwrite Body Whether to overwrite the message body with the response. The default is false. No Sample configuration <ai.getFromKnowledge> <connections> <embeddingConfigKey>EMBEDDING_CONN</embeddingConfigKey> <vectorStoreConfigKey>VECTOR_STORE_CONN</vectorStoreConfigKey> </connections> <input>{${payload.input}}</input> <needEmbedding>true</needEmbedding> <embeddingModel>text-embedding-3-small</embeddingModel> <maxResults>5</maxResults> <minScore>0.75</minScore> <responseVariable>ai_getFromKnowledge_1</responseVariable> <overwriteBody>false</overwriteBody> </ai.getFromKnowledge> Sample response The response received will be stored in the variable email_list_1 as a JSON object. The following is a sample response. [ { "score": 0.9066269765264976, "embeddingId": "1fd2ccb4-4317-4c3e-a598-1cbcce4ae5ab", "embedding": [ ... ], "embedded": { "text": "WSO2 Integrator: MI is a comprehensive integration solution that simplifies your digital transformation journey. The WSO2 Integrator: MI streamlines connectivity among applications, services, data, and the cloud using a user-friendly, low-code graphical design experience. Deployment options include both microservices and ESB styles for greater flexibility.", "metadata": { "index": "0" } } } ] Click on the Go to Tutorial button below to learn how to build your first AI integration using the above operations. The tutorial will guide you through the process of creating a simple integration that utilizes the AI capabilities of WSO2 Integrator: MI. Go to Tutorial →