Data Mapping using AI¶

Did you know that the WSO2 Micro Integrator component of WSO2 Integrator comes with built-in data mapping capabilities to streamline your integration workflows? This data mapping solution brings you a powerful graphical interface, combined with AI-driven assistance, to perform all sorts of data mappings in mere seconds.

In this guide, we are generating an API that processes weather data retrieved from the OpenWeather API for a mobile app. The app only requires a selected set of data from the API response and needs to modify some of the retrieved data to meet its requirements.

Check out this video on YouTube to see it in action.

Let's get started!¶

Prerequisites¶

You need Visual Studio Code (VS Code) with the Micro Integrator for VS Code extension installed.

Info

See the Install Micro Integrator for VS Code documentation to learn how to install Micro Integrator for VS Code.

Step 1: Create an integration project¶

The Integration project will contain all the required artifacts for the integration solution.

Launch VS Code with the Micro Integrator extension installed.
Click on the Micro Integrator icon on the Activity Bar of the VS Code editor.
Click Create New Project on Design View.

Next, the Project Creation Form will be opened.
In the Project Creation Form, enter AIDataMappingDemo as the Project Name.
Provide a location under the Project Directory.
Click Create.

You will now see the projects listed in the Project Explorer.

Step 3: Design the integration¶

Create a REST API¶

Go to Project Settings > Add Artifact.
Select API under Create an Integration.
Enter Weather as the API Name. The API Context field will be automatically populated with the same value.
Click Create.
On the Service Designer, click on the three dots (⋮) and then edit to access the Properties of the default API resource.
Click + Add Query Param, provide city as the query parameter and click Add.
Click Update.

Define the mediation flow¶

You can now start configuring the API resource.

Navigate to MI Project Explorer > APIs > Weather > /?city={city} to open the Resource View.
Click on the + icon in the diagram to open the Mediator Palette.
Select Variable mediator under Mediators.

Specify the following values and click Add.

Info

This parameter is used to store the API key required for OpenWeather. It is recommended to use secure vault to store credentials. For more information, refer to Encrypting Secrets.

Parameter	Value
Name	`API_KEY`
Data Type	Select `STRING`.
Value	Your OpenWeather API Key

Click on the + icon and search for get in the Mediator Palette to add the HTTP GET operation for retrieving the coordinates of a location.
Click + Add new connection to create a new connection.
Select HTTPS and fill in the following details to create a connection to OpenWeather API. Finally, click Add in the Add New Connection form to create the connection.

Property Value

Connection Name OpenWeather

Base URL https://api.openweathermap.org/

Provide /geo/1.0/direct as the Relative Path, and click + Add Query Param to add the following query parameters.

Note

To add an expression as the query parameter value, make sure the EX button is checked as follows,

Query Parameter Name	Is Expression	Value
q	true	`params.queryParams.city`
limit	false	`1`
appid	true	`vars.API_KEY`

Click Submit to add the operation to the integration flow.
Next, click on the + icon in the diagram and search for get in the Mediator Palette to add the HTTP GET operation for retrieving the weather details using the coordinates.

Select the connection created in Step 6, provide /data/2.5/weather as the Relative Path, and click + Add Query Param to add the following query parameters. Finally, click Submit to add the operation to the integration flow.

Query Parameter Name	Is Expression	Value
lat	true	`payload[0].lat`
lon	true	`payload[0].lon`
appid	true	`vars.API_KEY`

Click on the + icon in the diagram and search for for the Datamapper mediator in the Mediator Palette.
Click + Add new, provide weatherDataMapper as the name and click Create in the Create New Datamapper form.
Click Add to add the data mapper to the integration flow.
Click on the name of the newly added Data Mapper mediator to open the input and output mapping window.

Click on the Import input schema to import the input schema. Since this scenario involves JSON to JSON mapping, you can import from either a JSON sample or a JSON schema. In this guide, we will be using the Import from JSON option.

Sample Input JSON

Here’s a sample response received from the OpenWeatherMap API. You can copy and paste this as your input schema in this scenario.

{
    "coord": {
        "lon": 79.8478,
        "lat": 6.9319
    },
    "weather": [
        {
            "id": 800,
            "main": "Clear",
            "description": "clear sky",
            "icon": "01d"
        }
    ],
    "base": "stations",
    "main": {
        "temp": 296.91,
        "feels_like": 297.61,
        "temp_min": 296.91,
        "temp_max": 296.91,
        "pressure": 1011,
        "humidity": 87,
        "sea_level": 1011,
        "grnd_level": 1010
    },
    "visibility": 10000,
    "wind": {
        "speed": 3.87,
        "deg": 39,
        "gust": 3.99
    },
    "clouds": {
        "all": 0
    },
    "dt": 1736385682,
    "sys": {
        "country": "LK",
        "sunrise": 1736384109,
        "sunset": 1736426384
    },
    "timezone": 19800,
    "id": 1248991,
    "name": "Colombo",
    "cod": 200
}

Follow the same steps to import the following output schema.
Sample Output JSON
The data mapper will map the above input schema to the below output schema. You can copy and paste this as your output schema in this scenario.
```
{
    "weather": {
        "main": "Clear",
        "description": "clear sky",
        "icon": "01d",
        "daylight_hours": 11.74
    },
    "temp": {
        "temp": 23.76,
        "temp_description": "23.76 Feels like 24.46"
    },
    "wind": {
        "speed": 3.87,
        "deg": 39
    },
    "pressure": 1011,
    "humidity": "87%",
    "visibility": 10
}
```
Tip

Micro Integrator allows you to do data mappings between different data types like XML and CSV. Check out more on this in our documentation.

Now that all our schemas are in place, how does the actual mapping happen?

You could always fall back on the age-old approach of manually mapping each field. But why go through all that hassle?
Click on the Map button.

The Micro Integrator Data Mapper leverages AI to generate various types of mappings. With a simple button click, you'll find that all of your mappings have been automatically completed.

The AI Data Mapper is capable of handling complex mapping scenarios as well.

For instance,
- In our output schema, daylight_hours is calculated from the formula (sunset - sunrise) / 3600 based on the input data.
- The humidity field in the input data is a percentage value and must be displayed accurately to the user.
The AI Data Mapper can recognize and manage these types of use cases effectively.
If you need to apply additional operations to define mappings between the input and output data, try using the Expression Editor. Explore the following scenarios to gain insight into how you can utilize the Expression Editor effectively.
- Display the weather description in upper-case letters
Tip

You can learn further about the Expression Editor and other features from our documentation.

Click on the + icon in the diagram, select the Respond mediator from the Mediator Palette, and click Add to add it to the integration flow.

You may refer to the following API, HTTP connection, and Data Mapper configurations for reference,

Weather API

Design ViewSource View

<?xml version="1.0" encoding="UTF-8"?>
<api context="/weather" name="Weather"
    xmlns="http://ws.apache.org/ns/synapse">
    <resource methods="GET" uri-template="/?city={city}">
        <inSequence>
            <variable name="API_KEY" type="STRING" value="REPLACE_API_KEY"/>
            <http.get configKey="OpenWeather">
                <relativePath>/geo/1.0/direct?q=${params.queryParams.city}&amp;limit=1&amp;appid=${vars.API_KEY}</relativePath>
                <headers>[]</headers>
                <forceScAccepted>false</forceScAccepted>
                <disableChunking>false</disableChunking>
                <forceHttp10>false</forceHttp10>
                <noKeepAlive>false</noKeepAlive>
                <responseVariable>http_get_1</responseVariable>
                <overwriteBody>true</overwriteBody>
            </http.get>
            <http.get configKey="OpenWeather">
                <relativePath>/data/2.5/weather?lat=${payload[0].lat}&amp;lon=${payload[0].lon}&amp;appid=${vars.API_KEY}</relativePath>
                <headers>[]</headers>
                <forceScAccepted>false</forceScAccepted>
                <disableChunking>false</disableChunking>
                <forceHttp10>false</forceHttp10>
                <noKeepAlive>false</noKeepAlive>
                <responseVariable>http_get_2</responseVariable>
                <overwriteBody>true</overwriteBody>
            </http.get>
            <datamapper config="resources:datamapper/weatherDataMapper/weatherDataMapper.dmc" inputSchema="resources:datamapper/weatherDataMapper/weatherDataMapper_inputSchema.json" outputSchema="resources:datamapper/weatherDataMapper/weatherDataMapper_outputSchema.json"/>
            <respond/>
        </inSequence>
        <faultSequence></faultSequence>
    </resource>
</api>

HTTP Connection

Design ViewSource View

<?xml version="1.0" encoding="UTF-8"?>
<localEntry key="OpenWeather" xmlns="http://ws.apache.org/ns/synapse">
    <http.init>
        <connectionType>HTTPS</connectionType>
        <baseUrl>https://api.openweathermap.org/</baseUrl>
        <authType>None</authType>
        <timeoutAction>Never</timeoutAction>
        <retryCount>0</retryCount>
        <retryDelay>0</retryDelay>
        <suspendInitialDuration>-1</suspendInitialDuration>
        <suspendProgressionFactor>1</suspendProgressionFactor>
        <name>OpenWeather</name>
    </http.init>
</localEntry>

DataMapper Configuration

import * as dmUtils from "./dm-utils";
/*
* title : "root",
* inputType : "JSON",
*/
interface Root {
    coord: {
        lon: number
        lat: number
    }
    weather: {
        id: number
        main: string
        description: string
        icon: string
    }[]
    base: string
    main: {
        temp: number
        feels_like: number
        temp_min: number
        temp_max: number
        pressure: number
        humidity: number
        sea_level: number
        grnd_level: number
    }
    visibility: number
    wind: {
        speed: number
        deg: number
        gust: number
    }
    clouds: {
        all: number
    }
    dt: number
    sys: {
        country: string
        sunrise: number
        sunset: number
    }
    timezone: number
    id: number
    name: string
    cod: number
}

/*
* title : "root",
* outputType : "JSON",
*/
interface OutputRoot {
    weather: {
        main: string
        description: string
        icon: string
        daylight_hours: number
    }
    temp: {
        temp: number
        temp_description: string
    }
    wind: {
        speed: number
        deg: number
    }
    pressure: number
    humidity: string
    visibility: number
}

/**
* functionName : map_S_root_S_root
* inputVariable : inputroot
*/
export function mapFunction(input: Root): OutputRoot {
    return {
        weather: {
            main: input.weather[0].main,
            description: input.weather[0].description.toLocaleUpperCase(),
            icon: input.weather[0].icon,
            daylight_hours: (input.sys.sunset - input.sys.sunrise) / 3600,
        },
        temp: {
            temp: input.main.temp - 273.15,
            temp_description: input.main.temp > 30 ? "Hot" : input.main.temp < 15 ? "Cold" : "Moderate",
        },
        wind: {
            speed: input.wind.speed,
            deg: input.wind.deg,
        },
        pressure: input.main.pressure,
        humidity: `${input.main.humidity}%`,
        visibility: input.visibility,
    };
}

Step 4: Build and run¶

Click on the Build and Run button to build and deploy the integration.
Runtime Services will open after the server is up and running, and from that panel, select the API that you need to try out using the Try It button.
Click the Try it Out button to provide the required query parameters.
Provide the required parameters and click the Execute button to invoke the API.
Check the response received from the server and confirm that the weather details for the specified city have been mapped to the output data schema you provided.

Try this feature along with our other exciting AI capabilities and discover how much simpler integration development can be.

Property	Value
Connection Name	`OpenWeather`
Base URL	`https://api.openweathermap.org/`