Transformations is one of RudderStack's key features. It gives you the ability to code custom JavaScript functions to implement specific use-cases on your event data, like:

  • Filtering/sampling events.
  • Enriching events by implementing static logic or leveraging an external API.
  • Cleaning/aggregating data.
  • Data masking or removing sensitive PII information in the events to ensure data privacy.
  • Implementing external actions on the events using an API.
Transformations can be used across your Cloud Apps, Reverse ETL and Cloud Extract pipelines.
You can refer to our Sample Transformations GitHub repository for some useful templates that you can use to create your own transformations.
Transformations only work for the Cloud mode destinations.

To know more about the Cloud mode in RudderStack, check out the RudderStack Connection Modes guide.

Adding a transformation

To add a new transformation in the RudderStack dashboard, follow these steps:

  1. Log into your RudderStack dashboard.
  2. Go to Enhance > Transformations, as shown:
Adding a Transformation
  1. Click New Transformation.
  2. Add a name for your transformation and an optional description.
  3. Next, add your JavaScript function in the Transformation window, as shown:
Adding a Transformation
You can also add other functions and call them from within transformEvent.
Refer to the Sample User Transformations GitHub repository for some useful templates that you can use to create your own transformations.
Remember to delete the pre-populated transformEvent function before adding or pasting your transformation code.
  1. To test your transformation on a sample event, click Run Test. RudderStack provides some sample events for you test if your transformation logic works as expected.
Adding a Transformation
  1. To save the transformation, click Save.

Connecting transformation to a destination

You can connect a transformation to a destination in two cases:

Case 1: While setting up a new destination

RudderStack gives you the option to add a transformation while setting up a destination, as shown:

Connecting a transformation

Case 2: Connecting to an existing destination

To add a transformation to an existing destination, follow these steps:

  1. Go to the Transformation tab in your destination dashboard and click Add a transformation, as shown:
Connecting a transformation to existing destination
  1. Then, choose the transformation to you want to connect to the destination.
Connecting a transformation to existing destination

transformEvent definition

RudderStack applies the transformEvent function on each event and takes two inputs:

  • event: This corresponds to the input event
  • metadata: This is an optional input corresponding to the JavaScript function which you can use to access the metadata of the given event.
For more information on metadata, refer to the Accessing metadata section below.

After the transformation is complete, transformEvent returns the final event to be sent to the destination.

Using transformBatch instead of transformEvent

In cases where you want to perform any aggregation or roll-up operation on a micro batch of events, you can use the transformBatch function instead of transformEvent, as shown:

export function transformBatch(events, metadata) {
return events;
}
If you are using transformBatch and want to ensure event ordering, make sure you pass on the messageId from the input event to the output event. Without the messageId, there is a strong possibility that RudderStack will not maintain the event ordering.
It is highly recommended to use transformEvent as much as possible, as it ensures event ordering in any case.

Libraries

One of the key features of RudderStack Transformations is the ability to reuse the code written for a transformation in other transformations using the Libraries feature.

To add a new library, follow these steps:

  1. In your RudderStack dashboard, go to Enhance > Transformations and click the Libraries tab, as shown:
Adding a library
  1. Click New library.
  2. Add the library Name, Description and write the function you want to reuse across different transformations.
  3. You can also add multiple functions in a single library, as shown:
Adding a library
Currently, RudderStack does not support deletion of libraries. This is to ensure that any existing transformations using the libraries do not break.
  1. Click Run Test to ensure the code included in the library has the correct syntax.

Using libraries in transformations

To use the libraries in your existing transformation, simply take the name of your library and convert it into camel case without spaces - this becomes your library handle.

For example, if your library name is is rudder email, then the library handle would be isRudderEmail.

Using libraries in transformations

Let's say you want to import a function called rudderEmail, which returns true for the emails from a specific domain (e.g. RudderStack) and false otherwise, from the is rudder email library. Also, you want to use this function to filter the events that don't have the email address of the specified domain.

The following code snippet demonstrates how we can implement this use-case:

import { rudderEmail } from "isRudderEmail";
export function transformEvent(event) {
const email =
event.context && event.context.traits && event.context.traits.email;
if (email) {
if (!rudderEmail(email)) return;
}
return event;
}

On running a test, an example event not having the specified email domain is filtered out, as shown:

Event filtered out without specifying email domain

Importing multiple functions from a single library

When importing a single function or multiple functions from a library, it is important to keep the import statement to one line. Breaking the import function into multiple lines will lead to an error.

The following snippets highlight how to properly import functions from a library:

// Correct Way
// ---------------
import { getLoss } from "getFinanceData";
// OR
import { getLoss, getRevenue, getProfit } from "getFinanceData";
import {
getLoss,
getRevenue,
getProfit
} from "getFinanceData";
// For default getPrice import
import getPrice, { getRevenue, getProfit } from "getFinanceData";
// alias imports
import getPrice as gp, { getRevenue as gr, getProfit } from "getFinanceData";
// usuage: gp(event), gr(event), getProfit(ev)
import * as GFD from "getFinanceData";
// usuage: GFD.getRevenue(ev), GFD.getProfit(ev)
// for default import: GFD.default(ev)
// Incorrect Way
// -----------------
import * from "getFinanceData";
// OR
import getPrice as gp from "getFinanceData";
getPrice(ev)

Accessing event metadata

RudderStack injects a function metadata(event) into your transformations as an argument. This allows you to access the event metadata variables that help you customize your transformations.

metadata() takes the event as the input and returns the metadata of the event.

The following properties, if available, are present in the metadata response:

PropertyDescription
sourceIdThe ID of the source configured in your RudderStack dashboard. Note that it is different from the source write key.
destinationIdThe ID of the destination configured in your RudderStack dashboard.
messageIdThe unique ID for each event.
sessionIdThe ID associated with a session. This is applicable only if sessions are enabled.

Since you may not need the event metadata in every transformation, you can include it as an optional argument in your function signature whenever required.

An example of using metadata is shown below:

export function transformEvent(event, metadata) {
const meta = metadata(event);
event.sourceId = meta.sourceId;
return event;
}

External API requests

You can make any number of external API requests in your transformation functions and use the response to enrich your events data. RudderStack injects an asynchronous fetch(url) function into your transformations. It makes an API call to the given URL and returns the response in a JSON format.

An example of how to use the fetch function in transformations is shown below:

export async function transformEvent(event, metadata) {
const res = await fetch("post_url", {
method: "POST", // POST, PUT, DELETE, GET, etc.
headers: {
"Content-Type": "application/json;charset=UTF-8",
Authorization: "Bearer <your_authorization_token>"
},
body: JSON.stringify(event)
});
event.response = JSON.stringify(res);
return event;
}
To see how this feature is used in more detail, refer to the Clearbit enrichment example which uses the fetch function.
For improved performance, we recommend using batch API requests instead of a separate API request for each event whenever possible.

FetchV2

FetchV2 is a wrapper around the fetch call to provide more access to fetch the response properties.

The following properties are present in a fetchV2 response:

Property NameDescription
statusStatus code of fetch response, e.g. 200.
urlURL of the Fetch API.
headersResponse headers
bodyResponse body in JSON or TEXT. By default, it is JSON.
Use the try-catch block to capture the cause for failures during the fetch call, e.g. when a timeout is applied.

An example of how to use the fetchV2 function in RudderStack transformations is shown below:

export async function transformEvent(event) {
try {
const res = await fetchV2("url", { timeout: 1000});
if (res.status == 200) {
event.response = JSON.stringify(res.body);
}
} catch (err) {
log(err.message);
}
return event;
}

Debugging with logs

RudderStack enables you to access any event-related information in the form of logs while running a test on your transformation. You can do this by including the log function in your transformation code.

An example of using the log function is shown below:

export function transformEvent(event, metadata) {
const meta = metadata(event);
event.sourceId = meta.sourceId;
log("Event Name is", event.event, ";", "Message ID is", event.messageId);
log("Source ID is", meta.sourceId);
return event;
}

On adding the above transformation and clicking Run Test, you can see the resulting log in the Logs section of the dashboard, as shown:

Transformation log
You can pass a string, number, or an object as an argument to the log function.

Limits

You should take into account the memory and time limits when adding a new transformation. Each transformation invocation should not exceed the following limits:

ParameterLimit
Memory limit8 MB
Time limit4 seconds
The user transformation fails if these limits are exceeded.

FAQ

Why do I need to use RudderStack's Transformations feature?

RudderStack's Transformations feature lets you transform the events collected from a source into a destination-specific format. Some key benefits of this feature are:

  • These transformations can be applied to your in-transit events, in real-time.
  • You can implement specific use-cases depending on your business requirements using this feature. Some examples include PII masking, event filtering based on specific criteria, aggregating and enriching events, and so on.
  • They're easier to build and manage, and can be reused as well.
Read more about why your data pipeline needs to have an efficient event transformations capability in our blog.

I used to write transformations like function transform(events) {}. Why am I not able to create new transformations this way?

We have changed the way transformations are written. RudderStack now supports writing functions that transform a single event instead of a batch of events. You can now define your transformation in the following manner:

export function transformEvent(event, metadata) {}

I want to write a transformation that can be applied to a small batch of events. Can I not do this anymore?

You can. Simply define your transformation in the following manner:

export function transformBatch(events, metadata) {}

This can cause a loss of the event metadata that RudderStack uses internally to maintain event ordering for you.

I cannot import libraries into my existing transformations. What do I do?

Previously created transformations shown as Version: V0 don't support libraries. If you want to use libraries, please create a new transformation with Version: V1 that support libraries. You can then reconnect your destinations with this new transformation that you have created.

I want to update my existing version v0 transformation code to version v1. What do I do?

You cannot update transformation Version: V0 directly to Version: V1. Please create a new transformation with updated code Version: V1. You can then reconnect your destinations with this new transformation that you have created and delete existing Version: V0 transformation.


Contact us

For more information on the topics covered on this page, email us or start a conversation in our Slack community.