RudderStack Docs – Developer Docs ↗️

Archive: Profiles Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

Profiles Tutorial (pb tutorial) is currently available only for Snowflake and Google BigQuery warehouses. Also, note that you can use it only with the Profiles Builder (PB) CLI tool.

Profiles Tutorial (pb tutorial command) is a guided interactive tutorial within the Profiles CLI.

This tutorial walks you through the key Profiles concepts and how they work. You will also build a basic Profiles project with an ID Stitcher model and a handful of features, ultimately producing an ID Graph in your warehouse and a view of the customers along with some customer features/attributes.

Overview

The Profiles Tutorial has multiple interactive components and most of the interaction takes place within a terminal session. The goal of this tutorial is to familiarize you with the Profiles product - this includes details on the YAML configuration, how data unification works, what the outputs look like after a run, and how to troubleshoot and build a solid ID graph around a defined entity.

As a part of the tutorial, you will build a demo Profiles project configuration YAML which you will edit directly following the directions provided in the terminal. You will also seed your warehouse with some sample data that is sent to a target schema within your warehouse. You can query this data directly as well as the materialized tables produced by Profiles within the tutorial session.

With the help of this tutorial, you can build your own Profiles project and extend it further to unify your data around a defined entity, building a C360 degree view of this entity, and much more.

Sample data

The sample data used for this tutorial is based on a fictional business called Secure Solutions, LLC.

This fictional business sells security IOT devices and a security management subscription service. They have a number of Shopify stores and a subscription management service, and one physical store where customers can buy security equipment and checkout at a kiosk.

This fictional business decided to use Profiles to more quickly and easily unify different data sources related to their customers as well as help produce a customer 360 table that they can activate in downstream tools.

Prerequisites

Python environment (v3.9.0 to v3.11.10).
Profiles v0.20.0 or above installed locally within the above Python environment.

pip3 install profiles-rudderstack

profiles-mlcorelib library (v0.7.0 or above) installed within the above Python environment (and same as the environment for the profiles-rudderstack library).

pip install profiles-mlcorelib>=0.7.0

Create a warehouse connection file that Profiles uses to access your warehouse. This creates a local site configuration file inside your home directory: ~/.pb/siteconfig.yaml.

Workflow

Retrieve the name of the connection you would like to use from the siteconfig.yaml. Then, run the below command in your terminal to start a new tutorial session:

pb tutorial --connection <connection_name>

For example, if your siteconfig.yaml looks as follows:

connections:
 snowflake-conn:
 target: snowflake
 outputs:
 snowflake:
 type: snowflake
 account: xxxxx

In that case, your connection name is snowflake-conn.

The session takes between 30 minutes to one hour to complete. Once the tutorial is complete, the session automatically ends within your terminal.

Tutorial components

Introduction to the tutorial and how to interact with it.
Introduction the the fictional business to provide context to the sample data.
Profiles project creation.
Profiles runs: Create an ID Graph.
ID graph QA - Phase 1
Fix the over stitching issues that happen in the first run.
ID graph QA - Phase 2: This includes more detailed analysis and sample queries to run to debug some users who got merged.
Run again to produce a healthy ID graph.
Feature creation: You will build a handful of features on the user entity defined in the tutorial.
Final run to produce a feature view, which is a view within the warehouse where each record is a unique user and each column is a defined feature or attribute on that user.

Get started with your own project

Once you exit the tutorial session and want to begin a a new Profiles project with your own data, follow this Quickstart guide to get started.

FAQ

In addition to the below FAQs, see the FAQs on Python-related errors for more information on debugging issues related to running the tutorial.

Is this tutorial powered by an LLM?

No, this tutorial is based on a simple static script meant as a step-by-step guide with some simple built-in validations.

Does the tutorial session seed my warehouse with data?

Yes, a part of the tutorial will seed your target schema (specified in the connection setup) with sample data for Secure Solutions LLC. It will add the below three tables: 1. PAGES 2. TRACKS 3. IDENTIFIES

These tables will serve as source data for the Profiles configuration built during the tutorial session, for runs.

Does the tutorial output any tables in my warehouse?

Yes, the source data (see above) is used as an input into Profiles’ semantic models configured in the tutorial session.

The materialized tables and views will output to the same target schema configured in the site configuration/connection setup step.

Archive: Create New Profiles Project

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers the detailed steps on creating a Profiles project from scratch. It also details how to:

Build a virtual Python environment
Install Profiles
Grant the necessary warehouse permissions for the project
Establish the warehouse connection
Generate the resulting project template

Video walkthrough

Create virtual environment

Follow these steps to create a Python virtual environment:

python3 -m venv .venv
source .venv/bin/activate

Install Profiles

Install the Profiles Builder (PB) tool:

pip3 install profiles-rudderstack

If you have already installed PB, you can update its version:

pip3 install profiles-rudderstack -U

Validate Profile Builder’s version after installing:

pb version

RudderStack recommends using a Python virtual environment to maintain an isolated and clean environment.
If you are an existing user, migrate your project to the new schema.
See the Profiles setup and installation FAQ for troubleshooting any issues.

Grant warehouse permissions

RudderStack supports Snowflake, Redshift, Databricks, and BigQuery for creating your Profiles project.

RudderStack requires specific warehouse permissions to read data from the schema having source tables (for example, tracks and identifies tables generated via Event Stream sources), and write data in a new schema created for Profiles.

RudderStack recommends keeping separate schemas for projects running via CLI and web. This way, the projects run from the CLI will never risk overwriting your production data.

Snowflake

Snowflake uses a combination of DAC and RBAC models for access control. However, RudderStack chooses an RBAC-based access control mechanism as multiple users can launch the Profile Builder CLI.

Also, it is not ideal to tie the result of an individual user run with that user. Therefore, it is recommended to create a generic role (for example, PROFILES_ROLE) with the following privileges:

Read access to all the inputs to the model (can be shared in case of multiple schemas/tables).
Write access to the schemas and common tables as the PB project creates material (output) tables.

To access any material created from the project run, the above role (PROFILES_ROLE in this case) must also have read access to all of those schemas.

The following sample commands grant the required privileges to the role (PROFILES_ROLE) in a Snowflake warehouse:

-- Create role
CREATE ROLE PROFILES_ROLE;
SHOW ROLES; -- To validate

-- Create user
CREATE USER PROFILES_TEST_USER PASSWORD='<strong_password>' DEFAULT_ROLE='PROFILES_ROLE';
SHOW USERS; -- To validate

-- Grant role to user and database
GRANT ROLE PROFILES_ROLE TO USER PROFILES_TEST_USER;
GRANT USAGE ON DATABASE YOUR_RUDDERSTACK_DB TO ROLE PROFILES_ROLE;

-- Create separate schema for Profiles and grant privileges to role
CREATE SCHEMA YOUR_RUDDERSTACK_DB.RS_PROFILES;
GRANT ALL PRIVILEGES ON SCHEMA YOUR_RUDDERSTACK_DB.RS_PROFILES TO ROLE PROFILES_ROLE;
GRANT USAGE ON WAREHOUSE RUDDER_WAREHOUSE TO ROLE PROFILES_ROLE;
GRANT USAGE ON SCHEMA YOUR_RUDDERSTACK_DB.EVENTSSCHEMA TO ROLE PROFILES_ROLE;

For accessing input sources, you can individually grant select on tables/views, or give blanket grant to all in a schema.

-- Assuming we want read access to tables/views in schema EVENTSSCHEMA
GRANT SELECT ON ALL TABLES IN SCHEMA YOUR_RUDDERSTACK_DB.RS_PROFILES TO PROFILES_ROLE;
GRANT SELECT ON FUTURE TABLES IN SCHEMA YOUR_RUDDERSTACK_DB.RS_PROFILES TO PROFILES_ROLE;
GRANT SELECT ON ALL VIEWS IN SCHEMA YOUR_RUDDERSTACK_DB.RS_PROFILES TO PROFILES_ROLE;
GRANT SELECT ON FUTURE VIEWS IN SCHEMA YOUR_RUDDERSTACK_DB.RS_PROFILES TO PROFILES_ROLE;

-- Assuming we want read access to tracks and identifies tables in schema EVENTSSCHEMA
GRANT SELECT ON TABLE YOUR_RUDDERSTACK_DB.RS_PROFILES.TRACKS TO PROFILES_ROLE;
GRANT SELECT ON TABLE YOUR_RUDDERSTACK_DB.RS_PROFILES.IDENTIFIES TO PROFILES_ROLE;

Redshift

Suppose the inputs/edge sources are in a single schema website_eventstream and the name of the newly created Profiles user is rudderstack_admin. In this case, the requirements are as follows:

A separate schema rs_profiles (to store all the common and output tables).
The rudderstack_admin user should have all the privileges on the above schema and the associated tables.
The rudderstack_admin user should have USAGE privilege on schemas that have the edge sources and input tables (website_eventstream) and read (SELECT) privileges on specific tables as well. This privilege can extend to the migration schema and other schemas from where data from warehouses comes in.
The rudderstack_admin user should have privileges to use plpythonu to create some UDFs.

The sample commands are as follows:

CREATE USER rudderstack_admin WITH PASSWORD '<strong_unique_password>';
CREATE SCHEMA rs_profiles;
GRANT ALL ON SCHEMA "rs_profiles" TO rudderstack_admin;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA "rs_profiles" TO rudderstack_admin;
GRANT USAGE ON SCHEMA "website_eventstream" TO rudderstack_admin;
GRANT USAGE ON LANGUAGE plpythonu TO rudderstack_admin;

GRANT SELECT ON ALL TABLES IN SCHEMA "website_eventstream" TO rudderstack_admin;

To give access to only specific input tables/views referred in your Profiles project, use the below command:

GRANT SELECT ON TABLE "<YOUR_SCHEMA>"."<YOUR_TABLE>" TO rudderstack_admin;

Supported inputs

RudderStack supports the following input types for Redshift warehouse/serverless:

Redshift cluster with DC2 type nodes with following types as inputs:
- Redshift internal tables
- External schema and tables only for inputs (not supported as output)
- CSV files stored on S3 as inputs
Redshift cluster with RA3 type nodes with following types as inputs:
- Redshift internal tables
- External schema and tables only for inputs (not supported as output)
- CSV files stored on S3 as inputs
  - Cross DB input tables
Redshift serverless with following types as inputs:
- Redshift internal tables
- External schema and tables (not supported as output)
- CSV files stored on S3 as inputs
  - Cross DB input tables RudderStack also supports various authentication mechanisms to authenticate the user running the Profiles project. Refer Redshift warehouse connection for more information.

Databricks

Open the Databricks UI.
Create a new user.
Reuse an existing catalog or create a new one by clicking Create Catalog.
Grant USE SCHEMA privilege on the catalog.
Create a separate schema to write objects created by RudderStack Profiles.
Grant all privileges on this schema.
Grant privileges to access relevant schemas for the input tables. For example, if an input schema is in a schema named website_eventstream, then you can run the following commands to assign a blanket grant to all schemas or only specific tables/views referred in your Profiles project:

CREATE USER rudderstack_admin WITH PASSWORD <strong_unique_password>;
GRANT USE SCHEMA ON CATALOG <catalog name> TO rudderstack_admin;
CREATE SCHEMA RS_PROFILES;
GRANT ALL PRIVILEGES ON SCHEMA RS_PROFILES TO rudderstack_admin;
GRANT SELECT ON SCHEMA website_eventstream TO rudderstack_admin;

GRANT SELECT ON ALL TABLES IN SCHEMA "website_eventstream" TO rudderstack_admin;

To give access to only specific input tables/views referred in your Profiles project, use the below command:

GRANT SELECT ON TABLE public.input_table TO rudderstack_admin;

BigQuery

For BigQuery, RudderStack recommends you to use a view instead of table for streaming datasets.

You must first assign the BigQuery Job User role to your service account. Follow these steps:

Open the BigQuery UI (Google Cloud Console).
Select IAM & Admin > IAM from the left sidebar.
Click GRANT ACCESS to grant permissions to your service account.
Enter your service account email in the New principals field.
In the Assign roles section, select BigQuery Job User role from the role list.
Click Save.

Alternatively, you can add the following IAM Policy Binding:

{
 "bindings": [
 {
 "role": "roles/bigquery.jobUser",
 "members": [
 "serviceAccount:your-service-account@your-project.iam.gserviceaccount.com"
 ]
 }
 ]
}

Then, assign the dataset or project level roles to your service account:

Open the BigQuery UI (Google Cloud Console).
Select BigQuery from the sidebar.
Select the dataset to which you want to grant access.
In the Dataset info window, click SHARING > Permissions.
Click Add Principal.
Enter the service account email in the New principals field.
In the Assign roles section, select the following roles from the role list:
- BigQuery Data Viewer: Allows read access to the dataset.
- BigQuery Data Editor: Allows read and write access to the dataset.

Alternatively, you can add the following IAM Policy Binding:

{
 "bindings": [
 {
 "role": "roles/bigquery.dataViewer",
 "members": [
 "serviceAccount:your-service-account@your-project.iam.gserviceaccount.com"
 ],
 "condition": {
 "title": "Access to dataset1",
 "description": "Allow data viewing access to dataset1",
 "expression": "resource.name.startsWith('projects/your-project-id/datasets/dataset1')"
 }
 },
 {
 "role": "roles/bigquery.dataEditor",
 "members": [
 "serviceAccount:your-service-account@your-project.iam.gserviceaccount.com"
 ],
 "condition": {
 "title": "Access to dataset2",
 "description": "Allow data editing access to dataset2",
 "expression": "resource.name.startsWith('projects/your-project-id/datasets/dataset2')"
 }
 }
 ]
}

If you have input relations in many datasets from a single project, you can assign the required privileges for the complete Google Cloud project. This allows your service account to access the relation from all the datasets of the project.

Open the BigQuery UI (Google Cloud Console).
Select IAM & Admin > IAM from the left sidebar.
Click GRANT ACCESS to grant permissions to your service account.
Enter the service account email in the New principals field.
In the Assign roles section, select the following roles from the role list:
- BigQuery Data Viewer: Allows read access to the dataset.
- BigQuery Data Editor: Allows read and write access to the dataset.

Alternatively, you can add the following IAM Policy Binding:

{
 "bindings": [
 {
 "role": "roles/bigquery.dataViewer",
 "members": [
 "serviceAccount:your-service-account@your-project.iam.gserviceaccount.com"
 ]
 },
 {
 "role": "roles/bigquery.dataEditor",
 "members": [
 "serviceAccount:your-service-account@your-project.iam.gserviceaccount.com"
 ]
 }
 ]
}

Create warehouse connection

After granting the required permissions to your warehouse, you must create the warehouse connection to allow Profiles to access your data.

Initiate the warehouse connection:

pb init connection

Then, follow the prompts to enter further details about your warehouse.

A sample connection for a Snowflake account is as follows:

Enter Connection Name: test
Enter target: (default:dev): # Press enter, leaving it to default
Enter account: ina13147.us-east-1
Enter warehouse: rudder_warehouse
Enter dbname: your_rudderstack_db
Enter schema: rs_profiles # A segregated schema for storing tables/views created by Profiles
Enter user: profiles_test_user
Do you want to use key-pair authentication? [y/N] y
--- If you select yes ----
Enter file path containing value for privateKey: /<path>/key.pem
Enter passphrase (leave blank if private key not encrypted):
-- If you select no----
Enter password: <password>
--common for both--
Enter role: profiles_role
Append to /Users/<user_name>/.pb/siteconfig.yaml? [y/N] y

Connection Name: Name of the connection in the project file.
Target: Environment name, such as dev, prod, test, etc. You can specify any target name and create a separate connection for the same.
Account: Name of your Snowflake account. Based on your cloud platform and region, you might need to append .aws, .gcp, or .azure in your account name. See Snowflake documentation for more information.
Warehouse: Name of the warehouse.
Database name: Name of the database inside warehouse where model outputs will be written.
Schema: Name of the schema inside database where you’ll store identity stitcher and entity features.
User: Name of the user in data warehouse.
Use Snowflake key-pair authentication: RudderStack supports Snowflake’s key-pair authentication mechanism to validate your Snowflake connection.

Security

For enhanced security, RudderStack recommends using the key pair authentication over the basic authentication mechanism (username and password).

See the following sections in the Snowflake documentation to generate and use the key pair:

If you opt for Snowflake key-pair authentication, then enter the following settings:

Private key: Enter the file path containing the private key, for example, /<path>/<private_key>.pem.
Passphrase: Specify the password you set while encrypting the private key. Leave this field blank if your private key is not encrypted.

The user authentication will fail if your private key is encrypted and you do not specify the passphrase.

Deprecated: If you opt for username and password authentication, then enter the following settings:

Password: Password for the above user.

The username/password authentication mechanism is deprecated. RudderStack recommends using key-pair authentication instead.

Then, continue with the connection setup by specifying the below settings:

Role: Name of the user role.
Append to site configuration file: Determines whether to add the connection details to the siteconfig.yaml file.

RudderStack supports various user authentication mechanisms for Redshift:

Enter Connection Name: test
Enter target: (default:dev): # Press enter, leaving it to default
Enter dbname: your_rudderstack_db
Enter schema: rs_profiles # A segregated schema for storing tables/views created by Profiles
Enter user: profiles_test_user
Enter sslmode: options - [disable require]: disable # Enter "require" in case your Redshift connection mandates sslmode.
How would you like to authenticate with the warehouse service? Please select your method by entering the corresponding number:
y
[1] Warehouse Credentials: Log in using your username and password.
 Format: [Username, Password]
[2] AWS Programmatic Credentials: Authenticate using one of the following AWS credentials methods:
 a) Direct input of AWS Access Key ID, Secret Access Key, and an optional Session Token.
 Format: [AWS Access Key ID, Secret Access Key, Session Token (Optional)]
 b) AWS configuration profile stored on your system.
 Format: [AWS Configuration Profile Name]
 c) Use an AWS Secrets Manager ARN to securely retrieve credentials.
 Format: [Secret ARN]

Connection Name: Name of the connection in the project file.
Target: Environment name, such as dev, prod, test, etc. You can specify any target name and create a separate connection for the same.
Host: Log in to AWS Console and go to Clusters to know about host.
Port: Port number to connect to the warehouse.
Database name: Name of the database inside warehouse where model outputs will be written.
Schema: Name of the schema inside database where you’ll store identity stitcher and entity features.
User: Name of the user in data warehouse.
Password: Password for the above user.

Warehouse credentials

If you choose to use the warehouse credentials (option 1), enter the following details:

Enter host: warehouseabc.us-west-1.redshift.amazonaws.com
Enter port: 5439
Enter password: <password>
Enter tunnel_info: Do you want to use SSH Tunnel to connect with the warehouse? (y/n): y
Enter ssh user: user
Enter ssh host: 1234
Enter ssh port: 12345
Enter ssh private key file path: /Users/alex/.ssh/id_ed25519.pub
Append to /Users/alex/.pb/siteconfig.yaml? [y/N] y

AWS Programmatic Credentials

If you choose to use AWS Access Key ID, Secret Access Key, and an optional Session Token from AWS Programmatic Credentials (option 2a), enter the following details:

For Redshift Cluster

Which Redshift Service you want to connect with?
 [c]Redshift Cluster
 [s]Redshift Serverless
c
Enter Redshift Cluster Identifier: cluster-id
Enter access_key_id: aid
Enter secret_access_key: ***
Enter session_token: If you are using temporary security credentials, please specify the session token, otherwise leave it empty.
stoken
Append to /Users/alex/.pb/siteconfig.yaml? [y/N] y

For Redshift Serverless

Which Redshift Service you want to connect with?
 [c]Redshift Cluster
 [s]Redshift Serverless
s
Enter Redshift Serverless Workgroup Name: wg-name
Enter access_key_id: aid
Enter secret_access_key: ***
Enter session_token: If you are using temporary security credentials, please specify the session token, otherwise leave it empty.
Append to /Users/alex/.pb/siteconfig.yaml? [y/N] y

If you choose to use AWS configuration profile stored on your system from AWS Programmatic Credentials (option 2b), enter the following details:

For Redshift Cluster

Which Redshift Service you want to connect with?
 [c]Redshift Cluster
 [s]Redshift Serverless
c
Enter Redshift Cluster Identifier: ci
Enter shared_profile: default
Enter region: us-east-1
Append to /Users/alex/.pb/siteconfig.yaml? [y/N] y

For Redshift Serverless

Which Redshift Service you want to connect with?
 [c]Redshift Cluster
 [s]Redshift Serverless
s
Enter Redshift Serverless Workgroup Name: serverless-wg
Enter shared_profile: default
Append to /Users/alex/.pb/siteconfig.yaml? [y/N] y

If you choose to use an AWS Secrets Manager ARN to securely retrieve credentials from AWS Programmatic Credentials (option 2c), enter the following details:

For Redshift Cluster

Which Redshift Service you want to connect with?
 [c]Redshift Cluster
 [s]Redshift Serverless
c
Enter Redshift Cluster Identifier: cluster-identifier
Enter secrets_arn: ************************
Append to /Users/alex/.pb/siteconfig.yaml? [y/N] y

A sample connection for a Databricks account is as follows:

Enter Connection Name: test
Enter target: (default:dev): # Press enter, leaving it to default
Enter host: a1.8.azuredatabricks.net # The hostname or URL of your Databricks cluster
Enter port: 443 # The port number used for establishing the connection. Usually it is 443 for https connections.
Enter http_endpoint: /sql/1.0/warehouses/919uasdn92h # The path or specific endpoint you wish to connect to.
Enter access_token: <password> # The access token created for authenticating the instance.
Enter user: profiles_test_user # Username of your Databricks account.
Enter schema: rs_profiles # A segregated schema for storing tables/views created by Profiles
Enter catalog: your_rudderstack_db # The database or catalog having data that you’ll be accessing.
Append to /Users/<user_name>/.pb/siteconfig.yaml? [y/N]
y

Connection Name: Name of the connection in the project file.
Target: Environment name, such as dev, prod, test, etc. You can specify any target name and create a separate connection for the same.
Host: Host name or URL of your Databricks cluster.
Port: Port number for establishing the connection, usually 443 for https connections.
http_endpoint: Path or specific endpoint you wish to connect to.
access_token: Access token for authenticating the instance.
User: Username of your Databricks account.
Schema: Name of the schema to store your output tables/views.
Catalog: Name of the database or catalog from where you want to access the data.

RudderStack currently supports Databricks on Azure. To get the Databricks connection details:

Log in to your Azure’s Databricks Web UI.

Click on SQL Warehouses on the left.

Select the warehouse to connect to.

Select the Connection Details tab.

A sample connection for a BigQuery account is as follows:

Enter Connection Name: test
Enter target: (default:dev): # Press enter, leaving it to default
Enter credentials: json file path: # File path of your BQ JSON file, for example, /Users/alexm/Downloads/big.json. Entering an incorrect path will exit the program.
Enter project_id: profiles121
Enter dataset: rs_profiles
Append to /Users/<user_name>/.pb/siteconfig.yaml? [y/N]
y

Connection Name: Name of the connection in the project file.
Target: Environment name, such as dev, prod, test, etc. You can specify any target name and create a separate connection for the same.
Credentials: File path of your BigQuery JSON credentials file.
project_id: Your Google Cloud project ID.
Dataset: Name of the dataset where the output tables/views created by Profiles will be stored.
Append to site configuration file: Determines whether to add the connection details to the siteconfig.yaml file.

This creates a local site configuration file in your home directory: ~/.pb/siteconfig.yaml. Your Profiles project uses this file to access the warehouse, Git credentials, and other details. If you don’t see the file, enable the View hidden files option.

Project template

Finally, create your Profiles project:

pb init pb-project -o MyProfilesProject

The above command creates a new project in the MyProfilesProject folder with the following structure:

See Project structure for more information on the Profiles project files.

Archive: Project Structure

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks you through the Profiles project structure generated once you initialize a new project using the pb init command.

Initial project structure

After initializing a new Profiles project, you will see the following project folder structure:

.
├── logs/logfile.log
├── models/
│ ├── inputs.yaml
│ └── profiles.yaml
├── output/
├── .gitignore
├── pb_project.yaml
└── README.md

Folder	Description
`logs/`	Stores the `logfile.log` file containing the logs from all `pb` runs.
`models/`	Stores the `inputs`, `profiles`, `sql_models`, and `macros` YAML files.
`output/`	Stores the compiled SQL files from each project run. These files are then executed in your warehouse.

File	Description
`.gitignore`	Specifies the files that Git ignores.
`pb_project.yaml`	Profiles project configuration.
`README.md`	Overview on using `pb` and the SQL queries for analysis.

Full project structure

After adding sql_models.yaml and macros.yaml files (if used), the project structure looks as follows:

.
├── logs/logfile.log
├── models/
│ ├── inputs.yaml
| ├── macros.yaml
| ├── profiles.yaml
│ └── sql_models.yaml
├── output/
├── .gitignore
├── pb_project.yaml
└── README.md

SQL output structure

Profiles projects are compiled into SQL files which are executed in your data warehouse.

These compiled SQL queries are stored in a hierarchy of folders in the output folder.

output/
└── <target>/
 └── seq_no/
 ├── 1/
 ├── 2/
 ├── ...
 └── latest/
 ├── compile/
 └── run/

The output folder contains folders for each warehouse connection target (for example, dev, prod, etc.).

Within each target folder is a seq_no folder containing separate folders for each sequence number (1, 2, 3, etc.), along with a latest symbolic link for the latest sequence number. The sequence number folders contain compile (compiled SQL files, staged for execution) and run (post execution SQL files) folders storing the SQL files and folders containing the SQL files.

Naming convention

The SQL files and folders containing SQL files are named in the following format: Material_<name>_<hash>_<seq_no>.

For example: Material_product_tier_32cf8f89_1

Archive: Site Configuration File

Mon, 01 Jan 0001 00:00:00 +0000

RudderStack creates a site configuration file (~/.pb/siteconfig.yaml) while creating a warehouse connection.

This guide walks you through the site configuration file spec in detail.

Overview

The site configuration file (~/.pb/siteconfig.yaml) contains the following details including secrets (if any):

Warehouse connection details and its credentials.
Git repository connection credentials (if any). Follow these steps to add the credentials:
1. Generate the SSH Key.
2. Associate the SSH key to your Git project.
3. Add private key as credentials in the siteconfig.yaml file under key field in the gitcreds field.

You can store the details for multiple warehouse connections in the same site configuration file. This is helpful in cases where you have multiple Profiles projects with different warehouse connections.

Example

A sample site configuration file containing multiple warehouse connection details is shown below:

connections:
 # connection name
 prod-db-profile:
 target: dev
 outputs:
 dev:
 account: inb828.us-west-3
 dbname: MAT_STORE_DEV
 password: password_dev
 role: PROFILES_ROLE_DEV
 schema: AB_SCHEMA_DEV
 type: snowflake
 user: profiles_demo
 warehouse: DEV_WAREHOUSE
 prod:
 account: inc654.us-west-3
 dbname: MAT_STORE
 password: password
 role: PROFILES_ROLE
 schema: AB_SCHEMA
 type: snowflake
 user: profiles_demo
 warehouse: PROD_WAREHOUSE
 test-db-profile:
 target: test
 outputs:
 db:
 access_token: dabasihasdho
 catalog: rs_dev
 host: adb-98.18.azuredatabricks.net
 http_endpoint: /sql/1.0/warehouses/919uasdn92h
 port: 443
 schema: rs_profiles
 type: databricks
 user: johndoe@abc.onmicrosoft.com
 dev:
 account: uk12.us-west-1
 dbname: RUDDERSTACK_DB
 password: password
 role: RS_ROLE
 schema: RS_PROFILES
 type: snowflake
 user: johndoe
 warehouse: RS_WAREHOUSE
 snowflake-keypair: # example of an unencrypted Snowflake key-pair
 type: snowflake
 account: vb8.us-east-1
 dbname: PROD_DB
 role: PROFILES_ROLE
 warehouse: RUDDER
 schema: RS_PROFILES
 user: PROFILES_USER_UNC
 useKeyPairAuth: true
 privateKey: <PRIVATE_KEY_WITH_BEGIN_AND_END_TAGS>
 snowflake-encrypted-keypair: # example of an encrypted Snowflake key-pair
 type: snowflake
 account: vb8.us-east-1
 dbname: PROD_DB
 role: PROFILES_ROLE
 warehouse: RUDDER
 schema: RS_PROFILES
 user: PROFILES_USER_EC
 useKeyPairAuth: true
 privateKey: <PRIVATE_KEY_WITH_BEGIN_AND_END_TAGS>
 privateKeyPassphrase: valuegoeshere!
 redshift_v1:
 dbname: warehouse_rs
 host: warehouse.abc.us-east-3.redshift.amazonaws.com
 password: password
 port: 5419
 schema: rs_profiles
 type: redshift
 user: redshift_user
 redshift_v2:
 workgroup_name: warehouse_workgroup
 region: us-east-1
 driver: v2
 sslmode: require
 dbname: warehouse_rs
 schema: rs_profiles
 type: redshift
 access_key_id: ******************
 secret_access_key: ******************************
 big:
 credentials:
 auth_provider_x509_cert_url: https://www.googleapis.com/oauth2/v1/certs
 auth_uri: https://accounts.google.com/o/oauth2/auth
 client_email: johndoe@big-query-integration-poc.iam.gserviceaccount.com
 client_id: "123345678909872"
 client_x509_cert_url: https://www.googleapis.com/robot/v1/metadata/x509/johndoe%40big-query-integration-poc.iam.gserviceaccount.com
 private_key: <PRIVATE_KEY_WITH_BEGIN_AND_END_TAGS>
 private_key_id: <PRIVATE_KEY_ID>
 project_id: big-query-integration-poc
 token_uri: https://oauth2.googleapis.com/token
 type: service_account
 project_id: rs_profiles
 schema: rs_profiles
 type: bigquery
 user: johndoe@big-query-integration-poc.iam.gserviceaccount.com
gitcreds:
 - reporegex: "git@github.com:REPO_OWNER/*" # in case of ssh url
 key: <OPENSSH_PRIVATE_KEY_WITH_BEGIN_AND_END_TAGS>
 - reporegex: "https://github.com/rudderlabs/*" # https url
 basic_auth:
 username: oauth2
 password: ... # your GitHub access token with read permission
block_store_creds:
 - type: minio # type Minio
 bucket: test
 endpoint: https://localhost:9000/
 access_key_id: access_key_id
 secret_access_key: secret_access_key
 - type: s3 # S3 credentials with access key
 bucket: shubham # Optional; use creds for this bucket
 region: us-east-1
 access_key_id: access_key_id
 secret_access_key: secret_access_key
 session_token: session_token
 - type: s3 # S3 credentials with ARN
 bucket: shopify
 region: us-east-1
 secrets_arn: secrets_arn
 - type: s3 # S3 credentials with shared profile
 bucket: shopify
 region: us-east-1
 shared_profile: shared_profile
py_models:
 enabled: true # in case you are using Python models in your project, else set it to false
 python_path: /opt/anaconda3/bin/python # the path where Python is installed (run `which python` to get the full path). If `py_models` is not enabled, set it to `""`. For Windows, you may pass the path value as: python.exe
 credentials_presets: null
 allowed_git_urls_regex: ""
cache_dir: /Users/YOURNAME/.pb/WhtGitCache/ # For Windows, the directory path will have forward slash (\)
filepath: /Users/YOURNAME/.pb/siteconfig.yaml # For Windows, the file path will have forward slash (\)

RudderStack recommends defining two target within a single connection- one for dev/staging and the other for production.

For example, see the connection named prod-db-profile in the sample file above. The default target is set to dev, as specified by the target key. To change the output to production, you can run pb run -t prod.

Archive: Create and Manage Profiles Projects with Profiles IDE

Mon, 01 Jan 0001 00:00:00 +0000

The Profiles IDE is a web-based editor in the RudderStack dashboard that lets you develop your Profiles projects in a production-ready environment.

You can build your project and manage the YAML configuration files completely within the RudderStack dashboard or convert it to a version control repository like GitHub, GitLab, at any time.

To use Profiles IDE as an AI-powered, web-based editor, see Use Profiles IDE with Rudder AI.

Profiles IDE features

Integrated editor for building and modifying YAML-based Profiles configuration files.
Safe validation of workflows through separate development schemas.
Easy rollbacks of uncommitted changes with a button click.
Automated notifications when multiple users are editing a project simultaneously.

Video walkthrough

Use Profiles IDE

You can perform the following operations using the Profiles IDE:

Create a new Profiles Project: Select the Web-based editor (IDE) option in the RudderStack dashboard and follow the instructions. Then, you can modify the inputs, build an ID graph and features, and validate outputs in your data warehouse.
Edit an existing Profiles Project: Edit the project even if it is stored and managed fully within RudderStack or connected to your version control repository, for example, GitHub, GitLab, etc.

In your Profiles project, navigate to Settings and click Editor mode.

Next, enter the schema name you want to use for development purposes for your editing session and click Continue.

RudderStack recommends using a development schema while editing your project so that you can modify your YAML configuration files, run your changes within the editor, and validate the outputs in a separate development schema.

This makes it convenient to test your changes in the development schema without affecting the outputs in your production schema.

Once you make the changes, you can easily commit them to your RudderStack managed production project.

If you’re using a version control repository, RudderStack will push the commits to the main branch of the remote repository so that you have full access to your commit history. This allows visibility into the historical changes, easy rollbacks, and a reliable project sourced from a repository that can be managed both from the IDE and your local environment.

FAQ

What is the difference between a RudderStack-hosted project and a Git-connected project?

RudderStack-hosted project	Git-connected project
The files are hosted completely within RudderStack.	The files are stored in the repository of your version control platform.
You need not set up and connect your version control provider but you will not have access to the commit history or the ability to revert changes.	You have access to your commit history and can easily rollback any changes.

You can convert your RudderStack hosted project to a Git repository at any point of time.

What is the difference between the save and commit operations?

Save operation: When a new session starts, the current project loads into the session’s browser storage. Any changes you save are added to the browser storage connected to that session and used during runs performed within that session. Note that the saved changes do not affect the production project.

Commit operation: Once you validate the changes and want to merge them into the production project, you can commit them to save the changes to your production project.

Why does the IDE ask me for a schema for each session?

By default, the production project writes its outputs (ID graph, feature views, etc) to a defined schema. Each session prompts you for a schema to save that session’s (or user’s) outputs to so that they are run and validated without affecting the production outputs.

RudderStack recommends using a separate schema for development and the production project so that the editing sessions do not interfere with connected pipelines, activations or audiences.

Multiple users can use the same or separate development schemas to work in isolation and validate outputs directly in the data warehouse.

Which Profiles versions can I run in the IDE?

The IDE sources the Profiles version in the same way as the CLI. So, RudderStack uses the version defined in your IDE configuration to run the project and it can be deployed to the production project as well.

What if multiple users make changes to the Profiles project?

All changes are tied to a user/browser session. The pushed changes will cause conflicting open sessions to reset to the commited changes.

Saving and running the changes within a given user session does not affect the production project (whether RudderStack hosted or Git hosted) until you commit them.

What if the origin changes during the editing session?

If the origin changes during an editing session(s), RudderStack pulls down and applies the remote changes before applying the editor session changes.

Do I need to provide write access to my SSH deploy key?

Yes, if you are using the IDE with a Git-connected repository, you will need to provide the write access to your SSH deploy key so that the RudderStack app can commit your changes directly to your remote Git repository.

If you did not initially provide write access, you can use a new SSH deploy key and grant write access to the repository.

Which Git URLs does the Profiles IDE support?

RudderStack supports the SSH URLs and the below ones while working with the Profiles IDE:

Writes to the main branch of the repository: git@<provider-host>:<org-name>/<repo-name>
Writes to a specific branch: git@<provider-host>:<org-name>/<repo-name>/tree/<branch-name>

RudderStack does not support the below URLs while working with Profiles IDE:

git@<provider-host>:<org-name>/<repo-name>/tree/branch-name/subfolder
git@<provider-host>:<org-name>/<repo-name>/tag/v1.0.0
git@<provider-host>:<org-name>/<repo-name>/commit/<commit-hash>

In addition, you can use these SSH Git URLs to read your Profiles project.

Why am I unable to see the past runs of my Profiles project in the History tab, even though I ran it through my IDE?

RudderStack stores the project runs triggered from your IDE based on your session ID, not the project ID. Hence, your past runs won’t appear in the History tab.

Why does the red dot keep showing next to my saved YAML file?

The red dot indicates that you have uncommitted changes to your file. To remove the dot, you’ll need to commit these changes to your remote repository.

Will my run history and preferences be the same if I access the same project from a different machine?

No, RudderStack ties your run history and user preferences to a specific session ID, which is unique to each machine. Therefore, accessing the same project from different machines will result in different histories and preferences.

Why can’t I see all the files in my Profiles project in the IDE?

Currently, the RudderStack IDE only supports viewing and editing the YAML and SQL files. Any other file types within your Git repository like text or image files won’t be visible in the IDE.

Can I change the Git URL of my Profiles project in the RudderStack dashboard after it’s been set up?

No, it’s not possible to modify the Git repository URL of an existing Profiles project. However, you can always create a new project with the required Git URL.

Archive: pb_project.yaml

Mon, 01 Jan 0001 00:00:00 +0000

The pb_project.yaml file contains the overall project configuration. This includes the entities and id_types that are used throughout the project.

Video walkthrough

Overview

The pb_project.yaml file is saved in the top level of a Profiles project.

name: project_name
schema_version: version_number
connection: connection_name
model_folders:
 - directories
entities:
 <entity-config>
id_types:
 <id_type-config>
python_requirements: # Optional
 - python_package==version
carry_forward_privileges: true / false

Field	Description
`name` Required	Project name.
`schema_version` Required	Project’s YAML version.
`connection` Required	Connection name from `siteconfig.yaml`.
`model_folders` Required	Path to the folder containing the model files.
`entities` Required	List of entities used in the project.
`id_types` Required	Configuration of all ID types across all entities.
`python_requirements`	List of Python packages and versions required for the project, specifically used for controlling the Profiles version.
`carry_forward_privileges`	Boolean that determines whether the privileges granted on the views should be retained after the view definitions are updated. This resolves the issue where recreating views for each model in the project led to revoked privileges. Note: Using this key can lead to some performance overhead.

Example

# Project name
name: sample_attribution
schema_version: 85
connection: prod-db-profile
model_folders:
 - models
entities:
 - name: user
 id_column_name: user_rud_id # Optional
 id_types:
 - user_id
 - anonymous_id
 - email
 feature_views:
 using_ids:
 - id: email
 name: customer_profile_by_email
id_types:
 - name: anonymous_id
 - name: email
 filters:
 - type: include
 regex: ".+@.+"
 maximum_edges:
 - anonymous_id: 2
 - user_id: 1
 - name: user_id
 filters:
 - type: exclude
 value: "test"

carry_forward_privileges: false

Archive: profiles.yaml

Mon, 01 Jan 0001 00:00:00 +0000

The profiles.yaml file defines the Profiles semantic models using models and var_groups.

models are defined by the model_type and consist of prepackaged SQL code that takes your inputs and runs it in your warehouse to output the defined views and tables.
var_groups define entity_var and input_var used to define the features for feature views.

models:
 <model-config>
var_groups:
 <var-group-config>

Key	Description
`model_config` Required	Model configuration - see the specific model for details.
`var_group` Required	Configuration for the features.

Models

Example

models:
 - name: user_id_stitcher
 model_type: identity_stitcher
 model_spec:
 entity_key: user
 materialization:
 run_type: incremental
 edge_sources:
 - from: inputs/rsIdentifies
 - from: inputs/rsTracks
var_groups:
 name: user_features
 entity_key: user #Name as defined in the project file.
 vars:
 - entity_var:
 name: first_seen
 select: min(timestamp::date)
 from: inputs/rsTracks
 description: First user appearance
 - entity_var:
 name: last_seen
 select: max(timestamp::date)
 from: inputs/rsTracks
 is_feature: false
 - entity_var:
 name: user_lifespan
 select: '{{user.last_seen}} - {{user.first_seen}}'
 description: Lifespan of a user

Archive: inputs.yaml

Mon, 01 Jan 0001 00:00:00 +0000

The inputs.yaml file contains the configuration of all input sources used in Profiles. This includes the ID graph, entity vars, feature view, etc.

inputs:
 - name: input_name
 app_defaults:
 table: schema.table # Key must be only one of table/view/s3/csv
 occurred_at_col: time_series_column
 row_identifier: [list of identifiers for composite key]  # Optional
 ids:
 <id-config>
 where: <filtering_condition>

Key Description

name
Required The input name.

contract

Helps you handle any potential errors if the required column doesn’t exist.

Key	Description
`is_optional`	Boolean that indicates if the model is optional.
`is_event_stream`	Boolean indicating if the data is Event Stream (contains a `timestamp` column).
`with_entity_ids`	List of all entities.
`with_columns`	List of all ID columns that the contract is applicable for.

app_defaults
Required Input configuration.

table\ view\ s3\ csv
Required Source name, see input sources.

occurred_at_col
Required Time series column.

row_identifier List of all the identifiers that combine to create a composite primary key.

ids
Required All IDs in the input source, mapped to entities and id_types

where The WHERE clause specifying the filtering condition.

Note: The filtered row will not be considered for ID stitching as well.

Input sources

A source can be a warehouse table, view, Amazon S3 bucket, or CSV file. Only one key can be used per source.

Key	Description
`table`	Warehouse table name You can also refer to tables from other databases/schemas in the same data warehouse.
`view`	Warehouse view name.
`csv`	CSV file path. Note that this path should be relative to the project folder. Note: RudderStack recommends using CSV file as an input only if you have limited amount of data.
`s3`	Amazon S3 URI path.

Video walkthrough

Archive: sql_models.yaml

Mon, 01 Jan 0001 00:00:00 +0000

You can write queries within the YAML file or in standalone sql files and reference them in the sql_template model. They have the same keys as an input model.

This data is saved in the model folder of a Profiles project, along with any referenced sql files.

models:
 - name: name_of_sql_model 
 model_type: sql_template 
 model_spec:
 <model-spec>

Key	Description
`name` Required	SQL model name.
`model_type` Required	Model type - set it to `sql_template`.
`model_spec` Required	Model configuration. See SQL Model Spec for more information.

Example

models:
 - name:
 model_type: sql_template
 model_spec:
 single_sql: |
 {% with input_table = this.DeRef("inputs/rsIDS) %}
 select
 id,
 first,
 last
 from
 {{input_table}}
 where
 id_type = 'email'
 {% endwith %}
 ids:
 select: "id"
 type: user_id
 entity: user

Archive: macros.yaml

Mon, 01 Jan 0001 00:00:00 +0000

Macros are reusable blocks of code that can be used in a Profiles project as a form of templating. They operate similar to functions in that you can reuse them with different parameters, reducing repetition and making the code more modular.

Define macros

You can define macros in the macros.yaml file in the model folder, and call them within any other profiles.yaml file.

macros:
 - name: macro_name  # Required
 inputs: # Required
 - list_of_parameters
 value: "code as string" # Required

Key	Description
`name` Required	Name of the macro - you can use it to call the macro.
`inputs`	Parameters for the macro.
`value` Required	Macro code in a string format.

Syntax

The syntax for Profiles macros is largely based on pongo2. The basic principles of writing a macro are as follows:

Macros operate on the YAML code itself, generating new code before execution. The inputs are parameters for adding strings at compile-time, expanding the reusability of the macro.
Input parameters are surrounded by double curly brackets (e.g. {{input}})
Control logic (if, else, endif) is defined within {% and %}.
Reserved inputs words are this and warehouse.

Example

# Macro with one input
 - name: array_agg
 inputs:
 - column_name
 value: "array_agg( distinct {{column_name}})"

# Macro with two inputs
 - name: macro_listagg
 inputs:
 - column
 - timestamp
 value: "LISTAGG({{column}}, ',') WITHIN group (order by {{timestamp}} ASC)"

# Macro with no inputs
 - name: frame_clause
 value: "frame_condition = 'rows between unbounded preceding and unbounded following'"

# Define If/Else logic
 - name: macro_listagg
 inputs:
 - column
 - timestamp
 value: "{% if warehouse.DatabaseType() == \"bigquery\" %} STRING_AGG({{column}}, ',' ORDER BY {{timestamp}} ASC) {% else %} LISTAGG({{column}}, ',') WITHIN group (order by {{timestamp}} ASC) {% endif %}"

# Define more complex logic — uses the built-in end_time_sql macro (v0.25+)
 - name: macro_datediff
 inputs:
 - column
 value: "{% if warehouse.DatabaseType() == \"bigquery\" %} date_diff(date({{ end_time_sql }}), date({{column}}), day) {% else %} datediff(day, date({{column}}), date({{ end_time_sql }})) {% endif %}"

# Use macro in a feature definition in the profiles.yaml file
- entity_var:
 name: all_anonymous_ids
 select: "{{ array_agg(anonymous_id) }}"
 from: inputs/rsIdentity

Archive: Incremental Features

Mon, 01 Jan 0001 00:00:00 +0000

Making a feature incremental improves performance and reduces costs by processing only new data instead of recomputing everything from scratch.

See Incremental Features Overview for more information.

Quick decision guide

Can you express your logic as…	Use this approach	Read more
A single composable function (`SUM`, `MIN`, `MAX`, `COUNT`)	Single Incremental Entity Var	Implementation Guide
Combine multiple composable aggregations (for example, `AVG()`=`SUM()`/`COUNT()`)	Express goal entity var as a composition of multiple Incremental Entity Vars	Implementation Guide
Window functions, PIVOTs, or advanced logic	Define an incremental SQL model and define entity vars (without merge) on top	Implementation Guide

Follow this decision tree if you’re unsure which approach to use:

Understand composable functions

Composable functions can be computed incrementally by combining results from different time periods:

SUM, MIN, MAX, COUNT are composable - they can merge results across checkpoints
AVG is NOT composable (you cannot average two averages correctly without knowing the counts)

Compound composable functions can be expressed as combinations of simple composable functions:

AVG(x) = SUM(x) / COUNT(x) - requires two composable components
WEIGHTED_AVG(x, w) = SUM(x * w) / SUM(w) - requires two composable components

This distinction matters because composable functions can use simple merge logic, while non-composable functions like AVG require compound aggregations.

Simple aggregations

Best for: Direct aggregations using a single composable merge function

When to use

Computing totals, counts, min/max values
Using SUM, MIN, MAX, or COUNT
Straightforward merge operations (for example, sum of previous + new, min/max of previous and new)

Why start here

Minimal configuration (just add a merge property)
Predictable behavior
Best performance
Works across all warehouses

Use cases

Use Case	Function
Total lifetime value	`SUM`
First purchase date	`MIN`
Last login time	`MAX`
Total event count	`COUNT`

Quick example

- entity_var:
 name: total_purchases
 select: SUM({{orders.value}})
 merge: SUM({{rowset.total_purchases}})  # ← Key: merge mirrors select
 from: inputs/orders

Detailed reference

See Simple Aggregations for implementation and detailed examples.

Compound aggregations

Best for: Metrics combining multiple simple aggregations

When to use

Need to calculate averages (sum ÷ count)
Combining multiple aggregations
Array operations (union, distinct, sort)

How it works

This approach breaks down complex metrics into simple components, then combines them:

# Components
- entity_var:
 name: order_value_sum
 select: SUM({{orders.value}})
 merge: SUM({{rowset.order_value_sum}})

- entity_var:
 name: order_count
 select: COUNT(*)
 merge: SUM({{rowset.order_count}})

# Combination
- entity_var:
 name: avg_order_value
 select: "{{user.order_value_sum}} / NULLIF({{user.order_count}}, 0)"

Use cases

Use case	Components
Average order value	sum + count
Conversion rate	purchase_count + session_count
List of visited countries	array aggregation + distinct + sort

Detailed reference

See Compound Aggregations for detailed examples and implementation.

Incremental SQL models

Best for: Stateful processing and advanced transformations

When to use

Use this approach only when entity vars cannot express your logic. For example:

Tracking activity windows (MAU/DAU with rolling dates)
Maintaining running balances or state
Multi-step transformations with conditional logic
Complex business rules requiring temporary tables

How it works

Reference previous model state and merge with new data:

-- Previous state
{{#with this.DeRef()}}
 SELECT * FROM {{this}}
{{/with}}

UNION ALL

-- New incremental data
SELECT * FROM new_data

Use cases

Use case	Description
Monthly Active Users (MAU)	Rolling 30-day window with date-based filtering
Wallet balances	Running balance with credits/debits
Multi-step ETL	Conditional logic with intermediate calculations

Detailed reference

See Incremental SQL Models for detailed examples and implementation.

Quick reference

Pattern	Complexity	Use when
Simple Aggregations	⭐ Easiest	Single aggregation function
Compound Aggregations	⭐⭐ Moderate	Multiple aggregations combined
Incremental SQL Models	⭐⭐⭐ Advanced	Stateful or complex logic

Tip: Start with simple aggregations first. If your use case doesn’t fit, move to compound aggregations. Use incremental SQL models only when simple and compound aggregations cannot express your requirements.

See more

How to Make Features Incremental: Feature-level migration guide
Entity Vars
Features
Checkpoints and Baselines

Archive: Profiles Optimizations

Mon, 01 Jan 0001 00:00:00 +0000

This feature is in Beta, where we work with early users and customers to test new features and get feedback before making them generally available.

Contact the Product team if you have any questions.

This guide introduces the available optimizations in Profiles and explains how to use them in your Profiles project to enhance execution efficiency.

Optimization options

Profiles provides the following optimizations to reduce the execution time of a pb run:

Input variable table optimization

Instead of copying the entire input table, Profiles automatically identifies and copies only the necessary columns, reducing data transfer and improving performance. PB identifies the necessary columns by parsing the SQL fragments provided in input_vars and entity_vars.

Set enable_input_columns_filtering to true in your optimizations.yaml file to enable this optimization.

Limitation

If Profiles fails to identify the necessary columns and the run fails, you need to specify the unidentified columns in the input model’s contract - this workaround should help you bypass the run failure. An example is shown below:

- name: rsIdentifies
 contract:
 is_optional: false
 is_event_stream: true
 with_entity_ids:
 - users
 with_columns:
 - name: missed_column # PB was not able to identify this column, it is added to the contract.
 app_defaults:
 table: identifies 
 occurred_at_col: timestamp 
 ids:
 - select: user_id 
 type: user_id 
 entity: users

Efficient main ID column addition

Instead of performing multiple joins between the input variable table and the ID stitcher, a case-based approach is used to add the main ID column. This method is particularly effective for large input tables.

Set main_id_addition_approach to case_based_join in your optimizations.yaml file to enable this optimization.

Input vars bundling

Profiles automatically groups the execution of related input_vars based on their definitions to optimize runtime performance.

Set bundle_input_vars to true in your optimizations.yaml file to enable this optimization.

Entity vars bundling

Similar to input variable bundling, this optimization groups the execution of entity_vars based on their definitions to improve efficiency.

Set bundle_entity_vars to true in your optimizations.yaml file to enable this optimization.

There are two bundling strategies - normal and aggressive, with normal as the default strategy.

In the normal strategy, PB groups similar bundles with inter-dependencies and with or without a window together.
In the aggressive strategy, PB groups regardless of similarity.

Optimization options summary

The following table summarizes the available optimizations:

Optimization	Allowed values	Default value
`enable_input_columns_filtering`	`true`, `false`	`false`
`main_id_addition_approach`	`plain_join`, `case_based_join`	`plain_join`
`bundle_input_vars`	`true`, `false`	`false`
`bundle_entity_vars`	`true`, `false`	`false`
`bundle_entity_vars_strategy`	`normal`, `aggressive`	`normal`

Usage

Specify the optimizations in the optimizations.yaml file.

Example configuration

optimizations:
 main_id_addition_approach: case_based_join
 enable_input_columns_filtering: true
 bundle_entity_vars: true
 bundle_input_vars: true

Make sure to place the optimizations.yaml file in the appropriate configuration directory of your Profiles project (within the models folder). Also, make sure that there is only one optimizations.yaml file in your project.

Archive: Advanced ID Stitcher Features

Mon, 01 Jan 0001 00:00:00 +0000

The Profiles ID Stitcher provides advanced features that give you fine-grained control over how identities are connected in your graph, helping you maintain data quality, prevent identity explosion, and enforce business rules about identity relationships.

Available features

This section describes the advanced features available in the Profiles ID Stitcher.

Inclusion/exclusion rules

You can define specific rules to control which nodes and edges are included in your ID graph using a warehouse table. These rules provide the highest precedence over any other filtering mechanisms.

Key capabilities

Exclude or include specific identifiers (node control)
Define specific connections between identifiers (edge control)
Manage rules through CLI tools
Handle conflicting rules with configurable priority settings

See ID Graph Inclusion/Exclusion Rules for more details.

Cardinality rules

Set maximum connection limits between different ID types to prevent over-connected identity graphs and maintain data quality.

Key capabilities

Define directional limits (for example, each email connects to at most 1 user_id)
Prevent identity explosion from shared identifiers
Maintain audit trail of violations
Support incremental runs with consistent behavior
Optimize performance through controlled graph size

See ID Graph Cardinality Rules for more details.

Advanced operations

Control how the ID Stitcher processes data and ensures stable user identifiers across runs.

Key capabilities

Rebuild the entire graph from scratch using rebase_incremental mode
Apply rule changes retroactively to historical data
Maintain deterministic user IDs across runs
Understand how anchor nodes ensure ID stability

See Advanced ID Stitcher Operations for more details.

How these features work together

Both features integrate seamlessly with your Profiles pipeline:

Input filters reduce data volume before graph processing.
Cardinality rules enforce connection limits during graph building.
Inclusion/exclusion rules provide final control over specific nodes and edges.

This layered approach ensures you have comprehensive control over your identity graph while maintaining optimal performance and data quality.

Archive: Run Profiles Project

Mon, 01 Jan 0001 00:00:00 +0000

This guide lists the commands required to run your Profiles project successfully.

Compile

Once you have created the Profiles project, make sure to compile it to verify that there are no compilation errors.

Run the following command:

pb compile

This generates SQL files in the output/ folder that you can run directly on the warehouse. In case of any compilation errors, you will see them on your screen and also in the logs/logfile.log file.

Run

Once the compilation is successful and there are no errors, run your Profiles project:

pb run

This command generates and runs the SQL files in the warehouse, creating the material tables.

Archive: Warehouse Output

Mon, 01 Jan 0001 00:00:00 +0000

Upon a successful run, the Profiles project creates a series of tables and views in your warehouse.

Views

The final output from Profiles are views that, after each run, are updated to the most recent materialized tables. This allows you to always work with the most recent data without having to change code.

ID Graph

The name of your ID graph’s output is the name defined in your model spec. The name of the resolved rudder_id is {ENTITY}_MAIN_ID by default.

USER_MAIN_ID,
OTHER_ID,
OTHER_ID_TYPE,
VALID_AT,
FIRST_SEEN_AT

Cohorts

{ENTITY}_ALL

{ENTITY}_{COHORT_NAME}

Using the names defined in profiles.yaml.

USER_MAIN_ID

Feature views

{ENTITY}_{FEATURE_VIEW_NAME}

Using the names defined in pb_project.yaml for entities and profiles.yaml for cohorts.

USER_MAIN_ID,
<FEATURES>

Features named defined in profiles.yaml.

Tables

Profiles creates a series of materialized tables that feed into the final views. RudderStack does not recommend using these tables directly as they are tied to individual runs.

These table include:

Entity var tables
Input var tables
Edges tables
Mapping tables

Note that the tables use the format: Material_<name>_<HASH>_<seq_no>.

Archive: Activation API v2

Mon, 01 Jan 0001 00:00:00 +0000

This feature is in Beta, where we work with early users and customers to test new features and get feedback before making them generally available.

Contact the Product team if you have any questions.

With RudderStack’s Activation API, you can fetch enriched user traits stored in your Redis instance and use them for near real-time personalization for your target audience.

You can sync all your customer 360 data from Profiles project to your Redis store. Then, use the Activation API endpoints to retrieve and use the enriched user data for personalization.

Prerequisites

To use the Activation API:

You must have a working Redis instance in place to use the Activation API
You must have at least one successful Profiles run
Your pb_project.yaml > entities must have a feature_views property
Generate a workspace-level Service Access Token in your RudderStack dashboard with the following permission — this token is required to authenticate and use the API

Resource	Permissions
PII Permission	Destination Data Access for the specific Redis destination

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have Admin permissions.

See this documentation for more information on generating the token.

Use Activation API

Your Profiles project already uses a warehouse connection whose user/role must have the required permissions for a successful Profiles run. Follow the steps below to create the RudderStack state schema (used to store the sync state) and grant that same user/role full privileges on it.

You need privileges sufficient to create a schema in your database (for example, the ACCOUNTADMIN role).

Run the following queries:

CREATE SCHEMA "<YOUR_DATABASE>"."_RUDDERSTACK";
GRANT ALL PRIVILEGES ON SCHEMA "<YOUR_DATABASE>"."_RUDDERSTACK" TO ROLE "<YOUR_PROFILES_ROLE>";

Replace <YOUR_DATABASE> with your Profiles database and <YOUR_PROFILES_ROLE> with the Snowflake role configured for your Profiles warehouse connection.

The _RUDDERSTACK schema is used by RudderStack for storing the state of each data sync. Do not change this name.

Run the following queries:

CREATE SCHEMA "_rudderstack";
GRANT ALL ON SCHEMA "_rudderstack" TO <YOUR_PROFILES_USER>;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA "_rudderstack" TO <YOUR_PROFILES_USER>;

Replace <YOUR_PROFILES_USER> with the Redshift user configured for your Profiles warehouse connection.

The _rudderstack schema is used by RudderStack for storing the state of each data sync. Do not change this name.

Run the following queries:

CREATE SCHEMA `_rudderstack`;
GRANT ALL PRIVILEGES ON SCHEMA `_rudderstack` TO `<YOUR_PROFILES_USER>`;

Replace <YOUR_PROFILES_USER> with the Databricks user configured for your Profiles warehouse connection.

The _rudderstack schema is used by RudderStack for storing the state of each data sync. Do not change this name.

Use the same service account as your Profiles BigQuery warehouse connection — this account should already have the project and dataset roles described in Grant warehouse permissions.

Create the RudderStack schema from the BigQuery SQL workspace:

create schema rudderstack_;

The rudderstack_ schema is used by RudderStack for storing the state of each data sync. Do not change this name.

Grant full access on that schema to your service account.

GRANT `roles/bigquery.dataOwner`
 ON SCHEMA rudderstack_
 TO "serviceAccount:<SERVICE_ACCOUNT_ID>";

Replace <SERVICE_ACCOUNT_ID> with the service account ID from your Profiles BigQuery warehouse connection (of the format your-service-account@your-project.iam.gserviceaccount.com)

In your Profiles project settings, scroll down to Activation API and turn on the Enable sync to Redis toggle.

Permissions required to enable sync to Redis

New Access Management system:

Admins can turn on the Enable sync to Redis toggle by default.

Members need to have the Profiles Edit permission for the specific Profiles project.

Legacy RBAC system: In the legacy Permissions Management (RBAC) system, only Org Admins can turn on this toggle.

Enter the account credentials for your Redis instance and click Create. This will also create a Redis destination in your dashboard.

Note the destination ID from the Settings tab of your Redis destination.
Use the Activation API endpoint to fetch user profiles from your Redis instance.

Authorization

This API uses Bearer Authentication for authenticating all requests. Set the Service Access Token as the bearer token for authentication.

Base URL

https://profiles.rudderstack.com/v2/

The URL is https://profiles-eu.rudderstack.com/v2/ if you are hosted in the EU region.

Get user profiles

POST

/activation

Request body

entity

Required

String

Entity type

destinationId

Required

String

Redis destination ID.

Required

Object

ID containing type and value

{
 "entity": <entity_type>, // User, project, account, etc.
 "destinationId": <redis_destination_id> , // Redis destination ID
 "id": {
 "type": <id_type>,
 "value": <id_value>
 }
}

Sample request

POST /v2/activation HTTP/1.1
Host: profiles.rudderstack.com
Content-Type: application/json
Authorization: Bearer <service_access_token>
Content-Length: 90

{
 "entity": <entity_type>,
 "destinationId": <redis_destination_id>, // Redis destination ID
 "id": {
 "type": <id_type>,
 "value": <id_value>
 }
}

curl --location 'https://profiles.rudderstack.com/v2/activation' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <service_access_token>' \
--data '{
 "entity": <entity_type>,
 "destinationId": <redis_destination_id>, // Redis destination ID
 "id": {
 "type": <id_type>,
 "value": <id_value>
 }
}'

const axios = require('axios');
let data = JSON.stringify({
 "destinationId": <redis_destination_id>,
 "entity": <entity_type>,
 "id": {
 "type": <id_type>,
 "value": <id_value>
 }
});

let config = {
 method: 'post',
 maxBodyLength: Infinity,
 url: 'https://profiles.rudderstack.com/v2/activation',
 headers: {
 'Content-Type': 'application/json',
 'authorization': 'Bearer <service_access_token>'
 },
 data: data
};

axios.request(config)
 .then((response) => {
 console.log(JSON.stringify(response.data));
 })
 .catch((error) => {
 console.log(error);
 });

Responses

You will get the following response if the Service Access Token is absent or trying to access a destination to which it does not have access:

statusCode: 401
Response: {
 "error": "Unauthorized request. Please check your access token"
}

You will get the following response if the destination is not Redis or the destination ID is absent/blank:

statusCode: 404
Response: {
 "error": "Invalid Destination. Please verify you are passing the right destination ID"
}

You will get the following response if the destination ID is present and valid:

statusCode: 200
Response:
{
 "entity": <entity_type>,
 "id": {
 "type": <id_type>,
 "value": <id_value>
 },
 "data": {
 <traits_from_Redis>
 }
}

You will get the following response if the destination ID is not present in Redis:

statusCode: 200
Response:
{
 "entity": <entity_type>,
 "id": {
 "type": <id_type>,
 "value": <id_value>
 },
 "data": {}
}

Delete user profiles

DELETE

/activation

Use this request to delete a single user profile. To delete multiple profiles in one request, see Batch delete user profiles.

Request body

entity

Required

String

Entity type

destinationId

Required

String

Redis destination ID.

Required

Object

ID containing type and value

Sample request

DELETE /v2/activation HTTP/1.1
Host: profiles.rudderstack.com
Content-Type: application/json
Authorization: Bearer <service_access_token>
Content-Length: 90

{
 "entity": <entity_type>,
 "destinationId": <redis_destination_id>, // Redis destination ID
 "id": {
 "type": <id_type>,
 "value": <id_value>
 }
}

curl --location --request DELETE 'https://profiles.rudderstack.com/v2/activation' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <service_access_token>' \
--data '{
 "entity": <entity_type>,
 "destinationId": <redis_destination_id>, // Redis destination ID
 "id": {
 "type": <id_type>,
 "value": <id_value>
 }
}'

const axios = require('axios');
let data = JSON.stringify({
 "destinationId": <redis_destination_id>,
 "entity": <entity_type>,
 "id": {
 "type": <id_type>,
 "value": <id_value>
 }
});

let config = {
 method: 'delete',
 maxBodyLength: Infinity,
 url: 'https://profiles.rudderstack.com/v2/activation',
 headers: {
 'Content-Type': 'application/json',
 'authorization': 'Bearer <service_access_token>'
 },
 data: data
};

axios.request(config)
 .then((response) => {
 console.log(JSON.stringify(response.data));
 })
 .catch((error) => {
 console.log(error);
 });

Responses

statusCode: 200
Response: {
 "deletedKeys": 20,
 "actualKeys": 30
}

statusCode: 400
Response: {
 "message": "id should have at least one item"
}

statusCode: 404
Response: {
 "message": "None of the provided userIds were found"
}

statusCode: 500
Response: {
 "message": "Internal server error"
}

Batch delete user profiles

DELETE

/activation

Use this request to delete profiles for multiple identifiers in a single API call. Pass an ids array instead of a single id object.

Request body

entity

Required

String

Entity type

destinationId

Required

String

Redis destination ID.

ids

Required

Array

Array of IDs, each containing type and value

{
 "entity": <entity_type>,
 "destinationId": <redis_destination_id>,
 "ids": [
 {
 "type": <id_type>,
 "value": <id_value>
 },
 {
 "type": <id_type>,
 "value": <id_value>
 }
 ]
}

Sample request

DELETE /v2/activation HTTP/1.1
Host: profiles.rudderstack.com
Content-Type: application/json
Authorization: Bearer <service_access_token>
Content-Length: 90

{
 "entity": <entity_type>,
 "destinationId": <redis_destination_id>,
 "ids": [
 {
 "type": <id_type>,
 "value": <id_value>
 },
 {
 "type": <id_type>,
 "value": <id_value>
 }
 ]
}

curl --location --request DELETE 'https://profiles.rudderstack.com/v2/activation' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <service_access_token>' \
--data '{
 "entity": <entity_type>,
 "destinationId": <redis_destination_id>,
 "ids": [
 {
 "type": <id_type>,
 "value": <id_value>
 },
 {
 "type": <id_type>,
 "value": <id_value>
 }
 ]
}'

const axios = require('axios');
let data = JSON.stringify({
 "destinationId": <redis_destination_id>,
 "entity": <entity_type>,
 "ids": [
 {
 "type": <id_type>,
 "value": <id_value>
 },
 {
 "type": <id_type>,
 "value": <id_value>
 }
 ]
});

let config = {
 method: 'delete',
 maxBodyLength: Infinity,
 url: 'https://profiles.rudderstack.com/v2/activation',
 headers: {
 'Content-Type': 'application/json',
 'authorization': 'Bearer <service_access_token>'
 },
 data: data
};

axios.request(config)
 .then((response) => {
 console.log(JSON.stringify(response.data));
 })
 .catch((error) => {
 console.log(error);
 });

Responses

statusCode: 200
Response: {
 "deletedKeys": 1,
 "actualKeys": 2
}

statusCode: 400
Response: {
 "message": "ids should have at least one item"
}

statusCode: 404
Response: {
 "message": "None of the provided userIds were found"
}

statusCode: 500
Response: {
 "message": "Internal server error"
}

Use case

You can use the Activation API for real-time personalization. Once you fetch the user traits from your Redis instance via the API, you can pull them into your client application to alter the application behavior in real-time based on user interactions.

You can respond immediately with triggered, user-focused messaging based on actions like page views or app clicks and provide a better customer experience.

Redis configuration

You must have a working Redis instance in place before setting up the connection.

Address: Enter the public endpoint of your Redis database. If you are using Redis Cloud, you can find this endpoint by going to your Redis database and navigating to Configuration tab > General.

Password: Enter the database password. You can find it in the Security section of the Configuration tab:

Cluster Mode: Turn on this setting if you’re connecting to a Redis cluster.
Secure: Enable this setting to secure the TLS communication between RudderStack Redis client and your Redis server.

Data mapping

RudderStack creates multiple Reverse ETL sources automatically based on your Profiles project. You will see separate sources connected to the same Redis destination.

The following pb_project.yaml snippet shows the sources to be created:

entities:
 - name: user
 id_types:
 - main_id
 - user_id
 - email
 - salesforce_id
 feature_views:
 using_ids:
 - id: email
 name: features_by_email # Optional. Takes default view name, if not specified.
 - id: salesforce_id
 name: salesforce_id_stitched_features

FAQ

How do I generate a workspace-level Service Access Token to use the Activation API?

Follow these steps to generate a workspace-level Service Access Token to use the Activation API:

Workspace-level Service Access Tokens are linked to a specific workspace — their usage is restricted to workspace-level resources (sources, destinations, transformations, etc.) and APIs.

Log in to your RudderStack dashboard.
Go to Settings > Access Management > Service Access Tokens.
Click the Workspace tab.

Click Generate new token.
Enter the name of the SAT.
Choose the relevant workspace (applicable for a multi-workspace organization) where the token will be applicable. Then, click Next.

Under Workspace SAT access policy, configure the access policy for the token with the following permission:

Resource	Permissions
PII Permission	Destination Data Access for the specific Redis destination

Important: Once generated, you cannot edit the access policy of the workspace-level Service Access Token.

Click Generate to generate the token.
Note the token and use it to authenticate the Activation API.

Secure the token value — you will not be able to see it again once you click Close.

Why am I getting an error trying to enable API in my instance for a custom project hosted on GitHub?

For GitHub projects, you need to explicitly add the IDs of the custom project that need to be served.

In your pb_project.yaml file, you can specify them as shown:

entities:
 - name: user
 id_types:
 - main_id
 - user_id
 - email
 - salesforce_id
 feature_views:
 name: user_feature_view
 using_ids:
 - id: email
 name: features_by_email
 - id: salesforce_id
 name: salesforce_id_stitched_features

Does RudderStack perform a full sync if I add a new feature to my project?

Yes, RudderStack updates the mappings and automatically sends all columns from the customer 360 view by triggering a full sync.

Suppose I’m running a full sync and the Profiles job is running in parallel and finishes eventually. What happens to the scheduled sync? Does it get queued?

RudderStack first creates a temporary snapshot copy of any sync when it starts. So its syncing the created copy. Even if a Profiles job is running in parallel, the sync - if started - is not impacted by it.

Archive: Profiles ID Stitcher Audit Tool

Mon, 01 Jan 0001 00:00:00 +0000

Profiles ID Stitcher Audit tool is embedded within the Profiles Builder (PB) core CLI tool and helps you analyze the health of your ID graph.

Note that:

Profiles ID Stitcher Audit tool is only available in Profile Builder (PB) v0.19.0 and above.

This tool is only available for the PB CLI.

If you are developing your Profiles projects within the web-based editor (IDE), then see IDE project migration for the steps to develop locally.

This feature provides the following insights:

A high-level aggregate analysis of your overall ID graph across all IDs for a given entity.
The option to select a single ID (typically, a customer) and provide a drilled-down analysis and visualization of their individual graph.
An optional interactive LLM session where you can ask questions about your ID graph and get answers to them.

Video walkthrough

Prerequisites

Python environment (v3.9.0 to v3.11.10).
Profiles v0.19.0 or later installed locally within the above Python environment.

pip3 install profiles-rudderstack

profiles-mlcorelib library (v0.6.0 or above) installed within the above Python environment (and same as the environment for the profiles-rudderstack library).

pip install profiles-mlcorelib>=0.6.0

The audit tool helps understand an ID graph that is already materialized. So, it expects a successful Profiles project run on the same pb version at least once that has generated an ID graph, and the ID graph definition does not change after the run.
Generate a workspace-level Service Access Token in your RudderStack workspace. Then, add it to your site configuration file.

The workspace-level Service Access Token has Read permissions by default. You do not need to assign any resource permissions while generating the token.

Token permissions for legacy RBAC system

If you are on the legacy Permissions Management (RBAC) system, your workspace-level Service Access Token should have minimum Viewer permission.

See this documentation for more information on generating the token.

IDE project migration

If you are developing a Profiles project in the RudderStack dashboard using the web-based editor (IDE), then follow these steps:

Navigate to your Profiles project in the RudderStack dashboard.
Make sure that your latest IDE session is saved to ensure that you have the latest changes. If your project is connected to a remote repo, you can commit those changes as well.
Download the project: - If in a remote repo, navigate there and download locally. - If the project is not connected to a remote repo, then navigate to the project’s settings and click Download this project, as shown:

Move the project configuration to the desired local directory.
Navigate to the root of that directory within your terminal.
Before performing the runs locally, verify that you have set up a warehouse connection. See Step 2: Create warehouse connection to create a local file containing your warehouse credentials.

You can successfully run PB and utilize the Profiles ID Stitcher Audit tool after completing the above steps.

Workflow overview

To use this feature, go to your Profiles project configuration and run the command below within your terminal:

pb audit id_stitcher

The audit tool then:

Scans the project to look for ID stitcher models.
Asks the customers to select one model in case there are multiple ID stitcher models.
For the selected model, presents the below three audit stages:

Note that the tool works only on top of a batch ID graph run. By default, an ID stitcher model runs incrementally and a batch run typically happens in the below cases:

First run of a Profiles project.
When you change the ID stitcher model’s definition, for example, adding/removing input sources or id_types, filters, etc.
When you upgrade your Profiles project to a new version.
If the underlying Profiles tables in the output schema are not found.

Usually, an audit is done after modifying the ID stitcher model - this should not cause an issue because the run happens in batch mode. However, in case an audit is being performed after an incremental run, you can force a one-time batch run by adding a flag rebase_incremental in the pb run command, as shown:

pb run --rebase_incremental

The above command forces a full batch run for one time, allowing for a full audit of the ID graph.

Static ID graph summary

The pb audit id_stitcher command gives you a detailed report on the ID stitcher as an output. It contains the ID graph summary that helps you understand:

ID graph’s definition, like ID types, main ID, etc.
Count of values, like:
- Counts of each ID types before and after ID stitching.
- Most widely connected IDs.
- Resolved main ID stats on the number of IDs connected together, like average, 99th percentile, etc.
- IDs that are never connected to any other ID.

Note that for all the counts, the tool shows some samples but they are at an aggregate level, not on an individual main_id level.

Click here to view a sample report. Note that the exact text and format may differ.

main id for entity user: USER_MAIN_ID


Analyzing ID Stitcher for entity: user
 Node types: ['id', 'anonymous_id', 'contact_id', 'email', 'lead_id', 'user_id']
 Unique IDs of each type and their counts:
 id: 10958
 anonymous_id: 4294899
 contact_id: 79854
 email: 122661
 lead_id: 75482
 user_id: 296346
Total Distinct IDs
 Before stitching: 4880200
 After stitching: 4290186


Top 10 nodes by edge count:
 ID: alex@example.com, Type: email, Edges: 2503
 ID: alex@example.com, Type: user_id, Edges: 2140
 ID: alice@example.com, Type: email, Edges: 1725
 ID: abc@example.com, Type: email, Edges: 946

Top 5 nodes of different types, before id stitching, by edge counts:

 type id by edge count:
 ID: a2ab0609d6c10187ff11, Edges: 2
 ID: a2ab060a860ee7b512, Edges: 2
 ID: a2ab0605e232d3c609, Edges: 2
 ID: a2ab060bb95dcaef14, Edges: 2
 ID: a2ab0600d159d259, Edges: 2

 type anonymous_id by edge count:
 ID: 49a8bebe-4d9d-47e5-8ea3-29153fc33dcb, Edges: 71
 ID: d81d2d84-71a8-46db-9d30-81502051c214, Edges: 61
 ID: 3421a313-189e-45d2-9270-e0ae3181e07d, Edges: 47

 type contact_id by edge count:
 ID: 003QP000001SRUjYAO, Edges: 7
 ID: 003QP000001LpBGYA0, Edges: 6

 type email by edge count:
 ID: alex@example.com, Edges: 2503
 ID: alice@example.com, Edges: 1725

 type lead_id by edge count:
 ID: 00Q5x00001zW1Y9EAK, Edges: 3
 ID: 00QQP00000800rR2AQ, Edges: 3

 type user_id by edge count:
 ID: alex@example.com, Edges: 2140
 ID: 28HhwSx1WEU56QM5sY7r74MEf9n, Edges: 1526

Average edge count per node (before stitching): 1.331919

Average edge count by node type (before stitching):
 id: 2.518149
 anonymous_id: 3.122989
 contact_id: 3.068089
 email: 3.406679
 lead_id: 2.453102
 user_id: 4.522037

 POST ID STITCHING ANALYSIS


Cluster size after stitching:
 No:of main ids with a single other id (Singleton nodes): 3998772
 Average other id counts: 1.137526
 Min other id counts for a single main id: 1
 Max other id counts for a single main id: 13711
 25th percentile: 1
 50th percentile: 1
 75th percentile: 1
 90th percentile: 1
 99th percentile: 4

Average number of ids of different id types, per main id, after stitching:
 id: 1.059563
 anonymous_id: 1.026212
 contact_id: 1.031319
 email: 1.049362
 lead_id: 1.104071
 user_id: 1.149782

Top 5 biggest clusters after id stitching (and the distinct id types in each cluster):
 Main ID: rid0e06a33cfdbccc7679e635e78f0f391f, Size: 13711
 anonymous_id: 11503
 user_id: 1205
 email: 457
 lead_id: 375
 contact_id: 117
 id: 54
 Main ID: rid039ea1d83a8484fa66f153143f013ac7, Size: 1526
 anonymous_id: 1525
 user_id: 1
 Main ID: rid25ddb86517babaa3f45dceef3c400330, Size: 964
 anonymous_id: 949
 user_id: 9
 email: 2
 contact_id: 2
 lead_id: 2
 Main ID: ridf0fef860592708f92f211df285bef625, Size: 833
 anonymous_id: 618
 user_id: 88
 email: 47
 lead_id: 47
 contact_id: 32
 id: 1
 Main ID: rid2e9139338ef8ccfec085e259da0ef3fb, Size: 719
 anonymous_id: 718
 user_id: 1

Singleton Node Analysis (after stitching):
anonymous_id: 3963171 nodes (92.28%) not connected to any other ID type
user_id: 28686 nodes (9.68%) not connected to any other ID type
contact_id: 5376 nodes (6.73%) not connected to any other ID type
lead_id: 1374 nodes (1.82%) not connected to any other ID type
id: 165 nodes (1.51%) not connected to any other ID type

Checking for potential issues:
WARNING:__main__:Potential overstitching detected: Node alex@example.com of type email has 2503 edges

Checking for missing direct edges between node types (but indirect edges can exist):
WARNING:__main__:Warning: No direct edges found between id and anonymous_id
WARNING:__main__:Warning: No direct edges found between id and contact_id
WARNING:__main__:Warning: No direct edges found between id and lead_id
WARNING:__main__:Warning: No direct edges found between id and user_id
WARNING:__main__:Warning: No direct edges found between anonymous_id and id
WARNING:__main__:Warning: No direct edges found between anonymous_id and contact_id
WARNING:__main__:Warning: No direct edges found between anonymous_id and lead_id
WARNING:__main__:Warning: No direct edges found between contact_id and id
WARNING:__main__:Warning: No direct edges found between contact_id and anonymous_id
WARNING:__main__:Warning: No direct edges found between contact_id and user_id
WARNING:__main__:Warning: No direct edges found between lead_id and id
WARNING:__main__:Warning: No direct edges found between lead_id and anonymous_id
WARNING:__main__:Warning: No direct edges found between lead_id and user_id
WARNING:__main__:Warning: No direct edges found between user_id and id
WARNING:__main__:Warning: No direct edges found between user_id and contact_id
WARNING:__main__:Warning: No direct edges found between user_id and lead_id

 Check for missing edges between node types (direct or indirect) after id stitching:
No missing edges found between node types. GREAT!!


ANALYSIS COMPLETE FOR ENTITY: user

Cluster analysis

The pb audit id_stitcher command also lets you understand specific clusters (main_ids), for example, how an email ID E1 is linked to a Salesforce ID S1.

The audit tools gives a snapshot of the complete cluster and also points to the IDs that may be problematic.

Click here to view a sample

You can explore specific clusters by entering an ID to see how the other ids are all connected and the cluster is formed.
The ID can be either the main ID or any other ID type.
Enter an ID to visualize (or 'skip' to skip this step):
Acme

Your network visualization is ready! We've saved an interactive map of your data connections here:
/Users/user_name/git_repos/profiles-prod/graph_outputs/Acme_graph.html
You can open this file in your web browser to explore the network visually.
Cluster Summary:

 Network overview:
 -----------------
 1. Size: 7 ids connected by 9 links
 2. Connectivity: On average, each id is connected to 2.6 other ids
 3. Spread: The farthest connected ids are 2 steps apart

 Key Players:
 ------------ 

 Most connected ids:
 - ID: acme.com, No:of edges: 6
 - ID: Acme, No:of edges: 2
 - ID: 000123XYZ, No:of edges: 2
 - ID: Acme America, No:of edges: 2
 - ID: 000456ABC, No:of edges: 2
 - ID: 000789DEF, No:of edges: 2
 - ID: Acme Inc, No:of edges: 2

 Critical Linking Nodes, and percent of id pairs they are connecting (these are the nodes that act as bridges across all nodes):
 - ID: acme.com, % of shortest paths: 80.00%

The audit tool also outputs an HTML file that helps you understand the cluster visually. It is stored locally and is loaded proactively by highlighting the path so you can revisit it later. A sample file is shown:

In the above example, you can see how acme.com joins Acme Inc and Acme America entities which are probably different. You can address this by removing the acme.com domain.

Click here to view a sample project that helps you identify a common ID that has multiple users stitched together.

<html>
 <head>
 <meta charset="utf-8">
 <script src="lib/bindings/utils.js"></script>
 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/vis-network/9.1.2/dist/dist/vis-network.min.css" integrity="sha512-WgxfT5LWjfszlPHXRmBWHkV2eceiWTOBvrKCNbdgDYTHrT2AeLCGbF4sZlZw3UMN3WtL0tGUoIAKsu8mllg/XA==" crossorigin="anonymous" referrerpolicy="no-referrer" />
 <script src="https://cdnjs.cloudflare.com/ajax/libs/vis-network/9.1.2/dist/vis-network.min.js" integrity="sha512-LnvoEWDFrqGHlHmDD2101OrLcbsfkrzoSpvtSQtxK3RMnRV0eOkhhBN2dXHKRrUU8p2DGRTk35n4O8nWSVe1mQ==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<center>
<h1></h1>
</center>

<!-- <link rel="stylesheet" href="../node_modules/vis/dist/vis.min.css" type="text/css" />
<script type="text/javascript" src="../node_modules/vis/dist/vis.js"> </script>-->
 <link
 href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/css/bootstrap.min.css"
 rel="stylesheet"
 integrity="sha384-eOJMYsd53ii+scO/bJGFsiCZc+5NDVN2yr8+0RDqr0Ql0h+rP48ckxlpbzKgwra6"
 crossorigin="anonymous"
 />
 <script
 src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/js/bootstrap.bundle.min.js"
 integrity="sha384-JEW9xMcG8R+pH31jmWH6WWP0WintQrMb4s7ZOdauHnUtxwoG2vI5DkLtS3qm9Ekf"
 crossorigin="anonymous"
 ></script>
 <center>
 <h1></h1>
 </center>
 <style type="text/css">
 #mynetwork {
 width: 100%;
 height: 1000px;
 background-color: #ffffff;
 border: 1px solid lightgray;
 position: relative;
 float: left;
 }
 </style>
 </head>
 <body>
 <div style="position: fixed; top: 20px; right: 20px; background-color: rgba(255, 255, 255, 0.9);
 padding: 15px; border-radius: 5px; box-shadow: 0 2px 5px rgba(0,0,0,0.2); z-index: 1000;">
 <div style="font-weight: bold; margin-bottom: 10px;">ID Types</div>
 <div style="display: flex; align-items: center; margin-bottom: 5px;">
 <div style="width: 20px; height: 20px; background-color: #b23535; margin-right: 10px; border-radius: 3px;"></div>
 <div>user_id</div>
 </div>
 <div style="display: flex; align-items: center; margin-bottom: 5px;">
 <div style="width: 20px; height: 20px; background-color: #99b235; margin-right: 10px; border-radius: 3px;"></div>
 <div>anonymous_id</div>
 </div>
 <div style="display: flex; align-items: center; margin-bottom: 5px;">
 <div style="width: 20px; height: 20px; background-color: #35b267; margin-right: 10px; border-radius: 3px;"></div>
 <div>sample_customer_id</div>
 </div>
 <div style="display: flex; align-items: center; margin-bottom: 5px;">
 <div style="width: 20px; height: 20px; background-color: #3567b2; margin-right: 10px; border-radius: 3px;"></div>
 <div>device_id</div>
 </div>
 <div style="display: flex; align-items: center; margin-bottom: 5px;">
 <div style="width: 20px; height: 20px; background-color: #9935b2; margin-right: 10px; border-radius: 3px;"></div>
 <div>email</div>
 </div>
 </div>

 <div class="card" style="width: 100%">
 <div id="mynetwork" class="card-body"></div>
 </div>

 <script type="text/javascript">

 // initialize global variables.
 var edges;
 var nodes;
 var allNodes;
 var allEdges;
 var nodeColors;
 var originalNodes;
 var network;
 var container;
 var options, data;
 var filter = {
 item : '',
 property : '',
 value : []
 };

 // This method is responsible for drawing the graph, returns the drawn network
 function drawGraph() {
 var container = document.getElementById('mynetwork');

 // parsing and collecting nodes and edges from the python
 nodes = new vis.DataSet([{"color": "rgba(153, 53, 178, 1.0)", "font": {"color": "black"}, "id": "alexkeener@example.com", "label": "alexkeener@example.com", "shape": "dot", "title": "ID: alexkeener@example.com\nID-Type: email\nDegree: 36"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "887d473fa43c", "label": "887d473fa43c", "shape": "dot", "title": "ID: 887d473fa43c\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "bc854f11fca5", "label": "bc854f11fca5", "shape": "dot", "title": "ID: bc854f11fca5\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "65dacdcce759", "label": "65dacdcce759", "shape": "dot", "title": "ID: 65dacdcce759\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "29fc92b6df57", "label": "29fc92b6df57", "shape": "dot", "title": "ID: 29fc92b6df57\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(178, 53, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "akeener", "label": "akeener", "shape": "dot", "title": "ID: akeener\nID-Type: user_id\nDegree: 2"}, {"color": "rgba(153, 53, 178, 0.8833333333333333)", "font": {"color": "black"}, "id": "alicekeener@example.com", "label": "alicekeener@example.com", "shape": "dot", "title": "ID: alicekeener@example.com\nID-Type: email\nDegree: 30"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "64e041ba1b8c", "label": "64e041ba1b8c", "shape": "dot", "title": "ID: 64e041ba1b8c\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "56ed43ca-22c3-40cd-858d-3e3711a10395", "label": "56ed43ca-22c3-40cd-858d-3e3711a10395", "shape": "dot", "title": "ID: 56ed43ca-22c3-40cd-858d-3e3711a10395\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(53, 178, 103, 0.4361111111111111)", "font": {"color": "black"}, "id": "1001724", "label": "1001724", "shape": "dot", "title": "ID: 1001724\nID-Type: sample_customer_id\nDegree: 7"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "37dd4c20-6bef-4de6-bedd-c4cc204ff1e1", "label": "37dd4c20-6bef-4de6-bedd-c4cc204ff1e1", "shape": "dot", "title": "ID: 37dd4c20-6bef-4de6-bedd-c4cc204ff1e1\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(53, 178, 103, 0.611111111111111)", "font": {"color": "black"}, "id": "1001601", "label": "1001601", "shape": "dot", "title": "ID: 1001601\nID-Type: sample_customer_id\nDegree: 16"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "61e3bcdc-d995-41d9-a6aa-1172af68f104", "label": "61e3bcdc-d995-41d9-a6aa-1172af68f104", "shape": "dot", "title": "ID: 61e3bcdc-d995-41d9-a6aa-1172af68f104\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(53, 178, 103, 0.7472222222222221)", "font": {"color": "black"}, "id": "8096804", "label": "8096804", "shape": "dot", "title": "ID: 8096804\nID-Type: sample_customer_id\nDegree: 23"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "da21019fe59d", "label": "da21019fe59d", "shape": "dot", "title": "ID: da21019fe59d\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "77f689d8-1d07-4ea2-a34c-a73aa9b76516", "label": "77f689d8-1d07-4ea2-a34c-a73aa9b76516", "shape": "dot", "title": "ID: 77f689d8-1d07-4ea2-a34c-a73aa9b76516\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "2001cbbf1ffd", "label": "2001cbbf1ffd", "shape": "dot", "title": "ID: 2001cbbf1ffd\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "2f41b084-7c53-4ef4-a746-a36c591aabdc", "label": "2f41b084-7c53-4ef4-a746-a36c591aabdc", "shape": "dot", "title": "ID: 2f41b084-7c53-4ef4-a746-a36c591aabdc\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "a207523c-d826-45c5-b104-d2f0a80dda12", "label": "a207523c-d826-45c5-b104-d2f0a80dda12", "shape": "dot", "title": "ID: a207523c-d826-45c5-b104-d2f0a80dda12\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "3407e0e3-4900-497c-bff8-8d370754e3ee", "label": "3407e0e3-4900-497c-bff8-8d370754e3ee", "shape": "dot", "title": "ID: 3407e0e3-4900-497c-bff8-8d370754e3ee\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(53, 178, 103, 0.7277777777777779)", "font": {"color": "black"}, "id": "8097816", "label": "8097816", "shape": "dot", "title": "ID: 8097816\nID-Type: sample_customer_id\nDegree: 22"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "dcf6d0f9-3710-4f97-98cb-75288033c56d", "label": "dcf6d0f9-3710-4f97-98cb-75288033c56d", "shape": "dot", "title": "ID: dcf6d0f9-3710-4f97-98cb-75288033c56d\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "efc1aeba-0a0d-40be-a24a-ee393299a480", "label": "efc1aeba-0a0d-40be-a24a-ee393299a480", "shape": "dot", "title": "ID: efc1aeba-0a0d-40be-a24a-ee393299a480\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "1bf9329f6a6e", "label": "1bf9329f6a6e", "shape": "dot", "title": "ID: 1bf9329f6a6e\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "c0ba22140cec", "label": "c0ba22140cec", "shape": "dot", "title": "ID: c0ba22140cec\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(153, 178, 53, 0.33888888888888885)", "font": {"color": "black"}, "id": "39974b70-b667-4540-8eaa-a91d0e5180a1", "label": "39974b70-b667-4540-8eaa-a91d0e5180a1", "shape": "dot", "title": "ID: 39974b70-b667-4540-8eaa-a91d0e5180a1\nID-Type: anonymous_id\nDegree: 2"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "f7d3790baeea", "label": "f7d3790baeea", "shape": "dot", "title": "ID: f7d3790baeea\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "2cf176c253e7", "label": "2cf176c253e7", "shape": "dot", "title": "ID: 2cf176c253e7\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.37777777777777777)", "font": {"color": "black"}, "id": "84f9eb766ea0", "label": "84f9eb766ea0", "shape": "dot", "title": "ID: 84f9eb766ea0\nID-Type: device_id\nDegree: 4"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "54cb22e5f0a7", "label": "54cb22e5f0a7", "shape": "dot", "title": "ID: 54cb22e5f0a7\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "6848d77e2c51", "label": "6848d77e2c51", "shape": "dot", "title": "ID: 6848d77e2c51\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "e54e6b399efc", "label": "e54e6b399efc", "shape": "dot", "title": "ID: e54e6b399efc\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "77e1b9368bd0", "label": "77e1b9368bd0", "shape": "dot", "title": "ID: 77e1b9368bd0\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "e14211ff4114", "label": "e14211ff4114", "shape": "dot", "title": "ID: e14211ff4114\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "278e2d59feeb", "label": "278e2d59feeb", "shape": "dot", "title": "ID: 278e2d59feeb\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "639d42ba6306", "label": "639d42ba6306", "shape": "dot", "title": "ID: 639d42ba6306\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "34ffd366-5581-404c-9de7-8e20958bd62f", "label": "34ffd366-5581-404c-9de7-8e20958bd62f", "shape": "dot", "title": "ID: 34ffd366-5581-404c-9de7-8e20958bd62f\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "1adf0fa5-91a7-4dd3-895e-75799df6bbc6", "label": "1adf0fa5-91a7-4dd3-895e-75799df6bbc6", "shape": "dot", "title": "ID: 1adf0fa5-91a7-4dd3-895e-75799df6bbc6\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "6758a04b-f52e-432b-9f42-54fb615baa63", "label": "6758a04b-f52e-432b-9f42-54fb615baa63", "shape": "dot", "title": "ID: 6758a04b-f52e-432b-9f42-54fb615baa63\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "ec58c825-69b5-4406-bb89-189aed588793", "label": "ec58c825-69b5-4406-bb89-189aed588793", "shape": "dot", "title": "ID: ec58c825-69b5-4406-bb89-189aed588793\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "e9779e82-9943-4039-a8c7-fc1609d99295", "label": "e9779e82-9943-4039-a8c7-fc1609d99295", "shape": "dot", "title": "ID: e9779e82-9943-4039-a8c7-fc1609d99295\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "c09ceb12-0242-4eed-b1f9-07416475978f", "label": "c09ceb12-0242-4eed-b1f9-07416475978f", "shape": "dot", "title": "ID: c09ceb12-0242-4eed-b1f9-07416475978f\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "cc66231a1a9b", "label": "cc66231a1a9b", "shape": "dot", "title": "ID: cc66231a1a9b\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "d8d63dac8d42", "label": "d8d63dac8d42", "shape": "dot", "title": "ID: d8d63dac8d42\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "d61186783374", "label": "d61186783374", "shape": "dot", "title": "ID: d61186783374\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "3aa23c92e2a2", "label": "3aa23c92e2a2", "shape": "dot", "title": "ID: 3aa23c92e2a2\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "dfd7297746b0", "label": "dfd7297746b0", "shape": "dot", "title": "ID: dfd7297746b0\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "450ba6e99bc6", "label": "450ba6e99bc6", "shape": "dot", "title": "ID: 450ba6e99bc6\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.3194444444444444)", "font": {"color": "black"}, "id": "324190ef-a1dc-4153-8a7a-1f61dc3a06e9", "label": "324190ef-a1dc-4153-8a7a-1f61dc3a06e9", "shape": "dot", "title": "ID: 324190ef-a1dc-4153-8a7a-1f61dc3a06e9\nID-Type: anonymous_id\nDegree: 1"}, {"color": "rgba(153, 178, 53, 0.3194444444444444)", "font": {"color": "black"}, "id": "610575ef-102f-4ad6-ad11-ce1ba8eaa7b1", "label": "610575ef-102f-4ad6-ad11-ce1ba8eaa7b1", "shape": "dot", "title": "ID: 610575ef-102f-4ad6-ad11-ce1ba8eaa7b1\nID-Type: anonymous_id\nDegree: 1"}, {"color": "rgba(153, 178, 53, 0.3194444444444444)", "font": {"color": "black"}, "id": "24b39873-2ba1-4e34-b172-eb9b6a086d4e", "label": "24b39873-2ba1-4e34-b172-eb9b6a086d4e", "shape": "dot", "title": "ID: 24b39873-2ba1-4e34-b172-eb9b6a086d4e\nID-Type: anonymous_id\nDegree: 1"}, {"color": "rgba(153, 178, 53, 0.3194444444444444)", "font": {"color": "black"}, "id": "877ee12f-180b-4051-8a25-3bec111f3076", "label": "877ee12f-180b-4051-8a25-3bec111f3076", "shape": "dot", "title": "ID: 877ee12f-180b-4051-8a25-3bec111f3076\nID-Type: anonymous_id\nDegree: 1"}, {"color": "rgba(153, 178, 53, 0.3194444444444444)", "font": {"color": "black"}, "id": "dc4a08da-d493-48b7-beb7-2bc045e6dd2c", "label": "dc4a08da-d493-48b7-beb7-2bc045e6dd2c", "shape": "dot", "title": "ID: dc4a08da-d493-48b7-beb7-2bc045e6dd2c\nID-Type: anonymous_id\nDegree: 1"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "71446f27d2a1", "label": "71446f27d2a1", "shape": "dot", "title": "ID: 71446f27d2a1\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(53, 103, 178, 0.35833333333333334)", "font": {"color": "black"}, "id": "f7d072960756", "label": "f7d072960756", "shape": "dot", "title": "ID: f7d072960756\nID-Type: device_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "677571db-4cac-4501-9a14-3dde045673f2", "label": "677571db-4cac-4501-9a14-3dde045673f2", "shape": "dot", "title": "ID: 677571db-4cac-4501-9a14-3dde045673f2\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "b1f5cac8-6cc2-4e4d-81c0-478210da138a", "label": "b1f5cac8-6cc2-4e4d-81c0-478210da138a", "shape": "dot", "title": "ID: b1f5cac8-6cc2-4e4d-81c0-478210da138a\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "823305a7-5740-42b4-b007-29e490e17b08", "label": "823305a7-5740-42b4-b007-29e490e17b08", "shape": "dot", "title": "ID: 823305a7-5740-42b4-b007-29e490e17b08\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "6e6e960d-4166-4a5e-a865-aaec299166b7", "label": "6e6e960d-4166-4a5e-a865-aaec299166b7", "shape": "dot", "title": "ID: 6e6e960d-4166-4a5e-a865-aaec299166b7\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "922d079e-4a67-47a5-b84e-e286242d5c9e", "label": "922d079e-4a67-47a5-b84e-e286242d5c9e", "shape": "dot", "title": "ID: 922d079e-4a67-47a5-b84e-e286242d5c9e\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "8ee77e07-5c95-4540-ba17-5ca052a36720", "label": "8ee77e07-5c95-4540-ba17-5ca052a36720", "shape": "dot", "title": "ID: 8ee77e07-5c95-4540-ba17-5ca052a36720\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "03499e02-dce1-43dd-bf1a-0c73ff75a36e", "label": "03499e02-dce1-43dd-bf1a-0c73ff75a36e", "shape": "dot", "title": "ID: 03499e02-dce1-43dd-bf1a-0c73ff75a36e\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "2fb65df5-633e-47e6-abcd-4496fe47e470", "label": "2fb65df5-633e-47e6-abcd-4496fe47e470", "shape": "dot", "title": "ID: 2fb65df5-633e-47e6-abcd-4496fe47e470\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "66c13c66-5df1-4af1-a38f-ffc0f65ac0cc", "label": "66c13c66-5df1-4af1-a38f-ffc0f65ac0cc", "shape": "dot", "title": "ID: 66c13c66-5df1-4af1-a38f-ffc0f65ac0cc\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "7e3cf9a0-4d7a-4a2b-a905-8385f9523baf", "label": "7e3cf9a0-4d7a-4a2b-a905-8385f9523baf", "shape": "dot", "title": "ID: 7e3cf9a0-4d7a-4a2b-a905-8385f9523baf\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "8ccd8f72-69ca-4e64-a195-adde694b14d3", "label": "8ccd8f72-69ca-4e64-a195-adde694b14d3", "shape": "dot", "title": "ID: 8ccd8f72-69ca-4e64-a195-adde694b14d3\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "97777211-6db6-4532-806d-8e44a7c1f83f", "label": "97777211-6db6-4532-806d-8e44a7c1f83f", "shape": "dot", "title": "ID: 97777211-6db6-4532-806d-8e44a7c1f83f\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "73a88af0-e480-4d7e-9ad4-647d221f273b", "label": "73a88af0-e480-4d7e-9ad4-647d221f273b", "shape": "dot", "title": "ID: 73a88af0-e480-4d7e-9ad4-647d221f273b\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "e843ead3-744f-4d53-bbd0-84c19fca6e87", "label": "e843ead3-744f-4d53-bbd0-84c19fca6e87", "shape": "dot", "title": "ID: e843ead3-744f-4d53-bbd0-84c19fca6e87\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "d39934c4-a029-4fd9-a04e-344ea251e89e", "label": "d39934c4-a029-4fd9-a04e-344ea251e89e", "shape": "dot", "title": "ID: d39934c4-a029-4fd9-a04e-344ea251e89e\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "0415f350-ed54-4037-9fed-40712a77fc43", "label": "0415f350-ed54-4037-9fed-40712a77fc43", "shape": "dot", "title": "ID: 0415f350-ed54-4037-9fed-40712a77fc43\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "22765460-7b19-4f2a-9138-ac601e1a824c", "label": "22765460-7b19-4f2a-9138-ac601e1a824c", "shape": "dot", "title": "ID: 22765460-7b19-4f2a-9138-ac601e1a824c\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "aaf7c363-922b-4443-a7d0-4073750b6f0c", "label": "aaf7c363-922b-4443-a7d0-4073750b6f0c", "shape": "dot", "title": "ID: aaf7c363-922b-4443-a7d0-4073750b6f0c\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "25727c7d-380f-4845-993e-b34406f9330f", "label": "25727c7d-380f-4845-993e-b34406f9330f", "shape": "dot", "title": "ID: 25727c7d-380f-4845-993e-b34406f9330f\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "4e26df5b-b1ac-4964-9f23-979d8237a2b4", "label": "4e26df5b-b1ac-4964-9f23-979d8237a2b4", "shape": "dot", "title": "ID: 4e26df5b-b1ac-4964-9f23-979d8237a2b4\nID-Type: anonymous_id\nDegree: 3"}, {"color": "rgba(153, 178, 53, 0.3194444444444444)", "font": {"color": "black"}, "id": "ddbbbca3-34d7-4e90-b9e5-de9407a9e0c7", "label": "ddbbbca3-34d7-4e90-b9e5-de9407a9e0c7", "shape": "dot", "title": "ID: ddbbbca3-34d7-4e90-b9e5-de9407a9e0c7\nID-Type: anonymous_id\nDegree: 1"}, {"color": "rgba(153, 178, 53, 0.35833333333333334)", "font": {"color": "black"}, "id": "38d72063-56fc-493c-9f27-dd17e58dcedd", "label": "38d72063-56fc-493c-9f27-dd17e58dcedd", "shape": "dot", "title": "ID: 38d72063-56fc-493c-9f27-dd17e58dcedd\nID-Type: anonymous_id\nDegree: 3"}]);
 edges = new vis.DataSet([{"color": "#888888", "from": "alexkeener@example.com", "to": "887d473fa43c"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "bc854f11fca5"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "65dacdcce759"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "29fc92b6df57"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "akeener"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "64e041ba1b8c"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "f7d3790baeea"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "2cf176c253e7"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "54cb22e5f0a7"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "6848d77e2c51"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "c0ba22140cec"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "77e1b9368bd0"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "34ffd366-5581-404c-9de7-8e20958bd62f"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "1adf0fa5-91a7-4dd3-895e-75799df6bbc6"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "ec58c825-69b5-4406-bb89-189aed588793"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "e9779e82-9943-4039-a8c7-fc1609d99295"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "d8d63dac8d42"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "d61186783374"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "610575ef-102f-4ad6-ad11-ce1ba8eaa7b1"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "877ee12f-180b-4051-8a25-3bec111f3076"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "dc4a08da-d493-48b7-beb7-2bc045e6dd2c"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "71446f27d2a1"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "f7d072960756"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "8097816"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "677571db-4cac-4501-9a14-3dde045673f2"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "823305a7-5740-42b4-b007-29e490e17b08"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "6e6e960d-4166-4a5e-a865-aaec299166b7"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "922d079e-4a67-47a5-b84e-e286242d5c9e"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "66c13c66-5df1-4af1-a38f-ffc0f65ac0cc"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "7e3cf9a0-4d7a-4a2b-a905-8385f9523baf"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "97777211-6db6-4532-806d-8e44a7c1f83f"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "73a88af0-e480-4d7e-9ad4-647d221f273b"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "e843ead3-744f-4d53-bbd0-84c19fca6e87"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "0415f350-ed54-4037-9fed-40712a77fc43"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "1001601"}, {"color": "#888888", "from": "alexkeener@example.com", "to": "4e26df5b-b1ac-4964-9f23-979d8237a2b4"}, {"color": "#888888", "from": "887d473fa43c", "to": "2f41b084-7c53-4ef4-a746-a36c591aabdc"}, {"color": "#888888", "from": "887d473fa43c", "to": "823305a7-5740-42b4-b007-29e490e17b08"}, {"color": "#888888", "from": "887d473fa43c", "to": "1001601"}, {"color": "#888888", "from": "bc854f11fca5", "to": "37dd4c20-6bef-4de6-bedd-c4cc204ff1e1"}, {"color": "#888888", "from": "bc854f11fca5", "to": "1001601"}, {"color": "#888888", "from": "bc854f11fca5", "to": "34ffd366-5581-404c-9de7-8e20958bd62f"}, {"color": "#888888", "from": "65dacdcce759", "to": "1001601"}, {"color": "#888888", "from": "65dacdcce759", "to": "39974b70-b667-4540-8eaa-a91d0e5180a1"}, {"color": "#888888", "from": "65dacdcce759", "to": "677571db-4cac-4501-9a14-3dde045673f2"}, {"color": "#888888", "from": "29fc92b6df57", "to": "1001601"}, {"color": "#888888", "from": "29fc92b6df57", "to": "a207523c-d826-45c5-b104-d2f0a80dda12"}, {"color": "#888888", "from": "29fc92b6df57", "to": "922d079e-4a67-47a5-b84e-e286242d5c9e"}, {"color": "#888888", "from": "akeener", "to": "alicekeener@example.com"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "da21019fe59d"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "2001cbbf1ffd"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "84f9eb766ea0"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "e54e6b399efc"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "1bf9329f6a6e"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "e14211ff4114"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "278e2d59feeb"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "639d42ba6306"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "1001724"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "6758a04b-f52e-432b-9f42-54fb615baa63"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "cc66231a1a9b"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "3aa23c92e2a2"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "dfd7297746b0"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "450ba6e99bc6"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "324190ef-a1dc-4153-8a7a-1f61dc3a06e9"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "24b39873-2ba1-4e34-b172-eb9b6a086d4e"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "8096804"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "b1f5cac8-6cc2-4e4d-81c0-478210da138a"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "8ee77e07-5c95-4540-ba17-5ca052a36720"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "03499e02-dce1-43dd-bf1a-0c73ff75a36e"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "2fb65df5-633e-47e6-abcd-4496fe47e470"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "8ccd8f72-69ca-4e64-a195-adde694b14d3"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "d39934c4-a029-4fd9-a04e-344ea251e89e"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "c09ceb12-0242-4eed-b1f9-07416475978f"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "22765460-7b19-4f2a-9138-ac601e1a824c"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "ddbbbca3-34d7-4e90-b9e5-de9407a9e0c7"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "aaf7c363-922b-4443-a7d0-4073750b6f0c"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "38d72063-56fc-493c-9f27-dd17e58dcedd"}, {"color": "#888888", "from": "alicekeener@example.com", "to": "25727c7d-380f-4845-993e-b34406f9330f"}, {"color": "#888888", "from": "64e041ba1b8c", "to": "efc1aeba-0a0d-40be-a24a-ee393299a480"}, {"color": "#888888", "from": "64e041ba1b8c", "to": "1001601"}, {"color": "#888888", "from": "64e041ba1b8c", "to": "1adf0fa5-91a7-4dd3-895e-75799df6bbc6"}, {"color": "#888888", "from": "56ed43ca-22c3-40cd-858d-3e3711a10395", "to": "1001724"}, {"color": "#888888", "from": "56ed43ca-22c3-40cd-858d-3e3711a10395", "to": "da21019fe59d"}, {"color": "#888888", "from": "1001724", "to": "77f689d8-1d07-4ea2-a34c-a73aa9b76516"}, {"color": "#888888", "from": "1001724", "to": "da21019fe59d"}, {"color": "#888888", "from": "1001724", "to": "2001cbbf1ffd"}, {"color": "#888888", "from": "1001724", "to": "b1f5cac8-6cc2-4e4d-81c0-478210da138a"}, {"color": "#888888", "from": "1001724", "to": "aaf7c363-922b-4443-a7d0-4073750b6f0c"}, {"color": "#888888", "from": "37dd4c20-6bef-4de6-bedd-c4cc204ff1e1", "to": "1001601"}, {"color": "#888888", "from": "1001601", "to": "2f41b084-7c53-4ef4-a746-a36c591aabdc"}, {"color": "#888888", "from": "1001601", "to": "a207523c-d826-45c5-b104-d2f0a80dda12"}, {"color": "#888888", "from": "1001601", "to": "39974b70-b667-4540-8eaa-a91d0e5180a1"}, {"color": "#888888", "from": "1001601", "to": "efc1aeba-0a0d-40be-a24a-ee393299a480"}, {"color": "#888888", "from": "1001601", "to": "677571db-4cac-4501-9a14-3dde045673f2"}, {"color": "#888888", "from": "1001601", "to": "34ffd366-5581-404c-9de7-8e20958bd62f"}, {"color": "#888888", "from": "1001601", "to": "1adf0fa5-91a7-4dd3-895e-75799df6bbc6"}, {"color": "#888888", "from": "1001601", "to": "823305a7-5740-42b4-b007-29e490e17b08"}, {"color": "#888888", "from": "1001601", "to": "922d079e-4a67-47a5-b84e-e286242d5c9e"}, {"color": "#888888", "from": "61e3bcdc-d995-41d9-a6aa-1172af68f104", "to": "8096804"}, {"color": "#888888", "from": "61e3bcdc-d995-41d9-a6aa-1172af68f104", "to": "84f9eb766ea0"}, {"color": "#888888", "from": "8096804", "to": "dcf6d0f9-3710-4f97-98cb-75288033c56d"}, {"color": "#888888", "from": "8096804", "to": "1bf9329f6a6e"}, {"color": "#888888", "from": "8096804", "to": "84f9eb766ea0"}, {"color": "#888888", "from": "8096804", "to": "c09ceb12-0242-4eed-b1f9-07416475978f"}, {"color": "#888888", "from": "8096804", "to": "639d42ba6306"}, {"color": "#888888", "from": "8096804", "to": "dfd7297746b0"}, {"color": "#888888", "from": "8096804", "to": "8ccd8f72-69ca-4e64-a195-adde694b14d3"}, {"color": "#888888", "from": "8096804", "to": "6758a04b-f52e-432b-9f42-54fb615baa63"}, {"color": "#888888", "from": "8096804", "to": "25727c7d-380f-4845-993e-b34406f9330f"}, {"color": "#888888", "from": "8096804", "to": "d39934c4-a029-4fd9-a04e-344ea251e89e"}, {"color": "#888888", "from": "8096804", "to": "cc66231a1a9b"}, {"color": "#888888", "from": "8096804", "to": "e54e6b399efc"}, {"color": "#888888", "from": "8096804", "to": "278e2d59feeb"}, {"color": "#888888", "from": "8096804", "to": "e14211ff4114"}, {"color": "#888888", "from": "8096804", "to": "8ee77e07-5c95-4540-ba17-5ca052a36720"}, {"color": "#888888", "from": "8096804", "to": "2fb65df5-633e-47e6-abcd-4496fe47e470"}, {"color": "#888888", "from": "8096804", "to": "03499e02-dce1-43dd-bf1a-0c73ff75a36e"}, {"color": "#888888", "from": "8096804", "to": "22765460-7b19-4f2a-9138-ac601e1a824c"}, {"color": "#888888", "from": "8096804", "to": "3aa23c92e2a2"}, {"color": "#888888", "from": "8096804", "to": "450ba6e99bc6"}, {"color": "#888888", "from": "8096804", "to": "38d72063-56fc-493c-9f27-dd17e58dcedd"}, {"color": "#888888", "from": "da21019fe59d", "to": "aaf7c363-922b-4443-a7d0-4073750b6f0c"}, {"color": "#888888", "from": "77f689d8-1d07-4ea2-a34c-a73aa9b76516", "to": "2001cbbf1ffd"}, {"color": "#888888", "from": "2001cbbf1ffd", "to": "b1f5cac8-6cc2-4e4d-81c0-478210da138a"}, {"color": "#888888", "from": "3407e0e3-4900-497c-bff8-8d370754e3ee", "to": "8097816"}, {"color": "#888888", "from": "3407e0e3-4900-497c-bff8-8d370754e3ee", "to": "c0ba22140cec"}, {"color": "#888888", "from": "8097816", "to": "c0ba22140cec"}, {"color": "#888888", "from": "8097816", "to": "6848d77e2c51"}, {"color": "#888888", "from": "8097816", "to": "77e1b9368bd0"}, {"color": "#888888", "from": "8097816", "to": "d8d63dac8d42"}, {"color": "#888888", "from": "8097816", "to": "4e26df5b-b1ac-4964-9f23-979d8237a2b4"}, {"color": "#888888", "from": "8097816", "to": "e9779e82-9943-4039-a8c7-fc1609d99295"}, {"color": "#888888", "from": "8097816", "to": "ec58c825-69b5-4406-bb89-189aed588793"}, {"color": "#888888", "from": "8097816", "to": "f7d3790baeea"}, {"color": "#888888", "from": "8097816", "to": "d61186783374"}, {"color": "#888888", "from": "8097816", "to": "54cb22e5f0a7"}, {"color": "#888888", "from": "8097816", "to": "2cf176c253e7"}, {"color": "#888888", "from": "8097816", "to": "f7d072960756"}, {"color": "#888888", "from": "8097816", "to": "71446f27d2a1"}, {"color": "#888888", "from": "8097816", "to": "6e6e960d-4166-4a5e-a865-aaec299166b7"}, {"color": "#888888", "from": "8097816", "to": "e843ead3-744f-4d53-bbd0-84c19fca6e87"}, {"color": "#888888", "from": "8097816", "to": "73a88af0-e480-4d7e-9ad4-647d221f273b"}, {"color": "#888888", "from": "8097816", "to": "7e3cf9a0-4d7a-4a2b-a905-8385f9523baf"}, {"color": "#888888", "from": "8097816", "to": "97777211-6db6-4532-806d-8e44a7c1f83f"}, {"color": "#888888", "from": "8097816", "to": "0415f350-ed54-4037-9fed-40712a77fc43"}, {"color": "#888888", "from": "8097816", "to": "66c13c66-5df1-4af1-a38f-ffc0f65ac0cc"}, {"color": "#888888", "from": "dcf6d0f9-3710-4f97-98cb-75288033c56d", "to": "1bf9329f6a6e"}, {"color": "#888888", "from": "1bf9329f6a6e", "to": "d39934c4-a029-4fd9-a04e-344ea251e89e"}, {"color": "#888888", "from": "c0ba22140cec", "to": "e9779e82-9943-4039-a8c7-fc1609d99295"}, {"color": "#888888", "from": "f7d3790baeea", "to": "73a88af0-e480-4d7e-9ad4-647d221f273b"}, {"color": "#888888", "from": "2cf176c253e7", "to": "4e26df5b-b1ac-4964-9f23-979d8237a2b4"}, {"color": "#888888", "from": "84f9eb766ea0", "to": "8ee77e07-5c95-4540-ba17-5ca052a36720"}, {"color": "#888888", "from": "54cb22e5f0a7", "to": "97777211-6db6-4532-806d-8e44a7c1f83f"}, {"color": "#888888", "from": "6848d77e2c51", "to": "6e6e960d-4166-4a5e-a865-aaec299166b7"}, {"color": "#888888", "from": "e54e6b399efc", "to": "22765460-7b19-4f2a-9138-ac601e1a824c"}, {"color": "#888888", "from": "77e1b9368bd0", "to": "ec58c825-69b5-4406-bb89-189aed588793"}, {"color": "#888888", "from": "e14211ff4114", "to": "38d72063-56fc-493c-9f27-dd17e58dcedd"}, {"color": "#888888", "from": "278e2d59feeb", "to": "03499e02-dce1-43dd-bf1a-0c73ff75a36e"}, {"color": "#888888", "from": "639d42ba6306", "to": "c09ceb12-0242-4eed-b1f9-07416475978f"}, {"color": "#888888", "from": "6758a04b-f52e-432b-9f42-54fb615baa63", "to": "3aa23c92e2a2"}, {"color": "#888888", "from": "cc66231a1a9b", "to": "8ccd8f72-69ca-4e64-a195-adde694b14d3"}, {"color": "#888888", "from": "d8d63dac8d42", "to": "e843ead3-744f-4d53-bbd0-84c19fca6e87"}, {"color": "#888888", "from": "d61186783374", "to": "66c13c66-5df1-4af1-a38f-ffc0f65ac0cc"}, {"color": "#888888", "from": "dfd7297746b0", "to": "25727c7d-380f-4845-993e-b34406f9330f"}, {"color": "#888888", "from": "450ba6e99bc6", "to": "2fb65df5-633e-47e6-abcd-4496fe47e470"}, {"color": "#888888", "from": "71446f27d2a1", "to": "7e3cf9a0-4d7a-4a2b-a905-8385f9523baf"}, {"color": "#888888", "from": "f7d072960756", "to": "0415f350-ed54-4037-9fed-40712a77fc43"}]);

 nodeColors = {};
 allNodes = nodes.get({ returnType: "Object" });
 for (nodeId in allNodes) {
 nodeColors[nodeId] = allNodes[nodeId].color;
 }
 allEdges = edges.get({ returnType: "Object" });
 // adding nodes and edges to the graph
 data = {nodes: nodes, edges: edges};

 var options = {"physics": {"forceAtlas2Based": {"gravitationalConstant": -50, "centralGravity": 0.01, "springLength": 100, "springConstant": 0.08}, "minVelocity": 0.75, "solver": "forceAtlas2Based"}, "nodes": {"font": {"color": "black"}, "borderWidth": 2, "borderWidthSelected": 4}, "edges": {"color": {"inherit": false}, "smooth": {"enabled": true, "type": "dynamic"}}};

 network = new vis.Network(container, data, options);

 return network;
 }
 drawGraph();
 </script>
 </body>
</html>

In the above project, there is a resolved ID that appears to have two users merged together. Note that user_main_id had 2 emails (alexkeener@example.com and alicekeener@example.com) within. Upon running the visualizer on this main_id - you can see that both the users associated with the emails share the same user_id (akeener). With this knowledge, you can quickly make changes to the Profiles configuration to unstitch these users or fix the source data.

Interactive LLM session

RudderStack supports this feature only for the Snowflake warehouse currently. Support for the other warehouses is coming soon.

The audit tool also provides an open-ended text2sql session where you can ask arbitrary questions related to the ID graph. The question goes to an LLM which prepares a SQL query.

The SQL query then runs on your data warehouse and generates an answer.

Click here to view a sample

Question:
 how many accounts do we have with more than one domain?

Thought:
 Ran following queries to get the answer:

 SELECT COUNT(*) AS accounts_with_multiple_domains
FROM (
 SELECT ACCOUNT_MAIN_ID
 FROM account_id_graph
 WHERE other_id_type = 'domain'
 GROUP BY ACCOUNT_MAIN_ID
 HAVING COUNT(DISTINCT other_id) > 1
) subquery;;

 SELECT COUNT(DISTINCT ACCOUNT_MAIN_ID) AS total_accounts_with_domains
FROM account_id_graph
WHERE other_id_type = 'domain';;

Response:

Now we have more context to provide a comprehensive answer:

There are 100 accounts that have more than one domain associated with them. This is out of a total of 25,000 accounts that have at least one domain.

To put this into perspective:
- Approximately 0.50% of accounts with domains (100 out of 25,000) have multiple domains associated with them.
- The vast majority (about 99.75%) of accounts with domains are associated with just a single domain.

This distribution suggests that while most accounts in the system are linked to a single domain, there is a small but significant number of accounts that operate across multiple domains. These could represent larger organizations, companies with multiple brands, or entities that have expanded their online presence across different domain names.

You can ask next question, type exit to quit, or back to select a different entity

Q: what are these accounts? why do they have multiple domains?
INFO:api_server.sessions.chat_session@2024-09-30 11:02:28,987: Getting response for prompt: what are these accounts? why do they have multiple domains?
INFO:api_server.sessions.chat_session@2024-09-30 11:02:28,988: Converted history to prompt
INFO:api_server.sessions.chat_session@2024-09-30 11:02:28,988: Making llm calls with the extended prompt

Response:

Based on the query results, we can answer the user's question:

These are the top 10 accounts with multiple domains, ordered by the number of domains they have:

1. An account (ACCOUNT_MAIN_ID: rid0a615df48d4ae923f518ab3d052ded1f) with 8 domains, including <domain1>, <domain2>, <domain3>, and others.
2. Apex (ACCOUNT_MAIN_ID: rid7492d64d982d1e6b8531e12a39513d6e) with 4 domains across different countries.
3. Aurora (ACCOUNT_MAIN_ID: rid149ba1fd2a2c59b30663fb22d4aabfaa) with 4 domains in different countries.
4. Acme (ACCOUNT_MAIN_ID: rid49a8de71ef3cbe73d0359965e000a20b) with 3 domains in different Nordic countries.
5. A company (ACCOUNT_MAIN_ID: rid0a256a02fc2aaf3bf812db4df78b95fa) with 3 domains.
6. Clement (ACCOUNT_MAIN_ID: rid44cb9148b8c8b5b608d5a8ecebc11e4b) with 3 related domains.
7. Hero (ACCOUNT_MAIN_ID: rid301a9453dddc0bca3a7315af38370c3e) with 3 domains in different countries.
8. Buffalo (ACCOUNT_MAIN_ID: rid0c22fb973e9b75dd2356072f29c4f91e) with 3 related domains.
9. Majority (ACCOUNT_MAIN_ID: rid5d63f8165e5fa418cef359f0dcde888a) with 2 domains.
10. Super (ACCOUNT_MAIN_ID: rid21bb08ad7fd688b23e43d00b0b54d3c0) with 2 domains.

As for why these accounts have multiple domains, we can infer several reasons based on the patterns observed:

1. International presence: Companies like Apex, Aurora, Acme, and Buffalo have domains for different countries or regions.

2. Brand diversification: The first account with 8 domains seems to represent multiple brands or services under one parent company.

...response continued...

These multiple domains allow companies to tailor their online presence to different markets, products, or strategies while maintaining a unified backend infrastructure for user identification and data management.

Privacy policy

For an interactive LLM session, RudderStack sends the user prompt and some data (for example, table and column names, SQL query response, etc.) to an external LLM service.

RudderStack currently uses two LLM service providers:

OpenAI (Privacy policy: here)
Anthropic (Privacy policy: here)

Both the OpenAI and Anthropic privacy policies state that the data in APIs is retained for 30 days to monitor for any policy violations.

It will not be used for training purposes and will be deleted after 30 days.

The tool asks for your consent to use these LLMs before making the first LLM call. You can opt out of the LLM services if there are concerns about sending your data to the external providers.

If you opt out of the LLM services, the audit tool never makes any calls to the LLM provider.

Any consent you provide (opt-in or opt-out) is stored locally in the ~/.pb/preferences.yaml file on the same device and gets reused in all the future profiles audit runs.

To reset your consent, manually delete the consent from the ~/.pb/preferences.yaml file.

Tracking

Note that for every command run, the tool tracks the audit_start and audit_end events.

It creates an anonymous_id in the ~/.pb/preferences.yaml file and uses this ID for future runs. All the runs get associated with this ID. On the very first run when anonymous_id is created, you will see the following disclaimer:

 Privacy Policy regarding Telemetry:
 - This tool collects usage statistics to help us improve the product.
 - We do not collect or store any sensitive information from your models.
 - You can opt out by adding 'telemetry_opted_out: true' to ~/.pb/preferences.yaml

The audit tool also collects the IP address of the device as a part of the standard RudderStack call. However, it does not collect any other user identifier.

You can add a line telemetry_opted_out: true in this preference file to opt out of tracking.

FAQ

See the FAQs on Python-related errors for more information.

Archive: Commands

Mon, 01 Jan 0001 00:00:00 +0000

The Profile Builder tool supports specific commands, making executing the usual operations easier. The basic syntax of executing a command is:

$ pb <command> <subcommand> [parameters]

Supported commands

You can use the following Profile Builder commands:

audit

Analyze your ID graphs, for example, count of IDs before and after stitching, distribution of ID counts, etc.

This command also lets you analyze a single cluster by understanding how different IDs get connected to one, by visualizing all IDs and their connections and highlighting the most important IDs that form connections in that group.

See the Profiles ID Stitcher Audit tool guide for more information.

pb audit id_stitcher

To use this command, make sure to:

Run the project successfully at least once so that it produces an ID graph.

Install the package profiles-mlcorelib (v0.6.0 or above) in your Python environment (v3.9.0 to v3.11.10) using pip install profiles-mlcorelib. Note that it should be the same Python environment as the profiles-rudderstack library.

Optional Parameters

Parameter	Description
`--rpc_python_path`	Absolute path of the Python executable to use for PyNative models.

cleanup

Displays and removes materials, older than the retention time period specified by the user (default value is 180 days).

pb cleanup materials -r <number of days>

Optional Parameter

Parameter	Description
`-r`	Retention time in number of days. Example: If you pass 1, then all the materials created prior to one day (24 hours) are listed. This is followed by prompts asking you for confirmation, after which you can view the material names and delete them.
`--retention_time_in_hours`	Retention time in hours. Example: To remove materials older than 3 hours, use `pb cleanup materials --retention_time_in_hours 3`. This is followed by prompts asking you for confirmation, after which you can view the material names and delete them.
`--retention_time_in_ms`	Retention time in milliseconds. Example: To remove materials older than 100 milliseconds, use `pb cleanup materials --retention_time_in_ms 100`. This is followed by prompts asking you for confirmation, after which you can view the material names and delete them.
`--src_url`	Location of the PB project used as a filter on `SRC_URL` while the cleanup runs. Default is your current working directory. Pass `*` as the value if all source URLs should be considered.
`--concurrency`	Enables concurrency by defining the number of concurrent workers for clean up. The default value is 1. Note: This flag is currently supported for Snowflake only.
`--expired`	Ensures that materials are only deleted if they are truly expired based on the model retention periods and dependency tracking. This is recommended when you are using timegrains.
`--remove_latest_view_ptrs`	Allows forced removal of expired materials from the latest run, if necessary. This flag is useful when you want to remove expired materials even if they belong to the latest run. It is also useful when clearing space but should be used carefully as it can remove materials that recent models still rely on.

compile

Generates SQL queries from models.

pb compile

It creates SQL queries from the models/profiles.yaml file, storing the generated results in the Output subfolder in the project’s folder. With each run, a new folder is created inside it. You can manually execute these SQL files on the warehouse.

Optional parameters

Parameter	Description
`clean_output`	Empties the output folder(s) before executing the command.
`-c`	Uses a site configuration file other than the one in `.pb` directory. Example: `$ pb compile -c MyOtherConnection/siteconfig.yaml`
`-t`	Defines target name (mentioned in `siteconfig.yaml`) or timestamp in building the model. Example: If your `siteconfig.yaml` has two targets, `dev` and `test`, and you want to use the `test` instance: `$ pb compile -t test`
`--begin_time`	Timestamp to be used as a start time in building model.
`--end_time`	Timestamp to be used as an end time in building model.
`--migrate_on_load`	Whether to automatically migrate the project and packages to the latest version. Defaults to false.
`--migrated_folder_path`	Folder location of the migrated project. Defaults to sub-directory of the project folder.
`-p`	Uses a project file (`pb_project.yaml`) other than the one in current directory. Example: `$ pb compile -p MyOtherProject`. Fetches project from a URL such as GitHub. Example 1:`$ pb compile -p git@github.com:<orgname>/<repo>`. You can also fetch a specific tag, like `$ pb compile -p git@github.com:<orgname>/<repo>/tag/<tag_version>/<folderpath>` Compiles project hosted on S3. Example: `pb compile -p s3://<url>`
`--rebase_incremental`	Rebases any incremental models (build afresh from their inputs) instead of starting from a previous run. You can do this every once in a while to address the stale data or migration/cleanup of an input table.
`--assume_all_inputs_exist`	Does not throw error if the specified table/column is absent in the warehouse. It is helpful in debugging issues when you want to compile the materials even if any table or column is absent.
`--end_time_offset`	Adds an offset to the end timestamp in a human readable format. It means that RudderStack does not use any data you load in the warehouse after the offset time has elapsed for that run. For example, `pb compile --end_time_offset=45m` ensures that RudderStack does not use any data older than 45 minutes from the run’s start time. Note that you can’t use this new flag with the `seq_no` or `end_time` flags.
`--passcode`	Snowflake MFA Passcode. Works only with Snowflake.

discover

Discovers elements in the warehouse, such as models, entities, features and sources.

pb discover

It allows you to discover all the registered elements in the warehouse.

Subcommands

Discover all the models, entities, features, sources, and materials in the warehouse.

$ pb discover models
$ pb discover entities
$ pb discover features
$ pb discover sources
$ pb discover materials

Optional parameters

Parameter	Description
`-e`	Discovers specific entities with their name. Example: `$ pb discover -e 'Name'`
`-m`	Discovers a specific model. Example: `$ pb discover -m 'MY_DATABASE.PROD_SCHEMA.CREATED_MODEL'`
`-c`	Uses a site config other than the default one. Example: `$ pb discover -c siteconfig.yaml`
`-s`	Discovers entities in a specified schema.
`-s "*"`	Discovers entities across all schemas (case-sensitive).
`-u`	Discovers entities having the specified source URL’s. Example: To discover all the entities coming from GitHub: `$ pb discover -u %github%`
`-t`	Selects target (mentioned in `siteconfig.yaml`).
`-p`	Uses project folder other than the one in current directory. Example: `$ pb discover -p ThisFolder/ThatSubFolder/SomeOtherProject/`
`-f`	Specifies a file path to dump the discovery output into a csv file. Example: `$ pb discover -f path/to/csv_file.csv`
`-k`	Restricts discovery of the specified model keys. Example: `$ pb discover -k entity_key:mode_type:model_name`
`--csv_file`	Specify this flag with a file path to dump the discovery output into a csv file.
`--passcode`	Snowflake MFA Passcode. Works only with Snowflake.

Examples

# Discover all the models
$ pb discover models

# Discover a model with specific name
$ pb discover -m 'RUDDER_WEB_EVENTS.PROD_SCHEMA.feature_profile'

# Discover all features having 'max' in their name
$ pb discover features -u %max%

# Discover all the entities for a specific profile
$ pb discover entities -c siteconfig.yaml

# Discover all materials for target dev
$ pb discover materials -t dev

# Export output of discover command to a CSV file in output folder
$ pb discover -f my-custom-name.csv

# Export all sources to a CSV file in output folder
$ pb discover sources -f my-custom-name.csv

help

Provides list information for any command.

$ pb help

Subcommand

$ pb help <command_name>

Get usage information for a specific command, with subcommands, and optional parameters.

init

Creates connection and initializes projects.

pb init

Subcommands

pb init connection

Inputs values for a warehouse connection and then stores it in the siteconfig.yaml file.

pb init pb-project

Generates files in a folder named HelloPbProject with sample data. You can change it as per project information, models, etc.

Optional parameters

Parameter	Description
`pb-project -o`	Creates a Profile Builder project with a different name by specifying it as an additional parameter. Example: To create a Profile Builder project with the name SomeOtherProject: `$ pb init pb-project -o SomeOtherProject`
`connection -c`	Creates `siteconfig.yaml` at a location other than `.pb` inside home directory. Example: To create `myconfig.yaml` in the current folder: `$ pb init connection -c myconfig.yaml`.

insert

Allows you to store the test dataset in your (Snowflake) warehouse . It creates the tables sample_rs_demo_identifies and sample_rs_demo_tracks in your warehouse schema specified in the test connection.

# Select the first connection named test having target and output as dev, of type Snowflake.
$ pb insert
# By default it'll pick up connection named test. To use connection named red:
$ pb insert -n red
# To pick up connection named red, with target test .
$ pb insert -n red -t test

This command is supported only for Snowflake currently.

migrate

Migrate your project to the latest schema.

Subcommands

pb migrate manual

Based on the current schema version of your project, enlists all the steps needed to migrate it to the latest one.

pb migrate auto

Automatically migrate from one version to another. To migrate your models, navigate to the folder where your project files are stored and execute one of the following:

pb migrate auto --inplace: Replaces contents of existing folder with the migrated folder.
pb migrate auto -d <MigratedFolder>: Keeps the original project intact and stores the migrated project in another folder.

Optional parameters

Parameter	Description
`-c`	Uses a `siteconfig.yaml` file other than the one in your home directory.
`-t`	Target name (defaults to the one specified in `siteconfig.yaml` file).
`-v`	Version to which the project needs to be migrated (defaults to the latest version).
`-d`	Destination folder to store the migrated project files. Example: `pb migrate auto -d FolderName`
`--force`	Ignores warnings (if any) and migrates the project.
`--inplace`	Overwrites the source folder and stores migrated project files in place of original. Example: `pb migrate auto --inplace`
`-p`	Uses a project folder other than the one in current directory. Example: `$ pb migrate auto -p ThisFolder/ThatSubFolder/SomeOtherProject/`
`--passcode`	Snowflake MFA Passcode. Works only with Snowflake.

run

Creates identity stitcher or feature view model in the warehouse.

pb run

It generates the SQL files from models and executes them in the warehouse. Once executed, you can see the output table names, which are accessible from the warehouse.

Optional parameters

The run command supports:

All the parameters of compile command except --assume_all_inputs_exist,
And the following ones:

Parameter	Description
`--force`	Does a force run even if the material already exists.
`--write_output_csv`	Writes all the generated tables to CSV files in the specified directory. Example: `$ pb run --write_output_csv WriteOutputHere.csv`
`--model_args`	Customizes behavior of an individual model by passing configuration params to it. The only argument type supported currently is `breakpoint` for feature table models. The `breakpoint` parameter lets you generate and run SQL only till a specific feature/tablevar. You can specify it in the format `modelName:argType:argName` where argName is the name of feature/tablevar. Example: `$ pb run --model_args domain_profile:breakpoint:salesforceEvents`
`--model_refs`	Restricts the operation to a specified model. You can specify model references like `pb run --model_refs models/<model-name> --seq_no latest`
`--seq_no`	Sequence number for the run, for example, 0, 1, 2,…, latest. If you do not pass this parameter, a new sequence number is used. You can check the run logs or use the discover commands to know about existing sequence numbers.
`--ignore_model_errors`	Allows the project to continue to run in case of an erroneous model. The execution will not stop due to one bad model.
`--grep_var_dependencies`	Uses regex pattern matching over fields from vars to find references to other vars and set dependencies. By default, it is set to `true`.
`--concurrency`	(Experimental) Lets you run the models concurrently in a warehouse (wherever possible) based on the dependency graph. In CLI, you can specify the concurrency level for running models in a project via `pb run --concurrency <int>` (default int value is 1). Note: This flag is currently supported for Snowflake only and the concurrency limit is `20`. To increase the limit, see the Snowflake documentation.
`--begin_time`	Timestamp to be used as a start time in building model.
`--end_time`	Timestamp to be used as an end time in building model.
`--migrate_on_load`	Whether to automatically migrate the project and packages to the latest version. Defaults to false.
`--migrated_folder_path`	Folder location of the migrated project. Defaults to sub-directory of the project folder.
`--disable_registry_cache`	Disables the feature where the registry is cached by default while running the project material, to reduce the number of warehouse calls. Default value is `false`.
`--rpc_python_path`	Absolute path of the Python executable to use for PyNative models.
`--passcode`	Snowflake MFA Passcode. Works only with Snowflake.

show

Obtains a comprehensive overview of models, id_clusters, packages, and more in a project. Its capacity to provide detailed information makes it particularly useful when searching for specific details, like all the models in your project.

$ pb show

Subcommands

models

pb show models

This command lets you view information about the models in your project. The output includes the following information about each model:

Warehouse name: Name of the table/view to be created in the warehouse.
Model type: Whether its an identity stitching, feature view, SQL model etc.
Output type: Whether the output type is ephemeral, table, or view.
Run type: Whether the model’s run type is discrete or incremental.
SQL type: Whether the SQL type of the model is single_sql or multi_sql.

pb show models --json

This subcommand saves all model details in a JSON file.

dependencies

pb show dependencies

This subcommand generates a DAG in a graph file (dependencies.png) highlighting the dependencies of all models in your project.

dataflow

pb show dataflow

This subcommand generates a DAG in a graph file (dataflow.png) highlighting the data flow of all models in your project.

idstitcher-report

pb show idstitcher-report --id_stitcher_model models/<ModelName> --migrate_on_load

This subcommand creates a detailed report about the identity stitching model runs. To know the exact model reference to be used, execute the pb show models command.

Note that:

By default, this command picks up the last run - you can change this behavior using the flag -l.
Use the --seed_clusters_csv flag to pass a CSV file with two columns - id and id_type. The cluster ID can be either rudder_id or other_id. A sample file is shown below:

id	id_type
`'8f81237f90'`	`'user_id'`
`'alex@example.com'`	`'email'`

Note that:

Each row specifies an ID to be mapped to a cluster main ID, which is then included in the report.

The ID need not be specific - for example, specifying alex returns all the pattern matches containing that name.

The output of the pb show idstitcher-report command consists of:

ModelRef: The model reference name.
Seq No: Sequence number of the run for which you are creating the report.
Material Name: Output name as created in warehouse.
Creation Time: Time when the material object was created.
Model Converged: Indicates a successful run if true.
Pre Stitched IDs before run: Count of all the IDs before stitching.
Post Stitched IDs after run: Count of unique IDs after stitching.

Profile Builder also generates a HTML report with relevant results and graphics including largest cluster, ID graph, etc. It is saved in output folder and the exact path is shown on screen when you execute the command.

entity-lookup

pb show entity-lookup -v '<trait value>'

This subcommand lists all the features associated with an entity using any of the traits (flag -v) as ID types (email, user id, etc. that you are trying to discover).

Optional parameters

Parameter	Description
`--entity string`	(Optional) Passes the entity value. (default `user`).
`-h`	Displays help information for the command.
`-p`	Specifies the project path to list the models. If not specified, it uses the project in the current directory.
`-c`	File location of the `siteconfig.yaml` (defaults to the one in your home directory).
`-t`	Target name (defaults to the target specified in `siteconfig.yaml` file).
`--include_disabled`	Lets the disabled models be a part of the generated graph image (applicable to `dataflow` and `dependencies`).
`--seq_no`	Specifies a particular run for an ID stitcher model (applicable for `idstitcher-report`).
`--passcode`	Snowflake MFA Passcode. Works only with Snowflake.

plan

pb show plan

This subcommand shows the detailed information about materials created by a specific run and their corresponding timegrains.

Optional parameters

Parameter	Description
`--show_dependents`	Shows the list of dependent objects along with dependencies for each material. For example, to view information about a material with `seq_no 2` and get information about its dependent objects, use `pb show plan --seq_no 2 --show_dependents`.
`--json`	Shows the information in JSON format.

query

Executes SQL query on the warehouse and prints the output on screen (10 rows by default).

pb query <query>

For example, if you want to print the output of a specific table/view named user_id_stitcher, run the following query:

pb query "select * from user_id_stitcher"

To reference a model with the name user_default_id_stitcher for a previous run with seq_no 26, you can execute:

pb query "select * from {{this.DeRef('path/to/user_default_id_stitcher')}} limit 10" --seq_no=26

Optional parameters:

Parameter	Description
`-f`	Exports output to a CSV file.
`-max_rows`	Maximum number of rows to be printed (default is 10).
`-seq_no`	Sequence number for the run.

validate

Validates aspects of the project and configuration.

$ pb validate

It allows you to run various tests on the project-related configurations and validate those. This includes but is not limited to validating the project configuration, privileges associated with the role specified in the site configuration of the project’s connection, etc.

Subcommands

$ pb validate access

Runs tests on the role specified in the site configuration file and validates if the role has privileges to access all the related objects in the warehouse. It throws an error if the role does not have required privileges to access the input tables or does not have the permissions to write the material output in the output schema.

version

Shows the Profile Builder’s current version along with its GitHash and native schema version.

pb version

Archive: YAML Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

YAML is the preferred choice for writing Profile Builder files due to its simplicity and ease of use.

This guide explains the base concepts, syntax, and best practices for writing code in YAML.

What is YAML?

YAML, short for YAML Ain’t Markup Language or Yet Another Markup Language, is a data serialization format often used in config files and exchange of data. A YAML file uses indentation, specific characters, and line breaks for representing various data structures.

Sample YAML file

Below is how a sample YAML document looks like. It contains key-value pairs where the keys are on the left, followed by a colon (:), and the associated values are on the right. The hierarchy and data structure is defined using indentation. The next section explains this in more detail.

# This is a comment
person:
 name: Ruddy Buddy # Note the spacing used for indentation
 age: 42
 is_employed: true
 address: # An object called address
 street: Jefferson Davis Highway
 city: Ruther Glen
 state: Vermont
 phone: 555-90-210
 favorite_sports: # A list
 - soccer
 - baseball

The above code has details of an object called person with properties like name, age, gender, is_student, address and favorite sports.

Here’s how the same YAML file looks in the JSON format:

{
 "person": {
 "name": "Ruddy Buddy",
 "age": 42,
 "is_employed": true,
 "address": {
 "street": "Jefferson Davis Highway",
 "city": "Ruther Glen",
 "state": "Vermont",
 "phone": "555-90-210"
 },
 "favorite_sports": [
 "soccer",
 "baseball"
 ]
 }
}

Indentation

In YAML, the indentation is done using spaces - to define the structure of data. Throughout the YAML file, the number of spacing should be consistent. Typically, we use two spaces for indentation. YAML is whitespace-sensitive, so do not mix spaces and tabs.

# Example of correct indentation
person:
 name: Ruddy Buddy # We used 2 spaces
 age: 42

# Example of incorrect indentation
person:
 name: Ruddy Buddy 
 age: 42 # We mixed spacing and tabs

Comments

As shown above, YAML has single-line comments that start with hash (#) symbol, for providing additional explanation or context in the code. Comments are used to improve readability and they do not affect the code’s functionality.

# YAML comment
person:
 name: Ruddy Buddy # Name of the person
 age: 42 # Age of the person

Data types in YAML

YAML supports several data types:

Scalars: Represent strings, numbers, and boolean values.
Sequences: Represent lists and are denoted using a hyphen (-).
Mappings: Key-value pairs used to define objects or dictionaries using colon (:).

# Example of data types in YAML
person:
 name: Ruddy Buddy # Scalar (string)
 age: 42 # Scalar (number)
 is_employed: true # Scalar (boolean)
 address: # Mapping (object)
 street: Jefferson Davis Highway
 city: Ruther Glen
 state: Vermont
 phone: 555-90-210
 favorite_sports: # Sequence (list)
 - soccer
 - baseball

Chomp modifiers

YAML provides two chomp modifiers for handling line breaks in scalar values.

>: Removes all newlines and replaces them with spaces.

description: >
 Here is an example of long description
 which has multiple lines. Later, it
 will be converted into a single line.

|: Preserves line breaks and spaces.

description: |
 Here is another long description, however
 it will preserve newlines and so the original
 format shall be as-it-is.

Special characters

You can use escape symbols for special characters in YAML. For example, writing an apostrophe in description can cause the YAML parser to fail. In this case, you can use the escape character.

Best practices

Follow these best practices for writing clean YAML code in your Profiles projects:

Always keep consistent indentation (preferably spaces over tabs).
Give meaningful names to your keys.
Avoid excessive nesting.
YAML is case sensitive, so be mindful of that.
Add comments wherever required.
Use blank lines to separate sections like ID stitcher, feature view, etc.
If your strings contain special characters, then use escape symbols.
Make sure you end the quotes in strings to avoid errors.
Use chomp modifiers for multi-line SQL.

References

YAML for VS Code: Extension for comprehensive YAML support in Visual Studio Code.
YAML Lint for linting.