RudderStack Docs – Additional Resources

Archive: Profiles Run Process

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains the run process of your Profiles project in detail along with the order of execution of all of its components.

Project assumptions

This guide assumes you have a Profiles project with the following specifications:

A user entity
ID stitcher model
entity_vars
A feature table model
Two Feature views models (default, and using email as the primary key)
Custom SQL model to create entity_vars
Cohort representing high value users

Run process

When you exceute the pb run command to run your Profiles project, RudderStack performs these steps:

Constructs a DAG to analyze the order in which each component should run based on the model dependencies.
Compiles the configuration specified in your project files and outputs the SQL files (that will run in your database) in the output directory of your project.
Creates an input var table for each input table/source and adds an input row ID to it. This is used to create the entity_vars.
Performs the ID stitching process.
Creates the ALL cohort which is a table of distinct list of all the main_ids created from the most recent ID stitcher run. It serves as the base for the entity var table as well as custom cohorts.
Uses main_ids created in the previous step and appends them to the input var tables (created in step 3).
Creates entity var tables which serve as the base for any feature table/views models built around this entity. The default output is a single column containing main_ids from the ID graph.
Creates features for actionable views by calculating entity_vars and input_vars.
Joins calculated features (via entity_vars) to the entity var table.
Creates and materializes the custom cohort list (if defined in the YAML).
Creates the custom cohort var table (similar to the entity var table) which serves as the target table for feature creation and the base table for any feature table/views created on top of custom cohorts.
Generates the final tables and views:
- Feature table model and default feature view using the entity var table
- Custom feature views using the default feature views
- Custom cohort views using the cohort var table

Archive: Python Multi-version Support

Mon, 01 Jan 0001 00:00:00 +0000

This feature is supported for the Profiles versions 0.10.8, 0.11.5, 0.12.0, and all versions after 0.12.0.

You can constraint your Profiles project to run only on specific version(s) by specifying it in the pb_project.yaml file, under the python_requirements key.

For example, the following snippet pins your project to v0.16.0:

python_requirements:
 - profiles-rudderstack==0.16.0

Use the following snippet to stay on any patch release within the 0.24 minor line (between 0.24.0 inclusive and 0.25.0 exclusive):

python_requirements:
 - profiles-rudderstack>=0.24.0,<0.25.0

If you do not specify any version in pb_project.yaml, the latest Profiles version is used by default. The version constraints follow the same syntax as those of Python dependency specifiers.

Make sure that the version of Profiles project is the same in your environment and the pb_project.yaml file. Otherwise, RudderStack will throw an error.

Archive: Snowflake Sample Data

Mon, 01 Jan 0001 00:00:00 +0000

RudderStack provides a sample dataset for the Snowflake warehouse, available in the Snowflake marketplace. You can use this data to run the Profiles project and Propensity Scores through the UI or the CLI.

Note that:

The number of columns in this dataset is intentionally limited to make the dataset easy to understand.

All email addresses are generated randomly.

No PII is used in the dataset generation.

The following tables, properties, and user information is included in the dataset:

Tables

This dataset includes the following RudderStack event data tables and the approximate number of rows in each table:

Table	Description	Number of rows
`PAGES`	Page view events from anonymous and known users.	~172k
`TRACKS`	Summarized tracked user actions (like `login`, `signup`, `order_completed`, etc.).	~56k
`IDENTIFIES`	Associated with `identify` calls when the user provides a unique identifier.	~22k
`ORDER_COMPLETED`	Detailed payloads from tracked `order_completed` events.	~1.2k

The events that generate these tables follow the pattern of a standard ecommerce conversion funnel (pageview, signup, order).

This data starts from June 2023 and is valid until mid 2027, ensuring that future users can still use it to run their Profiles projects.

Properties

This dataset includes a subset of the standard properties found in the RudderStack Warehouse Schema spec for each table. The required columns for running Profiles and Predictions projects are also present.

User information

The user data includes a subset of our standard properties for identify calls.

This dataset contains a total of ~10k unique users by anonymousId. About 5.8k of these unique users are known users (with an associated identify call).

Archive: Supported Git URLs

Mon, 01 Jan 0001 00:00:00 +0000

This section is applicable for Profiles CLI project.

To know about the supported Git URLs in the RudderStack dashboard, see the Import from Git guide.

Profiles supports Git URLs for packages in the ssh://git@host:port/path.git format.

For private repos, RudderStack only supports SSH Git URLs. You need to add credentials to the siteconfig.yaml and public SSH key manually to the platforms.

The URL scheme doesn’t depend on individual Git provider host. You can use the below-mentioned Git URLs:

	URL Syntax	Format (Example)
URL for the default branch of a repository	`https://<provider-host>/<org-name>/<repo-name>/path/to/project`	Github: `https://github.com/rudderlabs/librs360-shopify-features/shopify-features` Gitlab: `https://gitlab.com/rudderlabs/librs360-shopify-features/shopify-features` Bitbucket:`https://bitbucket.org/rudderlabs/librs360-shopify-features/shopify-features`
SSH URL for the default branch of a private repository	`git@<provider-host>:<org-name>/<repo-name>/path/to/project`	Github: `git@github.com:rudderlabs/librs360-shopify-features/shopify-features` Gitlab: `git@gitlab.com:rudderlabs/librs360-shopify-features/shopify-features` Bitbucket:`git@gbitbucket.org:rudderlabs/librs360-shopify-features/shopify-features` Azure DevOps: `git@ssh.dev.azure.com:v3/rudderlabs/profiles-project/librs360-shopify-features`
URL for a specific branch of a repository	`https://<provider-host>/<org-name>/<repo-name>/tree/<branch-name>/path/to/project`	Github: `https://github.com/rudderlabs/librs360-shopify-features/tree/main/shopify-features` Gitlab: `https://gitlab.com/rudderlabs/librs360-shopify-features/tree/main/shopify-features` Bitbucket:`https://bitbucket.org/rudderlabs/librs360-shopify-features/tree/main/shopify-features`
URL for a specific tag within the repository	`https://<provider-host>/<org-name>/<repo-name>/tag/<tag-name>/path/to/project`	Github: `https://github.com/rudderlabs/librs360-shopify-features/tag/wht_test/shopify-features` Gitlab: `https://gitlab.com/rudderlabs/librs360-shopify-features/tag/wht_test/shopify-features` Bitbucket:`https://bitbucket.org/rudderlabs/librs360-shopify-features/tag/wht_test/shopify-features`
URL for a specific commit within the repository	`https://<provider-host>/<org-name>/<repo-name>/commit/<commit-hash>/path/to/project`	Github: `https://github.com/rudderlabs/librs360-shopify-features/commit/b8d49/shopify-features` Gitlab: `https://gitlab.com/rudderlabs/librs360-shopify-features/commit/b8d49/shopify-features` Bitbucket:`https://bitbucket.org/rudderlabs/librs360-shopify-features/commit/b8d49/shopify-features`

Archive: Glossary

Mon, 01 Jan 0001 00:00:00 +0000

Cardinality Audit table

A system table that logs all cardinality violations with details on what was filtered and why.

Cardinality rules

A configuration that limits the maximum number of connections between different ID types in an identity graph.

Cardinality violation

Occurs when a node exceeds the maximum allowed connections defined by cardinality rules.

Cohorts

A Cohort refers to a subset of instances of an entity meeting a specified set of characteristics, behaviors, or attributes. Using cohorts, you can define core customer segments and drive targeted marketing campaigns and deliver personalized experiences.

For example, you can define a cohort for users who purchased items in the last 90 days, users based in a specific geographical location, etc.

Edge cardinality

Defines the number of connections a node has to nodes of a specific type.

Edge sources

The edge_sources field provides the input sources for an identity stitching model. You can choose from the input sources defined in the inputs.yaml file and list them under the edge_sources field in the models/profiles.yaml file.

Entity

Entity refers to a digital representation of a class of real world distinct objects for which you can create a profile. An entity can be a user, account, customer, product, or any other object that requires profiling.

Entity features (`entity_var`)

There are various attributes related to an entity whose profile you are trying to create. For example, they can be name, city, LastVisitTimestamp, etc. for the user entity. Each attribute is called an entity_var, and it is derived by performing calculation or aggregation on a set of values. Together, all the attributes create a complete picture of the entity. By default, every entity_var gets stored as a feature, such as days_active, last_seen, etc.

Feature views

If the features/traits of an entity are spread across multiple entity vars and ML models, you can use Feature View to get them together into a single view. These models are usually defined in pb_project.yaml file by creating entries under feature_view key with corresponding entity.

Features

Features are inputs for the machine learning model. In a general sense, features are pieces of user information we already know. For example, number of days they opened the app in the past week, items they left in the cart, etc.

Feature tables (Deprecated)

Feature tables are the outputs based on events, user attributes, and other defined criteria across any data set in your data warehouse. You can define models that can create feature tables for users with ID stitching, ML notebooks and external sources, etc.

Frontier view

A Frontier view is a dynamic view that always points to the output of the latest PB run’s models, ensuring that the queries retrieve the most up-to-date data without altering any downstream dependencies.

ID Stitcher

Data usually comes from different sources and these sources may assign different IDs. To track a user’s journey (or any other entity) uniquely across all these data sources, you need to stitch together all these IDs. ID stitching helps map different IDs of the same user (or any other entity) to a single canonical ID. It does this by doing connected component analysis over the Id-Id edge graph specified in its configuration.

Identity stubs

Identity stubs are nodes that had their connections removed due to cardinality violations.

Input var

Instead of a single value per entity ID, an input var represents a single value per row of an input model. You can think of it as representing addition of an additional column to an input model. It can be used to define entity features. However, it is not itself an entity feature because it does not represent a single value per entity ID.

Inputs

Inputs refers to the input data sources used to create the material (output) tables in the warehouse. The inputs file (models/inputs.yaml) lists the tables/views you use as input sources, including the column name and SQL expression for retrieving the values.

You can use data from various input sources such as event stream (loaded from event data) and any existing tables in the warehouse (generated by external tools).

Label

Label is the output of the machine learning model and is the metric you want to predict. In our case, it is the unknown user trait you want to know in advance.

Machine Learning model

A machine learning model can be thought of as a function that takes in some input parameters and returns an output.

Unlike regular programming, this function is not explicitly defined. Instead, a high level architecture is defined and several pieces are filled by looking at the data. This whole process of filling the gaps is driven by different optimisation algorithms as they try to learn complex patterns in the input data that explain the output.

Materialization

Materialization refers to the process of creating output tables/views in a warehouse by running models. You can define the following fields for materialization:

output_type: Determines the type of output you want to create in your warehouse. Allowed values are:
- table: Output is built and stored in a table in the warehouse.
- view: Output is built and stored as a view in the warehouse.
- ephemeral: Output is created in the form of temporary data which serves as an intermediary stage for being consumed by another model.
run_type: Determines the run type of models. Allowed values are:
- discrete (default): In this mode, the model runs in a full refresh mode, calculating its results from the input sources. A SQL model supports only the discrete run type.
- incremental: In this mode, the model calculates its results from the previous run and only reads row inserts and updates from the input sources. It only updates or inserts data and does not delete anything making it efficient. However, only the identity stitching model supports this mode.

Material tables

When you run the PB models, they produce materials - that is, tables/views in the database that contain the results of that model run. These output tables are known as material tables.

Material registry

This is an internal table created in the output schema within your warehouse and is used as a log by PB. Every model run is recorded in this table.

You will find two records for every model run - the first entry is logged when the model execution starts and the second one is logged when the execution is finished. For example, Table name: material_registry_4;

Predictions

The model’s output is called a prediction. A good model makes predictions that are close to the actual label. You generally need predictions where the labels are not available. In our case, most often the labels come a few days later.

Prediction horizon days

This refers to the number of days in advance when you make the prediction.

For example, statements like “A user is likely to sign-up in the next 30 days, 90 days, etc.” are often time-bound, that is, the predictions are meaningless without the time period.

Profile Builder (PB)

Profile Builder (PB) is a command-line interface (CLI) tool that simplifies data transformation within your warehouse. It generates customer profiles by stitching data together from multiple sources. It takes existing tables or the output of other transformations as input to generate output tables or views based on your specifications.

PB project

A PB project is a collection of interdependent warehouse transformations. These transformations are run over the warehouse to query the data for sample outputs, discover features in the warehouse, and more.

PB model

Any transformation that can be applied to the warehouse data is called a PB model. RudderStack supports various types of models like ID stitching, feature view, Python models, etc.

PyNative model

PyNative models are a set of Profiles models built in Python, and are run by Profiles through a gRPC Python server. Propensity, Attribution etc are some of the existing PyNative models.

Schema versions

Every PB release supports a specific set of project schemas. A project schema determines the correct layout of a PB project, including the exact keys and their values in all project files.

Source URL

This is a column in the material registry and represents the location of the PB project that was run.

The source URL also helps us isolate different projects. If you are running the same project from different locations, RudderStack treats them as different projects.

SQL template models

Sometimes the standard model types provided by Profiles are insufficient to capture complex use cases. In such cases, RudderStack supports the use of SQL template models to explicitly templatize SQL.

SQL template models can be used as an input to an entity-var/ input-var or as an edge-source in id-stitcher.

Timegrain

Using time_grain parameter for a model, you can restrict the context timestamp of that model to the specified time boundary. In other words, a feature v1 with time_grain value of a day will look at all the data up to UTC 00:00 hrs of any particular day only.

If you compute that feature at 3:00PM or 5:00PM, the result would be the same because its inputs change only at the very beginning of the day. Similarly, if time_grain for a model is set to a week, it needs to be run only once a week. Running it twice within the week won’t change its results.

Training

Training refers to the process of a machine learning model looking at the available data and trying to learn a function that explains the labels.

Once you train a model on historic users, you can use it to make predictions for new users. You need to keep retraining the model as you get new data so that the model continues to learn any emerging patterns in the new users.

Archive: Profiles FAQ

Mon, 01 Jan 0001 00:00:00 +0000

This guide contains solutions for some of the commonly asked questions on Profiles. For queries or issues not listed in this guide, contact RudderStack Support.

Setup and installation

I have installed Python3, yet when I install and execute pb, it doesn’t return anything on screen.

Try restarting your Terminal/Shell/PowerShell and try again.

You can also try to find the location of your Python executable. PB would be installed where the executables embedded in other Python packages are installed.

I am an existing user who updated to the new version, and now I am unable to use the PB tool. On Windows, I get the error: 'pb' is not recognized as an internal or external command, operable program or batch file.

Execute the following commands to do a fresh install:

pip3 uninstall profiles-rudderstack-bin
pip3 uninstall profiles-rudderstack
pip3 install profiles-rudderstack --no-cache-dir

I am unable to download profile builder by running pip3 install profiles-rudderstack even though I have Python installed.

Firstly, make sure that Python 3 is installed correctly. You can also try to substitute pip3 with pip and execute the install command.

If that doesn’t work, Python 3 is likely accessible from a local directory.

Navigate to that directory and try the install command again.
After installation, PB should be accessible from anywhere.
Validate that you’re able to access the path using which pb.
You may also execute echo $PATH to view current path settings.
If echo $PATH does not give you the path, then you can find out where Profiles Builder is installed using pip3 show profiles-rudderstack. This command displays a list of the files associated with the application, including the location in which it was installed. You can navigate to that directory.
Navigate to the /bin subdirectory and execute the ls command to confirm that pb is present there.
To add the path of the location where PB is installed via pip3, execute: export PATH=$PATH:<path_to_application>. This will add the path to your system’s PATH variable, making it accessible from any directory. Note that the path should be complete and not relative to the current working directory.

If you still face issues, you can try to install it manually. Contact RudderStack Support for the executable file and download it on your machine. Follow these steps afterwards:

Create rudderstack directory: sudo mkdir /usr/local/rudderstack.
Move the downloaded file to that directory: sudo mv <name_of_downloaded_file> /usr/local/rudderstack/pb.
Grant executable permission to the file: chmod +x /usr/local/rudderstack/pb.
Navigate to the directory /usr/local/rudderstack from your file explorer. Ctrl+Click on pb and select Open to run it from Terminal.
Symlink to a filename pb in /usr/local/bin so that the command can locate it from the env PATH. Create a file if it does not exist: sudo touch /usr/local/bin/pb. Then execute sudo ln -sf /usr/local/rudderstack/pb /usr/local/bin/pb.
Verify the installation by running pb in Terminal. In case you get the error command not found: pb, then check if /usr/local/bin is defined in PATH by executing the command: echo $PATH. If not, then add /usr/local/bin to PATH.

If the Windows firewall prompts you after downloading, proceed with Run Anyway.
Rename the executable as pb.
Move the file to a directory, for example, C:\Program Files\Rudderstack. You can create the directory if required.
Set the path of the pb.exe file in the environment variables.
Verify the installation by running pb in the command prompt.

When I try to install Profile Builder tool using pip3 I get error message saying: Requirement already satisfied

Try the following steps:

Uninstall PB using pip3 uninstall profiles-rudderstack.
Install again using pip3 install profiles-rudderstack.

Note that this won’t remove your existing data such as models and siteconfig files.

I am facing this error while ugrading my Profiles project: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. profiles-pycorelib 0.2.2 requires profiles-rudderstack!=0.10.6,<=0.10.7,>=0.10.5, but you have profiles-rudderstack 0.11.0 which is incompatible.

This is because you must uninstall and then reinstall the pycorelib library while upgrading to a recent version.

Warehouse issues

For my Snowflake warehouse, I have two separate roles to read from input tables and write to output tables? How should I define the roles?

You need to create an additional role as a union of those two roles. PB project needs to read the input tables and write the results back to the warehouse schema.

Furthermore, each run is executed using a single role as specified in the siteconfig.yaml file. Hence, it is best in terms of security to create a new role which has read as well as write access for all the relevant inputs and the output schema.

Can I refer a table present in another database within the warehouse?

Snowflake: You can refer tables from cross-database as long as they are in same warehouse.
Bigquery: You can refer tables from cross-projects.
Databricks: You can refer tables from database in the same or another warehouse.
Redshift: For the following setups:
- Cluster with DC2 Nodes: You cannot use tables from cross-database in the project.
- Cluster with RA3 Nodes/Serverless: You can refer tables from cross-database as long as they are in same warehouse.

While working with Profiles, how can I use the tables in my BigQuery warehouse that are partitioned on the time criteria?

To refer to the partitioned tables in your Profiles project, you must include a filter based on the partitioned column. To do so, add is_event_stream: true and use the partitioned filter as occurred_at_col: timestamp while defining your inputs.yaml file:

- name: rsTracks
 contract:
 is_optional: false
 is_event_stream: true
 with_entity_ids:
 - user
 with_columns:
 - name: timestamp
 - name: user_id
 - name: anonymous_id
 app_defaults:
 table: profiles_new.tracks
 occurred_at_col: timestamp
 ids:
 - select: "user_id"
 type: user_id
 entity: user
 - select: "anonymous_id"
 type: anonymous_id
 entity: user

Compile command

I am trying to execute the compile command by fetching a repo via GIT URL but getting this error: making git new public keys: ssh: no key found

You need to add the OpenSSH private key to your siteconfig.yaml file. If you get the error could not find expected afterwards, try correcting the spacing in your siteconfig.yaml file.

While trying to segregate identity stitching and feature table in separate model files, I am getting this error: mapping values are not allowed in this context

This is due to the spacing issue in siteconfig.yaml file. You may create a new project to compare the spacing. Also, make sure you haven’t missed any keys.

While using v0.13, I notice that two subfolders are created inside the output folder for compile and run, even if I execute only the pb run command. What exactly is the difference between them?

RudderStack generates two subfolders for easy debugging as compiling is a substep of running the project. If you encounter an error during the project run and are not able to get the corresponding SQL generated for this step, you can still rely on the SQL generated during the compile step to debug the error.

I want to build profiles over my Snowflake warehouse data which is pulled in using Salesforce (CRM tool). Is it necessary that the data in my Snowflake warehouse flows via RudderStack? Can I build an entity model for Salesforce users that references the Snowflake table?

RudderStack Profiles lets you use any data present in your warehouse. It does not need to come in via RudderStack. Further, you can define the entities in a pb_project.yaml file and use them declaratively while describing the columns of your input sources.

Command progress & lifecycle

I executed a command and it is taking too long. Is there a way to kill a process on data warehouse?

It could be due to the other queries running simultaneously on your warehouse. For example, for the Snowflake warehouse, open the Queries tab and manually kill the long running processes.

Due to the huge data, I am experiencing long execution times. My screen is getting locked, thereby preventing the process from getting completed. What can I do?

You can use the screen command on UNIX/MacOS to detach your screen and allow the process to run in the background. You can use your terminal for other tasks, thus avoiding screen lockouts and allowing the query to complete successfully.

Here are some examples:

To start a new screen session and execute a process in detached mode: screen -L -dmS profiles_rn_1 pb run. Here:
- -L flag enables logging.
- -dmS starts as a daemon process in detached mode.
- profiles_rn_1 is the process name.
To list all the active screen sessions: screen -ls.
To reattach to a detached screen session: screen -r [PID or screen name].

The CLI was running earlier but it is unable to access the tables now. Does it delete the view and create again?

Yes, every time you run the project, Profiles creates a new materials table and replaces the view.

Hence, you need to grant a select on future views/tables in the respective schema and not just the existing views/tables.

Does the CLI support downloading a git repo using siteconfig before executing pb run ? Or do I have to manually clone the repo first?

You can pass the Git URL as a parameter instead of project’s path, as shown:

pb run -p git@.....

When executing run command, I get a message: Please use sequence number ... to resume this project in future runs . Does it mean that a user can exit using Ctrl+C and later if they give this seq_no then it’ll continue from where it was cancelled earlier?

The pb run --seq_no <> flag allows for the provision of a sequence number to run the project. This flag can either resume an existing project or use the same context to run it again.

With the introduction of time-grain models, multiple sequence numbers can be assigned and used for a single project run.

What flag should I set to force a run for the same input data (till a specified timestamp), even if a previous run exists?

You can execute pb run --force --model_refs models/my_id_stitcher,entity/user/user_var_1,entity/user/user_var_2,...

Can the hash change even if schema version did not change?

Yes, as the hash versions depends on project’s implementation while the schema versions are for the project’s YAML layout.

Is there a way to pick up from a point where my last pb run failed on a subsequent run? For large projects, I don’t want to have to rerun all of the features if something failed as some of these take several hours to run

Yes, you can just execute the run command with the specific sequence number, for example, pb run —seq_no 8.

What is the intent of pb discover models and pb discover materials command?

You can use pb discover models to list all the models from registry and pb discover materials to list all the materials from the registry.

I got this while running pb show models. What is “Maybe Enabled”?

In the show models command, the enable status is computed without looking at tables in the warehouse. Imagine a modelM that has an optional input column. So, M is enabled if and only if the optional input column is present. Hence, it may or may not be enabled, depending on whether the input column is present or not.

How can I handle my Profiles project in the development and production workspace in RudderStack?

Profiles support git branches in the RudderStack dashboard. Refer Supported Git URLs for more information.

In case you wish to run only one project in the CLI and run them differently in dev and prod, you can use targets:

Create a connection using pb init connection and give a connection name (say test). Then, give a default target name, say prod. Enter remaining details.
Create another connection using pb init connection and give the same connection name as before (test). Then, give a different target name, say dev. Enter remaining connection details for connecting to your warehouse.
When you execute a command via CLI, you need to pass -t flag. The first connection you’ve defined is the default one, hence, you don’t need to pass a flag explicitly. However, you can pass it for the other one. For example, pb run -t dev.

Targets aren’t yet supported in the UI. So while you can run the same project on different instances (prod, dev) in the CLI; in the UI you have to make either a different project or a different branch/tag/subfolder.

I am getting an “operation timed out” error even though the pb validate access command worked fine.

Retry the command run after some time. It should resolve the issue.

I have defined a version constraint in my project and migrated it to the latest schema using pb migrate auto command. The project is migrated except the python_requirements key which has the same version constraints. How do I change that?

You need to manually change the project version in CLI as the version constraints don’t change automatically.

How can I generate the .sql output file for my Profiles project containing sql_models.yaml. It is currently not getting generated because the SQL model is not used anywhere.

Even if your SQL model is not used anywhere, you can generate the .sql output file in the warehouse by ensuring that:

Your model’s output_type is not set to ephemeral. Removing the materialization section from the model_spec and observe the outputs.
Your model’s requested_enable_status is not set to only_if_needed.

Identity stitching

There are many large size connected components in my warehouse. To increase the accuracy of stitched data, I want to increase the number of iterations. Is it possible?

The default value of the largest diameter, that is, the longest path length in connected components, is 30.

You can increase it by defining a max_iterations key under model_spec of your ID stitcher model in models/profiles.yaml, and specifying its value as the max diameter of connected components.

Note that the algorithm can give incorrect results in case of large number of iterations.

Do I need to write different query each time for viewing the data of created tables?

No, you can instead use a view name, which always points to the latest created material table. For example, if you’ve defined user_stitching in your models/profiles.yaml file, then execute SELECT * FROM MY_WAREHOUSE.MY_SCHEMA.user_stitching.

I want to use customer_id instead of main_id as the ID type. So I changed the name in pb_project.yaml, however now I am getting this error: Error: validating project sample_attribution: listing models for child source models/: error listing models: error building model domain_profile_id_stitcher: main id type main_id not in project id types.

In addition to making changes in the file pb_project.yaml file, you also need to set main_id_type: customer_id in the models/profiles.yaml file.

If a user account (user_id) is deleted, will the associated user_main_id be deleted as well?

If a user_id is not found in the input sources, it would not be tied to that user_main_id after a full run. However, the user_main_id would still exist if the first node was from an anonymousId for that user.

Suppose a user_main_id has two associated user_ids as they share the same phone number. If one of the user_id changes their phone number, will the user_main_id be updated to include only one of the user_ids? Will a new user_main_id be created for the other user_id?

In this case, as the common node (phone number) is removed, after a full run, the two users would not be associated to the same user_main_id and a new user_main_id would be created for the other user.

I ran identity stitching model but not able to see the output tables under the list of tables in Snowflake. What might be wrong?

In Snowflake, you can check the Databases > Views dropdown from the left sidebar. For example, if your model name is domain_profile_id_stitcher, you should be able to see the table with this name. In case it is still not visible, try changing the role using dropdown menu from the top right section.

I am using a view as an input source but getting an error that the view is not accessible, even though it exists in DB.

Views need to be refreshed from time-to-time. You can try recreating the view in your warehouse and also execute a select * on the same.

What might be the reason for following errors:

processing no result iterator: pq: cannot change number of columns in view. The output view name already exists in some other project. To fix this, try dropping the view or changing its name.
creating Latest View of moldel 'model_name': processing no result iterator: pq: cannot change data type of view column "valid_at" Drop the view domain_profile in your warehouse and execute the command again.
processing no result iterator: pq: column "rudder_id" does not exist. This occurs when you execute a PB project with a model name, having main_id in it, and then you run another project with the same model name but no main_id. To resolve this, try dropping the earlier materials using cleanup materials command.

I have a source table in which email gets stored in the column for user_id, so the field has a mix of different ID types. I have to tie it to another table where email is a separate field. When doing so, I have two separate entries for email, as type email and user_id. What should I do?

You can implement the following line in the inputs tables in question:

 - select: case when lower(user_id) like '%@%' THEN lower(user_id) else null end
 type: email 
 entity: user
 to_default_stitcher: true

How do I validate the results of identity stitching model?

Contact RudderStack Support if you need help in validating the clusters.

Which identifiers would you recommend that I include in the ID stitcher for an ecommerce Profiles project?

We suggest including identifiers that are unique for every user and can be tracked across different platforms and devices. These identifiers might include but not limited to:

Email ID
Phone number
Device ID
Anonymous ID
User names

These identifiers can be specified in the file profiles.yaml file in the identity stitching model.

Remember, the goal of identity stitching is to create a unified user profile by correlating all of the different user identifiers into one canonical identifier, so that all the data related to a particular user or entity can be associated with that user or entity.

If I run --force with an ID Stitcher model and also pass a --seq_no for the most recent run, will it still recreate the full ID Graph? Also, is there a way to know if the model was run incrementally or not?

This will re-run the ID stitcher and if it is incremental, it will look for the most recent run of the stitcher. After finding the existing run for that seq_no, it will use it as the base. This is because the base for an incremental run could be the current seq_no. If you do not want to do this, you can pass the rebase_incremental flag.

What is the difference between the valid_at and first_seen_at columns in the ID graph?

In the context of the ID Stitcher output, the columns valid_at and first_seen_at capture timestamps for each record related to the edges and clusters created during the connected component analysis.

first_seen_at: The earliest timestamp at which a specific identifier (node) was first observed in the source data. It is set once when the node first appears and does not change when the cluster is later modified or expanded. Each node retains its own independent first_seen_at value.
valid_at: Reflects the validity of this node’s current cluster assignment. It is initially derived from the minimum timestamp of the node’s source edges. However, during the stitching process, when clusters merge, valid_at is recalculated for affected nodes. This means valid_at may update for existing nodes in a cluster when new identifiers join, even if those specific nodes had no new activity.

Consider a situation where Node A and Node B are first linked on 2024-01-01. Both valid_at and first_seen_at for this cluster would be 2024-01-01.

Later, Node C connects to this cluster on 2024-02-15:

Node C’s first_seen_at = 2024-02-15 (when Node C was first observed in the source data).
Node C’s valid_at = 2024-02-15 (when its cluster connection was established).
Node A and B’s first_seen_at remains 2024-01-01 (unchanged).
Node A and B’s valid_at may update to 2024-02-15 as a result of the cluster merge during stitching.

In summary:

first_seen_at is a stable, per-node timestamp recording when each identifier was first observed. Use this field when you need a reliable timestamp for individual identifiers (for example, to determine a primary email address).
valid_at reflects the cluster assignment validity and can change when the cluster is modified. It should not be used as a stable per-node timestamp.

I am getting a bunch of NULL VALID_AT timestamps. Is it because the table where the data is being referenced from does not have a timestamp fields specified? Will this impact anything in the downstream?

Yes, if there is no timestamp field in the input table (or it is NULL for the row from where the edge source was pulled), then VALID_AT column would have NULL value. This only affects the VALID_AT column in the final table and nothing in the ID stitching.

Which identifiers should I include in my inputs.yaml file?

Include all the IDs that contribute to the ID stitcher model.

Should I re-run the stitching process once all user_id’s have been sorted out with market prefixes? I want to ensure that users are captured separately instead of being grouped under one rudder_id.

It is recommended to use the --rebase-incremental flag and re-run the stitching process from scratch. While it may not be necessary in all cases, doing so ensures a fresh start and avoids any potential pooling of users under a single rudder_id. It’s important to note that if you make any changes to the YAML configuration, such as modifying the entity or model settings, the model’s hash will automatically update. However, some changes may not be captured automatically (for example, if you didn’t change YAML but simply edited column values in the input table), so manually rebasing is a good practice.

While running my ID stitcher model, I get the error “Could not find parent table for alias “”

This is because RudderStack tries to access the cross-database objects (views/tables) for inputs, which is only supported on Redshift RA3 node type clusters.

To resolve the issue, you can upgrade your cluster to RA3 node type or copy data from source objects to the database specified in the siteconfig file.

I want to use a SQL model for an exclusion filter which references tables that are not used in the ID stitching process. Do I still need to add those tables to the inputs.yaml file?

It is not necessary to add the table references to the inputs.yaml file. However, it is advised to add it for the following reasons:

You can rule out any access/permissions issues for the referenced tables.
The contract field in inputs.yaml would help you handle errors if the required column doesn’t exist.

In my Profiles project, I am making changes to cohorts and var groups and triggering the run one after another. Will the ID stitcher model run every time?

Every time you trigger the project run:

The ID stitcher model runs unless you choose to run only specific models. In that case, only the specified models and their dependencies run unless they already exist.
Models with specified timegrains do not run again within the same time grain.
If you re-use the previous sequence number, the project run will resume the previous invocation and will not recreate the materials that got created previously.

I have provided around 50M ID types as inputs to my ID stitching model but the output is converging to a single user. What might be the reason?

That might happen due to the following reasons:

The identifiers you’re using might not be unique for each user, causing the ID graph to converge into one entity. Make sure that they are unique.
Sometimes, false edges are added to the ID graph if there is any user ID value that should have been ignored. You can exclude these entity ID values by listing them in the id_types declaration in the pb_project.yaml file:

id_types:
 - name: device_id
 filters:
 - type: exclude
 value: "alex@example.com"
 - name: anonymous_id
 filters:
 - type: exclude
 value: "unknown"
 - type: exclude
 value: "NaN"

Is it possible to define two ID stitcher models in a single Profiles project?

Yes, you can define different entities and an ID Stitcher model for each entity in your Profiles project.

How can I fix the issue of identity explosion, that is, too many connections between identities?

You can use ID Graph Cardinality Rules to limit connections between ID types.

Feature Table (deprecated)

How can I run a feature table without running its dependencies?

Suppose you want to re-run the user entity_var days_active and the rsTracks input_var last_seen for a previous run with seq_no 18.

Then, execute the following command:

pb run --force --model_refs entity/user/days_active,inputs/rsTracks/last_seen --seq_no 18

I have imported a library project but it throws an error: no matching model found for modelRef rsTracks in source inputs.

You can exclude the missing inputs of the library project by mapping them to nil in the pb_project.yaml file.

Can I run models which consider the input data within a specified time period?

Yes, you can do so by using the begin_time and end_time parameters with the run command. For example, if you want to run the models for data from 2nd February, 2023, use:

$ pb run --begin_time 2023-01-02T12:00:00.0Z

If you want to run the nmodels for data between 2 May 2022 and 30 April 2023, use:

$ pb run --begin_time 2022-05-01T12:00:00.0Z --end_time 2023-04-30T12:00:00.0Z

If you want to run the models incrementally (run them from scratch ignoring any previous materials) irrespective of timestamp, use:

$ pb run --rebase_incremental

Is it possible to run the feature table model independently, or does it require running alongside the ID stitcher model?

You can provide a specific timestamp while running the project, instead of using the default latest time. PB recognizes if you have previously executed an identity stitching model for that time and reuses that table instead of generating it again.

You can execute a command similar to: pb run --begin_time 2023-06-02T12:00:00.0Z --end_time 2023-06-03T12:00:00.0Z. Note that:

To reuse a specific identity stitching model, the timestamp value must match exactly to when it was run.
If you have executed identity stitching model in the incremental mode and do not have an exact timestamp for reusing it, you can select any timestamp greater than a non-deleted run. This is because subsequent stitching takes less time.
To perform another identity stitching using PB, pick a timestamp (for example, 1681542000) and stick to it while running the feature table model. For example, the first time you execute pb run --begin_time 2023-06-02T12:00:00.0Z --end_time 2023-06-03T12:00:00.0Z, it will run the identity stitching model along with the feature models. However, in subsequent runs, it will reuse the identity stitching model and only run the feature table models.

While trying to add a feature table, I get an error at line 501, but I do not have these many lines in my YAML.

The line number refers to the generated SQL file in the output folder. Check the console for the exact file name with the sequence number in the path.

While creating a feature table, I get this error: Material needs to be created but could not be: processing no result iterator: 001104 (42601): Uncaught exception of type 'STATEMENT ERROR': 'SYS _W. FIRSTNAME' in select clause is neither an aggregate nor in the group by clause.

This error occurs when you use a window function any_value that requires a window frame clause. For example:

 - entity_var:
 name: email
 select: LAST_VALUE(email)
 from: inputs/rsIdentifies
 window:
 order_by:
 - timestamp desc

Is it possible to create a feature out of an identifier? For example, I have a RS user_main_id with two of user_ids stitched to it. Only one of the user_ids has a purchase under it. Is it possible to show that user_id in the feature table for this particular user_main_id?

If you know which input/warehouse table served as the source for that particular ID type, then you can create features from any input and also apply a WHERE clause within the entity_var.

For example, you can create an aggregate array of user_id’s from the purchase history table, where total_price > 0 (exclude refunds, for example). Or, if you have some LTV table with user_id’s, you could exclude LTV < 0.

Is it possible to reference an input var in another input var?

Yes - input vars are similar to adding additional columns to the original table. You can use an input var i1v1 in the definition of input var i1v2 as long as both input vars are defined in the same input (or SQL model) i1.

I have not defined any input vars on I1. Why is the system still creating I1_var_table?

When you define an entity var using I1, an internal input var (for entity’s main_id) is created which creates I1_var_table. RudderStack team is evaluating whether internal input vars should create the var table or not.

I have an input model I1. Why is the system creating Material_I1_var_table_XXXXXX_N?

This material table is created to keep the input vars defined on I1.

I am trying to run a single entity_var model. How should I reference it?

The right way to reference an entity var is: entity/<entity-name>/<entity-var-name>.

I have two identical named fields in two user tables and I want my Profiles project to pick the most recently updated one (from either of the user tables). What is the best way to do it?

Define different entity_vars (one for each input) and then pick the one with a non-null value and higher priority.

What does running material mean?

It means that the output (material) table is being created in your warehouse. For example, an output table named material_user_id_stitcher_3acd249d_21 would mean:

material: Prefix for all the objects created by Profiles in your warehouse, such as ID stitcher and feature tables.
user_id_stitcher: View created in your schema. It will always point to latest ID stitcher table. This name is the same as defined in the models/profiles.yaml file.
3acd249d: Unique hash which remains the same for every model unless you make any changes to the model’s config, inputs or the config of model’s inputs.
21: Sequence number for the run. It is a proxy for the context timestamp. Context timestamp is used to checkpoint input data. Any input row with occured_at timestamp value greater than the context timestamp cannot be used in the associated run.

YAML

Are there any best practices I should follow when writing the PB project’s YAML files?

Yes, refer to the YAML best practices guide for detailed information.

Can I use double quotes when referencing another entity_var in a macro?

You can use an escape character. For example:

 - entity_var:
 name: days_since_last_seen
 select: "{{macro_datediff('{{user.max_timestamp_bw_tracks_pages}}')}}"

Also, if have a case statement, then you can add something like the following:

select: CASE WHEN {{user.max_timestamp_tracks}}>={{user.max_timestamp_pages}} THEN {{user.max_timestamp_tracks}} ELSE {{user.max_timestamp_pages}} END

Is it possible to define default arguments in macros?

No, RudderStack does not support default arguments in macros.

Propensity Models

Why is my Profiles project taking so long to run?

The first Profiles project run usually takes longer, especially if you are building predictive features. Training ML models requires generating multiple snapshots of historical data, which significantly increases the initial processing time.

The computational workload varies between different phases:

The training phase is computationally intensive as it processes historical data snapshots.
The prediction phase is relatively faster once the model is trained.

Your project’s runtime will typically see periodic spikes coinciding with model refreshes, which occur based on the training validity time configured in your propensity model. Between these refreshes, routine runs should complete more quickly.

Activation API

While using Redis destination, I am facing an error: These sample records were rejected by the destination?

This error is observed if you have enabled Cluster mode setting for Redis in the RudderStack’s configuration settings but you are on the Redis free plan.

To overcome this, ensure that the Redis plan you are using allows clustering. Alternatively, you can turn off the Cluster mode setting.

Does the user-profiles API (old) and activation API (new) behave differently in updating a key that maps to two different primary keys? For example:

Primary key	user_id	Feature_1	Feature_2
PK1	U1	F1	null
PK2	U1	null	F2

Yes, in user profiles API, RudderStack updates the value for a specific key (that is, feature_1 in this case):

{
 "userId": "U1",
 "profile": {
 "feature_1": "F1",
 "feature_2": "F2"
 }
}

Whereas, in activation API, RudderStack syncs the entire row as value for the model_name key:

{
 "entity": "entity_name",
 "id": {
 "type": "user_id",
 "value": "U1"
 },
 "data": {
 "model_name": {
 "feature_1": null,
 "feature_2": F2
 }
 }
}

Is it possible to use the Activation API without any Profiles project?

Unfortunately, no! The Activation API works only with a Profiles project.

I have toggled on the Activation API option in the RudderStack dashboard to generate a Reverse ETL pipeline (connected to the Redis destination) and have defined a single ID in the feature_views key. However, two Reverse ETL pipelines are generated on running the project. Which one should I use and what is the difference between the two?

Profiles generates two feature_views models if you define a single ID under the feature_views key. One is the default feature view with main_id as the identifier and the other is based on the identifier you have defined.

RudderStack assigns the default names to the view such as user_feature_view (default one with main_id as the identifier), or feature_view_with_email (email as the identifier), etc. You can also specify the final view’s name in the name key.

Profiles UI

I have included some features in the RudderStack dashboard while creating the Profiles project but when I click “download this project”, my project files does not include any feature. What might be the reason?

If you have selected pre-defined features from any library project, they are referred to as profiles-multieventstream-features in the project by default.

If you have created any features using the custom feature functionality, they will be a part of your models/resources.yaml file.

Can I use main_id when searching for a profile in the Profiles Lookup tab?

Yes. In the Profile Lookup tab, you can search using any available unique identifier such as email, phone number, user id, anonymous id, or the entity’s main_id (for example, user_main_id). Enter the value and click Search user profile to view the stitched profile.

While choosing pre-defined features in the RudderStack dashboard, I can preview code for only some of the features. What might be the reason?

You can preview the code only for entity var based features. This functionality is not available for features built from ML and SQL models.

While creating a Profiles project by importing from Git, I dont see any warehouse options in the dropdown selector in the Validate Profiles project section. What might be the reason?

A Profiles project looks for the supported warehouse destinations configured for that workspace. Hence, make sure you have configured any of the following warehouse destinations in your RudderStack dashboard:

Snowflake
Databricks
Redshift
BigQuery

Why am I not able to see the Concurrency option in the Settings tab of my Profiles project?

RudderStack supports the Concurrency option only for the Snowflake warehouse currently. You will not be able to see this option if you have configured your Profiles project using the Redshift, BigQuery, or Databricks warehouse.

I have chosen some pre-defined predictive features while creating a Profiles project in the RudderStack dashboard but my project fails on running. What might be the reason?

One of the probable reasons could be the lack of adequate data in your input source. Try following the steps suggested in the error message. In case the issue still persists, contact our support team.

I am creating an activation in the RudderStack dashboard but do not see the ID field being populated. What might be the reason?

To populate the IDs in the dropdown, make sure that your Profiles project has feature_views defined in the pb_project.yaml file under the entity.

Can I specify any git account like CommitCode while configuring a project in the web app?

Profiles UI supports repos hosted on GitHub, BitBucket, Azure DevOps, and GitLab.

How do I resolve the error `rpc init: timeout waiting for python client to initialize` when running the PyNative models (propensity, audit, tutorial)?

PyNative models are written in Python where the code is first compiled into bytecode and then executed by the Python interpreter.

Sometimes, compiling Python code may take more time than expected. You can keep retrying the commands if they do not succeed the first time - the error will resolve on its own.

RudderStack highly recommends using an isolated and clean Python virtual environment.

If the issues persist, you can enforce pb to use the Python present in the virtual environment by passing its path as the value for the parameter --rpc_python_path.

Make sure to pass the absolute path and not the relative path. For example:

pb tutorial --connection <connection_name> --rpc_python_path=/Users/<name>/.env/bin/python3.10

You can use the which python or which python3 command to get the Python path.

Miscellaneous

Why am I getting Authentication FAILED error on my data warehouse while executing the run/compile commands?

Some possible reasons for this error might be:

Incorrect warehouse credentials.
Insufficient user permissions to read and write data. You can ask your administrator to change your role or grant these privileges.

Why am I getting Object does not exist or not authorized error on running this SQL query: SELECT * FROM "MY_WAREHOUSE"."MY_SCHEMA"."Material_domain_profile_c0635987_6"?

You must remove double quotes from your warehouse and schema names before running the query, that is SELECT * FROM MY_WAREHOUSE.MY_SCHEMA.Material_domain_profile_c0635987_6.

Is there a way to obtain the timestamp of any material table?

Yes, you can use the GetTimeFilteringColSQL() method to get the timestamp column of any material. It filters out rows based on the timestamp. It returns the occurred_at_col in case of an event_stream table or valid_at in case the material has that column. In absense of both, it returns an empty string. For example:

 SELECT * FROM {<from_material>}
 WHERE
 <from_material>.GetTimeFilteringColSQL() > <some_timestamp>;

What is the difference between setting up Profiles in the RudderStack dashboard and Profile Builder CLI tool?

You can run Profiles in the RudderStack dashboard or via Profile Builder CLI.

The main difference is that the RudderStack dashboard only generates outputs based on the pre-defined templates. However, you can augment those outputs by downloading the config file and updating it manually.

On the other hand, the CLI tool lets you achieve the end-to-end flow via creating a Profile Builder project.

Does the Profiles tool have logging enabled by default for security and compliance purposes?

Logging is enabled by default for nearly all the commands executed by CLI (init, validate access, compile, run, cleanup, etc.). Logs for all the output shown on screen are stored in the file logfile.log in the logs directory of your project folder. This includes logs for both successful and failed runs. RudderStack appends new entries at the end of the file once a command is executed.

Some exceptions where the logs are not stored are:

query: The logger file stores the printing output and does not store the actual database output. However, you can access the SQL queries logs in your warehouse.
help: For any command.

How can I remove the material tables that are no longer needed?

To clean up all the materials older than a specific duration, for example 10 days, execute the following command:

pb cleanup materials -r 10

The minimum value you can set here is 1. So if you have run the ID stitcher today, then you can remove all the older materials using pb cleanup materials -r 1.

Which tables and views are important in Profiles schema that should not be deleted?

material_registry
material_registry_<number>
pb_ct_version
ptr_to_latest_seqno_cache
wht_seq_no
wht_seq_no_<number>
Views whose names match your models in the YAML files.
Material tables from the latest run (you may use the pb cleanup materials command to delete materials older than a specific duration).

I executed the auto migrate command and now I see a bunch of nested original_project_folder. Are we migrating through each different version of the tool?

This is a symlink to the original project. Click on it in the Finder (Mac) to open the original project folder.

I am getting a ssh: handshake failed error when referring to a public project hosted on GitHub. It throws error for https:// path and works fine for ssh: path. I have set up token in GitHub and added to siteconfig.yaml file but I still get this error.

You need to follow a different format for gitcreds: in siteconfig. See SiteConfiguration for the format.

After changing siteconfig, if you still get an error, then clear the WhtGitCache folder inside the directory having the siteconfig file.

If I add filters to id_types in the project file, then do all rows that include any of those values get filtered out of the analysis, or is it just the specific value of that id type that gets filered?

The PB tool does not extract rows. Instead, it extracts pairs from rows.

So if you had a row with email, user_id, and anonymous_id and the anonymous_id is excluded, then the PB tool still extracts the email, user_id edge from the row.

In the material registry table, what does status: 2 mean?

status: 2 means that the material has successfully completed its run.
status: 1 means that the material did not complete its run.

I am using Windows and get the following error: Error: while trying to migrate project: applying migrations: symlink <path>: A required privilege is not held by the client.

Your user requires privileges to create a symlink. You may either grant extra privileges to the user or try with a user containing Admin privileges on PowerShell. In case that doesn’t help, try to install and use it via WSL (Widows subsystem for Linux).

If I want to run multiple select models, how can I run something like: pb run --model_refs "models/ewc_user_id_graph_all, models/ewc_user_id_graph, models/ewc_user_id_graph_v2

You can do so by passing --model_refs multiple times per model:

pb run -p samples/test_feature_table --model_refs 'models/test_id__, user/all' --migrate_on_load OR pb run -p samples/test_feature_table --model_refs models/test_id__ --model_refs user/all --migrate_on_load

How can I keep my Profiles projects up to date along with updating the Python package and migrating the schema version?

You can check for the latest Profiles updates in the changelog.

To update the Python package and migrate the schema version, you can standardise on a single pip release across the org and use the schema version that is native to that binary. When you move to a different binary, migrate your projects to the schema version native to it.

Contact Profiles support team in our Community Slack for specific questions.

I am facing this error on adding a custom ID visitor_id under the id_types field in the pb_project.yaml file: could not create project: failed to read project yaml Error: validating project sample_attribution: getting models for folder: user: error listing models: error building model user_default_id_stitcher: id type visitor_id not in project id types. What might be the reason?

While adding a custom ID type, you must extend the package to include its specification in the pb_project.yaml file as well. In this case, add the key extends: followed by name of the same/different id_type that you wish to extend, and corresponding filters with include/exclude values like below:

id_types:
 - name: visitor_id
 extends: visitor_id
 filters:
 - type: exclude
 value: "someexcludedvalue"

Can I keep multiple projects in a Git Repo?

Yes, you can create multiple folders in your project repo and keep different projects in each folder. While running the project, you can use any suitable URL to run a specific project:

https://github.com/<org-name>/<repo-name>/tree/<branch-name>/path/to/project https://github.com/<org-name>/<repo-name>/tag/<tag-name>/path/to/project https://github.com/<org-name>/<repo-name>/commit/<commit-hash>/path/to/project

See Supported Git URLs for more information.

Can a models folder contain subfolders?

Yes, you can manually add subfolders to the models folder and reference their path in the pb_project.yaml file:

model_folders:
 - models/inputs
 - models/inputs/web.yaml

A sample folder structure is shown:

.
├── models/
│ ├── inputs/
| │ ├── web.yml
| │ ├── mobile.yml
| │ └── server.yml
│ └── ...

How is Activations different from Audiences?

Activations qualify as Audiences with a minor exception of having a Profiles project as a source instead of a Reverse ETL source (with schema, database, table etc).

What do the sequence numbers under Artifacts in the History tab mean?

A unique sequence number is generated every time you run a Profiles project. Each project run generates one sequence number. However, note that defining timegrains can result in multiple sequence numbers (based on the timegrain).

I am running a Profiles project with the timegrains parameter and noticed that multiple subfolders having different seq_no are generated. Which seq_no should I use to resume an earlier run?

For the CLI project, you can resume the project run using CLI commands(like run, compile, etc.) and passing the --seq_no displayed at the top of the terminal output. For the UI project, you cannot choose to stop/resume the project run.

How do I identify the most recently-generated tables for a particular Profiles project?

Check the material_registry_4 table in your schema. The seq_no column increments with each run - if you are not using the timegrains feature, then for each run there will be only one seq_no.

Note that there will be multiple rows in this table corresponding to each model for each seq_no.

What is the purpose of the PTR_TO_LATEST_SEQNO_CACHE view in a Profiles schema?

The PTR_TO_LATEST_SEQNO_CACHE view scans the registry to build metadata for finding the latest table or view of any model name or model hash. This metadata then goes into the creation of views named model - for each model named model - and points to the latest Material_model_xxxxxxxx_n after every run.

I have multiple models in my project. Can I run only a single model?

Yes, you can. In your spec YAML file for the model you don’t want to run, set materialization to disabled:

materialization:
 enable_status: disabled

A sample profiles.yaml file highlighted a disabled model:

models:
- name: test_sql
 model_type: sql_template
 model_spec:
 materialization:
 run_type: discrete
 enable_status: disabled // Disables running the model.
 single_sql: |
 {%- with input1 = this.DeRef("inputs/tbl_a") -%}
 select id1 as new_id1, {{input1}}.*
 from {{input1}}
 {%- endwith -%}
 occurred_at_col: insert_ts
 ids:
 - select: "new_id1"
 type: test_id
 entity: user

Why am I getting the error could not build project from package corelib: getting ProjectFolder: getting tags list: repository not found?

You might get this error due to changes in your Git credentials. Try resetting them in your system configuration and try again.

What factors should I consider while defining the edge sources for my user ID graph?

The ID graph needs only a minimum set of inputs for capturing all the identifiers. For example, you might not need to include the tracks table to just capture the client side track calls. As the anonymous_ids, user_ids, and emails are probably captured in the pages and identifies tables.

Why is my Profiles project run failing since I enabled SAML SSO on GitHub?

My Profiles project run is failing since I enabled SAML SSO on GitHub. What might be the reason?

After enabling SSO on GitHub, you must edit your Profiles project and deploy the updated SSH key. Also, make sure you are logged in to GitHub using SSO while doing that.

RudderStack Docs – Additional Resources

Archive: Profiles Run Process

Project assumptions

Run process

Archive: Python Multi-version Support

Archive: Snowflake Sample Data

Tables

Properties

User information

Archive: Supported Git URLs

Archive: Glossary

Cardinality Audit table

Cardinality rules

Cardinality violation

Cohorts

Edge cardinality

Edge sources

Entity

Entity features (entity_var)

Feature views

Features

Feature tables (Deprecated)

Frontier view

ID Stitcher

Identity stubs

Input var

Inputs

Label

Machine Learning model

Materialization

Material tables

Material registry

Predictions

Prediction horizon days

Profile Builder (PB)

PB project

PB model

PyNative model

Schema versions

Source URL

SQL template models

Timegrain

Training

Archive: Profiles FAQ

Setup and installation

Warehouse issues

Compile command

Command progress & lifecycle

Identity stitching

Feature Table (deprecated)

YAML

Propensity Models

Activation API

Profiles UI

Python-related errors

How do I resolve the error rpc init: timeout waiting for python client to initialize when running the PyNative models (propensity, audit, tutorial)?

How do I resolve the Python package-related issues when trying out PyNative models?

Miscellaneous

Entity features (`entity_var`)

How do I resolve the error `rpc init: timeout waiting for python client to initialize` when running the PyNative models (propensity, audit, tutorial)?