RudderStack Docs – 0.25

Archive: Profiles Overview

Mon, 01 Jan 0001 00:00:00 +0000

Modern data teams rely on their warehouse as a single source of truth for customer data. RudderStack’s Profiles is a declarative framework that transforms configuration into optimized SQL — letting you define what a customer looks like (identity rules, features, cohorts) rather than how to compute it.

Unlike traditional semantic layers that abstract SQL, Profiles inverts the paradigm: you declare entities and their attributes in version-controlled YAML, and the system generates warehouse-native SQL that handles identity resolution, incremental computation, and feature aggregation automatically.

With Profiles, data teams can stop maintaining thousands of lines of brittle SQL and start maintaining clean, reviewable configurations that compile to a comprehensive customer 360 table.

Check the comprehensive 12-part video series on RudderStack Profiles to build an end-to-end Profiles project that helps you:

Solve identity resolution at the root

Create an identity graph with all your customer data

Model powerful Customer 360 views in your data warehouse

Why use Profiles?

Data teams typically choose between two painful paths: maintaining sprawling SQL codebases with complex dependencies, or locking into rigid SaaS platforms that can’t model their unique business logic.

Profiles offers a third way — declarative customer data modeling:

Traditional approach	Profiles approach
Write complex SQL joins across identity tables	Declare identity types and stitching rules in YAML
Build and maintain aggregation pipelines	Define features as `select` expressions; system handles grouping
Manage incremental logic manually	Declare once; incremental computation is automatic
Debug SQL across dozens of files	Review diffs in version-controlled configuration

This declarative DNA means your team defines the semantics — what constitutes a user, what “lifetime value” means for your business — while Profiles handles the mechanics of Identity Graphs, table scans, and efficient computation.

Move faster with an end-to-end platform

Profiles integrates directly with RudderStack’s other pipelines:

Event Stream pipelines have known schemas and unique identifiers through the ingested customer data. Profiles can produce a baseline identity graph, and user features out of the box. Data teams can then augment the graph and features using any other data in their warehouse.
Reverse ETL pipelines make it easy to send data from the customer 360 table directly to the downstream tools used by marketing, customer success, product, and other teams.

Enrich user profiles with features

You can enhance user profiles with additional data points and features. When new data sources are added, discovered, or calculated, data teams can add them to their Profiles configuration without having to clean data and update complex models and dependencies.

The features/traits can include demographic information, preferences, purchase history, browsing behavior, or other static or computed data points.

You can define the associated properties or attributes that provide detailed information for each new feature. For example, if the feature is purchase_history, its properties can include the date of purchase, product category, or order value.

Unlock deeper insights

Profiles extends its capabilities to support features derived from complex concepts such as funnels, organizational metrics, and machine learning models. You can understand your customer’s journey through your sales funnel or locate each user across the histogram of customer metric values by simply defining a trait.

Deliver personalization and recommendations

Using Profiles, you can ship projects like personalization significantly faster by focusing entirely on activating key user features instead of cleaning and modeling data to build them.

Predict user conversions and churn

You can leverage Propensity Scores to build predictive features that help you predict in advance whether a lead is likely to convert, or a customer is likely to churn or make a purchase.

Who can leverage Profiles?

Profiles is built for data engineers, data scientists, and technical marketers who want configuration over code.

You define identity stitching rules, entity relationships, and user features in YAML—no hand-written SQL for identity resolution or feature computation. The declarative model means:

Reviewable changes: Pull request diffs show what changed in your customer model, not SQL syntax changes buried in logic.
Composable features: Entity vars reference each other naturally; the system resolves dependencies.
Portable definitions: Your customer model is decoupled from warehouse-specific SQL dialects.

Archive: Concepts

Mon, 01 Jan 0001 00:00:00 +0000

Archive: Developer Docs ↗️

Mon, 01 Jan 0001 00:00:00 +0000

Archive: Management, Scheduling, and Data Activation

Mon, 01 Jan 0001 00:00:00 +0000

Archive: RudderStack Data Apps

Mon, 01 Jan 0001 00:00:00 +0000

RudderStack’s Data Apps help data teams accelerate key projects like attribution, propensity scoring, and real-time personalization.

You can leverage the power of your Customer 360 without the need for complex SQL or Python coding and enable your data teams to deliver impactful results quickly.

Use cases

Use case	Description
Attribution	Deliver accurate, report-ready first and last-touch attribution data for paid campaigns, including RoAS.
Propensity scores	Generate propensity scores using your own churn or conversion definitions, then automatically sync them to user profiles in business tools.
Real-time personalization	Make your customer 360 available via API to drive real-time website and app personalization, enhancing user engagement, and conversion rates.

Archive: Additional Resources

Mon, 01 Jan 0001 00:00:00 +0000

Archive: Profiles Changelog

Mon, 30 Mar 2026 00:00:00 +0000

Version 0.26.0

23 July 2026
Schema version: 99

Release Schedule

Milestone	Date
CLI release	July 23, 2026
Deployment to dev workspaces	July 23, 2026
Deployment to production workspaces	July 28, 2026

What’s New

Rule-based filtering (ID Graph Cardinality Rules) is now supported in ID Stitcher for BigQuery, in addition to Snowflake
ID Stitcher metadata logging (log_direct_edge_info and log_node_metadata) is now supported for BigQuery, in addition to Snowflake. See ID Stitcher
Per-source edge attribution: Each bucket in the edges_by_type metadata column now carries a sources[] array listing the source models that contributed the edge. See Identity Graph
The <material>_<entity>_CARDINALITY_AUDIT table includes a new source_models column, so you can trace a capped or dropped edge back to the source models that produced it
Incremental entity var bundling: Support to compute incremental entity vars in a bundle is now available.
Profiles writes user-facing output tables to a schema separate from its internal working tables.
The pb CLI supports shell autocompletion

Miscellaneous

BigQuery queries now enforce GoogleSQL, with normalized JSON and time expressions
S3 configuration resolution and the Redshift S3 write path support IAM role authentication, in addition to key-based authentication
The pb version is included in the task summary
Snowflake OCSP cache miss errors can be suppressed using a PB environment toggle

Bug Fixes

Rebase-affected default materials are now reused at compile time
rebase_incremental is no longer silently skipped in already-materialized timegrain buckets
Material cleanup now tolerates dependency cycles instead of failing
Connection leaks and broken pipe errors caused by unbounded connection pools are fixed
Query tags work again on Snowflake

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross-database references can fail on Redshift for a few clusters.
While creating activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs — you can ignore it:

WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes.

To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null

pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get the ssh: handshake failed error, then you will have to manually remove the entire folder from WhtGitCache to make it work.

Version 0.25.7

15 June 2026

Bug Fixes

Projects that use case_based_join optimization no longer fail with an ambiguous column name error when an input’s id select reuses an id-stitcher column under a non-main_id type; the var-table → id-stitcher join keys are now properly qualified.

Version 0.25.6

4 June 2026

Bug Fixes

To avoid out-of-memory issues on BigQuery, var-table bundles for inputs with no identifier column now generate input_row_id via the native GENERATE_UUID() instead of an unpartitioned ROW_NUMBER() OVER() window.

Version 0.25.5

29 Apr 2026

Bug Fixes

Fixed a bug where features with non-default time grains were not getting included in the feature view.

Version 0.25.3

16 Apr 2026

Bug Fixes

Fixed a bug to prevent an occurance of empty entity var bundle.

Version 0.25.2

15 Apr 2026

Miscellaneous

Suppressed unnecessary warning logs.

Version 0.25.1

31 March 2026
Schema version: 97

Release Schedule

Milestone	Date
CLI release	March 31, 2026
Deployment to dev workspaces	March 31, 2026
Deployment to production workspaces	March 31, 2026

What’s New

Incremental features: You can compute features incrementally. See Incremental Features for more details.
You can now materialize ID stitcher models as tables instead of views. Configure via materialization: output_type: table on your ID stitcher model for better downstream query performance.
Dry-Run Mode: New --mock_material_run flag (used with --assume_all_inputs_exist) lets you simulate a full pb run without executing any SQL against the warehouse. This is useful for validating project definitions, debugging model graphs, and testing configurations without live warehouse access.
You no longer need to write {{entity.Var("var_name")}} — you can now simply use {{entity.var_name}} with dot syntax to access entity vars, input vars, and columns in SQL templates.

Miscellaneous

Better error messages for incomplete incremental yaml configuration
BigQuery legacy SQL queries have been migrated to GoogleSQL (standard SQL)
Python models are removed

Bug Fixes

GRPC connection issues and file path issues encountered on Windows are now fixed

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross-database references can fail on Redshift for a few clusters.
While creating activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs — you can ignore it:

WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes.

To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null

pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get the ssh: handshake failed error, then you will have to manually remove the entire folder from WhtGitCache to make it work.

Version 0.25.0 (Beta)

10 December 2025
Schema version: 96

Release Schedule

Milestone	Date
CLI release	December 10, 2025
Deployment to dev workspaces	December 15, 2025
Deployment to production workspaces	December 17, 2025

A major re-architecture is introduced in this beta release. The core engine has been rebuilt from the ground up to support a powerful new capability — accessing and instantiating any past version of any model in your project.

Why this matters: This foundational change is essential for enabling true incremental computation. By tracking model versions over time, Profiles Builder (PB) can now intelligently determine what needs to be recomputed, dramatically improving performance and reducing costs for large-scale projects.

🔒 Safety First:

As this is a beta release with significant architectural changes, you must explicitly opt-in to use it. Set allow_beta: true in your pb_project.yaml or use the CLI beta flag.

What’s New

Building Incremental Features is now possible.
Added explicit consent requirement for beta releases: Users must now opt-in to run PB commands with a beta release, either by passing a CLI argument (--allow_beta) or by setting allow_beta: true in your pb_project.yaml file.

Bug Fixes

Fixed a nil pointer exception in some cases while loading the projects
Fixed an issue with Redshift during runs, where internal table creation fails if it already exists

Miscellaneous

Major internal rearchitecture of PB to enable incremental support
Propensity model’s internal training and prediction table names changed
Model names and paths have changed for cohort and feature views. However, there will be no change in frontier view names to avoid impact on downstream pipelines.
Removed unnecessary warning logs while loading projects
Shows the right error message that explains why a project failed instead of showing context cancelled as the message

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross-database references can fail on Redshift for a few clusters.
While creating activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs — you can ignore it:

WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes.

To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null

pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get the ssh: handshake failed error, then you will have to manually remove the entire folder from WhtGitCache to make it work.

Version 0.24.6

2 February 2026

Bug Fixes

Fixed wrong model reference path issue faced in Windows OS

Miscellaneous

Try both IPV4 and IPV6 addresses for GRPC connectivity between internal processes

Version 0.24.4

15 January 2026

Miscellaneous

Added retry functionality to retryable errors thrown by BigQuery

Version 0.24.3

8 December 2025

Miscellaneous

Added timeout to queries run on cleanup command

Version 0.24.2

21 November 2025

Bug Fixes

Fixed a nil pointer exception in some cases while loading the projects
Fixed an issue which occurs in Redshift during runs, where internal table creation fails if it already exists

Miscellaneous

Removed unnecessary warning logs while loading projects
Shows the right error message that explains why a project failed instead of showing context cancelled as the message

Version 0.24.0

4 November 2025
Schema version: 93

Release Schedule

Milestone	Date
CLI release	November 4, 2025
Deployment to dev workspaces	November 4, 2025
Deployment to production workspaces	November 6, 2025

What’s New

Rule-based filtering (ID Graph Cardinality Rules) is now supported in ID Stitcher for Snowflake
pb init pb-project generates project with optimizations configuration file by default

Bug Fixes

IPv4 addressing is used for Go and Python RPC servers

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross-database references can fail on Redshift for a few clusters.
While creating activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs — you can ignore it:

WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes.

To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null

pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get the ssh: handshake failed error, then you will have to manually remove the entire folder from WhtGitCache to make it work.

Version 0.23.3

2 Oct 2025

Bug Fixes

Fixed an issue where enabling ID stitcher metadata resulted in a SQL error.

Version 0.23.2

18 Sep 2025

Bug Fixes

Fixed an issue where referencing models in a case-insensitive manner caused errors.
Resolved a regression affecting PyNative package registration.

Version 0.23.1

9 September 2025
Schema version: 92

Release Schedule

Milestone	Date
CLI release	September 9, 2025
Deployment to dev workspaces	September 10, 2025
Deployment to production workspaces	September 15, 2025

What’s New

Added metadata in ID stitcher internal mapping table — this metadata helps you understand how identifiers are connected and trace the lineage of your identity graph. To enable, toggle on log_direct_edge_info and log_node_metadata flags under id stitcher model spec. An example is shown below:

models:
 - name: user_id_stitcher
 model_type: id_stitcher
 model_spec:
 entity_key: user
 log_direct_edge_info: true
 log_node_metadata: true

pb show models --tree prints output in a tree format.
Added support for hooks on feature-views. An example is shown below:

entities:
 - name: user
 id_stitcher: models/test_id__
 feature_views:
 hooks:
 pre_run: "CREATE OR REPLACE VIEW {{warehouse.NamedWhObject(name='x1', type='VIEW')}} AS (SELECT 1 as testCol);"
 post_run: "CREATE OR REPLACE VIEW {{warehouse.NamedWhObject(name='x2', type='VIEW')}} AS (SELECT 1 as testCol);"

Removed ID collator as a model type.
PB logs are now redirected to stderr. pb show models --json > output.json output is a valid JSON file now.

Bug Fixes

Fixed the bug wherein a non-converged ID graph (after maximum iterations) resulted in an error.

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross-database references can fail on Redshift for a few clusters.
While creating activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs — you can ignore it:

WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes.

To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null

pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get the ssh: handshake failed error, then you will have to manually remove the entire folder from WhtGitCache to make it work.

Version 0.22.4

18 Aug 2025

Bug Fixes

Fixed a Databricks OAuth authentication issue affecting PyNative models such as Audit, Propensity, etc.

Version 0.22.2

14 May 2025

Miscellaneous

Maintenance release: Updated dependencies to the latest compatible versions.

Version 0.22.1

6 May 2025

Bug Fixes

Fixed a pb audit issue.

Version 0.22.0

24 April 2025
Schema version: 89

Release Schedule

Milestone	Date
CLI release	April 24, 2025
Deployment to dev workspaces	April 24, 2025
Deployment to production workspaces	April 29, 2025

What’s New

Support for using the OAuth protocol for authorization and authentication with Databricks.
Added ID stitcher rules — this feature allows you to control which nodes and edges are included in your ID graph by defining rules in a warehouse table.
Improved logging and error handling.
MFA token caching support is added for Snowflake. Pass passcode with --passcode flag. Ex: pb run --passcode 123456.
Log file is renamed to pb.log.
pb tutorial now accepts the --rpc_python_path flag.

Bug Fixes

Fixed PyNative packages loading issues.
Fixed an issue where duplicates under model_folders were resulting in an error.
Fixed an issue with PyNative models when using with a Snowflake unencrypted private key.

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross-database references can fail on Redshift for a few clusters.
While creating Activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs - you can ignore it: WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get ssh: handshake failed error, then you’ll have to manually remove the entire folder from WhtGitCache to make it work.

Version 0.21.6

1 April 2025

Bug Fixes

Fixed the Bitbucket Git URL parsing issue.
Fixed an issue to control the number of warehouse calls made for pre-existing checks when running with high concurrency.

Version 0.21.5

26 March 2025

Bug Fixes

Corrected the protobuf dependency versions.
Fixed the Databricks validating privileges issue when catalog has special characters.

Version 0.21.4

13 March 2025

Bug Fixes

Fixed an issue wherein the metadata of column materials’ parent field was set wrong.
Boundaries of month and year timegrains are now made to coincide with week boundary.

Version 0.21.3

11 March 2025

Bug Fixes

Fixed Snowflake key-pair auth regression.

Version 0.21.2

6 March 2025
Schema version: 88

Release Schedule

Milestone	Date
CLI release	March 6, 2025
Deployment to dev workspaces	March 6, 2025
Deployment to production workspaces	March 11, 2025

What’s New

Schema has been updated from 85 to 88.
Introduced Timegrains - timegrains give you the ability to control the frequency at which certain models run irrespective of the frequency at which you trigger the Profiles project runs.
Introduced a change wherein the IDs to be considered for ID stitching must be of string data type. If ID columns in your inputs are not strings, you must cast them explicitly.

Optimizations

These optimizations are still in beta and are released behind feature flags. See Optimizations for more information.

Input var bundling: With this optimization, PB automatically bundles the execution of some input_vars together during runtime, based on the var definitions.
Entity var bundling: With this optimization, PB automatically bundles the execution of some entity_vars together during runtime, based on the var definitions.

Improvements

ID stitcher now runs in incremental mode by default.
Reduced the package size by 50% by removing debugging information while building the binary.
pb show plan now shows the disabled materials. This is helpful when working with timegrains to understand which materials run in a particular timegrain.

Deprecation

Username/password authentication is now deprecated for Snowflake - use key-pair authentication instead.

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross database references can fail on Redshift for a few clusters.
While creating Activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs - you can ignore it: WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get ssh: handshake failed error, then you’ll have to manually remove the entire folder from WhtGitCache to make it work.

Version 0.20.3

3 February 2025

Bug Fixes

Fixed a performance issue for nested column models by adding LIMIT 1 to the query that checks if a column exists or not.

Version 0.20.2

28 January 2025

Bug Fixes

Fixed an issue where project fails on Snowflake and Redshift when the ID columns are of non-string types.

Version 0.20.1

27 January 2025

Bug Fixes

Fixed an issue wherein any changes to id_types are not considered in the ID stitcher model hash computation.

Version 0.20.0

Schema version: 85

Release Schedule

Milestone	Date
CLI release	January 21, 2025
Deployment to dev workspaces	January 21, 2025
Deployment to production workspaces	January 25, 2025

What’s New

Schema has been updated from 84 to 85.
Added a column description: The YAML description of each feature now gets added in feature_views as a comment in the corresponding column.
Azure DevOps Git URLs are now supported while creating a Profiles project in the dashboard. See Import Profiles Project from Git for more information.
WHERE clause support: Input YAMLs now support the WHERE clause, helping you filter out any unnecessary rows from the source and reduce the need for an additional SQL model.
A new command pb tutorial is now added. Profiles Tutorial is a guided interactive tutorial within the Profiles CLI and walks you through the key Profiles concepts and how they work. See Profiles Tutorial for more information.

Improvements

ID stitcher performance improvements: Rows in the ID stitcher table are compacted based on node_id, node_id_type, and main_id to remove duplicates. Also, some unnecessary steps are removed from the ID stitching process to make it faster.
Removed empty lines from the generated SQL to make it more readable.
The SQL errors now show the file path and the corresponding error, instead of printing the entire query in the console/logs.
Removed some unnecessary internal models to simplify the model graph in case of column models.

Bug Fixes

Fixed the issue wherein running the var materials failed sometimes if the PB process was killed forcefully, by making the process idempotent.
You can now refer to features on SQL models and Python model (non-entity-var features) also using the general path reference for the feature, i.e. <entity>/<cohort>/<feature-name>. You can also refer to the inherited features in the same way.
Fixed the issue wherein --output_folder flag was not respected in pb run and pb compile commands by supporting absolute paths.

Deprecation

python_model model type is now deprecated.

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross database references can fail on Redshift for a few clusters.
While creating Activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs - you can ignore it: WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get ssh: handshake failed error, then you’ll have to manually remove the entire folder from WhtGitCache to make it work.
Timegrains is an experimental feature. There might be some undiscovered issues.

Version 0.19.3

9 January 2025

Bug Fixes

Properly sets the Snowflake connection parameter application to Rudderstack_Warehouse.

Version 0.19.2

6 December 2024

Bug Fixes

Fixed a common tables migration bug where PB may go into an unrecoverable state when you run it in two parallel processes.
Fixed an inconsistency in incremental ID stitching due to differences in how warehouses treat NULL values sorting.
Fixed a bug in incremental ID stitching related to occasional changes in the main IDs when you add new edges to the cluster.

Version 0.19.1

29 November 2024

Bug Fixes

Fixed a bug causing an unexpected change in the profiles ID when the ID stitcher is running in incremental mode.
Fixed a crash that occurs while parsing a YAML which starts with a list at the root level.

Version 0.19

14 November 2024
Schema version: 84

What’s New

Schema has been updated from 80 to 84.
Project created using pb init pb-project now locks the PB version.
Task summary is now shown at the end of each run, including relevant command details like start time, end time, sequence number, and total models processed (if any).
The number of iterations/loops during ID stitching are now logged.
A new tool named Profiles ID Stitcher Audit tool is introduced. It is embedded within the core PB CLI tool and helps you analyze the health of your ID graph by giving you a high level aggregate analysis of your ID graph across entities. You can also analyze a single cluster by visualizing all IDs and their connections, along with highlighting the most important IDs that form connections in that group.
A key named carry_forward_privileges (type Boolean) is added in the pb_project.yaml file. This key determines whether the privileges granted on the views should be retained after the view definitions are updated. This resolves the issue where recreating views for each model in the project led to revoked privileges. Note that using this key can lead to some performance overhead.
A new column named pb_version is added to the material_registry table in the database which denotes the PB version that created the entry.
A new flag pb show idstitcher-report --seed_clusters_csv is added which you can use to pass a CSV file with two columns - id and id_type. The cluster ID can be either rudder_id or other_id. Each row specifies an ID to be mapped to a cluster main ID, which is then included in the report.

Improvements

An error is now thrown if a model has a feature without any IDs.
User-friendly messages are shown in case a package isn’t installed.
Error is now thrown if duplicate feature names are present for a cohort.

Bug fixes

Resolved the cleanup issue where some materials cleanup failed due to dependencies.
Fixed an issue where a non-existent folder inside model_folders caused the project to fail.

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross database references can fail on Redshift for a few clusters.
While creating Activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs - you can ignore it: WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get ssh: handshake failed error, then you’ll have to manually remove the entire folder from WhtGitCache to make it work.
Timegrains is an experimental feature. There might be some undiscovered issues.

Version 0.18.5

9 January 2025

Bug Fixes

Properly sets the Snowflake connection parameter application to Rudderstack_Warehouse.

Version 0.18.4

25 October 2024

Bug Fixes

The schema name has been prepended to the drop statements executed during cleanup. This ensures that deletions are always performed in the correct schema.

Version 0.18.3

16 October 2024

Bug Fixes

Fixed a bug that causes cleanup of materials fail due to current transaction is aborted error. With the fix, if cleanup of one material fails (for some reason, ex: other objects depend on it), the cleanup of other expired materials should continue.

Version 0.18.2

4 October 2024

Bug Fixes

Resolved a migration bug which occurs when there are nested model folders containing non yaml files.
Cleanup with flag --remove_latest_view_ptrs was not respecting retention time period set in pb_project.yaml. This is fixed.

Version 0.18.1

3 October 2024

Bug Fixes

Fixed issue in default ID stitcher for run_type: discrete.
Resolved a bug that occurs when a project contains multiple entities with same cohort name.
During migration, pb was skipping non-YAML files, which caused scheduled runs to fail. This is fixed.

Version 0.18

27 September 2024
Schema version: 80

What’s New

Cohort model now lets you perform filtering using a filter_expression followed by AND/OR list of expressions, for example:

models:
 - name: high_value_us_residents
 model_type: cohort
 model_spec:
 ...
 filter_expression:
 AND:
 - {{ user.Var('country') }} = 'US'
 - {{ user.Var('salary') }} > 10000

You can define the retention_period for each model of a project. Further, the pb cleanup materials --expired command cleans up the materials beyond the defined retention period.
Referring other entity_vars/input_vars is now simplified. You can use {{entityName.entity_varName}} instead of the earlier one {{entityName.Var("entity_varName")}}. Note that the earlier syntax also works fine.
You can use features of an SQL model while using a cohort. To do so, specify the entity_key or entity_cohort in the model_spec of an SQL model.
pb cleanup materials --concurrency - A new command which enables concurrency for cleanup, by defining the number of concurrent workers for cleanup. The default value is 1.
The default offset value while executing pb run command is now updated to 0. It was 30 minutes earlier.
A new flag --end_time_offset is added to the compile/run commands for adding an offset to the end timestamp, in a human readable format. It means that RudderStack does not use any data you load in the warehouse after the offset time has elapsed for that run. For example, pb run --end_time_offset=45m ensures that RudderStack does not use any data older than 45 minutes from the run’s start time. Note that you can’t use this new flag with the seq_no or end_time flags.
You can now import Packages starting with SSH URLs, for example, ssh://git@host:port/path.git.
You can run or import projects hosted on S3 as packages by adding block_store_creds in your site configuration file. To run the project, execute pb run -p s3://<url> command.
Running a project with the --migrate_on_load flag now stores generated artifacts in the output subfolder instead of migrations.
For an entity_var/input_var, the default key has been renamed to default_value.
Simplified the project created using pb init pb-project by removing the dependency on corelib package , sample SQL model, model contracts and CSV’s in the inputs file.
RudderStack now uses INNER JOIN instead of RIGHT JOIN when calculating entity_vars. This results in performance improvement and also prevents some values from getting lost.
Feature view model with main_id as an identifier is created by default.
Schema has been updated from 72 to 80.

Improvements

By default, RudderStack ignores all the blank values in the ID stitcher model.
There is a slight aesthetic improvement in HTML reports generated using pb show idstitcher-report command.
Relevant errors are now thrown if you specify an unknown YAML key in the model definition.

Bug Fixes

validity_time key has been removed.
The pb validate access command, for Databricks, now checks only for the necessary permissions and not for ALL the privileges.

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross database references can fail on Redshift for a few clusters.
While creating Activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Databricks

Concurrency does not work for cleanup.

Other issues

Linux users might see this warning for all command runs - you can ignore it: WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get ssh: handshake failed error, then you’ll have to manually remove the entire folder from WhtGitCache to make it work.
Timegrains is an experimental feature. There might be some undiscovered issues.

Version 0.17.1

9 January 2025

Bug Fixes

Properly sets the Snowflake connection parameter application to Rudderstack_Warehouse.

Version 0.17

14 August 2024
Schema version: 72

What’s New

pb show plan: A new subcommand is added to show detailed information about the materials with different timegrains along with their dependencies in the order of execution.
The casing for the input tables names defined using double quotes is preserved in the warehouse. For example, the table defined as table: "TableName" will be referred as TableName in the warehouse.
Schema has been updated from 71 to 72.

Improvements

Relevant error message is shown if the computed status for a material is not possible.
An error is thrown if multiple feature views are defined with the same name.

Bug Fixes

Fixed the issue where input_vars was not getting defined on the packages.
Resolved the bug where the project was failing due to empty or commented YAML files.

Known Issues

BigQuery

pb validate access command does not work for BigQuery.

Redshift

If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
Cross database references can fail on Redshift for a few clusters.
While creating Activations, validation for Redshift does not work correctly in the RudderStack dashboard.

Other issues

Linux users might see this warning for all command runs - you can ignore it: WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
pb insert does not work for Redshift, Databricks, and BigQuery.
If you are referring a public package in the project and get ssh: handshake failed error, then you’ll have to manually remove the entire folder from WhtGitCache to make it work.
The code for validity_time is redundant and should be removed.
Timegrains is an experimental feature. There might be some undiscovered issues.

Version 0.16.1

6 August 2024

Bug Fixes

Fixed the concurrency issue where project run was failing because the same ID stitcher model was served as an input source to two models.
Resolved the ambiguous column issue by using an alias for selecting main_id in the entity_var.

Version 0.16

1 August 2024
Schema version: 71

What’s New

You can now choose between the username-password and key-pair authentication modes while running the pb init connection command for Snowflake.
Added support to remove materials based on time in hours and milliseconds. For example, to remove materials older than 3 hours, use pb cleanup materials --retention_time_in_hours 3. To remove materials older than 100 milliseconds, use pb cleanup materials --retention_time_in_ms 100.
Schema has been updated from 69 to 71.
Edge type preferred has been renamed to coercive. Here’s a sample code:

 - name: example
 model_type: sql_template
 model_spec:
 single_sql: |
 select * from {{this.DeRef(test, dependency = "coercive")}}

Improvements

Changed the casting for timestamp columns for Snowflake, Redshift, and Databricks warehouses to avoid a timestamp with string comparison. CAST(timestamp_column AS TIMESTAMP) < ‘2024-07-28T23:00:00Z’ has been changed to CAST(timestamp_column AS TIMESTAMP) < CAST(‘2024-07-28T23:00:00Z’ AS TIMESTAMP)

Bug Fixes

Fixed the Sequence number conflict found error on BigQuery.
Fixed the bug in RudderStack dashboard where entities weren’t showing up, in case no features were defined in the project.
Resolved the intermittent timeout waiting for python client to initialize error for projects using PyNative models.
Fixed the directory not empty error when Profiles tries to clone a git repository.
Fixed the issue where the registry entry for materials was not using the same casing due to which the relations were getting created in the warehouse. Now the cleanup is able to delete materials that were skipped earlier due to case mismatch.
Fixed the issue where the same hash was calculated for different cohorts, as the filter pipeline wasn’t being considered.
Fixed the issue where some procedures on Snowflake and BigQuery were not getting removed after the run.

Known Issues

RudderStack does not support accessing input sources in a different project for the BigQuery warehouse.
Linux users might see this warning for all command runs - you can ignore it: WARN[0000]log.go:228 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
Redshift: If two different users create material objects on the same schema, RudderStack gives an error during cleanup when trying to drop views created by the other user, like user_var_table.
pb validate access command does not work for BigQuery.
pb insert does not work for Redshift, Databricks, and BigQuery.
Cross database references can fail on Redshift for a few clusters.
If you are referring a public package in the project and get ssh: handshake failed error, then you’ll have to manually remove the entire folder from WhtGitCache to make it work.
The code for validity_time is redundant and should be removed.
Timegrains is an experimental feature. There might be some undiscovered issues.
While creating Activations, validation for Redshift does not work correctly in the RudderStack dashboard.