RudderStack Docs – Advanced ID Stitcher Features

Archive: ID Graph Inclusion/Exclusion Rules

Mon, 01 Jan 0001 00:00:00 +0000

The ID graph exclusions/inclusions feature lets you determine which nodes and edges are included in your ID graph by defining rules in a warehouse table—giving you fine-grained control over the graph structure beyond what’s possible with regex-based filters.

These rules take the highest precedence over any other filtering mechanisms, including regex filters and rule-based filters.

Key features

Node control: Exclude or include specific nodes (identifiers) in the ID graph.
Edge control: Define specific connections between the identifiers to exclude or include.
Rule management: Get CLI tools for creating and populating rules tables.
Precedence control: Specify how conflicting rules should be handled.

Set up the rules table

This section describes how to create and populate the rules table that controls ID Graph inclusions and exclusions.

1. Create the rules table

Use the following CLI command to create a dedicated table in your warehouse:

pb idstitch rules create

This creates a table named id_stitch_rules by default. You can specify a custom name as follows:

pb idstitch rules create --name my_custom_rules

2. Add rules to this table

You can add rules in two ways:

Interactive mode

Run the following command:

pb idstitch rules update --interactive

You will then be prompted to enter the following details:

Entity name (for example, user, account)
ID1 value and type
Optional ID2 value and type
Action (EXCLUDE or INCLUDE)
Optional reason for the action

Batch mode

Run the following command:

pb idstitch rules update -f rules.csv

This rules.csv file will contain the required columns: entity, id1, id1_type, action, and optional columns: id2, id2_type, reason.

Configure your project YAML

To enable the rules for your project, add the following configuration to your YAML file:

# Existing configuration...

entities:
 - name: user
 id_stitcher: models/rudder_user_id_stitcher
 # Other entity configuration

# Add the id_stitcher_rules section
id_stitcher_rules:
 enabled: true # Optional, defaults to false
 table_name: id_stitch_rules  # Optional, defaults to id_stitch_rules
 priority: exclude  # Optional, defaults to exclude

Configuration parameters

Parameter	Description
`enabled`	Set it to `true` to activate the rules feature.
`table_name`	The name of your rules table. It defaults to `id_stitch_rules`.
`priority`	Defines how to handle conflicting rules. Acceptable values are `exclude` and `include`. The default value is `exclude`.

Rule behavior and examples

Note that the rules fall into two categories:

1. Single ID rules (Node rules)

These rules apply to a single identifier and affect the entire node.

Example: Exclude a test email

Entity: user
ID1: test@company.com
ID1_Type: email
Action: EXCLUDE

Effect: The node for test@company.com will be completely removed from the ID graph.

Example: Include a specific user ID

Entity: user
ID1: 12345
ID1_Type: user_id
Action: INCLUDE

Effect: The node for user_id 12345 will be added to the graph even if it doesn’t appear in the input data.

2. Paired ID rules (Edge rules)

These rules apply to the connections between two identifiers.

Example: Exclude a specific connection

Entity: user
ID1: john@example.com
ID1_Type: email
ID2: user_67890
ID2_Type: user_id
Action: EXCLUDE

Effect: The direct connection between john@example.com and user_67890 will be removed but both nodes remain in the ID graph. You can still connect them through other paths.

Example: Include a specific connection

Entity: user
ID1: jane@example.com
ID1_Type: email
ID2: user_54321
ID2_Type: user_id
Action: INCLUDE

Effect: A direct connection between jane@example.com and user_54321 will be added to the ID graph even if this connection doesn’t exist in the input data.

Conflict resolution

When rules conflict, the resolution depends on your priority setting:

With priority: exclude (default): If a node is excluded but appears in an edge inclusion rule, the node exclusion takes precedence and the edge inclusion is discarded.
With priority: include: If a node is excluded but appears in an edge inclusion rule, the edge inclusion takes precedence and the node will remain in the graph.

If multiple rules exist for the same entity and identifiers, the most recent rule takes precedence.

How this feature works

During pb run or pb compile, the system checks for any enabled rules.
Rules are applied while preparing the raw_edges table. Note that excluded nodes are not added to this raw_edges table.

Archive: ID Graph Cardinality Rules

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains how to use cardinality rules to limit the maximum number of connections between different ID types in your identity graph.

Cardinality rules are supported for Snowflake and BigQuery.

Rule enforcement, the <material>_<entity>_CARDINALITY_AUDIT table, and the per-edge cardinality_info metadata are emitted on both warehouses.

Overview

ID Graph cardinality rules let you define maximum connection limits between different ID types in your identity graph, providing fine-grained control over how identities are stitched together.

Cardinality rules help prevent identity explosion and maintain data quality by limiting how many connections a node of one type can have to nodes of another type.

This is particularly useful for maintaining data quality when dealing with shared identifiers like email addresses, device IDs, or anonymous IDs that might be used across multiple users.

Cardinality rules work seamlessly with other Profiles features:

Input filters reduce data volume before graph processing. The cardinality enforcement step comes after this in the pipeline, so if expected nodes are missing from your graph, input filters can be the reason.
Manual exclusion/inclusion rules provide explicit control over specific connections and happen after the cardinality enforcement step in the pipeline. If you find identity clusters that violate cardinality rules despite having the rules configured, those extra edges might be coming from manual inclusion overrides.

Key features

Data quality: You can prevent identity graphs from becoming overly connected due to shared or recycled identifiers.
Performance: Maintain optimal graph size by controlling edge proliferation.
Compliance: Enforce business rules about identity relationships.
Transparency: Get full audit trail of which edges were filtered and why.
Incremental consistency: Historical violations persist across runs for reliable behavior.

Performance considerations

Consider the following when implementing cardinality rules:

Initial runs may experience increased runtime as the system processes and enforces the new rules across your entire dataset.
Incremental runs should see minimal performance impact once the rules are established.
Trade-off evaluation: The increased runtime in initial runs delivers more precise ID stitching and better data quality. Evaluate whether the improved accuracy justifies the additional processing time for your use case.

How cardinality rules work

Directional rules

Cardinality rules are directional — they apply from a source ID type to a target ID type. For example:

user_id → email: max_edges = 2 means each user_id can connect to at most 2 email.
Note that this does not limit how many user_id an email can connect to (unless you add a separate rule)

Rule enforcement

When a rule is violated:

All edges from the violating source are removed (not just the excess ones)
Nodes are preserved as isolated stubs to maintain graph integrity
Violations are audited with full details about what was removed and why
Historical context is maintained across incremental runs

Incremental behavior

The feature works seamlessly with incremental runs:

Historical violations persist: If a node violated a rule previously, new edges to that node type are automatically blocked.
New violations break all edges: If a new edge causes a violation, all current edges from that source are removed.
Rule removal: If you remove a rule from your configuration, the entire id graph is built from scratch, so historical violations for that rule are no longer enforced.

Configuration

You can add cardinality rules to your pb_project.yaml under the id_types section, as shown:

entities:
 - name: user
 id_stitcher: models/customer_id_graph
 id_types:
 - user_id
 - email
 - anonymous_id

id_types:
 - name: user_id
 maximum_edges:
 - email: 2
 - anonymous_id: 1
 - name: email
 maximum_edges:
 - user_id: 1
 - name: anonymous_id
 maximum_edges:
 - email: 1
 - user_id: 1

Configuration limits

The following limits ensure optimal performance for incremental runs:

Maximum edges per rule: 10 (The maximum_edges value for each rule cannot exceed 10)
Maximum target types per source: 5 (The maximum_edges list for each source ID type cannot have more than 5 entries)

The above limits only apply when using cardinality rules. Without these rules, the regular ID stitcher can connect unlimited edges, as usual.

Examples

This section provides examples of how to configure cardinality rules for different use cases:

id_types:
 - name: user_id
 maximum_edges:
 - email: 2 # Users can have up to 2 emails
 - device_id: 5 # Users can have up to 5 devices
 - name: email
 maximum_edges:
 - user_id: 1 # Each email belongs to only 1 user
 - name: device_id
 maximum_edges:
 - user_id: 2 # Shared devices (family tablets)

id_types:
 - name: anonymous_id
 maximum_edges:
 - email: 1 # Anonymous sessions link to 1 email
 - name: email
 maximum_edges:
 - user_id: 1 # Emails don't get shared between users

How rule application works

This section explains how cardinality rules are applied to your identity graph.

Fresh and incremental runs

Cardinality rules behave differently depending on whether you’re running a fresh or incremental run:

Fresh runs: Rules are applied to all incoming data. If a node stays within its limits, all edges are preserved.
Incremental runs: Rules consider historical violations. If a node previously violated a rule or if new data causes a violation, all edges from that source are removed, including previously valid ones.

Fresh run

# Input data
user_1 → email_a
user_1 → email_b # Valid: within limit of 2

# Result: Both edges allowed

Incremental run

# Previous run: user_1 had 2 emails (at limit)
# New data: 
user_1 → email_c  # This causes violation

# Result: ALL edges from user_1 are broken (including previous ones)

Example

Given these rules:

id_types:
 - name: user_id
 maximum_edges:
 - email: 2
 - name: email 
 maximum_edges:
 - user_id: 1

And the following data:

user_1 ↔ email@example.com
user_2 ↔ email@example.com 
user_3 ↔ email@example.com

What happens

email@example.com connects to 3 user_ids, violating the email → user_id: 1 rule
All three connections are broken, not just the excess ones.
All nodes are preserved as isolated stubs.
Three audit entries are created documenting the violation.

Monitoring and debugging

This section explains how to monitor and debug cardinality rules.

Cardinality Audit table

Each violation is logged in the cardinality audit table, which resides in the same schema as the ID Stitcher output tables. The table name is derived by appending the suffix _CARDINALITY_AUDIT to the name of the ID Stitcher model.

The audit table is created on both Snowflake and BigQuery and contains the following columns:

Column	Description
`run_id`	ID of the run when the violation occurred.
`model_hash`	Hash of the model configuration.
`id1`	Source ID value.
`id1_type`	Source ID type.
`id2`	Target ID value.
`id2_type`	Target ID type.
`reason`	Always `"CARDINALITY_VIOLATION"`.
`rule_details`	JSON with `max_edges` and `current_count`.
`source_models`	JSON array (BigQuery) or native ARRAY (Snowflake). Each element is `{source_model, first_seen_at}` and records every input model that contributed the dropped edge, with the earliest timestamp at which that model first observed it.

Sample rows

The following rows are excerpted from a qa_metadata_validation_user_CARDINALITY_AUDIT table after a run where two users (viol_user and mviol_user) each exceeded a user_id → email: max_edges = 2 rule. mviol_user saw one of the violator emails (mviol_shared@test.com) from two different input models, which is why its source_models array has two entries:

id1, id1_type, id2, id2_type, model_hash, reason, rule_details, run_id, source_models
viol_user, user_id, viol_a@test.com, email, ffffe585, CARDINALITY_VIOLATION, {"current_count":3,"max_edges":2}, 1, [{"source_model":"events_app","first_seen_at":"2022-03-01 10:00:00.000"}]
viol_user, user_id, viol_b@test.com, email, ffffe585, CARDINALITY_VIOLATION, {"current_count":3,"max_edges":2}, 1, [{"source_model":"events_app","first_seen_at":"2022-03-01 10:00:01.000"}]
viol_user, user_id, viol_c@test.com, email, ffffe585, CARDINALITY_VIOLATION, {"current_count":3,"max_edges":2}, 1, [{"source_model":"events_app","first_seen_at":"2022-03-01 10:00:02.000"}]
mviol_user, user_id, mviol_shared@test.com, email, ffffe585, CARDINALITY_VIOLATION, {"current_count":3,"max_edges":2}, 1, [{"source_model":"events_app","first_seen_at":"2022-04-01 10:00:00.000"},{"source_model":"events_web","first_seen_at":"2022-03-25 08:00:00.000"}]

`cardinality_info` in the metadata column

When edge metadata logging is enabled (see Metadata and observability), the per-node METADATA column on the ID stitcher output reflects rule enforcement in two ways:

Capped edges for non-violator nodes carry a cardinality_info field on the stub edge, recording the max_edges and current_count that caused the cap.
Violator nodes (nodes that exceeded a rule and had all their edges removed) appear with an empty direct_edge_info.edges_by_type array. Their node_info is preserved so the node still exists as an isolated stub in the graph.

For example, both a single-source violator (viol_user) and a multi-source violator (mviol_user) collapse to the same shape in the metadata column — only node_info survives:

{
 "direct_edge_info": {
 "edges_by_type": []
 },
 "node_info": [
 {
 "first_seen_at": "2022-04-01 10:00:00.000",
 "source_model": ["events_app"]
 },
 {
 "first_seen_at": "2022-03-25 08:00:00.000",
 "source_model": ["events_web"]
 }
 ]
}

Because dropped edges no longer appear in the metadata column, the audit table’s source_models column is the only surviving trace of which input model contributed each violator edge. See Interpret the audit table for a debugging workflow.

Best practices

This section provides best practices for configuring the cardinality rules correctly in your Profiles project.

1. Start conservatively

Begin with lower limits and gradually increase based on your data patterns:

# Start here
id_types:
 - name: email
 maximum_edges:
 - user_id: 1

# Adjust based on audit results
id_types:
 - name: email
 maximum_edges:
 - user_id: 2 # If you see legitimate shared emails

2. Monitor audit tables

Make sure to review the Cardinality Audit table regularly to understand:

Which rules are triggering most often
Whether legitimate connections are being blocked
If limits need adjustment

3. Test with historical data

Before deploying rules to production:

Run rules against historical data in a test environment.
Analyze the audit results.
Adjust limits based on findings.

4. Consider bidirectional rules

Make sure to consider both directions of a relationship:

# Often you want both directions
id_types:
 - name: user_id
 maximum_edges:
 - email: 2 # Users can have multiple emails
 - name: email
 maximum_edges:
 - user_id: 1 # But emails belong to one user

5. Add rules for shared identifiers

Cardinality rules help when a single identifier can legitimately appear across many users but you don’t want the graph to merge those users. Common cases:

Shared corporate emails such as info@, support@, or hr@ addresses that show up in many sessions.
Bot or test accounts that share a fixed anonymous_id or device_id across thousands of synthetic users.
Sentinel and placeholder values such as 0, unknown, n/a, or a default device identifier that slipped past upstream filters and would otherwise stitch unrelated users into one cluster.

If a few well-known sentinel values are responsible for most of the spread, prefer adding them to your input filters so they never enter the graph in the first place. Cardinality rules are the right tool when the offending values are not known in advance and you need the graph itself to flag them.

6. Interpret the audit table

To triage cardinality violations:

Group by reason and model_hash to confirm the violations belong to the rule you expect. rule_details carries the configured max_edges value and the current_count the source node reached, so a row with current_count: 12, max_edges: 2 is qualitatively different from one with current_count: 3, max_edges: 2.
Sort by id1 to find the worst offenders. A node that appears as id1 many times is over-connected on one or more target types; the count of distinct id2 values for that id1 reflects how many edges triggered the violation in the current run.
Use source_models to trace each dropped edge back to its input. This is the only surviving source-attribution trace for violator edges, because the per-node direct_edge_info.edges_by_type array is emptied when a node violates a rule. If a single id2 carries multiple source_models entries, multiple input models independently reported the same edge — useful for deciding whether the offending edge is a data-quality problem in one source or a real cross-source pattern.
Cross-reference with the metadata column. For non-violator nodes that share an id2 with a violator, look at the cardinality_info field on the stub edge to see whether the rule capped the edge or whether it survived intact.

7. Trace source attribution in the metadata column

For nodes that did not violate a rule, the metadata column carries full source attribution per edge. The direct_edge_info.edges_by_type[].edges[].sources[] array lists every input model that contributed the edge, along with the timestamp each one first observed it. Combined with the audit table for violators, these two sources provide complete source-attribution coverage across the graph.

See Metadata and observability for the full per-source contract.

Troubleshooting

Issue	Solution
Legitimate connections being blocked	Review audit table and increase limits if needed.
Identity graph still too connected	Add more restrictive rules or lower existing limits.

Debugging

Check rule configuration in pb_project.yaml.
Query audit table to see what is blocked.
Review stub metadata in the final ID graph.
Compare fresh vs. incremental run results.

Migration guide

This section provides tips on managing cardinality rules in your Profiles models.

Enable rules on existing models

Add rules gradually: Start with one rule and observe effects
Test in development: Use a copy of production data to validate
Monitor after deployment: Watch audit tables and graph metrics
Adjust as needed: Fine-tune limits based on real-world results

Disable rules

To stop enforcing a rule, simply remove it from your configuration:

# Before
id_types:
 - name: email
 maximum_edges:
 - user_id: 1

# After (rule removed)
id_types:
 - name: email
 # maximum_edges section removed

This triggers a complete fresh run of the model. Historical violations are no longer enforced, but previous audit records remain for reference.

Archive: Advanced ID Stitcher Operations

Mon, 01 Jan 0001 00:00:00 +0000

This guide explains advanced operations available in the Profiles ID Stitcher, including the rebase incremental mode and how ID determinism ensures stable user identifiers across runs.

The `rebase_incremental` flag

While the ID Stitcher typically runs in Incremental Mode (processing only the data that arrived since the start of the last run and merging it with previous run’s graph), you can use a powerful reset mechanism called rebase_incremental.

This flag is currently only available in the CLI.

How it works

When you set this flag to true, the pipeline changes behavior in three critical ways:

Ignores history: It does not load the previous mapping table (the graph state from the previous run).
Full ingestion: It removes the timestamp filters on the input tables, reading the entire history of edges from the beginning of time.
Fresh calculation: It rebuilds every cluster and recalculates every user ID from scratch using the current logic.

Use cases

Applying new rules retroactively: If you update your blocklist (for example, adding a new spam IP), a standard incremental run only blocks future edges. A rebase applies this block to the entire history, splitting any clusters that were incorrectly merged by that IP in the past.
Graph reset: If the graph has become too “messy” (too many older, irrelevant edges), a rebase allows you to clean up the state based on current data retention policies.

Why IDs are deterministic

A common concern with re-running a graph algorithm is that IDs might shuffle (for example, user ABC becomes user XYZ). The ID Stitcher logic prevents this.

Logic used

The user ID is not a random number — it is mathematically derived from the anchor node of the cluster.

Anchor node: The node with the earliest occurred_at_col timestamp in the cluster.
Formula used:

ID = ID_GEN_OPERATOR(ANCHOR_NODE_VALUE + ANCHOR_NODE_TYPE)

Why IDs persist during incremental runs

The system carries forward the rudder_main_id from the previous run via the previous mapping table. New nodes simply adopt the existing ID of the cluster they join.

Why IDs persist during rebase or fresh runs

Even if you delete the previous mapping table, the anchor node for a specific user (for example, the first cookie they used in 2021) effectively remains the same in the raw data.

The algorithm re-groups the cluster.
It identifies the node with the minimum timestamp (the same 2021 cookie).
It hashes that value.

Result: The generated ID comes out identical to the previous run.

RudderStack Docs – Advanced ID Stitcher Features

Archive: ID Graph Inclusion/Exclusion Rules

Key features

Set up the rules table

1. Create the rules table

2. Add rules to this table

Interactive mode

Batch mode

Configure your project YAML

Configuration parameters

Rule behavior and examples

1. Single ID rules (Node rules)

Example: Exclude a test email

Example: Include a specific user ID

2. Paired ID rules (Edge rules)

Example: Exclude a specific connection

Example: Include a specific connection

Conflict resolution

How this feature works

Archive: ID Graph Cardinality Rules

Overview

Key features

Performance considerations

How cardinality rules work

Directional rules

Rule enforcement

Incremental behavior

Configuration

Configuration limits

Examples

How rule application works

Fresh and incremental runs

Fresh run

Incremental run

Example

Monitoring and debugging

Cardinality Audit table

Sample rows

cardinality_info in the metadata column

Best practices

1. Start conservatively

2. Monitor audit tables

3. Test with historical data

4. Consider bidirectional rules

5. Add rules for shared identifiers

6. Interpret the audit table

7. Trace source attribution in the metadata column

Troubleshooting

Debugging

Migration guide

Enable rules on existing models

Disable rules

Archive: Advanced ID Stitcher Operations

The rebase_incremental flag

How it works

Use cases

Why IDs are deterministic

Why IDs persist during incremental runs

Why IDs persist during rebase or fresh runs

`cardinality_info` in the metadata column

The `rebase_incremental` flag