You are viewing documentation for an older version.

Click here to view the latest documentation.

ID Graph Cardinality Rules

This guide explains how to use cardinality rules to limit the maximum number of connections between different ID types in your identity graph.

Cardinality rules are supported for Snowflake and BigQuery.

Rule enforcement, the <material>_<entity>_CARDINALITY_AUDIT table, and the per-edge cardinality_info metadata are emitted on both warehouses.

Overview

ID Graph cardinality rules let you define maximum connection limits between different ID types in your identity graph, providing fine-grained control over how identities are stitched together.

Cardinality rules help prevent identity explosion and maintain data quality by limiting how many connections a node of one type can have to nodes of another type.

This is particularly useful for maintaining data quality when dealing with shared identifiers like email addresses, device IDs, or anonymous IDs that might be used across multiple users.

Cardinality rules work seamlessly with other Profiles features:

• Input filters reduce data volume before graph processing. The cardinality enforcement step comes after this in the pipeline, so if expected nodes are missing from your graph, input filters can be the reason.
• Manual exclusion/inclusion rules provide explicit control over specific connections and happen after the cardinality enforcement step in the pipeline. If you find identity clusters that violate cardinality rules despite having the rules configured, those extra edges might be coming from manual inclusion overrides.

Key features

• Data quality: You can prevent identity graphs from becoming overly connected due to shared or recycled identifiers.
• Performance: Maintain optimal graph size by controlling edge proliferation.
• Compliance: Enforce business rules about identity relationships.
• Transparency: Get full audit trail of which edges were filtered and why.
• Incremental consistency: Historical violations persist across runs for reliable behavior.

Performance considerations

Consider the following when implementing cardinality rules:

• Initial runs may experience increased runtime as the system processes and enforces the new rules across your entire dataset.
• Incremental runs should see minimal performance impact once the rules are established.
• Trade-off evaluation: The increased runtime in initial runs delivers more precise ID stitching and better data quality. Evaluate whether the improved accuracy justifies the additional processing time for your use case.

How cardinality rules work

Directional rules

Cardinality rules are directional — they apply from a source ID type to a target ID type. For example:

• user_id → email: max_edges = 2 means each user_id can connect to at most 2 email.
• Note that this does not limit how many user_id an email can connect to (unless you add a separate rule)

Rule enforcement

When a rule is violated:

1. All edges from the violating source are removed (not just the excess ones)
2. Nodes are preserved as isolated stubs to maintain graph integrity
3. Violations are audited with full details about what was removed and why
4. Historical context is maintained across incremental runs

Incremental behavior

The feature works seamlessly with incremental runs:

• Historical violations persist: If a node violated a rule previously, new edges to that node type are automatically blocked.
• New violations break all edges: If a new edge causes a violation, all current edges from that source are removed.
• Rule removal: If you remove a rule from your configuration, the entire id graph is built from scratch, so historical violations for that rule are no longer enforced.

Configuration

You can add cardinality rules to your pb_project.yaml under the id_types section, as shown:

entities:
  - name: user
    id_stitcher: models/customer_id_graph
    id_types:
      - user_id
      - email
      - anonymous_id

id_types:
  - name: user_id
    maximum_edges:
      - email: 2
      - anonymous_id: 1
  - name: email
    maximum_edges:
      - user_id: 1
  - name: anonymous_id
    maximum_edges:
      - email: 1
      - user_id: 1

Configuration limits

The following limits ensure optimal performance for incremental runs:

• Maximum edges per rule: 10 (The maximum_edges value for each rule cannot exceed 10)
• Maximum target types per source: 5 (The maximum_edges list for each source ID type cannot have more than 5 entries)

The above limits only apply when using cardinality rules. Without these rules, the regular ID stitcher can connect unlimited edges, as usual.

Examples

This section provides examples of how to configure cardinality rules for different use cases:

id_types:
  - name: user_id
    maximum_edges:
      - email: 2          # Users can have up to 2 emails
      - device_id: 5       # Users can have up to 5 devices
  - name: email
    maximum_edges:
      - user_id: 1         # Each email belongs to only 1 user
  - name: device_id
    maximum_edges:
      - user_id: 2         # Shared devices (family tablets)

id_types:
  - name: anonymous_id
    maximum_edges:
      - email: 1           # Anonymous sessions link to 1 email
  - name: email
    maximum_edges:
      - user_id: 1         # Emails don't get shared between users

How rule application works

This section explains how cardinality rules are applied to your identity graph.

Fresh and incremental runs

Cardinality rules behave differently depending on whether you’re running a fresh or incremental run:

• Fresh runs: Rules are applied to all incoming data. If a node stays within its limits, all edges are preserved.
• Incremental runs: Rules consider historical violations. If a node previously violated a rule or if new data causes a violation, all edges from that source are removed, including previously valid ones.

Fresh run

# Input data
user_1 → email_a
user_1 → email_b  # Valid: within limit of 2

# Result: Both edges allowed

Incremental run

# Previous run: user_1 had 2 emails (at limit)
# New data: 
user_1 → email_c  # This causes violation

# Result: ALL edges from user_1 are broken (including previous ones)

Example

Given these rules:

id_types:
  - name: user_id
    maximum_edges:
      - email: 2
  - name: email  
    maximum_edges:
      - user_id: 1

And the following data:

user_1 ↔ email@example.com
user_2 ↔ email@example.com  
user_3 ↔ email@example.com

What happens

1. email@example.com connects to 3 user_ids, violating the email → user_id: 1 rule
2. All three connections are broken, not just the excess ones.
3. All nodes are preserved as isolated stubs.
4. Three audit entries are created documenting the violation.

Monitoring and debugging

This section explains how to monitor and debug cardinality rules.

Cardinality Audit table

Each violation is logged in the cardinality audit table, which resides in the same schema as the ID Stitcher output tables. The table name is derived by appending the suffix _CARDINALITY_AUDIT to the name of the ID Stitcher model.

The audit table is created on both Snowflake and BigQuery and contains the following columns:

Sample rows

The following rows are excerpted from a qa_metadata_validation_user_CARDINALITY_AUDIT table after a run where two users (viol_user and mviol_user) each exceeded a user_id → email: max_edges = 2 rule. mviol_user saw one of the violator emails (mviol_shared@test.com) from two different input models, which is why its source_models array has two entries:

id1,        id1_type, id2,                    id2_type, model_hash, reason,                rule_details,                              run_id, source_models
viol_user,  user_id,  viol_a@test.com,        email,    ffffe585,   CARDINALITY_VIOLATION, {"current_count":3,"max_edges":2},         1,      [{"source_model":"events_app","first_seen_at":"2022-03-01 10:00:00.000"}]
viol_user,  user_id,  viol_b@test.com,        email,    ffffe585,   CARDINALITY_VIOLATION, {"current_count":3,"max_edges":2},         1,      [{"source_model":"events_app","first_seen_at":"2022-03-01 10:00:01.000"}]
viol_user,  user_id,  viol_c@test.com,        email,    ffffe585,   CARDINALITY_VIOLATION, {"current_count":3,"max_edges":2},         1,      [{"source_model":"events_app","first_seen_at":"2022-03-01 10:00:02.000"}]
mviol_user, user_id,  mviol_shared@test.com,  email,    ffffe585,   CARDINALITY_VIOLATION, {"current_count":3,"max_edges":2},         1,      [{"source_model":"events_app","first_seen_at":"2022-04-01 10:00:00.000"},{"source_model":"events_web","first_seen_at":"2022-03-25 08:00:00.000"}]

cardinality_info in the metadata column

When edge metadata logging is enabled (see Metadata and observability), the per-node METADATA column on the ID stitcher output reflects rule enforcement in two ways:

• Capped edges for non-violator nodes carry a cardinality_info field on the stub edge, recording the max_edges and current_count that caused the cap.
• Violator nodes (nodes that exceeded a rule and had all their edges removed) appear with an empty direct_edge_info.edges_by_type array. Their node_info is preserved so the node still exists as an isolated stub in the graph.

For example, both a single-source violator (viol_user) and a multi-source violator (mviol_user) collapse to the same shape in the metadata column — only node_info survives:

{
  "direct_edge_info": {
    "edges_by_type": []
  },
  "node_info": [
    {
      "first_seen_at": "2022-04-01 10:00:00.000",
      "source_model": ["events_app"]
    },
    {
      "first_seen_at": "2022-03-25 08:00:00.000",
      "source_model": ["events_web"]
    }
  ]
}

Because dropped edges no longer appear in the metadata column, the audit table’s source_models column is the only surviving trace of which input model contributed each violator edge. See Interpret the audit table for a debugging workflow.

Best practices

This section provides best practices for configuring the cardinality rules correctly in your Profiles project.

1. Start conservatively

Begin with lower limits and gradually increase based on your data patterns:

# Start here
id_types:
  - name: email
    maximum_edges:
      - user_id: 1

# Adjust based on audit results
id_types:
  - name: email
    maximum_edges: 
      - user_id: 2  # If you see legitimate shared emails

2. Monitor audit tables

Make sure to review the Cardinality Audit table regularly to understand:

• Which rules are triggering most often
• Whether legitimate connections are being blocked
• If limits need adjustment

3. Test with historical data

Before deploying rules to production:

• Run rules against historical data in a test environment.
• Analyze the audit results.
• Adjust limits based on findings.

4. Consider bidirectional rules

Make sure to consider both directions of a relationship:

# Often you want both directions
id_types:
  - name: user_id
    maximum_edges:
      - email: 2        # Users can have multiple emails
  - name: email
    maximum_edges:
      - user_id: 1      # But emails belong to one user

5. Add rules for shared identifiers

Cardinality rules help when a single identifier can legitimately appear across many users but you don’t want the graph to merge those users. Common cases:

• Shared corporate emails such as info@, support@, or hr@ addresses that show up in many sessions.
• Bot or test accounts that share a fixed anonymous_id or device_id across thousands of synthetic users.
• Sentinel and placeholder values such as 0, unknown, n/a, or a default device identifier that slipped past upstream filters and would otherwise stitch unrelated users into one cluster.

If a few well-known sentinel values are responsible for most of the spread, prefer adding them to your input filters so they never enter the graph in the first place. Cardinality rules are the right tool when the offending values are not known in advance and you need the graph itself to flag them.

6. Interpret the audit table

To triage cardinality violations:

1. Group by reason and model_hash to confirm the violations belong to the rule you expect. rule_details carries the configured max_edges value and the current_count the source node reached, so a row with current_count: 12, max_edges: 2 is qualitatively different from one with current_count: 3, max_edges: 2.
2. Sort by id1 to find the worst offenders. A node that appears as id1 many times is over-connected on one or more target types; the count of distinct id2 values for that id1 reflects how many edges triggered the violation in the current run.
3. Use source_models to trace each dropped edge back to its input. This is the only surviving source-attribution trace for violator edges, because the per-node direct_edge_info.edges_by_type array is emptied when a node violates a rule. If a single id2 carries multiple source_models entries, multiple input models independently reported the same edge — useful for deciding whether the offending edge is a data-quality problem in one source or a real cross-source pattern.
4. Cross-reference with the metadata column. For non-violator nodes that share an id2 with a violator, look at the cardinality_info field on the stub edge to see whether the rule capped the edge or whether it survived intact.

7. Trace source attribution in the metadata column

For nodes that did not violate a rule, the metadata column carries full source attribution per edge. The direct_edge_info.edges_by_type[].edges[].sources[] array lists every input model that contributed the edge, along with the timestamp each one first observed it. Combined with the audit table for violators, these two sources provide complete source-attribution coverage across the graph.

See Metadata and observability for the full per-source contract.

Troubleshooting

Debugging

• Check rule configuration in pb_project.yaml.
• Query audit table to see what is blocked.
• Review stub metadata in the final ID graph.
• Compare fresh vs. incremental run results.

Migration guide

This section provides tips on managing cardinality rules in your Profiles models.

Enable rules on existing models

1. Add rules gradually: Start with one rule and observe effects
2. Test in development: Use a copy of production data to validate
3. Monitor after deployment: Watch audit tables and graph metrics
4. Adjust as needed: Fine-tune limits based on real-world results

Disable rules

To stop enforcing a rule, simply remove it from your configuration:

# Before
id_types:
  - name: email
    maximum_edges:
      - user_id: 1

# After (rule removed)
id_types:
  - name: email
    # maximum_edges section removed

This triggers a complete fresh run of the model. Historical violations are no longer enforced, but previous audit records remain for reference.