You are viewing documentation for an older version.

Click here to view the latest documentation.

Identity Graph

This guide introduces you to the concept of identity graphs in Profiles and shows how to use them in your Profiles project.

Overview

Multiple identities is the biggest obstacle to getting a full picture of your customers, user, or any other entity.

Companies gather user data across digital touchpoints like websites, mobile apps, enterprise systems like CRMs, marketing platforms, etc. During this process, a single user is identified with multiple identifiers across your systems, like email address, phone number, device ID, anonymous ID, user name, etc. To create a unified user profile, it is vital to collect and stitch all of those different identifiers into one canonical identifier.

You can leverage Profiles’ id_stitcher model to take your inputs and stitch all the identifiers together into an identity graph.

An identity graph consists of two main components:

• Nodes: Represent individual identifiers that belong to an entity.
• Edges: Represent the relationships observed between these identifiers in your input data.

Requirements

• Defined id_types for your entity and mapped to your input sources.
• Identifiers from your inputs must be of the string data type. If the columns in your input sources are not of string types already, you must cast them explicitly.

Usage

The identity graph is located in the profiles.yaml file as an id_stitcher model.

models:
  - name: user_id_stitcher
    model_type: id_stitcher
    model_spec:
      entity_key: user
      edge_sources:
        - from: inputs/rsIdentifies
        - from: inputs/rsPages

Link the identity graph to your entity

After you define an id_stitcher model in profiles.yaml, you must reference it from the corresponding entity in your pb_project.yaml file. Set the id_stitcher key on the entity to the model path (for example, models/user_id_stitcher for a model named user_id_stitcher).

Always set id_stitcher on each entity that has a custom ID stitcher model defined in profiles.yaml.

If the entity does not specify id_stitcher, the intended id stitcher is not used and you may get unintended results.

Example entity configuration in pb_project.yaml:

entities:
  - name: user
    id_stitcher: models/user_id_stitcher
    id_types:
      - user_id
      - anonymous_id

... # Rest of the configuration

See Entity for full entity configuration options.

Metadata and observability

To provide transparency into the stitching process, you can enable metadata logging using the following flags in your id_stitcher model specification. These flags control what metadata is logged for the identity graph components:

Both flags are supported on Snowflake and BigQuery. Other supported warehouses do not currently emit the METADATA column.

An example of how to enable these flags is shown below:

models:
  - name: user_id_stitcher
    model_type: id_stitcher
    model_spec:
      entity_key: user
      log_direct_edge_info: true
      log_node_metadata: true
      edge_sources:
        - from: inputs/rsIdentifies
        - from: inputs/rsPages

This metadata helps you understand how identifiers are connected and trace the lineage of your identity graph.

Metadata structure

The generated metadata contains two main objects:

The following example shows one node with a single email edge from one input model:

{
  "direct_edge_info": {
    "edges_by_type": [
      {
        "count": 1,
        "edges": [
          {
            "first_seen_at": "2022-01-01 10:00:00.000",
            "id": "s1_email@test.com",
            "sources": [
              {
                "first_seen_at": "2022-01-01 10:00:00.000",
                "source_model": "events_app"
              }
            ]
          }
        ],
        "type": "email"
      }
    ]
  },
  "node_info": [
    {
      "first_seen_at": "2022-01-01 10:00:00.000",
      "source_model": [
        "events_app"
      ]
    }
  ]
}

When the same edge is observed in more than one input model, each contributing model appears as a separate entry in the edge’s sources array. The edge-level first_seen_at is the earliest timestamp across all sources:

{
  "direct_edge_info": {
    "edges_by_type": [
      {
        "count": 1,
        "edges": [
          {
            "first_seen_at": "2022-02-10 08:00:00.000",
            "id": "ms_email@test.com",
            "sources": [
              {
                "first_seen_at": "2022-02-15 10:00:00.000",
                "source_model": "events_app"
              },
              {
                "first_seen_at": "2022-02-10 08:00:00.000",
                "source_model": "events_web"
              }
            ]
          }
        ],
        "type": "email"
      }
    ]
  },
  "node_info": [
    {
      "first_seen_at": "2022-02-15 10:00:00.000",
      "source_model": ["events_app"]
    },
    {
      "first_seen_at": "2022-02-10 08:00:00.000",
      "source_model": ["events_web"]
    }
  ]
}

Incremental runs do not merge metadata across runs

On each incremental run, the per-node METADATA column reflects only the source attribution observed during the current run’s data window. If a node was contributed by events_app in a previous run and only by events_mobile in the current run, the metadata column shows events_mobile alone — the previous run’s source list is not carried forward.

This applies to both node_info[].source_model and direct_edge_info.edges_by_type[].edges[].sources[]. The stitched ID graph itself (the id and other_id columns) does carry forward across incremental runs as expected; only the metadata column resets per run.

Reading the metadata column

The METADATA column is a per-row JSON document attached to every row of the ID stitcher output. Each row describes one identifier (other_id) and how it was observed during the current run.

The top-level structure is fixed:

• node_info is a deduplicated array, one entry per input model that contributed the node. Each entry carries the first_seen_at timestamp for that model and a one-element source_model array.
• direct_edge_info.edges_by_type is an array of buckets. Each bucket has a type (the ID type of the connected nodes), a count, and an edges array.
• Each edge in edges has the connected node’s id, the edge-level first_seen_at, and a sources array. The sources array carries one entry per contributing input model with that model’s own first_seen_at.

Multi-source edges

The same edge can appear in multiple inputs. The number of entries in an edge’s sources array shows how many inputs reported the edge in this run — the source_model and first_seen_at per entry trace the edge back to a specific table and time.

A single entry in sources means only one input model contributed the edge in this run — not that the edge is weaker.

Incremental run behavior

Two incremental runs of the same project can produce different sources arrays for the same edge, depending on which inputs landed new rows in each run.

When you compare metadata across runs, treat the sources array as a per-run snapshot, not as lifetime history. To preserve cross-run source history, snapshot the column into a downstream table on each run.

For more information on how violator edges interact with the metadata column under cardinality rules, see ID Graph Cardinality Rules.

Metadata size and limits

Each row’s METADATA JSON is bounded by structure, not by data scale. Two internal caps keep each row’s JSON small even for highly-connected nodes:

The two hardcoded limits are not currently configurable from YAML. In practice, the defaults are large enough for almost all production identity graphs. If you lose meaningful detail, consider whether ID Graph Cardinality Rules on the entity can tighten the underlying graph rather than expanding the metadata caps.

Detecting edges hidden by the cap

Each edges_by_type[] bucket reports the uncapped total in its count field, while the edges array is capped at 10. A row where count exceeds the length of edges means the metadata column shows a truncated view of that node’s neighbors.

You can run this check directly against the METADATA column without consulting the cardinality audit table — it complements the cardinality_info field used for rule-capped edges. See ID Graph Cardinality Rules for the rule-capped case.

Best practices

• For the identity_stitcher model, use only the id_types and inputs necessary to stitch the identifiers relevant to an entity. For example, if you don’t care about anonymous traffic, do not add anonymous_id as an id_type or any inputs related to anonymous traffic.
• Conceptually, you can think of your identity_graph table as a kind of fact table in star or Snowflake schema. All of your core identifiers are in the center table and are used to connect to your inputs.
• Only include inputs as edge sources that include id_types directly related to the entity (the first level out from the fact table in a Snowflake schema). For example, you may need to include an id_type for product names because they are in a separate table but you do not want to include them in the identity_graph.