How to Use Databricks Metric Views in Lookout

Rudder Lookout is currently in Public Beta. Contact the Product team if you have any questions.

Lookout reads the metric views your data team defines in Unity Catalog. It then answers questions and builds audiences with the governed descriptions, display names, and synonyms on those views.

Trace a metric view through Lookout

Lookout reads the definition, not your data. It runs two metadata queries, and only when a question needs them. One query lists the metric views in the connection’s catalog. One query per view reads that view’s descriptions, display names, and synonyms. Lookout screens and caps that text, then matches your words to the view’s fields.

When one field matches, Lookout writes the answer or the audience rule with the real field name. When a word fits two fields, Lookout asks which field you mean.

Both queries read catalog metadata only. Lookout creates nothing in your Databricks workspace.

Lookout runs these queries from Chat and Ask Questions and from Audiences and Activation. Lookout in Slack does not read metric views yet.

Connect a Databricks catalog

Lookout reads metric views only in the catalog on that connection. One connection covers one catalog. When the views sit in two catalogs, add a connection for each.

Only an admin can add the connection — see Roles and Permissions for more information.

1. Define the metric views in Unity Catalog. Display names and synonyms need YAML version 1.1. On a cluster, that metadata needs Databricks Runtime 17.3 or above. A SQL warehouse already runs a current Databricks SQL version, so you don’t pick a runtime for it. On version 0.1, Lookout still reads each column’s Unity Catalog comment. See Metric View Feature Availability in the Databricks documentation.
2. In Lookout, go to Settings > Connections > Warehouses.
3. Select Add account, then Databricks.
4. Enter the Workspace Host and HTTP Path. Choose an Authentication type, then enter a Personal Access Token, or an OAuth Client ID and OAuth Client Secret.
5. Set Catalog to the catalog that holds the metric views. The form marks it optional, but Lookout reads metric views only when it is set.

If your warehouse restricts network access, allow the Lookout IP addresses before you test the connection. See Allow Lookout to reach your warehouse.

Grant catalog privileges

The principal on the connection needs to see the metric views and read them. Display names and synonyms need no extra privilege. They come back in the same read as the definition.

Grant those privileges to the principal Lookout uses. Replace main, sales, and the principal name with your own:

GRANT USE CATALOG ON CATALOG main TO `lookout-service-principal`;
GRANT USE SCHEMA ON SCHEMA main.sales TO `lookout-service-principal`;
GRANT SELECT ON SCHEMA main.sales TO `lookout-service-principal`; -- or ON VIEW, one view at a time

Write a metric view

Lookout uses the same fields your team already writes for Databricks AI/BI dashboards and Genie. Nothing in the YAML is specific to Lookout. See Agent Metadata in Metric Views in the Databricks documentation.

CREATE OR REPLACE VIEW main.sales.orders_metrics
WITH METRICS LANGUAGE YAML AS $$
version: 1.1
source: main.sales.orders
fields:
  - name: order_status
    expr: o_orderstatus
    comment: "F = fulfilled, O = open, P = partial"
    display_name: Order status
    synonyms:
      - fulfilment state
      - status
measures:
  - name: order_count
    expr: COUNT(1)
    synonyms:
      - orders
      - number of orders
$$;

A few rules keep the match unambiguous:

• Give each word to one field. If two fields share a synonym, or one field’s synonym is another field’s name, Lookout asks which field you mean.
• Keep synonyms short. Lookout drops any synonym over 100 characters.
• Keep personal data and credentials out of names and synonyms. Lookout withholds any term that looks like one, and says it did.

Lookout reads only catalogs, schemas, and views whose names use ASCII letters, digits, and underscores, and don’t start with a digit.

Confirm Lookout can read the metric views

Three questions in the web app’s chat confirm the setup. Replace the names with your own.

1. “Which metric views are in the main catalog?” Lookout lists them, with their owners.
2. “In main.sales.orders_metrics, what does fulfilment state mean?” Lookout names order_status, quotes its description, and says it matched the synonym “fulfilment state”.
3. Ask with a word you gave to two fields. Lookout asks which field you mean.

If any of these fail, see Troubleshooting for help.

Review what Lookout reads and keeps

• Lookout reads the list of metric views in the catalog, and one view’s definition at a time.
• Lookout doesn’t cache a definition between messages. When a later message needs the view, Lookout reads it again. Within one reply, Lookout reuses a definition it already read.
• Like any query result, the definition becomes part of the chat’s history, which Lookout stores encrypted. Delete the chat to remove it from the history. Memories that Lookout proposes are separate. Delete those in the Context Hub.

When a limit applies, Lookout says so in its notes.

RudderStack support can stop Lookout from reading display names and synonyms for a workspace. The same switch also turns off Unity Catalog table tags. Lookout still reads the rest of each definition. The change takes effect from the next message, within about a minute. Words already in a chat stay in that chat.

Troubleshooting

Next steps

• How to Connect a Warehouse to Lookout if this Databricks account is not connected yet
• How to Build an Audience with the Agent from the metric view
• Security and Data Handling for how Lookout treats warehouse data