Summary of Field-level lineage with dbt, ANTLR, and Snowflake

00:00:00 - 00:25:00

This video discusses how field-level lineage can be used to identify data issues and root causes. It shows how to use the Upstream and Downstream Dependencies table in Snowflake to do this. Finally, it shows how to use DBT to calculate a metric that gives an estimate of the number of users for a given customer.

• 00:00:00 The speaker introduces DBT, ANTLR, and Snowflake, and describes how field-level lineage can be used to identify data issues and root causes. They also explain how data observability can help companies prevent data incidents from happening.
• 00:05:00 The Field-Level Lineage product offers a way to assess the downstream impacts of fields on other fields, as well as root cause analyzes for errors in those fields. Field-Level Lineage is built on the premise that field relationships are complex and need to be presented in a clear and concise manner. The product integrates with popular data analysis tools, including DBT Snowflake, bigquery, and others. Despite the complexity of building field lineage, awareness of the complexity is low, and existing solutions are not adequate.
• 00:10:00 The video describes how Monte Carlo built a field-level lineage for SQL, using dbt, ANTLR, and Snowflake. In order to achieve 80% coverage of the different SQL operators, the team added coverage for different classes of operators one by one.
• 00:15:00 In this video, Elena from the engineering team at Monte Carlo shows how field-level lineage works in Monte Carlo. She starts by showing the catalog page for the table, which includes field-level lineage for the four fields in the table. She then demonstrates how to select one of the fields and view the field-lineage for that field. Finally, she shows how to use field-level lineage to navigate through the table's data.
• 00:20:00 The video discusses how field-level lineage can be used to analyze the upstream and downstream dependencies for a field. It shows how to use the Upstream and Downstream Dependencies table in Snowflake to do this. Finally, it shows how to use DBT to calculate a metric that gives an estimate of the number of users for a given customer.
• 00:25:00 The speaker provides a brief overview of field-level lineage and how it can be used to calculate metrics. They then discuss how Monte Carlo can be used to check the accuracy of a model, and how DVD can be used to check the accuracy of a test. The speaker invites the audience to attend next week's Impact Summit, where they will hear from two speakers.