Summary of Workshop: Get more out of your DAG

00:00:00 - 00:40:00

This workshop covers how to get more out of a DAG by recording when models are materialized and how long it took, building on that data, and implementing a warning when models reference low maturity parent models. The presenter demonstrates how to use a macro to store materialization results and how to use logical checks to determine if a table exists.

• 00:00:00 This workshop will teach attendees how to use macros and deep structures in DBT, as well as how to enforce model referencing rules and perform Dynamic modeling. Attendees are expected to be familiar with DBT and macros, and have experience working with DBD Cloud.
• 00:05:00 The workshop discusses how to scale a DBT project and address pain points that come with growth. The first goal is to record when models are materialized and how long it took, and the second goal is to build on top of that data. The third goal is to implement a warning when models reference low maturity parent models.
• 00:10:00 The workshop discusses how to get more out of your DAG, which includes access to the results object and metadata for every event that happens as part of an operation. You can use these results to debug and understand model materializations, as well as iterate over the list of results.
• 00:15:00 The presenter demonstrates how to create a macro to store materialization results, and how to use logical checks to determine if a table exists and, if not, to create it.
• 00:20:00 In this workshop, attendees learn how to get more out of their DAGs, by crafting a select statement and executing it as part of the on run hook. They also review statement blocks and load result functions, and see how they can use them to pull data from a data warehouse.
• 00:25:00 The presenter demonstrates how to use a reporting model to extract data from the executed application. The model includes a unique list of departments and a for loop to sum execution time for each department.
• 00:30:00 The workshop discusses how to materialize a model and its parent has low maturity. This can help contributors be more mindful when their upstream data have not been well characterized while tested well documented. The last piece of this puzzle is the graph object and its one useful characteristic is that within nodes, there is a nice Python dictionary where the keys are the unique ID for each node and the values are the node.
• 00:35:00 The presenter covers the goals of today's workshop, which involve identifying nodes and models for a given model of interest, and adding them to a ref maturity warning macro. Additionally, they discuss how node objects contain check sums for models, and how materialization metadata can be used for multiple purposes. Lastly, they discuss how metrics can be developed using materialization models.
• 00:40:00 The presenters gave a presentation on getting more out of a DAG, and afterward, attendees were invited to submit questions or comments in the Slack channel. There are several other sessions scheduled after this one, including a Back to the Future-themed workshop.