Summary of Outgrowing a single `dbt run`

00:00:00 - 00:25:00

The speaker discusses how to move away from using a single "dbt run" to a more scalable model that can handle concurrent data ingestion. He provides tips on how to stage data for use in a warehouse and how to use an orchestration tool to manage your staging data.

• 00:00:00 In this talk, prathik discusses the different types of orchestration used in data engineering, discusses why proactive orchestration wasn't working for VOX Media's DVD project, and provides tips on how to orchestrate your data projects.
• 00:05:00 The author discusses their experience of transitioning from a single DBT run to a more scalable model using a combination of DVT and dialectical dialectical behavior therapy. He advises not overcomplicating things early on in a project's development, and notes that running queries on the weekends is not ideal.
• 00:10:00 The speaker discusses how their company has grown and evolved over time, specifically mentioning that they have transitioned away from using disconnected Cron schedules in order to account for the varying schedules of their data sources. They go on to say that any data source should be accounted for, including Downstream use cases such as abandoned cart messaging.
• 00:15:00 The video discusses how to stage data for use in a warehouse, and how to run staging models whenever new data is available. It also mentions reactive orchestration, an orchestration technique for handling concurrent data ingestion.
• 00:20:00 The author provides a tutorial on how to outgrow a single "dbt run" and how to ensure that all of your staging data is always fresh and ready to go. He also explains how to use an orchestration tool to manage your staging data.
• 00:25:00 The speaker discusses how successful data management for DBT can involve navigating the tension between upstream systems that place freshness constraints on what can be done with data, and downstream users or use cases that demand certain freshness slas. He provides recommended reading material and encourages attendees to stay for the following talk, which will cover how to build a data team like a community.