Summary of Dagster: A New Programming Model for Data Processing | Elementl

00:00:00 - 00:35:00

Dagster is a new programming model for data processing that makes it more efficient and reliable. The model separates business logic from environmental concerns, and uses materializations to represent side effects. Dagster is open source and can be downloaded and tried out. The team is looking for partners to help them build the system.

• 00:00:00 The speaker, Nick Shrock, discusses Dagster, a new programming model for data processing. Dagster is an open source project that aims to make data processing more efficient and reliable. Shrock describes how data scientists often have to repeat tasks that should be handled by someone else, and how businesses are struggling with data failures. He suggests that data cleaning refers to a wider range of activities, including engineering and data science work, and urges listeners to raise concerns with their leaders.
• 00:05:00 The video discusses the Dagster programming model, which is designed to make data processing more manageable and easier to work with. The model acknowledges the complexity of the problem and focuses on gradual adoption, preserving existing tools, and providing value gates to help increase productivity.
• 00:10:00 Daggett is a new programming model for data processing that provides a better local development experience. Daggett serves as an environment for developing data applications. The model includes a pipeline, a solid, and metadata.
• 00:15:00 The video discusses the Dagster programming model, which separates business logic from environmental concerns. The video demonstrates how Dagster can be used to automatically persist intermediate materializations of a pipeline, and how it can be used to debug pipelines.
• 00:20:00 The Dagster platform provides a suite of tools for data processing, including a Python library and an API. The platform is designed to be easy to use and extend, and can be used by developers of all skill levels.
• 00:25:00 The video discusses the "paper mill" model of data processing, which allows for easy integration of data science and engineering workflows. The model is based on the idea that data scientists and engineers have different, but overlapping, roles.
• 00:30:00 The Dagster programming model provides an elegant, functional, and incremental approach to data processing. It integrates with existing tools and infrastructure, making it easy to deploy and adopt.
• 00:35:00 Dagster is a new programming model for data processing that uses "materializations" to represent side effects. Dagster is open source and can be downloaded and tried out. The Dagster team is looking for "two to three teams" to work with deeply embedded, and is seeking a "design partner," "founding customer," or "founding partner" to help them build the system.