Summary of Building a Data Platform from Scratch with dbt, Snowflake and Looker

This is an AI generated summary. There may be inaccuracies.
Summarize another video · Purchase summarize.tech Premium

00:00:00 - 00:25:00

In this video, Prateek Shrotriya discusses the basics of building a data platform from scratch. He covers the layers of a data platform, the importance of adaptability, and the impact of data downtime. He also provides key takeaways for modern data management, including setting up a data pipeline and addressing data latency.

  • 00:00:00 In this video, DBT Labs' Ben Warshaw and Monte Carlo's Pratik Chawla discuss how to build a data platform from scratch. They highlight the importance of data ingestion, data processing, and data storage, and discuss some of the tools available to help with these tasks. Finally, they discuss orchestration and workflow automation.
  • 00:05:00 The speaker discusses how they built a data platform from scratch, starting with the ingestion layer and moving on to data transformation and modeling, data discovery, and data platform observability. They say that while their platform was not perfect, it was very scalable and maintainable due to their use of DBT.
  • 00:10:00 The data platform described in this video has remained largely unchanged since its inception two years ago. Improvements have been made to the underlying architecture, including the use of Snowflake for data ingestion and managed tables for faster Transformations.
  • 00:15:00 Data Platforms are becoming more complex, and in order to keep up with changes, testing is not enough. Data observability tools help identify when data is incorrect or unavailable.
  • 00:20:00 This video introduces dbt, Snowflake, and Looker, three software tools for building data platforms. dbt tracks data freshness, distribution, and volume, while Snowflake and Looker help understand data schema, lineage, and volume. Monte Carlo provides holistic application observability across all data assets.
  • 00:25:00 In this presentation, Prateek Shrotriya discusses the basics of building a data platform from scratch. He covers the layers of a data platform, the importance of adaptability, and the impact of data downtime. He also provides key takeaways for modern data management, including setting up a data pipeline and addressing data latency.

Copyright © 2024 Summarize, LLC. All rights reserved. · Terms of Service · Privacy Policy · As an Amazon Associate, summarize.tech earns from qualifying purchases.