Summary of Keynote - Data Mesh by Zhamak Dehghani

00:00:00 - 00:35:00

Zhamak Dehghani discusses the importance of data mesh technology in effectively managing data and transitioning to a more connected world. She explains how data mesh works and some of the challenges associated with migrating to this new system.

• 00:00:00 Zhamak Dehghani is a principal consultant at ThoughtWorks, and she will be speaking about data mesh today. Data mesh is a new paradigm in data management that aims to improve the reliability and scale of big data systems. Dehghani discusses the trend of big data platform failures in the US and how this inspired her to explore data mesh. She shares data from an annual report by NewVantage Partners that shows that executives' confidence in big data is decreasing. Dehghani explains that data mesh is a new paradigm designed to improve the reliability and scale of big data systems.
• 00:05:00 This video discusses how the world of data is divided into two planes: operational data, which is databases that capture the current state of applications, and analytical data, which is data that is used to optimise the business. The architecture behind data management for analytical data is inspired by water analogies, which is one of the reasons data mesh was not named after a water analogy. Data mesh is a technology that helps to move data between these two planes by copying data from operational systems and transforming it into a virtual data format.
• 00:10:00 Zhamak Dehghani discusses the challenges of building big data platforms using a centralised, file-based architecture, and how data mesh can help solve these problems. He explains that while data mesh is not new, the concept has been articulated in a more unified way and provides a set of principles that can be used to solve various big data platform challenges.
• 00:15:00 Zhamak Dehghani discusses the importance of domain-driven architectures, and how data mesh can help organizations decompose data and align it with the domains within their businesses. He also introduces the concept of data products, which are microservices that act as a historical view of a customer, and are accessible through an API.
• 00:20:00 The presenter reviews how data products can be created and managed autonomously, through abstraction of complexity into the data platform layer. This layer is accessible through simple declarative interfaces, allowing data product developers to focus on the logic of transformation and cleansing rather than infrastructure provisioning. This model of federated governance allows for cross-domain communication and the minimization of global decisions.
• 00:25:00 Zhamak Dehghani defines the pillars of data mesh, which are domain oriented ownership and decomposition of the data, treating each data product as a product with quality metrics, a self-serve data infrastructure to create autonomy and remove complexity from each data product, and a federated governance that focuses on ecosystem thinking. He provides an example of an insurance domain, describing the need for a data product to produce daily snapshots of claims data, as well as an online claims system that consumes those data products. He also discusses the need for data scientists to be happy with data products that produce short-term and low-retention data, and the need for data products to produce events that can be consumed by other data products.
• 00:30:00 Zhamak Dehghani discusses the new paradigm of data mesh, which is a decentralized data management system that allows for multiple sources of data to be integrated and aggregated into a single product. He says that this new model of governance is necessary in order to embrace change and move away from the past model of centralized systems. He concludes by saying that we need to learn a new language in order to create this future.
• 00:35:00 Zhamak Dehghani speaks about the need for data Mesh technology, which will allow people to manage their data more effectively and as a result, create a more connected world. He also discusses some of the challenges associated with migrating to data Mesh, and how be aware of these traps in order to successfully transition.