Summary of The Whys and Hows of Database Streaming

00:00:00 - 00:50:00

This video explains the reasons why databases should be streamed in real time, the challenges of currently streaming data, and how Kafka can be used to solve these problems. The video also discusses the use of a materialized view to avoid querying the underlying table, as well as some ongoing work on streaming Cassandra into BigQuery.

• 00:00:00 The speaker discusses the reasons why data should be streamed in real time, describing the challenges of currently streaming data and how air flow can be used to improve the process. The talk also covers the use of a materialized view to avoid querying the underlying table, as well as some ongoing work on streaming Cassandra into bigquery.
• 00:05:00 The video discusses the problems associated with database streaming, including the reliance on microservice owners to not delete data, the error-prone nature of the process, and the potential for data inconsistency. Kafka is used to solve these problems, as it allows for streaming of data from databases without the need for a two-phase commit or a distributed transaction. However, there is one potential issue with this approach - stale data being read by users.
• 00:10:00 The BGM (BigQuery metastore connector) is an open source project that helps connect databases in a streaming fashion. This can be used to improve write performance, crash recovery, and replication.
• 00:15:00 The video explains how a database streaming connector works and how it takes data from a Kafka topic and puts it into a BigQuery dataset. The connector is able to handle failover and has a distributed mode for high throughput.
• 00:20:00 The video discusses how a database streaming system can be designed to be resilient to failure. The system includes features to improve data deduplication and compression, and to determine which data to show to the user. The system also uses Kafka to store schemas.
• 00:25:00 In this video, the presenter explains how Cassandra's peer-to-peer replication model makes it difficult to implement change data capture (CDC), and how Kafka can be used to address this issue. The presenter also discusses how a commit log is used to ensure that data is always available when a node in the cluster becomes unavailable.
• 00:30:00 This video explains how Cassandra's commit log can be used to stream data to Kafka, and how the Cassandra pipeline can be adapted to handle incomplete change of event data.
• 00:35:00 In this video, the presenter describes how a stream of change event can be used to efficiently query data from a Cassandra database. The presenter also discusses the costs and benefits of this approach.
• 00:40:00 The speaker discusses why they decided to use Cassandra as their database for streaming data pipelines, noting that Cassandra offers better distributed transaction capabilities and more reliable read consistency than bigquery. They also mention that it can be difficult to keep Cassandra and streaming data pipelines in sync, and suggest using events to communicate between the two.
• 00:45:00 The video discusses the differences between bigquery and a database, and how Cassandra might be a good option for streaming purposes. It discusses how Cassandra's peer to peer architecture could make it difficult to ensure data is not lost in a network failure, and discusses the costs and benefits of using the windowing and stream processor.
• 00:50:00 The presenter discusses the benefits of using a database streaming feature in Cassandra, which allows for more consistent data storage.