Summary of AWS re:Invent 2021 - Introducing Amazon Kinesis Data Streams On-Demand Mode

00:00:00 - 00:55:00

Amazon Kinesis Data Streams on-demand mode is a new pricing model for the Kinesis data streaming service that eliminates the need to pay for data retention periods of up to 24 hours or 7 days. Instead, customers pay per gigabyte stored. This mode is adaptable, growing in nature, and does not put the responsibility for scaling on the user.

• 00:00:00 Kinesis Data Streams is a serverless streaming data service offered by AWS that allows you to ingest high volumes of data from a wide variety of input sources, store the data, and make it available for low-latency processing. Kinesis Data Streams also enables you to do multiple things with the same data.
• 00:05:00 Kinesis Data Streams allow for a high degree of parallelism and predictability in processing, making it easy to scale capacity when needed. Highly variable workloads can be handled by provisioning for peaks rather than averages, which results in less unused capacity.
• 00:10:00 The new feature, Kinesis Data Streams On-Demand Capacity Mode, automates the process of scaling streams as and when there is an increase in traffic, ensuring that streams are kept running smoothly and without disruption.
• 00:15:00 Kinesis Data Streams enables customers to manage their streaming data in a hands-free, on-demand capacity mode, while retaining the ability to scale up or down as needed.
• 00:20:00 The video describes how Amazon Kinesis Data Streams work and how they can be switched from on-demand to provision. The new stream creation experience allows for up to 200 megabytes per second of data throughput, while the provision stream experience scales up to 8 megabytes per second for writes and 4 megabytes per second for reads. The scaling mechanism is based on two key factors: headroom for burstiness and enough capacity to accommodate future increases in data traffic. If you want to add more consuming applications, you can use the enhanced fan-out feature.
• 00:25:00 The presenter introduces Amazon Kinesis Data Streams, describing how on-demand mode allows for the scaling of streams without the need to pay for fixed capacity. CloudWatch alarms are triggered when the streams' provision throughput is exceeded. The presenter then discusses pricing, explaining that data retention is included with the stream's per-hour charge, and that the cost of retaining data for 24 hours is bundled into the stream's data end pricing.
• 00:30:00 In the 2021 re:Invent conference, Amazon announced a new pricing model for its Kinesis data streaming service - on-demand mode. This mode eliminates the need to maintain separate pricing for data retention periods of up to 24 hours or 7 days, and now customers pay per gigabyte stored rather than per hour. In addition, a feature called long-term retention allows customers to retain data for up to 365 days for a reduced price of 2.3 cents per gigabyte per month.
• 00:35:00 The on-demand mode of Amazon Kinesis data streams enables you to use data for workloads that are unpredictable and have a cyclical pattern, while the provision mode is ideal for use cases where you want predictable costs and don't need high performance or availability. The on-demand mode has lower costs of ownership than the provision mode, but it may not be the best option for streams with a limit on the amount of data that can be written. Finally, the provision mode has no limits, making it ideal for test/dev workloads.
• 00:40:00 Amazon Kinesis Data Streams On-Demand Mode is available to all AWS customers, and includes the ability to move an existing stream to on-demand mode. The mode is adaptable, growing in nature, and does not put the responsibility for scaling on the user.
• 00:45:00 The presenter discusses how on-demand mode and resharding differ from provisioning and how resharding can speed up the process of adding shards to a stream. There is a possibility of throttling in on-demand mode if traffic increases suddenly and an unbalanced partition key is used.
• 00:50:00 The video explains that on-demand mode is more cost-effective than provision for high peak to mean workloads. Kinesis provides stable read write ordering, which allows for semantics like at least once processing to be retained.
• 00:55:00 In 2021, Amazon introduced Kinesis Data Streams on-demand mode, which allows customers to omit the need to buffer data before sending it to the stream. This mode can be more expensive than the provision mode, but it can provide more throughput.