Summary of SREcon22 Americas - Principled Performance Analytics

This is an AI generated summary. There may be inaccuracies.
Summarize another video · Purchase summarize.tech Premium

00:00:00 - 01:00:00

This video discusses how performance analytics can be used to improve the reliability of services. It explains that the three critical components of reliability are availability, performance, and correctness, and that there is a lot of ambiguity in how these factors are measured. The video discusses how slos can be helpful in identifying and quantifying reliability issues, but notes that they have limitations in capturing the full richness of system failures.

00:00:00 The speaker at SREcon22 Americas describes work that they have done to improve reliability of Google Analytics services. They note that this work is difficult because it requires a deep understanding of what reliability means and how to measure it. They explain that the three critical components of reliability are availability, performance, and correctness. They mention that there is a lot of ambiguity in how these factors are measured and that much more needs to be done to improve reliability.
00:05:00 The video discusses how slos are used in the field of service management, and how they can be misused. It goes on to say that availability and performance slos can be difficult to measure, and that probers can cause more problems than they solve.
00:10:00 SREcon22 Americas discusses how principled performance analytics, or slos, can be helpful in identifying and quantifying reliability issues. Slos are error-based measurements that can help identify problems and optimize system performance. However, slos have limitations in capturing the full richness of system failures.
00:15:00 SREcon22 Americas discusses the need for reliable performance analytics, and how quantitative methods are important in order to understand the objective reality of systems.
00:20:00 The video discusses how service providers and customers view performance analytics. The service providers see aggregate data across all customers, while the customers see performance data for their specific queries. When the service fails, both the customer and service provider want the service just restored. To help service providers and customers understand each other's expectations, the video discusses how service providers view performance analytics, and how customers view performance of their queries.
00:25:00 The video discusses how a service should be consistent over time, and how performance and reliability can be measured. Service providers and customers need to share little information to measure reliability. To improve reliability, both sides need to be able to see performance across all workloads. This is an observability problem, because the two sides cannot observe performance across different workloads.
00:30:00 The video discusses a technique called "two-sigma" which uses historical data to approximate performance. The technique is used to figure out the expected performance of workloads with the same intent, day over day. If performance differs from the expected performance, it is an indicator that something has changed on either side. By approximating intent, cohorts can be created which approximate the intended workload. These cohorts can then be used to compute likelihood scores which can be used to determine if there is an issue on the service side.
00:35:00 This video explains that normal distributions do not accurately model performance, and that, in order to model performance, it is necessary to use z-scores. Z-scores are a good way to measure the likelihood of a given event happening, and, by monitoring the fraction of z-scores greater than two, a company can determine when a shift in the long tail indicates that a change in intent has occurred.
00:40:00 This video explains how performance analytics can be used to identify anomalous behavior in data distributions, and how simple clustering algorithms can be used to achieve good coverage. In some cases, it is possible to achieve good performance without needing to tune the algorithm.
00:45:00 This video discusses how SREcon22 Americas has implemented a more forward-looking approach to performance analytics, which has noticed that in the nine months that they've been running the approach on production data, they've only found one false positive. Everything else that they've found has been accurate, even if the user wasn't able to see it in their current monitoring. This is a tall order, and every time that they implement the approach they go through a very extensive back testing process. This data was from a real service, and you can see how the distance between the two lines is 18 hours, which shows the sensitivity of the system.
00:50:00 The video discusses a two-sigma analysis approach for performance analytics, which helps to identify problems and fix them before they become more complex and difficult to fix.
00:55:00 This video discusses how performance analytics can be used to diagnose problems in systems, and how monitoring and understanding system excursions can help identify which parts of the system are impacting performance.

01:00:00 - 01:05:00

This talk provides an overview of how performance analytics can be used to answer questions about how a system is performing, how it changes over time, and how reliable it is. The speaker also mentions that performance data is often more useful than availability data, and that there are many more invariants that can be validated.