Flaky tests and developer experience in software teams | Pakka Nimbu Podcast 04

This conversation discusses the challenges of dealing with flaky tests and the impact they have on the development process. The speaker shares their experience of struggling with identifying and fixing flaky situations, as well as the psychological and operational impact it had on them. They explain how they decentralized the problem by defining test health and flakiness, building a service catalog, and routing alerts to the appropriate teams. The speaker also discusses the importance of approaching engineering problems with calmness and the need to improve code organization. They highlight the efforts made to speed up the CI suite and reduce test coupling. The conversation concludes with a discussion on introducing static analysis tools and the cultural change and adoption process. In this conversation, Puneet shares his experience of implementing changes to improve the CI/CD process. The transfer of ownership from the central team to developers was facilitated by minimizing additional work through documentation and providing examples. Quantifying the impact and value of the changes helped in gaining developer buy-in. Enforcing ownership and responsibility was driven top-down by management, with metrics and performance indicators supporting the shift. The differentiation between critical and non-critical tests allowed for targeted alert routing and escalation. The involvement of managers and automation helped create urgency and behavior change. Puneet advises focusing on solving patterns rather than instances and investing time in learning from others' experiences. This episode is also available as podcast on all major platforms, including Spotify, Apple Podcasts, and Amazon Music. This episode's sponsor is ⁠⁠⁠BlueJay by 10xEngg⁠⁠⁠, they are building a very promising product "Bluejay" that makes it easier for developers to create and manage alerts. If you aren't using them, you are missing out!! It automatically handles the creation of alerts and adds a ton of intelligence to them going forward! Request a demo from them TODAY! Chapters in this video: 00:00 Introduction 00:59 Challenges with Flaky Tests 02:23 Decentralizing the Problem 04:17 Defining Test Health and Flakiness 05:16 Building a Service Catalog 06:16 Approaching Engineering Problems with Calmness 07:44 Routing Alerts and Ownership 08:39 The Impact of Routed Alerts 09:09 The Challenges of Alert Routing 10:34 Improving Code Organization 11:33 Identifying and Fixing Flaky Situations 16:19 Predicting Test Failures 17:18 Debugging Flaky Tests 18:16 Struggling with Building a Complete Process 19:44 Escalating the Problem to Developers 21:09 The Impact of Flaky Tests on Production 23:36 The Inconvenience of Flaky Tests 25:02 The Cycle Times of CI Runs 26:27 Speeding Up the CI Suite 28:21 Improving MongoDB Performance 30:40 Reducing Test Execution Time 32:08 Reducing Test Coupling 35:30 Introducing Static Analysis Tools 39:19 Adopting Linters and Formatters 41:42 Cultural Change and Adoption 46:29 Improving Pre-Commit Checks 47:57 Transfer of Ownership 48:26 Minimizing Additional Work for Developers 49:26 Documenting for Easy Learning 50:50 Quantifying the Impact and Value 51:48 Enforcing Ownership and Responsibility 52:43 Metrics and Management Support 53:13 Measuring Flakiness and Ownership 54:09 Creating Urgency and Behavior Change 55:07 Differentiating Critical and Non-Critical Tests 56:06 Failure Scenarios and Ownership 57:36 Escalation and Manager Involvement 59:01 Democratizing Alerts and Involvement 59:56 Differentiating Infrastructure and Application Failures 01:03:20 Solving Patterns, Not Instances 01:05:46 Learning from Others' Experiences 01:06:15 Investing Time in RCA and Learning 01:07:34 Avoiding Overcommitment and Seeking Feedback Find me on other platforms: LinkedIn: ⁠⁠⁠https://www.linkedin.com/in/bkdonline/⁠⁠⁠ Twitter: ⁠⁠⁠https://twitter.com/PakkaNimbu⁠⁠⁠ Telegram: ⁠⁠⁠https://t.me/pakkanimbu⁠⁠⁠ Youtube: https://youtube.com/@pakkanimbu