A/B Testing Secrets from LaunchDarkly | Why Data Can't Replace Human Curiosity

What if your dashboard is lying to you? 📊❌ In this episode of UX Stories, host Lihong Hicken sits down with Aaron Montana, Head of Experimentation & Product Analytics at LaunchDarkly, to explore the real intersection of engineering, A/B testing, and user research — and why the future of testing might include machines testing for machines. 🤖 Whether you're a product manager, engineer, or UX researcher, this conversation will challenge the way you think about features, data, and what it really means to understand your users. 🕐 TIMESTAMPS 00:00 – Welcome & Introduction 01:20 – What a dashboard can never fully explain 💡 02:05 – The role of human curiosity in experimentation 03:05 – What is LaunchDarkly and who uses it? 03:27 – Why feature flags are the best foundation for A/B testing 05:45 – Bridging the gap between engineers and marketers 08:25 – Quantitative vs. qualitative: what's the right sequence? 09:25 – How to design a research study from scratch 11:15 – Qualitative finds problems; quantitative solves them 15:48 – The psychological challenge of killing your own feature 😬 18:45 – Why high-velocity teams are already running experiments — just badly 22:14 – Vibe coding vs. vibe testing: the speed gap 🚀 24:50 – Is qualitative research becoming too expensive? 26:20 – Synthetic data and AI panels — promise vs. reality 🤔 28:08 – Early-stage vs. late-stage testing: when to use AI 30:00 – Are we about to test for machines, not humans? 🤯 32:31 – The future: machines running experiments for machines 💬 If technology is the rain and our ecosystem is a garden — more jobs, more testers, more ecosystems to validate. 🌱 Like, Subscribe & hit the 🔔 if you're building products and want to ship with confidence — not just speed. UXResearch #ABTesting #LaunchDarkly #ProductManagement #UserTesting #VibeCoding #FeatureFlags #Experimentation #StartupGrowth #UXStories