Running Effective In-App Experiments — Giancarlo Musetti, Ad Hoc Labs

[00:00:00] **David:** Hello. I’m your host, David Barnard. With me today is RevenueCat CEO, Jacob Eiting. Our guest today is Giancarlo Musetti, Growth Product Manager at Ad Hoc Labs. Ad Hoc Labs makes several apps including Firewall and Dialed, but is best known for Burner, an app that allows you to create multiple phone numbers to protect your privacy and better manage communication. On the podcast we talk with Giancarlo about soft vs. hard paywalls, how to think about product experiments, and why removing friction from onboarding didn’t actually help. Hey, Giancarlo, thanks for taking the time to be with us on the podcast. [00:00:56] **Giancarlo:** Hey, David. Thanks so much for having me. [00:00:58] **David:** I want you to kick things off by talking about the company you work for, Ad Hoc Labs. It’s one of those companies, like FishBrain and so many others we’ve had on the podcast, where it’s a huge business but you don’t really know about it unless you know about it. So tell us a little bit about Ad Hoc Labs, and Burner, your flagship product. [00:01:16] **Giancarlo:** Ad Hoc Labs are the makers of Firewall, Burner, and Dialed. Firewall is a robo-calling solution. Burner is our flagship product, and it’s a second phone number app. You can get a second phone number, and make calls and texts from the app. It’s very popular for people working in sales. They want a specific area code, a specific city, etc. It’s popular for people who want to keep their business number private and have anonymity when they meet people. [00:01:59] **Jacob:** When I get sales calls that come from my tiny 3,000 person town, I know who to blame. [00:02:07] **David:** My wife has actually used Burner professionally. It’s one of the many apps we’ve found on the podcasts that I’m a fan of. She’s a licensed professional counselor, so when she puts her number out there she doesn’t want it to go to her cell phone, and doesn’t want callbacks to come from her cell phone. So yeah, it’s a super cool product. You mentioned, leading up to the podcast when we were chatting, dating. I hadn’t thought about that as a use case. It’s like if you’re putting your number out there on Tinder and other dating apps, you don’t want your real number that people are going to have forever. So it seems like there’s all sorts of use cases, professionally and personally, putting your number out on Craigslist or Facebook marketplace. There’s so many use cases to have that second number that you can just burn, which is cool, like the whole name and branding around Burner. Yeah. So you know, one of the things you put in, in your notes for our conversation today, it’s a top a hundred, 150, revenue grossing app in the app store. It’s a huge business. So a lot of people, have this need. how long have you been at the company and, and, you know, what have you seen growth wise, or like, what is really resonating with. [00:03:19] **Giancarlo:** Yeah. so I’ve been with a company for a year now. it’s funny that, you know, Burner had steady growth for, you know, several years. I mean, it’s been a top five revenue grossing app in the utility category, for, I think over five years now. And yet in the last couple of years, growth has like, has started. Grow even faster. and I think a lot of that is thanks to the experimentation, that we’ve been doing, which has been awesome to see, [00:03:52] **Jacob:** I want to know more about experimentation. Tell me what I gotta do to make my app go in select this clearly, whatever you discovered is universal and will apply everywhere. That’s a well-known fact in apps, so, [00:04:03] **Giancarlo:** Oh yeah, of course. [00:04:05] **David:** Yeah. So several, experiments that I wanted to dig into. one of the ones you, you talked about was a soft paywall. Can you tell us a bit about that experiment and what kind of results, you said. [00:04:17] **Giancarlo:** Definitely. So the soft paywall was, there was a time when you could download the Burner app and just create a sample Burner, without starting a free trial. And so generally we’ve been shifting towards this subscription model where people need to subscribe to use Burner rather than just paying one-off for a number. And so the soft paywall. Basically a seven day free trial, a paywall where you, you subscribe to a seven day free trial skew. And it’s a soft paywall because there’s a maybe later or, either a maybe later button or just some button at the bottom that lets you like pass the paywall and get into the app. And so the soft pay will we notice, really helped increase subscription. and then a little bit after that we tested the hard day wall and that’s where we removed, that like bypassing button and people just had to start a seven day free trial. and that’s where we, we started to see tons of growth. That’s kind of roughly when I joined the team. and then since then we have been bread crumbing our way from our previous revenue model, which is credits to subscription. Hmm. And so my time here has been focused on increasing monthly recurring revenue. and we’ve had several different experiments that have helped us get there. [00:05:47] **David:** Yeah. Before we move on to those experiments, I wanted to ask a few more followups. yeah, it sounds like, the hard paywall was done around the time that you got there. Do you recall, and have you talked to any of your colleagues, did apple hassle y’all at all? I know, like in the past, the hard paywall is something they’d been sensitive to like a free app has to have some kind of free component, like. When you say hard paywall, is there any functionality in the app at all? If you don’t start a free trial or subscribe, or is it just completely blocked? There’s nothing you can do in the app without subscribing. [00:06:24] **Giancarlo:** Actually if you start the free trial and then you decide you don’t want the subscription, You use prepaid numbers instead of subscribing. So subscriptions, aren’t the absolute only way to use burden. [00:06:36] **Jacob:** Did you have to start like some IEP trial to get past that screen though? Yes. Okay. Yeah. I mean, we’ve done that. I mean, I, this is not me personally, but elevate we’d done that. I mean, Apple’s okay with it, but it’s interesting because you can always like start the. And cancel and then, and then, and then you have to have it’s it’s it’s odd because at that point you have to have a free experience. Like you can’t like kick somebody back out, like once they’ve canceled a free trial, maybe you can, after it expires, I suppose, but, they’ve been okay with that. but it doesn’t. So, so the way you described it, it sounds like the soft paywall performed less well than the hard paywall. yeah. which this is one of these things though. And why testing and data are always so powerful over any sort of intuition, because you think like, oh, that’s a bad user experience. How this is going to turn a lot of people off. And you’re like, yes, but also you’re showing it to a lot more people just like the way that usage decay is an interest, the case so quickly in an app usage, like the more you push to the beginning, the more like, urgency kind of you drive for the user experience. It’s not necessarily in the book of like user-friendly UX. It’s not, oh, I wouldn’t say always it’s the most user-friendly right. Like user you most user-friendly we hear it’s a free app that, you know, it’s going to run out of money and crashing, you can really can only figure this out, through experimentation. there are some like high-level things that work, but, you know, every app is, if I’m in it’s interesting to me though, is I have a there’s this app called open phone, which is a. Phone number for small businesses. It’s like, if you’re a plumber, you run a small business, you can have this like shared phone number and stuff like this. This was a big problem. They actually had was people trying to do the Burner use case with their product because they had free trials upfront. And they were like, They were not tooled. They were not tooled to, accept this kind of usage right. Where it was like, get a phone number because because of the backing costs, like the underlying costs of getting a number and things like that, going back to the discussion, and maybe we can expand a little bit here on how you’ve navigated this transition of usage based or like consumption-based pricing to subscription. Isn’t there some difficulty with, you know, somebody wants to use a hundred numbers. Does the subscription package, like give them infinite numbers or is it like some number per month? Or how does that, how did you package it? [00:09:00] **Giancarlo:** Yeah, so. you, you can’t get infinite numbers. You right now can get either a one, one, three or 10 phone numbers, subscription package, and you can still purchase prepaid numbers in addition to that. So if you want seven numbers, you can get a three phone numbers subscription, and then a few other prepaid numbers, right? the way we package. There was some discussion around, should we be able to give them the ability to purchase any number of subscription numbers? Right. 4, 5, 6, 7, 8. And we basically opted not to do that because the UX would be kind of weird, [00:09:38] **Jacob:** Especially like upgrades and downgrades and, and, just skew management on your side as well. It’s such a pain. [00:09:46] **Giancarlo:** Right. And one thing we’ve noticed is that ski management does take some time, but it doesn’t take that much time. And so we’ve generally when we are considering creating a new skew, we’re not too worried about the ski management part of it. but we do kind of think about what’s, what’s going to actually help the user in the best way and also increase revenue. Right. And so we have. Launching the 10 line experiment. and that was basically a 10 phone number, subscription package, which is doing really well. a lot of users have liked that. I guess if you have habitual [00:10:23] **Jacob:** Use, you need as many as you can get. Right. you know, if you’re somebody who uses. Every day, like, is it 10 is probably not even enough. Right. [00:10:32] **Giancarlo:** Yeah. And it also depends on the use cases, right? Cause there’s people who have one phone number, they use that specific phone number every day, but then there’s people who are like dating for example, and they want a separate number for each person. Right. Because if one person becomes spammy, they want to delete that number, but they don’t want to delete all of the contacts they have. Right. And so the same, I’ve heard this in like the business use case as well, where someone. Has a ton of different contractors and, has a different number for each contractor and has like 50 different numbers, you know? So, one other takeaway there is talking to users is really interesting because you learn a lot about your. And how it’s used, especially for an app [00:11:14] **Jacob:** Like that, where the it’s a utility that has, a number of different applications. Right? did you line the subscription products up with like tiers of consumables you were already selling, like, did you on the consumer. Could you buy credits at any granularity? Like 1, 2, 10, 20, or like, did you have packages as well? [00:11:31] **Giancarlo:** Their packages for credits [00:11:33] **Jacob:** You align them like the subscription ones and the no. So you have like, they’re all offset from each other. [00:11:40] **Giancarlo:** Yeah. So the way it works is to buy a number. You can either buy a prepaid Burner or a subscription. Yeah. The prepaid Burners. There’s a list. There was like five of them. there was like a mini Burner, which would give you X number of text messages per month and Y minutes of calling. And so there were different types of prepaid numbers that offer different amounts of calling and texting in order to buy a prepaid number you had to buy credit. And there was like five tiers for buying credits. You can buy like 25 credits at one. Yeah. And the [00:12:12] **Jacob:** Credits had a different, like they had the different tiers of number had like a credit card. Yes, exactly interesting. That’s much more like a Zynga land. Like when I used to do free to play stuff. Right. You had like, you had some intermediary currency, right? Cause I, what, what I was going to, I was curious about is like, how did you discount? Cause because you know, people are gonna buy bundles of numbers. because they get some discount, right? Like why would I buy three? You know, assuming I’m not, maybe I don’t need three. Maybe I only need two. And then I lost money because I didn’t use all of them for the month. I was curious, like, how did you think about pricing for those? Like how did you think? Or like actually not even how you think, like right now, like how much do I save if I buy three on a subscription versus three ad hoc? [00:12:58] **Giancarlo:** Yeah, for sure. So generally subscriptions are going to be wasted. And we intentionally did that because subscription revenue is more reliable. It makes our business more valuable and it’s also giving a discount to the user. So it’s like a win-win situation. Right. and that’s kind of. The reason we’ve been kind of aiming towards increasing MRR. [00:13:20] **Jacob:** Yes, it is. Is it like 10%, 20%? Do you know offhand, like roughly, like what I can save by signing up versus, buying ad hoc. [00:13:29] **Giancarlo:** So just comparing the like one phone number plan to the 10 phone number plan is a significant discount. It’s like, it can be 15, 20. and then like going to the annual plan from the monthly plan for any like the three-line or the timeline is another is like 20% sure. So the ideas, you know, the more you pay upfront in like the Supreme. You’re going to save a lot more. [00:13:58] **Jacob:** I mean, you’re buying, you’re selling a futures contract, right? Saying, we’ll give you a discount if you commit, you know, if you put up this much capital now and then you’ll have access, and then it’s up to the consumer to decide like what their future consumption is going to be. But then you can afford to discount, right? And then, like you said, it gives your business a lot more price predictability and like, and stuff like that. I’ll just pause and say, y’all have a. Complicated setup compared to most apps. [00:14:24] **Giancarlo:** Totally. That’s what I [00:14:25] **David:** Was going to jump in on and say that like my eyes were glossing over trying to like, [00:14:32] **Jacob:** I’m grilling you, John Carlo and life. Exactly what your tears are. And so I’m trying to put it together, but this is [00:14:37] **David:** Great because this is something I think. A lot of apps and myself included are scared of the complexity. Like don’t force users to make choices, you know, too many skews is going to just confuse them. And so I’m curious your perspective because it sounds like, you know, hitting that inflection point recently was in part. Just trying to push everybody to subscription instead of having all that complexity is, am I understanding right? That part of that inflection, it was that simplification or is there still like a lot of skews and a lot that the consumer has to understand to use the product and subscribe? [00:15:14] **Giancarlo:** Yeah. I mean, I imagine the inflection, isn’t just. That specific experiment, right? Growth growth is happening. For many reasons. We have some, going on on the marketing side as well. We have a lot of stuff, in the experimentation world. The growth team’s working on. And then we also have things outside of that too. So I think it’s a mixture of different factors that’s causing the growth. And, and not just one thing, [00:15:40] **David:** But today is it, is there less cognitive load? Like how many skews do you present, in that initial, paywall, [00:15:49] **Giancarlo:** Right? Right. So in that initial paywall, we just offered. Right now a three phone number package that has a monthly and annual option. And part of the bread crumbing kind of plan of moving from the credit Allah cart model to the subscription revenue model. The hypothesis was that there are too many products, right? And that’s a little too confusing. And so one of the most successful experiments was removing a bunch of prepaid numbers rather than offering five different prepaid numbers. We just offered one, the one that was, it’s called the picture Burner, and kind of hat. Texting calling and you can send pictures. [00:16:31] **Jacob:** And when you say, when you say remove numbers, you mean these different tiers, like these different flavors of numbers with limits and capabilities. [00:16:37] **Giancarlo:** Yeah. Just like removing four products that you could purchase to buy a prepaid number. So rather than having five, which we did before the treatment in this experiment only had one and it just. Drastically outperformed control. Interesting. [00:16:56] **Jacob:** Yeah. So that [00:16:57] **David:** Makes sense. Cause all of that complexity, you and Jacob were going through about the prepay analysis. [00:17:03] **Jacob:** Oh, well, when I heard there were three different levels, I was like, oh boy, we just added another dimension. This one from a. a spreadsheet to a tensor real fast. [00:17:12] **David:** So, so that’s great to hear. So, so it was really that simplification process that that was the successful experiment. And so, so you initially present something very simple. And then if they decline that, then they can buy those prepaids and then they kind of so are how many different paywalls do you end up showing? Like, so once they, get rid of that initial paywall and they get dropped into the app where they can buy these prepaids, is that a completely separate paywall or do you still have the, free trial option with that single product monthly or annual. [00:17:48] **Giancarlo:** Once they go past the paywall, they can go to subscription management to manage their subscription, upgrade to submissions with higher numbers, or they can go to another place in settings to buy the prepaid number. [00:18:01] **David:** Gotcha. Okay. yeah. Yeah. That’s also fascinating. I, I, this is where I wish we had slides. We should be doing this, I guess, a webinar [00:18:11] **Jacob:** I’d love to like, have these, it’s all about [00:18:12] **Giancarlo:** Webinars now, [00:18:13] **David:** But I did want to talk through some of the, unsuccessful experiments. I think, you know, everybody loves to talk about, oh, this did a 10% lift that did a 25% lift, and this is what you should do, but understanding the experiments. Failed, I think helps inform, the kinds of experiments you want to try and, and just not enough people talking about the failures. So, there were a few here that you listed as failed experiments. So I’d love for you to talk through the, the annual subscription price testing and what failed about that experiment. [00:18:47] **Giancarlo:** Yeah, so that experiment was testing the current annual subscription price on the paywall 59 99, again, 49 99 and 69 99. and you know, the KPI for success here was roughly, I think like there had to be 15% more often, for the 49 99 package or 15%, no, less than 15%, 15% drop in opt in for the 69 99 to break even on the, price difference. And that unfortunately didn’t happen. We were just looking back at this experiment a couple of days ago. And it’s tough because you can actually determine whether the treatment is successful. You can estimate whether it’s successful by predicting the retention for the annual, subscribers. Right. But like for, you know, there was a significant increase in optics. For the 49 99 plan, right? You dropped the price of the annual skew. More people are opting in, but you can’t actually measure immediately, what the longterm benefit to the businesses, because you don’t know how that cohort of users is going to retain. Are they going to resubscribe after one year? Presumably it’ll be higher retention. 59 99 or 69 99. [00:20:11] **David:** Yeah. And then what was the control price you said was [00:20:15] **Giancarlo:** What? 59? 99, [00:20:17] **Jacob:** 60 bucks. so you saw some of the plasticity standard, right? Like you got higher conversion at a lower price and lower conversion at a higher price, but then, was it that you didn’t feel confident not knowing those renewal rates changes to make a decision or was, was the where the change is just not big enough to [00:20:35] **Giancarlo:** Justify? We estimated the retention and, you know, we landed on, oh, it looks like it’s roughly breaking. Yeah, but I, I’ve very excited to like revisit this a year from now and see if we’re wrong. Right. Cause that’s [00:20:49] **Jacob:** The beautiful cohorts, right. You actually started, we have the plan, so you can look at it, you know, but a year after it’s not really agile. [00:20:58] **Giancarlo:** Yeah. It’s not. And so we, obviously we try to be agile. Part of the, part of like the experimentation flow is we need roughly one to two weeks for an external. During onboarding to get statistical significance. and I like to generally run on-boarding experiments in parallel with experiments down in the funnel, as long as they’re not like price oriented or, going to like confound [00:21:26] **Jacob:** The result of wording around the thing you just showed somebody or something like this. Right? Yeah. that’s great. I mean, price test. I, when you first said like, oh, 49 99 at 69, 9 or 79 99, whatever it was, I was like, oh, those prices are too close. And like you, the closer you get, like the more data you need to show a significance. Right. [00:21:45] **David:** Might have huge volume too though. [00:21:47] **Jacob:** So, but there’s, there’s benefit there. If you can’t limit the uncertainty on the renewal rates, you’re only going to be able to, no matter how much conversion rate data you get, you’re only going to be able to get so accurate until you have like trustworthy renewal rate data and. It’s hard. Like the apps from, I looked at how like the, after my changed the way rules work in the next 12 months. Right? Like there’s all kinds of reasons to not like, you know, so you could, you know, you could have said like, we’ll assume that they’re the same, but it’s like, For the 49 99 case, you’re adding a marginal, whatever it is, 10% more users. Right. And you it’s, it might be a lot to assume that those users who were already on the price margin are going to behave in a similar way. Like I think it would be pretty foolhardy to be like they will be at or above the average. in terms of renewal behavior, as the folks who are all buying it, you know, 59 99. Right? So, you almost have to assume it’s going to be worse, right? Because they’re already marginal customers. so yes, price experimentation is our, we know that it’s like we, when we’re going on this pricing fermentation product for years, the easy part is running the tests. The hard part is the predicted LTV because there is just like a lot of parameters and. Often at the model’s very sensitive to, right. Like small changes in your renewal rate create can create huge swings in your lifetime LTB. and so it’s pretty easy. Like, let’s say you were testing $10 a year and $60 a year. You can get significance there pretty fast, but anything more fun. It gets close. There’s a, how did you like how you were talking about like different amounts of experiments you run? Like how does, how did that experience. Inform the kind of tests and things that you’ve you focus on now, because that’s what I always found with running a test is like, when you have a test that’s inconclusive, it’s not really a failure. I feel like that’s actually, it’s an all result is actually interesting because it tells you like, Hey, there’s, unless you just screwed it up, which happens. But like a lot of times it tells you like, this was a, a path that’s probably not super rich. Right. So like, how does it focus on, well, how did you change your philosophy after going through that? [00:23:55] **Giancarlo:** Yeah. I think philosophy is generally pretty, pretty much the same after that. Right. Because pricing, in my opinion is still very, very important, right? Like it’s, I think one of the biggest levers, and I think even though this experiment in the long run probably has roughly the same LTV for both of those cohorts There’s a lot of opt-in in one of the cohorts and not the others. And so that speaks to the power of price. Right. And that’s kind of backing up. the other experiments, we brought the soft paywall, the hard paywall removing specific prepaid Burner plans. And so. [00:24:33] **David:** Another experiment you mentioned was removing the screen that lets users choose their full number. I’m curious about that way, because again, I have used the product and I think it remember, was like years ago when I set up for my wife. and so I’m assuming it was like a whole screen where you get to like go through a bunch of different numbers. And I remember like, kind of like when I got the option to pick a number, I, you know, you spend time, like you get invested in like, oh, I want to find the best possible number with the right. You know, maybe I can spell something or whatever. so yeah, I’m curious about this specific experiment and then why you considered it a failed experiment. [00:25:12] **Giancarlo:** Yeah. So to give context, before we jump into the experiment, I break down in amplitude, which is a analytics platform for those that don’t know it’s, it helps you understand the user journey. And I break down each step of the funnel for onboarding, just to see what is the drop-off at each screen, right? Every screen has a drop off. There’s always some amount of users that stop using the app at every school. And so the idea is if the screen doesn’t add lots of value to the user let’s test removing it so that we can get less drop off on that part of the onboarding. And so there’s a screen. So the way the funnel works is you have a welcome screen. When you open the Burner up, you then enter your phone number. Then you select an area. And then after you select an area code, you get a list of a few different options with a full Burner number. And the idea here was choosing the area code is really important, but users probably don’t want their full number. They don’t want to select the full number. and so [00:26:23] **Jacob:** It’s a lot of numbers I have to pick. It’s like nine numbers. [00:26:26] **Giancarlo:** I have to pick lots of, lots of information and things you have to enter. And users are trying to like get into the app really quickly. And so like, let’s try to remove that friction. and then we tested that and the way we set up the experiment is you choose your area code and then it jumps you to the screen that usually shows after the full number screen, which is like a congrats. You have. and then we also added this little button at the bottom of the next button that basically takes them back so that if they want to go and select the full number, they can actually go do that. And so it kind of creates this loophole, like, okay. If someone really cares about the full number, that he can go back and choose that if not, they can click next and get to the paywall. But in the end, this didn’t actually outperform. and so the primary KPI here was free trial start. It’s very difficult to get like lots of data on what percent of each of users in each cohort are actually paying for a subscription seven days after the free trial start. So for onboarding, I think we were mostly looking at like the free trial opt-in as like a higher level metric to give us an indication of whether this was going to. Make us lots of money. And in the end we decided, to keep the control. [00:27:48] **David:** So I kind of spoiled the results because I mean, it was exactly like my use case is that after I spent a minute, like finding that perfect number and fussing around, like finding a number, I was like more invested and more likely to start a free trial. So maybe the cognitive load was actually creating a little bit more. Buy-in a little more sunk costs along the path to starting that free trial. Did, were you able to like measure, or did you look at as part of this experiment, like how much time was spent on that or like how many different numbers people cycle through before they moved to the next step? [00:28:27] **Giancarlo:** How much. I don’t think we have that data conversion [00:28:32] **Jacob:** Rate. That’s all you care about. You’re talking about your funnel. Yeah is just like, oh yeah, we’re on this page. How many of you have made it to the next page? And then ultimately how many people started a trial, right? Like that’s what you care about. and then, you know, the, the, the details are probably all, not at all, not that informative unless you, you know, maybe one day you’ll hire a product manager. Who’s like the product manager of phone, cell phone number selection. Right. Then they’ll have the time to dig into it. but it’s interesting, Funnels or especially in consumer, just so important. Right? You have only have a limited amount of time to make a pitch to somebody. And it really can just be how you present it is like, do I convert this person or not? Right. Like some amount of people probably were never going to convert. Some amount of people probably, always were, but there’s, I think, I don’t know what a number is, but there’s people on there that if you just presented in the right, I know I’m like this. Like I judge a product. Like this, if I’m three pages in and I don’t feel like, okay, we’re heading towards me solving my problem. I’m bounced. Like, give me, I’ll try another app I’m out of here. and especially if I’m not impressed with how the apps put together, I, it doesn’t bode well for the rest of the usage experience. Right. but, so it’s really important to study these pages, but nothing’s in a vacuum. I always find this with experimenting, with onboarding funnels, like you’ll take something. But it’s all, you know, you might just be you’re right in that there’s drop-off in every page, but it doesn’t necessarily mean that page is causing that dropout. Right? Like you pull that page out and then that dropout is going to spread between the page before and the page after potentially. so not always the case, right? Like you could put up a screen that is just like very confusing and that would probably call. More dropout. but you know, generally yes, the fewer pages is better, but also sometimes not if they’re really good pages. Right. and that like, like David was saying, it creates, anecdotal. we did this at elevate where we had a very, very light onboarding. When we launched, it was like three questions, sign up, bang, you’re done, and then launched you into some brain training grant games. And then we added this ma literally added 25 steps to the onboarding, like as an experiment, it was like a personalization test kind of thing. And I was like, this is not going to work. This is going to ruin the funnels. This is, this is going to make our app is everybody’s good, churn, whatever. So we AB that and. It drove down a lot of like number of users getting to the end, but it turned out it didn’t affect trial start rate. It didn’t affect, day one and retention. It didn’t affect a bunch of numbers that I was convinced it would ruin, but honestly, They were good pages and it was shaken off people that were going to turn anyway. Right. It was just moving some of that term further up. so anyway, it’s all goes back to the point. It was like, testing is important because there’s literally no way you could think about this and figure out what’s going to [00:31:11] **Giancarlo:** Happen. I’m curious. Did you end up keeping that 25 step onboarding? [00:31:16] **Jacob:** I haven’t used the app in a while, but yeah, it became a big, it was called on Borneo. It was a whole game. but, but it, it was this like questionnaire we had, you play like a few mind games, like, and then we actually, that, that became the seating data for what customized training you received after that. and it wasn’t like super crazy scientific, like, do you like, do you want to work on words? Okay. Yeah, you’ll get some more words. I’m sure it’s very scientific and whatever, but like we were trying to ship a and get it out and, but it created, it gave us, it was two things. One. It gave the opportunity for us to show people what the product was going to do more directly. Right. and I assume like numbers is kind of a good way to show that like, Hey, like we have control of the phone system and you can type in a phone number that’s kind of cool. So it showed people more what the product would do before they had to commit anything. Right. Like they were just playing. and then, to, yeah, it did kind of, turning people out that are not going to convert to meaningful users early or later, doesn’t really matter. Right. In fact, probably in the benefit of your company, if a user’s not going to be a high value or like good for your brand, or they’re not going to be satisfied, it’s probably better, you turn them out as early as possible. Right. and so by saying like, Hey, this is kind of what the experience is gonna be. People were like bounced. And so, yeah, it turned out it was a, it was a good move. It made the product, it didn’t affect any metrics negatively. It might’ve even, I don’t remember everything, but I remember being one of these things where Mr. Smart math me with like down, you know, like funnels and all this stuff. It was like, this is no way. There’s no way this is going to work. Right. I like to think of, onboarding funnels, like slides, like you just want users to be able, like if they. Take one step onto your onboarding. Like, whoops, suddenly I’m a user, right? Like it’s just like impossible not to. but I’ve since learned that that’s not always the case, especially I’ll just last complicating anecdote is in B2B, which is what I work in now. All that physics changes because people have like a very different experience with a product and you can actually have quite a bit of friction and it doesn’t affect things too. It may affect, like now this is, everybody’s going to tweet at me about how painful entitlements are in RevenueCat. And they’re like, I know, I know we’re going to fix it, but people will work over it. Right. And so like, but, yeah, I still think it’s, it’s good to have that mentality that like, make it as smooth as possible. But, you know, it, doesn’t always, it’s not always the case, right? Like, yeah. Sometimes a little bit of onboarding friction creates a better product, [00:33:37] **Giancarlo:** For sure. Yeah. And to David’s point, it sounds like that creates a little bit more commitment, right? When you have to choose your full, not only the area code, but like the full number. And so if anything, it, it makes me wonder, what else should we be adding [00:33:50] **Jacob:** Asking them like, Hey, what are you going to use? Well, maybe that’s not good for a thing called Burner. [00:33:55] **Giancarlo:** Yeah. Actually you considered that though. That screen what’s your use case. Yeah. I mean, it’s helpful for down down the funnel customization, like what type of content are we feeding them? What type of models are we sending? What, what’s the email flow and now we’re starting to really optimize for that. for CRM. and, and that’s going to be very exciting, [00:34:17] **Jacob:** I think for in, especially a product in a utility category. Like if you can like smartly integrate some of that stuff, that’s going to help on. Sure. You know, like it’s a great thought commoditize, you guys have a brand, but like, you know, I can set up a Twilio account and make a Burner thing. Right. But so like the more you invest in. Yeah. Getting people to lock in. There’s this it’s a billion years old now, but this like back in those Evernotes when they were raising their series B their deck leaked. And it was really interesting how they thought about. Inputting data creating a lock-in. Right. And they credited that a lot with their really great consumer subscription growth. And I think not enough apps, especially in probably utility categories, like think about smart ways to do that. Right. and so, yeah, th if you can figure out ways to get users to invest in your app, That’s going to just help you right down the line. So we’ve [00:35:07] **David:** Been talking a lot about, experiments and getting into specifics and tactics and everything, but I, I do want to take a step. And talk about the ideation for roadmap and experiments. yeah. I’d love to just get your thinking about like what experiments to run and, you know, sometimes it’s so often counter-intuitive like Jacob was saying, like, you don’t want to test the twenty-five page onboarding, but you probably should. So how do you think about like what crazy experiments to run? What, you know, outside the box thinking is maybe you get an actual. produce better results. And so how do you pick what to work [00:35:46] **Giancarlo:** On? Yeah, so it usually starts with a brainstorm and it’s important to me that the brainstorm has like a bunch of different people. Ideally an analytics person, a designer, hopefully an engineer, customer support people like I want everyone. Who touches the product to be included in that brainstorm a lot of the time. And that’s just really helpful because there’s so many different things you can possibly test and you never know where the next great idea is coming from. And so that’s like, step one is having that brainstorm. And then from there, discussing technical complexity with engineering to figure out how long will each of these take, right. Cause there might be something that’s. Medium to high opportunity, but insanely high complexity and it’s going to take months to build. Right. And so at that point, is that worth it? Or is the high, like the medium to high opportunity with like a very low lift worth it, right. Like that second option is going to be where you’re going to want to start. So as an engineer, you’re [00:36:51] **Jacob:** Going to want to find that out early in the process. [00:36:54] **Giancarlo:** Right. Right. And so this is all happening before the roadmap. and so, you know, just mixing the brainstorming with, the complexity estimates. Another thing is also looking at like the market, right? There’s a bunch of second phone number apps. There’s a bunch of apps in general. I gone through dozens and dozens of apps, screenshotted, every single screen in their app. Put it all in. And then use that as a starting point for the brainstorm to like get everyone’s creative juices flowing. and I think that’s really helpful as well as to see how other people are doing it. And then, you know, try to take the good things that they’re doing and improve on those. from there. Just synthesizing into. [00:37:36] **David:** How do you balance the, technical complexity component? Because some of the, some of the experiments you, like, I mean, again, get to back to Jacob’s story. Putting 25 pages and asking for information that you then have to store and then personalize the rest of the app. Like that was probably a massive technical lift. and you know, and Jacob intuitively it’s like, this is not. [00:38:01] **Jacob:** Luckily I had a visionary boss pushed me through it. [00:38:05] **David:** And when do you push past those technical complexities? Is it, there’s a hypothesis that it will be a huge benefit, but that there’s going to be this massive change. And how do you, how do you think about [00:38:17] **Giancarlo:** That? Yeah, I think sizing the opportunity is really helpful, right? so a good example is that we’re working on. a premium Burner, like a plus Burner plus, and we’re thinking through what are the different features, and kind of going through this entire process that I just mentioned was that limited numbers [00:38:36] **Jacob:** On there. I’ve been meaning to ask you about that. [00:38:37] **Giancarlo:** So actually all of our subscription plans have unlimited calls. not unlimited numbers. That would, right. That would be crazy. I don’t think cost basis, [00:38:46] **Jacob:** I think with like the backing networks. Right. It’s like they charge you per, number, so it might be undoable, but I was like, somebody who’s going to pay a thousand dollars a month for that. [00:38:55] **Giancarlo:** Yeah. We have to, we have to deal with, spam. We don’t want that to be on [00:39:03] **Jacob:** Sense. Yeah. I’m sorry. So you’re saying opportunities, right? Like the sizing, the opportunities. [00:39:07] **Giancarlo:** Yeah. Part of that involves, sometimes it’s just estimating, right? Like, oh, if X percent purchase at this rate, what is the revenue going to be? And then for something as big as like a plus product, you want to make sure you’re including the right features. Right. So we’re doing tons of surveying to make sure, like, what are people not only interested in, but what are they willing to pay for? and so I use two pricing models, to do that the government Granger model and the van Westendorp model. And so these are two different survey models that basically ask questions about pricing. I think, the van Westendorp model is something like, are you interested in buying Burner plus with XYZ features for this price? If yes. Survey done, if no. What about for this price, $10 cheaper. And then you start to see, the percentage of people who are extremely likely, likely not that likely, definitely not likely to purchase. And then you calibrate that with like assumptions and there’s like, step-by-step instructions online that you can find on YouTube. you should, it’s a great source. I highly recommend it. and so we use that to come up with a pricing, a price. That will, you know, balance maximizing revenue, but also getting as many users as possible on it. And, we also are doing smoke desks. So basically showing users Burner plus, and seeing if they’re interested in purchasing before they can actually get it. And so then we measure [00:40:51] **Jacob:** Like a CTA that’s like actually a purchase or one that’s like, tell me later, or. No, like actually a purchase. And then you give them the womp womp paywall, like, are those that modal that’s like, sorry, not ready yet, but thanks for your interests [00:41:02] **Giancarlo:** Coming soon, coming very soon. [00:41:04] **Jacob:** I love consumer because you have so many users, you can like just tie a little thing. You can do a little weird experiment like that, and like can be so informed the same with the pricing experimentation, right? Like you have so many users with a consumer app. Like you can get surveys out and get significance on something. That that finicky. Right. And I don’t think I’ll say from personal experience, like in my past knit was not nearly. As bullish on surveys. Cause like, I think I was just, I didn’t like talking to people. Right. but surveys are great. You don’t actually have to talk to them. Like you send the link, you send an email and people tell you and it’s, and it’s in some ways scientific and stuff like that. So that’s interesting what to put a link in the notes to those models, because I think that’s something I’ve never heard of before that it’d be interesting to. [00:41:47] **David:** Well, there’s like a million other things I wanted to talk about, but we do need to wrap up. Any last thoughts as we wrap? I know a Burner is hiring. Are you hiring on your team specifically, or any specific job postings you want to shout out, and anything else you want to share with the listeners as we wrap up? [00:42:05] **Giancarlo:** Yeah, Burner is hiring. If you’re interested in the product, if you’re interested in working at a high growth, successful business, this is a great opportunity. The team is really great. I don’t think I touched on why I joined Burner, but to be completely honest I didn’t know too much about Burner when I first got my “thumbs up” for the first round interview. I just applied to a bunch of places, and Burner he came back. So I was like, okay, let me do some research. Did lots of research. It was really cool. Talked to my manager in the first round, and I was like, “Oh, this person’s really cool. I would love to work with all of these people.” That’s something I value a lot, and the workplace is just a great environment. We are hiring on lots of different teams. I think we’re hiring a Director of Customer Support, and finance, lots of engineering. You can check out the roles online, but if you don’t see the role that you think you would like, we accept all applications. We just want great talent. If we think it’s a good fit, we’re going to make an opening for it. [00:43:16] **Jacob:** How many folks are there? [00:43:18] **Giancarlo:** Thirty. [00:43:20] **Jacob:** Oh, that’s a great size. We’re slightly a little bit bigger than that, but like, we’re like 50, but like it’s small enough you can know everybody, but big enough that you have help, like you can do things. It’s a great time, especially if you’re focused on an app that’s super fun. [00:43:37] **David:** I would’ve thought y’all were much bigger. [00:43:40] **Jacob:** Apps, David. Huge leverage, leveraging. [00:43:44] **David:** Yeah, it’s funny. I mean, as an indie developer, I often get people emailing as if I’ve got 20 people on staff, and it’s like, it’s just me, part-time, and my developer part-time, and a designer we contract with a few hours every once in a while. But then I make the same assumptions, right? Four years ago Burner was a huge company. So that leverage is incredible. It’s amazing what y’all are doing with just the 30 person team. We’ll include links to the hiring page, and to Giancarlo’s Twitter and LinkedIn, and all that kind of stuff in the show notes. But again, thank you so much for being on. A lot of great stuff that we were able to talk about today. Thank you. [00:44:24] **Giancarlo:** Thanks so much for having me. This was a lot of fun.