How to Create a Culture of Experimentation with Brendan Burke

Brendan is currently Senior Marketing Growth Manager at Baden slash Boomerang . Maybe he can explain what the difference is there . Prior to joining Boomerang , brendan did digital marketing and strategy in the world of politics , including working on campaigns . Brendan , thank you for joining us today .

I was working in , I was working for a member of Congress and they were having trouble hiring a digital communications director and someone brought up in one of the hiring meetings that I had like a funny Snapchat , and so that has led to me getting the job , to me getting getting the job , um .

I've been reading stories about they've passed the fewest bills in 100 years or something . So I don't know . I'd like to say that they all should use Boomerang for email productivity , but I don't know that that would solve all the political problems in Washington with one app All right , all right , all right .

That would be a totally normal and expected topic . All right . So when we first talked , yeah , you did say that your , your company , has a culture of experimentation , which I thought was interesting , because I think a lot of companies say they do , but in my experience , most don't really , and in fact , the company's referring to 2024 as the year of experiment .

So walk us through that and what it means and maybe a little bit of the background about how that came about .

The most with what they had has really been the culture of the company , and that means you know investigating what is working and you know promoting that and cutting what isn't working in terms of tactics and things like that .

So it's not that there was an experimentation framework at the company really until this year , but it was definitely part of the culture that that was going to make data-driven decisions and make sure the results are measured and go with what's working , um .

But when this year came around , um , it was really just a conversation between rolled that out in our uh , we call it a work away , we're fully remote . So we were actually all together in person , um , at the beginning of this year , um , and said you know we're going to set a goal of 26 experiments and that that was quickly , quickly , uh , doubled to 52 .

What you were describing is that the company is willing to cut things that aren't working or stop things that are not working , and I think just my commentary there is like I think that's actually highly unusual , even though I don't think it should be right . There's a lot of .

I've seen many places where you've get into this fallacy , where we sunk X amount of money and time and effort into something we just just a little bit more and I'm sure it'll work right , when in fact it's just throwing good money at bad stuff and I so applaud that .

So just to be clear like , this is a company-wide thing , this is not just a marketing thing or a sales thing or a development thing . Right , Product thing , Is that right ? Is it a full company ?

Definitely , one could argue that it's extra work that doesn't align with your core goals , right , probably as a marketer or product or whatever . So I think that's really interesting . A marketer or product or whatever , but , uh , so I think that's really interesting . Um , what , what ? Um ? So what kind of impacts have you seen from the experience that you've done ?

Uh , you keep it to marketing , if you want , or overall or both ?

Um , you know tabulation earlier this year and it was about uh,000 in new ARR on top of our about 8 million ARR . So not insignificant in terms of a growth on top of the company's existing revenue no-transcript .

Um , others are much more top of funnel , like changing a landing page or landing page template . So the proximity to revenue varies . But that's the top line impact that we , or the bottom line impact that we arrived at , and it is a I should say it's a product led um company . You know , that's how they , how they grew the company .

Now it's um , it's becoming more of a mix of like B2B Um , but historically the user base and the customer base have been sort of the prosumer customers . So that product-led motion is a really I think is really ripe for this type of experimentation .

It's um people who may be independent contractors or freelancers , or you know they may work for a company , but um , they're , they're buying it for themselves in their own use for themselves but for their professional , mostly for their professional life , got it okay , yeah , um .

And then how did you go about even coming up with and maybe you didn't start with 26 , right , initial experiments , but like , like , how did you like where the experiments come from ?

.

So any you know when , when they announced that this was going to be part of our um focus for the 2024 um , you know everyone was encouraged at that outset to contribute ideas for for things that you know they saw that could be improved , that could be tested , questions that they had um , and that led to this huge list of like over 90 um experiments or , uh

you know , places that we could look at to make an improvement .

So he's a big believer that marketing ops is kind of more aligned with product management than I tend to think , although I'm coming around . I'm coming around .

I think I think you know if you didn't use it in yours . I think that's fine too , right ? I think I think the key , the idea , is you've got something that you're using that's trying to take some of the uh , the bias out of the prioritization that is more than one one person's opinion's opinion , I think , which is often how these things happen .

Um , and sometimes you even have a framework and then , yeah , person with the highest title gets to decide anyway . So it's like , why do you do it ? Mostly , mostly bigger companies , I think , but , um , yeah , that's that's what I think .

You know , even if you're not going to do a year of experiments , like to run , run things this way because , um , it does take that bias out of which direction you're going to go in right and it , um , you know the best idea wins and you know ideas that aren't working .

Um , you don't need to argue about whether it's going to be effective or not , because you can just measure it . Um , and so , as long as people are willing to accept what , uh , the results are , um , you know it can take a lot of whatever politics or or whatever , out of the out of the equation .

So do you do like , are you doing a regular review , applying that rice model with the full the list as it stands after you've tried some so they're no longer on the list , or added ones that weren't originally on the list ?

For example , when new users sign up , they're on a trial of our product .

You know a little bit better and better and I think that that's kind of been key , as , as you do these and you get the results , sometimes there are like the thing that you're and sometimes the thing that you're testing .

The area of the website visitor journey , user journey that you're focusing on is just a high leverage spot and so it's worth it to keep iterating in the same area , even if you get a win or two wins . It can really make a big impact if you keep going .

And you're now at a pace of one a week in terms of how are you like , how are you managing that whole effort and how do you , how are you able to move that quickly ?

Um , to make sure that there's going to be the resources for it . Um , I think one thing that sort of quickly became apparent , though , was you can only do one test at a time on on one thing . You know , if you , if you have this landing page , you're like , oh , I want to , you know , convert more people on this landing page , um , well , you can .

You can probably only run like one ab test at a time , and that might take a couple weeks , and so if you want to do something in those two , three weeks , then you can look at other areas , other parts of the surface area to run a different test on . We're able to do so many over the course of a year is .

There's a lot of different places from , uh , lifecycle email website , um , you know , landing pages in product user activation stuff . There's a lot of different places that you can be running these on um , sometimes simultaneously . Um and so you know , once , once , you , uh , once you get started , there's really no end to what you can , what you can do .

So if we know it's probably going to take three weeks to get enough data for one of them , something will automatically move into a column where it tells us it's time to decide side um . So , yeah , each , each experiment , each ab test , is its own card in trello um and that way we can comment on it in that place . It .

You know , we use other tools to actually do the . Maybe , if there's development work required , that's , you know , going to go into our , our ticketing system , um . But as far as managing the experiments themselves , it's it's uh sure , uh , yeah , yeah I , I was just .

So I think we actually have talked to somebody about this . We've had a couple people on recently who , I would say , came out of a science background , right so familiar with the scientific method , which should be of a science background , right so familiar with the scientific method , which should be a highly structured approach , right ?

You develop a hypothesis and then you develop a test , you run the test , evaluate the results . Does your hypothesis hold ? So ? And for anyone who's a scientist out there , if I butchered that a little bit or missed it by a little bit , please don't come get me . I think I'm close enough . But do you have a structured process like that you mentioned ?

Some of these happen simultaneously or overlap , I assume . What does that process look like for any given experiment , so that you can , like I said , ultimately you want to know at the end did it work or not and what was the impact , Right ?

Um , so we do have that at the top of each Trello card is the hypothesis for this experiment .

Right , because you know you can see , you might be able to see , you know other effects , but you need to be able to identify , you know , what does success look like and and should it make the number go up or go down ? All of our experiments are that hypothesis , the success criteria , the instrumentation for how we'll actually measure the results .

This is another area that we have had to improve on , because there was some , there is some communication breakdown . Basically , we've been good at saying you know , okay , this screen looks like this , we want to make it look like this other thing as part of this test . That part of the communication has been very straightforward .

The part that we've had to um improve a bit when we got started was how are we going to measure whether the first screen or the second screen , you know , converts more ? Whatever result , we're looking for um , because sorry , I've got this .

So I don't know , maybe we could do 104 with a platform or something . So instrumentation is a key part that we've learned needs a lot of focus and very explicit communication as an experiment moves from team to team . And then also guardrails , um , so there's the number that we want to see go up or down as an indicator of success .

But you know , it's like , um , it's like sending an email . Yeah , you want clicks on that email , but you might also want to keep an eye on people who unsubscribe um , to see if the net effect is more harmful than is actually , uh , going to be helpful , right ?

So , um , for each one , just as a thought exercise , like , what is the guardrail that we want to prevent ? Um , I think , uh , you know , email unsubscribed is is kind of the clearest thing there .

, though . Let's say you say we've defined , this is the metric , we want to increase . But let's say you run the experiment it starts to decrease . I assume you also have a floor right it says we're gonna this is where the reversibility comes in like we're gonna undo that experiment before it .

You know it has a truly negative impact on the business Is that what you're describing Okay .

if , if necessary , um for the design , and then onto our developers , um , and then we will line it up to um to be to be launched you know it was as you were describing this .

What you described is having like the one sentence description of what you're doing with the desired outcome and then the measurement or metrics behind it . Right , those , basically , are the first two letters in the smart rate , specific and measurable .

Um , and and I think a lot of people miss that in how they do goal setting and probably would miss it in the context of doing experiments too yeah , yeah , yeah , the measurement I think can be intimidating , um , but it it's really , it's really key and say we want to do an A-B test and when they describe it I'm like that's not really an A-B test , it's

like two completely different scenarios , right , because you're not really testing one small , I wouldn't even call it multivariate . Right , it's like we're going to do two completely different landing pages and see which one quote wins , but then they don't really do the test of the wins , they run the whole campaign .

They run the whole campaign and then they may or may not take the insights from that campaign and apply it to a future campaign , which is not really an A-B test . I'm not saying you can't learn from it , but it's not really what an A-B test is .

It's because I can get energized by the creative process , but at the end of the day , I'm going to be disappointed if I don't know the impact or results of the thing that I did , um , and I think that's one thing that sets marketing operations apart . But the repeatability that you mentioned is another thing .

That's , I think , tricky because you know case in point , like we actually have sort of two products we have Boomerang for Gmail and Boomerang for outlook um , that people can use with either of those email accounts , and that's one been .

One of the surprising things is we'll take a result from um , you know our gmail user base and try to do the same thing with our outlook product , and the results are not repeatable .

I've worked working right now where I've got on one side a client or an employer that I've worked for that had outlook , but I use gmail as on personal stuff and they I mean they operate differently enough where , like I can imagine that how you , you would use a product like yours would also vary .

But I remember and I still see this , like people agonizing over the copy you put into a paid search ad right , what's your headline going to be , what's the words you're going to use . And I remember one time I was like I can't remember what the specific scenario was , and this was for something like , really like specific .

It was for semiconductor , like semiconductors , so it was like not a consumer product by any means , although we did have a challenge with audio semiconductors being confused with search terms that were for people who are audiophiles . But that's a whole other part of it .

But one of the things I remember one time my friend was like look , we're debating on this , one word in this line of copy of like , there's's like the , the relative cost for us to just try both and see which one works , cause we think that like , we think they're synonyms and the , but the reality is one ended up , and then we're all like .

Everybody had an opinion . I had an opinion about which one would perform better . Once we did this and I was wrong . And so it was like the first time when I was like , oh , we don't need to agonize over this if we've got something that we can move quickly on right , yeah , and also I was like I may have my opinion about what's going to work .

It was like one of those . I was like I need to be ready to admit when I was wrong , because at the end of the day , I , if I let my ego get tied up in , yeah , whether or not my choice of , in this case , copy was right or better or whatever , and it's not , I'm going to be like I'm .

I'm going to go down a path where I'm going to sub optimize what we're going to get to . So I was like it was really , really important for me to get that . I love the idea we could get that quick feedback and and stuff . But , um , it was like so are you like you're running into surprises too ? I mean , you mentioned that one .

Right , where are there other examples of things where you were , you did an experiment , you thought something was going to work or did not work , and and then you saw the opposite or something different . Maybe totally , yes , um , definitely , and that's a .

And that's a great case in point in myself , as I've , you know , gone on in my career is like I'm it's not that I I mean different or don't care anymore about you know the choices that are made , but I know that I might be wrong .

Uh , you know , I've had that happen to me enough , where I don't want to get too wrapped up in thinking that one way or another is the right answer , even if I do have an opinion , um , because I just want to get it out into the world and and see what happens

. Um , so , yeah , I mean another example of uh , I want to be glib and say another example of where , uh , you know the user was wrong . Uh , right , those pesky customers right uh , but no , the customer is always right , as the data informed us .

Um , we tried to redesign some pages in our purchase flow that used an older , we would say outdated , form of our blend , our branding , and we could not , um for the life of us , improve the conversion rate on in some steps of the purchase flow .

Basically , that we , you know , subjectively thought was um going to be an improvement uh , on the design , and I think , objectively , like , following design principles , actually like would have considered it uh a more intuitive , you know , higher converting um example of a of like a purchase page , um design , uh , but the numbers , you know I guess the numbers didn't

lie Um , and so there's some , there's some compromises that you have to make there . Um but yeah , um , I think those are , those are some of the most interesting examples , um , and just really the , the ones that that reinforce why , why it's important to , or you know why it's helpful to test everything .

Uh , yeah , as as time and resources allow yeah , no , I think I also .

Is you get this never-ending or really extended review and approval process right really extended review and approval process Right ? And um , what I have tried to do is educate the rest of those teams about not again to your point , like it's not that I want .

I'm not discounting the importance of caring about the work to a point and and really I go , I try to get tell them like you need to think about , um , if something is either wrong or not right , like suboptimal right , I don't even know there's right or wrong , right , there's like there's always trade offs in these things is , consider how , how easy is it to

change it or fix it ? Should you realize that something has gone wrong ? and if you think about it that way .

Yeah , you want to invest a little more time and effort on uh email , because once it's gone , it's gone right especially if it's an email that has important dates or times , like invitations to a webinar or a you know event , that you're having something like that , um , and know that you're going to miss stuff , because it always happens .

No , but if you're doing like we're putting up a landing page , right , well , you can fix a landing page like that , right , it's seconds , theoretically , so should you just publish whatever , right ? No , but I'm like the level of review and approval should be aligned with the risk associated with something going out again .

Now , if you have again , if you have content that's wrong or misleading or whatever , you should fix it and that needs to be fixed . And if you have to do some sort of communication , um , then do it . But otherwise , like , don't get so caught up in this and that's what I'm like . The . The other one is interesting me .

So , again , going back to the search marketing service , it was . It's been amazing to me how many times I was like people get caught up in the uh , the grammatically correct copy or ads , especially in search ads , when what I find , what you find , is actually misspelling sometimes .

Do better , because people don't know how to spell or they're typing fast and they misspell . It is astounding to me and like people don't really read this stuff and so I'm like try misspelling .

But the one in particular , like you mentioned that this is a cross-functional thing and there's other teams Do you feel like and I know you're a small organization anyway , but do you feel like this is like this ?

This culture of experimentation , this year of experimentation , has fostered more and better ability to collaborate across those functional what is often divides I don't want to use the word , but that's what it is , I think , for a lot of people .

Um , in the past you know for me to be working with developers or something like you know , if we have a new feature or a feature update , um , those would be much more effort intensive um projects that would be on the scale of like weeks and months and not right days , um , and so being able to increase that cadence of collaboration through these experiments

small changes , like you know , tweaking landing pages , things like that um has definitely uh allowed for more collaboration , just better , um , you know , better working relationships um improve communication between these teams .

You think , yeah , knowing what's important to communicate cross-functionally , yeah , Um , you know , just knowing what things they need to do their work and what things you know letting them know what things I need to do my work , Um , those kinds of process things . It's an iterative , you know .

It provides opportunity for a lot more iteration when you're doing a lot of small changes rather than these . You know , a few big changes each year .

So one last question , and then we'll have to wrap up what's the most painful lesson you've learned from this ? Like , what is like ? Yeah , it sounds like a lot of positives Like have there been ? Like what's the biggest challenge you've run into as part of this ?

It's interesting to observe that the strategic decision that has to be made there about whether you're going to go with something that might seem more aesthetically pleasing , but you know , however small , however small the difference is could hurt the business sure um and you know this company's three co-founders bootstrapped for over a decade , now profitably .

I think that their success is enough evidence that you've got to be able to make those decisions . Back to one of our earlier points about letting things go is like a rebrand might look nicer , but if it's going to harm the business is , is that really the right thing to do ?

Yeah , that's , you know , sort of been a learning experience for me , coming from you know , the marketing side of things , where you want every I to be dotted and t to be crossed . Um , and you know , have everything sort of picture , picture , perfect . Um , you know , it's , it's , uh , it's not easy to no , I mean like , so this is .

So , brendan , thank you for sharing what you've done , what you learned . I'm sure that our listeners have probably taken away some ideas that they can take and try to bring back to their organizations . If folks want to , you know , learn more about what you're doing or reach out to you or whatever . What's the best way for them to do that ?

If you have ideas or feedback on topics or guests , or want to be a guest , feel free to reach out to Naomi , Mike or me , and we would be happy to talk to you about that . Until next time , bye , everybody .