Okay. So you guys tell me if this is too stupid, but well, if you have to ask, I like how this is starting. Sold. They're just three amigos making their way in the crazy old world of software as a service. Welcome to FounderQuest. Yeah. So, yeah, let's talk about the hackathon, guys. Cool. Okay.
so last week uh we had a our first ever honey badger hackathon and the idea was that we would take a break from working on our For more mundane lives, you know, working on error tracking, uptime tracking, stuff like that, and work on something completely different to kind of clear out the cobwebs and have a little fun.
Yeah, we chose to build a product in Elixir and Phoenix because we don't really use those too often. So it's a nice chance to do something different. So what do we hope to achieve by doing this? What was the goals? So I think like just, I mean, mainly have fun. um one of the things i really liked about the um the hackathon was we also this was like the first thing we did coming back from like a uh like what like a three-week vacation for christmas over which we i think we all worked on an elixir
like a Udemy course. So it was kind of a chance to practice some of the stuff we learned in the course and kind of get back into the swing of things. Yeah, that was nice. Are we going to do that vacation like all the time? Is that going to be a regular thing? We'll break it up by hackathons.
Otherwise, I'm cool with just being on vacation. Okay. Can we have like a hackathon driven company? Is that a thing? Yeah, I think so. Would we be like hip then? Would we be popular? Probably. What do you think, Ben? Can we be more popular? There are limits, right? Yeah. Let's think back to it. We first were talking about the hackathon at one of our conclaves. If you don't know us, we work remotely.
We do everything in Slack and stuff, but we meet up once a quarter for what we call conclaves at an undisclosed location in Western Washington. Yeah, so we went through lots of different ideas for the hackathon. Who was it that came up with the idea we went with? I think it might have been Ben.
We're kind of talking about the model of deploying the applications that we're interested in building. I think we were talking about things that are easy to do on-site or on a single server if they're a self-hosted type thing. That's kind of what led us to talking about Elixir and stuff. But I think it was Ben that came up with the...
Yeah, I think we do a lot in our day job with high traffic sites, right? And we do a lot of processing. And one thing that was really interesting was, as far as Elixir is concerned, is that the high concurrency that it can support. And so we're like, oh, what can we do?
build that would be interesting that would be you know kind of in our wheelhouse but but still kind of fun and you know we did that like you said we did the elixir over the christmas break but uh we also did the the advent advent of code and uh oh yeah I didn't finish that. I didn't finish it either. We didn't have to talk about that. But yeah, we definitely started with the right intentions. I know that for me, I was doing them first in Ruby, and then I would do it in...
fixer and see how different it was. Having the idea to play with that, I think was a lot of the fun motivation behind Hackathon. Yeah. So eventually, I think Ben was like, let's build a segment replacement because we use segment. to send various... Well, segment is sort of like, works kind of like a repeater, right? You send...
events that happen to your users, you send your user data to Segment, and then it sends out to your vast array of third-party services that consume that data, like Intercom, I think we use Mixpanel, we use Drip, maybe Google Analytics? I don't know. Yeah. Yeah. Like it costs a lot of money, right? Yeah. It costs a lot of money. Well, a fair amount of money. It depends how much. Right. Yeah. Yeah. But seeing like we basically only use it to broadcast.
request to other services. So it seems like it should be pretty cheap, but it's not pretty cheap. And we've had, yeah, we've had some trouble. We've been, yeah, we've talked about kind of, you know, building some sort of internal thing to do this for us. Just because we're really not, we're not fully utilizing its full capability yet either. Cause really what I mean, like I think the core of what it is, is like a centralized customer information database and warehouse really. And then it.
handles like you said sending all those events to all the different places, like third-party software-as-a-service tools and things, even to the point where it can even replay events, which I think is kind of a cool feature that we're not using at all. So is it actually a database, though? Can you go in and query your users?
into segment? Well, one of the destinations that you can configure is like a Postgres database, which we do. We dump them into Postgres. So yeah, you can go and query the events. Yeah, I think one of the things that was interesting about using that as a hack.
project was that it was it's very similar to what we do right we take in a bunch of events and we spit them out to different places like slack or you know github issues or whatever and so we thought hey we kind of know what this is like we can We can build something that will take in events and spit out things. Yeah, so we didn't end up replacing segment. Note for future hackathon planners. Yeah, you probably can't duplicate a complex.
product built by a whole team of people with a couple of guys working a week. Yeah. And I guess we should qualify like what our hackathon was, because I think some people might think when you say hackathon, think, oh, like. You start on a Friday and it's Red Bull fueled craziness until Monday morning. But ours is nothing like that. We just did it during the week. Yeah, it was just a normal work week. I don't know. You guys did. I was chugging the Red Bull. i was like where is everybody
That explains a lot. So we kind of divided up the responsibility. I ended up doing sort of the backend routing of the requests as they come in and sort of fanning them out to different notifiers that would send the request then on to intercom, to drip, to stuff like that.
that. Ben worked on deployment and Josh worked on the API sort of intake side of things. It also worked on the notifiers as well. But I feel like Josh sort of did the most research and had the best grasp of all of us into the sort of
problem domain. Yeah, what I started with was working out like the data model, which kind of helped a lot with that. Segment is really awesome in their documentation in that they share a lot about how they do things on the backend anyway. So they have a really amazing API spec.
which kind of outlines how their data model works. And I borrowed mostly from that, but did a few interesting things to kind of make it faster for us just so we could get more decoding and not trying to figure out the whole domain. and architecture and everything. I think one of the things that you did that was really cool, talking about the API and stuff, was that you built those JSON schema documents. Oh, that was really nice. Yeah, that really helped, I think.
make it really smooth to build out the code once you had that in place. Yeah, yeah, the JSON schema, they weren't... Should we explain a little bit what we actually did? Sure, yeah, I can basically, like I mentioned, there's a nice API spec that says... segment has on their website, it's got, you know, like example payloads of like the kind of things that you can send to segment, which is basically what models your customer data. So if you think like,
I'm a person that's coming to your, uh, to your website or your app. Like I've signed up and I'm using your app and you want to tell, you know, your. all of the tools you use that I just showed up and logged in. You can send an event to segment called identify, which basically includes a bunch of information that you have about me, like my email address and that sort of stuff that creates like the customer.
record. And then from there, segment would forward it on to drip or intercom or any of the other tools we're using. And so these events are kind of the core of the data model. They basically like describe everything that could happen with a customer in your product. JSON schemas are handy for basically describing the model of what that data is. So the data is modeled in JSON. So it's basically just key values and objects and stuff. And JSON schema is a standard that you can use.
to describe JSON data. It's like a structured format. So you basically define what each property in your JSON object is. And from there, you can actually run it through a validator that would check your actual data object against the schema. And if it doesn't match up, it'll throw an error. Yeah. So the idea is that if you submit a bad API request, it'll just reject it.
outright instead of sending it on to the model layer or further code that then errors because it was wrong you basically validate it when it comes in yeah yeah and so that's that's the once i had the schemas which Kind of helps make sure we understand what the data model is. Then the next step was to, I added a little, actually I found this little.
Elixir package called xjson schema. And luckily, it pretty much did all the work for me to actually like take an object and verify it against a schema. I plugged I put that into a little in Phoenix, I put into a plug, which is kind of like a middle way. that when a request comes in, I can actually just run that payload through the schema validator. And if it was wrong, then it would return an error to the user, basically.
I went into this not really knowing much about Elixir, not much about OTP, which is the sort of foundations that Elixir and Erlang applications are built on. And I found it really fascinating to learn about all this. It was a bit hectic at first. At first, I felt like I'm not making any progress.
At all. This is incredibly slow. Like, why am I doing this? But it actually helps to know the language and the platform that you're using before trying to build things in it. Eventually, by the end of the week, I was actually surprised at how... quickly, we're able to stand up, I wouldn't call it a product, but a robust system.
By the end of it, we had a system that would take an events, process them where for any given user, you always use the same queue and events always went in the same order so that you didn't have the problem where the same user sends events in and they get assigned to two different queues. One queue is running faster than the other, so then they get processed out of order. I was super impressed by...
by Erlang and Elixir by the end of the week. Yeah, I really like the code that you came back with, like the repeater code, which is what we ended up calling it. It didn't seem like a whole lot. of actual code that was written, but like what it actually did was kind of crazy to me. Like, like the way it basically had kind of like, what was like a routing system that would control the flow basically of data through the, through the repeater system.
Yeah, I guess it's not built in, it's a package, but it's called GenStage, which lets you process streams of data in a way that takes in... to consideration the fact that processing doesn't happen immediately. It was all pretty cool. What were you going to say, Ben? One of the things that I thought was neat, one of the things that we thought was interesting about going with Elixir for this was, we're used to the idea and the...
in the Ruby land of you take in some work, you shove it into a sidekick queue, and then you have a job somewhere that's processing it right. The only problem is that you have to now worry about Redis, right? You got to keep that up and you got to deploy it and so on. The thing I think is really cool about how Star did this is that it's all self-contained, right? You're writing an application.
And it's handling that, queuing that processing just in one place. So you can reason about what's going on more easily and you can handle the back pressure that happens when you're trying to deliver to a target that can't handle it as fast as work's coming in. I thought that was pretty neat.
We do stuff mostly in Ruby. We work in AWS. We need a given system that we push up. We may have several separate services that support it. We may have a data store. We may have caching. And it's just kind of crazy to me that in... Elixir and Erlang, you can put all of that or a lot of it into the application itself. That's just really cool to me.
By the end of the week, we had, what, like two actual destinations done, Drip and Intercom? Yeah, that was it. We had two destinations up. We could handle... Well, I started Postgres, too. Did I? Oh, my God. I forgot about that. Friday, you just busted out. It was all the Red Bull. I was in a Red Bull-fueled haze of productivity. By the end of the week, we had two events.
two types of events done which i think are the kind of the main two we had the identify event that i mentioned um which is like basically telling the system that a user exists and that they're you know that they're here And then a track event, which is actually telling the system when a user performs an action. And that's where it gets really interesting, sending this data to other systems like Drip or Intercom, because those systems will use these events.
to do things like an intercom, for instance, when someone signs up, we start triggering like various onboarding actions, like emails and stuff that go out to them. And that's all handled through events. So that's like the end result of this data.
coming through our system and going out to these sources. Yeah. On the actual destination side, we had drip intercom post grass. We had a, I forgot to mention the debugger that we, that I built. Oh yeah. That was actually, yeah. So like, We could see the.
we could see the events actually coming in. Like when you would post an event to the API, it would put it out into the, like the actual web browser using a Phoenix channel over a web socket. So it would actually like show the, you know, like the format of JSON data in the browser basically. in real time, which was, uh, that was all lines of code, right? There was just a couple, couple lines, mostly boilerplate. Um, it was more than a couple lines, but it was more mostly boilerplate.
Phoenix code. So that was pretty cool. Yeah. Can I, can I tell you guys a secret? Yeah, of course. The whole event routing thing I did, that was boilerplate too. I copied most of it straight from the Elixir docs. And most of the time I spent on it was just trying to understand how it all worked. Oh my gosh, star. Yeah. Am I fired? Well, while we're talking about copying and pasting. Yeah, what did you copy and paste, Ben? We are experienced senior engineers here.
so yeah so my job was the upside right and so i had to figure out like how would you get this into production and uh you know i'm somewhat colored by having to keep a honey badger up and running all the time so sometimes you know i over engineer but i held back and
I just copy-pasted a bunch of Docker stuff from the distillery documentation, which is fantastic. They have great examples on how to deploy to various providers. And so I decided to run with the Docker version because, I mean, you can run a Docker container anywhere, right?
So the Docker that I got from Distillery was pretty cool. They talked about how you can build. And this was kind of new to me. I haven't spent a lot of time with Docker. So I got to learn about this, which was fun. You can build. you know your one image kind of like a like a build image and then you can use that build image and another image
so that you don't have to have all of those build artifacts along with the final image that you send out. So, you know, with Elixir, you got to do a lot of compiling. You got to have, you know, Erlay and all that stuff installed. And you might not need all those things in your final production image, right? So you can... Step one is you compile the app using Mix and you get everything into a release. It's horrible.
And then the second step, the second image that you make, grabs that tarball, puts it in the right place, and then packages it up. So instead of having a few hundred megabytes worth of image, you can have 40 or 50. I thought that was pretty cool. So yeah, there's documentation. the distillery side about how to do that.
It works out pretty well. Another one trick being that you do really want to configure your configuration of like, for example, Intercom and Drip have API keys that you have to use, right, to be able to talk to their services. So the only little... of snag is that you want to have the Elixir application configured by a little code that actually hangs outside of the compilation step.
because those API keys aren't necessarily available when you're compiling, but they will be available when you run it in production. You toss a little code to the side there that loads those things from the environment at runtime rather than compile time so that you can configure on the fly with those API keys.
On the whole, what do you guys think of Hackathon? Would you do it again? Totally. Awesome. Yeah, it was great being able to work on something that didn't have to do serious stuff in production where we could fail and it didn't matter. So it's kind of nice. Yeah, I had a lot of fun. All right. Well, I guess that wraps up our podcast. I think Ben has somewhere he has to be. So ThunderQuest is a weekly podcast by the founders of Honey Badger.
Zero instrumentation, 360-degree coverage of errors, outages, and service degradations for your web apps. If you have a web app, you need it. Available at HoneyBadger.io. Want more from the founders? Go to founderquestpodcast.com. That's one word. You can access our huge back catalog or sign up for our newsletter to get exclusive VIP content. Founder Quest is available on iTunes, Spotify, and other purveyors of fine podcasts. We'll see you next week.