So you've created a Python-based open source project, and it's starting to take off. You're getting contributors, lots of buzz in the podcast space, and more. But you have that day job working on Java still. How do you make that transition from popular hobby project to a full-time job? After all, you're giving away your open source project for free, right? Well, on this episode, I put together an amazing panel of guests who all have done exactly this.
Turned their project into full-time work and even companies in some cases. We have Samuel Colvin, Gina Häußge, Sebastian Ramirez, Charlie Marsh, Will McGugan, and Eric Hulsher on to share their stories. This is Talk Python To Me, episode 448, recorded December 7th, 2023. Welcome to Talk Python To Me, a weekly podcast on Python. This is your host, Michael Kennedy. Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython, both on fosstodon.org.
Keep up with the show and listen to over seven years of past episodes at talkpython.fm. We've started streaming most of our episodes live on YouTube. Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode. This episode is sponsored by BaseDash. BaseDash uses AI to build a dashboard for your database. Get a custom admin view in your Postgres, Microsoft SQL Server, MySQL, MariaDB, or Redshift database.
Get started for free at talkpython.fm/BaseDash. And it's brought to you by Sentry. Don't let those errors go unnoticed. Use Sentry. Get started at talkpython.fm/sentry. Hello, everyone. Charlie, Will, Eric, Sebastian, Gina, and Samuel. What a big group of people we have here today. An awesome, awesome group. Thanks for being here, everyone. It's going to be super fun to have you on the show.
You know, I know there's so many people out there that are dreaming of an open source project or even working on open source and contributing to it, but it's something they squeeze in the last hour of their day. And at some point, all of you have made this amazing transition to where there's enough support, there's enough interest in what you all built that becomes your full-time thing, right? Which is, I know for many out there, living the dream. So you all are living the dream.
I'm hopefully you feel that way. But it's going to be really great to kind of just explore your projects, how do you make that transition, and, you know, what you're up to. So we have Samuel Colvin on from Pydantic, and he has the tightest schedule. He's squeezing us in on a far-flung trip, a work trip. So let's go to you first, Samuel. Just, you know, tell people a bit about who you are and what your project is to get started. Yeah, I'm Samuel.
I maintain Pydantic, which is a data validation library for Python that uses PyTynx. So those weird things you've seen after colon and Python that mostly do nothing unless you run mypy or Pyright. Pydantic basically enforces them. Almost exactly a year ago, I was sort of working on Pydantic full-time back then, but really spending my own money to do so. And a big American VC got in touch. And fast forward, I raised some money and started a company at the beginning of this year.
I hired now 10 people. And I'm actually, as you mentioned, I'm in Istanbul at the moment with the whole team. It turns out if you have a team with Americans, Iranians, and Russians, that Istanbul is one of the few places that you can actually meet efficiently. So, yeah, we're having a really fun week in Istanbul, mostly working together, and then going after dinner, which is where we are at the moment. I hope that's a good summary. Absolutely.
And what a cool experience that you get to hang out with all these people talking about your project in Istanbul. I just had, by the way, I just had... I mean, just think that the other thing I would add is that one of the best bits of it has been being able to hire friends of mine from open source, people who had, I think, all but two of the people here have contributed to Pydantic before we started the company, which is how I hired the first, I think, six people. And then a few more.
being able to pay the property. Yeah, awesome. I see a lot of people shaking their head out there as well. Just coincidentally, I recently had Sydney Runkle on Who Works With You, and she's such an inspiration. That show is not out yet, but has already been done yet through the weird time shifting of podcasting. So it'll be out before this, but it's not out yet. So anyway, very, very cool team you got there. I guess we'll go around the order of the video. Pretty bunch pictures here.
So Charlie, you're up next. Quick, quick bit about you. Hey, so I'm Charlie. I am the author of Ruff, which is a Python linter code formatter and code transformation tool written in Rust. I started working on Ruff about a year ago, maybe like October last year. Similar time frame to Samuel, yeah. Yeah, and I worked on it full time for a while, similar to Samuel, although probably not for quite as long. I was working on it full time without any funding.
And then similarly, I started a company around Ruff and really around the vision of trying to build high performance Python tools. So take some of the things that make Ruff nice to use and popular and well-liked and try to apply those principles to other parts of the Python tool chain. Yeah, we're a team of six and we're fully remote. So no two of us live in the same place. I guess a couple of us live in the same country, but most of us live in different countries.
And we span basically U.S. central time to, I don't know, we have one person in India and one person in Minneapolis. And that kind of stretches like the full time zones of the team. That's a good stretch. That's a good stretch there. Well, absolutely. Congratulations on Ruff. It is really, really taken off. And I know the guy right below you and this squares, Sebastian, I saw him saying nice things about it with FastAPI and so on. Yeah, yeah, I know. Thanks a lot.
It's just funny because I think I was saying before I started recording, I think when I came on Talk Python last time, I think that might have been my first time ever on a podcast. And it was to talk about Ruff. And so I was like, oh, wow, people care about what I'm doing. That's super cool. And Pydantic and FastAPI were some of the earliest adopters that people had heard of.
So they were like very early on in the life cycle and helped a lot with, you know, getting people to see Ruff and sort of care about Ruff and know that smart people are using it. That's awesome. Well, when you visit Talk Python or the course's website or others, it's all Ruff-ified. I don't know what the adjective of Ruff is, but it's been Ruff. It's whatever you want to say. It's amazing. It's been Ruff-ed up and it works great. It's really amazing. That makes the code sound really bad.
But yeah. True. Yeah. That's maybe not the best. It's like saying it's sick. Like, oh, that was a sick trick they did. It means good. Yeah, that code is rough. Yeah, that code is rough, man. All right. Will, is your code rough? Who are you? It's got some rough edges, certainly. Welcome. Yeah. I'm Will. I'm probably best known for a library called Rich. And Rich is a library for writing formatted content in Terminal.
You can write tables and progress bars and syntax highlighted code, all sorts of things. And then that became quite popular in the Python world. I think there was a need for it. And two years ago, I founded Textualize, which is a company which builds on Rich to produce a library called Textual, which allows you to build user interfaces inside Terminal. And these are kind of more like web interfaces and less like Cursus applications. Yeah, I was just thinking almost like Flexbox.
You can dock stuff to the side and that's part of a UI. But it's pretty amazing what you've built. I think it's pushed the Terminal far beyond what many people thought the Terminal should be doing. It's been very cool. It's been a process of discovery. It's kind of strange. You've got technology that's been around for decades and are still discovering things they can do, which people haven't tried before. So, yeah, it's been a lot of fun.
And it's been great to work with a small team of developers. We've got a small core here in Edinburgh, Scotland, and we've got one developer in Portugal. So, yeah, it's been great. And I love the reception and I love what people are building with it. Yeah. It's so many people are interested in using Rich. And there's a ton of plugins, right? There's like Rich Click and there's pytest Rich and all these things like, well, whatever we're doing, it should also have what you've built, you know?
Yeah. I love to see that. I love to see the ecosystem building. Our adjective is richified, which is not as good as possible. Richified. Okay. Yeah. Yo, that's rich. My code is rich. Okay. Richified. Got it. I love it. We all have adjectives that we've got to put on things. Yeah. I'm a big fan of you and your team's work, by the way. I mean, I'm sure you hear it all the time that you build awesome stuff. Cheers. I appreciate it. Yeah, indeed. We all are. Eric, welcome.
You and I are coming in somewhat local here out of Portland, not Istanbul. I'm actually out in Bend at the moment. So yeah. I absolutely love Bend. Lots of mountains and whatnot. But yeah, so my background in Eric Holscher, folks probably know me from kind of the Read the Docs ecosystem. I feel like I've been doing this for a while compared to everyone else here. The project actually started in 2010, which is kind of amazing. And then we actually started the company around it in 2014.
So coming up next year, that'll be 10 years. And so, yeah, and we're all bootstrapped. So we haven't done any venture capital. It's all been kind of just built up the open source and then kind of turned it into a company on top of that. We kind of have a business model of doing paid for private hosting, you know, companies as well as advertising on open source.
And we spun out a kind of a separate business called Ethical Ads that we built kind of to basically make, you know, better advertising on the internet, not doing any of the creepy tracking stuff. Read the Docs is so critical to the open source space, not just Python, but especially Python. Yeah. Yeah. Yeah. I mean, so awesome work on that. And then just, you know, good job on the ad stuff as well. It's the tendency of the whole ad space is to be super creepy.
And just like, how much can we resell and how much can we buy to mix in to like create these shadow profiles? And it's not a good social trend. It's not good for people. And it's not really necessary, right? Like if I want to, you know, I've sponsored ads on Read the Docs or had run ads on Read the Docs before. And we had a Flask course. And let's just, you know, run that ad on Flask instead of like, let's see if we can track somebody through Instagram over to Facebook. Who knew, right?
Yeah, exactly. We call it, yeah, newspaper advertising. And it's sort of been really cool with the advances of ML. Like the content targeting has gotten pretty good, right? So it's definitely a lot easier technically these days to kind of do that content matching. But yeah, I think in terms of the kind of open source, maybe the way that I'm a little different than other folks here is we're actually running something more akin to a service.
So Read the Docs, all the code is open source, but a lot of the usage is open source projects using our hosted service. We do obviously ship a lot of code. Probably our Sphinx theme is probably the most kind of well-known actual software that people are running. A little bit similar to the folks here, the kind of sidebar on the left. And yeah, so I think that's kind of the background is, yeah, we have a little bit of everything.
But yeah, it's a little more akin to something like a SaaS app, I would say. But the code is all open source. We do get a contribution in that way as well. Yeah, awesome. And, you know, it sounds a little similar to what Textualize is working towards as well, hosting. I mean, it's more hosting people's apps than hosting people's docs, but not terribly different. Paying for services is a classic open source monetization strategy, right?
Like you have some way to kind of build the monetization on top. Yeah, it's like what you build is great, but it's kind of hard for me to run. Could you just do that? Like, yeah, sure, we can do that. Sebastian, always good to see you. Welcome back. Thank you very much for having me. Hello, everyone. I'm Sebastian Ramirez or Tiangelo. I created this little tool for building web APIs that is called FastAPI. And it has been growing quite a bit. People have liked it, fortunately.
And like, yeah, that's it. I'm probably the only one that hasn't built a company here, I guess. Okay, baby. Okay. There are two of us. Nice. I want to hear that. Okay, so this year I have been able to work full-time on open source because Sequoia Capital, the VC firm, started this open source fellowship. And I am the first fellow, so I am kind of the test trial of the program. So I get, you know, just to work full-time on open source.
And that's how I've been able to just, yeah, release that stuff in FastAPI and the ecosystem, like the other libraries. FastAPI, SQL model, type, and a couple of other things. But yeah, that's pretty much it. Yeah, awesome. Well, FastAPI is definitely the poster child for modern Python on the web. You know, and Samuel's work, obviously, is central to that as well, right, with Pydantic being such a core element there. So very awesome.
Yeah, and like, I mean, FastAPI is built on top of like great open source tools and their needs. So Pydantic does all the data validation, serialization, documentation. And on the other side, Starlette does all the web parts. Yeah, it's like, and like, you know, like it's a very close, how do you call it? Like a close friendship between all the projects. Pydantic, Starlet, Ubicorn, FastAPI. Like, you know, like we actually know each, by this point, we actually know each other in person.
Like the current maintainer of Starlette and Ubicorn works for the Pydantic company. And he's like the top FastAPI expert right now. But, you know, like it's very nice because helps the dynamism of like the speed of building stuff. It's great. Yeah, I imagine a lot of people don't realize the interconnections between these different projects that are at play here. Yeah, it's definitely super fun. Absolutely.
And then it's worth saying we in Pydantic use FastAPI as well as contributing to the libraries that build it. We also use FastAPI for the services that we're building. So we're like consumers both directly and indirectly of the things that we maintain in open source. Awesome. Chat point off and splash is awesome. It's turtles all the way down, basically. Exactly. This portion of Talk Python To Me is brought to you by BaseDash.
BaseDash is the custom admin panel for your database that you don't have to build. We've all dealt with endless ad hoc requests for data access. Your support team wants user records, your non-technical co-founder needs to see charts, or your engineering team needs an easy way to share your SQL queries.
These all sound like relatively simple tasks, but between building an internal tool, setting up an analytics product, and then switching back and forth between that and your SQL client, you've lost so much time. BaseDash gives you that time back. With BaseDash, you can instantly generate a visual UI for your database. All you have to do is connect your data source. It uses AI under the hood to generate the perfect admin panel for your data. Voila!
Now anyone in your company has access to the data they need. They're rolling out a ton of new features like the ability to create charts using only natural language. You're going to get hours back in your day, and I promise you won't want to build an internal tool ever again. Check them out by visiting talkpython.fm/BaseDash. That's talkpython.fm/BaseDash. The link is in your podcast player show notes. BaseDash is free for small teams, so give it a try. You've got nothing to lose.
Thank you to BaseDash for supporting the podcast. Gina, also awesome to have you back. Welcome in. Very happy to be back. That's really, really great to be here now, I think, for the second time. And the third if we call Python Bytes.
So, yeah, my name is Gina Heuske, and I'm probably, yeah, I don't know if I'm that well-known throughout the Python community, but I'm quite well-known in the 3D printing community for having created Octoprint, which is the snappy web interface for U3D printer, and the server part of that is all written in Python. And I started a project back in 2012 when I got my first 3D printer as your regular pet project on the side after hours vacation, blah, blah, blah.
In 2014, I got hired by a 3D printing company to work on it full-time, and in 2016, that company ran out of money, and I suddenly found myself with a full-fledged, full-grown open-source project and no funding whatsoever. So I tried this whole crowdfunding stuff with Patreon, and that is pretty much how I've been working ever since. So I'm self-employed under German law. I'm a freelancer. And, yeah, all my income comes from people who just sent me money for working on Octoprint.
Did you say since 2016? Since 2016, yeah. Wow, okay, cool. So full-time since 2014, and self-employed and crowdfunded since 2016. So it's been a while. That's awesome. I also went full-time on all my stuff here in 2016, although it's technically not open-source. It talks a whole lot about open-source stuff. So awesome. It's been a while. Congratulations. Thank you. I'm still surprised it works, to be honest. I'm waiting every day that it stops working.
And, yeah, actually, this year it came a bit close, and then I put out a blog post basically along the lines of, you know, the income is going this, but the usage numbers are going that, so something is amiss here and we need to talk. And then the community rallied and stuff fixed itself again. So that is pretty amazing. That's really awesome. Congratulations.
It's easy for people to just go, oh, it's free, and there's some people supporting it, and just kind of assume that things are taken care of. But, yeah, really good. Really good. Yeah, you have a ton of supporters on GitHub sponsors as well, right? Actually, I've diversified over the years.
In the beginning, it was only Patreon and PayPal, and then I started adding new payment platforms and options because people prefer to be able to use the stuff that they know, and some people don't like Patreon, some people don't like and don't know GitHub sponsors and stuff, and so you just give them the options to choose whatever they feel comfortable with, and this increases the likelihood that they actually will go through with throwing something in your head, so to speak.
At least that's my feeling. I don't know. I think that GitHub sponsors has been really positive for open source. I think it's made it pretty easy to just check a box. They already have your credit card, potentially, and you just want to give a little support instead of a one-time PayPal donation here that you forget about after once and all that. Before we move on here, there's a funny comment, nice comment out. Kanishk says, this group, look at all of you.
You guys are the Avengers of Python open source, right, which is kind of like the superheroes. I love it. It's somewhat actually true, it seems. Okay. So the next thing I want to ask is a little bit of an origin story. Just what projects did you try and they didn't really get traction, and then how do you think that your project got traction here? And I know, Samuel, you might vanish any minute, so you're next. You're up first on this one. That's a really interesting.
I've built a number of things in open source. I started something last week. I open source on Friday that has got more stars over the last three days or, I guess, week now. as the first project I started, ARQ'd, has over, I think, coming up on six years. So, I mean, that's front end, and front end gets a lot more traction than like queuing an RPC does. But, yeah, I don't know how, obviously there were some things I played with back then that never took off.
But, yeah, it was this, like, I think having the right time, right place for Titans. They'd obviously been around in Titans for a long time, but back in 2017 when I started Pydantic, their usage was growing really fast. And I think there were lots of people like me who found it kind of frustrating that they were there and didn't do anything. So, yeah, I think being the right time, right place was super valuable for me.
And, obviously, like, great projects like FastAPI, adopting Pydantic, has made a big difference. But I don't know. What's weird, if you look at the download chart, it's like there was a protection point at the beginning of 2021. So then we had, like, 5 million downloads a month. And since then, there's been kind of almost exactly linear growth to now 125 million downloads a month. So something weird happened in 2021, not after the project started, I don't know.
But, yeah, I think it was right place, right time. And that frustration was the starting point. On that note, I'm going to have to rush guys a bit. It's great to see everyone. And I'm sure I'll meet you all in a conference soon. Bye-bye. Thanks for dropping in. Good to see you. Bye-bye. All right. Charlie, keep going around the circle here just so I don't... Yeah, you know, it's interesting because I, like, this is... Ruff is really my first time being a maintainer, like, publishing open...
Like, I've been a consumer of open source, right, for my whole career, but I was never really, I guess, a creator or a publisher or a maintainer of open source. You know, around the time that I started working on Ruff, I was working on, like, a couple different projects. And they all... I was kind of trying to figure out what I wanted to do next. I'd left my job recently.
And Ruff was kind of motivated by a lot of the experiences I had at my job, where I was, like, maintaining a large Python code base. Did you leave your job with the intent of going to work on Ruff? Or were you just like, I'm going to leave, and then I'm going to figure something out. And you're kind of like, well, what is the process there? I left my job with the intent of starting a company, but I did not think it would be Ruff.
And Ruff was, like, this distracting side project that I was working on while I was, like... I was working with a friend, and we were, like, trying to figure out, like, the Venn diagram of interests, of, like, what should we... What are we willing to work on full-time together? What makes sense, given, like, our interests? And then, you know, in all my free time, I was like, I just want to work on developer tools. And, like, I worked on... I built Ruff, or I started building Ruff.
I worked on, like... Yeah, I worked on a lot of different... Not a lot, but, like, I did... We were working on a couple of different projects. I did, like, a sort of CICD thing where you, like, write Docker files and, like, CI files in TypeScript, and they, like, transpile down. And I was like, oh, I think that has a lot of cool ideas. I worked on, like... It was pretty early in a lot of the LLM stuff.
I worked on, like, a code-based-wide refactoring tool where you, like, give it examples of, like, before and after, and then it tries to find examples. It was, like, a co-pilot for your entire code base. Kind of... It didn't work that well, like, at scale, but it was, like, a cool idea. So I was, like, working on the stuff that was all open source. Maybe it could have been a timing thing. Like, maybe today that would be all the business. Yeah, I mean, it's also, like, an...
Yeah, it's also an effort thing, I guess. Like, I put a lot more into Ruff. And it was interesting because I kept viewing Ruff as, like... You know, it was probably open source at that point, but I don't think I had launched, like, launched it. And I kept viewing it as, like... Like I said, like, a little bit of a distraction. And my friend was, like...
You know, I think you should really, like, push to, like, release this because, like, if you think it's, like, interesting, then, like, other people will probably think it's interesting. And he was the person that really, like, motivated me to actually, like, see it through to doing the release. And then I did the release and, like, a lot of people were actually interested in it.
So that kind of gave me, like, the energy to, like, really start working on it full-time and just kind of see where it went. And it was really, like, projects like Pynantic and, like, Sebastian started, like, commenting on issues and stuff. And I was like, wow, like, real serious projects are looking at this, like, crazy tool. And let me just see.
I'm just going to do whatever it takes to make it, like, a feasible choice and just started, like, cranking through, like, all the issues and all the things that we were missing. It escalated pretty quickly, I think. Did it surprise you? Oh, yes. Yes. Absolutely. I was convinced that I had faced, like, this problem around tooling as, like, our code base got large in my last company and that there was an opportunity to, like, build, like, better, more performant tooling.
I genuinely wasn't sure, like, how widely that message would resonate with people. And so, I don't know, it's like a little bit of luck, right? Like, sometimes you work on great projects and then, like, they go nowhere and, like, just no one happens to see it. And then sometimes you work on a good thing and, like, it does go somewhere. And so, there are certain things that are within your control, certain things that aren't.
I feel like I did a good job of, like, communicating the project and, like, why it was interesting and why it was exciting. But I also feel like I got a little bit lucky that it just, like, gravitated towards the right people and got attention in the right ways. Sure. Also, maybe that Black had existed, which is a little ironic because Black kind of solves a real similar problem. But, like, people were, okay, we embrace the idea of the thing that Ruff does really well.
You know, you just convince them to use Ruff in that sense. Yeah. I mean, we have a little bit, I mean, it gets back to this. Like, we have a little bit of, like, a second mover advantage, right? Yes, exactly.
There's a lot of existing tools that people use already and, like, a lot of those practices, like, the idea of using a format or the idea of running code mods, like, the idea of using a linter and, like, the knowledge of, like, what kinds of rules are valuable and, like, what kinds of analysis we can do. Like, all this stuff existed. And that was part of the story, really, was I was, like, these tools are great and I get a lot of value out of them, but I want them to be, like, easier and faster.
And so, yeah, there's definitely a strong timing thing. I think, like, I think Rust, too, this is maybe, like, a little bit more specific to what we're doing. But, like, you know, I think, like, the intersection between Python and Rust has, or the, I guess, the interoperability between them and sort of the ecosystem around it has just grown a lot. I mean, still, I would say, like, pretty early, but it's, like, matured and grown a lot over the past few years.
And so even just the fact that, like, at my last company, we started to introduce Rust and we started to, like, move some of our core systems into Rust and expose them over PyO3. Like, the fact that that existed and the fact that I was, like, exposed to that, like, that's all just, like, pure chance. Otherwise, I never would have thought to do this. Very cool. Well, congrats. Well-deserved. We're all doing the, all using Ruff, as we said, is excellent. Will, Rich is awesome.
How, why do you think it took off? Did you try stuff before? What's the story? I tried a lot of stuff before, before GitHub. You know, coding has always been my hobby. You know, I'd do it for work and I'd come home and work on a hobby project. And it just seemed natural to want to share it. Pre-GitHub, I would just put stuff on my blog and get some feedback from it there. I quite enjoyed that. And yeah, I have a number of open source projects. Pre-Rich had a BB code parsing library.
I had a chess library. I had a web toolkit. So yeah, when Rich came along, it was another hobby project, something to keep me entertained. Why is this terminal so boring? Come on, see what we can do about that. You know, I'd always use the terminal and I'd always struggled when you've got a page of white text on black background and you're trying to pick out an IP address from somewhere. And, you know, I'd always wished, oh, I wish it would just format it and colorize it for me.
I know this is possible, but it doesn't happen. So then, yeah, I just, I started it and it came together quite well. When I released it, it was like, boom, the stars just started accumulating. Got lots of feedback. It was very exciting. And I kept building on it. It was kind of like issue-driven development. So people would just ask for something. Oh, that's a good idea. And go ahead and implement it. And it grew from there and it became really large.
I did actually, prior to Rich, I had a library called PyFileSystem. And this is kind of like a wrapper to file systems. So you can have the same interface for your FTP server as a hard drive for a zip file, S3 bucket. And that got some use in the community. It wasn't enormous. But that taught me a lot about building open source, you know, managing feedback issues, et cetera. So that was a great experience when I started working on Rich.
And I was very surprised, actually, how successful it became. I remember the first time I realized that this was bigger than just a hobby project is when someone told me off for violating Semver, I released a clear, a clear breaking change because I thought, nobody's using that yet. But they were. So the next day I got told off quite appropriately. And then I thought, okay, I'll have to take this more seriously.
If people are using this in their day job, they can't just have someone who's just like throwing new bits of code and changing functionality. They had to treat it like it was an actual my day job. Yeah, well, everyone on the call here probably has a little bit of nervousness about like, if I break this, there are a lot of people that depend on this thing. And it's so different. It just changes so much.
When you were talking about the issue-driven development thing too, I just remember that phase of Ruff where it was like, anyone who cared all about the project, all I wanted to do was make them happy. And I was like, oh, wow, that seems like a cool idea. Let's definitely do it. And I was just like, it was the point in time where I could fix the bug, cut a release the same day, and then their thing is fixed. And then it's like, okay, great. Now we have a relationship. Thanks for using my thing.
I just think like, but that's changed so much. Right? Because then I went through the same experiences of like, I ship a release with a breaking change that I didn't really document. A lot of people get upset. And then you realize, okay, I actually have some more responsibility now. Yeah, these days, it's more like trying to find the balance between saying no to stuff that you then have to end up maintaining and not like disappointing people too much because you say no or things like that.
And then trying to keep this whole interaction with people nice, even though you don't want to do stuff that they want you to do because you know it's better for the project. This portion of Talk Python To Me is brought to you by Sentry. You know Sentry for the air monitoring service, the one that we use right here at Talk Python. But this time, I want to tell you about a new and free workshop. Heaming the Kraken, managing a Python monorepo with Sentry.
Join Salma Alam Nayour, senior developer advocate at Sentry, and David Winterbottom, head of engineering at Kraken Technologies, for an inside look into how he and his team develop, deploy and maintain a rapidly evolving Python monorepo with over 4 million lines of code that powers the Kraken utility platform.
In this workshop, David will share how his department of 500 developers, who deploy around 200 times a day, use Sentry to reduce noise, prioritize issues, and maintain code quality without relying on a dedicated Q&A team. You'll learn how to find and fix root causes of crashes, ways to prioritize the most urgent crashes and errors, and tips to streamline your workflow. Join them for free on Tuesday, February 27th, 2024, at 2 a.m. Pacific time. Just visit talkpython.fm slash sentry-monorepo.
That link is in your podcast player show notes. 2 a.m. might be a little early here in the U.S., but go ahead and sign up anyway if you're a U.S. listener, because I'm sure they'll email you about a follow-up recording as well. Thank you to Sentry for supporting this episode. PRs are like cake or puppies. If it's a simple bug fix, that's like cake. You'd like, thank you. I enjoyed that cake. Then move on. But some PRs are like a puppy. It's like terrific.
You like puppies, but you've got to feed them and clean up after them. Why am I standing in the rain with this puppy on like a Friday night? How do I get here? Collecting the book is poop. Yes, exactly. Sometimes you say thank you, but I just can't, I just can't look after another puppy right now. That's a difficult thing to come to terms when the project gets a bit more mature. Because previously, you're accepting all the puppies, but then you have to start saying no, no to puppies.
And Rich definitely got there because it accumulated. Do you hate puppies? Come on, that's a pretty hard stance. I'm kind of a cat person, to be honest with you. But yeah, I mean, you have to say no. Unacceptable. Eventually, yeah. I think this is where plugins always kind of like take the center stage, right? You're like, just, oh, that sounds like a great idea for you to maintain external to my library. That was exactly my approach to the whole situation. Yep. Here's a plugin API. Have fun.
But also the probability of some little change breaking someone's code and like being considered a breaking change grows as the project grows in usage. And like, you know, like at some point it's like almost any change will end up breaking someone in very unexpected ways because they are doing something really, really weird. But you know, like someone is doing it. So like it becomes more and more difficult to know like, is this a breaking change or not?
Like no one should be using this variable here or this parameter, but like, you know, there's someone out there doing that. So I feel defining what is actually a breaking change and what is, what is this trick somewhere gets more difficult as things grow. Yeah, I don't know. One of the things I like that the Django project did was kind of basically only things that are documented are supported.
Basically, that was kind of the line they drew and I thought that was a pretty good, pretty good way to draw it. But yeah, you're always, you know, that never actually works. That doesn't make people happy. It just gives you plausible deniability. Don't yell at me. It's not my fault. I think adding a formal like versioning policy is one of the best things that we did because for a long time rough was just we only used patch releases. So we got to like zero, zero, like 285 or something.
And we had basically no guarantees about what would or wouldn't change like across releases. And as the project got more and more popular, like that started to cause more and more problems. And so Zany, someone on our team, like when they joined, one of the first few things they did was like create an actual formal versioning policy and we added like preview behavior. So like you can opt similar to what block has kind of opt in to like breaking or more experimental changes.
and so now we have like clear expectations around what it means to like bump a minor release, et cetera, et cetera. And like that has made our lives a lot easier, like actually having clear expectations around that that are communicated and respected. But it's the kind of thing that you just don't think about at all until at least I didn't. Until I checked on figure. Yeah. Some people's code runs things like FastAPI or Octoprint or whatever. Your stuff rewrites people's code.
Yeah, but at least it doesn't run at runtime. I don't know. I actually think it's easier. Yeah. Indeed. All right. I find you talk about the plugins thing because like I've actually very intentionally taken the opposite approach, which is we have like almost no public API because like the only public API is the CLI and like we don't expose our API in any other way. And that's because like we know we're going to change like everything internally, like pretty dramatically.
And so we wanted to have like full control over that without having to worry about breaking people's stuff yet. But it's like it's starting to become more of a problem because more and more people want to use it as a library in like different ways. But it's sort of a counterintuitive way to like make our lives easier as maintainers was like not expose any public API apart from the CLI. Yeah, very interesting. That's a funny, funny sort of anecdote.
I'm rewriting a lot of the documentation for SQL model because I want to have examples. Right now I have examples that are compatible with Python 3.7 and above, but actually 3.6 and above. But I want to have also the syntax for 3.9 and 3.10. 3.10, you can have like the unions using the vertical bar and these things. And I want to have examples for each one of those.
So the approach I did was to write a script that will automatically update each one of the files by calling rough as a soup process. So that's the API. And then like, you know, like I mean, the process is like doing all that stuff. But like, yeah, it's like another fanboy of trying to use it as an API before it's available. Yeah, with lack of an API, right? An API will be created. Exactly.
I've heard this before as like Hiram's law, which is like with a sufficiently large number of users, like any implementation detail become someone will eventually rely on an implementation detail. Like any arbitrary implementation to someone is probably relying on that behavior, which is basically the behavior of the program is the API. And unfortunately, they'll find the underscore functions.
You've tried to dissuade them from using and all the things, you know, Eric, what's the origin story for Read the Docs and what did you try and how do you think it caught on? Yeah, so I mean, this was kind of way back in the day, but yeah, I kind of got started writing Django plugins, you know, apropos to the conversation.
You know, I was working at the Lawrence Journal world where Django came from and it was kind of early in my career and I was just kind of getting excited about open source and blogging and just basically built a few of these kind of testing related open source projects and then that was kind of where I got started with open source and then basically it was the classic scratch-on-itch thing, right? Like it's like I have a bunch of open source projects I want to write documentation.
How do I solve that problem? And back in the day that was a much harder problem to solve, right? It was basically just like build a zip file and upload it to packages.python.org if folks remember that one or was it packages or whatever the docs, there was a docs hosting on PyPI basically. And yeah, basically just wanted to kind of build a better version of that that in the integrated with GitHub. I feel like web hooks were like the cool new thing back in 2010.
And so that was really kind of the insight, right? It's like let's build this kind of like CICD workflow on top of web hooks and actually did build kind of a version of that the previous year around kind of like code quality stuff. So it actually ran kind of like a linter and built a website on commit and had a similar kind of workflow, right? Where it like gave you a grade and did all this kind of stuff, which is like 2009. And yeah, that didn't really catch on at all.
But then I think Read the Docs just we built it and then I was using it for my own projects and I actually had the need to kind of maintain it and keep it updated. And then I think people just kind of grew naturally, you know, gave some conference talks, that kind of stuff. But I think it just solved a problem that people had and that's always going to be the best way to grow a thing. Yeah, absolutely. Awesome. What's Read the Docs written in? Is it Django? Yeah, it's all Django and Python.
Funny coincidence. I went to college in Lawrence at the University of Kansas. Oh, nice. I was right there, right at the heart of Django, but I moved off to grad school like a couple years before all that happened. So I missed, missed the excitement. I grew up in Virginia and everyone was like, you're going to Kansas? You graduated school? Like, how are you going there? Like, what?
Everybody thought it was the weirdest decision, but yeah, like I really do think, you know, ending up in that Python and Django ecosystem has been pretty transformative to my life. So, you know, worked out. Lawrence is a pretty cool little town, actually, of all the places. Yeah. Sebastian, how did you come about this crazy idea to put types into our web apps? You know, Python, that's dynamic. It doesn't have types. What are you doing? Yeah, I don't know. It's so crazy.
It's so fun to see it said Michael Larson in the chat, like, they maintain our URL E3 and the Python security developer in Resilience just, like, chatting along with us. Like, he probably has had to deal with so much stuff as us. Yeah, I'm sure.
So, like, some of the first things that I did in open source were actually Docker images for deploying Flask because I was working with Flask and deploying Flask was difficult and I needed to be able to combine Nginx with UWSGI and, like, a bunch of things and, like, they all had their own custom configuration files and it was, you know, like, so difficult.
I didn't like doing that and then I just had to study how to do that stuff and then after going through all that I wanted to save everyone else's time doing that. So, we're like, well, let's just put a Docker image with this and a lot of documentation of how this Docker image works and how you can use it with some sensible defaults. So, you know, it was just like a weird contraption that was it but it actually grew and, like, got, like, a few stars, like, a bunch of stars.
For me, it was a lot, you know, like, 100 stars. Oh, my gosh, I have an open source developer. And then, like, at some point it had, like, a thousand stars or something like that. I was so happy about that. And at some point it was kind of the de facto standard for doing Flask, Docker, Flask-indocker. that was the first thing that I did. I ended up with FastAPI. I was avoiding building FastAPI for a long while and I was trying all the other frameworks and all the other tools.
I was convinced that there was something that would do the things that I wanted. I just had to find it. And as I was trying different frameworks, also in different languages, I was extending the list of things that I wanted to have and also the list of things that I wanted not to have. For example, I didn't want to... That takes you farther and farther away from any framework so you're out. having one that would tickle the boxes. So it was like, ah.
But then I realized I really like this thing of having types because you get out of completion and inline errors and this is so cool. It's so cool to be able to have this. I want to have this in Python and then Python added type annotations and it was like, this is great. How do I use them? There's no way to use them with the current frameworks. So at some point I actually found the right framework. It was called API Star by the same author of Django REST framework.
It was just missing some authentication stuff. I said like, okay, I'm going to contribute to the other stuff. When I was about to jump into the code, he said, I have to deprecate this. I will go focus full on Starlet. And then he went to build Starlet. This is Tom Christie, which is super prolific and an amazing open source person in general. And then at that point is when I said like, ah, just have to try it. Let's just do it.
So I'm just going to try to build something that will be kind of a spiritual successor to API Star built on top of Starlet. So I was, you know, like I was actually narrating all the learnings from Flask, Django, Django REST framework and all the ecosystem and just like bringing all those ideas together. At least that was my intention. And then I wanted to have a bit of better type annotations. So I saw that Pydantic was using standard type annotation.
So people wouldn't have to learn this like, you know, from FastAPI for special string or something like that. Just instead of that, just use pure string.
So I wanted to have something based on standards like OpenAPI, JSON schema and all that stuff and based on standard Python and to have like the simplest syntax possible and to, you know, like give the best developer experience possible while, you know, like not adding like any additional steps for developers to build something that by default will have all the best practices built in. That was the intention. And I was just like trying to solve it for the things that I was working on.
I was supposed to be doing AI and machine learning and stuff, but like I had to stop for a bit to solve APIs. I got stuck in APIs. That's how I ended up with FastAPI. Oh, I think you made a pretty decent choice. It seems like FastAPI is doing all right. Yeah. Yeah, I just saw you. It's much better than what I ever had expected. Yeah, I know. Congratulations.
Well, it's also deserved, but I also just saw an X, I don't know, a post on X Twitter, whatever you, however you, addressed these things, where you said you showed a graph where the number of GitHub stars for FastAPI just passed Flask. And, you know, I have a lot of respect for Flask and the Palette team and David Lord and all those folks, but, you know, awesome, awesome that your stuff has taken off so much. That's really cool. It's super cool. It's amazing.
And, you know, like Flask was one of the big inspirations for FastAPI, and I've been able to be at this point. It's crazy. Like, yeah, I still can't get over it. Yeah, super cool. super cool to be able to build all this and to keep building more stuff. So, yeah, super nice. one final comment before we move on to Octoprint. But, one of the things I kind of see you doing in the world is you're like the combiner.
You're like, oh, we've got this cool stuff with Pydantic and Starlet, and how can we combine it in this way to make this, you know, and Swagger. And, like, similarly with SQL model, you're like, well, Pydantic is cool, but SQLAlchemy is kind of cool, but it could be more or better, you know, like, it could be better, right? So, anyway, good job. Yeah, and also, like, Typer, that is the library for building command line applications.
It just click with the type annotations, the same ideas from Pydantic, and now with, you know, like, integrated building support for reach. So, like, also, picking Will's work, trying just to put a bunch of things together. I'm just, you know, like, I'm just making cocktails everywhere. Yes, exactly. It's more than that, but there is a lot of value in combining things in a smart way and accessible way. Yeah, awesome.
All right, Gina, you told us a little bit about the origin story, but like, what did you try before? And then I have a follow-up question that I think is unique to your project. I wouldn't necessarily say that I tried anything before because the whole thing was completely unintentional. I basically bought myself a 3D printer in late 2012. I wanted to be able to put it in my spare bathroom and monitor it from afar because back then this thing was tying up my PC.
You had to constantly keep it connected so it could operate and that for hours and hours and it made noises and it produced fumes and I just wanted it out of my office and to be able to play games on my PC again instead of having it basically communicate constantly with a 3D printer.
And so I bought myself a Raspberry Pi and I was looking online for something to be able to just put on this Pi and attach to the printer and throw a Wi-Fi dongle in because back then the Pi didn't have Wi-Fi built in and throw that in the spare bathroom but there wasn't anything. So over the course of my Christmas break in 2012 I sat down and changed that and this is basically the origin story of Octoprint. I just wanted to scratch my own itch. I wanted to put my printer in my spare bathroom.
This is all. And apparently a lot of people had the same problem because I just you know like how we like to do these things. We build something that we think might be interesting.
We throw it up on GitHub and go just like here go nuts enjoy have fun with that and I suddenly started getting emails from all around the world like hey I have this in this printer can you also make it work with that and so this escalated and I went like oh someone is using it of course I'm going to add this support and of course I'm going to add this feature and it just grew and grew and grew and grew and apparently I just hit a nerve so this was utterly unplanned this was never
my intention at all and I just wanted to solve my own problem and have continued to solve my own problems and the problems of other people ever since.
Before Octoprint I actually was also quite active in a little project called DocuViki where I was developing some plugins for that and I even did open source before I knew what open source was with some PHP scripts when I was 18 or so that I threw up on my website but yeah Octoprint was like it took over my life it just was an accident a happy little accident maybe. How did you decide okay this is a job for me rather than just a thing you worked on?
The thing is by mid-2030 2013 I went to 80% on my regular day job I used to be a software architect slash consultant with big corporation Java World all the enterprise stuff and to be able to dedicate one day per week fully to Octoprint plus of course the weekends and the after hours and the vacations and all of that and that still didn't suffice anymore by 2014 and I noticed that it was impacting my health it was impacting my relationships and that was actually quite perfect timing really when
this company that hired me initially approached me and said hey do you want to maybe fly out to us and we'll chat and if you like all of that what we have to propose then we can just do that and this is how it then continued to go from 2014 until 2016 and yeah it was never my intention to become self-employed I'm a quite risk-averse person actually so when this point came where I was like okay either I find a way to keep funding this or I have to really drop it because it was way too big
by then to be kept as a pet project without being utterly utterly unhealthy yeah I decided to jump into the cold water and have been trying to keep up at the surface ever since basically so yeah all of that really completely unplanned cool what an adventure indeed yeah and the good thing is even if people don't understand what open source is or what code is I always have quite a topic at parties so that is fun because yeah like people give you money for something they can get for free what
exactly yeah if you're not familiar with open source then it definitely is a weird it was tricky to find a text consultant who understands the concept so yeah I can imagine that's interesting I'm sure all right I was very recently I was like what do you do what is your day oh I'm a software developer so you work for a company not really what is it that you build well I build a platform that is free for others to use and who's paying you a company that pays money to companies to build companies
so you're building a company no it's actually you know like it was a long conversation they got bored and later this doesn't make any sense yeah I've just taken to say I'm a software developer and I work with 3D printers and then people usually stop asking questions because 3D printers are this mysterious thing that no one understands anyhow usually at least and then yeah unless they say oh 3D printers and then I say yeah do you know 3D printers and they go yeah yeah I have one
and then I can say oh do you know Octoprint yes oh yeah I made that and then oh yeah it's tricky to explain to people I'm sometimes not even sure my parents understand what I do so I'm sure that my parents don't understand what I do that's okay I'm sure they're still proud of you anyway it's fine yeah my definitely are proud of you I would like to just maybe getting a little short on time you're coming up to the end of our Avengers meeting let's close it out with this go around one more time
let y'all or maybe just as a group kind of chime in on this speaking to the people out there listening who want to start an open source project or want to contribute to open source or somehow kind of get involved in similar ways what would you do different if you started now many of you have been working on this for a long time you've had a lot of experience like if somebody said well the world is somehow the memory has been erased from the fact that rough or rich or octafrid or whatever existed
to start over what would you do different or the same I would use FastAPI yay good use of a time machine there everybody I've had multiple conversations over the years where people are like why don't you use this library I'm like oh it didn't exist exactly yeah this is the problem exactly when you actually have a plug-in system then swapping out stuff like that can become very tricky absolutely when I get asked what to work on what to study what to focus how to get into open source or like
almost any of those questions I always say that the main advice I give is just to focus on a problem that is important to you more than you know like innovation market disruption or war whatever just like focus on a problem that is actually important to you hopefully that is important to others as well and if it's not a problem that affects you directly hopefully it affects someone that is very close to you so you can get like a very tight feedback loop of what you are building and then try to
solve it and then use that as the guide of what to do what to learn what to focus on what to do in many cases it's just learning a framework that is already there you know like why would I go and build a system to control 3D printers instead of just like learning how to use auto print in many cases solving the problem is just like using the tool that is already there but then in some cases you end up figuring out that there's no tool and you just have to build it there's actually no lint there
and formatter that can be super fast so you can run it every time you just hit save so then you just have to build it in Rust and create rough or like you know like there's always like this thing that you are trying to solve that is just not solved there yet and that is what gives you the best value and the best outcome in many cases it can be just like you know contributing to another open source project or building something new but the thing is like for me it's just like focusing on a problem
that is important that is what has worked for me at least good advice I would add to that that you really shouldn't try to you need to be really passionate about the thing as well right it needs to be something that is really something you're into something that will that you will want to spend a lot of time on that is actually not your working hours and such because otherwise the whole jog until this becomes anything viable anything big enough to support you in any kind of way even if it's just
being able to drink a coffee per week or something that will be a quiet long slog and a lot of work and a lot of blood and sweat and tears and so you really need to be into this so that you want to do that if you are just in that for the end goal of I don't know becoming rich through open source which by the way will probably not work only will I go rich then this will not work out you need to focus on the path to the goal a lot of people try to reach for the end goal before they are willing to
walk the path I happen to be in the GitHub star program and I've had a lot of people ask me how do you become a GitHub star and that is the wrong question to ask how do you become an open source maintainer of a popular project you just have to do whatever you are interested in you have to do work and you have to be passionate about your work and then maybe if you hit the right nerve then that will happen and maybe it won't I think it's very interesting that none of us had a career path to where
we are now we didn't set out to be where we now are it just feels like we followed our intuition and it worked out which means it's very difficult when someone asks me how do you become an open source developer how do you start a company I'm not quite sure I can tell you my path to there but it's very hard for me to articulate to someone else how to get there I think there's a lot of luck involved as well right it's like you have to do all this stuff and roll this dice if it gets a six you can
be successful there's an element of luck but it's kind of luck you make for yourself and it might take you to somewhere else or it might not but as long as you're passionate about it and you enjoy it then great things will happen I think also Will and Charlie you both took some specific time on your own money to really level up what you're working on which is pretty brave not a lot of people would say I'm just going to spend my savings to work on this project and see if I can make it go so
you've earned some of that through taking that chance and putting that time and effort I think for me that didn't last long because I'm after 15 months I was living on GitHub sponsorship VCs came along and then there was cash everything changed and you had to learn a whole new set of skills I'm sure I do feel lucky that like I said this is my first time being a maintainer and it was very clear to me quickly that for Ruff to have the pace of development and the scope that it had to be a full time
thing and that was pretty obvious to me quickly and I have a lot of respect for people who have been maintainers for longer periods of time and in a way where it's not their full time job or they have to do it on the side I have been in that position but it's already clear to me that that takes a lot of dedication and commitment so I feel very lucky that I get to work on open source full time I know it's not all entirely luck right like you increase your luck surface area and opportunities come
your way but I do think it's a fortunate position that these kinds of opportunities do exist because maintaining popular stuff is a lot there are privileges that come with it 100% but it's a lot of work and a lot of stress and a lot of responsibility and you don't necessarily have to start your own right away I didn't have gray hair before this you don't have to start your own either you could contribute to a really popular one and get into the scene and you know like Eric that sounds a little
bit like your story like you were in the Django scene and then spun off from that what you are doing right yeah definitely and you know part of an open source community and definitely going to PyCon and just being surrounded by those people because that was always the it's very lonely work just to have an inbox of github issues and no inbox thank you you know and like going to conferences and that kind of stuff was really where you feel the appreciation and you really kind of actually feel the
value that you're providing as well as just the stress of the things you're breaking so absolutely the conferences are great to really feel the appreciation rather than just the request they'll say thank you and then they'll give you a bug report but at least the thank you I'm talking to you there is this can I just show you this an interesting observation for me a year ago I didn't know anyone in Python open source at all I was a user of Python in my day job every day but I was not interacting
with Python in a year now like Sebastian we've interacted a bunch there's a lot of people that I've come to know and think of as friends so I think especially if you're interested in getting involved and putting in work I really don't think there are significant barriers and you get out of it what you put in so I think I've been very impressed with just how welcoming and friendly the community has been especially other maintainers yeah absolutely talk to a lot of people who show up at PyCon
and they're like I was really nervous to come here or I feel like it didn't fit in and they're you know just had such a great experience and I said well did you feel like that was my first PyCon this year I'd never been to a fight on conference and so I was like we did up at such an amazing party that's true that was interesting all right guys well I think we are pretty much out of time anyone want to have some final thoughts for listeners before we wrap it up go and build some cool stuff who's
brave enough how can maintain it today I think give cake not puppies let's leave it with that huh all right give cake not puppies and be really careful before you start a popular source project it might take over your life if you want this great but if not then yeah careful what you wish for you might get it exactly Gina Sebastian Eric Will Charlie thank you all for being on the show this has been a a ton of fun thank you very much for having us a pleasure and honor to be with these amazing
people thank you for the invitation I agree honestly I had a blast just hearing everyone's stories because I hadn't heard any of this before so thanks thanks to everyone else and thanks Michael yeah you bet bye bye bye bye bye this has been another episode of talk python to me thank you to our sponsors be sure to check out what they're offering it really helps support the show this episode is sponsored by base dash base dash uses AI to build a dashboard for your database get a custom admin view
in your postgres Microsoft SQL server my SQL Maria DB or Redshift database get started for free at talkpython.fm slash base dash take some stress out of your life get notified immediately about errors and performance issues in your web or mobile applications with Sentry just visit talkpython.fm slash Sentry and get started for free and be sure to use the promo code talkpython all one word want to level up your python we have one of the largest catalogs of python video courses over at talkpython
our content ranges from true beginners to deeply advanced topics like memory and async and best of all there's not a subscription in sight check it out for yourself at training.talkpython.fm be sure to subscribe to the show open your favorite podcast app and search for python we should be right at the top you can also find the iTunes feed at slash iTunes the Google play feed at slash play and the direct RSS feed at slash RSS on talkpython.fm we're live streaming most of our recordings these days
if you want to be part of the show and have your comments featured on the air be sure to subscribe to our YouTube channel at talkpython.fm slash YouTube this is your host Michael Kennedy thanks so much for listening I really appreciate it now get out there and write some Python code Music Music Music Music Music Music