Hey folks, Michael here. Brian and I put together a special podcast episode for the top Python stories of 2018. We originally recorded this as a talk Python to me episode. That's my other podcast. Since it's a perfect match for what we do every week here on this show, Brian and I decided to share it with all of you for our end of year countdown on Python bytes. We hope you enjoy the episode and thank you for making this show so successful in 2018. We promise we'll keep it
going strong into 2019. Brian, Dan, both of you, welcome back to talk Python. Thank you. Hey, thanks. Great to be back again. Yeah, it's great to have you back as always. It's going to be so much fun and it is time to look back on 2018. Can you believe it's the end of 2018? I just remembered how to write 2018 and not 2017 on all the stuff I have to sign. Yeah, it's, it's, it's gone by fast. It definitely has. It definitely has. So it's good to have you back. We're going to go through
what the three of us have decided are some of the top stories in the Python space for 2018. And I just want to quickly throw out there for folks listening. This mostly does not include data science stories. There's a little bit here and there, but it's, it's mostly pure Python. And the reason is I'm also doing a show on data science year in review. So save those big stories for that one. Don't want to overlap.
All right. Well, we're, we're going to start and do a top 10 countdown and we're going to go from really interesting to the absolute big news at the end. And you can see what story we have picked to be the number one story of 2018 for the Python space. Let's do a little warmup here and Dan, I'll let you kick this off. What was our number 10 item? Sounds good. So our number 10 is the Python 3.7 release and more specifically the performance
optimizations in there. And I know you had a really good show on the podcast here with Anthony Shaw, where you guys talked about, you know, some of the changes in 3.7. And I just thought that 3.7 should be on this list because it was a really, really good release. You know, there's a lot of like free performance improvements. It's always nice to see that. And it's just been awesome to see Python 3 and see Python 3 specifically gets so much love and that it's under active development and it's
making this forward progress. And I think it's also the release, or I think it was either 3.6 or 3.7 where Python 3 is now faster in all of the benchmarks than legacy Python, except for startup time. So I thought, you know, that's kind of a nice milestone for Python 3. I think that's a huge milestone for Python 3. I mean, this is sort of the year that the tide has turned for Python 3 in a big way. I think it was starting to turn before, but really this year, I think that's,
that's one of the themes. And this is a major aspect of it, right? That Python 3 is now faster. There were always these couple of reasons people say, I'm holding out to Python 2. You guys with your fancy Python 3 and your new keywords can just go play because I, my code is faster and faster beats everything, right? Well, maybe not. Yeah. Well, I mean, faster never really beats everything. It's,
it's faster development time too. And when you have a better data structures, you can sometimes develop faster. However, you're right that a lot of people were holding out because of the, some of the speed and, you know, and for certain, some applications that makes sense. But yeah, I think 3.7 is definitely faster. And, and the other thing is a lot of people had already, some people for some of their projects
had already switched to Python 3.6. And that was probably from 2.7 to 3.6 was not an easy switch. Having a switch from 3.6 to 3.7 be like, oh, I don't have to do anything. It's just better now. That was a nice thing to have. Yeah. That's super nice. And I think that's a really great point you raised, Brian, that, you know, careful what you're optimizing for, right? Like if it was really all
about speed, wouldn't we just be writing assembly code still, you know? Yeah. I bet somebody good at writing assembly could crush the performance of Python or a lot of C++, but you know, they would release an app, a small app every 10 years and it would be buggy, right? Like there's just, it's not the most important thing. That's, that's for sure. One of my, my favorite examples there is, what's this, it's, what's this game called? It's Rollercoaster Tycoon, which was written by one person
in Win32 x86 assembly. It's like a hundred percent assembly code. And it's actually an amazing, amazing game. And it's, it's just, it just blows my mind. Like everything was written in assembly language. Wow. And it probably runs like totally smooth on like a 286 or something, right? Yeah. No, I think you needed a Pentium or something, but, you know, now somebody's working on like a Python, going to need a, I don't know, multi-core CPUs and whatnot to run
at the same performance. Yeah. Who knows? But you know, another, another interesting thing that's coming out is more and more talk about actually compiling Python, which I think, you know, didn't make the list, but it's definitely something that, that came up like my PyC for example,
and Cython and whatnot. Yeah. Yeah. So if we're going to release a new library or an awesome game, like Rollercoaster Tycoon or something like that, and we were going to do it in open source, probably the main full release after it's like totally stable and spent out for a year, it'd be like 0.01 or, or 0.3, maybe an 0.3, like a 0.12, I don't know, something really, really small, right? Like that. Right. And that's been a theme that's gone through a lot of,
a lot of open source. So I think I'd like to highlight a little bit of shift in just the versioning that seems to be happening around projects. I want to kick off this part of the conversation with talking about Mahmoud Hashemi's project called Zerover. Brian, do you remember covering that on Python Bytes? Yeah. He released it like as an April Fool's thing, but it actually highlighted a lot of, a lot of things about different projects that were zero over after like,
you know, years. Yeah. So the idea is kind of like I was describing at the beginning, like so many projects just never even reach version 1.0. And in some sense, it doesn't matter, right? Is the version number smaller or bigger? That tells you whether you have an older or new one, right? So that's fine. But there's a lot of people, especially coming from the enterprise space or from the commercial software space that sees things like 0.20 and go, oh, that thing's like a
super alpha. We're going to stay away from that until that reaches 1.0. And I think it has a tangible effect on turning off a certain category of folks, especially people who are maybe unfamiliar with open source and things like that. Yeah, I think definitely. We, I mean, a lot of people do commercial software development and like I do and, the zero, zero versions always, or almost always mean the API is in flux. You can't trust it. You shouldn't program against it because it can, the
developers can change their mind at any time. And it isn't until the 1.0 version where we have a, a stable, a stable API that people can count on. Yeah. And so I think a lot of people perceive it that way, right? Yeah. It's interesting to consider the, the version number as part of the user interface, you know, for, for an open source project, which makes absolute sense because of those, those effects where people are like, yeah, you know, is this, is this really ready to go here?
Is this a, an early release or whatever? I mean, I'm wondering if it's really going to have a big impact, you know, if people are moving to something like, like dating their releases, like just calling them 2018 dot something, something, and you sort of get the November
release, but it's, it's an interesting experiment for sure. I mean, it definitely looks a little bit nicer than, you know, having, I don't know, like somebody put pandas, pandas here, which had like 83 releases and it's been running or it's been active for eight years and it's sitting at 0.24 .0.dev. .dev. .dev. .dev. .dev. Nothing wrong with that, but I can see how it turns some people off. Yeah. I don't know how dev zero means production, but you know, to me that just says like, dude,
stay away from this. Don't ship this. That's scary. Yeah. I think the, you know, maybe the grandfather of zero for might be putty, you know, that little, SSH tool you get for windows cause it doesn't come built in. Uh, that'll trade app. it's 20 years old and it's 0.7.
One concern that I have with moving to the date based or like year based version numbers is how, like if you have the semantic versions, you can do things like, okay, I want, give me all the minor patches for this package and all the security updates, but I don't want to jump to the next major version, which maybe has a breaking changes in it. And I know that's not a perfect system. Generally you want to pin your packages, but it's kind of nice to have
the ability to do that. And with, you know, who knows what's going to happen in like July, 2018 or something 19. Yeah, absolutely. So if you look at some of the tools that I know you use and I use, for example, I use pyup.io and use something slightly different, but they let you go to your requirements file or your pip file and say, I want you to keep this on the latest pin version possible with this constraint where you could say like 0.1.star or something like that. Right.
Right. Yeah. But like, how would that work with, you know, date based formats? Exactly. So let's dig into that a little bit. So what are the two options? I guess the two options are just like to start shipping like full versions, like Django is now Django two and the Django two, the main change from one to two is we're dropping Python to support, which is kind of ironic. Maybe it should be Django three, but whatever. You know, it indicates a major change, like you were saying there,
right? So one of the projects that's switching to calendar based version is Sanic, right? The async web framework. And what they're doing is they're trying to ship a version, a major release every quarter. So they might be, you know, 2018.07.0, then .1.2 to for like incremental changes. But they're also doing LTS releases around their calendar version. So every December is an LTS version. So 2018.12.0 and so on would be
supported for a year and would be the LTS equivalent, the long-term support equivalent. If you don't want to switch to something beyond that, that's how that group is handling it. Interesting. Yeah. I kind of like that. Then you can glance at a version and I mean, if everybody sort of supports something like that, you can glance at all of your versions and go, Oh, everything's within a year or so. So it's, it's probably fine. Exactly. Like Brian, you know a lot about testing. If I told
you I'm using web test.2.0.32, is that new? Is it super old? Is that a beta version? Like what is it? Like, what is that? Right. You don't know. Yeah. But if I told you I was using web test 2019.1.0, like, Oh, you must be like testing the new one that's coming or, you know, something I, it's just communicates it. So Sanic is doing this. pip is doing this. and then you threw some others in there, Brian, what are these? I'm not familiar with them. There's some descriptions of calendar
version and semantic version, calver.org and semver.org. I love that those are domains. I think Mahmoud did at least one of them, maybe both. He has this knack for creating these little sites and these little projects that just pop out of nowhere. But the, the, the, one of the things I want to point out is I was talking with somebody about calendar versioning and that versus semantic versioning. And one of the things is that the major number, whether it's two or three or it's a date
based often means, you're not going to, you're not going to remove features. So you promise not to break customer code within a major version. The date base kind of says we're at least going to stay. it's safe for a year. Yeah. Whereas a semantic versioning, you know, a development team can go through major versions as fast as they want to, if they feel like breaking things fast. That's true. That's true. It's just when you get the feeling to switch the major version from two to three or three
to four, you just do it, right? Yeah. But it has impact. So, yeah, I think there's some interest there. So I think another thing that was really interesting about the calendar versioning that I learned talking to Adam Hopkins from the Sanic project was that there's this sense of you need to keep shipping, right? Like if you have a calendar version and it's from last year and that's the latest version that sends a message in and of itself that this project is, is not getting a lot of activity.
Whereas without it, you know, you kind of got to be more deeply familiar to the project to understand its rate of improvement. Yeah. I think even it'd be cool to have people, even if there's not really any changes needed just to release it and say, Hey, we tested it. It's, this has all been tested on the new versions and you're good to go. We updated like the readme file or something. Right. Well, there's almost always like some minor little thing that can be,
can be improved and shipped. Right. Yeah. Another story of 2018 and maybe slightly before, but really it was coming in strong in 2018 is Python's kind of popular. Yeah. So, Python's very popular. It's taking over the world. There was a neat economist article that came out and that, I guess that's just sort of related to the, they're highlighting a trend. And the article was, Python has brought computer programming to a vast new audience.
But some of the things they point out within there is that, right now, according to a stack overflow survey, 40% of professional developers use Python and another 25% wish they did, which is interesting. It just, just go do it, man. Also the, they interviewed, code Academy, which is an online training site and their biggest increase in demand, for 2018 is with Python. Yeah. That doesn't surprise me. I mean, you think about a very fast growing technology that also has
a lot of career possibilities. Definitely people are going to be wanting to learn that, right? So if you go to somewhere like code Academy or any other online trading place, like there's going to be, unless they're doing something wrong, there should be a lot of demand for Python. They noted that since 2014, at least in us from a survey in 2014, Python was the most popular language, intro language at American universities, but that was amongst, people, studying
sciences or computer science or something engineering. But what we're seeing now in 2018 is the growing movement of people that are not traditionally think of themselves as programmers, primarily learning Python. And, we'll see that in data sciences, embedded development is now moving towards Python and in a lot of maker senses. And then also the article also pointed out that more and more people are just sort of worried that all jobs will be taken over by automation. So they want
to be part of that. So in order to, to learn some automation on your own, Python is a good way to, to be part of that. You either control the robots or you get replaced by the robots. Take your pick, right? Yeah. Yeah. Also to throw in there, I saw this on, Nina Sakarenko's Twitter feed recently. so Python
is becoming, or has been nominated as the official programming language for education in France. So I think it's going to be part of the high school curriculum there, which is, that's super cool. I think it's, we're seeing the snowball effect, you know, where, where people, it's really becoming the dominating language for beginners and educational language. And I think that's super cool. That is so cool. And when I saw that on Nina's post as well, I'm like, Oh, that is amazing. And you
know, I've been thinking a lot about this. Like why is, is Python both popular as a teaching language and popular as a professional programming language? Like C++ is popular as a program, professional programming language, but they don't suggest that kids start with it. Right. Things like this. I think there's this special balance that when the language was created, it has this way to get started easily and to pull in just the aspects that you want. Right. So you can start out with not even
functions and then like, okay, I need some structure. I'm going to put some functions in here. Oh, let's start working with classes and things like generators, but none of that you don't have to understand or use or care about that until you're ready. So like Python has a special ability to start easy and small, but expand into professional type of apps. Whereas a lot of them are either you're always a beginner
type of language or you're always like, yeah, that's a void star star kids. Let's talk about that. Totally. Yeah. And actually, Mike, I think you, you called it a full spectrum language way back when, when we did an interview on my, my blog with you. And that just really stuck with me because I think that's, that's so true. You know, it's, it's not just, it's not like you're learning, you know, like the little like turtle, like logo or something like that, where it's like you learn the basics of
programming. Then you're sort of stuck and you have to move on and learn something else, but no, you can, you know, all the skills you learn working with Python, they kind of scale and, and grow with you. And that's, that's just a beautiful thing. It's special. There's not many other languages like that. What a time to be alive. It really is an amazing time to be alive. Before we get on the next one, let me just tell you all about digital ocean. They're doing all sorts of
cool stuff. Our infrastructure runs on it really, really nice and reliable. One of the things I want to highlight this time is their work with Kubernetes, Docker and coordinating Docker, orchestrating Docker stuff with Kubernetes is a big deal these days. And so they're launching a new Kubernetes cluster over at digital ocean. So a really nice way to manage and deploy your container workloads in the cloud. And if you go to pythonbytes.fm/digital ocean, and you're a new user, you get a hundred
dollars credit to Kubernetes all the way. If you want, you can run a lot of Kubernetes for a hundred bucks in the cloud. So that's, that's pretty awesome. That's yeah. Very cool. Yeah. So check them out. Pythonbytes.fm slash digital ocean. They're big supporters of the show and they keep us going strong each week, don't they? Yeah. I'm very grateful for them. Yep. Brian, you, you touched on that. A lot of the growth of Python was centered around people who
are using Python as a tool, maybe not as their identity, right? Like Python developers are like, I'm a Python developer, but other people are just using Python, maybe for data science or things like that. And so number, number seven in our list is that I'm going to declare 2018 as the year that the number of data scientists have come to equal the number of web developers in Python for professional Python developers or users, let's say. Okay. You're going to have to back that up a little bit.
All right. So, so there was a really cool survey done by the PSF in conjunction with JetBrains. So JetBrains did the infrastructure for sort of analyzing it and hosting it and so on, but it was promoted just through the PSF and other public sources. So it's not like tied to JetBrains, but they did this really great survey and it's called the Python developer survey 2017, which might sound like I'm wrong in my year and sort of, I am, but this is at the very end. This is like November,
December or something like that of 2017. So it's pretty much there. And they did a bunch of interesting, they asked a bunch of interesting questions and they found out a lot of things like, for example, four out of five Python developers use Python as their main language. So when you see people are using Python, mostly they're using that as their main language. It's not like, well, I write and see, but I also have this script that like compiles it in Python or some weird thing like that. Right.
So the majority of people using Python use it as their main language. And that's, that's pretty interesting. But the thing that I want to point out is they said for, you know, what do you use Python for? And if you pick where Python is main versus where Python is secondary, you get slightly different answers. But generally it says for data analysis and data science, 50% of the people use it for that. And 49% of the people use it for web development. And then there's a whole bunch
of other areas that like, you know, it's a multiple answer type of thing. So they've also checked those off. But in this survey, basically they said 50% of the people said data analysis, 49 web development. Does that surprise you guys? No, not really. I think it surprised me. I know there's a lot of data scientists. I feel like on a web development team, there's more participants than on a data analysis team.
I feel like a data analysis is often done by a smaller group of people than like the 30 people that maintain the website or, you know, whatever. Right. Like that was my conception. I don't know if that's accurate, but apparently no. Either there's even more data scientists than I guess, or the there's larger teams of data scientists and smaller teams of web developers.
So this is all interesting. Another thing that I thought was pretty interesting around this is they asked people to guess that ratio in this survey. And they gave them like a five choices, four choices, something like that. And they said, what do you think the ratio is five times the number of web developers to every data
scientist? One to one, you know, maybe worse. I can't like 10 to one. I can't remember. But when they asked that question to the group, half of whom were data scientists and half of whom were web developers based on the original question, the responses they got with only one, only 10% of the people got it right. 20%. And, most thought that it was five to one, five times web developers to one data scientist. So I think the general perception is there's more stuff happening on
the web around Python than there are data scientists. But I think also what the most surprising thing to me when I saw that result was half of the people that answered that question were data scientists, you know? And so to me, my interpretation is that means that a lot of the data scientists feel like they're not as big of a representative group of the community as they are. They feel like, oh, we're like 20% of the community. But in fact, they're like half. And I don't know why that is,
but I think it's interesting. It is interesting. Well, there's definitely like a, if you look at the hype around different packages, for instance, there's more packages that are web related than there are
data science related. Mostly I think because a lot of the data science tools have been bundled into these large groups of tools, even though they're like a common set of things that they were like data scientists like to all use sort of this, you know, a handful of different tool chains, whereas web developers like to go off and create their own asynchronous library. Yeah. Everyone's got their own library to match their new JavaScript library. Yeah. I wonder if the
anaconda distribution and things like that also play into that feeling, right? You just install it and you just have the tools, right? You don't necessarily go and hunt for every little library as much. This sort of reminds me of who coined that term? Scott Hanselman, I think the concept of the dark matter developer. Yes. I was thinking that too.
Yeah. Yeah. I think, I think you actually told me about it. So I guess the idea is that there's a large segment of the programming population or, you know, people who program for, even for a living sometimes that are just not visible in the community. They, you know, get their work done and it's just not, maybe they don't identify as a programmer or it's not just something, it's just not something that they do publicly or have like side projects and stuff like that. It's a job.
Nothing wrong with that. Yeah. And most people, like they don't go to conferences or go on social media and go crazy about their job. Like, like if I was a barista, nothing wrong with that, but like, I probably wouldn't go to barista conferences and go to barista meetups and like, you know, talk about being a barista on social media very much, right? Like it just, it just, it just, yeah, you actually have some awesome stories if you did that. Make coffee for me.
Hey, anyone out there listening, go for it. That's all you. I'm going to get that. I would listen to that. I think there's a lot of, a huge number of companies that have one data scientist or data analyst in their company. And so they feel outnumbered because they're the only one in their company or there's like the, the ratio is very much skewed within their company, but they're kind of all over the place. Yeah. Yeah. I totally agree. And I do think there's a little bit of this dark matter
stuff possibly happening here. Maybe you're not allowed to talk about your analysis that you're doing. So you just don't, maybe you've backed into becoming this data scientist business sort of analyst type person. You started out as like a trader and then you, you kind of were the person that had enough code to like figure out how to do the overall reporting and analysis and whatnot. And you still don't think of yourself as a developer, but you effectively are. Right. So
I don't know. I think there's a lot of interesting angles here. Yeah. Yeah. Also, I was just thinking like, you know, a lot more secrecy maybe in the, in those areas. Like if you're working on some state of the art, like machine learning thing, I don't know, self-driving cars or whatever, maybe you don't necessarily want to go out there and share with the world what you just did. Whereas in the web development work, I feel like we're sort of,
you know, we figured most of it out. And so people are more willing, I think, to, to share what they're working on. At least that's the impression I got. And often it's, yeah, I agree. And it's on the web already. Yeah. Whereas if you're working on the Apple self-driving car, like you can't even acknowledge its existence, right? You're like working for the CIA. Yeah. I work for the state department. Okay.
So anyway, if you want to dig into more of this, I had Eva Jelowalska and Dmitry Filipov on, they're the two folks who did the, this and they printed it in a Python. So on Talk Python 176. All right. Item number six, Brian, what do we got? We have a darkening of the community. This is a very dark topic. It is a little dark, but kind of in a good way. Black is taking over. Black is, there's, not in a bad way. So the project black,
black is a code formatter that just sort of formats all your code for you. And, and, and it like Pep8 and some of those flight gate and stuff tools were to tell you where places where you might want to look at your code and change it. Whereas black is, yeah, I'm not, I can tell you if you want me to tell you, but I'll just go ahead and change it. That's awesome. Sort of taking, there's a lot of projects around the web or around Python that are picking it up and,
and using it as their standard. And I think it's, it's been definitely very interesting in 2018 watching this happen. I love that tool. So it's so good. Like, especially if you, if it's a community project, you're working on a project with several developers and it's just so easy to standardize everything and you don't have to have the back and forth conversations of defining your own code style. And I mean, assuming you like the output that black produces, which I personally really like,
and it's, it's been super helpful for real Python. We have like an materials repository, some sample projects. So like every single folder in there was created by a different person, my different author. And so I went down that, that route of like, okay, we're going to define like a beautiful code style and then tell people how to format everything and make sure it's, it's sort of adhere to. And that's really, really hard to do. And so I just ran black on the whole
thing, set it up on CI. So we'll check, it'll check the formatting problem solved. And it took 15, 20 minutes to do that. So yeah. But what a lifesaver, such a great tool. So a couple of things about it. It's taken, it's one of the things that takes things like style and format and stuff like that, and takes those out of code reviews. And so code reviews can just be focused on algorithm and design and not on whether you use single quotes or versus double quotes or
something. The other thing is a black shows an example. It has part of it is a pre-commit hook so that you can add a black pre-commit hook to your code. So everybody just as they're, as they're checking it in, we'll have black run on their code. And that highlighted the pre-commit tool for me. And I think pre-commit is being used more and more because of black using it. Yeah. I've heard that more than once. That's pretty awesome. You're like, wait,
what's a pre-commit hook? That thing exists in Git? Yeah, definitely. Nice. I like it. The slogan, like you can have it in any color you like, as long as you went in black. Yeah. I didn't know that that was from Henry Ford until somebody told me about Henry Ford saying that about the Model T. Does it have like a theme song or what's going on here? Oh, well, every time I use black, I hear Black Hole Sun from Soundgarden. So I went ahead and just mentioned that in the hard notes.
Black Hole Sun is, Soundgarden was such a good band. They really came from like a special era of music. We had them, Pearl Jam, Alice in Chains, Dinosaur Jr. Like there's a lot of good stuff back there, but this is the only one that seems to be making it to a Python package. So way to go. All right. Dan, for item number five, I feel like for a really long time, it was like, well, pypi.python.org slash pypi is a little bit out of date and a little bit crummy, but we're working
on it. And it just, it seemed to be in this like stasis of like half improved forever, but not anymore, right? Yeah. The new pypi, new Python packaging index went live this year and what an amazing improvement. I mean, like, you know, throughout the whole project and I know you had a really, really great show where you interviewed, you know, a bunch of people working on that transition and launching the new and improved pypi. And it's just crazy, you know, like throughout the whole thing, it's,
it's been like rewritten from the ground up the way I understand it. It's new hosting infrastructure and it's much more friendly now for new contributors. Like beforehand, I think it was sort of like a home grown web framework because, well, it was that pypi, old pypi was created before web frameworks really were a thing in Python. Here's the socket. Let's start with that. Like, wait, wait, wait, wait, what?
Yeah. And so, yeah, I just love it. You know, like every time I'm searching for a package now, it just, it just looks so much better. And it looks like, it looks good. You know, it's like, it's when you compare it with JavaScript, like the Node.js world and NPM, I felt like, yeah, that was sort of the gold standard for a while for what a online package repository should look like and the features it should have. And I feel like PyPI is now playing in that league and it's super
nice. I think it is super nice as well. And it's silly to say that it matters that it looks good because it's just pip install a thing. But I think it really communicates a message to people who come into Python or who are using Python, but especially people who are new and they don't, they're like, oh, what is this stuff? Oh, there's a package injection. You're like,
whoa, that thing is like from the nineties. What is that? Right? Like that, I mean, that communicates something about how much the community cares for her for that kind of stuff. Right. Yeah. They also rolled that over with like everybody in the world using it and very little hiccups. Yeah, it definitely is really impressed. It's super impressive. So a couple of interesting notes here when they rolled it out first, it was pypi.io. And I thought, oh, they're just being
part of this hip new.io. You know, it's like, it's like the .ly of yesteryear, you know, it's like Libya all of a sudden became a super awesome place to have like your domain name because it had ly in the end. And so io, right, it's super popular now, but now it's pypi.org. And the story is the pypi.org was owned by somebody else and it took a long time for the PSF to get it from them. Whereas they could go and buy pypi.io. So that was like an intermediate step in the whole rollout.
Yeah. Some of the nasty details of web development. Exactly. And then the other one is, this is a pretty interesting example of a popular Pyramid web app, right? They started out in Flask and actually said, no, we'd rather go with Pyramid, which I thought was a pretty interesting choice being a fan of Pyramid myself. And then Brian, finally Markdown, right? Yeah. So the old pypi did not support readme.md. It only was readme. What is that? RSL or something?
Yeah. RST. Yeah. So I try to avoid restructured text as much as I can. No offense to everybody that loves it. Yeah. I hear you. It's like Markdown, but it's broken. The stuff you put in there doesn't quite work. Just kidding. No comment. Just kidding. Just kidding. I'm just saying the Markdown stuff doesn't always work in there. All right. Item number four, Brian, what we got? You touched on this a little bit before.
Yeah. I think it's been fun to watch. It was unexpected, the rise of Python in the embedded world. We actually did a segment on this on Python Bytes, but there's so many, partly from the maker movement and a lot of people trying to get into controlling their, not just their computers, but their world with the internet of things and also just playing with, you know, adding lights to your bicycle and things like that with microcontrollers and hardware hacking.
Adafruit's been a big part of that, of pushing it. But there's a lot of projects around Python and embedded. MicroPython is one of those. And MicroPython is being used by both hobbyists and serious developers. Hobbyists are serious developers too, but I think some companies are using MicroPython as well. Adafruit has its own version of this called CircuitPython. Actually, I don't know if it's related to MicroPython or not, but Adafruit has CircuitPython. And then MicroBit is code Python
base for the MicroBit processor. And then we even have Mew, which is a editor that directly interacts with microcontrollers running Python. It's been pretty fun to watch this year. I really think the rights of IoT and Python is just taking root now. And MicroPython is so interesting to me. I mean, the fact that I can take a Lambda expression and tie it to a hardware interrupt just like never ceases to amaze me, you know?
Yeah, that's really cool. And so, and a lot of this is because somebody wants to learn electronics or something, but they also want to make it do things. So you have to learn a programming language. And it used to be we taught people C and like the barrier to entry for C is huge. Having Python be able to do that is wonderful. Yeah, it is. So you talk of the IoT stuff. I learned something from Matt McKay recently on an episode, and I want to share this with you and just see what you think. So you
guys click on this link here. So here's an IoT device using Python. And the idea is the title of the article. This is on the Twilio blog. It's how I potty trained my kids using Twilio and an AWS IoT button. So it's a little button like those Amazon dash buttons. And he put it near his son's bed. And apparently there are kids sharing the room and wanting to make sure he didn't wake up and make a lot of noise and wake up with the other kids. So he put this button that basically has like a poop
emoji and you click it and it says, I got to go. And the parents would come in and like help out the kid while he was learning. And this is written using like the Python API on like AWS Lambda and stuff like that. So yeah, the rise of Python and IoT. So Q4 10 years and his kids are like, dad, did you have to put that on the internet? This is awesome. Exactly. Yeah, this is great. Isn't this funny? So yeah, I mean, funny and like an amusing, but also like, actually, that's kind of helpful.
Like I can see how that would be helpful as a parent. Because you definitely don't want your kids waking up. And you have to be up for a few hours in the middle of the night. So if an IoT button can help that, then get the IoT button. All right, item number three. And it's packed full of goodness here, you guys. So legacy Python's days are fading. What do you think? Well, I haven't used it for ages. So yeah, I haven't either. Dan, is real Python running on like what,
two, six? Yeah, not a lot of legacy Python for me anymore. Thankfully. That's awesome. And yeah, it's I mean, it's kind of crazy that it's, you know, it's a sort of crap on us, the Python 2 end of life. And now we're not that far away anymore from it. So yeah, we're actually super close. And the official item that I'm linking to as this piece of news is Guido Von Rossum's official statement. So I don't know, four years ago or something, it said, yeah, Python 2,
it'll be done in 2020. Whatever, right? Like that's like a broad range for a period where things stops a whole year. So we needed a more clear date. People are speculating maybe the date of PyCon might be a cool one, maybe the end, maybe the beginning. So he came out and said, let's not play games with semantics. The way I see it for the situation for Python 2.7 is that end of life is January 1st, 2020. And there will be no updates, not even source only security patches after that date,
period. That sounds like the end. Yeah, it's a good end, though, because the, and some people might complain about the whole security patches, but this is volunteer driven. So it's volunteers time. And I think that to let them work on new things is good. Yeah, absolutely. And it's not like people are surprised, right? It's not like, wait, last month, they didn't, nobody told us there's going to be a new Python. Like it's been going on for 10 years. This will make it 12. It's time.
Yeah. And there's a couple of loudmouths that two podcasts that have been talking about it for a couple of years. So yeah, I think we're actually making a dent with this referring to Python 2 as the legacy Python. I think it's making a difference. And not everybody loves it. But I know some people it's been working for. Do you guys know, you know, with the no more official security patches after the end of life date? I mean, obviously, this is a big business opportunity for other companies to
come in and offer, you know, long term Python 2 updates and stuff. Does anybody know who's doing that right now? Like some Linux distributions, I would guess, but I don't really know what's going on in that space. Yeah, possibly. The big news that I just heard. So, you know,
there's Red Hat Enterprise Linux, right? And they do a lot of that kind of stuff. And the versions that they have live on for a long period of time, but they just announced that they're dropping Python 2 support from Red Hat Enterprise Linux, which to me sounds like, okay, even the laggards are stepping forward. So I think that that's pretty big. But I suspect that there's still a lot of support on the
backend. There's been a few people who've like talked about forking Python 2 and just supporting it themselves as a single individual. But it seems, I'm sure there's going to be some of that. But like, as a company, like, let's say, I don't know, your Bank of America, you're like, dude, we don't need to upgrade. There's this guy. He forked it. We're good. You know what I mean? Like that just doesn't
seem realistic. It's also not just the C run, the CPython runtime anymore. It's like so much more than that. So Django no longer supports Python 2, like the scientific stack, like NumPy and stuff are dropping support for Python 2. So even if you still have the older stuff, you know, you're like, well, we just keep running. It's not just about what's running your code. It's also what you can use in
your code. And you know, that's going to start looking like it's just zooming off into the distance. So not good. Well, I mean, I think Python 2 will actually live on forever in some situations where, for instance, you got something that doesn't need a security patch because it doesn't touch the internet at all. And as long as it will run on whatever CPU you're sticking it on for some people will have like a huge program that they never touch because the requirements never change. And
they use it every day, but it doesn't need to be flexible. It doesn't need to be worked on that much. they'll probably upgrade when they have to change the program. So yeah, I'm wondering if that also means that Apple is finally moving to Python 3 and making that the default for macOS because the latest release, it still ships with 2.7.10, I think. Yeah, that would actually be really, really nice.
You know, it's starting to shift in Ubuntu and just different Linux distributions, but also on macOS. Yeah, that would be super cool. Brian, you're talking about things that don't require security patches. I just saw a video from NASA talking about their trip to the moon and setting up a base there, which is just awesome. So they had like all this sort of quick, different scenes of people doing stuff. And one of the scenes was them working, they had Sublime open, like editing a
bunch of Python code controlling like something about the moon mission, which was pretty cool. But I didn't catch whether it was two or three. I couldn't tell. That's pretty cool. Yeah. Someone else on Reddit did catch that the Sublime was unregistered, by the way. I guess a final thought here is, we've also talked about this on Python Bytes, is that there's the PythonClock.org, the Python 2 death clock. And it has been updated for this new end of life statement.
And it reads one year, one month, two days. So if people are out there and they got to start upgrading, like the death clock is ticking. Yeah. All right. I also got item number two, and that I would categorize as the end of the innocence for PyPI and PIP. So we've heard about a couple of issues, one like right before 2018, like late 2017. And then another one again this year about people posting or taking over control and posting some form
of evil PyPI package. And primarily they look like they're kind of either a research project, a prank, or some kind of like Bitcoin theft. So one of them installed like a clipboard monitor that would watch for Bitcoin addresses. And as soon as it saw one, it would like ship that off to like the mothership or something to that effect. But I don't know of anything super bad that's happened, but it, I guess until, you know, 2016, 2017, PyPI as a place was just, it's incredible to me. It's a place
on the internet that was entirely unprotected and yet safe. I can imagine that. Yeah. If you had like a open FTP server and you just put it on the internet, like all sorts of badness would happen to that thing in a day. It would be full of all sorts of things you don't want it to be full of. And here PyPI was going along and it was just, you know, the community sort of policed itself and took care of itself. Plus it used to be really hard to become a person that could push things to PyPI.
Yeah. Maybe it correlates to the shipping of pypi.org. Yeah. There seemed to be a lot of, like, not a lot, like I'm trying to be more specific, but I think NPM, again, in the JavaScript, like Node.js world, they had a big, they actually was, it was something similar. I think it started with a typo squatting attack,
but then it was actually a worm that would then go. So if you got the malicious package, nothing would really happen except that package as it was installing was looking for your load. Like if you were registered with the NPM package repository and it would go in and patch the packages that you owned to spread that worm further. I don't know if it was some payload, but it was, you know, really exploding quickly.
And I guess the same thing could happen with PyPI. Like I don't, I don't see why not. Yeah, it definitely could, but it hasn't. And let's, let's not give any people ideas. Sounds cute. We're going to cut that out. Yeah, exactly. Don't say that. Security by obscurity. Come on.
Exactly. Right. Security by lack of creativity. Okay. So the two main problems were basically, like you pointed out, Dan type of squatting, which is like, we're going to upload a package that takes a copy of Django and then slightly modifies it set up.py and publish it under Django or Django or whatever, like how you pronounce it without the J. Right. So if people pip install and they miss a spell, they get your bad thing. Right. So that's one. And the other was to take advantage of folks who
didn't understand why something wasn't available to them. So like, if you tried to use regular expressions and you say re dot, and there's like, it says, well, re is not defined. You're like, well, maybe I need to pip install re. No, you just need to import it. Right. It's built in. So they were targeting the built-in libraries and I believe all the built-ins are not patched. And Brian, didn't we
cover something on Python bytes that would help guard against typo squatting? It would like try to grab packages that also had misspellings around your package name. I think there was something like that. Yeah, I think so. I don't remember what it was called, but I think that exists and people could find it. Oh yeah. Yeah. I remember listening to that episode and you guys were talking, I forgot what it's called too, but like it generates different variations of your own package names. Yeah. And
try and like register all of those. Yeah, exactly. It's very work intensive. Yeah. So it's interesting on what is the fix? And I don't know what the fix is. The PSF is trying to get more resources to support PyPI and the Python packaging infrastructure in general, but like largely that's been improve it. Do things like you talked about previously, Dan, on just shipping pypi.org. Right. But this validation, if you have so many releases of so many packages, how do you even verify it? I don't know.
It's tricky. It's not necessarily something that could be, that can be solved with a straightforward technology fix, right? Like essentially you're, it's a problem that all of these, let's call them
online marketplaces for code repositories or whatever you want to call them. You know, if you look at the Android app store, Apple app store, I mean the Apple app store, they have these, these like crazy intensive review processes, you know, considering that there are millions of apps in this app store and
they're reviewing every single one. And I think there's even some manual steps involved there. And even there we've had malicious apps that were trying to steal people's, you know, private information and it's a really locked down platform. So I think it's really, it's a challenge that, that we're facing now because we're all connected over the internet and, you know, people are really
relying on these, on these package platforms. So, it seems like it's something we'll, we'll have to try out a bunch of solutions and see what will work there. I think we should get some of the, all of these new machine learning Python people to get involved in, and take a look at it because conceptually the normal updates for a package are little tweaks on, what it's doing. And if suddenly it starts, connecting to different servers all over the
world, or even just one where it used to not connect to anything, that would be a change. It would be a different behavior than it was ever doing before that, that should be something that a machine could catch. Yeah, actually, I think you're right. I mean, there's almost no scenario where pip install a thing should create outbound connections besides to download the package. Or, you know, even a package that doesn't normally use any connections or use requests suddenly adds, you know, socket
connecting or something. that's something to take a, you know, at least we can filter it down to things people could take a look at and say, is something weird going on here? Well, it would actually be sweet to have like a CPython interpreter that is fully sandboxed that, you know, can't like, you could just run code that people send you and you can run it safely and it wouldn't be able to write to any folder on your disk or open sockets and whatnot. That would actually be
kind of sweet. I mean, it's all interpreted. So it's seems like that should be possible. I mean, it's going to take a lot of work, I guess, but I think it's possible as well. You know, Dan, you mentioned the online app store as maybe something. I wonder if there's something like that in terms of packaging that we could add. And what I'm thinking is like, if I go build an iOS app, I have to declare the things it's going to ask for, like it has to ask for access to contacts.
It has to ask for access to GPS. So, you know, you declare it needs the GPS feature. It needs these other features, right? So what about a package that declares, like I don't declare any network capabilities. So I don't talk on the network and things like, you know, I can depend upon packages that do, for example. On the flip side, none of these have caused a huge amount of damage. And I don't want to go to the point of overreacting so that everybody has to take off their shoes to travel.
Yes, I know. Things like that. I know. We don't want feel good security that doesn't actually make any difference, but makes life hard for everybody. Yeah. Yeah, I agree. That's a good point. I don't know what the answer is. I don't know what the fix is or if there is a fix, but I would definitely declare the end of innocence for PyPI. That's my declaration as item two for 2018. Dan, you have the honor of kicking off what is the number one story in the Python space for 2018. What happened?
Oh, man. What a sad honor, though. Well, yeah, I know. It's not about the messenger, right? Well, yeah. I mean, Guido van Rossum stepped down as the BDFL, which is both sad and ironic, because I guess BDFL stands for Benevolent Dictator for Life, which I think he still is now technically, but he's just going to be less involved with the stewardship and leadership of CPython. Yeah. He's a little more like the queen of England now, in a sense, right?
Like, he's still representing Python. He's still very active in the community, but he just said, I'm going to let other people deal with the decisions around it. And basically, went so far as saying, like, I'm not going to decide for you how you decide. Like, it was even a meta non-decision. So he said, all right, you all have to figure out how to govern yourselves. And it looks like he's coming back in a little bit more in terms of participating.
Well, right. I mean, I think that's one of the reasons is because his involvement with Python was just the stuff that wasn't fun. And now he gets to be involved with some of the things that are fun, like speaking and mentoring and things like that. You know, I think that's a really interesting point. I'm not sure I would want his job the way it kind of looked a year ago, right? Just, you know, arguing over peps and stuff like that.
And you know that if you do stuff on the internet, that it's, there's definitely a very small minority of people that are, you know, abrasive, unfriendly. It just sucks. But even just a few people like that will suck your energy for sure. Yeah. And we still don't know what's going to happen. So we'll, maybe that'll be the story for 2019 is what happens now. I think you're right. I think 2018 is the year Keto stepped back.
2019 will be the year that whatever, whatever takes place going forward is going to be that year. So regarding the new governance structure for CPython, which obviously, you know, for any Python developer, that should be really dear to our heart because it's going to have such a big impact on the future of Python and the CPython project.
And so the last update that I saw, basically we have PEP 8000 now, which is a PEP that discusses the various new proposals that have been forward, that have been put forward regarding the Python language governance. And I just recently saw, it was like two or three days ago, Victor Stinner's post on discuss.python.org, where he's comparing the seven Python governance PEPs. So I guess like actually like the different proposals are separate PEPs, but they're also listed under PEP 8000.
So there's different models that have been put forward, like, for example, leading the project with a trio of three co-leaders. There could also be a community governance model where there's no central authority, sort of like a total 1080 from, you know, a dictatorship to no central authority. I don't know how that would look like, you know, in concrete terms, but there's like different models that are being discussed right now.
And I think it's going to be very interesting to see what the CPython team ends up with there. I agree. And it's pretty important, like, even simple stuff like Lucas Lenga, also creator of Black, what we spoke about earlier, he's doing the release management for CPython for the next couple releases. And he was thinking like, hey, it would be great to release this yearly, but we can't even discuss having it released yearly instead of every 18 months.
Because we don't even know how to decide whether we, how would we, who decides, how do we vote? We don't know. So, so many things like that are just like on hold until this gets figured out. So there's a lot of pressure to make it happen. Definitely. Yeah. It's like, it's a total bottleneck, I guess, for, yeah, like you were saying, you know, just to put out a new release, you would have to, they would have to figure that out first.
And so I'm pretty sure that also means it's going to be a decision that's going to be made relatively quickly there. So we can all move on. Yeah, absolutely. One thing you did note, though, is that Guido is starting to come back and be more active in the community again, just not in this BDFL decision maker way, right? Yeah, I thought that was kind of cool. You know, I don't know, it might be coincidence, but I don't think it was.
So as I was putting together, you know, the recent PyCoders Weekly issue, I saw that there were a couple of interviews with Guido out there, articles that, you know, had like little sound bites from him. So he did a really long like video based interview on the MIT AI podcast. So it's actually on YouTube. It's more like a video based show, but I think it's also podcast that I thought was really interesting. And he also started blogging again.
So he has this personal blog at I think it's neopythonic.blogspot.com or something like that. And he put out a new post there. And I think it was kind of nice to see some activity there again, because I'm sure it's also a sign that, you know, now he maybe feels like he has more time to do that or feels inspired to do things like that again. And I think that's a really, really positive sign. And just for his personal well-being, I think. I think it's a positive sign as well.
And just like Brian and I were talking about earlier, a lot of his energy had previously gone into like debating PEP 572 and just dealing with all the decision making. And now he can focus back on, you know, what most people would want to focus on coding, software, cool stuff like that. And it sounds like he's getting back into it.
So that's awesome. Yeah. And I mean, you know, what a crazy responsibility to be at the center, like at the focus of it all, you know, and to be the one person that needs to make all of these decisions. And sure, you know, like he was delegating a lot of these decisions and everything. But the way the CPython governance was set up before, it was like all focused on one person. And yeah, I mean, it's tough to do that for several decades, right?
Yeah. Honestly, I was surprised that he had kept going in that role for that long. I mean, it's sad to see him step down. But at the same time, that was 25 years. Like how many other people have a job that goes 25 years? That's high pressure, right? Yeah, it's incredible. I mean, thank you so much, Guido. Yeah, absolutely. Python's a special place and it has a lot to do with him. All right, guys, that's it. Those are our top 10 for 2018. It's been a fun year, hasn't it? Absolutely.
Yeah. Yeah, I think 2018 just has more good stuff to come. So it doesn't seem like anything is slowing down then. And that's pretty awesome. Now, before we round out the show, you have to answer the two questions, both of you. So, Brian, let's start with you. If you're going to write some Python code, maybe test a little something or other, what editor would you use? PyCharm. Right on. And Stan. Yeah, good one. Dan?
I'm still a sublime guy, although I have been experimenting with Visual Studio Code. And it's pretty sweet, actually. Yeah, I would say, like, the sublime Visual Studio Code, there's probably a lot of bleed over between those. And with all the activity around Visual Studio Code, it's got to be eating into the sublime world. Yeah, definitely. Probably. I like what VS Code is. I like watching what VS Code is doing. They're just not quite there with the testing yet.
And I'll take a look again when they get that up to speed. Yeah, I agree. I think they're doing really sweet stuff. Okay. Brian, notable PyPI package? One, preferably, that doesn't have any badness in it? pytest. pytest. Right on. Dan? Sweet. So I was going to say black again, but we talked about this at length. So I'm going to say pytest-ICDIF. So it's a pytest plugin that gives you better error messages for your pytest assertions in some cases.
So it'll basically do, like, a more extensive, like, diff on, let's say, when it's comparing two dictionaries. And it'll tell, you know, those keys were added, those keys were removed, and it has, like, color highlighting. And I just found that this is, like, a really, really sweet little tool. And I started installing it in all of my projects that have tests. And I highly recommend it. Yeah, that sounds great. I may have to go check this out. Cool. All right.
So a straight sweep for pytest there. That's great. All right, guys. Thanks so much for being on the show and being part of this. It's so many cool things to talk about, and I really appreciate it. Yeah, thanks for having us. Thanks. Yep. Bye. Bye-bye. Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm.
If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues. Thanks.