How BitTorrent Works

related stuff in the near future. So we wanted to kind of lay the groundwork and explain what exactly is torrenting and how does that that technology actually work and and why is there so much controversy around it. But to start off, we kind of need to talk about just how you would typically get a file, right, how

how files are transferred period at all? Under so the normal process, yeah, by either by by any of the protocols that you would use over the internet, be the FTP or HTTP that's file Transfer protocol or hypertext Transfer protocol excellent acronyms. So traditionally, your your computer, which in this scenario we call the client, is going to contact a a host computer a server in order to to say, hey, I want that thing exactly. Yeah, whether that thing is a web page. So if your web browser can be

a client. Yeah, that thing could be a web page. That thing could be a music file, it could be a movie file, it could be uh an email. It all depends on the people content, right that that you want to get. Um. The server would then send the file to your computer according to whichever protocol you are using, right, And it's using the basic set of rules that we all know from the Internet, the idea of everything's in packets. Those packets travel through different routes and then they get

reassembled on your computer. But yeah, it's essentially a one to one relationship, right, I'm asking the server to send me something, the server send something to the client. Done, done, and you know, and the speed with which this can happen depends on the amount of traffic on the server the size of the file. Yeah. So let's let's boil this down to an analogy. Alright, So, Lauren, you are a server, I am, yes, So now you have entered into the uh the service industry. You are a server

in a coffee shop. Okay, you serve terrible at that job. But let's let's imagine Lauren, that this is a reality where you're not terrible at your job, you're actually pretty good. You can serve up to three people simultaneously, so amazing, but only three people and uh, and the amount of time it takes you to produce the order for the customer depends upon what they ordered, like what size of it,

and how complicated is the drink. Obviously these things will factor into how long it takes you to prepare it. But you can still handle up to three. But then there's a rush, right like, uh, I don't know, there's some hipster band is in town and everyone needs their coffee, so they all rushed the coffee shop at the same time.

And now you've got twenty people asking you to make very complicated drinks, but you can only serve three at a time, and these drinks are getting progressively more complicated and larger. Things start to slow down, so that person who's you know, seventeen people back, feels like they're waiting forever to get a coffee, and they're thinking, what's the big deal. It should only take It's just a coffee,

You should just take two minutes. I should be in and out in two minutes, and instead I've been sitting here for fifteen minutes. And the lines barely moved. That's the same thing we experience with this traditional approach. If you're talking about a single server hosting a lot of files, a lot of clients contacting the server for the files, and those files are big, it just means that things

can get bottlenecked and slow down. So there were some different approaches to trying to figure out how to make this uh more streamlined and put. One way is to just increase broadband speed. Right, So, essentially, if you're talking about the Internet being a series of tubes, that's it. It's a bigger, bigger, longer tubes. Yeah, bigger tubes and

more tubes. Yeah, those are your two choices, right. You make them bigger so that you can shove more stuff through the tubes, or you make a lot more of them so that there are new ways for it to pass through. But you're still kind of ball necked by the the server itself and how quickly it can respond to requests. Sure, so if this approach is really slow, then one alternative is to distribute a file across a network so that you have lots of different options when

you need to get that file. Right. So, in this instance, instead of saying that Lauren is the only person who can serve coffee within a twenty mile radius. And that's why we're all kind of uh, you know, up the creek when we walk into the coffee shop and we see there's fifteen people ahead of Instead, we have a coffee shop on every corner and across the street from each other, and next door to one another. Their coffee

shops everywhere. So really I just have to walk into any coffee shop and I really just stand in the middle of the street and show coffee and someone will bring one. So that's that's kind of the idea of the peer to peer network. In this sense, you have some form of software that allows you to connect to a distributed network where when you participate in this network,

you're essentially giving permission to access part of your hard drive. Yeah, you just set aside a little folder on your hard drive that is okay for this program to access, and then everyone else who is running the same program also has folders set up. Hopefully they have stuff in them, right, Yeah, Because one thing you could do is you can set up this folder and then immediately move stuff out of the folder, and then you're leaching. That's the term, right right.

It's it's considered a faux pause in sharing circles just because you're you're not putting anything back out into the community. Right. So with peer to peer, let's say you've got this folder that's designated as a shared folder, and when you're connected to the software, anyone who's also connected the software, when they're searching for a specific file, it your file. If you happen to have that file within your folder, you come up as an option to connect to download

from your computer. Now, the problem with peer to peer, it solves one issue, right. The one issue is that if you go with the clients server uh model, then you are limited by the server. Like if the server is the only machine out there that has an existing version of this file, then that's the one destination you can go to, and you are stuck with whatever problems that might have. In peer to peer, you're adding more tubes,

but you're not making them any bigger. Right, So in other words, like I could you know, if we're going back to our coffee shop, example, and we have these coffee shops everywhere, I can walk to the coffee shop that's closest to me, but there's no guarantee that that coffee shop won't have the world's slowest server, or you know, like that someone who's just they they are very meticulous about the way they make their coffee, or there might be two or three people in front of me at

that coffee shop and I have to wait for them. Anyway, it gives me more options. It does not necessarily mean it will be faster, and the really good software will identify the person with the best connection who has that file exactly yes, So that way it gives you the best chance of having a smooth transfer. And it's also good to note that this creates more stability when you're

trying to get a file. So the other big issue with the old clients server approach is that if that server suffers a problem, if it goes down for some reason, power outage or whatever, then you're stuck. You know, you

don't you do not get that file. And if you were trying to download a big file and it was maybe seventy of the way done and the power goes out, there's a chance that file could get corrupted in the process and then you have to start all over again, right, which was a real bummer when peer to peer was was really big back in the early two thousand's, like late nineties, um, because you know, it was about that modems that could that could handle broader band than fourteen

four speeds became available for consumer purchase. And after that, you know, when when people started getting connected to the ethernet was really when Peter Pierre happened. Yeah, and then you you were able to you know, one of the downsides is that traditionally, with most Internet service providers, your upload speed is a fraction of what your download speed is. Right.

The reason for that is that ages ago, when companies began to offer Internet service, they looked at the broadband that they had available, like the bandwidth they had available to the size of the tubes. Essentially, they're saying, all right, here's here's what we are capable of delivering to our customers. How are we going to determine how much is down link versus uplink? And then they said, well, you know, probably people aren't sending stuff up to the Internet that frequently.

They're mostly trying to consume stuff pull stuff down. This was before we were uploading millions of years worth of YouTube videos every second. Yeah, this was this was back when that was not even a consideration. So the idea was that, well, you know, we can just make it fraction upload speeds will be a fraction of download because

you there's nothing as frustrating really as sitting down. Well this is probably that's that's exaggerating, but it's very frustrating to sit down and try and get at some sort of content online and then just see either a buffering thing or a loading screen and it just goes on forever.

If you're talking about the old old days of the Internet, there's nothing like trying to look at a picture of something and watch it slowly load pixel line by pixel line, and you're just thinking, I have no idea what I'm looking at. I won't know for another forty five minutes. I would I would like the record to state that I just used I just gestured at Jonathan to help you guys understand what he was talking about and exactly how ineffective that was. That was a great visual representation

on an audio podcast. Yeah, so you know that this approach meant that we suddenly had all these other options, this peer to peer approach, and this is the way a lot of those file sharing services back in the day worked because lime Wire, Napster, that kind of thing. They well, yeah, it's all about let's find people who have the stuff you want and connect this create a

connection directly to them. Yeah, so it almost becomes like a direct phone line in a way, where the connection is between the host computer and your computer, so you can get the file and then of course once you have the file, your computer can become a potential host computer. So now an improvement on that was bit torrents approach, right, and I think it will be really interesting to get into the intricacies of how bit torrent works. But before we do that, let's take a quick break. Let's get

back to the show. Alright, we're back, and Lauren, you

alluded to bit torrent. Now, this was a totally different approach in the sense that it was someone coming up with the idea of how can we make these download speeds faster, not just more reliable so that I have more options, but how can I get the stuff I want faster, particularly if it's a really large file, you know, and if you're if you're like Jonathan said earlier, if your internet connection is maybe not the best and something cuts out somewhere that you won't lose all of the

progress on downloading a file you had already made. So this was the idea of a programmer named Bram Cohen. Now, Bram Cohen had worked for several dot com startups that never really took off, and they just kind of failed over and over, and he really wanted to work on something that worked. That was a big ambition of his understandable And in two thousand one he was also getting really irritated by these problems we're talking about the slow

download speed of large files. He liked the idea of the peer to peer network, but he didn't care for the execution. So in two thousand one he begins work on a new protocol. Now we've used the word protocol several times in this podcast, just to remind you. A protocol really is just a set of rules. It's kind of a kind of a seven instructions for a computer to follow. So he designed a protocol that he called BitTorrent. Now, this was an attempt to solve that connection problem um

and the speed problem at the same time. So you know, peer to peer adjusted the connection. He wanted to do speed as well, So how do you do that? Well, his approach was to create this protocol that would allow files to be distributed in pieces, and as you receive a piece, you are also able to upload that piece even before your file finishes downloading. So I might have a file that's at twenty seven percent as it's slowly downloading, that is completely available for other people on the network

who are also trying to get that file. You could also get pieces of these files from multiple sources, so as long as everyone had the same file, then you could get, you know, a piece here and a piece there.

So imagine that everyone has at least some part of a five piece puzzle, and one person down the street is giving you pieces uh one through twenty seven, and someone on another side of the street is giving you pieces forty three through fifteen, and you're just getting You're getting the whole puzzle, but you're getting them from different locations all at the same time. It's like it's like if everyone in that in that coffee house street has a cup of coffee and you just kind of run

down and they all pour a little bit into your cup. Right. Yeah, it's similar to that, except except with less like gross spittle contamination. And also there's a pretty design at the top. I don't know how they managed that while you running down the street like a crazy person, but you know they're talented, that's all. So he was thinking like this would be the way of solving this problem of slow download speeds. Now, he did not necessarily go about testing

this in a very scientific, scientific PC way. Did you see how he tested Okay, folks, here's how he tested it. He collected a batch of freely available pornographic material and then invited beta testers to see about getting hold of this stuff using the bit torrent protocol. Stay classy, Cohen, But now he figured that this was kind of a sure fire away. And because it was freely available stuff,

it wasn't like it wasn't copyright. I didn't have to worry about running into problems with torrent systems have having fact run into lots of copyright issues. We'll talk about that towards the end of the show. You know, that's that's a that's a totally different discussion. But yeah, he he wanted to try and test it out, and apparently that wasn't considered a terribly auspicious beginning. But he eventually did launch a working version of bit torrent for the

general public in October two thousand two. Now two years later in two thousand four, he had a five person company working on a search engine that would work alongside his protocol. So he designed the protocol, but one of the issues people were running into was how do you find the torrent files which are pointer files. I'll talk

about them a little bit more in a second. How do you find these files that then facilitate the downloading process when you're you're actually using the bit torrent software and this is by a tracker server. Right, So he created he worked with this five person company. Actually, this five person company worked for him to create the search engine that would look for these torrents to help make this process work more smoothly. Uh. He was also then

looked at by a little company called Valve. Yeah, so Valive. Valve is thinking, you know, we want to do a downloadable distributed network approach for our games. We want to have this ability to deliver games to our users that is fast and reliable and doesn't put a lot of pressure on our own servers. Right, we don't that way, we don't have to build out a whole data center just to support this. This model, this distribution model, so they contacted Cohen and they hired him to work on

a platform that we now call Steam. Cohen was one of the people who helped build that program out, and it was all because they saw what he was doing with bit torrent, and they said, that's the approach we want to take. And in fact, Valve and Blizzard are two companies that use bit torrent for a perfectly legitimate means of distributing their files. It's a good time to just mention there's nothing illegal about torrent files. No, no,

it's it's just a matter of distribution. It's kind of like if you were to say, yeah, but you can get illegal material that way. You can get illegal material through the mail, But you wouldn't say that means we should shut down the postal service. You might say we should shut down the postal service for other reasons, but

not for that one. And same sort of argument you could use for bit torrent, like to say, let's get rid of bit torrent because some people are sharing illegal files, even lots of people sharing illegal files, doesn't mean that

the tool itself is wrong, right. The technology itself can be used for lots of really cool things like yeah, like like letting people like Valve put games up online without having to buy giant servers, or independent artists put their music online, right right, Yeah, And there there are a lot of reasons there their entire you know, whether it's software or video files, music files. You know, there are a lot of reasons why you would want to

take this approach. And uh, one of the big ones is that it, like you said, it takes the strain off the provider. Right. So, if I'm a small business or an artist, an independent artist, I might not have the resources available to me to create a dedicated server where people can come and download stuff, especially if I'm

having to pay lots of fees to maintain that. And you know, I mean, you know, for example, if you post some music to your personal website and uh, Neil Gaiman or Will Wheaton link it on Twitter, and all of a sudden, you're completely overwhelmed. This, by the way, is a complete invitation for both Will Wheaton and Neil Gaiman to tweet about our podcast. We would love that. That would be gorgeous. Um. But yeah, so I wouldn't even know what to do with myself. I would probably

totally flip out. No, I would totally flip out. I wouldn't probably there. There's no probably there. But yeah, in two thousand five, already by this time, okay, the bit torns only three years old from when it officially launched. Even by then, Hollywood had taken notice and was not terribly happy. Yeah, the Motion Picture Association of America and also the Recording Industry Association of America. That's um, the m P A A and the R double A R I double A is is like that's music. Essentially an

m P A is film. Uh So m P A A. UH in particular was concerned because the bit torrent protocol did make moving large files much easier, which meant that suddenly people were able to upload and download large files like rips of films. They take a DVD and ripped the film from the DVD and then share it and you would get illegal copies. Or if you were working on a digital film and you were part of a digital film production and you had access to the file,

there were movies that leaked out. Um some famous instances of movies that leaked out before they had even hit the theaters where people were getting hold of pirated copies, and Hollywood was really taking him a bit torrent, even though again, as we said, but torrents a tool, it's

not like it was specifically facilitating the illegal activity. However, on November twenty two, two thousand five, Brand Cohen had a joint news conference with the chairman of the m p A A. Dan Glickman, and announced that he had agreed to prevent his own bit torrent website from linking to torrents pointing to illegally available movies. Now, before that point, their policy was that if they were alerted to a torrent that linked to those pointing to they would take

it down. So that was there, that was their policy all all along. But they said, well now we're just

going to make sure that that's much more streamlined. Which if you've followed these kind of these kind of cases not just with bit torrent, but with other providers, other platforms like YouTube for example, there are examples of takedown notices that were improperly um yeah, in probably dealt with or improperly submitted people who didn't actually have the rights to something demanding they get taken down and then it

gets taken down, um. And then there then there's some embarrassing ones I think Microsoft just recently issued a takedown request to itself. Yeah, so sometimes these sort of things can end up being kind of embarrassing, but he was saying that we want to, you know, we don't want to. So um, today you can find bit torrent on lots of different platforms. In fact, bit torrent today allows you to do things like produce content and then distribute it.

So it's designed more now to help, like I said, the small businesses, the independent artists to create content, and then even to to enable it so it'll perform properly on things like smartphones, tablets, consoles, creating a huge distribution network that people wouldn't have had access to, you know, ten years ago. So there are many reasons why it's a very useful tool. So let's talk a little bit

about how it actually does what it does now. First of all, it's open source, which kind of tells you that Cohen wasn't necessarily looking to create like a a uh multibillion dollars a commercial venture. No, he wanted he really believed in this. An open source means you can see the source code. You can go and get the source code for the bit torrent, play around with it. Yeah, you can change it, you can make your own product based upon it. It's open source UM. And then there

are sites that house torrent files. Torrent files are not they don't have any material in them other than pointer information that will point a your software to the right destinations that will have the actual file you're looking for. It's it's a little bit like that that like a protocol that we were talking about, but just in a in a file format. It's and it's kind of like metadata. It's it'share information about the file you want, the information

mainly being where you can get it. Like it's it's allowing the software that you have on your computer and when when you when you run a program to uh to to go go out and find the little pieces of this file across the internet, across the swarm of computers that contain it. That's good. Yeah, it's so. Yeah. Your basic computer that has you know, the file, the full file on it, you would call it a seed.

It's a seed computer. UM. And then the swarm is all the different computers that are connect to the network that have some portion of that file and are actively downloading and or uploading that file right that are also running this This torrent sway, right, everything has to be working running the software at the same time. If you stop running the software on your computer, you disconnect from this network. You're probably you're still on the internet, but

you're not part of the bit torrent network anymore. So, Yeah, the torrent file just kind of points the software in the right direction so you can get these pieces. It gives them the software the information it needs to identify and pull those those pieces of file into your computer. Classically, this is all organized by a central server called a tracker like I mentioned earlier. Um, these these days, that's it's it's a little bit um a little more complicated

than that. But yeah, your classic bit torrent has a tracker server that kind of acts like a traffic driver, like it's it's the one that's making sure everyone is going to the right place. Um, and you're so, you've got your seed, you've got your swarm. You are constantly downloading as long as you're connected to this or downloading end upling as long as you're connected to this, uh, this network. The cool thing is with bit torrent that

your download speeds depend upon your participation within the network. Right, you get a rank based on how many files you are allowing the system to upload from you, and and your and your actual upload speed as well, So those

two things factor in. So if you are being you know, if you are altruistic and you are sharing a lot, then you can also download faster, right because your rank goes up and that means that your download speed's improve and so you don't have to wait eight hours to get you know, that music file you wanted to get. It will download in a matter of minutes maybe or or faster, depend upon your broadband speed and the speed of course of the various computers that are hosting pieces

of that file. But it does mean that you go much faster than you would with your traditional peer to peer or client server relationship kind of a protocol. You end up getting the pieces of the file you need, and then once you've got all the pieces you you're good to go. So you don't have to depend upon one computer and its connection to the network. You're depending upon the entire network and anyone who has pieces of that particular file. Now, like we said, there's nothing illegal

about this. This is just a distribution network. So if your file that you're distributing doesn't have any sort of copyright to it, or if it's licensed under something like creative comments that gives the person who has the file the the ability to distribute it however they like. Then there's no issue there. That's completely legal, and in fact, that might even be the intent of that content, you know, the person who made it. Maybe I want this to

be shared with as many people as possible. That's why I uploaded it to the bit torrent network. Uh. However, it does also mean that it can create an environment that allows for illegal sharing in a way that is a lot more difficult to fight than the clients server approach right, Right, Because it's not located in a single place. You can't go knock on one particular person's door and say you're holding this file. Because it's spread out over so many people that exactly I can't it's a lot

harder to trace back. I can't go into lawrence coffee shop and tell her that she has to stop serving this one type of coffee. If everyone around her is serving that same type of coffee, it doesn't do any good, right, same sort of thing. Yeah, So this is although I mean, I don't know you could. You can make some arguments that even even the illegal portions of torrenting are beneficial in a way because you know, first of all, it's it's made the industry um create its own legal ways

of distributing files the faster, you know. I I think that the direct streaming on like Netflix and Amazon Instant all that kind of stuff is of direct response. I agree to torrenting. I agree. And it also means that, you know, encouraging the studios to find new ways to get content to people who who want want to buy it, let me give you my money. They're eager to participate in this. If you make it easy enough and you

don't price gouge, then you're not really inspiring piracy. There's also been several surveys of varying reliability that have suggested that people who pirate stuff also tend to be some of the it's right right, which you know, people who are torrenting are probably more likely to buy things online than your average consumer anyway, because they're more computer and

internet savvy to begin with. But but you know, but the number that gets tossed around a lot is from a study by the American Assembly, which is connected to Columbia University. So it's more or less on the up and up. Yeah, this is that that's a that's a fairly reputable institution. They at least call themselves nonpartisan. They do, they do have a little bit of a liberal slant, I would say, but um, but they found that UM peer to peer file share users purchase more music online

than non peer press. And then of course there's also the argument that a lot of these organizations make the m p A and r I double A is that UH stolen property UH directly translates to lost revenue. And we've seen multiple studies, including studies that were specifically UH

funded by the government. You know, there were government agencies that looked into this to see how much, how how much in damage is really is caused by piracy, and the conclusion is that you cannot come up with a number because you cannot say for certain that someone who pirates something would have purchased that something had they not had access to the piracy software, and furthermore, that they're not going to go out and purchase it afterwards, that

they weren't trying before they bought it. Right, So in both cases, you don't know if they went out and bought it anyway, and you don't know if they would have bought it. So without knowing those two factors, you can't say that this actually cost anybody anything. And and you know, it's not like a physical copy of something, where if I walk into a store and I shoplift, that store is out a physical piece of inventory. When you download a digital file, you're making a copy of something.

The original version of that file is still on a person's computer somewhere else, so it's not like they have lost that. So with all of these factors, it really means it's so complicated that we cannot put an actual dollar amount. Not that that has stopped anyone from doing so when creating massive lawsuits against either a company that

creates the software or the users of that software. Absolutely, And we we will talk a little bit about that in a future episode that you'll hear probably in just a couple of days, because I think we're recording get immediately after we finished this one. Yes, we're talking about

specific right. Um. My favorite unofficial study, by the way, is an anecdote on the effects of purchasing and and free downloads was when uh Neil Gaiman in his publisher, which I believe in this case is HarperCollins, um put up a copy of American Gods for free and just just free ebook download. Go do it, people, And I don't think it's up anymore. I might you might not be able to find it. I could have just lied

to you tremendously. But they had. They had a free copy up for a certain period of time, and sales increased, definitely increased, like appreciably increased in the immediate future after it was that. I will say that, um there, you know, And I've told this story several times in the past, so you know I'm comfortable telling it again. Uh. The you know, I I definitely was one of the people

who downloaded something outside the realm of the law. Because I downloaded a television show that was made in England that was not available in the United States. There was no way for me to purchase it legally. Not that that justifies illegal behavior. It doesn't. So I'm still in the wrong, you know, I'm still in the wrong for

doing it. But I downloaded Space, the show that Edgar Wright and Simon peg And and several other people did, and uh, and I loved it, but you know, there was no way for me to legally get it in the United States at that time. As soon as it became available, as the day that it came out on DVD, I went on purchase it because I loved it, I wanted to support it, and I wanted to have a

really good copy of it. UM. That's another reason that things that like the studio backed stuff is getting more popular, because the quality tends to be better and you don't have to worry about like about malware packages usually unless the DRM is also malware. Sony, I'm looking at you, UM much better recently, by the way, but you know

that that CD thing still sticks in my craws. I think even classically though that I mean, you know, this conversation is reminding me of you know how you used to go to NERD conventions and they would have all of these illegal Hong Kong VHS tapes of various UM

did Battle Royal that way. But I thought that as soon as it came available to and and I think that similarly that really that kind of underground black market VHS sort of thing, aside from inciting nostalgian people of a certain age, really pushed movie companies to say, oh, hey, there's there's a reason like there's a reason for us to publish this in America, people in other countries outside of their original origin. People love this. There is a

market for it. We can you know, if we provide it and we price it properly, we will make money. Like That's that's the lesson that a lot of these companies have learned. Uh. And people can argue that, you know, these are things that various industries have had to learn, uh in a staggered amount. So like the music industry learned it first, and now the book industry also because with the books that definitely raised it. But as broadband speeds have improved for the common users. Yeah, so so

it's interesting. I mean, and again, you know, BitTorrent is just a means of distribution. You don't necessarily have to have it be something where you're pirating movies and tv um um there there is a little bit of a fight back. As of April, mccaffee had patented a system that identifies pirated content and can prevent users from downloading it by um either blocking it entirely depending on how you set up the software. UM it's it's through side advisors,

So depending on how you set up side advisor. It would either block the software entirely, um or block the download entirely I'm sorry, or just give you a little pop up window that says, hey, it looks like you're trying to use a torrent file that is torrenting illegally copied material. Do you really want to do that? Maybe here's some legal alternatives that you could use to download that that on the up and up for money you give to the creator. I kind of wonder how John

McAfee would feel about that. He's, of course the guy who who founded that company, and then we did a whole podcast about him. He crazy, he's uh, he is a character. Yeah, he's to have more character than a Tolstoy novel. Alright. So yeah, guys, I hope you enjoyed this discussion about bit torrent and torrent technology in general. It's really an interesting innovative way of getting around that

speed problem. And uh, and it leads us into a discussion about a particular website that we will talk about in our next podcast, which I don't want to give it away here, I'll just say y anyway, Uh, if you have any suggestions for if you your topics that we should cover here on tech Stuff, Send this an email let's know our addresses tex stuff at Discovery dot com,

or find us on Facebook or Twitter. You can do that by looking for the handle tech stuff hs W because that's us and we'll talk to you again really soon. For more on this and thousands of other topics, does it How staff works dot com