Near Protocol: 'Blockchains Cannot Scale Without Sharding!' - Illia Polosukhin & Alex Skidanov

You need to actually store all that state somewhere. And for a billion users, it's actually going to be a lot of state. And this is usually everybody talks, you know, talks about GPS. But actually one of the bigger bottlenecks right now across the space is state. And so obviously no single computer, no single kind of node can process all that can

validate the whole network. Now you can rotate this, validators validate what chart all the time, every block they can be. Actually, you know you can randomly select set of validators and they are able to validate this because they don't need to sink into the Shard. They don't need to process every single transaction that ever hit that Shard. They only need to look at this specific block. The data availability and consensus are merged together into a single single mechanic.

The essence of it is you you want to have a design that is not dependent on the underlying hardware itself improving at a certain rate to be able to service user demands. You want to have mechanisms beyond that type of scaling. This episode is brought to you by Gnosis.

Gnosis builds decentralized infrastructure for the Ethereum ecosystem with a rich history dating back to 2015 and products like Safe Cow Swap or Gnosis Chain, Gnosis combines Needs Driven Development with deep technical expertise. This year marks the launch of Gnosis Pay, the world's first decentralized payment network. With Agnosis Card you can spend self custody crypto at any Visa accepting merchant around the

world. If you're an individual looking to live more on Chain or a business looking to white label the stack, visit gnosispay.com. There are lots of ways you can join the Gnosis journey. Drop in the Gnosis Dow Governance form, become a Gnosis validator with a single GNO token and low cost hardware, or deploy your product on the EVM compatible and highly decentralized Gnosis chain. Get started today at Gnosis dot IO.

First One is one of the biggest node operators globally and help you stake your tokens on 45 plus networks like Ethereum, Cosmos, Celestia and Dydx. More than 100,000 delegators stake with Chorus One including institutions like Bit, Go and Ledger. Sticking with Chorus 1 not only gets you the highest years, but also the most robust security practices and infrastructure that are usually exclusive for

institutions. You can stake directly to Chorus One's public note from your wallet, set up a white label note, or use the recently launched product Opus to stake up to 8000 ETH in a single transaction. You can even offer high yield staking to your own customers using their API. Your assets always remain in your custody so you can have complete Peace of Mind. Start staking today at Chorus .1. Hello everyone, welcome to Epicentre. Today I have an amazing episode lined up for you.

We're talking to Alex and Ilya, the Co founders of Near, which is a in production sharded blockchain with a lot of value riding on it. Specifically, we will cover how near sharding actually works, try to build it concept by concept into an integrated hole, and then understand where they are in their journey to implement sharding. Alex and Ilya, welcome to Epicenter. So at this point, we've kind of done a few episodes with both of

you. Maybe you could give you know a short introduction to your to your backgrounds. Hi, yeah I think at this point my background is all near. Everything that was before is irrelevant. But yeah, I I was working on the sharded database called Mem Sequel before near for five years which is right now it's an in production sharded database and and before that I was at Microsoft. Yeah, I mean my background is

actually in machine learning. AII was at Google research prior to our startup adventures and I was one of the class. I was in a paper that introduced Transformers which is technology powering ChatGPT majority and other AI advancements.

And then with Alex, we actually started originally near as AI company and realized we need SAS, cheap, easy to use, easy to build on blockchain because we wanted to use it ourselves for data crowdsourcing and some other data use cases for our AI company and adaptivity to that in 2018 and yeah, focusing on that ever since. Yeah, let's get into sharding, blockchain scalability. So what is sharding overall and why has it been a generally difficult target industry wide?

I think I mean maybe broader question is if if you imagine having a billion users coming in and using blockchain as a means, payments as a mean of tracking ownership as a way to kind of coordinate resources and efforts. You imagine that you need to have kind of a few things happening, right? One is you need to actually store all that state somewhere and for a billion users it's actually going to be a lot of state. And this is usually everybody talks, you know talks about GPS.

But actually one of the bigger bottlenecks right now across the space is state and kind of its growth and so. So that's probably problem number one. Problem #2 is, I mean you have billion users try, they transacting. There's a, you know, hundreds of thousands of applications. Obviously there's a lot of transactions flying around and and a lot of processing that needs to happen and kind of people want to put more and more complex smart contracts and complex logic indices.

So you need to have throughput, bandwidth and processing power to do this. And so obviously no single computer, no single kind of node can process all that can validate the whole network. And so the way to support this is one way or the other, partition this computation, partition this storage and partition this kind of bad list to receive this. And so people have kind of historically been talking about sharding because that's how the

two companies doing this, right. So Alex mentioned, you know, doing this kind of in web two world, then SQL single store now is used by Fortune 500 companies. You know Google and Facebook have their own solutions and it kind of seemed reasonable that this is an approach that you

should take in blockchain. Now there are a lot of kind of problems that arise when you actually move into a permissionless setup, kind of compared to a permission setup that usually left two companies to deal with. I also want to add, Ilya said. It's obvious that you need to share processing. I don't think it was obvious to

everybody until recently. So there were multiple block chains which were like the favorite thing they were, they like to say was like, look, look at Visa, look at how many transactions Visa processes. And that's a world scale. Obviously, we can do it on a single computer, right? But as a user, you don't use Visa frequently, right? You use it like 3 * a day on a good day, right? And so finally now generally when Neo launched we made multiple bets which were not obvious to everybody.

You know like when near from day one we had that named accounts where you can rotate keys and and we had charding and it was an obvious to everybody that it's useful and now suddenly very scalable block chains which are not charted yet you know congested and they have no way of getting any more performant,

right. And and similarly like you know when it comes to account abstraction like a theorem right now is switching to to that, so, so now finally it is becoming obvious to everybody that those were correct decisions. Yeah, I mean to give an example, right, like any, any kind of a single, I call it single node blockchain, right, which is something that every single node in the network needs to process every single transaction and store all the state, right.

What it means is as soon as like let's say that that work has a high capacity, they also have a huge state growth. They have kind of limit on how many transactions they can process because of the bandwidth and execution. And so at some point there will be more demand like if if and it is very natural thing, right, the price for the transactions

usually based on supply demand. And so while transactions are cheap, don't you know at some point there will be more demand because they're so cheap to even just spam spam it to try to get some financial benefit from because the transaction fees are so cheap. And so when that happens, right, you don't have a way to expand capacity, so your your prices start to grow for everyone, right.

And so, so this leads to now you kind of pricing out people who are using this blockchain originally for normal use cases because of the spam and kind of people trying to run arbitrage for some other application. And so that's kind of just the principle where like any single like kind of state machine, right singular thing machine will get over on and start rising fees. Right. So kind of Solana is the, you know, contrasting example.

Of course, even Ethereum and Bitcoin are based on the idea that the miners or the validators and even the full nodes of these system have to process every transaction that is happening in the system. Solana has taken that idea and said yes, we have a bunch of validators at the 1800 or something like that currently, and every transaction that goes to the Solana system has to be processed by every one of these

validators. They are assuming that, OK, these validators can be placed in data centers where networking bandwidth is very high, which means a lot. They can ingest a lot of transactions from the network at a very high rate. They also assume that the machines are very performant, so the work of accounting can be. You can assume that each machine can handle lots of transactions do do their, do their accounting work and then Solana would also

assume that. OK, maybe the history doesn't need to be stored by these machines, they only need to store the currently. What are the different accounts and what what balances they own and what they they kind of like a project like Solana assumes is the improvement in in compute in terms of like bandwidth, processing power, every resources it kind of double S on some time scale so some resources double on 12 month time scale.

Another one doubt might download three-year time scale, but because of this, doubling the capacity of the blockchain would keep growing at a certain rate and the hope is that the user growth is actually slower than the doubling rate of the underlying hardware and therefore you continue to have like a cheap blockchain. Whereas in the near case like near the opposite approach where we where you say user growth can be way faster than the improvement in hardware.

So fundamentally you need to move away from the paradigm of every validator or every node needing to store. First of all what are the balances of every account in the system are, and then you also need to. So basically no single machine may have a complete view on what the balances of every account on the near system is.

And then it might also be the case that there's some machine and there's some transaction on the network and that transaction happened, but this machine is part of is is is part of part of processing that network, but it never actually executed the transaction itself. And so the essence of it is you you want to have a design that is not dependent on the underlying hardware itself improving at a certain rate to be able to service user demands. You want to have mechanisms

beyond that type of scaling. Yeah, I would add that as I said, it's not even about users, it's it's actually in in in a way it's sadly the economics, the economics of this blockchain isn't such that you need to have kind of a way to expand capacity otherwise if you want to maintain low fees because that's that's at some like at some level there will be such like saturation where some subset of people are willing to pay higher fees because they need planning like they try to capture some

economic value from an exchange token, launch whatever trading and that in turn increases price for everybody else, right. And so so Solana and and some other like kind of high capacity networks even though the idea that like hey we we have to have capacity and it will grow over time. The reality what happens is since that it gets slotted by transactions that are all trying to hit kind of the same economic

opportunity and extracted value. And so that people are willing to pay way higher fees than folks that are potentially using it for you know other use cases right for payments for example and others. And so that's kind of the, the point is and this is to add to the side that like in the state grows and everything requires validators to continuously expand their hardware even it just to continue maintaining the

network. So, so I think like to me actually like past like 3-4 months have been a really great validation and then this is not just about salon base and other like kind of even you know bases centralized sequencer, right. There's a single server effectively. But if the Nets cannot catch up with all the transactions that

they need to process. And so that's kind of an example of just like as soon as you have enough economic activity, you starting to get kind of this slot of transactions trying to capture that. And you don't have any way to either isolate it and add that extra capacity for everybody

else to to do this, right. And so example I like to use is imagine Netflix. You go to Netflix and first of all, like in the CDO ecosystem, it would ask you to choose which data center you want to watch from, right? Do you want their arbitrary data center or optimism data center,

a base or you know, blast. And then when you go there, it says like actually, first of all, you need to bring your money from the other data center where you have your money if you want to pay for this movie, pay for watching the movie here. And then second one is actually you know because somebody else watches a very popular movie, you cannot watch this movie right now at the lower price, you need to pay more.

So that's kind of the current stage right is and what we want to do is you know you go you can pick any movie and you watch it and you pay kind of fixed fee, right. That's like it's predictable for everyone.

And so to do that right. Similarly how NASA need to use Amazon that kind of scales under the hood right and is able to build more data centers that had of the demand that cats lakes has kind of similarly you need a network that is able to scale with demand and kind of you know in a way you know you have the supply demand curves and so like you want to flatter the supply curve such that even as demand grows you kind of can maintain the their fixed. Fees, right.

So this is the distinction between burst capacity and like. Average capacity in a sense where like a a system might only be using like X capacity is sometimes in a year, but then suddenly like one application or the old system might require 5X or 10X and that burst might happen very quickly.

And what you're saying is essentially that if the if the scalability properties of a system are only dependent on the underlying machines that the validators use, then that cannot change very quickly to adjust to burst demand. Like certainly lots of demand comes in, machines can't be changed across the whole network that fast.

So there needs to be like some other mechanism where a burst happens and the system is also able to somehow respond and being and be able to scale dynamically on a on a shorter, shorter time horizon. And this is this is a property nearly every blockchain kind of like lacks today, which is why

you have gas congestion. Yeah. And so specifically in last months, we went from 4 shards to six shards, increasing our capacity by 50% because we had a couple of applications who had like massive growths. We have heart which grew from zero to 5 million users. It's like over a month, million daily actives within a month. And so we started to having this actually congestion to South of

shards because of that. And so instead of just OK, everybody's now paying higher fees, your network added more capacity. So yeah, let's get into what you know what. What are the different challenges with with building building a sharded Shard blockchain? So there's a couple of them.

First of all, there are certain changes to the user experience because since nobody maintains the state of all the accounts, if the transaction has to touch multiple accounts, something needs to be done about it, right? And it's a, it's a, it's a very large design space and generally we refer to those transactions across short transactions and that's that's one big challenge. The second challenge is in the network, in which every node processes every transaction.

If you're if you have a node and you're at a particular state, you have very high certainty that every transaction was processed properly, because you literally processed each and every one of them, right? In the worst case, you can be in a situation where you're looking at a network which is not canonical, right? So maybe some other part of the network believes that another set of transactions happen on

this one, right? But that's a very different problem from someone literally, you know, like making something that doesn't make any sense according to the rules of the chain. And so in the target chain, because every node only applies a subset of transactions, you need some checks and balances which ensure that that what you see is actually a correct state that results from the properly

executed transactions. And when you start digging deep into that problem, into the problem of ensuring that everything is executed correctly, then you you start facing another problem where in order for like for almost any mechanic you can come up with that ensures that everything is executed correctly. Maybe with an exception of ZK proofs like. Once we start digging, today we.

Will we will see it? You need to be able to access certain information in order to perform the validation, and that information could be made unavailable by malicious actors. And so you need to have a sort of native you need to have a native mechanic on the blockchain which ensures that certain pieces of data are available to certain participants and cannot be concealed right. And so those two challenges are like one of the biggest challenges that exist. There are there are some others.

One interesting one would be that because the state is so big, you need to have a way for people to either synchronize state very quickly or work without synchronizing state. So I I think those four would be the most interesting ones. Right. So like, yeah, essentially, like imagine yourself like being an accountant of the near blockchain. It's it's a massive data structure. You only have a small part of

it, the transaction comes. So the first problem is OK, you might not be able to process the transaction fully yourself because you only have a part of that entire data structure. So you can maybe make some changes to that part, but then the transaction might hit OK. Now it needs to do a change on some other part that you don't have and that's a completely

different accountant. So you need to process you know some only a part of the transaction and then it needs to be like that that rename beta needs to be kind of handed over to some other party and then they do that and and so on. So like that's that's one, that's one issue. Second issue is if you're only handling a part of that data structure which has contains all the account balances, you receive that data from some place and then how do you even know that that data is genuine, right?

So that's that's the problem of stateless validation or like how do I know that this data I'm receiving it's actually processed correctly in the past? Then? Kind of like, OK, if there's any mechanic to know that any kind of certificate that would tell me that this was processed correctly in the past, then the generators of that certificate kind of need to have had access to that data in the 1st place.

But if that data wasn't there to the generators of like these certificates, then then they won't be able to generate certificates. A certain data needs to be kept in a state where you can reach it in order to do something with it. Yeah, so, so a couple of comments. So the first one would you, you mentioned that you need to process part of it and then send

it to someone else. So it's also important that that message you're sending is not getting lost, right, but it has to be delivered with certificates. You mentioned that you need to be to have certain data to create certificate. It's also in many situations the case that you need certain data to be able to verify the

certificate. Actually, like in Near in in one sense like there's a lot of complexity because fundamentally like kind of like the accountants in your system do not have, may not have access to the full data and so that leads to a lot of complexity. But in a different sense, Near is Near is similar to other block chains in IT that it's just one data structure containing accounts and like

smart contracts and their data. And this is exactly the same as kind of like Bitcoin and Ethereum where it's you're dealing in the end with a single data structure. NIR is managing the data structure in a different way. But ultimately it's it's a single data structure like that's that's a unifying property across all of these systems. That's right. Yeah. So yeah, you can think of NIR as like a very large mapping from account IDs to state.

A big difference from other block chains is that the account ID is not your key account ID. You think of it as like a domain name, right? So let you know. I would be Alex dot near. And it's not just a convenience, it's not just something that is easier to convey to other people. It's also like if you if you think about it, if if the key is what identifies your account, then if for any reason you have reason to believe that the key is compromised, your account is gone, right?

You have to create a new one. You have to try to move the assets. Not all the assets are movable, right? Like you can imagine an NFT which is not movable by design, an NFT. You know like the first user of this feature, it's not a movable NFT. That NFT is gone forever, right? So near. If if I have a reason to believe that my key is compromised, I just change it, right? It also allows you to to do to like auction your account.

Like like your account has a particular state that you think is of value, you can literally sell your account. There are services in there that allow you doing that right? And then this massive state of mapping. At the end of the day, it's still a mapping from an account to to state, similar to Bitcoin, Etherium, Stalana, any other blockchain, but that state is short. Just just to be clear, not Bitcoin because you J exos but yes Etherium so like that every account based blockchain.

But yeah, you know, like you DXL, you can think of it is as even less persistent account than than just a key. And so this state of all the accounts, it's split into multiple subsets, right, into multiple sets. So today we split by by name, right. So there would be a contiguous set of accounts that lives on Shard one, and there is a contiguous set of accounts that lives on Shard 2. And. That those boundaries are not they not immutable through the through the life of the blockchain.

And as a matter of fact they did change multiple times. When NEO launched, it was a single set. NEO launched with a single Shard, right? And then it was split into four and then in the recent months it was split two of them were split twice into two again. So it's and in the future that will be dynamic. In the future of the system will be changing the the boundaries as the you know like as the load changes automatically.

So I mean, maybe one way to imagine it is like in your like in the postal system in your city most likely like your city is kind of like divided into these different regions, each with a different postal code, right? Where I live there called yeah PLZ or something like that. And so each kind of like region of the city will have will have a number and the city will be partitioned into various different postal codes and you'll have like a post office

in every every code essentially. And Can you imagine like in near it's it's taking that data structure, you can think of that as the city and then breaking it down into like these these

areas, these shards. And the dynamism is like if you have like a region in the city that has a postal code and suddenly lots of letters are being sent there and they're like, oh, now actually we need 2 post offices, then maybe they will divide the region in the city into into two different regions with two different post office with two different numbers. And in practice that does happen

right? Like postal codes change over over long horizons and in near similarly like OK, the whole blockchain is broken down into like the shards. The definition of a Shard can also change in order to kind of route around the capacity demanding in in some way. If you think of near today, you can think of it as there is a you know, we all observe what happens in the city and how much mail goes to every post office, right?

And at some point we realize that for a particular post office, there's a lot of mail coming in and it's, you know, it's getting harder for the employees there to handle it, right. And so there could be a proposal that says, hey guys, let's build another post office half a mile away and split it like this, right. And then there there's a separate entity which is validators of the network which which either choose to, you know, to go ahead with this change or not.

And it's sufficient percentage of them, which I think is 80, you know, wants to go with this change, then it's then it happens, right. And you know, if you think of near of the future when the name of your sharding is there, it's slightly different. It's, you know, as mail starts coming into the building, another building just spontaneously pops in without without any human being involved, right. It's entirely. More like that building splits into two and.

Moves away. Exactly, exactly OK And your wall appears, yes, to building separate or like on a on a set of rails. And that happens without any involvement from any natural intelligence, right. It just it it just occurs and then it's at some point, you know two post notices become, you know it's it's getting chill there. So they can, you know come together and it all disappears. Yes. Yes, exactly. But, but I think the important

parts here right? As you said, zip codes changes, which means like when you're sending your mail, you need to like. Now whoever is in the new zone needs to update everyone. They're like in a new zip code, but. So on here, you never see the zip code on here you say I want a mail to be sent to Ilya, right? And Nir figures out the zip code itself. You don't need to know it.

That's the beauty. Yeah. And and to compare with this with other approaches like subnets, roll ups, etcetera, I mean in a way they're trying to emulate the same behavior, right? It's like oh you know this this roll up is too busy. Instead of launching there, you know you can spin up a new roll up and now everybody can go there and you have more capacity. But it's a very, you know, not just manual and expensive process, right.

I mean it's like each roll up you know cost is at least $1,000,000 a year just to run between sequencer explorers, RPCS etcetera, all the infrastructure. But it's also now every user, every developer, every sort of contract of us actually trying to use it. Now it is to figure out how to go there, how to bridge there, how what gas tokens is used there etcetera, right.

So it's it's a huge load on the whole kind of understanding of the network process which we are actually addressing with kind of chain abstraction and chain signatures as well because we do believe this is kind of a unit like what we're trying to do with Near is a universal problem, right. It's like the capabilities of the network should be able to change dynamically and everybody should be able to route things without thinking about the underlying infrastructure.

But on near we kind of solved it in a very like direct way by having this kind of namespace that is common for everyone and using that to route like our transactions and messages mail between between different participants. Yeah, that is so cool. I actually I actually own meher dot near and and yeah I've never needed to think about what chart

it is on right? So to me I I only need meh dot near and it's journey through time maybe like it was processed in Shard #1 then Shard #3 and then it's changing and I never need to know about it right. Like that's that's that's like really. Yeah, I think it, it wasn't Shard. Yeah, Shard 0, then Shard 2 and then Shard 3. Now it's on probably short 3 right now, but yeah, you don't, you definitely don't need to know about that. And like even I am like you know, there is a way to look it up.

But we actually don't show it in Explorer usually because I mean some new explorers show sometimes, but because we don't actually want people to know because it's it's irrelevant information. It's like knowing which which exact computer in on which rack in AWS is, you know, providing us with this interface we're looking at right now. Yeah, yeah. So maybe like an interesting imagination for Near is because, like, ultimately it's like this human memorable names at the bottom.

And maybe, you know, like each human memorable name actually corresponds to a person or a company because that's how the world is partitioned. And because like the Near system breaks into shards, you can almost imagine like a virtual connection of people that are transacting their business on a Shard at a certain point and these are the users of of the Shard itself. And of course like as the Shard boundaries change, the collection of people transacting

on a Shard is also changing. But maybe if you imagine it as like being you know stationary or constant for a certain while which is true for near. We can we can think of this like virtual collection of people in the in the near suburb on a sense that's transacting on a Shard. And then the on the other side corresponding to a Shard you need kind of servers or validators or postman in our early analogy that are kind of processing the mail that's coming to that particular area or Shard.

I guess like one of the question starts to become so in any blockchain network you have these validator machines or minor machines that are ultimately like kind of like this postman or accountants of the system that are doing the processing. And even in near I would imagine OK there's a set of validators that are that are found through

proof of stake. Now these validators sort of like need to be assigned to shards that hey you go and process the transactions in in this in this Shard you other one you do it there and how does that process work?

So I I first want to to correct slightly the first time I I think the 2nd analogy was good, but the first analogy was not quite correct because even though Shard boundaries do not change as frequently today as they will be at some point, it it it has been the case from day one or near that two accounts residing on the same chart has absolutely no significance for those two accounts. So I think a better analogy for that would be not the post office but like cell towers,

right. We could be next to each other and I can call you and we will be you know, served by the same cell tower or we could be into in different parts of the world and I can call you and other different cell towers but we have no benefit. You know, like I, I will never know that you're on the same cell tower and I will never

care. And it is the case of Mir that if you and I are on the same chart and they send you money or you know, like we transact on some smart contract or we'll if we are in different charts, from the user's perspective, there's no difference in, in experience, right. So you know, fees are not affected, the performance is not affected. You know like like you know, sharding is completely obstructed to it, right. And so there's no incentive, for example, to try to be on the

same chart. There's no incentive to grant, for example, account IDs or the contention was at the same, you know, accounting in the same. Chart. When it comes to the second analogy, you can think about this way. You can think, I I like, I like going in multiple steps and effectively saying, you know, let's say we're designing a new blockchain and we want it to be sharded right on. How do we ensure security when not everybody processes every transaction?

And the first idea would be, let's say we have a massive set of validators, right? So we set the, you know, the minimum key to be relatively low and we say we have hundreds of thousands of validators or even millions, I don't know. And then every Shard, even though it has only a subset of valid daters, it still has a massive set of validators, right? So it's. So we have a million total, 100 charts, and every chart has

10,000 validators. Then you can say, well, if if you sample them randomly and we relatively certain that the total set of validators has up to a certain percentage of bad games, right, we say we believe that the total set has up to 25% bad games and not more, right? Then you can do the math and you say, well, if I sample 10,000 of them, then the percentage of the bad guys exceeding 33% is so unlikely that we can consider it to be impossible either to be like, you know, 1 / 10 to the

power of some large number. And then you say, well, and because there's no more than 33% of bad guys in the chart, we can just assume that they adhere to the protocol that any state transition they approve of, Like if it has, you know, as a particular percentage of signatures of those people who are validating the chart, then we believe that that's the transition was valid because the number of bad guys is limited and the good guys signed off. So it's, you know, good to go.

It has practical issues. We don't actually have a million of validators and we do want to have more than 100 charts in in the limit. But it has a bigger problem which is the contract of a big guy or a good guy is very abstract. At the end of the day, everybody who's on the blockchain, you know, they want to make money. That's the that's the ultimate goal.

You know majority of validators, I'm sure there are some validators who are there to build the, you know the decentralized world of the future where everybody's a happy corgi owning their data but invalid. The majority of them are there because you stake money. So or rather like you have people delegate to you, you you keep the percentage, you make money, right? And so correspondingly we should think about the security in the presence of the bad guys who will try to corrupt other

participants, right? And people talk. There are ways for people to talk, you know a lot of validators are just sitting on telegram, right? And it makes sense for them to be in the same telegram groups because they they run into issues. You know, like the network is too slow. The validator, they need to know that they need to operate the validator so they all in the same telegram channels.

They all easily reachable if the bad guy wants to come and say, hey guys, I want to do this act of being a bad guy and I need that percentage of validators to cooperate, right? I can DM each of you and say this is how much money I'm willing to pay you because there's something for me to gain from the blockchain game going down. It could be some minor extractable value.

It could be that I'm, you know I'm the Solana investor and I just want you to go down and Solana to go up etcetera right and so. The system needs to be designed in such a way that a very large percentage of the validators could be corrupted and, you know, incentivize to do something bad. And we need to correspondingly, let's say we have 100 or 1000 of validators and we have on the small sets subset of them in every Shard, We should expect that almost all of them will get corrupted, right?

Or even all of them. And so the system needs to be designed in the way that, yeah, there are those, you know, postman in the post offices, but it could be that the bad guy enters the post post office, gives all of them, you know, $1000 and asks them to do something bad, you know, like a route and a mail which was not actually sent by the originator or something like this. And so we designed systems in a way that that is prevented or made very, very difficult to

execute, if that makes sense. So not only mathematics of it. But if you if you take that that scenario that you sketched out, first there's a million validators and then I'm sampling. What I mean by sampling is, you know, it's a huge set and I'm taking 10,000 and I'm choosing randomly because the blockchain ultimately needs to choose randomly and. Then if I get the. Set of 10,000 my my mathematical intuition says that if it's like 25. Percent of that.

Set of million is 250,000. They are malicious in some way and 750,000 are kind of honest. If I'm choosing 10,000 randomly, my odds. Of. Kind of choosing a set where the majority, maybe 66% is malicious, means 6666 are kind of malicious and the rest are good guys. I would think like that would happen pretty frequently, right? Like if I'm if I'm if I'm creating these sets like once every second, I would think every six months or every year I'm going to get a set which is

which is full. Of full of malicious. Now so if you have. If you have 250 out of a million, then even sampling 1/3 will never happen. If you sample 10,000 out of a million, then even then sampling 1/3 would not happen. It's extremely unlikely. Oh, it's. Extremely unlikely. So the so the my my my physical intuition is wrong.

And if I'm sampling 10,000 out of a set of million and I keep doing that, keep creating new samples once, you can do it like billions of. Billions of billions of times. Like you can do it every nanosecond for a billion years and it will not happen. We actually had the. Math in. One of our original papers, right. And it was I I don't remember like that exact constants, but yeah, it was like 8:50 or. 20

minus. Studying something like that, Probability so like longer than universe exists kind of thing. But to an extent, it doesn't. Matter because we still like if your intuition. Was. Correct. Or if my math was wrong, which is possible, it would only make the. Situation worse. Right. But we're saying we we're saying, hey, let's say the math is such that and and and I think it is right, let's say the math is such that it's very unlikely to sample a large percentage of

bad guys. It still doesn't matter because good guys can become bad guys when they sufficiently incentive us to be bad guys. And and then in the world of blockchains, incentives are often such that you can, you can benefit a lot by by corrupting, you know, a set of validators. And so you you would every now and then the validation will choose to do, will choose to do so, right? And. So correspondingly you. Want to design a system where even if the set of validators in

the chart is corrupted, right? Either either due to sampling or because they were corrupted, what we call adaptively right? So after the fact, then the system still operates reliably, and so there are. Multiple you know there. Are many ways of ensuring that and it's it's the area of research that has been developing and this is where in sharding we we use. You know this is the the biggest user of Ziggy proves in sharding.

Because you know, if I can just prove that my transition is correct, then the problem is. Solved. Right, instead of sending a bunch of validators, if you can just have a proof that says everything is correct, that it's and the problem is solved. But maybe building building this? Up from the from bottom up, right. I mean it it starts with so kind of we discussed right. There's there's account space, right. You know, think of it as a city with people and you know we have the cell tower.

So when people, you know call each other like for, for, for use of the of the analogy when people call each other in the city, right, Kind of bounces to the cell tower and then goes to another cell tower to kind of connect them and so. So the first thing that we need to do right is to ensure that every second, every transaction is recorded, right, ordered and kind of across all. Charts.

And that no, there's no way even for if everybody is corrupted in that chart to be able to kind of change that order and and for other use cases also, you know, potentially introduce something that's not valid, right. And so that's called data

availability problem. The near had kind of data availability from from 2019 then the design Nightshade and then you know as other approaches like roll ups etcetera started becoming more popular, they also needed data availability and that's kind of where a lot of the current data availability offerings are coming to market like a very short. Primer would be, you know, like let's use roll ups as an

example, right? So there's, let's say this hypothetical optimism and they they do transactions, but they do not use zero knowledge proofs, right? Actually don't know if optimism is planning to use zero knowledge proofs, right? And so corresponding we're using zero knowledge proofs for fraud. Proofs, I see. Interesting. Interesting. And so they check in the, you know, the the, the state route to Ethereum every now and then. And they say, you know I applied

all the transactions correctly. And if you think you did not, I posted just enough of cryptographic information of like you know, a gestation that that if I did something wrong, you would be able to come and prove that it was wrong, right. And if you do that, I will, I will lose a lot of money. And so I have a strong incentive not to do so. Right. And so and so then you observe the the roll up and you see that some transaction was applied incorrectly. You can go to a theorem and say

this is a transaction. It was applied wrong and this is the proof, right? But you can only do that if you actually see the transactions, right? So if the if the if the roll up was able to operate in such a way that the validators cannot see the transactions, then the roll up would be snapshotting something. But nobody can prove anything wrong because nobody sees what's happening. It's like you know it's a filled

room. So data availability is effectively this concept of ensuring and proving that the transactions that you claim to apply and the state on on top of which you claim to apply those transactions are all visible to everybody, right. And and that's something that Near had from we were either first or. Second to. Have it like I I don't know when I don't remember when polka got launched and whether they had the data will be with you from day one. OK, so you. Have like a Shard.

And then there's a. There's a set of validators that are processing transactions in the Shard. Let's first get an. Intuition of like. This. Set of like let's say imagine. Like Shard 2 or whatever the set of validators at a processing Shard too. Is it like a static set or does it keep changing, changing with time? Dynamically it's? Changing with time so.

The idea is and and so there is like what's right now lives and what's been lives for for a few years and also what's we launching with status validation. I'll probably just. Start with about. Status validation just for for ease of explanation. So the status validation there is a two set kind of two roles that eval that it can play and and all of this is repeatable once. One role. Is so-called chunk producer. Which? Is somewhat similar to what you enroll up school sequencers

right to this. The node that receives the transaction orders it in the block in the chunk, in your case. Responsible for their Shard. It sends out this chunk to others as well and then executes the transactions. And receives the kind. Of the result and so importantly, kind of where comes in the data availability when when the chunk producer. Sends. Out the the chunked information

they include. They kind of so-called erasure coded, which means they replicate this information in such a way that, you know, they send it to other cell towers such that even if everybody who's in, you know, servicing this cell tower is offline, goes malicious, etcetera, other cell towers can completely replicate everything that happened in the cell tower. So that's kind of what data availability erasure coding is. So there's a chunk producer now.

There's a small set of chunk producers, kind of similar how you know there's single sequence are usually on roll ups, but you don't want that because of censorship and reliability now. For validators. Actually have a different story, right? So what? Expansion you have this. Adaptive corruption problem,

right? So if you have validators which is sitting there for a long term for a long time, it's it's possible that you go and say like hey, if you're in this Shard and I see you in this Shard for example, or I can bribe you to you know do something for the shark and so and then you need fraud proofs. And fraud proofs are complicated and kind of require additional timelines. And so with the. Status of additional.

Say actually the chart producer not just produces kind of the transactions but also includes all of the state required to execute this transactions. And so that's so-called state witness. And so. Then any other. Node in this in the network right can receive this block and execute it without knowing anything else about the network except kind of like client of the of the network. So you receive everything, kind

of you. Need to validate that if you apply this transaction and you have the state and the state was included in the previous block, then the result of this and you can confirm that and kind of send the confirmation and so that's kind of I mean in a way it's you know kind of stateless validation is ability for any node to come in and say like hey I'm ready to validate. I don't need to sync synchronize the network state, I don't need to maintain the state in on my

desk right. Which again reduces the needs for the validator node requirements they can. Just like make sure that. Everything's OK. And so now you can rotate this validator's validate what chart all the time every block they can be actually. You know you can randomly select set of validators.

They can be overlapping. You can select you know out of the million you can select you know 100,000 per Shard and they've evaluated 10 shards for example each if if there's enough capacity or or any kind of parameters and they are able to validate this because they don't need to sync into the Shard. They don't need to process every single transaction that ever hit that Shard. They only need to look at this specific block.

So that kind of opens up a lot, you know, both kind of, you know, you can imagine I'm actually really excited that potentially, you know, not probably this year but soon enough somebody can open up a new tab, you know type in the URL which has kind of a validator node in in a browser and actually start receiving blocks and validating them because again like you don't actually need anything else and and browsers have web assembly to execute the the transactions

embedded. So that's kind of. The like. Very very light, lightweight validation that you can rotate all the time. So. Like I'm imagining. This state. Witness is as more like. So let's say like there was block N and. Then the transaction. Came in and the chunk producer made n + 1. But is it the case that? For every transaction, almost like for every transaction that they processed in in that block, they are creating individual

witnesses for each transaction. So if you take transaction one, they'll say, OK, transaction 1 modifies these two. Accounts these. Two accounts had this balance previously after the modification. The result is these two accounts have this balance now. So our state witnesses this was. What was before? This transaction was processed. This was after the transaction. Process. But the data that's being supplied is only those two accounts that are hit by the

transaction. And so for every transaction you are just creating like these like these bread crumbs, these bare minimum amount of info that is needed to kind of validate that transaction and that's the state witness for that transaction. So every transaction that comes in the block of the Shard by these this chunk producer entity is kind of broken down into these individually verifiable

pieces. And then those individually verifiable pieces are scattered across all of the validators of the Shard and they can kind of do the job of verifying these individual pieces one by one. And because each of these piece can be verified individually you that's why you are able to run the validator in your browser because. While your browser. May not be a powerful. Machine, It can still validate a few of them. Yeah, yeah. That's, that's right. So. Is it the case that?

This chunk producer that's kind of like more a more long. Lived. Role and then the validators of a Shard is like a more short lived role. Meaning as a validator I'm doing some verification in this Shard, then few seconds later in a different Shard than a third few seconds later than a different Shard. Like that I'm constantly switching as a validator but as a chunk producer and few seconds later is a little bit. Is a little bit consulting. We produce a block every second

but. Yeah, it's like every second, every. Block you can be valid in a different Shard because there is actually no difference. Like you, you don't actually care which Shard it's on because for you it's just a block result like a set of transaction to resolve the information you need and so that's why you can rotate validators now every second. I mean there's like networking requirements and you know data information propagation, but in

principle, yes it can. It can rotate every second and then for chunk producers they rotate every epoch right now, which is 12 hours or 12 to 14 hours and there. Where? You because you need to actually sync the state. Like if you're moving to the next chart for the chunk producer, you actually need to know everybody's balances, right? And so you need to actually download all that, make sure

it's correct, consistent. And then kind of while you're downloading, you actually need to now receive new blocks and apply them as well. And so you're kind of doing 2 jobs in parallel. You validate it, you kind of chunk producing the Shard you're in and you're getting ready to produce Shard that you are the next time. And so that requires kind of more sophisticated setup. Right. So, OK, so then you have. These validators that are constantly jumping.

From Shard to. Shard validating some small pieces and when is it the case? In like when I'm validating a certain piece, I'm also adding my signature saying yes I checked it and it's correct and every transaction and it's it's witness is kind of getting more and more signatures or attestations on the validators and that's how you're building up trust, but it's not. Accumulating over time. All the signatures happen on the same block, right?

So, so every block you need to do a sign off and it's not, it's not the case that everybody signs of every block, right. We we just need a certain majority. And you know, like because blocks are created every every second and the validators are running on the relatively community hardware, sometimes you will miss a signature and like if you go to an explorer you will see that like nobody has 100% up time, people have like 99%, right. But yeah, but the idea is that, yeah.

There's a. Set of validators. They validating a particular block, We know whom we expect to sign off on the block and then you know a majority, a percentage of them signs off and the block is created. You can look at it and say, well, that many validator signed off at this point. I know which charge they. Were supposed to validate. I know that unless someone corrupted them in .1 milliseconds, you know it's There's a relatively high certainty that the state

transition is valid, right? Because on the. Other side, the mathematics says that when I'm sampling these validators randomly, my odds of getting a completely bad set of validators. If 25% of the set in the bigger network is corrupt, it's kind of low. So because of of that sampling, you're able to kind of trust that OK, while these validators are considered or you are sampling them constantly every second you are changing the samples, but because the prob. You.

Calculate the. Probability of 1 sample being. Being malicious like. So 66% extent or something has been very low, you are able to trust kind of like the signatures on the witnesses of your transactions and be be sure of the state of of a particular Shard and. Meanwhile like these chunk. Producers, when they are producing these blocks, they are also forwarding the data corresponding to these blocks. To a set of validators.

Now these this this other set of validators may be different from the validators that are checking. The witness the. Witnesses of the transactions like they don't need to be the same set and. So so. There's two things that happened. One is. You I. Mean the the. Validators. That are receiving to check the witnesses that actually received the data as well, right? Because they actually can validate the transactions. But also you want to send to other shards as well in case you

know this whole Shard of sale. But also like just you want to route outgoing messages and so we're actually combine the message routing data availability. And. Consensus into kind of one process where let's say you have you know let's say like you withdrew money from my account on Shark One and then you're sending money to Alex within Shark Two. So now there's a message going to Shark Two saying you know you should debit Alex you know 10 year.

And so now that message is not just a message, it also includes a so-called eraser coded part of the transaction data that this Shard was producing. And so kind of this process ensures few things. One is everybody you know then goes and when they actually signing and confirming their own information and and sending the their approval, it also confirmed they received the needed chunks, the needed kind

of parts from other shards. And so that's also provides us guarantees to this this message delivery from from Shard to Shard, it provides the data ability guarantees and it's all kind of, you know integrated into the consensus messages that are being sent by validators to each other to actually accumulate the BST consensus let's say. It's worth mentioning you there's an. Extra role called block producers right.

So there's an actual is the blockchain, is the near blockchain So it's not like because often when people think of sharding and you know many sharding blockchains do work this way. People think of multiple chains, right? So every Shard is a chain. It is not the case on near. On near there's only one block chain and and there are block producers creating blocks on the chain and when but but those blocks do not contain the actual transactions, right? Or or like logically they do

right? Like you can think of a block exactly the same way as you would think of a block on Etherium, where it has like a header, consensus information and a bunch of transactions with a difference that while logically transactions are there, physically it only contains information about what we call chunks, SO1 chunk per Shard, or rather up to 1 chunk per Shard. And physically, the block does not contain those transactions, it just contains the information about the chunks that were

produced, right? And at every particular block, some shards might might miss a chunk, right, because there's a particular chunk producer responsible at every particular moment. It could be a flying, it could be, you know, busy, etcetera. And so a chunk could be could could be missed, right? But if the chunk is produced, what happens is that the chunk producer, when they produce a chunk, as Ilya mentioned, they erasure coded and they send a part of it to every block producer, right?

And the block producer would only sign off on the block if they have a part of the chunk that that that is intended for them. And so this is where consensus and data availability are sort of mended together, is that, you know, in order to reach a consensus on the block, 2/3 of all the block producers waiting by stake have to sign off on it, right? It's a BFT consensus. If there's no 2/3 of signatures, the block, you know, we wait until we have, right? We we favour safety over

liveness. And correspondingly, if we cannot get 2/3 of signatures, we will stall, right? But we and if you have 2/3 of signatures, because the block producer would only sign off on the block if they have their part of every chunk in the block, right? Then you know that 2/3 of the block producers have, for every chunk included have their little part. And the eraser code is such that you need 1/3 to reconstruct any

chunk, right? So as long as you believe that no more than 1/3 of the block producers is malicious, and if you have 2/3 of signatures, in the worst case all the malicious actors are included in those 2/3. So you have 1/3 malicious, but you still have 1/3 honest and so you can you can reconstruct any chunk, right? So every chunk is available to everybody, guaranteed if you have consensus reached on the block. So data availability and consensus are merged together.

Into a single. Single mechanic. Right, so like, this is wicked. Cool. And it's also like hard to understand because this is actually unlike any other system where data availability and consensus are usually like like two very separate processes like whether you go from the Ethereum to Celestia to all of the. But in near it's almost essence right you you want. To it, it sort of begins with with the your goals, right? And then and then we were going back right.

So the goal was to have crossword transactions and generally communication to be with a delay of one block, right. So if I effectively, if in a particular block a transaction initiated and it wants to do something in another Shard, we want that to happen with a very high probability in exactly the next block, right? And so if the data availability was separate from consensus, that would be extremely hard to ensure, right? Because we need to be certain that data is available as of the

moment. When the block. Is produced as opposed to that being a separate process, right? And similarly Ilya mentioned there are three things which are merged together, a data availability, consensus and the message passing right. So together the chunk that is now totally available and the moment of consensus being reached also contains the messages that needs to be routed

to the to another chart. And it is designed in such a way that it is ensured that by the time the chunk producer or that other chart is producing the chunk, they not only know that the messages exist, but they also have them right and so they can immediately act upon it. And moreover they have to act upon them, right? So a chunk producer, the chunk would not be valid if the chunk producer did not act on the messages that were sent from another sharp.

It could be that they don't act upon them immediately because they because of congestion, like you mentioned, everybody sending receipts to the same sharp right. So that is automatically handled, right? It could be that the receipt was not processed immediately, but the receipt was acknowledged immediately and it's put on the queue immediately. And most of the time it is also acted upon immediately because congestion is there. You know, most of the time there's no congestion, right?

But it is all. Part of the same. Process where we ensure that if something happens in block M and that something wants something else to happen in another shirt, that something else will very likely happen in N + 1. And maybe to to use like. ECDM and roll ups and allergy here that you know you have optimism producing a block, right? That block immediately sent to ECDM validators who included and as well as they include every

other role. Let's say we have our optimism arbitral trying to to send money directly between each other, right? So like both of their blocks need to be included at the same time immediately in the same ECDM block, right? To guarantee data availability? Because now the kind of our like let's say optimism sense something to arbitrary connect. In it but. If it's just that right, then arbitrary needs to read out the state from from the CDM, right? Which adds extra latency.

And So what happens here is you assume that optimism and Arbitrary are that those sequences are also validators in the network. And so you send them, you kind of optimism order and send their blocks to each other, right. They confirm it and kind of those such stations are now allowed to progress forward the blockchain.

And so like as if all the roll ups themselves formed a CEO consensus, right and we're sending information to each other directly and in turn that allows to optimize for latency and for kind of this cross short communication because everybody talking to each other directly and but rely but then you know sending confirmation to the to the whole United system. So I'll I'll try. To state this in kind of my own understanding of how it works. So so it's like we need two

different views. One is kind of like the Shard view and then when we need like the global view because there's a global block. So you know, we we went to the Shard view pretty well. It's like there's a huge set of validators, Some of them are assigned to the Shard for. For a block and then then we. Reassigned to other shards, and they're kind of like validating the parts of the transactions.

In the Shard. Now these validators some sometimes they also get the responsibility of being these chunk producers which are more like long lasting entities for 12 hours for particular Shard they are. They are producing like OK for this block. This is the set of. Transactions. And here are all of the witnesses, witnesses for it. So these are like long lasting entities.

So you could have like a short lasting role and then you could also get a long lasting role as a as as a validator, right. So that's kind of like the local view of the Shard. Now in the network there's like a global view where there's actually a single blockchain with like block after block after block like like Bitcoin or Ethereum, but what that block contains is not the transactions themselves, but the chunks, probably different shards that are sort of accepted.

Post. Consensus as being correct. Like, OK, so here's block N it contains chunk chunk 1 from this Shard, like chunk X from this Shard, chunk Y from that Shard, chunk Z from that Shard and so on. And it just contains what chunks the networks is is considering, like finalized. And so all of the validators are trying to build this blockchain that just contains contains chunks and. The. Validators are kind of like signing off on that on that

block containing chunks. They are trying to add their signatures to it and their logic is something like what they are checking is corresponding to each of the chunks that are part of the block. Did I get the the slice of data in order to ensure data availability for the whole network? So as a validator, when I'm signing off on a block, what I'm checking is OK, this block contains these chunks. If it contains these chunks, I should have received the data corresponding to these chunks.

Do I have it in my hard drive? Yes. OK, so that's one sort of validation passed. And then I sign and like all of the other validators are signing that. So fundamentally the network is coming to agreement that we have the data that we are going to need to reconstruct any part of the actual transactions in the block in the future. That's the thing people are

coming to consensus. So in a in a normal like Bitcoin like the when a block gets finalized, the network comes to consensus about these transactions that are processed in near. When a block gets finalized, network is coming to consensus about the validators collectively agreeing that. They all have. The data needed to reconstruct every part of the block should the need ever arise. Is that right? Is that intuition right? And they also. Like in the world of. Stateless validation.

We also have the information that each of those parts were reconstructed and validated by by a set of validators, right? Right. Not only that, the. The that the data is there that can reconstruct everything in every transaction in the chunks, but also the data corresponding to that every. Transaction. With its state witnesses was. Validated by a. Certain number of validators of the shards where the state witnesses originated. So it's coming. To consensus. About data that's yeah, that's

really interesting. It's really cool. Yeah, I mean. The big the big benefit. Was and I mean we do get this question which is like, you know why ECDM kind of shifted from sharding to roll up architecture and I mean. First of all, obviously not. Talking for anybody individually, ECDM or ECDM is

not an agent themselves either. But practically speaking, to design something like this right, you need to build everything from ground up. You see how consensus, data availability, message passing and kind of validation correctness all layered in into one system and that requires kind of this like approach and as well as the VM itself, right?

Because VM now needs to be aware that compared to you EDM where everything is kind of available, right, you can always say like hey, give me a state of that other account, like tell me how much tokens does you know Alex have in, in the case of the charted system like that requires a cross chart message, right? That requires saying like, well go find where Alex lives, right? And you know and ask him how

much tokens he has. And so you need to design kind of everything from scratch, from bottom up with this understanding given the goal we had, which is like you know, ultimate horizontal scaling that is hidden from the users and developers in many cases. And so for ECDM like they, they can do not have such a luxury, right. They have a working system, extremely valuable, extremely

kind of integrated everywhere. And so they needed something that is kind of an evolution of their existing system versus a complete rebuild right from scratch. And I mean roll up architecture, you know, we use it as an analogy because a lot of it is similar. ECIDIUM provides the availability and consensus roll ups are able to communicate with each other, but they need to go through Etherium pretty much to kind of validate and and settle

before a message can be passed. And so a lot of it is kind of spiritually the same and it's just because of the kind of legacy of like well now each EVM is the whole separate universe of accounts, right? And so now you I am as a user have account that every chain right. I don't have a singular balance now that I can use everywhere. Like all of that is kind of a legacy, you know how do we kind of upgrade the existing system into more scalable platform.

So that's kind of really the, the biggest obviously you know in a way better that we had which was like starting from scratch and kind of designing it. Was this the luxury of the clean slate? Yeah, luxury of the. Clean slate is. Is what you had, right? Right. So as a. Validator. Sometimes I am. Taking on this role of being a long. Lived Chunk. Producer for 12 hours in a particular Shard.

I am constantly taking. The role of validating the state witnesses of a Shard and I'm being assigned from Shard to Shard to Shard and then. Every time a kind. Of like a block is produced in the global main network what I'm. Signing off on. Is corresponding. To that block. I should have received some data to backup the data of the state of the entire network. The data of the entire network, I should have received some small chunk of it. I can identify it, have I received it.

And then I should not only should I have received data, but I should have also validated some of the witnesses in the global network or in the Shard. Have I done that? And if I have done that, I sign off.

And if a majority signs off, that is when Near says OK, we have consensus over all that the data is backed up properly and all parts of the update had witnesses and they were validated by enough, enough, enough validators and therefore like this block is correct and like then that's done and then the chain moves on. Yes. Yeah, that's right. And. At the end of the day, all of that complexity.

Boils. Down to you know, like there are checks and balances that people do, but at the end of the day, all you care about as the observer is do I have signatures from the sufficient set to validate, and if I do, I know that they have done all the work, or at least they claim to, to have done so, right? But that's. But that's a whole set of validators, right? So for them to be corrupted, you need you.

Need to corrupt. Like a whole big proof of stake system, which, you know, yeah, it has its own, you know, design. There are certain design principles that don't allow it to be corrupted without a massive amount of money being lost. And yeah, and so that allows also, you know, other chains to easily create light clients, right, So. So near itself. Encompasses a lot of processing

internally. It could also be a part of a bigger ecosystem because building a lifetime for near is a relatively straightforward process. And near having you know general purpose WASM machine can can run any light client for any blockchain that allows a light client and that allows you to have two directional bridges that effectively say as long as the light client of both chains is not compromised. And light clients are very hard to compromise despite the term light client like you know

compromising. In the theorem. Light client or near light client is. Extremely hard. Right. And so then you say, well for as long as those are not corrupted, Near is also part of the bigger ecosystem of of more. Chains. So I guess. Like every creator, when they make something, they end up having like there's usually certain element of the system that they wished was better. And then do you have those? Like what? What for you individually?

What parts of Near are you kind of unsatisfied about in the design? What I mean, there's few things that we're still. Finishing up or like. And and still on the road, I think. So we mentioned dynamically sharding, right. So right now as Alex mentioned, right, this is more of a kind of a governance, technical governance process.

But the benefit with the stateless validation approach is that because we rotate validators, you know, every second, now you can actually change the number of shards for validators very quickly, right? Because they they again, they don't really care how many shards in the network, they don't care which Shard, they just received the block. And so the benefit? Here if. We have sufficient kind of redundancy on chunk producers.

We can split the chunk producer group and say like you know as of next block you don't care about half of the chart that you were in, right. So comparative you know to to where we are now, where we need to like everybody agrees you know new client has been drawn, validators spit it up.

You know, this shortly happens, this can happen literally instantly where everything like, OK, now we split into two shards, you know, now I'm ignoring just half of the transactions at that and they still receive because people are still routing to me and I'm just processing this half And then on the junk produce production side and then on the valid side, you know, now you're just assigning. Yeah, sampling just from a different for the listener, this is the.

Cell tower splitting into two dynamically, or the post office splitting into two exactly and this can happen.

Like immediately, right, which is you know huge, huge benefit for the spike in usage where you know sub applications just went you know viral and esteem and or you know talking claim or whatever and you can like hey let's just pull it out into a separate Shard let it you know go nuts and then maybe merge it back in like a day or so. So that that's that's you know definitely huge kind of benefit of debt and and for context I mean status validation is something we published in

February and I mean been building last year and should be launching within the next few months and then we're going to start working on dynamically sharded now. From my. Perspective. There is a few. Few things that. Beyond that that would be that's what we should be working on. One is and and now Alex actually worked on some designs for this earlier is needless check production. So right now there's one chunk producer at a time that is responsible for producing chunk right.

And I mean they kind of rotate and they randomly assign, but still you know if that chunk producer is offline now we have a gap on that time slot. And you know when you drop in GPS you have you know high latency with the users. And so the idea definitely is they can also be. They can also be deduced. Which is, yeah. Something that happens on other networks right now? Not not. Not because we sound.

Resilient to that in Just bad Guys shows another network to to do those, not ours, but it can happen eventually, right? So also leaderless chunk production consensus is something we need to implement and then we have a a a pretty clear way of achieving it. So leading by leading. This chunk production SO when you say that I'm kind of like reminded of algorand where where the the essential idea is like.

When you think of a network. Like like you when you think of like a Cosmos network, the block chains rolling. Along. And blocks have been produced. Everybody knows who should be producing block n + 1, right? It's publicly known. And if they fail to produce, the network waits and then. Realizes fails to produce. And then the network also knows if that guy fails to produce the block, then this other guy should produce the block. And there's kind of like a almost a line, a queue made of

like who has rights to produce. And so that's only what you mean when you were saying leader. So if you have a Shard, you have a multiple validate like multiple chunk producers, but which one exactly produces the chunk for this block? It's currently. Known. In near just like Cosmos, but what you would like is a system where there are N chunk producers and then when a. Certain block rolls along. One of the chunk producer realizes they have some. Kind of winning.

Lottery ticket and they can produce the chunks. So this is no, this is not so you're explaining algorithm. And it is not. I'm explaining algorithm, so is, is. Is that division for lead now? There's still a leader in what you? Explain. So the only difference is that the leader is not known in advance, right. So partially solves the problem. It's much harder to give those. For example, you cannot give those someone you don't know,

right. And so by the time you know that they won the lottery ticket and we actually do the the way we think of it is similar. So an algorithm, you actually don't know in advance that you won a lottery ticket. You, you know like like a fact. What happens is that everybody looks at their lottery ticket and sees the number, and the highest number wins. But you don't know if you have the highest number because you don't know the numbers of

others. If you did and others did, then it would be no different from Cosmos. Then everybody would know, right? So instead you say, well, I have a sufficiently high number for me to believe that I might be the winner. So I will produce the block. I will publish it, maybe someone else will publish. And you know, given my number was 97 out of 100, there's a good chance that I was the highest, right? But maybe there was 98.

This approach has a has a minor problem that still multiple blocks will have to be broadcast, right? Because like effectively either the threshold is too high that every now and then nobody will broadcast a block because nobody want the ticket above a certain threshold. Or it's like sufficiently low that multiple blocks will do the

broadcast but in our. But in our case, something interesting to think about is that at the end of the day, the chunk will be broadcast to everybody like a little part of it, right? And so everybody will have to receive. But within the network, within the, within the set of the chunk producers, what they can do, they will have to send the chunk in its entirety for validation or like with with the validators of the of the chart, right.

So you can think of a system where on the high level what happens is that every single chunk producer generates a chunk, not just some people who who are beyond certain threshold. Everybody produces A chunk and then everybody stands to every person in the Shard. So like every chunk producer every you know, like eraser coded part. And so at the end of the day the network overhead is not chunk size times number of participants, It's still proportional to the chunk size,

right? But but now every but now you have as many chances, you have chunk producers and then you still do the lottery ticket. Then you review your lottery ticket and if you want your chunk is is accepted right? But there's no issue with the choosing the threshold and they be spending the network with multiple chunks that have to be exchanged in its entire team. So that's the high level idea,

right? But the high level idea is that now you can literally never a chunk will be skipped, no matter you know, like as long as there's one person you didn't do, the chunk will not be skipped. And it's, it's a slightly, slightly improvement. Over. An AL grant idea where you, you know, have a threshold and but still exchange the whole block. Yeah, this is really. This is really. Fascinating. So I think the essence of the design being that. In Bitcoin or in?

Ethereum When When I? Have a block. I have to forward. That. Entire block to everybody, and there's a certain like diameter of the network. So I'm a validator and there's some validator to which I have the worst connection, like there's multiple hops and that entire block must be now sent across the diameter of the other side to that worst other validator. But in near almost, it's like I have a more efficient. Microphone or. Megaphones I produce the block I.

Cut it into lots of pieces. And I only have need to send the small pieces to all of the validator. So there's some validator to which I have the worst connection, but I only have to send a small piece to through that. Worst. Connection through that diameter of the network and because of. That because. Because like this, this broadcast of is only piece piece wise and this broadcast is kind

of like efficient. You can afford to have a design where in a Shard, all of the chunk producers produce a block. They broadcast their pieces and like. Then they compare OK, which of these pieces has some highest amount of randomness and that becomes like the canonical chunk for for that block. Yes, yes, that be a. Chunk of the block, Yeah. And it's like depending like that person I have the worst connection to depending on why

they need my block, right? If they just the block producer and they just need to to sign off, it's sufficient for them to have just that little piece. We don't need to reconstruct the block or a chunk. But if they do need the whole chunk, it's still more performing than sending a whole chunk to some, right? Like I sent little pieces and then that person will collect little pieces from different

entities, right? So it's still a faster way to propagate information and and I guess the advantage here for us in building that leaderless consensus, sorry, leaderless chunk production is that we already have a concept of this erasure code, right? We already sent small pieces, we already have the mechanic to gather them, right? So that's it's much less invasive change.

Therefore, a network where today blocks are being sent fully right, so for them to implement the same feature would be implementing a lot of new mechanics. Well, in our. Case it's just plugging into existing machinery I I wanted. To touch, So you asked. A question which I think is interesting and and nearly covered it from a very different perspective of sort of, you know, a. Something that. Design wise is not great and we would like it to be different.

I I think something I thought a lot from the day we launched near is that the accounting model for sharding is quite different, right. Like you cannot have like flash loans for example easily because accounts on. Different. Shards and everything takes a hope, right? And we've been trying to solve this problem like we were trying to find a way to have atomic transactions since day one with many different designs and it's

a drawback of sharding. But what's interesting is that I think slowly we're coming to to realisation that long term not having sharding is not an option, right. So in essence the highest throughput block chains today which are not sharded are getting congested and that it's only that much they can squeeze more, right? Like they they work day and night to remove all the sub optimalities, but at best they will squeeze out another 20%,

right? Let's say 50% that will get congested again like that optional blockchain today is a fraction of what we want it to be to consider the whole ecosystem to be successful, right? And so sharding will have to happen. And when sharding has to happen, people will have to deal with this disadvantage of of this different account model. And so Near in this case is positioned extremely well because from day one every application near was built with that account model in mind, right?

So we have tools, we have understanding developers in the ecosystem are used to working in this set up, right? While in the rest of the ecosystem people are still operating in this atomic transactions mindset which they will have to abandon eventually. Because at some point if their application is to scale and applications they depend upon are to scale, they will not be able to maintain this atomic guarantees and scale to the user tried.

So they will have to abandon this mindset and we're positioned uniquely in the sense that everything that is built on Near this, you know, reach ecosystem applications is built in the future proof way. You know, as we create more charts, every application in Near gets to take advantage of that, while for any application that is built on, you know what we call synchronous. Runtime. Then then they will have to to do right their applications I I. Think that that actually the really.

Interesting part is because near is kind of a synchronous in in the way every contract, every account and every contract is designed to be independent. It actually doesn't. Matter if that. Account is on near or not and so that's why kind of a lot of the chain kind of abstraction ideas becoming because well it. Actually doesn't matter for like

for. REST finance like an AML on near doesn't matter, the token is a year on the stadium and based on Salina like you can actually deposit any token and the rest side answers will be able to handle it and so. So like generally speaking nearest design with every like every contract, every account is kind of handling assets that are living on like in a synchronous way, right. And obviously it's good to have them on near because you get like one second communication time.

So like the latency is very low and you know that counts faces is nice but it can be living somewhere else and it's like if we have kind of a message passing way of sending this then you know near smart contracts know how to deal with that And all our standards like here C20 on analogous standard is designed with school backs and kind of that's the passing mind. And so again like compared to right now in the VMS they tried to figure out how to do cross layer two communication.

The challenge is really lies and like all of the standards. So I can imagine trying to say there's a 20 from 1 chain to another like there's no standard address space or anything that supports that. It's also synchronous like expectation is that that transaction will execute same block but actually it needs to you know be scheduled, message passed settled, you know send somewhere else that revalidate

etcetera. So, so really that's kind of the IT it, it was a very non trivial trade off, right. And and we kind of I would say had a had a period of time where we're like did we do a right trade off kind of but but right now yeah we're seeing like we're literally seeing the validation of our pieces kind of throughout, yeah. So. The point being like something. Like Flash loan where in the near design it's hard it's not that flash yeah. It's not like Iron Man Low.

Yeah, yeah, it's. Not that flash. So at some point in 2021 or something like that, it must have seemed that, oh, Ethereum has flash nodes but near wooden and that's a problem or is it perceived? Problem. But when you look at like Ethereum in 2028, which is Ethereum plus all of these L twos and L threes, you can't have flashlights across their entire ecosystem and you can't have flashlights across near. So it's fine like like it's a. It's a downside, but not really

exactly like the modern. G 5/20/28 will be what what people been building G5 near the past 2-3 years and as Ilya mentioned, this entire. Ecosystem of L2's and L3's will also be part of near ecosystem because because of chain obstruction, right? Any account on any chain can be, you know, with a very thin abstraction layer perceived as a near account, just with a higher latency, right? So so like a near account to near. Account will be one second always but near account to

optimism account. It's going to be exactly the same protocol on near site, but the latency will be higher because there's communication between them. Yeah, I'm not going to chain abstraction. Because I feel like we've already covered so much, and I did cover chain abstraction with in the previous episode with Ilia. So I mean avoiding that because things things start to become only too complex. Curious listener? Can Google near chain? Abstraction. Yeah, I I just.

Wanted to to to give an understanding why like like chain obstruction didn't come from nowhere. Chain obstruction was the mentality we took was near when we designed near it's just like now we expanded that to kind of all chains but it's still kind of you know the sinking we put in into designing near is still there.

Now how do we apply it to the whole that three again assume like you can think of the optimism just being another Shard of near and this is where actually like you know for example ZK proofs it and Aglayer what Aliga is working on is all coming together because like if we can unify security if we can kind of provide kind of common security layer then on top of this you know and. Again, we have. Near DA, we can actually settle DA of other layer twos.

Well, now they're actually not that different from other shards of near. I mean there's differences in sequencing and production. And so like there's things to

handle under the hood. But again we can kind of extend our layer of abstraction that we provide to users and developers to kind of cover up that and say like actually you know if there's decay proofs and data availability, we can actually like say security is the same and now we can message pass and we can do gather species this way and so.

So that's kind of the idea is like you know how do we, how do I apply the same methodology and and kind of user experience, developer experience but but then expanded back more to the rest of it 3. So like from. My perspective, you know when I look at like Ethereum Ethereum's road map and then like the near and near road map, one of the things that stands out to me is in Ethereum the road map is based on like scaling where these L twos and L threes.

But the relationship between ether, the asset and the core asset of the L2 can be synergistic in at times but it it can be non synergistic at other times right Like so. So it's like the L2 pays the main Ethereum chain for certain services and the service intended is usually data availability. But it can be the case that the L2 generates 100 million in fees, but it only pays 500,000 in fees to the main chain. It's this this relationship.

Is kind of like great if the L2 is kind of building a. Completely new. Market that Ethereum never had right? Like imagined? I don't know, some some AI decentralized app comes and the L to capitalize on it built it. They got 100 million in transaction fees and paid 500,000 to Etherium main chain. It's great for the Etherium main chain because there's a new revenue stream coming. It went 500,000 but it's new.

The interesting case becomes when some app that was massive like that's like massively popular on the Etherium main chain generating millions in fees ends up thinking it's better I migrate to that L2. And so they might be making like 10 million in fees on the main chain and then they migrate to the L2 and then the fees get cut to a million and then Ethereum's only making 100,000 on it. So, so it was making 10 million in fees and now it's only making 100,000 in fees because the

L2's. Ecosystem exists. And they're kind of like the relationship is well from the Ethereum ether holders perspective. That isn't so ideal, right? Because you're you're losing adapt that might have been cultivated by these EM network over years and then now it's kind of like migrated away. And in practice this has happened with something like DYDX. But what's really cool about near is like this sort of system doesn't exist like the shards.

Like the relationship between like, near and the shards is kind of the near token. Kind of like. All the revenues made by all shards and kind of there is not in order to. Scale it. Doesn't need to have these complex economic James be present between kind of. Like a main chain. And an execution layer which is very much there in Ethereum, and I believe that this will be, this will become a relevant feature of Ethereum's ecosystem. Politics in the.

Future and NIA will just not have any of it. Cool. So yeah. I guess we can we can keep it at that and it was it was great to have both of you on the podcast. Maybe we should have. Another one to discuss on how Alex is planning to use recent developments in the AI technology and what he's building there. I was going to say that it's not a coincidence. That AI stands for Alex Amelia. Yeah. I mean, thanks.

Thanks for having us. Obviously this is a highly technical topic that I think it's it's been really hard to explain in general. I mean, we've been trying to do this for years now, but. There kind of the. Core idea is also like it was really hard to prove it out when you just launched because when you just launched you don't have anything so there's no users. So there's no need for sharding.

And I think like we was going to have that problem in Web 3 where everybody was claiming scale, but until you actually have like real world you know, massive user base to actually transact, you know, just like a a general improvement was enough. And I think only in last probably like 3-6 months we've seen you know on near for example kind of multiple kind of million user. Applications.

Launching right near right now somewhere between 1.5 to 2 million daily active right, which is more than any other blockchain right now. We have more transactions usually than all layer two combined at least on some days and so like that that's kind of where this started to prove out, right? And for context, we're still under Solana transaction numbers, so Solana counts. Solana counts. The consensus transactions. So it's. I don't know how we compare.

To the to the actual member. Yeah, but. But generally speaking, yeah. Like we have more data active users than Solana and most of the days Tron because Tron is kind of second the biggest right out. But again like this, this is the point that you know as we started to see kind of this growth like we started to see you know congestion and sounds of shards. And the idea was that, you know, we can just expand capacity without increasing fees versus every other blockchain,

including Tronic chips. Really good example because the transaction fees went from being very cheap to actually now being like, you know, 30-50 cents for their users even though they are running, you know, it was like a small subset of validators, you know? A a kind. Of modified Indian chain. So I think like that's kind of where we're starting to see these things play out. And obviously again it took a while right to ecosystem to mature the applications to to you know build and launch as

well As for them to gain users. But now we're starting to see this story really playing out and you know on real sweet. It's it's exciting and it's also kind of tell now is really good time to tell the story and kind of explain how it works. Cool. Then I'll catch you again on. Epicenter in the end, Alex, thank you. Thank you for being there. Thank you. Thank you. Thank you for joining us on this week's episode. We release new episodes every

week. You can find and subscribe to the show on iTunes, Spotify, YouTube, SoundCloud, or wherever you listen to podcasts. And if you have a Google Home or Alexa device, you can tell it. To listen to the latest episode of the Epicenter podcast, go to epicenter.tv/subscribe for a full list of places where you can watch and listen. And while you're there, be sure to sign up for the newsletter so you get new episodes in your inbox as they're released if you want to interact. With us guest.

Or other podcast listeners, you can follow us on Twitter and please leave us a review on iTunes. It helps people find the show and we're always happy to read them. Well, thanks so much and we look forward to being back next week.