Building Trust in Peer Grading with David Kofoed Wind of Eduflow

online peer feedback. David earned his PhD at the Technical University of Denmark, with a focus on machine learning data science and educational technology. David has co authored six scientific publications on abstract math, algorithms, network analysis, and statistics before becoming an ad tech entrepreneur, David Kofoed Wind Welcome to EdTech insiders.

right. So as a teaching assistant in a ton of courses as a student, and when I got to my, my PhD, I realized pretty quickly that I was, I like teaching a lot more than researching. So I was trying to kind of do this research. But then I was also getting myself into teaching a large course, I wanted to teach a course about data science, I was like, more practical than the theory heavy courses we had. So I got permission, I set up a new course in the cost base, and I put it out there, and I called

it something with big data. And then I think we expected like 20 students for the course. But 150 people enrolled the first semester, I think, mainly because a lot of people come to our university from abroad. And then they go to the course business search for data, big data, whatever this was when I Coursera, which is starting up with a big data courses in machine learning, and so on. So it was pretty hyped at that

point. So I learned this like, three months before the semester basic like, Okay, you're going to have 150 students. And I basically decided the course around extreme amounts of work, like you have to work a lot in this course not just listen, so they had to do weekly assignments and like write this

long papers. And suddenly, I could just do the back of the envelope math, right and say, okay, 150 students, a weekly paper, I spend five minutes reading that even then it's just like, 40 hours of grading a week. That sucks. Like, that's, that's not gonna be fun for me, it's not going to work, and it's not going to be useful for the

students. So that's kind of part of why I started this whole educational technology thing that I had to solve this problem, then I guess, kind of a side story into that as well as that a couple years earlier. I've been doing some research and another university in like assessment, like voting systems, like how do you catch fraudulent

votes in a voting system. And then my mind gravitated towards like, I guess you could do kind of like peer review here where like, the students would give feedback to each other in my course, then I don't have to give them any feedback. And then I could use that cool algorithm from back then to find the fraudulent students. So my mind was like, okay, I can build some peer feedback software that will allow me to let the students do the hard work of giving

feedback. And then maybe there's even a research paper this where I can deploy that algorithm. I've always been thinking about and catch the students who are giving fake feedback or something that was the motivation originally. And then, as every software developer, you always make the mistake of like, yeah, I guess I can build this in a weekend, right? It's not it

shouldn't be too complicated. So I just started building and then pretty quickly got a very bad version set up and my supervisor found out about and use, like, an entrepreneurial kind of guide. So he was like, Yes, this is amazing. He also liked not grading, right? So like most people do. So he was like, This is great. David, you should sell this. And what do you mean I should sell sell this to whom like I, I building this for myself at my own job, like what doesn't make any sense? Like,

come with me? We'll go to restaurants, restaurants was the department head and he was like, You come and sell it to restaurants and I Go up there, and we'd knock on his door and I still have no real clue what's going on here. And he's like, Oh, David has to sell you something. And he just looks at me and I might cry. Okay, I guess I built this peer feedback software. So I can, students will get feedback, and it'll be learning something and

I don't have to grade. And he's like, oh, so like, what does it cost? It cost $1,000 per course. And then he just looks at me in silence for a long time. Like, I guess if I if you buy it, I don't have to get a teaching assistants. And he's like, Okay, sounds fair. Like it's a deal. And then now I've sold a product that doesn't really exist and that I had no plans of selling. And then that's where it

started. Right? So now I, I had to kind of product make a product on how to build a company I had to like, because I had to send an invoice, right? How do you send the invoice, you have to have a company thing. So I set up a company. And then I called my old high school friend Melda, who's a better programmer than I am, and said, like, I've gotten myself into trouble. You're smart. I have one teaching assistant position left, do you want to be my TA and then just build this with me

instead? And then I'll we'll figure out how to run the course. So that's, that's how we started pa grade. Like, basically, we sold the first license to my boss, and then melden, we joined forces to build the first version while teaching this course on data science. So yeah, that's how I got it to pure grade and educational technology, I guess, kind of by accident. But yeah, too many car non accidental things. That's a fantastic

story. I love I love that idea that before that you were just thrust into selling it, you know, sort of before it existed, that is a perfect crash course in an entrepreneur, entrepreneurial, safe to do, right. They're always like, to pre sell and everything. And I honestly I think if I had not done this, just because I did it and somebody and I was just like, I want to go to company. So somebody told me to sell it before I built it, I would never have been able to pull it off.

But I was never really given an option. Right? It was just like, what was the cost was the first question from the customers. I was like, something and then then we're going right? So we actually sold a second license that way as well. I send an email to University of Copenhagen, and I went there talks to the elearning people and, and I gave them a pitch with like slides. No point, there was still no real product, right? So it's just slides and they're like, go buy 10 licenses. Okay.

Gotta gotta make it work. So it was pretty good start, I think a lucky lucky start as well. Absolutely. Well, you know, sometimes they talk in product about, you know, is your product, vitamins or medicine? Is it optional or necessary. And I think you by pinpointing the pain of having to grade many papers, and many projects that instructors go through, I think you got a lot of leverage from that right off the bat, because instructors feel that pain as you were feeling in your class.

Yeah. And I think it's always been this weird balancing act for us. Because peer feedback, as was, of course, the whole product of peer graded is also a big part of our Nick our new product at flow. And peer feedback has always been this weird thing where people will talk about it. For the learning reasons. There's a lot of learning benefits to giving and

receiving peer feedback. But underlying all of that there's this secret motivation that nobody wants to talk about that like I don't have to great, and that's awesome. So go and give presentations about peer feedback to like 50 people, and they would all talk about learning and pedagogical development. And then after the talk, somebody would come up sneaking to me and say like, I love this, I don't have to create anymore, it's gonna be great. That was often the

motivation, right? But they couldn't say that to their colleagues or their students. So there's always this kind of interesting, double benefit product, I guess. Absolutely. So you mentioned edu flow. Let's, let's talk a little bit about edge of flow. And we'll come back to talk more about this peer grading because it's a great, it's a really interesting

topic. So NGO flow is a modern learning management system has a lot of really interesting features, intuitive authoring, peer grading, as you mentioned, embeddable content SCORM integration. So that's, you know, learning objects, group work, and a lot more. Tell us about how you came up with Agile flow, and how its approach to learning management is different and how you're going to move the LMS field forward with it. Yeah, I think it's always interesting to look back at the story,

right? And then you'll kind of you end up somewhere sometimes by the time of weird happenstance. Right, and then you try to rationalize the path there. So how we got to flow and what flow is it connects a lot, right. So what happened was that we we built peer grade, and that worked well. So for four years, we just worked on peer grade, I think. And we raised some venture capital, we went through Y Combinator, we got customers

and we did well right. And then we're seeing a lot of different challenges with pure grade as a as a company like people love the process. still do. But it was always about like this very niche. We had to create a market for peer feedback software, there was no real market. And honestly, we're better product people than we're salespeople. So I think it was always hard to like, go and

create a new market for us. And then we kept getting feature requests for pure gray to be just a little bit more flexible, like, Oh, I really like it. But could I do this thing differently? Or could I also do that? Or could I disable that. And we had this like, really want to solve all these problems our users have. So we had this long list of things we would love to build on here, great.

But it became pretty tricky, eventually, to extend your grade, just because the technical debt, right, it became kind of a big mess. My old code was still in there somewhere. And I think we got to the point when they were like, Okay, if we knew all of these things from the beginning, we should just start in a different way. So let's go back. Let's build pier grid. 2.0 was the codename for

this thing. And there's this build a flexible fee peer feedback to where we can let people do all the things they did in peer grade. But where they could also do more. So like peer feedback, and resubmissions. And so feedback and group work in a smart way, and so on. So, so we basically started from scratch after four years and said, Let's build a

better peer feedback tool. But then quickly, it kind of turned into a monster of its own and became eventually a learning management system, which was kind of ironic, I think, because if you talk to me, in year, two of pure grade, I think I could have said something like we will never build a learning management system. Or you joked about writing that on the wall or something because it was like, smart, smarter to not do that. But everything turns into another mess eventually, right.

And I think we did too. But then, because of the way we came there, pick you flow has always been a learning management system with a very heavy focus on feedback, collaborative learning, etc, which comes from the peer great days. And then it's always been a product built on a foundation of flexibility. So we didn't feel period was flexible enough. So we kept getting small feature requests that were impossible to cater to. So we build flexibility into the core of effective law as

well. So it's a very flexible product that allows you to build exactly the learning experience you want. Do you want peer feedback, editing, if you don't want it, nobody will ever know that it's a possibility. Netiquette does not add. And then it has a ton of functionality around collaborative learning, active learning, things that, like we believe that online learning should be more than videos and quizzes and SCORM files as well. Right? It should be activity should be doing things with

other people, and so on. So that's a lot of the stuff we've been focusing on over the last couple of years.

nobody will learn anything. So we decide a ton of functionality around like, how do we educate people in a smart way? How do we assign good feedback rubrics that lead people to give good feedback? How do we give feedback on the feedback and so on, and all of those things are in a flow as well. So every LMS out there will say we have peer feedback. But when you get into it, it sucks. Like nothing works. It's not designed well at all. And we really have this like, because we come from

higher education. We have this deep care for pedagogy and like actually making something where people learn something, which it's not immediately visible on the landing page. But once you get even the product, you'll see that there's a lot of details that are cared for more than other places, I think.

right? The teachers and professors just don't have enough time to give really good feedback to that many people individually. So if peer grading can be even nearly as effective as expert feedback, it really allows scaling. So you've mentioned some of the ways that you do that in peer grade where you can get feedback on the feedback or you use rubrics, tell us a little bit more dive in a little deeper, because this is such an important part of the

EdTech. World. Tell us a little bit more about what peer grade does to make sure that peer grading actually ends up with with useful feedback for the students. Yeah,

right? It's like this spider man quote, or whatever, great power, great responsibility. And I think that's what many people do wrong, right, is that they think it's just like a quick, vain day, things like cluck peer feedback on the thing, and there'll be good, but then it just turns worse, actually. So. So there's a ton of things you have to do, right? But the main thing is to get buy in from the learners, right? They have to trust the system they have to trust you and why you're doing

this. And they have to trust and understand why this is even happening, right? Like, does it even make sense for me to, to give and get feedback, I think one of the simplest mistakes to fixes that learners tend to have a misunderstanding that in peer feedback, it's about getting good feedback. That's usually how it is when you get feedback from your instructor or your teacher is about getting some

good feedback from them. And you can learn, but in peer feedback, most of the learning opportunities in giving the feedback to others, but students will generally focus narrowly on like, I don't think I'm going to get good enough feedback in this. So like, I'm not sure I trusted, but it doesn't matter. Like nobody has, they don't even need to read their feedback more or less. It's just in the giving, giving portion. That's

not intuitive. So as an instructor, you have to like submit that and like really build the case for why it's important to give feedback. And also means even if you see if like somebody was reviewing you doesn't know a lot, it doesn't matter too much. And if you review somebody who doesn't know, like, you can still teach themselves. That's a big part of

it. Then, in my own course, we we tried to do different things to make it work, right, because I went in to my class, I told them about this idea on the first day, and I was like, this is three months in right? The product is working, we sold the first license, I go to the class and I tell them about this idea. And they're like, No, thank you. We do not want to do this. And I'm like shit, I just want what are we going to do now? And

well, why not? Like, please tell me and then some brave student raises his hand and say like, I don't I don't trust the feedback from these other people. And I like on the spot brainstorm in my head, and I said, Okay, what if I build a feature where there's a flag, you can click, and then if there's some feedback you get, and you don't like it, you click the flag, and then I'll review that, that your work, and then you'll get my

feedback instead. So like, if you really hate this process, just click the flag everywhere. And I'll just grade you. And they're like, Okay, I buy this, like, it's, it's a safety valve for me in the system. And so we built this flagging functionality in here, great, which turned out pretty important for people. In the end, not that many people flex things, right, I had to grade 10% of the work, maybe. But it ended up being extremely fair. And he will have suddenly got

this trust in the system. The other thing we did was that we implemented this thing called feedback reflections, where when you got feedback, you had to give feedback to the feedback. So basically, I would get feedback from you. And then I had to give feedback to that feedback using another rubric of like, measuring how how good your feedback was. And this allowed me to see which students were getting good feedback,

generally. And who would just skip and write great, great, great through the whole review. And then I told them, 30% of your final grade is going to be your feedback quality. And that mattered a lot, right? So and I told them, like, I have smile algorithms to find you and like I did not, but like they didn't know, right, I just like a four course, person writes like they couldn't really know what I was doing. So I just they just trusted that I had to support my algorithm, and then they all

behave very nicely. So those were some of the things that worked extremely well. And then rubric development over time I made beta and beta rubrics to to help guide them through the process.

You know, in my olden days when I was at Coursera, the the most reliable complaint about the learning experience at Coursera was almost universal was the variable quality of the peer review system because the peer review system there was not fully featured, it was pretty basic. And you know, just like you said, students would spend lots of time and effort on their projects and receive really low level sort of careless responses, like you said, like, just great, or Nice work, great

job. And they'd be like, Why did I spend all this time if all I'm going to get back is a great job or Nice work. Some there also, sometimes language barriers if you had people who, you know, couldn't communicate that clearly. So I love everything you're saying about peer grade, the use of rubrics is fantastic that feedback on the feedback? What do you think peer grading is going to look like in the future? How is it going to continue to evolve? And just as a side note, you're a machine

learning expert. Do you see the there being a role for for NLP or machine learning in that future? Or is it really just about good pedagogy or both?

And they always see like, Oh, this is machine learning guy, he's got to make some cool AI with his pure grade. And they asked me like, what, where do you use AI? And like, nowhere, we don't we use none of it. And they're always like, very positive by like, why wouldn't

you use it? And I think it's, it's that the obvious way, some of these people think is like, okay, so, but David, you have this great data set, right, you have all these submissions, and you have all this feedback to these submissions, and you even have quality of the feedback. So couldn't you train a machine to give the feedback, and there'll

be even smarter? And then, um, yeah, I guess I could, but then if the learning isn't giving the feedback, I just removed all the learning, like, that's not very smart. That's kind of takes you out of business and the students out of the learning so So that sucks. What we have been working on like it not so much recently, but generally over the years is

a different approach. Right. So if you remember, I talked about this kind of algorithm I had in my mind back then of like, finding these fraudulent students. So we implemented this, me and some thesis students I had, and it would automatically adjust all the feedback and grades to be more fair. So basically, like it would take all their peer reviews, and then it will balance them out and say, Okay, this person's pretty high. So I'm gonna change that a little

bit. So they've been nicer. And then we would give the students all these grades. And they would all say, I don't understand why I'm getting a 3.8 here, because this guy has given me a 3.2. And they just didn't get it, right. It's just, it might be correct, but they didn't really understand what was going on.

And I started kind of thinking more about this, and kind of coming to the conclusion that it's kind of like, people very concerned about like, machine learning, playing a part of their grading, they didn't really trust the machine here, kind of like with self driving cars, right? It's, they might be a bit on average, but it's not good enough. If you get hit by a self driving car, you're gonna be really mad, even if it's bad on average, right? So this had

to be perfect, which was hot. So what we ended up doing instead for pure grade, and maybe we'll do the same for at flow was building kind of a a human computer combination thing where we could get a system to find all the feedback that we thought was wrong, and recommend what we think it should have been. And then we could prompt the instructor to say, okay, David is giving Alex a four out of five here, we think it should be a five out of five. Do you

agree? And then if the instructor said yes, then from the students perspective, it was the instructor who corrected the grade and not the machine. But we basically made it very easy for the instructor to like accept these machine learning, like collected motivations. And I think that was a kind of a nice, middle ground where the instructor got a motivation interface, essentially, we call it fleck bot back then it was a bot that automatic reflect

things. And then the students would just see it as like a very smart instructor who was doing a lot of work. I think that has a lot of potential, where just looking at a human in the loop, you just make their learning or their life a lot easier.

right? That say this a score one to five, and you get three peer reviews, and two of them give you one because it's a one, and then one of the people get your five, you're not gonna flick that you're gonna see if that slides, right, you're like, Okay, this guy really likes my bad work, but still like the flat but say, hey, wait a minute, everybody's saying you're bad, except this one guy, it's like, you're probably bad.

Versus if you get if you're really a five, and one person gives you a one, you're going to flick that yourself, right? Always. So that we don't need to get to those we just need to catch like the positive outliers. For example, we did some more advanced stuff of like actually reading the feedback. And then checking what how good we think this feedback is versus what the feedback on the feedback set. But that was a bit easier, because feedback in general is more similar than

sufficient. So patience can be very different, right? But we could see if you're using words, such as figure, page shot, and so on, that means you're pointing to something specific, if you're saying things like could have should have might have whatever, then you're suggesting something and that, feed that into machine learning classifier, and then we could give a pretty good estimate of how good we think this feedback is, and then correlate that with a, okay, this looks like good

feedback. This guy says it sucks. Like, that's probably wrong. That's your to the instructor.

And, you know, I think that's a really thoughtful approach, you have such a sophisticated machine learning background, and people like the venture capitalists, you named have been looking for a long time to sort of understand what role machine learning will play in the tech ecosystem. I think your take is really unique and interesting.

And I'd love to hear you just zoom out and think, a little bit even more broadly for our listeners about, you know, how machine learning can play that type of role that you're mentioning, in other edtech applications. And it's a big question, but sort of how can entrepreneurs and innovators and VCs think about machine learning in a way that it isn't just a replacement for learning, as you mentioned, but it's really a supplement?

most people don't. And which is this old paper where they look at students in a class and then like, what happens if you give them some cool master learning tools and so on, and then what happens if you give them each an individual tutor, and when you give people an individual tutor, they become extremely good. Like, they'll all become like,

top 1%? Basically, if you give people individual tools, but you can't afford that, right, so and then they post this question a long time ago, like how do we automate the individual tutor? How do we like how we build technology to be that right? So like, I guess, on a high level, the easy answer is robot tutors, which sounds kind of lame. And, but but it is, in theory, pretty smart, right? If you could actually build a personal automated tutor for everybody that would bring a lot of value,

but it's very hard. And I see a ton of entrepreneurs getting blindsided by the complexity of this right, they started like, oh, we'll just like build a tutor, like, but like, I when I asked my Okay, Google to play music, it still sucks, right? So like, you're not going to be able to build a tutor in like two years, even if you get a ton of venture capital, because not even Google can, like play my Spotify playlists. So not yet. Not yet, my friend, like you have to start somewhere else

now. And then maybe we'll do this in five years. But I think this human computer interaction thing is kind of interesting, because by using the human in the loop, you can kind of do 80%, which is easy. It's always easy to do 80% In machine learning, it seems it's almost impossible to do 100%. But 80% is enough because the human will

then add the last 20. So I think there's a lot of potential around this where you can and I think people are looking at this right where you can give people an 80% tutor, and then whenever there's a problem that the computer is like, Okay, I'm not sure I can solve this, send it off to a human in the loop who can then go in and help for example. So I think that's a big potential, but I also think that most products probably don't need machine gun. A lot of people also like just do ml for

the sake of ml. They're like how can we Add some cool AI to a product. But they'll probably have more benefit. If they just spend that time on the onboarding process in their product and make that better, they'll probably make the whole business better instead of like trying to add AI to everything, especially in the LMS space where we're at, right? Every learning management system wants to be aI powered, is all an NS, right? Nobody can it doesn't matter. Like no LMS needs AI.

Honestly, it's all fake. And they're all trying to compete on this. And then we'll talk to customers. And they'll be like, Oh, but this other one has AI in it. Like, what do you even mean? What does it mean that it has AI in it, you don't need a I in your LMS, you just need, like, just needs to work, these good user experience, you need to upload your SCORM file. So you don't need an AI to go in there.

But it's very hard, right? So it's kind of frustrating sometimes to be be knowledgeable on machine learning, because you realize how much of a scam everybody's pulling it off to be.

just pick the top one. Honestly, it's going to be like 90% as good as your deep learning model and it will be like implemented in a weekend. And that's that's all it is right then they they they branded sai but they just count on sold almost holy ISIS counting and Sony but nobody realizes that

countries. And you know, pure greed and as you flow are based in Copenhagen, as are you as you flows on Holan Ickes list of the Nordic and Baltic 50, which are sort of the most promising countries in that region along companies like lobster and Kubo, which are also from Denmark, sauna labs, Cognetti, which are Swedish scrim GBA, which is Norwegian. We've seen a huge

increase in edtech. interest and investment in Europe in general, a report just this week, you know, said it tripled the investment in Europe VC investment in the last year. And, you know, we've also seen that, particularly in the Nordic countries, and with the one big, big, big, big breakout company being Norwegian Kahoot, which is, you know, claims to have reached 1.9 billion users, it's a little bit hard to swallow, but maybe. So tell us a little bit about from your perspective

being incited for years. What does the EdTech landscape look like in Denmark, in the Nordic countries, you can even talk about Europe. Tell us about your perspective on that growth?

Honestly, I remember meeting mess when he was like, just starting Napster earlier. And he was were at the same university back then. And he was like, trying to recruit me. It's like one of the first two or three people that I left back then that's important role. And I was like, this will never work. This is the worst idea I've ever heard. You're going to build it in, in flesh. And boy, was I wrong. So happy for footmaster have succeeded there. Despite my

bad, bad feedback back then. I think when people look at at the Nordics, they generally look at at the Nordics as kind of like a place where we teach differently, right? It's a lot more feedback, and it's a lot more leveled. And I remember so I don't know if this meaningful, right? But I remember going to the UK to try to sell here Great. And this UK is even in Europe, but it's not in the Nordics. And whenever I got to the UK they didn't understand

it. Because in the UK university systems people are in hierarchies and like the professor is correct and the students are wrong. So having feedback from the students make no sense. It was kind of weird for them. And then what they really wanted was students to greet each other as group members. So they could see who were the best group members in the group, which is kind of weird for me. It's like, why do you want you don't want that, that sounds like a Hunger Games

kind of thing going on. So but there was their model and the same in France and so on. Right. So even inside Europe, you'll have very different schools of education. I think in the in the Nordic countries in Denmark, and Sweden, and Norway, and so on, it's, the students have a lot of power. So you'll see these kinds of initiatives like like peer grade, I think, makes sense

here. But honestly, most of these, such as their random accidents like mine, and then they were started over here, I think a lot of its tickets in the US, right. And the US has a very large school system that works in peculiar ways. So a lot of it sick in the US. So us specific, because the school system is big enough, it's not big enough here, right? We cannot do anything for the Dainius system, because it's not big enough, we have to like break out of that Mark

immediately, basically. So I think one of the things over here is that we build internationally from the beginning, most of us at least, we of course, we also have like local players, but but most of us learn from beginning to start thinking about the US from day one, right? Which is kind of interesting, as a company,

effective in the world. And people have have, just, as you say, thought about it as a system that's much more freedom oriented with a lot of equity built in less hierarchical than some of the other stars like South Korea, and, and Hong Kong. And there's something very interesting about the idea of an education system that values feedback, and collaboration and freedom. That also works

incredibly well. And I have one more question just about this topic, which is, you know, when we look at ed tech around the world, certain cultural features sometimes seem to influence the landscape of what companies rise that you mentioned, a great example before, which is, the US has this really unusual, federated, you know, strange school system. So a lot of companies have our birth

specifically to adapt to it. You also see a good amount of entertainment influenced at tech companies like Duolingo, which is totally game based or masterclass, which uses celebrities. And you know, we've talked on this show about how in India, which the tech scene is enormous, but a lot of the biggest companies focus on test prep, because that's such a big part of the education landscape there. And it's something that

families spend money on. So you've already touched on this, but I'd love you to, you know, double click on what do you think are some of the sort of cultural features you mentioned, there are some happy accidents, but are there some cultural features of the Nordic countries that you think influence how edtech has developed there?

students, right. So there was just not a big thing in Denmark, like, because the trust in the Danish society, for example, was very high in government and politicians and so on. So I think trust is very large in the Nordics. And I also think there's, I think that kind of plays into this seriousness as well, right? We trust researchers, we trust teachers, we trust the students, which means a lot of this, you can go

from deep learning, right? And I think that's kind of a trend we not connect with deep learning. But like this trend of like, Can we do some learning things that are deeper? And we have, like counterpoint to this, like Kahoot? Kahoot is great, right? It's very entertaining. But it's very entertaining. And maybe it's too entertaining sometimes, right? It's, it's it has the same function as peer paid, where it has this double thing where it's fun for the students,

then they like it. So it's easy for the teacher to like, pull a Kahoot out of the head and make everybody happy. But was it really the best way to spend 20 minutes to like, have a lot of fun with Kahoot? Most of the time? Yeah, it's probably a good idea. But not all the time. Right. So I think there's more room for experimentation and trusting the students to do complicated things. And I think

that that helps a lot. Right. i One of the things I did in my own course that I kind of liked was that one of the semesters I taught the course I had nothing planned. For the last two weeks, and I couldn't come up with a topic, and I was kind of panicking a little bit. So what I did instead is that I said, Okay, imagine you had to teach

this course. And you had two weeks more for curriculum, what would you teach, go and teach that make a video of a lecture that you give, put it on YouTube, and submit the link in geogrid watched three other people's videos and give them feedback. Here we go, I didn't have to do any work. And like people loved it, right, they went and did some incredible tutorials on TensorFlow and different machine learning theories, and so on. And some of these videos got like hundreds

of 1000s of views. And honestly, they learned a lot more from this than from watching me drone on for an hour about some MapReduce function or something like so easy for me, love learning for them. But it only works if you trust that your students are smart. And these people are smart, right? They're much smarter than me. So maybe individual students wouldn't be more knowledgeable about it than me. But as a as a combination,

right? I had 150 smart people in the room, they could go in everything, and they could teach each other and so on. So I think this trust in the students and their competencies is kind of hypnotic, I guess. That is

in education. And that lack of ego can go a really long way in sort of getting to the right answer instead of instead of just the loudest voice in the room. So I really appreciate that. It's

be bad, and it can be good. But this is also a place where it belongs right? That I'm a teacher, but I'm not that smart. Right? Like the students also, they are smarter than me. And often they are right. So it's, it's good to give them the space to be smart as well. 100%

learning, and so on. Most implementations, so far, there's a few startups in this space who are really like loud. And honestly, it's just glorified very expensive zoom calls, right? So every week, we'll go and assume with 100 other people, and they'll call it a cohort based course. But it's just 10 webinars, right that you pay $1,000 for. And that's not I'm not involved with that

model. But but the idea of taking self paced courses, for example, and then community find them or adding collaboration elements to it. I think that's kind of interesting. And I think we'll see more demand from, from my coach consumers to, to get more than a list of videos, we're running our own record base calls right now. And it's a lot of work. But it's also very interesting to see all the dynamics that come out of having people do things together in

different ways. So I think that's a trend that has a lot more potential than then it's your show. And so far.

doing it on your own. I still think it's a little bit elusive. So I think that it's a great trend to keep our eye on and will those CBC providers start to break through and figure out you know, what is so special about being in the room at the same time? With a lot of other peers and learners, we're trying

great, right? So he's like he's been working in it take from an academic perspective, I think he's like, he has this pet peeve that everybody thinks they're reinventing, or like building new things all the time. But he's like, it's all been done before guys, like it's always been there. So he goes back 25 years. And then he takes every year he's like, this is the year of the Does everybody remember the year of the look right? And

everything. And he goes back all the way and said, Okay, 25 years ago, we started using discussion forums in our university, it's not new. We've been doing online learning for 25 years, at least,

right. And it's like, he goes through the history with like, a thematic story per year, talking about the first time an LMS becomes important and discussing goals and moves and, and like why it happened at different times, which is, and one of the interesting things he hasn't had is that we all kind of have it backwards, like we think of like, okay, we had video courses, selfies, video calls, and now we're adding all this

collaboration. And it was actually the other way around, you had all these discussion forums, and everything back in the days before video could be online, because you couldn't run like massive video at scale is streaming back then. So you could only do text. And then as soon as videos become possible, you get things like Coursera, and so on becoming really big. And then now we're moving away, again, kind of backwards in time to discussion forums and

communities and so on. So it's kind of it's just a humbling experience of like, shit, these people thought about my good idea. 25 years ago, I'm not that smart. And it kind of grounds here the debate in the in the tech world, like, people have been researching this for a long time. So I think it was a fairly quick read and kind of interesting to see historical perspective on edtech, which is always about the future.

enjoyable conversation. I've learned a huge amount about both of your companies and about the tech scene in general, and about machine learning and how it doesn't really work. Great. I'm kidding. I really, really appreciate your time. And thank you for being on Ed Tech insiders.