Game-Based Learning and the Future of Education: Insights from Rebecca Kantar of Roblox

educators and students. Roblox education connects educational organizations with extraordinary Roblox developer studios, through Roblox community fund grants. Rebecca came to Roblox in 2020 when Roblox acquired Imbellus Inc, a simulation based assessment technology company

Rebecca founded and led. Rebecca founded Imbellus with conviction that better educational assessments would drive deeper teaching and learning and in turn, more students graduating ready for adulthood in delivering operational fair, valid and reliable assessments at scale, Imbellus demonstrated the potential for simulation based assessments to measure deep thinking skills in high stakes, workforce and

educational contexts. Rebecca led Imbellus in raising over $23 million, signing multimillion dollar development contracts with Fortune 500 corporations, and in completing successful m&a transactions with Roblox Corporation and McKinsey and

Company. Prior to launching Imbelsus, Rebecca founded an expert network that Gerson Lehrman Group (GLG), acquired in 2012, Rebecca serves as the vice chair of the Museum of Science Board of Trustees, where she serves on the campaign cabinet and as vice chair of the trustee Nominating Committee. She has also served as an advisor to the common app, and to the Massachusetts common Start

Coalition. Rebecca grew up in Newton, Massachusetts, and currently lives in Massachusetts, Rebecca Kantar, welcome to Edtech Insiders.

through that work. And like the folks I worked with every day day in and day out on these nonprofit projects, to really apply whatever it is we were learning to more contextual and real world settings or

challenges. So at that time, I'd say became really interested and how might we imagine an education system that more deeply focused on some of these core skills that we would all need as adults, as opposed to just kind of surface level mastery of specific domains of knowledge, which certainly matter, but matter because we go on as adults to apply them to hopefully to solve problems and make the world slightly better.

So as I developed an interest in a lot of the young people who I was exposed to through my nonprofit work, I became interested in what their schooling experiences were like, and I left college thinking that I would maybe someday go back maybe not ended up not being the right thing for me to ever

return. But during my time, starting to explore what were journeys and my peers who had kind of similar paths during high school and college, or maybe they were working in the adult world, while being in school, maybe they were confronting different social challenges. I became interested in the rules or incentives that either guided their educations to be more traditional or less traditional. And the number one kind of rule system or incentive system I'd say that fascinating.

enemy and seem to have a major shaping force on all of our trajectories was testing. And in particular assessment in the high stakes context of a transition between high school and college, which is not a transition everyone makes, nor should it be necessarily. But it is a transition that the entire system revolves around kind of sorting out a very fortunate elite few who get to go to some of our nation's you know, top

300 colleges. And that admissions process ends up shaping a lot of what becomes before in terms of how our default settings of public schools, private schools, etc, are all kind of merging and how they're all coordinating what kids are learning and more importantly, how students are

learning. So I became interested in games and simulations through some outstanding research that EA Pearson, ETS, and GlassLab, a fantastic organization that kind of was a consortium of those groups came together and began on researching the Sims and how the Sims could be adapted for seventh grade, I believe it was seventh grade, don't quote me on that one. But systems thinking for middle school students in California was kind of the subject of their adaptation of

The Sims game. And I saw just such promise and the nature of the types of contexts they were developing for students to both learn and demonstrate learning within. So I became curious about how we could adapt game based assessment or simulation based assessment that had often been used in formative contexts or maybe in the workplace, how we might think about those same kinds of assessments for high

stakes context. So contexts like college admissions contexts, like state summative assessment, those end of year tests that are required by federal law in grades three through 10, and then also in to college admissions assessment, and how we might better sort on the kinds of skills that all kids need for adulthood as opposed to just figuring out did they retain some bits of information that certainly matter, but matter in the context of application that folks go on to

do later in life. So that was my background. I was so fortunate to meet Roblox via that work in 2020, and the height of COVID, when all states froze all standardized testing, and I was just starting to bid on state

standardized tests. I was fortunate that the corporate side of umbrellas the company I started to work on these kinds of high stakes game based and simulation based assessments had been thriving and had piqued the interest of numerous employers, Roblox among them, and Roblox then decided that what we'd created and what they'd seen out in the wild was so compelling that it would be pretty interesting to have a assessment on Roblox for Roblox product managers and engineering

candidates to demonstrate how they think and that's very much what our team, my assessment team with my colleague, Jack Buckley and I have focused on shepherding into being at

Roblox. I was also fortunate, as you mentioned, to come in to a place that had such a rich history of working in the education, informal education market, I should say, with some bleed over into the formal education market, and to come in here to a very open field for how Roblox might provide outstanding learning opportunities across subject areas and across grade levels for students the world over. So that's my journey to date.

pandemic, which is crazy. But that just makes it even more important to think about where the equity is going to come from a post, you know, the Supreme Court, primitive action world. Fascinating story, and it's really interesting to hear that the game based simulation based authentic assessment was resonating in the corporate training and with the Roblox

group. So you just mentioned a little bit about Roblox is incredible reach but Roblox, as you say, sometimes it's sort of considered an ed tech sometimes not. I've seen it described different ways. But no matter what it is, it is a powerhouse of a platform. It's everywhere it has reached all around the

world. Tell us a little bit about Roblox is reached for those of us who may not know the extent of it in this particular era, and how you're thinking about it as an educational platform to foster creativity and learning among its users.

or unavailable to us. A lot of folks found solace and connection during the pandemic, for example, and it was unavailable for some of us to meet in person. So this world which is created by developers of all stripes, who imagine what would be interesting, what would be fun, what would be meaningful, what could help folks learn and then use Roblox studio, which is the crafting section of the platform to realize those imagined ideas.

And the beauty of Roblox is that anyone can jump in and be a creator, they could work on digital fashion and helping folks bring their identity to life on the platform, they could work on the gorgeous 3d and immersive art that's present across these experiences. Or maybe they're working on the game design side of things, thinking about how to build a compelling and rich narrative

that's multi layered. And that really brings users in through a story and transport them through it experience that might travel through space or time or history. So that's what was so exciting to me. When I came to Roblox and learn more about Dave's vision, his history and his roots are CEOs history, and his roots are very much in education. His first company was focused on educational software

and technology. So I knew that we had a strong history and connection to thinking about this outstanding canvas and community that Roblox had already fostered and developed. Being a bigger part of both formal and informal learning contexts and allowing for students to translate the enthusiasm they feel already playing Roblox experiences with their friends, visiting a virtual museum or experience on Roblox with their parents or

grandparents. Finding new ways to gain skills be those skills and problem solving or language learning on the platform. I knew we had such a exciting future to capitalize on that enthusiasm and redirect it as students are going through curriculum during the school day, for example, that lends itself well to 3d digital immersive environments

with multiplayer components. And to think about how we might not kill the fun, as many students cautioned us not to do as we try and translate that enthusiasm to that learning setting and maybe upgrade some of the materials through which students are engaging with those underlying deep thinking skills and the surface level whatever content or domain they're kind of acquiring those skills by way of, of learning. So that's what

was so compelling to me. ROBLOX is scale is an unprecedented opportunity to bring access to outstanding curricular materials should we have them on the platform, which now we do to folks the world over, we have 66 million daily active users 1.4 billion hours of play time in the last year. I mean, that's just outstanding engagement and reach. So when I thought about our education strategy, globally, I saw an opportunity

for us to do three things. The first was to continue fostering outstanding computer science instruction and game design game development instruction around Roblox studio. So we've made a number of investments. And we can talk about our partnerships

more in a moment. But we made a number of investments in curricula providers who work during the school day, say through AP Computer Science Principles type curriculum, as well as outside of the school day through camps, boot camps, educator training materials to make sure that learning on Roblox studio has no high barrier to entry, and is imminently available both in school and out of school workstream two, and our strategy is all about offering a compelling content library of

outstanding educational experiences that are standards aligned, that could be international standards that could be national standards that are kind of bottoms up emergent like next generation science standards, or that could be top down state standards. We're flexible on depending on an organization's context. And depending on their vision for what they want to build on the platform are flexible, and kind of what sets of standards we would see as most appropriate.

But principle for us here is we want to make sure that whatever is developed on Roblox doesn't die on the vine that really makes it into classrooms, educators feel like they can slot it in, they know exactly how it fits in with their piecing in their schedule, and therefore it becomes a rich and reusable resource. So that's the aim of content stream to for our

education strategy overall. And then the final one is were well aware that Roblox as it stands now as a direct to consumer platform is not always that school friendly. So we're constantly working in thinking on how do we improve the school friendliness and reduce friction for robots to be present during the school day in a way that's obviously appropriate and respectful. and fully compliant. So that's what we've been endeavoring on and more to come

on that from us. But those are the kind of big three points of our strategy and what I helped shape once arriving here.

incredibly interesting. And it's really about content and school friendliness, just as you say, this is an extension of what Roblox has been doing. It's taking the informal learning from Roblox and allowing it to work in formal settings. So tell us about what that looks like for you.

on Roblox. But what's important and the reason that we harp so much on students learning to code with Roblox is first and foremost, Roblox is a place where for young developers, they see a closed feedback loop with

a reward. That's not like some arbitrary points to spend at arcade or some, you know, parent constructed point system in the home, which is fantastic, but not always the same level of reward and a feedback loop that comes when you build something, share it with your friends, and instantly you're in there in the space, you create it. That's a pretty positive feedback loop.

So when I think about Roblox studio as a tool for students to master some early computer science skills, some game design and most importantly, some computational thinking, I'm thinking about that set of transferable skills and the likelihood that that becomes an appealing set of skills to leverage over and over again, as I also think about where students are going, particularly with generative AI. And I know you wanted to chat about this a

little bit today anyway. But was it for an example that all of our technology over the last 100 years has been moving humans to like higher and higher orders of thinking, where we're doing a lot of the strategy work asking the right questions, considering what to invent understanding what needs there are, and then we're relying ever more so of on technology to help us do the laborious parts of executing on

the ideas we come up with. And I think generative AI is just the logical epitome of so much of that class for us, certainly since like the 1990s, where

computation has just boomed. So when I think about what students are going to be doing in the future, it's even more important to focus on these deeper thinking skills, these higher order reasoning skills, and that's where I come to something like computational thinking, and being able to understand and architect how a system works, how components work, to anticipate orders a consequence and design for those. That's what I want students to take away from learning to interact

with Roblox studio. More than just learning to interact with Roblox studio. There's a level of abstraction above that. So I see that as vital for our

schools to adopt. And I think that speaks to my broader views on what generative AI should be doing for us all, as Roblox incorporates it as others incorporated in schools, we want to up the ante on what students are able to and expected to achieve, to incorporate the deeper thinking that they're going to need forever to work alongside these tools successfully, as opposed to trying to shut those tools out, right, and kind of keep students in an era where they're protected from some of the

higher demanding type skills that they're really going to have to leverage as adults. So anyway, that's an aside, Roblox overall, in our education strategy, recognize that one of the challenging aspects of bringing educational games to life is that you often have studios who are either pretty good at games and decide to work on something that's educational, or you have teams that are pretty good at curriculum and instructional design, but maybe

have less gaming experience. So what we wanted to do was actually merge the best of both worlds by pairing outstanding developers who had experience on Roblox and other gaming platforms, with outstanding educational providers who have a pretty good sense of what's required for use and widespread

adoption in schools. So our Roblox Community Fund, which maybe you've seen a little bit about online, we're now launching our second fund coming up, but our roebucks Community Fund has existed to helped match those types of organizations. For example, Project Lead the Way with tipping point media rate outstanding curricula developer with an outstanding development studio, filament games and Robo co sports league

with FIRST Robotics. So thinking about Robo co sports league as the epitome of digital social, robotics, education administered at scale, and remotely outstanding, that's a team who we helped to fund and make sure that they could bring an ambition they've long had to life. And then another example would be the Museum of Science and filament games working on mission Mars a chance for students to practice engineer design thinking to get iterative

feedback. And they're really experiment and learn about how engineers would design rovers for Mars in that case. So these kinds of partnerships are the bread and butter of workstream. Two for us, which is always about bolstering this compelling content library about standing standards aligned digital experiences on the platform. Hopefully, that gives you a little bit of a sense of how we've been approaching that,

lived in that one. So the idea of working with these top notch studios and you know, filament games was how we connected in the first place because they're doing really fascinating. You mentioned two projects you're doing with them, work with you. It makes a lot of sense as a strategy to actually allow the Roblox world to sort of meet with the standards based world that schools and need to actually teach curricula.

listeners might notice that as you talk about these co experiences and this immersive world, Roblox actually has a pretty viable case to be made that it is already the main Metaverse for children and teens. The demographic usually is in the young teens, but it's been expanding very quickly. The fastest growing demographic on Roblox is 17 to 24 year olds, and all of these have exploded since the pandemic especially.

So there's a case to be made that you know, in terms of what does an educational Metaverse look like? ROBLOX might be it already? And how do you see that? And how do you see the relationship between immersive worlds that are populated with user generated content and characters and experiences like Roblox and the concept that we've been hearing about for the last year or two about a metaverse.

touch on filament games. And FIRST Robotics because I love the nature and ethos of that partnership, particularly what FIRST Robotics brings to the equation as the values they hope to instill in all of their participants. Those are values like coopetition values, like graceful professionalism, these tenants of what it means to work in a team and to compete with other teams. Whereas first once told me, you want to beat folks

at their best, right? So your goal as a team is to lift up all these other teams when you're at the World Championship, and to make sure that everyone's top notch in their form, and they're putting their best foot forward.

I think Roblox is developer community and user community benefits from this goal and ambition to connect folks with optimism and civility and these co experiences, because it means that all of our developers and the experiences they build are striving to incentivize that type of behavior on the platform. So in that way, thinking about robotics is a place for digital co experience. And seeing so many of those experience imbue this ethos of civility and optimism and positivity and the nature of

interactions on the platform. I think there's a great case to be made for our viability as one of the major if not the major player to bring this kind of immersive and dynamic learning to schools and students outside of school. That said, I always like us to be humble about the role we can play in an education system that is diverse, that has constant challenges, that has really taxed people and resources depending on where you

are. And that has an uneven terrain that it's trying to overcome every day depending on the school depending On the students, depending on the educators, I have a lot of empathy, for how difficult it is for everyone involved in the education system. Pretty much no matter where you are, right, there are different challenges. But it's just a hard thing to take kids through 12 years of learning and across so many subject areas and to have that work for each kid who has

different needs. So I like to think about us as being there to provide remote instruction for anyone who can't access it, because they were born in the wrong zip code in the world, or a zip code that didn't provide opportunity, I should say, for outstanding curricular

materials. And also to think about a district who has everything but would still benefit from having an outstanding experience like the one Project Lead the Way offers were students to learn about the immune system by being inside the immune system navigating a viral or bacterial infection as cells in our bodies actually do. So I think there's a role for us to play in all sorts of

contexts. And certainly the Headstart Robotics has and understanding how to bring not just huge numbers of people together, but huge numbers of people together in ways that are civil and respectful, and that bring out the best and folks, that's the mission we need to continue to pursue. And the one I'm pretty excited about for us as we work our way into the education market.

other in the same experience. It's like it is ever more than half of the kids under 17 in the US are using Roblox probably a lot more than that now. So it's everywhere. What's so interesting about your opportunity is that you're taking something that is incredibly popular, almost ubiquitous among young people right now, and instilling some of these professionally made games and experiences but that build on the ethos and the culture of Roblox that's already

been created. And I'm sure there's some interesting tensions there, but also a huge amount of opportunity to really learn from what your users have already shown that they love. So, you know, you mentioned you're in the second round of the community fund. And you've mentioned a couple of these projects. But can you elaborate a little more on you know, when you're bringing these professional game developers and studios into the Roblox universe? How do you make sure you're doing something users

love? And that doesn't feel like chocolate covered buckling?

testing and shipping. And that's the loop or the relationship that I insist on all of the teams we fund getting right from the beginning. So what we look for, is an outstanding coupling of exactly what it is students are supposed to be learning, we'll take the Museum of Science example, who I mentioned previously, there, we're focused on learning the engineering design process, it's aligned to middle school NGSS standards

around engineering design. So what we want the core game loop to be is not have fun search for objects, and then use those objects to do a round of engineering. That's tutorial alized. And very prescriptive. What we want the core game loop to be is engineering design. What I'm looking for is to see rich coupling of these systems, the core game design systems and the sets of learning objectives, such that the deeper students go on understanding the system in a game, the more they're learning

and vice versa. And the richer the possibility space of how they're thinking what they're required to demonstrate becomes the deeper they get in that game experience. So that requires, from Roblox anyway, real alignment with the partner teams who were making grants to through the roebucks Community Fund. Because it's brutally

hard. Yeah, it's just intellectually difficult to anticipate and map out those tightly coupled types of learning objective ladders, where you're leveling up to perhaps higher and higher orders of thinking as you're working through familiarity with a domain or a new subject, and the corresponding game design loop that needs to be rich and interesting and fun, but also fun because of how it you uses those learning standards or

those learning objectives. So we focus so hard core on that at the beginning of all partnerships that we are unrelenting in further testing and further phases, let's say the alpha or the beta phase of a partner bringing an experience to life. Were unrelenting when they're kind of early signs or symptoms that maybe that core

loop isn't quite working. And I think a lot of what folks like us can do to level up the entire educational game design community is just figure out how do you develop a good sense of when it's working. And when it's

not? Well, one thing that's powerful about Roblox is you can test with a whole bunch of players, right, you can release something that's half baked in a beta, and tell the community that you're looking for feedback on it, and ask for volunteers, you know, in a dev forum or community Chat, where you can find young people who are ready and willing and able to hop in and test out your super gray box

prototype you build. And that's important, because it means that we don't end up with experiences that look really nicely done, but in fact, lacks some of that rigor and some of that meat that we're hoping for in all of our experiences and actively trying

to shape. So that the kind of a high level of what I'm always emphasizing in our partnerships, past that Robox encourages this transposing of what students expect in at home play or unsupervised play when they're just enjoying the platform as they are versus a formal learning context. One of the aspects that we have to transpose carefully is the Avatar. The Avatar is so key to students developing a bond with the platform, feeling like it

helps them build confidence. I mean, we've talked to a lot of young people about this and the avatar, what they're wearing each day is very important. The nature of who they're interacting with, in which experience might dictate changes to the avatar that they want to make that avatar agency has to be with them in class, or in any

educational experience. But so too, does being an A class experience require for all students to abide by some norms that are maybe different than when we're playing at home and just having fun, right? So we might want to think about maybe a little bit more serious dress, maybe we don't want to have fantastical creatures that are a quarter donkey, a quarter Penguin, and two quarters mermaid when we're there during the school day, right, maybe

that's not the right context. So we want to think about transposing an expectation like complete agency over an avatar, to a place where we don't ruin it and make it unfun but to a place where it's manageable for a group of 30 students in an educator together participate in a class period where you know, you don't have people distracted by certain types of avatars during the class and kind of detracting from the educational

experience at beat. So that's a snapshot insight into the types of considerations we make when we think about how to preserve what's engaging and exciting about the platform that game design has to be there to maintain that interest and intrigue. We often look for multiplayer to add to that engagement, because that's part of what students seek on the platform naturally, as well. And then we think about what's the context, all of this experience in school, at least may take

place in? And how do we make sure for each of these experiences, that context is well managed, and minded as well.

action. One of the things I appreciate and sort of notice as a through line through your career in edtech is really this focus on realism, authenticity, you know, getting as close as possible to the actual goal rather than sort of hoping to bounce off something that's on the path. So you know, in Bellis was simulation based assessment, all about trying to create educational assessments that were actually assessing the actual thing rather than sort of

proxy symbols. And when you say something like, oh, when we're creating these incredible experiences in Roblox, we want the loop to be engineering design, not something that sort of says that, like, you know, evokes engineering design or it's not just a game mechanic like World of Warcraft, you go find 10 skins and trade them in for this. Like, it's so clear that this has been a guiding principle for you throughout your career. And I'd love to

hear you talk more about it. So you told us a little bit of the actual history behind In Dallas, but tell us about why authentic and simulation based assessments are so different than what traditional testing does. And in the context of workplace hiring of transitions, what advantages do you get if you're actually assessing people on something very, very close to what you want to know, versus something that sort of potentially hints at it?

learning and assessment. But in assessment and in teaching, we can't directly impute something into your mind and say, there we go, we installed algebra one, we also can't directly slice open a brain and say, well, there was the critical thinking I saw it right there at that point. And so where that leaves us is in a constant state of striving for the highest fidelity proxies to use your word that we can possibly get, and how do we know when a proxy is trustworthy or

not? That's the central question that makes assessment and in particular game based assessment challenging, because what I'm trying to do is first, if we zoom out right to the highest level, I'm trying to figure out, what does assessment do in the

system. And as I mentioned earlier, I've focused in on assessment because I see it driving a lot of the opportunity that we have, and the opportunities we shape by way of how we structure public school in particular, and what we do in each grade within public school. So assessment is this important lever when we go in next level down? The reason assessment is so important is because of what's on those tests, what are

we evaluating? When we choose to evaluate something and we don't choose to evaluate something else, we make an implicit judgment that one thing matters more than something else. And that to study, something that's not related to the assessment is to incur a potential opportunity cost where you could have been spending time as an educator or student preparing for that incentive, that end of the year evaluation, for example, and instead, you chose to do something else that could have a

repercussion. So assessment, the first question we like to ask is, what are we measuring? And why? How do we know that it's the right thing? Why is it the important set of skills to evaluate, given the context of the person we're evaluating and the opportunity for which we're

evaluating their fit? And so in the college admissions context, the most famous assessment context in the world, in the college admissions context, when we talk about the LSAT or the AC T, or any other comparable assessment, what we want to think about first is, are we measuring what matters? And my simple answer to that question, it's different from the usual critique, I think of perhaps shortcomings of some of the college admissions assessment.

My personal evaluation of those assessments is they're not measuring a lot of what matters, they measure some of what matters, I don't mean to, you know, disagree with measuring critical thinking and reading and writing, you have to have those skills without a doubt. But you also need to be able to come into novel problem solving contexts as an adult, and not panic and make your way through.

You also need to be able to understand and evaluate how systems work and parse those systems and anticipate their consequences in their mechanics. You also need to be creative, and be dexterous with thinking about new inputs, new challenges, new parameters, and coming up with streams of ideas. You also need to be able to construct a compelling argument using evidence and distinguish what evidence is good and what

evidence is not up to par. So that constellation of skills, I felt could be drawn into much sharper relief for our education system to serve as a more powerful incentive and shaping force on what schools we're trying to teach and why. And the why part is because I believe fundamentally, that the point of school past being tremendously valuable in creating social contexts and teaching us about norms and expectations, if behaving with our peers, the main point of school is to

prepare us to be adults. That is what it is for. That's why we're there for 12 years. And so what a shame for us to go through that system, and to spend all that time and fall short of holding the sets of skills that we're going to be required to use just years later, when were suddenly out in the world on our own and not in this prescribed and safe environment of a school day with a lot of pre scheduled activities and expectations

spelled out clearly for us. So my interest was in leveraging an assessment format that could adequately mirror the complexity and conditions that those of us who are adults now have encountered in the real world. And I felt it was important to think about in an abstracted sense, the dynamics of real world life and real expectations, be they in the workforce, when we're at home, when we're interacting with civil society, or when we're interacting with the internet, whatever it is, all of those

contexts have features. And those features could be something like degrees of complexity, or orders a consequence that we're thinking about when we're interacting with those contexts or dynamism. How dynamic are the variables that are their permanence versus

temporality, right? How do we need to think about time with regard to everything we're optimizing for in any of these contexts, what's beautiful about game based or simulation based assessment is you can mirror all of those features, but you can design them in a format that's hospitable to a testing period, that's 30 minutes or one hour, where you can drop folks right in to the meat of the matter, as opposed to having to spend a lot of time as one would in the real

world. If you were to build say, like a fish tank type evaluation setting where I wanted to ramp you up on a work environment, then watch how you execute on the work environment and then evaluate you that could take a month. So the beauty of the game based or simulation based format, is I can instantly get you into the kind of heightened level of complexity that resembles what we might have as

adults in real life. That was my central interest in understanding the power of game based and simulation based

assessment. As we go down a level deeper, right, so we've talked a little bit about how you measure why simulation based simulation based assessment, what you might be trying to measure how you build these assessments is really complex as well, because now I'm trying to see these amorphous features or cognitive skills that again, you can't precisely slides open your brain and see at any one point, and I'm trying to do so in a way that compresses the nature of context, I'd have the luxury of

observing in real life over months, perhaps into a short time period. And in order to do that compression, I need to comply with the nature of expectations you hold around a game or a simulation. What that means in our case, is I needed to have an environment that you could jump right into, not everyone knows about hospitals, so I can't use a deep hospital simulation, not everyone knows

about airports. So maybe I can't use an airport environment, I need something that's universal and accessible enough for any student to come into one of these places. Or I need to have ramped a group of students up on a particular context well enough that I can kind of take as given that they have familiarity with the pieces I'm going to then use to try and assess these other

skills. So the first area where we can introduce some compounding error is in picking the wrong place to situate this evaluation or this examination of the cognitive skills in our case that we're interested in evaluating. But from there, the potential for compounding error just escalates. What if something's off with our art, what if we've implied that it's cold somewhere in an environment, and then we failed to have snow or frost there overnight where one would expect

it to be. So there are all these opportunities. With regards to the setting I've chosen the background context, which we can often take advantage of assuming someone has some basic familiarity with and that visa vie the art and the way we bring whatever environment to life and again, based or simulation based setting. And then of course, we have the challenge of measurement, it's not good enough for me to just drop you in an environment, I need you to

do things. So I need to have stimuli that motivate the given cognitive skills, I want to see you exercise, it could be that I want to see you solve problems, it could be that I want to see you come up with creative solutions and lots of ideas and see how you whittle those down. But I have to create a game or a simulation environment that asks you to do that. Now back to your question. As I'm thinking about each phase of developing these kinds of game based or simulation based assessments.

I'm thinking about how do I keep the totality of what we're developing as close to the totality in the ways that matter of the settings we're going to encounter in real life. So I'll use a real example to drive this home at Roblox. We use a Roblox assessment in the hiring process for all new engineering hires of certain levels and for all new product manager hires of certain levels, and you can read about this more online if folks are

interested. But this assessment looks at some of the skills that are relevant to working at Roblox as a product manager and engineer. We've done a lot of research internally and I externally with literature reviews and subject matter experts to understand what is that constellation of skills that are required for success here. And then we take that definition of those skills, we call it an ontology. And we bring that back as the blueprint for what we're trying to

measure. And then we do our best to create a design and a setting and a context for a game or a simulation, it's usually about 30 minutes long to try and delve into each one of those skills. And we might stack several of those together, depending on the role and depending on the complexity of the skills we're trying to measure. And as we're evaluating the quality of those assessments, we're looking at different types of validity indicators to understand are we measuring the thing we think

we're measuring? Are we measuring in a way that's fair across all demographic groups, and across all levels of background experiences, and that could be college degree based background experiences, that could also be gaming proficiency. Now, one would argue at Roblox that you need to have some basic gaming proficiency, if you're coming to work there, it's kind of job relevant or construct relevant. But in other cases, it might not

be. And so you want to make sure that you're not advantages or disadvantages certain types of gamers who maybe are really into Candy Crush, or really into role playing games, if that's not something you're actually intending to evaluate. So we're constantly thinking about those themes, as well as what we call

reliability. And this pertains more to the structure of these assessments, what you're trying to do is sense as you have different stimuli throughout a simulation, or throughout a game, you want to start to see different types of stimuli and the responses folks offer in their performance in those settings, you want to start to see those hang together or not, what does that mean, if I were giving you a math test, and I want to evaluate your degree of understanding of subtraction

versus multiplication, I kind of want to see you get a bunch of the subtraction items either right or wrong, and a bunch of the multiplication items really right or wrong, if you sporadically get half of each one right and half wrong, like I don't know, do you know subtraction, and multiplication, it's hard to conclude, right? So what I want to see is some clustering around a latent skill or ability that we've assumed we're measuring, I want to see

that bear out in the data. So I want to start to see that aspects of the test where we expect folks to have consistent performance. And this is now zooming out to across a cohort of test takers versus just you as an n of one, I want to start to see that for a population generally, what happens when they go through this assessment is we're able to see a demonstration with some confidence of a particular skill or not. And if the answer's or not, then you have to go back

and do the whole process. Again, back to the drawing board, figuring out where did your design go wrong? Where did your art go wrong? Where did your setting go wrong, right, and start to parse it that way. So it's a terribly laborious process to develop these kinds of game based assessments. It's extremely technical. And it's almost like casting, a film production, and full cast of characters for a movie or a show because everyone involved has a

different skill set. And they all have to work together as a tight pod from beginning to end, and you kind of flex who holds most of the burden were but at each point, you're asking a group of pretty trained experts to rely on someone else's perspective or agenda across the table from them in a totally foreign domain to them as kind of the dominant strain to consider at that phase in the

design or development, right. So for a game designer, now I have this other stakeholder who matters just as much, maybe more in the early phases, or it was a learning scientist, I've never had that before, as a game

developer. Now I have to worry, I can't do things the way I usually do, because I need to collect these specific pieces of evidence that the UI engineer is telling me because of the learning scientist or the psychometrician involved, that I need to make sure that we document and collect all the way through. So these kinds of tests are very challenging scientific instruments to develop. And I think that's why we've seen relatively few places, build them and use them to great

success. But embellish was a magical team and remains in both incarnations at Roblox and our kind of sister team, who was still at McKinsey and Company, a longtime client of analysis. We see both of these teams continue to execute on these kinds of assessments with grace and with success, because this group of people who we've had working together in some cases for you know, six to 10 years now have learned how to dance that dance and go through that development

phase. And I think everyone remains excited by it because of the high fidelity picture or snapshot. We can get of someone's capacities visa vie an opportunity to make sure that people no matter their background, no matter their opportunities in life to this point, that if they have the goods, we give them the visa airy best way to step forward and demonstrate those in a context that's super closely aligned to what they'd actually

have to do on the job. therefore making it a pretty good mirror when you step back and look in and come out on the other side. So hopefully that addresses some of what you were hoping to understand.

environment like Roblox? Is this part of what you're going for? I know that the things you're building are standard based experiences. They're not you know, psychometric assessments. But do you see a future at some point where people can either get into college or get a job or you know, move to the next grade?

I'm excited for other educational providers, other employers to use robots build their own assessments, the eliminate reagent there is in no way the tech like the platform is an outstanding Canvas for building these kinds of assessments. It has all the features you need, it's already multiplayer, it's relatively easy and lightweight to get ramped up and roebucks studio.

The challenge and constraint same as in strong Game Design for Learning Experiences, is testing design and thinking about how you construct a team that's capable, again, of balancing the thing you're trying to assess with the expectations, norms and realities of using a game based medium. So I'm hopeful and enthusiastic about seeing a lot of providers, perhaps start to build formative tools on the platform and weta is an early example of one of those. And I think more to come on that

shortly. We'll have to confirm with our team that you can keep that one in there. But NWA you know, longtime assessment provider of interim and formative assessment, you're going to see an experience that already leverages Roblox for assessment coming out from NW EA shortly. So I'm optimistic that we'll continue, I think state summative assessment and high stakes assessment, it depends on

the type of test. So could I imagine a world in which a state summative assessment or several state summative assessments chose to bring a deep kind of experiential item to life via Roblox and use that as part of a

bigger test footprint? Sure. I think we're a little ways off from that because of the nature of state's interest summative assessment, but I could imagine that I think admissions testing and employment testing, you'll see imminently on the platform, because it just lends itself well, if you have a team who's

capable of building it. And if you already have that team, it's just such a lightweight place to get started and easily find all the tools you're used to having to kind of stand up my team used to work not on Roblox prior to joining robots and robots is just easier to configure a lot of what we need in an assessment context. So I'm excited to see that evolve. And I'm particularly excited as more colleges look at these kinds of open ended projects or alternative examples of student

greatness. To imagine students using Roblox as part of a portfolio or as part of a long form project or as part of an intervention with a community that's supposed to, you know, be an example of outstanding community service or engagement. I think all of those things will happen.

Roblox. It can be built by these outstanding teams, I imagine that some of our listeners that may be thinking, how can I become one of those people that could be on that incredible team that could be the learning designer or the game developer, or you know, somebody who puts all the pieces together? It is a really exciting world as somebody who has been hoping for a sort of game based educational and assessment experience for a long time. This is the most vivid picture I've heard of it

ever. I can't let you go without talking about artificial intelligence. You obviously have this incredibly wide ranging expertise about so many different things in educational technology and the idea of the day of the year maybe even we'll see maybe even of the coming decade could be artificial intelligence and what it's going to do With education, two questions, how is Roblox currently leveraging AI to enhance the learning experience?

And what do you see as the future of how AI can contribute to the vision that you've been painting over this last hour?

And it's exciting to hear from the developer community, what they want, what kind of integrations they're excited about, what kinds of creations they want to make that right now might be challenging or time consuming or tedious and to think about how we can support

them in those workflows. So as for Roblox, that's what I've seen so far on the developer side, what's interesting is a number of our grantees have already been thinking about the client or platform side, how to integrate generative AI into the experiences they're building that let people build

experiences. So to think about if there are levels that can be customized if there are coding challenges, for example, Code Combat is launching an experience that does an outstanding job welcoming in some generative AI and assisted AI for students when they're

learning to code. And I'm excited about that merging in robotics experiences and the full hearted embrace of AI, obviously, responsibly, but generative AI as a fellow developer, or a fellow classmate, or a fellow research assistant, I mean, the beautiful thing speaking as someone who works with chat GVT all the time and has been really enthusiastic and early adopter on my journey, Dali, three stable diffusion, I mean, a lot of the different tools that have been coming out,

I really enjoy just the endless array of roles, you can if you know how to drive these tools, you can drive them to assume the ways in which they can help our workflows, the ways in which they can educate us, the ways in which these tools can make us more efficient and allow us to spend more time again at the higher orders of our functioning and less time on tedious crap.

That's exciting. And I hope that educators and schools everywhere lean in to preparing students to interact with these tools in

that kind of capacity. So I'm very optimistic in terms of the impact and educational environment, I think this is a helpful push towards reckoning with the reality of where our job markets have moved anyway, right, I don't understand a education system that would deliver students who can do the bare minimum of computational processing instead of really indexing on the human processing and the human framing and the human perception of the world.

And then thinking about using computational tools to one's benefit, and hopefully to the world's benefit. I think that should be core to all curriculum that we're teaching everywhere. As you know, there are so many problems that are so worthy of all of us spending time working

on and solving. And I think we should be challenging all of our students to think about how generative AI has one tool among many, but how they can play a role and perhaps leverage these new tools to do even more to better their local communities to improve conditions for people for animals for the planet. So I'm pretty excited about offering students that freedom and additional horsepower and capacity that comes from thinking about leveraging a suite of generative AI tools on

Roblox and elsewhere. And I hope educators and schools take advantage of the opportunity to do so and radically, accommodate any necessary changes to projects or changes to assignments, to ask more of our students in some ways and ask less of them and others as is going to be the case when they enter the workforce and operate as adults in the world.

on Ed Tech insiders. We usually end with two quick questions. I'd really love to hear your answers on these given what you've been talking about. The first is what's the most exciting trend you see in the Ed Tech landscape right now that you think our listeners should keep an eye on. As things move forward, what's right around the corner.

Because if you look historically, at past generations, like some of the best mobilizing moments, and the times when people have greatly advanced the human condition, and, and made real progress have been on the tails of really

harsh crises. And I think for our young people, to graduate with such maturity and grace and triumph over challenging home circumstances, challenging macro economic conditions, challenging macro pole, geopolitical conditions, the atrocities that we're seeing all the time, be they due to climate change, or natural disasters, or, you know, this week, the atrocity committed by Hamas and Israel and the conflict in Palestine and the innocent lives lost

everywhere. That should be the ultimate motivating force for all of our education and all of our collective aspirations to help empower young people. So I'm pretty excited about the opportunity for generative AI to help folks feel like there is a way for each one of us to extend our reach, extend our capacity, and constantly learn and

upskill. So I'm excited to see a next generation feel like they have the fire in their belly from surviving and fighting through so much, and seeing a bright light, hopefully, in leveraging all the tools at their disposal to try and imagine and create a better future for themselves and others.

and information. I think broadly, the other thing I'd encourage for all educators, and hopefully, I'm not too far out ahead of my skis here, but is to recognize that there's a real threat to public schooling right now in America in particular, and a real quest to normalize rhetoric around shutting down federal funding and support for schools around banning books around deflating educators and their ability to do their jobs.

And I would just ask that your listeners try and support some of the organizations and individuals read the substacks, you know, really immerse themselves in some of these discussions to make sure that we have a school system, we kind of have always taken it as given that the education system is going to be there, and that it's going to withstand all manners of challenges. And I'm not sure

that's the case anymore. So I want to make sure that all of us who depend on the education system, who believe in it, are here around the table staying abreast of news and attempts to perhaps undermine or discredit our public education system in this country. And not that there's anything wrong with private schools. But we

certainly need both. And so making sure that all folks continue to have an opportunity to high quality education and following librarians associations, getting involved in your local school board, making sure you're appraised of of what's happening on the ground for you, I think is more important than any one resource or any piece of thought leadership is to really have an active stake and your school in your district.