Evidence as a Service in Edtech with Karl Rectanus of Learnplatform

companies. Karl Rectanus is an educator, serial entrepreneur and the co founder and CEO of learning platform, a mission driven for benefit corporation where he leads a team of educators, researchers and technologists committed to expanding equitable access for students learn platform is an edtech effectiveness system, working to deliver safe equitable cost effective learning for districts, states and their partners such as solution providers, learn platforms, technologies are used

by over 8000 schools, serving more than 7 million students across the United States. Karl Rectanus Welcome to Ed Tech insiders.

classroom. And then I actually became a CFO for schools, especially in the early days of charter schools, because I felt like the back office was screwing up what I was trying to do in the classroom as a teacher, and I wanted to understand what there's got to be a better way. And what I found was, as a CFO, I was consistently having to use last year's data and old information to inform next year's budgets

and decisions. And of course, you have, you know, a third of your students by design are going to turn over and middle schools where I was working or high schools, but certainly teacher turnover, etc. It was just very difficult to run organization that way. And so when we launched learn platform in 2014, our goal was to help people figure out what they were using that works to bring, you know, essentially, there was no data around ITT Tech, and what was working. But you know, as we

came to market, we help. We said, We help you figure out what you're using that works. And they said, could you just tell us what we're using? Because we're not even sure. And we realized there was a much, much larger need and larger opportunity.

years. But we've also realized that on average, school districts are accessing over 1400 Different ed tech products every month now, since COVID, started, I mean, that is overwhelming for most districts and states. And so understanding what's working in which situations becomes even more critical. The other thing we've learned, while most of our work has been with districts and states, is that providers want and need evidence to inform their own decision making to make better products to support

their work. So we've launched this new service evidence as a service for providers. I want to answer your question on why we've launched this new service evidence as a service. And there's really three reasons why now, the first, as I mentioned, the learning environment has become more tech enabled and more complex, over 1400 Different edtech products being used in any school district in the US that is totally

overwhelming. Most LMS is only integrate with, you know, three to 500 tops, even your largest LMS is that means almost 1000 or more interventions are happening outside of the learning management system. I mean, there's, there's just a ton going on. So it's it's overwhelming is the first one.

Secondly, we know that it's not the same for all students, students of color, suitor students in lower affluent, less affluent areas, their edtech engagement is significantly lower than their peers in more affluent districts, our data shows that there was a digital learning equity gap before COVID. And expanded, it was

actually a K shaped recovery. So those who had access and were affluent, had more ad tech engagement, and others did not, they actually engage less, and that expands that divide that is so critical. So the second reason is equity. We know it's a persistent problem, and we have to answer it. And the third is

actually regulatory. For a long time evidence in education has meant the randomized control trial, you had to have a What Works Clearinghouse or gold standard research that took 18 to 36 months, it was costly, it was difficult to run it, these randomized control trials often weren't read by practitioners to make decisions. And because of software development, life cycles, they actually are out of date by the time they're printed, because the interventions have evolved

during that time. Well, that there's a place for those strong for those randomized control trials for that gold standard. But the Every Student Succeeds Act actually outlines four levels of evidence. And that RCT that strong evidence is the top of the line, there's no doubt about it, there's a important place for that. But the Every Student Succeeds Act actually creates a regulatory on ramp for any product provider, any solution to be deemed evidence base, which is highlighted with

an ESA and stimulus funding. It can start as basic as a logic model that can be shared to outline that they demonstrate a rationale for what's going to happen. That's how those are words in Every Student Succeeds Act, and in the guidance from US Department of Ed. And you'll notice that that doesn't include a need to share private data or anything, it's a logic model with a clear and supportable

approach. The second and third levels of evidence that lead on to or into a randomized control trial are called promising and moderate and Every Student Succeeds Act. And that is a correlative comparative or control based analysis, that can be done at a much smaller in size, and much more easily than traditional research has been able to do or has been acceptable. Now, that is

acceptable. And through our technology, through advancements in work, and through collaboration with providers and practitioners, we've launched evidence as a service, to make it more cost effective. And quicker to be to show and gather your evidence in weeks and be compliant with the requirements of SSA and stimulus funds.

But it's very slow, and it can be costly, and not and for that reason, not that many educational, you know, tech companies pursue it, especially if they're very small. So when you're saying is that there are other tiers that still meet compliance standards and provide the type of evidence that can support you know, product direction, it can create equity, as you mentioned, it can give schools insight, but it can all be done much faster and at a lower cost. Is that,

control trial. But it is very valid evidence to inform instructional decisions, or, you know, how we focus. And so what is great about this, Every Student Succeeds Act is it actually makes room for a logical approach to gathering and sharing evidence. And whether that's within a district, you know, an individual local education agency, or to make it accessible for solution providers to really test and vet their solutions and

make them better over time. So they get to a point and more of them get to a point where they're ready and able to do those types of RCTs.

compliance reasons. Let's talk about some of the other stakeholders like you know, the parents and students and other people in this ecosystem, maybe administrators in schools, principals who might care about the evidence, talk about the sort of entire entire circle of people looking at why and tech evidence and efficacy?

pillars. One is evidence, you know, which we've talked a lot a little bit about, and the other is efficiency. That is how do we automate processes to ensure those stakeholders, whether they're a teacher, or a parent, or a district administrator, or a solution provider, have the information they need to inform the decisions they're making. So the stakeholders you outline are really important. Let me give

you an example. On our system, school districts have run literally 1000s of rapid cycle evaluations, they've used our technology called Impact, to do the equivalent of a third party evaluation in a matter of

minutes instead of months. And that has created visualizations for the district administrator, say at the end of the first semester, to have a sense of when we're using a product or intervention, how much are we using it and which, which groups are using it the most, and when they use it, in which situations does it appear to be having, you know, positive effects on benchmark assessments or other other data, you know, one district share with me a story that they had done this at the

end of the first semester, and they found that their math intervention was having an outsized, positive effect for English language learners in their district. They also saw that their highest achieving students weren't, were already there, they weren't having that same positive effect, they needed something else there. And that they they were using, because of that their student number of licenses they needed, was actually different than what they had previously projected in

their coming budget. And so the curriculum instruction stakeholders understood and thought, hey, this has impact on our instructional design and our professional development for our teachers, they were able to communicate to parents, you know, these interventions are going to be used with, you know,

students in this situation. And this other intervention is used for, you know, students in these classes, for example, and those parents could understand what was being used by whom, in which situations and why which obviously, leads to better student retention, better parent communication, etc. But the CFO also had more clarity on the number of licenses they needed, or were projecting, moving

forward. And the Technology Office knew what to approve and not approve, or what they needed to get vetted or for things like interoperability and back office management. So each of those stakeholders has different takes and needs, but provided using that evidence to inform them with the information they need is really important. And one partner that I didn't highlight in that. That's really important though, is the solution provider

themselves. ELS. And in that case, this was a couple of years ago, the the district actually came to us and said, we're a little worried to share this with a solution provider, because we're not going to use it for all students, but we're having an outsized effect, we're getting positive effects, but we're not going to use it for all students. And we'd like you to walk through the data with

them. And the solution provider, they thought they were, the district was concerned that the solution provider was upset would be upset that they weren't going to purchase quite as many licenses as they had before. The solution provider actually thanked them vigorously, and ended up writing case studies and validation and informing their product development based on this information, because they were having this outsize positive effect with in math with students who are non native

English speakers. And that's a group that they wanted to focus on and could could add more value. So each of the stakeholders in the cycle, you know, gains from having a shared fact based on what's actually happening. And, you know, in our experience, most of them are focused on what's best, best for every single student.

school. So the ability to do that quickly, and then share back with a provider that things that it's working in a particular context, especially something as important as English language learners, I have a feeling they will be very happy, not not just upset about not selling enough licenses. So I love that story. And that's

same page. Nobody sees any data they're not supposed to, but be able to directly deliver evidence to all parties to help inform what's best for students, you know, it was meaningful. And that's what led us to launch evidence as a service.

worked, in what contexts. And that was one of her main impetus, you know, it just starting at search, which you might sound a little strange, because that's not exactly what that search ended up doing. But it was a really interesting insight. And it sounds like something that you've been doing

a lot to address. And I, you know, I wanted to talk specifically about the K 12 context, because one thing we hear pretty consistently on this podcast is that at tech entrepreneurs, when they're deciding what company to start and who to focus on and what their market is, sometimes get a little nervous about, you know, k 12, they get a little nervous about creating companies where they have to sell into school districts, because there are procurement processes and data

laws, as you mentioned, and, you know, complex decision making ecosystems. And it sounds like, you know, what you're doing with evidence as a service, as well as what you're doing for schools and districts, maybe even more so can begin to address this. So tell us a little bit about the role of evidence in sort of making it easier for Ed Tech to work within the complexities and bureaucracy of the school system? Yeah,

about that. You know, and that's for a couple of reasons, in particular, because of regulatory requirements, because of the way, you know, finance and offices of procurement, and generally in the public sector set up to avoid the bad thing from happening, not necessarily help the good thing, not that they're against it, but it's, it's really focused on how to

avoid the bad thing. Right. And so they, you know, as a former CFO or and leading procurement, we were sometimes considered the office of No, you know, we just sort of slow things down. And, you know, and obviously, we want to make sure that there's the bad things being graphed. The bad things get being, you know, loss of data privacy, or other overcharging, you name it use of public funds. And so traditionally, it's a very

cumbersome process. And so people try to figure out a better way to do these things and how to work around it. And that leads to decentralized purchasing, or freemium models or other things. And that's certainly been important within technology space. The other thing it's led to is people sort of saying, like, it's a beast, there used to be a saying and other in the corporate world, you don't get fired for hiring IBM, they were the safe that

right? It was okay, because and so, you know, in our space, you know, it was safe, if you were purchasing the large provider, right, they weren't going to fall apart, they weren't going to go somewhere, they weren't going to, you know, be acquired necessarily, literally all of those things is not the case, and change, but that was the perception. And so the highest indicator of purchase, is the nearest district to to have purchased your stuff before, right referrals or what have

you. And part of that is because evidence has been somewhat convoluted. Those randomized control trials, all of those are driven by bespoke reports and deep research, academic work that don't translate very well to administrators. They're sometimes written in a way that are inaccessible for educators. They're housed in journals that you have to pay for, and practitioners don't don't use

them. And practitioners have also told us for a long time that we don't trust individual marketing, because it's not third party validated. So their third party has been their peers and partners and other districts. The opportunity by lowering the cost of third party evaluation evidence is a service you can think of it almost like fractional third party evaluation that any solution provider can access and afford.

The equips a trust mechanism that equips school districts and states to increase their trust and what's going on for providers. And we're excited about where that fits evidence has been useful, but not necessarily a highest indicator of purchase or procurement. It's been difficult to build into

that that procurement cycle. But we believe that by equipping districts and states with the ability to activate their own evidence to share that across their own districts, and then equipping solution providers to gather and share theirs through trustworthy, but more cost effective third party validation, that evidence will become a much more important part of the procurement process. And we're already starting to see that play out with stimulus

dollars. And Every Student Succeeds Act requirements for evidence based interventions.

whole world. And as you said, it really had almost no impact on what schools were doing because of the format because of the structure because it was just written in a language that was just inaccessible for too many

decision makers. I'd love to just double click on what you are mentioning here in terms of the format of the evidence that comes out you mentioned earlier that you they can do visualizations, that part of what learning platform enables is, you know, an administrator can make a visualization that can be shared and used for

decision making. Tell also a little bit more about what you're doing to make sure the evidence is not only being collected and reported on, but it's also shareable and actually being made usable for decision making. Yeah,

districts. One example of how to show and share this is we're launching for the 2223 school year designation called Impact ready, impact ready will be is a visual badge that literally any, any solution provider can earn and gain and we're doing, we're working with solution providers right now. And it will be able they can show it on their site that will be visible within learn platform as a searchable field and, and that impact ready badge turn that you only have to

do three things. One, you need to have a lot a shareable logic bottle, right? A visual logic model. With our subscription, we help design it, you can have our research team do it. But you need to have a visual logic model or more an evidence base. So you have to reach demonstrates rationale, or above. And be committed to having an evidence based too, you need to share your data privacy agreements, every solution provider should have and make their data privacy agreements visible and easy to

find. So sharing them on our system. And then thirdly, you have to be willing and commit to sharing usage data, for the purposes of research in order to be able to do you know that those correlative comparative or other analyses moving forward, these three basic things are available, any solution provider that wants to work in education

should be able to do this. And by setting that bar, we're not, you know, making it difficult, it's not a subjective decision, you do those three things, you will have impact ready, the value proposition for the reason we set it there is because districts and states are so overwhelmed by all these

different edtech products. They said, If you help us see which products are committed to evidence and committed to data privacy, that will help us that will smooth out our purchasing cycle that will make life easier for us, we will at times automatically approve them for use, we will help those move more quickly through our procurement process that is valuable, we're not going to take away procurement decision making at a local level, we're not going to take away obviously, you know, that will

always be a local decision. But by making it easier and tying evidence to show how this can help the whole market move much more quickly. We're excited about what that can mean for using evidence in the

procurement cycle. Thereafter, there's tons of opportunities because of stimulus funding and ESA, to show that you're an evidence based intervention on an ongoing basis, by lowering the cost of doing, you know, third party evaluation to, you know, just a fraction of what it used to cost, do this that impact ready badge, we're really excited about that as a way to help create a shared fact base in the market to drive evidence into the procurement cycle.

a sort of obvious hurdle or, you know, a condition that needs to be met to smooth out the

procurement process. I you know, I think that's a really, it's a really interesting answer because it takes the enormous complexity of these randomized controlled trials and you know, the academic papers that are that are now used for this and turn in sort of boils it boils it down a little bit into a switch into an on off switch that allows the two sides to speak to each other without anybody needing to come through, you know, come through abstracts or read, read, you know, the

methodologies of studies.

But one of the other things that I want to highlight is is something that we've been working on with a number of foundations, the federal government, researchers, practitioners, especially those in districts with high free and reduced lunch, and high populations of non white students, is something we're calling a universal evidence

report. The you er, for sure universal evidence report is an answer for a challenge that you just highlighted, those RCTs, the research that's in the What Works Clearinghouse, all these academic studies, they're not computer readable, they're bespoke, they're challenging to understand, they're almost impossible to translate or compare across different groups

or studies. The Universal evidence report is another step that we have been working with a collaborative of organizations to create, you can think of it as a cover sheet, or even like, I sometimes call it a 1040, EZ form for evaluation, right, we all have different family and financial situations. But every year we fill out a set of forms. And maybe we use TurboTax, or CPA or something to help us. But we fill out a set of forms that make it easy to understand what

the situation is. And to make sure that we pay the right amount of taxes. The Universal evidence report is that cover sheet that highlights a study the intervention, the core components, the context and the findings into a set of fields that ultimately can be computer readable and serve as a receipt, or a validation that we've done this this analysis that there was an analysis, this will be open sourced and available for anybody to use, it doesn't matter who does the research.

But with the use of the universal evidence report, it would equip districts, states, providers, and others, we would have more computer readable edtech evidence than we've had in history, by agreeing on standard, just a just a form to report on analysis, not judgment, just the facts, right. And so that universal evidence report is just one other sort of evidence based solution that we're excited about coming into

2022 and 23. Being into the market, we're right now getting feedback from supporting a feedback loop with researchers and stakeholders from across the market to inform what should be included and sort of the the beta versions of that. But you know, there's not only at the beginning, but evidence is an ongoing process. So being able to learn together is we're really on the front edge of that free the US education system.

across the tech landscape. I agree that the computer readable nature of it, the idea of being able to compare efficacy across different companies across different populations across different ages, is just almost impossible to do now. And that's what the you know, what tort Clearinghouse has been trying to do. But it's just almost impossible to do given the current format. So that really does add up. I want to talk a little bit about some of the

obstacles to this. I think it's a you know, it's it's so exciting to get evidence into the conversation in this way. But I think it's pretty fair to say, having been in the field for a while that, you know, pretty much all stakeholders would, in an ideal world loves to have evidence that the educational products they're buying or building or using, or that their kids are using are effective. In practice, it hasn't historically been simple. And we've named some of the

reasons already. Why, you know, there's data privacy issues, there's the complexity of communication. But I wanted to dig into a couple of other obstacles. The first one I wanted to ask you about is you mentioned the story that the math provider who works out and they were having outsized, positive effects, and they went back to the provider, and everybody was very happy about that. What happens if, you know as, as edtech tools start getting compared what would happen to the tools that aren't

as effective? How would they if evidence is more transparent? What what might be the outcome there?

But it's been difficult. We've created these barriers, some of which you outlined, which make it really difficult to make those decisions or to even have the data to inform those decisions by equipping, whether it's districts, just decision makers, practitioners, and providers with the tools to do this more effectively, I think we get to the right answer more quickly, I think we eliminate a lot of waste in the process, and we are able to reinforce, and recommit the investments we're

making into education into things that are working better than the other thing. I think we get to a place where not every new entrepreneur has to reinvent the wheel with limited

information. I don't think there's any doubt that people who, like myself and others who have been in this space for decades, have a leg up on how to do this work, just because we've got decades of experience compared to others who are new to the space, when in every other market, you can enter a space building on the shoulders of those others who have worked

before you. We have not invested in the research and development infrastructure and education, the way we need to and have in other and other sectors to be able to do that. And I think what we're doing now equips better decisions that will lead to better outcomes for students, and frankly, better economics for solution providers.

quickly. And so I love just to be clear, I love the shift towards evidence, I think it's very good for the field, I just, you know, have a little bit of a small cynical bone only because education is hard and edtech is hard. And for every really terrific tool that is going to be proven out through to be effective and effective with with all sorts of populations. There may be others that when the data is actually gathered are on the shorter end of the

stick. But I think that's I think all in all, that is definitely positive for the field, it's better for education, it's better for procurement is better for students. And that's obviously better for everybody.

tail. But what our initial looks at this work is the size of the organization is not necessarily a determinant of the evidence of effectiveness. But traditionally, that procurement has gone outsize to those larger organizations, you know, the bigger you are, the better your sales are. That's the definition of success, whereas newer entrants, and earlier stage organizations may have solutions that really outshine, you know others, but haven't had the

chance. And quite frankly, I think we're leveling the field. And for those larger organizations, many of those larger organizations do have very effective tools and products, but they should be able to, you know, communicate that as well.

that tech a lot. So that, you know, in the long run, what this should do the introduction of evidence that sort of a basic level and comparative evidence is allow the decisions to be made purely on the benefit to students rather than some other factors, which traditionally have been used to make that kind of large scale decision.

around evidence. So you know, other than learning platform, you've been in the space a long time, who are some of the other players in the Ed Tech landscape that you feel are, are helping drive this movement towards evidence based decision making? This could be nonprofits, companies, you mentioned a coalition working on this universal evidence report, we'd love to hear a little bit because you're so close to this space a little bit about who else is sort of joining you on this journey?

You know, Mark Schneider, and the team at IES are fantastic and really leaning in, they, you know, for all intents and purposes is over the last 25 years, has been the sort of sole investor at the federal level, or at least the primary investor at the federal level and education, research and development or research specifically, and you know, that team and the grants that they're making the efforts that they've launched, most recently, it would include, you know, digital

promise won an award to lead something called Cyr net. And so Jeremy Rochelle and the team, they're working with large platforms to do some great work around evidence creation and

sharing and tooling. Certainly philanthropies we've been honored to work with folks like the Bill and Melinda Gates Foundation Foundation, the Chan Zuckerberg Initiative, Michael and Susan Dell Foundation, and other philanthropies that have been making grants into this space and, and have been leaning in on how to collect and share and make evidence more valuable, especially as it relates to

equity. I think the intentionality of many nonprofit groups like Arif, based on sort of the DARPA work or education, innovative to you, has done some great work with Project unicorn and a new framework called birdie. That is a framework for safe data sharing is d is and other associations like Kosan. And the Council of Great City Schools are are all supporting practitioners and the solution provider community to engage together in a way that's safe and effective and sharing data.

But ultimately, the two groups that are we've just been so empowered by our district administrators on one hand, who are absolutely like leaning in at a time when they've never been asked to do more. So really caring about how do we solve these problems around making decisions on what's best for students? And then solution providers are I mean, when we engage with solution provider, they are all in we hear the concerns of what about this? And what will this be for the

market? What if, what if I don't find X or Y, but the the commitment from organizations. I mean, some of our early adopters are small, medium and large. They're in the traditional curriculum, areas, like age of learning, math and reading. They're an assessment like formative there in non curricular net press and some others who are doing some great work, their commitment to evidence is really leading this

charge. And the way they're working as partners with districts and states is just is so excited for the field.

in the EdTech lands? Skype right now what's something that our listeners should keep their eye on? From your perspective?

than they have been. But the opportunity to modernize learning to not waste a crisis, that's a trend that we feel like we're seeing across the entire landscape that people are trying to, you know, not get, quote back to normal, but they recognize it's time for the next normal. And they're really leaning in, we're seeing, you know, the trend we're watching and supporting is leaders modernizing learning. I

hashtag edge chat. It's getting into whether it's Twitter or some other feed, the edge chat edge chat or superintendent chat or those any given day or night. Digging into that is such a great finger on the pulse of what what's really going on in school districts and states right now that that's my recommendation.