[MUSIC PLAYING]
Hello and welcome to Data Nation, a podcast from MIT's Institute for Data Systems and Society. I'm Liberty Vittert, and I'm here with my co-host, Munther Dahleh, the head of MIT's Institute for Data Systems and Society. In this episode, we talk with Fox Harrell. Fox is professor of Digital Media and AI in both the comparative media studies program and the Computer Science and Artificial Intelligence Laboratory at MIT.
Professor Harrell's research investigates and innovates ways that the computational world can mix and layer with our actual world. You hear the word "augmented" reality or augmentation of reality. We hear the word "virtual" reality. Can you give me just a layman's version of what are all these different realities? What do they actually mean? I'd be happy to share with you a layperson's introduction to this medium.
In some way, we all have virtual extensions of ourselves, because if you use a social media profile, you have a virtual self. If you have any kind of e-commerce account, you have a virtual self. If you have any kind of avatar or character, you have a virtual self. But when it comes to augmented reality, virtual reality, extended reality, and so forth, I think about it as a spectrum.
The reason that we call the center that I founded and directed here at MIT, the MIT Center for Advanced Virtual Reality, is because I'm interested in all of the kinds of technologies that blend the physical world with the computer, with the computationally-generated world. And you could say each of those technologies you mentioned, does something similar. So, for example, virtual reality blocks the world and replaces it with something else.
Augmented reality overlays something on top of the world. Mixed reality is typically thought of as augmented reality, but handling occlusion. So it looks like things are actually embedded in the world. And extended reality is often used as an umbrella term for all of those types of technology. That's great. Actually, that's nice positioning. I feel like that's the first time I understand what all these things mean. Yeah, I agree with liberty.
And having heard these expressions, even from you, Fox, I think this is the first time you actually define it for me, and it's very helpful. But then, given that we have this spectrum, it's clear that there are different applications and different value for different aspects of the spectrum. A lot of time, we think of gaming as one way where there is aspect of virtual reality, maybe even augmented reality. How big is the spectrum of applications of these ideas? It's a great question.
So one way that I think about it is, first of all, just thinking about computational media more broadly, because unlike some other prior forms of media, the computational medium, some people refer to it as a media medium, that is, it can emulate previous forms of media, that is, we can watch cinema on the computer, we can listen to radio on the computer, and so forth. We can play games that were formerly board games on the computer. But it also has its own characteristics.
And those characteristics, you could say, at the core, is that, it's algorithmic and data structural, and everything that falls out from that. And so that's just, I guess, an entry point to say that the applications are very broad when you think about it as a media medium, you could just sit in a virtual reality space and watch a screen in front of you. It wouldn't be the most innovative use of the medium.
In terms of gaming, like you mentioned, I think some people these days are surprised just to learn how far the gaming industry has outpaced the film industry globally, sports industry, globally, in terms of revenue. So it's just a pervasive medium that has massive uptake these days. But increasingly, for education, for business, for such a range of applications, different types of virtual worlds are being used.
A Lot of times, following the idea of computational media and its data structure and algorithmic nature, is the same content can be broadcast, streamed, participated in in different ways on different platforms. So the same content that could be experienced through, say, a VR headset could also be experienced through a mobile phone and also experience through your browser, and so forth. So when you think about it that way, it's just in massive use and pervasive.
Although, the uses don't always maximize every affordance, just meaning that not every single use is going to be an immersive virtual world in 3D, where you take into account proprioception and the sense of the user in space and all of these sort of things. This is really interesting, Fox, because I'm beginning to think about the breadth by which this mixed reality plays a very important role in our society. And I'm thinking in a very primitive way.
I was flipping through my TikTok the other day, and there's all these advertisements about how you actually look different. You can run your filters through your images and you create an image of you, but a lot better than you. And I think of that all the time. So what is real? What is the real, and what is augmented, and what is virtual? And AI is doing this a lot to us. You can frame yourself differently, but it's still you in some level. So what is real and what is virtual?
It's a great question. And I'm trying hard to resist the low-hanging fruit of the joke you mentioned, when you say it's like you, but better than you. I say that, yeah, many users find that to be the case, although, I've not found the better version of me yet. [INAUDIBLE] You're the lucky one. I will interleave that kind of joke with a lack of humility with the self-deprecating one next, just so you get a sense of the full spectrum of personality.
But when you mention this question of what is real, of course, it's such a profound question. And one way that I think about it is like this, because people sometimes distinguish the virtual from the real and they say, that was a virtual world, and this is the real world. That was the virtual character, that's the real you, and so on.
And I don't think about it exactly that way, because let's say that you're in an online course for learning computer science and you learn breadth first search, depth first search, and you begin to learn more advanced concepts, and so forth, then you step out of that platform. It's not as if you didn't actually learned something.
You learned something, it just was computationally mediated the same way you could say that a conversation that you have on the telephone isn't a fake conversation just because it's not in-person. It's just mediated through the telephone. Similarly, if somebody has an experience of harassment, threat, and so forth online, mediated by the computer, I wouldn't want to dismiss it to say something like, well, that wasn't a real experience. That was just a virtual experience.
Again, it was just mediated through the computer. So when I think about the real and the virtual, I tend to use the terms "virtual" and "physical world." And I use physical world in place of what people tend to refer to as the real world. But there are so many more nuanced ideas about this kind of question. So, for example, in Jean Baudrillard's book, Simulations, he begins to talk about notions the like the hyper real.
And this is where you have experiences in the physical world that are so based in the artificial, whether that's because of synthesized, manufactured environments, or advertising, or all of these things that mediate modern life and you begin to lose that sense of what feels like something essential and authentic. I mean, the examples that are given, one often is Celebration Florida, which is a city that is like Main Street in Disneyland, but it's this entire town. It's a manufactured town.
And you can say, is that really a small town? It didn't come about organically. It was planned. And so that sort of thing, he begins to refer to as the real. And it might be a bit theoretical, but I mention it just because when you talk about a term as loose reality, you can also talk about the ways that what we experience in the physical world can feel more or less real in a way. And then you can also distinguish between what happens in the virtual world.
To point this out, in my class, I have the students read both about Celebration Florida, and there is a Village Voice article from the 1990s with a harrowing title by Julian Dibbell, which is called A Rape in Cyberspace. And this is about a platform which is entirely text only, everything is described. And you'd describe what you'd, say, go north, go south, pick up object, and so forth, but with other people there.
And somebody had an artifact where they could control another person's character and they basically harassed them in a horrible way. And although it was a virtual character, they felt really disturbed by the trauma of the experience. And it became an inflection point because people would say clearly something traumatic occurred, but we can't conflate this with trauma in the physical world at the same time and say that they are the same.
So what does this mean, that there was some psychological damage that was inflicted? And so I contrast that experience because it's something that somebody certainly experienced as real with something in the physical world that feels quite artificial so the students can begin thinking about exactly that question that you mentioned from day one. I think that brings up such an interesting concept of, do we get programmed, in a way, to perceive things that aren't 100% real?
Whether it's something artificial in the physical world. I put on a ton of makeup. So I'm artificial in the physical world. Or I go on-- [INAUDIBLE], I'm shocked to hear you have a TikTok. I think that's fabulous. But to go on TikTok and I put a filter on. I have no makeup on, but I put a full filter on, and then I'm in the virtual world, but someone completely different. Would you almost say those are the same thing?
Everyone's fear of, we're getting programmed to perceive things that aren't real by the virtual world, is that not even necessarily fair because we're also programmed to see things as not 100% real that are in the physical world? I would say that some of the social phenomena are the same, and we could go back to seminal work in sociology, such as Erving Goffman's work on self-presentation in everyday life, that is, how people manage the impressions that others have of them.
And in the physical world, of course, we are doing impression management frequently. And people do the same thing in virtual worlds. When it's based on algorithms and data structures, then that allows you to do different things than you could in the physical world. So for example, with virtual makeup, the developer might have access to when you purchased it, from whom, or was it traded to you from which online store. There might be a lot of metadata around that.
You could track trends quite easily. Somebody could change the implementation of it so that the same metadata then refers to a differently-appearing makeup. For example, you could make it dynamically change over time, and say, process it algorithmically. So there's a lot of different things on top of that technology that you could do differently than in the physical world.
Although, of course, with physical world makeup, there would be some things that you couldn't do in the virtual world very easily that have to do with tactile sensation, maybe, with the method of application, with the blurring, blending. It could be done, but it would take, maybe, some more advanced techniques. And so a lot of my work is looking at the intersection of issues, like what persists from the physical world in terms of social phenomena? Because I do a lot of social simulation.
One of my friends, a professor at UC Santa Cruz, likes to refer to it as social physics. In the same way that games have physics engines to model physics relationships, and gravity, and those sorts of things, this is a work that's modeling social phenomena, like social physics engines. Then what also becomes new in those environments, what new forms of experience, whether empowering, disempowering, and so forth, come about because of using the computer.
One key aspect of it is that, in the physical world, I mean, we're interacting with other people. And all of the phenomena just fall out naturally from that. Sometimes in the computational world, people think that just because you change the physical appearance of an avatar, for instance, then you will know what the experience of somebody else in the world would be. And I don't know why people make that kind of mistake.
It's like if somebody puts on, to continue with your example, makeup and they look like a different person in the physical world, I think people are less likely to think that now they know entirely what it's like to be that other person now. But in the computer, sometimes people like to think, just because you use an avatar of one gender, or one ethnicity, or nationality, you now somehow know what it's like, like you've walked in the shoes of somebody from that group.
And I feel like that's a misconception about the medium. But this is actually really interesting. And we're segueing a little bit from images into, essentially, social experiments. I mean, you talk about social simulation, but what about social experiments? And in many ways, it's very expensive and it's very hurtful to have an actual social experiment, where physical people are interacting with each other because that's not that easy. You cannot intervene.
You cannot do things that cannot be done. But you can do them in an augmented reality. And both of us can be interacting with the game or interacting with the platform. And what you're highlighting, it's not always the case, that we behave the same way. And so I'm curious to your thoughts about using this medium as a social experiment and to what extent the information that we draw out of it is biased or is not necessarily valid. I think about it in this way. It's a kind of parable.
There is a story by the author Jorge Luis Borges called On Exactitude in Science. If people know this author, he writes these parables just to think about the nuances of life. There always some tricky outlook on life. So this is one about a society that creates a map that has a one to one relationship with the physical world. And so you could think, if there was a map that was like this, then, essentially, it's large enough, it would cover the world.
And so it's an impossibility, in a way, to create a map that has complete fidelity to the actual physical world. Because then, in a way, almost, it would be the world. That's the idea of the parable. I mention it just because I think when we have these simulations within virtual worlds that are like this, it's similar to creating a map. If you create a map of the globe, that's a flat map. There's going to be some distortion that takes place or there's going to be some elements that are omitted.
Do you show certain geographical features? Do you show national features? And so forth. And I think of the process of designing these experiments as a similar kind of mapping. You have to decide what you're going to map from the physical world phenomenon to the simulated phenomenon. And there are disciplined ways that we can do this. I use an approach, which is called isomorphic semiotics sometimes. It can be used informally or very formally.
But essentially, it's looking at the sorts of things that are being modeled that is like the data types, looking at the ordering, the relationship of them to one another, and so forth, to look at a structure of the social phenomenon and then think through, carefully, what we want to preserve in the simulation. And that decision of what we want to preserve is typically driven, in the case of social science simulation, by looking for the latest social science theories in that particular area.
So when we did work on modeling sexism in the workplace, we looked at a model, which is called the ambivalent sexism inventory. And we thought, what do we want to incorporate and what do we want to leave out? Because it's such a pervasive phenomenon and, in some ways, so well-known, but not well-explained, and not well-recognized, sometimes, just in terms of the outcomes not being commensurate with the injuries that are caused.
So we chose this particular model because it highlighted a few phenomena. Hostile sexism, which is what is covered, # say, for example, in the #MeToo movement, but also forms of everyday sexism, like certain paternalistic attitudes, complementary gender differentiation, like assuming people will be good at one skill or another, not good at one skill or another because of gender in part.
And so we wanted to take that model and then think, how can we preserve that in the simulation so that people can recognize, name, see the systematicity in those kinds of phenomena? So that's a typical way that we work. And I think if we work it with discipline and care in this way, then you're creating a reasonable distillation of the phenomenon to study.
If you just do something as simplistic as, say, give somebody an avatar that has a differently-appearing body, then you're not modeling any of the experiences, the situations, the systematicity of that kind of experience, then I think it will be a very poor simulation, most likely. If you're looking to study an experiment to study a particular phenomenon, I think you actually have to model that phenomenon, and not just expect it to fall out of a simulation.
So that's a pretty nuanced answer to your question. I think this medium can be quite important for experiments, but we have to be very careful in how we conduct the simulation and run the experiments. Let me just follow up quickly on this question. So this is Grayscale that you're talking about? Yes. Yeah, but I wanted to understand that a little bit better, since I haven't used the platform.
So was the definition of the ambivalent sexism something that emerged from the narrative or that was built in and narratives were used to simulate that sort of outcomes or neither? So Chimeria:Grayscale, it's an interactive narrative that was built using a patented platform that came out of my lab and my research. And that platform is used for, one, generating interactive narratives, and then also modeling how people are categorized in a nuanced way.
A lot of computational systems can model categorization, but usually, it's just a binary, like a set that you're either in or out, some set of fields that determine whether you're in the category or outside of the category.
What we know from the cognitive science work on categorization is that, in practice, it doesn't work that way, that we have degrees of membership in categories, multiple membership in categories that our category memberships fluctuate and change over time, over the course of our lives, and so forth. And so Chimera can model some of those types of phenomena. In this case, the protagonist works as an HR manager in a company called Grayscale, and they're faced with different dilemmas at work.
And those dilemmas can range from somebody comparing two different resumes of job applicants and using thinly-veiled language. They look like a real go-getter and a team player, dynamic person, or beautiful CV, they look like they'll be a real people person. And one might be the resume of someone with a name that's assumed to be male and the other one is assumed to be female.
We have a range of different kind of circumstances, people trying to regulate what other people are wearing in the workplace and these sorts of things. I see. And as you make those choices, each of those choices is aligned with the categories in the ambivalent sexism inventory. And they move you, as the player, as this manager, in and out of those multiple categories.
And then based upon the categories that you're exhibiting traits of, if you're seen as supporting ambivalent sexism in some forms, you have different outcomes and responses to the story. And at the very end, you get a performance review. And that performance review is narrative, and it's not just meant to be a boring training tool that sometimes people encounter, but an interesting story, where you can then reflect on what happened.
You might even have something like, well, things seem to be getting better there. People are pretty happy. But if this doesn't persist, we might have to go back to the old ways. So it might make you reflect on productivity within the company versus supporting anti-sexist measures and how they're judged against one another, and so forth. That kind of narrative, by the way, it plays out like an email system. It looks almost just like an email system, just a bit aestheticized and streamlined.
And you're selecting how you respond to emails. That's called an epistolary narrative. Epistolary narratives are those that are told through different fragments of communication, like diary entries, journal entries, emails, letters, those sort of things. So essentially, it's an interactive epistolary narrative that you're immersed in.
And the idea is that should be more engaging, narratively, than something which is just a training tool that tells you how to think because you're actually exploring the phenomenon. And then with debriefing, afterwards, after you're done with the experience, it can highlight and name the phenomena that you have just experienced. I have to say, it sounds amazing. I just the sexual harassment training at work thing. You watch two videos. It's the most boring thing ever.
You click through the answers. You answer some multiple choice questions that anyone with any common sense would get 100% on. And you get a score at the end of this miserable click through. And I learned nothing, and it was pointless. And so this sounds like 1,000 times better. But is there any way to actually measure it?
Is there any way to measure that this system, this Grayscale experiment, is better than the traditional harassment training videos with multiple choice questions that you click through at work? Well, you do run studies on this. And so, in that case, we were looking for a phenomenon, called perspective transformation, that comes from the learning sciences. And we have various ways of reading user data.
It can range from biometrics based on galvanic skin response, and heart rate, and those sort of things. We have surveys that we can use that are validated, instruments created by social scientists, and more. A lot of our work, we've relied on these surveys. And in the case of some of this work, perspective transformation, we're looking at, did people challenge their prior presuppositions that they came in with?
And the learning sciences theory on this, it suggests that there are multiple stages of prospective transformation, and essentially, we're looking to see if people transform their perspectives from when they began. And I mean, there are a lot of instruments. We've done work on discrimination in schools. So my former PhD student who graduated a year ago, Danielle Olsen, thesis was on VR simulation of racist experiences in education.
And that used a model called racial and ethnic socialization model. And in that case, it both assesses users to see what category of socialization that they've been raised through, that is, through parents, through media, through peers, and so forth. There were four categories in that case. One is called a colorblind category, where people say, I don't see race. It doesn't exist. We shouldn't talk about it.
There's another category, which is distrust of other people, which fortunately is the one that you see the least. But it is out there. I mean, distrust of people from different groups. You have prepared for bias. Sometimes people think about it these days as related to the talk as an example of it.
That's the talk that people in underrepresented or oppressed groups in society might say give their children when having to potentially deal with, say, biased authority figures, and say, you might experience this bias. Here's how you might want to comport yourself, and so forth.
And then the final one is pride and legacy reinforcement, which is a social acceptance and pride of not only your group, but the groups of others, and looking at the positive characteristics that might exist within those different groups. And so we essentially were able to, through this interactive experience, where the user interacts through selecting gestures for the character, assess how they were socialized to think about race from the experience.
And then it can also be an intervention to help them then reflect on and help with a kind of perspective change based upon them thinking through the issues as they go through the simulation. Yeah, so I have one last question from my side, and it has to do a little bit with the metaverse. As you know, at MIT, a lot of people have been talking about metaverse and the creation of, I would call it, augmented reality to navigate the complex world of the internet, and so forth.
Do you think that this is going to help us understand the world a little bit better, or is it going to isolate us even more from human beings and the rest of the world? And I mentioned this as an example, where recently, we visited Peru and we went to Machu Picchu. And it's a trek, and it's actually complicated to get to Machu Picchu. And the question is, I could have done a metaverse on this and got some idea of the whole place. Is that a good alternative?
How is that affecting us as human beings focusing so much more about this augmented reality? Actually, I will clarify, just to say, because I don't think it's just augmented reality. I think it's something like virtual worlds with social media, affordances, and networked computing all coming together. And you could think about the metaverse as something like this. And I think that, like a lot of things, it depends on the use.
My PhD advisor, Joseph Gauguin, was staunchly against what's known as technological determinism, the idea that any technology is a unilateral force that will shape the future of society, for better or for worse. Any technological determinist statement is false, is what he used to say. And people would argue, they'd say, but certainly, the cell phone, that's changed life. Or people might say, nowadays, certainly, the metaverse, that has changed society.
And in that case, there would be missing the key word in this, which is the unilateral side of it. And so the alternative way that I think about it, I think it's in line with thinkers like sociologists Bruno Latour and others, is it's a ecology. There's a network of different forces at work that includes people's, say, everyday physical relationships, the relationships online, the technology itself, the documentation of the technology, the way it's reported on in the media, and so forth.
You could say, the effects come from that whole network of forces. And I think that's what makes it a complex question to answer.
But if I wanted just to give a personal response, just my own feeling about it, is that, one, I think, if the experiences people have ultimately pushed people back to the physical world and physical world relationships, I think there's something about not making it a surrogate for physical world relationships and experiences, but something that loops you back towards those kind of experiences and needs. If it just absorbs you into something which is simulated, there's just so many dangers.
It can give the illusion of the experiences you would have in the physical world with all of that nuance, but really, it's highly regulated, created by a system of industry, and so forth. So the nutshell version is I would say that there's quite a danger that there, although, if designed in careful and sensitive ways, there also are profoundly positive opportunities that are available. And hopefully, many of the designers are cognitive of this issue when they design aspects of the metaverse.
I hope, increasingly so. I mean, it just depends on the designers. But they're getting to be some practices. I guess, you could say it's like in other areas of software engineering that there are many cases where people only think about individual users or only think about, let's say, the uptake of the platform, and so forth.
But there are strands of human-computer interaction and software engineering, where people think about social impact, build that into the process of design, work with various stakeholder groups, involved the stakeholder groups in the design process and evaluation process, and so forth. And I think it's similar. We begin to see some works that are developed hand-in-hand with community members.
Just now, I'm on a grant with researchers in Canada, and also, I mean, just internationally on indigenous cultures and artificial intelligence, and just thinking about the impacts on different global indigenous groups, and so forth. And so I think that there are an increasing number of initiatives that take this kind of work into account.
I think that's maybe one of the hallmarks of the Institute for Data Systems and Society at MIT, is just the fact that people are both looking at those social factors, as well as the information and data economy side of the work. If you use that as a parallel, you could ask, how many departments are doing something similar that take into account the technical and the social, and how many are focused primarily on just the end use in a more technical sense?
And that might be parallel to other types of software, like entertainment and metaverse, too. Thank you. So Fox, this is totally out there and your personal opinion on this, but in terms of what is the future of this, is there a world in which everyone walking around has virtual reality goggles on and everyone is living in this simulated world that's not physical, or is this just the stuff of movies?
I think if we extrapolate from where we are now, as everybody who has even a few years ago, we call it a smartphone, but now it feels almost redundant to say smartphone because everybody just has a phone that is of this sort carrying around a computer in your pocket. And so I think the idea that people will be networked, that people have virtual surrogates for themselves, different worlds to explore, and ways to engage with computational media, that certainly seems to be the case.
Whether that looks like something like a headset or glasses, and so on, I think that would remain to be seen. I think it will happen in a seamless kind of way, like the integration of the phone and the camera. At first, people thought these are just two devices. Why would they have to be paired together? And incrementally, people began to be used more and more to having different forms of mediation of life that you're carrying around with you.
But I think there's a problem with the headset, in the idea that, again, blocking the physical world and then replacing it with something else, but overlay in ways that seem banal initially.
But the same way that we scan a grocery store card and that connects us to a national or global network of data, and then that data transforms our experience of the store because it guides which products are going to be placed there next, and so forth, it's very easy to imagine picking up your phone and now you see information about those vegetables or whatever is in front of you. And then just incrementally, a bit more and more. Maybe advertising creeps in, and so forth.
So I think it will happen more like that, not a spectacle, in which, within a few years everybody is immersed in this global illusion. But we'll become less and less sensitized to the way that data is overlaid on our daily experiences.
And it might look not so much like being immersed in some exciting virtual world, like the Holodeck in Star Trek, The Next Generation, but it will be more like what we experience of going online now with our phones in the store or walking around, where you're just a little less connected to where you are. You have access to a lot of different types of information. You want to know a little bit about that musical instrument you're going to buy. So let me look it up.
But now, instead, you just see it laid on top. I could imagine things just going in that direction if I were to extrapolate in a fanciful way just based on where we are now. Thank you so much for doing this. We know how busy you are, so we really can't thank you enough. Thanks, Fox. Every time I listen to you, I learn something more about that word. Thanks. It was absolutely my pleasure. Thank you for listening to this month's episode of Data Nation.
You can get more information and listen to previous episodes at our website, idss.mit.edu or follow us on Twitter and Instagram @MITIDSS, @LibertyVittert, and @MuntherDahleh. If you like this podcast, please don't forget to leave us a review on Spotify, Apple, or wherever you get your podcasts. Thanks for listening to Data Nation from the MIT Institute of Data Systems and Society.