We’re All Being Played By Metrics

Hey, it's Flora Lichtman and you're listening to Science Friday. Today in the show Are you being played? Point systems are everywhere. Ready for movie night? Consult rotten tomatoes. Vetting a new pediatrician? How many stars do they have? And at work it's even more pervasive. There's KPIs, ROIs. Success has to be measurable. And you may ask, well, what's wrong with that? How else do you figure out if something is good or not, if something is working or not, if you don't score it?

Plus is all scoring created equal? Like scoring helps make games great, so what's that about? These are some of the big, messy questions that Dr. C. T. Nguyen has been wrestling with. He's a philosopher at the University of Utah and he just published his second book, The Score, How to Stop Playing Somebody Else's Game. T welcome to Science Friday. Hello, hello. I loved your book. I stayed up late in the night reading it.

Which is a big compliment'cause I get very sleepy and I had millions of thoughts and I know that philosophers love one sentence over simplifications. So If I had to boil the book down, I would call it hate the metric, don't hate the game. What do you think? That's better than anything I came up with. I I feel like you're buttering me up. But thank you. Okay, so I think it's gonna be music to people's ears to hear you rail again.

metrics, but I think the thing that's really interesting and that I want to talk about right off the bat is that You say that they're not just irritating, that they're actually powerful and that they can change us. Will you walk th walk us through that idea? Yeah. One of the Core experiences I had in my life was this kind of drift in my values, uh, where I started doing things like

Starting to diet, trying to get better at rock climbing, trying to get better at philosophy. And at first there was a scoring system nearby and I could mostly ignore it and be like, What I cared about? Is the interesting questions? Is feeling better in my body? And over time there's this drift. What happens with the drift is that I started caring deeply and immediately about the metric. So I've been calling this value capture.

And value captures what happens when your values are rich and subtle and you get put in a setting, typically an institutional setting. that presents you with some simplified and quantified version of your values. And then that starts to take over. You start caring immediately about getting more likes or getting more KPIs or getting higher in those rankings. Right. So instead of like making a great podcast, you try to optimize to get the most downloads as possible.

Um, the thing that's really worrying about value capture to me is when you get this slide and kind of like deep inside your soul, the thing that you think is important changes and your core expression of your value becomes likes, tweets, KPIs, page views, the status ranking of my university. Um another one of my favorite example, I I was talking about this stuff and a pastor said that uh Inside his church, they've constructed an internal leaderboard keeping track of baptism rates.

and that all the pastors were getting super competitive and that he'd found his own sermons changing and that he was no longer most attentive to like The spiritual growth of his flock, and he was trying to write like more poppy peppy sermons, it would suck in more people to get up his baptism rates.

Yeah. Okay, here's but on the flip side, what if your your values uh could be better. You know, because a lot of metric are are put in place to eliminate bias, for example. Like we talk about this in hiring as a way to help us be more objective or combat our own bad values. What do you think about that? I mean Here's I think

You started at the top by saying, you know, we all love to hate on metrics. And I think hating on metrics with this kind of like pure cold fury actually is kind of a cheap way out. Because it we we imagine Like you imagine this world this better world without metrics. I think the more painful

painful truth is to realize that metrics are both incredibly powerful and incredibly limited and that their power and their limitations are part and parcel of their core function. They offer us clear, simple, objective, unbiased counts, but that there's also an incredible price to them. One way to put it is that metrics are incredibly good at capturing the kinds of things that we can all count together easily. And I think the kinds of things they miss systematically are the kinds of things

That people are going to count differently, that requires some kind of discretion, some kind of judgment, kind of some kind of sensitivity. When I was thinking about bad metrics, One common response people have is to say, look. Maybe the problem is just this particular metric is locally bad. Maybe we can just find a better metric, right? Let's just get a more accurate way of counting good education or good health.

And I had this gut feeling in my heart that we weren't gonna get those kinds of metrics for certain kinds of things. That it was actually impossible. Yeah. That some things defy measurement. Yeah, the something defy measurement. I was I was actually sitting there in a conference about six or seven years ago, with a bunch of uh machine learning experts building the first generation of art making AIs.

And they were saying, Oh, we're do we're doing all this stuff to optimize for really good art making AI And I was the one philosopher of art in the room and I was like, Hold on, hold on, how are you defining Good art, right? Like what's your operationalization? It'd be amazing if they if those tech bros had figured that out. And what they said was, and this was really striking, they said, um, we're just using the Netflix database uh for engagement.

And I lost my mind. I said that's not that's not good art. That is um that's not good art. That is addiction. You're optimizing for addiction. And what they said back was Show us a better large scale database with a better metric for good art, and we'll use that instead. And I said, I don't think there's gonna be one. And that's I think the kind of path I've been going on. I've been trying to get find a philosophical account of why certain kinds of values will ultimately defy metricization.

I think I found the answer actually in a bunch of historians, particularly two amazing figures, Theodore Porter and Lorraine Dastin, and they really unlocked for me um the kind of core dynamic underneath metric. And it's actually, I mean, it's not about it's not about like pure science and the pure heart of what is countable or not. It's about the logic of bureaucracy. It's about how large scale institutions work together. Uh Theodore Porter puts it this way.

for quantitative forms of justification, even when they're really inappropriate. So what he says is that qualitative knowledge is rich and sensitive and open-ended and dynamic, but it travels really badly between context and So my favorite example as a professor and a teacher is the evaluations I write on my students' essays. But what I write is not going to be comprehensible. to a business school professor or a C S professor and what they write in their students won't be comprehensible to me.

So what Porter says is that when we make quantitative data, we pick a kind of chunk that's going to be steady and stable across context. We we find a chunk that everyone can understand, no matter their background, across the institution or across the world. So in education, that's letter grade, right? GPA. And I think you can kind of get a glimpse of what's going on there by just thinking about how different and how rich a qualitative evaluation is and how hard it is to understand at a distance.

and how quick and easy a letter grade is to understand, and how thin and simplified it needs to be. I felt like this was my like matrix the veil is ripped from my eyes moment was that metrics are powerful because they're designed to be stable across contexts, so we can all understand them and we can all collect into them. And that's what creates this kind of cross-cutting, massive, shareable piece of information.

But to make that we had to cut out the context, right? That is actually the design feature and the design bug in one. They are powerful because they're decontextualized. Right. Yeah, no, it's it's an interesting problem. I mean, y you know, so metrics are a way of scoring, right? And you talk a lot in the book about your own challenges with scoring and like getting sucked into the score, right?

How do you wake up from that? Like do you have a story that might confer some practical advice for our listeners? I mean, a lot of the stories involve I mean, I started doing philosophy out of love and because the questions were interesting. I mean

Philosophy is not a great thing to do with your life. I think all my parents and all my relatives thought I was throwing away a good education to go to philosophy, right? You're giving up on most things that might be a reasonable way to be a productive human adult to do this weird thing because I loved it.

And then I got I got professionalized. I went into graduate school and I learned that you had to that there was an internal ranking system in philosophy, and that internal ranking system uh was based on the status of the journal you published in. And high status journals really liked this very specific kind of extremely technical, extremely dry, extremely uh disattached from the world kind of work.

And so I started doing that and I became I kind of converted my life over to chasing this large scale scoring system and ended up kind of losing touch with the actual reason that I wanted to do philosophy and then I got miserable. I got super depressed. I almost quit the profession. And I think in my case, uh The signal was incredible. Boredom and despair.

Despair. Okay, good. I'll be on the lookout for that. Yeah, be on the lookout for despair. I mean there's uh there's a philosopher I love, Elijah Milgram, and one of the things he says is that boredom is this incredibly important signal because it tells you when you've picked bad values. It tells you when your values don't fit yourself and your context.

Oh that's a deep thought. I like that. And I think here here's another thought. If you listen to that, if you listen to that quiet voice, then you might be able to steer away. But If the clarity of the metrics by their clarity and their communicability is so strong that you let that Out shout, the kind of quiet voice of inner despair, uh, then you'll ignore that and you'll just keep following the metric.

Okay, we've been talking about, you know, some of the perils, complications of scoring, you know, for the first half of this conversation, but I love this twist.

In some context, You love scoring because you are game obsessed. And we're talking 90s European board games, social games. video games. And before we go to break, I just want to tell you what's coming next. We asked listeners to call and share stories about games that changed their lives. And like so many people had stories for us and were on board with this premise.

But not everybody. Yeah, this is rock and ravi in the drums. You know, I've been listening since about nineteen eighty seven to uh MPR and even unconsciously on my teachers car stereo probably in nineteen seventy seven. You freaking people with your video games, tell us about a video game that changed your life. And these Mambi Pambi topics for young kids who you know what changed my life? Here in Motown and Rock the Boat in nineteen seventy three or Led Zeppelin, the song remains the same.

Live. You know that album? No, you don't, just was uh hell of ticking music you're listening to. That's what changed my life. Let's up in live. Let's see if we can change Rock and Robbie's mind. I think I think I know it's a interruption wrong. Okay. Okay. Okay. Alright, after the break, what games can tell us about the meaning of life? Stay with us.

Okay, T, so for someone with, you know, serious grievances with scores and metrics, you also love games, does that mean you secretly love points? You know, this is This is actually where this entire book came from. And a lot of people, you know, when they talk about games, they'll talk about

fancy graphics and movie like dialogue and they'll talk about how much like movies they are and they won't talk about the thing I love about games, which is the freedom and the choice and the difficulty and the creativity. And when I was trying to understand that, I found this amazing

Moment from Reiner Knittia, my favorite board game designer. And what he said was that The most important design tool in his toolbox was the scoring system, because the scoring system tells people what to care about during the game. It sets their desires to And the thing that was super interesting to me that was that he had like calmly put his finger on this like weird deep part about games. Would you you can just open a game box up?

and read the rules and it tells you what to care about and suddenly you just do it, right? You can literally even my my wife and I will open up a board. Like amassing hotels. You're like, okay, I want the green ones. I gotta get the green ones.

Or building a better network of railway trades or killing each other or like and I mean it's so deep that you can open up a board game. My wife and I will open up a new board game and then it'll tell us whether we're cooperating to beat the game or whether we'll try to kill each other.

Right. And you take it and you're like, sure. Great. Thanks. Right. And with games, what this does is I think games are kind of What they're doing is they're shaping ourselves and our agency. They're telling us a different desire set to take on. And then they're designing, they're using that to design this incredibly interesting experience. Like so many board games will carefully design.

this intricate lock of what you're trying to achieve and what the constraints and methods are. And then suddenly you have to like do hyper careful logical calculation. Or you have to like try to guess what's in people's mind. Or, you know, in other parts of life you have to In my case, like climb a rock. A game rock climbing is a game that tells you to get to the top of the cliff using only your hands and feet. And then suddenly you're plunged into this life of the

of like balancing and delicacy. And the first thing I want to say, wait, who who is the person that was hating on you for loving video games? Rock and Robbie. Okay, rockin' Robbie. First claim. Games are bigger than video. My favorite definition of games comes from Bernard Suitz, who said playing a game is taking on a voluntary obstacle to make a certain kind of experience of struggling possible.

What he was really interested in was that sometimes we take on constraints specifically'cause we like that struggle. In a marathon, you could have taken a taxi, you don't, because you want to be running.

With climbing a cliff, I could have used a rope to like scale a tree, but I don't, right? I use my hands and my feet on the rock'cause that's more interesting. Here's another one, Rock and Robbie. Um I think You could have for in a lot of cases, you could have had more perfect music better by playing a recording or playing a written score, but a lot of the times we like to do it ourselves and we like to improvise it ourselves.

because the challenge is interesting, because the experience is interesting. One more thing for rock and robby. And I think this is really related to like my core deep weird fascination here, which is I'd written a whole book Talking about how scoring systems were beautiful in games. And then I was trying to write a bunch of papers about how scoring systems suck the life out of us in education and institutions. And I was trying to figure out why. And here's the first big difference.

It's fine that Rock'n'Robbie l hates video games. That's cool. That's because nothing in our normal lives, in our life with in most of our lives with games at least, forces us to play a particular game. Right. I can play ultra complicated strategic war games and someone else can play like chill zen out rhythm games and someone else can go fly fishing. And the interesting thing about games

is that you can maneuver through this ecosystem of different scoring systems and different constraints to find the action you love, right? And you can't with metrics. Metrics are inescapable.

Well speaking of of using constraints to make a beautiful experience, this is a game that a listener invented, and I want your take. Hello, my name is Connor. I'm calling from New Milford, Connecticut. When one of my dogs passed away. I decided that a good way to uh help my younger dog who was still alive was to hike on the Appalachian Trail.

So I started to take her out once or twice a week on the Appalachian Trail hiking and um eventually I realized that it was feasible for us to reach three hundred miles that year. um if I had just really focused. So I decided to make a spreadsheet and record all my heights and I just became absolutely obsessed with it. And uh

I have always loved hiking but this um this game where I was always checking the spreadsheet has just uh really changed my relationship with my dog and given me a new hiking companion. Amazing. I have so much to say about the story. So I think one of the interesting things about scoring systems is they can hyper motivate us and they can create this kind of very motivational clarity that can plunge us deeper into a kind of action.

Um, the question is, is it an action that you like or you don't? Right? Some scoring systems drag you into boring grinding action. Some scoring systems drag you into something that you find rich and fulfilling. And this story is

Like the ideal story. This is someone that added a scoring system, it created an experience that was better, and they acknowledge that. They're not being s they're not being sucked into an experience that they do not want, right? They're using the scoring system under control to give them the life they want. Yeah. Well here's another amazing game story. This story changed the way that this listener saw the world. Here's Elizabeth in Rochester, New York.

So one evening I was playing Yahtzee with this friend And he'd never played before. And we were at the point in the game where we had filled in all the categories and it was his turn and he only had to get Yahtzee. So he only had one role left. And he declares that he's gonna roll five sixes. And I was trying to stop him, I'm like, wait a minute, do you know what the odds are? And it's one in seven thousand seven hundred and seventy six. Or zero point zero one two eight six percent.

But there's no convincing him and so he scoops up all the dice and you guessed what happened. Otherwise I wouldn't be calling you about this. He got the five six And anyway, so the way that that changed my life is l any time I hear these crazy odds, like there's a zero point zero zero one percent chance that a giant meteor will hit the earth in the next few months

I think about this Yahtzee game and I'm like, it could happen. Yeah. Um I think one of the things that's really interesting in games is that we can take wild risks in them that we can't take elsewhere. Because they're detached from ordinary life. This is one of the questions. The stakes are actually low. The stakes are actually re I mean, so it this is not true of every game, right? Some games, y you know, if you're a professional sports player

There's real incentives. But for most of us, the game there there's this concept in game scholarship that games occur in a magic circle, and the idea is that. the things that happen in the game that we're trying for, the points, don't actually connect to anything in the outside world. And so we're kind of forced to be much more risk averse.

in the ordinary world because we're tied to various incentives, because those points in the real world are tied to things that we can't escape. They're they're tied to resources that we actually need. In games, right? The interesting thing about a game is you can set up a world, you can go all out, you can try to kill each other, you can try to take huge risks, you can win or lose. And then it just Evaporate afterwards.

The points don't actually matter. And this enables, right, this kind of wild risk taking. It enables the drama. Because if this person did that and they failed, it wouldn't actually matter. I think that's again one of these big differences between games and metrics. The scoring systems of games are temporary and disattached and distinct from ordinary life.

And the scoring systems and metrics there aren't. They're they're kind of inescapable, which means we can't have this kind of playful attitude towards them.

What's interesting to me is that games also can be attached to your real life. Here's another call we we got um about a game that many people have strong feelings about. Hi, this is Fred in Virginia. The game that has changed my life is Dungeons and Dragons. I found out about it when I was a young teenager. And it changed my life in I don't know how many ways. I learned about social interaction. I learned about probability.

I learned about storytelling. I ended up researching politics and geography and mythology and history. it sparked an entire scholarly Interest absolutely changed my life. Are you are you wiping tears from your eyes? Yeah, no, I mean I thi I I've had the same experience, I think this stuff is like You have? Yeah, yeah.

Some of the most holy and precious moments to me are in the space of Dungeons and Dragons and other tabletop role-playing games. For me in my book, Dungeons and Dragons players in particular, people that start modding the system are the real heroes. This is the opposite of bureaucracy is house rule dungeons and dragons. I think it really, I mean for me there's this one of the core questions behind all of this stuff is the question of what strict rules are doing for us and to us.

And you might think that strict rules push us away from play. I was trying to understand this, and I had a yoga teacher explain to me that. People think that total freedom is good, but if you give people total freedom of movement, they just repeat their normal habits and you need the strict rules of yoga to push you into a new posture and help you be flexible. And I think Dungeons and Dragons is one of the clearest cases where the strict rules

force you into a new habit of soul. They ask you to look at things from a different angle, to approach things from a different character, to approach things in a different world. Uh one of my favorite variants of Dungeons and Dragons. Um uh is Apocalypse World and it often depending on the character you play

it often restricts what you can do. Like one of the characters, the only thing you can do, you can't really attack people, you can't really make money. The only thing you can do is probe their soul for vulnerabilities. And if you're forced to play a character whose only way of moving through the world is getting people to open up about their vulnerabilities. This forces you into a completely different

angle on the world. And I think games are their most freeing when they let us like kind of force when they push us and encourage us to take on different angles. And they're they're least freeing when they lock us into a kind of single pervasive value system that we cannot leave and that we're kind of trapped in. A new habit of soul is such a beautiful turn of phrase. And also I feel like the D and D. people are like, Finally, someone gets it.

T your philosopher, uh I understand that a biggie for for y'all is understanding the meaning of life, so I thought you might enjoy this caller. Hi, this is Jenny. I'm calling from Arizona and wanted to share the story of my grandfather who did crossword puzzles every day. And I used to sit at the kitchen table while he did crossword puzzles at a very young age and he would ask me questions.

And when I got older I started delving back in and now I do the New York Times crossword puzzle daily, the Los Angeles T crossword puzzle daily. I love them all. And it just opens your world to so much to learn just what one word can be.

And you just keep looking and searching and that's what life is all about, finding the answers. So The secret truth is I've written a book about games and metrics, but it's actually secretly not that secretly, it's a book about the meaning of life.'Cause I think games and metrics actually teach you something really intensely about the meaning of life. Uh the philosopher of games that influenced me the most, Bernard Suit. At the end of his great book, he has this moment where he says

Imagine utopia, where we've solved all our practical problems. What would we do with our time? We would play games or we would be bored out of our minds. So games must be the meaning of life. What he's saying, it's actually an idea from Aristotle. What Aristotle thinks is that the meaning of life is in the actions we do, the exercise of our capacities, not our outcomes. And the the one

I think one of the things that the modern world has convinced us of is that the only thing of value are kind of portable outputs. And I think what suits reminds us of and what Aristotle was pushing was the idea that it's actually the process of inquiry, the figuring of things out, the exercise of your abilities, figuring out how to balance something, figuring out the puzzle. That is actually the thing that makes a human life valuable.

And I think we kind of all know that in a sense, and yet we've somehow been persuaded that a delightful, beautiful, fascinating, lovely process of doing an interesting puzzle that we should feel ashamed of that or that it's weird to love it because it doesn't result in some kind of portable outcome that you can like sell or process down the line.

There you have it, the meeting of life, right here on Science Friday. Doctor C. T. Nguyen is a philosopher at the University of Utah and author of The Score, How to Stop Playing Somebody Else's Game. T, thank you so much. This has been truly a pleasure. Thank you so much. I didn't realize but the format that I've always wanted was to riff on people calling in about the games they love. That's the ideal thing that I've always wanted to do.

To read an excerpt from Tees Book, head to our website, sciencefriday.com slash the score. This episode was produced by Rasha Aridi. Thank you all for listening and happy gaming, especially you, Rock and Robbie. I'm Flora Lixman. We'll see you next time.