Great Reviews and Terrible Tacos - Sharpening Substitute Questions with Counterfactuals

Substitute questions on the showbook. The idea of a substitute question Is that you'll take a cognitively taxing question and replace it Without your own realization of this replacement, you'll replace it with an easier to answer question. This question operates as a heuristic. on a way of answering something close to, something approximating or pointing towards the original concern.

So for example, you might ask the question, what is a good restaurant in my area? And then you'll substitute the question, what restaurant has good reviews in my area? Now, for many people, these are synonymous questions. They believe that a restaurant that has good reviews is entirely going to be representative of a restaurant that is good.

The difficult part is that defining what a good restaurant is is cognitively taxing. That is It's very hard to take in all of the variables that might be necessary to define good for a given person. Very often we will substitute questions by looking at historical answers to the same question rather than future casting answers. In other words, what do you want out of your career may be answered in multiple ways. For example, what have you enjoyed in your career previously?

Or another substitute question might be, what have I imagined my career might look like? This imagination, this visioning of your career, very often turns into what you expect from your career. This is true in a lot of our life experiences. We begin to desire or expect the things that we imagine are most likely to happen. There's a lot of reasons for this. Uh our desire for stability, for example. Additionally, the pain that we experience when something unexpected occurs.

Amazingly, sometimes this pain is felt even when the unexpected thing is a positive thing, something that we otherwise may be able to, on many objective measures, say was a good occurrence.

These substitutions happen all the time, and sometimes we do them consciously as well. For example, We substitute a very difficult or perhaps impossible to answer question like, will I enjoy this car if I purchase it now? Will I enjoy it? How long might I enjoy it? Another example of this might be uh will this person, this candidate that I'm considering hiring, will they do well in their role? These are questions about the future, questions about uncertainty.

And so what we do instead of trying to answer these impossible to answer questions is we break them down into various criteria that we hope correlates to an answer. We try to imagine what kind of thing will I appreciate about a car in the future? And does this car match that kind of thing? What kinds of things predict whether a candidate will be successful?

All of these criteria that we are using are various types of substitute questions or a substitute operation, multiple questions that we are substituting. To try to approximate a belief or an assertion about another question. So y you could kind of formulate uh the substitution as If you had a question X You're going to instead answer question why, because it's much easier to answer. And you're going to say, because of answer y, I believe the answer to x.

is something. Right. I believe that because there are a lot of reviews, positive reviews for this restaurant, then so that's question or uh that's that's the substitute question, question why. Uh since there are a bunch of reviews, positive reviews, then my belief about question X, which I'm not going to try to answer directly, but it's instead I'm going to use the information from question Y. My belief

Is that it is a good restaurant. Right? Uh uh my answers to all of the various criteria for this candidate are XYZ, right? And therefore, because of the criteria question answers, my belief about their potential is that they will do well. Now of course there are plenty of ways that this can go wrong. There are plenty of reasons why our heuristics are not always tuned perfectly.

And I want to give you a tool, a very simple tool, that you can use to try to check your own heuristics, try to check your own substitute questions. uh those substitutions to determine whether your heuristics are actually meaningful or not. Right, so in in effect what you want to do is determine how well correlated

Your substitution is with the thing you're trying to substitute. That is the fundamental task at hand. You're trying to determine if your substitute question has a high cohesion to the question that you really care about. So let's take our restaurant review example. This restaurant has a bunch of good reviews and therefore I believe it is a good record. What I want you to do is ask the question. Is that necessarily true?

Right, so that's gonna give you a very clear uh cohesion if it is a one hundred percent cohesion. Right. So then really your substitute question isn't really a substitute question. It's more like an evidence question or a measurement question. It has a higher cohesion rate because it's not really uh, you know, uh asking a different question. It's just asking a different way.

Okay, this is kind of what our brains are tricking us into believing we are doing with all of these substitute questions. But uh if if there is a possibility that that is not necessarily true. Right, that uh a a restaurant that has a bunch of good ratings may not necessarily be a very good restaurant. The way you arrive at this. is called counterfactual thinking.

All right, so this is a technique. Uh there's a bunch of ways you could do this, but I'm going to talk about counterfactual thinking in today's episode. And the basic idea of counterfactual thinking here is what else could be true? So instead of trying to find why this thing might be false, You are instead providing an alternative explanation. What else could be true? What is another good explanation for why there may be a bunch of good reviews?

Perhaps the restaurant uh rewards people for leaving a good review. Maybe they have hired a bunch of uh review farming uh you know, uh a bunch of people to leave reviews. um that never even ate at the restaurant. There's a bunch of possible counterfactual Um and as it turns out, there are plenty of opportunities for those sites, for example, that host those reviews to try to reduce those counterfactuals. If you've ever seen the kind of like verified buyer.

uh reviews. This requires that the person who's leaving review has actually bought the item. Right. So uh that's trying to cut down on some of the counterfactuals to increase the cohesion between that uh substitute question, is this product a good product?

with uh the the substitute question for that, you know, core question is how many positive reviews or what are the star ratings, you know, on Amazon or whatever, right? So increasing that cohesion rate, we're gonna address some of the counterfactual What do you want in your career?

This one's a little bit more complicated. Right. It's it's a little bit harder to come up with counterfactuals here because you may say, Well, I w I wanna continue making money, I wanna continue being being better at my job, I wanna uh you know, get a get another promotion. All of these things are hard to say, well, what else could be true? That's not really how we would do a counterfactual in this situation. Instead, what we would do is we would play uh a few thought experiments out.

Right, so let's say that your intent is to uh to get a promotion. So a thought experiment might be, would you want a promotion if it did not include a pay rate? Most people would say, well, but that's not realistic. And thought experiments fortunately don't have to be realistic. The whole idea here is to produce some kind of counterfactual thinking.

In other words, you are Uh kind of poking at the question and you're saying okay you you mentioned that you want a promotion is that actually the thing you want is that precise enough is the thing that you want a promotion and a pay rate And if you were to say, okay, well I I do want a promotion and I do want a pay raise, another thought experiment you might run is are you willing to work sixty hours a week for a promotion and a pay raise?

Well the answer might be, well, of course not. Uh nobody would ever make me do that. Once again, it's a thought experiment. So you can explore to find out more and now you might adjust your assertion. That you want a promotion, a pay raise, and a balanced work uh environment such that you don't have to work more than forty, forty-five hours a week, something.

Uh or it it may be that through this exploration you realize, you know what, actually, I don't really care so much about the pay raise. I don't care so much about the promotion, what I really want is to reduce the overall amount of work I do. I'm actually okay with how much I'm getting paid, but I'd like to work less and get paid the same amount. There's a totally different career goal.

And so through this thought experiment exploration, you may realize that actually what you what you really want is to give get paid more per hour, not a total amount more. So when you're looking at these assertions and you're looking at these substitute questions that you ask yourself. Counterfactual thinking either through that what else might be true uh frame, the that's a little bit easier in that first example about you know restaurants or product reviews, what else might be true?

and counterfactual thinking through the lens of a thought experiment. These are both going to provide you insight. For talk about hiring, for example. You can imagine the feedback for a good candidate coming in that says he immediately solved the coding problem. Therefore, I believe he is a good engineer. She asked great questions, therefore I believe she is a good communicator. These things are substitute criteria. Right, you're trying to evaluate whether someone is a good uh communicator.

And you're using a very simplified uh set of criteria to try to gain signal. If you've been interviewing for very long at all, you've heard that word over and over. signal in this case is some kind of indicator Again, this is a heuristic, some kind of indicator that tells you another thing about a person. If you were to simply ask a person, are you a good communicator? They're always going to say yes.

So that's not a good indicator. You instead you're looking for a signal. You're looking for some kind of sign. that this person is a good communicator. And so you might use the heuristic They asked a question. They asked a good question, a thoughtful question, something that wouldn't necessarily come to mind. They are a thoughtful person. They are a good communicator.

But all of these are heuristic assumptions and even the good communicator criteria ends up being yet another substitution for will this person communicate well when it really matters in their job? So let's think about a couple of these interview question uh or outcomes that we could apply a counterfactual. Let's say that you had an interviewee that did poorly on let's say like elite code style questions.

Now, the immediate assumption is this person is not going to cut it. They don't have the chops that they need to have as an engineer. But one counterfactual and actually a pretty compelling counterfactual is that this person has not been practicing leak code in quite some time. Instead, they've been working. They've been doing not uh you know actual engineering work, which very rarely actually requires the kinds of skills that you use when doing leak code interviews.

So in an odd way, counterfactual thinking may inform us that people who do exceedingly well at leak code interviews are good at well, leak code interviews. They may actually have a negative signal or uh perhaps even a neutral signal here. When somebody is particularly good at leak code. Now, it tells us something. It doesn't necessarily tell us only bad or only neutral things. There may also be some good things.

There's a signal here that the person is willing to put in effort, for example. There's also signal here that might suggest that the person is able to comprehend. complex topics, leak code can often be very complex. And so if they can comprehend a leak code complexity, then they probably can also comprehend the complexity of whatever is in your domain model.

But the leak code measurement itself, as studies have shown, and this is kind of interesting, studies have shown that it doesn't really predict future job performance. So if your goal as an interviewer is to answer that core question, how well we're will this person do in their job, then it turns out that leak code as a criteria Is probably a bad substitute question. Success on a leak code interview may not be predictive of success on the job at all.

So what these counterfactuals do is they help you hone in on better substitutes. Now it would be unrealistic for us to say you need to answer the core question. The only way to answer the question of whether somebody will s succeed on the job is to hire them. And the whole purpose of of having the criteria that we have in an interview interviewing process.

is to actually perform the substitution in a high cohesion manner. In other words, we want to find substitute criteria that allows us to test this criteria In advance he had in the cheapest way and the highest signal way as possible. That means getting rid of your leak code questions, most likely. That's a low signal or a mixed signal way of looking at your candidate. And we can use counterfactuals to drive what kinds of things should we be talking about, what kinds of things should we discuss?

If you're looking at a candidate and you're trying to understand why they behaved in a particular way, you're looking at feedback from the candidate, ask what else might be true. Is it possible that the thing that is being presented in this feedback about this candidate, that there's actually more to the story? There's another reason. There's another reason why they got this feedback.

Now once you've done the counterfactual, here's a critical part of this and I don't want you to uh you know, miss this particular aspect. There are potentially many, many, many explanations. There are many explanations. It's possible that that person was extremely distracted. Possible, right? Uh if somebody is distracted in an interview, they're not gonna do as well as they would if they were fully focused. Now you should ask yourself next, how likely?

For my counterfactual, how likely is it that that was the case? Now, whether or not your likelihood percentage or or your ratio or whatever you want to use here in terms of the metric itself. If you're saying, Oh, it's you know twenty percent, twenty five percent likely, or it's seventy five percent likely These ratios, these thresholds, whatever you want to call these, are fairly arbitrary. It's not necessarily the case uh that you should care about any particular percentage.

You may have something that you care about at a 0.001%, right? Think of the justice system. Counterfactuals are very important when you're sitting on a jury. Because of this simple phrase beyond the shadow of a doubt. What percentage would we assign to the shadow of a doubt? If there is some counterfactual that has even a point one percent likelihood, then there might be what some would call reasonable doubt in that situation.

And so the the point of this is not to identify a particular percentage, but instead to recognize what percentage you begin to care about. If I have an otherwise qualified candidate for uh a particular role and I run some counterfactuals and I think, okay, this person failed one of uh you know multiple interviews. Right. They they failed one of many interviews. Otherwise they're fairly well qualified. And I'm gonna look at the interview feedback and I'm gonna try to determine

Counterfactuals, is something else possibly true? Maybe one out of five, one out of three chance that something else is true? And I'm looking at all of the information together. I may make the decision that this interview itself used the wrong substitute question. We took something to mean one thing when actually the counterfactual is strong enough. that we shouldn't have taken it to mean that thing.

We need to ask more higher cohesion questions. We need to have closer cohesion between our substitute question, our heuristics. And the core thing we're trying to figure out, the criteria in this case. For a candidate we're trying to hire. Thanks so much for listening to today's episode of Developer T. I hope you will begin to think more critically.

about your substitute questions, the cohesion rate between your substitute question, your target question, or your uh, you know, your underlying evaluation, the things that you actually care about versus the cognitively cheaper thing that you're asking. And try to come up with better, uh, better questions by using counterfactuals. This is a way to improve your hiring process.

pretty drastically. It's a way to improve uh the way that you think about uh you know your e even debugging, right? Uh we can end up falling prey to a bunch of biases around debugging, uh you know, confirmation bias. We believe that a particular bug is caused by one thing. It turns out it's caused by something totally different. But As we were debugging, we substituted uh a bunch of kind of debugging steps for our confidence, right?

Uh, instead of debugging properly, we used confidence, we used our pre existing belief to trace down a particular path. Turns out that path was wrong, and so we fell prey to confirmation. But we could have used counterfactual. Could have used counterfactuals in the in that situation. What else might be true and how likely is it that that thing might be true? Uh and and that could help us avoid these. these poor pathways of uh of biased logic.

Thanks so much for listening. Um hopefully this this thoughtful uh this conversation is thoughtful to you and and that you would uh adopt some of these ideas into your workflows. Um I'd love to hear about it. You can reach out to me on the Developer T Discord. Head over to developer T dot com slash Discord. If you enjoyed this episode, I'd encourage you to share it with somebody that you think will benefit from counterfactual thinking, will benefit from learning about this.

Certainly we will talk about it. Yeah. Whatever it is, it's not a little bit Yeah. Thanks so much for listening and