Strategy. Innovation. Brand.
Are Machines Better Than Judges?
No bail for you.
Police make about 10 million arrests every year in the United States. In many cases, a judge must then make a jail or bail decision. Should the person be jailed until the trial or can he or she be released on bail? The judge considers several factors and predicts how the person will behave. There are several relevant outcomes if the person is released:
- He or she will show up for trial and will not commit a crime in the interim.
- He or she will commit a crime prior to the trial.
- He or she will not show up for the trial.
A person in Category 1 should be released. People in Categories 2 and 3 should be jailed. Two possible error types exist:
Type 1 – a person who should be released is jailed.
Type 2 – a person who should be jailed is released.
Jail, bail, and criminal records are public information and researchers can massively aggregate them. Jon Kleinberg, a professor of computer science at Cornell, and his colleagues did exactly that and produced a National Bureau of Economic Research Working Paper earlier this year.
Kleinberg and his colleagues asked an intriguing question: Could a machine-learning algorithm, using the same information available to judges, reach different decisions than the human judges and reduce either Type 1 or Type 2 errors or both?
The simple answer: yes, a machine can do better.
Klein and his colleagues first studied 758,027 defendants arrested in New York City between 2008 and 2013. The researchers developed an algorithm and used it to decide which defendants should be jailed and which should be bailed. There are several different questions here:
- Would the algorithm make different decisions than the judges?
- Would the decisions provide societal benefits by either:
- Reducing crimes committed by people who were erroneously released?
- Reducing the number of people held in jails unnecessarily?
The answer to the first question is very clear: the algorithm produced decisions that varied in important ways from those that the judges actually made.
The algorithm also produced significant societal benefits. If we wanted to hold the crime rate the same, we need only have jailed 48.2% of the people who were actually jailed. In other words, 51.8% of those jailed could have been released without committing additional crimes. On the other hand, if we kept the number of people in jail the same – but changed the mix of who was jailed and who was bailed – the algorithm could reduce the number of crimes committed by those on bail by 75.8%.
The researchers replicated the study using nationwide data on 151,461 felons arrested between 1990 and 2009 in 40 urban counties scattered around the country. For this dataset, “… the algorithm could reduce crime by 18.8% holding the release rate constant, or holding the crime rate constant, the algorithm could jail 24.5% fewer people.”
Given the variables examined, the algorithm appears to make better decisions, with better societal outcomes. But what if the judges are acting on other variables as well? What if, for instance, the judges are considering racial information and aiming to reduce racial inequality? The algorithm would not be as attractive if it reduced crime but also exacerbated racial inequality. The researchers studied this possibility and found that the algorithm actually produces better racial equity. Most observers would consider this an additional societal benefit.
Similarly, the judges may have aimed to reduce specific types of crime – like murder or rape – while de-emphasizing less violent crime. Perhaps the algorithm reduces overall crime but increases violent crime. The researchers probed this question and, again, the results were negative. The algorithm did a better job of reducing all crimes, including very violent crimes.
What’s it all mean? For very structured predictions with clearly defined outcomes, an algorithm produced by machine learning can produce decisions that reduce both Type I and Type II errors as compared to decisions made by human judges.
Does this mean that machine algorithms are better than human judges? At this point, all we can say is that algorithms produce better results only when judges make predictions in very bounded circumstances. As the researchers point out, most decisions that judges make do not fit this description. For instance, judges regularly make sentencing decisions, which are far less clear-cut than bail decisions. To date, machine-learning algorithms are not sufficient to improve on these kinds of decisions.
(This article is based on NBER Working Paper 23180, “Human Decisions and Machine Predictions”, published in February 2017. The working paper is available here and here. It is copyrighted by its authors, Jon Kleinberg, Himabindu Lakkaraju, Jure Lesovec, Jens Ludwig, and Sendhil Mullainathan. The paper was also published, in somewhat modified form, as “Human Decisions and Machine Predictions” in The Quarterly Journal Of Economics on 26 August 2017. The paper is behind a pay wall but the abstract is available here).
Managing Agreement: The Abilene Paradox.
I want to be a team player, but….
I used to think it was difficult to manage conflict. Now I wonder if it isn’t more difficult to manage agreement.
A conflicted organization is fairly easy to analyze. The signs are abundant. You can quickly identify the conflicting groups as well as the members of each. You can identify grievances simply by talking with people. You can figure out who is “us” and who is “them”. Solving the problem may prove challenging but, at the very least, you know two things: 1) there is a problem; 2) its general contours are easy to see.
When an organization is in agreement, on the other hand, you may not even know that a problem exists. Everything floats along smoothly. People may not quiver with enthusiasm but no one is throwing furniture or shouting obscenities. Employees work and things get done.
The problem with an organization in agreement is that many participants actually disagree. But the disagreement doesn’t bubble up and out. There are at least two scenarios in which this happens:
- The Abilene Paradox – in the original telling, four members of a family in Coleman, Texas drove 53 miles to Abilene in a car without air conditioning in 104-degree heat to have dinner at a crummy diner. After driving 53 miles back, they ‘fessed up: not one of them had wanted to go. Each person thought the others wanted to go. They agreed to be agreeable. (A variant of this is known as the risky shift).
Similar paradoxes arise in organizations all the time. Each employee wants to be seen as a team player. They may have reservations about a decision but — because everyone else agrees or seems to agree — they keep quiet. Perhaps nobody agrees to a given project but they believe that everyone else does. Perhaps nobody wants to work on Project X. Nevertheless, Project X persists. Unlike a conflicted organization, nobody realizes that a problem exists.
- Fear – in organizations where failure is not an option, employees work hard to salvage success even from doomed projects. Admitting that a project has failed invites punishment. Employees happily throw good money after bad, hoping to snatch victory from the jaws of defeat. Employees agree that failure must be delayed or hidden.
The second scenario is perhaps more dangerous but less common. A fear-based culture – if left untreated – will eventually corrupt the entire organization. Employees grow afraid of telling the truth. The remedy is easy to discern but hard to execute: the organization needs to replace executive management and create a new culture.
The Abilene paradox is perhaps less dangerous but far more common. Any organization that strives to “play as a team” or “hire team players” is at risk. Employees learn to go along with the team, even if they believe the team is wrong.
What can be done to overcome the Abilene paradox in an organization? Rosabeth Moss Kanter points out that there are two parts to the problem. First, employees make inaccurate assumptions about what others believe. Second, even though they disagree, they don’t feel comfortable speaking up. A good manager can work on both sides of the problem. Kanter suggests the following:
- Debates – include an active debate in all decision processes. Choose sides and formally air out the pros and cons of a situation. (I’ve suggested something similar in the decision by trial process).
- Assign devil’s advocates and give them the time and resources to develop a real position.
- Encourage organizational graffiti – I think of this as the electronic equivalent of Hyde Park’s Speaker’s Corner – a place where people can get things off their chests.
- Make confronters into heroes — even if you disagree with the message, reward the process.
- Develop a culture of pride – build collective self-esteem, not just individual self-esteem. We’re proud of what we have, including the right (or even the obligation) to disagree.
The activities needed to ward off the Abilene paradox are not draconian. Indeed, they’re fairly easy to implement. But you can only implement them if you realize that a problem exists. That’s the hard part.
Three Decision Philosophies
I’ll use the rational, logical approach for this one.
In my critical thinking classes, students get a good dose of heuristics and biases and how they affect the quality of our decisions. Daniel Kahneman and Amos Tversky popularized the notion that we should look at how people actually make decisions as opposed to how they should make decisions if they were perfectly rational.
Most of our decision-making heuristics (or rules of thumb) work most of the time but when they go wrong, they do so in predictable and consistent ways. For instance, we’re not naturally good at judging risk. We tend to overestimate the risk of vividly scary events and underestimate the risk of humdrum, everyday problems. If we’re aware of these biases, we can account for them in our thinking and, perhaps, correct them.
Finding that our economic decisions are often irrational rather than rational has created a whole new field, generally known as behavioral economics. The field ties together concepts as diverse as the availability bias, the endowment effect, the confirmation bias, overconfidence, and hedonic adaptation to explain how people actually make decisions. Though it’s called economics, the basis is psychology.
So does this mean that traditional, rational, statistical, academic decision-making is dead? Well, not so fast. According Justin Fox’s article in a recent issue of Harvard Business Review, there are at least three philosophies of decision-making and each has its place.
Fox acknowledges that, “The Kahneman-Tversky heuristics-and-biases approach has the upper hand right now, both in academia and in the public mind.” But that doesn’t mean that it’s the only game in town.
The traditional, rational, tree-structured logic of formal decision analysis hasn’t gone away. Created by Ronald Howard, Howard Raiffa, and Ward Edwards, Fox argues that the classic approach is best suited to making “Big decisions with long investment horizons and reliable data [as in] oil, gas, and pharma.” Fox notes that Chevron is a major practitioner of the art and that Nate Silver, famous for accurately predicting the elections of 2012, was using a Bayesian variant of the basic approach.
And what about non-rational heuristics that actually do work well? Let’s say, for instance, that you want to rationally allocate your retirement savings across N different investment options. Investing evenly in each of the N funds is typically just as good as any other approach. Know as the 1/N approach, it’s a simple heuristic that leads to good results. Similarly, in choosing between two options, selecting the one you’re more familiar with usually creates results that are no worse than any other approach – and does so more quickly and at much lower cost.
Fox calls this the “effective heuristics” approach or, more simply, the gut-feel approach. Fox suggests that this is most effective, “In predictable situations with opportunities for learning, [such as] firefighting, flying, and sports.” When you have plenty of practice in a predictable situation, your intuition can serve you well. In fact, I’d suggest that the famous (or infamous) interception at the goal line in this year’s Super Bowl resulted from exactly this kind of thinking.
And where does the heuristics-and-biases model fit best? According to Fox, it helps us to “Design better institutions, warn ourselves away from dumb mistakes, and better understand the priorities of others.”
So, we have three philosophies of decision-making and each has its place in the sun. I like the heuristics-and-biases approach because I like to understand how people actually behave. Having read Fox, though, I’ll be sure to add more on the other two philosophies in upcoming classes.
Arguments and Sunk Costs
Why would I ever argue with her?
Let’s say that Suellen and I have an argument about something that happened yesterday. And, let’s say that I actually win the argument. (This is highly theoretical).
The Greek philosophers who invented rhetoric classified arguments according to the tenses of the verbs used. Arguments in the past tense are about blame; we’re seeking to identify the guilty party. Arguments in the present tense are about values – we’re debating your values against mine. (These are often known as religious arguments). Arguments in the future tense are about choices and actions; we can decide something and take action on it. (Click here for more detail).
In our hypothetical situation, Suellen and I argue about something in the past. The purpose of the argument is to assign blame. I win the argument, so Suellen must be to blame. She’s at fault.
I win the argument so, bully for me. Since I’ve won, clearly I’m not to blame. I’m not the one at fault. I’m innocent. Maybe I do a little victory dance.
Let’s look at it from Suellen’s perspective. She lost the argument and, therefore, has to accept the blame. How does she feel? Probably not great. She may be annoyed or irritated. Or she might feel humiliated and ashamed. If I try to rub it in, she might get angry or even vengeful.
Now, Suellen is the woman I love … so why would I want her to feel that way? If someone else made her feel annoyed, humiliated, or angry, I would be very upset. I would seek to right the wrong. So why would I do it myself?
Winning an argument with someone I love means that I’ve won on a small scale but lost on a larger scale. I’ve come to realize that arguing in the past tense is useless. Winning one round simply initiates the next round. We can blame each other forever. What’s the point?
In the corporate world, the analogue to arguing in the past tense is known as sunk costs. Any good management textbook will tell you to ignore sunk costs when making a decision. Sunk costs are just that – they’re sunk. You can’t recover them. You can’t redeem them. You can’t do anything with them. Therefore, they should have no influence on the future.
Despite the warnings, we often factor sunk costs into our decisions. We don’t want to lose the money or time that we’ve already invested. And, as Roch Parayre points out in this video, corporations often create perverse incentives that lead us to make bad decisions about sunk costs.
Our decisions are about the future. Sunk costs and arguments about blame are about the past. As we’ve learned over and over again, past performance does not guarantee future results. We can’t change the past; we can change the future. So let’s argue less about the past and more about the future. When it comes to blame or sunk costs, the answer is simple: Don’t cry over spilt milk. Don’t argue about it either.
My Social Media
