Strategy. Innovation. Brand.

Critical Thinking

1 2 3 50

Are Machines Better Than Judges?

No bail for you.

Police make about 10 million arrests every year in the United States. In many cases, a judge must then make a jail or bail decision. Should the person be jailed until the trial or can he or she be released on bail? The judge considers several factors and predicts how the person will behave. There are several relevant outcomes if the person is released:

  1. He or she will show up for trial and will not commit a crime in the interim.
  2. He or she will commit a crime prior to the trial.
  3. He or she will not show up for the trial.

A person in Category 1 should be released. People in Categories 2 and 3 should be jailed. Two possible error types exist:

Type 1 – a person who should be released is jailed.

Type 2 – a person who should be jailed is released.

Jail, bail, and criminal records are public information and researchers can massively aggregate them. Jon Kleinberg, a professor of computer science at Cornell, and his colleagues did exactly that and produced a National Bureau of Economic Research Working Paper earlier this year.

Kleinberg and his colleagues asked an intriguing question: Could a machine-learning algorithm, using the same information available to judges, reach different decisions than the human judges and reduce either Type 1 or Type 2 errors or both?

The simple answer: yes, a machine can do better.

Klein and his colleagues first studied 758,027 defendants arrested in New York City between 2008 and 2013. The researchers developed an algorithm and used it to decide which defendants should be jailed and which should be bailed. There are several different questions here:

  1. Would the algorithm make different decisions than the judges?
  2. Would the decisions provide societal benefits by either:
    1. Reducing crimes committed by people who were erroneously released?
    2. Reducing the number of people held in jails unnecessarily?

The answer to the first question is very clear: the algorithm produced decisions that varied in important ways from those that the judges actually made.

The algorithm also produced significant societal benefits. If we wanted to hold the crime rate the same, we need only have jailed 48.2% of the people who were actually jailed. In other words, 51.8% of those jailed could have been released without committing additional crimes. On the other hand, if we kept the number of people in jail the same – but changed the mix of who was jailed and who was bailed – the algorithm could reduce the number of crimes committed by those on bail by 75.8%.

The researchers replicated the study using nationwide data on 151,461 felons arrested between 1990 and 2009 in 40 urban counties scattered around the country. For this dataset, “… the algorithm could reduce crime by 18.8% holding the release rate constant, or holding the crime rate constant, the algorithm could jail 24.5% fewer people.”

Given the variables examined, the algorithm appears to make better decisions, with better societal outcomes. But what if the judges are acting on other variables as well? What if, for instance, the judges are considering racial information and aiming to reduce racial inequality? The algorithm would not be as attractive if it reduced crime but also exacerbated racial inequality. The researchers studied this possibility and found that the algorithm actually produces better racial equity. Most observers would consider this an additional societal benefit.

Similarly, the judges may have aimed to reduce specific types of crime – like murder or rape – while de-emphasizing less violent crime. Perhaps the algorithm reduces overall crime but increases violent crime. The researchers probed this question and, again, the results were negative. The algorithm did a better job of reducing all crimes, including very violent crimes.

What’s it all mean? For very structured predictions with clearly defined outcomes, an algorithm produced by machine learning can produce decisions that reduce both Type I and Type II errors as compared to decisions made by human judges.

Does this mean that machine algorithms are better than human judges? At this point, all we can say is that algorithms produce better results only when judges make predictions in very bounded circumstances. As the researchers point out, most decisions that judges make do not fit this description. For instance, judges regularly make sentencing decisions, which are far less clear-cut than bail decisions. To date, machine-learning algorithms are not sufficient to improve on these kinds of decisions.


(This article is based on NBER Working Paper 23180, “Human Decisions and Machine Predictions”, published in February 2017. The working paper is available here and here. It is copyrighted by its authors, Jon Kleinberg, Himabindu Lakkaraju, Jure Lesovec, Jens Ludwig, and Sendhil Mullainathan. The paper was also published, in somewhat modified form, as “Human Decisions and Machine Predictions” in The Quarterly Journal Of Economics on 26 August 2017. The paper is behind a pay wall but the abstract is available here).

The Dying Grandmother Gambit

The government abandoned her!

I teach two classes at the University of Denver: Applied Critical Thinking and Persuasion Methods and Techniques. Sometimes I use the same teaching example for both classes. Take the dying grandmother gambit, for instance

In this persuasive gambit, the speaker plays on our heartstrings by telling a very sad story about a dying grandmother (or some other close relative). The speaker aims to gain our agreement and encourages us to act. Notice that thinking is not required. In fact, it’s discouraged. The story often goes like this:

My grandmother was the salt of the earth. She worked hard her entire life. She raised good kids and played by the rules. She never complained; she just worked harder. She worked her fingers to the bone but she was always the picture of health … until her dying days when our government simply abandoned her. As her health failed, she moved into a nursing home. She wanted to stay. She thought she had earned it. But the government did X (or didn’t do Y). As a result, my dying grandmother was abandoned to her fate. She was kicked to the curb like an old soda can. In her last days, she was a tiny, wrinkled prune. She couldn’t hear or see. She just curled up in her bed and waited to die. But our faceless bureaucrats couldn’t have cared less. My grandmother never complained. That was not her way. But she cried. Oh lord, did she cry. I can still see the big salty tears rolling slowly down her cheeks. Sometimes her gown was soaked with tears. How much did the government care? Not a whit. It would have been so easy for the government to change its policy. They could have cancelled X (or done Y). But no, they let her die. Folks, I don’t want your grandparents to die this way. So I’ve dedicated my candidacy to changing the government policy. If I can save just one grandma from the same fate, I’ll consider my job done.

So, do I tell my classes this is a good thing or a bad thing? It depends on which class I’m teaching.

In my critical thinking class I point out the weakness of the evidence. It doesn’t make sense to decide government policy on a sample of one. Perhaps the grandmother represents a broader population. Or perhaps not. We have no way of knowing how representative her story is.

Further, we didn’t meet the dear old lady. We didn’t directly and dispassionately observe her conditions. We didn’t speak to her caretakers. Or to those faceless bureaucrats. We only heard the story and we heard it from a person who stands to benefit from our reaction. She may well have embroidered or embellished the story.

Further, the speaker is playing on the vividness fallacy. We remember vivid things, especially things that result in loss, or death, or dismemberment. Because we remember them, we overestimate their probability. We think they’re far more likely to happen than they really are. If we invoke our critical thinking skills, we may recognize this. But the speaker aims to drown our thinking in a flood of emotions.

In my critical thinking class, I point out the hazards of succumbing to the story. In my persuasion class, on the other hand, I suggest that it’s a very good way to influence people.

The dying grandmother is a vivid and emotional story. It flies below our System 2 radar and aims directly at our System 1. It aims to influence us emotionally, not conceptually. It’s influential because it’s a good story. A story can do what data can never do. It can engage us and enrage us.

Further, the dying grandmother puts a very effective face on the issue. The issue is no longer about numbers. It’s about flesh and blood. We would be very hardhearted to ignore it. So we don’t ignore it. Instead, our emotions pull us closer to the speaker’s position.

So is the dying grandmother gambit good or bad? It’s neither. It just is. We need to recognize when someone manipulates our emotions. Then we need to put on our critical thinking caps.

Designing A Life — Fate and Gravity

Design constraint.

I was one of the taller kids in my high school class. I thought – and hoped – that I might use this size advantage to become a star basketball player.

Alas, it was not to be. I had a bad case of what’s often called “white guy’s disease”. Simply put, I couldn’t jump. Though I was over six feet tall, I could barely touch the rim even with my mightiest leap.

Van Jones would call this my fate. In a memorable commencement speech at Loyola New Orleans, Jones distinguished between fate and destiny. He defines fate as “those things that we have no control over” and suggests that the “people who are most miserable in life are the ones who spend their time cursing their fate.” (Click here for the video).

As it happens, the field of design thinking has a similar concept. Dave Evans, a design engineer, calls it the gravity problem. No matter how hard we try, we can’t change gravity. Indeed, we can’t even suspend it temporarily. Wouldn’t it be great to suspend gravity while we’re building a new house and then reinstate it when we move in? Unfortunately, we can’t. Time to move on. (For a podcast featuring Evans, click here).

Gravity is a fact of life. My inability to jump is a fact of my life. Instead of asking, “How can I change my fate?” it’s better to accept it and ask more useful questions. A useful question is one that we can actually do something about. A designer would say that we need to design around the constraints.

As Evans describes it, we’re looking for room to maneuver around the facts that define our products or our lives. I couldn’t jump very high. That’s a design constraint. So I might ask a different question: “How can I make basketball an important part of my life, even though I can’t play very well?” Once I ask the how can I question, I can dream up alternatives. I might become a coach. Or a sportscaster. Or I might decide to take up a sport that doesn’t require jumping.

Van Jones calls this destiny as opposed to fate. We have no control over fate. But we can respond to destiny. As Jones points out, “The world is not going to tell you every day about …” your destiny. We have to live our lives, and respond to our challenges, to discover our destiny.

Whether we call it destiny or design thinking, when we bump up against gravity, we need to change the question. By doing so, we can find an array of alternatives. Once armed with a list of alternatives, we can design a life or a product. Which alternatives fit the constraints? Which ones don’t?

We don’t design a product and then launch it. Rather we design it, then re-design it, then re-design it as we discover new constraints. Similarly, it’s difficult to design a life before we launch it. To overcome fate and discover our destiny, we need to design our lives as we live them.

How To Save Democracy

Back off! I know rhetoric.

Most historians would agree that the arts and sciences of persuasion – also known as rhetoric – originated with the Greeks approximately 2,500 years ago. Why there? Why not the Egyptians or the Phoenicians or the Chinese? And why then? What was going on in Greece that necessitated new rules for communication?

The simple answer is a single word: democracy. The Greeks invented democracy. For the first time in the history of the world, people needed to persuade each other without force or violence. So the Greeks had to invent rhetoric.

Prior to democracy, people didn’t need to disagree in any organized way. We simply followed the leader. We agreed with the monarch. We converted to the emperor’s religion. We believed in the gods that the priests proclaimed. If we disagreed, we were ignored or banished or killed. Simple enough.

With the advent of democracy, public life grew messy. We could no longer say, “You will believe this because the emperor believes it.” Rather, we had to persuade. The basic argument was simple, “You should believe this because it provides advantages.” We needed rules and pointers for making such arguments successfully. Socrates and Aristotle (and many others) rose to the challenge and invented rhetoric.

Democracy, then, is about disagreement. We recognize that we will disagree. Indeed, we recognize that we should disagree. The trick is to disagree without anger or violence. We seek to persuade, not to subdue. In fact, here’s a simple test of how democratic a society is:

What proportion of the population agrees with the following statement?

“Of course, we’re going to disagree. But we’ve agreed to resolve our disagreements without violence.”

It seems like a simple test. But we overlook it at our peril. Societies that can’t pass this test (and many can’t) are forever doomed to civil strife, violence, disruption, and dysfunction.

The chief function of rhetoric is to teach us to argue without anger. The fundamental questions of rhetoric pervade both our public and private lives. How can I persuade someone to see a different perspective? How can I persuade someone to agree with me? How can we forge a common vision?

Up through the 19th century, educated people were well versed in rhetoric. All institutions of higher education taught the trivium, which consisted of logic, grammar, and rhetoric. Having mastered the trivium, students could progress to the quadrivium – arithmetic, geometry, music, and astronomy. The trivium provided the platform upon which everything else rested.

In the 20th century, we saw the rise of mass communications, government sponsored propaganda, widespread public relations campaigns, and social media. Ironically, we also decided that we no longer needed to teach rhetoric. We considered it manipulative. To insult an idea, we called it “empty rhetoric”.

But rhetoric also helps us defend ourselves against mass manipulation, which flourished in the 20th century and continues to flourish today. (Indeed, in the 21st century, we seem to want to hone it to an even finer point). We sacrificed our defenses at the very moment that manipulation surged forward. Having no defenses, we became angrier and less tolerant.

What to do? The first step is to revive the arts of persuasion and critical thinking. Essentially, we need to revive the trivium. By doing so, we’ll be better able to argue without anger and to withstand the effects of mass manipulation. Reviving rhetoric won’t solve the world’s problems. But it will give us a tool to resolve problems – without violence and without anger.

Why Do Smarter People Live Longer?

What’s related to what?

A study published last week in the British Medical Journal states simply that, “Childhood intelligence was inversely associated with all major causes of death.”

The study focused on some 65,000 men and women who took the Scottish Mental Survey in 1947 at age 11. Those students are now 79 years old and many of them have passed away. By and large, those who scored lower on the test in 1947 were more likely to have died – from all causes — than those who registered higher scores.

This is certainly not the first study to link intelligence with longevity. (Click here, here, and here, for instance). But it raises again a fundamental question: why would smarter people live longer? There seem to be at least two competing hypotheses:

Hypothesis A: More intelligent people make better decisions about their health care, diet, exercise, etc. and — as a result — live longer.

Hypothesis B: whatever it is that makes people more intelligent also makes them healthier. Researchers, led by Ian Deary, call this hypothesis system integrity. Essentially, the theory suggests that a healthy system generates numerous positive outcomes, including greater intelligence and longer life. The theory derives from a field of study known as cognitive epidemiology, which studies the relationships between intelligence and health.

Hypothesis A focuses on judgment and decision making as causal factors. There’s an intermediate step between intelligence and longevity. Hypothesis B is more direct – the same factor causes both intelligence and longevity. There is no need for an intermediate cause.

The debate is oddly similar to the association between attractiveness and success. Sociologists have long noted that more attractive people also tend to be more successful. Researchers generally assumed that the halo effect caused the association. People judged attractive people to be more capable in other domains and thus provided them more opportunities to succeed. This is similar to our Hypothesis A – the result depends on (other people’s) judgment and there is an intermediate step between cause and effect.

Yet a recent study of Tour de France riders tested the notion that attractiveness and success might have a common cause. Researchers rated the attractiveness of the riders and compared the rankings to race results. They found that more attractive riders finished in higher positions in the race. Clearly, success in the Tour de France does not depend on the halo effect so, perhaps, that which causes the riders to be attractive may also cause them to be better racers.

And what about the relationship between intelligence and longevity? Could the two variables have a single, common cause? Perhaps the best study so far was published in the International Journal Of Epidemiology last year. The researchers looked at various twin registries and compared IQ tests with mortality. The study found a small (but statistically significant) relationship between IQ and longevity. In other words, the smarter twin lived longer. Though the effects are small, the researchers conclude, “The association between intelligence and lifespan is mostly genetic.”

Are they right? I’m not (yet) convinced. Though significant, the statistical relationship is very small with r = 0.12. As noted elsewhere, variance is explained by the square of r. So in this study, IQ explains only 1.44% (0.12 x 0.12 x 100) of the variance in longevity. That seems like weak evidence to conclude that the relationship is “mostly genetic”.

Still, we have some interesting research paths to follow up on. If the theory of system integrity is correct, it could predict a whole host of relationships, not just IQ and longevity. Attractiveness could also be a useful variable to study. Perhaps there’s a social aspect to it as well. Perhaps people who are healthy and intelligent also have a larger social circle (see also Dunbar’s number). Perhaps they’re more altruistic. Perhaps they are more symmetric. Ultimately, we may find that a whole range of variables depend partially – or perhaps mainly – on genetics.

1 2 3 50
My Social Media

YouTube Twitter Facebook LinkedIn

Newsletter Signup