If taking one vitamin pill per day is good for you, then taking two pills per day should be better, shouldn’t it? If two is better than one, then three should be better than two.
If we continue to follow this logic, we’ll soon be taking a whole bottle of vitamins every day. Why don’t we? Because of two limiting factors:
It’s easy to figure this out for simple items like vitamin pills. But, in more complex decisions, we tend to have a linear bias. If there’s a linear relationship between two variables, we assume that the line continues forever.
Let’s take schools, for instance. In the United States, we’re obsessed with measuring student performance in schools and tracking it over time. We create performance tables to identify the schools that provide the best education and those that provide the worst.
You may notice that small schools are often at the top of the charts. You might conclude that school size and performance are linearly related. It might be wise to build more small schools and fewer large schools. Unfortunately, you’re suffering from linear bias.
To find the error, just look at the bottom of the performance charts. You’ll probably find many small schools there as well. Small schools dominate the top and bottom of the chart; large schools tend to fall into the middle range.
What’s going on? It’s the variability of small samples. If you flip a coin ten times, you might get eight tails. If you flip the same coin a thousand times, it’s very unlikely that you’ll get 800 tails. With larger samples, things tend to even out.
The same happens in schools. Larger schools have larger samples of students and their performance tends to even out. Performance in small schools is much more variable, both upward and downward. The relationship between school size and performance is a curve, not a straight line.
For the same reason, I was briefly (but gloriously) the most accurate shooter on my high school basketball team. After three games, I had taken only one shot, but I made it! In other words, I made 100% of my shots – the top of the performance chart. But what if I had missed that one shot? My accuracy would have fallen to 0%, the very bottom of the chart. With one flip of my wrist, I could have gone from best to worst. That’s the volatile variability of small samples.
A straight line is the simplest relationship one can find between two variables. I generally believe that simpler is better. But many relationships simply aren’t simple. They change in nonlinear ways. By trying to make them linear, we over-simplify and run the risk of significant mistakes. Here are two:
What’s the bottom line here? If someone argues that the relationship between two variables is a straight line, take it with a grain of salt. And if one grain of salt is good for you, then two grains should be better. And if two grains are better, then three grains … well, you get the picture.
(I adapted the school example from Jordan Ellenberg’s excellent new book, How Not To Be Wrong: The Power Of Mathematical Thinking).
Are humans fundamentally biased in our thinking? Sure, we are. In fact, I’ve written about the 17 biases that consistently crop up in our thinking. (See here, here, here, and here). We’re biased because we follow rules of thumb (known as heuristics) that are right most of the time. But when they’re wrong, they’re wrong in consistent ways. It helps to be aware of our biases so we can correct for them.
I thought my list of 17 provided a complete accounting of our biases. But I was wrong. In fact, I was biased. I wanted a complete list so I jumped to the conclusion that my list was complete. I made a subtle mistake and assumed that I didn’t need to search any further. But, in fact, I should have continued my search.
The latest example I’ve discovered is called the survivorship bias. Though it’s new to me, it’s old hat to mathematicians. In fact, the example I’ll use is drawn from a nifty new book, How Not to Be Wrong: The Power of Mathematical Thinking by Jordan Ellenberg.
Ellenberg describes the problem of protecting military aircraft during World War II. If you add too much armor to a plane, it becomes a heavy, slow target. If you don’t add enough armor, even a minor scrape can destroy it. So what’s the right balance?
American military officers gathered data from aircraft as they returned from their missions. They wanted to know where the bullet holes were. They reasoned that they should place more armor in those areas where bullets were most likely to strike.
The officers measured bullet holes per square foot. Here’s what they found:
Engine 1.11 bullet holes per square foot
Fuel System 1.55
Rest of plane 1.8
Based on these data, it seems obvious that the fuselage is the weak point that needs to be reinforced. Fortunately, they took the data to the Statistical Research Group, a stellar collection of mathematicians organized in Manhattan specifically to study problems like these.
The SRG’s recommendation was simple: put more armor on the engines. Their recommendation was counter-intuitive to say the least. But here’s the general thrust of how they got there:
Clearly, the second explanation is more plausible. Conclusion: the engine is the weak point and needs more protection. The Army followed this recommendation and probably saved thousands of airmen’s lives.
It’s a colorful example but may seem distant form our everyday experiences. So, here’s another example from Ellenberg’s book. Let’s say we want to study the ten-year performance of a class of mutual funds. So, we select data from all the mutual funds in the category from 2004 as the starting point. Then we collect similar data from 2014 as the end point. We calculate the percentage growth and reach some conclusions. Perhaps we conclude that this is a good investment category.
What’s the error in our logic? We’ve left out the non-survivors – funds that existed in 2004 but shut down before 2014. If we include them, overall performance scores may decline significantly. Perhaps it’s not such a good investment after all.
What’s the lesson here? Don’t jump to conclusions. If you want to survive, remember to include the non-survivors.