Strategy. Innovation. Brand.


How Do You Know If Something Is True?

True or FalseI used to teach research methods. Now I teach critical thinking. Research is about creating knowledge. Critical thinking is about assessing knowledge. In research methods, the goal is to create well-designed studies that allow us to determine whether something is true or not. A well-designed study, even if it finds that something is not true, adds to our knowledge. A poorly designed study adds nothing. The emphasis is on design.

In critical thinking, the emphasis is on assessment. We seek to sort out what is true, not true, or not proven in our info-sphere. To succeed, we need to understand research design. We also need to understand the logic of critical thinking — a stepwise progression through which we can discover fallacies and biases and self-serving arguments. It takes time. In fact, the first rule I teach is “Slow down. Take your time. Ask questions. Don’t jump to conclusions.”

In both research and critical thinking, a key question is: how do we know if something is true? Further, how do we know if we’re being fair minded and objective in making such an assessment? We discuss levels of evidence that are independent of our subjective experience. Over the years, thinkers have used a number of different schemes to categorize evidence and evaluate its quality. Today, the research world seems to be coalescing around a classification of evidence that has been evolving since the early 1990s as part of the movement toward evidence-based medicine (EBM).

The classification scheme (typically) has four levels, with 4 being the weakest and 1 being the strongest. From weakest to strongest, here they are:

  • 4 — evidence from a panel of experts. There are certain rules about such panels, the most important of which is that it consists of more than one person. Category IV may also contain what are known as observational studies without controls.
  • 3 — evidence from case studies, observed correlations, and comparative studies. (It’s interesting to me that many of our business schools build their curricula around case studies — fairly weak evidence. I wonder if you can’t find a case to prove almost any point.)
  • 2 — quasi-experiments — well-designed but non-randomized controlled trials. You manipulate the independent variable in at least two groups (control and experimental). That’s a good step forward. Since subjects are not randomly assigned, however, a hidden variable could be the cause of any differences found — rather than the independent variable.
  • 1b — experiments — controlled trials with randomly assigned subjects. Random assignment isolates the independent variable. Any effects found must be caused by the independent variable. This is the minimum proof of cause and effect.
  • 1a — meta-analysis of experiments. Meta-analysis is simply research on research. Let’s say that researchers in your field have conducted thousands of experiments on the effects of using electronic calculators to teach arithmetic to primary school students. Each experiment is data point in a meta-analysis. You categorize all the studies and find that an overwhelming majority showed positive effects. This is the most powerful argument for cause-and-effect.

You might keep this guide in mind as you read your daily newspaper. Much of the “evidence” that’s presented in the media today doesn’t even reach the minimum standards of Level 4. It’s simply opinion. Stating opinions is fine, as long as we understand that they don’t qualify as credible evidence.

Does Smoking Cause Cancer in Humans?

You can’t prove nothing.

Let’s do an experiment. We’ll randomly select 50,000 people from around the United States. Then we’ll assign them — also randomly — to two groups of 25,000 each. Let’s call them Groups X and Y. We’ll require that every member of Group X must smoke at least three packs of cigarettes per day. Members of Group Y, on the other hand, must never smoke or even be exposed to second hand smoke. We’ll follow the two groups for 25 years and monitor their health. We’ll then announce the results and advise people on the health implications of smoking.

I’ve just described a pretty good experiment. We manipulate the independent variable — in this case, smoking — to identify how it affects the dependent variable — in this case, personal health. We randomize the two groups so we’re sure that there’s no hidden variable. If we find that X influences Y, we can be sure that it’s cause and effect. It can’t be that Y causes X. Nor can it be that Z causes both X and Y. It has to be that X causes Y. There’s no other explanation.

The experimental method is the gold standard of causation. It’s the only way to prove cause and effect beyond a shadow of a doubt. Yet, it’s also a very difficult standard to implement. Especially in cases involving humans, the ethical questions often prevent a true experimental method. Could we really do the experiment I described above? Would you be willing to be assigned to Group X?

The absence of good experiments can confuse our public policy. For many years, the tobacco industry claimed that no one had ever conclusively proven that smoking caused cancer in humans. That’s because we couldn’t ethically run experiments on humans. We could show a correlation between smoking and cancer. But the tobacco industry claimed that correlation is not causation. There could be some hidden variable, Z, that caused people to take up smoking and also caused cancer. Smoking is voluntary; it’s a self-selected group. It could be — the industry argued — that whatever caused you to choose to smoke also caused your cancer.

While we couldn’t run experiments on humans, we did run experiments on animals. We did essentially what I described above but substituted animals for humans. We proved — beyond a shadow of a doubt — that smoking caused cancer in animals. The tobacco industry replied that animals are different from humans and, therefore, we had proven nothing about human health.

Technically, the tobacco industry was right. Correlation doesn’t prove causation. Animal studies don’t prove that the same effects will occur in humans. For years, the tobacco industry gave smokers an excuse: nobody has ever proven that smoking causes cancer.

Yet, in my humble opinion, the evidence is overwhelming that smoking causes cancer in humans. Given the massive settlements of the past 20 years, apparently the courts agree with me. That raises an intriguing question: what are the rules of evidence when we can’t run an experiment? When we can’t run an experiment to show that X causes Y, how do we gather data — and how much data do we need to gather — to decide policy and business issues? We may not be able to prove something beyond a shadow of a doubt, but are there common sense rules that allow us to make common sense decisions? I can’t answer all these questions today but, for me, these questions are the essence of critical thinking. I’ll be writing about them a lot in the coming months.

Ice Cream and Muggings

Feel like mugging someone?

Did you know that the sale of ice cream is strongly correlated to the number of muggings in a given locale? Could it be that consuming ice cream leads us to attack our fellow citizens? Or perhaps miscreants in our midst mug strangers to get the money to buy ice cream? We have two variables, X and Y. Which one causes which? In this case, there’s a third variable, Z, that causes both X and Y. It’s the temperature. As the temperature rises, we buy more ice cream. At the same time, more people are wandering about out of doors, even after dark, making them convenient targets for muggers.

What causes what? It’s the most basic question in science. It’s also an important question for business planning. Lowering our prices will cause sales to rise, right? Maybe. Similarly, government policies are typically based on notions of cause and effect. Lowering taxes will cause the economy to boom, right? Well… it’s complicated. Let’s look at some examples where cause and effect are murky at best.

Home owners commit far fewer crimes proportionally than people who don’t own homes. Apparently, owning a home makes you a better citizen. Doesn’t it follow that the government should promote home ownership? Doing so should result in a safer, saner society, no? Well… maybe not. Again, we have two variables, X and Y. Which one causes which? Could it be that people who don’t commit crimes are in a better position to buy homes? That not committing crimes is the cause and home ownership is the result? The data are completely tangled up so it’s hard to prove conclusively one way or the other. But it seems at least possible that good citizenship leads to home ownership rather than vice versa. Or maybe, like ice cream and muggings, there’s a hidden variable, Z, that causes both.

The crime rate in the United States began to fall dramatically in the early 1990s. I’ve heard four different reasons for this. Which one do you think is the real cause?

  1. Legalized abortion — in 1973, the Supreme Court effectively legalized abortion in the United States. Eighteen years later, the crime rate began to fall precipitously. Coincidence?
  2. The “broken windows” theory of policing — police traditionally focused on serious crime while ignoring petty crimes. In the 1980s, sociologists began to argue that ignoring petty crime sent a signal to would-be criminals that citizens will tolerate crime in a given area. Even minor crimes like broken windows could send the wrong message. Police adopted the idea and started cracking down on petty crimes. The message? If minor crimes are not tolerated, just think what they’ll do for bigger crimes!
  3. The aging population — we’re getting older. Young people commit a disproportionate number of crimes, especially violent crimes. As our nation ages, we become more sedate.
  4. The “get tough” sentencing movement — politicians in the 1980s began to sponsor legislation to “get tough” on crime by imposing longer, mandatory sentences. One result has been a dramatic rise in our prison population. (In fact, I read recently that the U.S. has 700 people incarcerated for every 100,000 citizens. In Sweden, the equivalent rate is 70 prisoners. Could it be that we’re ten times more criminal than Swedes? Swedes are blonde and they don’t commit crimes. Cause and effect, right? Perhaps we should all dye our hair blonde.)

Which of the four variables actually caused the declining crime rate in America? A lot is riding on the answer. Unfortunately, the data are so tangled up that it’s difficult to tell what causes what. But here are some rules for thinking about correlation and causation:

  • If you think X cause Y, always ask the reverse question. Is it possible that Y caused X?
  • Always look for a hidden variable, Z, that could cause both X and Y.

Actually, the only way to prove cause and effect beyond a shadow of a doubt, is the experimental method. Which leads us to our question for tomorrow: does smoking cause cancer in humans?

My Social Media

YouTube Twitter Facebook LinkedIn

Newsletter Signup