by Conor Friedersdorf
The reader writes:
As an epidemiology doctoral student, I'm constantly amazed by the ways in which people misunderstand, misconstrue, and overestimate the available evidence on health-related topics. Few things touch as many aspects of our lives as intimately as health, and yet we consistently see high percentages of individuals making use of therapies with little or no value, sometimes at great expense to both their health and their pocketbooks. We have around 300 years of statistical and epidemiological methods for evaluating health effects, and despite controversy at the periphery, these methods work. Not only that, they're the only methods that do.
The question of how to behave in the absence of evidence is an open one; how to behave with evidence of absence is not. It's hard to begrudge the desperate family for seeking unproven cancer treatments, but as a society we need to think about how we value false hope, and whether or not we can discourage those hopes in the first place, rather than humoring them or crushing them after they take hold (neither of which are appealing choices).
And so a handful of rules & suggestions for the ill and ill-informed on how differentiate between absence of evidence and evidence of absence:
1. Ancient wisdom is the weakest possible form of plausibility. Given what we know about our extensive irrationality, it isn't hard to imagine a therapy existing for thousands of years because it plays on our biases but doesn't do anything.
2. "Scientific" does not apply to the act of cherry-picking desirable outcomes from among many, many contradictory examples. In the absence of large, diverse, double-blinded randomized trials, we have to look at the totality of evidence from different sources using different methods.
3. Recognize that our ability to evaluate evidence is colored by our level of personal investment. Wishin' and hopin' has never changed a p-value nor shifted a confidence interval. We could all take a page from Adam Smith's Theory of Moral Sentiments, and try appealing to the "impartial observer" within and without. Don't wait until you have cancer to engage in the health arena. Teach yourself some science and statistics and form opinions now, before you have a diagnosis.
4. There is no such thing as "alternative medicine", only "alternatives to medicine". Once something has been proven efficacious, it simply becomes medicine. Physicians are desperate for treatments, and researchers are desperate for high-impact studies; if a treatment was ripe for the picking, you can bet your ass there'd be a stampede to demonstrate its effectiveness.
5. Always beware of "individualized" therapies. These may be crucially important in years to come, but only in very circumscribed areas (personal genomics). Charlatans have always appealed to our vanity; remember that often the more personally tailored a treatment becomes, the farther it deviates from accepted standards of treatment.
5a. And walk the other way when you hear the claim that a treatment is too individualized to be evaluated by randomized trials. This is often the post hoc cry of those who had a hand in designing the trial in the first place.
6. Don't put your faith in experts, but do conditionally trust them (trust, but verify). The more information we have, the harder it is to evaluate and synthesize it. There are more and more great resour
ces whose sole purpose is to evaluate the evidence on complex topics; while these may not answer your questions, they are an essential starting point.