EBM Tools for Practice: Clinical Trials of Alternative Medicine: Testing Whether Magic Works

I have in mind a prospective clinical trial in which I study a common children’s phenomenon, detailing the differences between genders, location, ethnicity, biometrics, dental data, monetary value, etc. It’s a common phenomenon that most parents have photos of and have experienced ourselves. I’m sure we could generate a lot of highly statistically significant data and make many useful insights.

I’m talking about what our children find left under their pillows by the Tooth Fairy. One problem — the Tooth Fairy doesn’t really exist.

This phenomenon, Tooth Fairy Science,¹ happens daily throughout America, at many of our hospitals and institutions of higher learning. Under pressure from alternative medicine (Alt-Med or Complementary and Alternative Medicine [CAM]) practitioners and societies, and even Alt-Med-friendly government officials,²  such prescientific and long-discredited practices as homeopathy, acupuncture (“a theatrical placebo”), reiki, and others are infiltrating our medical schools and hospitals, without scientific proof of their efficacy and safety. These practices often come under the guise of integrative medicine, an attempt to “treat the whole patient” by integrating traditional medicine with Alt-Med.

Good clinicians attempt to treat the whole patient in the first place. However, many physicians, being rushed by the many pressures engendered by electronic medical records and volume-based reimbursement, have ceded to the Alt- Med practitioners the beneficial effects of talking to the patient and a more extended “laying on of hands” experience. As far as combining traditional medicine with Alt- Med, as a famous phrase goes,³ “Mixing apple pie with cow pie doesn’t make the cow pie taste better, it makes the apple pie taste worse.”

Two recent articles are pertinent. Maurizio Pandolfi and Giulia Carreras4 discuss why “the type of inferential statistics used in medicine have intrinsic flaws to which CAM interventions appear to be particularly vulnerable.” When even a well-designed clinical trial is performed, the predictive power of the results is dependent on the prior plausibility of the phenomenon being studied. How to establish this prior plausibility? Conformity to scientific laws and principles is one criterion – the Tooth Fairy, homeopathy (in which the “treatment” is so diluted that it contains not even one molecule of the original substance!), “energy medicine,” reiki, Bach Flower Therapy, acupuncture (there is no identifiable “chi” nor “meridian”), and many others all fail to meet this criteria. Another criterion is “falsifiability”: Science aims to produce falsifiable hypotheses; pseudoscience often cannot. Another criterion is “Ockham’s razor,” according to which if two models describe the observations equally well, the simpler one is more likely to be true, e.g. the placebo effect in Alt-Med interventions. Applying Bayes’ theorem, that is the probability of an inference’s being true is dependent on prior plausibility — Bayes’ theorem provides a principled way of combining new evidence with prior beliefs — it is inopportune to interpret P values at face value, especially when they are calculated from results obtained by testing hypotheses with low prior probability/plausibility. Clinicians tend to be easily swept up in P values, often neglecting how heavily dependent they are on prior probability/plausibility of hypotheses being tested. (This is very low for Alt-Med modalities such as those mentioned above.) It is critical to realize that the P value tends to exaggerate support for the hypothesis tested, especially if the scientific plausibility of the hypothesis is low. Any difference can be shown to be statistically significant if the numbers are large enough. What counts is clinically meaningful clinical differences found to be statistically significant.

David Gorski and Stephen Novella⁵ lament that the usual process of evidence-based medicine (EBM), wherein treatments proceed to randomized clinical trials after biological plausibility is determined and compelling evidence from preclinical studies has been amassed, has been upended in the case of Alt-Med treatments, for which clinical trials are conducted prematurely. Some of these trials will have “positive P values” but, as explained above, the low prior plausibility undermines the trials’ support for the tested hypothesis.

It is important to note that “biologically plausible” does not mean “knowing the exact mechanism.” It means that the mechanism should not be so scientifically implausible as to be reasonably considered impossible and not violate well-established laws of biology, chemistry, and physics (as reiki, homeopathy, and acupuncture do, for example).

“Extraordinary claims require extraordinary evidence,” as philosopher David Hume once said.⁶ Many clinical trials are imperfect and bias-prone and should not be weighted as superior evidence of truth in circumstances where prior plausibility is low. In RCT’s testing modalities with low plausibility, confounding bias effects are very much magnified, producing a high false-positive rate.⁷  Despite their lack of biological plausibility and pre-clinical studies, Alt-Med clinical trials are, in fact, often driven by popularity with the public and funding sources. The authors have distinguished science-based medicine (SBM) from EBM by the recognition that, before doing a clinical trial on a medical treatment, there must be a reasonably high prior plausibility that the treatment will work.⁸

An example perhaps familiar to many readers is the Trial to Assess Chelation Therapy (TACT), the $30 million multi- center trial funded by tax dollars and the National Center for Complementary and Alternative Medicine (NCCAM) to determine the safety and efficacy of ethylenediaminetetraacetic acid (EDTA) chelation therapy for individuals with coronary artery disease (CAD) and prior myocardial infarction (MI). The trial was conducted despite a lack of prior plausibility, possible mechanism of action and preclinical trial data, as well as multiple failed smaller trials. Many study sites were dubious alternative medicine clinics, and the treatment itself poses risks to patients.⁹ When finally reported, results were negative except for one subgroup (diabetics), and all results have come under much deserved and damning criticism.¹⁰

An esteemed lipidology colleague, when engaged in debate over these issues, accused those of us practicing SBM of being biased and closed minded in our assessment of prior plausibility for Alt-Med treatments. He’s certainly correct. SBM is biased in favor of science because science works. Science also does not pretend to know everything. Who knows what pharmacologic treatments await discovery out there? But unless new treatments are consistent with the laws of science, their assessment in clinical trials is unrevealing, predicts little, and is even unethical.

“Keep your minds open and stay thirsty my friends, but don’t let your brains fall out!” — The Most Interesting Scientist in the World.

Disclosure statement: Dr. Pokrywka received consulting fees from Amarin and AstraZeneca. He received speaker honoraria from Amarin, AstraZeneca, Daiichi Sankyo Inc., Kowa Pharmaceuticals, LipoScience Inc., Health Diagnostics Labs, Genzyme, Metagenics, and Genentech.

References are listed on page 35 of the PDF.