I’m a bit tired of blogging about nothing but COVID-19; so I thought I’d take some time to “dunk on a 7′ hoop” and look at homeopathy. Obviously, I’m being sarcastic here, because, no matter how much the precepts of homeopathy violate multiple well-established laws of physics and chemistry, no matter how, for homeopathy to “work,” huge swaths of well-documented physics, chemistry, and biology would have to be not just wrong but spectacularly wrong, or how often clueless academics dismiss skepticism that debunks quackery as too easy and not worthy of their big brains, it’s often not at all easy to explain to the lay public why homeopathy is The One Quackery To Rule Them All, particularly when a homeopath starts touting what looks like a positive controlled randomized clinical trial of homeopathy for some ailment or other, saying something like, “If homeopathy is quackery, explain this, skeptic!”
Challenge accepted, although in a more general fashion, thanks to a paper published last week in BMJ Evidence-Based Medicine entitled Assessing the magnitude of reporting bias in trials of homeopathy: a cross-sectional study and meta-analysis. What’s reporting bias? Hang on, and I’ll explain. Before I go on, I feel compelled to add something. When I make this reference to The One Quackery To Rule Them All echoing One Ring To Rule Them All, it occurs to me that I should add my apologies to J.R.R. Tolkien. At least in The Lord of the Rings, the One Ring had actual power, unlike, of course, homeopathy. Moving on to mix fantasy novels and movies, as I like to say when I compare quackery to magic at Hogwarts, in the Harry Potter universe magic actually works—and it even works in a fashion sufficiently predictable to be studied scientifically, which is rather what this paper tries to do, at least in terms of clinical trials.
Traditionally, whenever the topic of homeopathy comes up, even though I know that a lot of my readers are already very familiar with it, I feel an obligation to include a brief tutorial for the newbies rather than just linking to past discussions (of which there are many going back to 2005) and making them click through if they aren’t familiar with the topic. The first thing one has to understand is that homeopathy is privileged, particularly in European countries such as Germany and France (although also in the US, just somewhat less so) because of history. Basically, it was grandfathered in, such that it is accepted without being held to the same rigorous standards of evidence that pharmaceutical drugs are by the US FDA and its equivalent in other countries. Indeed, until recently in France, the government fully paid for homeopathic remedies. Even now, physicians who speak out against homeopathy can find themselves at the receiving end of lawsuits and government sanctions.
Back to the brief primer, though.
Homeopathy is, as I said above, what I like to refer to as “The One Quackery To Rule Them All,” although other equally magically ridiculous (or even arguably more magically ridiculous) alternative medicine treatments like reiki, distance healing, and the like do give homepathy a run for its money for the title. When I first learned what homeopathy really is, I was gobsmacked. Indeed, most people are blissfully unaware of the magical principles of homeopathy, such as the law of similars (i.e., “like cures like,” the principle that states that, in order to relieve a symptom, you should use an herb, medicine, or other compound that causes the symptom) and the law of infinitesimals (which claims that diluting a remedy makes it stronger). And don’t even get me started on homeopathic “provings,” in which healthy people take the substances used in homeopathic remedies and then report their findings. I’ve discovered that even most physicians are unaware of the true precepts of homeopathy, with most of them thinking it’s just herbal medicine.
Although the law of similars is without a basis in science, biology, or physiology, and homeopathic provings result in some truly hilariously ridiculous nonsense, it is the law of infinitesimals that best illustrates the utter absurdity of homeopathy. Here’s the idea. This law states that, to make a remedy stronger, you dilute the remedy. And, wow, do homeopaths ever do that! A typical homeopathic remedy is 30C, with “C” signifying a 100-fold dilution. So a 30 C homeopathic dilution is equal to thirty 100-fold dilutions or (10-2)30, or a 1060-fold dilution. Those of you with a chemistry background will notice right away that this is an incredibly large number compared to Avogadro’s number, which is 6.022 x 1023 and is the number of molecules in a mole of a chemical. So, even if one starts with a mole of a substance (whose weight equals its molecular weight in grams), the resulting 30C dilution will dilute it over 1036-fold beyond the number of starting molecules. In other words, it’s incredibly unlikely that there will be a single molecule of starting substance left, other than potentially any that might “carry over” between serial dilutions sticking to the glassware. How do homeopaths explain this? They claim that water has “memory” and that it “remembers” contact with the active ingredient.
As is usually the case whenever I write about homeopathy, I like to include this clip of Richard Dawkins from Enemies of Reason. Whatever other failings Dawkins has, in this case he explains the ridiculousness of The One Quackery To Rule Them All in two and a half minutes better than I can in two thousand words:
So, from a basic science standpoint, homeopathy can’t work. To repeat for emphasis what I said above, for homeopathy to work, huge swaths of physics, chemistry, and biology would have to be not just wrong, but spectacularly wrong. This brings us to clinical trials.
Noting, as I did above, that the perception still persists in a number of countries that homeopathy is a “safe, holistic and comparable alternative to modern medicine” and adding that “up to 9.2% of adults in mostly western countries have relied on homeopathic remedies during the past 12 months,” the authors then provide a bit of background:
For years, sceptics and homeopathic practitioners have engaged in a fierce debate on whether homeopathic treatments are more effective than placebo.5–11 Advocates of homeopathy often refer to two systematic reviews and meta-analyses by Mathie et al that reported statistically significant differences of homeopathic treatments compared with placebo.12 13 According to Google Scholar, these reviews have been cited up to 200 times in other research publications. Sceptics counter that homeopathy’s effectiveness is no different from a placebo’s effectiveness when effectiveness is based on methodologically sound studies.14 The Australian National Health and Medical Research Council concluded that according to an assessment of 57 systematic reviews on 68 conditions, ‘there are no health conditions for which there is reliable evidence that homeopathy is effective’.15 Institutions in the European Union16 and the UK17 have issued similar statements that support this stance.
An important factor often missing from the debate, however, is that published RCTs of homeopathic treatments might not represent the totality of conducted scientific studies but rather only a selected proportion with positive results. This phenomenon, known as reporting bias, occurs when the publication or non-publication of studies or outcomes depend on the nature and direction of results, with statistically significant findings having a higher likelihood of publication than non-significant findings.18 Because statistical methods to detect or correct for reporting bias have limitations, meta-analyses of published studies like the ones from Mathie et al12 13 can lead to inflated and misleading results because positive trials are overrepresented.
The same sort of issue applies to acupuncture studies, by the way, but that’s a topic for another post.
Unfortunately, they also say what I said above in cringeworthy milquetoast way:
Homeopathy, developed by Samuel Hahnemann in Germany almost 200 years ago, is largely inconsistent with current scientific concepts.1 For example, the principle of similarity (like cures like) and the claim that an increasing dilution of a substance leads to a stronger treatment effect (potentiation) lack evidence and contradict medical and physical principles.2
“Largely inconsistent with current scientific concepts”? “Lack evidence and contradict medical and physical principles”? I guess you could say that. At least they cite David Grimes.
Another name for the phenomenon of publication bias is the “file drawer effect,” so named because negative studies can be more likely to get filed in a drawer somewhere rather than submitted for publication in the peer-reviewed literature. There are, of course, a number of techniques to assess for publication bias in systematic reviews and meta-analyses. One of the most commonly used is the funnel plot, the main assumption behind which is that studies with high precision will tend to end up plotted near the average, while studies with low precision (i.e., lower quality studies) will be distributed evenly on both sides of the average, creating a roughly funnel-shaped distribution. However, funnel plots are imperfect, as are many of the other methods of detecting publication bias just from the published literature, which brings us to this study.
One strategy that has been implemented, with varying degrees of rigor and success, is to require pre-registration of clinical trials, including the primary and secondary outcomes, before the trials are begun, which can allow the determination of whether a file drawer effect exists for a treatment and its clinical trials, and, if so, how large it might be. The authors note:
To reduce reporting bias, public trial registries such as ClinicalTrials.gov in the USA and ClinicalTrialsRegister.eu in the European Union have been founded to promote the prospective registration of all trials. Since an amendment of the Declaration of Helsinki in 2008, prospective trial registration and publication of results are regarded as an ethical obligation of investigators.19 In 2005, the International Committee of Medical Journal Editors (ICMJE) adopted a policy that the journals they oversee would only publish results of clinical trials which have been prospectively recorded in a public registry.20 These measures have led to an increase in the number of clinical trials that are prospectively registered21; however, researchers are not obligated to publish results of such trials22 and the proportion of non-publication remains high.
Because regulatory agencies do not require proof of effectiveness for homeopathic products, little attention has been paid to the non-publication of homeopathy trials and its consequences. An assessment by Thomas et al reported that up to the year 2013, only 46% of registered homeopathic trials (16/35) were published.23 When no information is publicly available about the majority of homeopathic trials, sound conclusions about the efficacy and the risks of using homeopathic medicinal products for treating health conditions are impossible.24 25
This is, of course, true for all clinical trials, regardless of whether the treatment being studied is from alternative medicine or science-based medicine, as the authors note later, in their conclusion:
The non-publication of trial results and selective outcome reporting, however, is not a phenomenon that is limited to homeopathy.60 Over the past years, numerous studies reported that, despite registration, large proportions of completed trials remained unpublished.22 61–65 For example, an assessment of 2132 registered clinical trials in Germany between 2009 and 2013 showed that 33% remained unpublished after 5 years.62 Likewise, of 4347 clinical trials conducted in academic centres in the USA, 34% remained unpublished.61
Let me intercept one criticism that I can foresee right here, namely the response that there are high numbers of unpublished clinical trials of pharmaceuticals too. This is nothing more than “whataboutism”—or “whataboutery”—a propaganda technique designed to distract from a legitimate criticism by deflecting that criticism back at the critic and insinuating hypocrisy. Let’s just say that proponents of science-based medicine apply the principle of insisting on preregistration and publication of clinical trials to all medicine, not just homeopathy and other alternative medicine. Let me also observe a general principle about clinical trials. Low prior plausibility (and homeopathy has, in essence, zero prior plausibility) plus equivocal clinical trial results equal, “the treatment doesn’t work or has effects too small to be clinically useful.” (I frequently say this about ivermectin for COVID-19 and here note that ivermectin, as poor as its prior plausibility is for COVID-19, is still way more plausible a treatment than homeopathy is for anything.) In contrast, pharmaceutical studies almost always have a much higher level of prior plausibility at their heart than any study of any homeopathic remedy.
Now that that’s out of the way, let’s move on. The current study had four objectives:
- To determine the proportion of registered trials assessing homeopathy that remains unpublished.
- To examine whether registered primary outcomes are consistent with published primary outcomes.
- To assess the proportion of published randomised controlled trials (RCTs) on homeopathy that have been registered in a public clinical trial registry.
- To gauge the impact of reporting bias on evidence syntheses of homeopathy trials.
To accomplish this, the authors did this:
Two persons independently searched Clinicaltrials.gov, the EU Clinical Trials Register and the International Clinical Trials Registry Platform up to April 2019 to identify registered homeopathy trials. To determine whether registered trials were published and to detect published but unregistered trials, two persons independently searched PubMed, Allied and Complementary Medicine Database, Embase and Google Scholar up to April 2021. For meta-analyses, we used random effects models to determine the impact of unregistered studies on meta-analytic results.
The primary outcomes and measures were the proportion of registered but unpublished trials and the proportion of published but unregistered trials. The authors also assessed “whether primary outcomes were consistent between registration and publication.” Why did they do that? Because changing the primary outcome of a clinical trial after the trial starts is a no-no, the reason being that such changes are not infrequently made because the results for the primary outcome were negative and the authors switch to a different outcome for which a “statistically significant” result can be reported.
Here’s the schema for the searches:
So what were some key findings? First, the authors found that close to 38% of homeopathy trials registered since 2002 remain unpublished. The authors also found that over half (53%) of published randomized controlled trials of homeopathy since then were not registered and that, even for the trials that were registered, retrospective registration (registration after the trial was started—ore even after it was published) was more common than prospective registration (registration before the trial was started). It’s gotten somewhat better recently, but even over the last five years nearly 30% of homeopathy RCTs published had not been registered.
Here’s a graph of the results by year:
A meta-analysis of unregistered RCTs yielded a statistically significant treatment effect favouring homeopathy (SMD: −0.53, 95% CI −0.87 to −0.20). By contrast, a meta-analysis of registered RCTs did not show a statistically significant difference between homeopathy and placebo (SMD: −0.14, 95% CI −0.35 to 0.07). Meta-regression revealed that the difference in effect sizes between registered and unregistered studies did not reach statistical significance (difference in SMDs: 0.39, 95% CI −0.09 to 0.87). Figure 4 presents the meta-analyses of registered and unregistered RCTs.
I like to say that homeopathy is a perfect tool to probe the weaknesses of clinical trials in that any homeopathic remedy over around 12 C is indistinguishable from nothing and therefore most homeopathy trials compare placebo to placebo. Thus, even under perfect conditions, by random chance alone we would expect to see that roughly 5% of clinical trials of homeopathy would be positive, just because we arbitrarily choose as our cutoff for statistical significance p-values less than or equal to 0.05. Of course, far more than 5% of homeopathy clinical trials appear to be positive and that’s where the biases and shortcomings of randomized clinical trials come in. This study adds to the literature yet another observation that publication bias is a major problem in homeopathy and likely explains its higher than expected (based on random chance alone) number of positive clinical trials.
The bottom line, however, remains that homeopathy is based on prescientific concepts more akin to sympathetic magic than it does to any science. For all the reasons I included in my introduction, that means that, arguing from basic science alone, homeopathy can’t work. Unfortunately, that doesn’t stop naturopaths and other quacks from believing that it does work or evidence-based medicine methodolatrists who worship the randomized controlled trial as the only valid form of medical investigation from ignoring how the precepts of homeopathy violate multiple laws of physics and chemistry and observe that more clinical trials need to be done.