If there’s one thing that the COVID-19 pandemic has taught me, is that, no matter what else is going on, antivaxxers gonna antivax (that is, continue to spread antivaccine propaganda). It’s certainly true that very early on during the pandemic antivaxxers formed an unholy alliance with pandemic deniers/minimizers, antimaskers, anti-lockdown protesters, and promoters of unproven treatments for COVID-19, such as hydroxychloroquine—and QAnon. However, that new alliance, which, thanks to the pandemic’s being the all-consuming crisis and story of 2020, brought antivaxxers to arguably more prominence than they’ve ever achieved before, hasn’t stopped antivaxxers from continuing to do what they do to provide fodder for their disinformation blaming vaccines for autism and other neurodevelopment disorders, as well as all manner of other diseases and chronic health conditions. A favorite among these is what I like to refer to as the “vaxxed/unvaxxed” study, and another such study was just published this week by antivax pediatrician Dr. Paul Thomas and scientist turned antivax crank, James Lyons-Weiler.
What is a “vaxxed/unvaxxed” or “vaccinated/unvaccinated” study? Simple. It’s any study that compares health outcomes between a vaccinated cohort of children and those of an unvaccinated cohort. In its purist form, it is a randomized, double-blind, placebo-controlled study comparing “vaxxed” to “unvaxxed.” Of course, such a study would be highly unethical because, by its very design, it would require that a group of children be intentionally placed in a control group that would be left vulnerable to vaccine-preventable diseases because the control group would only receive placebo vaccines. Antivaxxers don’t like it when that simple fact is pointed out to them, of course, because they really and truly believe that vaccines do more harm than good. Of course, it never occurs to them that their belief would make such a randomized, controlled clinical trial unethical as well, because they would be intentionally placing children into a group who would be—to them—being exposed to something they view as a grave threat to their health, vaccines. (After all, they believe that vaccines are toxins-laden interventions that cause autism, autoimmune diseases, obesity, alter DNA, render females infertile, and have created the “sickest generation” of children—and in some cases even kill.) True, their belief is erroneous, but intent matters. It all comes down to clinical equipoise, which is the genuine uncertainty over whether an intervention is on balance beneficial, harmful, or without detectable effect. A randomized controlled trial (RCT) of “vaxxed/unvaxxed” children violates clinical equipoise.
As a result, some of the savvier antivaxxers, who know that a “vaxxed/unvaxxed” RCT would be highly unethical (or who are at least willing to concede that the overwhelming view of physicians and scientists who do clinical trials is that a “vaxxed/unvaxxed” RCT would be highly unethical) have retreated back to suggesting observational “vaxxed/unvaxxed” studies, such as retrospective and epidemiological studies. The whole idea is that vaccines cause autism, the obesity epidemic, and in general the “sickest generation” of children, even though there is no evidence that they do. Of course, doing such a study is a hell of a lot more difficult than antivaxxers think, given how difficult it is to account for confounders and how many subjects are needed to provide sufficient power to detect differences in a condition whose prevalence is in the low single digit percentage range. Still, none of this has stopped antivaccine “scientists” and physicians from trying to do such studies. Unsurprisingly, the results are always dismal in that the studies are inevitably positive (i.e., claim to find that unvaccinated children are healthier than vaccinated children) but so poorly designed and executed that they are singularly uninformative and their conclusions are not supported by their data and design. Examples abound, unfortunately, such as risibly incompetent “vaxxed/unvaxxed” studies by Brian Hooker and Neil Z. Miller, studies by Anthony Mawson that were retracted and republished and retracted, Internet surveys by quacks about vaccinations, and a number of others that I could reference.
This brings us to Thomas and Lyons-Weiler’s incompetent study, Relative Incidence of Office Visits and Cumulative Rates of Billed Diagnoses Along the Axis of Vaccination, published in a journal I’d never heard of before (or at least don’t currently recall having heard of before), the International Journal of Environmental Research and Public Health. It was published on Sunday, and antivaxxers, such as Jennifer Margulis, are already crowing about it, claiming that it shows that vaccinated children are sicker. First, though, Margulis denies that doing a “vaxxed/unvaxxed” RCT would be unethical:
For over two decades, vaccine safety advocates have wondered if vaccinated children are sicker or healthier than their unvaccinated peers. These children’s health advocates have asked the CDC to conduct studies to compare health outcomes in completely unvaccinated children to those in vaccinated children, as per the CDC’s recommended schedule.
Every parent in America, and every doctor recommending vaccines, should want to see these studies done.
After all, in order to put the vaccine debates to rest, we must demonstrate that the current CDC-recommended vaccine schedule is actually safe.
But, despite constant hand-wringing over declining public confidence in vaccines, the CDC refuses to do these studies.
Their excuse has been that “it would be unethical” to withhold vaccines in order to study their safety.
This excuse assumes that randomized controlled trials, where children are randomly separated into two groups, one which gets vaccines and one which does not, are the only way to do such a comparison.
(If you’re wondering right now why it’s “unethical” to withhold vaccines to determine their safety but not unethical to recommend them for every infant before they are determined to be safe, you’re not alone.)
Note the antivax propaganda: “Vaxxed/unvaxxed” studies are not unethical because it hasn’t been shown that vaccines are safe and effective yet, and there are no long term studies showing vaccine safety. (They have been, many times, and there are several long term vaccine safety studies to confirm this.) There are no vaxxed/unvaxxed studies. (There are, and they don’t show what antivaxxers think they should show.) Not quoted above, but that all the “vaxxed/unvaxxed” studies out there currently don’t truly study unvaccinated children but “vaxxed versus slightly less vaxxed children. (Again, not true.) You get the idea. Margulis thinks that Thomas and Lyons-Weiler’s study is slam-dunk evidence that unvaccinated children are healthier, too:
Aware of the firestorm surrounding vaccine science, Lyons-Weiler and Thomas’s analysis is scrupulous and thorough. They analyzed the data several different ways to account for potentially confounding factors, such as an increasing tendency to avoid vaccination. They compared the new metric (RIOV) with the old measure of incidence. And they found that the new method correlates well but is more sensitive, thus more likely to reveal a true negative effect than the old one.
So what did Lyons-Weiler and Thomas find?
The results: cumulative office visits for asthma, allergic rhinitis, breathing issues, behavioral issues, ADHD, respiratory infection, otitis media, ear pain, other infections, eye disorders, eczema, and dermatitis were all much higher in vaccinated children than unvaccinated children.
In even the most conservative analysis, the study finds statistically significant elevated risks of anemia and respiratory virus infection in the vaccinated children.
Time to go to the tape, so to speak, and look at the “study” (such as it is) itself. The first thing to look at is the study design. Basically, if we’re to believe Thomas and Lyons-Weiler, it’s a retrospective comparison of all patients that were born into Dr. Thomas’ practice between June 1, 2008 and January 27, 2019, with a first visit before 60 days of life and a last visit after 60 days. The inclusion/exclusion criteria (basically stated just now) whittled down over 21,000 records to 3,324 patients, of whom 2,763 were “variably vaccinated,” defined as “having received 1 to 40 vaccines.” But what is the primary outcome studied? For this, Thomas and Lyons-Weiler make up a brand new metric that I’d never heard of before, the Relative Incidence of Office Visit (RIOV).
Before I get to that, I note that there are a number of epidemiological studies that rely on billing records and diagnoses submitted to insurance companies and third-party payors. The advantages of using such metrics are simple. They’re there. They exist already. They’re in the medical record. They’re relatively easy to access and fairly standardized. In the case of hospital systems and national insurance programs (like several in Europe), utilizing insurance data can provide huge numbers of data points to mine for correlations. There are a number of problems, however, as well, given that insurance and billing data are collected for financial reasons primarily, rather than for medical reasons. There can be selection bias (particularly in a concierge practice run by an antivax pediatrician) and the question of external validity (applicability to a more general situation outside of the practice) is a huge issue. Moreover, what is coded for a diagnosis for billing purposes often maps imperfectly (or even poorly) to actual clinical diagnoses.
At this point, I’d also like to make a general sort of comment about epidemiological studies (and, make no mistake, that’s what this study is, as it is, in essence, a retrospective cohort study). If you’re going to do a study like this, you need a competent statistician involved before you collect the data. The only two authors listed are Thomas and Lyons-Weiler, and the statistical methods are not well-described. It really also would have behooved them to have an epidemiologist or at least a clinical investigator with experience doing retrospective analyses to help them. Clearly, they did not. Margulis claims that they did, but this “independent statistician” (as she puts it) is not identified anywhere that I can see. (I’ll gladly identify the statistician allegedly involved with this study if I missed it somehow.)
Then there’s Table 7. Before I go on to the rest of the obfuscation of this paper, look at Table 7. Basically, Table 7 shows that vaccines work. Basically, the rates of any diagnosis of a vaccine-preventable disease were as follows: Vaccinated, 7/2647 (0.00264) vs. unvaccinated, 34/561 (0.0499). Personally, I was surprised that the difference was what it was, but it’s much more striking to note that all of the cases of vaccine-preventable disease in the “vaxxed” population were chickenpox (6) and pertussis (1), while the “unvaxxed” population had a lot more pertussis, chickenpox, and rotavirus. In any event, at the very least, this table shows that, even in Dr. Thomas’ practice, vaccines work and that likely the low level of vaccine-preventable disease in the unvaccinated is due to herd immunity.
But back to the RIOV. I did some PubMed searches, and I couldn’t find a single paper that used this metric as described by Thomas and Lyons-Weiler. Certainly, the authors do not cite any papers that have used this particular made-up metric before to justify its use, to demonstrate its advantages and disadvantages, and in general to provide the sort of information that any clinical investigator would want about an unfamiliar metric. Indeed, I’m always suspicious when I see a metric like RIOV. It strongly suggests to me, particularly in the case of a retrospective study, that the authors tried to do an analysis looking at more defined, traditional primary outcomes and failed to find any statistically significant results. In other words, this paper reeks of p-hacking, the practice of doing comparison after comparison until a “statistically significant” result is tortured out of the data. To see this, you have only to look at Table 2, where Lyons and Weiler do comparisons for 18(!) health conditions, after which they layer on analyses for family history (Table 6) for each condition, gender,
So, unable to dazzle us with brilliance, Thomas and Lyons-Weiler try to baffle with bullshit, presenting graph after meaningless graph of RIOV results. One that caught my eye was this one, Figure 3:
Notice the correlation between vaccine acceptance and visits for fever. This could well just be because fever is a common complaint after many childhood vaccines coupled with the likelihood that parents who are more accepting of vaccines will have different health-seeking behavior than those who do not, being far more likely to bring their children in to be seen when they have a fever. The authors claim that RIOV “reflects the total number of billed office visits per condition per group, reflecting the total disease burden on the group and the population that it represents,” but no good analysis or references are provided to show that RIOV does, in fact, correlate with disease burden, particularly when using billing data. Doing matched analyses for patients with similar “days of care” (DOC) in the practice, which is claimed to be “unbiased” (excuse me if I doubt this, given that there wasn’t really a good demonstration that this the choice of children with matched DOC was, in fact, “unbiased”) doesn’t change this. If the primary outcome is a new, unvalidated metric, it is incumbent upon the investigator to demonstrate its robustness.
Moreover, here’s another issue. These are billing records being mined for diagnoses and billed visits. What about unbilled services? What about phone calls, for example? Fever, for instance, is a biggie, a very common reason that parents call their pediatrician’s office. After such a call, it is up to the pediatrician to decide if the child needs to be seen in the office (or be sent to the emergency room) or if reassurance and home measures can be provided over the phone. Come to think of it, because it is billing records being analyzed, it is impossible to know how many emergency room visits there were in each group or how many unbilled telephone calls were received from parents of children in each group. At least, these measures cannot be determined without a chart review, which appears not to have been done. Thomas and Lyons-Weiler just looked at the number of billed visits there were for each condition, did some perfunctory attempts at correcting for length of time in the practice, and called it a day.
Finally, if there’s one thing I’ve learned about practices like this, it’s that “integrative” physicians (particularly antivaccine) integrative physicians like Dr. Thomas don’t practice the same way science-based pediatricians do. For example, remember Dr. Mayer Eisenstein? He famously claimed that he had zero children with autism in his practice, which is not plausible. Dr. Thomas claims that he has zero patients with autism among his unvaccinated patients and that all the cases of autism in his practice are among the group that is “most vaccinated,” whatever that means.
Similarly (and perhaps most importantly) the children in Dr. Thomas’ practice are likely to be quite different than children in a typical pediatrics practice, as I discussed before. For instance, there’s likely to be ascertainment bias, which is the systematic distortion in measuring the true frequency of a phenomenon due to the way in which the data are collected. How could this happen? Think about it. Dr. Thomas believes that vaccines cause autism. That right there introduces unconscious bias that could affect how likely he and his staff are to investigate subtle signs of autism and refer out to for evaluation based on vaccination status and how likely he is to ascribe various diagnoses to “unvaxxed” children compared to “vaxxed” children. One can easily imagine this bias leading to unvaccinated children to be less likely to be given an autism diagnosis than vaccinated children or to be—dare I say?—brought in to the office as often for various conditions that Dr. Thomas attributes to vaccines.
It also doesn’t seem to occur to Thomas and Lyons-Weiler that measuring the number of office visits for given diagnoses is likely to magnify differences between the cohorts, whether there are real differences or not or whether the differences between “vaxxed” and “unvaxxed” children observed are due to ascertainment bias due to Thomas’ quack practice and antivaccine beliefs plus differences in health-seeking behavior. Think about it. Any of the diagnoses examined in this study is likely to result in multiple office visits per diagnosis. Instead of one child with a diagnosis compared to one child without the diagnosis, you get many visits due to one child with a diagnosis compared to zero visits from one child without the diagnosis. RIOV looks custom-designed to amplify small differences, whether they’re clinically relevant or not. Maybe that’s the point. Heck, the authors even admit it when for Figure 6 they perform simulations to show that RIOV is “more powerful” than disease incidence:
The simulations were not intended to precisely model the data from the current study; instead, it is intended to demonstrate the principle that the loss of information caused by using the incidence of health condition rather than the more sensitive measure of the number of office visits results in a loss of power to detect adverse events.
Over the range studied, the average increase in power achieved from the analysis using RIOV compared to the odds ratio of diagnoses was doubled over that of odds ratio on incidence of diagnoses (133%) (Figure 6). RIOV was more powerful compared to OR on rates of diagnosis over the simulated range. Our results demonstrate that drug and vaccine safety studies should employ RIOV rather than OR on rates of diagnosis of health conditions that might be attributable to the treatment, therapy, or vaccine.
See what I mean? What they are doing is cranking up sensitivity at the cost of specificity. As for their argument that RIOV should be used rather than incidence of adverse events/diseases in vaccine safety studies, that’s utterly ridiculous, particularly given that they make this argument based on billing data and that prospective ascertainment of adverse events based on actual medical records will provide a far more accurate estimate than some made-up metric in a retrospective study using billing records.
The bottom line is that, COVID-19 or no COVID-19, antivaxxers gonna antivax, and that’s just what Thomas and Lyons-Weiler are doing here. They published a crappy study based on a metric they made up that hasn’t been validated elsewhere and use that to claim that “vaxxed” children are much less healthy than “unvaxxed” children, when in fact the data we have suggest emphatically that the opposite is true. This study is propaganda, not science, and there’s going to be more to come, as Margulis crows:
Researchers at IPAK (The Institute for Pure and Applied Knowledge) tell me they are currently working on Phase 2 of the study. Phase 2 will specifically focus on whether there are different risks associated with aluminum-containing virus. aluminum-free vaccines, differences in health outcomes in children receiving live virus inactivated vaccines, and whether specific vaccines are associated with specific poor health outcomes.
Oh, goody. More p-hacking. How will I ever be able to predict the results of the followup study?