From time to time, there are dubious studies that pop up that I mean to discuss and, for whatever reason, never get around to or, having failed to discuss them in a timely fashion, decide that no one cares any more and don’t come back to. On the other hand, sometimes there are studies like what I just described that keep bugging me, that keep coming back in a niggling fashion in such a way that eventually I just say, “WTF?” and discuss them, even though I probably should have done so a week or two ago. A new study by Peter Doshi and colleagues that purports to be a reanalysis of the adverse events data from the original phase 3 clinical trials of the Pfizer and Moderna mRNA-based COVID-19 vaccines that the companies used to gain Emergency Use Authorizations (EUAs) for their products over a year and a half ago is one such study, and last night I just gave in.
The study was published as a preprint in early June, but, as often happens with studies antivaxxers widely share as evidence that vaccines don’t work or are dangerous, it took at least a couple of weeks before it started to gain traction on antivax Twitter. Personally, I trace its real takeoff to when Jordan Peterson cited it a week ago:
Much was also made of Peter Doshi’s status as a senior editor at The BMJ, one of the oldest and most respected journals in the world:
As of this morning (when I decided to make one last check before publishing this post), this is what the analytics for the preprint look like:
Unfortunately, the manuscript, entitled Serious Adverse Events of Special Interest Following mRNA Vaccination in Randomized Trials, authored by Joseph Fraiman, Juan Erviti, Mark Jones, Sander Greenland, Patrick Whelan, Robert M. Kaplan, and corresponding author Peter Doshi, is making a splash.
The first thing I wondered when I read this study for the first time is more of a meta issue, specifically: Why was this study done? First, let’s look at the introduction to Doshi’s paper:
We sought to investigate the association between FDA-authorized mRNA COVID-19 vaccines and serious adverse events identified by the Brighton Collaboration, using data from the phase III randomized, placebo-controlled clinical trials on which authorization was based. We then use the results to illustrate the need for formal harm-benefit analyses of the vaccines that are stratified according to risk of serious COVID-19 outcomes, as well as contextualize the findings against post-authorization observational data.
What is the Brighton Collaborative? (I’m actually surprised that I hadn’t heard of it before.) It’s a group dedicated to vaccine safety and improving the scientific rigor of vaccine science. Learning that, to be honest, made me a bit annoyed at some criticisms of the study that the serious adverse events (SAEs) had been pulled out of someone’s nether regions. That the Brighton Collaborative doesn’t absolve Doshi for his chicanery in this paper, but we should be careful regarding criticisms we make.
Back to my meta question, though: Why does this study exist? I’ll preface my answer by pointing out a simple observation. It’s been well over a year and a half since the randomized clinical trial (RCT) results for the Pfizer and Moderna vaccines were first reported. Both of them involved only ~43K and ~30K participants, respectively. Next, I will point out that even large randomized clinical trials used to approve drugs and vaccines miss less common adverse events (AEs), including SAEs. That’s why we do postmarketing surveillance studies, particularly for vaccines. Less common AEs sometimes don’t show up until after a vaccine is rolled out and distribution goes from a population of tens of thousands to administration to millions, tens of millions, hundreds of millions, and even billions, as has happened with the Pfizer and Moderna COVID-19 mRNA vaccines during the more than a year and a half since EUAs were granted. In other words, if you are interested in the safety and efficacy of COVID-19 vaccines right here, right now, in mid-2022, over a year and a half after the EUAs were granted, the original RCT data are not the best data to use to estimate rates of adverse events. Over 10 billion doses have been administered since then, and numerous countries have safety and efficacy data.
So, given that background, why reanalyze the original RCT results from Pfizer and Moderna at all? There’s one reason, and one reason only, that scientists might want to reanalyze data from a completed and published clinical trial, specifically if they think that there was something wrong with the original RCT or how it was analyzed. The subtext, of course, is that they might do it if they suspect some sort of serious flaw in the RCT design or how the RCT was carried out. They might even suspect outright fraud. Doshi and his colleagues don’t explicitly say this, but if you know Doshi’s history you’ll understand that the real reason he undertook this analysis was almost certainly because he thought that the RCTs for the Pfizer and Moderna vaccine didn’t show what they claimed to show and were analyzed in such a way to exaggerate efficacy and hide adverse events. It’s not as though he’s made a secret of this belief, given that he’s a senior editor at The BMJ who’s been allowed to use the journal as his own soapbox.
Before I discuss this paper, let’s look at a bit of Doshi’s history on this issue, which is pretty clear. He’s argued in editorials for The BMJ that the vaccines aren’t as effective or safe as advertised, starting in January 2021, when he outright tried to argue (using truly awful methodology) that the efficacy of the Pfizer was much lower than reported because a large number of COVID-19 cases were missed. He was wrong. In fact, he was worse than wrong. Doshi’s methodology turned into an antivaccine meme that still circulates today, albeit in a somewhat different form. It’s similar to a technique that he’s used to attack influenza vaccines going back at least to 2006 and continuing for years and years after that, with his articles being approvingly cited by antivaxxers as prominent as Robert F. Kennedy Jr. This preprint is nothing more than the latest in a long line of articles by Doshi calling the safety and efficacy of vaccines into doubt and asking “Do we need more evidence?” Unsurprisingly, his conclusions are always that the vaccines aren’t safe and effective as the studies claim and that we do need more evidence. While no vaccine advocate would ever dispute that more data are a good thing, Doshi’s calls for “more data,” more than anything else, resemble JAQing off, rather than an honest call for more information and study.
It turns out that Doshi isn’t the only author of the paper known for misinformation about COVID-19 and COVID vaccines. The first author is Joseph Fraiman. The name sounded very familiar to me, but I couldn’t place him, although he’s billed as an emergency medicine physician in Louisiana. So I searched the blog, and it turns out that I’ve mentioned him before. He was a speaker at antivaxxer Steve Kirsch’s quackfest in which he claimed that COVID-19 vaccines have killed a half a million people. It further turns out that Fraiman has no expertise in vaccines, infectious disease, epidemiology, or pandemics, which is probably why he appeared on an “Urgency of Normal” panel organized by Florida Gov. Ron DeSantis to argue against vaccinating children. Dr. Fraiman, it turns out, is the first author of the paper. In papers in the biomedical literature, the two most important authors are the first author and corresponding author, the latter of whom is the senior author under whose supervision all the research is carried out and who is the point person for submission of the manuscript, answering peer review issues, and postpublication contact. Under “Author Contributions,” it’s noted that Fraiman and Doshi drafted the manuscript, which makes a lot of sense given how bad it is and how it tries to promote an antivaccine narrative.
Fraiman and Doshi aren’t the only COVID-19 contrarians on the author list either. Another author, Patrick Whelan, has popped up before in my reading. He’s a pediatric rheumatologist who was referred to in a positive light all the way back in February 2021 and before that supported the fear mongering of Human Noorchasm in a positive light by Lynn Redwood on Robert F. Kennedy Jr.’s Children’s Health Defense website for having claimed that the spike protein in the Pfizer and Moderna vaccines can cause heart attacks and strokes. Meanwhile, Robert Kaplan is Stanford faculty who, interestingly, expressed disbelief that antivaccine sentiments were more widespread than expected. Now, by being a co-author on this study, which is designed to imply that COVID-19 vaccines do more harm than good, he’s contributing to them.
No doubt at this point Doshi’s defenders (and there are a significant number of them, unfortunately) will claim that I’m basing my criticism on an ad hominem attack. Such claims misunderstand the nature of ad hominem. If I dismissed this study solely because Doshi had served as corresponding author on it, then that would be an ad hominem. It is not, however, an ad hominem to point out Doshi’s history and how the methodology of this study is consistent with his history of doing everything he can to portray vaccines as less effective and safe than accepted based on current evidence and then to discuss the problems with this study.
Enough about the authors. Let’s get back to the paper, which is yet another example of what I like to refer to as weaponizing the medical literature to spread disinformation. Here’s one big “tell” regarding what this is really about:
The Pfizer and Moderna trials are expected to follow participants for two years. Within weeks of the emergency authorization, however, the sponsors began a process of unblinding all participants who elected to be unblinded. In addition, those who received placebo were offered the vaccine. These self-selection processes may have introduced nonrandom differences between the vaccine and unvaccinated participants, thus rendering the post-authorization data less reliable. Therefore, to preserve randomization, we used the interim datasets that were the basis for emergency authorization in December 2020, approximately 4 months after trials commenced.
Here Doshi is echoing a common antivax talking point, in which it is claimed that the unblinding was carried out to hide AEs and much lower efficacy than reported based on the data used to obtain EUAs for the vaccines. Of course, the question of whether or not to unblind a clinical trial is a complex issue and depends on the intersection of bioethics and science. In the case of COVID-19 vaccines, after efficacy and safety were demonstrated in the first analyses, it became unethical to leave the control groups of those studies unprotected against COVID-19, which was surging around the world and causing mass illness, disability, and death. I won’t go into more detail, as I discussed the issue of unblinding the trials in detail over a year ago. Moreover, multiple assessments of the trials have concluded that both were rigorously conducted.
As I read this study, the first thing that came to mind was p-hacking. This study reeks of it. In brief, p-hacking (also known as data dredging) is a technique in which multiple analyses are done by exhaustively searching and comparing the variables alone and in combination, until a “statistically significant” result is found, regardless of whether that result is scientifically or clinically meaningful. Basically, p-hacking is a technique to make nonsignificant results seem “significant.” Basically if you compare enough things or combine and then compare enough things, you can almost always find a spuriously “statistically significant” result.
So let’s look at the paper.
Instead of looking at all AEs, as the papers and reports analyzing the data from the clinical trials did, Doshi and his colleagues decided to focus on “serious adverse events of special interest” (SAESIs, sometimes use called AESIs in the manuscript, something that at times confused me about what the authors meant in any one section). The first version of this list was published early in the pandemic based on five reports from China and has undergone a total of four updates, the most recent of which was published last September. These SAESIs were determined, as described by Doshi:
This effort created an AESI list which categorizes AESIs into three categories: those included because they are seen with COVID-19, those with a proven or theoretical association with vaccines in general, and those with proven or theoretical associations with specific vaccine platforms. The first version was produced in March 2020 based on experience from China. Following the second update (May 2020), the WHO Global Advisory Committee on Vaccine Safety (GACVS) adopted the list, and Brighton commenced a systematic review process “to ensure an ongoing understanding of the full spectrum of COVID-19 disease and modification of the AESI list accordingly.”7 This resulted in three additional AESIs being added to the list in December 2020. The subsequent (and most recent fourth) update did not result in any additional AESIs being added to the list.
We matched SAEs recorded in the trial against an expanded list of AESIs created by combining Brighton’s SPEAC COVID-19 AESI list with a list of 29 clinical diagnoses Brighton identified as “known to have been reported but not in sufficient numbers to merit inclusion on the AESI list.”7
So right away, I wondered how these diagnoses were being combined, mixed, and matched. If you look at the Brighton Collaborative document, you’ll see a lot of unremarkable standard AEs and SAEs, but you’ll also find ones that require intepretation. Here is the table defining its AESIs in Brighton’s Safety Platform for Emergency Vaccines (SPEAC):
The AESIs included because they have a theoretical or proven association with specific vaccine platforms are interesting, mainly because none of them are associated with the mRNA platform, but rather platforms that existed before the mRNA-based COVID-19 vaccines were released. Also note how the AESIs are (mostly) listed as broad categories, rather than specific diagnoses. Exceptions include, of course, myocarditis, which is associated with COVID-19 and has been associated in safety data with COVID-19 vaccines, but mapping the AEs in the clinical trials to these categories requires some subjectivity. There are more than just what’s listed above, delineated in a number of charts under each organ system. Colitis is listed in Annex 6, which encompasses the gastrointestinal system.
Indeed, here’s a passage that jumped out at me as a surgeon (who was well trained in dealing with abdominal pain, for instance) and clinician and suggested the subjectivity to me:
For SAEs that described symptoms, not diagnoses, the clinician reviewers independently judged whether each SAE type was likely to have been caused by an AESI. For example, the SAE “abdominal pain” is a symptom based diagnosis, which was judged as fitting within the SPEAC clinical diagnosis of “colitis/enteritis.” Disagreements were resolved through consensus; in two cases, consensus could not be reached and were resolved by the judgment of a third clinician reviewer (PW) to create a majority opinion. For each included SAE, we recorded the corresponding Brighton Collaboration AESI category and organ system.
Here’s a hint: Not all abdominal pain is due to colitis (inflammation of the colon) or enteritis (inflammation of the intestines). True, these are common causes, but there are so very many others. Similarly, what mapped to myocarditis and pericarditis? There are lots of causes of chest pain other than myocarditis and pericarditis.
For completeness, let’s look at the list (taken from Supplemental Table 1) of the included AESIs:
Included SAE types (matching AESI list): Abdominal pain, Abdominal pain upper, Abscess, Abscess intestinal, Acute coronary syndrome, Acute kidney injury, Acute left ventricular failure, Acute myocardial infarction, Acute respiratory failure, Anaemia, Anaphylactic reaction, Anaphylactic shock, Angina pectoris, Angina unstable, Angioedema, Aortic aneurysm, Aortic valve incompetence, Arrhythmia supraventricular, Arteriospasm coronary, Arthritis, Atrial fibrillation, Atrial flutter, Axillary vein thrombosis, Basal ganglia haemorrhage, Bile duct stone, Blood loss anaemia, Bradycardia, Brain abscess, Cardiac failure, Cardiac failure acute, Cardiac failure congestive, Cardiac stress test abnormal, Cardio-respiratory arrest, Cerebral infarction, Cerebrovascular accident, Chest pain, Cholecystitis, Cholecystitis acute, Cholelithiasis, Colitis, Coronary artery disease, Coronary artery dissection, Coronary artery occlusion, Coronary artery thrombosis, Deep vein thrombosis, Dermatitis bullous, Diabetic ketoacidosis, Diarrhoea, Diplegia, Dyspnoea, Embolic stroke, Empyema, Facial paralysis, Fluid retention, Gastroenteritis, Gastrointestinal haemorrhage, Haematoma, Haemorrhagic stroke, Hemiplegic migraine, Hepatic enzyme increased, Hyperglycaemia, Hyponatraemia, Hypoxia, Ischaemic stroke, Laryngeal oedema, Multiple sclerosis, Myocardial infarction, Noncardiac chest pain, Oedema peripheral, Pancreatitis, Pancreatitis acute, Pericarditis, Peripheral artery aneurysm, Peritoneal abscess, Pleuritic pain, Pneumothorax, Post procedural haematoma, Post procedural haemorrhage, Postoperative abscess, Procedural haemorrhage, Psychotic disorder, Pulmonary embolism, Rash, Rash vesicular, Respiratory failure, Retinal artery occlusion, Rhabdomyolysis, Rheumatoid arthritis, Schizoaffective disorder, Seizure, Subarachnoid haemorrhage, Subcapsular renal haematoma, Subdural haematoma, Tachyarrhythmia, Tachycardia, Thrombocytopenia, Thyroid disorder, Toxic encephalopathy, Transaminases increased, Transient ischaemic attack, Traumatic intracranial haemorrhage, Type 2 diabetes mellitus, Uraemic encephalopathy, Uterine haemorrhage, Vascular stent occlusion, ventricular arrhythmia.
And now let’s compare to the list of SAEs that were excluded by Doshi because they didn’t match the AESI list:
Excluded SAE types (not matching AESI list): Abdominal adhesions, Abortion spontaneous, Abortion spontaneous incomplete, Accelerated hypertension, Adenocarcinoma gastric, Adrenal gland cancer, Alcohol abuse, Alcohol poisoning, Alcohol withdrawal syndrome, Animal bite, Ankle arthroplasty, Ankle fracture, Anxiety, Anxiety disorder, Aortic stenosis, Appendicitis, Appendicitis perforated, Arteriosclerosis, Asthma, Atelectasis, Autonomic nervous system imbalance, B-cell small lymphocytic lymphoma, Back injury, Back pain, Benign prostatic hyperplasia, Bipolar disorder, Breast cancer, Breast cancer stage I, Breast hyperplasia, Bronchitis, Cartilage injury, Cellulitis, Cervical radiculopathy, Cervical spinal stenosis, Cervical vertebral fracture, Choroidal neovascularisation, Chronic kidney disease, Chronic lymphocytic leukaemia, Chronic myeloid leukaemia, Chronic obstructive pulmonary disease, Clostridium difficile colitis, Clostridium difficile infection, Colon cancer stage III, Colon injury, Colorectal cancer, Completed suicide, Complicated appendicitis, Concussion, Confusional state, Constipation, Cough, Craniocerebral injury, Dehydration, Depression, Diplopia, Diverticular perforation, Diverticulitis, Dizziness, Drug hypersensitivity, Duodenal ulcer, Duodenal ulcer haemorrhage, Emphysema, Facial bones fracture, Fall, Feeling hot, Femoral neck fracture, Femur fracture, Fibromuscular dysplasia, Flail chest, Flank pain, Food poisoning, Foot fracture, Foot operation, Forearm fracture, Fracture nonunion, Gastric cancer, Gastric perforation, Gastrooesophageal reflux disease, Gout, Gun shot wound, Head injury, Heart disease congenital, Hepatic cancer metastatic, Hepatic mass, Hepatitis A, Hernia, Hiatus hernia, Hip arthroplasty, Hip fracture, Humerus fracture, Hypertension, Hypertensive emergency, Hypertensive urgency, Hypoglycaemia, Hypokalaemia, Hypomagnesaemia, Hypotension, Idiopathic intracranial hypertension, Immunisation anxiety related reaction, Incarcerated hernia, Incision site pain, Influenza like illness, Intentional self-injury, Interstitial lung disease, Intervertebral disc degeneration, Intervertebral disc protrusion, Intestinal obstruction, Intestinal perforation, Intraductal proliferative breast lesion, Invasive ductal breast carcinoma, Invasive lobular breast carcinoma, JAMMED RIGHT INGUINAL HERNIA@@, Jaw operation, Joint injury, Knee arthroplasty, Large intestine perforation, Lead dislodgement, Leiomyosarcoma metastatic, Leydig cell tumour of the testis, Ligament rupture, Loss of consciousness, Lower limb fracture, Lung cancer metastatic, Lymphadenopathy, Major depression, Malignant melanoma, Meningioma, Mental disorder, Metabolic acidosis, Metastases to central nervous system, Migraine, Multiple injuries, Musculoskeletal chest pain, Nausea, Neck pain, Nephrolithiasis, Neutropenia, Obstructive pancreatitis, Oesophageal carcinoma, Oesophageal food impaction, Organising pneumonia, Orthostatic hypotension, Osteoarthritis, Osteochondritis, Osteomyelitis, Ovarian cyst, Ovarian mass, Overdose, Pancreatic mass, Papillary thyroid cancer, Paraesthesia, Pelvic neoplasm, Penile cancer, Penile neoplasm, Peritonitis, Pharyngitis streptococcal, Pleural effusion, Pneumonia, Pneumonia aspiration, Pneumonia staphylococcal, Pneumonitis, Polymyalgia rheumatica, Postoperative wound infection, Precancerous condition, Prostate cancer, Prostate cancer metastatic, Pulmonary mass, Pyelonephritis, Pyelonephritis acute, Rectal prolapse, Renal cancer, Renal cell carcinoma, Renal colic, Retinal detachment, Retinal tear, Rib fracture, Road traffic accident, Salivary gland calculus, Salpingitis, Sepsis, Septic shock, Sexual abuse, Shoulder injury related to vaccine administration, Skin laceration, Small intestinal obstruction, Speech disorder, Spinal cord injury cervical, Spinal fusion surgery, Spinal stenosis, Staphylococcal infection, Streptococcal sepsis, Suicidal ideation, Suicide attempt, Suspected COVID-19, Swelling face, Syncope, Systemic inflammatory response syndrome, Tendon rupture, Thoracic vertebral fracture, Thyroidectomy, Toxic shock syndrome, Toxicity to various agents, Transient global amnesia, Traumatic liver injury, Ulna fracture, Umbilical hernia, Unevaluable event, Urinary bladder polyp, Urinary tract infection, Urosepsis, Uterine leiomyoma, Uterine prolapse, Vertigo, Viral pharyngitis, Volvulus, Vomiting, Wound infection, Wrist fracture.Note: I’m not sure why “jammed right inguinal hernia” is in all caps. I’ll presume it’s a typo.
A lot of the SAEs in the second list make sense given that they include fractures, gunshot wounds, head injuries, and the like, but a number do not, such as viral pharyngitis, volvulus, vomiting, and others. Dr. Susan Oliver has posted a video discussing the problems with this preprint. She didn’t really so much discuss the meta problems with it and Doshi’s history, but she did note many of the same things that I did, in particular the odd choices of what was and wasn’t included as SAESIs. For example, Doshi included diarrhea, but not vomiting (or, as the surgeon in me can’t help but note, intestinal perforation or volvulus, the latter a known complication of a certain vaccine); hyperglycemia (high blood sugar) but not hypoglycemia (low blood sugar); gastrointestinal hemorrhage but not duodenal ulcer hemorrhage (which is a form of gastrointestinal hemorrhage); and coronary artery disease but not atherosclerosis (which causes coronary artery disease). It’s all very curious. Perhaps the most important issue is that “events related to COVID-19” were excluded, which on the surface makes sense, but, given that COVID-19 cases were much more common in the placebo controlled group, automatically biases the results for the remaining SAEs to the vaccine-group.
That’s not all, though. Instead of comparing the number of people who had SAEs, they did this:
In their review of SAEs that supported the authorization of the Pfizer and Moderna vaccines, the FDA concluded that SAEs were, for Pfizer, “balanced between treatment groups,”14 and for Moderna, were “without meaningful imbalances between study arms.”15 In contrast to the FDA analysis, we found an increased risk of all cause SAEs in the Pfizer trial. While our analysis excluded SAEs related to COVID-19 (because it is an efficacy outcome), this exclusion did not explain the difference given the low risk of SAEs attributed to COVID-19 (0 in the vaccine arm, 1 in the placebo arm). Instead, the difference in findings may in part be explained by the fact that the FDA analyzed the total number of participants experiencing any SAE, whereas our analysis was based on the total number of SAE events. Given that approximately twice as many individuals in the vaccine group experienced multiple SAEs than the placebo group (there were 24 more events than participants in the vaccine group, compared to 13 in the placebo group), FDA’s analysis of only the incidence of participants experiencing any SAE would not reflect the observed increase in multiple SAEs in the vaccine group.
To put it briefly, they compared number of SAEs, not the number of patients who suffered an SAE. This sort of analysis is guaranteed to double count SAES—at least!—because some of the SAEs or groups of SAES will be linked. For example, as Dr. Oliver points out, abdominal pain often goes along with diarrhea, to which I would add that colitis or enterocolitis can lead to gastrointestinal hemorrhage. Moreover, formal reporting systems for clinical trial AEs require that all AEs be entered, even when they are related, which is why analyses are usually done at the patient-level, as in “number of patients who suffered this AE,” rather than in total AEs reported in each group independent of the number of patients. I’d be willing to bet that if the same statistical analysis were done using per-patient-level data rather than SAE-level data the statistical significance would likely disappear. At the very least, if I were reviewing this paper, I would refuse to publish it until such an analysis was included for comparison.
The “money” chart that antivaxxers are sharing is Table 2:
First of all, this chart demonstrates the power of cherry picking. Notice first the huge confidence intervals. Next, notice how for the combined data for all SAEs there is no statistically significant difference. Now notice how, even for the SAESIs, for the individual trials there is no statistically significant difference until you combine the two. Moreover, all they could find was an additional 12.5 SAESIs per 10,000 participants (with, I can’t help but adding, a 95% confidence interval of 2.1 to 22.9, again a huge uncertainty).
But that’s not all. Perhaps the most dubious—dare I say dishonest, even?—part of the paper is a comparison that Doshi makes between the number of SAESIs reported and hospitalizations due to COVID-19 observed in the placebo control group:
In the Moderna trial, the excess risk of serious AESIs (15.1 per 10,000 participants) surpassed the risk reduction for COVID-19 hospitalization relative to the placebo group (6.4 per 10,000 participants).3 In the Pfizer trial, the excess risk of serious AESIs (10.1 per 10,000) surpassed the risk reduction for COVID-19 hospitalization relative to the placebo group (2.3 per 10,000 participants).
Think of it this way. You can only hospitalize patients, not SAESIs (or AESIs or AEs). An individual patient in the vaccine group could suffer more than one AE, but a a patient in the placebo control group could only be hospitalized once (in the context of the limited timeframe of the clinical trial) for COVID-19. Here Doshi is comparing apples and oranges in order to make it look as though the vaccines were more dangerous than actually getting COVID-19, which is a ridiculous contention given what we know. Moreover, in clinical trials in general a lot of the “serious adverse events” are not serious enough to warrant hospitalization. In fact, according to the standard terminology used to rate SAEs in clinical trials grade 3 events and above (on a five-point scale) are rated severe. If you look at the list of specific AEs, you’ll see that some grade 3 AEs require hospitalization; some don’t. Grade 3 is defined as an AE that:
- Is severe or medically significant but not immediately life-threatening; OR
- Requires hospitalization or prolongation of hospitalization indicated; OR
- Limits self care/activities of daily living (ADL)
For completeness, I’ll mention that grade 4 AEs are by definition life-threatening events that require urgent intervention and that grade 5 events are by definition AEs that result in death. Any rigorous evaluation would compare hospitalizations due to AEs in control versus hospitalizations due to AEs in the vaccine group, not AEs (regardless of whether they are AESIs or just AEs). Again, Doshi’s comparison is deeply intellectually dishonest. The only thing Dr. Oliver didn’t consider in her discussion was AE grades.
There’s another issue here as well. The rate of hospitalizations in the placebo control group would be expected to be highly dependent on the level of COVID-19 that was circulating in the populations tested during the time period in which the clinical trial was carried out, as these trials were not challenge trials, in which subjects are intentionally exposed to the virus. As a result, most people in the placebo and vaccine groups were not exposed to COVID-19, because these trials were carried out in the summer and early fall of 2020, before the really big winter surge hit.
Here’s a graph of COVID-19 cases in the US in 2020:
Enrollment for the Moderna trial ended on October 23, 2020; for the Pfizer trial, November 14, 2020. Note that this was before the winter surge took off. Had the trial started a few months later and ended in, for example, February 2021 or later, you can bet that the rates of hospitalization for COVID-19 would have been much higher in the placebo control group.
The bottom line is that this study is deeply misleading based on what sure looks like p-hacking combined with misleading comparisons, further combined with a low enough risk of COVID-19 in the two populations to allow for a low rate of hospitalization when normalized to the entire population in the control group.
This brings me back to Peter Doshi. If you didn’t know Doshi’s history, you might very well take this study at face value. Sadly, The BMJ hired Doshi, who is now a senior editor, despite his long history of playing footsie with the antivaccine movement since at least 2009, amplifying antivaccine conspiracy theories, downplaying the severity of influenza and thus feeding antivaccine narratives, using sleight-of-hand to downplay the effectiveness of flu vaccines, and generally playing the role of a false skeptic with respect to vaccines, as well as having signed a petition in 2006 “questioning” whether HIV causes AIDS. It continued to employ him even after he’d fallen for a conspiracy theory that the Vaccine Adverse Events Reporting System (VAERS) database was being made inaccessible to suppress report. Even worse, Doshi has also served as an expert witness for the plaintiffs in antivaccine leader Robert F. Kennedy Jr.’s lawsuit against the University of California’s influenza vaccine mandates and taken part in a “roundtable” organized by Sen. Ron Johnson to go dumpster diving in VAERS to find “vaccine injuries” due to COVID-19 vaccines, whether the injuries were caused by them or not.
In his testimony at Sen. Johnson’s quackfest, Doshi denied that COVID-19 at the time (November 2021) was a “pandemic of the unvaccinated”, citing a report from July from the UK that most hospitalizations are among the fully vaccinated. It turned out that this report was in error, substituting “vaccinated” for “unvaccinated” and the majority of hospitalizations were among the unvaccinated, even though they made up only 31% of the population at the time. He even cited cherry-picked tables to claim that the vaccine wasn’t saving lives in what was basically an updated rehash of the nonsense he had peddled a few months earlier in which he claimed that there was “no biodistribution data” for COVID-19 vaccines and made a number of other negative false claims about the vaccines (also deconstructed by Dr. Hilda Bastian). In a truly risible moment, Doshi even cited the Merriam-Webster definition of “antivaxxer” as opposed to those supposedly opposed to vaccine mandates to argue that he and his fellow COVID-19 contrarians were “not antivaccine” and that large numbers of people would qualify as “antivaccine”. He even parroted the antivaccine talking point that mRNA vaccines are not really vaccines and therefore shouldn’t be mandated like vaccines.
I’ll conclude by scratching my head and wondering why Sander Greenland signed on to Doshi’s “study,” which was predestined to find something bad about mRNA-based COVID-19 vaccines. For those of you who haven’t heard of him, Greenland is a giant in the world of statistics, and I myself have cited his work on Bayesian statistics and reasoning when giving talks about the differences between science-based medicine and evidence-based medicine to support my argument that science-based medicine is superior and, in fact, necessary. Given that Greenland apparently had no role in deciding which AEs were included (at least not as far as I can tell) and Doshi is listed as having been the investigator solely responsible for data acquisition, I can only conclude that Greenland mistakenly trusted Doshi and did his standard skilled statistical analysis on a fatally flawed dataset. At least, I hope that that’s what happened. Unlike, for example, John Ioannidis, for now I have to give Sander Greenland the benefit of the doubt.
Unfortunately, Peter Doshi has spent his career amplifying antivax narratives, either wittingly or unwittingly, disguised as demanding more rigor in scientific trials. Even worse, The BMJ continues to employ him, thus providing him with a platform that gives him the appearance of scientific authority that allows him to attract respected academics to help him spread dubious studies custom-made for weaponization by antivaxxers, making its leadership complicit in his spreading misinformation.