Clinical trials Complementary and alternative medicine Skepticism/critical thinking

The return of the revenge of a “more fluid concept of evidence”

Dr. David Katz is undoubtedly a heavy hitter in the brave new world of “integrative medicine,” a specialty that seeks to “integrate” pseudoscience with science, nonsense, with sense, and quackery with real medicine. In fairness, that’s not the way physicians like Dr. Katz see it. Rather, they see it as “integrating” the “best of both worlds” to the benefit of patients. However, as we’ve documented extensively here, on our personal blogs, and even in the biomedical literature (plug, plug), what “integrative” medicine means in practice is indeed what I characterized, the infiltration of woo into medicine. This infiltration seems to have started mainly in academia—hence the term “quackademic medicine” and “quackademia”—with the steady infiltration of nonsense into medical schools and academic medical centers, but has since metastasized to the world of community hospitals. This “integration” (or, as I like to refer to it, “infiltration”) has become so pronounced that a few years ago The Atlantic published an article entitled “The Triumph of New Age Medicine“, and just last December the Journal of the National Cancer Institute (JNCI) published a monograph full of articles touting “integrative oncology,” including guidelines recommended by the Society of Integrative Oncology (SIO) for the “integrative” treatment of breast cancer symptoms.

I mention Dr. Katz for two reasons. First, he’s taken another broadside at us at Science-Based Medicine in blog entry at The Huffington Postwhere else?—entitled “Holism, Holes and Poles” that I’ve been meaning to address for a while. But before I address Dr. Katz’s most recent complaint against science-based medicine (SBM), it’s necessary to step back and look at some history.

A more “fluid” concept of evidence

Longtime readers might recall that Dr. Katz is on faculty at Steve Novella’s institution, Yale. More importantly he is (in)famous for making a statement that utterly blew me away seven years ago when I first encountered it, courtesy of David Colquhoun, portrayed in this video, specifically beginning at around 6:05:

Unfortunately, the original video I cited in my original post was taken down due to a copyright claim from Yale, but this one remains. The problem is that it is missing bits that were discussed when the talk was widely discussed in the medical blogosphere. However, the money quotes are still there, such as:

I think we have to move beyond the results of RCTs in order to address patient needs, and to do that I’ve arrived at the use of a more fluid concept of evidence than many of us have imbibed from our medical educations…[Referring to an anecdote of a patient with chronic pain Dr. Katz went on.]…Now, we don’t want you on narcotics any more than you want to be on narcotics. We initiated a course of acupuncture and over the next two to three months weaned him off narcotics. He was pain-free on acupuncture and subsequently transitioned into homeopathy. Now, I don’t care to get into a discussion of how or even whether homeopathy even works, but this guy had tried everything.


And the anecdotal evidence that homeopathy might be effective was brought up by the naturopaths. We tried it. It worked.

Later in the talk, he mentions an NIH consensus statement about the use of multivitamins that concluded that there was insufficient evidence to determine if they work and says:

And so I would argue that here, too, is an invitation to think more fluidly about evidence. Absence of evidence is not evidence of absence.

Whether or not absence of evidence is evidence of absence depends, of course, on how hard scientists have actually looked for evidence. Of course, the proper retort to such seemingly-sensible nonsense is to suggest that maybe—just maybe—you shouldn’t be making such confident recommendations until there is actually some convincing supportive evidence. Moreover, in the context of discussing a bunch of negative clinical trials that Dr. Katz apparently doesn’t believe, it is fairly clear that he’s implying that negative clinical trials are an “absence of evidence.” They are not. They are evidence. While from a frequentist perspective, it is true that a negative clinical trial does not necessarily mean that a treatment does not work (it just means that the results observed in the treatment and control groups were statistically indistinguishable, to put it in simple form), when negative clinical trials, particularly if they are well-designed, pile up, that absence of positive evidence that a treatment works becomes fairly powerful evidence that it doesn’t work. In any case, this “more fluid concept of evidence” clearly involves patient anecdotes, which are, for the many reasons explained here over the years, often unreliable and even misleading. That is why they are relegated to a much lower rung on the ladder of evidence than clinical trials.

As you might imagine, quite a few skeptical bloggers have discussed Dr. Katz’s idea of “a more fluid concept of evidence,” including Steve Novella, Kimball Atwood, and myself. More recently, Dr. Katz was very unhappy over how Steve Novella and I had criticized the Cleveland Clinic for opening a traditional Chinese medicine clinic run by a naturopath. I have characterized David Katz’s attitude as a false dilemma: Abandon patients or abandon science. And so it is. Unfortunately, it is a false dilemma that lies at the very heart of the very concept of “integrative medicine,” as you will see.

Dr. Katz lashes out again

Dr. Katz is once again very unhappy with us over here at SBM. One can’t help but notice right away how he refers to us (without naming us, although he does at least link to us) as the “self-proclaimed guardians of the true definition of evidence-based medicine” (EBM). Ouch. He sure makes us sound as though we are uptight, doesn’t he? Except that we don’t really do that. Rather, we advocate changing the definition of EBM to be more like SBM because of the obvious defects in the definition of EBM that allow pseudoscience to infiltrate medicine. It’s not as though we haven’t discussed this more times than I can remember.

I get it, though. No one likes being criticized or having his favorite ox gored. I can even sort of understand why Dr. Katz seems to think that we are a bunch of peons unworthy to criticize him, as once again he has to lay down his bona fides in a rather obvious appeal to authority:

I opened an Integrative Medicine Center some 15 years ago, and have directed it, and seen patients in that context, since. I did so not because of any long suppressed desire to practice medicine under that banner, and certainly not because of any inclination to don Birkenstocks.

In fact, I was — as I remain- a card carrying member of the evidence-based medicine club. I was already running then, as I still am, a federally funded clinical research lab. I was already then, as I am now, routinely publishing studies in the peer-reviewed literature. I was teaching then, as I did for roughly ten years, biostatistics and clinical epidemiology to Yale medical students. And while back then I had co-authored a textbook on epidemiology and biostatistics, I have since co-authored four editions of that textbook, and a textbook on evidence-based medicine as well.

In other words: How dare you miserable, worthless, cretins criticize me? Me! Yes, Dr. Katz has been using variants of this same whine since he dismissed David Colquhoun, Steve Novella, Dr. R. W. Donnell, and myself as not having “earned the right to express an opinion” on these issues and of “simply having an opinion, an Internet connection, and some time to kill.” He even referred to us as “health hazards of the blogosphere” and quacks.

Although not to the extent that Dr. Katz has, I, too, have been federally funded and I, too, routinely publish in the peer-reviewed literature. Unfortunately, I probably stuck too closely to basic science too long before transitioning it into the realm of clinical research, given that a surgeon will always have a great deal of difficulty directly competing with basic scientists at basic science, hence my difficulties retaining external funding more recently. In retrospect, I’m amazed I did as well as I did for as long as I did competing with basic scientists and am hopeful that my recent forays into much more directly clinically relevant areas are not too late, given my priority score on a recent grant that just missed the draconian cutoff for funding in today’s—shall we say?—highly constrained funding environment.

Of course, Dr. Katz’s impressive qualifications, which I concede are more impressive than mine, do not matter much when he routinely spouts nonsense such as his justification of the use of homeopathy, naturopathy, and traditional Chinese medicine, as he has unfortunately done so often in the past. Only the quality of his arguments for the “integration” of various alternative medicine practices into conventional medicine matter, and these arguments have been uniformly of very poor quality. This time around is no different.

I’ll start with his third complaint, because it is a complaint so frequently used by advocates of quackery (whom, whether Dr. Katz realizes it or not, he is emulating by repeating that complaint), which is all about the money:

And third, there is the thorny little issue of money.

The horses that pull toward “evidence” pull a cart full of money. The average cost of bringing a new FDA-approved drug to market now approximates a billion dollars. Spending millions, tens of millions, or even hundreds of millions on clinical trials makes sense if a return in the billions is the offing. It’s pretty much a non-starter otherwise, and a problem even a budget the size of the NIH’s cannot solve. In fact, the price tag for one, new FDA-approved drug is nearly ten times the annual operating budget of the National Center for Complementary and Integrative Health.

Hmmm. Where have we heard this one before? I wonder…

Oh, yes. It becomes clear when Dr. Katz proceeds to use his favorite tale to illustrate how money supposedly rules everything in medicine. Basically, it’s a tale of co-enzyme Q10 (CoQ10), a tale he’s told many times before going back to his original “more fluid concept of evidence” talk in 2008, with periodic repetitions, up to and including Katz’s current post. CoQ10 is also known as ubiquinol and is found widely in plants. It acts at the level of the electron transport chain to facilitate the production of chemical energy through oxidative metabolism in our mitochondria. According to Dr. Katz, we tend to get most of what we need from our diet because our bodies don’t make CoQ10 very efficiently, although a recent review calls into question Dr. Katz’s claim that we get most of our CoAQ10 from our diet rather than our body’s own synthesis of it.

In any case, it was thought that CoQ10 would be a good adjunct in heart failure, in order to improve the function of the heart. In 2000, there was a small randomized double-blind placebo-controlled clinical trial that concluded that CoQ10 “does not affect ejection fraction, peak oxygen consumption, or exercise duration in patients with congestive heart failure receiving standard medical therapy,” for which investigators concluded “we detected no objective benefit from coenzyme Q10 administration in patients with heart failure” and that their “study shows no benefit to adding coenzyme Q10 to the standard treatment of heart failure.” Dr. Katz was very unhappy about this and characterized this study as being used to “drive the final nail in the CoQ10 hypothesis,” while mentioning that within a year a large RCT of a prescription drug carvedilol (trade name: Coreg) was published showing that it was safe and effective for the treatment of heart failure. To Dr. Katz, this is the evidence of big pharma primacy and the premature rejection of a promising treatment.

He then goes on:

It took more than a decade after that to undo the damage, but eventually it happened: trial data came in to show that co-Q10 could reduce heart failure mortality by about 50 percent. In other words, ten years after being declared defunct, the co-Q10 “hypothesis” was not only vindicated, but seemed to represent the greatest advance in the treatment of heart failure in years.

First of all, I can’t help but note that if the CoQ10 hypothesis was truly dead, then why was a relatively large randomized, double-blind study begun a mere three years later? Something doesn’t jibe here. If the 2000 study had truly “closed the door,” then no IRB would have approved a bigger study. Clearly, the forces of big pharma were not as powerful as Dr. Katz insinuates. I also find it odd that he didn’t actually link to the primary study by Mortensen et al, which was published in JACC Heart Failure in December. In any case, one thing that I noticed about this study right away is that there were no differences in short term endpoints; so in that, at least, this study by Mortensen agrees with the 2000 study. It was only in long-term outcomes that it showed a significant difference, with a major decrease in mortality.

The other thing that I noticed is that, judging from the introduction and discussion by Mortensen et al, Dr. Katz appears to be giving a highly biased account of what happened. There were clearly a lot more uncertainty and conflicting data than his accounts would leave you to believe, and, of course, it was completely appropriate for the original study authors to conclude that, based on their results, it was not appropriate to recommend CoQ10. That’s a provisional conclusion justified by the study at the time. That’s the way clinical science progresses after preclinical validation. Pilot studies are done. If they are promising, then small RCTs are done. If those are promising (and sometimes if they are mixed, as judging from the introduction and discussion of this paper), then larger RCTs are done. There’s nothing outrageous here, and Mortensen et al also acknowledge that their study might have enrolled a larger proportion of patients with milder symptoms than other trials.

In an accompanying editorial, Justin Ezekowitz notes that there were accrual problems, which is why the study took ten years, noting that it’s “unclear whether this situation resulted from patients’ acceptance of the experimental therapy, the ability of the sites and investigators to conduct a trial, the intensity of follow-up, or other features that are not further elucidated.” He also notes that it is premature to recommend CoQ10 as part of standard guidelines for treating heart failure, noting that these “striking results should be interpreted with caution given the small population and event numbers and the large treatment effect; many of these results were not replicated when a subsequent adequately powered trial was done (11).”

Note that the reference refers to a study by John Ioannidis that found that smaller studies generally produce large treatment effects that decline when larger trials are done.

To complete my deconstruction of Dr. Katz’s complaint about no money for studying supplements like CoQ10, I like to echo Dr. Ezekowitz’s remonstration with Mortensen et al:

Enthusiastically, the investigators have said that CoQ10 “should be added as standard therapy” based on these results, and additionally, it is a “natural and safe substance” (12). Perhaps this second statement is driven by the fewer side effects in the CoQ10 group than in the placebo group (13% vs. 19%). Further study in adequately powered clinical trials is required because it is premature to suggest that this finding reaches the necessary efficacy or safety bar required to consider prescribing this drug. Additionally, although CoQ10 is widely available and taken as a supplement, safety cannot be assumed for any drug in this context, and so much broader and more detailed experience is required to declare this substance acceptably safe. Finally, CoQ10 is natural, naturally made in a factory by pharmaceutical-grade processes of medicinal chemistry and yeast or bacterial fermentation.

Indeed. What was that about Dr. Katz denying that he believes that “natural is better,” again? After all, if CoQ10 does “work,” improving outcomes in heart failure—and it very possibly does, given the results of Mortensen’s trial—it’s working as a drug, and the actual supplement used is manufactured in a manner very much like the methods used to manufacture pharmaceuticals. In particular, I like how Dr. Eskowitz fires a shot across the bow of advocates like Dr. Katz and the supplement manufacturers:

Nutraceutical manufacturers must step up and, like pharmaceutical and device companies, fund a trial for which an indication is sought or advertised, not solely through governmental funding agencies. If 1% of the estimated $1 billion annual market for CoQ10 was reinvested into an adequately powered, appropriately designed, academically led trial to demonstrate the efficacy and safety of CoQ10, then perhaps we would have a scientific advance worth being excited about.

Yes, the lack of patentability is a problem, but it’s not an insurmountable one. The problem is that supplement manufacturers want it both ways, and, thanks to the DSHEA of 1994, a law that basically lets supplement manufacturers get away with all sorts of vague health claims without evidence, they can have it both ways. Why would they want to spend even 1% of their profits and subject themselves to FDA pharmaceutical regulation if they don’t have to? There’s more greed over nutraceuticals than just big pharma and the lack of patentability. Funny how Dr. Katz never seems to acknowledge that.

Integrative medicine “individualization”

Now let’s go back to Dr. Katz’s “more fluid concept of evidence,” which, given his arguments, sure sounds like an excuse to change the rules of evidence in the middle of the game, as Steve Novella has charged. In the case above of CoQ10, he argues that if only there had been a more “fluid” concept of evidence applied to the original 2000 trial, then maybe 13 years might not have been “wasted” and we might have been able to incorporate this miraculous supplement into treatment regimens of people with heart disease. I view it more as a cherry picked example, of Katz being lucky in probably having guessed right. How much more often do clinical investigators guess wrong after an initially-promising clinical trial? That’s what the whole concept of the “decline effect” is about, where initially-impressive effect sizes tend to decline with time as further studies are done. As I like to say, the “decline effect is science correcting itself over time.

Be that as it may, Dr. Katz, not satisfied with complaining about funding and nefarious pharmaceutical companies, launches into what I like to call the “personalization” gambit:

First, the evidence from any given clinical trial, however methodologically robust, may or may not pertain to any given patient. The application of evidence from trials to the care of individuals not in those trials is itself a product of the art of medicine, not the science. To my knowledge, there has never been a randomized trial examining different ways clinicians might decide if trial results pertain to a specific patient. That may result from the developing field of pharmacogenomics, but we have a ways to go. In other words, for now, the application of the evidence in evidence-based medicine to the actual care of patients is not, itself, evidence-based. That’s a fact.

No, actually, it is not a “fact.” It is Dr. Katz’s opinion. Also notice how Dr. Katz subtly attacks a straw man version of EBM by emphasizing only RCTs, implying that nothing is known because there hasn’t been an RCT of applying evidence-based practice (EBP) guidelines. For one thing, it’s just not true that nothing is known about the application of EBM. True, not enough is known, but a lot is known. Moreover, there have been studies of EBP guidelines, both prospective and also both simulated and in the real world. Has Dr. Katz ever heard of a surgical checklist? These are checklists that must be done before surgery to ensure that various EBP items on the list are “checked off,” thus ensuring adherence to EBP guidelines. No doubt Dr. Katz would claim that these are not applicable because they are used in a much more controlled, regimented world and don’t show how physicians apply guidelines to individual patients, but surgical patients are individual patients and checklists allow for not performing an intervention if it is deemed inappropriate for a given patient.

I also can’t help but note that in the case of CoQ10, Dr. Katz seems rather eager, based on one RCT, to adopt a “one size fits all” guideline recommending the addition of CoQ10 supplementation to the treatment of heart failure patients.

Be that as it may, Dr. Katz is also attacking a straw man. No one—and I mean no one—arguing for SBM and against the sort of integrative relativism espoused by practitioners like Dr. Katz claims that randomized clinical trial (RCT) results must always apply to a given patient with a disease presenting for evaluation. That’s why it’s called science-based medicine. Heck, that’s why EBM was originally referred to as evidence-based medicine. That’s because, ideally, medicine is based in science and evidence, but science and evidence are not the be-all and end-all.

What the patient values, for example, matters. For example, I routinely tell women diagnosed with breast cancer that mastectomy and lumpectomy followed by radiation therapy both produce equivalent survival rates. In women for whom it is appropriate, I generally recommend breast-conserving surgery (lumpectomy) first. However, even though the survival rates are the same for the two procedures, there is a somewhat higher chance of a local recurrence (recurrence in the breast) with breast-conserving surgery. The reason that that local recurrence doesn’t portend a higher risk of death is because such local recurrences can be “salvaged” with mastectomy. Some women have a very hard time dealing with even the relatively low risk of local recurrence inherent in breast conserving surgery even if the survival is the same. There’s also the issue that radiation therapy, as given, can be rather inconvenient. Most regimens involve daily treatments for four to six weeks (it used to be always six weeks), which can be a pain if the woman doesn’t live close to a center with a radiation oncology facility.

So is it “wrong” to do a mastectomy to treat a small breast cancer that could be treated with breast-conserving surgery? Of course not. I admit that I will at least try to persuade a woman whom I deem a good candidate for breast conserving surgery out of a mastectomy because it’s a much bigger operation, but if she insists on mastectomy I will do it. The point here is that informed consent is a critical component of EBM and SBM, but that informed consent must be truly informed. In other words, the information given to the patient must be accurate and based on the best science available. That’s where integrative medicine fails, both as EBM/SBM and ethically. I’ve frequently referred to the concept of “misinformed consent.” What integrative medicine does is to introduce treatments not based in science as though they were.

Dr. Katz then invokes a common scenario in medicine:

Second, the needs of patients all too often go on when evidence runs thin, or out entirely. This is why I went into “integrative” medicine in the first place — and why I am now involved in the online delivery of holistic care to a larger population.

By working with colleagues trained differently than I, naturopathic physicians in particular, we had more treatment options together than we did alone. When patients had been everywhere, tried everything subtended by RCTs and still weren’t better, they came to us. And we, generally, could come up with something reasonable to try. Admittedly, it wasn’t yet in the textbooks — but that was the point. These were the folks that had already run off the pages of textbooks, and still needed help.

This, of course, is another straw man view of EBM and SBM. Indeed, one of our criticisms of EBM is similar, but different, than Dr. Katz’s. I like to refer to it as methodolatry, defined as the profane worship of the randomized clinical trial as the only valid method of investigation. So there we agree! The RCT should not be the be-all and end-all of investigation! After all, there will never be large, double-blind RCTs to answer every medical question that needs answering. Sometimes it’s unethical to do an RCT. Sometimes it’s too impractical, either because of expense, time, or both. Sometimes the condition is just so uncommon that there will never be enough patients to do an RCT. For instance, at the recent Society of Surgical Oncology meeting in Houston, it was discussed how doing an RCT comparing outcomes in patients who undergo standard pancreatectomy to those who undergo minimally invasive pancreatectomy (laparoscopic or robot-assisted) for pancreatic cancer with overall survival as the primary outcome would require over 6,000 patients and several years. In such cases, we are forced to use the best evidence available, because sometimes the necessary trials will never be done.

Here’s the problem, and here’s where Dr. Katz and I diverge. We at SBM advocate that patients are best served when they are not subjected to unscientific and pseudoscientific medicine and when the treatments recommended to them are firmly rooted in science. Just because there is a lack of evidence regarding the best management for a given patient does not mean you can scientifically or ethically justify recommending pseudoscientific treatments like homeopathy as though they have evidence to support them! Irish comedian Dara O’Briain has a brilliant retort to this kind of argument, which is basically of the same general form of (medical) “science doesn’t know everything”:

Jesus, homeopaths get on my nerves with “science doesn’t know everything.” Well science knows it doesn’t know everything. Otherwise, it’d stop…But just because science doesn’t know everything doesn’t mean you can fill in the gaps with whatever fairy tale most appeals to you.

That’s exactly what integrative medicine does by advocating the use of modalities such as reiki, homeopathy, naturopathy, and much of traditional Chinese medicine. And that’s exactly what Dr. Katz is doing when he advocates a “more fluid concept of evidence.” Don’t believe me? Behold these words from Dr. Katz:

Nowhere in the argument for a more fluid application of evidence to the care of people is there an argument to reject evidence when it is clear. The evidence in support of immunization is clear. A connection between the two argument is specious; a non sequitur.

Note that apparently Dr. Katz was criticized that his a more “fluid” concept of evidence has been used by the antivaccine movement. Now contrast the above passage with these words from Dr. Katz:

And the anecdotal evidence that homeopathy might be effective was brought up by the naturopaths. We tried it. It worked.

If there’s a form of quackery other than homeopathy for which the evidence is more clear that it does not work better than a placebo, that, physically according to what we know from chemistry and physics, it cannot work, I am unaware of it, with the possible exception of reiki. In 2008, Dr. Katz was clearly willing to reject evidence that homeopathy is pseudoscience. What other clear science has he been willing to reject when he sees fit? How “fluid” is his concept of evidence? In the final section, we shall see.

Dr. Katz’s lovefest with naturopaths

Dr. Katz works a lot with naturopaths. We’ve known this for nearly eight years. Apparently, his definition of evidence is “fluid” enough to embrace a specialty with as tenuous a relationship with science as naturopathy whole-heartedly, to the point where he’s recently given testimony to the Public Health Committee in the Connecticut State House of Representatives in support of a bill that could expand the scope of practice of naturopaths and—or so naturopaths hope—give them prescribing privileges. In his written testimony, he brags about how he has “worked extensively with naturopathic physicians in every context relevant to the advancement of medicine” and gushes shamelessly about how he has been “richly rewarded by all of these collaborative activities.” He even refers to naturopaths as “ideally suited to meet the primary care needs of Connecticut residents.”

What is it about naturopaths Katz so admires that he’s willing to go to bat for them in front of the legislature in his own state and speak of them in such glowing terms? Is it their use of quack treatments, such as unsupported thyroid tests and treatments for fake diseases like “systemic candidiasis” and Chronic Lyme disease? Given Katz’s admirable defense of vaccines in wake of the Disneyland measles outbreak, is it their near-universal antivaccine views, even among naturopaths who think of themselves as “not antivaccine”? Is it their advocacy of major quackery, including chelation therapy, a potentially deadly pseudoscientific treatment, to treat autism? Is it their advocacy of intravenous peroxide to treat a wide variety of conditions? Is it their advocating “detoxification,” herbs, and acupuncture to treat infertility? Is it how naturopaths embrace the four humors? Is this what he means by embracing a “more fluid concept of evidence”? Apparently so. After all, the humors were believed to be fluids.

Or perhaps Dr. Katz really doesn’t know what naturopathy is actually about. If that’s the case, I’ll take the opportunity to educate him and suggest that he read Kimball Atwood’s classic critical appraisal of the vitalistic pseudoscientific mystical practice that is naturopathy, after which he might be interested in what naturopaths say amongst themselves when they think no one is listening. Then he should read Britt Hermes’ confessions of a former naturopath and her description of naturopathic education. Let’s just say, the facts about what naturopathy is and how naturopaths practice contrast starkly with the naïve view Katz presents in his testimony. A lot.

Seven years after he introduced his more “fluid” concept of evidence, Dr. Katz has failed to give a satisfactory or convincing answer to the question of just what he meant. The best he can come up with, after whining about how nasty we “self-appointed guardians of the definition of EBM” with an Internet connection and too much time on our hands are, seems to be either to substitute anecdotal evidence for RCT evidence whenever he thinks it appropriate or to seek a “middle way” between SBM and quackery. If that means embracing homeopathy, traditional Chinese medicine, and naturopathy, apparently that’s fine with him. Unfortunately, Dr. Katz is not alone. He is on the vanguard of a whole movement advocating a more “fluid” application of evidence who, with good intentions, have brought rank pseudoscience into bastions of academic medicine.

Additional reading:

  1. “Integrative” medicine at Yale: A more “fluid” concept of evidence?
  2. Changing the Rules of Evidence
  3. The false dilemma of David Katz: Abandon patients or abandon science
  4. Science, Reason, Ethics, and Modern Medicine, Part 2: the Tortured Logic of David Katz



By Orac

Orac is the nom de blog of a humble surgeon/scientist who has an ego just big enough to delude himself that someone, somewhere might actually give a rodent's posterior about his copious verbal meanderings, but just barely small enough to admit to himself that few probably will. That surgeon is otherwise known as David Gorski.

That this particular surgeon has chosen his nom de blog based on a rather cranky and arrogant computer shaped like a clear box of blinking lights that he originally encountered when he became a fan of a 35 year old British SF television show whose special effects were renowned for their BBC/Doctor Who-style low budget look, but whose stories nonetheless resulted in some of the best, most innovative science fiction ever televised, should tell you nearly all that you need to know about Orac. (That, and the length of the preceding sentence.)

DISCLAIMER:: The various written meanderings here are the opinions of Orac and Orac alone, written on his own time. They should never be construed as representing the opinions of any other person or entity, especially Orac's cancer center, department of surgery, medical school, or university. Also note that Orac is nonpartisan; he is more than willing to criticize the statements of anyone, regardless of of political leanings, if that anyone advocates pseudoscience or quackery. Finally, medical commentary is not to be construed in any way as medical advice.

To contact Orac: [email protected]

39 replies on “The return of the revenge of a “more fluid concept of evidence””

By working with colleagues trained differently than I, naturopathic physicians in particular, we had more treatment options together than we did alone.

Indeed he does! He has worthless treatment options as well as ones that work!

When I hear concepts like ‘personalisation’ and ‘every person responds differently’, apart from obvious cases like a person with an egg allergy experiencing anaphylaxis with some vaccines (as a rough example), what I ultimately decode this into are two things:

1. That some people (beyond the medical oddities, rare cases, etc etc) are so keen to believe they are special that they are willing to buy into the idea that their body works in different ways to ever other body o the planet for absolutely no reason. It is startlingly narcissistic.

2. That what is expected of doctors is so completely inappropriate and beyond the bounds of what we might expect of other professionals: we don’t want or need our lawyers or accountants to ‘work with the whole client’: we are perfectly fine to understand the concept of professional expertise and limitations here, but we seem to find it especially challenging to not expect the services of a chaplain, a psychologist, a friend and a drill-sergeant from our PCP. Once again, also narcissistic and demanding.

It is rather depressing. This Yale cat should really know better with his degrees and the textbook on epidemiology.
Oh, well.


@Bec #2

I don’t think that most of what is standard sorts of personalized medicine initiatives involve narcissism. I mean one can acknowledge that there is a diverse set of alleles that determine if one responds to a drug or not, or at what dose one should get because you metabolize it faster or slower than other people.

Plavix is one of the early examples of this coming into medical practice as if you don’t metabolize it properly there is no point taking it and one should use a different drug in the class.

Yes there are a lot of I’m a special snowflake types out there, but there are genuine biological variations based on gender and genes that shouldn’t be ignored.

KayMarie you’re very right and I thought I’d worded my response better: it effectively agrees with your comment.

My thought is that thinking ‘every body reacts differently’ to things that are patently woo (eg that reiki or homeopathy magically works for them and them only, despite the endless reams of evidence to say it works on nobody else but them) – like, because they are special, maybe green juice and coffee enemas will work for them. I articulated my idea poorly – mea culpa.

I imagine the logic works very similar to this:

Strange…here is a quote from Katz taking down gluten-quack and “too much too soon, I’m not anti-vaccine”David Perlmutter.

“You’re only being a good scientist. Iif you say, ‘I’m going to try to read the literature in as unbiased a manner as I possibly can, see where it leads me, and then offer the advice that I have based on that view from an altitude.’ I don’t see that going on here, and again, I think it’s kind of sad because I think the public is being misled.”

I guess he does not apply that reasonable attitude everywhere.

Hey Bec, no harm no foul. Just it does seem a lot of the time the alternative world picks up on the new medical buzzwords faster than the medical world and too often flings them around inappropriately.

Probably because evidence-based medicine tends to be more cautious with new things and doesn’t have to rely so hard on marketing to get patients. When most of what you have to sell is the marketing it helps to be on the cutting edge of new buzzwords.

Most of the buzzwords do have a legitimate use but often get tainted by the marketeers leading the charge.

Although it is funny here watching some of the conventional medical world leaping into more marketing as we are in a bit of a battle of the giants to see who is going to carve up what parts of the state into mega-care systems. So many billboards and TV ads from the systems gobbling up bits and pieces of the state. I do like the how long the wait time is at the ER billboard. I suspect that is trying to compete with the Urgent Care centers as a lot of them have had a time to be seen info on the websites for a few years and you can get in line on line and get a text when it is time to head to the clinic.

but we seem to find it especially challenging to not expect the services of a chaplain, a psychologist, a friend and a drill-sergeant from our PCP

That’s in part because, not so long ago, a physician was expected to be all of the above. A hundred years ago, there were many ailments for which there was no adequate treatment available, let alone a system of RCTs to prove that a treatment is adequate. Even today, psychological factors can affect the prognosis for some conditions, so a good PCP needs to at least have an instinct in that direction, and know when the patient needs help from a specialist. There is also the well-known issue of medication interaction–even the ads that urge you to consult your doctor about Medication X will say that you shouldn’t take it if you are also taking Medication Y.

I might specify that I come from a non-American context here (a country with both socialised medicine and a ban on prescriptive pharmaceutical advertising on TV, in magazines and on billboards) where we do have our PCPs/GPs address these issues. For what it’s worth, they do what they need to quite well – at least mine does. But I don’t expect her to have the ability to write a detailed dietary guide for me, or to diagnose my psychological/psychiatric issues, or to have time to counsel me through every single underlying factor which contributes to my experience of a medical condition – which is where I see the delineating point between what proponents of sCAM believe a doctor should do, and what really happens in practice.

As good and as thorough as my doctor is, I am not her only patient and as such I am fine with her knowledge extending only so far as to know when to direct me to someone with that degree of specialty. An ‘integrative practitioner’ who might only see eight patients a day while giving an ‘exact’ diagnosis for everything wrong with their patient, plus a very exacting dietary restrictions guideline, can give the perception of ‘personalisation’ while not really knowing how to actually do anything of value other than listen.

I wonder how Dr Katz would like being accused of a crime based on a “more fluid concept of evidence”?

The “everybody is different and what works for one may not work for another” mantra so commonly heard in alt med goes beyond narcissism.

First, it is key to explaining alternative therapy failures. Don’t be discouraged, go out and try something else that worked for my co-worker’s cousin’s sister-in-law. And keep on trying different stuff until something works, or at least has a temporary placebo effect. Not so good from the standpoint of prolonging suffering, delaying proper diagnosis and treatment and wasting money, but great for the quack practitioner and supplement dealers.

Secondly, claiming we’re all different is a way to attack evidence-based medicine as supposedly cramming everyone into hte same category, while alt med “individualizes” treatment. The truth obviously is that while individual variance affects some EBM treatments, it’s a gross exaggeration to suggest that proven therapies cannot be expected to work for the vast majority of people.. And when variance in response is found in EBM, it’s based on actual genetic/molecular parameters, not speculative woo.

Dr. Katz needs to pay a lot more attention to the characters he’s “integrating” with, or those shiny items on his CV are going to get increasingly tarnished.

I’ve known about his position on this for a while, and find it very disappointing, especially since when it comes to obesity and weight loss, I’ve found is information to be excellent. I’ve interviewed him a couple of times (for the Chicago Tribune) and the quotes he provided were outstanding, without a hint of quackery. The articles of his I’ve read on the subject of nutrition, obesity and weight loss are equally excellent.

But then this stuff. As great as his information is regarding weight loss etc., I can no longer consider him as a reliable source, because his affiliation with woo could be harmful to any article or book I write as a result. It’s a shame, really. I will still read his work, but can’t quote him anymore.

Dr. Katz is once again very unhappy with us over here at SBM.

Phew! I’m very relieved to hear it.

we seem to find it especially challenging to not expect the services of a chaplain, a psychologist, a friend and a drill-sergeant from our PCP.

Speaking as one of the many commenters here who actually has up close and personal experience with a drill sergeant, that’s the last thing I want from my doctor.

Speaking as one of the many commenters here who actually has up close and personal experience with a drill sergeant, that’s the last thing I want from my doctor.

Based on what others say, the apparently want their doctors to tell them how to eat, how to dress, and how to stay in shape in straight, no nonsense, possibly abusive terms, possibly standing over them and yelling to get them to do what they need to do.

Not my idea of a good time either.

Here is a vivid example of the outcome of Complementary and Alternative Medicine. Perhaps Dr. Katz might speak to this in his next glorious review of CAM!

Sadly, it is also an example of a horrific miscarriage of justice. I know Quebec jurisprudence is governed by civil law rather than common law, but I see no reason this verdict is plausible (if you aren’t entitled to administer injections, you admit doing so and the patient dies; the outcome would seem to be fairly straight forward!)

My son recently had an issue, he was septic due to kidney stones. White count hit 47,000, oh, and he has lung damage from a life time of medical interventions due to his Truncus Arteriosus.

Anyhow, he was offered naturopthy, accupuncture, and reiki upon being flown to a university ICU.

They took VERY good care of him, but the fact that they had the gall to offer such quackery infuriates me.

Oh here, this kid is septic, lets wave hands and stick pointless needles in him….

@17 Chemist.

If I am reading that link correctly that was a criminal trial not a civil one. As far as I can tell the Civil vs Common Law distinction would not apply. Any Quebecois lawyers here for a better opinion?

Agree with Bec and Dangerous Bacon – the ‘we’re all so individual’ line serves many purposes. It also probably helps to reduce cognitive dissonance within and between woo modalities because if every case has such a unique constellation of factors, then it’s ok if two practitioners prescribe two completely different treatments. Added bonus: pretty damn hard to establish consistent standards of care.

I keep meaning to ask my TCM acquaintance how ‘the ancients’ originally established which treatments to use for which conditions given that their patients were presumably also unique individuals requiring completely improvised, I mean personalised care.

kind of how I feel about battlefield acupuncture

Plus, “auricular acupuncture” was made up out of whole homoncular cloth by a Frenchman and “brought to market” by the Red Army.

Regarding the “absence of evidence is not an evidence of absence”.

Would dr Katz try, despite absence of evidence for human ability of unassisted flight, jumping off a skyscraper based on my strong belief that he’ll float like a baloon, so full of hot air?

Some of what Dr. Katz says sounds good in the abstract:

the needs of patients all too often go on when evidence runs thin, or out entirely. …
When patients had been everywhere, tried everything subtended by RCTs and still weren’t better, they came to us. And we, generally, could come up with something reasonable to try. Admittedly, it wasn’t yet in the textbooks — but that was the point. These were the folks that had already run off the pages of textbooks, and still needed help.

I’ve been in that position. I’ve had to do the best I could to “come up with something reasonable” when I had puzzling illnesses.
I’ve read research to try to pick up on things that haven’t yet made in into medical practice; tried many different things that I thought might help or that were suggested by doctors; adopted some unproven treatments that seemed reasonable, and listened to other people with similar problems. And some things did seem to help, and some of what people said did pan out, and in the end my experimentation did get results.
But Dr. Katz’s idea of “coming up with something reasonable” actually seems like a waste of time and energy. Stuff like homeopathy, massage, etc. etc.
He talks well when he’s being abstract about it, but in reality he seems to have been “wooed by woo”. It seems like a lazy way of “coming up with something reasonable”.

@ Beth

But Dr. Katz’s idea of “coming up with something reasonable” actually seems like a waste of time and energy.

It’s not just Dr Katz, but the general modus operandi of con men and true believers.
These people are good at selling that they sell. They drape themselves into the mantles of reasonableness, compassion and hope. It’s when you start looking at the unlikely proposed way of action – or at the big mansion these good Samaritans bought with the cash they earned – that you start going like “wait a minute…”
The way the debate is framed makes it all the more difficult to fight their nonsense. One comes across as a bully for daring to question someone who is just trying to help.

It’s annoying, sometimes. Despite all the children pointing out that the Emperor is naked, there are still plenty of courtiers following the Imperial @ss.

And in the meantime, legit endeavors by NGOs and other people honestly trying to do some good don’t get the time in the spotlight they really deserve.

Re. Narad @ 22:

For which reason we shouldn’t use the term ‘Traditional Chinese Medicine’ (‘TCM’) but instead call it ‘Traditional Maoist Medicine’ (‘TMM’).

Re. homeopathy:

I wonder about this: offer it to chronic pain patients an an option instead of narcotics. Provide full disclosure: ‘according to SBM this is so much rubbish,’ but let the patients choose. If some of them get placebo relief either in whole or in part, good for them: fewer narcotics.

But what’s more likely to happen is that the overwhelming majority will experience no relief whatsoever, the reputation of homeopathy will take another devastating hit, and perhaps various authorities will become more realistic about the needs of chronic pain patients.

To my mind there’s nothing unethical about this: patients would be suitably warned, there is no risk to their lives, and they can get real medicine any time they ask for it.

As for Dr. Katz, there is one thing to be said for an advocate of quackademic medicine who has no tolerance for anti-vaxxers, namely that he can be used as an example of ‘someone who supports naturopathy’ but stands on the right side of the vaccine issue. That might at least be worth some cognitive dissonance on the part of your anti-vaxxers in California, Birkenstocks and all.

I’m old-school enough to believe that the only sharp pointy objects you should be sticking into people on the battlefield are bayonets.

@ shay:

Take a look at TMR – today they feature a tale about vaccine injury in a guy who was sent overseas several times as part of his military assignment.

There’s no denying that some individuals have suffered severe reactions to the plethora of vaccinations military personnel have to undergo. However — it’s still about relative risk.

One can console oneself with the knowledge that no single person or organization is ever going to get the DOD to stop vaccinating.

Do they have any reason why they think the gov’t is trying to ensure the fighting forces are too sick and weak from the vaccines to fight?

Do they acknowledge that there is any other issues with all the other exposures one has in battle?

OT – but is AoA ever TRULY off-topic at RI? – there’s a new Gamondes rant over at the wretched hive. It opens with a Foucault quote. I feel like a kid on Christmas morning.

@ JP:

Well, they say that imitation is the sincerest form..

but I *might*have said…

OT but is rambling, free associative prose ostentatiously decorated by bad photo shopped images created by post moderne wannabe investigative journalists
EVER truly OT @ RI, I ask you?

Gamondes comments on her own article saying that (paraphrase) our era reminds her of the film, Close Encounters, wherein random people get messages from the aliens which she compares to anti-vax parents “getting the message” about autism and vaccine.

Actually, her writing reminds me of quite a few other films which I hesitate to name.

herr doktor bimler@26

Company president and naturopathic physician Mort Fishman, who leads the team and is certified in Acupuncture Cardiovascular Life Support (ACLS), says that in situations like this there is a golden hour where appropriate intervention can make the difference between life and irreversible chakra degradation.

Clay Jones has also been writing satire like that for a while.

That was great.

@Helianthus #27

It’s not just Dr Katz, but the general modus operandi of con men and true believers.
These people are good at selling that they sell. They drape themselves into the mantles of reasonableness, compassion and hope. It’s when you start looking at the unlikely proposed way of action – or at the big mansion these good Samaritans bought with the cash they earned – that you start going like “wait a minute…

Yes, what Dr. Katz does is to provide a frame that tells you what to think about his alt-med approaches.
But what he talks about in his frame is a real problem in mainstream medicine. Mainstream doctors do often quit looking for an effective treatment when the standard answers run out. Perhaps because they feel safer from malpractice suits if they do things in a standard way – going outside the box is risky for doctors.

Comments are closed.


Subscribe now to keep reading and get access to the full archive.

Continue reading