Here we go again.
Oh, well. These things come in waves, and sometimes I have theme weeks. Right now, this week appears to be developing into a week of quackademic medicine involving dubious acupuncture studies. Yesterday, it was acupuncture for lymphedema after breast cancer surgery, a study coming right about what is rapidly becoming the Barad-dûr of cancer quackademia, Memorial Sloan-Kettering Cancer Center. How a hospital that is so awesome in every other way can have such a blind spot bigger than the Eye of Sauron, I don’t know, but it does, and the result is a steady stream of embarrassing forays into quackademic medicine like yesterday’s.
Of course, there is one institution that far surpasses even MSKCC in the power of its quackademic woo, and that is the University of Maryland, home to Brian Berman, king of acupuncture quackademia, and my Google Alerts did there job and, well, alerted me to a new meta-analysis published online late last week in the Journal of Human Reproduction Update. Berman is the corresponding author (of course!), and a research associate by the name of Eric Manheimer is the lead author, and together with other colleagues, they have produced yet another fine analysis of tooth fairy medicine entitled, The effects of acupuncture on rates of clinical pregnancy among women undergoing in vitro fertilization: a systematic review and meta-analysis.
Now, remember all the things I said about the utter lack of prior plausibility for acupuncture for treating lymphedema? You remember, how there’s no plausible biological mechanism that would lead one even to suspect that sticking needles into parts of the body completely unrelated to the physiological mechanisms that result in lymphedema after mehanical interruption of regional lymphatics by surgery? All of that goes double—nay, triple!—for using acupuncture for infertility and improving the pregnancy rates after in vitro fertilization (IVF). I mean, seriously. Think about it. How on earth would sticking needles into the skin improve the odds of conception after this procedure? It wouldn’t, and it doesn’t. That doesn’t stop acupuncturists and acupuncture apologists from heavily selling acupuncture as somehow managing to do just that, against all physiology and reality.
So here’s how the systematic review is being sold:
Acupuncture, when used as a complementary or adjuvant therapy for in vitro fertilization (IVF), may be beneficial depending on the baseline pregnancy rates of a fertility clinic, according to research from the University of Maryland School of Medicine. The analysis from the University of Maryland Center for Integrative Medicine is published in the June 27 online edition of the journal Human Reproduction Update.
“Our systematic review of current acupuncture/IVF research found that for IVF clinics with baseline pregnancy rates higher than average (32 percent or greater) adding acupuncture had no benefit,” says Eric Manheimer, lead author and research associate at the University of Maryland Center for Integrative Medicine. “However, at IVF clinics with baseline pregnancy rates lower than average (less than 32 percent) adding acupuncture seemed to increase IVF pregnancy success rates. We saw a direct association between the baseline pregnancy success rate and the effects of adding acupuncture: the lower the baseline pregnancy rate at the clinic, the more adjuvant acupuncture seemed to increase the pregnancy rate.”
It’s hard not to be a bit snarky here and say that if your clinic is doing well with its pregnancy rate, then obviously you don’t need mumbo-jumbo. However, if you’re not doing so well, maybe some bread and circuses will help.
So let’s look at the study itself or, as I like to say, go to the tape (or journal, or whatever). Basically, Berman and company examined sixteen trials with a total of 4,201 participants that compared needle acupuncture administered within one day of embryo transfer to shame acupuncture or no treatment. They left out studies that examined electroacupuncture (a famous bait-and-switch form of acupuncture frequently mixed in with regular acupuncture). Well, actually, not exactly. Teir rationale for not using electroacupuncture studies was not what you would think, namely because they didn’t want to mix acupuncture studies with studies involving electricity, which didn’t exist at the time when acupuncture was allegedly invented. Rather, the rational was that these studies involved studying electroacupuncture as an alternative to conventional anesthesia during oocyte retrieval and the points are therefore not chosen to improve fertility bur rather to reduce pain. Electroacupuncture was okay, as long as its intent was pregnancy. I kid you not. Or, as the authors say:
We included only trials in which acupuncture involved the insertion of needles into traditional meridian points. The needles could be inserted into tender points in addition to the traditional meridian points, and the needles could also be electrically stimulated.
In any case, the methods used were fairly standard meta-analysis methodology, and when they were through they had 16 randomized trials. Now here’s the annoying thing. This is a negative study. Oh, the authors, as you can see from the press release, jump, jive, and wail to try to extract something positive out of it, but the bottom line is this. When they looked at the pooled studies, there was no statistically significant difference in pregnancy rates between the acupuncture and control groups. None. Nada. Zero. Zip. And any other word you can think of for “no” or “zero.” Or, as the authors put it, there was “no statistically significant difference between acupuncture and controls when combining all trials [risk ratio (RR) 1.12, 95% confidence interval (CI), 0.96–1.31; I2 = 68%; 16 trials; 4021 participants], or when restricting to sham-controlled (RR 1.02, 0.83–1.26; I2 = 66%; 7 trials; 2044 participants) or no adjuvant treatment-controlled trials (RR 1.22, 0.97–1.52; I2 = 67%; 9 trials; 1977 participants).”
In other words, that’s a negative trial. Acupuncture does not improve pregnancy rates. Did I say that enough times? Heck, consistent with what one would expect for an intervention that doesn’t have an effect, the asymmetric funnel plot showed a tendency for the intervention effects to be more beneficial in smaller trials. We see the same thing in virtually all clinical trials, but especially in clinical trials of CAM modalities. Homeopathy, in particular, is notorious for demonstrating this effect. So is acupuncture.
Of course, in so-called “complementary and alternative medicine” (CAM) or “integrative medicine” trials, there’s never such a thing as a negative study. So we have subgroup analysis, the better to seek statistically significant results in smaller pools of patients, where there might be more variability. All too often it’s not clear whether that subgroup analysis is prespecified or cooked up post hoc. To Berman’s credit, in this case, the subgroups were specified before the meta-analysis:
We conducted subgroup analyses on five clinical characteristics that might influence the effect of adjuvant acupuncture on clinical pregnancy success rates: (i) two acupuncture sessions or more than two; (ii) selection of meridian acupuncture points the same as the points selected in the first published trial (Paulus et al., 2002) that evaluated acupuncture as an adjuvant to embryo transfer, and which showed a large effect, or a modified version of this trial’s acupuncture point selection protocol; (iii) control group clinical pregnancy rate (as an estimate of the baseline clinical pregnancy rate) dichotomized as higher [32% or greater, which is the European average of pregnancy rate per embryo transfer (de Mouzon et al., 2012)] or lower; the control group clinical pregnancy rate was also analysed as a continuous variable to test whether the relation was linear and consistent with the findings of the categorical analysis; (iv) explanatory trials conducted to test the effects of adjuvant acupuncture under controlled conditions in which the acupuncture was administered onsite at the IVF clinic or pragmatic trials conducted to test the effects of adjuvant acupuncture delivered off-site, which might better approximate every day, ‘real life’ conditions since most IVF clinics do not have onsite acupuncturists (Arce et al., 2005); and (v) trials that involved a treating acupuncturist who was judged as adequately experienced or not adequately experienced, with such judgments made by acupuncturist assessors who were blinded to the identities and results of the trials.
One wonders what objective criteria these acupuncturist assessors use to judge other acupuncturists as “adequately experienced.” They also prespecified six “risk of bias” domains to be examined: random allocation sequence generation; concealment of allocation of randomization sequence; blinding of patients (i.e. use of sham control); blinding of embryo transfer physicians; incomplete outcome data; and unequal co-intervention. And guess what? These were nearly all negative, too. There was only one exception, which of course was touted in the press release above. Oh, the authors dance around several variables that they describe as “almost” statistically significant, but “almost” only counts in horseshoes and hand grenades (and nuclear weapons). To be honest, these subgroups weren’t even that close to being statistically significant. If I were reviewing this paper, I would have told the authors to cut out at least a couple paragraphs worth of verbiage dancing around these topics, although it is somewhat interesting to note that one of these “almost significant” subgroups was whether or not the physician doing the IVF embryo transfer was blinded to the acupuncture status of the subject.
In any case, there was an inverse correlation between the baseline pregnancy rate observed in a study and the effect of acupuncture. Studies reporting greater than average pregnancy rates (32% or greater) showed no effect of acupuncture. In fact, the risk ratio for such studies was 0.90 [95% confidence interval 0.80 to 1.01]. That’s almost, but not quite, a statistically significant negative effect on pregnancy rates! In contrast, studies for which the baseline pregnancy rate was less than 32%, produced a risk ratio of 1.53 [95% confidence interval 1.28 to 1.84]. Why is this? Who knows? The authors speculate that additional interventions have little or no value when pregnancy rates are already high, but acupuncture can help when pregnancy rates are low, but there are so many factors that determine pregnancy rates, including embryo selection, prevailing practice, number of embryos transfered, and many others. There could very well be a confounder there that the authors didn’t pick up.
The authors seem to recognize this, if grudgingly and/or subconciously, in their conclusion:
For future trials, if the objective is to investigate the mechanism of IVF adjuvant acupuncture, and specifically whether any effect is mediated through the point-specific needle placement, or alternatively through a non-needling-related psychobiological placebo mechanism, both a sham control arm and a no adjuvant treatment arm would be necessary. A sham control arm might also be necessary for recruitment, if potential trial participants would be unwilling to be randomized to a no adjuvant treatment control (Smith et al., 2012). However, because it is difficult to assess whether or not a sham control intervention has acupuncture-specific effects that may increase IVF success rates, sham-controlled trials in this area can also potentially complicate the interpretation of the overall evidence (Manheimer, 2011). For addressing the more clinically relevant question of the total effects of acupuncture (i.e. specific needling effects plus any non-needle-related placebo effects) in contributing to any increases in IVF pregnancy success rates, a sham control seems unnecessary (Manheimer, 2011). This question about the need for sham controls may also apply to other invasive, difficult to blind adjuvant procedures evaluated in IVF RCTs (Holt et al., 2009).
This is of course, veyr silly. There is no need for further stuies. Acupuncture does not work. It does not improve the pregnancy rate. There is no physiological mechanism to think that it should, and this meta-analysis confirms it. The questions of whether a sham control intervention is needed or not, whether multiple treatments are needed or not, whether sticking to the meridians matters or not are all the equivalent of what Harriet Hall likes to call Tooth Fairy science.
32 replies on “More Tooth Fairy Science: Acupuncture does not improve in vitro fertilization success rates, no matter what acupuncturists say”
Do stress levels have an effect on success of IVF? Or is that one of those old wives tales?
I seem to remember reading that high levels of a stress hormone (Cortisol?) are detrimental to successful pregnancies, as in higher chance of miscarriage or low birth weight, as well as lowering the chances of getting pregnant in the first place.
If that is actually the case and not a figment of my imagination I could possibly see how acupuncture might seem to help. If I stand on my head and squint.
There are far more pleasant and less dangerous ways of reducing stress levels.
There is, however, a plausible mechanism for acupuncture needling of a condom increasing <i?in vivo fertilization rates.
argh–apologies for the typo:
There is, however, a plausible mechanism for acupuncture needling of a condom increasing in vivo fertilization rates.
I never cease to be amazed by the “one size fits all” properties of quackery. Lymphadema? Stick needles into the guy. Fertility? Needles. Pain? Needles. Dampness? Needles.
Yet I am sorely disappointed by no mention of de qi sensation. How can we asses the results of this acupuncture study if we don’t know wheather or not reaserchers have taken de qi sensation into account?
Wait…are they using the needles on the male half of the couples, or the female?
Inquiring minds want to know.
Can I ask a question in all seriousness? Maybe there is someone on this site who has worked in scientific journal publishing that can answer. What is the function of a journal referee, and how much notice is taken of their judgement/recommendations? I’m presuming that the Human Reproduction article was refereed and that the faults pointed out here were noticed by one or both referees. Would an editor be likely to publish anyway, even if the referee objected? Can they be overruled?
@kathy: I speak as a practicing physicist. Yes, journal referees are supposed to look for nonsense like this. However, most journals allow authors to suggest potential reviewers, and in many cases the editor will choose one (or sometimes more) reviewer from that list. The bigger the population of scientists likely to publish in a journal, the more likely this is to happen. The editor chooses the reviewers, so if he happens to be a woo-believer himself, he can choose woo-friendly reviewers. And yes, the editor has the power to overrule the referees, or to send to a third reviewer if the first two are divided in their opinion. So there are plenty of chances to game the system.
The only kind of “puncture” I can think of that might improve fertility in IVF is intracytoplasmic sperm injection, which, I guess, is a lot more “accu” than most punctures.
@ Eric Lund :
What do you think of the journal in which the article appeared?
Regarding the worst-performing clinics having the biggest response, how much of that is just mean reversion?
It seems to me that the results of this analysis are almost the definition of regression to the mean – if the success rate of a clinic is below the mean it is surely likely to increase, whatever you do.
Any mention of needles and fertility reminds me of Professor G.S. Brindley’s unusual yet memorable way of reporting his success in using papaverine injections to treat erectile dysfunction, at the 1983 Urodynamics Society meeting in Las Vegas.
Finally, I am grateful to Orac for allowing me to add “shame acupuncture” to my collection of solipsistic typos.
@Denice: I’m a physicist, not a physician. I’m not qualified to have an opinion of the journal in question.
Okay, so “we limited ourselves only to traditional acupuncture, except when we didn’t.”
Who seriously writes this with a straight face? I mean, even ignoring the underwhelming data, this is so sloppy it is meaningless.
If somebody didn’t mention “shame acupuncture” I sure was. That is so apropos.
Ack. I should have thought of that, particularly given that the result for the higher performing clinics was an almost statistically significant decrease in pregnancy rate, with a relative risk of 0.90…
I agree with Krebiozen. This is classic regression to the mean.
The under-performing sites did better; the over-performing sites did worse.
Well, I for one am convinced. If ever I want to get pregnant, acupuncture will do the trick.
Hey, what? You mean it doesn’t work like that? Man, are you saying I don’t have the qi? I don’t have the qi?!
I am too scared to look up “acupuncture contraception”.
I’m a physicist, not a physician.
This needs to be said in a Dr McCoy voice.
In addition to regression to the mean, what about the Hawthorne effect? We have some poorly-performing clinics, possibly due to some sloppiness in procedures. You run a trial there, any trial, and suddenly more eyes are on them and they run things a little more tightly. With a clinic that is already doing well and has good procedures in place, that’s less likely to make a difference…?
@herr doktor bimler – I think i was involved in a case of acupuncture contraception. I’ve got the scars to this day and learned a new respect for women…
Ignorance and arrogance.
Speaking as someone who writes about Dark Ages crafts where there’s not a lot of evidence, this bit jumped out at me: the more adjuvant acupuncture seemed to increase the pregnancy rate.”
When you’re pretty sure of something but you can’t prove it, phrases like “seems likely” or “it’s probable” is how you present your hypothesis.
Of course, I’m usually saying something like “Well we have no extant Anglo-Saxon tunics but we do have pictures. Here’s this tunic from Denmark that’s the same age as the picture, and the tunic looks like the picture, so it’s probable the Anglo-Saxons were making their tunics the same.” With pictures so people can follow my logic. These clowns are, to use another re-enactor phrase, “talking out of their butts.”
So if they disagree with you, they automatically become quacks, even if it’s a respected institution? All hail Orac, infallible God of Medicine. What splendid hubris and arrogance!
Sandrop – So you’re dismissing Orac’s post while being unable to make an argument against it and show him he’s mistaken? All hail Sandrop! Whlat splendid hubris and arrogance?
The only ‘splendid hubris and arrogance’ is yours. It isn’t a matter of disagreement, it’s a matter of the evidence not supporting a hypothesis and sloppy science to try and make it so. Who gives a toss where and what reputation the institution has. Since when is that insulation against stupidity? That’s the beauty and integrity of SBM dumbass; it doesn’t matter who or where a study emanates from. What matters is a supportable conclusion.
How come nobody mentioned the story about IVF causing autism?
A question for you guys (about acupuncture but not with respect to effect on IVF):
Was in a heated argument with a colleague regarding acupuncture when he triumphantly shoved this article in my face. (http://www.ncbi.nlm.nih.gov/pubmed/21344469) It’s a double blind RCT comparing acupuncture at actual acupoints vs. random skin punctures or placebo for the relief of dental pain control showing a significant improvement in analgesia requirement. This is unlike most other acupuncture studies which compare placebo vs. acupoints. My statistics is really weak and I’m afraid I did not have a rejoinder then. I was wondering if anyone could help me with the data?
anonymous of asia (#28):
The RI search function indicates that Orac has regularly dealt with the study’s third author (one Brian Berman), but not with this particular study.
Berman’s previous results for acupuncture include the argument that “sham acupuncture” — inserting needles but not in the approved acupoints — is actually *real acupuncture* (i.e. that the needling is the crucial part and not the location per se), thus explaining the lack of difference between them. In this paper there *was* a difference between ‘approved’ and ‘sham’ acupuncture, with the conclusion that acupoints *are* crucial. In fairness, Berman was not a primary author so there is no need for consistency with his earlier publications.
But the procedure was not the same in the two treatments; for ‘real’ acupuncture the needles were manipulated to be sure that the patients experienced “soreness, heaviness, numbness, or distention”… with no manipulation in the sham treatment. Then there was a third control treatment where “a plastic needle tube was taped on the bony area”.
I don’t feel qualified to criticise the stats. Suffice to note that the authors report small, borderline-significant differences in the time taken until patients needed real pain relief; differences entirely consistent with three different strengths of placebo effect.
Sandrop: ALL respected institutions have their share of quacks, dropkicks and embarrassments on the academic staff. In this, academia is no different to any other industry/workplace.
As someone who suffers from infertility quacks preying on desperate couples really chaps my hide. I have to admit to the hubby and I being scammed into a couple of sessions of accupuncture before going to a reputable fertility clinic. The accupuncture was not unpleasant, but was spectacularly uneffective and the cost was prohibitive, particularly since we were ‘recommended’ to receive 3-5 treatments per week (at $150 per treatment) for the forseeable future but at least 6-8 weeks for ‘optimum’ results. Given the desperation one feels in the search to have a child, it is galling how much money and time is bilked out of the desperate. Particularly for these couples who have already tried (and failed) other avenues for pregnancy and are now up to the expensive, painful, and depressing IVF. We were quoted here in the US $10,000-25,000 for a single round of IVF (if necessary) with approximately a 33% chance of a live birth. As an epidemiologist I did not like those numbers. Thankfully after a year in treatment we conceived a healthy son without having to resort to IVF.
Eye of Sauron? More like eye of Saturn!