SkepticblogSkepticblog logo banner

top navigation:

Revenge of the Woo

by Steven Novella, Mar 11 2013

Sometimes the targets of our skeptical analysis notice, and they usually are not pleased with the attention.

Last year the Acupuncture Trialists Collaboration published a meta-analysis of acupuncture trials in which they claim, “The results favoured acupuncture.” The report was widely criticized among those of use who pay attention to such things. In my analysis I focused on the conclusions that the authors drew, rather than their methods, while others also had concerns about the methods used.

The authors did not appreciate the criticism and went as far as to publish a response, in which they grossly mischaracterize their critics and manage to completely avoid the substance of our criticism.

To review, the original meta-analysis concluded:

Acupuncture is effective for the treatment of chronic pain and is therefore a reasonable referral option. Significant differences between true and sham acupuncture indicate that acupuncture is more than a placebo. However, these differences are relatively modest, suggesting that factors in addition to the specific effects of needling are important contributors to the therapeutic effects of acupuncture.

In my critique I pointed out that the results do not show that acupuncture is effective, nor that it is a reasonable referral option. What they characterize as “modest” differences were, rather, not clinically significant. Further, such tiny differences are most parsimoniously explained as the result of researcher and publication bias, two phenomena that are well established in general and specifically within the acupuncture literature. Unblinding alone would be sufficient to explain these results.

What they call “factors in addition to the specific effects of needling” the rest of the scientific community would call “placebo effects,” which are not an indication that a treatment works, but rather the result of bias, noise, and statistical illusions. These results are due to unblinded comparisons with untreated groups in clinical trials – they are not evidence of any kind of efficacy.

Their conclusions are part of a pattern visible within the acupuncture community – attempting to parlay placebo effects into the mirage of a real effect from acupuncture. I commented in my original article that such a conclusion was evidence of pro-acupuncture bias in the authors.

In their response, the authors write:

Although there was little argument about the findings in the scientific press, a controversy played out in blog posts and the lay press.

Only one substantive critique of the paper has appeared in a scientific forum.

We find that there is little argument in the scientific press because most scientists pay little attention to what they consider fringe practices. That is precisely why it is left to those of us who do care and pay attention to fringe medicine to provide a detailed analysis and point out the flaws in reasoning used by proponents.

In fact we did submit a letter in critique of the study, in a traditional scientific forum, but it was not published. Only the brief letter by David Colquhoun was.

This represents a typical strategy by proponents of dubious fringe medicine – interpret lack of resistance by mainstream scientists as acceptance. Whey they do encounter resistance, they try to minimize it as irrational – as Vickers et. al. have done here.

They continue:

This controversy was characterised by ad hominem remarks, anonymous criticism, phony expertise and the use of opinion to contradict data, predominantly by self-proclaimed sceptics.

This is a remarkable exercise in cherry picking and distortion. Their example of “ad hominem” remarks was my article in Science-Based Medicine (linked above) in which I said their conclusions were not justified and were therefore evidence of pro-acupuncture bias. This was followed by a substantive critique of their analysis, demonstrating the bias.

The majority of the criticism was not anonymous. All the usual players (myself, David Colquhoun, Edzard Ernst, Mark Crislip, Andy Lewis) posted articles or comments under our names. There are a few medical bloggers (like Orac) who prefer to remain anonymous (although they also blog under their real name) so as to preserve their rhetorical freedom and minimize professional harassment. To characterize the criticism as “anonymous criticism” is extremely unfair.

Under “phony expertise” they explain:

Many blog posters threw around methodological concepts such as I2 or funnel plots, or made claims about the nature of chronic pain or acupuncture placebo techniques. At the same time, many admitted to not having read the paper,4 and none appear to have published scientific research on pain, acupuncture or meta-analysis.

The reference is to Orac’s criticism. I reread it, and nowhere do I see a statement that Orac did not read the study. In fact he read not only the study but analyzed the studies in the meta-analysis themselves. Perhaps they meant to reference another article.

They also deride the concept of “Science-Based Medicine” as if that is a strange concept. What they fail to realize is that our collective expertise is in the distinction between science and pseudoscience, and the various mechanisms of self-deception. Most of us are also physicians, and we share our respective specialty expertise when collectively analyzing such studies. The original article, and this response, are excellent evidence of why such expertise is desperately needed in medicine, especially when dealing with unusual claims, such as acupuncture.

In response to my article they wrote:

One blogger asserted that acupuncture ‘has an effect size that is very small and, in my opinion, overlaps with no effect at all’.3 It is simply bizarre to dismiss years of careful statistical analysis on the grounds that results ‘might’ change; similarly, it should go without saying that whether an effect size overlaps with no effect is not a matter of opinion but of CIs.

Wrong and wrong. I did not simply substitute my opinion. My criticism was also not based on confidence intervals. We are not talking about statistical analysis, but systematic bias. I specifically cited the paper on “researcher degrees of freedom” to document this point. I further cited the authors in admitting that unblinding is a source of bias.

The point is that you can generate a small statistically significant result even when a treatment has zero effect. The authors falsely and naively assume that statistical significance equals a real effect, and they retreat to this position as if that counters the meat of our criticism. But it is simply not true. This is the point that those who are not aware of the principles of science-based medicine often miss. This is precisely why we advocate using a Bayesian analysis rather than p-values to assess clinical data.

Researcher and publication bias tend to produce a small (but statistically significant) positive result in clinical trials, and a meta-analysis will show that. We reject the results because:

- The effect size is not clinically significant

- There is a-priori and empirical evidence of bias in the acupuncture research

- Acupuncture is inherently implausible

- There is a clear pattern in the research that the best controlled and designed trials have no effect (no difference between true, sham, and placebo acupuncture.

There are two ways to interpret the acupuncture literature. One is that there is a real effect but it is too small to be clinically relevant. The second (the one I advocate) is that the small effects that tend to emerge from the research are likely not real and due to well-established sources of bias in clinical research. This is the most parsimonious interpretation. It is partly justified by the fact that the effect sizes and patterns in the research are similar to other phenomena, such as homeopathy and ESP, that are almost certainly not true.

We also criticized this statement from the authors:

With respect to the debate about clinical implications, the Collaboration argued that, while a treatment should ideally be shown to be superior to placebo, evaluation of clinical significance should be based on overall benefit, including any non-specific effects.

Yes – this should be debated. We maintain that clinical significance should absolutely not include “non-specific effects,” because such effects do not support a specific benefit from the procedure in question and are largely the product of illusion and bias. Further, useful non-specific effects, such as a therapeutic relationship between doctor and patient, can be had with legitimate treatments that are not based upon dubious principles.

Conclusion

Criticism of the Vickers et. al. article have been substantive and perfectly legitimate. Some were indeed intended for a lay audience, and meant to counter gullible treatments in the lay press. The authors, however, are unfair to dismiss all of this as “political muckraking” as they do in their response.

They did try to address some of the substantive criticism, but failed to do so, in my opinion. They entirely missed the main point of the effect of systematic bias in clinical research. They do so, it appears, because they lack expertise in pseudoscience – the very expertise they derided.

The pattern is very clear. Acupuncture is an implausible treatment with a pattern of clinical evidence that mirrors other highly implausible treatments. Researcher degrees of freedom alone is enough to explain the small residue of positive results, and it should not be ignored that the best designed studies tend to be entirely negative.

The unblinded comparisons to a no-treatment group do not justify acupuncture. Imagine if a pharmaceutical company tried to get away with such an argument. What Vickers and his coauthors have demonstrated is the dire need for science-based medicine.

 

GD Star Rating
a WordPress rating system
Revenge of the Woo, 5.0 out of 5 based on 17 ratings

Recommended Reading

17 Responses to “Revenge of the Woo”

  1. RCAF says:

    From personal experience, I believe that there may be another issue. I believe that due to the saliency of the needle, there is actually different degrees of the placebo effect.

    In my case, I had pain in my right forearms from lifting too much weight in the gym too quickly. This pain lasted for 6 months and continued even after two treatments with NSAIDs. At this point, my doctor sent me to a PT for treatment, and she decided to use a technique called dry point needling; read acupuncture for PTs. I was, of course, very skeptical about its efficacy, but I thought it would be interesting to try.

    After the treatment, I had the feeling of “pins and needles”, and warmth in my forearms for several hours. I was quite amazed that the pain in my left forearm disappeared within a week, and the pain in my right forearm is almost gone – note it is about 7 weeks after treatment.

    I don’t believe that this is anything other than a placebo, but this is quite a substantial one. Something you probably couldn’t get without sticking the needle in someone’s arm because you wouldn’t experience the same intensity without the penetration of the needle.

    From my experience, I think that it is quite possible that the studies are just showing the statistical difference between two placebo effects. It must be kept in mind that placebo effects are real, and I don’t know any reason not to think that some placebos are more powerful than others.

    • Luara says:

      How in this case is saying “this works by placebo” different from saying it doesn’t?

      • RCAF says:

        If you re-read my post, you’ll see I say clearly it is a placebo. I’m just saying the statistical difference could be based on two placebo effects that are in themselves different.

    • Mr Pogle says:

      It could also be a real effect that is a result of inserting needles into you, regardless of where they are inserted. Acupuncture claims that inserting needles in specific points is better than at random points (which it isn’t) but you may be benefiting from an effect of inserting needles versus not inserting needles. Either way, there will also be an associated placebo effect.

  2. It is well established that the more invasion a procedure the greater the measured placebo effect.

    Also, the more expensive or the more difficult to obtain, or the perception of rareness or novelty.

    Also, pill color affects placebo response.

    The term “placebo effects are real” is problematic. The research also shows it is mostly transient and illusory, not a real biological effect.

    • Luara says:

      Last I looked, it seemed people could get rid of warts by placebo effect. So that might be a case of a real biological effect via one’s state of mind.

    • Luara says:

      Also, when you say, the more invasive the procedure the greater the measured placebo effect, do you mean the more invasive someone is told the procedure is, or the more invasive the procedure actually is?
      Someone could be told they’re being poked with a needle when their skin hsan’t been penetrated, etc.

    • RCAF says:

      That is what I was thinking, there are differences in the perception based on the “treatment”

      However, I don’t think there is an issue in saying it is “real” as there is a perception that something has changed, which if it isn’t real, then what would it be? It’s like calling something “unnatural”, just what does that mean?

      Of course, I don’t suggest that anything has changed at a biological level, but when you’re trying to treat something that is subjective, i.e. a mater of perception, and not something purely at the biological level, then a positive outcome would be real.

      Psychotherapy, obviously, has not effect on a person on a biological level, but is effective for a wide range of mental health issues. For example, for clinical depression, psychotherapy isn’t effective as there is a biological underpinning. OTOH, pharmacotherapy for alcoholism is just about useless, but counselling and cognitive behavioural therapy can produce good results, in many patients.

      • RCAF says:

        Just to clarify, I’m not in any way defending the lying woo clan. I’m just suggesting that they could get a statistical difference based on a placebo vs a placebo.

      • Luara says:

        I didn’t think you were defending the “woo clan”.
        But, to put my question more exactly:
        In this case, what is the difference in what would be observed, between “it’s a placebo effect” and “it’s not placebo”???
        What experiment do you have in mind, that would differentiate between those two possibilities?
        In other words, I don’t know what exactly it means to say something is placebo, in this case.

      • RCAF says:

        Ok, I think you would need three groups. In my case, you’d require a group getting a known placebo, such as a sugar pill. One getting the dry needle, and the third getting a known medical intervention, say a cortisone shot.

        Then you could do an analysis between the three groups. If there were a significant difference between the pill and the dry needle, and a significant difference between the real treatment, and the dry needle, then you could say the difference was that the there was a stronger placebo effect for the dry needle.

        I hope that makes sense.

      • Luara says:

        RCAF,
        You said:
        I don’t believe that this is anything other than a placebo
        What do you mean by saying that the dry needle is a placebo?
        What experiment would tell you that the dry needle is a placebo, vs is not a placebo???

      • RCAF says:

        Laura, I thought that my answer would have cleared that up. The dry needle, is a placebo because it isn’t a medical intervention. Any relief I felt, or anyone feels is not do to a known medical treatment. The experiment that I proposed with a cortisone shot, should show if it is a placebo.

      • Luara says:

        RCAF,
        I added a reply to you, by accident to the end of the comment section.

  3. Archie Clebberdale says:

    When I read up on the saga, it seemed to me more like Vickers e.a. simply know they’re full of shit.
    Why concoct a complicated story about them being so wrong in such a dumb way that it’s inexplicable they’ve managed to live to the present, when a much simpler explanation beckons: they’re on damage control. Create enough fake controversy and it’s impossible for the lay public (intelligent or not) to discern where the truth lies and in the face of uncertainty people tend to stick to their old truths. Mission accomplished.

  4. Luara says:

    The dry needle, is a placebo because it isn’t a medical intervention.
    That seems circular – translated, it seems to mean “the dry needle hasn’t been shown to be better than placebo”.
    But in this case, if the dry needle weren’t a placebo, what would the placebo be, that it works better than? If the dry needle does have an effect, then it can be used in medical treatment, and placebos can have a real physiological effect. So saying that something is a medical treatment is
    not the same as saying it isn’t a placebo.
    Calling dry needling a “placebo” might imply that it can’t be doing anything except “in your head”, but that’s far from clear. It could be having an effect locally on your nerves, or in your spinal cord.
    If you give someone a sugar pill for a cancer, it’s pretty clear that if it has a physiological effect, it must be caused by something happening in their brain (unless they had an allergic reaction to the sugar pill). Thus the sugar pill would be considered a placebo.
    But in this case things are much less clear.
    Could your pain nerves have acquired a “habit” of pain from long term pain perception, and breaking this “habit” with the counterstimulation, ended the “habit”? Perhaps long-term pain tends to be self-perpetuating, in other words?
    Counterstimulation has been used since ancient times as a treatment for pain, and it works, if only by diverting one’s attention from the pain.
    For people who have amputated limbs, stimulating the opposite limb with TENS might even work to reduce phantom limb pain! See http://www.ncbi.nlm.nih.gov/pubmed/19788713
    These things are very complicated.
    It’s also possible that your pain was about ready to go away anyway, and you attributed the effect to the dry needling just because us humans make stories, attribute causation.
    Dry needling is used by physical therapists in many countries, so it is a medical treatment, see http://en.wikipedia.org/wiki/Dry_needling