SkepticblogSkepticblog logo banner

top navigation:

I Was a Skeptic Too Until I Tried…

by Brian Dunning, Mar 11 2010

We’ve all heard the infamous anecdote used in a multitude of advertisements for various flavors of woo, “I was a skeptic too until I tried (insert your favorite alternative woo product here).” It’s an effective and compelling sales pitch to laypeople, so much so that it’s become so old and hoary that it usually provokes a laugh from experienced skeptics.

Of course we answer “The last thing a skeptic would have done is rush right out and give their money to the (insert the salesman of your favorite alternative woo product here).” And here is where the heads butt. Believers often feel that a truly skeptical and scientific process would have tested the product by trying it firsthand. And, to the average layman, inexperienced in science, that sounds extremely reasonable.

This is going to be a very short blog post, because this question is very simple. No, trying a product personally is not a useful way to assess its value. A scientist understands that, a layperson does not; it sounds thoroughly counterintuitive.

The scientist knows that anecdotes and testimonials, while they can be interesting, are not useful evidence, not even the scientist’s own. Anecdotes are usually of unknown origin; and even when they’re not, they are uncontrolled, unscientific, and subject to a myriad of biases, influences, random variances, and external factors. Well designed scientific tests control for, and eliminate, as many of those influences as possible — and often, all of them.

The skeptical process does not ever require personally sampling the woo. Why would it? You’re certainly welcome to try the product if you want, but doing so is completely outside of a meaningful scientific assessment.

Does this mean that every time we hear a woo claim we must go out and perform a thorough scientific test? Of course not. Well performed tests take time and cost money, and it’s the party selling the product who bears that obligation. In science, good evidence is required to move from the null hypothesis (the null hypothesis states that the claims about the product are unsupported until they are proven true). So when I’m told about a woo product, I’m probably willing to ask to see good scientific evidence, but I’m not obligated to provide such evidence myself. I’m sure as hell not obligated to buy the damned product.

So the next time you hear “I was a skeptic until…”, you know that (a) that person is inexperienced in science, and (b) their endorsement of the product was born from an invalid process.

Your thoughts and comments are welcome.

71 Responses to “I Was a Skeptic Too Until I Tried…”

  1. Jeshua says:

    I was a skeptic about this site until i tried it for myself and now i’m addicted!

    Not your most interesting essay, but true nonetheless.

  2. johnc says:

    “You’re certainly welcome to try the product if you want, but doing so is completely outside of a meaningful scientific assessment.”

    You seem blissfully unaware of the established human need to try things because they might be fun. You have my pity.

    • Peter says:

      Your comment is an utter non sequitur to the point of being actually painful to read. You have my smug pity.

    • Kris says:

      Yes, and while penis enlargement pills “might be fun”, it doesn’t change the fact that they don’t work.

      • Brian the Coyote says:

        Not that any of we gentlemen here would be in need of any such product, real or imagined….

    • Helfrick says:

      @johnc
      I have no idea who urinated in your cornflakes the last couple of days, but it seems to be preventing you from comprehending what is being written. Take a break from the trolling and try contributing.

    • CW says:

      That’s about as unrelated as two statements can possibly get. I hope that you’re just being sarcastic, johnc.

  3. karatex says:

    JohnC, establishing the scientific validity of a claim has nothing to do with trying things “because they might be fun.”

  4. Chris Howard says:

    Wait a minute! I just purchased two whole cases of Woo (now with 10% more sex appeal)… Very good explanation of anecdotal evidence, and why it isn’t considered “strong” evidence. May I use your example when attempting to sell my friends New and Improved “WOO 3000 Super Happy Fun-Time Sauce and Laundry Detergent”? (absolutely, 100% guaranteed not to cause kidney failure, 60% of the time)

  5. Claire B says:

    Can’t blindly trust anything.

  6. captnkurt says:

    As they say, the plural of “anecdote” is not “evidence”.

    Also, @#4: Do not taunt Super Happy Fun-Time Sauce and Laundry Detergent.

    • Jim Shaver says:

      Nice SNL reference there, cap’n.

    • Retired Prof says:

      Well, I don’t know. The plural of “anecdote” is “evidence” if what you are studying is anecdotes themselves. Then you need to collect enough of them to form a body of evidence for statistical analysis.

  7. CW says:

    “Well performed tests take time and cost money, and it’s the party selling the product who bears that obligation.”

    What about videos of seeing a product in action? Surely those can be faked, but after showing video proof and testimonial support (I won’t call it “evidence”), what else does the party have to do in order to gain scientific acceptance or (at the very least) gain scientific interest?

    • Brian M says:

      @CW

      Perform proper scientific studies research. There are plenty of “see it in action” bigfoot videos, but those aren’t proof it exists. One must properly do the test with double (or triple blinding). Once you do that, you get the attention of others.

      Take, for example, homeopathy. Nobody in the scientific community gave it much thought, and assumed the null hypothesis about it. Then, suddenly, some research came to light, that was carried out by a reputable scientific lab. It immediately garnered the attention of scientists. And, of course, James Randi was called in. After some more science, it was proven to be false, and the original research was questioned for not being conducted properly.

      Thats how things get the attention of scientists. Even things that are wrong can get the attention of scientists. But there needs to be some due diligence on the part of the claimant first.

      • MadScientist says:

        I haven’t got any videos, but I’ve got my UFO photos. :) I’ll have to check them again to see if they’re grainy enough to remain unidentifiable.

  8. Dennis says:

    Your concluding statement is almost right. “So the next time you hear ‘I was a skeptic until…’, ” is a legitimate prefix if what follows is a rational examination of the evidence, of course.

    • Shahar Lubin says:

      Except that you don’t want to stop being a skeptic after…
      I was a skeptic until enough evidence to support the H. pylori hypothesis of peptic ulcer came about. I was still “a” skeptic after, and as a skeptic accepted the evidence and changed my mind on that subject.

      SO it might be legitimate if the “a” is dropped. Nothing(except for flying unicorns bearing bigfoots) would make me “a” non skeptic but possibly make me non skeptic of…

  9. Jim Shaver says:

    Anecdotes are unusually of unknown origin; and even when they’re not, they are uncontrolled, unscientific, and subject to a myriad of biases, influences, random variances, and external factors.

    Brian, I think that’s a typo there. I think you mean anecdotes are usually of unknown origin.

    Another version of the “I used to be a skeptic” phrase that I also hate to see in junk advertising is “I’m just a skeptic, so I didn’t expect it to work.” As if the guy is admitting his “skepticism” is some kind of character flaw. Oh, brother.

  10. dglas says:

    Dunning’s is a worse than worthless post.
    It offers us absolutely nothing.

    So what are *we* supposed to do? *We* don’t have access to laboratories. Dunning tells us the people putting the product out must provide the scientific evidence, but they won’t (it is expensive, right?) And then Dunning tells us us critics shouldn’t have to do the work either. The net result is that the work does not get done. So, guess where that leaves the rest of us – lost in the wilderness of mirrors of conflicting claims. We don’t have anything left to go on but our personal experience, and now Dunning denies us that. What now?

    Dunning’s point may or may not be true (his derision of the mere vulgar aside), but as useful information goes, it provides us absolutely nothing at all. The above boils down to “trust us; not them.” If popularizing science is the goal, Dunning could not have done worse (except, perhaps, by having a larger audience).

    People need real, empowering advice for the real world. We don’t need to be left paralyzed and without recourse by impractical idealism. Certainly, divorcing us from the process doesn’t do anything at all to help anyone with anything – especially the promotion of science.

    • Cthandhs says:

      I find this post *very* useful. This is a good response to a question I have gotten a couple times. Next time I have a discussion about woo with a friend and they ask if I have actually tried Super Happy Fun-Time Sauce and Laundry Detergent, I can frame a more cogent answer about why I haven’t and why I (and they) don’t need to.

      You seem to write from a perspective as you have to buy something, but you need good information about which one to buy. Well, you don’t. You don’t have to buy any of it. The key point you’re missing, is that *we* don’t have to experiment on ourselves. If *they* won’t give us evidence, *we* don’t need it.

      There’s no “trust us;not them” here. It’s all “trust no one” and “make your own decision based on available evidence”. If you don’t get evidence, don’t get the product.

      “People need real, empowering advice for the real world.” Is an oxymoron. Empowerment is all about being able to make my own decisions and to understand why I’m making them. Empowerment is not someone telling me what to do.

      • dglas says:

        “You seem to write from a perspective as you have to buy something, but you need good information about which one to buy.”

        Tell that to a patient with a consent form in his/her hands looking for a treatment for a lethal ailment. Reality imposes decision-making on us, whether you wish to admit it or not.

        “make your own decision based on available evidence”

        We cannot trust our get evidence from the advocate. We cannot get evidence from the critic. We cannot get our own evidence according to Dunning. What are we supposed to use for evidence? Never mind. I see you will not understand the point.

        ““People need real, empowering advice for the real world.” Is an oxymoron. Empowerment is all about being able to make my own decisions and to understand why I’m making them. Empowerment is not someone telling me what to do.”

        Talk about a miscaricature and a complete misunderstanding. I am not asking to be told what to do. However, science promises us certain things, including sound advice for matters of empirical study. Dunning’s little blurb makes all such utility unreachable. What precisely is the purpose of science if it isn’t to provide information (via predictive power) which we can use to make decisions? I talk about empowering people and you take that to mean controlling people? You aren’t even saying anything remotely intelligible. You are only trying to depict me as recommending enforcing a particular viewpoint. A cheap rhetorical ploy. I have no patience for disingenuousness; don’t try that again.

        You missed the point altogether.

      • Cthandhs says:

        “Tell that to a patient with a consent form in his/her hands looking for a treatment for a lethal ailment. Reality imposes decision-making on us, whether you wish to admit it or not.”

        What’s the decision? Between medical treatment that has scientific evidence and one that doesn’t? Between two “treatments” that both do not have scientific evidence? If I had a disease which could not be treated by science, I could (and would) choose not to be treated by pseudo-science, no matter how many anecdotes supported it.

        “We cannot trust our get evidence from the advocate. We cannot get evidence from the critic. We cannot get our own evidence according to Dunning. What are we supposed to use for evidence?”

        The scientific consensus. My doctor. Another doctor if I need a second opinion. Research into the scientific plausibility of effectiveness… I cannot begin to count the ways the scientific method and the professional community of science based medicine is capable of helping me and my loved ones through difficult medical decisions. As far as advocates and critics go, if the advocate will not or cannot provide science to back their claims, and the critics do, I will go with the critics. vice-versa is also true. Lots of evidence for vaccines, no plausible evidence against, I’ll get my shots. It’s really not that mysterious.

        “However, science promises us certain things, including sound advice for matters of empirical study. Dunning’s little blurb makes all such utility unreachable.”

        How is Dunning making science inaccessible to you? Dunning is specifically commenting about anecdotal evidence, which is not evidence at all. How is this making you powerless? There’s still a lot of science out there.

        “A cheap rhetorical ploy. I have no patience for disingenuousness; don’t try that again. You missed the point altogether.”

        Either I misunderstood your point OR I understood it well enough to make a rhetorical ploy. You can’t get away with calling me stupid AND cunning in the same breath. Make up your mind. I have no patience for disingenuousness. :P

      • dglas says:

        “The scientific consensus.”

        Apparently, you have forgotten the topic under discussion.

        Dunning posits a situation in which there is no scientific study to support nor refute. I wrote my example with that in mind. You changed the context to now include evidence that Dunning specifically did not allow for.

        This is why I say you completely missed the point. More interested in winning the argument than learning anything, I see. Are you done embarrassing yourself yet?

      • Cthandhs says:

        Evidently you missed my point. You don’t have to buy any of it. If there is *no* science to support it, *choose nothing*. Should I say that more slowly?

        Choosing nothing is a valid choice.

        More interested in throwing ad hominems than learning anything, I see. Are you done embarrassing yourself yet?

    • dglas says:

      Of course, none of Cthandhs’ nonsense addresses the concern I put forward. Contrary to his assumptions, we are often put in positions where we *must* make decisions, often without the support of rigorous scientific study (for or against). We can just try to wish that never happens, like Cthandhs, but wishful thinking doth not reality make. Refusing to ever buy anything without a thoroughgoing study to support it is simply impractical.

      So we, outside the laboratory, are left with precisely what to help us with our decision-making, now that Dunning has taken our own experience away?

      Dunning takes away, but doesn’t offer anything. Upon what basis are we supposed to make decisions?

      Do we roll dice? Do we consult a ouija board? Do we just refuse everything, lie down and die, like Cthandhs suggests? Do we magically grab “scientific consensus” out of thin-air when it has already been determined that we do not have it?

      How are we helping the cause of popularizing science by taking away all bases for decision-making from people?

      • Gonzo Skeptic says:

        For a worthless article you sure are spending alot of time trying to prove how much smarter you are.

        Perhaps you could write Brian and get your money back.

      • mikerattlesnake says:

        I feel like you’ve expended a lot of energy ignoring the obvious.

        1)Any medical product worth anything will have some evidence to back up its claims. Extensive research is expensive, but pilot studies that provide plausibility (and therefore funding for further study) are not particularly expensive, especially considering the profits that would come from an effective treatment. Medical products that make claims without research probably don’t have any interest in doing honest research as it would invalidate their claims.

        2)The default position should be to not use an unproven treatment. My mom had cancer (and fortunately it was breast cancer, which is relatively treatable and survivable) and she was offered reiki, german new medicine, and other treatments from her friends. She declined (or in the case of reiki, patiently went through it to make her friend feel better) because it’s crap. Your hypothetical situation of which box to check leaves two choices for an unproven and implausible treatment: check the “no” box and rely only on proven or plausible treatments, or check the yes box and try every single alternative modality until you are broke and still dying. The former is the correct answer because we NEED to have standards for what we will buy and treat ourselves with, though I would never fault a terminally ill person for choosing the latter.

        The whole point of this article is that you can reject a treatment without disproving it personally. I would imagine that would be liberating for the person you describe, aleviating some of the “what if I had done X?” guilt. You may not be getting the answer you want, but the claim that he’s leaving you without an answer is total bunk.

  11. Max says:

    Of course we answer “The last thing a skeptic would have done is rush right out and give their money to the (insert the salesman of your favorite alternative woo product here).”

    Maybe the skeptic got a free sample or a gift.

  12. Jim Lippard says:

    I’m not sure the proffered suggestion works for claims in certain realms of human life–aesthetic, artistic, emotional, sexual, psycho-pharmacological, gustatory, recreational… in some areas of life it is the experience that matters for its own sake, rather than intellectual claims about explanation and meaning. It doesn’t mean those things aren’t subject to scientific explanation (as well as being ripe for exploitation by bogus claims of woo), but sometimes first-hand experience gives you an understanding of kinds of value that you don’t get from a third-party description.

    • Shahar Lubin says:

      You mixing two modalities.

      There cannot be a scientific discussion on whether Dali’s exploration of surrealism lack because of they are too busy and as such prevent you from having that feeling that the logic of the picture is just out of your grasp, and that that feeling is what makes surrealism successful.

      There can be one about what pigments he chose to use.

      When a factual claim is made(a claim that, whether we know, or even technically are capable of knowing at the moment what it is, has only one true yes or one answer) science is the only useful tool we have, when dealing with non-factual claims, it is not. Now maybe science cannot prove or disprove that factual claim at this time, and so we rely on the null hypothesis for moment.

  13. Max says:

    Do you really need a thorough scientific test to see whether a scratch remover removes scratches? I’m skeptical that it works “As seen on TV” but if my friend tries it and says it works, I can believe that.

    • tmac57 says:

      Of course you may also have a gullible friend (don’t we all?), that didn’t tell you that the scratch remover only worked for a day or two until the oils dried out, and the scratch came back.

  14. I do think context is important. Brian’s post applies mainly to things with medical claims, like supplements, or certainly anything with a subjective but empirical outcome, or one that is not obvious to perceive (like products that claim to increase gas mileage).

    However, some applications are obvious. The classic example is the parachute – we don’t need double-blind controlled trials to prove they work. The concept is straightforward. All you really need is product reliability testing. I guess a scratch remover might be similar, if the results are obvious.

    For may products, like soap, for example, the basic concept (it makes you cleaner) does not need testing, but product comparisons might (if they claim soap A works better than soap B).

    None of this applies to aesthetics (I like the way soap A smells better than soap B).

    • Max says:

      Right, you also need more controls when there are more confounders. If a scratch goes away immediately after using a scratch remover, the cause is obvious, but if gas mileage improves after switching to a new motor oil, there could be other factors like changing weather.

    • Max says:

      Steve,

      I have a question about experimental controls.
      Let’s say there’s a correlation between alcohol and lung cancer, but when you control for smoking, the correlation goes away. Would it be fair to conclude that drinking alcohol doesn’t cause lung cancer? What if it causes lung cancer indirectly by increasing the urge to smoke? The original correlation caught this, but the controlled experiment didn’t.

  15. Max says:

    Ideally, the product test should be well-designed and independent.
    If the company’s well-designed test of its own product says it works, but my friend tries it and says it doesn’t, I might be more willing to believe my friend.

    • Johnny Slick says:

      Then no offense but you’re coming to your (possibly correct) conclusion for the wrong reasons. I like to think of skepticism as a mindset where you require a certain amount of evidence to believe in anything other than the null hypothesis. Think of it as a line the evidence must cross. A company’s own test, well designed or not, is probably not going to cross that line for me. If a friend then says it doesn’t work, that might actually shrink the overall level of evidence I have – get it further away from that line – but that doesn’t mean I’m “more willing to believe my friend”. It means that I happen to concur right now with my friend that there’s no evidence for this claim.

      As for anecdotes not being evidence, I think that sometimes we lose track of exactly why that is. If you just stopped 100 people on the street and asked them about Product X, 10 of them used it, and then documented their stories, those are still 10 anecdotes but they’re also 10 anecdotes that count for something in that they eliminate a lot of biases (specifically confirmation bias). Of course, there may be other biases that haven’t been controlled for, and sample size may be an issue, but at the least these 10 anecdotes count for *something* – a blip, maybe not enough for total proof, maybe more than the aforementioned company study, there’s not exactly an established weighting system here.

      • Max says:

        My ulterior motive for that post was that I suspect that Brian Dunning is a little too willing to believe industry studies of its own products. For example, his Skeptoid episode on the safety of aspartame cites a review by “a group of scientists from the NutraSweet company.”

      • Johnny Slick says:

        Okay, point taken. I certainly haven’t listened to every Skeptoid podcast; in fact, one of the things that I like about it is that there are so many episodes that I can listen to the ones that sound interesting to me and avoid the ones where his politics get in the way of his skepticism. It’s not so much about not wanting to hear the “skeptic’s” side of things as it is my not wanting to put a guy as fun to listen to at times as Dunning is in a bad light. I guess it’s similar to the way I don’t try to watch interviews of Tom Cruise or for that matter just about any other actor out there unless they have something to say about their craft.

  16. Bob Mcbride says:

    @the comment that “we don’t have access to labratories to conduct research. If there is some treatment modality out there the medical providers would jump all over it. You do have to show a mechanism for being better than placebo.

  17. tmac57 says:

    On the topic of research, a fun thing (though not scientific) to try at a party is to do a double blind beer (or wine) tasting with the guests favorites as part of the lineup, along with ones that they swear are not worth anything and that they would “never” drink. I’ve done this several times, and the results can be funny and quite surprising. The result that I loved the best was when one friend of mine who was adamant that he hated “X” beer, ended up choosing it as his favorite when blinded, and his favorite came in next to last out of six!
    The point is that this can open peoples eyes to be a bit more skeptical of the way that they make choices in life. Of course this is all an anecdotal story, so be skeptical (until you try it yourself of course).

  18. Tuffgong says:

    I’m sorry Brian but I have to respectfully disagree. I am willing to say there IS value in anecdotal evidence but the issue isn’t that it is anecdotal evidence, but whether or not people understand the context of that personal account and the context of who is saying it. The entire “reviews” industry surrounding movies, books, games, music and such rely on essentially “anecdotal” evidence to determine the quality of a product. However if one is properly skeptical, value can be found in that kind of information regarding any product.

    If I were to test a consumer product (not something that should be tested in a controlled environment) then my anecdotal experience shouldn’t automatically be useless just because I didn’t perform scientific experiments. That is the touchy problem with this kind of thing is it really does come down to whether or not people had the right idea and process in evaluating something (being skeptical, citing bias etc.) and amongst ourselves (humans in smaller groups and larger culture) it is hard to ascertain whether said person did or not. They could or could not have but it does take something like skepticism to cut through the confusion.

    This article felt a little off because I feel like I can make an informed or educated (not academic or scientific) statement about my personal experience with a product if I do it correctly. The problem is that “I was a skeptic until…” is 9 times out of 10 an empty statement that is either pure marketing or someone doesn’t understand what skepticism means. I’m looking this through the lens of consumer products as opposed to more medicinal (which I agree should never be relied on anecdotal evidence) but this is as much a misunderstanding about skepticism as it is science as a formal process.

    In the end I may or may not have actually disagreed with Brian, but I didn’t get the impression that this was about one type of product over another or just “products” in general.

    • Bob Mcbride says:

      Anecdotes are only useful in cases in which the only question is personal opinion. Things like what color you want to paint a room or was that performance (say in a movie or theatre etc) enjoyable. I’m hoping you are not suggesting that product safety (ie will that pill increase my odds of a heart attack) should be based on personal stories.

      In the case of how well a pill did performance wise, the improvement over placebo would come out in rigerous double blind testing.

      • Alan says:

        What about instances where you are trying to grade levels of service? For example, does a particular medical practice or even medicine within a region or country provide a sufficient level of service? There is a certain “personal opinion” there, but it certainly not just a matter of random choice as if you were debating whether Coke is better than Pepsi. Poor service can lead to bad outcomes, yet it is something that can’t easily be expressed in scientific terms.

        I make this example because I’ve seen skeptics use “oh, it’s just an anecdote” as code for “Oh, I can just ignore that.” No, in the proper circumstance you can’t just ignore it.

        That doesn’t mean that we should from there make absurd generalizations like “Doctor X treated me bad so all doctors are bad”. However, that bad experience is evidence of poor service within the medical industry. It’s just not convincing evidence of a broader pattern of bad service by itself. When combined with enough similar evidence then you would have something. However, using this “anecdotes are worthless” attitude can effectively negative entire categories of evidence with the rhetorical wave of the hand.

        In other words, we should make sure we are properly evaluating potential evidence and not just coming up with clever rationales for ignoring what we don’t want to hear.

  19. Wilson says:

    I always thought the problem with “I was a skeptic until…” was that, to put in bluntly, accepting a proposition doesn’t cause one to stop being a skeptic. If you stopped being a skeptic when you accepted the proposition (in this case, the proposition that the product works) then your acceptance is not really worth that much; if you accepted the proposition *as a skeptic*, then it’s worth a lot, because it means that you have taken evidence into account and arrived at a well-considered opinion.

    So, if you need to stop being a skeptic to accept the product, that’s probably a bad sign.

    • Sophie says:

      Good point. That said, sometimes, “I was a sceptic until I tried X,” is shorthand for, “I was sceptical about whether X would be any good until I tried it,” which is rather different from, “I was sceptical about everything in life, and then I tried X, and now I am a True Believer in anything you care to throw at me,” and there are shades of grey in the middle as well. It’s worth keeping an eye on semantics here.

  20. Lone Wolf says:

    Many people do not understand how science, evidence and even logic works. So they think anecdotes are evidence when logic will tell them anecdotes are not evidence.

    It’s the educational systems failure to teach basic science, logic and critical thinking that leads to many of these claims.

  21. scott says:

    This post is spot on, at least in the way that it was intended. Of course one can come up with exceptions (like the ‘scratch remover’ case mentioned on the thread), and it doesn’t mean that advice from friends should always be ignored (like a recommendation for a good restaurant from someone who has given good advice on such matters before, for example).

    The kind of cases that Brian is clearly referring to is exemplified in some experiences I’ve had which are all to common.

    When I was an MA student in linguistics, one of my professors who was into exercise praised a supplement that he claims gave him a tremendous amount of energy (obviously, he was way off topic in his lecture). I tried it out (I believe the cost was around $40 a bottle), and at first I thought it worked well. However, after buying it a second time, I began to doubt that I really was feeling any more energetic. I stopped taking it and didn’t notice any difference. My conclusion from this experience was that, contrary to ‘common sense’, we are not really that good at judging the effectiveness of health products through our own experience. This lesson cost me $80 (at a time when I didn’t have enough money to afford health insurance, by the way).

    A friend recently got into MonaVie fruit juice (as a distributor, of course) and was extolling its supposed miraculous benefits. I asked if there were studies done on it that he could point me to. He referred me to the testimonials page on the MonaVie website, but admitted he didn’t know of any actual studies. But, he claimed, all that didn’t matter. ‘Just try it for a few weeks and you’ll see for yourself’. Through Brian’s podcast and some other sources, I quickly could see that the evidence for the product and the plausibility of the claims were severely lacking. ‘Testing’ it for myself would most likely be a waste of time and money. I believe the going price of Monavie is also around $40. This time, I think I’ll keep that money in my wallet.

    Thank you for reading about my anecdotal evidence that we cannot trust anecdotes as evidence. ;)

  22. Max says:

    I got some of those green bags that are supposed to extend the life of produce, put half of tomatoes in a green bag, half in a regular plastic bag. They spoiled at the same rate. There’s my anecdote.

    • Shahar Lubin says:

      No, that’s your scientific experiment.

      Admittedly it’s a small sample size and as such do not provide yet with any conclusiveness, but the difference is, that if you conduct multiple tests of this nature and get the same results, that would establish useful evidence, but multiple anecdotes would not.

  23. Shahar Lubin says:

    One problem is that the story of science as usually told is the story of successes. We hardly ever told and learn of the failures, and as such all most all the science stories he hear are about how skeptics stood in the way until reluctantly made to change their mind.

    “Skeptics questioned so-and-so theory of what-ever until he was finally vindicated by the evidence”

    I hear that often from non-skeptics as an argument agains skepticism. What we might need is to incorporate not only the success stories into the science curriculum but the failures as well.

  24. Funkydebunker says:

    I could relay an anecdote to you all in support of my position. However, I would prefer to just tell you a story about myself instead. However, before I begin I would like to say that I dislike being called “a skeptic or a anything. I have the conceit that my nature and existence require more than a one word definition. I will grant that for our purposes I will go so far as to refer to myself as having a skeptical perspective. It was my understanding that skepticism, like science, is a process and not a position. Put it this way: it would take more than being proven wrong about some sketchy medical nostrum to completely overturn my science based world view. I like to think that this world view is different than faith, in that it is the best tool we have to model reality. So when I was recently diagnosed with osteoarthritis, my first reaction was to learn all I could about it. There is a lot of woo about this, and I believe that I can offer some first -hand explanations as to why this is so.
    The first thing I learned was that this was not curable, but there is some measure of control possibly and some symptomatic relief. And that is it. The reason that I can accept this as fact is because of my evidence based world view. So why did I dig $25 that I couldn’t spare out of my pocket and purchase a large bottle of woo? There were two reasons, one is about lottery tickets and one is about love. Why do I buy a lottery ticket once a week? The odds of my winning the big one are literally astronomical. If I bought a ticket with any real expectation of winning, I would be foolish. If I just put that bit of money in an account every week, I would eventually be guaranteed to have some money with interest over a long time. Yet probability has another side: the odds of any one particular person winning are immense, yet SOMEONE always ends up with the jackpot. I will get back to that in a moment.
    I had read the reputable reports about things like glucosamine, MSM, plus a lot of other herbal remedies. You will see huge stacks of these “supplements” at most drugstores these days. I guess that is because osteoarthritis is one of those diseases that a lot of people get who live long enough, and the boomer generation should be there about now, like me. I saw no evidence that proved that anything was effective. As in a lot of studies of herbal remedies, there was either no measurable effect compared to a placebo, or the evidence supporting it was at the extreme end of measurability. There just isn’t any repeatable evidence of any significant effect that I could find, except from the people who sold the stuff. I don’t need to try something for myself to see if it works: The scientists are much smarter and better at that sort of thing than I and I will take their word for it. At least the scientists will provide the evidence they used to form opinions, best guess, hypothesis or theory. Yet, I still bought the damn pills! Why?
    My wife had a problem with her knee a few weeks ago, and (according to her) this stuff fixed her right up. When I explained to her that I had a hip problem, not a knee problem, and that her anecdotal evidence was not creditable enough to refute the findings from people whose job it is to test this stuff, I was met with the response “ Fine- go through your life with a limp, or try something that worked for me!” I thought to myself, “would I rather be right, or would I rather be happy?” Yes, I could have lectured her about all the reasons anecdotal evidence is unreliable. I could have discussed the double or triple blind method. I could have self righteously lectured myself into a night on the couch. So I agreed to try one bottle, because I like her too much argue with her long. I did not, however, mention that we are expected to take these pills every day for the rest of our lives: at an average of 50 cents a day for these things, that adds up to quite a lot, if you live long enough. No, I would try one bottle, and if it didn’t work then there would be no harm, no foul. Even still, I was the one who her accused her of being closed minded by not considering the evidence that disproved her beliefs. As for her reaction to that folks, well you just had to be there.
    Essentially, at this point I had not altered my skeptical outlook. I had just been a coward. I am sure that there is some bright anthropologist or sociologist or something who could do a paper on the evolutionary advantage of cowardice. Send me a copy when you do, I could use the validation. Anyway, as I was paying for these “herbal supplements” that I knew were useless, I thought about lotteries. (See, I TOLD you that I would get back to that in a moment!). I thought about the fact that any measurable effect is so very small as to be negligible, a statistical anomaly, regression to the mean, incorrect diagnosis or whatever. I know that it is like saying “this stuff really works, if you don’t count the people that it did nothing for.” Perhaps this was similar to the lottery (although the price to buy in was much higher and the payout was much smaller-nothing). Someone always wins (eventually), and the alt-med helps some people, sometimes (maybe, but probably not). Call it cognitive dissonance, selling out, or being seduced by the Dark Side. Caught between a lot of pain and fear of future pain, and the well meaning pressure from a loved one, I caved and bought the damn woopills. If a month from now I am somehow all better, I will not re-consider my skeptical outlook. I will just figure that I got lucky.
    In conclusion, I would have to say that perhaps some of the reasons people buy into woo of any description have nothing at all to do with logic. There are plenty of reasons, such as emotional investment, wishful thinking, hope, fear, advertizing (!), peer pressure and etc., but they all add up to the fact that we are human, swimming in a soup of hormones and conflicting motivations. It is possible that those who do not believe by reason cannot be convinced by reason alone. I hope that I am not being a troll or doing something else bad, because I am a relative blog virgin, but I would like to suggest that it is not enough to be right, and there is a need for something more convincing. Whatever the solution is, well I am sorry to disappoint you but I have no idea. I do not suggest the abandonment of all that we do skeptically, far from it. Nor do I suggest that in the spirit of fair play or equal time that we let the Intelligent Design folks or Creationists (or whatever they are calling themselves these days) have the same authority as those whose knowledge is based on the laws of cause and effect, logic and evidence based reason. Science and faith are diametrically opposed, and can only weaken each other by trying to reconcile the two. Yet if we are to weaken the hold of the big bad woowoo, I believe we are going to need something beyond arguments that prove conclusively that we are right and they are wrong. For many, apparently evidence is not enough.
    When my child was little, he was scared of monsters under the bed. No amount of discussion, logical argumentation or philosophical discourse was going to change that! He needed a nightlight, and a hug, a bedtime story and some reassurance. You can’t just take something away from a child- you have to replace it with something else. I think that may be part of what the credulous are purchasing along with their “supplements”- reassurance and hope. Can anyone recommend somewhere else to buy that? Big Herbal costs too much.

    • Max says:

      You can find reassurance and hope in real medicine. Where are the doctors in your story?

      • Funkydebunker says:


        ■Max says:
        March 12, 2010 at 1:35 pm
        You can find reassurance and hope in real medicine. Where are the doctors in your story?

        Reply:
        Once it became obvious that this was a medical issue, I saw a doctor immediately and continue to see specialists. I talk to them about everything I have read about the disease, and took the real medicine they gave me. My sources are the public library, the internet, the back of the bottles of woopills and hearsay. The medicine did not claim to be a cure, but something that would help, and provide some relief. I eventually make up my own mind after doing a bit of research, but usually side with the scientists and doctors, because their arguments are more compelling.Yet when I am diagnosed with something incurable, I see no opporunity for reassurance or hope. In point of fact, I am enough of a realist that I don’t need them, and I see a doctor offering such platitudes as being dishonest. But that is just me- I can deal with the truth, and complain to my rabbi about it later if I need to vent. None of that excuses my deliberately purchasing woopills that I knew wouldn’t work. What I have trouble undertanding is not the people who purchase something homeopathic because no one has explained to them what it really is that they are buying. No, the ones I have trouble with are the willfuly ignorant, the ones who either refuse to examine what they believe or deny any and all evidence to the contrary. Some people refuse to accept that they have some incurable disease, and woo can become a faith that sustains them, hence the willful ignorance. But I can guarantee you my friend that the doctors were in on this from the moment I knew there was a need for them. Believe it or not, it is easier to care about the evil “Big Pharma” when you are not in constant pain. I have never been the kind of person to go to the herbal counter first. The rare exception to this is when I need a sleep aid. The ones I buy work not because I believe in them, but because they have chemical compounds clinically proven to be effective. If I wanted a homeopathic sleep aid, well I guess that I could just drink a glass of water couldn’t I? According to the theory, there is bound to be some of whatever I want in the water’s memory, isn’t there? I am sorry for climbing back into the pulpit, but it does give me an opportuntity to state the obvious once more: what we do here, and what skeptical organizations in general do is increase awarness, which is part of the solution. I am just saying that it is not the whole solution, but unfortunately have nothing further to suggest (YET!).

  25. Jon says:

    One way an anecdote can gain weight is through the duration of the trial. In my personal experience with placebo medicine, I’ve found that, eventually, the little bit of effect I’d thought I’d experienced dropped out. These attempts include baldness remedy (can’t remember what), Avodart for BPH, Sawgrass for BPH, chiropractic for lower back pain, and acupuncture for lower back pain. In all the above, I initially felt that I was responding. And, in all the above, after a while I realized there was nothing there. The longest trial was for Avodart. I gave it six months since it was an approved medical treatment for the condition and seems to work for some people. After an initial positive result, I realized after six months that I was wasting my money.

    • Max says:

      The longer duration certainly paints a fuller picture.
      The placebo effect usually doesn’t last very long, whereas some treatments need time to show a real effect.
      Still, a sample of one person is small. A treatment that doesn’t work for you may work for others. “Results may vary.” Medicine is not an exact science, but I hope it’ll be much more exact in the future.

  26. Max says:

    captnkurt,
    > As they say, the plural of “anecdote” is not “evidence”.

    Check out this story on “Patient 2.0″
    http://www.time.com/time/magazine/article/0,9171,1957460,00.html

    “In 2008, in response to the publication of a small study indicating that lithium helps people with ALS, the site [PatientsLikeMe.com] evaluated the collective experience of its members who had taken the drug and determined that lithium didn’t work — a conclusion it reached six months ahead of similar findings from conventional clinical trials.”

  27. Max says:

    captnkurt,
    > As they say, the plural of “anecdote” is not “evidence”.

    Check out this story on “Patient 2.0″
    http://www.time.com/time/magazine/article/0,9171,1957460,00.html

    “In 2008, in response to the publication of a small study indicating that lithium helps people with ALS, the site [PatientsLikeMe] evaluated the collective experience of its members who had taken the drug and determined that lithium didn’t work — a conclusion it reached six months ahead of similar findings from conventional clinical trials.”

  28. MadScientist says:

    I was a skeptic until I discovered El Chupacabra – since then I just knew that homeopathy and naturopathy work!

  29. I was highly skeptical until I tried a fuel additive that promised to half my cost of fuel, increase mileage and boost power upwards of 50Kw. I threw that sucker on the Pacer to test it’s authenticity and I’m glad to say I am no longer highly skeptical. I’m now extremely skeptical.

  30. Max says:

    Funkydebunker, here’s a silly question. Why try glucosamine instead of, say, jelly beans? At least jelly beans haven’t been found to be as effective as a placebo in a large clinical trial. Plus, jelly beans taste better and have a better safety profile.

  31. I was skeptical but then I…accidentally the whole thing.

  32. Sophie says:

    Excellent post, but there’s one area where I disagree with you. You seem to have a rather extreme approach, with science = reliable clinical trials = good on the one hand, and woo stuff = no trials at all (or trials showing that it doesn’t work = bad on the other hand. I think it’s a spectrum instead. There are plenty of things which fall in the middle, both in the realm of orthodox medicine (since we mainly seem to be talking about medical stuff here) and alternative therapies.

    My own background is that I have severe ME, and have had it for fifteen years. It’s currently incurable, and in terms of treatment for individual symptoms, there are plenty of things you can try, but no one thing will help everyone, and the vast majority of things won’t help you at all (yes, including orthodox medicine). Problem number one is that most drugs are trialled on healthy people, or on people who only have one medical problem, and very few of them are trialled on people with ME. And folks with ME are extremely ill and tend to react badly to a lot of medication, so this is something of an issue. The first time I went on hormonal contraception, for instance, the Family Planning Clinic assured me that it was entirely safe, that it never caused anyone any problems, and that they had given it to other women with ME without any problems. They stuck to this story for the next six months, and eventually gave me hormone tests and discovered that yes, my hormones were completely messed up. In later years, they stopped handing out Depo Provera so readily, as it’s now well-established that its side effect profile is lousy. Rinse and repeat with plenty of other meds.

    You soon become very sceptical when you realise that you are in a patient group which reacts atypically to most things, and which is very poorly researched. (Come to that, being a woman puts me in a patient group which is poorly researched – clinical trials hugely prefer using men, as women’s hormonal fluctuations make them more changeable as subjects. It’s only recently that they’ve noticed such problems as women reacting quite differently to heart medications to men.) You also notice that there is quite a lot of bad science out there. Not to mention that media reporting of science is downright terrible, and will sometimes manage to report the opposite results to what the clinical trial actually found. Patients with ME are often told that CBT or graded exercise will fix them, which has somehow become the recommendation as a result of trials which did such things as not using proper diagnostic criteria (it’s thought that at least half of the participants actually had depression instead), ignored the large proportion of drop-outs, used short-term and/or subjective appraisals of improvement, and then classified the “got slightly worse” group in the “improved” category. I’m just citing these ones as they’re what I’m most familiar with, but these problems occur all over the place, and seem to be a particular problem with the area of psychotherapy. It can take a while before errors with clinical trials become apparent, and sometimes big drug companies have an interest in pushing the trials which favour them and hushing up the trials which don’t. There is also the matter of using a drug off-label, when the evidence to support it for this use is scantier but it may be worth a try. I think a sceptical approach is always useful with medicine, which in my experience tends to involve doing some googling (sometimes medicine information sheets are more useful from one country than from another – hallucinations are acknowledged as a side effect of Ambien in the US but not in the UK, for instance), asking around in patient communities (ones which I know and can rely upon to some degree, as opposed to random strangers), and having a chat with my local pharmacist.

    When it comes to alternative therapies, again I have quite a lot of experience here. Most people with ME have tried pretty much anything in their first few years, out of desperation. Then you realise that you’re pouring a lot of money down the drain for little if any benefit, and develop a shorter temper with the idiots who tell you to drink dangerous quantities of olive oil in order to “flush” out gallstones, or chi machines, or “earthing sheets”, or the Lightning Process (a seriously nasty scam which actually endangers people). But there are a few things that help some people, which may be worth a try. Take acupuncture, for instance. The theory behind Chinese acupuncture is something that we can all agree is woo of the first degree, and which incidentally is very similar to the theory of “humours” that was the norm in medieval European medicine. And lots of claims are made for acupuncture, many of them quite dotty. However, it does have a reasonably good track record for pain relief, and there are now organisations which deal in medical acupuncture, where it is studied and practised within the confines of Western medicine. Personally, I’ve only tried Chinese acupuncture (years ago, when I was less sceptical, and unaware of medical acupuncture), and found that depending on the practitioner, it made me feel noticeably better, noticeably worse, or no different. I gave it a shot, but I don’t like anything that doesn’t give reasonably reproducible effects. However, if I had the resources to get to a medical acupuncturist now (I’m housebound these days, plus they’re quite hard to access on the NHS and I can’t afford private treatment), I would definitely be interested in trying it. My expectations would still be low, but then my expectations of pretty much any treatment are low by now.

    Then there are things such as massage for pain relief – researched up to a point, but a very individualised thing, and there are lots of schools of massage out there. Or two treatments which I personally have found very effective: bright light therapy and darkness therapy for sleep disorders. There’s a fair bit of research behind them by now, solid stuff as far as I can tell, and I’ve certainly found that they work, but they are still making their way into mainstream medicine. This roughly means that the sleep specialist encouraged me to use a lightbox and gave me instructions on how to use it, but couldn’t prescribe one on the NHS, and the GP said, “Oh yes, that makes sense,” when I explained it to her, rather than having been taught about it already in medical school.

    So there are plenty of areas in the middle. Personal anecdotes can be useful when you don’t have a failsafe level of medical support to fall back on. For instance, since having gallbladder surgery I have problems with severe upper abdominal pain if I take opioids, something which turned out to be on the medication leaflet. A few people mentioned that they find that taking the medication on a full stomach circumvents that problem, I tried it, and it pretty much works. At that point, I was getting no help from my doctors or pharmacists, who weren’t aware of this issue (though I eventually pointed out to one that if it’s mentioned on the medication leaflet, it must be documented somewhere, so could they please look it up), and who could only make guesses. I was faced with the possibility of losing the only type of pain relief which actually works for me. Anecdotes were useful here. Similarly, when I experienced bad nausea after the surgery, I asked around for advice on foods which are easier to eat when you’re nauseous, and got a better range of suggestions from friends than I did from my GP.

    …Which is a very long way of saying “It’s complicated”! I’ve heard people saying things like, “I was sceptical that a low-dose antidepressant could work for pain relief, and then I tried it.” It simply means that it’s not 100% mainstream (it’s pretty well accepted by the medical community as something that works for some patients), and sometimes it just means that it’s something the patient hadn’t heard of before and finds counter-intuitive. The usefulness of anecdotes depends on the situation, and sometimes a patient will have more to gain than they have to lose by trying something which looks reasonably promising, rather than the things which have “total scam” written all over them. There is no point in trying a chi machine, when someone has recommended it to you in a vague “it cures everything!” way, but you don’t actually have any medical need for anything of the sort, plus all evidence points to its being a scam, and an expensive one at that. There is every point in trying the Miracle Ball Method if you have bad muscular pain in your back, since despite the daft name it’s a fairly sensible little thing, cheap and easy, follows known laws of how the body works (it’s basically a way of getting your muscles to relax, with added deep breathing for relaxation) – and in this case the anecdotal evidence of reviews on Amazon has a certain purpose.