We’ve all heard the infamous anecdote used in a multitude of advertisements for various flavors of woo, “I was a skeptic too until I tried (insert your favorite alternative woo product here).” It’s an effective and compelling sales pitch to laypeople, so much so that it’s become so old and hoary that it usually provokes a laugh from experienced skeptics.
Of course we answer “The last thing a skeptic would have done is rush right out and give their money to the (insert the salesman of your favorite alternative woo product here).” And here is where the heads butt. Believers often feel that a truly skeptical and scientific process would have tested the product by trying it firsthand. And, to the average layman, inexperienced in science, that sounds extremely reasonable.
This is going to be a very short blog post, because this question is very simple. No, trying a product personally is not a useful way to assess its value. A scientist understands that, a layperson does not; it sounds thoroughly counterintuitive.
The scientist knows that anecdotes and testimonials, while they can be interesting, are not useful evidence, not even the scientist’s own. Anecdotes are usually of unknown origin; and even when they’re not, they are uncontrolled, unscientific, and subject to a myriad of biases, influences, random variances, and external factors. Well designed scientific tests control for, and eliminate, as many of those influences as possible — and often, all of them.
The skeptical process does not ever require personally sampling the woo. Why would it? You’re certainly welcome to try the product if you want, but doing so is completely outside of a meaningful scientific assessment.
Does this mean that every time we hear a woo claim we must go out and perform a thorough scientific test? Of course not. Well performed tests take time and cost money, and it’s the party selling the product who bears that obligation. In science, good evidence is required to move from the null hypothesis (the null hypothesis states that the claims about the product are unsupported until they are proven true). So when I’m told about a woo product, I’m probably willing to ask to see good scientific evidence, but I’m not obligated to provide such evidence myself. I’m sure as hell not obligated to buy the damned product.
So the next time you hear “I was a skeptic until…”, you know that (a) that person is inexperienced in science, and (b) their endorsement of the product was born from an invalid process.
Your thoughts and comments are welcome.