SkepticblogSkepticblog logo banner

top navigation:

Defending the Null Hypothesis

by Brian Dunning, Aug 11 2011

When an “expanding Earth” crank demands to see “peer-reviewed evidence” that zero point energy is not causing the Earth to expand from within, what do you show him?

When a homeopath demands you produce experimental proof of the nonexistence of water memory, what do you show him?

The past couple of weeks I’ve done a half dozen or so show radio interviews regarding a growing movement in Canada to get radio transmitters banned because of a belief that RF injures people. From all we’ve learned from evolving on an RF-rich planet, it’s pretty clear that’s not the case. I’m in the position of defending the null hypothesis, stating that no sudden, implausible, non-evidenced threat is known to exist. But I’ll assure you: Producing published evidence that shows the null hypothesis is often harder than it sounds.

Publication bias is a part of the problem. This is the tendency for research that rocks the boat to receive more attention than research that tells us nothing new or interesting. The overwhelming majority of actual research in the real world tells us nothing new or interesting, but you rarely see a cover story in Nature trumpeting “People Can’t Fly!”

Null hypothesis papers are also much less likely to be submitted for publication in the first place. This results in what we call the file drawer effect. Most research, which tends to have non-startling results, gets filed away; while the small number of results that show unexpected results tend to be submitted, published, and to receive more attention. Therefore, if you were to do a survey of all the published research on any given question, you’d likely get a skewed perspective; and it would be skewed away from the null hypothesis. Thus, on any given question, the real answer is probably closer to the null hypothesis than a thorough meta-analysis would show.

The inevitable result is that to cite research that most accurately reflects what we’ve truly learned, it’s necessary to cherrypick. An unsavory thought, to be sure.

The larger and better the study, the more likely it is to be accurate. Therefore, when we logically put all of this together, larger and better studies are more likely to show results nearer the null hypothesis. Conversely, published papers are more likely to be less accurate, further from the null hypothesis, and to be from a smaller study of lower quality.

The job of promoting public understanding of science has never been easy and probably never will. Presenting information in a fresh, interesting, and commercial light is one side of the coin; going the extra mile with your research and digging past the popular volumes and into the better performed, more accurate, less interesting research is the other. Then it’s a matter of putting the two together. If it sounds hard to do, it is; and that’s why bad information will probably always be more popular.

27 Responses to “Defending the Null Hypothesis”

  1. John K. says:

    If we could get everyone to let go of ideas based on a lack of empirical evidence, there would be little need for a skeptical movement anymore. “You can’t have absolute certainty that it is not” is one of the first things you hear from anyone defending anything that has no evidence to back it up.

    I think making efforts to eliminate personal bias and discarding ideas that are not supported by the preponderance of evidence is the very definition of “being a skeptic”.

  2. Labels says:

    “You can’t have absolute certainty that it is not” is often followed by “you’re not the boss of me”.

  3. With respect to the expanding earth crank and the homeopath, it’s worth emphasising: the onus is on THEM to provide evidence for their claims.

    The world wouldn’t function if things were the other way round, if it were acceptable to make any crazy claim whatsoever and then demand that other people spend their time debunking it.

    Part of communicating skepticism to people should include emphasising these points. When YOU say something YOU should give me a reason to believe it; it’s not I who should give you a reason to DISBELIEVE it.

  4. Animal says:

    At some point debating with such people becomes counter-productive. I say this as a person whose two brothers-in-law are raving, foaming-at-the-mouth religious loonies who maintain that the earth is but 6,000 years old. (My wife has much more sense, else I would probably not be married into the family.)

    There will always be people who believe bizarre crap for bizarre reasons.

  5. Somite says:

    What I try to tell people is that things for which there is no evidence are indistinguishable from imaginary objects.

    This is why it is so upsetting that we have to deal with the “testability” canard that some skeptics are promoting lately. Lack of testability = a null hypothesis = imaginary. There should be nothing else to say. The recent trend of framing the null hypothesis as “untestable” sounds to people like “possible” when it should be imaginary.

    • BillG says:

      Careful we should be as imagination is the germ of scientific discoveries and revolutions: “imagination is more important than knowledge” – Einstein.

      Obviously though, it’s still fantasy until testability.

  6. MadScientist says:

    There’s a lot of experimental evidence for the non-existence of water memory – in fact, given all the numerous uses of water, there are absolutely no observations consistent with the claims of water memory.

    What’s the “expanding earth”? I’ve never heard of that nonsense.

  7. Max says:

    “I’m in the position of defending the null hypothesis, stating that no sudden, implausible, non-evidenced threat is known to exist.”

    What about a non-sudden, plausible, evidenced threat?

    It’s good to ask where the null hypothesis came from in the first place. If you’re dropped in the middle of the jungle, do you assume that everything is harmless until proven otherwise? No, you assume everything is harmful until proven otherwise, which is basically the precautionary principle that libertarians want to do away with because it hurts financial interests.

    So if you claim that something is harmless, you must have a good reason.
    “From all we’ve learned from evolving on an RF-rich planet, it’s pretty clear…”

    It’s the appeal to nature! It’s only a fallacy when others use it.
    But if you ignore unnaturally high levels of “natural” things, you end up sounding like Michele Bachmann, who argued that CO2 “is a natural byproduct of nature,” and, “As a matter of fact, carbon dioxide is portrayed as harmful, but there isn’t even one study that can be produced that shows that carbon dioxide is a harmful gas.”

    The file drawer effect often results from funding bias, where studies that don’t support the interests of the financial sponsor get filed away.
    http://en.wikipedia.org/wiki/Funding_bias

    “A 2006 review of experimental studies examining the health effects of cell phone use found that studies funded exclusively by industry were least likely to report a statistically significant result.”

    “The FDA determined in 2008 that the bisphenol A (BPA) in plastic containers is safe when leeched into food, citing chemical industry studies. Independent research studies reached different conclusions, with over 90% of them finding health effects from low doses of BPA.”
    http://www.washingtonpost.com/wp-dyn/content/article/2008/07/14/AR2008071402145.html

    • In case anybody but Max missed it, my main point in this post was to promote a pro-corporate, pro-libertarian agenda. WTF?

      Max my friend, I don’t what I’d do without you. :) :)

      • Max says:

        You got it, buddy.

        It’s funny to hear the anti-regulation people resort to the appeal to nature when it comes to carbon dioxide, sugar, radiation, tobacco, etc. It’s all harmless because it’s natural, right?

      • Max, for your birthday, I’m going to ask you to send me all of your political preferences and ideologies. Then I’ll post a blog espousing whatever the opposite is. Think how happy you will be! It’s like your heaven!

    • SocraticGadfly says:

      Max, I’ve been on vacation. Thanks for standing in. And, yes, publication bias can rock the boat with big money behind it, like the Galileo Project’s recent claim about CO2.

      That said, Wiki’s page on the null hypothesis does specifically note that it is vulnerable to publication bias criticism as well as straw man criticism. (I can certainly see that; there’s no guarantee that a particular null hypothesis will be well framed.)

      http://en.wikipedia.org/wiki/Null_hypothesis

      • Max says:

        What’s special about the null hypothesis?
        If I flip a coin, is the null hypothesis that it’s a fair coin? Why? Because it’s more probable? Maybe I pulled it from a bag of two-headed coins.
        If I pick a random industrial solvent or an all-natural mushroom, is the null hypothesis that it’s safe to drink or eat?

  8. Ashley Harron says:

    I couldn’t disagree with you more Max. I am on Brian’s specially patented wellness plan whereby I eat nothing but radioactive sugar, breath nothing but cabin dioxide, and get my vitamins from an IV tobacco drip and I feel fine.

    He may have used examples you disagree with but he makes a valid point about how we process information.

    • Max says:

      “When a homeopath demands you produce experimental proof of the nonexistence of water memory, what do you show him?”

      Show him the experimental proof “that liquid water essentially loses the memory of persistent correlations in its structure within 50 fs.”
      http://www.ncbi.nlm.nih.gov/pubmed/15758995

      But if you have absolutely no clue whether or not water has a memory, then you have no good reason to assume it doesn’t. To put it in Bayesian terms, your prior odds are 1:1.

      So if you’re in the position of defending the null hypothesis, you should do your homework and learn the evidence supporting it. If there’s no evidence supporting it, why defend it?

      • The homeopathic claim of water memory is that it retains a “spiritual essence”, not that it retains “persistent correlations in its structure”.

    • Somite says:

      But Max; just because someone says something it doesn’t magically gain any probability of being true. Nothing is real until there is evidence for it or at least it is mechanistically possible.

      • Max says:

        Of course, but Brian is saying that even when there is evidence against the null hypothesis, it’s probably just due to publication bias.
        Just because someone calls something a null hypothesis, it doesn’t magically gain any probability of being true either.

      • I also said that regulation of industry is bad (don’t know who could have missed that). I wish I had your reading comprehension skills…

      • Max says:

        You take every opportunity to defend big industry from “alarmism,” but you never criticize the industry’s doubt-mongering campaigns.
        You’ve cited biased sources like JunkScience on DDT safety, the Heritage Foundation on nuclear safety, ACSH and CGFI on organic food myths, and the American Chemistry Council and ILSI on plastic bottle safety. These are basically industry front groups. You do not cite opposing groups like CSPI or the Union of Concerned Scientists.

        All this suggests a pro-industry anti-regulation bias.

  9. Wally says:

    Selling the sensational, not the null hypothesis, makes the media more revenue from it’s advertisers by increasing it’s ratings.
    Advertisments sell sensational distortions much better than the truth. Claims of Clinically Tested or Natural have no real meanings.
    Everthing that exists is part of nature thus is natural. Natural does not make something good or bad, healthy or unheathy. Death is Natural. Whether it is healthy or unhealthy depends on perspective. It may be healthy to the carnivore but unhealthy to the prey.

  10. Collin says:

    I think it’s fairly easy, as a rudimentary thought experiment, to defend the null hypothesis. You simply have to ask them which is more numerous: the number of events X which have no effect on some phenomenon Y, or the number of eventz Z which definitely have an effect on Y. No matter how you frame the question, X is much larger than Z, possibly infinitely so. It’s therefore overwhelmingly more likely that a proposed cause is a member of X and not of Z.

    The null hypothesis isn’t just a fiat, a priori rule, but an adaptation from this kind of thought experiment.

    • Max says:

      The null hypothesis doesn’t have to be probable.
      If you test whether Tylenol is effective at relieving pain, the null hypothesis is that it’s not effective, even though that’s improbable.
      If you test whether gasoline is safe to drink, the null hypothesis is that it’s safe, even though that’s very improbable.

  11. Bill Minuke says:

    Message to Max

    Here is my analogy Max.
    If I make many specific medical claims, some of which are contradictory and publish them saying, these are true, there is no evidence to the contrary. I’m right, there is no evidence to the contrary. But that doesn’t make claims true. Where did the claims come from? Somebody made them up. If someone disproves a few of them, I could move the goal posts a little or I could claim “conspiracy” or that science will one day catch up with me and then it will understand the truth. But basically, I’ve constructed a body of untruth and probably damaged some lives as a result.
    The damage occurs because I’ve made claims that were accepted on face value. If I had the burden of proof, where I needed to show rigorously that each of my claims was valid, then there would be no harm. Likewise, if my audience/followers accepted the null hypothesis, then I would be forced to shoulder the burden of proof.
    It is not reasonable to transfer the burden of proof either. In five minutes I could probably propose more claims than could reasonably be researched in the next few decades and with billions of dollars of research. That’s not an efficient use of resources and it’s not how science works.

    Thus, if I make a claim and believe it, I need to prove it.

    Advice to Max. I can’t speak for others, but when I’m enjoying a debate or argument about a topic, and one of the debaters uses an ad hominem attack, I lose interest in that persons message. Sure we all know that Brian is part of the Illuminati who is secretly funded by big pharma and lizard aliens from area 51, but so what? His arguments stand or fall on their own. If you disagree with his facts, there’s a whole internet out there for you to amass opposing evidence. Rash generalizations get you nowhere with good skeptics.

    • Max says:

      If you claim that something is effective or harmful, you need to prove it.
      If you claim that something is ineffective or harmless, you need to prove it.

      Where do you see any ad hominem?
      I pointed out that anti-regulation people often criticize their opponents for their appeals to nature, but don’t notice their own appeals to nature when it comes to outputs of big industry like radiation, CO2, and sugar.
      Brian insinuated that my opinion that he’s anti-regulation is due to poor reading comprehension, so I gave some of the reasons why I think he’s anti-regulation.

  12. Bill Minuke says:

    Max

    If you claim that something is effective or harmful, you need to prove it.
    If you claim that something is ineffective or harmless, you need to prove it.

    This is true. We agree. The only stipulation I would add is that, if you assert that something is effective, i.e. homeopathy, I don’t need to prove that it is not. It is not effective until the initial claimant proves that it is effective.

    “Ad Hominem” from wikipedia short for argumentum ad hominem, is an attempt to negate the truth of a claim by pointing out a negative characteristic or belief of the person advocating it.

    Your point whether true or false, is off topic. I read it as an ad hominem, as defined above ( not as a personal attack).