SkepticblogSkepticblog logo banner

top navigation:

Gambling on ET

by Michael Shermer, Jul 19 2011

How to compute the odds that claims of extraterrestrial life discovery are real and reliable

The Search for Extra-Terrestrial Intelligence (SETI) has to be the most interesting field of science that lacks a subject to study. Yet. Keep searching. In the meantime, is there some metric we can apply to calculating the probability and impact of claims of such a discovery? There is.

In January, 2011 the Philosophical Transactions of the Royal Society published 17 articles addressing the matter of “The Detection of Extra-Terrestial Life and the Consequences for Science and Society,” including one by Iván Almár from the Hungarian Academy of Sciences and Margaret S. Race from the SETI Institute, introducing a metric “to provide a scalar assessment of the scientific importance, validity and potential risks associated with putative evidence of ET life discovered on Earth, on nearby bodies in the Solar System or in our Galaxy.” Such scaling is common in science—the Celsius scale for temperature, the Beaufort scale for wind speed, the Saffir-Simpson scale for hurricane strength, and the Richter scale for earthquake magnitude. But these scales, Almár and Race argue, fail to take into account “the relative position of the observer or recipient of information.” The effects of a 7.1 earthquake, for example, depends on the proximity of its epicenter to human habitations.

An improvement may be found in the Torino Scale that computes the likelihood of an asteroid impact and the risk of its potential damage—from 1, a near miss with no danger, to 10, certain impact with catastrophic consequences. But Almár and Race note that “the scale does not include any consideration of the observations’ reliability.” Building on SETI’s Rio Scale for evaluating the effect on society of an ET discovery, Almár and Race propose the London Scale that multiplies Q x δ, where Q (scientific importance) is the sum of four parameters:

  • life form (1–5, from Earth-similar life to completely alien),
  • nature of evidence (1-6, from indirect biomarkers
    to obviously organized complex life),
  • type of method of discovery (1–5, from remote sensing
    to return mission sample), and
  • distance (1–4, from beyond the Solar System to on Earth).

This sum is then multiplied by δ (a reliability factor) ranging from 0.1–0.5, from probably not real to highly reliable. The maximum Q can be is 20 x .5 = 10.

For example, Almár and Race compute the odds that the Allan Hills 84001 Martian meteorite contains alien life as (2+2+4+4)0.3 = 3.6 for scientific importance and credibility, noting that “several scientific counter-arguments have been published and the discovery has not been generally accepted.” I would assess the recent claim of arsenic-based life in Mono Lake as (2+1+4+4)0.2 = 2.2, fairly low by comparison.

Such scientific scales attempt to bring some rigor and reliability to estimates of events that are highly improbable or uncertain. The process also reveals why most scientists do not take seriously UFO claims. Although the first two categories would yield a 5 and a 6 (completely alien and complex life) and its distance is zero (4, on Earth), the method of discovery is highly subjective (perceptual, psychological) and open to alternative explanation (1, other aerial phenomena) and the reliability factor δ is either obviously fake or fraudulent (0) or probably not real (0.1), and so Q = (5+6+1+4)0.1 = 1.6 (or 0 if δ = 0).

The Phoenix lights UFO claim, for example, was a real aerial phenomena witnessed by thousands on the evening of March 13, 1997. UFOlogists (and even Arizona governor Fife Symington) claim it was extraterrestrial, but what is δ for this event? It turns out that there were two independent aerial events that night, the first a group of planes flying in a “V” formation at 8:30 that started a UFO hysteria and brought people outdoors with video cameras, which then recorded a string of lights at 10:00 that slowly sank until they disappeared behind a nearby mountain range. These turned out to be flares dropped by the Air National Guard on a training mission. Ever since, people have conflated the two events and thereby transmogrified two IFOs into one UFO. So δ = 0 and Q shifts from 1.6 to 0, which is how much confidence I have in UFOlogists until they produce actual physical evidence, the sine qua non of science.

53 Responses to “Gambling on ET”

  1. Chris says:

    “the Richter scale for earthquake magnitude”

    My geologist senses are tingling. And twitching.

    Earthquakes are measured on the moment magnitude scale (for the most part), not on the Richter scale. The Richter scale was developed to give a value to earthquakes in Southern Calirfornia, within ~100 km of the epicentre between magnitudes 3 and 7. If you hear abuout the magnitude of an earthquake, it’s most likely on the MMS (unless you’re talking to seismologists about a particular event, and then there’s multiple ways to measure the energy released and recorded).

  2. Trimegistus says:

    The trouble with this scale is that it seems like an objective, quantifiable number, but it’s actually quite subjective. How do you rate the “alien-ness” of life? And the “reliability factor” is entirely subjective — who decides? Scientists think other scientists are reliable, but the history of science is full of crackpots and brilliant researchers who were incredibly gullible.

    I’d like to see a way to genuinely quantify this probability.

    • Agreed. The confidence in the sum of a bunch of wild guesses is pretty darn low.

      Additionally, I have some other complaints:
      – I convinced that they should be summed rather than multiplied.
      – These four factors are not ‘orthogonal’ in other words if a discovery is far from Earth it is much more likely to be done by remote sensing.
      – Finally, why does it matter how similar to us these aliens are? I mean, if we discovered Vulcans and Wookies wouldn’t that be more Biology shattering than finding pseudo-cellular biochemical entities?

    • In Wiki’s article on the Drake equation, a quote from Michael Crichton:

      The problem, of course, is that none of the terms can be known, and most cannot even be estimated. The only way to work the equation is to fill in with guesses. [...] As a result, the Drake equation can have any value from “billions and billions” to zero. An expression that can mean anything means nothing. Speaking precisely, the Drake equation is literally meaningless…

      I think the same applies here.

      The first parameter tells us nothing about the rareness or not; per Bad Boy, it’s not “sieving” at all. The second parameter as it relates to SETI sums up three original Drake parameters:
      fi = 0.01 (1% of which will be intelligent life)
      fc = 0.01 (1% of which will be able to communicate)
      L = 10,000 years (which will last 10,000 years.)

      The third and fourth parameters actually are good, in the sense they focus on the *search* and not the likelihood of existence.

      That said, they both should be set at 1. We’re “never” going to have a return mission to/from the nearest planetary star, let alone beyond that. And parameter 4 is barely a 1, even.

  3. “I would assess the recent claim of arsenic-based life in Mono Lake as (2+1+4+4)0.2 = 2.2, fairly low by comparison.”
    I’m not sure how you arrived at these figures, and am not sure I agree with them, either.

    The arsenic-tolerant life (arsenic-based is a bit of a stretch) found in the recent study is pretty far from alien. It’s Earth bacteria that evolved a survival strategy in a very harsh environment. I’d call that a 1 on the Life Form scale. And the nature of evidence wasn’t indirect markers, but actual living bacteria. That’d be a 5 or 6 on the Nature of Evidence scale.

    So by my calculations, (1+5+4+4)0.2 = 2.8 for Mono Lake. Which implies that it’s somehow more alien than your calculation. And is completely opposite of the point I was trying to make.


      • ArsenicGate – cute… misleading but cute.

        If only we could get the term “StupidGate” to catch on ;)

        IIRC: NASA didn’t trot this out as something well established (like say, Ponds and Fleishman did with Cold Fusion). It may have been over-hyped, but it is hard to tell how much of that was from NASA and how much was from the Media. (and some of it may have been due to some researchers who puffed it up to sound more important to a reporter).

        Cut NASA some slack. It’s hard to do astronomy research, launch satellites, and support human space flight with such a small budget – especially when they have an unfunded mandate to guard secrets about the Moon Landing Hoax, Alien space craft, and Sentinel Prime. ;)

      • That’s exactly the point of “NASA PR.” Budget worries lead federal agencies to this type of fluffery. Not misleading at all.

        And, a fair amount of the fluffery was either directly caused by NASA or else NASA’s fault.

        Anyway, back to fluffery and the federal budget:

        Another example? A Department of Energy lab director claiming the old tired claim that “fusion is just around the corner.”

        Scientists can have “follow the money” motives just like anybody else.

  4. Ok, let’s try another example. Tea Party Presidential Candidate Michele Bachmann:

    Life Form: 1 (grudgingly, but available evidence points to Earth-similar life)
    Nature of Evidence: 5 (1 point removed from the intelligent design proponent out of pure spite)
    Method of Discovery: 3 (I only have evidence from print and electronic media)
    Distance: 4 (Whether we like it or not)
    And for δ I’m going with 0.45, for the slim but still not-zero chance that she’s some sort of elaborate hoax generated by the major media outlets to test American gullibility.

    So Michele Bachmann’s Almár-Race score is 5.85, demonstrating that she’s more likely than not to be extraterrestrial. Which is exactly the result I expected.

    I’m gonna like this new toy. :)

  5. MadScientist says:

    Oh goody, yet another scheme for inventing numbers. Now the Richter Scale, Beaufort Scale, and so on actually have some meaning – but what do these ET numbers mean? Are they a genuine measure of anything? Can they even be used to make sensible decisions?

    • Max says:

      Are rating scales for mental disorders like depression a genuine measure of mental disorders?

      • It depends.
        What do you mean by ‘genuine’?
        What do you mean by “measure”?

      • Max says:

        Whatever MadScientist meant.

      • There’s lies, damn lies, statistics .. and what Twain never knew about … metadata, metastudies and similar!

      • MadScientist says:

        I don’t know – but what do those scales have to do with this “ET scale”? Shermer mentions some scales like Richter and Beaufort which do have some practical use and then lumps in this ET scale as if all ranking schemes were somehow automatically valid.

    • Paul Davies, in his book, The Eerie Silence: Renewing Our Search for Alien Intelligence, says that Drake wayyyy overestimated the probability of many of the Drake equation numbers, as well as many SETI-ans today not thinking outside the box enough. He goes on to wonder if hardcore SETI-ans aren’t quasi-religious.

      The book got a definite five-star rating from me.

      • I attended a talk once where it was said that physical science types tend to give substantially higher estimates for the biological terms in the Drake Equation. Life science types give estimates several orders of magnitude smaller.

        The terms are: f_l, f_i, f_c (fraction – or probability – of life developing on an inhabitable world, fraction of worlds with life evolving intelligent life, fraction of worlds with intelligent life which develop a ‘communicative civilization’). For some reason those who know something about life seem to think it’s much harder than those of us who know physics.

        I have tried finding a reference to this in vain.

      • MadScientist says:

        Aside from the requisite intelligence and development of technology such as radio (or light) communication, one frequently touted reason for ‘listening’ just doesn’t sit right with me: the argument of terrestrial radio broadcasts reaching into space. Communicating with the interplanetary probes, especially those out by the gas giants, is quite a task. Although TV and radio stations put out much more power than the radio telescopes used in the satellite communications, by the time those photons get to the nearest star system they will be well below the fundamental threshold of detection – and that’s assuming that those photons are not blocked at all. Unless we have some aliens already nearby (perhaps somewhere in the Kuiper belt or just beyond) and with fairly powerful radio transmitters at that, we can expect SETI to never detect a signal from intelligent aliens.

        Now let’s assume aliens had developed technology to turn entire stars into ‘stellagraphs’ – powerful omnidirectional transmitters which would broadcast to pretty much everyone within thousands of light years. If intelligence were encoded in a star’s light, the rate at which data can be received will depend on the telescope/imager’s exposure times, and even assuming an observatory like Hubble which has no atmospheric interference we’re looking at pretty low data rates. So unless we already have aliens in our vicinity watching us and attempting to communicate, I’d put SETI’s chances of finding an intelligent alien radio signal at 0. They’re better off using optical telescopes and hunting down intelligently modulated light sources (not that I think their chances of success will be significantly different from 0).

      • Mad, one related issue, re extragalactic civilizations.

        And that is communications being redshifted. We might be looking in the wrong spectrum area.

  6. DeLong says:

    The Math Skeptic should “leak” his assessment of Michele Bachmann to the nearest tabloid! The next headline you might see at the grocery store check-out lane may be something like “MICHELE BACHMANN AN ILLEGAL ALIEN!” Of course, the tabloid would have to confuse the “alien” from “not-of-this-planet” with the “alien” from Mexico. I’m sure they could find a terrible photo or two to hype the headline. Too bad the Murdoch tabloid chain is suffering from a separate scandel, they might have run with this story…

  7. John K. says:

    This really only strikes me as a variation on the Drake equation, but at least Drake limited himself to probabilities. This system seems to yield a more or less arbitrary number from 1 to 10, if I understand it correctly. So what does a 1 or a 10, or any value really mean? Why give “nature of evidence” more weight than “distance”. How exactly does one come to a life form factor of, say, 3? How are any of these factors objectively determined?

    Just because you plug in a bunch of subjective numbers that are more or less pulled out of thin air into an equation does not give the factors any greater credibility. Without a sample size or denominator, probability just becomes a way to sound more science-like than you really are. Throwing out percentages can make someone sound impressive, even if the numbers are completely made up. If someone says X% of people think something, then they better be able to point to an actual study with a sample size, otherwise it is all tricky language to give the illusion of more authority.

    Or in this case, if even one of these factors is just determined by the gut feel of the calculator, the Q number doesn’t have much meaning at all. The calculation then just becomes an attempt to lend credibility to an argument by blinding people with mathematics.

    • Pardon this repost of a comment I made elsewhere in your thread, but it fits perfectly.

      Paul Davies, in his book, The Eerie Silence: Renewing Our Search for Alien Intelligence, says that Drake wayyyy overestimated the probability of many of the Drake equation numbers, as well as many SETI-ans today not thinking outside the box enough. He goes on to wonder if hardcore SETI-ans aren’t quasi-religious.

      The book got a definite five-star rating from me.

    • The Drake equation wasn’t really intended to answer the question “How common are extra-terrestrial civilizations in our galaxy?” – it was intended to ‘organize ignorance’ so searchers could start asking better questions. Questions like “Where is the best place to look for (ET) life?”

      Really, the Drake Equation is a series of dependencies – each term depends on all the terms to the left (well, except the life-expectancy term). Another way to look at it is a sieve.

      • I understand that, Bad Boy; but then, why did Drake try to nail down specifics on the variables?

        To take up your analogy, it’s like everybody says it’s a sieve, but different people have different ideas of how fine or loose that sieve is.

        It’s like asking if we’re looking for minnows or sharks.

        That said, I agree with Davies. At the minimum, I highly doubt we will find carbon-based life in our galaxy.

      • MadScientist says:

        Drake started with “these are the things which I think are obvious and need to be considered” – that’s all perfectly normal for any hypothesis. The next step in science of course is to test the hypothesis – and in the ET case, with Drake’s equation in particular, this has proven elusive.

  8. Max says:

    Any metric can be converted into probability if you can validate it. Like if you come up with some formula that outputs a medical diagnosis as a function of biomarkers, you validate it by testing it on subjects who are known to have or not have the disease. That gives you the metric’s sensitivity and specificity.

    Of course the problem with SETI is that we don’t have any known ET life to validate the metric, but we don’t need to convert it into probability if we just use it as one way to rank or prioritize evidence.

    We can’t validate the Drake equation either, because we can’t test it on multiple universes, but at least it calculates a probability directly from probabilities.

    • Max says:

      Although we have no alien life to validate the metric, we have plenty of non-alien life to test for false alarms. Your score would be (1+6+5+4)*0.5=8.
      The Phoenix lights UFO should really have the same score, because the sighting itself was reliable, it just wasn’t alien life.

    • What if the metric is non-linear or discontinuous?
      What if the metric is categorical? (ie. you cannot say that one value is greater than any other value).

    • Of course the problem with SETI is that we don’t have any known ET life to validate the metric, but we don’t need to convert it into probability if we just use it as one way to rank or prioritize evidence.


      SETI has not found any evidence of ETI, true but does the metric of ETI’s need ‘validation’? Maybe I am naive, but I think that the natural coice of a metric for ETI’s is the count of those found (how you establish when a candidate is ‘found’ is another matter). Call me crazy but, I don’t think that you need to actually *find* anything to validate that counting the number you have found is a good measure of them. I don’t have to have any money in my bank account to know that keeping tabs on my account is a good idea.

      • Max says:

        In that sentence, “the metric” is the contrived London Scale. It must be validated to assign it a physical meaning.

        Obviously, if the metric is the very measurement you’re interested in, there’s no need to validate it. So if you want to know the number of ETIs, and your metric is the number of ETIs, it’s redundant to validate the metric against itself. Validation is needed to correlate a metric with something else, like blood pressure with heart attack risk.

    • Here’s Wiki on the Drake equation and its background:

      nsiderable disagreement on the values of most of these parameters exists, but the values used by Drake and his colleagues in 1961 were:

      R* = 10/year (10 stars formed per year, on the average over the life of the galaxy)
      fp = 0.5 (half of all stars formed will have planets)
      ne = 2 (stars with planets will have 2 planets capable of developing life)
      fl = 1 (100% of these planets will develop life)
      fi = 0.01 (1% of which will be intelligent life)
      fc = 0.01 (1% of which will be able to communicate)
      L = 10,000 years (which will last 10,000 years.

      The first two parameters’ estimates aren’t controversial to me. From what we’ve seen of early exoplanetary findings, the third could be moderately too high.

      From there, we really have shinola hitting the fan, IMO. (And I’m not alone; the Wiki article details alternative current values for these subsequent terms.)

      Fl at 100 percent? No way, Jose. Let’s put it at 10 percent instead. Fi at 1 percent? No way again. Even if “intelligent” is defined loosely, let’s take that down to 0.1 percent.

      And, there again, we have problems with the “sieve,” which modern evolutionary biology teaches us.

      What’s “intelligent” mean? Crows, ravens and other corvids are arguably intelligent, we now know. What’s “communicate”? An animal like a vervet monkey not only has different calls/alerts for different predators, members of a tribe can fake the calls to steal food.

      And “L”? Given exactly how we define “civilization,” our own 10,000 years may be just about up, if we start with the development of agriculture. If we start with, say, the steam engine, we have plenty of life left. This is just Drake spitting in the wind on this number.

      • Oh, per Wiki, Shermer himself once set “L” at just 420 years, not 10,000.

        And, Michael Crichton also is quoted on Wiki:

        The problem, of course, is that none of the terms can be known, and most cannot even be estimated. The only way to work the equation is to fill in with guesses. [...] As a result, the Drake equation can have any value from “billions and billions” to zero. An expression that can mean anything means nothing. Speaking precisely, the Drake equation is literally meaningless…

        A basically undefinable sieve isn’t much of one.

      • Max says:

        Useless doesn’t equal meaningless. Like, here’s a pretty useless equation whose meaning is clear.
        N = number of aliens in our galaxy + number of aliens outside our galaxy.

        The starting point for “L” is the ability to communicate with aliens.

      • Well, this gets back to what you mean by “meaning.” Tis true that L has a specifically defined framework, the estimate of the length of civilization. But, given that we’re clueless as to what that might be, because:
        1. We’re using “survey size = 1″;
        2. There’s a question as to how we define “civilization”;
        3. Per Shermer as cited on Wiki, there’s a question of whether we’re talking “Civilization” with a capital or individual “civilizations” …

        L is the most “content-free of the Drake variables, and IMO so content-free as to be useless.

        That said, if the unspoken presupposition of L is “communication with aliens,” Davies covers all the *problems* with that one, too.

        How do we know that aliens communicate the same way we do? Take the gold “CD” sent with Voyager. If aliens don’t “hear” at all … the music and other sounds on it mean not a damned thing to them.

        In other words, I think SETI is still too anthropocentric and so does Davies.

      • MadScientist says:

        “fi = 0.01 (1% of which will be intelligent life)”

        Hmm .. that’s not even true of Planet Earth. Maybe the intent was “1% of planets with life will develop intelligent life at some point in time”.

    • MadScientist says:

      The ranking itself is questionable – where does the weighting come from? For me, the ranking is absolutely useless – I wouldn’t waste time ranking a claim, I’d simply investigate and come up with conclusions/recommendations.

  9. WetRock says:

    A variable, subjective magic number multiplied by four subjective parameters. I think most gamblers would even reject this one.

  10. Daniel says:

    I am an atheist who believes there probably ETs. In 1977, my brother and I and another person who was not with us (we found out about her later), saw a sphere while we were driving at night. It was hovering in the sky, with three long antennae-like protrusions. The sphere was apparently electrically charged, and the sphere and antennas were all the same color, a mixture of red, white, and blue (and possibly green, I don’t remember for sure). We stopped, and after watching the sphere for about 30 seconds, the antennas fairly slowly retracted into the sphere at the same time, and when they were level with the surface of the sphere, there was a slight pause and the craft took off at “instant acceleration” and turned into a red streak at what looked like Star Trek’s “warp speed” until it left Earth’s atmosphere into space across the horizon. My brother and I don’t drink or take drugs (I don’t know about the woman), so we know it was not a hallucination. How could this be explained away (since three of us saw it) if there are no ETs visiting earth (assume I’m telling the truth–and I am)? By the way, the reason we found out about the woman witness is because she was being teased at work the next day for claiming she saw a UFO, but the co-workers shut up when my brother told them that he saw the UFO too.

  11. Jonathan Ray says:

    xkcd has a better version of this:

  12. PZ Myers has a good blog post on how many searchers for extraterrestrials, not limited to SETI-ans, make the anthropocentric mistake on discussing exobiology:

  13. Peter Spargo says:

    Why all these rather grumpy,petulant queries as to why people are attempting to devise scales such as this one? The answer is simply that, for all their manifold deficiencies, scales such as these concentrate our thoughts on the topic in hand, which almst invariably means that our thinking – even that of the Old Grumps! – is made just that bit crisper and sharper. Surely that is worth doing? Let’s be positive rather than negative . . .

  14. Ivan Almar says:

    Dear Mr Shermer,
    Just by chance I have found your very correct report on our London scale paper in Phil. Transactions. Thank you!
    I have two remarks:
    1/ Paul Davies proposed to look after a “shadow biosphere” and his group “discovered” something interesting at Mono Lake. I have discussed the problem with Paul and he agreed that since such cases (i.e. the discovery of something unusual among lifeforms on Earth) are by definition NOT extraterrestrial, the London Scale should not be used. (Nevertheless astrobiologists in Hungary agree that the “discovery” is probably not real.)
    2/ You use the London Scale for UFO sightings as well. Although I agree with your conclusion, but have to emphasize that the Rio Scale and not the London Scale is proposed to evaluate a putative discovery of ET technology or intelligence. The result would be the same, but we wanted to make a distinction between the detection of ET life and the discovery of traces of ET intelligence.

    Best regards Ivan Almar (

  15. Douglas Caines says:

    What validity and/or value do such mathematical equasions/ suppositions have outside of the planet from which they originate?

    If you truly believe that you can learn about the Universe simply thru Observation & Measurement, It is you who are gullible, and deluded.