Science Journalism
I recently got into a small kerfuffle with a journalist, actually a sports writer who decided to dabble in science journalism. The exchange started at science-based medicine when I wrote a piece critical of the claims being made for a new device called the GyroStim, which is being offered as a treatment for brain injury.
In this article I linked to a piece in the popular press about the treatment, in the Denver Post by a sports writer, Adrian Dater. Dater thought I was being unfair in my criticism of his piece, and so wrote a response on his blog. The exchange and the comments have exposed many of the problems with journalism in general and science journalism in particular, that I would like to explore further here.
First I have to say that there are many excellent journalists and science journalists out there. I am not implying that that there are no good journalists. I do find, however, that the baseline quality of science journalism is lacking and, if anything, getting worse. Part of the problem is the evaporating infrastructure for full-time journalists. Many outlets no longer maintain specialist journalists, and use generalists (including editors) to cover science news stories.
What follows can be seen as a quick primer, or at least a list of helpful suggestions, to journalists who wish to cover science topics. I will primarily use examples from the recent exchange over the GyroStim.
Think About the Narrative
Almost all news stories have a clear narrative. The facts of the story are presented in a way to create a meaningful story. Even if all the facts are individually correct, the choice of which facts to present, in what balance, and in which order affect the bottom line impression left by the article. The choice of headline is also important, and I know for big news outlets the article author is often not the headline writer, but that doesn’t mean the headlines don’t matter also – they tend to frame the article.
For example, Dater defended his article by claiming that he got all the facts right, that he included “balance” (more on that below) by indicating the Gyrostim is not FDA approved and quoting a doctor saying more evidence is needed, and that he did not directly endorse the treatment.
My criticism, however, was based on the narrative that he blatantly created. I find it interesting that he seems to be unaware of this narrative or its effects.
The story was framed as a touching account of a father who is an engineer who decided to build a machine to cure his daughter of cerebral palsy. Right out of the gate the reader is rooting for this machine to work. The pull-out quotes include, “Spinning stimulates the brain,” and “Miracles almost every day.” He also ended with a hopeful anecdote:
“The machine is amazing, it really is,” said Hishon, who played nine games for the Lake Erie Monsters this season after nearly two years of concussion symptoms. “It just seemed to wake something up in my brain. I can’t explain it, but it definitely worked wonders with me.”
The fact that there is some token skepticism tucked in the middle is just part of the narrative – every story needs a villain, right? Dater may not have intended the skeptics to be the villain of his narrative, but that is the role he assigned them. On the one hand you have a loving father, hope, the device is being researched, you have excited practitioners, and many patients who are thrilled with their results, and on the other side some talking-head canned skepticism – “more evidence is needed, not FDA approved, blah, blah.”
It seems to me that many journalists don’t even think about the narrative – it just emerges as a default story format. Start with a human interest angle to draw in the reader, then just report what both sides are saying, be sue to include plenty of anecdotes, and then end on a hopeful note.
What many non-science journalists don’t seem to get is that this is not a proper narrative for a science story. Further – all journalists need to decide what their narrative is before writing the story (although hopefully after they have researched it – often journalists decide on their narrative first then just backfill the facts and anecdotes).
Here are some other narratives that journalists covering science stories might consider:
– The allure and harm of false hope, and the exploitation of false hope by dubious practitioners and companies.
– A cautionary tale about getting excited prematurely by some newfangled treatment before it is adequately tested, given that most new treatments do not pan out.
– Is new and high tech always better – does this machine that costs tens of thousands of dollars work better than a $20 device that has a similar function?
Put the Story Into Context
The most challenging part of science reporting is putting a new story into a deeper scientific context. This requires background research and talking to a variety of experts – and asking the right questions and really listening to what they say. This context includes:
– What is the plausibility of the new claim? Does it confirm or contradict what is currently believed to be true?
– Does the new device, treatment, product resemble anything that has come before? Is it truly new, or just a rebranding of an old concept – and if the latter, how have previous incarnations fared?
– What is the current consensus, if any, on this new claim? Is it truly controversial, or very one-sided with the majority of scientists taking one position and only a few outliers disagreeing with the consensus?
– What are the credentials and backgrounds of the experts on which you are relying. Is their degree generally recognized as valid? Do they have a history of making other dubious or controversial claims? Do they have a history of fraud?
– Overall, how does the new discover, claim, treatment, etc. fit into existing evidence and scientific theories?
– What are the implications of all this for the current stance one should have toward the claim – should it be considered experimental, should it be taught in public science classes, should it be legal, etc. ?
– What steps are needed in the future? What questions need to be resolved?
Adding the above context is exactly what we do at Science-Based Medicine, and at many other “skeptical” blogs. Good science journalists also do this. This is the real story, not the fluff narrative that is better suited to covering the local dog show.
False Balance and Token Skepticism
The balance of the article should generally reflect that balance of scientific acceptance. If 95% of the scientific community accepts one consensus, then that is what the bulk of the article should reflect. If you feel the other 5% deserves a mention, then it should be given appropriate space, and also put into context (as above).
Stories about politics and social issues requires obsessive balance, because these are mostly based on value-judgments and opinions. For these stories a journalist needs to get the facts right, and make sure that all credible sides have their say.
Science does not work like that. In science, some opinions are objectively better than others. Science stories are about the evidence and the process of science – about finding the best current answer. Science articles need to reflect that.
As soon as you put a pseudoscientist up against a genuine and respected scientist, you have elevated the pseudoscientist to a stature they likely do not deserve. You have framed the story in a very deceptive way that does not reflect the reality.
Bad science journalism generally falls into one of three categories in this regard. Some stories have false balance, where a pseudocontroversy is presented as if it is a real scientific controversy. This is the false-balance fallacy.
Other stories have what we call token skepticism – most of the article dedicated to giving a forum to the crank and glowing anecdotes, with scant mention of doubt and/or quick commentary by a real scientist. Dater’s article fell into this category.
The third is when even token skepticism is lacking – the story is presented without a hint of skepticism or actual investigation.
Conclusion
Good science journalism requires putting a science news story into a proper context, making sure the narrative that emerges is fair and appropriate to the actual story, and properly balancing different points of view to the scientific consensus and to scientific legitimacy.
This is not easy. It requires, in my opinion, at least a baseline of scientific literacy. It also requires significant background research into the topic, and into any experts upon which the journalist relies.
What we have from Adrian Dater is an excellent example of what happens when a non-science journalist thinks they can dabble in science reporting, without understanding any of the special requirements of competent science reporting. Even more telling that the article itself is Dater’s defense of his journalism. As is often the case with defensive overreaction, he just dug himself in deeper and deeper.
The exchange also highlights for me the new role that science blogs are playing in the reporting of science news. Journalists who write bad science news stories now have to contend with the second wave of science blog analysis. Now actual scientists, or at least dedicated science journalists, can add the missing context, deconstruct a misleading narrative, and rebalance a science news story.
Journalists, like Dater, who encounter this science-blog pushback when they write a naive and misleading piece would be better off if they embrace the criticism and try to learn from it, rather than get into an online fight with someone who actually knows what they are talking about.
You should be as skeptical of AGW claims.
I hope some journalism professor assigns this post as required reading for any aspiring reporter.
It seems, to me, that so many reporters begin with the spin–they know the effect that they want and craft the story around that, as any good fiction author should. The facts are secondary. From the description, this was not an attempt at science journalism–it was a feature-reporter bit, a feel-good piece, far easier than actual reporting. Just imagine the confusion that would have arisen from inclusion of concepts like placebo effect, confirmation bias, etc.
In the mid 90s, Mickey Mantle was waiting on a liver transplant. He ended up getting one, from an accident victim that had made arrangements to donate multiple organs should the situation arise.
The doctor who performed the operation gave a press conference, and took questions. One of the reporters commented on how giving many organs was such a generous gift. He then asked the doctor “May we speak to the donor?”
The doctor just looked at the man and said “Are you a sports reporter, sir?”
I don’t see this particular article as that egregious. At bottom, it’s a sports section piece, particularly hockey — which is what Dater writes about primarily — where concussions have sidelined some really good players. Evidently the biggest star in the NHL used the machine and is now playing again. I follow hockey — although Crosby is a giant baby — so it’s an interesting story that there’s this machine that relatively sophisticated business operations are using.
I read the whole article, and it was clear to me anyway that the treatment was unproven, and that most of the medical profession has its doubts. Plus the machine is pretty expensive to begin with, so you’ll probably go through some kind of gate keeper that knows what he’s talking about before you use it to the exclusion of other potentially useful treatments. It’s not quite the same thing as Steve Jobs foregoing real medicine to treat his pancreatic cancer.
I would have thought Mr. Jobs was a bit too intelligent to fall for snake oil, just goes to show. One born every minute.
Jobs was heavily into woo from the get go. The name Apple came from his early interest in “healthy eating”. I use quotes because there is so much pseudoscience attached to nutrition, not because I think an apple is not a healthy food. Jobs was into far more than the idea of “an apple a day…”.
To be accurate (and resist temptation), David Gorski did a post at SBM on Jobs after his death which concludes that we can’t be sure Jobs would have been saved even if he had followed standard treatment–because of the nature of his particular (and rare) cancer. I’m sure it’s in the archive.
Steve, you’re wasting your words. I’ve checked some of the stuff this Adrian Dater wrote and he’s perfectly well aware of everything you mentioned, *especially* the narrative. That’s what the guy starts with, and he picks this one because he thinks it’s the one that’ll get the most eyeballs.
It’s important to remember: all news stories are like this. Whenever you know something about the facts behind a news story, it’s always shocking how wrong the published report is. That’s true for all stories. The ones that seem accurate are the ones you don’t have any other information about.
I was unable to read the story referenced in the post, but I wonder if Mr. Dater wrote it himself or just polished up a press release from whoever is selling this new miracle treatment. A shocking proportion of science and technology stories are just press releases. (I worked at a paper when the American Chemical Society held its meeting in my city — all the news stories about the meeting were straight from the “press packet.”)
The original article in question, written by Adrian Dater, does not seem out of line to me for two reasons.
The first reason is that I don’t feel like this narrative is as one sided as Dr. Novella suggests. In fact, there were so many disclaimers in the first half of the article that I thought it must have been rewritten since the criticism. In it’s current form, I do not feel the narrative is deceptive.
The second reason is the audience: The Denver Post is not New Scientist or Science Daily. If it had appeared on a more science-focused venue, I would expect a more clinical treatment of the facts. If I read this in /r/science I would be upset that someone even posted it. But this is the Denver Post. I think the type of article that Dr Novella envisions may not be entertaining enough to their readership to make it in the paper.
That said, this post by Dr. Novella on science journalism is amazing, and I am very grateful for the free education. I have bookmarked it and may try to use it in my school’s cross-subject curriculum.
I think if I were a journalist and a Professor of Clinical Neurology at Yale University went out of his way to advise caution on an article I’d written, I’d get an interview with said Professor of Clinical Neurology at Yale University, write another article and milk the story for all its worth! I’d then go on to foster that relationship so I could have a credible, expert opinion on tap.
The very last thing I’d do is get into an acrimonious comment war on the internet.
Perhaps I should go into journalism. My tradecraft seems about a million times better than some folks’ we could mention.
The poor writing skills of Mr. Dater in the blog comments in question convince me that he isn’t much of a writer to begin with and supports the contention of Trimegistus above. Mr. Dater reacted like a little child wrongly accused of doing something his little brother actually did. He refused to accept the slightest criticism and just dug himself deeper into a hole of exhibiting his ignorance.
Thanks for this article. It took a while to check on the links etc, but it was very entertaining.