Skepticblog

Most Americans have used the internet to look for health information (a recent survey reports that 59% of adults have searched for health information on the net). Yet there are serious concerns about the accuracy and reliability of that information. There have therefore been many studies looking at the accuracy of health information, and not surprisingly the results are concerning.

Most of these studies pick a specific topic and then have one or more experts on that topic review websites obtained through specific search terms. For example, a British study looking at the treatment of fever in children concluded:

Only a few web sites provided complete and accurate information for this common and widely discussed condition. This suggests an urgent need to check public oriented healthcare information on the internet for accuracy, completeness, and consistency.

Only 4 of 41 websites they reviewed gave complete and accurate information. Another study looking at information for carpal tunnel syndrome found:

Thirty-three percent of the sites sold commercial products for the evaluation or treatment of carpal tunnel syndrome. An additional 30 percent were commercial web sites that did not sell products. Only 23 percent of the sites were authored by a physician or an academic organization. Fewer than half of the sites offered conventional information. Twenty-three percent of the sites offered unconventional or misleading information. The mean informational value of the web sites was 28.4 of a possible 100 points.

There has also been a systematic review of studies that review health information on the internet. This review found:

Most frequently used quality criteria used include accuracy, completeness, readability, design, disclosures, and references provided. Fifty-five studies (70%) concluded that quality is a problem on the Web, 17 (22%) remained neutral, and 7 studies (9%) came to a positive conclusion. Positive studies scored significantly lower in search (P =.02) and evaluation (P =.04) methods.

Therefore most studies concluded that the quality of health information on the internet is poor, and those that had a positive evaluation were less rigorous. From existing research it seems fairly clear that the quality of health information on the internet is variable and generally poor. This is not surprising given the “wild west” nature of the internet. Anyone can buy a domain name, set up a reasonably impressive website, and then start spewing whatever nonsense they wish. Other sites may intend to provide accurate information, but it takes a lot of work to provide accurate and up to date health information, so therefore high volume sites, especially those that are monetized, likely take many shortcuts that compromise quality.

There are many efforts to improve the quality of health information on the net, or at least provide consumers a way to judge the likely accuracy of the information they find. Several studies have looked at predictors of accuracy of information. A recent study of information about infant sleeping, for example, found:

Government and organizational websites had the highest percentage of accurate information (80.1% and 72.5%, respectively).  Blogs, retail product reviews, and individuals’ websites had the highest percentage of inaccurate information regarding infant sleep safety (30.9%, 36.2%, and 45.5%, respectively).  News websites were accurate only one-half of the time.

This is in line with previous studies that gave similar results. The most reliable sites are institutional and education sites (whether government, university, or patient-advocacy). These are sites that likely have the resources and access to dedicated professionals to ensure high quality information. They also have another thing in common – their professional reputation is their most treasured asset, and so they care about their public face on the internet. Having wrong or misleading information branded by their institution for the world to see is something to be avoided at all costs.

Individual websites are highly variable – they run the full gamut from an individual professional blogger (journalist, scientist, or other expert) to cranks or misguided crusaders of all stripes. Even the best individual sites, however, still represent the knowledge and opinions of an individual. This by necessity (and yes, of course this includes my own humble blog) is limited to an individual’s perspective and biases. It’s always a good idea, therefore, to balance that with the opinions of others, especially institutions that can represent a consensus of expert opinion.

Commercial sites generally should be looked at with skepticism. The bottom line is that they are trying to sell you something, so their motivation is very different from an academic institution. They may be directly trying to sell you a product, and all of the information they are providing is essentially a commercial (perhaps disguised deliberately as news or an information resource). Or they may simply be driving traffic to their website, and providing provocative opinions and information is a good way to do that.

Another study also looked at indicators of accuracy and found slightly different results:

Three indicators correlated with accuracy: displaying the HONcode logo, having an organization domain, and displaying a copyright. Many proposed indicators taken from published guidelines did not correlate with accuracy (e.g., the author being identified and the author having medical credentials) or inaccuracy (e.g., lack of currency and advertising).

The HON code stands for “Health on the Net” and is a seal of approval, meaning that the HON organization has looked at the material and found it to be reliable. It makes sense that this should be a predictor, because this review can weed out blatantly inaccurate, pseudoscientific, or commercial sites. Displaying a copyright is interesting – I wonder if that is just a marker for having institutional resources.

It is interesting but not surprising that having a medical degree did not predict accuracy. In my experience many crank or commercial websites use degrees and credential as a mark of legitimacy – it’s a marketing strategy. It apparently is very easy to find someone with an MD or PhD to endorse your product, and often times they are part owners of the company. Also, having a degree is no guarantee that one is not a crank.

Also interesting is the fact that having advertising on the website is not a negative predictor. At times I have run Google ads on this site, for example, and I often get comments from people that use this fact as an indicator that the site is somehow less legitimate. Placing a strip of ads on a website with decent traffic, however, is an easy way to support the site. In my case it generates enough money to pay for bandwidth, and that is about it, but it’s nice for a site to at least pay for itself.

Having generic ads on a site to cover costs, however, is different from a highly monetized or commercial site – making millions off of selling products, and gearing their editorial policy toward maximizing sales.

I also found a number of studies looking at the process of evaluating websites itself. The internet (despite the fact that many of us take it for granted now) is a relatively new phenomenon in our culture. Methods for evaluating information on the net are still evolving. One study looked at inter-rater reliability among the experts that are evaluating the information for accuracy. They found:

The medical experts showed a low agreement when rating the postings from the newsgroup. Hence, it is important to test inter-rater reliability in research assessing the accuracy and quality of health-related information on the Internet. A discussion of the different measures of agreement that could be used reveals that the choice of statistic can be problematic. It is therefore important to consider the assumptions underlying a measure of reliability before using it. Often, more than one measure will be needed for “triangulation” purposes.

In other words, when more than one expert is used to evaluate information, they often do not agree with each other about the accuracy of the information. Further, there are various statistical methods that can be used to measure how much they agree, and these produce different results. The authors therefore conclude that multiple experts should be used, and multiple statistical methods should be used to measure their agreement.

The same study also found that research that uses a single expert to evaluate health information may be unreliable itself. The knowledge and opinions of an individual, even an expert, are quirky.  In general it is more reliable to use a consensus of multiple experts than any individual.

Conclusion

We are still living in the wild west of the internet. The net is becoming a critical resource, and health information is near the top of the list. It is difficult, however, to create and maintain accurate resources of health information. The topics are complex and the information is constantly changing. In my experience the more controversial the topic also the more misinformation there is out there clogging up Google searches.

There is no single guarantee of information accuracy. The features I discussed above, however, are a useful guide, at least to a first approximation of which information is likely to be accurate. There are some additional rules you can follow also.

– Don’t rely on any one site. If the information is important, verify it on multiple reliable sites.

– Always seek out the counter opinion for anything that seems controversial. Until you have heard both sides (or multiple sides) to a controversy, do not think you understand it or which side is likely to be correct.

– Don’t be afraid to get help from those with more knowledge and expertise than you (including your personal doctor). Even when information on the net is accurate it may be difficult for the non-expert to understand and put into proper context. I see patients every day with information they got from the internet which is not inaccurate, they are just unable to put the information into a proper context and apply it properly to their situation. In many cases I see patients or family members very confused by accurate information, but they don’t know how to plug that information into the specific situation they are dealing with.

And – as always – be skeptical.