The trope of the journalist writing in all caps and misrepresenting, exaggerating scientific findings is already well established. But while examples like these do exist (and they should be taxed), journalists in general are pretty careful when communicating scientific findings — says exactly a scientific study.

In science, absolute certainty is often hard to achieve. Uncertainty is a factor of the process and doesn’t mean that a theory is wrong. Scientists have developed specialized, often murky ways to discuss scientific uncertainty, but outside the scientific community, these methods and the associated terminology can be confusing and lead to incorrect conclusions.

This is especially true for science journalists, who have to report regularly on scientific research and can sometimes ignore or fail to understand scientific uncertainty. However, this is the exception rather than the rule, according to a new study. Researchers found that journalists are overall careful when communicating science.

“I feel like when we talk about the potential of journalists exaggerating claims, it’s always these extreme cases,” David Jurgens, assistant professor at the University of Michigan, said in a statement. “We wanted to see if there was a difference when we lined up what the scientist said and what the journalist said for the same paper.”

Understanding science communication

Science journalists have an important role in shaping public understanding of science as they broker information between the scientific community and the general public. They are intermediaries between scientists and the general public, and their judgment and expertise allow the public to engage with new scientific data. The way they frame new scientific findings influences public opinion — because let’s face it, when’s the last time you read an actual scientific study?

Many studies have looked at journalism amid scientific uncertainty over the years. Last year, a group of US researchers looked at how journalists communicated preprints in the early moments of the Covid-19 pandemic, finding that half of the stories analyzed from digital media outlets contained framing devices emphasizing uncertainty.  

In a new study, David Jurgens and Jiaxin Pai from the University of Michigan looked at how scientific uncertainty is being communicated by journalists and whether scientific claims are being exaggerated or not. They also wanted to explore how science claims in the news changed between well-respected and less rigorous publications

“Our findings suggest that journalists are actually pretty careful when reporting science,” Pei said in a statement, highlighting the skills needed to translate science to a general audience. “Journalists have a hard job. It’s nice to see that they really are trying to contextualize and temper scientific conclusions within the broader space.”

The researchers gathered news data from Altmetrics, a company that follows mentions of scientific papers in news stories. They collected about 129,000 articles that mentioned scientific papers. In the stories, they analyzed sentences that included words such as “conclude” to see how journalists were stating the claims of the paper.

They established certainty levels in more than 1,500 scientific discoveries. Jurgens and Pai then built a computer model to see if they could replicate the certainty levels that readers pointed out. The model was very correlated with human assessments of how certain a claim was. The model is not perfect, but it’s good enough to get the idea of what’s going on. Overall, journalists were staying true to the level of certainty presented to the study, the researchers note.

However, Pei said the work of the journalists can sometimes get trickier when looking at the quality of the journal. News writers tend to report the same levels of certainty no matter where the study was published, he argued. This can be problematic for the audience, Pei said as the journal impact factor is an indicator of research quality (though this is also a matter of debate).

The researchers believe their work is a big step forward in understanding and quantifying how uncertainty is communicated in scientific news. To further help reporters and scientists, they created software that helps to calculate the uncertainty in research and reporting. This could also benefit the general audience, they argue, providing a “calming effect” to some degree.

The study was published in the Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.