Recent research: When fact checkers agree

FCP logoWhat does it mean when fact-checking journalists review campaign claims and come to the same conclusions?

Michelle Amazeen, an assistant professor of advertising at Rider University in New Jersey and one of API’s fact-checking researchers, takes a look at fact-checking agreement in a recently published study.

Amazeen revisits a study by two University of Miami scholars whose criticisms of journalistic fact-checking included what they deemed media’s “naive” methodology and a failure to adhere to methods used by the scientific community.

But journalism itself is not scientific, Amazeen says. “Does that mean we should do away with news reporting, too?”

“What the study suggests is that we should not be blinded by the criticism that just because fact-checking is not scientific, it holds no societal value,” says Amazeen.

The paper cites examples of clear-cut deceptions that fact-checking has exposed, Amazeen notes.  “And when not one, not two, but three different fact-checkers come to the same conclusion about the accuracy of a claim using different evaluation methods, the general public should have more confidence that there’s cause for concern,” she says.


Fact-checker agreement during U.S. presidential campaigns

How often the primary fact-checkers in media reach similar conclusions

2008 2008 2008 2012 2012 2012
Fact checkers No. of ads No. of claims Agreement (%) No. of ads No. of claims Agreement (%), Fact Checker, PolitiFact 2 6 100 13 17 100, Fact Checker 5 9 100 12 49 98
Fact Checker, 1 2 100 8 19 95, 18 36 98 6 12 100
TOTAL 26 53 39 97

Data Source: Dr. Michelle Amazeen, Rider University


Beyond agreement on evaluations, finding statements that multiple fact-checkers agree are verifiable is also noteworthy. Out of all of the statements in a political ad, most are not factually verifiable. For instance, “We can restore America’s leadership in the world” is not something that fact-checkers can check, Amazeen notes.

“Thus, these leading fact-checkers have to be disciplined in carefully selecting the types of claims to check,” she says. “It’s a task that’s fraught with peril.”

Amazeen acknowledges that the authors “do offer some fair criticisms” of current fact-checking practices in U.S. journalism. For instance, she says, fact checkers would benefit from adopting a more systematic and transparent method of claim selection. However, she adds that “as long as unambiguous practices of deception continue, fact-checking has an important role to play in the United States and around the world.”

You can read the full text of Amazeen’s study, “Revisiting the Epistemology of Fact-Checking,” here.

Read more about political fact-checking and accountability research to be released by the American Press Institute this spring.