What does it mean when fact-checking journalists review campaign claims and come to the same conclusions?

Michelle Amazeen, an assistant professor of advertising at Rider University in New Jersey and one of API’s fact-checking researchers, takes a look at fact-checking agreement in a recently published study.

Amazeen revisits a study by two University of Miami scholars whose criticisms of journalistic fact-checking included what they deemed media’s “naive” methodology and a failure to adhere to methods used by the scientific community.

But journalism itself is not scientific, Amazeen says. “Does that mean we should do away with news reporting, too?”

“What the study suggests is that we should not be blinded by the criticism that just because fact-checking is not scientific, it holds no societal value,” says Amazeen.

The paper cites examples of clear-cut deceptions that fact-checking has exposed, Amazeen notes.  “And when not one, not two, but three different fact-checkers come to the same conclusion about the accuracy of a claim using different evaluation methods, the general public should have more confidence that there’s cause for concern,” she says.

 

Fact-checker agreement during U.S. presidential campaigns

How often the primary fact-checkers in media reach similar conclusions

	2008	2008	2008	2012	2012	2012
Fact checkers	No. of ads	No. of claims	Agreement (%)	No. of ads	No. of claims	Agreement (%)
FactCheck.org, Fact Checker, PolitiFact	2	6	100	13	17	100
FactCheck.org, Fact Checker	5	9	100	12	49	98
Fact Checker, PolitiFact.com	1	2	100	8	19	95
FactCheck.org, PolitiFact.com	18	36	98	6	12	100
TOTAL	26	53		39	97

Data Source: Dr. Michelle Amazeen, Rider University

AMERICAN PRESS INSTITUTE

Beyond agreement on evaluations, finding statements that multiple fact-checkers agree are verifiable is also noteworthy. Out of all of the statements in a political ad, most are not factually verifiable. For instance, “We can restore America’s leadership in the world” is not something that fact-checkers can check, Amazeen notes.

“Thus, these leading fact-checkers have to be disciplined in carefully selecting the types of claims to check,” she says. “It’s a task that’s fraught with peril.”

Amazeen acknowledges that the authors “do offer some fair criticisms” of current fact-checking practices in U.S. journalism. For instance, she says, fact checkers would benefit from adopting a more systematic and transparent method of claim selection. However, she adds that “as long as unambiguous practices of deception continue, fact-checking has an important role to play in the United States and around the world.”

You can read the full text of Amazeen’s study, “Revisiting the Epistemology of Fact-Checking,” here.

Read more about political fact-checking and accountability research to be released by the American Press Institute this spring.