The question of how to measure the impact of media coverage has been around for more than 100 years.
In the 1920 and 1930s, scholars worried about the persuasive power of mass media and propaganda. As data collection and research methodologies became more rigorous in the second half of the century, scholarly views on media effects grew more nuanced. For example, researchers found that even when media does not directly convince the public what to think, it can be highly influential in what the public thinks about, an influence known as “agenda setting.”
Media messages may also influence public opinion by priming citizens to evaluate political candidates along certain dimensions and framing stories within a certain field of meaning (e.g. discussing budget deficits in the context of corruption).
Finally, scholars argued that the media may influence local elites such as civil society and religious leaders who in turn influence members of their community.
Today, media impact is not just an academic preoccupation. Journalists and media organizations around the world invest in documenting and quantifying the impact of their work. These efforts are driven in part by a desire for rigorous self-evaluation, and in part by the need to justify their work to donors, investors and the general public.
Drawing on interviews with a broad range of scholars and media practitioners, and a survey of more than 50 years of research on media effects, our team of five researchers, spanning three academic disciplines, gathered decades of research on media effects to develop a taxonomy of media impact that can be used by media practitioners and researchers. The results are presented in our recent paper, “Understanding journalism impact: A multi-dimensional taxonomy for professional, organization, and societal change.”
We hope the taxonomy will help organize discussions between journalists, editors, donors and scholars about why media programs matter, and how to quantify their effects.
Our analysis yielded the following primary findings:
(1) The most fundamental channel of media influence is that of providing information to the public.
In the past 20 years, social scientists have used a range of methods to document the power of journalism and mass media to affect public knowledge, attitudes, perceived norms, and political behaviors like voting and protest participations. The effect of media and journalism on public opinion varies by program and context, but it remains the most fundamental measure of media impact.
(2) But effective journalism programs have impacts beyond public opinion.
We identified a broad set of media impacts that extend beyond providing information to the public. For example, media programs may catalyze and influence interest group advocacy, build social networks between media professionals, social movements, and insiders. Journalists everywhere are not just agenda-setters for society, but have impact when they work with other groups trying to bring about change.
(3) There are limits to cost-benefit analyses of journalism.
Democracy’s Detectives, written by economist James T. Hamilton, applies the ideas of the classic Journalism of Outrage in his analysis of the career of Pat Stith – a Pulitzer-prize winning reporter from North Carolina who carried out more than 300 investigations over 36 years. Of the 300 investigations, 149 generated substantive changes, 110 produced deliberative outcomes, and 43 generated individual impacts. Hamilton also tried to assess the cost-benefit of some of the stories.
Even so, it’s not really possible to do a broad cost-benefit impact of journalism for a whole lot of reasons including the fact that costs vary widely (think of the salary gap between a reporter in a low-income country and a reporter in a high-income country), and that in general it’s impossible to put a monetary value on a public good such as accountability or truth that helps all of society.
(4) The impact of journalism can be divided into three phases.
The 1991 book The Journalism of Outrage is a key text. This book, which looks at U.S. journalism after Watergate, provides case studies of investigative reporting from around the U.S. The authors find that the impact of journalism unfolds in three phases, beginning with effects on individual knowledge and attitudes, transitioning into public discussion and deliberation, and culminating in substantive policy or material change.
(5) Using online tools to measure engagement/reach is only part of the story.
Many media organizations measure impact by scraping social media data reactions, doing surveys and charting reach and engagement. But there are other options available. Academics have developed a broad range of new tools for measuring impact, including natural experiments that compare outcomes inside and outside newspaper media markets and radio/television transmission ranges and collaborative experiments that work with journalists to vary targets of their reporting and track impact.
(6) Cross-border investigative reporting educates the journalists who do it.
Impact can include the effects that working on a story, or series of stories, has on journalists and the profession. We are using our matrix to research how cross-border outlets, such as the International Consortium of Investigative Journalists (ICIJ), have not just changed the world, but also the community of journalists who work on those projects. We’re starting to measure the effect of cross-border investigations on the reporters and networks of reporters who carry them out.
An early notable finding, from our survey of reporters working with ICIJ, is that the majority said they learned new skills and developed their understanding of the subjects they covered. New skills learned include working with large datasets, data scraping techniques, deciphering offshore financial account operations, and the handling of encrypted communication.
(7) Impact can include backlash against journalists.
Unfortunately, one impact of investigative reporting is when it creates a backlash against journalists. Shooting the messenger is all too common, and a powerful set of stories that has impact can lead to negative consequences for journalists, such as arrests, prosecutions and violence.
We’ve developed a set of surveys that can be used to measure impact and Lindsay Green-Barber’s Impact Architects provides a one-hour free consultation. Please contact us if you want to know more.
Schiffrin and Groves wrote the paper with SIPA adjunct associate professor André Corrêa d’Almeida, Columbia University PhD student Adelina Yankova and Lindsay Green-Barber, the co-founder of Impact Architects.
Photo by Markus Winkler on Unsplash.