Media have only scratched the surface in assessing their value to the world. To better understand and deliver that value, we need to improve how we measure the true impact of journalism. Analysis technologies have never been more abundant and accessible, and can almost do what we need, but better tools are needed.
These issues attracted me to the recent Brown Institute for Media Innovation and Tow Center for Digital Journalism event, "Research, Three Ways." The event brought together journalists, technologists and media experts to spark research on increased transparency, innovative uses of data in journalism and measuring impact.
Measuring impact occupies much of my attention in the Knight International Journalism Fellowships. Our Fellows run projects around the world to improve people’s lives through journalism. We track and communicate our program’s impact so that we understand what works and what we can improve, our supporters know the value of this work and the journalism community can benefit from our lessons learned. We are launching our new program – one focused on creating cultures of news innovation to improve the flow of quality information – which means I am immersed in the question, “How do we measure and communicate the impact of journalism innovation in a rapidly changing technological environment?”
Thanks to the event’s format, in which people created discussion groups by topics of interest, I engaged brilliant minds – such as Brian Abelson, the Open News Fellow and media measurement expert who just joined the New York Times – to take on impact measurement. Our task: identify key questions that often go unasked, and surface new ideas for tackling them.
We agreed it is still very difficult to assess the impact of a given story. What reporting did it trigger by other journalists and media? Did it change the course of reporting? Did it change human behavior or policy? How can we automate this kind of tracking? Can today’s technology do it? Is anyone already automating?
Certainly, there is tech-based impact measurement already happening in the form of Web analytics, social mentions and related metrics. There are dashboards to let us watch it all in one place. But these measure impact on attention. The model and technology are still very nascent for measuring impact on behavior and policy beyond the website, the TV program, the mobile app.
Google Alerts can track issue keywords on the Web. If This Then That can feed alerts – and various other mention trackers, like Twitter – to Google Spreadsheets. Fusion Tables can turn this data into visualizations to create makeshift aggregation dashboards. With code, one can set up a one-stop shop for tracking and monitoring. Unfortunately, relatively few journalists know code, and few media can afford newsroom developers. And a lot of code is needed to create the sophisticated, user-friendly, open source tools we need specifically for measuring journalism impact.
Measurement can be done manually. For instance, through stakeholder interviews and surveys. Or Web searches to see what stories emerge, and visually connecting them by hand (think: posting links in order on a Web page or Storify). But these are expensive, time-consuming and difficult to scale.
Automating impact measurement, therefore, would be powerful.
Imagine it. Journalists and media showcasing their impact, building trust in audiences, as they see clearly who has the best information available and who makes a difference, rather than having to rely merely on belief or reputation. In turn, this could push journalists and media to think and measure more comprehensively, backed by data, about their value to the world.
Of course, important counter questions were raised. Do we lose something important if stories are chosen only if they target impact? Do we risk logical fallacy and misrepresenting the whole story if we assume journalism is the sole catalyst of a given impact, when there are typically larger governmental or societal forces at play? Is it really the role of a journalist to always have an impact? If not, does that raise questions about support from foundations, which increasingly pay for impact through journalism?
We ultimately agreed that more needed to be done by journalists and media to make a clearer case for their value, and that better automated tools would greatly simplify measurement of impact beyond news products. Doing so would build trust, which would make news media more competitive in the fight for audience and revenues. It may even make the difference between viability and extinction.
Ben Colmery is the deputy director of the Knight International Journalism Fellowships at the International Center for Journalists.
Image CC-licensed on Flickr via Michael Gallagher.