Use algorithms, not interns, for newsroom grunt work

by Margaret Looney
Jun 13, 2013 in Data Journalism

It's a common sight in newsrooms that practice data-driven journalism: interns with dark circles under their eyes and a glazed look that comes from spending hours combing through each entry in a vast database, trying to standardize it so it can be analyzed for patterns.

Derek Willis, interactive news developer for The New York Times, believes there is a better way. "Interns are great, but this is a way to kill them," he told attendees at the Online News Association (ONA) June meetup in Washington, DC. Instead, he and his fellow presenter suggest saving time by letting a computer algorithm deal with this grunt work.

Essentially, an algorithm is just a “glorified how-to list,” said Justin Myers, interactive producer with The Chronicle of Higher Education. Likened to a recipe, it’s a set of instructions your computer will repeatedly follow at your behest.

Algorithms offer journalists value that goes far beyond time-saving benefits. Reporting methods are essentially the same across beats, following a typical ask-a-question-then-receive-an-answer system. A major advantage of teaching a computer to do these tasks is that it’s creating a replicable system that can continue long after a reporter decides to move on to another job, Willis said.

The Times uses algorithms for stories such as its campaign finance tracker, to make simple comparisons of political campaign donors based on name, recipients, occupation and other demographics. There is a motivation behind any political campaign donation, and they typically follow a pattern that an algorithm can identify.

Once you've used an algorithm to eliminate manual tasks and you start collecting data on an ongoing basis, you will be able to ask better questions based on the trends the computer spotted for you.

Algorithms can unearth stories the human eye might have missed. Willis noted how journalists can use algorithms to monitor the routine actions of political figures, by tracking their social media feeds, where their press releases are distributed or how often they use a key phrase in a speech. This could lead to story ideas explaining the “why” behind the politician's strategic actions.

Embracing the efficiency of algorithms is also a great way “to kill stories that suck,” Willis said. By having a machine learn to gather the cold hard facts for you, you can avoid anecdotal stories that lack ongoing significance.

While there will always be things that journalists (and their beleaguered interns) need to check by hand, if you choose a good threshold for error (somewhere around a 90 percent certainty that your data is clean, Willis said), algorithms can reduce the list of items to check and decrease the chance that your interns flee the scene.

You can check out an in-depth Storify recap of the ONA event, along with more tips on using algorithms in the newsroom here.

IJNet Editorial Assistant Margaret Looney writes about the latest media trends, reporting tools and journalism resources.

Image CC-licensed on Flickr via dullhunk.