VizPol: Real-Time Symbol Recognition for Field Reporting and Image Tagging
by Ishaan Jhaveri
VizPol App Returning Results for a Search
Editor’s Note: The first five introductory paragraphs below have been paraphrased from “When is a frog not a frog? Building a new digital tool to track political symbols," Columbia Journalism Review, September 26, 2019.
On August 12, 2018, Nina Berman, a photojournalist who covers American political movements and a professor at Columbia University’s Graduate School of Journalism, was photographing a “Unite the Right 2” rally in Washington D.C The event was organized by one of the leaders of the deadly 2017 Unite the Right Rally in Charlottesville, Virginia. Berman hoped to see what issues motivated the participants and how the press covered the event.
© Nina Berman
Berman spotted a couple, a man and a woman, with tattoos down their arms and their faces partially hidden by scarves. She took a picture. When Berman reviewed her photos after the rally, she looked more closely at the tattoos, and a series of numbers on the woman’s forearm caught her eye. They read “1488,”a white supremacist meme (the 14 represents white nationalist idealogue and convicted murderer David Lane’s 14-word slogan, “We must secure the existence of our people and a future for white children,” and the 88 represents “Heil Hitler,” “H” being the 8th letter of the alphabet.)
At the time, the symbol was new to Berman. She reflected that had she been able to identify this symbol in the moment,
she would have been able to ask the woman about the apparent discrepancy between what she claimed she was demonstrating for and what she was telegraphing on her body.
In the three years since then, reporting on symbols has increased in frequency and relevance. Across print, visual, and digital media, photographs of a shirtless Viking impersonator, a man leading rioters wearing a t-shirt with a large “Q” inlaid with the US Flag on it, and a young man invading the senate chamber and sitting in the Presiding Officer’s chair were some of the many widely shared and analyzed from the January 6th Capitol Riots.
These photos introduced the QAnon movement to a wider audienceand revealed the little-known white supremacist America First movement, popular mostly among young far-right activists.
January 6th also catapulted coverage of far-right groups like the Proud Boys, the OathKeepers, and Christian Nationalist symbols like the obscure Pine Tree Flag.
Several news outlets ran articles with variants of the title “Identifying far-right symbols that appeared at the U.S. Capitol riot” (such as this one from the Washington Post), attempting to categorize the overwhelming political imagery and branding on display.
New York Times article titled “Decoding the Far-Right Symbols at the Capitol Riot”
While guides like these are invaluable to journalists, researchers, and law enforcement analyzing events like the Capitol riots after the fact, they often come too late. Photographs and interviews from reporters in the thick of the fray at protests are essential in building a body of evidence which can then be studied to understand the larger narrative of the event, and even identify key actors within the agitating crowd.
Berman saw this lack of visual literacy about political symbols as a missed opportunity for reporters to include crucial context about events they were covering. She felt that if a reporter had in-the-moment context about a symbol a potential source was spotted with, the reporter would be able to conduct a better interview. Specific questions about their political ideology or demands would allow for fuller reporting. But expecting reporters, particularly those sent to report on protest or similar events to stay up-to-date on the large and transient body of political iconography is a tall order (and even more so for those who don’t usually report on political beats).
Berman wanted to solve the problem of poor visual literacy about political symbols for herself and for other journalists who would no doubt encounter it. But she wanted to design a tool that wouldn’t force journalists to memorize an ever-increasing and ever-changing roster of symbols.
This brings us to VizPol. Over the last two years, an interdisciplinary team of researchers from Columbia University’s journalism and computer science departments have created an app that serves as a resource for on-the-ground reporters to quickly search for crucial, in-the-moment context as they interview, photograph, or collect evidence from a protest, political demonstration, riot, or related event.
The VizPol app is designed to be an up-to-date database of political symbols, queryable in real-time using only an image of the unknown symbol for use by on-the-ground reporters, image captioners, and photo editors. In just under two years, our team designed and built VizPol for iOS, Android, and the Web. We tested the tool in the field in the summer of 2020 by giving it to about 70 real-world reporters and photographers who report on political events. Below is a summary of the app’s goals and suggested uses.
Project Goals For Reporters
Identifying symbols in the field in real time to help with determining reporting context and give insight into the individual being pictured or interviewed
Communicating identified symbols back to a central database, constantly building up a collection of symbols that can evolve over time with input from on-the-ground reporters in aid of future reporting
Our continued reporting on political symbols has shown us that many have originated, evolved, or been given a meaning in obscure online spaces that would be difficult for reporters to keep abreast of. We wanted to create a system that lets a reporter add a political symbol to a centralized database that is also available to other reporters so that when it is used again, other reporters with access to the database can instantaneously learn what it is.
Identifying symbols in the editing stage following visual reporting of an event to make sure relevant symbols are included and identified properly in captions.
Identifying extremist symbols which may not be suitable for publication or may require a warning.
In an October 2019 story about clashes between protesters and counter-protesters over a Confederate monument outside a county courthouse in North Carolina, WRAL, an NBC-affiliated TV station posted a video to their website showing a man stressing that the statue had nothing to do with slavery. However, in plain view, dangling from his chain was the white supremacist version of the pre-Christian Celtic Cross:
Had the web editor known the meaning of the chain, they might have not aired the interview, or at the very least pointed out that though the man claimed that the Confederate monument had nothing to do with slavery, he was wearing a white supremacist symbol as he said so.
For Image Taggers
Tagging images with symbol identifiers so that people can search photographic databases for images by symbol.
VizPol was created as a hyper-specialized, instantaneous resource for people needing to conduct image-based searches for information on visual graphic political symbols. In the research on the app to be published in a forthcoming white paper with the Tow Center, we explain how the tool works better than several other popular generalized reverse image search platforms. We present a thorough blueprint of a product aimed at journalists with the primary objective of giving them another resource in their toolkit to increase accuracy and reduce misinformation in their reporting. While VizPol didn’t end up receiving as much usage from our first batch of “alpha testers” as we had hoped, we believe that we have successfully prototyped a system and mobile technology for newsrooms to be able to make up-to-date information on a wide range of visual political symbols available in real-time to their correspondents. Furthermore, there is nothing in the algorithm that conducts the visual query, nor in our crowd-sourcing approach to keep the database current that is intrinsic to political symbols, and we believe this method could have uses in other news and non-news domains as well.
Editor’s Note: This is an excerpt from a forthcoming white paper with the Tow Center about the VizPol project.