Multimodal Social Media Analysis for Gang Violence Prevention

Gang violence is a severe issue in major cities across theU.S. and recent studies have found evidence of social me-dia communications that can be linked to such violence incommunities with high rates of exposure to gang activity. Inthis paper we partnered computer scientists with social workresearchers, who have domain expertise in gang violence, toanalyze how public tweets with images posted by youth whomention gang associations on Twitter can be leveraged to au-tomatically detect psychosocial factors and conditions thatcould potentially assist social workers and violence outreachworkers in prevention and early intervention programs. Tothis end, we developed a rigorous methodology for collectingand annotating tweets. We gathered 1,851 tweets and accom-panying annotations related to visual concepts and thepsy-chosocial codes:aggression,loss, andsubstance use. Thesecodes are relevant to social work interventions, as they repre-sent possible pathways to violence on social media. We com-pare various methods for classifying tweets into these threeclasses, using only the text of the tweet, only the image of thetweet, or both modalities as input to the classifier. In partic-ular, we analyze the usefulness of mid-level visual conceptsand the role of different modalities for this tweet classifica-tion task. Our experiments show that individually, text infor-mation dominates classification performance of thelossclass,while image information dominates theaggressionandsub-stance useclasses. Our multimodal approach provides a verypromising improvement (18% relative in mean average preci-sion) over the best single modality approach. Finally, we alsoillustrate the complexity of understanding social media dataand elaborate on open challenges. The annotated dataset willbe made available for research with strong ethical protectionmechanism.