No. 60 - Cornelius Puschmann, Hendrik Meyer: Gaging LLMs’ strengths and weaknesses in political content analysis

Abstract

Recent innovation and growth in the capabilities of generative artificial intelligence have opened options for the scholarship of political communication. While large language models (LLMs) have much to offer to improve content analysis workflows in political communication, there is as of yet no consensus regarding which options deliver the best results to the field and scale best computationally. This paper investigates the performance of five distinct classification models across three content analysis tasks of varying complexity: identifying political content, classifying political issues, and detecting stance towards climate policies and movements in news articles. Our automated coding is validated by 36,320 individual human coding decisions. We show that (political) task complexity is a crucial determinant for model selection. While simpler, efficient models perform well on lower to moderate complexity tasks, both proprietary and commercial LLMs exhibit superior performance on the nuanced stance detection task when appropriately instructed. These findings underscore the need to balance performance gains with computational costs and CO2 emissions, advocating for a “frugality approach” to automated coding.

Read more.

About the authors

Cornelius Puschmann is Professor of Communication and Media Studies with a focus on digital communication at ZeMKI and leader of the digital communication and information diversity (DCID) lab. From 2012 to 2016, he headed the DFG project “Networking, Visibility, Information? Usage motives of informal digital communication genres among scientists” at the Institute for Library and Information Science at Humboldt-Universität zu Berlin. Between 2014 and 2015, he held a professorship for communication science with a focus on digital communication at Zeppelin University Friedrichshafen. After conducting research at the Alexander von Humboldt Institute for Internet and Society as part of the project “Networks of Outrage: Mapping the emergence of new extremism in Europe” from March to September 2016, he moved to the Leibniz Institute for Media Research / Hans Bredow Institute at the University of Hamburg in 2016. Cornelius Puschmann has been a visiting scholar at the Oxford Internet Institute at the University of Oxford, the Berkman Klein Center for Internet and Society at Havard University and the Department of Media Studies at the University of Amsterdam, among others.

Hendrik Meyer has been a research associate at the Faculty of Economics and Social Sciences since May 2021. His research interests lie primarily in the study of structures and amplifications of media-networked political discourses – especially their supposed “polarisation”. He is also interested in how the society negociates social media and platform regulations as well as public discourses on media technologies based on algorithms and so-called “artificial intelligence”. Currently, he is also working on the data processing and analysis of the project “Climate Futures” in the Cluster of Excellence CLICCS and researching potential polarisation dynamics of (climate) political online discourses. In his dissertation, he analyses and classifies findings generated from this research through communication studies and network sociology.