Auditing Disinformation Spreading by Social Media Recommender Systems
Nowadays, spreading of online false information threatens the democratic fabric of our society. Meanwhile large social media platforms continue to exploit their market position and recommendation algorithms to extract ever more revenue from promoting arousing and controversial content (which disproportionally includes false information). Despite promises, platforms fail to self-regulate themselves to curb this trend, which has been going on for almost a decade. Public pressure on social-media has been steadily increasing, but so far, social media could escape the criticism by slamming the available evidence as anecdotal (and were, unfortunately, right in many cases).
To address this, a more systemic response is mounting. Public authorities, policy-makers and regulators prepare new legislation and oversight frameworks for large online platforms (e.g. EU’s Digital Services Act and Artificial Intelligence Act). Meanwhile, researchers focus on developing scalable oversight tools, capable of independent and systematic (quantitative) collection of social media misconduct evidence.
In this talk, we explain the concept of automated auditing of recommender systems. In this context, an audit is a systematic probing of a recommender system behavior. As a case study, we take our sockpuppeting, black-box audit of YouTube’s recommenders and search engine, which focused on disinformation filter bubble creation and bursting. In a sockpuppeting audit, a collection of agents-bots poses as users of a given platform. They stimulate its recommender system through execution of pre-defined scenarios of user actions (e.g. video watches, likes, subscribes) and record system responses (recommendations). In a following analysis, the recommended items are annotated, and false information quantified. In future works, new methods of automated content annotation and realistic audit scenario generation will need to be researched to make audits fidelitous and ubiquitous.
Jakub Šimko
is a researcher in the areas of artificial intelligence, human-computer interaction, human computation, crowdsourcing and human-AI symbiosis. His research domain is disinformation combatting through automatic detection of disinformation content, fact-checker support and auditing of social media recommender systems. He graduated from Slovak University of Technology in Bratislava, where, after receiving his PhD, he worked for 7 years as a researcher and teacher. Since 2020 he has been an Expert researcher at Kempelen Institute of Intelligent Technologies, where he also leads the Web and User Data Processing team. He co-authored more than 30 internationally recognized publications, together receiving more than 200 citations. Currently, he is participating on 4 European research projects. In 2021, the auditing research he participated in was awarded the Best Paper Award at the prestigious ACM RecSys conference