Social media recommender systems play a huge role in today’s information gatekeeping. They act as filters and amplifiers of messaging, and are frequently gamed by bad (including FIMI) actors to spread false information, propaganda, scams, and other toxic content. Many existing studies already showed their considerable influence on society, including minors. For these reasons, social media platforms gradually become subjects to legislation, regulation and public oversight. Algorithmic auditing is one of the few ways to independently assess properties of social media recommender systems. In algorithmic auditing, a set of agents that impersonate real platform users is stimulating the recommender with user actions (e.g., views, likes, shares, flags) and record platform’s reactions (e.g., the recommended content). When done over time and with controlled behavior of the agents (which are typically implemented as bots), this process yields sufficient data to disclose tendencies, biases and possible legislation violations of the audited recommenders. Such approach allows for quantitative assessment of platform’s behavioral traits, something which would otherwise require direct access to data and logs of the system, which is typically not available (due to the lack of cooperation of the platform). Algorithmic auditing has been known for some time, but remains technically and methodologically challenging. In this paper, we present our 3-year experience of designing, implementing, and executing auditing pipelines. We discuss issues such as anti-bot defenses of the platforms or scalability. We demonstrate our findings on several cases studies (of audits), which include: (1) demonstrating the difficulty of audit replication, (2) measuring personalization drift for polarizing topics, and (3) advertisement appearance for minor users and adherence to minor protection as stipulated in the European Digital Services Act (DSA).

Jakub Šimko is a researcher in the areas of artificial intelligence, human-computer interaction, human computation, crowdsourcing and human-AI symbiosis. His research domain is disinformation combatting through automatic detection of disinformation content, fact-checker support and auditing of social media recommender systems. He graduated from Slovak University of Technology in Bratislava, where, after receiving his PhD, he worked for 7 years as a researcher and teacher. Since 2020 he has been a researcher at Kempelen Institute of Intelligent Technologies, where he also leads the Web and User Data Processing team. He co-authored more than 50 publications and received more than 900 citations. He has participated in several European research projects. In 2021, the auditing research he participated in was awarded the Best Paper Award at the prestigious ACM RecSys conference.