Recommending Toxicity: The role of algorithmic recommender functions on YouTube Shorts and TikTok in promoting male supremacist influencers
Summary Report
Dr Catherine Baker, Prof Debbie Ging and Dr Maja Brandt Andreasen
DCU Anti-Bullying Centre
Dublin City University
April 2024
Acknowledgements
This project was funded by the Department of Justice with DCU Anti-Bullying Centre, Dublin City University, Ireland.
In line with DCU’s Strategy, the core mission of DCU Anti-Bullying Centre is to be a future focused and globally connected European centre of excellence for research and education on bullying and digital safety.
We are grateful to the Reset Australia team for sharing their methodology and offering advice and guidance on our research design.1
- Cover illustration: Willow Cahill
(willow.cahill123@gmail.com) - Data visualisation: Finnian O’Cionnaith2
(finnian.ocionnaith@gmail.com)
1 Reset Australia (au.reset.tech/) is a not-for-profit charity which specialises in research methods on digital risks and online harms.
2 Data visualisation was funded by DCU Faculty of Humanities and Social Sciences Staff Journal Publication Scheme
Research Team
- Principal Investigator:
Prof Debbie Ging,
School of Communications and Anti-Bullying Centre,
Dublin City University
(debbie.ging@dcu.ie)
- Researcher and Lead Author:
Dr Catherine Baker,
Anti-Bullying Centre,
Dublin City University
(catherine.baker@dcu.ie)
- Researcher:
Dr Maja Brandt Andreasen,
University of Stavanger, Norway
(maja.b.andreasen@uis.no)
What is this study about?
There has been growing concern in recent years about the role of recommender algorithms in promoting extreme content to social media users. The growth of influencer culture on TikTok, in particular, has platformed a significant number of highly influential ideological entrepreneurs such as Andrew Tate, Myron Gaines and Sneako. This monetization of male insecurity not only serves to mainstream anti-feminist and anti-LGBTQ ideology, but may also function as a gateway to fringe Far-Right and other extreme worldviews.
Most social media companies do not disclose how their algorithms work, which presents challenges to researchers investigating this phenomenon. We lack evidence on the experience of ‘real’, logged-in users traversing personalised algorithms based on viewing history. Given the recent surge in popularity of short video content, evident in the rise of TikTok and YouTube Shorts, research is needed to explore how platform recommender algorithms function in these new format domains.
The current study simulates the digital reality of boys and young men using TikTok and YouTube Shorts, who are most likely to be targeted by the manosphere. Our findings have significant implications for social media platform governance as well as for the development of educational and technological interventions for boys, men, parents and teachers to prevent radicalization into these ideologies.
Method
This study tracked, recorded and coded the content recommended to 10 experimental or ‘sockpuppet’ accounts on 10 blank smartphones, 5 on YouTube Shorts and 5 on TikTok. On each platform, we set up 5 types of accounts: one 16-year old boy and one 18-year old boy who sought out content typically associated with gender-normative young men (e.g. gym content, sports, video games), one 16-year old boy and one 18-year old boy who actively sought out content associated with the manosphere (e.g. Andrew Tate, anti-feminist), and one blank control account that did not deliberately seek out or engage with any particular content.
Our research team watched, recorded and coded over 29 hours of videos: 12 hours 43 minutes of TikTok videos (or an average of 2 hours and 32 minutes per account) and 16 hours 41 minutes of YouTube Shorts videos (or an average of 3 hours and 20 minutes per account). The recordings were manually and systematically coded to identify the most frequent thematic categories, actors, and hashtags as well as the most dominant myths or ‘talking points’ in the dataset. By coding and sub-coding all content related to men’s rights, anti-feminism and neo-masculinist influencers, we were able to determine the frequency and nature of manosphere recommendations, based on different age profiles, interests and types of interaction.
Key Findings
- Content featuring ‘Manfluencers’ (male influencers) accounted for the vast majority of recommended videos in the dataset, demonstrating their centrality in the current manosphere ecosystem.
- Overall, YouTube Shorts accounts were recommended a larger amount of toxic content3 (on average 61.5% of the total recommended content) than TikTok accounts (34.7%).
- In the case of YouTube Shorts, the manosphere-curious accounts were recommended a significantly higher amount of toxic content (71.4%) than the gender-normative (generic) accounts (51.6%).
- For the TikTok accounts, relatively similar levels of toxic content were recommended to both the manosphere-curious accounts (32.5%) and the gender-normative (generic) accounts (36.6%).
- All of the accounts, both those which sought out manosphere content and those which sought out gender-normative male-interest content, were fed toxic content within the first 23 minutes of the experiment, and manosphere content within the first 26 minutes.
- On TikTok, the gender-normative (generic) 16-year-old and 18-year-old accounts were recommended manosphere content after less than 9 minutes and 15 minutes, respectively. On the manosphere-curious accounts, this happened after 10 minutes and 25 minutes of viewing, respectively.
- On YouTube Shorts, the gender-normative (generic) 16-year-old and 18-year-old accounts were recommended manosphere content after 17 and 2 minutes of viewing, respectively. On the manosphere-curious accounts, this happened after 8 minutes and just under 2 minutes of viewing, respectively.
- Once an account showed interest by watching manosphere content, the amount rapidly increased. By the last round of the experiment (i.e. after 400 videos or 2-3 hours viewing), the vast majority of the content being recommended to the phones was problematic or toxic (TikTok 76% and YouTube Shorts 78%), primarily falling into the manosphere (alpha male and anti-feminist) category.
- Many of the phones were also shown reactionary right-wing and conspiracy content (13.6% of recommended content on TikTok and 5.2% of recommended content on YouTube Shorts). Much of this was anti-transgender content.
3 Toxic content was defined as all coded content, excluding the category ‘Known actor generic content’ (for coding breakdown see Methods section and Appendix 1 in full-length report at antibullyingcentre.ie/recommending-toxicity/).
TikTok Overall Content Prevalence
YouTube Overall Content Prevalence
Top Myths/Talking Points of Manosphere Influencers
The most frequent myths and disinformation / talking points in our dataset fell into three main categories, namely crisis narratives, motivational scripts and gender ‘science.’
- Men and masculinity are under threat, allegedly due to feminism but also more broadly as a result of liberal government ‘brainwashing’ and ‘women-centric’ legal systems.
- Men are being falsely accused of rape. This claim drastically exaggerates the incidence of false rape claims, despite overwhelming evidence that sexual offences are significantly under-reported, under-prosecuted, and under-convicted.
- The heteropatriarchal nuclear family is under threat, allegedly due to female promiscuity, childlessness, divorce, LGBTQ rights and the transgender movement (often referred to as the ‘trans cult’).
- Boys raised without fathers are emasculated and more likely to be rapists, addicts and criminals, while girls raised without fathers are more likely to be promiscuous.
- If you work hard, you’ll make it. This myth is a clear attempt to reconnect male identity with economic status, a relationship which has been substantially disrupted by neoliberal capitalism and its attendant erosion of salaries, career stability, and the social safety net.
- Stoicism is the solution to depression. Rather than acknowledging that the pressure to conform to hetero-patriarchal norms is harmful to men, the manosphere construes male suffering as attributable to feminism and to its alleged agenda to make men ‘more like women’.
- Self-discipline, physical self-improvement and the suppression of emotion are key to manliness, financial success and good mental health.
- Men and women experience (rather than express) emotion differently. Most manfluencers frame depression as weak, emasculating and the result of laziness or lack of motivation.
- Men are hardwired to fight, protect and provide, women are hardwired to reproduce, nurture and stay at home. Debunked theories from evolutionary psychology about sex differences underpin most of the manosphere’s claims.
- Any form of female empowerment necessarily leads to male disempowerment. Most manfluencers claim that feminism has upset the ‘natural order’, made women unhappy and destroyed the family.
- All men want to marry a virgin, yet also desire to have sex with multiple women. This sexual double standard is used to control women through strategies of public sexual shaming.
- All women are hypergamous. This ‘theory’ claims that women strive to ‘marry up’ by seeking out alpha males to optimise their genetic reproductive opportunities.
- Heterosexual relationships and sex are transactional. This concept places strong emphasis on the attractiveness of (sexually) submissive women.
Conclusion
TikTok’s and YouTube Shorts’ algorithms promote toxic content to boys and young men. As the study progressed, each account was recommended an increasing amount of manosphere content, with the majority of messages promoting rigid and harmful masculine norms, misogyny, and spurious advice on mental health and wealth accumulation. The findings of this report point to urgent and concerning issues for parents, teachers, policy makers, and society as a whole.
In particular, our findings highlight the ineffectiveness of social media platforms in protecting children and young people. Ultimately, girls and women are the most severely impacted by these beliefs, but they are also damaging to the boys and men who consume them, in particular in relation to mental wellbeing. We hope our findings will compel the social media companies, government, and policy makers to take urgent action.
Key Recommendations4
- Stricter and more sophisticated content moderation. The proliferation of Andrew Tate content in our dataset at a time when he was de-platformed demonstrates that content moderation needs to be both content- and account-focused.
- Social media companies should work closely with Coimisiún na Meán (Ireland’s new media regulator) and trusted flaggers to highlight illegal, harmful and borderline content.
- Safety-by-design principles should be embedded in product development. This should involve collaboration with diverse experts in gender-based abuse.
- As per the recommendation of the Irish Council of Civil Liberties (ICCL) and 60 other organisations, recommender algorithms should be turned off by default.
- Schools should create a safe space in which young people can be heard. Peer-to-peer learning approaches and positive male role models should be emphasised to promote an educative rather than punitive response to boys’ behaviours.
- Schools need to invest in teacher training in this area, and in teaching critical digital literacy skills.
- Parents should encourage and allow open discussions without fear of rebuttal.
- Parents should discuss why their child is attracted to celebrity influencers, and encourage engagement with relatable resources. There are lots of YouTube videos and podcasts which discuss and debunk the manosphere’s key ideas in a thoughtful and rational way.
4 See full-length report for detailed recommendations (antibullyingcentre.ie/recommending-toxicity/).
Useful Resources
Webwise webwise.ie
The Positive Masc Project positivmasc.ki.se
Beyond Equality beyondequality.org
Hope Not Hate hopenothate.org.uk/communities/in-schools/
BBC Trending: How to Exit the Manosphere - bbc.co.uk/programmes/w3ct5d95
European Institute for Gender Equality: Gender Equality Index, Ireland (2022) eige.europa.eu/modules/custom/eige_gei/app/content/downloads/factsheets/IE_2022_factsheet.pdf
Rape Crisis Network Ireland (RCNI) 2022 Statistics rcni.ie/wp-content/uploads/RCNI-Rape-Crisis-Statistics-2022.pdf
Who is Andrew Tate? The Journal.ie Explainer podcasts.apple.com/ie/podcast/who-is-andrew-tate/id1452246930?i=1000595850526
Now and Men podcast: Men, Masculinities and Gender Equality menengage.org/resources/now-and-men-podcast-men-masculinities-and-gender-equality/
More Information
This is an overview of the report ‘Recommending Toxicity: The role of algorithmic recommender functions on YouTube Shorts and TikTok in promoting male supremacist influencers’.
The full document is available online at: