Co-Designing Artificial Intelligence-Based Cyberbullying Interventions on Social Media with Children

Qualitative Research Findings

By: Tijana Milosevic, Kanishk Verma, Samantha Vigil, Michael Carter, Derek Laffan, Brian Davis, & James O’Higgins Norman

Download "Co-Designing Artificial Intelligence-Based Cyberbullying Interventions on Social Media with Children: Qualitative Research Findings" PDF

Abstract

This report details the results of a qualitative research study (focus groups and indepth interviews) with children and teens aged 12-17 (N=59) in Ireland about the perceived effectiveness of Artificial Intelligence (AI)-based cyberbullying enforcement mechanisms on popular social media platforms. The adoption of the UN General Comment No. 25 established that children’s rights, as outlined in the UN Convention on the Rights of the Child (UNCRC), apply in a digital environment. We therefore examine children’s perceptions about how AI-based enforcement mechanisms affect their rights to protection (safety), participation and privacy. We inquire into how children perceive the effectiveness of the proposed mechanisms; and how these could be made more effective from their perspective; and which changes or alternatives they propose. The proposed interventions are based on social learning and social norm theories, and they include designated support contacts, bystander and school involvement, and systems that are designed to reward prosocial behaviours and deter perpetration. We find that children would welcome many interventions but raise concerns around their privacy and effectiveness of what has been proposed. We provide policy recommendations for the technology industry and policy makers.

Tijana Milosevic, Elite-S Research fellow, DCU Anti-bullying Centre (ABC) and ADAPT SFI
Kanishk Verma, Irish Research Council PhD Candidate, DCU School of Computing, ADAPT SFI, ABC DCU 
Samantha Vigil, PhD Student, Department of Communication, University of California, Davis
Michael Carter, PhD Candidate (ABD), Department of Communication, University of California, Davis
Derek Laffan, ABC DCU
Brian Davis, Professor, DCU School of Computing and ADAPT SFI
James O’Higgins Norman, Director ABC, DCU, Professor at DCU, and UNESCO Chair on Tackling Bullying in Schools and Cyberspace

Introduction

During Covid-19 lockdowns, youth overwhelmingly relied on the Internet for activities that normally take place offline, such as schooling. While also for socialising and leisure, in some countries, this uptick in mediated activities coincided with an increased rate of cyberbullying victimisation for particular age groups (Lobe et al., 2021). Cyberbullying, or the enactment of repeated and intentionally hurtful behaviour (Hinduja & Patchin, 2015), is a serious problem across social media platforms and can take on various forms. For example, cyberbullying can span mean or abusive comments, posts or direct messages (DMs); creating a fake profile of someone for the sake of mocking them; excluding someone from an activity on purpose, revealing their private information (eg, doxing); and so on (Smith, 2016; O’Higgins Norman, 2020). Given its complexity, cyberbullying definitions¹ remain a matter of academic and policy debate, and cyberbullying is sometimes considered as interchangeable with harassment, especially in social media platforms’ policies. Nevertheless, platforms typically do not allow activities deemed as abuse, cyberbullying, and/or harassment on their platforms, as stipulated in their Terms of Service, Community Standards, Guidelines, or other comparable documentation (Milosevic, 2016, 2018).

Common mechanisms implemented over popular social media platforms as tools to target instances of cyberbullying also vary in their level of human involvement. Conventionally, users often have access to options for reporting on abusive content uploaded or sent over a platform. This typically initiates a moderation process to determine whether reported content or activities violate the company’s policy and if any content should be taken down. With millions of users and vast amounts of content, it is impossible for companies to rely on human moderators alone to facilitate this process, however (Gillespie, 2018). Algorithmic applications, such as natural language processing (NLP), machine learning (ML) and deep learning (DL), remain common among popular apps to help automate the process of content moderation; these approaches fall under the umbrella of “artificial intelligence” or AI (Gorwa et al., 2020; Milosevic et al., 2022). Given the capacity of AI-based moderation techniques, companies have begun to use AI applications to try and proactively moderate content by detecting and removing content before it is even reported by users (Community Standards Enforcement Report).² Notwithstanding such innovations, the legitimacy and efficacy of the use of AI for content moderation over social media remains under ongoing scrutiny (Heldt & Dreyer, 2021). In the end, platforms often have to partially rely on users to take the initiative, whether by reporting content or by implementing various forms of user blocking (eg, unfriending, blocking), content restriction (eg, segregating audiences, muting), and/or content filtering (eg, Hidden Words on Instagram) to manage their experiences on the platform.³

As a result of ongoing developments in platform-based practices for addressing instances of cyberbullying and the persistent reliance of platforms on user involvement to intermediate their experiences with forms of online abuse, it is critical to understand youths’ perspectives on the efficacy of platform tools in this context. This is especially true given the rise of legislative frameworks focusing on systemic changes to content circulation (for an example, see Douek, 2022), which require the provision of evidenced effectiveness of platform tools and AI-based moderation internationally (eg, Online Safety and Media Regulation Bill⁴, Ireland; Online Safety Bill⁵, the United Kingdom; Digital Services Act,⁶ the European Union; Online Safety Act,⁷ Australia), which were in part created to protect young users. Lastly, provided the pace at which the commercial social media landscape changes over time, there remains ample need to explore innovative and novel platform mechanisms targeting the mitigation of online forms of cyberbullying and abuse in particular.

Therefore, the present study sought to advance understanding of youths’ perspectives towards AI mechanisms and platform tools targeting cyberbullying and online forms of abuse over multiple social media apps. To do so, we conducted 6 focus groups and 15 semi-structured in-depth interviews with children and adolescents (N =59) to assess their views towards platform mechanisms targeting online abuse through a set of five hypothetical cyberbullying scenarios illustrated via a set of realistic, yet mock user interfaces mirroring core aspects of commercially available social media apps (i.e., TikTok, Instagram, Trill Project). Scenarios included an array of novel platform mechanisms (eg, designated support contact, bystander notifications, and rewards) to explore a range of possible intervention designs. The study represented the qualitative phase of the research project “Co-designing with Children: A rights-based Approach to Fighting Bullying” funded by Facebook/Meta Content Policy Award, Phase 2.⁸ In all, results help to inform the developing policy-making environment globally in the context of social media by highlighting themes in youths’ perspectives towards different types of platform mechanisms targeting instances of cyberbullying and abuse online.

Children’s rights in digital environments: implications for safety by design⁹

The adoption of the United Nations’ General Comment No. 25¹⁰ in 2021 established that children’s rights as specified in the United Nations Convention on the Rights of the Child (UNCRC) apply in the digital world (Staksrud, 2016; Livingstone et al., 2016). This signifies that children have, among others, rights to protection, and freedom from abuse and cyberbullying is considered as a right to be protected and safe online and offline. They also have the right to provision, which encompasses the right to education and quality media content, for example. Bullying and cyberbullying in schools are considered as an affront to education because they interfere with the child’s ability to learn. Therefore, ensuring the right to protection from bullying and cyberbullying is a critical enabler of other rights. Finally, children also have the right to participation which includes their right to express views on matters that concern them (such as cyberbullying moderation on social media platforms); and also to participate in environments that provide them with leisure and socialisation opportunities, such as social media.

Another important right that directly concerns this study is children’s right to privacy. In online environments, children’s right to privacy can be violated and infringed upon in several ways (Livingstone et al., 2019). Firstly, children can unwittingly share too much information about themselves which can jeopardise their safety. For example, in more extreme cases, they can publicly reveal information about where they live or leave geolocation traces which can allow for their tracking by strangers. Children’s privacy can be jeopardised in the social context too, for example when their friends or parents/ caregivers take their photos without their consent and then post them on social media, a phenomenon known as sharenting (Livingstone et al., 2020). Data collection for commercial purposes such as tracking, which is the basis of social media platforms’ business models, can also constitute an affront to children’s privacy (Mascheroni & Siibak, 2021). Among other issues, this includes sharing their data with third parties and data brokers that can lead to data breaches and profiling that can hamper their future education and job opportunities (Montgomery et al., 2017). In this study, we are interested in children’s perceptions of privacy in the context of deploying AI-based cyberbullying interventions. Specifically, we are focused on hearing about how children perceive their right to privacy from AI-based monitoring of public posts and private messages, as well as the use of facial recognition.

Protection vs. participation?

In addition to reporting abusive content for company moderation, social media platforms tend to provide users with automated options designed to assist with various forms of cyberbullying where the platform does not get involved. For example, users can block and mute others or discussions that they find insulting; they can restrict users if they do not wish to see their comments and posts (a polite way of blocking), they can also turn off comments so that no one can comment on their content; or they can turn on comment filtering (“Hidden Words”¹¹ on Instagram), whereby all comments which are detected to have abusive content are not shown/visible to the target.

Such features, however, place the onus on the young user to deal with problems on their own without the platform’s assistance. Secondly, the features that restrict activity or access, prioritise children’s safety over their ability to fully participate in online environments (Livingstone & Third, 2017). For example, having an Instagram account that is private or if comments are switched off for safety reasons can limit the possibilities of engagement, thus prioritising children’s right to protection over their right to participation, as we detail below. Companies often recognise that their support systems face significant challenges and are not fully adequate; some companies also have a trusted flagger system (also known as trusted reporter) in place, which allows individuals and various third-party organisations such as nongovernmental organisations the option to draw companies’ attention to individual cases of abuse; and escalate cyberbullying, hate speech and other violations to companies.¹² 

Theory and research informing the design of our interventions

Interventions tested in this study (please see section “Interventions tested in this study,” which details each intervention) are informed by social learning and social norm theories, which posit that maladaptive behaviours such as cyberbullying are reinforced by role models who behave in an overtly or covertly aggressive manner; and when these behaviours are supported by the social environment or considered as acceptable or normative (Espelage et al., 2012; Hinduja & Patchin, 2013). For example, when the perpetrator receives tacit support or active encouragement from those who witness the abuse, such behaviour enables the perpetrator to continue with abusive behaviour, and it also sends a message to those who witness the abuse (bystanders) that such behaviour is allowed. Much research has therefore focused on the conditions that determine whether a bystander will get involved to assist the victim in a cyberbullying situation (therefore becoming an “upstander”).

Furthermore, offline and online bullying tend to go hand in hand, with an overlap between victimisation and perpetration (eg, bully-victim phenomenon); those who were once victims can also be perpetrators at the same time or later on (eg, getting back at someone; Kowalski et al., 2014; Görzig, A., & Macháčková, 2015). Thus, while online the perpetrator can operate anonymously by hiding behind a username or a fake profile/account, young people often know who bullies them, especially if an incident happens in the context of peer relationships or at school (Mishna et al., 2009; Mishna et al., 2021; O’Higgins Norman, 2020).

Existing research has explored social and technology design-related conditions that increase the likelihood of bystander involvement to assist the victim rather than the perpetrator (Bastiaensens et al., 2016; DeSmet et al., 2014). It has been examined whether a lack of empathy and accountability contributed to failure to get involved to help the victim, with some findings suggesting that empathy could prevent negative bystander behaviour (Barlińska et al., 2013; Macháčková & Pfetsch, 2016). As for accountability, it has long been established in research on offline bullying that the presence of more bystanders can diffuse the sense of responsibility whereby each of them believes that someone else will help the victim (Latane & Darley, 1970). Accountability, or the belief that one will be held responsible for one’s actions leads to a sense of personal responsibility which motivates action to help the victim, and research has recently explored how technological design can promote a sense of personal responsibility (van Bommel et al., 2012). For instance, it has been found that a sense of awareness that one is being watched in public and that they are not anonymous was found to be conducive to prosocial behaviours (Pfattheicher, & Keller, 2012). For example, when bystanders were informed about the size of the audience and when they received a notification from the platform that they have seen an abusive post/message, they were more likely to intervene on the side of the victim by reporting the bullying content to the platform (DiFranzo et al., 2018). Reporting abusive behaviour, content or blocking the perpetrator is considered as indirect support for the victim, whereas direct support would entail writing to the victim to offer help or responding to attacks or addressing the perpetrator.

Following this line of research, we created a set of demos whereby support from designated contacts/helpers and bystanders is solicited when abusive behaviour such as cyberbullying is detected by AI (as described below). Earlier research conducted by Meta/Facebook in collaboration with Yale Centre for Emotional Intelligence and University of California, Berkeley, explored social reporting, a process which allowed users to solve conflicts amongst themselves by reaching out to others for help with pre-made messages; or to perpetrators with requests to take content down (Anderle, 2016; Milosevic, 2018). The research attempted to test whether users would reach out to third parties for help using pre-made messages and whether this process would result in the perpetrator taking content down or apologising for their actions. According to the findings presented at Facebook’s Compassion Research Day, of the 25% of children who used social reporting options, 90% messaged the person they had a problem with, and over a third of those who posted something problematic deleted such content once they had been contacted and asked to do so (Milosevic, 2018, p. 129). Hence, we provided the option in some of our demos for the victim, support contact or bystander to reach out to the perpetrator asking to take the content down or apologise. We also created a variation on this type of a response by allowing children to create an anti-bullying video with pre-made text which tells the perpetrator that such abusive behaviour is hurtful or not ok (variations on the type of message were possible and open to children for feedback).

Reflective messages are a widely researched intervention which was shown to be effective in reducing abusive behaviours, and some platforms already have them in place (Ashktorab & Vitak, 2016; Lieberman et al., 2011; Van Royen et al., 2021; Van Royen et al., 2017; Van Royen et al., 2016). Before posting/ messaging, the content of the post is screened by AI for abusive content and if it is detected, the poster/sender is provided with a reflective message prompting them to reconsider if they really wish to post/send it. Since this is a widely researched type of intervention, we used it as an optional demo, and asked participants about desirability and perceived effectiveness of this tool.

Finally, research has suggested that interface design which rewards prosocial behaviours should be explored further in terms of effectiveness in reducing undesired behaviours (Wu et al., 2022). We solicit youth views on the idea to gain access to more platform features and increase one’s supportiveness score as a reward for helping others.

The current study

The following research questions guided this phase of the project: 

RQ1: How can we design automatic tools that support effective proactive bullying interventions that assist victimised children while ensuring children’s rights to privacy, freedom of expression and other relevant rights as outlined in the UNCRC? 

RQ2: How can we leverage children’s feedback to optimise the effectiveness of such tools?

Interventions tested in the study

Interventions we designed in this study involve not only the target (victim) and the perpetrator but also those who witness cyberbullying incidents, the so-called “bystanders” (Rudnicki et al., 2022). Bystanders can remain neutral and not become involved in the incident they are witnessing; or they can support the perpetrator or support the victim (at which point, they are considered to be “upstanders”). Furthermore, we have included a feature called “support contact/helper/friend” whom children can add upon sign up and who can be contacted when abuse is detected by AI. The idea behind the support contact is based on peer mentoring (Papatrainou et al., 2014; Bauman & Yoon, 2014), but we envisaged that the support contact can be an adult as well (parent/caregiver, or someone else who is close to the child).

Using a collaborative interface design tool, Figma,¹³ the research team created four core and two optional demos¹⁴ each showing a scenario with an example of abusive behaviour that could constitute a cyberbullying incident on Instagram, TikTok and Trill¹⁵ and a subsequent intervention. Core scenarios were shown in each interview and focus group while the optional ones were shown if there was additional time in the session. Each scenario then showed examples of how the incident could be detected by AI proactively and a subsequent intervention based on research into bystander involvement in cyberbullying incidents (Bastiaensens et al., 2014; DiFranzo et al., 2018; Macaulay et al., 2022). The proposed interventions as designed in this study are hypothetical and only some components of these are available currently on certain social media platforms. For example, “hidden words” on Instagram allow the user to turn on comment filtering, which removes abusive comments which the user can later on nonetheless view if they would like to. All of the features we propose, should be, however, technologically feasible to implement, based on the current state of AI development for the purpose of detecting cyberbullying and harassment as previously identified by the authors of this report (Milosevic et al., 2021).

For example, we proposed that once children create an account on Instagram/TikTok/Trill, children be offered the option to add a support contact/helper/friend who could be contacted if AI detects cyberbullying or some other type of abuse on the platform. A support contact could be a friend, parent, teacher or someone else and the person need not be using the given platform. In Demo 1, (Image sequence 1) we showed an example of a girl receiving negative comments on her post on TikTok; once these are detected by AI, the girl receives a notice from TikTok that abusive comments have been detected, and she is prompted to review them (abusive comments are not displayed automatically in order not to traumatise her if she chooses not to see them); or to request help from the support contact. Demo 1 also showed the option to request support from those who have been detected by AI as bystanders (eg, they posted something positive or neutral on the post that received negative comments, or have merely been detected as having seen the abusive post). Those identified by AI as bystanders would receive a prompt from the platform that abusive comments have been detected on the person’s post and they’d be prompted to intervene by providing support to the person who was abused; or by reporting the abusive content or account to the platform; or by reaching out to the perpetrator asking them to take it down. We then asked children for feedback on the desirability of such options, perceived effectiveness of these interventions and their perceptions of how such deployment of AI might affect their privacy and freedom of expression.

In the second demo, we featured an example of cyberbullying by exclusion, which according to Instagram was a common way for teen girls to experience cyberbullying on the platform.¹⁶ For example, purposeful exclusion would be made visible and performative (Marwick & boyd, 2014) by tagging the person in a story or post featuring photos from the event to which she was not invited. In the demo, we showed three teen girls tagging the fourth one in a photo from an event where she was not invited

By photo analysis and facial recognition, AI application could detect that more people are tagged in the photo than are actually present in the photo; and establish that bullying has possibly occurred by further examination of direct messages (DMs) exchanged among the three girls who talked about not inviting the fourth one to the event, and then showing her that she was not invited by tagging her in the photos. Thereafter, the victim would receive a prompt asking her whether she’d like to review the post where she’d been tagged in and report it to Instagram, in case it was bullying. Any intervention that would prompt the victim to view an abusive message should contain a trigger warning as well. She would also be prompted to reach out to her support contact for help. The support contact would be provided with the option to reach out to the girls who engaged in exclusion and ask them to take the post/story down, explaining that such behaviour is hurtful. Both the victim and the support contact would have the option to restrict further sharing of this post/story on Instagram and other platforms, in addition to the regular options of reporting it to the platform and untagging themselves.

Demo 3 offered the possibility of reporting a cyberbullying incident on Instagram to one’s official school account which would be managed by a professional at their school. Under this scheme, every school in Ireland would have an official account on Instagram. Upon sign up, children would be given an option to confirm their attendance of a particular school and given the ability to report incidents to their school. This demo is a variation on Facebook/Meta’s earlier proposals and efforts in the United States (at the state level)¹⁷ to involve schools as escalators or trusted flaggers.

Under such a scheme, the school would be able to flag a case to the platform for prioritised handling as a trusted flagger (Milosevic, 2018). In the demo, we did not position schools as trusted flaggers, but rather we tested the desirability of school involvement into cyberbullying cases altogether. The demo shows a boy tagged in a post with abusive comments underneath; the post was then detected by AI proactively and the boy was prompted to report it to his school in addition to reporting it to the platform; like in previous demos, the option to reach out to a support person was provided; as well as the possibility of asking the perpetrator to take the post down. Furthermore, the perpetrator was punished by having less engagement on all his posts over the course of the following month (i.e., all his posts regardless of the nature of their content would have less visibility to other users on the platform, similarly to shadow banning¹⁸), following a notification and the option to appeal the decision.

Demo 4 took place on Trill and it showed homophobic bullying of a person via direct messaging. AI was able to scan DMs for abusive content and following the detection of such content, the sender was automatically blocked; and the victim received prompts with options to seek support from the support contact and report the content to the platform. Subsequently, those who engaged as support contacts were rewarded with support score points, which could be added to one’s account profile/username and they were also rewarded by being able to unlock additional platform features such as colours.

Demo 5 was an optional demo (we only showed it if there was enough time left in the end of each interview/FG session) which allowed users to create an anti-bullying video on TikTok and Instagram upon sign-up. The anti-bullying video could be tailored by the user and created together with the support contact/helper/ friend and feature any music/sound clips available. Users could incorporate a pre-made message such as “be kind” or “that was hurtful,” or “this is not ok,” asking the perpetrator to take abusive content down or stop the abuse (common messages in online safety campaigns¹⁹); or the user could write something that they thought was appropriate, which could even try to frame the situation in a joking manner or be more assertive in tone towards the perpetrator. The video could then be sent automatically when AI detects something abusive towards the user; or the user could choose whether and when it should be sent.

Finally, the last optional demo showed a “reflective message,” a well-researched intervention already used by some platforms, which prompts the user who is about to post something detected as abusive to think twice before posting it. The message that the poster was about to post was not necessarily abusive, it expressed a negative opinion “a bit dull if you ask me” in response to a throwback post of someone having fun in a photo of a pre-Covid lockdown party. The comment was trying to convey the message that their party did not seem like that much fun after all.