2.1 Toxicity in online conversations

Characterising toxic content is a difficult challenge as toxicity is a broad concept that can include threats, racial slurs, extremist views, sexism, insults, hateful, and derogatory comments, which can be both directed at specific users or at communities. In this context, Jurgens, Chandrasekharan and Hemphill (Reference Jurgens, Chandrasekharan and Hemphill2019) categorise toxic online content into two categories – (a) Overt, physically dangerous content: content that may or may not use abusive or hateful language and may include personally identifiable information to cause harm (e. g., doxxing). Content that promotes behaviours of criminal harassment, human trafficking, paedophilia or explicitly incites mobs for violence – all these constitute the category of overt and physically dangerous content; (b) Subtly harmful content: content that is linguistically subtle, where harm is implicit. We note that the latter category has been predominantly disregarded and is among the difficult sets of samples for annotation. Our work focuses mostly on the latter category.

There has been an increasing amount of work towards studying toxic online conversation in the last few years, which has focused on overt and physically harmful online behaviours (Waseem Reference Waseem2016; Vidgen and Derczynski Reference Vidgen and Derczynski2020). The topic has been studied under various lenses, including that of ‘abusive language’ (Nobata et al. Reference Nobata, Tetreault, Thomas, Mehdad and Chang2016), ‘harmful comments’ (Faris et al. Reference Faris, Ashar, Gasser and Joo2016), ‘toxic language’ (Jacobs et al. Reference Jacobs, Blodgett, Barocas, Daumé and Wallach2020), ‘online hate speech’ (Davidson et al. Reference Davidson, Warmsley, Macy and Weber2017) and incivility (Kumar et al. Reference Kumar, Ojha, Malmasi and Zampieri2018). Further, recent work has also analysed online conversation in the context of cyberbullying and trolling (Yin et al. Reference Yin, Xue, Hong, Davison, Kontostathis and Edwards2009; Cheng et al. Reference Cheng, Bernstein, Danescu-Niculescu-Mizil and Leskovec2017; Kumar, Cheng, and Leskovec Reference Kumar, Cheng and Leskovec2017; Liu et al. Reference Liu, Guberman, Hemphill and Culotta2018). Significant efforts have been made towards conducting shared tasks to increase community participation and benchmarking towards the task of detecting toxic behaviours. These include Zampieri et al. (Reference Zampieri, Malmasi, Nakov, Rosenthal, Farra and Kumar2019, Reference Zampieri, Nakov, Rosenthal, Atanasova, Karadzhov, Mubarak, Derczynski, Pitenis and Çöltekin2020) and Basile et al. (Reference Basile, Bosco, Fersini, Nozza, Patti, Rangel Pardo, Rosso and Sanguinetti2019) among others. There are various taxonomies for both overt and covert types of toxic conversations. We refer the reader for a detailed analysis of taxonomies to Vidgen et al. (Reference Vidgen, Harris, Nguyen, Tromble, Hale and Margetts2019).

Previous work has studied different aspects of toxic online conversation, including the linguistic variation, such as understanding linguistic changes in user behaviour in the context of community norms (Danescu-Niculescu-Mizil et al. Reference Danescu-Niculescu-Mizil, West, Jurafsky, Leskovec and Potts2013) and understanding the influence of community norms in the context of community moderation (Chandrasekharan et al. Reference Chandrasekharan, Samory, Jhaver, Charvat, Bruckman, Lampe, Eisenstein and Gilbert2018). Their study indicates the presence of macro-, meso- and micro-norms that are specific and unique to particular subreddit communities and modulate different types of disparate conversation. Norms play an important role in moderation of online communities. In our work, we are interested to explore how would a general populace perceive such moderated content. This helps identify potentially outward-facing moderation strategies.

2.2 Datasets

A variety of online content datasets have been created for studying toxicity in online conversations. However, the samples in most existing datasets are predominantly collected with the purpose of building and evaluating supervised machine learning models, to predict toxicity, rather than to obtain a deeper understanding of the topic. Most of the datasets have samples that consist of predominantly textual comments and with sample sizes ranging between 500-500,000.

We briefly describe some of the most popular datasets that have been used for studying toxic language and building predictors. We restrict ourselves to English language datasets and refer the reader to Madukwe, Gao and Xue (Reference Madukwe, Gao and Xue2020), Poletto et al. (Reference Poletto, Basile, Sanguinetti, Bosco and Patti2020), Vidgen and Derczynski (Reference Vidgen and Derczynski2020) for a more comprehensive review of these and other collections. Using Yahoo News Groups, Warner and Hirschberg (Reference Warner and Hirschberg2012) present a dataset based on comments and hyperlinks from the American Jewish Society. The comments are manually annotated using seven categories: anti-semitic, anti-black, anti-Asian, anti-women, anti-Muslim, anti-immigrant and other forms of hateful behaviour (anti-gay, anti-white etc.). Djuric et al. (Reference Djuric, Zhou, Morris, Grbovic, Radosavljevic and Bhamidipati2015) use comments from Yahoo Finance which are annotated by crowd-workers for the presence or absence of hateful comments, while Nobata et al. (Reference Nobata, Tetreault, Thomas, Mehdad and Chang2016) use comments from Yahoo Finance and Yahoo News annotated for abuse, simply whether they are abusive or clean, through crowdsourcing.

Using Twitter as platform, Burnap and Williams (Reference Burnap and Williams2016) investigate aspects of cyber-hate, with an emphasis on a set of triggers including sexual orientation, race, religion and disability on tweets. The dataset was annotated by crowd-workers who were answering if the sample is offensive or antagonistic in terms of the aforementioned triggers. Waseem and Hovy (Reference Waseem and Hovy2016) introduce a dataset with tweets annotated by the authors and validated by experts, where the focus was on annotating tweets that showcase racism or sexism. A further extension enhanced the dataset with annotations from both the experts and crowd-workers and analysed the influence of expertise in the annotation process (Waseem Reference Waseem2016). Reference Founta, Djouvas, Chatzakou, Leontiadis, Blackburn, Stringhini, Vakali, Sirivianos and KourtellisFounta et al. (2018) present a large dataset of 100,000 tweets, focused on annotations of abusive, hateful and spam-based categorisation by crowd-workers. The three categories are the result of an iterative preliminary analysis, where the authors experiment with a wide variety of often interchangeable labels, to distinguish the relationship between them and select the most representative. The various labels are either eliminated or merged, through statistical analysis, to produce a smaller robust set which covers a range of abusive behaviours.

In terms of datasets including modalities other than text, Gomez et al. (Reference Gomez, Gibert, Gomez and Karatzas2020) use Twitter to curate a multimodal – image and text-based – meme dataset, where memes are annotated by crowd-workers on six categories: racism, sexism, homophobia, religion-based hatred and attacks on other communities. A similar work by Kiela et al. (Reference Kiela, Firooz, Mohan, Goswami, Singh, Ringshia and Testuggine2020) presents a synthetic meme dataset, where the focus is on categorising samples as multimodally hateful, in which case the image and the associated text are hateful, or unimodally hateful, in which case either the image or the text is hateful.

A few recent works have focused on studying toxicity in a contextual setting, we here list the most prominent works. One of the earliest works to study the importance of context is by Gao and Huang (Reference Gao and Huang2017), where the authors present an annotated corpus constructed using articles and the corresponding discussion on a news website. This work considers the preceding discourse and the corresponding news item as relevant context. In Qian et al. (Reference Qian, Bethke, Liu, Belding and Wang2019), the authors collect a dataset of hateful comments and crowdsource interventional responses to these comments, from a small set of subreddits. The annotation task aims to identify toxic comments, as well as create human-written responses to the total conversation, which would intervene to the discussion and hold back hate. The annotations are collected through crowdsourcing, and the contextual intent is also provided to the annotator, in matters of title and content from the original submission. Voigt et al. (Reference Voigt, Jurgens, Prabhakaran, Jurafsky and Tsvetkov2018) release a large-scale raw dataset (not annotated) with Reddit comments and the associated user’s gender. Pavlopoulos et al. (Reference Pavlopoulos, Sorensen, Dixon, Thain and Androutsopoulos2020) also create a context-aware toxicity dataset from conversational comments from Wikipedia talk pages discussion pages. The authors are interested in similar questions as to whether context can modulate human judgements of toxicity and the utility of context on machine learning models. However, the contextual information is local and does not take into account global subject content under discussion. Shen and Rose (Reference Shen and Rose2021) investigate the influence of annotators’ political beliefs and familiarity of the source towards annotations of identifying political orientation of Reddit posts. Chung et al. (Reference Chung, Kuzmenko, Tekiroglu and Guerini2019) present a dataset that collects both hateful and toxic messages along with potential repairs that can provide counter-narrative information with fact-based information and non-offensive language to de-escalate hateful discourse. Ljubešić, Fišer and Erjavec (Reference Ljubešić, Fišer and Erjavec2019) present a dataset in both Slovenian and English annotated for a variety of annotations for socially unacceptable discourse. The dataset consists of conversational content from Facebook platform, such that the entire discourse thread is presented to the annotator during annotation for each potentially toxic comment.

We note that the predominant approach in curating samples from social media platforms is by using content that contains words from a small seed lexicon of hateful words. Most datasets focus on annotating comments, where the annotators are not given any context or additional information, such as what the comment refers to, or any history of conversation. The comments are usually taken as an independent entity. Even in the two multimodal datasets, the sample is a unique piece of content which, in this case, happens to have both image and text in it. Our dataset is based on comments from Reddit, where we focus on understanding the perception of toxicity. In contrast to other datasets, our work is a controlled study on how the labels change in the presence and absence of different types of context. We are also interested in understanding the influence of demographic information on annotations.

3.1 Definition of toxicity

Previous work in this domain has focused on fine-grained distinctions of toxic content in the form of hate speech, abusive language and other profanity-related terms. We note, however, that the definitions used for the fine-grained aspects of toxicity have been fairly ambiguous and have been used interchangeably (Founta et al. Reference Founta, Djouvas, Chatzakou, Leontiadis, Blackburn, Stringhini, Vakali, Sirivianos and Kourtellis2018; Vidgen et al. Reference Vidgen, Harris, Nguyen, Tromble, Hale and Margetts2019). In addition, recent work has studied the prevalence of biases due to annotation artefacts in datasets, including those on toxic online content (Davidson et al. Reference Davidson, Bhattacharya and Weber2019; Sap et al. Reference Sap, Card, Gabriel, Choi and Smith2019b). Among the reasons for the presence of such biases, one prominent aspect is the complex and stratified definitions of fine-grained forms of toxic online content. It has been shown that complex definitions need experts or trained annotators and may yield biased annotations, when seeking annotations from crowd-workers (Waseem Reference Waseem2016).

In this paper, we are interested in a simpler and broader definition of toxic content, as we focus on capturing the varying perceptions of toxicity. To this end, we conducted various pilot experiments on the crowdsourcing platform with a few samples to study annotators’ responses while varying the toxicity definition. Our goal was to estimate the difficulty of content annotation, based on a variety of definitions. After five rounds of experiments, we observed that a broader and simpler definition was conducive to annotations and also received positive feedback from the crowd-workers. We thus settled on the definition of toxicity proposed in Wulczyn, Thain and Dixon (Reference Wulczyn, Thain and Dixon2017) and Davidson et al. (Reference Davidson, Warmsley, Macy and Weber2017): “any form of content that is hurtful, derogatory or obscene, such that it can hurt a person either physically or emotionally, for example content threatening a person or spreading hateful messages against a person or a group”.

3.2 Contextual multimodal content

A fundamental concept of online social media platforms is the interaction among users and their use as conversational platforms, typically through comments or replies to original submissions. Therefore, the contextual aspect of these interactions should be considered in social scientific research, especially given the nature of toxicity. This has also been emphasised in previous work, such as Vidgen et al. (Reference Vidgen, Harris, Nguyen, Tromble, Hale and Margetts2019). In the previous section, we define toxicity as content that can hurt a person or a group, hence we presume that there is usually a target.

Despite the prominent role of context, it is not common in previous work to consider contextual information when annotating content. This is one of the most important contributions of our work – studying the effect of context on toxicity annotation while also focusing on considering demographic factors influencing annotations.

Furthermore, social media content exists in multiple modalities including textual, visual (such as images or videos) and audio forms. Various platforms support different modalities, and some modalities can be more prominent in conversations depending on the platform, such that the textual modality may not necessarily play the most important role. Both Kiela et al. (Reference Kiela, Firooz, Mohan, Goswami, Singh, Ringshia and Testuggine2020) and Gomez et al. (Reference Gomez, Gibert, Gomez and Karatzas2020) present multimodal datasets that contain memes or pictorials with textual information as samples. However, we classify these as a different type of content, which has a very different communication intent.

To measure the effect of context, we run two parallel experiments where the only difference is the availability of context during annotation for one of the experiments. In the first experiment, we provide comments obtained from Reddit to a group of annotators and ask the participants to annotate whether they believe the comment (which is only comprised of text) is toxic under our definition. For the second experiment, we provide the same samples and annotation setup, only this time participants have access to the additional context from the submission the comments belong to. This context includes the title (in the form of text), along with the image of the original submission, and the annotators are asked to decide upon the toxicity of the sample under the same definition, but now given the context. We restrict our definition of conversational context to this content since it is what prompts users to post comments on the platform. The context thus acts as the communication intent and we hypothesise that this adds important information to decide on the toxicity of the sample; however, this may further increase the subjectivity in analysing the samples due to the additional source of information. We present an overview of our experimental setup in Fig. 1 and instances of the samples in Fig. 2.

Figure 1. Overview of the experimental setup. In order to participate in the experiments, workers first need to pass a series of qualifications. Once they successfully qualify, they are separated into control groups based on the confounding variables of each experiment. Participants are then given access to the annotation platform and, based on the experiment type and provided information, are asked a series of questions related to toxicity. The final annotation scheme is rich in information, approaching toxicity in a holistic manner.

Figure 2. Examples from the dataset.

In the rest of the paper, we consider the two different set of experiments as two different projects and represent them with the letters C and CTM, where C stands for Comment-only tasks, that is no context was presented to the annotators, and CTM stands for Comment-Title-Media, that is samples that are enriched with contextual information.

3.3 Sample selection

Our dataset is based on conversations from the Reddit platform. Most existing works use a form of seed lexicon or a set of seed topics to build the dataset. In this study, we base our collection on the strategy from Bamman et al. (Reference Bamman, O’Connor and Smith2012).

We gathered a list of 400 most popular subreddits based on the comment traffic on the subreddits.Footnote ^e Like Twitter and other social media services, Reddit provides developers with open APIs to build services, including access methods to storing a snapshot of a post and the corresponding comments. We also exploit Pushift API (Baumgartner et al. Reference Baumgartner, Zannettou, Keegan, Squire and Blackburn2020) which additionally provides access to retrieve historical changes in Reddit data. This functionality allows to especially track comments and posts that were made on the platform which were then subsequently removed by the subreddit moderator. We note that the associated norms may not always be relating to the ‘toxic’ content, for example a subreddit dedicated for discussion of football-related news may consider comments and posts that discuss political issues to be breaking the norms of that specific subreddit.

In order to build our dataset, we queried Reddit at fixed intervals using both Reddit and Pushshift API to retrieve a sample of messages. Every message in our corpus was initially written between 1st October 2019 and 30 October 2020. We then use Pushshift API to identify messages that have been deleted by the moderator. The API returns the message “[removed]” for content that is removed by the moderator of the subreddit, while it returns the message “[deleted]” for content that is deleted by the user themselves. The comment is usually removed by the moderator as it breaks one or more of the norms associated with the subreddit. Of these, we focus only on content of the type where we have access to a fully formed title, an associated image and the corresponding comment that is removed. In this version of our dataset, we focus on top-level comments. In this study, we deliberately choose top-level comments in order to remove possible confounding due to the discourse. Therefore, every sample in our dataset consists of a title, an associated image and a removed comment.

We observed that there are a substantial set of the samples that were not appropriate for the purposes of annotation. These were messages that were strictly not textual (such as hyperlinks), or the posts were typically inappropriate (such as constituting gore and other bad content). Over the course of the whole year, we sampled over 60,566 piece of toxic content. From this, we randomly sampled, without replacement, a subsample of over 2500 samples. Of these samples, the exclusion criteria for removal were as follows:

• • Content devoid of comments that were not predominantly text-based (i.e., comments that mostly comprised of hyperlinks/a single token)

• • Samples that broke our institution’s ethics guidelines: these were specifically related to imagery that consisted of visually harmful content, such as content showcasing extreme violence or animal abuse or sexually explicit forms of content.

• • We also removed content where either the title or the comments were not in English.

We then carefully curated a set of 1500 samples that were deemed appropriate for the crowdsourcing platform.Footnote ^f We also note here that the set of subreddits considered consist of subreddits that have left-leaning, right-leaning and politically neutral subreddits.

3.4 Annotation setup

We use Amazon Mechanical Turk (AMT)Footnote ^g for collecting annotations. AMT is a crowdsourcing website where individuals (or requesters) can collect annotations using a distributed set of crowd-workers. It has been broadly favoured by the research community for data annotation tasks as it has a sufficiently expressive interface for controlled and randomised experiments. Annotations can be obtained relatively in a short period of time due to its large workforce.

AMT offers a set of tools for a multitude of projects. In our work, we exploit some of these existing API tools to implement the final custom platform, which can be seen in Figs. 3(a)– 4(b). Our task consists of three parts: the instructions, the practice trials, and the main annotation task.

Figure 3. Our annotation platform for the comments-only experiment.

Figure 4. Our annotation platform for experiments with comments and with the additional conversational intent.

The instructional part contains information which details the task requirements. Here, we briefly describe the task on a main page, but also offer more details as well as a video and a set of example annotations.

The other two parts of the platform are task-dependent: comment-only (C) versus context-rich (CTM) tasks. For the samples that relate to the comment-only tasks (C) each annotation sample consists only of the textual comment and no other information (Fig. 3(a)). On the other hand, the context-rich tasks include annotation samples with the title of a post, an image originally associated in the post and the comment related to this post (Fig. 4(a)).

The annotation scheme also changes for the two cases. Specifically, the rudimentary annotation framework remains identical in both cases, that is identifying whether the sample is toxic. The details, however, change for the latter, where annotators are asked to judge not only the comment but also the context for their toxicity. In both cases, the first annotation task is a simple classification of whether the sample, as a whole, can be considered toxic, that is based on the specific definition of toxicity (described in Section 3.1) workers are asked to judge if the whole sample is toxic. If for any reason, they are unable to decide or have other issues with the sample, they have the option to skip. Even though workers rarely opted to skip in our pilot experiments, we received feedback from the annotators expressing the importance of this option. A participant, for example, mentioned “Having an option to skip is a very important button I’m glad you included, because for some comments I feel like I don’t have enough authority to really say whether or not something is toxic”. [sic].

We first present the common setup across both C and CTM experiments. Following the toxicity judgement, we are interested in understanding the degree to which annotators are confident in their decision, so we ask them to provide an estimate from a scale of 0 to 5. We also ask annotators to highlight the portion of the comment text that makes the sample to be perceived as toxic. This allows us to study reasons (i.e., rationales) that led the annotator to identifying the sample as toxic.

During our data curation, we observed that a significant number of comments are sarcastic. As the samples in our dataset are mostly of subtly harmful content, the use of sarcasm to mock or convey contempt is a common observation. Previous work has also suggested that such behaviour is more frequent in subtle and covert forms of abusive and hateful language (Malmasi and Zampieri Reference Malmasi and Zampieri2018). Further, it has also been pointed out that it is difficult to get correct judgements in such cases as it requires a broader knowledge and understanding of the specific communities and commonsense assumptions (Nobata et al. Reference Nobata, Tetreault, Thomas, Mehdad and Chang2016). We refer the reader to Vidgen et al. (Reference Vidgen, Harris, Nguyen, Tromble, Hale and Margetts2019) for a more comprehensive survey on the challenges of detecting and annotating subtly harmful content. To allow for such analysis, we further ask the annotators to decide whether the sample was seen as sarcastic.

In our setup, the above format remains consistent across experiments. In addition to the previous questions, for the CTM experiments we further ask the annotators whether the comment and/or context are toxic. Note that, in this case either any of the comment or the context or both can potentially be annotated as toxic. Finally, when the content (in the form of comments) or the context (in the form of titles and images) are considered toxic, we want to further identify the parts of the sample that make it toxic. We also ask the participants to highlight the parts of text and the image, which led them to that opinion. Examples annotated as toxic can be seen in Figs. 3(b) and 4b.

We also note that the supplementary questions regarding the annotator rationales are only visible if the sample is considered toxic. This conditional nature of the annotation scheme was deliberate to avoid confirmation biases. In our pilot experiments, we observed that the workers might be prone to exaggerate labelling content as toxic if they see all the questions and requests a priori; thus, a set of questions focusing on the analysis of toxic parts could lead them to annotate the sample as toxic.

After the experiment, we ask annotators to optionally provide feedback. We remark that the feedback presented by all the workers in all our experiments was that they had a positive experience. Workers have praised the clarity of the instructions and how the platform facilitates the task; they judged the task as straightforward.

3.5 Qualification of annotators

To participate in our annotation experiments, workers first had to qualify, answering several questions that are similar to the samples in the experiments. More specifically, the initial requirement is to ensure all workers understand the adult nature of the tasks. The participants are required to be over 18 years old and have to consent to work on assignments that may include adult content.

We also wanted to establish a baseline of common understanding of the task and prime the annotators with regard to specific definition of toxicity (Section 3.1). Therefore, as quality control, we required all workers to answer whether a set of eight examples are toxic or not. These examples are provided in Table 1. Qualifications required answering at least seven out of the eight examples correctly.

Table 1. The examples used to qualify workers. All the examples have been collected during the Black Lives Matter movement during the summer months of 2020

We also collect demographic information for every annotator. We request all participants to complete a demographics survey and provide their socio-economic information to understand their background. Specifically, our survey consists of questions on the annotator’s gender, age group, annual income range, education level, nationality, ethnicity, country of residence and political orientation. We restrict ourselves to self-reported political orientation on a simple scale from left to right. Most political charts, such as Nolan’s chart (Eysenck Reference Eysenck1968) or Mitchell’s chart (Mitchell Reference Mitchell2007) are focused on either American or European political contexts. Our goal was to use broader scales to accommodate other parts of the world. Our decision was inspired by experimental research in political philosophy by Kroh (Reference Kroh2007) and Zuell and Scholz (Reference Zuell and Scholz2016), which has been shown to capture general trends using a 10- or 11-point scale. We ask the annotators to rate how they self-identify themselves politically, from 0 to 10 (0 being left and 10 being right). We note that the annotators cannot pass the qualification round if they have not filled the demographics survey. We use some of this demographics information to employ the controls we describe in Section 5.

4.1 Vote aggregation strategies

We consider several aggregation methods. Starting with a simple majority vote (MV), that is the most commonly observed value among annotations, which is later enriched with ‘confidence in decision’ information to produce high-confidence majority votes (HCMV). These correspond to the most frequently agreed votes, taking only annotations where the annotators declared that they are very confident about the annotation (confidence score of 4 or 5, on a scale of 0–5). It is noted that, for the rest of the paper, when referring to MV and HCMV both values show the percentage of annotations that indicate toxicity. We also wanted to get an understanding of the annotators who are most confident about their annotations, therefore we calculate the rate of fully confident annotations, regardless of toxicity.

As toxicity is a very subjective concept, we also considered investigating how many of the non-toxic samples, as decided by the majority, have at least one person disagreeing. Therefore, we also calculate at least one toxic vote (ALOTV) score.

To look at sarcasm, we follow the same procedure as presented above where we calculate the majority sarcasm vote (SMV) between annotations of a sample, as well as the number of samples with at least one sarcasm vote.

Finally, we also study the inter-annotator agreement. We note that we do not expect high agreement, this is due to the prevalence of subjectivity in the task, the inherent biases we observe in our controlled experiments, and the potential random noise effects due to the between-subject experimental design. We measure agreement using P(A), the percentage of cases where annotators agree (100% refers to full agreement) and Fleiss Kappa (Fleiss Reference Fleiss1971), a common inter-annotator agreement metric which also takes into account the probability of agreement by chance.

4.2 Summary of results

Collectively, we have a total of 21,000 annotations for the 1500 total samples; however, it is noted that there are a varying number of annotations per sample, from 12 to 18, depending on the batch it belongs to. Samples annotated while controlling for political orientation, for example, have been examined by three different groups of workers, twice each (once without and another including the context), hence a total of 18 times.

According to the majority vote, around 30% of the samples are annotated as toxic; however, we note that nearly all samples (94%) have at least one worker suggesting they are toxic (ALOTV). Furthermore, 80% of all majority votes (and 33.5% of toxic votes in specific) are attributed with high confidence, as their annotations show a median confidence score higher than 4 out of 5. We also observe that a small percentage of samples are annotated to be sarcastic (SMV), even though the total rate of sarcasm-positive answers is more than 30%. Finally, only half of the total annotations are marked as fully confident (confidence score = 5), showing the difficulty of individuals to decide upon such a subjective matter as toxicity.

5.1 Controlling for gender

Recent research and surveys (Ipsos Reference Ipsos2016; Capezza et al. Reference Capezza, D’Intino, Flynn and Arriaga2017) have highlighted a consistent pattern where the perception of toxic content varies between genders. Further, practitioners who use machine learning and natural language processing for the detection of toxic online content have considered using gender as an additional feature and it has been seen to improve predictive performance in such models (Waseem and Hovy Reference Waseem and Hovy2016). With these studies as background, we conduct a controlled experiment in order to understand the degree to which gender can potentially act as a confounding factor in perceiving toxicity.

During the preliminary analysis of data, we observed that the self-reported gender identities of our workforce were predominantly represented by males and females, all other genders were considerably under-represented on AMT. We therefore restrict our experiments to the dominant representation and use the self-reported gender to separate males from females. Even though we allow other individuals who self-report as any other gender to participate in both tasks, our final dataset consists of only the two most representative genders in these tasks, therefore we characterise the control groups hence referred to as males and females. We will release the full dataset with the all annotations on a public platform for full analysis.

Overall, the total toxicity rate of all annotations controlled for gender is 51.45%. There are 246 toxic samples, and approximately 50% of the batch is annotated as toxic according to MV. There are 12 annotations for every sample, therefore we consider the final decision of the toxicity to be fairly robust. When we consider only high-confidence annotations, most of the votes over samples remain the same and only a few change (approximately 90% remain intact). Additionally, nearly 50% of the toxic votes are attributed with high confidence (according to HCMV). We note that more than half of the annotators exhibit full confidence (5 out of 5) in their annotations.

We further note that most samples contain at least one annotation as toxic (96% of the samples with ALOTV equal to toxic). We also note that samples are not frequently annotated as sarcastic: only 15% of the samples appear to have sarcasm as the majority vote (SMV), even though the total sarcasm rate of all 6000 annotations is more than 30%. We observe that half of these sarcastic samples are also toxic.

Contextual & Control Group Findings. Table 4 presents an overview of the aggregations related to toxicity. For the case of the simple majority vote (second column), we observe that the proportions of annotations annotated as toxic increases with the provision of context. However, when we consider annotations that are highly confident on the other hand, we do not observe any changes. We also observe that context plays an important role for females. Aggregated annotations by the male group, on the other hand, do not seem to be affected by the presence of context.

Table 4. Annotation summary of toxicity for gender-controlled experiments. The third column exhibits the rate of fully confident annotations, regardless of toxicity. The abbreviations F and M stand for Females and Males, accordingly

Samples that have at least one toxic annotation also consistently increase considerably in the presence of context, from around 70% to around 80%. We observe that the introduction of context also introduces subjectivity making individuals disagree more when compared to the annotations for which comments only are provided, where the tendency is to agree on no-toxicity. We also perform a Wilcoxon signed-rank test (Wilcoxon Reference Wilcoxon1992) in order to assess the differences between male and female annotations by focusing on MV and annotator confidence. We consider the confidence as a continuous variable between 0–5. We observed that the annotations are significantly different among comments with context setting (with $p\lt 0.05$ ) between the two groups.

Considering the summary of sarcasm annotations (presented in Table 5), we again observe that the there is a marked difference between the C and CTM for female group for annotations of sarcasm. This seems to be reflected in all of the settings. It also seems that sarcasm is correlated with toxicity for the female group. None of these effects are noticed as strongly on the male group.

Table 5. Annotation summary of sarcasm, for gender-controlled experiments. The third column indicates the proportion of the sarcastic samples that are also toxic (according to MV). The last column indicates the non-aggregated (for all annotations) rate of sarcasm

Annotator Agreement. We also compute the per cent agreement and Fleiss Kappa to obtain the inter-annotator agreement. We observe that for the whole experiment, the per cent agreement – P(A) – is 0.59 and within each annotation task this number rises to up to 0.65. With the introduction of context, however, we see a drop in agreement rates. This drop is even more apparent when looking at Fleiss Kappa scores – $\kappa$ . The agreement among annotators when they only see comments is 0.36, but it decreases to 0.2 with context. Further investigating into each group, the trend is the same for all cases, both for P(A) and for Fleiss Kappa.

5.2 Controlling for political orientation

Previous work in the area of political science and the area of political psychology has studied the question of the influence of political orientation on the perception of both hateful language and free speech (Davis and Silver Reference Davis and Silver2004; Lindner and Nosek Reference Lindner and Nosek2009). Rossini (Reference Rossini2019) studies the conversational dynamics when it comes to political discourse on online platforms. In this work, we are interested in understanding the differences in perception of similar types of content due to political orientation.

Table 6. Annotation summary of sarcasm, for political orientation-controlled experiments. The second column indicates how many of the sarcastic samples are also toxic. The last column indicates the non-aggregated (for all annotations) rate of sarcasm

We recognise that it is very difficult to cover the broad spectrum of political beliefs and distinguish political orientation, especially considering that AMT workforce is spread worldwide. To avoid any definitions, we let annotators decide how they identify politically, on a scale of 0–10. Zero, in our case, indicates left-leaning political beliefs and 10 is right-leaning. We make the scale as such to provide a solid middle ground, an option for those who do not identify politically with either side and consider these annotators as a separate group. Therefore, we consider three control groups for this experiment – left-oriented, right-oriented and politically neutral annotators.

We randomly select a set of 500 samples from the 1500 samples main set, without replacement. Note that none of the samples that appear in Section 5.1 are considered for this experiment. It is interesting to observe that the rate of samples with at least one toxic annotation is at the same level as in Section 5.1 (ALOTV 94%). High-confidence toxic aggregations decreased (HCMV 18%), while fully confident annotations also decreased by a narrow margin (42.8%), and we tend to observe that annotators tend to give annotate with lower confidence. Lastly, this pattern is also consistent in answers for sarcasm (in Table 6), where sarcasm rate is fairly high (a little less than 30%) and nearly all samples have at least one individual saying they are sarcastic, yet the total set of sarcastic samples is very small (32 units or 6.5%).

Contextual & Control Group Findings. We present our results in Table 7, we observe that right-leaning individuals tend to overlook toxicity compared to the other groups. The tendency of left and centre is to annotate around a quarter of the samples as toxic, a rate that further increases with the introduction of context. The numbers we see in the MV column for these groups are almost double the rates of the aggregated scores discussed earlier. This shows major differences between the perceptions of the groups (predominantly the right-leaning v/s others). We also see that the right-leaning sub-group tends to disagree with the other groups. We also observe similar trend for the right-leaning sub-group, where we see that for the comment-only tasks we see 10.2% toxic samples are annotated as toxic. However, with conversational context (or the intent), this percentage drops to a staggering 4.8%.

Table 7. Annotation summary of toxicity, for experiments controlled about political orientation. The third column exhibits the rate of fully confident annotations, regardless of toxicity. The abbreviations L, C and R stand for Left, Centre and Right political orientations, accordingly

In addition, for ALOTV results we find that while the total rates relatively drop compared to Section 5.1. We observe that there is a high degree of consensus between the right-leaning annotators. We also notice that the right-leaning annotators consider more than 50% of samples to be totally non-toxic which increases to approximately 70% with the added context.

Lastly, it is interesting to note the divergence in fully confident annotations. With the comment-only annotation tasks, left-leaning annotators are half the number of times fully confident about their answers, whereas the other two show much less confidence. Once the context is introduced, however, the centre-leaning group shows a sudden increase to 56.5% fully confident annotations, although the other two rates drop slightly.

With the Wilcoxon signed-rank Test on MVs, we observe significant differences in annotations across all the groups (with $p\lt 0.01$ ). This is true for both the comments and comments with context settings.

Overall, these findings strongly indicate that there are extreme differences in the perception of online content between people with different political orientations, especially considering that our data originates from a platform widely used for political expression. We believe this is an important finding and expect further research in this area from the community. We also hope this finding will influence practitioners in computational social sciences to re-examine some of the experimental designs, especially in the context of automated detection of toxic online content.

Annotator Agreement. Regarding the inter-annotator agreement scores of the Political Orientation experiment, we see similar behaviour as Section 5.1. Agreement rates are lower than the ones from the previous experiment, but the tendency remains the same. Additionally, looking at each individual group, all scores follow the same pattern, even though there are slight differences in similarities within groups.

5.3 Controlling for age

Finally, with the last batch of 500 samples, we perform an experiment where we control for age. Previous research has examined the influence of age in terms of the perception of online privacy, with reports of marked difference among the age groups (Zukowski and Brown Reference Zukowski and Brown2007). However, there is much less research in the context of toxicity judgements regarding the influence of age. This has also been discussed in the context of recent US House Judiciary Subcommittee on Antitrust law, where senators of different age groups had a markedly different understanding of online harms.Footnote ^h

In this experiment, we want to investigate the influence of age towards perceiving toxicity. Our hypothesis is that younger annotators will have a different perception of toxicity. We believe there is a strong effect of the Internet culture on younger individuals, as they grew up with the Internet and social platforms incorporated in their everyday lives in an increasingly ubiquitous manner. The Internet had a principal role in the development of their belief system; hence, we expect this to have an impact on how they perceive toxicity.

In our annotation experiments for the pilot study, the study in Sections 5.1 and 5.2, we observed that a large proportion of approved annotators fell in the range of 20–30 years old (about 40%) and the range of 30–40 years old (about 42%), and smaller proportions of people below the age of 20 or above the age of 40. This observation led us to considering a split of the age group in the range below 30 and the range above 30 for our experiments to investigate the effect of age in the perception of toxic online content.

Overall, we observe a total toxicity rate of more than 30%, which means that a third of all annotations for this batch were annotated as toxic. Each sample was annotated collectively by 12 annotators, and a quarter of the batch (123 samples) were judged as toxic (MV). Once again, we notice that most of the samples have at least one annotation stating they are toxic (ALOTV 92.6%). Additionally, we also notice that half of the annotations are fully confident (49.85%). Finally, as an aggregate, there are only a few sarcastic samples (47 out of the 500, or 9.4%), although total sarcasm rate is once again relatively on the higher side (33.2%).

Our experiment suggests that the differences between the two age groups, with regard to the perception of toxicity, are seemingly weak. To further understand the dynamics, we now look at each group results separately and with reference to the effect of context.

Contextual & Control Group Findings. Our results are presented in Table 8. We observe that annotators with age above 30 years tend to annotate more samples as toxic, compared to their younger counterparts. We also note that they predominantly annotate with higher confidence (column 4). We also observe that the presence of context in both groups leads annotators to perceive more samples as toxic. This effect is seemingly stronger on younger participants, for whom we notice an increase in HCMV and ALOTV – from approximately 30 and 50% in the case of non-contextual tasks (C) going up to 43.6 and 72.4% when associating context (CTM), accordingly. In the case of highly and fully confident annotations, there is a large contrast among groups, regardless of the presence of context. Annotators under 30 years old tend to display low confidence. While those aged 30 or above showing the opposite – the highest confidence. The conversational context causes a slight drop in both.

Table 8. Annotation summary of toxicity, for age-controlled experiments. Similarly with the two previous cases, the third column exhibits the rate of fully confident annotations, regardless of toxicity. As described before, the splitting point for the groups is the age of 30; therefore, annotators of the first group are younger than 30 years old and of the second are thirty years old or over

Finally, as seen in Table 9, the annotation of sarcasm rises considerably with context, more so for the group of 30 and over. It is likely that this particular group has difficulties understanding sarcasm by seeing only the comments, which probably explains why there are almost double the number of samples annotated as sarcastic in the presence of the context. They also seem to associate sarcasm with toxicity more than their younger counterparts.

Table 9. Annotation summary of sarcasm, for age-controlled experiments. The third column indicates how many of the sarcastic samples are also toxic. The last column indicates the non-aggregated (for all annotations) rate of sarcasm

Based again on the Wilcoxon signed-rank test on the MVs annotations, we observe that annotators under 30 and over 30 do not show significantly different annotations (with $p{\gt }0.05$ ). This is true for the both the comments and comments with context settings. These observations indicate that despite the marked differences in understanding toxicity and sarcasm among the two groups, there are stronger differences in degrees of belief, as over 30 annotators show stronger confidence and higher cohesion.

Annotator Agreement. Finally, we compute the inter-annotator agreement across the Age experiment. Table 3 suggests that context results in reduced agreement among the annotators and leads to a bigger variability in terms of the annotations.

6.1 Contextual toxicity

We now present our analyses on the annotations of the contextual components. As we described in Section 3.1, when annotators are provided with the context, we also request that the annotators highlight specific components in the sample that makes it toxic. Over the entire dataset, 81.2% of the toxic samples have their comment annotated as toxic and only 39.8% have the context annotated. Additionally, in more than half of the toxic annotations, it is only the comment that is found toxic (57.2% to be exact). Only 24% of cases have both comment and context toxic, and for 15.7% it is exclusively the context that is toxic. Despite the latter percentage being much lower than others, we note that it still is a substantial set of annotations which would have been considered neutral if the additional information was absent. Also, we note that titles and images played no role in the collection of the data, which was based only on comments. We expect these numbers to be reflective of real-world scenarios. Indeed, most of the state of the art in automated detection of toxic online comments, unfortunately, seem to ignore contextual cues (Vidgen and Derczynski Reference Vidgen and Derczynski2020).

Distinguishing the samples among the various controls results in comment toxicity range at a similar 75–85%, indicating that our dataset is certainly rich in toxic comments. For the case of context, there seems to be a higher amount of variance. The experiments that control for gender show a rate of 52.7%, showing that more than half of the toxic annotations display a toxic bit of context, while the same rate for female exclusive group indicates it to be approximately 58.2%.