4.1 Weaknesses in data

Spurious correlations: Correlations between input data and the expected prediction are ‘spurious’ if there exists no causal relation between them with regard to the underlying task but rather they are an artefact of a specific dataset. They are also referred to as ‘(annotation) artefacts’ (Gururangan et al. Reference Gururangan, Swayamdipta, Levy, Schwartz, Bowman and Smith2018) or ‘(dataset) biases’ (He et al. Reference He, Zha and Wang2019) in literature.

In span extraction tasks, where the task is to predict a continuous span of token in text, as is the case with MRC, question and passage wording, as well as the position of the answer span in the passage are indicative of the expected answer for various datasets (Rychalska et al. Reference Rychalska, Basaj, Wróblewska and Biecek2018a; Kaushik and Lipton Reference Kaushik and Lipton2018) such that models can solve examples correctly even without being exposed to either the question or the passage. In the ROCStories dataset (Mostafazadeh et al. Reference Mostafazadeh, Chambers, He, Parikh, Batra, Vanderwende, Kohli and Allen2016) where the task is to choose the most plausible ending to a story, the writing style of the expected ending differs from the alternatives (Schwartz et al. Reference Schwartz, Sap, Konstas, Zilles, Choi and Smith2017). This difference is noticeable even by humans (Cai, Tu and Gimpel Reference Cai, Tu and Gimpel2017).

For sentence pair classification tasks, such as RTE, Poliak et al. (Reference Poliak, Haldar, Rudinger, Hu, Pavlick, White and Van Durme2018) and Gururangan et al. (Reference Gururangan, Swayamdipta, Levy, Schwartz, Bowman and Smith2018) showed that certain n-grams, lexical and grammatical constructs in the hypothesis as well as its length correlate with the expected label for a multitude of RTE datasets. For example, the word ‘no’ in the premise occurs more often with the label Contradiction than with the label Entailment in the SNLI and MNLI datasets. This correlation is spurious because although true for the datasets, the appearance of the word ‘no’ in the premise is not indicative of contradiction. For example, ‘No cats are in the room.’ entails ‘No cats are under the bed.’. Similarly, McCoy et al. (Reference McCoy, Pavlick and Linzen2019) showed that lexical features like word overlap and common subsequences between the hypothesis and premise are highly predictive of the entailment label in the MNLI dataset. These correlations, too, are artefacts of the data collection method rather than robust indicators of entailment – for example ‘I almost went to Vienna.’ does not entail ‘I went to Vienna’ despite a high lexical overlap.

Beyond RTE, the choices in the COPA dataset (Roemmele et al. Reference Roemmele, Bejan and Gordon2011) where the task is to finish a given passage (similar to ROCStories) and ARCT (Habernal et al. Reference Habernal, Wachsmuth, Gurevych and Stein2018) where the task is to select whether a statement warrants a claim contain words that correlate with the expected prediction (Kavumba et al. Reference Kavumba, Inoue, Heinzerling, Singh, Reisert and Inui2019; Niven and Kao Reference Niven and Kao2019).

Other data quality issues: Pavlick and Kwiatkowski (Reference Pavlick and Kwiatkowski2019) argue that when training data are annotated using crowdsourcing, a fixed label representing the ground truth, usually obtained by majority vote between annotators, is not representative of the uncertainty, which can be important to indicate the complexity of an example. Sometimes, data annotations are factually wrong (Pugaliya et al. Reference Pugaliya, Route, Ma, Geng and Nyberg2019; Schlegel et al. Reference Schlegel, Valentino, Freitas, Nenadic and Batista-Navarro2020), for example, due to limitations of the annotation protocol, such as when only one ground truth label is expected but the data contain multiple plausible alternatives. In ‘multi-hop’ datasets, such as HotPotQA and WikiHop where the task is to find an answer after aggregating evidence across multiple documents, this process can be circumvented in the case of examples where the location of the final answer is cued by the question (Min et al. Reference Min, Wallace, Singh, Gardner, Hajishirzi and Zettlemoyer2019). For example, consider Figure 2: while initially this looks like a complex question that requires spatial reasoning over multiple documents, the keyword combination ‘2010’ and ‘population’ in the question is unique to the answer sentence across all 10 documents, allowing to find the answer to the question without ever reading Passage 1. The initially complex question can be substituted by the much easier question ‘What is the 2010 population?’ which does not require any reasoning and has a unique answer that coincides with the expected answer to the original question. This is especially true for the multiple-choice task formulation, as the correct answer can often be ‘guessed’ by excluding implausible alternatives (Chen and Durrett Reference Chen and Durrett2019), for example, by matching the interrogative pronoun with the corresponding lexical answer type. Sugawara et al. (Reference Sugawara, Inui, Sekine and Aizawa2018) show that multiple MRC benchmarks contain numerous questions that are easy to answer, as they do require little comprehension or inference skills and can be solved by looking at the first few tokens of the question. This property appears ubiquitous among multiple datasets (Longpre, Lu and DuBois Reference Longpre, Lu and DuBois2021). Finally, Rudinger, May and Van Durme (Reference Rudinger, May and Van Durme2017) show the presence of gender and racial stereotypes in crowdsourced RTE datasets.

Figure 2. Example for a dataset artefact where the requirement to synthesise information from 2 out of 10 accompanying passages can be circumvented by exploiting simple word co-occurrence between question and answer sentence.

There are multiple reasons for data quality issues, among them the carelessness of the annotators, usually hired via a crowdsourcing platform (Brühlmann et al. Reference Brühlmann, Petralito, Aeschbach and Opwis2020). This also provides a possible explanation to the existence of dataset artefacts: as annotators are paid per annotation, they often adapt simple strategies to maximise the output. For example, deriving a contradicting premise by simply negating the hypothesis might lead to the spurious correlation discussed before. Thus, it is important to establish the quality of data during and after collection, by filtering low-quality examples, an increasingly employed practice, as we discuss in Section 5.1.

In any case, these data quality issues diminish the explanatory power of observations about models evaluated on these data: the presence of cues casts doubts on the requirements of various NLU capabilities, if a simpler model can perform reasonably well by exploiting these cues. The situation is similar, when expected answers are factually wrong.

4.2 Model weaknesses

In this section, we discuss whether data-driven approaches to NLU are in fact affected by these data quality issues. We discuss multiple works that reveal their dependence on dataset-specific artefacts. This is further evidenced by their poor generalisation on data that stem from a different distribution than their training data, suggesting that they in fact overfit to the patterns inherent to different datasets rather than reliably learning the underlying task. A suspected reason for this is that state-of-the-art NLP approaches rely increasingly on no-assumption architectures – little to no expert knowledge is encoded into the models a priori and all necessary information is expected to be derived from the data.

Dependence on dataset-specific artefacts: Given the data-related issues discussed earlier, it is worth knowing whether models optimised on this data actually inherit them. In fact, multiple studies confirm this hypothesis (McCoy et al. Reference McCoy, Pavlick and Linzen2019; Niven and Kao Reference Niven and Kao2019; Kavumba et al. Reference Kavumba, Inoue, Heinzerling, Singh, Reisert and Inui2019). This is usually evidenced by the poor performance of models evaluated on a balanced version of the data where the spurious correlations have been balanced. To illustrate it by means of the previous example, a balanced set would have equally many instances with the word ‘no’ appearing with the label Contradiction and Entailment. Note, that the act of balancing the data alters the underlying generative process; thus, this type of evaluation is performed on data outside of the training data distribution; therefore, the same considerations regarding poor out-of-distribution generalisation, discussed in more detail in the following section, apply.

Neural models tend to disregard syntactic structure (Rychalska et al. Reference Rychalska, Basaj, Wróblewska and Biecek2018b,a) and important words (Mudrakarta et al. Reference Mudrakarta, Taly, Sundararajan and Dhamdhere2018), making them insensitive towards small but potentially meaningful perturbations in inputs. This results in MRC models that are negatively impacted by the presence of lexically similar but semantically irrelevant ‘distractor sentences’ (Jia and Liang Reference Jia and Liang2017; Jiang and Bansal Reference Jiang and Bansal2019), give inconsistent answers to semantically equivalent input (Ribeiro, Singh and Guestrin Reference Ribeiro, Singh and Guestrin2018) or fail to distinguish between semantically different inputs with similar surface form (Gardner et al. Reference Gardner, Artzi, Basmov, Berant, Bogin, Chen, Dasigi, Dua, Elazar, Gottumukkala, Gupta, Hajishirzi, Ilharco, Khashabi, Lin, Liu, Liu, Mulcaire, Ning, Singh, Smith, Subramanian, Tsarfaty, Wallace, Zhang and Zhou2020; Welbl et al. Reference Welbl, Minervini, Bartolo, Stenetorp and Riedel2020). For RTE, they may disregard the composition of the sentence pairs (Nie, Wang and Bansal Reference Nie, Wang and Bansal2019).

Poor generalisation outside of training distribution: Mediocre performance when evaluated on RTE (Glockner, Shwartz and Goldberg Reference Glockner, Shwartz and Goldberg2018; Naik et al. Reference Naik, Ravichander, Sadeh, Rose and Neubig2018; Yanaka et al. Reference Yanaka, Mineshima, Bekki, Inui, Sekine, Abzianidze and Bos2019b) and MRC data (Talmor and Berant Reference Talmor and Berant2019; Dua et al. Reference Dua, Gottumukkala, Talmor, Gardner and Singh2019a) that stems from a different generative process than the training data (leading to out-of-distribution examples) reinforces the fact that models pick up spurious correlations that do not hold between different datasets, as outlined earlier. Limited out-of-distribution generalisation capabilities of state-of-the-art models suggest that they are ‘lazy learners’: when possible, they infer simple decision strategies from training data that are not representative of the corresponding task, instead of learning the necessary capabilities to perform inference. Nonetheless, recent work shows that the self-supervised pre-training of transformer-based language models allows them to adapt to the new distribution from few examples (Brown et al. Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever and Amodei2020; Schick and Schütze Reference Schick and Schütze2021).

No-assumption architectures: Note that these weaknesses arise because state-of-the-art end-to-end architecturesFootnote b (Bahdanau, Cho and Bengio Reference Bahdanau, Cho and Bengio2015), such as the transformer (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), are designed with minimal assumptions. As little as possible prior knowledge is encoded into the model architecture – all necessary information is expected to be inferred from the (pre-)training data. The optimisation objectives reflect this assumption as well: beyond the loss function accounting for the error in prediction, hardly any regularisation is used. As a consequence, there is no incentive for models to distinguish between spurious and reliable correlations, so they follow the strongest signal present in data. In fact, one of the main themes discussed in Section 5.3 is to inject additional knowledge, for example, in the form of more training data or heavier regularisation, as a countermeasure, in order to make the optimised model rely less on potentially biased data. For example, models that operate over syntax trees rather than sequences tend to be less prone to syntactic biases (McCoy et al. Reference McCoy, Pavlick and Linzen2019).

5.1 Data-investigating methods

Methods in this category analyse flaws in data such as cues in input that are predictive of the output (Gururangan et al. Reference Gururangan, Swayamdipta, Levy, Schwartz, Bowman and Smith2018). As training and evaluation data from state-of-the-art NLU datasets are assumed to be drawn from the same distribution, models that were fitted on those cues achieve high performance in the evaluation set, without being tested on the required inference capabilities. Furthermore, methods that investigate the evaluation data in order to better understand the assessed capabilities (Chen, Bolton and Manning Reference Chen, Bolton and Manning2016) fall under this category as well. In the analysed body of work, we identified the types of methods discussed in the following paragraphs. In Table 1, we summarise them with their corresponding investigation goal.

Table 1. Summary of data-investigating methods with the corresponding research questions as described in Section 5.1

Partial Baselines are employed in order to verify that all input provided by the task is actually required to make the right prediction (e.g., both question and passage for MRC, and premise and hypothesis for RTE). If a classifier trained on partial input performs significantly better than a random guessing baseline, it stands to reason that the omitted parts of the input are not required to solve the task. On the one hand, this implies that the input used to optimise the classifier might exhibit cues that simplify the task. On the other hand, if the omitted data represent a specific capability, the conclusion is that this capability is not evaluated by the dataset, a practice we refer to as Data Ablation. Examples for the former include training classifiers that perform much better than the random guess baseline on hypotheses only for the task of RTE (Gururangan et al. Reference Gururangan, Swayamdipta, Levy, Schwartz, Bowman and Smith2018; Poliak et al. Reference Poliak, Haldar, Rudinger, Hu, Pavlick, White and Van Durme2018) and on passages only for MRC (Kaushik and Lipton Reference Kaushik and Lipton2018).Footnote c For the latter, Sugawara et al. (Reference Sugawara, Stenetorp, Inui and Aizawa2020) drop words that are required to perform certain comprehension abilities (e.g., dropping pronouns to evaluate pronominal coreference resolution capabilities) and reach performance comparable to that of a model that is trained on the full input on a variety of MRC datasets. Nie et al. (Reference Nie, Wang and Bansal2019) reach near state-of-the-art performance on RTE tasks when shuffling words in premise and hypothesis, showing that understanding the compositional nature of language is not required by these datasets. A large share of work in this area concentrates on evaluating datasets with regard to the requirement to perform ‘multi-hop’ reasoning (Min et al. Reference Min, Wallace, Singh, Gardner, Hajishirzi and Zettlemoyer2019; Chen and Durrett Reference Chen and Durrett2019; Jiang and Bansal Reference Jiang and Bansal2019; Trivedi et al. Reference Trivedi, Balasubramanian, Khot and Sabharwal2020) by measuring the performance of a partial baseline that exhibits an Architectural Constraint to perform single-hop reasoning (e.g., by processing input sentences independently).

Insights from partial baseline methods bear negative predictive power only – their failure does not necessarily entail that the data are free of cues, as they can exist in different parts of the input. As an example, consider an MRC dataset, where the three words before and after the answer span are appended to the question. Partial baselines would not be able to pick up this cue, because it can only be exploited by considering both question and passage. Feng, Wallace and Boyd-Graber (Reference Feng, Wallace and Boyd-Graber2019) show realistic examples of this phenomenon in published datasets. Furthermore, above-chance performance of partial baselines merely hints at spurious correlations in the data and suggests that models learn to exploit them; it does not reveal their precise nature.

Heuristics and Correlations are used to unveil the nature of cues and spurious correlations between input and expected output. For sentence pair classification tasks, modelling the co-occurrence of words or n-grams with the expected prediction label by means of Pointwise Mutual Information (Gururangan et al. Reference Gururangan, Swayamdipta, Levy, Schwartz, Bowman and Smith2018) or conditional probability (Poliak et al. Reference Poliak, Haldar, Rudinger, Hu, Pavlick, White and Van Durme2018; Tan et al. Reference Tan, Shen, Huang and Courville2019) shows the likelihood of an expression being predictive of a label. Measuring coverage (Niven and Kao Reference Niven and Kao2019) further indicates what proportion of the dataset is affected by this correlation. Manually inspecting these correlations can help to identify whether they can simplify the task. For example, if an expression perfectly correlates with a label and covers some subset of the data, then this subset can be solved correctly by relying on the appearance of this expression alone. They are further spurious, if they are not indicative of the underlying task but rather artefacts of the dataset.

These exploratory methods require no apriori assumptions about the kind of bias they can reveal. Other methods require more input, such as qualitative data analysis and identification of syntactic (McCoy et al. Reference McCoy, Pavlick and Linzen2019) and lexical (Liu et al. Reference Liu, Zheng, Chang and Sui2020b) patterns that correlate with the expected label. Furthermore, Nie et al. (Reference Nie, Wang and Bansal2019) use the confidence of a logistic regression model optimised on lexical features to predict the wrong label to rank data by their requirements to perform comprehension beyond lexical matching.

It is worth highlighting that there is comparatively little work analysing MRC data (4 out of 18 surveyed methods) with regard to spurious correlations. We attribute this to the fact that it is hard to conceptualise the correlations of input and expected output for MRC beyond very coarse heuristics such as sentence position (Si et al. Reference Si, Yang, Cui, Ma, Liu and Wang2020) or lexical overlap (Sugawara et al. Reference Sugawara, Inui, Sekine and Aizawa2018), as the input is a whole paragraph and a question and the expected output is typically a span anywhere in the paragraph. Furthermore, the prediction labels (paragraph indices for answer spans or the number of the chosen alternative for multiple-choice type of questions) do not bear any semantic meaning, so correlation between input and predicted raw output such as those discussed earlier can only unveil positional bias. For RTE, in contrast, the input consists of two sentences and the expected output is one of three fixed class labels that carry the same semantics regardless of the input; therefore, possible correlations are easier to unveil.

Manual Analyses are performed to qualitatively analyse the data, if automated approaches as those mentioned earlier are unsuitable due to the complexity of the phenomena of interest or the output space discussed earlier. We posit that this is the reason why most methods in this category concern analysing MRC data (seven out of nine surveyed methods). Qualitative annotation frameworks were proposed to investigate the presence of linguistic features (Schlegel et al. Reference Schlegel, Valentino, Freitas, Nenadic and Batista-Navarro2020) and cognitive skills required for reading comprehension (Sugawara, Yokono and Aizawa Reference Sugawara, Kido, Yokono and Aizawa2017a).

5.2 Model-investigating methods

Rather than analysing data, approaches described in this section directly evaluate models in terms of their inference capabilities with respect to various phenomena of interest. Released evaluation resources are summarised in Table 2.

Table 2. Proposed adversarial and challenge evaluation sets with their target phenomenon, grouped by task and, where appropriate, with original resource name. The last column ‘OOD’indicates, whether the authors acknowledge and discount for the distribution shift between training and challenge set data (Y), they do not (N), whether performance under the distribution shift is part of the research question (P), whether an informal argument (I) is provided or whether it is not applicable (-)

Challenge Sets make for an increasingly popular way to assess various capabilities of optimised models. Challenge sets feature a collection of (typically synthetically generated) examples that exhibit a specific phenomenon of interest. Bad performance on the challenge set indicates that the model has failed to obtain the capability to process the phenomenon correctly. Similar to partial baselines, a good result does not necessarily warrant the opposite, unless guarantees can be made that the challenge set is perfectly representative of the investigated phenomenon. Naik et al. (Reference Naik, Ravichander, Sadeh, Rose and Neubig2018) automatically generate RTE evaluation data based on an analysis of observed state-of-the-art model error patterns, introducing the term ‘stress-test’. Challenge sets have since been proposed to evaluate RTE models with regard to the acquisition of linguistic capabilities such as monotonicity (Yanaka et al. Reference Yanaka, Mineshima, Bekki, Inui, Sekine, Abzianidze and Bos2019a), lexical inference (Glockner et al. Reference Glockner, Shwartz and Goldberg2018), logic (Richardson et al. Reference Richardson, Hu, Moss and Sabharwal2019) and understanding language compositionality (Nie et al. Reference Nie, Wang and Bansal2019). With respect to MRC, we note that there are few (11) challenge sets concerning rather broad categories such as prediction consistency (Ribeiro et al. Reference Ribeiro, Guestrin and Singh2019; Gardner et al. Reference Gardner, Artzi, Basmov, Berant, Bogin, Chen, Dasigi, Dua, Elazar, Gottumukkala, Gupta, Hajishirzi, Ilharco, Khashabi, Lin, Liu, Liu, Mulcaire, Ning, Singh, Smith, Subramanian, Tsarfaty, Wallace, Zhang and Zhou2020), acquired knowledge (Richardson and Sabharwal Reference Richardson and Sabharwal2020) or transfer to different datasets (Dua et al. Reference Dua, Gottumukkala, Talmor, Gardner and Singh2019a; Miller et al. Reference Miller, Krauth, Recht and Schmidt2020).

Notably, these challenge sets are well suited to evaluate the investigated capabilities, because they perform a form of out-of-distribution evaluation. Since the evaluation data stem from a different (artificial) generative process than the crowdsourced training data, possible decision rules based on cues are more likely to fail. The drawback of this, however, is that in this way the challenge sets evaluate both the investigated capability and the performance under distribution shift. Liu, Schwartz and Smith (Reference Liu, Schwartz and Smith2019a) show that for some of the challenge sets, after fine-tuning (‘inoculating’) on small portions of it, the challenge set performance increases, without sacrificing the performance on the original data. However, Rozen et al. (Reference Rozen, Shwartz, Aharoni and Dagan2019) show that good performance after fine-tuning cannot be taken as evidence of the model learning the phenomenon of interest – rather the model adapts to the challenge-set-specific distribution and fails to capture the general notion of interest. This is indicated by low performance when evaluating on challenge sets that stem from a different generative process but focus on the same phenomenon. These results suggest that the ‘inoculation’ methodology is of limited suitability to disentangle the effects of domain shift from evaluating the capability to process the investigated phenomenon.

Furthermore, a line of work proposes to evaluate the systematic generalisation capabilities of RTE models (Geiger et al. Reference Geiger, Cases, Karttunen and Potts2019; Geiger, Richardson and Potts Reference Geiger, Richardson and Potts2020; Goodwin et al. Reference Goodwin, Sinha and O’Donnell2020), concretely the capability to infer and understand compositional rules that underlie natural language. These studies concern mostly artificial languages, however.

Adversarial Evaluation introduces evaluation data that were generated with the aim to ‘fool’ models. Szegedy et al. (Reference Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow and Fergus2014) define ‘adversarial examples’ as (humanly) imperceptible perturbations to images that cause a significant drop in the prediction performance of neural models. Similarly for NLP, we refer to data as ‘adversarial’ if it is designed to minimise prediction performance for automated approaches, while not impacting the human baseline. Thus, patterns in the behaviour of a certain model or a range of models of certain architectures serve as a starting point for the development of adversarial evaluation methods. This is different from challenge sets, which focus on phenomena of interest without assumptions about the evaluated model. Notably, these two terms are sometimes used interchangeably in literature.

Adversarial methods are used to show that models rely on superficial, dataset-specific cues, as discussed in Section 4.2. This is typically done by creating a balanced version of the evaluation data, where the previously identified spurious correlations present in training data do not hold anymore (McCoy et al. Reference McCoy, Pavlick and Linzen2019; Kavumba et al. Reference Kavumba, Inoue, Heinzerling, Singh, Reisert and Inui2019; Niven and Kao Reference Niven and Kao2019), or by applying semantic preserving perturbations to the input (Jia and Liang Reference Jia and Liang2017; Ribeiro et al. Reference Ribeiro, Singh and Guestrin2018). Note that this is yet another method that alters the distribution of the evaluation data with respect to the training data.

Adversarial techniques are further used to understand model behaviour (Sanchez, Mitchell and Riedel Reference Sanchez, Mitchell and Riedel2018), such as identifying training examples (Han, Wallace and Tsvetkov Reference Han, Wallace and Tsvetkov2020) or neuron activations (Mu and Andreas Reference Mu and Andreas2020) that contribute to a certain prediction. Among those we highlight the work by Wallace et al. (Reference Wallace, Feng, Kandpal, Gardner and Singh2019) who showed that malicious adversaries generated against a target model tend to be universal for a whole range of neural architectures.

5.3 Model-improving methods

Here, we report methods that improve the robustness of models against adversarial and out-of-distribution evaluation, by either modifying the training data or making adjustments to model architecture or the training procedure. We group the methods by their conceptual approach and present them together with their applications in Table 3. In line with the literature (Wang and Bansal Reference Wang and Bansal2018; Jia et al. Reference Jia, Raghunathan, Göksel and Liang2019), we call a model ‘robust’ against a method that alters the underlying distribution of the evaluation data (hence making it substantially different from the training data) through for example, adversarial or challenge sets, if the out-of-distribution performance of the model is similar to that on the original evaluation set. They have become increasingly popular: 30%, 35% and 51% of the surveyed methods published in the years 2018, 2019 and 2020, respectively, fall into this category (and none before 2018). We attribute this to the public availability of evaluation resources discussed in Section 5.2 as they facilitate the rapid prototyping and testing of these methods.

Table 3. Categorisation of methods that have been proposed to overcome weaknesses in models and data. To indicate that a method was applied to improve performance on a challenge set, we specify the challenge set name as presented in Table 2

Data Augmentation and Pruning combat the issues arising from low-bias architecture by injecting the required knowledge, in the form of (usually synthetically generated) data, during training. There is ample evidence that augmenting training data with examples featuring a specific phenomenon increases the performance on a challenge set evaluating that phenomenon (Wang et al. Reference Wang, Singh, Michael, Hill, Levy and Bowman2018; Jiang and Bansal Reference Jiang and Bansal2019; Zhou and Bansal Reference Zhou and Bansal2020)—for example, Yanaka et al. (Reference Yanaka, Mineshima, Bekki, Inui, Sekine, Abzianidze and Bos2019b) propose an automatically constructed dataset as an additional training resource to improve monotonicity reasoning capabilities in RTE. As these augmentations come at the cost of lower performance on the original evaluation data, Maharana and Bansal (Reference Maharana and Bansal2020) propose a framework to combine different augmentation techniques such that the performance on both is optimised.

More interesting are approaches that augment data without focussing on a specific phenomenon. By increasing data diversity, better performance under adversarial evaluation can be achieved (Talmor and Berant Reference Talmor and Berant2019; Tu et al. Reference Tu, Lalwani, Gella and He2020). Similarly, augmenting training data in a meaningful way, for example, with counterexamples, by asking crowdworkers to apply perturbations that change the expected label (Kaushik et al. Reference Kaushik, Hovy and Lipton2020; Khashabi et al. Reference Khashabi, Khot and Sabharwal2020), helps models to achieve better robustness beyond the training set distribution.

An alternative direction is to increase data quality, by removing data points that exhibit spurious correlations. After measuring the correlations with methods discussed in Section 5.1, those training examples exhibiting strong correlations can be removed. The AFLite algorithm (Sakaguchi et al. Reference Sakaguchi, Bras, Bhagavatula and Choi2020) combines both of these steps by assuming that a linear correlation between embeddings of inputs and prediction labels is indicative of biased data points. This is an extension of the Adversarial Filtering algorithm (Zellers et al. Reference Zellers, Bisk, Schwartz and Choi2018), whereby multiple-choice alternatives are automatically generated until a target model can no longer distinguish between human-written (correct) and automatically generated (wrong) options.

A noteworthy trend is the application of adversarial data generation against a target model that is employed during the construction of a new dataset. In crowdsourcing, humans act as adversary generators and an entry is accepted only if it triggers a wrong prediction by a trained target model (Dua et al. Reference Dua, Wang, Dasigi, Stanovsky, Singh and Gardner2019b; Nie et al. Reference Nie, Williams, Dinan, Bansal, Weston and Kiela2020). Mishra et al. (Reference Mishra, Arunkumar, Sachdeva, Bryan and Baral2020) combine both directions in an interface which aims to assist researchers who publish new datasets with different visualisation, filtering, and pruning techniques.

Architecture and Training Procedure Improvements deviate from the idea of data augmentation and seek to train robust models from potentially biased data. Adversarial techniques (Goodfellow et al. Reference Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio2014), in which a generator of adversarial training examples (such as those discussed in Section 5.2, e.g., perturbing the input) is trained jointly with the discriminative model that is later used for inference, have been applied to different NLU tasks (Stacey et al. Reference Stacey, Minervini, Dubossarsky, Riedel and Rocktäschel2020; Welbl et al. Reference Welbl, Minervini, Bartolo, Stenetorp and Riedel2020).

Specific knowledge about the type of bias present in data can be used to discourage a model from learning from it. For example, good performance (as indicated by a small loss) of a partial input classifier is interpreted as an indication that data points exhibit spurious correlations. This information can be used to train an ‘unbiased’ classifier jointly (Clark et al. Reference Clark, Yatskar and Zettlemoyer2019b; He et al. Reference He, Zha and Wang2019; Belinkov et al. Reference Belinkov, Poliak, Shieber, Van Durme and Rush2019). Alternatively, their contribution to the overall optimisation objective can be rescaled (Schuster et al. Reference Schuster, Shah, Yeo, Roberto Filizzola Ortiz, Santus and Barzilay2019; Zhang et al. Reference Zhang, Sheng, Alhazmi and Li2020b; Mehrabi et al. Reference Mehrabi, Morstatter, Saxena, Lerman and Galstyan2021). The intuition behind these approaches is similar to Adversarial Filtering which is mentioned earlier: the contribution of biased data to the overall training is reduced. For lexical biases, such as cue words, Utama et al. (Reference Utama, Moosavi and Gurevych2020) show that a biased classifier can be approximated by overfitting a regular model on a small portion of the training set. For RTE, Zhang et al. (Reference Zhang, Bai, Liang, Bai, Zhu and Zhao2020a) compare the effects of different proposed debiasing variants discussed in this paragraph. They find that these approaches yield moderate improvements in out-of-distribution performance (up to 7% using the method by He et al. Reference He, Zha and Wang2019).

In an effort to incorporate external knowledge into the model to increase its robustness, multi-task training frameworks with semantic role labelling (SRL) (Cengiz and Yuret Reference Cengiz and Yuret2020) and explanation reconstruction (Rajagopal et al. Reference Rajagopal, Tandon, Clark, Dalvi and Hovy2020) have been proposed. It is interesting to note that SRL is a popular choice for incorporating additional linguistic information (Wu et al. Reference Wu, Huang, Wang, Feng, Yu and Wang2019; Chen and Durrett Reference Chen and Durrett2020) due to the fact that it exhibits syntactic and semantic information independent of the specific dataset. Additional external resources encoded into the models during training are named entities (Mitra et al. Reference Mitra, Shrivastava and Baral2020), information from knowledge bases (Wu and Xu Reference Wu and Xu2020) or logic constraints (Minervini and Riedel Reference Minervini and Riedel2018).

Interestingly, inconsistency on counterexamples, such as those used for training data augmentation, can be explicitly utilised as a regularisation penalty, to encourage models to detect meaningful differences in input data (Teney et al. Reference Teney, Abbasnedjad and van den Hengel2020a; Asai and Hajishirzi Reference Asai and Hajishirzi2020). Countermeasures for circumventing multi-hop reasoning are providing labels as strong supervision signal for spans that bridge the information between multiple sentences (Jiang and Bansal Reference Jiang and Bansal2019) or decomposing and sequentially processing compositional questions (Tang et al. Reference Tang, Ng and Tung2021).