2.1 Explanations for correct answers to elementary science questions

As alluded to above, QA pairs in the WorldTree corpus (2018) are annotated with explanations of up to 21 facts (see in Figure 1 the distribution of facts in the explanations in the training and development sets).

Figure 1. Facts in explanations per question-answer pair in the training and development datasets.

Based on corpus design decisions, the inclusion criteria for facts in explanations were: lexical overlap—facts lexically overlap with the question or answer, or with other facts in the explanation; and coherency—the explanation facts form a logically coherent discourse fragment. As a consequence of the lexical overlap characteristic, a traversal path can be traced between each QA pair and its explanation facts via multiple lexical hops (depicted in Tables 1 and 2 via the underlined words). Further, as an additional annotation layer, facts in each training and development set explanation were categorized as one of three classes. These classes were determined by the role played by the fact in the explanation. Specifically, the classes were Central, Grounding and Lexical Glue. Central facts were defined as core scientific facts relevant to answering the question. For example, facts such as “as the amount of rainfall increases in an area, the amount of available water in that area will increase.” Grounding facts were those which connected to other core scientific facts present in the explanation. For example, “rain is a kind of water” would connect “rain” and “water” present across two or more Central facts in the explanation. Finally, lexical glue facts expressed synonymy or definitional relationships. For example, “rainfall is the amount of rain an area receives.” Table 3 offers statistics on the overall prevalence of explanation facts across QA pairs in the training and development sets, and also per explanation fact category.

Table 3. Corpus statistics for QA pairs w.r.t. their explanation facts from the WorldTree (2018) training and development corpora combined

We now elaborate on the facts’ tablestore that formed the reference set for constructing the explanations per QA pair. The tablestore facts were authored based on the elementary science themes of the ARC question-answering data. They are organized in 65 tables representing relation predicates such as kind of (e.g., an acorn is a kind of seed), part of (e.g., bark is a part of a tree), cause (e.g., drought may cause wildfires); or the actions of organisms (e.g., some adult animals lay eggs); or the properties of things (e.g., an acid is acidic); or if-then conditions (e.g., when an animal sheds its fur, its fur becomes less dense). In Table 4, we depict the table types whose facts belonged to at least 1% of the explanations in the training and development sets. We see that only 21 tables from 65 in total were represented in at least 1% of the training and development explanations. Of the 44 remaining least frequently selected tables, in Table 5, we show only nine selected ones as examples. Based on the table sizes, the least frequently occurring tables have fewer facts than most of the 21 frequently selected tables; however, there are one or two exceptions (e.g, the COUNTRY-HEMISPHERE table with 269 facts).

Table 4. Percentage occurrences of facts from 21 table types (of 65 total) that participated in at least 1% of the training and development explanations. The numbers in parenthesis are the table sizes in terms of facts

Table 5. Nine randomly chosen tables as examples with less than 1% percentage fact occurrences (of 44 in all) in the training and development explanations. The numbers in parenthesis are the table sizes in terms of facts

Figure 2 depicts the top six of the 21 frequently selected table types for their fact rankings in the explanations. The six tables are KINDOF, SYNONYMY, ACTION, IF_THEN, CAUSE, and USED-FOR. For these tables, we explicitly show the proportions of their facts at ranks 1 to 10 in the explanations and aggregate the remaining lower-ranked facts in a single proportion. In the figure, we can see that except for SYNONYMY, all the remaining tables have a major proportion of their selected facts appear between ranks 1 and 5. For SYNONYMY, however, we see that only 5% of its facts appear at rank 1. This comparatively low proportion is meaningful in the context of the role played by its facts in the explanations—more often than not, they supplement the information of a previous fact. For example, an explanation has the fact “the moon orbiting the Earth approximately occurs 13 times per year” at rank 1 which is supplemented by the SYNONYMY fact “approximately means about” at rank 2.

Figure 2. Ranked proportions of a subset of facts from the training and development set explanations. These facts are only those in the top 6 most frequently selected fact table types (shown as Figures 2(a)–(f), respectively). Each pie chart depicts the respective % proportion of occurrences of facts for ranks 1 to 10 explicitly, with one additional unlabeled category for the ranks after 10.

5.1 Pairwise Learning-to-Rank (LTR) for preference ordering

The next question is how can our preference ordering problem be formulated in terms of binary classification. This is possible by the pairwise LTR transformation. Vaguely, this is done by modeling: (1) whether a candidate fact is a valid candidate or not and (2) for the collection of valid explanation facts, the logical precedence of one fact over another. Thus, these decisions are identified in a relative sense, that is to say, by determining the pairwise preferences between facts in the explanation compared w.r.t. each other and w.r.t. the remaining facts in the tablestore.

Our data set originally is

\begin{equation*}S=\{x_{ij},y_{ij}\}\ \textrm{where}\ x_{ij} = \phi((q_i,ca_i),f_{j})\end{equation*}

$(q_i,ca_i)$ is the ith QA pair instance, $f_j$ is the jth explanation fact from the tablestore where the ordering between facts is known during training and is unknown during development and testing. $\phi$ is a feature transformation function, and $y_{ij} \in \{1,2,3,...K\}$ denotes an order between the $(q_i,ca_i)$ pair and the explanation fact $f_j$ as a graded order w.r.t. the other relevant and irrelevant explanation fact candidates.

By the pairwise LTR transformation, our original data set S then becomes:

\begin{equation*}S\text{'}=\{(x_{ij}-x_{il}),(y_{ij} \theta y_{il})\}\end{equation*}

where $\theta$ is the rank difference so that $(y_{ij} \theta y_{il}) = 1$ if $y_{ij} > y_{il}$ and –1 otherwise, resulting as a binary classification task. The goal of the LTR algorithm is to acquire a ranker that minimizes the number of violations of pairwise rankings provided in the training set which is attempted as the above classification problem.

Essentially, since pairwise LTR only considers the labels where $y_{ij} > y_{il}$ between relevant candidates and $y_{il} > y_{ij}$ between relevant and irrelevant candidate pairs, respectively, we assign (a) higher ranks to the relevant facts to indicate precedence and (b) equal ranks of 1 to all irrelevant facts. Thus, for consecutive relevant instances, we offset the ranks by 1. That is, for a given $x_i$ , if there are $n_i$ correct facts in the explanation, then the first fact in the correct ordering receives a rank of $|n_i|$ + 1, the second is ranked as $|n_i|$ , and so on; the last correct fact receives rank 2, and all irrelevant facts have rank 1.

5.2 Pointwise preference ordering by regression

SVM regression differs from the SVM classification objective in that instead of optimizing over binary targets, the optimization is performed for real-valued targets. To facilitate this, regression is then defined in terms of an $\epsilon$ -precision objective. In other words, we do not care about training errors as long as they are less than $\epsilon$ . Further, as in the classification objective with soft decision boundaries, similar allowances are made with slack variables in the regression context, but defined over targeted regression precision. Formally, the regression optimization problem is defined as follows:

(3) \begin{equation}\begin{aligned}\min_{w,\ b,\ \xi,\ \xi^*} \quad & \frac{1}{2}w^{T}w+C\sum_{i=1}^{N}{(\xi_{i} + \xi^*_{i})}\\ \\[-10pt]\textrm{s.t.} \quad & y_{i} - w \cdot \phi(x_{i}) - b \leq \epsilon + \xi_{i}\\ \\[-10pt]\quad & w \cdot \phi(x_{i}) + b - y_{i} \leq \epsilon + \xi^*_{i}\\ \\[-10pt] &\xi_{i},\xi^*_{i}\geq0 \\ \\[-10pt] \end{aligned}\end{equation}

where $i = 1,...,N$ for N training instances, $\phi$ is a feature transformation function for input $x_i$ , w is the features’ weight vector over all instances, $y_i$ is a real-valued target, and $\epsilon$ is the regression targeted precision. The constant $C > 0$ determines the tradeoff between the norm of the weight vector and error margin defined by slack variables $\xi,\xi^*$ .

Next, an important question is, how to represent our ordering problem in terms of a regression objective. We do this by defining regression targets in terms of the preference ordering expectations (Kamishima et al. Reference Kamishima, Kazawa and Akaho2005, Reference Kamishima, Kazawa and Akaho2010) rather than true regression quantifications. In our data set $S=\{x_{ij},y_{ij}\}$ where $x_{ij} = \phi((q_i,ca_i),f_{j})$ , the label $y_{ij}$ for the correct candidate explanation facts is indicated as unit graded relevance in order of their preference, while all the incorrect candidates are relegated to a uniform least rank. The facts are assigned ranks in exactly the same manner as the pairwise LTR setting.

By using regression for the preference ordering of facts in explanations, we make the assumption that all facts can be treated independently w.r.t. each other. Such assumptions are highly contingent on the properties of the underlying data set and may not apply in all preference ordering or ranking scenarios. In contrast to the regression setting, the pairwise LTR is, in principle, applicable in any ordering scenario. Evidently, in the WorldTree corpus (2018), the order of facts in the explanations are based on logical precedence rather than discourse, wherein the discourse linguistic cues are not readily apparent in most explanations (consider the example depicted in Table 1). Thus, from the perspective that considers a pure logical precedence between the explanations’ facts, a regression setting that predicts logical precedence weights as target values for the facts is nonetheless relevant and a sound modeling of the task.

6.1 Bags of lexical features (70,949 total features)

This feature group most generically encodes the lexical overlap criteria by including features as lemmas of $q/ca/f$ ; lemmas shared by q and f, ca and f, and q, ca and f; 5-, 4-, and 3-gram prefixes and suffixes of $q/ca/f$ ; 5-, 4-, and 3-gram prefixes and suffixes shared by q, ca, and f; and f’s table type from the provided annotated tablestore data. The lemma and n-gram features are filtered for common pronoun and prepositional stop words.

6.2 ConceptNet (294,249 total features)

We hypothesize that semantic features, in particular commonsense knowledge, could be useful for the explanations to elementary science QA pairs. Since elementary science questions query general knowledge about common nouns like animals, planets, occupations, etc., we find that ConceptNet (Speer et al. Reference Speer, Chin and Havasi2017) as a resource with its focus on the general meanings of all words, whether they be nouns, verbs, adjectives, or adverbs, and less on named entities, is perfectly suited to our task. Let us illustrate with an example.

In this example, ConceptNet tells us that the answer “rabbit” is an “animal” and a “herbivore”, among other things. Extending the answer with this knowledge enables better semantic connection between the q, ca, and all three explanation facts, in the absence of which, the ranking could experience a semantic drift toward irrelevant explanation facts such as “long ears are a part of a rabbit” or “a jackrabbit is a kind of rabbit”.

Given the potential usefulness of ConceptNet for our task, we create conceptualization features as follows. The top 50 conceptualizations of q/ca/f words; top 50 conceptualizations shared by q and f, ca and f, and q, ca and f words; and the relation names that $q/ca/f$ words as ConceptNet facts participate in such as FormOf, IsA, HasContext, etc. For example, for word “tea” in $q/ca/f$ , the ConceptNet facts are “tea ReceivesAction brewed”, “tea HasA caffeine”, “teaIsA beverage”, etc., from which the features are “ReceivesAction_brewed”, “HasA_caffeine”, and ‘IsA_beverage’.

Note, since ConceptNet returns an ordered list of conceptualizations for queried terms ordered w.r.t. precision as most precise to generic, therefore, by selecting only the top 50 for the first two features, we control for genericity of the conceptualizations in the features considered.

6.3 OpenIE relations (36,989 total features)

We introduce features computed as open information extraction relation triples using the OpenIE tool (Angeli, Premkumar, and Manning Reference Angeli, Premkumar and Manning2015). We observed that within the triple representations of the question, correct answer, and explanation fact sentences were the content words that were needed to link across other feature groups (e.g., ConceptNet). This linkage also enabled indirect connections between (q, ca) and f. Let us illustrate with an example:

In the example, the given fact is top-ranked in the explanation. For it, from OpenIE we get the relation triple (hardness $\rightarrow$ is a property of $\rightarrow$ material). Further, ConceptNet tells us that the answer Hardness is related to concepts “property,” “material property,” etc. We see how pooling these information units together enables a unified word cloud involving the question, correct answer, and explanation fact for the terms “hardness,” “property,” and “material.” Features that enable grounding externally computed terms to the lexical items given in the QA pair or explanation facts create a tighter overlap improving task performance.

Given the potential usefulness of intersentence OpenIE triples for explanation generation, we create features as follows. For each triple produced by the parser, the features are the $q/ca/f$ lemmas in the relation subject role; shared q, ca, and f subject lemmas; $q/ca/f$ lemmas in the relation object role; shared q, ca, and f object lemmas; and $q/ca/f$ lemma as the relation predicate.

6.4 Multihop inference specific features (2,620 total features)

These features are a more selective bag of lexical features for obtaining matches with a positional emphasis. We find that adding positional information for lexical matches is a useful heuristic to identify the concepts that are the focus of the (q, ca) and explanation facts. Consider the underlined words in the two subsequent examples in this section.

As shown in the examples, often the focus word of the (q, ca) are at the start or end and also at the start and end of the f. Further, for one- or two-word ca, we can directly infer it as a focus concept, in which case we try to find a match with f where they are the first or last word. And for focus words that are verbs, they tend to occur in the middle.

The following features are considered in this group: length of q and ca; positions of q/ca verbs in the phrase (as 0 if it is the first word, 1 if it is the second word, and so on); of the verbs shared by q and f do they occur among the first few words or middle or last words.Footnote ^b; if ca is a uni- or bigram, does f contain all its words/lemmas?; does f contain the last q lemma/word?; is the last q lemma/word in the first position of f?, is it in the last position of f?; is the first q lemma/word in the first position of f?

Notably, the positional emphasis observations made for the features in this category are contingent on the language of the corpus. For example, we observe verbs as focus words occurring in the middle of the sentence because the WorldTree corpus is annotated for English sentences which follow the subject-verb-object (SVO) sentence structure. Thus, it is necessary to highlight that if the WorldTree corpus was annotated for a language with a different sentence structure, For example, Persian which follows the SOV order, the verbs would be expected at the end of the sentence and our “multihop inference specific features” would need to be accordingly adapted. Further, since the WorldTree corpus handles the Elementary Science level, it contains fairly short atomic sentences and this accounts for us finding focus nouns at specific locations as the sentence start or end and the focus verbs in the middle. A worthwhile consideration then is that for higher scientific levels the positions of the focus words could be different and harder to pin down as a fixed location as was possible in our case for the WorldTree corpus.

6.5 TF-IDF ranking (750,283 total features)

The explanation regeneration task performance via ranking based on cosine similarities between TF-IDF weighted (q, ca) appended text and each fact candidate proves surprisingly effective for the task (see scores in Evaluation section). We use the TF-IDF Iterated variant by Chia et al. (Reference Chia, Witteveen and Andrews2019) to encode the text. The ranks obtained by cosine similarity on these instances are then used as features for the SVM learner. We hypothesize that employing the TF-IDF-based cosine similarity ranks as features will provide a baseline ordering signal to the learning algorithm.

Our TF-IDF features per (q, ca, f) are the following: f’s rank; f’s binned rank in bins of 50; f’s binned rank in bins of 100; whether f is in top 100 or 500 or 1000?

6.6 BERT embeddings

BERT-based (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2018) context embeddings are our last features category. The out-of-box BERT model is pretrained on millions of words from Wikipedia which as a commonsense knowledge source is already pertinent to elementary science QA. Thus, we simply query the BERT embeddings from the pretrained model using the bert-as-a-service library. Thus, for each data instance word, we extract their BERT embedding features that can easily can be combined with the other linguistic features. This can be viewed as a semantic projection of an elementary science concept in the Wikipedia encyclopedia space. Specifically, we query the Bert $_{Base}$ Uncased English model: 12 layers, 768 hidden units, 12 heads, with 110M parameters that outputs a 768 dimensional vector for a given input text. We treat each dimension of this context vector as a separate feature for representing the instance.

While the earlier five feature categories enabled extending the (q, ca, f) vocabulary beyond the given words both lexically and conceptually, with BERT embeddings we aim to leverage semantic abstractions as features. We hypothesize such features would be useful in creating semantic associations between the elements in the (q, ca, f) triple, which are topically similar based on knowledge from Wikipedia. As in the following example.

In the example, considering the focus words “diamonds,” “earth,” and “minerals” that reflect the topics of the QA pair, the word “minerals” in the fact is neither present in the q or ca, but is relevant to the semantic topic of the (q, ca). We hypothesize that BERT features will help capture such topicalized semantic abstractions of similarity.

We tested two ways of obtaining BERT features for (q, ca, f) triples: (i) query BERT separately for the question, correct answer, and fact embeddings, respectively, obtaining three 768 dimensional feature sets and resulting in 2304 additional features from BERT per instance and (ii) query BERT for aggregate 768-dimensional embedding features for the (q, ca, f) triple. In this configuration, we use the special [SEP] token to demarcate the q, ca and f segments. The start-of-the-sequence [CLS] token then learns the encoding for the entire input sequence. Experiments indicated that the latter method is a better-suited representation for the task, while the former method is ineffective. This can be attributed to two reasons: (1) The second approach allows the model to capture not only the representations of the sequences but also the proximity between the input segments. Thus, the [CLS] token in this case is a latent similarity vector for the q, ca, and f. In the first case, it merely represented the sequences separately. And (2) the length of the vectors produced in case (i) is three times the length of the vectors produced in case (ii). This results in sparse features which in turn limits the performance of the SVM classifier.

Thus, all six feature categories used to represent (q, ca, f) triples, when taken together, should readily address the multistep inference process between (q, ca) and f candidates. This is since we have features extending the given information in the (q, ca) with world knowledge generically (e.g., ConceptNet, BERT) with other features providing lexical glue (at generic and task-specific levels) enabling traversing the (q, ca, f) via multiple hops (e.g., multihop inference lexical features, OpenIE relations). Therefore, for themes such as about vehicles, as an example from Jansen et al. (Reference Jansen, Balasubramanian, Surdeanu and Clark2016), describing its mechanisms, its purpose, its needs, and its functions, our various feature groups can take into account such diverse aspects of the real world.

Finally, in Figure 3, we depict our overall approach including the feature modules and the two SVM-based machine learners we employed.

Figure 3. Overall representation of our approach. For representing (q, ca, f) triples, the feature categories used include Lex (lexical features), TF-IDF (information retrieval features), and BERT-based features among others. The data instances are then represented as a features matrix, separately as training data, development data, and testing data. Two variations of the SVM algorithm ( $SVM^{Rank}$ and $SVM^{Reg}$ ) are then used to learn Explanation Regeneration models.

7.1 Experimental setup

Dataset

The experimental corpus of this study is the Worldtree corpus (2018), introduced in detail in Section 2. In our experiments, we maintain the same data set fold splits as provided by the data set creator.

Evaluation Metrics

We report one set of results in terms of the mean average precision mAP metric which is a standard in IR ranking tasks. With the mAP metric score, we see to what extent our system returns the relevant explanation facts as top ranked. To evaluate our system for ordering the relevant explanation facts w.r.t. each other, we employ the Precision@k and Recall@k metrics, where k is the group of top-ranked facts ranging between 2 and 50 facts in increments of 2. Note that for these latter metrics, a correct score is considered for each fact returned at its exact given position in the ordered facts in the explanation. For instance, to get a score of 1.0 by the Precision@2 metric, the predicted top-scoring fact should be the top-ranked fact and the predicted second top-scoring fact should be the second ranked fact in the gold data. These latter evaluations are a closer test of our system as satisfying the task defined in the WorldTree corpus (2018), in other words, to see if it logically orders the relevant facts at all.

Parameter Tuning

For the SVM ^rank and the SVM ^reg systems, we jointly tune the C and the number of negative training instances parameters on development data. Our best SVM ^rank model when evaluated on development data was obtained with C = 0.8 and 1000 negative training instances, while our best SVM ^reg model was obtained with C = 0.005 and 900 negative training instances.Footnote ^c

We compare our models with nine existing systems as reference performances, where the systems we compare with have varying degrees of complexity from simple IR approaches to neural-based machine learning approaches. Further, as we show in the results eventually, the systems also have varying degrees of performances not necessarily correlated with the system complexity—TF-IDF approaches prove surprisingly effective on this task. In the following section, the nine systems we evaluate against are briefly described.

7.2 Nine reference evaluations

TF-IDF Baseline

Facts are ranked by cosine similarity of their TF-IDF representation with the TF-IDF representation of the query string composed of the question and all the available answer choices.

TF-IDF Baseline features + SVM ^rank ( Reference Jansen and Ustalov Jansen and Ustalov 2019 )

For each data instance, two cosine similarity scores were computed: one between the question TF-IDF vector and the candidate explanation fact vector and another between the correct answer and the candidate fact vectors. These scores were used as features within an SVM ^rank (Joachims Reference Joachims2006) setting and a ranked list of facts were predicted.

Generic Feature-rich SVM^rank (D’Souza et al. Reference D’Souza, Mulang and Auer2019)

Our previous system had five main feature categories including OpenIE (Angeli et al. Reference Angeli, Premkumar and Manning2015), ConceptNet (Liu and Singh Reference Liu and Singh2004), Wiktionary, and FrameNet (Swayamdipta et al. Reference Swayamdipta, Thomson, Dyer and Smith2017) representations for each (q, ca, f) triple are employed, which are then ranked by SVM ^rank (Joachims Reference Joachims2006).

Rules + Generic Feature-rich SVM ^rank (D’Souza et al. Reference D’Souza, Mulang and Auer2019)

In this hybrid model, the Generic Feature-rich SVM ^rank system output is corrected for obvious errors by a set of 11 re-ranking rules applied sequentially, pipelined to the SVM system output. As an example of a rule consider: all facts that contain the bigram or unigram correct answer word are to be top-ranked.

BERT Iterative Re-ranking ( Reference Banerjee Banerjee 2019 )

The system models explanation regeneration using a re-ranking paradigm, where BERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2018) transformer models are used to provide an initial ranking, and the top-15 facts output by the BERT model are re-ranked using a custom-designed relevance ranker to improve overall performance. Note that in comparison with the top-performing BERT-based model (Das et al. Reference Das, Godbole, Zaheer, Dhuliawala and McCallum2019), described last in this section, this system is run in a BERT out-of-box configuration.

Optimized TF-IDF (Chia et al. Reference Chia, Witteveen and Andrews2019)

This system differs from TF-IDF Baseline in the following ways: all incorrect answer choices are dropped from the query string; the query and the fact strings are additionally preprocessed by lemmatization and the removal of their stopwords.

Iterated TF-IDF (Chia et al. Reference Chia, Witteveen and Andrews2019)

Where in the Optimized TF-IDF system, the query string consists of only the question and the correct answer; in this system, the query string is iteratively expanded to include the top-ranked fact. After each expansion step, cosine similarity is rerun on the remaining facts to obtain the next top-ranked fact. This process is iteratively repeated until all facts are ranked.

BERT Re-ranking with Iterated TF-IDF scores (Chia et al. Reference Chia, Witteveen and Andrews2019)

A BERT regression module is trained to predict the relevance score for each $(q+ca,f)$ pair, where the relevance score is the Iterated TF-IDF system rankings; and in the interest of lowering the computational complexity and runtime, the model is trained and tested to rerank only the top 64 of the Iterated TF-IDF system output.

BERT Re-ranking with inference chains (Das et al. Reference Das, Godbole, Zaheer, Dhuliawala and McCallum2019)

It is an ensemble model composed of BERT-based path ranker and a more advanced reranking system (Nogueira and Cho Reference Nogueira and Cho2019) which they employ as a ranker. The BERT-based path ranker uses a sophisticated multistep design. The initial step involves obtaining the top 50 facts based on TF-IDF similarity with (q, ca) query. In the next step, 1-hop lexical similarity paths are traced from each fact in the retrieved 50 facts list to the remaining facts in the tablestore. Finally, the BERT path ranker is trained on pairwise fact instances. Instances are formed by exhaustively pairing each fact in the top 50 TF-IDF list with all the corresponding retrieved facts at a 1-hop lexical distance from it such that the pairs where both facts constitute the explanation for the given query are a true instance for the BERT path ranker, and others are false. The overall ensemble system then relies on this BERT-based path ranker output for a score threshold of above 0.5, else it uses a reranker (Nogueira and Cho Reference Nogueira and Cho2019).

With its chaining of facts, this system models a vital aspect of the corpus: that is, some valid explanation facts directly lexically overlap with the QA pair, and others lexically overlap with other valid facts. As we will see next, this system has the overall best performance and is the only system we do not outperform. We note, however, that it also presents a high degree of computational complexity. Consider in our system, during training, each QA pair is linked with only roughly 1000 explanation facts; whereas the Das et al. system would construct training instances as follows: for each QA pair, given top-50 ranked facts, assuming each fact has a 1-hop chain to at most 200 other facts, this would result in nearly 10,000 instances. A larger training data set generally implies a larger training time—a fact that is particularly true in the case of BERT models as the Das et al. system, while also true for SVMs, although in the SVM case, the number of features would also matter. Further, in the test scenario, while the Das et al. system would still evaluate for 10,000+ odd chains, we would merely check for all the facts in tablestore which presently is about 5000. Thus, the Das et al. (Reference Das, Godbole, Zaheer, Dhuliawala and McCallum2019) system is the most effective and at the same time the most computationally intensive of all the systems including ours.

7.3 Results and discussion

Table 6 shows the elementary science QA pair explanation fact preference ordering results in terms of mAP with best results from the reference system and ours (last two rows) in bold.

Table 6. Mean Average Precision (mAP) percentage scores for Elementary Science Q&CA Explanation Regeneration by our systems (last two rows) compared with nine reference systems on testing (Test) and development (Dev) data sets, respectively

Between our models, we find that SVM ^reg is significantly betterFootnote ^d than SVM ^rank by applying the paired t-test to their adjacent scores in the table. Thus, given our underlying data set, a pointwise learning approach proves better suited to it than a pairwise learning approach. Nevertheless, the latter is still a valid model, in principle, for the task as it does not rely on the strong, and seeming unrealistic, independence assumption between instances made by the SVM ^reg model. However, since SVM ^reg significantly outperforms SVM ^rank at $p < 0.05$ , it proves practically better suited on this data set implying that the nonindependence assumption between facts by SVM ^rank is not a crucial factor in learning the task defined in the data.

Compared with the nine reference systems, our SVM ^reg approach significantly outperforms eight of the models. This set of systems also includes the neural ranking model by Banerjee (Reference Banerjee2019), which our system surpasses by +9.4/+10.9 points in mAP, as well as a neural re-ranking model by Chia et al. (Reference Chia, Witteveen and Andrews2019) which we surpass by +3 mAP. Although we observe lower performance when compared to the best-performing approach (–5.6/–5.3) by Das et al. (Reference Das, Godbole, Zaheer, Dhuliawala and McCallum2019), theirs is a significantly computationally complex system than ours as explained earlier (Section 7.2). Finally, in terms of scalability, next to our feature-rich SVM ^reg are the Iterated TF-IDF or Optimized TF-IDF models by Chia et al. (Reference Chia, Witteveen and Andrews2019). Where the simplistic Optimized TF-IDF system, as expected, significantly underperforms feature-rich systems, nevertheless its ranking output as features in our system proves effective as we will see in the ablation analysis results (Section 7.3.1).

With our re-engineered system leveraging domain-targeted features, we have significantly outperformed our earlier system that was based on generic linguistic features (2019). Our SVM ^rank is at +9.2/+8.8 compared to the system without rules and at +3.9/+1.5 compared to the hybrid system, and our SVM ^reg is at +16.6/+16.1 to the without-rules system and +11.3/+8.8 to the hybrid system. Thus, we see in contrast the task impact obtained from an effective learning algorithm and a set of features that specifically models the domain in our new system version.

In the preceding paragraphs, we have discussed the performance of our system for producing top-ranked relevant facts. Next, we briefly examine the performance of our system for ordering relevant facts w.r.t. each other among the top-ranked. This is presented by the Precision@k and Recall@k evaluations depicted in Figure 4. We see that at low values of k (i.e., ${\leq}$ 26), the ordering performance of SVM ^reg is distinctly better than SVM ^rank in terms of both precision and recall. However, beyond 26 facts there is no evident difference between them. At $k=2$ , SVM ^reg has a recall rate of 0.42 at a precision rate of roughly 0.38. This indicates how often the top two automatically ranked facts are correct and in the top two order. At $k=26$ , we see that 75% of the facts are retrieved (recall rate of 0.75), however, at low precision of 16% (precision rate of 0.16). Part of the low precision score can be attributed to retrieving also the irrelevant facts for explanations with less than 26 facts; nonetheless, these standard metrics are not modifiable for cases when all the relevant facts are already retrieved within the 26 for such explanations. Aside from the spike in the precision score at $k = 4$ for SVM ^rank , both SVM ^reg and SVM ^rank show fairly stable precision and recall rates with steady expected decline and climb rates, respectively.

Figure 4. Fact ordering evaluations of our system for top-ranked explanation facts in terms of Precision Rate @ k (a) and Recall Rate @ k (b) on the test data.

As a summary statement for our results presented in this section, we see that at 50.7% mAP of ranking the valid facts as top-ranked, only a small proportion of the predictions are in exact order based on the gold standard.

7.3.1 Feature ablation results

To provide further insights on the impact of adding different feature groups, we show ablation analysis results in Table 7. Our ablation analysis strategy is to append each of the six feature groups, one at a time, to the baseline features as individual ablation experiments.

Table 7. Ablation results of SVM ^rank and SVM ^reg on the dev and test sets, respectively, in terms of percentage mAP with feature groups (from six feature types considered) added one at a time to the bag of lexical features. TF-IDF and BERT features have highest impact

From the reported scores in the table, we observe that on both the SVM ^rank and SVM ^reg learners, respectively, that TF-IDF and BERT features show highest impact; multihop and ConceptNet features were the second most impactful feature types. And the least impact was from the OpenIE features that showed no improvement with SVM ^reg and a minor improvement with SVM ^rank . Nonetheless, we retain this feature group since its ablation analysis does not show a negative impact on system performance.

Next, we select a development set example for two of the most impactful feature groups, that is ConceptNet and BERT features, for qualitative analysis. The rest of our features can be similarly justified. In both examples, r $_{p}$ is the predicted rank, and r $_{g}$ is the gold rank.

First, considering the ConceptNet features, a qualitative examination of our results showed that its added commonsense world knowledge prevented semantic drift in several cases. We explain this with the help of the selected example below. In the example, the concepts “tree,” “photosynthesis,” and “leaves” are the content concepts in the question and the correct answer. These three concepts are also the content concepts in the three gold explanation facts taken together; however, “plant” is an additional content concept in the explanation facts. The task then is to link the facts with the “plant” concept better with the question and the correct answer so that they can be ranked higher. In this respect, from ConceptNet we have the information that the “plant” entity has a class “photosynthetic organism.” Our hypothesis was that this additional information should help boost the ranks for the 2nd and the 3rd gold facts. As we can see in the example, adding the ConceptNet feature has indeed helped boost the ranks for the 2nd and 3rd gold explanation facts since it linked “plant” with a focus concept from the question, that is “photosynthesis.” Further, comparing the Before and After sets, we can see that the semantic coherence from ConceptNet lower-ranked facts with concepts as “fruit,” “eating,” “digestion,” “animals,” and “consumers.”

In the second example, we qualitatively depict the impact of adding BERT features. We glean the theme of the QA pair as “falling under gravity.” While the dotted phrases “gravitational force,” “fall,” and “falling” encompass the theme, they are not directly present in the (q, ca) pair unlike the underlined phrases. The third fact, that is “come down is similar to falling,” that contains one of the (q, ca) absent thematic phrases, viz. “falling,” after adding BERT features, attains a significant performance boost by 7 ranks shown in After ranked collection. We posit this is by the better abstract theme modeled by BERT features. As a consequence, several of the unrelated facts were then ranked lower. Consider the facts such as “rubber is a kind of material,” “to bounce back means to reflect,” “objects are made of materials or substances or matter” that were no longer intervening in the returned After result collection.

7.3.2 Evaluating SVM ^reg for multihop inference

In this section, we present our best system performance, that is the SVM ^reg system with all six features’ categories, for multihop inference as a function of explanation lengths, that is the number of facts. Presumably, the more the facts in the explanation the greater the implication of the presence of the multihop phenomenon, that is lexical hops across explanation facts to attain shared lexical matches with other explanation facts and the other explanation facts then with the (q, ca) pair. This result is depicted in Figure 5.

Figure 5. Percentage mAP of the SVM ^reg system on Development data (in light gray) and Test data (in dark gray), respectively, on different length explanation sentences.