4.1 Data extraction

As indicated above, MBL needs labeled training data to be stored in memory, on the basis of which unlabeled test examples can be classified. We extracted the data from two large newspaper corpora. For the (ND) data, the 500-million-word Twente News Corpus (TwNC) was used (Ordelman et al. Reference Ordelman, de Jong, van Hessen and Hondorp2007), while for the BD data we used the highly comparable Leuven News Corpus (LeNC), which was compiled by the QLVL research group at the University of Leuven and currently contains 1.3 billion words. Both corpora are similar in design, consisting of materials from a collection of major Dutch and Flemish newspapers which have been published around the turn of the current millennium and are distributed and read nationwide.

The compilers of TwNC and LeNC have parsed all data with the Alpino parser, which is the current state-of-the-art dependency parser for Dutch (van Noord Reference van Noord2006). This allowed us to conveniently retrieve relevant instances of adjunct-initial existential sentences, which would be considerably harder using regular expressions on non-parsed text, as existential sentences are lexically highly, and in instances of the er-less variant even completely un(der)specified. Specifically, we extracted all sentences with the following syntactic structure:

1. (2) adjunct PP + main verb (+ er) + indefinite subject NP

In order to make extraction as efficient as possible in terms of precision and recall, we imposed a number of heuristic restrictions. For the indefinite subject, we limited ourselves to NPs with a nominal head node (thus excluding pronouns and proper names). Sentence-initial adjuncts were restricted to prepositional phrases (PP) with a nominal head node.Footnote ^d Next, since previous research had focused exclusively on variation in existential sentences with temporal and locative adjuncts (including their metaphorical uses, see Grondelaers et al. Reference Grondelaers, Speelman, Geeraerts, Kristiansen and Dirven2008, pp. 173–5), we limited the query to adjuncts which contain one of the prepositions the reference grammar ANS lists as typical or frequent prepositions in these types of adjuncts (cf. Haeseryn et al. Reference Haeseryn, Romijn, Geerts, de Rooij and van den Toorn1997, pp. 1194ff.). Finally, er itself could either or not occur between the main verb and the post-verbal subject.

For the present study, we exclusively extracted materials from the Dutch newspapers NRC Handelsblad (NRC) and Algemeen Dagblad (AD), and the Flemish newspapers De Standaard (DS) and Het Laatste Nieuws (HLN). In both countries, we selected newspapers along a formality dimension because this register variable was found to impact er-preferences in BD but not in ND (Grondelaers et al. Reference Grondelaers, Speelman, Geeraerts, Morin and Sébillot2002b; Reference Grondelaers, Speelman, Geeraerts, Kristiansen and Dirven2008; Reference Grondelaers, Speelman, Drieghe, Brysbaert and Geeraerts2009; and cf. supra): while many consider NRC and DS as “quality newspapers” that are geared toward an educated audience, AD and HLN are generally classified as “popular newspapers.” As could be expected from the separate regression analyses in Grondelaers et al. (Reference Grondelaers, Speelman, Geeraerts, Morin and Sébillot2002b, Reference Grondelaers, Speelman, Geeraerts, Kristiansen and Dirven2008), we found no statistically significant difference (at $\alpha=0.05$ ) between er’s distribution in the ND quality and popular newspaper ( $\chi^2_1=1.97$ , $p=0.16$ ). For BD, for which there is much more data, the difference is statistically significant ( $\chi^2_1=9.12$ , $p=0.003$ ), but the effect size is very small (Cramér’s $V=0.02$ ; this is also apparent visually in Figure 1, in which the proportional difference between the latter two columns is very small). Given all this, we will proceed without taking the register dimension explicitly into account.

Figure 1. Mosaic plot of er’s distribution across the four newspapers (NRC and AD for ND; DS and HLN for BD). The area of each tile of the plot is proportional to the number of observations it represents.

As can be appreciated in Figure 1, adjunct-initial existential constructions are much more frequent without (-ER) than with er (+ER) (an asymmetry that is well known, see Grondelaers et al. Reference Grondelaers, Speelman, Geeraerts, Morin and Sébillot2002b, Reference Grondelaers, Speelman, Geeraerts, Kristiansen and Dirven2008). However, this skewed distribution causes baseline accuracy for our models to be fairly high, especially in the TwNC data, which we remedied by downsampling the non-er cases, so that baseline accuracy for both the TwNC and LeNC data is set to 50%. For each national variety, from both newspapers, an approximately equal number of random instances were extracted and manually checked until we reached a total of 1000 instances; 500 with er and 500 without er.Footnote ^e

4.2 Feature selection

As one of the main goals of this article is to test the predictability of syntactic variation (within and across national varieties) on the basis of lexical input, we need a way to represent each corpus instance in a format that is interpretable for the TiMBL classifier, yet still meaningful in the light of our RQs. In view of RQ1—that is the question whether we can predict er-preferences on the basis of raw lexical material or whether some form of additional linguistic knowledge is needed—we created two types of datasets: one in which the features provided to the classifier consist of nothing more than the immediate lexical context, and one in which we used the Alpino dependency parses to extract from each instance those lexical features which have been shown in previous research to represent the key elements which determine er-choices. We will refer to the first type of instance representation as the window-based (WIN) approach, and to the second one as the parse-based (PAR) approach (cf. Pijpops Reference Pijpops2019, Chapter 7 for a similar procedure).

For the WIN approach, we identified for every instance in the balanced datasets the syntactic position (or “slot”) in which er occurs, or, in the instances where it is not realized, could have occurred. We subsequently extracted 5 items to the left and to the right of this position, which yielded 10 features in total. All words were converted to lowercase. In sentences with less than five words to the left or right of the er slot, an underscore was inserted as a padding character. Each feature in the dataset was separated by a white space, with the final feature being the class label, viz. +ER or -ER (cf. Daelemans et al. Reference Daelemans, Zavrel and van der Sloot2018, Chapter 4). Instances like (3)–(6) are thus represented as in Table 1 in the WIN format. The resulting datasets are henceforth referred to as LeNC-WIN and TwNC-WIN.

1. (3) In de Brusselse ziekenhuizen is er een schrijnend tekort aan vroedvrouwen.

   in the Brussels hospitals is there a harrowing shortage of midwives

   “In Brussels’s hospitals there is a harrowing shortage of midwives.”

   (DS, 11 September 2003)

2. (4) In de badkuip stond geen water.

   in the bathtub stood no water

   “In the bathtub was no water.”

   (HLN, 14 July 2000)

3. (5) Bij de fabriek in Amsterdam werken 285 werknemers.

   at the factory in Amsterdam work 285 employees

   “At the factory in Amsterdam work 285 employees.”

   (NRC, 26 April 2004)

4. (6) Over twee jaar is er een WK.

   in two years is there a World_Cup

   “In two years there will be a World Cup.”

   (AD, 26 June 2004)

Table 1. Slice of the WIN datasets (the first two examples are from LeNC and the last two are from TwNC)

L/Rn: nth word to the left/right of the er-slot.

Table 2 lists the number of distinct values (types) for each feature in LeNC-WIN and TwNC-WIN, as well as the number of types that occur in both datasets (overlap). This table shows that the overlap between the feature values in both varieties is generally rather sparse. The overlap is somewhat higher for features which tend to capture closed-class lexical items, such as prepositions and determiners, or several high- to mid-frequent verbs and nouns (e.g. the verb zijn “to be”; we will return to this in Section 6).

Table 2. Number of feature values in the WIN datasets

For the PAR approach, we employed a linguistically more informed way of feature selection. Although the WIN approach includes five words on either side of the item to be classified, it is uncertain that all the predictors known from previous research to be determinants of er’s distribution are actually included. This is the case in instances with long adjuncts, for example, like in (7), in which the temporal adjunct constituent op het moment dat De Visser en Rzasa hun gezicht lieten zien is 12 words long. In such cases, the nominal head of the adjunct, that is moment, falls out of the range of the window.

1. (7) Op het moment dat De Visser en Rzasa hun gezicht lieten zien weerklonk er slechts

   at the moment that D. V. and R. their faces let see resounded there merely

   een lauw applause.je

   a tepid applause.dim

   “The moment D. V. and R. showed their faces only a small applause resounded.”

   (AD, 29 July 1999)

In order to control for the presence of a number of canonical er-predictors, four syntactically informed lexical features were included in this respect: the preposition of the adjunct PP, the lexical head of the adjunct PP (which, in combination with the preposition, is a crude proxy for temporal or locative adjunct type), the verb, and the lexical head of the subject NP (the concreteness of the subject head noun was found in Grondelaers et al. Reference Grondelaers, Speelman, Drieghe, Brysbaert and Geeraerts2009 to significantly impact er-preferences). Applying this type of feature selection to the examples in (3)–(6) above yields a dataset as in Table 3. We will henceforth refer to the resulting datasets as LeNC-PAR and TwNC-PAR.

Table 3. Slice of the PAR datasets (the first two examples are from LeNC and the last two are from TwNC)

Here, too, the number of types for the four features as well as the overlap between LeNC-PAR and TwNC-PAR are given in Table 4. There are fewer feature values for closed word classes like the adjunct preposition, which because of its closed nature results in a high overlap between the LeNC and TwNC datasets. By contrast, the overlap is again quite low for more populated features, which contain open-class lexical items like nouns and verbs. We have also plotted the distribution of the individual feature values across LeNC and TwNC for both WIN and PAR in Figure 2.

Table 4. Number of feature values of the PAR datasets

Figure 2. Number of feature values by feature value frequency in LeNC (light gray) and TwNC (dark gray), for both WIN and PAR.

Finally, let us briefly illustrate how TiMBL calculates the distance between any two instances in the WIN and PAR feature formats. By default, TiMBL uses the overlap metric, which calculates the distance between two feature vectors X and Y as the sum of the differences between the features (Daelemans and van den Bosch Reference Daelemans and van den Bosch2005, pp. 28–9):

\begin{equation*}\Delta(X,Y) = \sum_{i=1}^{n}\delta(x_i,y_i)\end{equation*}

where

\begin{equation*}\delta(x_i,y_i)=\begin{cases}\frac{x_i - y_i}{\max_i - \min_i} & \text{if numeric, otherwise}\\[4pt]0 & \text{if $x_i = y_i$}\\[4pt]1 & \text{if $x_i \neq y_i$}\end{cases}\end{equation*}

All features are nominal, both for WIN and PAR, meaning that according to the above equation, a comparison of any two features yields a value of 0 in case of an exact match (i.e. overlap), and 1 otherwise (i.e. no overlap). We illustrate this for sentences (3) and (5). Starting with the WIN approach, we can see from Table 1 that these sentences (rows one and four) have two feature values in common, viz. is in the L1 position and een in the R1 position. The other eight features all have different values, resulting in a distance of $2 \times 0 + 8 \times 1 = 8$ . For the PAR approach (Table 3), both instances share only one feature, viz. the Verb feature is. In this case, as three out of four features are mismatches, the distance is equal to $1 \times 0 + 3 \times 1 = 3$ .

The overlap metric as illustrated here is TiMBL’s most basic implementation of similarity/distance between instances, but other, more fine-grained metrics can be used as well (e.g. character-based measures like Levenshtein distance). In addition, the example above assumes that each feature is equally important, but this needs not be the case, and various feature weighting methods are available. The task of hyperparameter selection is discussed in Subsection 5.1 below.

In view of the hypothesis that the ND distribution of er is determined by stronger lexical collocation, whereas BD speakers need additional higher-level cues (cf. Section 2), we predict for RQ2 that the WIN approach works better for ND, while it is less suited for BD. Conversely, BD can be expected to gain more from syntactically informed lexical information compared to ND, so one could expect the PAR approach to yield comparatively better results for BD.

5.1 Experimental setup

As indicated in Section 3, TiMBL operates with a number of hyperparameters, including the number of nearest neighbors taken into account for extrapolation (k), the similarity metric (m), the feature weighting metric (w), and the distance weighting metric (d). In addition to using TiMBL’s default parametric settings ( $k = 1$ , $m =$ overlap, $w =$ gain ratio, $d =$ no weighting), one can opt to use one or more non-default values in the function of a higher classification accuracy (cf. Hoste et al. Reference Hoste, Hendrickx, Daelemans and van den Bosch2002).

We wanted to take the influence of varying hyperparameter settings into account by iteratively using different combinations of hyperparameters from the possibilities in Table 5. Note that these do not exhaust all possible values; for m, w, and d, we selected the best-performing settings from earlier experiments in which we used van den Bosch’s (Reference van den Bosch2004) wrapped progressive sampling algorithm Paramsearch for automatic hyperparameter optimization.Footnote ^f For k, we selected values between 1 and 30, with increments of 5.

Table 5. Selected values for the TiMBL hyperparameters

The TiMBL experiments were carried out along two principal dimensions. The first dimension pertains to the respective feature representations for each national variety—window-based (WIN) versus parse-based (cf. Subsection 4.2). The second dimension involves the varieties, both in the training data and in the test data. Intra-varietal experiments (viz. training and testing within the same language variety) were contrasted with cross-varietal experiments (viz. training on the one variety and testing on the other), in both the WIN and the PAR approach. Following RQ3, we carried out this cross-varietal training because an additional way of gauging the syntactic individuality of BD and ND is measuring to what extent we can learn er-choices in the one variety on the basis of training material from the other.

Finally, we let sample size vary between 50 and 500 instances with and without er, with increments of 50. We did this because there is no way to know a priori how many instances the classifier needs to pick up on certain patterns in the data which are cues to er’s appearance. Each individual constellation of hyperparameters, feature representation, training and test variety, and sample size was repeated 10 times, each time randomly picking new training and test items from the balanced datasets, yielding 201,600 TiMBL runs in total.

5.2 Results and discussion

The results of the 201,600 experiments are visualized in Figure 3, by means of boxplots (with each box representing 2520 experiments). The four panels depict the two main experimental dimensions introduced in the previous section: on the horizontal axis, the WIN approach is contrasted with the PAR approach, while the vertical axis captures the distinction between intra- and cross-varietal training and testing. In each panel, the y-axis represents the accuracy (with the gray dashed line indicating the baseline of 0.5), while increasing sample sizes are plotted along the x-axis. The BD predictions are plotted in dark gray and the ND ones in light gray.

Figure 3. Boxplots capturing the accuracy for increasing sample sizes, both for WIN versus PAR feature representations and intra- versus cross-varietal training and testing.

First, across all four conditions (WIN and PAR, intra-varietal and cross-varietal) and varying sample sizes, the MBL classifier is able to predict er-choice fairly well, both for BD ( $M=0.717$ , $SD=0.105$ ) and ND ( $M=0.731$ , $SD=0.107$ ). These findings demonstrate that a learning algorithm which relies on no more than lexical input can rival sophisticated regression analysis with researcher-defined abstract predictors. As such, RQ1, repeated here, can be answered positively: with a fairly straightforward implementation of bare lexical context, we are able to predict er’s distribution correctly in over 70% of the cases on average.

1. RQ1 Can a learning algorithm such as MBL learn the distribution of er in BD and ND on the basis of raw lexical input—and lexical input only—or does the classifier need linguistically enriched information?

Crucially, however, the accuracies are consistently higher for ND than for BD—at least in the intra-varietal experiments (cf. the top two panels in Figure 3). In the cross-varietal experiments, the difference is much smaller (cf. the bottom two panels, ibidem). In addition, the interquartile range in accuracy scores in the WIN approach is much smaller than in the PAR approach (cf. the left-hand panels versus the right-hand panels, ibidem), suggesting that a crude bag-of-word feature representation may in fact be slightly more predictive than a syntactically more informed representation such as PAR—at least in our implementation of it. This is especially the case for smaller sample sizes; with increasing sizes, the difference decreases.

In order to verify if the patterns in Figure 3 hold under multivariate control, we fitted a linear regression model predicting TiMBL’s accuracies while taking all significant $2\times2$ interactions between the hyperparameters (cf. Table 5) into account, as well as a three-way interaction between train/test variety, sample size, and feature type; polynomials of degree 3 were used for the predictors k and sample size (F(78, $201,521)=6657$ , $p<0.001$ ; $R^2_{adj}=0.72$ ). The effect plots in Figure 4 visualize the predicted accuracies resulting from the regression model, both for the WIN and PAR feature representations. The intra- and cross-varietal experiments are diagrammed together in order to allow for easier comparison. We have used different line types for the training varieties and different grayscales for the test varieties. The lighter bands around the lines indicate the 95% confidence intervals of the fitted values.

Figure 4. Effect plots for a linear regression model predicting TiMBL accuracies.

These results can be used to answer RQ2:

1. RQ2 Does the alleged lexical specialization of er in ND materialize in comparatively higher predictive success on the basis of raw lexical input than in BD? And, conversely, does the MBL analysis of er’s BD distribution require linguistic enrichment?

The fitted values plotted in Figure 4 reveal that, under identical conditions, the ND distribution of er (i.e. ND trained on ND, the black dashed lines) is comparatively easier picked up by the classifier than er’s BD distribution (i.e. BD trained on BD, the gray solid lines). Focusing on the left-hand panel (the WIN approach), it appears that ND does not need a lot of lexical training material—100 cases with and without er suffice—to attain accuracies of over 0.7, and the models quickly reach a saturation point in accuracy of 0.76 around a sample size of 250 +ER and 250 –ER cases. This is also borne out by the trend that can be discerned in Figure 3. The PAR approach initially lags behind, but the slopes are slightly steeper, and with higher sample sizes the difference decreases. This means that our earlier observation that WIN outperforms PAR should be reformulated more strictly: it does so, but only for small sample sizes. The rapid increase in accuracy with waxing sample sizes may be taken as an indication that the classifier actually benefits comparatively more from syntactically informed lexical information—given enough instances. It could be that PAR will, at some point, with still larger sample sizes, overtake WIN, but more data are needed to verify if this is actually the case.

However, contrary to our expectations, BD does not benefit significantly more from syntactically informed lexical information than ND: both in the WIN and PAR conditions, the ND models consistently outperform the BD models, although the latter only “lag behind” by about 2–3%.

Finally, we turn to RQ3:

1. RQ3 Does the lexical specialization which is hypothesized to streamline er-production in ND entail that the much “messier” BD data are less suited to learn er’s ND distribution? And, conversely, can a Belgian classifier learn er-preferences from the more “lexically entrenched” ND material?

Two important results can be pointed out here. The first one is that the ND data appear to be a slightly better training model for the BD distribution of er than the BD data, especially within the PAR condition, where the difference is much more outspoken (compare the gray dashed and solid lines). Second, when trained on BD data, the ND classifier performs somewhat worse compared to the cases where it is trained on ND data.