2.1 Motivation for our selection of structures and proposed hierarchies

Three of our four experiments involve German structures. It is interesting to focus on German for the following reasons. First, in contrast to English, there are few experimental data available on the flexibility of German idioms (see Section 3.2). Second, there are German structures for which it is particularly contested to what extent there are limitations in the compatibility with idioms and whether these can be attributed to semantic restrictions: German prefield fronting, scrambling, and left dislocation (LD).

A hierarchy similar to Fraser’s (Reference Fraser1970) has been proposed for German by G. Müller (Reference Müller2000, Reference Müller2019). According to G. Müller (Reference Müller2000), prefield fronting is one of the least restrictive structures in German, followed by passivization, wh-movement (in particular, which-questions), and finally LD. G. Müller (Reference Müller2019) additionally proposes that scrambling is more restrictive than wh-movement. Like Fraser’s proposal, G. Müller’s hierarchy is also based on the compatibility of the structures with different groups of idioms.Footnote ⁴ The structures are illustrated in (8) for a non-idiom; examples from the literature containing idioms are provided throughout this section.

In our fourth experiment, we also test English structures. The experiment provides a link between our studies and previous experimental research on idiom flexibility, which has focused on English. Experiment 4 also serves to test the empirical adequacy of Fraser’s (Reference Fraser1970) hierarchy, according to which English nominalization without ‘of’ is one of the least restricted operations, passivization is more restricted, followed by nominalization with ‘of’, and clefting as the most restricted structure.

In Sections 2.2–2.9, we provide background on the structures that we test. We aim to provide examples from the literature showing that their compatibility with idioms is controversial and give illustrative examples of hypotheses about semantic restrictiveness.

2.2 German prefield

Prefield fronting refers to placing a constituent in the sentence-initial position preceding the finite verb in German main clauses. G. Müller’s (Reference Müller2000) assumption that it is one of the least restrictive structures in German is in line with the fact that even examples involving a non-compositional idiom in the prefield have been judged as felicitous in the literature. For example, the sentence in (10a) from Ackerman & Webelhuth (Reference Ackerman and Webelhuth1993) is discussed as a grammatical example by Nunberg et al. (Reference Nunberg, Sag and Wasow1994); see also Webelhuth & Ackerman (Reference Webelhuth and Ackerman1999).

However, the status of sentences like (10b) is not completely uncontested. Nunberg et al. (Reference Nunberg, Sag and Wasow1994: 512) report ‘a good deal of variability regarding these judgments’, and disagreements are also sporadically found in the literature. For example, (10b) is judged as well-formed by Fanselow (Reference Fanselow2004) but not by Frey (Reference Frey2004a).

Examples of theoretical approaches assuming prefield fronting to be semantically restricted are e.g. Fanselow (Reference Fanselow2004), who discusses a connection between the prefield position and focus or topic interpretation, and Frey (Reference Frey2004a, Reference Frey2010), who proposes a contrastive or emphatic interpretation of objects in the prefield, which should not be possible if the object is part of a non-compositional idiom. On the other hand, there are also analyses according to which prefield fronting does not have a semantic effect but rather is purely formal; this is the explanation that Nunberg et al. (Reference Nunberg, Sag and Wasow1994) propose for the high acceptability of idiom parts in the prefield; see also e.g. Fanselow & Lenertová (Reference Fanselow and Lenertová2011).

2.3 German scrambling

Scrambling refers to placing a constituent toward the left periphery of the sentence but without crossing the finite verb in main clauses or the complementizer in subordinate clauses. In G. Müller’s (Reference Müller2019, Reference Müller and Englisch2020) hierarchies, German scrambling is assumed to be among the most restricted structures, even more restricted than which-questions.

A study reported in Fanselow (Reference Fanselow, Kučerová and Neeleman2010) suggested that scrambling is acceptable at least for idiom parts that are definite (see Section 3.3). This view is not shared by everyone; a negative judgment for scrambling of a definite idiom part is e.g. reported by S. Müller (Reference Müller2010: 610), see (11).

There are approaches according to which scrambling moves specific (Diesing Reference Diesing1990) or topical (Frey Reference Frey2004b) objects toward the left periphery of the clause; this would predict that scrambling of idiom parts should be limited.

2.4 German left dislocation

German LD is a structure in which a constituent is placed in sentence-initial position and followed by a pronominal element. Cardinaletti (Reference Cardinaletti1986: 226, endnote 24) and Grohmann (Reference Grohmann, Billerey and Lillehaugen2000: 144) argued that prefield fronting and LD involve similar syntactic structures and report (12) and (13) as acceptable, respectively.

In contrast, negative judgments are provided by Jacobs (Reference Jacobs2001: 677, endnote 33), who notes that ‘idiom chunks (which are clearly non-referential) cannot be left-dislocated’, and by G. Müller (Reference Müller2000), who assumes that LD belongs to the most restricted structures.

An example of a theoretical approach that assumes prefield fronting to be semantically restricted is Frey’s (Reference Frey, Lohnstein and Trissler2004c) analysis of LD as a topic-marking structure.

2.5 Pronominalization

Pronominalization is another case in which the degree of semantic restrictiveness is controversial and thus worth testing. Nunberg et al. (Reference Nunberg, Sag and Wasow1994) argue that the antecedent of a pronoun needs to (individually) refer to something, which is the case for parts of compositional idioms like keep tabs on (figurative meaning: ‘to monitor someone’). They judge (14) as felicitous. The example stems from Bresnan (Reference Bresnan and Bresnan1982), who judged it as ungrammatical and argued that idiom parts cannot be antecedents for pronouns.

In contrast to German prefield fronting, scrambling, and LD, the potential restriction does not have to do with information-structural properties like topicality. Rather, the question is whether an idiom part like tabs it is able to introduce a referent that can be picked up by a pronoun like them in (14).

2.6 Passivization

In Experiments 3 and 4, we also test passivization. It has been argued that German passive is relatively unrestricted semantically – Nunberg et al. (Reference Nunberg, Sag and Wasow1994) proposed that this is the explanation for the observation that in German more idioms can undergo passivization than in English, which requires the promoted argument to be a topic (as discussed in Section 1). This is in line with G. Müller’s (Reference Müller2000) placement of German passive toward the unrestricted end of the hierarchy.

2.7 Nominalization

Nominalization is a structure that we would not expect to be restricted for semantic reasons. It is thus interesting and worth testing that one type of it (nominalization with ‘of’) has been claimed to be one of the most restricted structures in English by Fraser (Reference Fraser1970), while another type (without ‘of’) has been claimed to be one of the least restricted structures.

2.8 Which-questions

Which-questions are a structure that we definitely expect to be semantically restricted and to not be compatible with all idioms. A sentence like Which bucket did Mary kick? is associated with question semantics along the lines of ‘For which x, x being a bucket, is it true that Mary kicked x’?Footnote ⁶ For the question to make sense semantically, it is necessary that bucket has a meaning (in particular, that it denotes a property); otherwise, x is a bucket could not be evaluated. If semantic restrictiveness can be detected using test sentences with idioms at all, we have the clear expectation that which-questions will be restricted. In that sense, which-questions serve as a methodological sanity check; it can help to make sure that method is sensitive enough to detect semantic restrictiveness at all.

2.9 Cleftlike structures

Another structure that we would expect to be highly restricted and that can thus also serve as a methodological sanity check are English clefts. In Fraser’s (Reference Fraser1970) hierarchy, clefts are a highly restricted structure. We discuss cleft semantics in Section 4.4 in the context of our Experiment 4 on English.

2.10 Aims and limitations

We have mentioned a range of theoretical approaches as examples of assumptions about semantic restrictiveness of structures. We want to reiterate that the focus of this paper is not on evaluating the models’ assumptions about the exact nature of the potential semantic restrictions, i.e. we do not aim to answer questions like, Is prefield fronting restricted because it involves topicality or contrast?

What the various approaches crucially have in common is that they all make empirical predictions concerning the compatibility of certain syntactic structures with certain idioms. In view of the controversies concerning the acceptability of the discussed examples, we think it is useful to start at the basic level of determining which of these claims are descriptively accurate: Can we e.g. find evidence that prefield fronting is degraded with idioms, only degraded with some idioms (potentially non-compositional ones), or compatible with all idioms? What about other structures like scrambling and LD: Can we find evidence that they are more restricted than prefield fronting in this respect? Answering these empirical questions contributes to answering our two research questions: (i) Are some structures more restricted than others due to semantics, and (ii) Are some idioms more flexible than others due to compositionality? In Section 4.6, we give an outlook on the question of how a theoretical model’s exact predictions could be investigated in more depth in future work.

3.1 Previous theoretical approaches

The line of thought – that the acceptability of sentences containing idioms depends on how semantically restricted the structure is and how compositional the idiom is – can already be found in Nunberg et al. (Reference Nunberg, Sag and Wasow1994), even though this argument is not the main focus of their paper. As discussed in Section 1, they argued that English passivization is semantically restricted because it involves topicality. For further approaches that take up and discuss this idea, see also Kay & Sag (Reference Kay and Sag2014) for English and Bargmann & Sailer (Reference Bargmann, Sailer, Sailer and Markantonatou2018) for German. Bargmann & Sailer (Reference Bargmann, Sailer, Sailer and Markantonatou2018) argue against Nunberg et al.’s (Reference Nunberg, Sag and Wasow1994) categorical split between compositional and non-compositional idioms, but they agree with the idea that there is a crucial interaction between semantic properties of the idioms and semantic (language-specific) restrictions of syntactic structures.

3.2 Previous experimental research on English

Gibbs & Gonzales (Reference Gibbs and Gonzales1985) conducted experiments in which they first collected ratings of syntactic flexibility or frozenness and then collected processing times. In the frozenness pre-test, participants were asked to rate how similar the meaning of sentence pairs is, where one sentence contained an idiom and the other a non-idiomatic paraphrase:

The rating was intended to reflect the extent to which the idiomatic reading stays intact even when the syntax is modified. Gibbs & Gonzales tested five syntactic structures: nominalization without ‘of’, adverb insertion,Footnote ⁷ particle movement, passive, and nominalization with ‘of’. Low average similarity ratings across all tested structures were taken as an indicator of a ‘frozen’ idiom and high average similarity ratings as an indicator of a flexible idiom. They found that nominalization with ‘of’ received the highest similarity ratings, whereas the other structures did not differ significantly from each other. A by-item analysis showed a gradient continuum of syntactic flexibility among idioms. The flexibility estimate for each idiom gained in this first experiment then served as a basis for follow-up experiments on the processing of idioms. In the follow-up experiment, they found, among other results, faster reaction times to frozen idioms than to flexible ones in a lexical decision task.

In another influential study, Gibbs & Nayak (Reference Gibbs and Nayak1989) investigated whether the variability in syntactic flexibility found by Gibbs & Gonzales (Reference Gibbs and Gonzales1985) can be attributed to the semantic compositionality of the idioms. They collected judgments of compositionality in a first experiment and then tested whether this factor interacted with syntactic flexibility in a second experiment. Participants were asked to divide a set of idioms into three categories: ‘normally decomposable’ ones, in which each word makes a ‘unique contribution to the phrase’s nonliteral interpretations’ (Gibbs & Nayak Reference Gibbs and Nayak1989: 108); ‘abnormally decomposable’ ones, where the relation between the literal and non-literal interpretation is less direct; and ‘non-decomposable’ ones. For the experiment on syntactic flexibility, Gibbs & Gonzales’s (Reference Gibbs and Gonzales1985) method was adopted. They did not replicate Gibbs & Gonzales’s (Reference Gibbs and Gonzales1985) finding of highest ratings for nominalization with ‘of’; instead, nominalization without ‘of’ and adverb insertion received significantly higher similarity ratings than all other structures across all idioms. There was a significant interaction with compositionality: ‘non-decomposable’ idioms received lower ratings for adjective insertion and passive than the other idiom groups. A significant influence of compositionality was also found in a third experiment on pronominalization: ‘normally decomposable’ idioms received higher similarity ratings than the other categories.

Gibbs & Gonzales’s (Reference Gibbs and Gonzales1985) and Gibbs & Nayak’s (Reference Gibbs and Nayak1989) studies were influential with respect to the methods that they introduced for collecting compositionality categorizations and flexibility ratings. In both studies, variability between the idioms with respect to syntactic flexibility and between structures with respect to restrictiveness was found, but the findings were not consistent with respect to the question of which structure is the most restricted one.

3.3 Previous research on German

For German, fewer experimental data on syntactic flexibility of idioms are available than for English. Soehn (Reference Soehn2006), Bargmann & Sailer (Reference Bargmann, Sailer, Sailer and Markantonatou2018), Fanselow (Reference Fanselow2018), and Fellbaum (Reference Fellbaum2019) discuss corpus and web examples that provide the valuable insight that for most syntactic modifications, it is possible to find examples involving idioms, even with ones that are intuitively non-compositional. However, there are some challenges in interpreting the corpus data: without the possibility of comparing minimal pairs, it is difficult to assess which of the occurrences can be considered as evidence for syntactic flexibility and which cases are instances of deliberate bending of linguistic rules for rhetorical purposes.

An acceptability rating experiment on German idioms is reported by Fanselow (Reference Fanselow, Kučerová and Neeleman2010), who tested whether idioms parts can be scrambled. No significant difference was found between sentences with and without scrambling when the scrambled element was definite (5.8 vs. 5.4 on a 7-point scale), whereas a significant difference was found when it was indefinite (5.9 vs. 4.4). These results tentatively point toward the conclusion that scrambling is not so much restricted in terms of semantics or compositionality but rather by the formal property of (in)definite marking.

A detailed experimental investigation comparing a range of syntactic structures has not been done yet for German.

3.4 Methodological remarks

Because one of our goals is to investigate the properties of various syntactic structures, it is crucial to employ a method that allows to quantify as exactly as possible whether a certain structure is less acceptable with idioms in comparison to non-idioms.

In the studies reported in Section 3.1, the results for the tested structures varied between experiments. Whereas Gibbs & Gonzales (Reference Gibbs and Gonzales1985) found the highest mean rating for nominalization with ‘of’ in their Experiment 1 (5.58 on a 7-point scale) and the lowest for adverb insertion (4.22), Gibbs & Nayak (Reference Gibbs and Nayak1989) report the opposite for their Experiment 2: nominalization with ‘of’ received the lowest mean rating (4.44) and adverb insertion received the highest (5.28), although the same methodology was used.

Furthermore, Gibbs & Nayak’s (Reference Gibbs and Nayak1989) was replicated for Italian by Contrary to Tabossi et al.’s (Reference Tabossi, Fanari and Wolf2008) expectations and to Gibbs & Nayak’s findings for English, adverb insertion was the structure that showed the largest difference between compositional and non-compositional idioms in Tabossi et al.’s (Reference Tabossi, Wolf and Koterle2009) study, while no differences in this respect were found between the other tested syntactic structures (adjective insertion, passive, and LD). Tabossi et al.’s (Reference Tabossi, Fanari and Wolf2008) findings thus further support the view that there is a systematic relation between compositionality and syntactic flexibility, but their findings deviate from Gibbs & Nayak’s findings with respect to which structures show the strongest effect of compositionality, although the same methodology was used and no difference was expected between the languages on theoretical grounds.

These studies had a different focus; thus, the deviations are not crucial for the main conclusions that the authors of these studies draw, e.g. concerning correlations between compositionality and processing measures. For our purposes, however, it is crucial to get reliable estimates of the acceptability of each structure; thus, it is important to consider what could have caused the deviations between experiments.

The deviations might have several reasons. First, the employed methodology requires to choose a paraphrase for each idiom. This might introduce a confound: when participants are asked to judge how similar e.g. a sentence like ‘the law was laid down’ is to the paraphrase ‘strict orders were given’ (Gibbs & Gonzales Reference Gibbs and Gonzales1985: 245), the judgment arguably depends not only on the passivizability of ‘lay down the law’ but also on the passivizability of ‘give strict orders’; a different paraphrase might lead to a different result. Second, it is possible that even though the ratings are intended to reflect whether the expression retains its idiomatic meaning in the given syntactic structure, the ratings might also be influenced by other factors like plausibility or complexity – for example, it might be more difficult to judge whether two sentences have the same meaning when the sentence is harder to process, which might result in a lower rating. Because no minimal pairs or sets of items were used across structures, e.g. ‘They will lay the law down / give strict orders if the party gets too wild’ for particle movement vs. ‘Her father’s laying down the law / giving strict orders prevented her from going to the dance’ for nominalization (Gibbs & Gonzales Reference Gibbs and Gonzales1985: 245), differences between the conditions might result from such non-syntactic factors.

We therefore employ a different method, namely acceptability ratings, as also used by Maher (Reference Maher2013).Footnote ⁸ This exempts us from the need to provide a paraphrase for the idioms and thus eliminates the risk of confounds connected to this. Furthermore, we aim to construct minimal pairs that only differ in the syntactic manipulation of interest. Using acceptability instead of similarity ratings also comes with the advantage that it allows adding an important baseline: in addition to comparing different types of idioms, we can also compare them directly to non-idioms. We also include a baseline with canonical word order. The baselines help to quantify exactly to what extent any observed differences are due to the specific interaction between idiomaticity and syntax that we are interested in.

A further relevant finding, reported by Tabossi et al. (Reference Tabossi, Wolf and Koterle2009), is that providing a pragmatically suitable context significantly raises the ratings for syntactically modified idioms (except when the modification violates a grammatical requirement). We also provide contexts to ensure that a sentence is not just rejected because pragmatic motivation to use this structure at all is lacking.

We employ this methodology throughout our four experiments.

4.1 Experiment 1 (German, first set of structures)

4.1.2 Design and materials

Three factors were manipulated in this study in a 3 × 2 × 5 design. The first factor was compositionality of the VP with three levels: non-idiomatic, compositional idiom, and non-compositional idiom. The second factor was context with two levels: a broad focus context and a polarity focus context (illustrated below). The third factor was syntactic structure with five levels: canonical word order (baseline), fronting to the prefield, LD, scrambling, and pronominalization or anaphor. We describe the factors and their levels in in more detail below.

We selected German VP idioms from a corpus-based collection by Jan-Philipp Soehn. The idioms are listed in Table 1. All consist of a definite determiner phrase (DP) and a verb. Two of us (native speakers of German) categorized them independently for compositionality based on our intuition, following the criterion of whether both the verb and the DP have their own individual figurative meaning that combine compositionally to form the figurative meaning of the VP. We selected 12 idioms with congruent annotation, aiming at choosing maximally clear cases: six idioms that we categorized as compositional (i.e. as having individual figurative meanings for the DP or verb) and six that we both categorized as non-compositional.

Table 1 German idioms used in Experiments 1–3.

Note that the notion of semantic restrictiveness that we are interested in is not included explicitly as a factor in our experiment. We are able to draw conclusions about semantic restrictiveness by considering the factors structure and compositionality. Our assumption is that if a syntactic structure is more semantically restricted than another, this should be reflected in a certain kind of interaction with our idiom categories: a highly restrictive structure should be compatible with fewer idioms (by assumption, only with compositional ones). For this to work, it is crucial that the same idiom groups are used in all critical conditions (i.e. that minimal pairs or sets of items are used) and that one of the categories is overall more likely to contain compositional idioms than the other – thus, our intuitive categorization should provide a viable proxy of compositionality for our purposes, even if the distinction between the categories is not categorical and completely clear-cut. However, it is important to note that the intuitive categorization comes with certain caveats – we return to this issue in Section 4.5, where we take a closer look at individual idioms, and in the outlook on directions for further research in Section 4.7.

We added six non-idiomatic VPs: den Bus verpassen ‘to miss the bus’, den Manager verärgern ‘to upset the manager’, die Fenster putzen ‘to clean the windows’, den Hausschlüssel verlieren ‘to lose the housekey’, den Rasen mähen ‘to mow the lawn’, and die Anlage ausmachen ‘to turn off the stereo’.

The same 12 idioms and six non-idioms were used throughout Experiments 1–3.

The target sentences were presented in one of two different contexts. Context was a between-subjects manipulation: each participant either saw all items in the first type of context or all items in the second type of context. The first context type always contained a why-question, inducing broad focus in the answer sentence. The second context type contained a yes / no-question, inducing polarity focus in the answer sentence. The VP in the target sentence was discourse-given in the latter context type by virtue of a synonym used in the yes / no-question. The two context types are illustrated in (16).

One purpose of systematically controlling the context is to get an impression of whether decreased acceptability can be alleviated by providing a suitable context.

In addition, we hypothesize that the context manipulation can potentially provide a further way to identify semantically restricted structures. The two contexts induce different information-structural effects in the target sentence. If it is the case that some of the tested syntactic structures involve a topical and/or contrastive interpretation of the object (as it has been proposed for prefield fronting, scrambling, and LD), we might see an informative difference between the context types. In the broad focus context, the VP provides new information. Thus, the context is not compatible with interpreting the VP or a part of it as given information or topic. The polarity focus context, on the other hand, leaves more scope for interpretation in this respect. Here, the denotation of the VP is discourse-given by virtue of the synonym in the question (e.g. give up – throw in the towel). Interpreting the VP as topical and/or contrastive would require a certain amount of accommodation but would not be at odds with the context: the reader could construe an implicit contrast like ‘As for throwing in the towel, the two of them would never do that (in contrast to other activities)’. For non-idioms and compositional idioms, the possibility of interpreting the direct object on its own as a contrastive topic could also be available at least to some extent (‘As for the ice, the two of them would never break that (in contrast to other things)’, while this should not be possible with non-compositional idioms. The predictions are spelled out in more detail in Section 4.1.3.

We constructed five syntactic variants of each of the 18 items (the 12 idioms and six non-idioms).Footnote ¹⁰ This is illustrated in (17) for the polarity focus context. The structure of the anaphor condition is constructed similarly as Nunberg et al.’s (Reference Nunberg, Sag and Wasow1994: 502) example We thought tabs were being kept on us, but they weren’t, which is a modification of one of Bresnan’s (Reference Bresnan and Bresnan1982) examples.

The stimuli were distributed such that every participant saw every item in all levels of the factors structure and compositionality (only context was manipulated between subjects). The motivation for this was that we assumed that there might be variation between participants with respect to how familiar they are with each idiom and how acceptable they find it in general; by collecting ratings for each idiom in all conditions, we can be more confident that any differences that we find can be attributed to the syntactic manipulation. This decision led to a high number of critical items that each participant saw. To limit the overall length of the experiment, we included only a relatively small number of stimuli that were not part of the critical items described above. These stimuli partly served to check some potential caveats concerning judgments of idiomatic expressions, e.g. the question of whether participants tend to generally tolerate grammatical violations with this type of material. The design and results of these additional stimuli are reported in Appendix A.

4.1.4 Results

The results of Experiment 1 are summarized in Figure 1 and Table 2.

Figure 1 Experiment 1 – mean ratings (close-ended 1–7 point scale; error bars represent 95% confidence intervals).

Table 2 Results of Experiment 1 – mean ratings (standard deviations).

For statistical analysis, the factor context was sum-coded. For the factor compositionality, forward-difference coding was used (i.e. the level compositional idiom is compared to non-idiomatic, and non-compositional idiom is compared to compositional idiom). For the factor structure, treatment coding with canonical as the baseline was used (all other structures are compared to canonical). This also means that the model output for context, compositionality, and context*compositionality is to be interpreted as simple effects or interactions, i.e. effects within the baseline level of structure (canonical word order).

A linear mixed model was used for inferential statistical analysis.Footnote ^{11 ,} Footnote ¹² For our research question, it is relevant which interactions with structure are significant, and we therefore only report on these below; the simple effects or interactions are not informative with respect to our hypotheses. The exact specification of the models and the full output can be found in Appendix B2.

We found a significant two-way interaction between context and structure for each level of structure in the following direction: the acceptability difference between the canonical baseline and the other structures was reduced in the polarity context in comparison to the broad focus context for prefield (t = –8.40, p < 0.001), LD (t = –3.78, p < 0.001), scrambling (t = –6.07, p < 0.001), and anaphor (t = –4.05, p < 0.001). We also found a significant two-way interaction between compositionality and structure for each level of structure when comparing compositional idiom to non-compositional idiom: the acceptability difference between these two idiom categories was larger than in the canonical baseline for prefield (t = –5.10, p < 0.001), LD (t = –3.63, p < 0.001), scrambling (t = –2.74, p = 0.01), and anaphor (t = –3.88, p < 0.001). We did not find significant interactions between compositionality and structure when comparing non-idiomatic to compositional idiom.Footnote ¹³ We did not find significant three-way interactions.

Besides the planned contrasts concerning the factor structure (comparing canonical to all other levels), we also conducted a post hoc analysis to test if the structures prefield, LD, scrambling, and anaphor differ from each other with respect to the interaction with the factor compositionality. The goal of these additional tests is to check to what extent our results are compatible with the proposed hierarchies holding between syntactic structures. The post hoc analysis was done by running models in which another level of the factor structure was set as the baseline. Bonferroni correction was used to compensate for the higher likelihood of erroneous inferences in multiple testing. None of these additional pairwise comparisons was significant. Numerically, the largest differences were found for prefield vs. LD and prefield vs. anaphor when comparing compositional idioms to non-compositional idioms. The detailed results of the post hoc analysis can be found in Appendix B3.

4.2 Experiment 2 (German, replication of Experiment 1)

One of the goals of the second experiment is to replicate Experiment 1 in order to corroborate our findings about the tested structures and to make sure our methodology yields replicable results. A second goal is to include a new set of additional (filler) stimuli for a better grasp of the acceptability level that we observe; these are discussed in Appendix A.

4.2.3 Results

The results of Experiment 2 are summarized in Figure 2 and Table 3.

Figure 2 Experiment 2 – mean ratings (close-ended 1–7 point scale; error bars represent 95% confidence intervals).

Table 3 Results of Experiment 2 – mean ratings (standard deviations).

The same analysis procedure as reported above for Experiment 1 was used.

We found a significant two-way interaction between context and structure for each level of structure in the following direction: the acceptability difference between the canonical baseline and the other structures was reduced in the polarity context in comparison to the broad focus context for prefield (t = –8.04, p < 0.001), LD (t = –6.47, p < 0.001), scrambling (t = –5.79, p < 0.001), and anaphor (t = –3.30, p = 0.002). We also found a significant two-way interaction between compositionality and structure for each level of structure when comparing compositional idiom to non-compositional idiom: the acceptability difference between these two idiom categories was larger than in the canonical baseline for prefield (t = –2.27, p = 0.03), LD (t = –3.64, p < 0.001), scrambling (t = –3.02, p = 0.006), and anaphor (t = –4.18, p < 0.001). We did not find significant interactions between compositionality and structure when comparing non-idiomatic to compositional idiom. We did not find significant three-way interactions. The complete fixed effect results can be found in Appendix B2.

Again, we conducted a post hoc analysis to see if prefield, LD, scrambling, and anaphor differed from each other with respect to the interaction with compositionality. As in Experiment 1, none of the pairwise comparisons was significant. The full results are shown in Appendix B3.

4.3 Experiment 3 (German, second set of structures)

Experiments 1 and 2 showed a systematic difference between the idioms that we categorized as non-compositional and those that we categorized as compositional, but as for the tested structures, no consistent differences in the idiom behavior were found. The goal of Experiment 3 is to test further structures, namely passive, nominalization, and which-questions. For which-questions, there is the clear expectation that they should be restricted in their compatibility with idioms because which-questions are uncontroversially semantically restricted, as discussed in Section 2. Including this condition thus should help to see whether our method is sensitive enough to detect structures that are more semantically restricted than others at all and what kind of pattern we should expect to see for them.

4.3.2 Design and materials

For this experiment, we dropped the context manipulation. In Experiments 1 and 2, we were mainly interested in testing structures that potentially involve an information-structural restriction, which motivated comparing two information-structurally different contexts, whereas this is not the focus of Experiment 3. Experiments 1 and 2 showed that the polarity context had a general facilitating effect: it raised the acceptability of marked structures in comparison to the broad focus context (without erasing crucial contrasts between conditions). In our view, it is beneficial if the items are as acceptable as possible with non-idiomatic idioms – this helps ensure that any observed deviations in acceptability can specifically be attributed to the manipulated factors (compositionality and structure) rather than to the lack of a suitable context. We thus chose to adopt the polarity focus rather than the broad focus context for Experiment 3.

As in Experiments 1 and 2, we tested three categories of VP (non-idioms, compositional idioms, and non-compositional idioms). The canonical baseline, prefield fronting, and LD were replicated from Experiments 1 and 2. In addition, passive, nominalization,Footnote ¹⁴ and which-questions were tested. The additional conditions are illustrated in (18). The same VPs as in Experiments 1 and 2 were used. For the which-questions, the contexts were slightly adjusted: instead of providing a yes or no question in the context (which would be odd to follow-up by another question), we only provided a declarative sentence.

4.3.3 Results

The results of Experiment 3 are summarized in Figure 3 and Table 4.

Figure 3 Experiment 3 – mean ratings (close-ended 1–7 point scale; error bars represent 95% confidence intervals).

Table 4 Results of Experiment 3 – mean ratings (standard deviations).

The same analysis procedure as reported above for Experiments 1 and 2 was used.

We found a significant interaction between compositionality and structure for the following levels of structure when comparing compositional idiom to non-compositional idiom: the acceptability difference between these two idiom categories was larger than in the canonical baseline for prefield (t = –2.56, p = 0.02), LD (t = –3.62, p < 0.001), and passive (t = –2.22, p = 0.04). The difference between these idiom categories was not significantly different from the canonical baseline for nominalization (t = –1.06, p = 0.30) and which-question (t = –1.31, p = 0.20).Footnote ¹⁵ The interaction between compositionality and structure when comparing non-idiomatic to compositional idiom was only significant for which-question (t = –3.62, p = 0.002). The difference between these idiom categories was not significantly different from the canonical baseline for prefield (t = –0.74, p = 0.47), LD (t = –0.07, p = 0.95), passive (t = –0.20, p = 0.84), and nominalization (t = –1.57, p = 0.13).Footnote ¹⁶ The complete fixed effects model results can be found in Appendix B2.

In post hoc comparisons, we tested whether prefield, LD, passive, nominalization, and which-question differed from each other with respect to the interaction with compositionality. The numerically largest contrasts were found when comparing the which-question to all other structures with respect to the gap between non-idioms and compositional idioms; the contrast was significant for which-question vs. LD. See Appendix B3 for detailed results.

4.4 Experiment 4 (English)

In the fourth and final experiment reported here, we test a range of English structures: nominalization with and without ‘of’, passive, a cleftlike structure, and pronominalization. These structures have been experimentally investigated before using the similarity-rating method by Gibbs & Gonzales (Reference Gibbs and Gonzales1985) and Gibbs & Nayak (Reference Gibbs and Nayak1989). The first goal of retesting is to see if we can find further corroboration of the assumption that semantic restrictiveness of syntactic structures is detectable in our paradigm: among the English structures that we test is a cleftlike structure for which we assume that it is clearly semantically restricted and which can thus serve as a further test case, similar to the which-questions tested in Experiment 3: it should show a clearly different pattern than the canonical baseline. If this premise holds, a second goal is to get a clearer impression of the more controversial conditions: our methodology allows for a more direct comparison of the syntactic structures than the previous similarity-rating studies that were designed with different primary research goals. In particular, it will be informative to see whether the passive is indeed limited with respect to idiomatic expressions (as claimed, among others, by Nunberg et al. Reference Nunberg, Sag and Wasow1994) and whether we can find support for a difference between nominalization with and without ‘of’, e.g. ‘Mary’s kicking (of) the bucket’, as claimed by Fraser (Reference Fraser1970) and not consistently resolved by previous studies.Footnote ¹⁷

4.4.2 Design and materials

As in Experiment 3, we only tested one type of context (polarity focus). We selected the idioms for our study based on the results of Gibbs & Nayak’s (Reference Gibbs and Nayak1989) categorization task: we adopted six of the idioms that were mostly categorized as ‘normally decomposable’ by their participants (we assume this to be the category most closely corresponding to our ‘compositional’ idioms in Experiments 1–3) and five of the idioms that were mostly categorized as ‘non-decomposable’. To the latter group, we added one idiom (bite the bullet) that we considered to be non-compositional and that was not in Gibbs & Nayak’s list, based on the intuition that the other reported idioms might not be familiar enough to the young speakers of British English whom we intended to test. The idioms are shown in Table 5.

Table 5 English idioms used in Experiment 4.

We added six non-idiomatic VPs: forget the timer, eat the cake, cut down the hedges, reveal the trick, paint the car, and throw away the cutlery.

We tested six syntactic structures, as illustrated in (19). We refer to condition (19d) as ‘cleftlike’, because it shares properties with clefts: it is a biclausal structure involving a copular construction and relativization. We chose this structure rather than it-clefts (‘It is the question that he popped’) or pseudo-clefts (‘What he would never pop is the question’) because it was easier to construct plausible items while keeping the context constant. It-clefts are semantically restricted because they have exhaustive semantics roughly along the lines of ‘There is only one x for which it is true that Harry would never pop x, and x is the question’ (see e.g. Velleman et al. Reference Velleman, Beaver, Destruel, Onea and Coppock2012 for a formal analysis). Our cleftlike condition does not involve exhaustivity, but its meaning can be paraphrased as ‘There is a set S of elements that Harry would never pop, and the question is an element of S’ – similar to our reasoning concerning the semantics of which-questions and it-clefts, this should also be impossible to interpret or evaluate if the expression the question lacks an individual denotation.

4.4.3 Results

The results of Experiment 4 are summarized in Figure 4 and Table 6.

Figure 4 Experiment 4 – mean ratings (close-ended 1–7 point scale; error bars represent 95% confidence intervals).

Table 6 Results of Experiment 4 – mean ratings (standard deviations).

The same analysis procedure as for Experiments 1–3 was used.

We did not find a significant interaction between compositionality and structure when comparing compositional idiom to non-compositional idiom: the acceptability difference between these two idiom categories was not different from the canonical baseline for any of the tested structures (nominalization with ‘of’: t = 0.52, p = 0.60; passive: t = –1.62, p = 0.12; cleftlike: t = 0.45, p = 0.66; anaphor: t = –0.82, p = 0.42; and nominalization without ‘of’: t = 1.55, p = 0.13). The interaction between compositionality and structure when comparing non-idiomatic to compositional idiom was only significant for cleftlike (t = –3.89, p = < 0.001). The difference between these idiom categories was not significantly different from the canonical baseline for nominalization with ‘of’ (t = 0.85, p = 0.40), passive (t = –0.30, p = 0.76), anaphor (t = 0.00, p = 1), and nominalization (t = 1.88, p = 0.07). The complete model results for the fixed effects can be found in Appendix B2.

In post hoc comparisons, we tested whether the structures differed from each other with respect to the interaction with compositionality. The numerically largest contrasts were found when comparing the cleftlike condition to all other structures with respect to the gap between non-idioms and compositional idioms; the contrast was significant for cleftlike vs. nominalization with ‘of’, cleftlike vs. nominalization without ‘of’, and cleftlike vs. anaphor. See Appendix B3 for detailed results.

4.5 Exploratory by-item analysis

Our conclusions about the effect of compositionality on idiom flexibility rely on the assumption that our categorization of ‘compositional’ and ‘non-compositional’ idioms really reflects the intended property. We conducted a post hoc by-item analysis to explore other potential influencing factors and individual differences between the tested idioms.

For this, we pooled the data from all German experiments. The means for canonical word order, prefield, and LD are therefore based on data from Experiments 1–3; the means for scrambling and anaphor are based on data from Experiments 1 and 2; and the means for passive, nominalization, and which-question are based on data from Experiment 3. The results for each of the 18 items (12 idioms and six non-idioms) are shown in Figure 5. The by-item results for the English idioms can be found in the OSF repository. By-participant results are presented and discussed in Appendix C.

Figure 5 By-item results based on pooled data from Experiments 1–3.

Visual inspection shows that while on average, the non-compositional idioms (in the first row of Figure 5) show a larger acceptability difference between canonical word order and the other structures than the compositional ones (second row), there are also individual differences. When we focus e.g. on the distance between canonical and prefield, the non-compositional idioms that show the largest acceptability gap involve a unical element, i.e. one that does practically not occur outside of the idiom, namely ‘Garaus’. It is notable that the compositional idiom that shows the largest gap is ‘den Laufpass geben’, which also involves a compound that only occurs within this idiom. Another property that these two VPs have in common is that they require a dative object; this also applies to ‘die Leviten lesen’.

A further observation is that for the passive condition, there are large individual differences, even within our compositionality categories: for example, within the idioms categorized as compositional, there are some that can be felicitously passivized (das Kriegsbeil begraben) and some that cannot (den Faden verlieren).

4.6 General discussion

Our results are compatible with the view that there are syntactic structures that are semantically restricted, meaning that some types of constituents (in particular, expressions that do not have an individual meaning, like elements that are part of an idiom) cannot appear in them felicitously. For German prefield, LD, scrambling, anaphor, and passive, we found that their acceptability with a part of our idiom set (the idioms categorized as non-compositional) is significantly lowered in comparison to non-idioms; for German which-questions and the English cleftlike structure, we found lowered acceptability with the idioms categorized as compositional in comparison to non-idioms.

Our results are informative with respect to proposed restrictiveness hierarchy of the tested syntactic structures. For English, we observed similar patterns for nominalization with and without ‘of’, which speaks against the assumption that these two structures are at opposite ends of a restrictiveness hierarchy with respect to idioms, as proposed by Fraser (Reference Fraser1970). As for German, our results go against the view that LD and scrambling are more restricted than which-questions while prefield fronting is almost unrestricted, as proposed by G. Müller (Reference Müller2000, Reference Müller2019). Our results do not exclude the possibility that there are more fine-grained differences between prefield fronting, LD, and scrambling that we failed to detect, but if they exist, they seem to be perceived as more subtle by most speakers than has been proposed. Regarding the German passive, the high variability observed in the by-item analysis suggests that compositionality is not the (only) influencing factor here – the availability of passivization seems to depend not only on compositionality but also on further properties that varied both between and within our idiom categories.

Our experiments also provide methodological insights. First, context plays a role: there was a consistent increase of acceptability in the polarity context in comparison to the broad focus context in Experiments 1 and 2, suggesting that this is a factor that should not be neglected when estimating the syntactic flexibility of idioms; without a suitable context, it might be underestimated how acceptable even non-compositional idioms can be in non-canonical structures. Second, as shown in particular by Experiment 4 on English, it is important to include non-idioms and sentences with canonical word order as a baseline; otherwise, the importance of observed contrasts in marked syntactic structures could be overestimated.

The next logical question is what the findings mean for grammatical models of the tested structures. To answer it, it is helpful to take into account absolute ratings and effect sizes.Footnote ¹⁹ An observation that we can make based on visual inspection of our data is that there are some structures which are consistently rated above the midpoint of our scale in the facilitating polarity context (4 on the 7-point scale), even with non-compositional idioms (in particular, the German prefield condition), while others are clearly rated in the range below the midpoint when occurring with idioms (e.g. which-questions). This observation suggests that prefield fronting is semantically less restricted than which-questions; but does it support models that predict parts of non-compositional idioms to be ungrammatical in which-questions and grammatical in the prefield position? To evaluate this, we need to compare the observed effect sizes to other types of grammatical violations in future work.

To illustrate this suggestion, take for example Frey’s (Reference Frey2010) theory of the German prefield. In this model, an object occurring in the prefield position needs to have a certain information-structural interpretation (emphasis or contrast), which is not possible if it is a part of a non-compositional idiom. However, if this requirement is not met, the model does not predict ungrammaticality. The requirement is encoded in terms of a conventional implicature that is violated when the constituent in the prefield does not have the required interpretation. In order to test the model’s exact predictions empirically, we could compare the effect size observed for test sentences with idioms (as in our experiments) to other well-established cases of conventional implicature violations (Potts Reference Potts, Ramchand and Reiss2007), ideally within one experimental set-up. Our experiments provide a starting point for such more in-depth investigations.

4.7 Outlook

Our experiments open up further directions for future research. First, potential differences between the syntactic structures could be investigated in more detail in an experiment specifically designed toward detecting more fine-grained contrasts between a smaller number of structures.

Second, our manipulation of the factor compositionality in the German experiments was based on our own intuitive categorization. There are certainly cases that other speakers perceive differently. For example, as pointed out by a reviewer, while we categorized das Kriegsbeil begraben ‘to bury the hatchet’ as compositional, based on the intuition that the hatchet figuratively refers to a conflict and burying means ending it, a different plausible view is to see the action as a whole as a symbol for making peace, similar to how throwing in the towel (which we categorized as non-compositional) in the boxing ring symbolizes surrendering. To take into account the subjectiveness of compositionality and to increase generalizability, compositionality ratings could be collected from a large number of speakers in future experiments (following Hubers et al.’s Reference Hubers, Cucchiarini, Strik and Dijkstra2019 assessment that judgments of idiom properties can be collected reliably). This would also make a more fine-grained approach to compositionality possible, rather than splitting the idioms into two categories.

Third, as it has been shown that compositionality correlates with other idiom properties, there is the question of whether there really is a causal relation between this factor and syntactic flexibility. The by-item analysis (Section 4.5) suggested that unicality is a factor in which the tested idioms vary and which could be related to limited syntactic flexibility. If this is the case, then a part of the observed difference between the idiom categories could be attributed to the fact that our set of non-compositional idioms contained several unical elements – Garaus, das Zeitliche (nominalized form of zeitlich ‘temporal’), and Leviten. Unicality can be seen as an extreme case of non-compositionality; it is thus important to check whether our conclusions hold up when a broader spectrum of idioms is tested. More generally, it could play a role for syntactic flexibility in how likely the DP is to occur with this verb and vice versa (conditional frequencies, as discussed by Müller & Englisch Reference Müller and Englisch2020) or whether DP and verb independently co-occur with similar words and phrases (similarity of semantic vectors, as discussed by Gehrke & McNally Reference Gehrke and McNally2019). A related question is what role unicality plays in the categorization of an idiom as compositional or non-compositional. A reviewer expressed the view that idioms consisting of a unical element and a semantically light verb like den Garaus machen ‘to kill’ are special in that neither the DP nor the verb has a figurative meaning: since the DP only occurs in the idiomatic sense, there is no distinction between literal and idiomatic meaning; and since the verb is semantically light, its function within the idiom is not really different from literal uses, either. In our experiments, we decided to subsume this case under ‘non-compositional’, following the criterion that it is not possible to assign individual figurative meanings to the DP and verb.Footnote ²⁰ However, we agree that this reinforces the view that idioms are a highly heterogeneous set of expressions and that a more fine-grained analysis is desirable in follow-up studies.

Fourth, our findings are based on 12 idioms that were deliberately homogeneous with respect to their structure (verb and definite direct object, without negation) to avoid confounds and to ensure that all tested syntactic structures could be constructed with them. It is desirable to test whether our findings generalize to other types of idioms. On the other hand, there are also structural properties in which our items varied (e.g. whether they require a dative object), which could be controlled and studied more systematically.

A fifth and final direction for future research (suggested by a reviewer) is embedding idiomatic materials like ours in acceptability studies with a higher proportion and wider range of non-idiomatic sentences, as figurative expressions are much rarer in natural speech than in our experiments.