Idioms are often seen as a “fuzzy category” (Nunberg et al. 1994:492) that is not easy to define precisely. Prototypical examples of idioms like kick the bucket or spill the beans share the characteristic that the meaning and/or function of the whole expression is conventionalized, i.e., it is not fully derivable from knowledge about its parts (Nunberg et al. 1994, ibid.). For example, a speaker who knows the meaning and use of kick and bucket in isolation but is not familiar with the idiom kick the bucket could not predict that the expression is conventionally associated with the meaning to die. A similar concept is that of “constructions” as expressions whose “form or function is not strictly predictable from its component parts” (Goldberg 2003:219). However, not all expressions that are “conventionalized expressions” or “constructions” in this sense would standardly be referred to as idioms. Idioms also prototypically involve some kind of figurative/metaphorical speech: e.g., collocations with a specific, conventionalized, but non-figurative meaning like the fixed expression tax and spend (referring to a certain government policy; Nunberg et al. 1994:494) are usually not categorized as idioms; but the distinction is often not clear-cut.

The fuzzy and multi-dimensional nature of idioms makes it difficult to conceptualize idiomaticity as a formal property in the grammar: it is not straightforward what formal properties a [+/– idiomatic] feature would correspond to. Thus, if there is indeed an observable difference in syntactic flexibility between prototypical idioms like kick the bucket and non-idioms like eat the apple, an explanation in terms of more precisely definable linguistic properties is called for.

We take a complex expression to be (semantically) compositional if its meaning can be derived from the meaning of the individual parts and general semantic mechanisms, like combining functions and arguments (going back to ideas by Frege 1891; formalized, a.o., by Montague 1974). We follow Nunberg et al.’s (1994) view that all idioms are highly conventionalized, but some of them can nevertheless be compositional with respect to their figurative meaning.1 To illustrate this idea, let us elaborate more on the examples spill the beans vs. kick the bucket. Both are conventionalized: a speaker who knows the meaning of spill and beans does not necessarily know the idiomatic meaning of spill the beans; the same holds for kick the bucket. However, a speaker who is familiar with the idiom spill the beans might be able to establish a relation between the literal and the figurative meaning of the idiom in a way not available for kick the bucket. In spill the beans, spill can be understood as figuratively corresponding to the concept of revealing, and the beans to the secret; and these individual figurative meanings then combine compositionally to the complex meaning reveal the secret. Thus, in a sense, both the verb and the object have an individual (figurative) meaning within the idiom here. For kick the bucket, finding such individual figurative meanings for kick and the bucket that would then combine compositionally is less straightforward.

However, if the compositionality of an idiom is conceived of as a native speaker’s intuitive ability to assign individual figurative meanings, compositionality is subjective: possibly there are speakers who do not perceive spill the beans as compositional, and, conversely, also speakers who see kick the bucket as compositional (e.g., by having kick correspond to end, and the bucket to life). We will treat compositionality as a subjective and gradient property here: in Experiments 1 and 2, we collect individual speakers’ judgments and derive a gradient compositionality measure for idioms.

Gibbs & Nayak (1989) collected compositionality judgments and tested syntactic flexibility (as well as other idiom properties, a.o., transparency); see 2.2.2. for their method. They report an effect of compositionality on syntactic flexibility and conclude that compositional idioms are “more syntactically flexible” (Gibbs & Nayak 1989:100) than non-compositional ones with respect to some of the tested structures, in particular adjective insertion and passive. They report that compositionality also significantly affected the participants’ reactions to pronominalization of a part of the idiom.

Tabossi et al. (2008) replicated the study for Italian. They also report an effect of compositionality on syntactic flexibility, but only for adverb insertion, while no contrast was found for any of the other tested structures.

In three previous experiments, we investigated the syntactic flexibility of German and English idiomatic verb phrases (Wierzba et al. 2023). The following German structures were tested in these experiments: object movement to the prefield (the pre-verbal position in V2 clauses), object left dislocation, object scrambling, object pronominalization, passive, nominalization, object which-question. The following English structures were tested: passive, pronominalization, a cleft-like construction (“Kicking the bucket is something that…”), and two types of nominalization.

We found a robust effect of compositionality on syntactic flexibility in German: non-compositional idioms consistently showed lower acceptability in non-canonical syntactic constructions than compositional ones, while they were similarly acceptable in sentences with canonical word order.

For English, the picture was different: a gap between non-compositional and compositional idioms was observed as well, however, it was not significantly larger in the non-canonical structures than in the canonical baseline. Thus, in contrast to German, the gap could not be attributed to limited syntactic flexibility of non-compositional idioms in English.

In both languages, compositional idioms were as acceptable as non-idioms in most of the non-canonical syntactic structures, but some deviated from that pattern: for which-questions in German and the cleft-like construction in English, a larger part of our idiom set (including some of those that had been categorized as compositional) was judged as degraded than in the other constructions. This result was in line with the theoretical assumption that these two syntactic constructions impose a semantic requirement on the object and are thus less compatible with idiomatic VPs (where the object does not necessarily have an individual meaning, in particular when it is perceived as non-compositional, see Section 2.1.2) than constructions without such a requirement.

To estimate the syntactic flexibility of idioms, Gibbs & Nayak (1989) asked participants to rate how similar the meaning of sentence pairs like (5) was. One sentence contained an idiom (lay down the law) and the other a non-idiomatic paraphrase (give strict orders). The sentence pairs were presented in various syntactic variations, a.o., structures containing passivization as in (5), or nominalization. High similarity ratings across all tested structures were interpreted as an indicator of ‘syntactically flexible’ idioms, whose idiomatic reading remains intact even with modified syntax.

The use of idiom-paraphrase pairs to assess syntactic flexibility has been adopted by later studies (e.g., Abel 2003, Tabossi et al. 2008), but criticized by Maher (2013) and Wierzba et al. (2023) – one problem is that similarity judgments of pairs like (5) can depend not only on the passivizability of the idiom, but also of the paraphrase. We follow the latter approaches in using acceptability ratings instead of similarity ratings to estimate the syntactic flexibility of idioms.

Gibbs & Nayak (1989) also used idiom-paraphrase pairs to estimate idiom compositionality. They presented 40 idiom-paraphrase pairs similar to (5) and asked the participants “to decide whether the individual words in each expression made some unique contribution to the phrase’s nonliteral interpretations” (Gibbs & Nayak 1989:108). For example, in (5), participants judged whether the verb lay down and the object the law each make an individual contribution to the idiom’s figurative meaning of giving strict orders. If a participant said no, the idiom was categorized as “non-decomposable”; if they said yes, the idiom was categorized as “decomposable”, and the participants were asked to further judge whether the relation between the literal and figurative meaning was direct (“normally decomposable”) or indirect (“abnormally decomposable”); the latter more fine-grained subcategorization will not be relevant here.

Methodological criticism was raised by Maher (2013): all idioms tested by Gibbs & Nayak (1989) consisted of a transitive verb and an object, but this was not the case for the paraphrases. While some of them also involved a transitive verb and an object in parallel to the idiom (lay down the law/give strict orders), others only consisted of an intransitive verb (kick the bucket/die), an intransitive verb and an adjunct (chew the fat/talk aimlessly), or a predicative construction (pack a punch/to be powerful). All idioms whose paraphrase deviated from the idiom’s transitive verb + object structure were categorized as “non-decomposable” by the majority of the participants. Maher (2013) hypothesized that the form of the paraphrase might have influenced participants’ reactions in the categorization task: it could be easier to draw a connection between an idiom’s individual literal parts and its figurative meaning if the idiom and the paraphrase have parallel structures.

In our previous experiment on English, which examined whether compositionality affects syntactic flexibility, the categorization was adopted from Gibbs & Nayak’s (1989) empirically based lists of “decomposable” and “non-decomposable” idioms. In our previous experiment on German, we intuitively categorized the idioms as “compositional” or “non-compositional”. In this paper, we will replicate these studies, but will use a method to estimate the compositionality of an idiom neither requiring introspective categorization by the authors nor choosing literal paraphrases. Thus, we are going to reassess the previous findings with a new empirically grounded measure.

The research reported here builds on previous studies, including our own previous experiments (Wierzba et al. 2023). We would thus like to clarify the relation to the present study and to stake out the scope and limitations of this paper.

The focus of our previous experiments was on providing experimental data concerning the effect of compositionality on syntactic flexibility of idioms in German and English. The methodological focus was on finding a way to estimate syntactic flexibility reliably. The theoretical focus was on assessing previously proposed hierarchies of syntactic structures with respect to their compatibility with idioms, like the hierarchy proposed for English by Fraser (1970) mentioned in Section 1.2 (a more detailed review of the structures and the motivation for selecting them can be found in Wierzba et al. 2023).

In the new experiments presented here, we reassess the compositionality factor, both methodologically and theoretically. Our first methodological goal is to reassess previous findings using an adjusted compositionality task. We also test a new set of materials that is larger and contains a wider range of idioms in order to test whether the compositionality effect is generalizable.

Our conceptual goal is to reconsider why and how we should expect compositionality to affect syntactic flexibility and what its explanatory limitations are. In the previous experiments, non-idioms just served as a baseline against which the idiom behavior was interpreted. In the present paper, we go a step further and ask whether we can find common sources of variability (beyond compositionality) in syntactic flexibility for both idioms and non-idioms. In the next section, we will discuss some recent research concerning parallels between idioms and non-idioms in various grammatical domains.

Tabossi et al. (2009) report a piece of evidence revealing parallels between idioms and non-idioms. They found that providing a suitable context significantly raises the ratings for syntactically transformed idioms – except when the transformation violates a general formal requirement: in Italian, (non-idiomatic) bare nouns cannot appear in preverbal subject position. The same restriction was also found for bare nouns contained in idioms, and it was not alleviated by providing a context.

Horvath & Siloni (2009, 2019) observed, based on corpus searches of Hebrew and English idioms, that for most diatheses (voices) of the verb, there are idioms that uniquely occur in this diathesis (e.g., some idioms only occur in the adjectival passive: caught in the middle ‘fig. be between two opposing sides’, #catch X in the middle), but there are also diatheses for which this is not the case (e.g., verbal passive). Horvath & Siloni argue that idiomatic VPs are not stored as independent entries in the lexicon, but by “subentry storage”; for example, kick the bucket is stored as a subentry of the head kick. The argument is that if the idioms were stored as independent, holistic entries, it would be difficult to capture the systematic dependence on the head’s diathesis. Horvath & Siloni show that their observation also holds for non-compositional idioms and thus, the representation of all idioms needs to have an internal structure to some extent (similar to non-idiomatic VPs). This proposal goes against Nunberg et al.’s (1993) proposal of distinguishing between compositional idioms with internal structure, and non-compositional idioms with holistic VP representations.

Bargmann & Sailer (2018) also argue against the view that non-compositional VP idioms should be represented as holistic chunks, based on web examples from German and English. They show that even non-compositional idioms are syntactically flexible to some extent (a finding corroborated by Fellbaum 2019), and that their syntactic modification underlies similar restrictions as in the case of compositional idioms and non-idioms. They propose an analysis in which even non-compositional idioms consist of individual lexical entries, but they differ from compositional idioms in that the entries involve semantic redundancy: for example, in their analysis of kick the bucket, the semantic contribution of the and bucket is contained in the contribution of kick.

Gehrke & McNally (2019) show that idioms that canonically contain a certain determiner can also occur with other determiners under certain circumstances; determiner variability is found with respect to definiteness and number, e.g. (based on examples by Bruening et al. 2018 and Everaert 2017):

Gehrke & McNally model such cases by separating the descriptive content (which can be compositional or non-compositional) from the potential to introduce discourse referents (anaphoric potential). The anaphoric potential interacts with compositionality: if an idiom is compositional for a speaker and it contains a DP that has its own figurative meaning, e.g., a rat in (6), the DP can be assumed to have its normal function of introducing a referent, in this case, a suspicious entity. When changed to ‘many rats’, several individuals are introduced as referents. As for an idiom like kick the bucket, whose descriptive content is non-compositional for most speakers, Gehrke & McNally (2019:794–796) discuss the possibility that it only introduces a referent at the level of events, but no referents at the level of individuals. The determiner could then express whether one or several events – as in (7) – are introduced.

The approaches above have in common that they stress commonalities between idioms and non-idioms and argue in favor of an analysis that reflects the parallels. Horvath & Siloni (2009; 2019), Bargmann & Sailer (2018), and Gehrke & McNally (2019) all propose models in which even non-compositional idioms are represented in a way that allows access to its individual parts, for purposes of modifying diathesis, word order, or determiners. The latter two approaches nevertheless provide ways to represent differences between compositional idioms and non-compositional idioms without assuming that non-compositional idioms are unanalyzable chunks.

In this paper, we contribute experimental data to this line of research by studying parallels between idioms and non-idioms, in particular in Experiment 3.

The materials were adapted from our previous experiments on German (Wierzba et al. 2023). They included twelve idiomatic verb phrases, e.g., das Handtuch werfen (lit. ‘to throw the towel’, fig. ‘to give up’) and six non-idiomatic ones, e.g., den Bus verpassen ‘to miss the bus’. All idioms (throughout our Experiments 1–3) consist of a verb and a direct object. For a complete item list, see Appendix A.

All items were presented in the context shown in (8). In our previous experiments, we found that this type of context, which induces polarity focus in the target sentence, leads to higher ratings across various syntactic constructions than a broad-focus context. This helps to avoid the potential problem of comparing conditions that are perceived as unacceptable for independent reasons.

Each item was constructed in eight conditions (levels of the factor structure):

In our previous experiments, structures a/b/c/d/e were tested first, and a/c/d/f/g/h were tested in a follow-up experiment (some of the structures were included again to make sure that the results were comparable across experiments). Every participant saw each of the 18 items (12 idioms, 6 non-idioms) in all of the tested structures. The motivation was to reduce the risk that potential differences are due to individual variation with respect to how acceptable participants find each idiom in general rather than the syntactic manipulation.

In our replication, we combine the predecessor experiments into one study. To nevertheless stay close to the original designs, we created two lists: half of our participants saw each idiom and non-idiom in structures a/b/c/d/e, and the other half saw them in a/c/d/f/g/h. As a consequence, more judgments were collected for structures a/c/d, resulting in an unbalanced amount of observations per structure. In our Experiment 3 with new materials, we will use a different item distribution; here, our priority was to mirror the studies we replicate as closely as possible for comparability.

20 filler items were also adopted from the original materials. 10 contained a singular/plural manipulation of idioms (e.g., in den sauren Apfel beißen, lit. ‘to bite into the sour apple’, fig. ‘to do something necessary but unpleasant’ / in die sauren Äpfel beißen ‘to bite into the sour apples’). The other 10 contained minimizers, which usually only occur under negation (keinen Schimmer haben ‘have no clue’, #einen Schimmer haben ‘have a clue’), tested with and without negation. Each participant saw half of the fillers of each type in a condition which we expected to be acceptable (idioms containing a DP in its canonical singular/plural form or a minimizer licensed by negation) and half in a condition expected to be degraded (due to a number deviation or unlicensed minimizer).

In sum, each participant rated either 90 or 108 critical items (18 items in five/six conditions) and 20 fillers. The presentation order of stimuli was randomized.

The experiment was set up using the online questionnaire platforms SoSciSurvey (Leiner 2019) and L-Rex (Starschenko & Wierzba 2023). There were two parts to the experiment.

In the first part, participants provided compositionality ratings for 12 idioms. They were instructed that they would be asked questions about expressions with a figurative meaning. Participants were given two examples: aus einer Mücke einen Elefanten machen (lit. ‘to turn a mosquito into an elephant’, fig. ‘to blow a small issue out of proportion’) and ins Gras beißen (lit. ‘to bite into the grass’, fig. ‘to die’). They were told that for the first example, it is possible to divide the expression up into two individual parts that each have their own figurative meaning, i.e., ‘a mosquito’ corresponds to a small problem, and ‘an elephant’ corresponds to a big fuss, and that this division is less straightforward for the second example. They then saw one idiom per page and answered the question ‘Can this idiom be divided up into two parts that each have their own figurative meaning?’. There were four response options: (i) Yes, namely: (here, participants could enter paraphrases of the figurative meanings), (ii) Yes, but I cannot really express the individual meanings, (iii) No, (iv) I am not familiar with this idiom.

In the second part, participants were shown short dialogs including the same idioms and asked to rate the acceptability of the answer sentence on a scale from 1 (unacceptable) to 7 (acceptable).

There was a gap between the two parts, during which participants were asked to answer two unrelated questionnaires. These questionnaires served as a buffer between our two experiment parts and allowed us to make sure that the compositionality and acceptability tasks, which both involved the same idioms, did not directly follow each other. On average, the whole study took about one hour to complete.

48 native speakers of German took part. They were recruited via prolific.co and received £10 for participation. The pre-screening filters were set in such a way that participants spoke German as their first language, had been raised monolingually, and were born in and current residents of Germany.

If a participant indicated that they were not familiar with an idiom in the pre-test, the participant’s ratings for sentences containing this idiom were not included in the analysis of the acceptability ratings.

The compositionality ratings are shown in Figure 2. The acceptability ratings are illustrated in Figure 3.

For statistical analysis, the factor idiomaticity (idioms vs. non-idioms) was sum-coded in order to treat the two levels symmetrically. The factor structure was treatment-coded with canonical as the baseline to which the other structures are compared. This contrast-coding allows us to test whether there is an acceptability contrast between idioms and non-idioms in the canonical baseline, and whether this contrast is larger in the other structures.

We ran linear mixed models (LMMs) as well as cumulative link models (CLMs). For reasons of space, we only report the main LMM results in the paper. We provide the detailed model specifications and output of the LMM analyses in Appendix C. The CLM results are available in our Open Science Framework (OSF) repository under https://doi.org/10.17605/OSF.IO/JPUFT. It has been shown that linear mixed models can increase Type I and Type II errors and distort estimates of effect size when applied to ordinal rating data (Liddell & Kruschke 2018; Veríssimo 2021). The theoretical conclusions that we draw in this paper are based on tests that consistently yielded significant results in both the LMM and CLM analysis. We summarize and discuss any deviations that occurred between the model types in Appendix B.2.

According to an LMM5 fit to the whole data set, no simple effect of idiomaticity was found, i.e., no significant idiom/non-idiom contrast in the canonical baseline. As for the interaction between idiomaticity and structure, a larger contrast between idiom and non-idiom than in the canonical baseline was found for anaphor (t = –2.54, p = 0.02), prefield (t = –3.65, p = 0.002), LD (t = –3.14, p = 0.005), nominalization (t = –4.39, p < 0.001), and which-question (t = –5.23, p < 0.001), while the contrast was not significantly different from the canonical baseline for scrambling and passive.

An additional LMM was fit to the data subset including only idioms in order to test the influence of compositionality on flexibility. The compositionality measure that we used as a dependent variable was the proportion of positive responses to the question that we asked in the compositionality task (excluding “I am not familiar with this idiom” responses). For example, the compositionality value for das Handtuch werfen ‘to throw in the towel’ was 0.32 (32% of participants said that each part of the idiom had its own figurative meaning), while it was 0.75 for das Kriegbeil begraben ‘to bury the hatchet’. We included this compositionality rating as a linear predictor in our model. No simple effect of compositionality was found, i.e., no significant linear effect of compositionality on the ratings in the canonical baseline. As for the interaction between compositionality and structure, a larger effect of compositionality than in the canonical baseline was found for anaphor (t = 13.78, p < 0.001), prefield (t = 2.45, p = 0.004), LD (t = 2.30, p = 0.04), scrambling (t = 3.04, p = 0.01), and which-question (t = 2.45, p = 0.03), while the effect was not significantly different from the canonical baseline for passive and nominalization.

The estimate of the effect of compositionality is illustrated in Figure 4. The crucial interaction between compositionality and structure is illustrated by the steepness of the fitted lines and the width of the confidence bands. For example, the steeply increasing fitted line in the scrambling structure with a narrow band indicates that idioms with a higher compositionality value tend to correspond to higher acceptability of scrambling with relatively high confidence. In contrast, the broader confidence band and the flatness of the line in the passive structure indicates that there is little evidence for a linear relation between compositionality and the acceptability of passivization in our data – there are some highly compositional idioms that can easily be passivized, but also some that cannot be passivized felicitously, and the same holds for highly non-compositional idioms.

All significant effects reported above according to the LMM were also significant in the CLM analysis. We will only interpret these effects in the discussion below. We summarize and discuss any additional significant effects found in the CLM but not in the LMM in Appendix B.2.

Our results replicate several of Wierzba et al.’s (2023) findings, with the crucial difference that this time, compositionality was measured empirically by collecting judgments from participants.

First, there is a significant gap between the acceptability of idioms and non-idioms in most of the tested syntactically marked structures. This supports the view that on average, there is an empirical difference in the syntactic behavior between idioms and non-idioms. Second, our results are in line with our previous finding that compositionality of German idioms is a significant predictor of their syntactic flexibility; we have replicated this using an empirically grounded measure of compositionality. Our results furthermore show that it is worth having a differentiated look at syntactic flexibility: idioms with a higher compositionality rating receive higher acceptability ratings in marked structures on average, but a by-structure look reveals that this does not necessarily hold for every tested structure in the same way. A significantly larger effect of compositionality (in comparison to canonical word order) was found consistently (in both the LMM and CLM analyses) for the following constructions: anaphor, prefield, LD, scrambling, and which-question.

Our results substantiate the view that there is not only variability in the syntactic flexibility of idioms, but also among non-idioms. For most of the tested structures, the interquartile ranges of the non-idioms suggest a similarly broad distribution of data points as for idioms. An approach in terms of compositionality can only explain the variability in the behavior of idioms, as non-idioms do not vary in compositionality. The question thus remains whether there are factors that systematically determine the syntactic flexibility of the non-idioms, and whether it is the right approach to assume that they are different from the factors determining the flexibility of idioms. We will pursue this question further in Experiment 3.

An additional post-hoc observation is that a subset of the tested structures seems to correlate with each other with respect to the acceptability of individual idioms. Visual inspection of Figure 4 suggests that the more acceptable an idiom is with prefield fronting, the better it works also with LD and scrambling. For initial notes and hypotheses on the observed correlations, see Appendix B.1 and our data repository; the discussion of these correlations is beyond the scope of the present paper, but is worth further inspection in future work.

A closer look at the compositionality ratings in comparison to the introspective categorization used in our previous experiments reveals some informative deviations. While most of the items previously categorized as non-compositional (items 1–6) received more ‘no’ responses than those previously categorized as compositional (7–12), some of the latter fall outside of the pattern, a.o. den Tiefpunkt erreichen (lit. ‘to reach the lowest point’, fig. ‘to be in the worst possible situation’). A special property of this item is that Tiefpunkt can be used in a metaphorical sense independently of the verb erreichen; thus, as pointed out by a reviewer, it is only partially idiomatic. A limitation of our method is that we ask participants whether both parts of the idioms (verb/object) have their own figurative meaning. In cases like den Tiefpunkt erreichen, we get a high proportion of ‘no’ responses; however, the reason is not that the verb and the object do not combine compositionally (they definitely do), but partial idiomaticity. It is thus probable that our method underestimates compositionality in cases like this, and partially idiomatic expressions should be avoided.6

Another note about the results in comparison to the previous experiments is that the average acceptability ratings are relatively similar numerically.7 This alleviates the potential worry (raised by two anonymous reviewers) that the compositionality rating task might have influenced participants’ acceptability ratings.

The materials were adapted from our previous experiment on English (Wierzba et al. 2023); the items were based on lists of compositional/non-compositional idioms provided by Gibbs & Nayak (1989). They included twelve idiomatic verb phrases, e.g., pop the question (fig. ‘to propose’) and six non-idiomatic ones, e.g., forget the timer. For a complete item list, see Appendix A. Each item was constructed in six conditions:

As in the original study we are replicating, each participants saw each of the 18 items (12 idioms and 6 non-idioms) in all of the tested structures a–f.

20 filler stimuli were adopted from the original study, which were designed in the same way as the German fillers in Experiment 1: 10 contained a singular/plural manipulation (e.g., turn over a new leaf / turn over new leaves), and 10 contained minimizers with/without negation (e.g., I did not get a wink of sleep / #I got a wink of sleep).

In sum, each participants rated 108 critical items (18 items in six conditions) and 20 fillers. The presentation order was randomized.

The experiment was set up in a similar manner as described for Experiment 1. There was a gap of at least two hours between the compositionality and acceptability rating tasks. In contrast to Experiment 1, participants were not asked to complete other questionnaires during the gap. In sum, the two study parts took about 40 minutes to complete. The pre-screening filters that we set on prolific.co only allowed speakers who spoke English as their first language and were current residents of the UK to take part. Participants received £6.25 for participation.

If a participant indicated that they were not familiar with an idiom in the pre-test, their ratings for sentences containing this idiom were not included in the analysis of the acceptability ratings.

The compositionality ratings are shown in Figure 5. The acceptability ratings are illustrated in Figure 6.

According to a LMM fit to the whole data set, there was a simple effect of idiomaticity. As for the interaction between idiomaticity and structure, a larger contrast between idiom and non-idiom than in the canonical baseline was found for passive (t = –2.52, p = 0.02) and the cleft-like structure (t = –5.00, p < 0.001). The contrast was not significantly different from the canonical baseline for anaphor and nominalization with “of”. A significant deviance in the opposite direction (smaller contrast between idiom and non-idiom than in the canonical structure) was found for nominalization without “of” (t = 2.19, p = 0.04).8

An additional LMM was fit to the data subset including only idioms in order to test the influence of the factor compositionality. The same compositionality measure was used as in Experiment 1. A simple effect of compositionality was found: ratings were higher for more compositional idioms in the baseline. This means that idioms that were perceived as more compositional tended to be judged as more acceptable even in sentences with canonical word order, perhaps due to higher familiarity (we will come back to this below). As for the interaction of compositionality with structure, a larger positive effect of compositionality than in the canonical baseline was not found for any of the tested structures. The effect of compositionality did not differ from the one found in the canonical baseline for anaphor, passive, and the cleft-like structure. A significant deviance from the canonical baseline (an in comparison less positive effect of compositionality) was found for nominalization with “of” (t = –2.87, p = 0.004) and nominalization without “of” (t = –4.15, p < 0.001). The estimate of the effect of compositionality is illustrated in Figure 7. As in Experiment 1, a steep fitted line with a narrow confidence band indicates a strong linear relation between compositionality and the acceptability of the syntactic structure at hand. However, it is important to note that in contrast to Experiment 1, there already is a linear relation, i.e., a steep fitted line in the canonical baseline condition. Thus, the steepness of the fitted line in the other conditions, e.g., the passive structure, needs to be compared to the canonical baseline: only if the line was significantly steeper for passive than for canonical, it would indicate that higher compositionality facilitates passivization; this is however not the case here, neither for passive nor for any of the other marked structures; in some of the structures (the nominalization structures) the positive linear relation is even significantly less pronounced.

The same pattern of significant and non-significant effects as in the LMMs was found in the CLM analysis with the exception of the deviance noted in footnote 9, which is not relevant for the discussion below.

Our results confirm previous findings of Wierzba et al. (2023), but also go against some previous claims. In particular we failed to find the positive effect of compositionality on syntactic flexibility that Gibbs & Nayak (1989) reported for English.

As in Experiment 1, we do see a gap between the acceptability of idioms and non-idioms in a part of the tested syntactically marked structures (passive, cleft-like) that is larger than in the canonical structure. This lends further support to the view that there is indeed an empirical difference in the syntactic flexibility of idioms and non-idioms that requires an explanation; however, as in Experiment 1 on German, a differentiated look is useful, as the gap is not present (at least not in a stronger form than in the canonical baseline) in all syntactically marked structures.

Like Wierzba et al. (2023), we found a general effect of compositionality for English in our Experiment 2: sentences with idioms judged as more compositional tended to be more acceptable. However, again, this effect was found to already be present in the baseline structure with canonical word order, and it was similar in size and partially even smaller in all of the syntactically marked structures that we tested. Thus, this experiment does not provide evidence that compositionality is the factor that explains the idiom/non-idiom gap in syntactic flexibility. The results confirm that it is important to interpret any contrasts found in the syntactically marked constructions against a canonical baseline (which was lacking in Gibbs & Nayak’s 1989 study) – otherwise, contrasts between various idioms that arise due to independent factors and have nothing to do with syntactic flexibility might be overinterpreted.

As for the question of why we see a compositionality effect even in the canonical baseline, we speculate that familiarity might play a role. Even though we excluded participants’ data points for idioms that they judged as non-familiar, there still might be gradient differences within the remaining idioms, potentially leading to lower acceptability ratings as a result of feeling unsure about some of the stimuli. If the less compositional idioms happened to be less familiar to the participants, this could be the reason for the observed correlation between compositionality and acceptability.

It can further be noted that the compositionality judgments participants gave in this experiment do not fully line up with the categorization reported by Gibbs & Nayak (1989) and adopted in Wierzba et al. (2023). We provide a more detailed comparison in Appendix B.3.

In 3.1, we discussed the caveat that one of our German items was only partially idiomatic. In the German example den Tiefpunkt erreichen (lit. ‘to reach the lowest point’, fig. ‘to be in the worst possible situation’), Tiefpunkt can signify a bad situation independently of erreichen. The English items of Experiment 2 illustrate a similar type of item for which compositionality might have been underestimated (we thank a reviewer for underscoring this point). The idiom miss the boat can be paraphrased as miss an opportunity. Boat can signify opportunity in other examples such as The boat has sailed or Look, the boat’s gone, let’s just cut our losses and go. For this reason, participants might have hesitated to say that miss has its own figurative meaning in miss the boat, as miss can be interpreted literally here if boat corresponds to opportunity. Similar considerations apply to the verb clear in clear the air and the nouns question and law in pop the question and lay down the law. However, visual inspection of the compositionality judgments in Figure 5 shows a much lower proportion of negative responses (<50%) to the question whether both idiom parts have a figurative meaning for all of these English examples than for den Tiefpunkt erreichen in Experiment 1. This suggests that the availability of a literal interpretation of the DP/verb did not necessarily prompt participants to give a negative response in the compositionality task, but it is nevertheless a point that is worth being considered in future work.

Finally, we do not want to argue that the results of Experiment 2 rule out the possibility that compositionality affects the syntactic flexibility of idioms in English – there are various potential reasons for the absence of an effect: it could be due to methodological limitations like the sample size or, e.g., the fact that we only tested for a linear relation between the two properties. It could also be due to differences between the tested constructions. Both our German and English materials included several structures that we assumed to potentially depend on compositionality for semantic reasons: for English, these included anaphor, passive (argued to be linked to topicality/referentiality a.o. by Nunberg et al. 1994 and reported to depend on compositionality by Gibbs & Nayak 1989), and the cleft-like construction (linked to focus/exhaustivity; see Wierzba et al. 2023 for more detailed discussion). However, a part of the German structures for which we found a compositionality effect did not have a direct counterpart in the English materials (a.o., prefield, LD, scrambling) – if manipulations of linear order are the type of structure for which one is most likely to find a compositionality effect in our type of experimental design, there was less opportunity to find it in the English experiment.

Despite these limitations, we would like to stress the insight that the choice of the task used to measure compositionality and syntactic flexibility influences the results of the task, as evidenced by the divergent conclusions of Gibbs & Nayak 1989 vs. Wierzba et al. (2023) and the present study. Thus, previous findings of a positive effect of compositionality on flexibility should be taken with a grain of salt, and the question seen as re-opened.

In Experiment 3, we test indefinite and definite DPs, which are related to reference in several ways. First, they can introduce discourse referents (Karttunen 1969), which can then be picked up by pronouns (e.g., Mary ate an apple. It tasted great.); we will refer to this ability as ‘anaphoric potential’. Definite expressions presuppose a (unique or familiar) referent to already be present in the discourse and point to it (Heim 1991; Schwarz 2009). However, in appropriate contexts, it is often possible for speakers to accommodate a unique referent even if it has not been introduced explicitly (Singh et al. 2016); in this way, definite expressions can also introduce referents. Thus, we assume that in principle, both definite and indefinite DPs have anaphoric potential.

What about idiomatic DPs? As argued by Gehrke & McNally (2019), definite and indefinite DPs that are contained in idioms can in principle also introduce referents at the level of individuals (i.e., they have anaphoric potential), as long as the idioms are compositional (see Section 2.2.4). Non-compositional idioms, on the other hand, are more limited in this respect. Since pronominalization is one of the phenomena that has been used to estimate syntactic flexibility, anaphoric potential is relevant for our research questions.

A further DP property related to reference is specificity. Specificity is a property that is often discussed in connection with different readings of indefinite DPs, as in (11).

According to Fodor & Sag (1982), the difference between the readings is whether the speaker has a certain referent “in mind”. Under this view, definite DPs are often specific, because they usually refer to an already established discourse referent.9 For indefinite DPs, we assume that a specific reading can be enforced by the context as in (14b), but that it is not always a salient interpretation.

And idiomatic DPs? Analogously to the discussion of compositionality and anaphoric potential above, we also think that compositionality is a prerequisite for interpreting a part of an idiom as specific: only if an expression has an individual (figurative) meaning of its own within the idiom, speakers can have a certain referent for it in mind.

Specificity is relevant for the discussion of syntactic flexibility of idioms, because it has been argued to have an impact on word order modifications, in particular scrambling (Diesing 1990; Lenerz 2001; Frey 2004; Fanselow 2018). Frey (2004) gives the example in (12) to illustrate that scrambling the object over the adverbial results in a specific interpretation of ein Kind ‘a child’, which is infelicitous in the provided context that enforces an unspecific interpretation.

There are approaches according to which there is a relation between scrambling and other structures, which might therefore also be impacted by specificity: e.g., it has been proposed that prefield fronting of objects involves a step of object scrambling (Müller 2004), and that LD, in turn, involves prefield fronting (Grohmann 2000).

We will use the term referentiality in a dual sense here: We take an expression to be referential if it has anaphoric potential or it is specific. In this sense, referentiality is a semantic property that idioms and non-idioms possess to varying degrees. Thus, it has the potential to provide a unified explanation for (a part of) the variability found in the syntactic flexibility of idioms as well as non-idioms. Under this view, we would expect compositionality to correlate roughly with syntactic flexibility – however, not due to a direct relationship between these two properties, but due to a more complex dependency: compositionality of an idiom is a prerequisite for referentiality, and referentiality, in turn, is a prerequisite for some syntactic modifications.

If the reasoning in 4.1.1–4.1.2 is correct, we would expect similar behavior of definite/indefinite DPs with respect to pronominalization. As for scrambling, a contrast between definite/indefinite DPs should be detectable if a specific reading is not salient in the given context. For idioms, the availability of pronominalization and scrambling should additionally depend on compositionality.

Besides idioms, there are also nominal elements which have special referential properties even though they do not involve figurative meaning. An example of this type of expression that we will focus on in Experiment 3 are nouns in German predicates like Zeitung lesen ‘read (a/the) newspaper’. We follow Frey (2015:258) in assuming that such predicates involve a “very close syntactic junction” between the verb and the noun and in referring to it as incorporation. This type of construction combines a verb with a (semantically/pragmatically) “typical” direct object. A special property is that the object occurs in a bare (articleless) form, even if it is a singular count noun – usually, only mass nouns and indefinite plurals can occur without an article in German. The construction shows several syntactic characteristics differing from typical complex transitive VPs, for example when it comes to the interaction with sentence negation (see Frey 2015 for details and further evidence).

Modarresi & Krifka (2021) report speaker judgments showing a gradient pattern of anaphoric potential for a set of items with a bare singular noun; e.g., while the noun in Zeitung lesen completely lacks anaphoric potential as illustrated in (13), the noun in other examples like Kuchen backen ‘bake (a/the) cake’ can serve as an antecedent to some (limited) extent.

With respect to specificity, we follow Frey (2015:230) in assuming that incorporated nouns are non-specific.

Our motivation for including incorporated nouns in Experiment 3 is that they can potentially help to disentangle referentiality from idiomaticity. If the assumptions above are correct, verb phrases with incorporated nouns should show limited syntactic flexibility, even though they are non-idiomatic.

The compositionality rating task was adjusted in comparison to Experiments 1–2. First, we asked which of the idioms participants were familiar with. Then, we only asked for compositionality ratings for the familiar idioms. We asked separately whether each of the two parts of the idiom (verb/DP) had its own figurative meaning. The motivation for this adjustment was that it highlights which idiom part participants failed to assign an individual meaning to. To nevertheless make the results comparable across experiments, we will assume that two positive responses in Experiment 3 (“Does the verb/DP have its own meaning?”) correspond to a positive response in Experiments 1–2 (“Does each part have its own figurative meaning?”) in our analysis. At least one negative response in Experiment 3 will correspond to a negative response to the single question in Experiment 1–2. We will return to the additional information provided by the more fine-grained task in the discussion below.

The acceptability rating task was the same as in Experiments 1–2.

60 native speakers of German (again recruited via prolific.co) took part. They received £9 for participation.

16 new idiomatic VPs were used. Eight of them included a definite DP, e.g., die Wogen glätten (lit. ‘to smooth the waves’, fig. ‘to calm things down’), and eight included an indefinite DP, e.g., eine dicke Lippe riskieren (lit. ‘to risk a fat lip’, fig. ‘to say something cheeky’). In addition, 16 non-idiomatic VPs were used. Four included a definite DP, e.g., die Preise senken (‘to lower the prices’), four included an indefinite DP, e.g., eine Mahlzeit zubereiten (‘to prepare a meal’), and eight included a non-figurative incorporated noun, e.g., Zeitung lesen (‘to read (a) newspaper’).10 We wanted to balance the number of expressions that we assume to have / not have anaphor potential (indefinite and definite DPs vs. incorporated nouns). In our items with an indefinite DP, a specific reading of the DP was not salient in the provided contexts; e.g., in ‘[…] Do you think that Leon will send us greetings via mail? – No, he would definitely never send us a postcard’, it is not a salient interpretation that the speaker has a certain postcard in mind that Leon would never send. A complete item list is provided in Appendix A.

A different approach to item distribution was taken in Experiment 3. Two lists were created. Every questionnaire contained each item in four out of the eight structures. For each item, it was randomly determined which of the structures belonged to which list. This way, the same amount of data points was collected for each structure, resulting in a more balanced distribution than in Experiment 1. The distribution scheme did however not guarantee that every participant saw the same number of items per condition. The presentation order of the stimuli was randomized.

As motivated in Section 4.1, the most important structures for evaluating the hypotheses on compositionality and referentiality are those that involve a word order modification potentially linked to referentiality (scrambling, anaphor). However, to make a direct comparison to the previous studies possible and to see which findings carry over to the new data set, as well as for exploratory purposes, we included all syntactic structures from Experiments 1–2.11

A summary of the compositionality ratings is shown in Figure 8. The acceptability ratings are shown in Figure 9. If a participant indicated that they were not familiar with an idiom in the pre-test, their ratings for sentences containing this idiom were not included in the analysis of the acceptability ratings.

The factor idiomaticity (idioms, non-idioms) was sum-coded and the factor structure was treatment-coded with canonical as the baseline. For the additional factor DP type (definite, indefinite, incorporated), forward difference coding was used, allowing us to compare definite DPs to indefinite ones and indefinite DPs to incorporated nouns.

We ran three analyses. First, we tested the effects of structure, idiomaticity, and DP type in the data subset of definite and indefinite DPs in order to see whether definiteness had a similar effect on idioms and non-idioms; incorporated nouns were excluded here, because they are a special case of non-idioms. Second, we tested the effect of compositionality within the idiom subset. Third, we tested whether there was a difference between the indefinite DPs and incorporated nouns within the non-idioms subset.

In the first analysis, we fit a model to the subset of the data with definite/indefinite DPs (excluding the items incorporated nouns), with structure, idiomaticity, and DP type as fixed factors. No simple effect of idiomaticity was found, i.e., no significant idiom/non-idiom contrast in the canonical baseline. No simple effect of DP type was found, either, i.e., no significant definite/indefinite contrast in the canonical baseline. There also was no significant interaction between idiomaticity and DP type in the canonical baseline. As for the interaction between idiomaticity and structure, a larger contrast between idiom and non-idiom than in the canonical baseline in the direction of higher ratings for non-idioms was found for anaphor (t = –3.75, p < 0.001), prefield (t = –2.42, p = 0.02), LD (t = –2.49, p = 0.017), nominalization (t = –5.62, p = 0.001), and which-question (t = –3.51, p = 0.001), while the contrast was not significantly different from the canonical baseline for passive. As for the interaction between definiteness and structure, a larger contrast between definite and indefinite DPs than in the canonical baseline towards higher ratings for definite was found for scrambling (t = 7.81, p < 0.001). A significant effect in the opposite direction (towards higher ratings for indefinite) was found for nominalization (t = –2.58, p = 0.02) and which-question (t = –5.20, p < 0.001), while the contrast was not significantly different from the canonical baseline for the other structures. The latter finding was qualified by a three-way interaction between structure, idiomaticity, and DP type: in which-questions, the contrast between definite and indefinite DPs was larger in non-idioms than in idioms (t = 2.92, p = 0.007). No other three-way interactions were significant.

In the second analysis, we fit an additional model to the data subset including only idioms in order to test the influence of the factor compositionality and its interaction with structure and DP type. The compositionality measure that we used as a dependent variable was the proportion of cases in which a positive response to both questions asked in the compositionality task was given (concerning an individual figurative meaning of the DP and of the verb). No simple effect of compositionality was found, i.e., no significant linear effect of compositionality on the ratings in the canonical baseline (rather, there was a trend towards a negative effect in the canonical baseline). No significant effect of DP type was found, either, i.e., no significant contrast between definite and indefinite in the canonical baseline, nor a significant interaction between compositionality and DP type in the canonical baseline. As for the interaction between compositionality and structure, a larger effect of compositionality than in the canonical baseline was found for prefield (t = 2.92, p = 0.007), scrambling (t = 4.24, p < 0.001), passive (t = 2.61, p = 0.017), nominalization (t = 3.03, p = 0.007), and which-question (t = 2.63, p = 0.017), while the effect was not significantly different from the canonical baseline for anaphor. The interaction between DP type and structure was not significant for any of the tested structures, but this was qualified by significant three-way interactions between structure, compositionality, and DP type for anaphor (t = 2.19, p = 0.04) and scrambling (t = 2.63, p = 0.02). In both cases, there was a stronger positive effect of compositionality with definite DPs than with indefinite DPs. The estimate of the effect of compositionality is illustrated in Figure 10.

In the third analysis, we fit a model to the data subset including only non-idioms in order to test whether incorporated nouns behaved differently from indefinite DPs. Structure and DP type (this time also including the third level, i.e., incorporated) were included as fixed effects. No simple effect of any of the DP contrasts was found in the canonical baseline. As for the interaction between structure and DP type, a larger contrast between definite and indefinite (towards higher ratings for definite DPs) than in the canonical baseline was found for scrambling (t = –6.38, p < 0.001). A significant interaction in the opposite direction (towards lower ratings for definite DPs) was found for which-question (t = 5.01, p < 0.001). As for the contrast between indefinite and incorporated, a larger contrast than in the canonical baseline (towards lower ratings for incorporated) was found for LD (t = –5.43, p < 0.001), passive (t = –2.31, p = 0.03), nominalization (t = –2.37, p = 0.03), and which-question (t = –3.94, p = 0.001).

All significant effects reported above according to the LMM were also significant in the CLM analysis. We summarize and discuss any additional significant effects found in the CLM but not in the LMM in Appendix B.2. It is particularly relevant for the discussion below that a significant three-way interaction between structure, compositionality, and DP type (towards a stronger positive effect of compositionality with definite DPs than with indefinite DPs) was found for anaphor and scrambling both in the LMM and in the CLM analysis. We will focus on scrambling and anaphor in the discussion in Section 5.2. In addition, this interaction was also significant for LD and nominalization in the CLM. We will remark on this in the discussion of potential directions for future research in Sections 5.2 and 5.4.

Experiment 3 extended the data set to a new set of idioms and non-idioms. The main finding of Experiment 1 was confirmed: again, we observe a systematic positive linear effect of our compositionality measure on most of the tested syntactic structures. This supports the view that this is a replicable generalization.

An additional finding of Experiment 3 was that compositionality interacts with definiteness. This interaction was particularly strong for scrambling and anaphor; in these conditions, compositionality had a much stronger positive effect for definite than indefinite DPs.

When comparing the results for idioms and non-idioms, we observed that definiteness had a similar effect on both sets of expressions: for those syntactic structures in which definite non-idiomatic DPs were more acceptable than indefinite ones (in particular scrambling), we see the same pattern for idioms. For those structures in which indefinite non-idioms were more acceptable than definite ones (which-question and nominalization), we see the same for idioms. This adds support to the view that idioms and non-idioms are subject to similar grammatical constraints. We will discuss the definite > indefinite effect in the scrambling condition in more detail in Section 5. As for the opposite effect in the which-question and nominalization (indefinite > definite), we can think of the following potential reasons for it. The which-questions were constructed a bit differently in this experiment in order for us to be able to use the same construction for all items (with definite/indefinite/incorporated nouns): instead of a question starting with Welche(r/s)… ‘Which…’, we used Was für ein… ‘What kind of…’. This was potentially perceived as pragmatically odd in the conditions with definite expressions, where a referent was already introduced in the context (e.g., ‘Sasha complains about his old sofa every day. Do you think he wants to get rid of it? – I don’t know, what kind of couch would he throw away?’). Similarly, for nominalization, we chose a specific way to construct it to make it work with all the DP types (using von ‘of’ rather than genitive inflection), and it is possible that this was perceived as odd with definite DPs. Even though the reasons for the definite/indefinite asymmetry in which-question and nominalization could have to do with the specific way we constructed these conditions, we still think it is informative that idioms and non-idioms behaved in a similar way.

While definiteness showed a clear effect on idiom and non-idioms, it cannot account for all the contrasts in the data. We still observed a gap in syntactic flexibility between definite idioms and non-idioms, and between indefinite idioms and non-idioms. In Section 5, we will discuss referentiality as a property that might provide a unified explanation for our observations, including the residual idiom/non-idiom gap.

An observation from Experiment 3 that is compatible with the idea that referentiality plays an important role concerns the items with incorporated nouns. Items with incorporated nouns are non-idiomatic (in the sense that they do not involve figurative meaning), but lack referentiality. On average, they showed significantly lower syntactic flexibility than the other types of non-idiomatic expressions that we tested.

On a methodological note, asking two separate questions in the compositionality judgment task revealed that participants more frequently failed to find a figurative meaning for the verb than for the DP. As discussed in Sections 3.1–3.2, this can either mean that the idiom was non-compositional, or that the availability of a literal interpretation of the verb interfered12 – this is an uncertainty that our method leaves open, and that could be addressed in future work by asking “Does the verb/DP provide its own meaning?” rather than “…its own figurative meaning”; this would also open up the possibility to have both idioms and non-idioms judged using the same task.