1.1 Background: Co-speech gestures

The kind of co-speech gestures that have been the focus of recent semantic/pragmatic work and that we will be focusing on in this paper are gestures made simultaneously with speech/sign that enrich utterance meaning by iconically depicting an aspect of the situation or event (Kendon 2004; Goldin-Meadow & Brentari 2017); see (1–6) for examples. In these and future examples, gestures will be written in all-capital letters, and we will indicate the spoken words that align with the gesture by placing them in square brackets.

All of the co-speech gestures in (1–6) convey some information which the literal speech stream does not. For example, the co-speech gesture PULL-BACK in (1) involves the speaker making as if to grip and pull back something flexible which is fixed at the base, such as a sapling. The iconic information communicated by the gesture but not the spoken words is that the object being pulled back is fixed at the base. Similarly, the SPIRAL-UP gesture in (2) alone conveys the fact that the staircase in question is a spiral staircase, and the gesture TRIANGLE in (3) entails that the geometric shape in question is a triangle. UP in (4) implies that the helping was done by lifting, and LARGE in (5) implies that the bottle was large. Finally, the SHOOT gesture in (6) indicates that the scoring was done by shooting a basketball.

Kendon (2004: 7) defines a gesture as a visible action “used as an utterance or as a part of an utterance” with the goal of communicating something. Gestures can be classified according to their communicative function, the kind of meaning they express, their co-occurrence with speech, and their degree of conventionality and arbitrariness, among other factors (McNeill 1992; Kendon 2004; McNeill 2005; Abner et al. 2015; among others). Other types of gestures we will not be addressing include emblems (culture-specific gestures with fixed, arbitrary meanings that can be used with or without accompanying speech, such as the “thumbs-up” gesture), pantomimes (sequences of complete gestures that iconically depict scenes and/or events and never co-occur with speech), beats (short and simple non-iconic movements that pattern closely with an utterance’s prosodic peaks), and deictics (pointing gestures commonly used with demonstratives and locative adverbs). Iconic co-speech gestures can be distinguished from these other types of gestures by their co-occurrence with speech, their lack of arbitrariness, and their conveyance of meaning through iconicity. However, like other gestures, they are naturally occurring linguistic phenomena that are part of the linguistic system (Kendon 1980; McNeill 1985; 1992; Goldin-Meadow & Brentari 2017); they are spontaneously produced by speakers, and are not formally taught to learners.

The modest amount of previous formal semantic/pragmatic work on co-speech gestures has generally focused on inferences related to their not-at-issue-ness (Ebert & Ebert 2016; Tieu et al. 2017; Esipova 2018; Schlenker 2018), as illustrated by (7) where denial of the utterance cannot target the content of the gesture as in (b); instead (7a) can be continued by (b′) or (b″).

Of not-at-issue types of meaning, Schlenker (2018) proposes that co-speech gestures are in fact a type of presupposition, specifically an assertion-dependent conditional presupposition he calls a cosupposition. Based on informal inference judgments, he demonstrates that these cosuppositions exhibit standard presuppositional projection behavior in embedded contexts, except that the inference that projects is conditional. (In the examples below, reported inferences are indicated with the symbol ⇒.)

In contrast, Ebert & Ebert (2014; 2016) suggest that the way in which co-speech gestures contribute not-at-issue meaning is most analogous to supplements, such as expressives or non-restrictive relative clauses (Potts 2005). It is a subtle distinction: consider the case of non-restrictive relative clauses outlined in Chierchia & McConnell-Ginet (2000), which exhibit projection out of several operators in (9).

Sentence (9a) has the implication that Jill lost something on the flight from Ithaca to New York; sentences (b–d) show that the inference survives under negation, in a question, and in a conditional. However, unlike classic presuppositions, in none of (a–d) is it assumed by the speaker that the hearer already knows that Jill lost something, so these inferences have typically not been classified as presuppositions since they impose no restrictions on the background content.

An example of a supplemental analysis of the meaning of a co-speech gesture is given in (10), where Ebert & Ebert (2016) illustrate how the supplemental analysis predicts that the material conveyed by the gesture goes through as an inference in positive contexts (10a) but not in negative contexts (10b).

Although both the cosuppositional analysis and the supplement analysis of co-speech gestures take them to be not-at-issue, they make different predictions in several respects, one of which is projection behavior under negation, shown in (10) above. We will focus on one previously unexplored contrast in this paper: the restrictions that the gestures place on the discourse, either in previous context or in the same sentence, specifically whether the gesture is trivial, duplicating content provided elsewhere. Supplements are awkward when trivial: compare (11a), where the supplement is trivial, to (11b), a similar structure but where the supplement contains nontrivial information.

Taking a supplement analysis of co-speech gesture off the shelf, we would expect degraded acceptability for co-speech gestures that are trivial, by analogy to speech supplements. The cosuppositional analysis makes a less overt prediction about triviality, but by analogy to presupposition we might in fact expect the reverse direction of acceptability: presuppositions are usually expected to be given/trivial, and so the cosuppositional analysis would be consistent with the finding that gestures are more acceptable when trivial.

Past experimental work on the cosupposition/supplement distinction comes from Tieu et al. (2017), who report evidence based on different patterns of projection in favor of the cosuppositional analysis, albeit one with some extra assumptions that need to be made. We note, however, that their work focuses on inferential judgments involving gesture, while we take as our starting point the quite different task of acceptability judgments of gestures in context. Our motivation for focusing on acceptability is that we begin with the impression that, to the extent that we have acceptability judgments about co-speech gestures, they vary greatly in ways that we cannot predict using current theories of co-speech gesture. This includes examples in existing literature, some of which are entirely natural (e.g., (2)) and others which we found to be less so (e.g., (8d)), and which cannot in our mind yet be entirely explained by processing constraints like complexity, frequency, familiarity, etc. We are therefore interested in asking what kinds of factors contribute to the acceptability of co-speech gestures, so that gesture researchers creating linguistic examples can make more informed choices, including for important patterns like inferential judgments.

To summarize, we are interested in focusing on acceptability given the not-at-issue tendency of co-speech gestures, since dependence on previous context (in particular, being trivial) is unexpected for supplements but a hallmark of one type of presuppositional content (so-called “hard triggers”). With an eye toward fieldwork (or, in our case, work on an understudied phenomenon), Tonhauser et al. (2013) propose a typology for projective content. One of the properties that they note forms a class of projective content is the strong contextual felicity (SCF) condition, which will be the focus of this paper. By directly applying their suggested methodology for investigating expressions that seem to show projective behavior, we hope to better understand how to classify co-speech gestures in English in comparison to other projective spoken language phenomena, and to gain some more empirical evidence to bear on the theoretical analysis of co-speech gesture.

1.2 Background: Testing strong contextual felicity

The goal of our study is to essentially test how easily the content of not-at-issue meaning can be accommodated by an interlocutor. In other words, we are interested in asking whether co-speech gestures are more or less acceptable when they duplicate content or are entailed by content in the preceding discourse, or when they contribute new content. That is, are gestures better when they are informative, either with respect to the past context or the utterance that contains them? We will get at the former by testing whether their content must be entailed by the preceding discourse. Tonhauser et al. (2013) refer to this sensitivity to discourse context as strong contextual felicity (SCF); more specifically, an item is said to be [+SCF] if and only if it is only acceptable in a discourse context that appropriately entails it, where context entailment is defined in (12):

Then the property of strong contextual felicity is defined as follows:

Informally, if some expression can be uttered in a discourse context that neither supports nor denies an inference m, then that expression does not need to be supported by the context, so it is [–SCF] with respect to m. Formally, the following diagnostic can be used to decide on the strong contextual felicity value of an expression:

Part (ii) of the diagnostic captures the intuition that if an expression is not felicitous in a neutral context, but then is felicitous in a nearly identical context in which the only difference is that now m is entailed by the context, then we can safely conclude that the expression is [+SCF]. Using this diagnostic, we devised an experiment using Amazon Mechanical Turk to test if the semantic content of co-speech gestures is [+SCF] or [–SCF] by constructing various scenarios that permit co-speech gestures and manipulating the context to be either m-neutral or m-positive, where m is the proposition conveying the semantic content of the co-speech gesture.

As an example, consider the English additive particle ‘too’, which is standardly analyzed as having an existence presupposition of a salient parallel alternative proposition.

Let m be the proposition that somebody besides the bus driver is eating empanadas. The context in (15a) is m-neutral, because it doesn’t specify that anyone else is eating empanadas (Malena is eating a hamburger), or that no one else is eating empanadas. In this context, the sentence with too is infelicitous. The minimally different context in (15b), by contrast, is m-positive, since Malena is now said to be eating empanadas, not a hamburger. In this context, the sentence with too is felicitous. Since the same utterance with too is infelicitous in an m-neutral context but felicitous in a minimally different m-positive context, we conclude by the diagnostic in (14) that too introduces a strong contextual felicity constraint, i.e., too is [+SCF] with respect to the implication m.

An example of a projection trigger that does not exhibit a strong contextual felicity constraint is the change-of-state construction stop X.

Let m be the proposition that Laura used to eat gluten. The context in (16) is neutral with respect to m since Laura’s parents are not asserted or implied to know about Laura’s gluten consumption or lack thereof. Because the utterance is felicitous in the m-neutral context, we conclude that stop does not impose a strong contextual felicity constraint, i.e., is [–SCF] with respect to the implication m.

Recall that we are interested in whether iconic co-speech gestures are better when they are informative, meaning either with respect to the past context or the utterance that contains them. We will address the second of these issues by adding an additional dimension to the SCF diagnostic, which is whether the semantic content of the gesture is duplicated in the same proposition. This is especially expected to bear on the analysis of co-speech gestures as supplements, which predicts that triviality/duplication should result in lower acceptability of co-speech gestures. Our hypotheses are as follows:

• Hypothesis regarding strong contextual felicity: A large category of expressions carrying not-at-issue content require that content to be entailed by the preceding context. We ask whether co-speech gestures have the same requirement.

• Hypothesis regarding matching speech cue: One hypothesized analysis of gestures is as supplements (Ebert & Ebert 2014), which are infelicitous when trivial. We ask how matching content in the speech stream affects felicity of co-speech gestures, with the expectation that it should decrease grammaticality under the supplement account.

We describe the methods and procedures of our experiment, including the implementation of both of these factors, in the next section.

2.2 Results

Out of 198 participants, 5 participants’ responses were excluded due to those participants failing at least one of the three attention checks. The results discussed below are for responses from the remaining 193 participants.

Recall that the experimental design included eight trial types resulting from all possible combinations of the experimental conditions GESTURECUE, SPEECHCUE, and CONTEXT. Table 2 reports the mean acceptance rates, standard deviations, and standard errors across the eight trial types, and Figure 3 shows mean acceptance rates for each trial type with standard error bars. Descriptively, we see an interesting pattern in which the presence of a co-speech gesture decreases acceptability (as indicated by the values of M for trial types 5–8), although the presence of a matching speech cue with the gesture somewhat mitigates this effect (as indicated by the values of M for trial types 7 and 8).

Figure 3

Experiment 1 mean acceptance rates, by experimental conditions GESTURECUE, SPEECHCUE, and CONTEXT.

With respect to the experimental conditions GESTURECUE, SPEECHCUE, and CONTEXT, trends were as follows. On average, trials with gestures (trial types 5–8) were rejected more often than trials without gestures (trial types 1–4) (with gesture: M = 0.65; without gesture: M = 0.77). Trials shown with a neutral context (1, 3, 5, and 7) were rejected at approximately the same rate as trials shown with a positive context (2, 4, 6, and 8) (neutral: M = 0.72; positive: M = 0.70). Finally, trials with a speech cue (3, 4, 7, and 8) were rejected on average at approximately the same rate as trials without a speech cue (1, 2, 5, and 6) (with speech cue: M = 0.73; without speech cue: M = 0.70).

Table 3 gives a breakdown of ratings by scenario. Scenario 11 trials were excluded from analysis because of an accidental name mismatch between the written contexts and the speech in the video. Means for the remaining 15 scenarios ranged from 0.57 (Scenario 1) to 0.82 (Scenario 5), with SDs ranging from 0.38 to 0.50.

Analyses of subjects’ judgment responses were conducted using the R programming language (version 3.2.3) (R Core Team 2016) to build generalized linear mixed effects models (Baayen et al. 2008), using the function glmer from the lme4 package (Bates et al. 2015). In the model with the most data variation coverage (determined by ANOVA testing), the three independent factors were GESTURECUE (no/yes), SPEECHCUE (no/yes), and CONTEXT (neutral/positive); the binary naturalness judgment (coded as 1 = “completely natural” and 0 = “not completely natural”) was the dependent variable, and scenario number and participant ID were coded as random effects.

Results of the model indicate that there was a significant main effect of GESTURECUE (β = –1.148, z = –4.908, p < 0.001), but also a significant interaction of the factors GESTURECUE and SPEECHCUE (β = 0.665, z = 2.507, p < 0.05). Neither SPEECHCUE nor CONTEXT were significant main effects (p > 0.1); these findings agree with other models of the data that we constructed in which the only fixed factor was SPEECHCUE or CONTEXT (p > 0.1 in each case). There were no other significant interactions between factors.

2.3 Discussion

The main experimental question of this study was how contextual support (within a discourse or the same utterance) influences the felicity of co-speech gestures. The first factor we tested was whether co-speech gestures need to be entailed by their preceding discourse context to be used felicitously. In the terminology of Tonhauser et al. (2013), we wanted to find out if co-speech gestures are [+strong contextual felicity (SCF)] or [–SCF]. Our experimental results indicate that manipulating the discourse context from being m-neutral to m-positive (where m is the proposition representing the semantic content of the co-speech gesture) had no significant effect on participants’ acceptance rates of stimuli items. Since participants were equally likely to judge an utterance as completely natural with or without the positive entailment of the content of the co-speech gesture, and assuming that participants were judging the video with the context paragraph in mind, we can tentatively conclude that co-speech gestures are [–SCF]. In other words, co-speech gestures can either be easily accommodated or need no contextual entailment to be used felicitously. This is in clear contrast to other projective content like the English additive particle too, a so-called “hard” presupposition trigger (Abrusán 2016); as we discussed in example (15) in Section 1.2, too is [+SCF] with respect to its implication of the existence of a salient parallel alternative proposition in the discourse context. The differing behavior of too and co-speech gestures with respect to the content of the contexts they appear in supports the claims that the former is [+SCF] (Tonhauser et al. 2013) and the latter is [–SCF]. In sum, our experiment has shown that co-speech gestures cannot be classified as “hard” presupposition triggers (assuming participants did indeed read and include the context paragraphs in their linguistic judgments, which concern we will address with Experiment 2 in Section 3).

The significant main effect of GESTURECUE in the data indicates that the presence of a co-speech gesture in a trial in general had a negative effect on the rating of the video. Intriguingly, this negative effect was mitigated when there was a speech cue present along with the gesture in the trial. One way to interpret this is that gestures are used most felicitously when they are semantically duplicating information already present in the speech/sign stream. This redundancy preference clearly pragmatically differentiates co-speech gestures from well-studied non-gestural supplemental material like appositives; appositives are not felicitous when they convey the same information as the main assertion (Potts 2005), as can be seen in (26):

The content of the appositive NP in (26), shown in brackets, cannot be reiterated as part of the foreground asserted content. However, as we see from the experimental results, partial and even complete redundancy is completely acceptable with co-speech gestures; (27) is an example of a trial from the experiment with both a co-speech gesture and a speech cue present (Scenario 3, Appendix A).

Here the gesture BIG is arguably contributing very similar content to the speech cue big.

The significant interaction between GESTURECUE and SPEECHCUE indicates that speech cues had a different effect on judgments depending on whether or not there was a gesture; namely, trials with gestures had a higher average acceptance rate with a speech cue than without, while trials without gestures had a lower average acceptance rate with speech cues than without. Presumably this has something to do with pragmatic calculations about the amount of information present in the utterance and any corresponding implications the hearer (our participants) might have drawn. Clearly, at this point our results raise more questions than answers, especially regarding whether the same pattern would emerge with different kinds of co-speech gestures.

Since there was such wide variation in ratings across scenarios, we did several follow-up analyses (which were not planned before the experiment was run) of scenario types to see how properties that varied by scenario affected judgments. We focused on three questions: (i) whether the gesture semantically modified a noun or verb; (ii) whether the speech cue that duplicated the gesture’s content came temporally before, during, or after the co-speech gesture; and (iii) whether the belief state of the addressee, given the context paragraph, affected judgments. For the first question, we found that, overall, co-speech gestures that appeared in scenarios where they were modifying nouns were more acceptable than those modifying verbs (nouns: neutral context: M = 0.83, SD = 0.38; positive context: M = 0.77, SD = 0.42; verbs: neut: M = 0.634, SD = 0.48; pos: M = 0.628, SD = 0.48). However, the number of scenarios is too small to generalize beyond the sample (gestures modified nouns in two scenarios and verbs in thirteen scenarios); see Figure 4.

Figure 4

Experiment 1 mean acceptance rates for type of syntactic item modified by gesture (noun vs. verb), by context type.

Next, speech cues that occurred after their corresponding gesture were generally less acceptable than those that occurred with or before the gesture (after the gesture (N = 238): neutral context: M = 0.63, SD = 0.49; positive context: M = 0.65, SD = 0.48; before the gesture (N = 93): neut: M = 0.78, SD = 0.42; pos: M = 0.75, SD = 0.44; during the gesture (N = 386): neut: M = 0.75, SD = 0.43; pos: M = 0.70, SD = 0.46), but again, the sample size of each type of scenario is small (speech cues occurred after the gesture in five scenarios, before the gesture in two scenarios, and with the gesture in eight scenarios); see Figure 5.

Figure 5

Experiment 1 mean acceptance rates for speech cue timing relative to gesture timing, by context type.

The differences between placement of speech cues (before, during, and after the gesture) may initially appear to be intriguing, but critically there was no difference between these timing conditions for speech cues in trials where the speech cue was present versus those where it was absent. Using the glmer function in R, we constructed a generalized linear mixed effects model (fit by Maximum Likelihood) on the subset of the data where a gesture was present [GESTURECUE = yes], with subject ID and scenario number as random effects, the participant’s naturalness judgment (RESPONSE) as the dependent variable, and SPEECHCUE (yes/no) and SPEECHCUETIMING (before/during/after) as the independent variables. Neither the interaction between the presence of a speech cue and the speech cue coming after the gesture, nor the interaction between the presence of a speech cue and the speech cue coming before the gesture, was significant (p > 0.1 in both cases). This suggests that the difference in ratings between the differing speech cue positions is an artifact of the large variation in trial types and not in the end about the speech cues themselves.

Another factor one might suspect of influencing acceptability judgments by participants is the belief state of the addressee in the “dialogue” between Eliza and her interlocutor described in the context paragraphs. Does the addressee need to know/believe/be able to infer the content of the gesture proposition in order for the experimental participant to judge Eliza’s use of the gesture felicitous?² We conducted a follow-up analysis on the data, classifying scenarios (yes/no) according to whether the addressee of the dialogue knows or could infer the content of the proposition in question based on the information provided by the “positive” context. We constructed two separate generalized linear mixed effects models, one on the “yes” subset of the data (where the addressee knows/can infer the semantic content of the gesture based on the positive context), and the other on the remaining “no” subset of the data, with RESPONSE as the dependent variable, SPEECHCUE, GESTURECUE, and CONTEXT as the independent variables, and subject ID and scenario number coded as random effects. Neither model showed a significant effect of CONTEXT on RESPONSE (p > 0.1). From this we conclude that the belief state of the addressee is not a significant factor in acceptability judgments of utterances with co-speech gestures based on prior discourse contexts.

In this experiment, there were several methodological issues which may have had an unintended effect and require follow-up studies. For one, we told participants that they were going to be asked to rate English sentences as completely natural or not completely natural; however, we did not explain or define the concept of linguistic naturalness. This may in fact have resulted in participants performing different rating tasks from each other due to interpreting “naturalness” in different ways. Some respondents, for example, may have been rating the “acting” performance of the speaker in the videos, rather than rating the felicity of the utterance (speech + gesture) alone. One way to address this might be to provide a simple, non-technical explanation of linguistic naturalness in the survey instructions so as to better communicate to participants what we as researchers are asking them to do. However, at this stage we are still unsure what exactly this would look like, and how one could dissociate performance factors from gesture with a prompt.

A more worrying methodological issue is that from Experiment 1, it is impossible to tell if participants were actually taking the time to read the written context paragraphs and if the contexts factored into their judgments of the video utterances. In fact this is related to our previous concern, because participants might rate the delivery (rather than the linguistic naturalness of the utterance) precisely because they weren’t taking context into account. The attention checks built into the experiment could only detect if a participant was not watching or paying proper attention to the videos. Without knowing more about the participants’ behavior with respect to the written contexts, we cannot fully conclude whether or not co-speech gestures need to be entailed by the discourse context. We address this concern in a follow-up experiment, Experiment 2, in the next section.

3.2 Results

Out of 90 participants, 2 participants’ responses were excluded due to those participants failing at least one of the three attention checks. The results discussed below are for responses from the remaining 88 participants.

Table 5 reports the mean acceptance rates, standard deviations, and standard errors across trial types, and Figure 6 shows mean acceptance rates for each trial type with standard error bars.

Figure 6

Experiment 2 mean acceptance rates, by experimental conditions GESTURECUE and CONTEXT.

Trends across the experimental conditions GESTURECUE and CONTEXT were as follows. On average, trials with gestures (trial types 4–6) were rejected slightly more often than trials without gestures (trial types 1–3) (with gesture: M = 0.62; without gesture: M = 0.67). Trials shown with an infelicitous context (1 and 4) were rejected at a much higher rate than trials shown with either a neutral context (2 and 5) or a positive context (3 and 6) (infelicitous: M = 0.43; neutral: M = 0.74; positive: M = 0.77). Note that neutral and positive context trials were rejected at roughly the same rate.

Table 6 gives a breakdown of ratings by scenario. Means for scenarios across trials with infelicitous contexts ranged from 0.23 (Scenario 12) to 0.66 (Scenario 2), with SDs ranging from 0.43 to 0.51. Means for scenarios across trials with either neutral or positive contexts, by contrast, ranged from 0.67 (Scenario 16) to 0.93 (Scenario 5), with SDs ranging from 0.25 to 0.49.

Analyses of participants’ judgment responses were conducted using the R programming language to build generalized linear mixed effects models using the function glmer. In the model with the most data variation coverage (determined by ANOVA testing), the independent factors in the model were GESTURECUE (no/yes) and CONTEXT (neutral/positive/infelicitous); RESPONSE (1/0) was the dependent variable, and scenario number and participant ID were coded as random effects.

Results of the model indicate that there was a significant main effect of CONTEXT: With “neutral” as the reference value, [CONTEXT = infel] was highly significant (β = –1.796, z = –7.887, p < 0.001), but [CONTEXT = pos] was not significant (p > 0.1). In a minimally different model with “infelicitous” as the CONTEXT reference value instead, both [CONTEXT = neut] and [CONTEXT = pos] were highly significant with (β = 1.796, z = 7.887, p < 0.001) and (β = 2.034, z = 8.590, p < 0.001), respectively. In other words, there was no significant difference between the neutral and positive context conditions, but an infelicitous context significantly decreased the acceptance rate compared to either neutral or positive contexts.

Unlike the results from Experiment 1, GESTURECUE was not a significant main effect (p > 0.1). We discuss possible reasons for this difference in Section 3.3. There were no significant interactions between factors.

We also analyzed the subset of participant responses for which the context presented was either “neutral” or “positive”, thereby excluding the data from “infelicitous” CONTEXT trials. Again we used R and the the function glmer to build a generalized linear mixed effects model, with independent factors GESTURECUE and CONTEXT, dependent variable RESPONSE, and random effects of scenario number and participant ID. Results of the model indicate no significant main effect of either GESTURECUE or CONTEXT, nor a significant interaction between the two factors (p > 0.1). This supports the conclusion from the full model reported above that there was no significant difference in acceptance rates between trials with neutral contexts and those with positive contexts.

3.3 Discussion

Since the trials with infelicitous contexts were rated significantly lower than other trials, this follow-up study is encouraging, suggesting that participants were reading the context paragraphs and, just as importantly, taking them into account when rating the video utterances. Similarly to Experiment 1, in this follow-up study there was no significant difference in ratings between trials with neutral contexts and trials with positive contexts. Taken together, we can conclude more confidently that co-speech gestures are not sensitive to entailment/implication by the discourse context, and hence are [–SCF] in the Tonhauser et al. (2013) terminology.

The factor GESTURECUE was not a significant main effect in Experiment 2. We suspect that this is likely due to the overwhelming effect of context in Experiment 2 with the introduction of infelicitous trials, minimizing the variability available among felicitous trials. To investigate this further, we compared the means for the same trials across both experiments (the two trial types [SPEECHCUE = yes, GESTURECUE = no] and [SPEECHCUE = no, GESTURECUE = yes]), and found that the range of means for Experiment 1 is 0.60–0.77 (mean SD 0.46), compared to the more compressed range of means in Experiment 2 of 0.70–0.82 (mean SD 0.43). In other words, the same trials in Experiment 2 had higher and more compressed ratings for positive and neutral contexts than those trials did in Experiment 1, which we attribute to the addition of infelicitous contexts in Experiment 2. We therefore suspect that this compression of contexts of interest (felicitous trials) may have contributed to the lack of effect for GESTURECUE in Experiment 2 compared to Experiment 1.

In Section 3.1.4, we noted that many of the infelicitous contexts for this experiment were created by simply swapping the names of the addressee and the person being talked about in the neutral and positive contexts. One might wonder if participants were paying sufficient attention to these “name-swap”-type infelicitous contexts to detect the cause of infelicity, and if they were not, whether this artificially raised the ratings of those trials. To test this, we coded the scenarios for Experiment 2 according to whether their corresponding infelicitous contexts were made bad by swapping names (“name-swap”) or by some other mechanism (“other”), such as introducing a contradiction between the context paragraph and the video utterance (e.g., the infelicitous context of Scenario 5, Appendix C). We constructed a generalized linear mixed effects model on the data from just those trials where the participant saw an infelicitous context (condition [CONTEXT = infel]), with RESPONSE as the dependent variable, type of infelicity (name-swap/other) and GESTURECUE as the independent variables, and subject ID and scenario as random effects. There was no main effect of type of infelicity on acceptance rates, nor a significant interaction effect between type of infelicity and GESTURECUE (p > 0.1 in both cases). We interpret these results as indicating that participants did not behave differently when presented with a “name-swap” infelicitous context than they did with other types of infelicitous contexts, and so the cause of infelicity did not have an overall effect on felicity judgments.

As in Experiment 1, we wanted to check whether the belief state of the addressee in the “dialogue” between Eliza and her interlocutor described in the context paragraphs influenced participants’ judgments. We conducted another follow-up analysis on the Experiment 2 data, using the same classification of scenarios (yes/no) according to whether the addressee of the dialogue knows or could infer the content of the proposition in question based on the information provided by the “positive” context. We constructed two separate generalized linear mixed effect models, one on the “yes” subset of the data (where the addressee knows/can infer the semantic content of the gesture based on the positive context), and the other on the remaining “no” subset of the data, with RESPONSE as the dependent variable, GESTURECUE and CONTEXT as the independent variables, and subject ID and scenario number coded as random effects. Just as in the Experiment 1 analysis, neither model showed a significant effect of CONTEXT on RESPONSE (p > 0.1 in both cases). From this we once again conclude that the belief state of the addressee is not a significant factor in acceptability judgments of utterances with co-speech gestures based on prior discourse contexts.

Overall, Experiment 2 lends support to our experimental methodology, providing encouraging evidence that participants judged the overall effect of context plus video when providing their ratings, although the addition of the infelicitous contexts meant that more subtle differences between generally felicitous trials in some cases disappeared. Together, we suggest that Experiments 1 and 2 provide a more clear overall picture of the empirical landscape for co-speech gesture pragmatics.

4.1 Directions for future research

By directly controlling whether the content of co-speech gestures is duplicated in the preceding context and/or the same utterance, the two experiments we report in this paper show how the acceptability of co-speech gestures may be affected by linguistic context. We hope that this can be a first step for future studies, and, in some cases, provide a foundation for interpreting results found in the few already existing studies on the semantics/pragmatics of co-speech gesture.

Through the notion of triviality, we connect to the existing literature on the way that gestures contribute content, which has been the focus of previous discussion of co-speech gestures as either “supplemental” (Ebert & Ebert 2014; 2016) or as “cosuppositional” (Schlenker 2018). Under a supplemental analysis, it is quite surprising that speech cues aid the acceptability of gestures, given that supplements are typically less acceptable if they are trivial. On the cosuppositional side, we have shown that co-speech gestures are not “hard” presupposition triggers since they need not be entailed by preceding context; however, many presuppositions are known to be easily accommodated, which could account for their acceptable nature in our study. Under this view, it remains a question why a positive preceding context doesn’t also provide the same kind of content support (and our second experiment suggests that it is not due to participants’ ignoring written context). Altogether, given the improvement of gestures with matching speech cues, our data are much harder to reconcile with the supplemental theory.

One further remaining question raised by these studies has to do with the choice of gesture targets in the development of the stimuli, discussed above in the Methods section (2.1.3) of Experiment 1. Recall that we chose semantically underspecified predicates so that the corresponding co-speech gestures would non-trivially contribute to the truth conditions of the utterances. Some underspecified predicates seem to have a preferred “default” manner or adjectival property that is assumed in the absence of further modification (either by speech cue or gesture cue). Ebert & Ebert (2016) look at some examples of these default interpretations for NPs in German, referring to the phenomena as the “typicality of a gesture for an NP concept”. A particularly interesting example they describe involves the NP Fenster ‘window’ and the two different shape properties of either square or circular; they assume the square option to be the more typically expected shape for windows. For an example from our experimental stimuli, consider the VP writing an essay:

In this scenario we chose the gesture TYPE, indicating that the writing was done on a computer as opposed to by hand with pen and paper. In the current decade, it would be a very natural inference to make that writing an essay means writing (typing) on a computer; anyone visiting a college campus these days will see ample evidence of this preferred mode of writing essays. Returning to the experiment, in Scenario 5 trials such as (30) we chose the gesture option with the “default” or more expected interpretation of TYPE instead of a more uncommon co-speech gesture indicating writing with a pen and paper, call it WRITE-WITH-PEN. An important question is whether this choice of gesture has a significant effect on utterance ratings and whether it interacts in a significant way with the presence of a speech cue in an utterance. The supplement analysis of co-speech gesture, although generally not supported by our findings, would predict that less trivial, less expected gestures (given speech cues) would be improved, and this does match our intuitions about examples like (2) from Goldin-Meadow & Brentari (2017), repeated below as (31), which we find especially natural/felicitous:

Given that the prototypical staircase is not a spiral staircase, the gesture content is unexpected and hence informative. An experiment could be designed with gestures varying along the dimension of unexpectedness, given context and speech cue.

4.2 Summary

The primary goal of these experiments was to diagnose the behavior of co-speech gestures in contexts that do and do not entail/imply their semantic contents, thereby gaining a better understanding of when and how co-speech gestures can be felicitously used in conversation. We implemented this question formally using the strong contextual felicity diagnostic proposed in Tonhauser et al. (2013), and by varying whether similar information was contained in accompanying speech cues. The results of our experiments show that co-speech gestures do not need to be entailed/implied by their preceding discourse context; hence co-speech gestures are [–SCF] and cannot be considered “hard” (i.e., unaccommodatable) presupposition triggers. We also saw that speech cues, or speech expressions that (approximately) duplicate the semantic content of a co-speech gesture, had a significant interaction with the presence or absence of a gesture in the trials. We take this to be an indication that there are other restrictions on the felifelicitous use of co-speech gestures that we do not yet know about and that involve when and how the gestures can contribute “extra” semantic content to the utterance meaning. We speculate that this may vary depending on the type of content conveyed in the gestures, for example whether they represent size-and-shape or manner information, and whether they modify nouns or verbs, among other potentially relevant dimensions. We hope that the data on co-speech gesture felicity judgments gathered through these experiments will pave the way for future research on co-speech gestures that addresses these larger theoretical questions in interesting and fruitful ways.