3.1 Delimiting feedback

In order to make transparent which kinds of signals we include in our study, in the following we propose a schema of conversational feedback that allows for an integration of different types of signals found in different languages, while at the same time making clear distinctions between signals that are oriented toward indicating the presence vs. the absence of trouble.

The first important distinction we draw is between feedback and other kinds of responses in interaction. Feedback events are fundamentally responsive in that they are by definition related to some preceding talk by another interactant. However, it is crucial to distinguish feedback from other types of responses: while feedback events can be and are regularly solicited by the current signer or speaker, it is crucial that they are not made conditionally relevant in the same way as, for instance, answers to questions. This means that, in principle, the recipient can decide about the placement of feedback signals and may even withhold relevant feedback (Schegloff 1982: 86). Although the lack of feedback can lead to conversational failure (Bavelas et al. 2000), which is also the case for conditionally relevant responses in that they are, if withheld, “officially absent” (Schegloff 1968: 1083), the consequences are different: Conditionally relevant responses—answers to questions, acceptance of invitations, etc.—are responses that are specifically relevant at the particular point in time, with restrictions on possible responses set by the initial utterance. Feedback, in contrast, may be relevant at any point in a conversation, due to the fact that communicative trouble may arise at any time and, consequently, make the initiation of repair—or the passing of that opportunity—necessary (Schegloff 1982). Thus, the conditions on sequential placement are very different for conditionally relevant responses and feedback. This is illustrated by Example (2) showing a question–answer pair, and Example (3) showing the use of a feedback signal. In (2), interactant A poses a polar question in the first line, which interactant B answers in the second line. The question restricts B’s possibilities for responding, as an answer of the type yes/no is made conditionally relevant by the question. B responds to the question by means of the response particle ja ‘yes’.

In (3), in contrast, A offers a piece of information to B in the first line. This action does not make any particular response relevant, so B can choose to provide feedback or withhold it. The possibilities for responding are thus not restricted in the same way as by the question in (2). This does not mean that feedback may not be preferred over silence in this context, it just means that it is not conditionally relevant in the same way as an answer to a question. Speaker B can also choose the type of feedback she provides. In this case, she chooses a verbal feedback token in the form of the response particle ja ‘yes’. Here we can observe the multifunctionality of response particles: ja ‘yes’ is employed to formulate an affirmative answer to a polar question in (2), and as a continuer in (3).

Another important distinction to draw is that between different types of feedback, based on whether it deals with some conversational trouble (repair) or indicates a lack thereof (non-repair feedback) (see Figure 1).

Figure 1: Feedback strategies.

Figure 1 illustrates various feedback strategies. Although repair mechanisms (Dingemanse & Enfield 2015; Dingemanse et al. 2015) are included in the figure for the sake of completeness, they are excluded from further discussion, as they are not the focus of this study. Instead, we investigate conversational moves that do not initiate or constitute repair, but rather imply that repair is unnecessary. One of the most well-known and widely studied types of feedback is what is often referred to as a continuer. Continuers convey at least the basic interactional function of passing on the opportunity for initiating repair (Schegloff 1982)—see Example (3) above.⁶ Moreover, feedback signals called ‘newsmarks’, in addition, explicitly treat the information given by the preceding interactant as new and mark it as ‘remarkable’ (Marmorstein & Szczepek Reed 2023). In this category, we include non-repetitional requests for reconfirmation (Gipper et al. 2023) such as German echt? ‘really?’, see Example (4), as well as change-of-state tokens (Heritage 1984) such as ah, see Example (5).

Lastly, there are feedback signals that overtly indicate some kind of evaluation of the preceding information, ‘assessments’ (Uhmann 1996), as in Example (6).

We include these three types of feedback events—continuers, newsmarks, and assessments—in our investigation, regardless of their sequential position (second, i.e., following a volunteered initial utterance, or third, i.e., following a response made conditionally relevant) or their turn-taking properties (passive recipiency vs. incipient speakership, see Sbranna et al. 2022).

For this study, we did not include repetitions (see, e.g., the first part in Example (6)), as at least for German it has been shown that they tend to fulfill relatively marked actions when used in requests for reconfirmation (Gipper et al. 2023). Given that it is not clear whether this is also true for the other three languages, we chose to exclude repetitions and leave their investigation for future research, as they may not be fully comparable across the languages in our sample.

3.2 A modality-agnostic and holistic approach to feedback

In this paper, we take a modality-agnostic (Hodge et al. 2023) approach to the comparison of sign and spoken languages, an approach that looks at all components of a feedback event without privileging any of the articulators. Most of the signals functioning as feedback in our study, produced by the various articulators—head, eyebrows, eyes, nose, mouth gestures, cheeks, manual gestures, and shoulders—are comparable across sign and spoken languages. We thus observe that their use in feedback in signed languages is not qualitatively different from that in spoken languages. While we acknowledge that the meanings in feedback need not be the same for all articulators in the four languages, we start out with a descriptive and exploratory approach allowing for the possibility that signals are used in similar ways across languages. A detailed analysis of the exact meanings in feedback and other interactional functions will be an intriguing topic for future research. Our data show, for example, that both eyebrow raises and nose wrinkles are used in all four languages. For eyebrow raises, we can say that they are used in very similar ways in a newsmarking function, albeit with different frequencies. With regard to nose wrinkling, DGS shows markedly more nose wrinkles than the other languages (8% vs. 0.5% or less) (see Table 8). A nose wrinkle is known to convey the meaning ‘that’s right’ in DGS interaction (Herrmann 2020), but its use in spoken German discourse has not been addressed in the literature. A fuller analysis of this difference is left for future research. Moreover, the plurifunctionality of non-manual gestures is a well-established phenomenon (Andries et al. 2023; Oomen & Roelofsen 2023), and we acknowledge that many of the signals examined in this study may serve multiple functions simultaneously.

At the same time, we recognize that some articulators may be argued to differ across sign and spoken languages in the production of feedback signals. In sign languages, feedback may be conveyed through manual signs (see DGS Example (7) or RSL Example (6)) as well as mouthings (see DGS Example (7) and RSL Example (8)), mouth movements resembling spoken or written forms of the surrounding language (Bauer & Kyuseva 2022). In contrast, spoken languages express feedback through spoken words (see Example (3)) or vocalizations (e.g., mhm) (Dingemanse et al. 2022).

Classifying these signals on the basis of the articulators involved (hand, mouth, and mouth, respectively), would obscure the fact that these differences are based on modality-specific constraints for the two types of languages. Therefore, rather than classifying these three types of signals on the basis of the articulators with which they are produced, we classify them as one single category which we call talk⁷ for all four languages, signed and spoken. This allows us to compare sign and spoken languages with respect to the extent to which they employ talk elements regardless of the modality.

In addition to this modality-agnostic approach, in the following we propose a novel holistic construal of conversational feedback, where different articulators are employed to produce signals that combine into what we call a feedback event—see Figure 2. The figure shows a stretch of conversation between two interactants. The dark green longer bars represent turns, while the shorter bars indicate feedback events. Different colors represent different types of feedback, such as continuers or assessments, with varying durations. The zoom-in window illustrates how various signals may be combined within a single feedback event—some lasting the full duration of the event, others occurring only briefly.

Figure 2: Feedback signals vs. feedback events.

As can be seen in the ELAN screenshot in Figure 3, the person produces distinct signals such as a manual sign ja ‘yes’ in her lap, a mouthing ah, a head nod (hnn), squinted eyes (esc) and a nose wrinkle (nw) with different articulators, such as the head, the eyes, the mouth and the nose. These collectively form a feedback event. While some feedback events may consist of a single signal, this study shows that they often comprise multiple simultaneous signals (see Figure 7). So, instead of analyzing one signal, such as head nods, we claim that it is essential to take all (potential) articulators into account to get a broader understanding of the composition of feedback events. Crucially, this perspective does not entail that all signals that constitute a feedback event necessarily convey one single meaning. Rather, our approach focuses on temporal aspects of co-occurrence.

Figure 3: A screenshot from ELAN showing an example of a multilayered feedback event in DGS (source: DGS corpus, Hanke et al. 2020, file koe_03_sachgebiete).

This redefinition allows us to investigate feedback from two perspectives, looking at the whole, potentially multi-layered feedback event, but also at its components. Moreover, it allows for a modality-agnostic approach to feedback where the employment of different articulators is compared across sign and spoken languages.

3.3 Research questions

In this study, we aim to contribute to our knowledge of the similarities and differences in the composition of feedback events between sign and spoken languages. For this purpose, we compare conversational feedback in four languages, two signed and two spoken, matched according to cultural background: German Sign Language (DGS), Russian Sign Language (RSL), spoken German (GER), and spoken Russian (RUS). We annotated feedback events in corpora of casual conversations according to our definition in Sections 3.1 and 3.2, developing a coding scheme based on previous research and our own findings during annotation (see the Appendix).

In order to investigate possible similarities and differences between the four languages, we employ a descriptive and exploratory approach. Research comparing sign and spoken languages is still too scarce to formulate meaningful hypotheses. Moreover, the formulation of hypotheses would pose unhelpful restrictions on our research, where an exploratory approach opens the possibility for unanticipated findings.

We start our research with the following questions:

• I. What are the typical components of feedback events across languages?

  Previous research suggests that signed and spoken languages both rely to a large extent on head movements when expressing feedback events (Lutzenberger et al. 2024). Moreover, we know that non-manual signals such as head nods can combine with other signals in order to formulate multi-layered feedback events (Dittmann & Llewellyn 1968). Building on this research, we aim to further investigate the components involved in the formulation of feedback events, with special attention paid to similarities and differences between sign and spoken languages.

• II. What is the role of language, language modality, cultural background, and individual signer/speaker in the formulation of feedback events?

  As initial research suggests that the formulation of feedback events may in fact be quite similar in sign and spoken languages (Lutzenberger et al. 2024), we use our matched datasets to look at the question of whether feedback producers of one language are more similar to each other than to those of another language, or whether similarities and differences can rather be explained in terms of cultural background or language modality.

4.1 Data

For all four languages, we investigated video-recordings of free mundane dyadic conversations, using four (DGS, RSL, RUS, GER) corpora. For each language, we included three such conversations. In almost all dyads, both participants were familiar with each other prior to the recording.⁸ In the Russian dataset, however, two participants met for the first time on the day of the recording. In the same dataset, one interactant features in two recordings, so we investigate data from five interactants for Russian and from six interactants for the other three languages. For each language, we annotated between 43 and 58 minutes of conversation. In line with our exploratory approach, we chose to annotate similar amounts of time in order to be able to compare languages with respect to how frequent feedback is.

Our DGS data has been taken from the Public DGS Corpus (Hanke et al. 2020). The DGS Corpus is an annotated reference corpus of German Sign Language, 50 hours of which have been made publicly available. Its 330 participants use DGS as their primary language of daily life and come from various regions of Germany (Schulder & Hanke 2022). The DGS content analyzed and presented in this paper is drawn from release 3 of MY DGS – annotated (Hanke et al. 2020; Konrad et al. 2020), a research dataset that provides Public DGS Corpus recordings with full sign annotations and translations in German and English.

Our RSL data come from two corpora. One file was sourced from the RSL Online Corpus, developed by Svetlana Burkova and her team at Novosibirsk University (Burkova 2015). This corpus currently comprises over 230 recordings from 43 RSL signers (both men and women, aged between 18 and 63) including Deaf and Hard-of-Hearing individuals. To obtain authentic conversational data, we selected a 20-minute unprompted conversation between two Deaf signers recorded in Novosibirsk.

Due to the limited amount of interactional data in this corpus, we also used a second corpus of RSL conversations. The additional two casual dyadic conversations, lasting between 40 and 60 minutes, were sourced from Bauer & Poryadin (2023) and feature Deaf native RSL signers. This recently compiled corpus includes data from signers who previously lived in St. Petersburg, Čita, and Černišov before immigrating to Germany. Participants discussed a range of topics, including their lives in Russia before immigrating and their experiences as Deaf individuals in Europe and Russia.

Our German data constitute a subset of the Münster Korpus (Hoffmann & Himmelmann 2009), an unpublished corpus of video-recorded conversations among students from different areas in Germany who speak Standard colloquial German as their first language. The conversations were recorded in the year 2009 in the city of Münster, Germany.

Our three spoken Russian conversations are part of a Russian Multimodal Conversation Corpus (Bauer & Poryadin 2023). This corpus features dialogues among Russian immigrants lasting 40–60 minutes each. Participants, aged 20–30 years old, are native Russian speakers residing in Germany for no longer than 5 years.⁹

We aimed to use data that were maximally comparable across both interactional type (free, unprompted conversations between acquainted interlocutors) and recording setup (two participants). Table 1 summarizes the sources, the amount of time and tokens for the data employed in this study. Table 2 lists details for the different recordings.

4.2 Annotations

All data were annotated in ELAN (The Language Archive, MPI Nijmegen, The Netherlands; e.g. Crasborn & Sloetjes 2008). We started out with an annotation scheme developed by the first author inspired by the RSL Corpus annotations (Burkova 2015). For each feedback event, we created an annotation on a separate tier in ELAN marking the length of the whole event. A feedback event may consist of a single or of multiple signals (see section 3.2). The length of a feedback event is defined by the start of the first signal involved and the end of the last signal. To separate one feedback event from the next feedback event, we determined two options. Either, two subsequent feedback events were separated by 300ms¹⁰ of no movements or talk related to giving feedback. Or, two different feedback events could occur consecutively, distinguished by a noticeable change in movement, shape, or direction—for instance, a head nod transitioning into a head shake. We then added annotations on separate tiers for each articulator involved as described in Table 3. A key distinction from the earlier modality-agnostic approach by Hodge et al. (2023) lies in our grouping of signs, words, vocalizations (e.g., mhm), and mouthings under a single category: talk (see Section 3.2). In addition, we introduce an annotation tier labeled feedback type, which specifies the design of each feedback event based on the articulators involved. This includes whether the feedback consists of non-manual signals only, talk only, manual gestures only, or a combination thereof. Figure 3 above shows a still from ELAN exemplifying our annotations and Table 3 gives an overview of the annotation for various articulators.

During the annotation process, we continuously refined the annotation scheme, incorporating additional variables and annotations. For instance, we reannotated smiles and laughter according to the Smiling Intensity Scale (Gironzetti et al. 2016). In this way, our annotation scheme is grounded both in the previous literature as well as in our data. As our coding scheme developed, we also performed various iterations of coding for all data. In addition, we did various rounds of corrections. In this way, most feedback events have been annotated by at least two annotators. Our coding scheme where all the tier values are explained in detail can be found in the Appendix. In total, we identified around 1,900 feedback events in our data, comprising roughly 3,500 feedback signals.

We faced challenges in annotating certain multimodal features; in particular, (mutual) eye gaze was excluded from the analysis. Annotating eye gaze in video data using ELAN proved difficult, leading to inconsistent results and hindering the integration of gaze into the analysis. To address this gap, future research will employ eye-tracking technology, which we plan to use to improve the accuracy of gaze annotations. Apart from the challenges with gaze, we did not encounter noticeable difficulties in identifying non-manual signals in our datasets, even though some dyads in these corpora (e.g., in the Russian and RSL data) were filmed from a more lateral camera angle than those in the DGS Corpus. Importantly, this did not result in a higher number of unclear annotation values for these dyads compared to the other languages.

In order to assess the consistency of the annotations across coders, we calculated inter-annotator agreement on the articulator most frequently involved in feedback—the head. To this end, we re-annotated a randomly selected subset of the data comprising roughly 50% of all annotated head movements (843 out of 1603) drawn from all languages. These items had not been previously annotated by the respective coder. The onsets, offsets, and durations of the annotations were pre-annotated by the authors and Deaf and hearing assistants and therefore held constant; only the head movement type and the feedback event type were annotated. The resulting two annotation sets were then compared for inter-rater agreement. We calculated Fleiss’ generalized kappa in R version 4.5.1 (R Core Team 2025) (unweighted, 0.95 confidence level) with the function fleiss.kappa.raw() from the package irrCAC (Gwet 2019). The values show an inter-annotator agreement much higher than chance agreement. With 0.72, the kappa coefficient indicates substantial agreement between the two annotators (Landis & Koch 1977: 165). While these inter-coder reliability calculations concern only the content of the annotations, not their temporal location or duration, the fact that all annotation procedures involved subsequent corrections provides additional assurance that the annotations reflect a reasonable degree of coder consensus. A further limitation that needs to be acknowledged is that the agreement score pertains exclusively to the head tier and does not include less frequent signals—such as eyebrow or mouth movements—which are known to be more challenging for annotators to classify consistently (Esselink et al. 2024).¹¹

For the talk category, we classified the transcribed elements into subcategories based on their function. These categories are summarized in Table 4.

4.3 Analysis

As our approach is completely exploratory, our analyses do not involve any hypothesis testing. All analyses were performed in R (R Core Team 2023). To investigate the variability between feedback producers of the same language and compare it with cross-linguistic differences, we created a heatmap dendrogram in order to expose the relative association between a feedback producer and an articulator. Our inspiration for employing this method for comparing signers and speakers stems from Hodge et al. (2023). The heatmap dendrogram in Figure 6 and the heatmaps in Figures 8 and 9 were created with the function pheatmap() from the package pheatmap (Kolde 2019). In the heatmap dendrogram, hierarchical clustering is employed in order to identify clusters among the signers and speakers in our sample, as well as the annotated articulators. We did not scale the data in the heatmaps, as we aim to show the overall frequency of a certain articulator in the composition of feedback events of a certain interactant on the basis of the percentage of feedback events that contain that particular articulator. The ideal number of clusters for the data used in the heatmap dendrogram was calculated with the function NbClust()¹² from the package NbClust (Charrad et al. 2014). The other plots were created with ggplot2 (Wickham 2016) and, in part, the GGally package (Schloerke et al. 2024).

5.1 Head nods constitute the most frequent feedback signal in both sign and spoken languages

Across modalities, all four languages show only small percentages of feedback events without any non-manual elements. This is visualized in Figure 4, showing for each language the percentages of different combinations of types of feedback signals. A large majority of feedback events in all four languages is constituted by or contains at least one non-manual signal. As some of the numbers are too small to be visible in the figure, we also provide the absolute and relative frequencies of the different feedback event configurations in Table 5.

Figure 4: Feedback events across languages: Most feedback events consist of or comprise a non-manual element.

Figure 4 also shows a higher frequency of talk for the spoken languages in comparison to the sign languages, particularly in German. However, most feedback events that contain talk also contain a non-manual element. We can also observe that feedback events consisting of talk alone are virtually absent in the two sign languages.¹³ In the two spoken languages, they exist but are relatively rare, German showing the highest proportion with roughly 15%.

In our data, we furthermore observe that the head is the most pervasively used articulator in all four languages. In Figure 5, the relative frequencies of articulators employed in feedback events are visualized. Each panel shows a language, with lines representing individual signers/speakers. On the x-axis, the different articulators are placed. The y-axis captures the proportion of feedback events containing the articulator.

Figure 5: Coordinate plot of articulators involved in feedback: Head is the most frequent articulator in all languages. Each line represents an individual interactant.

Figure 5 shows that in all languages, the head is the most employed articulator. For the spoken languages, the second most employed articulator is talk, followed by mouth gestures. For the sign languages, in contrast, mouth gestures are more pervasively used than talk. Moreover, in the two sign languages we can observe a high variability in the employment of talk, eyebrow, and nose signals. The use of manual gestures and eyes is somewhat more pervasive in sign than in spoken languages, but generally low. Cheeks and shoulders are only very seldomly mobilized to formulate feedback events across the four languages. In sum, the relative rankings of articulators in Figure 5 suggest that we are dealing with quantitative rather than fully qualitative differences between sign and spoken languages.

Regarding the actual shape of the head movements, these are also relatively similar across languages. In Table 6, the frequencies of multiple and single nods as well as other head movements are summarized. It can be observed that across languages, multiple nods constitute the most frequent type of head movement. Taken together, multiple and single nods account for the largest part of head movements during feedback across languages.

In order to investigate the similarity of feedback event configurations across languages, Table 7 shows some of the most frequent signal combinations from our data and their frequencies in the four languages. Multiple head nods without any other additional signal constitute the most frequently employed feedback event configuration in all four languages. In the spoken languages, this is followed by multiple head nods combined with a yes-like element (e.g., equivalents of yes or mhm). This combination does not play such a large role in the two sign languages, which, in contrast, have as their second most frequent configuration a single nod without any further signal. A further combination that is employed in all languages with some frequency is multiple head nods combined with a closed mouth smile. Regarding the use of a yes-like talk element without further signals, only speakers of spoken German reach a proportion above 10%. In sum, a head nod is the most pervasive head movement during feedback across languages (Table 6). Talk, e.g. in the form of a yes-like talk element, plays a more important role in the spoken than in the sign languages.

While a more detailed investigation of the exact shapes of feedback events is left for future research, Table 8 offers a glimpse of the frequencies of some of the most frequent non-manual signals (excluding head movements). The table lists absolute frequencies and the percentage of feedback events that contain the signal (alone or combined with other signals) per language.

5.2 Signers and speakers employ a range of feedback styles

To tackle the variability between signers/speakers of the same language that already becomes apparent in Figure 5, we created a heatmap dendrogram (Figure 6). The numbers in the cells indicate the proportion of feedback events that contain a signal from the respective articulator for the particular interactant. This includes both signals produced on their own or in combination with other signals. The dark color stands for a high percentage of use of a certain articulator by a given interactant, while the light color indicates a low percentage. For instance, the speaker GER3 represented by the first line in the graph employs head movements in 62% of all feedback events she produces. 51% of her feedback events contain a mouth gesture, and 72% contain talk.

Moreover, the heatmap dendrogram reveals how similar the different articulators and the interlocutors are to each other. This allows us to investigate whether feedback producers of the same language, the same modality, or the same cultural background will cluster together. Based on the calculation of the ideal number of clusters (see Section 4.3), the interactants in our sample form three basic clusters. These three clusters are separated from each other in the graph for better visibility.

Figure 6: Heatmap dendrogram: Interactants form three clusters representing three different feedback styles.

The heatmap dendrogram in Figure 6 echoes our findings from Figure 5 above: The head is the most pervasively employed articulator, followed by mouth gesture and talk. These articulators also play a role in distinguishing the three largest clusters of interactants in the data: There is one group of interactants (containing all speakers of German and two speakers of Russian) that mostly employ these three articulators. These speakers show a higher proportion of feedback events containing talk, and many of them somewhat lower values for head. The second group, containing five signers of RSL, two signers of DGS, and three speakers of Russian, is defined by particularly high percentages of head movements, and comparatively low percentages for all other articulators. The third group, featuring four signers of DGS, one signer of RSL, and one speaker of Russian is characterized by the employment of a more variable and broader range of non-manual articulators, showing higher values for mouth gesture, eyebrows, and eyes. We propose that the clusters in Figure 6 manifest three different feedback styles: A style that relies more on talk and less on head movements; a style that relies mostly on head movements; and a style that employs a broader range of non-manual articulators (for discussion see Section 6.2.).

Figure 6 shows a clustering according to language, which is however only partial and fuzzy. The speakers of German cluster together due to their higher frequency of the talk articulator, whereas most signers of DGS cluster together due to their higher rates of mouth gesture, eyebrows and eyes. The middle group, however, defined by its reliance on very high percentages of head movements, is composed of most RSL signers, but also contains speakers of Russian and signers of DGS. This shows that, while on average speakers of the spoken languages tend to rely on talk more than the sign languages, and some signers (mostly those of DGS) tend to rely on a broader variety of non-manual articulators, these two possibilities constitute two extremes on a scale of reliance on talk and visual multi-articulator expression. In between, we find signers and speakers who rely on head movements to a large extent. Moreover, in Figure 6 we can observe that cultural background does not seem to play a major role in the clustering, as there is a complete separation between DGS signers and German speakers. In all three clusters, we find interactants from both cultural backgrounds.

These observations are also reflected in the number of articulators employed in feedback events. In Figure 7,¹⁴ we can observe that signers of DGS show higher percentages of feedback events with three or four articulators involved (and consequently less with one articulator only). This fits very well with the observation that most DGS signers form part of the group employing the multi-articulator style in Figure 6. German speakers, in contrast, show a relatively high percentage of feedback events with two articulators, which fits the high rates of talk while still maintaining head as the most important articulator for those speakers in Figure 6. Moreover, we examined all feedback events in our data sample to determine how many consisted of a single signal versus multiple signals. Overall, multiple-signal events (n = 1,061) occur more frequently than single-signal events (n = 865), which supports our holistic approach to feedback.¹⁵

Figure 7: Proportions of feedback events composed of one to six signals across the four languages. Single- vs. multiple-signal events counts are as follows: DGS (202 / 383), GER (230 / 295), RSL (227 / 170), RUS (206 / 213). In total, 865 events contain a single signal and 1,061 contain multiple signals.

Furthermore, our data give a first hint at intra-speaker variability. The Russian speaker (RUS-1) who features in two of our recordings—one with a previously unknown person (where RUS1 is indicated as RUS1-0) and one with a person she knows (RUS1-1)—is assigned to two different groups, which means that she employs different feedback styles in the two conversations. In the conversation with the person she knows, she is in the group employing a broader range of non-manual signals, basically due to the higher percentage of mouth gestures. In the conversation with the stranger, she is in the group mostly relying on head movements, showing a much larger percentage of head movements (0.93 vs. 0.59). This intra-individual difference cannot be explained by interpersonal alignment (Rasenberg et al. 2020), as the two Russian speakers with whom she interacts are both in the group relying more on talk (RUS2 and RUS4). This finding suggests that it will be promising to put a focus on intra-individual variation in the future, paying attention to feedback styles in different situations.

Lastly, different types of feedback signals are clearly associated with different feedback functions, which is very likely to in turn contribute to the emergence of the feedback styles we observe.¹⁶ If an interlocutor follows a narration by the other interactant providing mostly continuers as feedback, their feedback style will differ from a situation where the same interactant is participating in a very involved way, responding with higher amounts of assessments that offer the interactant’s subjective evaluation, or of newsmarks that index the remarkability of information (Marmorstein & Szczepek Reed 2023). In the latter case, we can expect higher proportions of facial expressions such as smiles or laughter (assessments) or eyebrow raises (newsmarks). Feedback styles, then, are partly shaped by feedback functions that feature prominently in the particular interaction. Future research will therefore examine feedback functions and their correlations with both the feedback signals and the feedback styles proposed here. In addition, we aim to control for the content of each conversation in order to more fully disentangle the influence of feedback function from other factors influencing the distributional patterns observed, such as community-wide conventions and personal preferences.

6.1 The multimodal and multi-channel nature of feedback

In this paper, we compare the formulation of feedback events in four languages—two signed and two spoken—while controlling for cultural background. Our data replicate earlier findings, showing that feedback exhibits some variability across languages, individuals, and can also vary for a single individual according to the situation.

In addition, our data suggest that language modality does play a role in the relative ranking of the different articulators in the composition of feedback events. While talk (i.e., manual signs and mouthing) is available to and employed by signers to formulate feedback events, it is used less pervasively than talk (i.e., spoken words and vocalizations) in spoken languages. However, the head emerges as the most pervasively used articulator for formulating feedback events across all languages in our sample, with multiple head nods without any accompanying signals constituting the most frequent feedback configuration. Our findings are thus consistent with previous findings on conversational feedback and the proposal of a shared conversational infrastructure for feedback and social interaction in general (Lutzenberger et al. 2024).

This infrastructure is inherently multimodal and multi-channel, involving multiple articulators in both sign and spoken languages.

These findings warrant an explanation. What are the advantages of a multimodal and multi-channel system for communication? We would like to suggest that an inherently multimodal and multi-channelled infrastructure for feedback (and conversation in general) allows signers and speakers to solve at least three conversational problems. First, the use of head movements and/or other visual signals allows for providing feedback without intruding upon the interlocutor’s turn, in line with earlier proposals (Dingemanse et al. 2022; Börstell 2024). Small multiple head nods can be produced in overlap with an ongoing turn without interrupting the current signer or speaker (whereas a manual gesture might be interpreted as an attempt to take the floor) (see Bauer et al. (2024) for kinematic properties of feedback head nods). Second, the possibility of producing visual signals in overlap with the interlocutor’s turn also comes with the opportunity of early signaling which action a signer or speaker is intending to perform in the upcoming turn, which may help the interlocutor to identify that action more quickly (Holler 2025). Third, a multimodal infrastructure for feedback allows interactants to evade linearity and thus flexibly express different meanings, including potentially the expression of different interactional functions at the same time (e.g., in the form of a head nod functioning as a continuer combined with a signed or spoken lexical assessment). A single-channel infrastructure, in contrast, implies a linear delivery, allowing for meanings to be expressed only consecutively rather than in parallel. The multimodal and multi-channel infrastructure for conversation thus provides signers and speakers with a system that is much more flexible than a single-channel infrastructure could be.

In addition to offering solutions to these three conversational problems, we suggest that the multimodal infrastructure also raises the question of who is the beneficiary of feedback. What is usually emphasized is the function of feedback of offering interpretable information to the current signer/speaker. Under this account, the recipient of the feedback is the sole or main beneficiary. But the multimodal infrastructure questions this position, as head nods may not only indicate to the addressee that the producer of the nod is still attentive to their talk. Rather, we suggest that it may potentially afford processing benefits for the person who produces the nod. Furthermore, while it is clear that facial gestures in conversation are not mere emotional expressions but rather serve pragmatic and interactional functions (Bavelas & Chovil 2018), this does by no means imply that they may not in addition constitute expressions of emotion that allow the speaker to regulate their emotional state. This is particular relevant in feedback, as feedback is produced in reaction to the interlocutor’s turn. Here, emotional reactions may coincide with the pragmatic function to be conveyed, e.g., a state of surprise leading to a reaction in the form of raised eyebrows may coincide with the the pragmatic function of indicating that the information provided by the interlocutor is conceived as surprising. For manual gestures, it is well established that they are also produced when they cannot be seen by the interlocutor (Bavelas et al. 1992; Iverson & Goldin-Meadow 1998; Mol et al. 2011). This suggests that their production may potentially not only serve the addressee, but could also impact the cognitive processes of the producer (Goldin-Meadow & Beilock 2010). However, the potential role of non-manual gestures such as head nods in altering the producer’s cognitive processes has yet to be explored in future investigations (Mori et al. 2022). Our data strongly suggest that this will be a worthwhile path.

6.2 Modelling multimodal feedback styles

Based on our analysis using the heatmap dendrogram (Figure 6), we identified three distinct feedback styles in our data. These styles differ in the relative prominence of various articulators used during feedback events. These findings are compatible with a shared interactional infrastructure for feedback among sign and spoken languages (Lutzenberger et al. 2024), where, however, interactants can choose among different feedback styles. These styles, we argue, can be ordered on a scale based on the pervasiveness of the different signals, as shown in the heatmap in Figure 8. This heatmap offers a summary of the three feedback styles identified, where each tile in the heatmap shows, per channel, the mean of the proportions for all interactants that were classified to belong to that style in Figure 6. Dark tiles indicate a high mean proportion, light-color tiles a low mean proportion. We chose to build this figure with the mean proportion in order to provide some potentially generalizable observations on the three feedback styles.

Figure 8: Heatmap of feedback styles identified with mean articulator proportions.

The mean proportions in Figure 8 suggest that feedback styles may form a gradient pivoting around the head articulator. In the Head-dominant style in the middle, the head is the most prominent articulator. In the Talk-oriented style, talk becomes more frequent, whereas the proportion of head movements drops. In the Face-oriented style, finally, other non-manual articulators are used more often, while head also drops somewhat. This suggests that when articulators other than the head become more pervasive in the feedback of an interactant, head movements become less prominent. We propose that from this observation we can derive a model that provides testable predictions for future research on feedback styles.

In Figure 9, we visualize the proposed model and summarize its predictions regarding the distribution of articulators in feedback styles. The three styles in the middle represent the styles we discovered in this paper. The means are extrapolated from Figure 8. As the proportion of feedback events containing head movements must decrease when other articulators (such as talk or non-manual gestures) become more prominent, the model predicts the theoretical existence of two additional feedback styles: one that is Talk-dominant, and another that is Face-dominant. These two styles represent the hypothetical endpoints of the model’s continuum.

Figure 9: Heatmap visualizing the model with observed and predicted feedback styles.

Importantly, unlike the three empirically observed styles in our data, these endpoint styles are theoretical projections—they are not based on actual measured data but are included to illustrate the model’s full conceptual range. To our knowledge, such extreme styles have not yet been explicitly described in the literature, although some preliminary observations (e.g., on Yurakaré and Yélî Dnye¹⁷) hint at their potential existence.

The model also predicts that no viable feedback style should exhibit simultaneously high proportions of all three articulator types (i.e., talk, head, and other non-manuals). We speculate that such an all-dominant configuration would likely be inefficient or cognitively taxing for both the producer and the addressee to process, as activating and integrating signals across all available channels might overload the interactional system rather than enhance it. This suggests that efficient feedback systems may rely on channel selection and modulation, rather than maximal channel activation.

For testing these model predictions and the adequacy of the model for new data, it will be crucial to conduct further studies with pairs of sign and spoken languages matched for cultural background. In addition, it will be relevant to investigate languages in non-(Indo-)European contexts in order to arrive at a more diverse picture.

The conceptualization of the feedback space in terms of a multi-dimensional gradient as visualized in Figure 9 has another advantage. In addition to an analysis of inter-individual variation as in Figure 6, it allows for an investigation and theorization of intra-individual variation. It is quite probable that signers and speakers are capable to adjust their feedback style according to the type and topic of the conversation, their interlocutor, degree of familiarity and thus the extent of shared common ground and even their personal physical and mental state at the time of the conversation. As another testable hypothesis, we propose that signers or speakers will vary along these lines, and that their styles will correspond to those we found in this paper, summarized in Figure 9. Of course, it is well possible that other styles will be discovered once more languages—including non-(Indo-) European ones—are investigated and larger samples with more interactants form the basis of our investigations, hopefully made possible through automatized annotations.

6.3 Towards a multimodal and interactional theory of the Language Faculty

Our results furthermore have important implications for the conceptualization of conversational concepts, as well as for theories of the Language Faculty. Specifically, they demonstrate the dominant role of non-manual signals—such as head movements, facial gestures, and other visual signals—in the production of feedback. This challenges the prevailing emphasis on verbal feedback in accounts of interactional phenomena, which have largely neglected the visual dimension of communication.

Recent research, making use of advances in recording and experimental methodologies, has significantly deepened our understanding of co-present interaction, revealing the fundamentally multimodal nature of human communication (Gregori et al. 2023; Henlein et al. 2024). While linguists increasingly recognize the multimodal foundation of language and interaction (Perniss 2018; Holler & Levinson 2019; Özyürek 2021; Rasenberg et al. 2022; Hamilton & Holler 2023; Kendrick et al. 2023; Sandler 2024), fundamental theoretical concepts, such as conversational turns and interactional feedback mechanisms, are often theorized in unimodal terms, leaving aside their inherently multimodal character.

We propose a shift in this perspective: these concepts should be theorized as multimodal from the outset as also currently suggested by Holler (2025) for ‘social action’. For instance, if a hand gesture precedes speech, why should the conversational turn be defined as beginning with the onset of the first syllable?

We argue that the boundaries of conversational turns—both their beginnings and endings—cannot be defined solely by spoken or signed items (see also Kendrick et al. 2023). Manual and non-manual gestures should instead be regarded as intrinsic components of conversational turns, as we have demonstrated for feedback in this study. A single head nod or head tilt, alone or in combination with other non-manual signals (e.g., widened eyes, eyebrow movements) or with manual gestures (e.g., palm-up gestures), constitutes an integral element of a feedback event.

Our results furthermore reinforce the urgent need for models of language and the Language Faculty that engage with the inherently multimodal nature of human communication. The Multimodal Language Faculty (MLF) model, a cognitive framework recently developed by Cohn & Schilperoord (2024), aims to account for both unimodal and multimodal language use, as well as other forms of communication across various modalities. While this model shows considerable flexibility, particularly through its proposed Multimodal Parallel Architecture, it remains unclear how interaction and dialogue are formally represented, as these components are not explicitly addressed in the current formulation of MLF. Our data highlight the need to extend such models to incorporate fundamentally interactional phenomena such as feedback, where the focus is not primarily on truth-conditional meaning but rather on interactional function. In contrast, the Interactional Spine Hypothesis (Wiltschko 2021) offers a highly detailed theoretical account of interaction, providing new perspectives on several interactional mechanisms including responsive actions. However, this model does not currently provide a framework for incorporating meaningful visual signals as core interactional phenomena. Extending this model to incorporate visual signals will be a fruitful path in the future, as the model explicitly predicts multi-functionality of linguistic items and is thus very well-suited for the integration of multi-functional non-manual signals. Taken together, our results suggest that visual, non-manual signals are an integral component of human interaction. We are therefore in urgent need of theoretical models that integrate both the visual and interactional dimensions of human language, moving beyond speech-centered paradigms.