1 Introduction
Communicative gestures (i.e., “co-speech gestures” or “gestures”) are spontaneous body movements such as hand movements, which naturally accompany both the spoken and signed discourse (Kita et al., Reference Kita, Van Gijn, Van Der Hulst, Wachsmuth and Fröhlich1998; Kita & Emmorey, Reference Kita and Emmorey2023; McNeill, Reference McNeill1992). Gestures constitute crucial non-verbal elements with linguistic meaning, enabling successful communication. Efforts have been recently made to better describe the gestural profiles of children with Autism Spectrum Disorder (ASD),Footnote 1 notably for their potential to constitute early markers for ASD diagnosis (Stewart et al., Reference Stewart, Vigil, Olszewski and Thornock2022; Zwaigenbaum et al., Reference Zwaigenbaum, Bauman, Stone, Yirmiya, Estes, Hansen, McPartland, Natowicz, Choueiri, Fein, Kasari, Pierce, Buie, Carter, Davis, Granpeesheh, Mailloux, Newschaffer, Robins and Wetherby2015) and language impairment (Choi & Rowe, Reference Choi and Rowe2024; Goldin-Meadow, Reference Goldin-Meadow2015). While most work has focused on production, studies investigating their gestural comprehension abilities remain scarce. This study proposes to address this gap by investigating the gestural comprehension abilities of autistic children aged 3–12 embedded in a narrative, in comparison to neurotypical peers of comparable age, non-verbal reasoning, and morphosyntactic skills.
Gestures are essential for effective and efficient communication, benefitting both the speaker and the listener. On the one hand, they support message conceptualization and linguistic planning, enabling the transformation of abstract thoughts into structured information that can be verbalized (e.g., Kita, Reference Kita2000; Kita & Özyürek, Reference Kita and Özyürek2003). On the other hand, they promote understanding and disambiguating speech (Dargue et al., Reference Dargue, Sweller and Jones2019) while facilitating information retention in short- and long-term memory (Cook et al., Reference Cook, Mitchell and Goldin-Meadow2008). At times, gestures reinforce the spoken message, providing emphasis and enhancing comprehension; in other instances, gestures convey meaning that is absent from speech, providing additional information that enriches the communication by visually reflecting the speaker’s underlying thoughts (Goldin-Meadow, Reference Goldin-Meadow1999, Reference Goldin-Meadow2003). These visual cues arguably also have a positive impact on thinking, learning, and problem solving (Goldin-Meadow & Alibali, Reference Goldin-Meadow and Alibali2013), by reducing the cognitive load in speakers and listeners (Goldin-Meadow, Reference Goldin-Meadow2004, Reference Goldin-Meadow2011, Reference Goldin-Meadow2014; Goldin-Meadow & Alibali, Reference Goldin-Meadow and Alibali2013). While gestures often occur with concomitant speech, situations where they replace verbal information are also frequent, for instance in noisy settings (Meissner & Philpott, Reference Meissner and Philpott1975), in case of word retrieval difficulties (Krauss et al., Reference Krauss, Chen, Gottesman and McNeill2001), or taboos (Brookes, Reference Brookes, Müller, Cienki, Fricke, Ladewig, McNeill and Bressem2014; Kendon, Reference Kendon1988).
Not only are gestures pivotal in language production and conceptualization, but their role in spoken language acquisition is also well established: Gesture production and comprehension noticeably precede and predict speech outcomes in vocabulary and syntax, both in the neurotypical population and in children with ASD (see Choi & Rowe, Reference Choi and Rowe2024; Kilili-Lesta et al., Reference Kilili-Lesta, Voniati and Giannakou2022 for reviews). Indeed, the frequency and diversity of gestures used by infants reliably predict their subsequent vocabulary size, with early gestures corresponding to words that will subsequently appear in speech (Colonnesi et al., Reference Colonnesi, Stams, Koster and Noom2010; Rowe et al., Reference Rowe, Özçalişkan and Goldin-Meadow2008; Rowe & Goldin-Meadow, Reference Rowe and Goldin-Meadow2009). Gesture–speech combinations also precede and predict the onset of two-word phrases, underscoring their foundational role in the development of syntax (Goldin-Meadow & Butcher, Reference Goldin-Meadow, Butcher and Kita2003; Iverson & Goldin-Meadow, Reference Iverson and Goldin-Meadow2005; Özçalışkan & Goldin-Meadow, Reference Özçalışkan and Goldin-Meadow2005). As such, gestures constitute not only an indicator of overall communicative ability, but also a “harbinger” of subsequent linguistic milestones (Goldin-Meadow, Reference Goldin-Meadow2015, p. 3).
Importantly, not all gestures are the same, and different gesture types convey different meanings. A pioneer classification is McNeill’s taxonomy of gestures (McNeill, Reference McNeill1992), distinguishing different gesture types based on their semantic function. In this study, we will focus on: (1) deictic gestures, pointing or referring to specific objects or locations in space, (2) iconic gestures, directly representing an object or action, by depicting a characteristic of the referent (e.g., mimicking DRIVINGFootnote 2), and (3) emblems or conventional gestures, which are culturally shared gestures with arbitrary meaning (e.g., THUMBS UP to convey “good”; Ekman & Friesen, Reference Ekman and Friesen1969).
Co-speech gesture production is delayed, atypical, and/or impaired in children with ASD (see Ramos-Cabo et al., Reference Ramos-Cabo, Vulchanov and Vulchanova2019 for an overview). Its pervasive impairment even represents a core characteristic of the non-verbal communicative behaviours identified to establish the autism diagnosis (American Psychiatric Association, 2013). Autistic children indeed have been described to produce fewer gestures in comparison to their typically developing (TD) peers (Mishra et al., Reference Mishra, Ceballos, Himmelwright, McCabe and Scott2021; Perrault et al., Reference Perrault, Chaby, Bigouret, Oppetit, Cohen, Plaza and Xavier2019). This difference is notably observable in toddlers, who produce fewer deictic gestures such as pointing (Manwaring et al., Reference Manwaring, Stevens, Mowdood and Lackey2018; Mastrogiuseppe et al., Reference Mastrogiuseppe, Capirci, Cuva and Venuti2015). However, differences in gesture use and comprehension between autistic and neurotypical children are not merely quantitative, but also qualitative. Precisely, pointing gestures can be distinguished based on their pragmatic function, aim, and social load (Baron-Cohen, Reference Baron-Cohen1989; E. Bates, Reference Bates1976): protoimperative pointing, on the one hand, is used to request or obtain objects. On the other hand, protodeclarative pointing serves to share attention and interest by directing addressees to objects or events. Research shows that while children with ASD may demonstrate relatively intact protoimperative pointing, they exhibit marked deficits in the comprehension and production of protodeclarative pointing (Baron-Cohen, Reference Baron-Cohen1989; E. Bates, Reference Bates1976; Wetherby & Prutting, Reference Wetherby and Prutting1984). This lack of protodeclarative pointing has been associated with reduced interest in social interactions, as well as broader challenges in joint attention, Theory of Mind (i.e., ToM; the ability to attribute mental states to others and predict their behaviours accordingly; Premack & Woodruff, Reference Premack and Woodruff1978), and social communication (Baron-Cohen, Reference Baron-Cohen1989; Brinck, Reference Brinck2004; Camaioni et al., Reference Camaioni, Perucchini, Bellagamba and Colonnesi2004). They are considered hallmark characteristics of ASD. Lower rates of conventional gesture use have also been reported in four- to-twelve-year-old autistic children, in comparison to age-matched TD peers of comparable general language skills (Huang et al., Reference Huang, Wong, Lam, Cheng and So2020) and full-scale IQ (So et al., Reference So, Wong, Lui and Yip2015).
Many efforts have been engaged to characterize the productive gestural profiles of children with ASD to better inform our understanding of their (non-verbal) communicative behaviours. However, studies exploring the extent to which autistic children understand gestures remain limited to date (see Choi & Rowe, Reference Choi and Rowe2024 calling for more studies on this topic). Yet, a better understanding of the gestural profiles of autistic children is highly valuable, for both diagnostic and intervention purposes. A recent review noticeably emphasized that gestures constitute an early diagnostic marker for ASD (Stewart et al., Reference Stewart, Vigil, Olszewski and Thornock2022), helping to detect and identify children at higher risk of autism at earlier stages. Early detection consequently promotes chances to receive early tailored interventions (Zwaigenbaum et al., Reference Zwaigenbaum, Bauman, Stone, Yirmiya, Estes, Hansen, McPartland, Natowicz, Choueiri, Fein, Kasari, Pierce, Buie, Carter, Davis, Granpeesheh, Mailloux, Newschaffer, Robins and Wetherby2015), which further lead to better social and communicative outcomes (Dawson, Reference Dawson2008; Estes et al., Reference Estes, Munson, Rogers, Greenson, Winter and Dawson2015). In addition, better descriptions of gestural profiles in autism may also support the development and implementation of gesture-based therapies, which benefit autistic children across the spectrum (So et al., Reference So, Wong, Lam, Lam, Chui, Lee, Ng, Chan and Fok2018). Finally, in the light of the well-established relationship between gesture and language development, gestures can also serve as a screening tool for children at higher risk of language delays and impairments (Choi & Rowe, Reference Choi and Rowe2024; Goldin-Meadow, Reference Goldin-Meadow2015; Kilili-Lesta et al., Reference Kilili-Lesta, Voniati and Giannakou2022), a common feature in autistic children (Rapin & Dunn, Reference Rapin and Dunn2003; Schaeffer et al., Reference Schaeffer, Abd El-Raziq, Castroviejo, Durrleman, Ferré, Grama, Hendriks, Kissine, Manenti, Marinis, Meir, Novogrodsky, Perovic, Panzeri, Silleresi, Sukenik, Vicente, Zebib, Prévost and Tuller2023; Silleresi, Reference Silleresi2023).
Moreover, work on the comprehension of gestures has so far yielded inconsistent findings, which may be explained by the variety of tasks and methodology used. In a study investigating the recognition of video-taped decontextualized speechless gestures, Perrault et al. (Reference Perrault, Chaby, Bigouret, Oppetit, Cohen, Plaza and Xavier2019) showed that four- to-eight-year-old children with ASD recognized fewer gestures of iconic and conventional types, in comparison to both TD children and children with language impairment of the same age, suggesting difficulties in this realm beyond language impairment (Perrault et al., Reference Perrault, Chaby, Bigouret, Oppetit, Cohen, Plaza and Xavier2019). Yet, no mention of the non-verbal reasoning abilities of the sample was provided, nor of their linguistic abilities. Similarly, Stieglitz Ham et al. (Reference Stieglitz Ham, Bartolo, Corley, Rajendran, Szabo and Swanson2011) showed that 19 autistic children aged 7 to 15 recognized fewer video-taped pantomimes and actions depicting object use compared with their neurotypical peers matched on chronological age, gender, and full-scale IQ (Stieglitz Ham et al., Reference Stieglitz Ham, Bartolo, Corley, Rajendran, Szabo and Swanson2011). Evidence of impaired understanding of iconic gestures in combination with speech has also been previously reported in autistic adolescents (Silverman et al., Reference Silverman, Bennetto, Campana and Tanenhaus2010; Hubbard et al., Reference Hubbard, McNealy, Scott-Van Zeeland, Callan, Bookheimer and Dapretto2012), supporting the possibility of a cross-modal, audiovisual integration deficit.
These findings, however, do not align with previous work conducted by Dimitrova et al. (Reference Dimitrova, Özçalışkan and Adamson2017), exploring the gesture comprehension abilities of autistic children aged 2–12 in comparison to vocabulary-matched TD peers. They found no difference in the recognition performance of deictic, iconic, and conventional gestures when participants had to select the picture, out of two choices, corresponding to the gesture performed by the experimenter, either proposed alone or in combination with single words (Dimitrova et al., Reference Dimitrova, Özçalışkan and Adamson2017). Furthermore, in a pantomime comprehension task in which participants were required to select the picture corresponding to a video-taped actor performing a pantomimic action (e.g., a girl kissing a boy), 7- to-11-year-old autistic children also performed on par with their TD peers of comparable age and IQ level (Adornetti et al., Reference Adornetti, Ferretti, Chiera, Wacewicz, Żywiczyński, Deriu, Marini, Magni, Casula, Vicari and Valeri2019). Even better recognition performance was exhibited by children with autism compared with neurotypical peers matched on vocabulary (Hamilton et al., Reference Hamilton, Brindley and Frith2007), though in a task that did not involve live or video-taped gesturing. Precisely, participants were required to select the correct hand position, among three black-and-white pictures of a hand, that matched a drawing representing a protagonist performing a gesture with a missing hand (e.g., when shown a drawing of a soldier with a missing hand, participants had to choose the corresponding hand from three available options). The group of autistic children provided more correct responses than their neurotypical peers who were younger in chronological age but had comparable vocabulary knowledge (autistic children: 8;1 ± 1; neurotypical children: 4;1 ± 0). The offline nature of the task, which did not involve processing the motion of the gesture, may have favoured the autistic participants’ strength in analytic processing and attention to the details (Brosnan et al., Reference Brosnan, Lewton and Ashwin2016; U. Frith, Reference Frith2003).
Different explanations as to why autistic individuals may experience challenges in co-speech gestures comprehension can be put forth. First, under the integrated-system hypothesis where gestures and speech constitute a unified communicative and cognitive system (e.g., Goldin-Meadow, Reference Goldin-Meadow2003; Kendon, Reference Kendon2000; McNeill, Reference McNeill1992), difficulties in gesture comprehension may be linked to challenges with language processing (e.g., Goldin-Meadow & Alibali, Reference Goldin-Meadow and Alibali2013; Kelly et al., Reference Kelly, Özyürek and Maris2010). Second, difficulties with co-speech gestures might stem from reduced attention and appetence towards socio-communicative cues, widely documented in autism (e.g., Chevallier et al., Reference Chevallier, Kohls, Troiani, Brodkin and Schultz2012; Klin et al., Reference Klin, Lin, Gorrindo, Ramsay and Jones2009). Third, challenges may relate to a deficit in multimodal integration of speech and gesture, as suggested by neuroimaging studies (e.g., Hubbard et al., Reference Hubbard, McNealy, Scott-Van Zeeland, Callan, Bookheimer and Dapretto2012), as well as by the observation that challenges in understanding gesture emerge when these are presented simultaneously with speech in autistic adolescents (Silverman et al., Reference Silverman, Bennetto, Campana and Tanenhaus2010). Noticeably, this difficulty was not attributable to lower comprehension of either speech or gesture alone. As the authors point out, however, it remains to be determined whether the difficulty in integrating speech and gesture stems from the “cross-sensory nature of the task itself” (Silverman et al., Reference Silverman, Bennetto, Campana and Tanenhaus2010, p. 389) or rather from the higher-order process of combining the verbal and gestural information together.
Previous studies on gesture comprehension have helped us to better describe the gestural profiles of autistic individuals. They nevertheless present an incomplete picture of the gestural communicative processes at play in comprehension: Indeed, co-speech gestures, either replacing, interacting, or co-occurring with speech, serve a cognitive or communicative purpose during communicative acts (Clough & Duff, Reference Clough and Duff2020). Such gestures are thus to be understood in their linguistic context, an approach rarely adopted in previous research (e.g., Adornetti et al., Reference Adornetti, Ferretti, Chiera, Wacewicz, Żywiczyński, Deriu, Marini, Magni, Casula, Vicari and Valeri2019; Perrault et al., Reference Perrault, Chaby, Bigouret, Oppetit, Cohen, Plaza and Xavier2019; Silverman et al., Reference Silverman, Bennetto, Campana and Tanenhaus2010; Stieglitz Ham et al., Reference Stieglitz Ham, Bartolo, Corley, Rajendran, Szabo and Swanson2011). Instead, decontextualized stimuli were often used, in the sense that the gestures to be recognized were not integrated into a discourse, contrary to naturalistic situations in which they most likely appear, such as conversations and narrations. Moreover, despite a few exceptions (Dimitrova et al., Reference Dimitrova, Özçalışkan and Adamson2017; Perrault et al., Reference Perrault, Chaby, Bigouret, Oppetit, Cohen, Plaza and Xavier2019), previous studies have generally investigated a specific gesture type at a time (often iconic gestures), although autistic children tend to primarily face challenges with deictic and conventional gestures (Ramos-Cabo et al., Reference Ramos-Cabo, Vulchanov and Vulchanova2019).
To address these gaps, the present study explored the gestural comprehension abilities of children with autism aged 3–12, in comparison to their neurotypical peers carefully matched on age, non-verbal IQ, and receptive morphosyntactic skills. This age range was selected because gesture comprehension is an evolving process developing throughout childhood (Kelly & Church, Reference Kelly and Church1998). During this period, several cognitive skills that may be linked to gesture comprehension in narratives co-develop and undergo significant maturation: attention and executive functions (Anderson, Reference Anderson2002), social cognition such as ToM (Hughes & Devine, Reference Hughes and Devine2015), perspective-taking (Beatini et al., Reference Beatini, Cohen, Di Tore, Pellerin, Aiello, Sibilio and Berthoz2024), as well as narrative comprehension (Stein & Glenn, Reference Stein, Glenn and Freedle1979). In the present study, we assessed the recognition of three types of gestures (i.e., deictic, iconic, and conventional) that replaced verbal information embedded in an overall story.
Specifically, this study aimed to answer two research questions: (1) Do autistic and neurotypical participants differ in their ability to understand deictic, iconic, and conventional gestures when they are integrated into a narrative and replace verbal information? (2) What is the impact of linguistic (i.e., receptive morphosyntactic and vocabulary skills) and cognitive (i.e., working memory, WM) variables on the performance, so as to shed light on the underlying mechanisms at play in gesture comprehension?
We predicted that children with ASD would show one of two types of performance: On the one hand, they could display lower performance than their TD peers, even after controlling for their non-verbal reasoning and linguistic abilities, as difficulties with both speechless iconic and conventional gesture recognition have been reported in autistic children (Perrault et al., Reference Perrault, Chaby, Bigouret, Oppetit, Cohen, Plaza and Xavier2019). Investigating the respective impact of the linguistic and cognitive variables on the performance will help to elucidate why autistic children might display lower gesture comprehension performance: Indeed, autistic children may experience challenges (a) in parsing the verbal part of the stimulus (namely the spoken sentence preceding the gesture) or (b) in processing the subsequent gestural part of the stimulus; alternatively, challenges may stem from difficulties (c) in integrating these two sources of information (i.e., language and gesture). In the latter case, examining the impact of WM will indicate whether the difficulties in multimodal integration stem from challenges in sequentially maintaining and manipulating these sources of information or whether difficulties lie rather in a higher-order combination of these sources to correctly interpret the message. On the other hand, children with ASD could show similar performance to their TD peers, since our task differed from those of previous studies by only requiring a non-verbal response. This design may have facilitated task performance in children with ASD, by steering clear of their more pronounced challenges in the realm of language production (Rapin & Dunn, Reference Rapin and Dunn2003).
2 Method
2.1 Participants
2.1.1. Groups
A total of N = 146 children, predominantly White, participated in the study. The group of autistic children was composed of N = 73 participants aged 3;10 to 11;7 years (M age = 101 ± 25.6 months), to which corresponded N = 73 neurotypical children aged between 3;11 and 11;10 years (M age = 100 ± 27.4), carefully matched on age, non-verbal IQ, and receptive morphosyntactic skills with a propensity score (see Supplementary Material, Appendix 1 for further information on the matching procedure). There were more female participants in the neurotypical group than in the autistic group (χ2 (1) = 38.50, p < .001). TD children had no suspicion or diagnosis of ASD, nor of any other developmental disorder or known linguistic or cognitive impairment. Children had no reported visual or auditory impairment. All participants in the autistic group had an official diagnosis of ASD following the criteria of the DSM-5 (American Psychiatric Association, 2013), established by a certified clinician prior to the study with gold-standard tools (e.g., Autism Diagnostic Observation Schedule: ADOS-2; Lord et al., Reference Lord, Rutter, DiLavore, Risi, Gotham and Bishop2012, Autism Diagnostic Interview Revised: ADI-R; Lord et al., Reference Lord, Rutter and Le Couteur1994).
2.1.2. Sample recruitment and characteristics
Participants were recruited via flyers, contact with clinicians and hospitals, and websites of associations relevant to the autistic community. Interested families contacted the team via the project’s website, emails, or phone calls. As part of an international project, 30.1% of the participants were tested in Switzerland, 32.9% in Germany, 26.7% in France, 3.4% in the UK, 5.5% in the USA, and 1.4% in Canada. Because multilingualism is a widespread phenomenon (Grosjean, Reference Grosjean2010), we did not restrict inclusion in the study to monolingual participants, but instead carefully monitored the participants’ language backgrounds with the caregiver questionnaire Quantifying Bilingual EXperience (Q-BEx; De Cat et al., Reference De Cat, Kašćelan, Prevost, Serratrice, Tuller and Unsworth2022). The groups of children with ASD and TD children were balanced in their proportion of monolinguals and multilinguals (see Table 1, 47.9% of multilinguals in the group of children with ASD and 45.2% in the group of TD children; χ2 (1) = 1.40, p > .05), with participants considered multilinguals if they had been exposed to two (or more) languages more than 20% of their lifetime, following previous studies (Hantman et al., Reference Hantman, Choi, Hartwick, Nadler and Luk2023).
Parental educational level indexed socioeconomic status, given its important correlation with family income (Hauser & Warren, Reference Hauser and Warren1997), association with children’s academic performance and cognitive development (Rindermann & Ceci, Reference Rindermann and Ceci2018). The highest level of education achieved between the two caregivers was selected and converted into a five-point Likert scale, from (1) elementary school to (5) university. The two groups had relatively high scores of socioeconomic statuses (Table 1). Yet, it was not possible to match participants on this criterion in addition to age, non-verbal IQ, and morphosyntactic skills, and the group of TD children had higher socioeconomic status scores on average (W = 2179.5, p = .04). Socioeconomic status was added as a covariate in the statistical analyses to control for its impact.
Table 1. Sample description
2.2 Measures
Each participant was individually tested in a quiet room, always accompanied by an experimenter. The Gesture Comprehension Task required participants to (1) sequentially process both the verbal and gestural part of the stimulus and (2) to successfully integrate them together to select the correct answer. To account for the cognitive and linguistic resources involved in the task, we monitored and controlled for the resources mobilized during the task, namely linguistic skills, visual perception of communicative cues, and WM. As such, the assessment started with first a pre-assessment task ensuring that the participant had sufficient attentional resources towards gestural cues. This was followed by the Gesture Comprehension Task. Then, the Peabody Picture Vocabulary Test (Dunn & Dunn, Reference Dunn and Dunn2007) and the Test for Receptive Grammar (Bishop, Reference Bishop2003) monitored the participants’ linguistic abilities. Raven’s Progressive Matrices (Raven et al., Reference Raven, Rust, Chan and Zhou2018) were used to assess non-verbal IQ, and the Frog Matrices Task (Morales et al., Reference Morales, Calvo and Bialystok2013) measured WM.
2.2.1. Gesture Comprehension Task
To test the participants’ gestural comprehension abilities in a narrative context, a new gamified task inspired by Botting et al. (Reference Botting, Riches, Gaynor and Morgan2010) and Wermelinger et al. (Reference Wermelinger, Gampe, Helbling and Daum2020) was developed. The task was digitalized and included 12 items preceded by two warm-up trials with explanatory feedback. All items formed a coherent story. The task was introduced by Gabi, a 3D character with a natural human-like voice, who was asking for help to understand the story played on TV. All instructions were given orally by this character, ensuring similar quantity and quality of instructions across participants. Participants were asked to watch and listen carefully to a short video and to select, by touching the screen, the corresponding picture out of four response choices.
Language versions. The task was made equivalent in four languages: English, French, German, and Italian. All items and instructions were translated from French to the other languages and were recorded by native speakers of the respective languages. The complete list of items in all languages is available in Supplementary Material, Appendix 2.
Stimuli description. Each stimulus consisted of a video of an actor enunciating a sentence and replacing the last part with a speechless gesture, either of deictic (i.e., pointing gesture, four items), iconic (four items), or conventional (four items) type (see Figure 1 for an example of each gesture type). Natural congruent facial emotions were kept ensuring a naturalistic context. The stimuli always appeared in the same order for all participants (i.e., stimulus with a deictic gesture, followed by an iconic, and a conventional gesture). The videos were recorded by a professional cameraman, which was subsequently post-edited for sound normalization and noise reduction. The stimuli’s narrations were short sentences of maximum 12 words, with reduced morphosyntactic and vocabulary difficulties, to avoid penalizing participants with lower language skills (i.e., simple clause syntactic structures, verbs in present tense, words emerging in the first years of life present in the Mac-Arthur Bates Checklist Inventory in English (Fenson et al., Reference Fenson, Marchman, Thal, Dale, Reznick and Bates2007), German (Szagun et al., Reference Szagun, Schramm and Stumper2009), French (Kern et al., Reference Kern, Langue, Zesiger and Bovet2010), and Italian (Rinaldi et al., Reference Rinaldi, Pasqualetti, Stefanini, Bello and Caselli2019)). Gestures were selected to minimize cultural variations in the Western countries the study took place in.
Figure 1. Example of stimuli of deictic, iconic, and conventional gesture types in the four languages.
Response choices. Four response options (Figure 2) that did not require verbal naming were presented. This number was chosen following the original task (Botting et al., Reference Botting, Riches, Gaynor and Morgan2010) and enabled lowering the likelihood of hitting a correct response by chance to 25%. The verbal and non-verbal components of each item were intentionally designed to be ambiguous when processed separately. Accordingly, participants had to integrate both verbal and gestural parts to accurately select the correct item from the four presented options, except for the stimuli of deictic types where paying attention to the pointing gesture was sufficient to succeed. The response options were: (a) the correct answer, depicting the correct object or the protagonist performing the correct action, (b) a gestural distractor, referring to a gesture close to the target gesture that could be selected if the participant paid only attention to the gesture, but not to the verbal part of the stimulus, (c) a semantic distractor, that matched the verbal content of the stimulus but not the gesture seen, and (d) an oddball, matching neither the stimulus’ verbal information nor the gesture. The oddball pictures were of the same attractiveness and nature as the other response choices to ensure consistency (e.g., animate/inanimate object). For deictic gestures, the gestural distractor (c) was replaced by a visual distractor, namely a picture visually close to the target item in terms of colour and/or shape, as it was not possible to conceive a gestural distractor for this type of gesture. The response choices appeared simultaneously at the end of the stimulus in a random order.
Figure 2. Example of response choices for stimuli of deictic, iconic, and conventional type.
Performance calculation. For each item, 1 point was awarded for a correct picture selection, while 0 point was given for an incorrect response, irrespective of the distractor selected. The total Gesture Comprehension Task score varied between a minimum of 0 and a maximum of 12 points.
2.2.2. Pre-assessment task
A short six-item pre-assessment task ensured that the participant had sufficient attention towards video-taped gestures. Each item consisted of a video-taped speechless gesture performed by an actor. Participants had to select the video of the same gesture performed by another actor amongst three choices. Participants performing below chance level (i.e., 33%) were excluded. All participants (N = 146) included in this study performed above chance level at this task.
2.2.3. Morphosyntactic assessment
As the Gesture Comprehension Task involved the comprehension of gestures embedded in a sentence, receptive morphosyntactic skills were assessed using the Test For Receptive Grammar 2 (Bishop, Reference Bishop2003) and its respective adaptations in the different languages of testing, namely French (Lecocq Reference Lecocq1996), German (Fox-Boyer et al., Reference Fox-Boyer2006) and Italian (Suraniti et al., Reference Suraniti, Neri and Ferri2009). This standardized task assesses the comprehension of different grammatical structures with increasing complexity (e.g., passive sentences, “neither nor” constructions, etc.), by requiring participants to select, among four choices, the image corresponding to the sentence heard. Z-scores were used in the analyses.
2.2.4. Receptive vocabulary assessment
Given the predictive role of vocabulary knowledge on the gestural development in both autistic and neurotypical children (Choi & Rowe, Reference Choi and Rowe2024; Kilili-Lesta et al., Reference Kilili-Lesta, Voniati and Giannakou2022), the PPVT was administered to assess the breadth of receptive vocabulary (i.e., English: Dunn & Dunn, Reference Dunn and Dunn2007, French: Dunn et al., Reference Dunn, Dunn, Dunn and Thériault-Whalen1993, German: Lenhard et al., Reference Lenhard, Lenhard, Segerer and Suggate2015, Italian: Stella et al., Reference Stella, Pizzoli and Tressoldi2000). Participants selected, among four options, the picture corresponding to a word heard. Raw scores were converted into z-scores to allow comparison across languages.
2.2.5. Working memory (WM) assessment
As the Gesture Comprehension Task involved WM, the Frog Matrices Tasks (FMT) following Morales et al. (Reference Morales, Calvo and Bialystok2013) was administered. In this variant of the Corsi Blocks Task (Berch et al., Reference Berch, Krikorian and Huha1998), children had to recall the backward location of two to six frogs appearing sequentially. The memory span was determined as the longest sequence of frogs that the participant could recall accurately in at least one of the two trials proposed. Groups did not differ in their WM spans (W = 1884.5, p = .08).
2.2.6 Non-verbal reasoning assessment
The short version of the digitalized Raven’s Progressive Matrices (Raven’s 2; Raven et al., Reference Raven, Rust, Chan and Zhou2018) was used to assess and control for non-verbal reasoning. The participant selected the option, among five choices, that best fitted the visual geometric design displayed on the screen. The resulting standardized non-verbal IQ score was used in subsequent analyses.
Descriptive statistics of the linguistic and non-verbal measures are presented in Table 2.
Table 2. Descriptive statistics of the linguistic and non-verbal measures
3 Data analyses
A binomial generalized mixed-effects model with a logit function on accuracy was constructed to investigate whether the two groups (children with ASD and TD children) had different gesture comprehension performances, and whether their scores were differently influenced by the type of gesture. Analyses were run in R using R Studio (Version 2023.12.0 + 369) with the lme4 package (D. Bates et al., Reference Bates, Mächler, Bolker and Walker2015).
The fixed effects of the model included group (autistic children, neurotypical children), gesture type (deictic, iconic, conventional), and their interaction. In addition, the following covariates were added to investigate their impact on performance: Receptive morphosyntax, vocabulary knowledge, and WM. Age, the interaction between age and group, socioeconomic status, sex assigned at birth, non-verbal reasoning, and language of testing were also entered as fixed effects to account for their effect and potential differences between groups. Participants and items were entered as random effects. We began by including a maximal random effects structure and only simplified the complexity of random effects in case of convergence issues (Barr et al., Reference Barr, Levy, Scheepers and Tily2013). Information on contrast coding and model specifications is available in Supplementary Material, Appendix 3 and under: https://osf.io/vp764/
4 Results
Descriptively, both groups performed well on the task, above chance level with scores approaching ceiling performance for all gesture types (Figure 3). Overall, children with ASD produced an average of 12.10% incorrect responses, whereas TD children made only 3.54% errors, indicating a higher accuracy rate in the neurotypical group.
Figure 3. Mean recognition performance per group and gesture types.
The model showed a significant effect of group (β = − 1.64, SE = 0.68, p = .02) with autistic participants recognizing significantly fewer gestures than their TD peers (Supplementary Material, Appendix 4). However, the size of the difference was small, estimated at about 2.0% when accounting for all other variables. There was no significant effect of gesture types (all ps > .05), and the non-significant interaction between groups and gesture types (all ps > .05) suggests that both groups were not differentially impacted by the different types of gestures (deictic, iconic, and conventional).
Age was a significant positive predictor of the performance (β = 1.23, SE = 0.26, p < .001), suggesting better gesture comprehension in older children. However, the non-significant interaction between age and group (p > .05) suggests that age did not impact the group’s performance differently. Illustrations of the predicted performance as a function of age are presented in Supplementary Material, Appendix 6. They suggest that participants reach a performance plateau around age 8. More advanced analyses, such as generalized additive models (GAMs) would be required to confirm these observations.
A significant effect of vocabulary knowledge was found (β = 0.39, SE = 0.16, p = .02), showing that participants with higher vocabulary skills also exhibited better gesture recognition performance. However, there were no significant effects of either receptive morphosyntactic skills or WM on accuracy. Additionally, sex assigned at birth, socioeconomic status, non-verbal reasoning, and language of testing did not predict the performance either (all ps > .05).
To further examine the role of receptive vocabulary on the performance difference between autistic and neurotypical children, causal mediation analyses were conducted (see Supplementary Material, Appendix 5). Results showed that vocabulary knowledge significantly mediated the group difference in performance: On average, about 18.39% of the total effect of the difference between the two groups was explained by PPVT scores (proportion mediated estimate = 0.18, CI = [0.19, 0.35], p < .001; 25.96% in TD children, and 10.81% in children with ASD).
Explanatory post-hoc analyses were further conducted to investigate whether the monolingual versus multilingual status of the participants differently impacted the performance. Multilingual status (monolingual, multilingual) was sum-coded and included in the final model either as a fixed effect (model A) or in interaction with the group variable (model B). Results revealed no significant effect of multilingual status in either model (see Supplementary Material, Appendix 7).
To further investigate whether autistic and neurotypical children differed in their error patterns (e.g., selecting the semantic distractor, visual distractor, or oddball), a series of chi-square tests of independence were conducted. These tests allowed us to evaluate whether the observed error frequencies differed significantly between the two groups for each combination of gesture type and error type, compared with the expected frequencies under the null hypothesis of no association. Given that multiple chi-square tests were conducted, a Bonferroni correction was applied to control for the increased risk of type I error (risk of false positives). Results indicated that when they did not select the correct answer, autistic children were significantly more likely to select the semantic distractor than neurotypical children (χ2 (1) = 11.78, p < .001), but only in items with iconic gestures. No significant difference in the proportion of errors was evidenced for other gesture types, nor other error types. Moreover, both groups produced all three types of errors, without salient preference for a specific type over another. Supplementary Material, Appendix 8 shows a visualization of the distribution of errors for each group and gesture type.
5 Discussion
The first goal of this study was to examine whether and how autistic children understand three types of communicative gesture (i.e., deictic, iconic, and conventional) when these gestures replaced verbal information in a narrative. Their performance was compared to TD peers who were carefully matched on age, non-verbal IQ, and receptive morphosyntactic abilities. After accounting for their age, sex assigned at birth, non-verbal IQ, receptive morphosyntactic abilities, vocabulary knowledge, socioeconomic status, WM, and testing language, children with ASD recognized significantly fewer gestures than their TD peers, irrespective of the gesture type. Despite the small difference between groups, these findings suggest that autistic individuals may not fully grasp all the information conveyed by hand movements that naturally accompany speech in conversation and narration. Consequently, they may miss crucial elements of the intended message, potentially leading to challenges in understanding communicative partners who use gestures alongside speech. The findings go beyond previous work reporting challenges in gesture comprehension of iconic and conventional gestures in autistic children (Perrault et al., Reference Perrault, Chaby, Bigouret, Oppetit, Cohen, Plaza and Xavier2019; Stieglitz Ham et al., Reference Stieglitz Ham, Bartolo, Corley, Rajendran, Szabo and Swanson2011): In the present study, we observed subtle challenges that seem to arise with gestures being embedded in a richer linguistic context. The (relatively) greater difficulty in integrating multiple sources of information (i.e., here speech and gesture) relative to neurotypical peers aligns with previous research using multimodal audiovisual tasks (Mongillo et al., Reference Mongillo, Irwin, Whalen, Klaiman, Carter and Schultz2008; Silverman et al., Reference Silverman, Bennetto, Campana and Tanenhaus2010). It is important to acknowledge, however, that all participants, including those with ASD, demonstrated good performance, performing above chance levels already at a younger age. While the group difference was statistically significant, it remained relatively small and may only indicate subtle repercussions in daily living. Moving beyond a deficit-based view of autism (Mottron, Reference Mottron2017), these results can be interpreted as evidence that autistic and neurotypical children, when carefully matched on key variables, perform similarly well in integrating gesture following speech.
As expected, age was identified as a positive predictor of performance in both groups, with older participants recognizing more gestures than younger ones and performing at ceiling. This pattern suggests a gradual maturation of the ability to process gestures over time (Capone & McGregor, Reference Capone and McGregor2004; Perrault et al., Reference Perrault, Chaby, Bigouret, Oppetit, Cohen, Plaza and Xavier2019). The size of the age-related improvement in gesture recognition appeared similar between autistic and neurotypical children, although a higher variability in the autistic group was visible. Future studies may examine in more depth developmental aspects in the comprehension of different gesture types in both children with ASD and TD children, for instance with the help of GAMs.
The second goal of this study was to identify the factors that may explain the difference between the groups in the current task. Interestingly, better performance could not solely be attributed to a better comprehension of the verbal story in which gestures were embedded. Indeed, the sole reliance on the verbal information (i.e., the sentential part of the stimulus) was insufficient to select the correct picture, as parsing the visual gestural information was necessary to disambiguate between the correct response and the so-called semantic distractor. Similarly, parsing only the gestural part of the stimulus would have led to confusion between the correct answer and the gestural distractor, potentially leading to a greater proportion of gestural distractors. Further examination of the error patterns revealed no striking differences between autistic and neurotypical children, except for items with iconic gestures: When they produced an error in iconic items, children with ASD selected the semantic distractor significantly more often than TD children. This behaviour suggests that autistic children accurately parsed the verbal information provided in the stimulus and likely relied on this information more heavily (than on the gesture) when confronted with these items. In other terms, these findings could also suggest a lower reliance on the gestural part of the stimulus: This would align with previous work examining the production of gesture–speech combinations in school-aged children, which reported lower reliance on gestures by autistic children than their neurotypical peers to add information to their speech (Baumann et al., Reference Baumann, Özçalışkan and Adamson2019). Instead, the children with autism in that study relied more strongly on speech than on gestures. More effortful processing of gestures with high communicative load such as pointing, as compared to cues with lower communicative value (like grasping), has also been reported (Aldaqre et al., Reference Aldaqre, Schuwerk, Daum, Sodian and Paulus2016). In addition, previous research also suggests impaired perception of biological motion in autistic children (Blake et al., Reference Blake, Turner, Smoski, Pozdol and Stone2003), with a preference towards non-social cues over social stimuli (Chevallier et al., Reference Chevallier, Kohls, Troiani, Brodkin and Schultz2012). According to the social motivation theory, the challenges autistic individuals experience in social cognition, including the detection, interpretation, and use of non-verbal cues such as gestures in the social context (C. D. Frith, Reference Frith2008), may stem from reduced social motivation (Chevallier et al., Reference Chevallier, Kohls, Troiani, Brodkin and Schultz2012; Parish-Morris et al., Reference Parish-Morris, Schultz and Herrington2021). This reduced social motivation reflects a diminished tendency to orient towards social cues, seek social interactions, and experience reward from social engagement and bonding maintenance. This possibility was ruled out in our task, as we ensured sufficient attention towards gestural stimuli by proposing a pre-assessment task: Only the participants who performed above chance level to this task were included in the study. Therefore, it is unlikely that group differences were driven by salient reduced attention to the gestures.
Pertaining to potential linguistic factors, morphosyntactic skills did not significantly impact the performance. This is arguably due to the intentionally limited morphosyntactic difficulty of the stimuli’s verbal component, restricted to simple clauses with verbs in the present tense only. This reduced linguistic complexity was proposed to avoid penalizing children with language impairment, often represented in ASD (Rapin & Dunn, Reference Rapin and Dunn2003).
Vocabulary knowledge, in contrast, significantly predicted gesture comprehension, in both children with ASD and TD children. This finding emphasizes the close link between speech and gesture, an intertwinement that has intrigued researchers for decades (e.g., Goldin-Meadow & Alibali, Reference Goldin-Meadow and Alibali2013; Kita, Reference Kita2000; Kita & Emmorey, Reference Kita and Emmorey2023). It furthermore supports the claim of a probable bidirectional, mutual influence of gesture and language developments: Not only gesture (use) predicts later vocabulary size (e.g., Rowe & Goldin-Meadow, Reference Rowe and Goldin-Meadow2009), but vocabulary may in turn predict gesture production, as evidenced notably for word comprehension and gestures in children with high and low risk of ASD (Roemer et al., Reference Roemer, West, Northrup and Iverson2019). The present study aligns with this view and provides, for the first time, empirical evidence of a predictive role of vocabulary on gesture comprehension.
Mediation analysis enabled us to decipher the extent to which vocabulary knowledge mediated the group difference in performance and determine whether the effect of the group difference on gesture comprehension was a direct effect. Analyses showed that vocabulary knowledge significantly mediated the performance difference between autistic and neurotypical children, with about 18% of the total effect being mediated by vocabulary. While this contribution was pivotal, most of the effect (approximately 82% on average across groups) was therefore not explained by vocabulary alone. This indicates that other factors, or their combination, may have contributed to the difference in gesture comprehension between autistic and neurotypical children in this task.
Finally, the (small) gap in performance between groups might not stem from difficulties in recalling and holding both types of input (i.e., language and gesture) to further combine them to answer correctly. Indeed, neither group differed on average in their WM span, nor did this factor have a predictive impact on performance. Yet, challenges in integrating information from multiple sensory modalities, such as verbal descriptions and gestures, have been reported in autistic children and adolescents (Hubbard et al., Reference Hubbard, McNealy, Scott-Van Zeeland, Callan, Bookheimer and Dapretto2012; Silverman et al., Reference Silverman, Bennetto, Campana and Tanenhaus2010). These studies presented speech and gestures simultaneously, which may have contributed to the difficulties observed. In our task, while the combination of both modalities was necessary to succeed, speech and gestures were processed sequentially. This sequential presentation may have alleviated the challenges associated with simultaneous multisensory integration: Such difficulties are frequently reported in autistic individuals and have been shown to contribute to sensory overload and processing challenges (Iarocci & McDonald, Reference Iarocci and McDonald2006). Thus, the difference in the mode of presentation may explain the relatively spared performance of the autistic participants in our task. While further research should allow confirmation of this possibility, this finding highlights the potential of sequentially presenting information, such as verbal input followed by gestures, as a means to reduce sensory overload. This approach aligns with outcomes from sensory integration therapy (Ayres, Reference Ayres2005), where structured and gradual exposure to sensory stimuli has been recognized as enhancing sensory processing and adaptive responses in autistic children (Camino-Alarcón et al., Reference Camino-Alarcón, Robles-Bello, Valencia-Naranjo and Sarhani-Robles2024; Schoen et al., Reference Schoen, Lane, Mailloux, May-Benson, Parham, Smith Roley and Schaaf2019). Such interventions in occupational therapy are effective in improving sensory integration, communication, and functional outcomes in autistic children (see Camino-Alarcón et al., Reference Camino-Alarcón, Robles-Bello, Valencia-Naranjo and Sarhani-Robles2024 for a review).
These results may also inform augmentative and alternative communication interventions, such as key-word signing (KWS), which combines speech with simultaneous manual signs to emphasize key elements of the discourse. KWS has been shown to improve both language comprehension and production in autistic individuals (Tan et al., Reference Tan, Trembath, Bloomberg, Iacono and Caithness2014). However, as underscored by Tan et al. (Reference Tan, Trembath, Bloomberg, Iacono and Caithness2014), KWS relies on sustained visual attention to the interlocutor’s signs, and reduced attentional focus may impact its effectiveness. Individuals also show variability in their preferences for the modalities in which information is conveyed, depending on a range of factors such as the communicative situation and partners (Iacono et al., Reference Iacono, Lyon, Johnson and West2013; Wendt, Reference Wendt, Mirenda and Iacono2009). Therefore, some flexibility in the presentation of speech with gestures is required to tailor interventions to each child’s needs, thereby improving functional outcomes.
Two limitations pertaining to the representativeness of the sample might hinder the generalizability of the findings: First, our sample constitutes a WEIRD sample (i.e., Western, Educated, Industrialized, Rich, and Democratic; Henrich et al., Reference Henrich, Heine and Norenzayan2010), with the majority of the participants stemming from higher socioeconomic backgrounds. Second, only participants who completed the whole battery of tests were included in the study, potentially excluding children with greater attentional or behavioural difficulties or with higher severity of autism symptoms, a variable that was not monitored in this study. Still, as previous work has reported a negative correlation between autism severity and gesture production (Mastrogiuseppe et al., Reference Mastrogiuseppe, Capirci, Cuva and Venuti2015), a greater difference in gesture comprehension between our groups could be hypothesized if children with high severity scores were included.
Future work might investigate whether the understanding of co-speech gestures is differently impacted by the linguistic environment of autistic children. Indeed, existing work has shown that multilingual neurotypical children present an increased sensitivity to communication situations and can better interpret non-verbal elements such as pointing gestures (Yow & Markman, Reference Yow and Markman2011) in comparison to monolingual peers. In the present work, a post-hoc examination revealed no significant impact of multilingual status (i.e., being monolingual or multilingual) on performance. However, multilingualism is a complex and multifactorial phenomenon which groups comparisons based on arbitrary criteria are unlikely to fully capture (de Bruin, Reference de Bruin2019; De Houwer, Reference De Houwer2023). Therefore, additional investigations are warranted to examine which aspect of the multilingual experience may impact gesture comprehension. Determining whether the multilingual effects observed in neurotypical children hold in the autistic population would help to inform cognitive and communicative processes and, crucially, help to build evidence-based recommendations that are currently lacking (Beauchamp & MacLeod, Reference Beauchamp and MacLeod2017).
6 Conclusion
The present study provides novel insights into the comprehension abilities of autistic and neurotypical children, particularly in their capacity to understand communicative gestures embedded in a narrative. Precisely, it revealed subtle group differences favouring neurotypical children in their ability to combine speech and gesture into a unified representation which was necessary for task success. Interestingly, this difference could not be attributable to lower linguistic or WM skills. While statistically significant, the difference between groups was small, and children with ASD displayed overall good performance. This relatively spared performance in gesture comprehension may be attributed to the sequential presentation of speech followed by gestures, which likely alleviated sensory processing demands compared with simultaneous multimodal input. These findings have valuable implications for therapeutic and intervention programmes, highlighting the potential of structured, sequential presentation of multimodal cues to enhance comprehension and reduce sensory challenges in autistic children, thereby supporting effective communication strategies.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0305000925000157.
Data availability statement
Data, code to reproduce the analyses, and stimuli video examples are available under: https://osf.io/vp764/
Acknowledgments
We warmly thank all the families and children who participated in this study. Thanks also to our collaborators and students for their investment in both material creation and data collection. Special thanks to EIKON school and their talented collaborators for their drawings.
Funding statement
This study is supported by the Swiss National Science Foundation (SNSF) awarded to Stephanie Durrleman (grant PR00P1_193104/1).
Statement of Competing interest
The authors declare that they have no conflict of interest pertaining to the research, authorship, and/or publication of this article.
Compliance with ethical standards and informed consent
This study was approved by the Swiss Association of Research Ethics Committees Swissethics (Project ID-2022-00878) and is in accordance with the 1964 Helsinki declaration and its later amendments. All parents provided informed consent for their child’s participation prior to their inclusion in the study.
Authors contribution
PW: conceptualization, methodology, formal analysis, investigation, resources, data curation, supervision, visualization, and writing of the original draft. FB: methodology, investigation, resources, supervision, writing, review, and editing. DC: supervision, writing, review, and editing. ND: conceptualization, writing, review, and editing. ES: formal analysis, writing, review, and editing. SD: conceptualization, methodology, resources, supervision, writing, review, editing, project administration, and funding acquisition. All authors reviewed and approved the manuscript for publication.
Open access
