1. Introduction
When hearing or reading the word ‘coffee’, different perceptual experiences may be evoked. Our direct perceptual knowledge of that particular word referent influences the retrieval of its meaning (Juhasz et al., Reference Juhasz, Yap, Dicke, Taylor and Gullick2011; Juhasz & Yap, Reference Juhasz and Yap2013; Lynott et al., Reference Lynott, Connell, Brysbaert, Brand and Carney2020; Vergallito et al., Reference Vergallito, Petilli and Marelli2020). We can even re-construct that perceptual information to some extent, that is, thinking of a specific cup of coffee that we saw, recalling the experience of holding it with our hands, recalling how good it tastes and smells, or playing in our head the sound that originates from the coffee pot. Works within the embodied cognition framework suggest that direct experience is strongly involved in conceptual representations, along with linguistic information. That is, concepts are assumed to be grounded, at least partially, in the same systems that govern perception both at the cognitive and neural level (Barsalou, Reference Barsalou1999, Reference Barsalou2008; Glenberg & Kaschak, Reference Glenberg, Kaschak and Ross2003; Meteyard et al., Reference Meteyard, Cuadrado, Bahrami and Vigliocco2012; Pulvermüller, Reference Pulvermüller2018). Thus, information coming from our direct experience with the outside world is encoded in word meaning. If this is the case, to what extent and how does the deprivation of one sensory modality impact the organisation of the conceptual system subtending word meaning? According to theories of embodied cognition, the lack of visual and/or auditory experience might affect the organisation of the conceptual system, supporting the assumption that word meaning is actually grounded in the perceptual system (e.g., Gallese & Lakoff, Reference Gallese and Lakoff2005; Glenberg & Kaschak, Reference Glenberg, Kaschak and Ross2003; Meteyard et al., Reference Meteyard, Cuadrado, Bahrami and Vigliocco2012).
Perceptual norms are powerful tools to study the impact of perceptual grounding in forming conceptual representations through the generation and analyses of perceptual profiles of words. They also represent a useful tool to explore what happens to the conceptual system when one of the perceptual modalities is absent. Lynott and Connell (Reference Lynott and Connell2009, Reference Lynott and Connell2013) were the first to propose a measure capable of evaluating the perceptual experience associated with a word, separately for each of the five perceptual modalities, collecting perceptual strength norms for 423 English adjectives. Estimates of perceptual strength reflect the extension to which we can experience a certain object or property through the five senses, namely: vision, hearing, touch, taste, and smell. Each word was thus represented as a five-value vector that reflected its perceptual strength through all five dimensions. This procedure allowed for the building of ‘perceptual profiles’ of words, along with computing different composite metrics. Indeed, the authors introduced a series of composite measures of perceptual strength, with the aim of selecting a reliable operationalization for the construct. Particularly relevant appeared to be Maximum Perceptual Strength, which measures the highest rating across the five modalities, and Modality Exclusivity, an index of the extent to which a word is experienced by a single perceptual modality (Lynott & Connell, Reference Lynott and Connell2009). Minkowski Distance and Magnitude of Perceptual Strength have also been considered in a more recent study by Lynott et al. (Reference Lynott, Connell, Brysbaert, Brand and Carney2020) as indices for representing perceptual strength in all dimensions with and without attenuation of the weaker ones, respectively (Lynott et al., Reference Lynott, Connell, Brysbaert, Brand and Carney2020; Vergallito et al., Reference Vergallito, Petilli and Marelli2020). The effectiveness of these norms in predicting language processing was tested by comparing metrics of perceptual strength to concreteness and imageability (two psycholinguistic variables traditionally employed to capture grounded content in language; see Brysbaert et al., Reference Brysbaert, Warriner and Kuperman2014; Paivio, Reference Paivio1991, Reference Paivio2007) in a lexical decision task (Connell & Lynott, Reference Connell and Lynott2012, Reference Connell and Lynott2014). In particular, Maximum Perceptual Strength was found to be a better predictor of reaction times in word recognition tasks as compared to imageability and concreteness. The authors concluded that estimates of perceptual strength are likely to outperform concreteness and imageability ratings because they tend to evoke perceptual judgments that accurately reflect our own direct experience (Connell & Lynott, Reference Connell and Lynott2012; Speed & Brysbaert, Reference Speed and Brysbaert2022; but see Petilli & Marelli, Reference Petilli and Marelli2024).
If the conceptualization is heavily informed by direct experience, when perceptual information is disrupted following the deprivation of one sensory modality (e.g., vision, hearing, smell, etc.) changes in conceptual representations may be expected. In people with sensory impairments, however, the study of the impact of visual deprivation on the organisation of the conceptual system yielded contrasting results (e.g., Kim et al., Reference Kim, Elli and Bedny2019; Bedny et al., Reference Bedny, Koster-Hale, Elli, Yazzolino and Saxe2019; Bottini et al., Reference Bottini, Morucci, D’Urso, Collignon and Crepaldi2022; Lenci et al., Reference Lenci, Baroni, Cazzolli and Marotta2013; Marotta, Reference Marotta2014; Ricciardi et al., Reference Ricciardi, Vanello, Sani, Gentili, Scilingo, Landini, Guazzelli, Bicchi, Haxby and Pietrini2007; Ricciardi et al., Reference Ricciardi, Handjaras and Pietrini2014; Petilli & Marelli, Reference Petilli and Marelli2024; Speed et al., Reference Speed, Iravani, Lundström and Majid2022). The blind population has received extensive attention over the years, and some studies showed that blind individuals are able to acquire perceptual knowledge and to establish the corresponding representations independently of their visual deprivation through language and other sensory dimensions (Kim et al., Reference Kim, Elli and Bedny2019; Bedny et al., Reference Bedny, Koster-Hale, Elli, Yazzolino and Saxe2019). While perceptual ratings were never explicitly collected with blind individuals, other studies identified some differences in the conceptual organisation of sighted and blind individuals relative to perceptual features. For instance, Lenci et al. (Reference Lenci, Baroni, Cazzolli and Marotta2013) developed BLIND (BLind Italian Norming Data), a collection of semantic properties for Italian collected in both blind and sighted individuals, which represents a first insight on the conceptual representations in congenitally blind persons. Blind persons seem to produce significantly less direct perceptual features when describing concrete objects with respect to sighted individuals, while no substantial difference arises between groups with abstract concepts.
With regards to language processing, Bottini et al. (Reference Bottini, Morucci, D’Urso, Collignon and Crepaldi2022) revealed that blind individuals, who cannot rely on direct visual experience, are faster when processing concreteFootnote 1 words as compared to abstract ones, just as sighted individuals typically do, concluding that the concreteness effect seems not to be driven by the perceptual experience that we associate with concepts, but possibly by other modality-independent semantic features. In line with these findings, Petilli and Marelli (Reference Petilli and Marelli2024) have recently shown that direct visual experience has a partial but not critical role in concreteness and imageability judgments. In fact, the authors have shown that visual frequency predicts concreteness and imageability ratings in blind and sighted individuals alike, thus concluding that even in the absence of a direct visual experience, a concept can be perceived as concrete and imageable. Finally, Speed et al. (Reference Speed, Iravani, Lundström and Majid2022) have recently investigated language processing in another sensory-deprived population, that is, people with acquired anosmia. Investigating whether olfaction is needed to comprehend words related to odour, the authors found no difference between anosmic participants and controls in a lexical decision task and in a semantic similarity judgement task, finding no evidence of impairment in the processing of odour words following anosmia. Surprisingly, anosmic participants even outperformed controls when asked to recall odour and taste words from the lexical decision. However, only acquired anosmics took part in the study, that is, people with an intact sense of smell for a large part of their lives, who likely still had memories of olfactory experiences, with olfactory-related information being processed and stored without any detriment.
Taken together, these studies suggest that sensory deprivation does not necessarily prevent the acquisition of typical perceptual knowledge related to the corresponding perceptual modality, allowing for the establishment of the related conceptual representations. In fact, perceptual information about the impaired modality may not only be provided by the remaining sensory dimensions (i.e., Meteyard et al., Reference Meteyard, Cuadrado, Bahrami and Vigliocco2012), but also through language. Indeed, language plays a critical role in providing the missing input, thus establishing a link to process strictly sensory-related knowledge for concepts that cannot be experienced first-hand (Vigliocco et al., Reference Vigliocco, Meteyard, Andrews and Kousta2009; Campbell & Bergelson, Reference Campbell and Bergelson2022; Günther et al., Reference Günther, Petilli, Vergallito and Marelli2020; Lewis et al., Reference Lewis, Zettersten and Lupyan2019; Petilli & Marelli, Reference Petilli and Marelli2024). Not only that, but the linguistic experience is also a contributing factor to conceptual representation, whereby concepts are informed by both sensory-motor and linguistic experience (e.g., Barsalou et al., Reference Barsalou, Santos, Simmons, Wilson, de Vega, Glenberg and Graesser2008; Louwerse & Jeuniaux, Reference Louwerse, Jeuniaux, de Vega, Glenberg and Graesser2008; Vigliocco et al., Reference Vigliocco, Meteyard, Andrews and Kousta2009; Meteyard et al., Reference Meteyard, Cuadrado, Bahrami and Vigliocco2012; Andrews et al., Reference Andrews, Frank and Vigliocco2014). In fact, what we know about the world may be learnt by direct interaction with it, but also through language. Symbolic accounts of conceptual representation have argued that meaning can be (and is) extracted from language. We learn word meanings through explicit verbal descriptions and definitions (found in texts or delivered in conversations), but we can also learn word meanings implicitly through statistical distribution of words across texts (e.g., Landauer & Dumais, Reference Landauer and Dumais1997). It has also been proposed that language actually encodes perceptual information, suggesting that grounded information can be extracted from linguistic inputs without necessarily involving direct grounding (Louwerse & Zwaan, Reference Louwerse and Zwaan2009; Louwerse, Reference Louwerse2011; Gatti et al., Reference Gatti, Marelli, Vecchi and Rinaldi2022). The implication of these approaches for individuals with perceptual impairments is quite straightforward: if perceptual information about the world can be extracted from language, it is possible that perceptual information is still represented in their conceptual system. It remains to be seen whether qualitative or quantitative differences may be nonetheless present.
In the present study, we aimed to expand our understanding of the impact of perceptual experience on conceptual representation by studying early deaf individuals who had no or very limited access to auditory information. To this aim, we collected perceptual strength norms for 200 Italian words, using the task originally developed by Lynott and Connell (Reference Lynott and Connell2009) and later adapted to Italian by Vergallito et al. (Reference Vergallito, Petilli and Marelli2020). As anticipated, perceptual strength norms provide an estimate of the perceptual experience of a word associated with the five senses (Lynott & Connell, Reference Lynott and Connell2009, Reference Lynott and Connell2013); thus, they are ideally suited to investigate whether auditory information is (still) represented in the conceptual system of deaf individuals. Note that conceptual representations explicitly associated with perceptual modalities have never been investigated directly and systematically through the collection of perceptual norms in populations lacking one sensory modality. The norms we collected were thus compared to existing perceptual norms produced by hearing individuals with the aim of establishing whether (i) the auditory information is still represented in the conceptual system of deaf individuals, (ii) the perceptual strength of words evaluated by deaf individuals is reduced in comparison to hearing individuals, and (iii) the deprivation of one modality affects the reorganisation of representations involving other modalities.
2. Materials and Methods
2.1 Participants
A total of 37 early deaf individuals with moderate to profound hearing loss took part in the study.Footnote 2 Participants were recruited through personal contacts or advertising. They were first asked to fill in a questionnaire to collect information about their hearing loss and linguistic preferences. Participants’ details are summarised in Table 1.
Table 1. Demographic characteristics of the deaf participants (N = 37)
We assessed participants’ lexical proficiency in Italian using the LexITA test (Amenta et al., Reference Amenta, Badan and Brysbaert2021), which evaluates vocabulary knowledge in Italian L2 speakers. The test consisted of 60 Italian words and 30 non-words. Participants determined whether each item was an Italian word by responding with ‘yes’ or ‘no.’ The LexITA test was administered online, following the questionnaire and preceding the rating task. Results indicated that participants achieved accuracy scores consistent with advanced proficiency levels reported by Amenta et al. (Reference Amenta, Badan and Brysbaert2021).
Participants also provided information about their reading habits, including the frequency of reading books (novels, essays, biographies excluding school textbooks), newspapers (online and/or printed), and web pages and/or blogs. Responses were rated on a 6-point Likert scale from 0 (never) to 6 (every day). Participants reported reading web pages and newspapers quite frequently (M = 5.51, SD = 1.09 for web pages; M = 4.8, SD = 1.73 for newspapers), while their engagement with books was lower (M = 3.6, SD = 1.8). These self-reports suggest a high familiarity with written material in Italian.
Before beginning the experiment, participants were provided with informed written consent forms, which were structured in order to be easily understandable by the population of interest.
2.2 Materials
The item set used in the perceptual rating task contained a total of 200 words, which were pseudo-randomly extracted from the dataset by Vergallito et al. (Reference Vergallito, Petilli and Marelli2020), who collected perceptual strength ratings for 1121 Italian words in hearing adults. Given the length of the task, we could not submit the full dataset to our participants, therefore, we decided to extract a subset of 200 items so that the distribution of word frequency, word length, and composite metrics of perceptual strength (Magnitude, Minkowski Distance and Modality Exclusivity) was comparable to that of the original set. This ensured that the items rated in our study were a representative sample of the dataset by Vergallito et al. (Reference Vergallito, Petilli and Marelli2020) concerning relevant variables. Moreover, we sampled items considering only those words that, in the original set, were predominantly visual or auditory. The primary or dominant modality was assigned, in the original set, as the modality that received the highest rating on a Likert scale ranging from 0 to 5 (see Lynott & Connell, Reference Lynott and Connell2009; Vergallito et al., Reference Vergallito, Petilli and Marelli2020). Note that most words were visual dominant, and only a few were auditory dominant.
The sample set used in our study consisted of 73 primarily auditory words and 127 primarily visual words. The set included 147 nouns, 37 adjectives and 16 verbs. Distributions of relevant variables are reported in Table 2. A detailed description of item sampling procedure, along with the full list of items included in the study, is reported in the Online Additional Materials (https://osf.io/8nyrc/?view_only=9aca1e7a1a604702afeafa4c91549397).
Table 2. Frequency, length and perceptual strength metrics for the 200 words comprised in the selected set
Frequency values were extracted from Subtlex-IT (Crepaldi et al., Reference Crepaldi, Amenta, Mandera, Keuleers and Brysbaert2016) and Zipf-transformed (Brysbaert et al., Reference Brysbaert, Mandera and Keuleers2018). Word length was counted in letters. Perceptual strength metrics are extracted from Vergallito et al. (Reference Vergallito, Petilli and Marelli2020) and refer to hearing individuals.
2.3 Procedure
The study utilised Qualtrics (Qualtrics, Provo, UT) as an online platform, allowing participants to complete the rating task using their personal computer or smartphone. The 200 words were divided into two sets, with each set evaluated by a different group of participants. Each participant assessed a set of 100 words. The first set comprised 50 auditory-dominant and 50 visual-dominant words, while the second set included 23 auditory-dominant and 77 visual-dominant words. Additionally, the second set featured 30 additional words (15 auditory-dominant and 15 visual-dominant) that were repeated from the first set. These repeated words served as probes to assess consistency in evaluations across the two sets.Footnote 3
Words were presented individually, appearing in written form only within a sentence prompting participants with ‘To what extent can you experience [WORD] by’ (original Italian: ‘In che misura puoi avere esperienza di [PAROLA] attraverso’). Each prompt was accompanied by five scales representing the five perceptual modalities: hearing, taste, touch, smell, and vision. Following Vergallito et al. (Reference Vergallito, Petilli and Marelli2020), a six-step Likert scale ranging from 0 (not at all) to 5 (very much) was employed for each modality. Participants were required to actively select a value without a default selection. The scales were consistently presented in a fixed order, while word order was randomised for each participant. After rating each word across all five modalities, participants progressed to the next word by clicking a button at the bottom of the screen. The experiment was self-paced, and on average, participants completed it in approximately 30 min.
The rating task was preceded by instructions given both in Italian and in Italian Sign Language (LIS). Videos in LIS were included to make sure that all participants comprehended the instructions, thus completing the experiment without any issues.
The study was approved by the Ethics Committee of the University of Trento (protocol number: 2019-024) and was conducted according to the criteria of the Declaration of Helsinki.
2.4 Perceptual metrics
The aim of this study is to compare metrics of perceptual strength between deaf participants and those provided by Vergallito et al. (Reference Vergallito, Petilli and Marelli2020) for the same words based on ratings from hearing individuals. The hearing group data were obtained from https://osf.io/zdg59/ and made freely available by Vergallito and colleagues.
Following Lynott and Connell (Reference Lynott and Connell2009, Reference Lynott and Connell2013, Reference Lynott, Connell, Brysbaert, Brand and Carney2020) and Vergallito et al. (Reference Vergallito, Petilli and Marelli2020), for each word, we considered the following metrics:
-
– Perceptual modalities: The mean rating for each perceptual modality;
-
– Minimum perceptual strength: The lowest rating across all the five perceptual modalities;
-
– Maximum perceptual strength: The highest rating across all the five perceptual modalities;
-
– Mean perceptual strength: The average across all five perceptual modalities;
-
– Magnitude of perceptual strength: Euclidean vector length, corresponding to the distance of the vector length from the origin, including the values for all the five perceptual modalities without attenuation;
-
– Minkowski 3 distance: Similar to the Magnitude, reflects the perceptual strength in the five dimensions while attenuating the impact of the weaker ones;
-
– Modality exclusivity: Indicates the extent to which a word can be perceived through a single perceptual modality. It is calculated as the range of values divided by their sum, according to the formula [(max(x) − min(x))/sum(x)] × 100) where x is a vector of mean ratings for each of the five perceptual modalities. Its scores range from 0% to 100%, where a score of 0% indicates that a word is entirely multimodal (with equal scores across all modalities), while a score of 100% implies that a word is entirely unimodal (for the word scoring 0 on all modalities except for one).
Trial-by-trial data, mean ratings and composite metrics, as well as the full analysis code, are available at https://osf.io/8nyrc/?view_only=9aca1e7a1a604702afeafa4c91549397.
3. Analyses and Results
3.1 Ratings reliability
The first set of items was rated by 22 participants, while the second set was rated by 15 participants. We assessed the rating reliability across participants in the two sets. To do this, we relied on the Intraclass Correlation Coefficient (ICC), which determines if the items can be rated consistently across different raters. ICC was computed in R (R Core Team, 2022) using the irr package (Gamer et al., Reference Gamer, Lemon, Gamer, Robinson and Kendall’s2012). ICC was 0.913 (95% confidence interval: 0.899 < ICC < 0.926) for the first list and ICC = 0.931 (95% confidence interval: 0.918 < ICC < 0.942) for the second list. The two values indicate excellent reliability in both sets (Koo & Li, Reference Koo and Li2016). Crucially, these results show that notwithstanding some individual differences (intrinsic to a population that is generally not homogeneous) participants evaluated all stimuli in a consistent way.
Correlation of mean ratings for each modality on the probe items ranged from r = 0.71 (for auditory modality) to r = 0.89 (for visual modality), indicating that the two sets of raters evaluated items in a similar way.
Based on these results, ratings were aggregated and further analysed as a unique dataset.
3.2 Computing perceptual modality metrics
For each word, mean ratings were calculated separately for each modality (auditory, visual, haptic, olfactory, and gustatory). Thus, each word was represented as a five-value vector that reflects its perceptual strength for each modality. Table 3 reports the full distribution of values across the five modalities, in our sample and in the data collected by Vergallito et al. (Reference Vergallito, Petilli and Marelli2020) with hearing participants. Mean ratings were compared between groups through t-tests (also reported in Table 3). Perceptual strength mean ratings assigned to each of the five modalities by the two populations are instead shown in Figure 1A.
Table 3. Distributions of the ratings for each modality for the two groups of participants
Data from hearing participants have been computed from Vergallito et al. (Reference Vergallito, Petilli and Marelli2020) and refer only to the 200 words also included in our list.
Figure 1. Spider plots showing perceptual strength mean ratings for each modality in the group of deaf (red) and hearing (blue) participants (A). The same ratings are reported separately for the visual dominant words (B) and for the auditory dominant words (C). Perceptual modality dominance was attributed based on Vergallito et al. (Reference Vergallito, Petilli and Marelli2020).
Overall, words were rated by deaf individuals (red line in Figure 1A) as mostly experienced in the visual modality. The second strongest modality was haptic, while auditory modality elicited only slightly higher ratings than the two chemical senses (i.e., smell and taste; olfactory and gustatory modalities, respectively). In comparison with data collected with hearing participants (blue line), in the data collected with deaf participants, we observed a lower contribution of the auditory modality, which is, however, still significantly different from 0 (t(199) = 25.161; p = .0001). On the other hand, all the other modalities received higher ratings by the group of deaf participants with respect to hearing participants. In particular, we observed a much greater contribution of the chemical senses, that is, taste and smell, along with the haptic modality.
We also computed the mean ratings separately for the original visual-dominant and auditory-dominant words (based on Vergallito et al., Reference Vergallito, Petilli and Marelli2020). We observed that the perceptual profile of the 127 visual dominant words (Figure 1B) was different in the two groups. In particular, the auditory modality had a significantly weaker contribution in the group of deaf participants (t(172.5) = −9.84, p = .0001), but still remained significantly different from 0 (t(126) = 20.218, p = .0001). The haptic (t(229.4) = 2.085, p = 0.038), olfactory (t(250.6) = 5.284, p = .0001) and gustatory (t(247.5) = 7.109, p = .0001) modalities all had significantly higher mean ratings. Instead, the visual modality was similar in both groups (t(250.9) = −0.319, p = .75). Turning to the 73 auditory dominant words (Figure 1C), we observed a striking difference. Apart from auditory modality, all the other modalities were significantly more involved in the group of deaf participants compared to the group of hearing participants (visual: t(121.6) = 5.689, p = .0001; haptic: t(141.4) = 6.641, p = .0001; olfactory: t(119.1) = 5.8, p = .0001; gustatory: t(113.1) = 10.463, p = .0001). Conversely, the contribution of the auditory modality was strongly reduced (t(131.6) = −14.566, p = .0001), albeit remaining significantly different from 0 (t(72) = 18.731, p = .0001).
In sum, the reported involvement of the auditory modality, as expected, decreases for both visual dominant and auditory dominant words in the ratings provided by deaf participants. Yet, the contribution of this sensory modality does not disappear, indicating that, at some level, deaf participants conceptualise word meaning considering also the auditory modality. Furthermore, we can observe an increase in the haptic modality and in the chemical senses (taste and smell), while the visual modality remains unchanged in the visual dominant words and shows a slight increase in the auditory dominant words (the increase is statistically significant when considering the words altogether). It appears, therefore, that a decrease in auditory modality is accompanied by an increase in all other modalities.
For each word we also computed composite metrics of perceptual strength, as described in Section 2.4, with the aim to compare perceptual strength metrics in the two groups. As shown in Table 4, Modality Exclusivity, Minimum Perceptual Strength, and Mean Perceptual Strength were significantly different in the two groups, with higher mean values for deaf participants. Conversely, there were no significant differences between hearing and deaf participants in the other perceptual strength metrics, that is, Magnitude, Minkowski Distance, and Maximum Perceptual Strength.
Table 4. Distributions of perceptual strength metrics for the groups of deaf and hearing participants and comparisons between the two groups
Dataset of hearing participants was extracted from Vergallito et al. (Reference Vergallito, Petilli and Marelli2020).
These results indicate that there is no reduction in perceptual strength in the ratings provided by deaf participants; on the contrary Minimum and Mean Perceptual Strength present significantly higher values in deaf participants compared to hearing participants. The scores of Modality Exclusivity were significantly lower in the group of deaf participants, indicating that the words we used in this study have been considered in a more multimodal fashion compared to the hearing participants.
3.3 Relationships between modalities
The analyses described until now showed that, compared to hearing participants, perceptual strength is not reduced in deaf participants, but there are substantial differences in the distribution of perceptual modalities. To explore the relationships between modalities in the two groups of participants, we computed correlations separately for deaf and hearing participants. Correlations were computed with the rcorr and rcorr.adjust functions in, respectively, Hmisc (Harrell, Reference Harrell2019) and RcmdrMisc (Fox et al., Reference Fox, Muenchen and Putler2018) packages in R (R Core Team, 2022). Results (Holm corrected) are shown in Figure 2A and B.
Figure 2. Correlation matrices between perceptual-strength scores in the five modalities for the group of deaf participants (A) and for the group of hearing participants (B). Larger circles indicate stronger correlations, with red shades being negative correlations and blue shades being positive correlations. Only correlations with p-values lower than. 05 are shown.
In both groups, the strongest positive correlations were observed between the two chemical senses (olfactory and gustatory modalities), and between the haptic and visual modalities. Another strong positive correlation was observed between the haptic and the olfactory modalities, especially in the group of deaf participants. Comparatively, smaller positive correlations were also observed between haptic and gustatory, and visual and olfactory modalities in both groups. Overall, we could appreciate a similar pattern of correlations in the two groups, with the notable exception of the auditory modality. The auditory modality negatively correlates with the haptic and visual modalities in the hearing participants group, while no significant correlations could be reported for deaf participants. Moreover, the correlation between the haptic and the visual modalities appears to be reduced in the group of deaf participants, in comparison to the group of hearing participants (z = 3.08, p = .002), suggesting that while the significance pattern is similar, modalities may be clustered differently in the two groups.
To further explore how words cluster over different perceptual modalities, we ran two separate Principal Component Analysis (PCA), one for the group of hearing participants and one for the group of deaf participants. PCAs were run in R (R Core Team, 2022) using the psych package (Revelle & Revelle, Reference Revelle and Revelle2015). Ratings on the five modalities were reduced to two dimensions in both datasets, following the procedure adopted in previous studies (e.g., Vergallito et al., Reference Vergallito, Petilli and Marelli2020; Lynott & Connell, Reference Lynott and Connell2009, Reference Lynott and Connell2013). In the group of deaf participants, the first component accounted for about 46% of variance, and the second component accounted for about 22% of variance with a cumulative variance of about 68%. In the group of hearing participants, the first component accounted for 48% of variance, and the second component accounted for about 27% of variance with a cumulative explained variance of about 75%. Data are visualised in the two biplots in Figure 3. The pattern shown in the group of hearing participants (Figure 3B) resembles closely the one shown in previous studies (see, e.g., Vergallito et al., Reference Vergallito, Petilli and Marelli2020 for the Italian language). Figure 3B shows how words rated by hearing participants as being most experienced through the visual modality were also experienced through the haptic one (see the vicinity of the two red arrows labelled as ‘visual’ and ‘haptic’ in the left panel). The gustatory and olfactory modalities, instead, were slightly separated from the visual items and close to each other, while the auditory modality is completely separated from the other modalities. In contrast, Figure 3A shows how words appear to be less clustered in the group of deaf participants than in the group of hearing participants. While words rated as most perceived by vision are close to words rated as most perceived by touch, the proximity is less marked in the group of deaf participants in comparison to the group of hearing participants. It also illustrates how the items experienced through the haptic and olfactory modalities and through the olfactory and gustatory modalities are less separated from vision and touch in deaf participants. This holds also for the auditory modality, which is not as separate from the other modalities as it is in the ratings provided by hearing participants.
Figure 3. Principal Component Analysis (PCA) of perceptual mean ratings between modalities in the group of deaf participants (A) and in the group of hearing participants (B). The proximity of arrows indicates the degree of correlation: arrows close to each other indicate a high correlation and vice-versa.
Overall, the difference between the two PCAs shows that perceptual modalities are more interconnected (i.e., less clustered) in the group of deaf participants than in the group of hearing participants. This is consistent with the previous observations about ratings distribution and modality exclusivity indicating that deaf participants tended to rate each word through multiple modalities.
3.4 Dominant modality and modality shift
The previous analyses highlighted some crucial differences in the ratings of perceptual strength in the two groups. This left us to explore if (and how) words that are prevalently rated over one modality by hearing participants are predominantly rated over another one by deaf participants. With the aim to assess this modality shift, each word was assigned a first-dominant modality (i.e., the modality that received the highest mean rating), and a second- and third-dominant modality (i.e., the modality that received the second and third highest mean rating, respectively). Figure 4 illustrates the shifts for the two groups as a function of modality.
Figure 4. Bar plots illustrating the shift between modalities in the group of deaf participants and in the group of hearing participants.
Comparing the two groups, the first-dominant modality changed (recalling that the distribution of the first-dominant modality in hearing participants was decided by experimental design). Words originally rated by hearing participants as being mainly experienced through the visual (63.5%) or the auditory (36.5%) modality were rated by deaf participants as being mainly experienced through the visual modality (96.5%) or through one of the remaining four senses (3.5%). The first-dominant modality for deaf participants was visual for most words (N = 193 out of 200). The remaining seven words were rated as follows: two words as being mainly experienced through the haptic modality (i.e., ‘attrezzo’ ‘tool’ and ‘poltrona’ ‘armchair’); three words mainly through the gustatory modality (i.e., ‘stomaco’ ‘stomach’, ‘disgustato’ ‘disgusted’, and ‘uovo’ ‘egg’); one word mainly through the olfactory modality (i.e., the word ‘posacenere’, ‘ashtray’); and one word mainly experienced through the auditory modality (i.e., ‘fischio’ ‘whistle’).
As for the second-dominant modality, hearing participants did not show a prevalent secondary modality, since vision, hearing and touch could be considered to be equally alternative. By contrast, deaf participants showed a clear preference for the haptic modality, followed by the auditory modality, with some words also being associated with the three other senses. For deaf participants, 70.5% of the words (N = 141) were rated as being experienced with touch; 17% of the words (N = 34) experienced with hearing, 4% of the words (N = 8) experienced with smell, 5% of the words (N = 10) experienced with taste, and 3.5% of the words (N = 7) experienced with vision as second-dominant modality. Instead, for hearing participants, 36.5% of the words (N = 73) were rated as being experienced with vision; 35% of the words (N = 70) with hearing; 27.5% of the words (N = 55) with touch; 0.5% of the words (N = 1) with smell; and 0.5% of the words (N = 1) with taste as second-dominant modality.
Finally, for the third-dominant modality, the visual modality was entirely absent for both groups, since all words were associated with vision as the first or the second preferred modality. For hearing participants, 63% of words have touch as the modality that received the third highest mean rating (N = 126). The remaining words have smell (N = 30; 15%), taste (N = 10; 5%) or hearing (N = 34; 17%) as third modality. For deaf participants, the modalities were more equally distributed. Most of the words (N = 61; 30.5%) have the auditory modality as the one that received the third highest mean ratings, 56 words (28%) have smell, 44 words (22%) have touch and 39 words (19.5%) have taste as third modality.
3.5 Qualitative observations
Having established how modality shifts occurred between hearing and deaf participants, we focused on specific words to qualitatively express these changes in the distribution of ratings in our data. This analysis does not aim at being exhaustive, rather it is intended to provide specific examples that could help appreciating how perceptual modalities and perceptual strength seem to reorganise in the group of deaf participants. Thus, we will present only some of the most demonstrative instances from our dataset, while the rest of the plots can be accessed in the study’s OSF.
We started with those words that are predominantly auditory in the hearing dataset (Figure 5). For these words, a shift is particularly evident. For instance, the ratings of the word ‘ambulanza’ (‘ambulance’) in deaf participants show a marked decrease in the auditory modality with respect to those provided by hearing participants, while the visual and haptic modalities remained virtually unchanged. Interestingly, a slight increment in the olfactory and gustatory modalities was observed. Another relevant example of this phenomenon pertains to the word ‘arrabbiato’ (‘angry’) where, again, the auditory component is drastically reduced in deaf participants’ ratings, with an increase of values in both the visual and the haptic modalities, in comparison to the ratings provided by hearing participants. This pattern is even more evident in those words that are not only predominantly auditory (in the hearing group norms), but also conceptually associated with the domain of sound. The word ‘fischio’ (‘whistle’) is an auditory dominant word for both hearing and deaf participants, even if the auditory component is greatly reduced in deaf compared to hearing participants, in favour of the visual component. The same happens for the word ‘canzone’ (‘song’), which is predominantly auditory for hearing participants but it becomes visual (first modality) and haptic (secondary modality) in deaf participants.
Figure 5. Spider plots showing perceptual strength mean ratings for each modality in the group of deaf (red) and hearing (blue) participants for words with hearing as the dominant modality: ‘ambulanza’ (‘ambulance’), ‘arrabbiato’ (‘angry’), ‘fischio’ (‘whistle’), ‘canzone’ (‘song’), ‘urlo’ (‘scream’), ‘orchestra’ (‘orchestra’).
Something worth noticing is that both ‘fischio’ and ‘canzone’ present a high modality exclusivity for hearing participants. Instead, in deaf participants, their perceptual ratings are more distributed across other modalities. This aspect is common to most words evaluated by deaf participants (see Sections 3.2 and 3.3). Another example is ‘orchestra’ (‘orchestra’), for which a decrease in the auditory modality and an increase in the haptic component can be observed in ratings provided by deaf participants, although the visual modality remains unchanged, with respect to hearing participants. A similar pattern can also be found in other words which pertain to the sound domain and that were also rated as predominantly auditory by hearing participants (see also ‘urlo’, ‘scream’, below). It seems then that for this type of words, for deaf participants, a decrease in the auditory component is replaced by an increase in the haptic modality, and, in some cases, in the visual modality as well.
Another difference in the ratings seems to pertain to increased values in multiple modalities provided by deaf participants, which is particularly evident in abstract words. In Figure 6, we can observe the spiderplots of ‘idea’ (‘idea’), ‘desiderio’ (‘wish’), ‘mente’ (‘mind’), and ‘spirito’ (‘spirit’). In all these cases, we can observe relatively low scores for all modalities in hearing participants (blue) along with higher scores in all modalities for deaf participants (red). These qualitative observations seem, therefore, to suggest that abstract concepts may be more grounded for deaf individuals in comparison to hearing individuals.
Figure 6. Spider plots showing perceptual strength mean ratings for each modality in the group of deaf (red) and hearing (blue) participants for abstract words: ‘idea’ (‘idea’), ‘desiderio’ (‘wish’), ‘mente’ (‘mind’), ‘spirit’ (‘spirito’).
Multiple modalities are also more involved in concrete word representation (Figure 7). Interestingly, it appears that the modalities involved pertain to the experiential context. For example, the word ‘cucchiaio’ (‘spoon’) is largely visual and haptic for both groups, but in the ratings provided by deaf participants, increased values for the gustatory and olfactory modality can also be observed, hinting at the contexts of use of the spoons themselves. A similar pattern is evident for the words ‘bar’ (‘bar’) and ‘balcone’ (‘balcony’), where the actual experience with the objects appears to be translated into the ratings: thus, for deaf participants ‘bar’ loses the acoustic connotation and increases the gustatory and olfactory aspects, while ‘balcone’ increases the gustatory and olfactory aspects keeping everything else close to the hearing participants’ ratings.
Figure 7. Spider plots showing perceptual strength mean ratings for each modality in the group of deaf (red) and hearing (blue) participants for concrete words: ‘balcone’ (‘balcony’), ‘bar’ (‘bar’), ‘cucchiaio’ (‘spoon’).
Finally, increased values in multiple modalities suggest that words may be conceptualised in a more multimodal way. This is particularly evident in the examples in Figure 8. The words ‘buio’ (‘dark’), ‘soleggiato’ (‘sunny’), ‘rosso’ (‘red’) and ‘musica’ (‘music’) are all examples of words that are attributed predominantly to a single modality in the group of hearing participants. ‘Dark’, ‘sunny’ and ‘red’ are predominantly visual, with high scores in vision and low scores in all other modalities, while ‘music’ (but we have already encountered ‘whistle’ and ‘song’ above) is a predominantly acoustic word, with high scores in audition and low scores in all other modalities. However, when looking at the ratings provided by the group of deaf participants, the perceptual dominance of one modality recedes in favour of a more distributed involvement of multiple modalities.
Figure 8. Spider plots showing perceptual strength mean ratings for each modality in the group of deaf (red) and hearing (blue) participants for highly unimodal words: ‘buio’ (‘dark’), ‘soleggiato’ (‘sunny’), ‘musica’ (‘music’) and ‘rosso’ (‘red’).
This qualitative analysis had the aim of showcasing some of the most interesting patterns emerging in the data. In particular, we observed that the modality shifts from the auditory modality interested specifically the visual and haptic modality; moreover, we observed that words are generally rated through multiple modalities by deaf participants in comparison to hearing participants, and that this pattern, in some cases, leads to a wider distribution of ratings through the modalities at the expense of perceptual dominance of only one modality.
4. General discussion
In this study, our objective was to examine the impact of perceptual experience on conceptual representation. To achieve this, we focused on early deaf individuals who had little or no access to auditory information. In particular, we were interested in assessing whether hearing loss is associated with a reorganisation of the conceptualization of perceptual information related to words. To this aim we collected perceptual strength norms for 200 Italian words by asking deaf individuals to rate to what extent each word referent could be experienced through each one of the five perceptual modalities. Previous research has shown that perceptual strength norms are a reliable instrument to assess the perceptual grounding of words and to study conceptual representations through the generation and analyses of perceptual profiles of words (e.g., Lynott & Connell, Reference Lynott and Connell2009, Reference Lynott and Connell2013; Vergallito et al., Reference Vergallito, Petilli and Marelli2020). In the present study, we relied on these norms to investigate whether (i) auditory information is represented in the conceptual system of deaf individuals; (ii) if, given the hearing loss, the perceptual strength of words is reduced in deaf individuals in comparison to hearing individuals, and, (iii) the reduced information obtained through the hearing modality may have affected the organisation of conceptual representations involving other modalities.
Is the auditory modality still represented in the conceptualization of words in deaf individuals?
Our data showed that the contribution of the auditory modality is significantly reduced in deaf individuals in comparison to hearing individuals. This was evident when comparing the mean ratings for the auditory modality, but also when observing the perceptual profile of words that were auditory dominant for hearing individuals, or again, words related to sound. This result was quite expected since deaf individuals had limited or entirely lacked auditory experience of the concepts presented in the study. However, the auditory modality, albeit reduced, was still present in the perceptual ratings, suggesting that, at some level, deaf individuals could infer auditory-associated knowledge about the words they were evaluating. This result is in line with the body of literature showing that sensory impairments affect conceptualization only up to a certain point (e.g., Lenci et al., Reference Lenci, Baroni, Cazzolli and Marotta2013; Kim et al., Reference Kim, Elli and Bedny2019; Bedny et al., Reference Bedny, Koster-Hale, Elli, Yazzolino and Saxe2019; Petilli & Marelli, Reference Petilli and Marelli2024). Indeed, Lenci et al. (Reference Lenci, Baroni, Cazzolli and Marotta2013) showed that congenitally blind individuals were able to generate properties relative to visual verbs that were similar to those generated by sighted individuals. More recently, Bedny et al. (Reference Bedny, Koster-Hale, Elli, Yazzolino and Saxe2019), in a semantic similarity task, showed that similarity judgements on visual verbs made by blind or sighted individuals were similar. Finally, Petilli and Marelli (Reference Petilli and Marelli2024) showed that a measure of visual experientiability (based on the frequency of objects in an image database) could predict concreteness and imageability judgements in both blind and sighted individuals. Taken together, research on the blind population suggests that, at some level, information relative to the lacking modality could still be inferred.
How is it possible that individuals with a sensory impairment can infer information relative to the impaired sensory modality? Previous research discussed two main hypotheses on how this is possible. The first one posits that information associated with the lacking sense may be inferred through language use (e.g., Landau & Gleitman, Reference Landau and Gleitman1985; Gleitman, Reference Gleitman1990). In fact, if learning a language and the meaning of words requires individuals to form associations between a symbol and an object in the real world, Landau and Gleitman (Reference Landau and Gleitman1985, p. 2) make the point that ‘[…] the problem of language learning for any individual, however, circumstanced, is a problem of learning from partial information’. Indeed, young children manage to learn the meaning of words without being exposed to all instances that could be denoted by that particular word; moreover, they also learn the meaning of abstract words, that, by definition, lack a referent in the physical world. This is even truer for adult learners, who constantly learn new words without being confronted with their physical referent (e.g., Gleitman, Reference Gleitman1990). So, how is it possible to learn word meaning by bypassing the association between a word and its referent? A possibility is that perceptual knowledge is inferred via language use and linguistic associations. Landau and Gleitman (Reference Landau and Gleitman1985) proposed that blind children could use sentence contexts as frames to interpret the meaning of words related to vision. Starting on words whose meaning they already know and by exploiting the sentence context, blind individuals may easily infer also the meaning of words whose referent they cannot directly experience, as it happens for visual verbs (e.g., Bedny et al., Reference Bedny, Koster-Hale, Elli, Yazzolino and Saxe2019). In our specific case, it would mean that words like ‘orchestra’ or ‘ambulance’ were still evaluated along their auditory component because deaf participants knew that these words denoted objects that can be experienced through the auditory modality, and this knowledge has been established through linguistic experience, communication, and interaction with other individuals.
Another possibility (not necessarily antithetic to the previous one, but rather complementary) is that auditory information may be gained through related and intact perceptual modalities (e.g., Meteyard et al., Reference Meteyard, Cuadrado, Bahrami and Vigliocco2012). One suggestive piece of evidence in this direction in our data comes from the changes in the role played by the haptic modality. We observe that most words that were auditory dominant for hearing individuals were evaluated by deaf individuals with lower ratings for the auditory modality but higher ratings for the haptic modality. In addition, we found that the auditory and haptic modalities were significantly anti-correlated in the group of hearing participants, whereas such an effect disappeared in deaf participants. One way in which these results could be interpreted is by referring to the tight interconnection between these two sensory systems. From a physiological perspective, both auditory and haptic inputs are a form of mechanical energy, captured and transduced into neural impulses by mechano-receptors (inner hair cells in the cochlea vs. Meissner corpuscles and Pacinian corpuscles on the skin). There is also evidence for common genetic elements that contribute to touch and hearing (Frenzel et al., Reference Frenzel, Bohlender, Pinsker, Wohlleben, Tank, Lechner, Schiska, Jaijo, Rüschendorf, Saar, Jordan, Millán, Gross and Lewin2012). The brain areas traditionally associated with the processing of sounds (i.e., the auditory cortex) can respond also to tactile stimulation in humans (e.g., Foxe et al., Reference Foxe, Wylie, Martinez, Schroeder, Javitt, Guilfoyle, Ritter and Murray2002) and other animals (e.g., macaque monkeys: Kayser et al., Reference Kayser, Petkov, Augath and Logothetis2005). However, it is precisely in deaf individuals that this interaction becomes most pronounced, as evidenced primarily in brain imaging research that documented strong responses to haptic stimuli in the deafferented auditory cortex of congenitally deaf participants (Auer et al., Reference Auer, Bernstein, Sungkarat and Singh2007; Karns et al., Reference Karns, Dow and Neville2012; for review see: Villwock & Grin, Reference Villwock and Grin2022). In such a context, it may be speculated that for hearing individuals, it would be advantageous to keep audition and haptic inputs clearly distinct, given the great margin for ambiguity related to their mutual overlap from physiology to neural processing. By contrast, for deaf individuals, this overlap could make touch the ideal sensory system through which some information that is typically conveyed by hearing could be conveyed to the brain. If acoustic information assumes a predominantly haptic form in the perception of deaf individuals, this could explain why, at the conceptual level, we observed a shift in the sensory modality associated with words by deaf participants. In other words, what is a quite exclusively acoustic experience for a hearing person becomes a mainly haptic experience for a deaf person, and, in turn, predominantly auditory words become highly haptic words. We may, therefore, speculate that the residual auditory information we observed in our data, if not directly perceived, may have been inferred by haptic information or integrated with it.
Finally, it is important to consider the potential influence of residual hearing in our findings. If a deaf individual has or had some level of residual hearing, and if they could benefit from hearing aids amplifying certain frequencies, this could potentially reflect in the dimensional ratings provided in our task. In fact, the deaf person might consider it to be their personal auditory experience, and they might, to some extent, also integrate it with the information obtained through linguistic interaction or other sensory systems. Note, however, that this auditory information is partial and qualitatively different from what is perceived by hearing persons, and it is subject to significant interindividual variability.
Is the perceptual strength reduced, given the hearing loss?
Since composite measures of perceptual strength are based on ratings along all five modalities, we could have observed reduced scores of perceptual strength in the group of deaf participants, given the reduced importance of the auditory modality. However, this was not the case. Our results show that perceptual strength does not decrease in the group of deaf participants; rather perceptual strength was equal and, in some cases, even stronger than hearing participants. This pattern can be explained by looking at the distribution of ratings across modalities: notwithstanding lower ratings in the auditory modality with respect to hearing participants, deaf participants rated words with higher scores on all the other four modalities. Between-group comparisons relative to the composite measures of perceptual strength showed no significant difference between hearing and deaf participants for Magnitude, Minkowski Distance, and Maximum Perceptual Strength, while showing higher scores for Minimum Perceptual Strength and Mean Perceptual Strength in the group of deaf participants versus the group of hearing participants. Notably, scores of Modality Exclusivity are reduced in the group of deaf participants versus the group of hearing participants.
Taken together, these results suggest that, while the auditory modality has a limited role in the conceptualization of words, deaf individuals rely more heavily on information from other perceptual modalities to form concepts. In other words, the auditory modality appears to be ‘compensated’ by the remaining perceptual modalities. The outcome of this process is twofold: first, the perceptual strength of words does not change in the group of deaf participants versus the group of hearing participants, conversely it appears to be enhanced in many metrics; and, most interestingly, words appear to be conceptualised in a multimodal way. The latter outcome was particularly evident in the pattern emerging from the PCA, showing a less clustered distribution of ratings along the five modalities, and it was confirmed by the lower scores of Modality Exclusivity for the deaf participants in comparison to the hearing participants.
Are words represented differently by deaf participants versus hearing participants?
The patterns observed thus far point to the fact that words were rated in a qualitatively different way by deaf and hearing participants. Comparing the perceptual profile of words and assessing the modality shift, we were able to observe how each perceptual modality contributed to the conceptualization of word meanings in a different measure in deaf versus hearing participants. We have already discussed the lower contribution of the auditory modality and the enhanced contribution of the haptic modality, especially in words that were rated as auditory dominant by the group of hearing participants. While this was perhaps the most striking and significant case of modality shift, we were also able to individuate a few more examples that could provide insights on how the conceptual system may reorganise in deaf individuals.
A particularly interesting case is that of abstract concepts. In our qualitative analyses, we observed that abstract words seemed to have higher perceptual strength in the group of deaf participants in comparison to the group of hearing participants. This pattern is quite surprising, as, based on a previous work that investigated explicit conceptual representations in cases of perceptual impairments, albeit in the blind population (Lenci et al., Reference Lenci, Baroni, Cazzolli and Marotta2013), no difference was expected at the conceptual level for abstract words. In fact, both groups (sighted and blind) participating in Lenci et al.’s study produced less spatial and perceptual features and more features associated with events, objects, and evaluative features. This was expected as in the absence of a referent in the physical world, the experience with abstract concepts stems from events and language (e.g., Barsalou & Wiemer-Hastings, Reference Barsalou, Wiemer-Hastings, Pecher and Zwaan2005; Kousta et al., Reference Kousta, Vigliocco, Vinson, Andrews and Del Campo2011). This also emerges in studies requiring individuals to evaluate the perceptual strength of abstract words (e.g., in Vergallito et al., Reference Vergallito, Petilli and Marelli2020, the correlation between concretenessFootnote 4 and Maximum perceptual strength ranges around r = .71, suggesting that words that are rated as more concrete are also attributed higher degrees of perceptual strength, and vice versa). In other words, it appears, from previous research, that sighted and blind participants tend to attribute less perceptual features to abstract concepts. However, data from our own study suggest that abstract words maintain high degrees of perceptual strength for deaf participants. Correlational analyses confirm our qualitative observation, indicating that Maximum Perceptual Strength is highly correlated with word concreteness (r = .74) for the group of hearing participants, while the correlation is significantly weaker for the group of deaf participants (r = .44; z = 4.74, p = .001). Indeed, when selecting from our dataset only words with low concreteness ratings (using the median of the distribution as a cutoff), we could appreciate a significant increase in the perceptual strength for deaf (M = 3.73, SE = .068) versus hearing (M = 3.37, SE = .074) participants (t(188.5) = −3.56, p = .0005).
Why this pattern? A possible explanation may be found in the linguistic experience characterising our two groups, in particular, the fact that the deaf participants in our study were not only fluent in Italian but were also fluent in Italian Sign Language (LIS). As a sign language, LIS is grounded in sensorimotor experiences, and previous research has shown that abstract concepts present different levels of embodiment in LIS (e.g., Borghi et al., Reference Borghi, Capirci, Gianfreda and Volterra2014). It is thus possible that conceptual representation of abstract words in LIS has influenced the evaluation of words in Italian. Of course, this is only a speculative interpretation and should merit further systematic investigation in future research. Another explanation, however, pertains to the general tendency of deaf participants to evaluate words by relying on multiple perceptual modalities. In fact, data from the qualitative analyses, highlight an increased involvement of the chemical senses and the haptic modality in both abstract and concrete words’ perceptual profiles. Interestingly, chemical senses seemed to be enhanced in words like ‘bar’ or ‘party’ or ‘spoon’, that is, words implying contexts or objects where the chemical senses are experientially involved. This pattern directly recalls the study by Lenci et al. (Reference Lenci, Baroni, Cazzolli and Marotta2013), where blind participants produced a larger number of situational features (like events or properties connected to events and experiences) in relation to concrete words. It may thus be that, similarly to blind individuals, also deaf individuals incorporated information related to the situation in which the object was experienced into their representation. Differently from Lenci et al. (Reference Lenci, Baroni, Cazzolli and Marotta2013), however, it seems that this trend applies to both abstract and concrete words when considering deaf individuals: deaf people seem to evaluate both types of words in a richly grounded manner on the basis of situational or contextual experience.
Going back to the original question, we propose then that the conceptual representation of perceptual features in deaf individuals is qualitatively and quantitatively different from that of hearing individuals in terms of the involved modalities and of the relationship between modalities.
5. Conclusions, limitations and future directions
In the present study, we collected perceptual strength norms for 200 Italian words based on ratings provided by deaf individuals and compared them to identical norms provided by hearing individuals in a different study (Vergallito et al., Reference Vergallito, Petilli and Marelli2020).
As we expected, the contribution of the auditory modality was significantly reduced (albeit not absent) in the norms provided by deaf individuals, leaving room for an enhanced contribution of the other modalities, in particular the haptic, gustatory and olfactory, leading to a multimodal representation of concepts. The pattern we found suggests that when the auditory component was tuned out, information coming from the other senses could acquire a more dominant role.
Interestingly, the auditory modality was still represented in our ratings, allowing us to ponder what perceptual norms are truly capturing. There is an ongoing debate in the field juxtaposing accounts that propose that perceptual norms capture the direct experience of individuals with the words referents (e.g., Lynott et al., Reference Lynott, Connell, Brysbaert, Brand and Carney2020; Connell & Lynott, Reference Connell and Lynott2012, Reference Connell and Lynott2016) and those who propose that perceptual norms (as all ratings more in general) rather reflect the conceptualization of that experience which is fed both by the experience itself but also by richer information (e.g., linguistic, affective, etc.; e.g., Petilli & Marelli, Reference Petilli and Marelli2024; Westbury, Reference Westbury2014, Reference Westbury2016). In the present study, deaf participants who had no or very limited access to acoustic experience, still evaluated words along the auditory modality, suggesting that perceptual norms might indeed capture elements of direct experience, albeit processed and enriched with information accrued from other channels.
This was, to the best of our knowledge, the first attempt to collect perceptual norms on perceptually impaired individuals and it was the first foray into assessing perceptual-related conceptual representation in deaf individuals. The group of deaf participants who rated our stimuli presented high levels of agreement in the task, indicating that, notwithstanding some individual differences, their judgement about each word was overall similar. However, we believe that some individual differences among participants’ characteristics (e.g., different levels of hearing loss, use of hearing aids or knowledge and use of Italian Sign Language, reading habits, etc.) might, in principle, influence the conceptualization of words, and thus merit further investigation. In the current study, it was not possible for us to address the impact of these variables, mainly due to the fact that they were not sufficiently distributed (e.g., most of our participants presented profound or severe deafness and only a few presented moderate deafness to one ear; again, most of our participants declared to use a sign language very frequently, while only a few declared to never use it) and individual characteristics were not properly isolated (note however that it may prove to be extremely difficult to find completely homogeneous groups of participants where all possible confounds are controlled for). In order to systematically assess the impact of these variables, it is necessary to design new studies where each characteristic is stratified in large enough groups in order to guarantee a proper distribution and control. That said, there are many research questions that we hope could and will be explored in future research. One might want to explore the role of deafness onset, as we may speculate that late deaf individuals would behave similarly to hearing individuals since their perceptual experience growing up was comparable to that of hearing individuals. However, it may also be the case that prolonged hearing loss might bring along the emergence of a multisensorial dimension; hence, late deaf individuals might show higher ratings in the auditory dimension (similar to hearing individuals) along with higher ratings in other dimensions (similar to early deaf individuals). The level of hearing loss might be another variable worth exploring as the perceptual experience of profound or severe deaf individuals is significantly different from that of mild deaf individuals. Related to this, the role of hearing aids (HAs) might also be relevant for perceptual judgements, though it has been shown that even state-of-the-art HAs, when used by persons with degrees of hearing loss that make the perception of certain frequencies impossible, lead to a partial or distorted representation of sounds, and yield little benefit in restoring the acoustic experience in every-day noisy environment (Lesica, Reference Lesica2018). In light of these considerations, we should note that perceptual judgments are likely to remain qualitatively different from those of hearing individuals (as shown in the present work). We hypothesise that the observed difference may stem from the inherent dissimilarity between the acoustic experiences provided by HAs and the natural hearing, in particular, in terms of consistency of the auditory information obtained through HAs when compared to the one obtained through other sensory modalities.
Finally, we believe that aside from audiological variables, also linguistic ones should be explored, such as language experience with the vocal language or a sign language. Our study suggests that sign languages could and should represent a relevant venue of exploration. In our qualitative analyses, it emerged that the use of a sign language might have implications for the representation of abstract words: we speculated that abstract words appeared more grounded in deaf versus hearing participants due to the influence of a sign language, therefore, it would be interesting to investigate whether deaf individuals who do not use a sign language would still evaluate abstract words as more grounded. Similarly, it would be interesting to investigate how perceptual norms would change if words were presented through the signed modality instead of in the written one. Indeed, based on existing literature (e.g., Borghi et al., Reference Borghi, Capirci, Gianfreda and Volterra2014), we may speculate that presenting stimuli through signs might lead to perceptual strength ratings different from those obtained when presenting them through written words. It is possible that using a sign language in presenting concepts might activate grounded representations to a higher degree. On the other hand, there is evidence that in bilingual individuals who know and use one signed and one vocal language, both languages are co-activated during language processing even when the sign language is the non-target language (e.g., Meade et al., Reference Meade, Midgley, Sehyr, Holcomb and Emmorey2017). In addition, Mott et al. (Reference Mott, Midgley, Holcomb and Emmorey2020) reported that deaf native or early-exposed American Sign Language (ASL) signers who were also fluent in written English were faster in responding to either iconic and non-iconic ASL signs when the signs were preceded by their English translations rather than when they were preceded by unrelated English words. In accordance with Bosworth and Emmorey (Reference Bosworth and Emmorey2010), the authors concluded that iconicity does not modulate semantic priming in deaf signers, acknowledging that factors other than imageability or concreteness might have affected the processing of signs on a semantic level (Mott et al., Reference Mott, Midgley, Holcomb and Emmorey2020).
Of course, these are only some ideas to showcase how wide the venue of investigation for this topic can be. For those interested in pursuing further investigations into this issue, we released the full dataset, including perceptual norms and composite metrics at https://osf.io/8nyrc/?view_only=9aca1e7a1a604702afeafa4c91549397.
Data availability statement
All data supporting the findings reported in this paper can be accessed at https://osf.io/8nyrc/?view_only=9aca1e7a1a604702afeafa4c91549397.
Funding statement
F.P., P.R. and G.G. were supported by a grant from the Italian Ministry for Research and University (MUR, PRIN 20177894ZH). S.A. and F.P were supported by the Fondazione Cassa di Risparmio di Trento e Rovereto.
Competing interest
The authors have no competing interest to declare.
Open access
