Approaches

Participants

First case study

Following the IRB approval for nonmedical human subject studies, the first case study recruited 20 participants from Bay Area local artists and authorized in-person researchers from Stanford University’s CCRMA. The participants were invited via email and were provided with the information sheet and oral consent form when they arrived. Participants were selected primarily from artistic practices of music, movement, or both. Although no music or movement background was required, all participants reported that they had multidisciplinary artistic backgrounds or practices.

Table 1 presents the demographic information (age group [Age], music [Music] and movement [Movement] experiences in years and their dominant practices [Dominant]) of the participants in the first case study. These experiences are based on their reported experiences and range from amateur self-practice to professional levels. The study asked participants to perform the same tasks regardless of their prior experience.

Table 1. Participant demographics of the first case study

Second case study

The second study was conducted with eight participants as audience members over three performance sessions in a special immersive performance space, Listening Room, at Stanford University’s CCRMA. Participants were invited via email and they provided their oral consent before proceeding. Participants had considerable music training with an average of 18+ years and familiarity with new musical interface performance. They had no reported hearing impairments. Seven participants communicated with spoken English and had no background in signing English and only one communicated primarily with both speaking and signing.

Table 2 presents participants’ age group (Age), music experience (Music), how often they move to music (Movement), hearing impairments (Hearing), and experience with performance using American Sign Language (ASL). None of the participants experienced hearing impairments and some had experience with performing arts that uses ASL either as artistic material or for increased inclusion of presenting the performance to the deaf communities. All participants had prior experience in music. Their music experience in years is indicated in parenthesis.

Table 2. Participant demographics for the second case study

Third case study

In the third case study, the first workshop started with three participants, two of whom joined in person (P1 and P3) and one virtually (P2). The workshop series was conducted with the support of ShareMusic & Performing Arts CenterFootnote ² in Malmö, Sweden. This study focused on codesigning haptic interfaces and music-movement performance with one participant (P3) who is profoundly deaf and has a dance background. The music and movement composition was supported by another participant (P2) who has a background in physical theater and no reported hearing disabilities. P1 decided against continuing the workshop series after the first session. The rest of the workshop and performance series as well as the evaluation of the study was continued by actively involving P3 in the design process. The Deaf participant (P3) communicated with Swedish Sign Language and equally associated with both hearing and Deaf/Hard of Hearing communities. The sessions always included two Swedish Sign Language interpreters and P3’s assistant whenever P3 was present.

Table 3 presents the participants’ age group (Age), hearing abilities or impairments (Hearing), cultural associations with hearing and/or Deaf/Hard of Hearing (D/HoH) communities (Cultural Association), and communication languages (Language). One hearing participant (P1) communicated with both signed and spoken languages, one hearing participant (P2) communicated with only spoken languages, and one deaf participant (P3) only communicated with signed language. Both P1 and P2 reported their interest in the project because they had relatives with hearing impairments. Majority of participants equally associated with both hearing and D/HoH communities.

Table 3. Participant demographics for the third case study

Experiment designs and setup

Interface design

The Bodyharp consists of an instrument body and wearable parts, including an attachment to the performer’s arm and a hand controller. Figure 1 presents the Bodyharp’s most recent 3D printed interface components, showing the wearable hand controller enclosure with tactile sensors, wearable arm-string attachment piece, and the instrument body enclosing the string-pulley system. The connection between the instrument and the wearable parts completes the interface by integrating the performer’s body, thus the instrument cannot be considered without its performer (Cavdir, Reference Cavdir2021). This hybrid system, combining WT with the human body, offers new embodied ways of designing musical instruments and considers the instrument and the body as extensions of each other. (Nijs et al., Reference Nijs, Lesaffre and Leman2009; Mainsbridge, Reference Mainsbridge2016; Cavdir, Reference Cavdir2021).

Figure 1. Bodyharp consists of two wearable interfaces (attachment to the performer’s arm and a hand controller) and a main instrument body.

Our approach to Bodyharp’s design incorporates embodied musical expression (Leman, Reference Leman2008) and movement-based interaction (Loke and Robertson, Reference Loke and Robertson2013). It employs the design consideration that drives from simultaneously capturing nuanced musical gestures and large-scale body movements. We approach this concept by coupling the performer’s gestures with a wearable interface at two levels of body movements. Larger scale movements contribute to kinesthetic and visual aspects of the performance. They exude a dance-like quality that invites embodied skills and somatic expressions to be transferred into music performance. Smaller-scale gestures offer a nuanced control over musical events that are captured by more tactile sensors. Their interaction focuses on finger or hand gestures in a smaller periphery.

Same interaction principles are followed in two iterations of the Bodyharp’s interface. The performer starts interacting with the instrument by plucking or stretching the strings and continues by controlling the parameters with finger gestures. The sound excitation starts with playing individual strings and is followed by adding sound effects by larger arm movements. In the first iteration, the exoskeleton detects smaller-scale gestures and in the second iteration, the hand controller allows the performer to control the parameters of these sound effects. Similarly, larger body movements, either captured by the strings or by the accelerometer, change these parameters while simultaneously extending the musician’s performance space. These movements provide more freedom in space and expression, indirectly controlling but influencing the music.

Design approach: Body as an extension of the instrument

Bodyharp’s both prototypes focus on extending the musical interface with the performer’s body, directly incorporating their arms, hands, and torso as integral parts of the interface. In this design, the performer’s body acts as an extension of the musical interface and correspondingly the musical interface extends the musician’s body (Cavdir et al., Reference Cavdir, Michon and Wang2018; Cavdir and Wang, Reference Cavdir and Wang2020). The first design of the instrument includes a wearable arm piece and an exoskeleton (see Figure 2). The arm piece encapsulates the controller system and the accelerometer to detect the performer’s arm movements and to map them to sound parameters. The exoskeleton worn inside of the hand extends from the arm and detects finger movement with a series of flex sensors. The data from the flex sensors are similarly mapped to more nuanced controlled sonic events. This initial interface also includes the main instrument body that holds the string-pulley system, allowing the strings to extend from the instrument body and connect to the performer’s body.

Figure 2. Bodyharp’s first prototype included an exoskeleton and an instrument body. The exoskeleton in the second iteration is replaced by the arm-string attachment and hand controller.

The second design iteration addresses the design considerations of Bodyharp’s control mechanism and the technical challenges of the first prototype. In the first prototype, five flex sensors failed to provide a wide range of control for nuanced gestural performance and sound control. They also lacked passive force-haptic feedback since they only rely on the bend of fingers in a limited range. These sensors additionally offered less reliable data and more variance between mappings in each performance as their deflection profile and consistency change over their life cycle.

To address these challenges, we replaced the exoskeleton with a hand controller. The finger-bend interaction is replaced with buttons on top of the controller. Two force sensors are added: the first is accessible with the thumb on the right hand wearing the instrument, and the second is accessible with either the freehand, other body parts (chest, legs, arms, etc.), or the environment. We observed performers engaging with this second sensor using their chest, legs, arms, or heads. This additional sensor, facing outwards from the performer’s body, improved the performer’s interaction with their bodies and their environments. Similarly, the accelerometer, placed inside the hand controller, is relocated from the forearm to the palm. Changing the accelerometer’s location created new possibilities for hand gestures. In its earlier position (on the forearm), the sensor was able to capture the orientation and movements of the arm in a limited range. In its final position (on the palm), the controller can still detect the arm orientation and movements, but it can also extend these affordances with shaking and waving hand gestures. By capturing the performer’s hand gestures, our final design extended Bodyharp’s movement vocabulary.

Results: Movement supports sonic interaction

In these two studies with Bodyharp (Cavdir et al., Reference Cavdir, Michon and Wang2018; Cavdir, Reference Cavdir2021), we explored how body movement contributes to the sonic interaction when it is used as a sound-facilitating and sound-modifying gestures (Godøy and Leman, Reference Godøy and Leman2010). Similar to the first prototype, the second iteration maintains the interaction mechanism of dividing sound excitation and nuanced control of sound effects and parameters. In this second prototype, the sound is created by the performer’s interactions with the strings (plucking, stretching, or moving the attached arm). The sound effects are solely controlled with hand or finger gestures on the hand controller.

In a user study with the second iteration (Cavdir, Reference Cavdir2021), participants commented on these nuanced, small-scale gestural interactions to control the sound effects and parameters. Most participants stated that the interaction with the strings and the touch sensors was the most effective part of the gestural interaction. They reported that the touch (FSR) sensors provided a “nuanced dynamic control.” Their responses showed that force-sensitive control created a tactile interaction during which most participants engaged with both gestural interaction and sound creation.

In addition to the hand controller, interaction with the strings was reported to be intuitive, engaging, and effective. Table 4 presents their self-reported experiences as participants responded to touch sensing and string interaction. Strings enabled participants to interact with both small-scaled gestures, such as plucking the strings, and with body movements, such as dynamically controlling their kinesphere and instrument-body shape. One participant emphasized the effectiveness of the dual gestural interaction by highlighting this combination of interaction with (a) musical gestures that directly influenced the sounds and (b) body movements that allowed investigation by being more elusive. Similarly, another participant commented on two levels of interaction: “I could alternate between using the whole-body, big, and emotionally charged movements and smaller, more delicate gestures.”

Table 4. Participants’ self-reported experiences with the tactile interaction of the wearable interface are presented

Note. Participants referred to the touch sensing as FSR, pressure, and touch sensors, they commented on the buttons and sliders, grouped under tactile sensing, and they described their experiences with the string interaction using plucking, stretching, string keywords.

All participants explored their space use and range of movement. Regardless of participants’ backgrounds, they all reported their relationship with Bodyharp’s space and shape affordances:

• • “Not every instrument allows one to stretch the limbs. This one creates an ambiance that allows the body to be playful.” (space and shape)

• • “…I can frame my body around it.” (shape)

• • “It was a joy to explore the sonic space while moving my body in the physical space.” (space)

• • “…Bodyharp similarly has a strong physical component to it due to the space its strings transverse. Quite graceful and also powerful.” (space)

Extending the performer’s use of space and movement vocabulary is crucial to offer an embodied music interaction in the Bodyharp’s design which necessitates a combination of wearable and free interaction with body movement. The instrument is not limited to a wearable interface that the performer acts on but merges both the interface and the body, both of which simultaneously react to each other. The Bodyharp’s wearable interface bridges the tactility of instrumental gestures and embodiment of the full-body movement interaction. This combination provides tactile interaction through touch on strings and sensors, creating opportunities for nuanced control over the sound that resembles musical gestural interaction. The bodily lived experience is built by physically incorporating the body (the performer’s arm) into the musical interface to form a single instrument as a whole through the performer’s movement, force interaction, and resistance from strings and touch.

Synthesis: Shared perspectives

Bodyharp encourages performers to leverage their artistic practice while developing awareness in either music or movement. In our user study, we worked with participants with various artistic practices, ranging from music performance, composition, and design to dance, choreography, performance arts, and poetry. Our goal was to investigate these artists’ varying perspectives and analyze differences and commonalities in how they interact with movement-based musical instruments. As a research outcome, the study led users to construct creative artifacts. As we explored both movement and music interaction, participants were asked to create a music composition and a choreography based on their own musical statements.

The results of our user study revealed that Bodyharp’s movement-based interaction provides a shared perspective to participants with musical and movement backgrounds. Based on the participants’ demographics and their common interaction patterns, we analyzed their response to Bodyharp from two perspectives: musicians and movers. Because musicians are traditionally trained to focus on the dexterity of instrumental gestures and nuanced control over the sound, when playing Bodyharp, they focused more on expressing articulate musical ideas and performing with musical gestures than they were with nonmusical body movements. However, many participants with music backgrounds reported that “the Bodyharp helped them to feel less self-conscious about their expressive body movements when they performed with the instrument compared to the free movement practice.” Similarly, participants with movement backgrounds translated their experience in dance and choreography into music-making despite little or no experience in music performance or composition. They reported that “they utilized their knowledge of choreography techniques to create a sound composition that is choreographed through movement interaction.”

Despite different backgrounds and expectations, musicians and movers reported shared experiences after performing with the Bodyharp. The instrument provided an introduction to each other’s practices. While Bodyharp offered a more embodied practice for the musician, it provided a more accessible musical interface for the mover, who already practiced a prior movement expression. Through qualitative questionnaires and interviews, we collected some of the shared experiences. Both musicians and movers shared that (a) Bodyharp was intuitive and encouraged them to explore new movement possibilities, (b) learning Bodyharp’s musical features based on its gestural affordance was effective, (c) Bodyharp enabled creating music through nuanced gestural interaction and body movements, and (d) physical constraints created new artistic possibilities.

Lastly, we asked participants how they imagined Bodyharp to be used in performances. Many of them reported that it can be played as a collaborative or duo performance. The performance settings included collaborative dance and theater pieces, duo performances with dancers, musicians mimicking dancers in interdisciplinary settings, and accompanying other musicians in ensembles. One participant with a dance and education background suggested using Bodyharp as a movement-based art piece supports “the creative process in returning to a place of not knowing” in dance teaching. Some of the feedback from participants were employed in real-life performance situations. Figure 3 shows three case studies of performance with Bodyharp, following the results of the user study: Bodyharp was played in (a) a duo performance with a flutist, (b) a quartet with three dancers where dancers used Bodyharp’s movement patterns as cues for choreographic events, and (c) a duo performance with a dancer, interacting together with the string and touch sensors.

Figure 3. Bodyharp’s performance practices were developed based on the participant’s feedback and reflections, showing (a) a duo performance with a flutist, (b) a quartet with three dancers, and (c) an interactive duo performance with a dancer.

Interface design

Felt Sound is a musical interface that incorporates ASL gestures into music performance. The Felt Sound’s performance is designed to offer an embodied, felt listening experience for Deaf and Hard of Hearing (D/HoH) audiences (Cavdir and Wang, Reference Cavdir and Wang2020) by combining low frequency and high amplitude in-air vibrations with physical gestures inspired by ASL. This instrument incorporates nonmusical communicative gestures (Godøy and Leman, Reference Godøy and Leman2010) to musical features that are composed to create physical sensations. The physicality from both the vibrotactile sensations as well as the gestural performance offers a shared, bodily, and felt listening experience for both D/HoH and hearing listeners.

In designing this accessible digital musical instrument (ADMI), we addressed two design considerations: (a) employing sound waves played through a subwoofer speaker array as a tactile feedback modality and (b) embodying ASL-inspired gestures for wearable interface design and musical expression. The overarching design objective draws from creating an inclusive music performance that offers an embodied listening modality for D/HoH and hearing listeners and provides a music composition that brings both audience groups’ experiences closer. This shared listening experience affords interaction with the subwoofer speakers, allowing the listeners to sit closer to the speakers. The audience members are encouraged to touch the speakers and sense the musical changes through their bodies when they simultaneously receive visual feedback of the gestural composition.

The musical interface maps the musical events to the gestural vocabulary of ASL-inspired finger, hand, and arm movements and nuanced gestural control. The gestural vocabulary consists of five ASL signs, performed in semi-structured improvisations (Table 5). The gestural compositions leverage ASL gestures’ communicative and expressive functions to provide a musical context to the D/HoH audience. Our motivation behind these compositions incorporates an unconventional gestural vocabulary outside music into the DMI design. Such design necessitates a wearable interface centered around the specific gestural vocabulary.

Table 5. Felt sound’s American sign language-inspired gesture-to-sound mapping

Design approach: Modular design

The wearable interface is modularly designed to allow designers and performers to customize the gestural interaction. The separate modules capture varying levels of gestures: nuanced finger gestures, single-hand gestures, and small arm gestures from both hands interacting with each other. The interface includes fingertip modules, passive elements like magnets for magnetic sensing, an accelerometer, and a controller module. These modules are prototyped using additive manufacturing where some parts such as magnets or accelerometers are embedded into the modules during the 3D printing and can be combined as desired on the left and right hands, wrists, and fingers (Cavdir, Reference Cavdir2020). The fingertip modules detect finger interaction with a hall effect sensor triggered by a wearable magnet and force-sensitive resistors (FSR) for more continuous control. While the FSR and hall effect sensors are fixed on the 3D printed fingertip structure, the wearable magnet can be placed in multiple places, such as on the palm, on the back of the palm, or on the wrist, depending on the desired gestures.

Since the detection mechanism is limited to available sensors, the modular design creates flexibility to customize gesture-to-sound mapping. For example, single finger interaction can be extended to multiple fingers to create fist opening and closing gestures (see Figure 4b,c). Similarly, more dynamic movements, captured by the accelerometer module, can be worn in multiple locations to detect different ASL signs (shown in Figure 4d). The modular design captures the “show” and “empty” signs with the same group of sensors at different locations in the hands.

Figure 4. Felt Sound’s first prototype: (a) All modules are worn on one hand, (b) Fingertip sensor and a magnet module interaction change the state of the hall effect sensor while the FSR sensors output continuous data to control frequency, gain, and filter and effect parameters, (c) The hall effect sensors and the magnet placed on the palm allow detecting first closing and opening gestures, and (d) Accelerometer module embedded into the 3D printed closure can be placed in multiple locations on the hand or wrist and coupled with other modules.

Although constructing the interface with only a set of sensors limits the gestural vocabulary, we chose a wearable design approach over detecting in-air gestures (Godøy et al., Reference Godøy, Haga and Jensenius2005) because the wearable interface offers tangible interaction. Felt Sound’s touch and force-sensitive interaction enabled users to embody the felt experience of performance beyond localized vibrotactile feedback. In addition to amplifying the physicality of the performance (see Figure 5), it enhanced awareness and sense of the performer’s own body by drawing the performer’s attention to the interaction of fingers, the relationship between the hands, and their movement in the space. This embodiment was visually and kinesthetically communicated with the audience through the performer’s presence on stage, supporting the vibrotactile whole-body listening experience that is equally shared between the performer and the audience.

Figure 5. Felt Sound’s performance with sign language-inspired gestures, showing “music,” “poetry,” and “empty” gestures respectively.

Results and synthesis: Shared listening experiences

The bodily listening experience significantly influences the perception of music and enjoyment of music performance. This kind of listening is provided to the audience through in-air sound vibrations that are perceived both on the surface of the body (Merchel and Altinsoy, Reference Merchel and Altinsoy2014) and inside the body (Holmes, Reference Holmes2017). While embodied listening and felt musical experiences support hearing audiences’ music performance appreciation, they play a significant role in D/HoH individuals’ understanding of music performance.

In designing Felt Sound, we addressed diverse hearing abilities in music performance. Our motivation behind Felt Sound’s performances is to create performance contexts that not only offer shared experiences for audiences with different hearing abilities but also invite each group to experience the music from the other’s standpoint. In addition to the composition providing similar bodily listening and felt sensations being shared among all listeners, the sign-language-inspired movement-based interaction offers a musical context to the signers. In this composition, ASL signers can relate to the musical context better while non-ASL signers can understand D/HoH listeners’ experience with music. Such shared listening experience is not limited to providing another modality for sensory deprivation but extends to developing a deep listening practice for all. One objective is to include the body in listening and develop an awareness of how the body plays a role in perceiving music that is traditionally dominated by auditory stimuli. This subtle perception relies on paying attention to the felt experience of how different pitches and sounds resonate in the body—the chest, stomach, and fingertips. The felt experiences are emphasized by the visual cues that appear when the performer creates music with movement-based instruments through nonmusical gestures. Another main objective is highlighting the moments of silence in music. Because music enhances the listening experience but at the same time, it can be overwhelming for the listeners to receive constant haptic stimuli on the body, balancing the experience with silent moments and pure tactile sensations becomes an important design consideration. One of our listeners commented that this aspect influences her bodily sensations and perception of movement qualities: “The felt sound highlighted moments of silence for me more so than traditional sound. I felt light and free in those moments and active in the moments of more intense vibration.”

We also discussed the deaf participants’ musical experience and preferences in semi-structured interviews. One deaf participant with profound hearing loss defined her musical experience as isolating since she needs to obtain additional information about music through nonauditory modalities. She explained that she understands the musical content, mostly the emotional content of music, through special lighting that is mapped to musical features. She positively responded to incorporating sign language gestures into music to provide context and meaning to music that deaf listeners frequently miss. She also expressed that she “felt the sound inside” of her body in the presence of strong vibrations. Another deaf participant reported that the composition felt like “the whole room was moving.”

All listeners reported some physical sensations through high amplitude, low-frequency sounds. One participant commented on the bodily listening experience: “I felt like I was using my torso to listen to music rather than my ears. The vibration seemed to be felt in and out of the torso.” Similarly, another participant reported that they “felt like the sounds are not perceived through pin-pointed sources, but rather through the entire body.” One participant with a singing background described her listening with movement qualities. She reported that “the felt sound highlighted moments of silence for me more so than traditional sound. I felt light and free in those moments and active in the moments of more intense vibration.” Some participants shared their experience with the gestural performance as they said “The premise of the piece felt like a physical expression of music through low-frequency sounds. Combining it with gestural elements created a powerful body-to-body connection.”

The audience members commented on the relationship between kinesthetic (movement and haptic) sensation and audio-visual feedback received from Felt Sound. They further expressed how the interface and the performance affected the communication between the performer and the audience:

• • “The sounds definitely embraced the bodies within the audience. … the connection was more one-to-one between the performer and myself.”

• • “I felt like physical and auditory movement were definitely related and emerging from the glove as the main controller. Responsive!”

This performance-oriented user study provided promising results and future directions. Building on our research with Bodyharp and Felt Sound and their performance practices, we created collaborative, shared performance practices that actively involve the participants in the design process. Bodyharp’s exploration on discovering shared perspectives of music and movement practitioners significantly contributed to the design of Felt Sound. Similarly, creating a shared listening experience that highlights the experience of the lived body and the expressivity of body movement contributed to improved inclusion and collaboration. The next section describes the final step in creating a shared performance space for a hearing musician and a deaf dancer that was presented to a mixed audience with diverse hearing abilities.

Interface design

When listening to music, the Deaf community experiences profound isolation, and its members need nonauditory modality to perceive the musical context, features, and emotions delivered in performance. Beyond listening, participating in music and dance practices contributes to their daily life by developing a better understanding of rhythm (Benari, Reference Benari2014), improving communication and connection with others (Darrow, Reference Darrow1993, Reference Darrow2006), or accessing opportunities for artistic and self-expression (Benari, Reference Benari2014). In this workshop series, we focused on creating a shared performance space for mixed performers and audiences that increased collaboration between hearing musicians and deaf dancers.

Felt Sound was designed as a movement-based musical instrument, primarily for D/HoH listeners. Based on our experiences from the musical performances and findings from the user studies and workshops, we developed new interfaces, including the next iteration of Felt Sound’s interface for the musician and wearable haptic modules for the dancer, and a new performance (Cavdir, Reference Cavdir2022). We collaborated with a deaf dancer to create both the interfaces and the performance. She has experienced profound hearing loss since birth, communicates primarily with sign language, and associates equally with deaf/hard of hearing and hearing individuals. In addition to her hearing impairment, she experiences some physical limitations and uses a wheelchair both in her daily life and in dancing. As a dancer, her need for movement and bodily expression necessitated a wearable design for vibrotactile feedback.

In addition to developing wearable haptic modules, we adopted Felt Sound’s wearable interface for easier connection and mobility in the second iteration. The wearable haptic modules were prototyped for the first time in collaboration with the dancer. During this codesign process, we constantly integrated her feedback and design considerations. We developed a shared performance space with the dancer where the dancer both contributed to the artistic creation and participated throughout the design process, specifically in ideation, prototyping, and performance design stages. Her participation was crucial in the design process because her persona brought her specific requirements and needs to the forefront and because she embodies a unique set of skills and artistic practices. Although the Deaf community includes diverse hearing abilities and musical interests, through this bespoke design, we observed that she still represents many deaf artists’ musical expectations, requirements, and engagements. This collaboration enabled us to codesign a shared performance space across diverse hearing and physical abilities.

Design approach: Cocreating music–dance interfaces

We conducted three workshops with the dancer as we codesigned the interfaces and the performance. The first workshop introduced her to the Felt Sound research, including our inclusive design approach, gestural vocabulary, and music composition for felt experiences. Through a semi-structured interview, we discussed her experience with hearing, community associations, music, and movement. Based on our discussions, we brainstormed interface ideas in three areas: (a) dancing, (b) listening experiences, and (c) collaborative performances with musicians.

In the second workshop, these prototypes were tested with the dancer (see Figure 6). We also explored which types of music compositions she perceived better and enjoyed more through the haptic modules. Initially, we tested four sound files with different musical qualities, including an excerpt from Felt Sound’s previous performance, African drumming, female voice singing, and piano, using a two-subwoofer speaker system. She needed to sit close to the speakers or touch them to feel the vibrations. She reported that she felt the Felt Sound and African drumming pieces more profoundly and more nuanced than the voice and piano pieces. Although the vibrotactile listening was less pronounced in voice and piano, she was able to recognize the pitch changes and onsets in the singing and she was able to recognize the instrument in the piano piece. She was also able to feel the music through in-air vibrations. She stated that “she can feel it inside” of her body, pointing to her chest and torso. She still preferred to touch the speakers to amplify the vibrotactile feedback. Because she moves in space when she dances, she wanted wearable modules on different locations on the body: one worn on the arm and another one in the chest area. Figure 6 shows the first prototype of the haptic modules which use Haptuator brand actuator.Footnote ³

Figure 6. The haptic module is prototyped with a Haptuator brand actuator, bluetooth audio module, and 3D printed enclosure for the haptic module. We tested this haptic module on her chest, arms, and hands.

After the second workshop, the first prototype of the haptic modules was redesigned (see Figure 7). We upgraded the haptic modules in shape and material for two reasons: (a) ergonomy and usability and (b) effectiveness of the actuation. Firstly, 3D printed modules are replaced with similarly sized fabric-foam enclosures based on the dancer’s feedback on the modules’ ergonomics. During the second workshop, the dancer commented that she needed a more flexible and softer module on her body. We also observed that she needed to hold and press the module to feel the vibrations closer to her skin. For more comfortable use, we designed both module enclosures and wearable straps that fixture the haptic modules on the dancer’s body with soft, stretchable, and stable fabric materials. Figure 7 shows the two haptic modules connected to the audio amplifier separately and embedded in the elastic straps of the wearable attachment pieces. Secondly, these interfaces were redesigned because the 3D printed enclosure failed the provide sufficiently strong vibrotactile feedback. The vibrations were more significantly perceived on the skin with a nondampening foam. The PLA material dampened some of the vibrations through its thickness and infill structure and the enclosure required a foam layer between the part and the actuator to avoid the rattling noise. Because this two-layer structure of the first prototype decreased the intensity of the vibrations and proximity to the skin, it is replaced with a fabric-foam enclosure. Additionally, because the wearable straps in the first prototype were unable to provide enough support during dancing, they are replaced by stretchable materials connected by the 3D printed fasteners.

Figure 7. Two haptic modules, including Haptuator Mark 2 brand, type C and D actuators, are connected to the bluetooth audio amplifier. A 3D printed enclosure is designed for the amplifier and its power, audio input, and output connections. Elastic straps are designed to enclose the new haptic modules to fixture them on desired locations on the body. Fasteners are 3D printed to adjust the tightness and stability of the wearable straps.

Results and synthesis: Shared performance spaces

Over three workshops and a demo session, we developed a performance setting across dance and music for deaf and hearing individuals. The performance practice was shared between the dancer and the musician through (a) on-body, vibrotactile music delivered from the musician to the dancer and (b) a narrative presented with sign language gestures and choreography. This performance practice focused on the interaction of the dancer with the musician in response to the musician’s live gestural performance.

In the last workshop, the interfaces were finalized where the dancer experienced listening to the vibrotactile music both through in-air vibrations using the subwoofer array and on-body sensing using the haptic modules. The dancer and the musician also developed a movement vocabulary in response to Felt Sound’s gestures. Because the dancer preferred performing a choreographed sequence more than improvisation, the dance movements were selected from the dancer’s repertoire and mapped to the musical gestures and vibration patterns. Different gestures and vibration patterns, performed by the musician, provided movement cues for the dancer. More specifically, the dancer received the music signals through the four-subwoofer speaker setup around the performance space and with the two haptic actuators (placed on the arm and the chest). In response, she performed her dance movements as she recognized specific vibration cues on her body from haptic modules and visual cues from the musician’s gestures. She reported that in order to follow the articulations in music, she needed the vibrations to be stronger, specifically to recognize the vibration-to-movement mapping during the dance. The in-air vibrations from subwoofer speakers only provided felt sensations on the body without the nuances of the sound. Due to these significant differences between the two vibrotactile modalities, we continued our dance–music collaboration with the haptic modules, using sounds from subwoofers to support the on-body listening.

The dancer’s experience with two vibrotactile modalities showed how to better approach creating felt experiences. We observed that the subwoofer array that surrounded the room created a more embodied, whole-body listening experience that was felt in the room and inside the body. The dancer reported that the in-air vibrations were felt on her body and the surrounding space and it is a form that she was used to experience sound. The workshop setup provided more amplified sensations of sound vibrations in the room. However, in-air vibrations challenged her to detect the nuances in the music. The on-body haptic modules offered more sensitive and articulate listening experiences, especially when simultaneously worn on more than one location on the body. She explained her experiences wearing the haptic modules: “I understood what the sound was about, I could feel the difference. I could feel the diversity and the flow in music.” The results of these experiments also showed that understanding the music through different modalities of vibrations, including learning how to feel and interpret the vibrations, requires ongoing practice for Deaf users.

The same performance was tested with hearing users. One participant wore the haptic modules and preferred to close her eyes while listening. She reported that “[she] had to listen in a different way, not only aurally but also in her whole body,” describing it as an internal listening experience. The vibrations created a sensation that resembled “a dialogue between skin and heart.” Similarly, one participant with a music background said that “she did not consciously separate two different modalities” (sound vibrations from the subwoofer speakers and on-body vibrations from haptic modules), instead “[the listening] became one experience.” Another participant reported that it was effective to feel the music through vibrations first, “neutralizing the listening experience beyond localized sensations on the body.”

These reports showed that the listening experience shifted the dancers’ focus onto their bodies, “regardless of an observer” as one participant expressed her first-person experience. This approach created opportunities to share embodied listening experiences among performers, connecting the vibrations created by the music with the movements of the dancer. One hearing participant with a dance background explained her experience with vibrotactile feedback and movement as “[…] the skin felt the vibrations that went into the body and the movements grew from there.” In addition to encouraging the inner motivation of the dancers to move, one participant reported both the dance and music performance using body movements and the sign language gestures expressed a connection, exuding dance-like qualities.

In addition to sharing the performance experience through felt sensations on the body, we cocreated a shared gestural vocabulary. A dance movement vocabulary was developed based on the connection between the musical gestures, their sonic response, and vibrations. Due to COVID-19 restrictions, the documentation of the collaborative performance was recorded virtually. Figure 8 shows some of the gestures composed by the Deaf dancer in response to Felt Sound’s gestures, sounds, and vibrations. She created a narrative choreography following a storyline of a storm where different parts of the story were associated with the vibration patterns and intensities. In Figure 8a, the dabbing gesture over the hand corresponded to the slowly increasing intensity of the vibrotactile sound waves, matching with Felt Sound’s discover gesture; Figure 8b shows the joint hands with fingers imitating a flying bird, representing the pulsating effect on the sound and vibrations, and finally the scanning gestures with one hand and the gaze in Figure 8c completes the performance as the sound and the vibrations fade out. These three dance movements are connected with the movement of arms waving in space, representing the high volume and frequency of the vibrations. Additionally, the poetry gesture in Felt Sound provided a movement cue to the dancer to change her location on the stage. Figure 9 shows the improvised movements of the hearing dancer while she listened to the music both with the subwoofers and the haptic modules on her body.

Figure 8. The deaf dancer composed four dance movements in response to the different vibration patterns. Three movements are connected with arms waving gestures in the air to represent the intensity of the vibrations and inspired by dance metaphors representing (a) raindrops with the dabbing gestures, (b) flying bird by flexing the fingers’, and (c) horizon by scanning space with the hand and the gaze.

Figure 9. The embodied listening experience where the sound is delivered by the haptic modules combined with the four subwoofer speaker arrays was also tested with a hearing dancer who improvised dancing to the music performed by the ASL gestures.

Awareness on the body

As previously described, the moving body is integral to the design of movement-based musical interfaces that positions one’s own bodily and felt experience at the center of musical expression. Our case studies demonstrate how wearable interaction supports first-person perspective through felt experiences, from building body-to-instrument connection and attachment to sensing music directly on the body.

Awareness on the listening

With Bodyharp, the participants’ focus shifted to the sonic outcome of their movement, making them feel less self-conscious. One participant with reported that “most of my attention is when I do something [a movement] on how the sound comes out.” Another participant shared similar experiences with focusing on the sound-creation process by saying, “The sound carried through my body […], I also felt very focused […]. There was an interesting process of attentive listening.” One participant commented on the composition practice with Bodyharp. “being asked to create [a musical statement] makes me much more aware of the sounds that my movements are creating.”

According to all participants in the Bodyharp study, the instrument encouraged them to move more while creating music and be more attentive to their body and body movements. For example, one participant highlighted that “the interaction between what you do and what you listen [to] really forces you to be really engaged and in tune in your movement as a source of inspiration.” Similarly, another participant reported that when she “felt out of control of the instrument,” she moved back to “listening to, thinking in terms of gesture, and feeling how to play, and letting go of the idea of always knowing how the instrument will sound.”

The participants’ responses to their listening experience highlighted how interconnected the awareness was built on the moving body and the listening. The Felt Sound study more specifically focused on shifting the listening experience to a more bodily experience using kinesthetic music composition. Participants used their bodies more to listen; in other words, they increased their awareness of how much the body was involved in the listening. One participant stated that “I felt like I was using my torso to listen to music rather than my ears. The vibration seemed to be felt in and out of the torso.” A similar experience was accessed in the Touch, Listen, and (Re)Act study. Hearing participants reported that they “felt it more than hearing [the music].” Similarly, one participant shared her experience of feeling the music “inside of [her] body […] in the torso and entire body.”

In the second case study, the biggest limitation was the lack of deaf and hard of hearing participants. Although, the study participants had experience with ASL performances (such as performing arts and new music, song interpretations, or theater), the study fell short in collecting the listening experiences of D/HoH members due to resource demanding process of reaching out to the D/HoH community. However, we were able to collect feedback on the listening experience with Felt Sound with one Deaf participant. Comparing the responses from both the second and the third case studies and from both hearing and deaf participants revealed how assistive wearable instruments and music performing and listening systems can support the experiences of hearing listeners. In the public performing session, we collected hearing participants’ experiences with the two haptic modalities.

For many, haptic modalities provided a listening experience outside the familiar. Such defamiliarization supported their increased attention on the bodily listening. They reported that they had to “listen in a different way, not only aurally but also in [their] whole body,” and they did not “consciously separate two different modalities” (sound vibrations from the subwoofer speakers and on-body vibrations from haptic modules), instead “[the listening] became one experience.” Exploring such an embodied form of listening expanded participants’ movement vocabulary in musical interaction. One participant highlighted how she freely moved while listening through both in-air and on-skin haptics as “[…] the skin felt the vibrations that went into the body and the movements grew from there.” The vibrations created a sensation that reminded her of “a dialogue between skin and heart.”

Additionally, bodily listening increased participants’ attention to musical qualities such as silences or intensities of sound effects. According to P5 in the Felt Sound study, the silences in sound and movement amplified the correspondences between music and gestural performance. These changes in silences and intensity of the vibrations were felt on the body. This feedback was reported in the third case study by the Deaf participant. She repeatedly reported the silences in vibrations and interpreted some of the low vibrations as silences. The level changes and varying amplitude envelopes were These similar responses from both participants with and without hearing impairments can inform the sound composition when designing bodily listening experiences with haptics. One of the limitations we experienced in the third case study was the time constraints for the participant to learn how to listen through haptics, considering that she had little experience listening to music. It highlights the need to develop a practicing framework to learn listening through haptics and how the vibrations are correlated to musical features. The same approach can apply to composition practices. For composers and designers, a framework to design music-vibrotactile mapping for increased awareness on the listening can further be studied.