The Embodied Turn

In recent years, there has been an increasing interest in using methodologies that go beyond the domain of verbal descriptions and include embodied aspects in research leading the so called “embodied turn” or “corporeal turn” in human sciences (Brown et al., Reference Brown, Cromby, Harper, Johnson and Reavey2011; Seikkula et al., Reference Seikkula, Karvonen, Kykyri, Kaartinen and Penttonen2015). Indeed, there is evidence to suggest that most of the communication between people takes place nonverbally (Mehrabian, Reference Mehrabian1981, Reference Mehrabian2009) estimated that non-verbal gestures and expressions communicated 93% of people’s feelings and attitudes. Similarly, Birdwhistell (Reference Birdwhistell1952) argued that no more than 30% of a social interaction and its attached meaning happens verbally. Even in an everyday context, if a person argues she feels sad but her facial expression and body gestures indicate that she is angry, people tend to believe more what is communicated nonverbally.

According to Denham and Onwuegbuzie (Reference Denham and Onwuegbuzie2013), studying nonverbal communication could help qualitative researchers in various ways, such as increasing validity through triangulation, capturing underlying messages, gaining new insights on nonverbal elements and discovering potential contradictions between verbal and nonverbal behavior. Yet, as Brown et al. (Reference Brown, Cromby, Harper, Johnson and Reavey2011) noted, embodiment in research poses ontological and methodological challenges. Although there has been a significant shift toward the inclusion of the body in research, little is mentioned regarding the ways in which current methodologies can move the focus away from discourse (Brown et al., Reference Brown, Cromby, Harper, Johnson and Reavey2011). Furthermore, there is currently no unified methodological approach. The existed embodied methodologies are informed by various theoretical frameworks ranging from phenomenology to neuroscience (Spatz, Reference Spatz2017).

Whilst these conceptualizations differ significantly, there is a common thread among them of approaching the embodied aspect of human experience and, in particular, incorporating the felt or affective dimensions. According to Willis and Cromby (Reference Willis and Cromby2020), this “affective turn” signifies an interest in emotion and affect, and places these phenomena both in biology and language. Therefore, this theoretical perspective does not seek to replace the focus on language but instead aims to integrate the verbal and embodied components of human experience. Indeed, our experience is always embodied experience and meaning is continuously generated both through the body and language (Cromby, Reference Cromby2012).

Qualitative researchers have used an array of methods for collecting and analyzing the nonverbal and embodied components of human behavior and interaction. According to Seikkula et al. (Reference Seikkula, Karvonen, Kykyri, Kaartinen and Penttonen2015), prosody-pitch, the rhythm and the timbre of the voice, body posture, gesture, and facial expression are some of the nonverbal or paraverbal elements that are vital to human communication and interaction. Other researchers have focused on mutual smiling episodes in a therapeutic context (Darwiche et al., Reference Darwiche, de Roten, Stern, von Roten, Corboz-Warnery and Fivaz-Depeursinge2008), on synchronized behaviors of participants, especially body movements (Seikkula et al., Reference Seikkula, Karvonen, Kykyri, Kaartinen and Penttonen2015) and on silences in people’s discourse (Mazzei, Reference Mazzei2008). Furthermore, Gorden (Reference Gorden1975) has developed a typology of nonverbal communication data. Some indicators for studying body language are kinesics (such as body movements and postures), proximity between people, temporal speech markers (such as silences and hesitation) and paralinguistic aspects (such as tenor, strength, and color of voice). Similarly, Ekman (Reference Ekman1999) studied how emotions are expressed through facial expression in various social contexts, (McNeill, Reference McNeill1992) focused on different types of gestures and Onwuegbuzie et al. (Reference Onwuegbuzie, Dickinson, Leech and Zoran2009) highlighted the meaning that can be communicated via gestures.

Consequently, researchers can use various frameworks for incorporating the embodied component of human interaction and communication. There are also numerous instruments that researchers can use to collect the data, such as audio and video recordings, photographs, checklists, matrices, and diagrams (Onwuegbuzie et al., Reference Onwuegbuzie, Dickinson, Leech and Zoran2009). More recently, partly due to technological advances, researchers have deployed physiological measures through the use of biometric devices and wearables (Cromby, Reference Cromby2012).

Sociodrama

A sociodrama is a dramatic play in which several individuals act out assigned roles for the purpose of studying and remedying problems in group or collective relationships. Its founder was Jacob Moreno, a social scientist who invented psychodrama and later sociodrama in an effort to explore his sociological interests while using psychodrama techniques. Sociodrama is an experiential group-as-a-whole procedure that can be used to explore social issues, social justice concerns, collective trauma, prejudice, stigma, interpersonal tension, intergroup conflict, healing, justice, reparation, and other social factors operating in complex human settings (Moreno, Reference Moreno1943; Marineau, Reference Marineau1989; Moreno et al., Reference Moreno, Blomkvist and Rutzel2000). It can be used for exploration, engaging, problem solving, and teaching (Agazarian and Carter, Reference Agazarian and Carter1993; Moreno et al., Reference Moreno, Blomkvist and Rutzel2000). Sociodrama is suggested, and has been used, as a creative means for groups to work through political and social issues that concern them (Browne, Reference Browne2005).

In the last decades, through the use of social theater (Boal, Reference Boal1979; Saldaña, Reference Saldaña2011) and other arts-based collaborative approaches (Conrad and Sinner, Reference Conrad and Sinner2015), sociodrama, apart from an innovative method of qualitative analysis of social issues (Saldafia, Reference Saldafia2003; Saldaña, Reference Saldaña2011), has developed into a major transformative intervention for community development (Weiner et al., Reference Weiner, Adderley and Kirk2011; Conrad and Sinner, Reference Conrad and Sinner2015).

Sociodrama is a methodology applicable to all sizes and types of groups, even to large groups. It starts with the present participants, in the here and now of the meeting. Instead of a group discussion a sociodramatic group demands individuals who are involved to engage each other in specific dramatic activities in order to discuss, explore and enact the possible solutions on issues that concern them (Kellermann, Reference Kellermann1998). Its purpose is by clarifying, educating and energizing, to make people find their spontaneity and creativity, love and empathy inside themselves, with the ultimate goal of increasing internal control, balance and integration as controversies and hostilities erupt in the world, in a local and/or in a larger area. That is why the sociodramatic groups are groups of horizontal hierarchy, that is, the group process is cocreated by the group leader as well as by all the participants (Moreno, Reference Moreno1953; Agazarian and Peters, Reference Agazarian and Peters1981).

Both verbal and nonverbal communication are involved in sociodrama. In verbal communication, thoughts, emotions or information are exchanged between individuals through the use of speech. Sociodrama also allows for a great deal of nonverbal communication that involves signals transmitted through facial expressions, posture, eye contact, gestures, tone of voice, body language, physical movements, space, touch, paralanguage, and other ways (Aguiar, Reference Aguiar2001; Bargh, Reference Bargh2013).

Wearables in Research

Wearables or biometric devices have been used in various studies ranging from analyzing the impact of music on humans (Rickard, Reference Rickard2004; Kelley et al., Reference Kelley, Andrick, Benzenbower and Devia2014), collecting data at events such as rituals and spectacles (Konvalinka et al., Reference Konvalinka, Xygalatas, Bulbulia, Schjødt, Jegindø, Wallot, Van Orden and Roepstorff2011; Fischer et al., Reference Fischer, Xygalatas, Mitkidis, Reddish, Tok, Konvalinka and Bulbulia2014), investigating consumers’ behavior (Chamberlain and Broderick, Reference Chamberlain and Broderick2007) and exploring interactions and dialogues within psychotherapeutic settings (Seikkula et al., Reference Seikkula, Karvonen, Kykyri, Kaartinen and Penttonen2015; Kykyri et al., Reference Kykyri, Karvonen, Wahlström, Kaartinen, Penttonen and Seikkula2017). Many researchers have measured cardio-vascular activity, such as heart rate (HR), heart rate variability (HRV), and finger temperature (Kreibig, Reference Kreibig2010; Seikkula et al., Reference Seikkula, Karvonen, Kykyri, Kaartinen and Penttonen2015).

Similarly, electrodermal activity (EDA) is often used to capture changes in skin conductance and galvanic skin response (GSR) (Sorinas et al., Reference Sorinas, Ferrández and Fernandez2020). In addition, changes in blood pressure, as well as respiration rate and variability are common physiological responses measured by biometric devices (Calvo et al., Reference Calvo, Member, Mello and Society2010; Sorinas et al., Reference Sorinas, Ferrández and Fernandez2020). HR and skin conductance are the measures most often reported in studies (Kreibig, Reference Kreibig2010), and they have both been used with the purpose of capturing levels of arousal (Kreibig, Reference Kreibig2010). There is also some evidence suggesting that changes in skin conductance reflect the general level of activation while performing a task, while HR depends more on the emotional response rather than on task requirements (Iffland et al., Reference Iffland, Sansen, Catani and Neuner2014).

The use of biometric devices in psychology and the social sciences relies on an underlying assumption that physiological measures capture bodily reactions that are not observable in visible behavior or in verbal accounts (Chamberlain and Broderick, Reference Chamberlain and Broderick2007; Seikkula et al., Reference Seikkula, Karvonen, Kykyri, Kaartinen and Penttonen2015). Drawing on neuroscience research and theories of emotion, bodily changes and in particular physiological arousal are linked with the rising of emotions (Rickard, Reference Rickard2004). For example, EDA has been found to increase with most emotions (Kreibig, Reference Kreibig2010). According to Rickard (Reference Rickard2004), skin conductance is a more reliable indicator of emotions than language-based methods, such as self-reports. HR changes have also been strongly associated with different affective states and responses, such as anger, anxiety, joy, fear, and sadness (Chamberlain and Broderick, Reference Chamberlain and Broderick2007; Kreibig, Reference Kreibig2010).

In many recent theories of emotion, the autonomous nervous system (ANS) is perceived as a major component of emotional responses (Kreibig, Reference Kreibig2010). The ANS is both an activating and restorative system. Specifically, it consists of two subsystems, the sympathetic nervous system (SNS), that increases activation and stimulates, and the parasympathetic nervous system (PNS), that has a restorative function. The ANS, also called an involuntary nervous system, responds to stimulus immediately, before conscious thought. Therefore, studying physiological reactions can provide valuable information for the affective dimension during social interactions (Seikkula et al., Reference Seikkula, Karvonen, Kykyri, Kaartinen and Penttonen2015). For example, when a stimulus is perceived as significant, there is a sympathetic activation of the ANS and an increase in physiological arousal occurs. Arousal is defined as the level of activation, ranging from total calmness to high alertness (Chamberlain and Broderick, Reference Chamberlain and Broderick2007).

It should be noted that there is no preexisting and specific procedure or mapping when analyzing biometric data within the context of qualitative methodology. Furthermore, given the fact that biometrics measure arousal levels and no specific emotions, the data is analyzed in combination with qualitative observations. As mentioned earlier, this “affective turn” in research focuses on both the verbal and nonverbal elements of human experience, which that are perceived as occurring simultaneously and continuously (Willis and Cromby, Reference Willis and Cromby2020). Developing a method of mapping biometric data onto different layers of verbal and nonverbal data in order to prepare the ground for a form of combined analysis of several data layers is the goal and, we contend, the innovative contribution of this study.

Study Design

The study aims to explore the perspectives of residents in Eleusis regarding their living conditions and their impact on their lives, at both individual and collective levels. The study design, incorporates sociodrama sessions, in which Eleusis residents are called upon to express themselves and deliberate as a group on the topics of environment, unemployment, and migration, that have been identified as central concerns for the Eleusis community. In line with multimodal research, the sessions are video and audio-recorded and bio-signals are obtained from selected participants, generating visual, audio and bio-signals. After preprocessing and synchronizing, the data are subjected to several layers of analysis, resulting in complex multilayered depictions of the patterns of expressions of Eleusis residents’ perspectives regarding living in the community of Eleusis. It is important to clarify that in article we contribute with the methods and a pilot study and we report on the technical and practical challenges, while the complete ethnographic research and interpretation of the verbal aspects is an ongoing process.

Epistemology

This study adopts a critical realist epistemological perspective. Critical realism is ontologically realist, assuming the existence of an external reality that is independent of human minds, and epistemologically relativist, acknowledging that different methods produce different perspectives on reality (Williams, Reference Williams2018). Although it offers a general theoretical framework, critical realism is not linked to a specific methodology, hence it can inform a wide variety of quantitative, qualitative, or mixed-methods designs (Sturgiss and Clark, Reference Sturgiss and Clark2020). Moreover, as it focuses on understanding the process, how and why things happen, it is most appropriate for qualitative research approaches that explore the dynamic nature of social processes (Sturgiss and Clark, Reference Sturgiss and Clark2020). Moreover, critical realism uses a theoretical and researcher driven analytical process (Fletcher, Reference Fletcher2017), which, in our study, fits the sociodrama process. At the same time, as we are interested in exploring participants’ experiences and perspectives on their conditions of living in Eleusis and the impact of this on their lives, we also adopt a phenomenological perspective. Phenomenological epistemologies focus on capturing and being true to participants’ experiential worlds (Willig, Reference Willig2013). Combining a critical realist approach to knowledge production with a phenomenological focus on participants’ subjective experience is considered appropriate in qualitative research, as they are epistemologically compatible and share enough common premises (Willig, Reference Willig2012).

Ethics and Quality Assurance

When it comes to complex, real-world contexts, mixed methods research offers a larger set of more flexible tools for dealing with ethical dilemmas than single-method approaches (Hesse-Biber, Reference Hesse-Biber2010; Preissle et al., Reference Preissle, Glover-Kudon, Rohan, Boehm and DeGroff2015). Reflexivity as a methodological consideration is central to ethics in mixed methods research, along with positionality and efforts to minimize bias (Cain et al., Reference Cain, MacDonald, Coker, Velasco and West2019).

Study procedures are in accordance with the ethical standards of the responsible institutional or regional committee on human experimentation or in accordance with the Helsinki Declaration of 1975 as revised in 1983. Ethics approval was sought and gained from the Ethics Committee of Athena Research Center.Footnote ¹

Written informed consent, after full briefing regarding the requirements and ethical safeguards of the study, will be provided by each participant as a prerequisite to their participation in the sociodrama sessions. Files of the nonpseudonymised video, audio and bio-signals will be kept in an encrypted server, to which only the research team members will have access, following signing a confidentiality agreement. In-depth qualitative analysis will be conducted to pseudonymized data, after all identifying features have been removed. Only processed and pseudonymized data will be given to artists and used in research reports and academic publications. The study complies with the Data Protection Regulation (EU) 2016/679 of the European Parliament. Specifically, a detailed management and storage plan for all data is envisioned, including storage of data on encrypted and password-protected files, secured on University computers, defining persons responsible for data storage and management and regulating access to data by research team membership status.

Data quality and analysis may be a measure of ethics in mixed methods studies and, thus, should be addressed (Cain et al., Reference Cain, MacDonald, Coker, Velasco and West2019). We have designed data collection and analysis processes that endeavor to fulfill the quality assessment criteria for validity and reliability, as defined in both qualitative and (Lincoln et al., Reference Lincoln, Guba and Pilotta1985; Walsh and Downe, Reference Walsh and Downe2006). More specifically, designing a multimodal study that incorporates several layers of qualitative and quantitative data within a multidisciplinary team requires intensive processes of building common ground, through familiarizing with other research approaches, translating concepts and tools to other research and disciplinary languages and building shared technical capacity in order to arrive at commonly agreed decisions and research protocols. Through systematic team work, triangulation and consensual validation processes have been adopted throughout the design, data collection and data analysis phases, and this we consider to be a major strength of this study. This collaborative process also ensures that the study fulfills the quality criteria of transparency, as all steps and aspects of the research process are continuously recorded and explained, transferability, as the knowledge acquired is already made relevant to different disciplines, as well as applicability, as the potential impact of the study is at the center of the team’s concerns.

Sample

Residents of Eleusis will be asked to participate in sociodrama sessions, with a minimum number of 20 adult individuals per group. In total, two groups per subject will be conducted, with a total number of approximately 120 participants. Regarding inclusion criteria, participants must be adult residents of Eleusis, who speak and understand the Greek language, as the sessions will be conducted in Greek. There will be no exclusion criteria. We aim to recruit a wide range of participants, with varied characteristics, in order to have a sample that will be as representative of the population of Eleusis as possible. This will be ensured through a broad dissemination of the participation call to the population of Eleusis through local agencies and through distributing participants in sociodrama sessions in a way that ensures participant variability within each session.

Data Collection Setting and Process

Data collection will take place in the framework of sociodrama sessions in Eleusis city in Greece, in a dedicated venue, in a big room with the capacity for allowing 20–25 people to comfortably walk and move in space. Sociodrama is an experiential procedure for social exploration characterized also as a “potential stage” for creating transformative research on social issues (Ius, Reference Ius2020). Participants will be asked to work through issues that have been designated as topics of concern for Eleusis, namely environment, unemployment and migration. Each sociodrama session will be dedicated to one of the issues above, and will last for approximately one-and-a-half hours. The sessions will be facilitated by two experienced sociodramatists, members of the research team.

Prospective participants are aware beforehand that they are called upon to participate in a sociodrama session and the topics they will be asked to engage with during the session. Detailed information regarding sociodrama is given to participants beforehand and they are called to attend an introductory meeting with the other participants, if they wish. In the beginning of the session, the sociodrama facilitators introduce the rationale and process of the session and initiate activities, through which participants introduce themselves to one another and start getting a sense of belonging to a group. The facilitators then assign a series of roles to individuals and subgroups of participants, and instruct them to enact these roles in subgroup or whole group activities. Usually, participants alternate between roles and subgroups, so that they experience all the aspects of the dilemmatic topic under consideration. There is guidance on how to play a role; participants are encouraged to get into their roles and feel free to express them in any way that feels right to them. The detailed introduction to the setting, the initial team building exercises and the freedom in performing the assigned roles, all are designed to decrease performance anxiety, to foster ease and spontaneity, and immerse participants in the “here and now” of the enactment process. Moreover, it is important to mention that during a sociodrama we do not ask participants to act, but to participate in a specific task as if they were someone else and express their personal point of view. We clarify to them that they are not judged on how they perform as if they were theater actors but they are asked to act as they feel, expressing opinions that their role they play would support.

The main part of the session consists of several phases of role enactment and working through the designated issues. The session is concluded with participants’ reflections on the experience of participating in this process and the new perspectives on the debated topic that they have obtained through this.

Sociodrama provides an open stage and a procedure through which various collective dynamics can be played out. In the process, every participant has a voice as an individual and as a member of the group. Participants engage with the topic at hand through thinking, talking, feeling, moving and acting, allowing their thoughts, feelings and actions to unfold in a free-flowing way without having to follow strict rules. Participants may sit or stand and talk, they may move to parts of the space to engage in subgroup actions, they might move within the space while enacting a role and so forth Verbal, nonverbal and bodily signals, expressions and interactions all contribute to the unfolding process, and therefore have to be recorded.

Wearable, Audio, and Video Equipment

To record human and human body activity during each sociodrama session, three different types of equipment will be used: cameras, microphones and wearables to record physiological data and movement.

GoPro Hero4 Silver cameras will be placed around the room to record the activity of the crowd from a panoramic point of view utilizing the wide-angle-lens setting option. Also, two more cameras set on the regular-angle-lens (no wide-angle-lens) will be placed in locations that allow the recording of postures and facial expressions of participants. Figure 1 demonstrates the top view of the room and the placement of the equipment in it.

Figure 1. Top-view of the room, displaying the location of the cameras and microphones.

The sound will be recorded by all cameras. However, to achieve a better quality of sound for the thematic analysis that we describe in the sections below, we will employ Zoom microphones. The placement of the microphones can be seen in Figure 1.

The wearable equipment that will be used is two Heartypatch sensors from Protocentral, previously used in Kalampratsidou et al. (Reference Kalampratsidou, Zavorskas, Albano, Kemper and Torres2019) and Kalampratsidou and Torres (Reference Kalampratsidou and Torres2020). Heartypatch sensors record ECG (electrocardiogram) signal wirelessly. To achieve this, we place the sensor on the left side of the chest (by the area of the heart) of randomly selected sociodrama participants using nonreusable sticky gel electrodes and we turn the sensor on. The sensor is automatically connected to the WiFi networks that it is preprogrammed to connect; the PC should be connected to the same network. Then, we execute the Python code that we wrote to load and save the streamed data.

Recordings Synchronization

To synchronize the data recorded from the various devices, we will time-stamp all the data collected on the Heartypatch devices. To synchronize the audio and video data with the bio-signal recordings, one of the GoPro Cameras, named as main (see Figure 1), will be initiated through a Python script employing the Python module goprocam.Footnote ² In this code, the time that the recording starts will also be time-stamped and saved. Once we know the time that the main camera started and the times of the signals collected, we will be able to sync the rest of the cameras and audio data with the main camera by listening or watching the material, and therefore all audio and video data will be synchronized with the bio-signals.

In a secondary effort to have control of the device synchronization, during sociodrama one of the researchers, who is responsible for setting up the equipment, will time-stamp manually the times that each equipment piece is started as well as the timeline of the sociodrama and possible failures will be noted during the session.

Data Analysis

Different types of data from sound, video and bio-signals files will be analyzed comparatively and in conjunction. In the initial phase of the analysis, these different types of data files will be synchronized and will be coded for the presence of particular interactional and group process instances. Subsequently, particular interactional and group process episodes will be selected for further analysis. Selected fragments of specific episodes will be subjected to in-depth qualitative analysis by using the Video Annotation software program. Such software programs, such as ELAN by The Language Archive, Hyperresearch by Researchware Inc., and so forth offer the possibility to choose, code and analyze segments of video-recording, through inserting annotations on video episodes, transcribing and analyzing visual and verbal content. The software’s user interface intuitiveness is considered to be exceptional and easy to use. Particular attention will be paid to the coordination between different levels of participant expression—verbal, nonverbal and bodily—for each participant. We will also be analyzing how these levels interact at group level, in relation to happenings at the level of group interaction. Finally, we will attempt to associate these coordinated patterns of individual and group expression with the topics under discussion, in order to arrive at conclusions regarding how different aspects of living in Eleusis impact on its residents, both individually and collectively.

Once the analysis has been completed, the written result reports will provide accounts of the complex patterns of processes uncovered. At the same time, the pseudonymised extracts of raw data—video, audio, biometric, verbal—will serve as both testimonies and vivid enactments of the processes depicted through the analysis. In the first sense, as testimonies, the raw data extracts can be used to document and support the analysis, accompanying written reports of the study findings. As vivid enactments, on the other hand, they have an evocative power that goes beyond a proof of data. For this reason, selected extracts will be given to the artists to aid their attunement with Eleusis residents’ experiences and spark their inspiration for the music they will be called upon to produce.

Bio-Signals Processing and Metrics

To work with the EEG signal we first filtered it to remove instrumentational noise and then extracted the heart-rate signal. Also, the audio signal was employed which was examined as another bio-signal. The EEG signal was processed and plotted in Matlab R2018b. Raw ECG data were streamed from the Heartypatch sensor and saved on a Python interface developed by our team. The data were then filtered using a Butterworth IIR band pass filter for 5—30 Hz at second order, to remove any baseline wandering and to accurately detect the R-peaks. The range of the band pass filter was selected based on the fact that a QRS complex is present in the frequency range of 5—30 Hz. From the filtered EEG signal, the heart-rate was estimated by extracting the peaks (or beats) from the ECG signal and by counting the beats that happened within the past 1 min. This estimation repeats each time a new beat is detected. Therefore, we were able to generate a graph that demonstrates the change of the HR for every beat detected than the heart-rate updated every minute. The study of the audio signal was done through spectrograms. The spectrograms were generated with iZotope RX 7 Audio Editor software and visualize three parameters: time, loudness, and pitch. Time is measured in seconds and corresponds to the X-axis, loudness is measured in Decibels (dB) and corresponds to the Y-axis, and pitch is measured in Hertz (Hz) using the Mel scale and is represented by the color range. Lower frequencies are visualized in blue colors, mid frequencies in green colors, and higher frequencies in red colors.

Pilot Sociodrama Experimental Session and Sample

The pilot sociodrama session was held in June of 2021 in Eleusis city, and was coordinated by two experienced sociodramatists (one male and one female), the same sociodramatists who are scheduled to coordinate the main sociodrama sessions of the project. As the pilot session had a preparatory goal for the main sessions/experiments, we did not involve Eleusis citizens as participants but it was held among the project team members. In particular, participants of the session were nine persons (four male and five female), with three of them working and living in Eleusis, one of them being a native of the city. The participants belonged to all three research teams that collaborated in the implementation of the “Transition to 8” project. Specifically, three participants from the psychology research team (National and Kapodistrian University of Athens), three participants from the technology research team (Athena Research Center) and three participants from the culture management team (Mentor in Culture & Heritage company). The session lasted 2 hr. In addition, three more persons were present during the session: one technology expert from the technology research team, who made sure that the equipment was working properly throughout the duration of the session, and two psychology experts from the psychology research team were observing the session and taking notes, acting as participant observers in ethnographic work.

The theme of the sociodrama session was the notion of place and our connection with cities in general and Eleusis in particular. During the session the participants had to enact particular roles that relate differently to events and facts about the city. In particular, three subgroups were proposed by the sociodramatists: (a) European Commission members, who decide that Eleusis is an appropriate candidate for a European Capital of Culture and announce their decision, (b) residents of Eleusis, who react to these news, and (c) citizens of other Greek cities, who react to these news. The nine participants were randomly divided into three groups and they were asked to enact these roles. All the groups tried all three roles and at the end of the sessions, during debriefing, they were asked to reflect on which role was more difficult and why.

Distribution of Wearable Equipment

Two participants were invited to wear the HeartyPatch Sensors. The selection was random and only after their approval we helped them place the sensor on their chest. In the manuscript, these two participants are referred to as P1 and P2.

Pilot Sociodrama Structure

The pilot sociodrama was divided into three main sections: “Introduction,” “Sociodrama,” and “Conclusion.” Each section contains several subsections, during which participants were directed to execute specific tasks. Below we demonstrate the sections and subsections as well as the tasks contained within them.

1. 1. Introduction—Engaging interaction

   1. (a) Walk and team questions: Participants were directed to walk in the room, then find the location of their research team, present how many they are and if someone is missing.

   2. (b) Maps: Participants imagine that they walk on a map and position themselves depending on the answers to the following questions:

      1. (i) Where do you live?

      2. (ii) Where is the place of your origin?

      3. (iii) What is the place that has impacted you culturally?

         Note: Before each direction participants were asked to walk until they were given the direction. Then they had to place themselves in the map based on the direction and stand there until all participants present the choice of their location.

   3. (c) Project connection: Participants were asked to touch the shoulder of the person who connected them with (or brought them in) the project

   4. (d) How much do you know Eleusis?: The sociodramatist specifies that a book in the middle of the room represents Eleusis, then she directs the participants to place themselves in a distance that indicates how much they know about Eleusis.

   5. (e) Perspective toward Eleusis: Using a book to represent Eleusis, participants are asked to walk in space at various distances to Eleusis and express in words and with their body posture their perspective toward the city.

   6. (f) Creating the subgroups: The sociodramatist randomly splits participants in three subgroups

2. 2. Sociodrama—Role playing

   1. (a) Role 1: Greeks (non Eleusinians): The Greeks (all Greek population except Eleusinians) respond to the news that Eleusis will be a European Capital of Culture for 2023.

   2. (b) Role 2: Eleusinians: Eleusianians learn that their city is going to be European Capital of Culture for 2023.

   3. (c) Role 3: European Commission: The European Commission decides the city that will be European Capital of Culture each year. Eleusis was selected among other candidates as European Capital of Culture for 2023 and the Commission should announce the decision.

      Note: Roles 1, 2, and 3 are listed in the sequence that were performed by P1 (and her team, which includes P2), whose bio-signal data will be discussed later. As it was already mentioned, all the groups tried all three roles and of course the sequence of the roles differs from group to group.

3. 3. Conclusion—debriefing

   1. (a) Walk and talk to Eleusis: A book in the middle of the room represents Eleusis and participants are directed to approach Eleusis and talk to the city.

   2. (b) Discussion: All participants, sociodramatists, observers and the person responsible for the equipment, discuss their experience of participating in, coordinating or observing the sociodrama session.

   3. (c) Circle of trust: Sociodramatists invited all participants to join the circle of trust and after the end of the session take with them anything they enjoyed during the session and leave in the middle of the circle what they wish to leave aside.

Coordination of Qualitative and Quantitative Data

In the following section, we provide the details of a preliminary synchronization of data and data segmentation. In particular, we attempt data segmentation for synchronization and analysis in two different ways: firstly using bio-signals as a guide and secondly using audio data.

Video data annotation and timestamp extraction

ELAN is a tool that enables video and audio annotation. In this work, ELAN v6.0 was used to synchronize the video and audio data recorded on the different devices. The recording of the main GoPro camera was used as a point of reference. Next, we annotated the beginning and ending of all the tasks performed during each subsection. Examples of annotated sections and subsections can be seen in Figures 2–4.

Figure 2. ELAN annotation of the main section “Introduction” and its subsections. The total duration of the introduction was 30.83 min.

Figure 3. ELAN annotation of the “Sociodrama” section and its subsections. The sociodrama lasted 33.82 min.

Figure 4. ELAN annotation of the conclusionary section, which lasted 30.55 min.

The timestamps of the annotated sections are then extracted in milliseconds—among other options—in a CSV file format, as Table 1 demonstrates. The timestamps were transferred in Matlab or other tools, where the bio-signal processing and observation took place. This step enables the synchronization of the quantitative and qualitative data, as well as the flexible move from one data-type to another for transferring and exchanging information.

Table 1. Times of main sections in milliseconds

Episodes localization through heart activity

After having specified the times and duration of each section, it is now possible to parameterize the bio-signals based on the section they belong. The ECG data of P1 is displayed in the upper plot of Figure 5. The signal was processed and plotted in Matlab R2018b. Raw ECG data were streamed from the Heartypatch sensor and saved on a Python interface developed by our team. The data were then filtered using a Butterworth IIR band pass filter for 5–30 Hz at second order, to remove any baseline wandering and to accurately detect the R-peaks. The range of the band pass filter was selected based on the fact that a QRS complex is present in the frequency range of 5–30 Hz. Figure 5 displays data from the “Introduction” section and its specific subsections “Walk and questions,” “Maps,” “Project connections,” and “How much do you know Eleusis” as an example of data color-mapped based on the corresponding activity section they belong. In the figure, the first approximately 5 min colored black are the data collected before the Pilot Sociodrama session started. The zero values were added to synchronized ECG with the video data of the main camera that was used as a point of reference to sync the data from the different equipment pieces. Therefore, at time 0 min the main camera started recording, the different equipment pieces followed up, at 4.17 min time the Heartypatch sensor started recording and the pilot session started at 4.50 min.

Figure 5. Segmentation of filtered ECG signal and the corresponding heart-rate based on the timestamps extracted from the ELAN software.

Then, the heart-rate is estimated by extracting the peaks (or beats) from the ECG signal and by counting the beats that happened within the past 1 min. This estimation repeats each time a new beat is detected. The outcome of the introductory section is presented in the lower plot of Figure 5.

Observing the HR fluctuations, it is easy to notice that there are specific time-zones during which high heart-rate values appear. For example, during the task “How much do you know Eleusis?” around 18–19 min, we can see the repetitive presence of high heart-rate values within the task. Therefore, the maximum heart-rate value was extracted for each task and its corresponding first and last time of appearance. The outcomes of these estimation are presented in Figure 6, where the blue vertical line defines the first and last appearance of max heart-rate value and the shaded area in between the duration of the time-zones (Table 2).

Table 2. Timestamps of the heart activity episodes extracted based on the max heart-rate value

Note. The table displays the start-time, end-time, and the duration of the episodes.

Figure 6. Visualization of the volume and time of the first and last max heart-rate value (blue lines) for each subsection. The shaded area between these two line indicates the duration of the occurrence.

The extraction of the times of these episodes, presented in Table 2, enable the transfer of the information to ELAN in order to annotate the media data (video and audio) and observe the physical activity that was taking place between the participants during and around these time-zones.

In this work, episodes refer to any specific pattern or outstanding behaviors that can be noted in either bio-signals or media data. By extracting their time and duration, we can then move to a different data type and observe what is happening during, before and/or after the episode in other data-types. Figure 6 and Table 2 demonstrate heart activity based episodes and specifically the time-zones that are characterized by maximum heart-rate values. In the following section of the manuscript, we will observe in ELAN the physical activities that are taking place before and during these episodes.

Observing the physical activity of heart extracted episodes in video data

By extracting the time-stamps of the heart episodes, it was possible to go back to ELAN and annotate the media files with the episodes. Then, we were able to observe in the media files what was happening during the time-zones of high heart-rate values but also what were the events before that caused the heart-rate increase. Thus, below the description of “before” and “during” each episode are listed.

1. A. Walk and questions

   Before: Starting at 00:06:45 the sociodramatist asked P1’s research team how many members were in their team and how many were missing. Figure 7 presents a video-frame of this interaction. During: The team started talking about the members and their participation.

   

   Figure 7. (a) P1 (indicated by a yellow dot) is with her research team and they communicate with the sociodramatist (indicated by a green dot) the participation of their team a few seconds before the high heart-rate is reached. P2 (indicated by an orange dot) is also part of the same research team. (b) P1 and P2 are touching their advisor and communicating with the sociodramatist before the high heart-rate (episode) of “Project connection” is reached.

2. B. Maps

   Before: P1 is walking around the room to finally move toward the place of her origin.

   During: P1 places herself on the imaginary map at the very first millisecond of the episode and stands there for the remaining time.

   Note: In this task, three mapping tasks were given (a) “where do you live?” (b) “where is the place of your origin?” (c) “what is the place that has impacted you culturally?”. Before each task was given, participants had to walk around in the room. Therefore, in this subsection we observe three main fluctuations of the HR.

3. C. Project connection

   Before: P1 is again with her research team and touching her advisor—as all participants were directed to touch the person who connected them with the project. Sociodramatist asks P1 and P2 about their connection to the project, they both confirm it was their common advisor. “Would you like to say something to your advisor?,” the sociodramatist asks. P1 and P2 pause for a few seconds, then they both agree that they want to thank her. “Would you like to respond to them?” the sociodramatist asks the advisor. “I want to say thank you as well,” the advisor responds and they all burst out laughing at 00:16:36.000. Figure 7 demonstrates a video-frame of this interaction.

   After: Another research team talks about who connected them to the project and how they joined.

4. D. How much do you know Eleusis?

   Before: All participants are walking in the room.

   After: Sociodramatist gives the task “I want you stand in a distance from Eleusis (represented by a book in the center of the room) that demonstrates how much you know the city.”

In two out of four cases, the HR increase seems to be connected with the walking task that was executed right before the max HR values are reached. For the other two subsections, “Walk and questions” and “Project connection,” that were characterized by the observers as emotional periods, the start and end time of activity that took place before the episodes were extracted (see Table 3). In the next section, the audio data of these two sections were investigated to “see” beyond what we could hear.

Table 3. Parts of the video empirically identified as before-episode activity

Observing the heart extracted before-episodes in audio data

From the audio files, the before-episode parts were exported and the spectrograms of those were further studied (see Figures 8 and 9). The spectrograms were generated with iZotope RX 7 Audio Editor software and visualize three parameters: time, loudness, and pitch. Time is measured in seconds and corresponds to the X-axis, loudness is measured in Decibels (dB) and corresponds to the Y-axis, and pitch is measured in Hertz (Hz) using the Mel scale and is represented by the color range. Lower frequencies are visualized in blue colors, mid frequencies in green colors and higher frequencies in red colors.

Figure 8. Spectrogram of the before-episode audio recording of subsection “Walk and questions.”

Figure 9. Spectrogram of the before-episode audio of subsection “Project connection.” At 23” a collective burst-out of laughter corresponds to a significant increase in loudness and pitch. From 9 to 14 s, there is a pause in conversation.

Figures 8 and 9 demonstrate an overall fluctuation in loudness and pitch during discussion. Specifically, Figure 8 demonstrates a continuous conversation without any interruptions. In contrast, Figure 9 demonstrates conversation with interruptions. In particular, in the period 9–14 s, we observe the pause, in the segments with the absence of magenta color, during the conversation between sociodramatist, P1, and P2 that was described in the previous section. Also at second 23, we observe a significant increase in loudness and pitch that corresponds to a collective burst-out of laughter and is visualized by the density of different colors.

Episodes localization through audio data

As mentioned earlier, in parallel with using HR as an index for the selection of episodes for further study, we adopted an alternative strategy of using audio data for the same purpose. This is part of a preliminary investigation of possible modes of selecting data segments, that we call “episodes,” for further synchronization and analysis, in order to determine which data modes bear more fruitful results. The audio data of the whole pilot session is used for segmentation and feature extraction. It is important to note that the audio data comes from the group sounds, in contrast to the heart data that corresponds to P1. So in this section the episodes are extracted not from a personalized but from a global measurement.

Figure 10 demonstrates the group audio data of the whole pilot session and the estimated root mean square (RMS) value of the audio volume of each subsection. It is noticeable that the “loudest” subsection is the third round of role playing, reaching as high as 0.1758. Role playing 1 and 2 and “the circle of trust” follow with approximately similar values (0.1259, 0.1266, and 0.1294 respectively). It is also important to observe the subsections with minimum values. In this case, first comes “How much do you know Eleusis?” with an RMS level of 0.0803, then follows “Walk and questions” (RMS = 0.0864) and “Walk and talk to Eleusis” (RMS = 0.0869).

Figure 10. The audio data of the whole pilot sociodrama session, colored based on main section segmentation (bottom graph). The top graph demonstrates the root mean square (RMS) of audio volume (y-value) for each subsection in sequence, whereas the x-value corresponds to the time that the subsection begins. These time values are highlighted with black lines in both bottom and top graphs. The text above the dots indicates the name of the subsection that the value corresponds to.

In the next section, we will define as audio based episodes the min and max RMS volume as extracted by the pilot session. As a result, through ELAN video annotation, we will observe the corresponding sections “How much do you know about Eleusis?” and “Role 3.”

Observing the physical activity of audio extracted episodes in video data

After extracting the audio based episodes, we move back to ELAN to observe what is happening in each of the two subsections.

1. A. “Role 3”—the loudest subsection Subgroup 1 is the European Commission (P1 and P2 are members of this sub-group). Initially they hold an internal meeting to decide about the European Capital of Culture. They decide for “Eleusis.” Then, P1 announces the news to Eleusinians, sub-group 2. The latter make a lot of noise and applaud to celebrate the event. One of the sub-group members enthusiastically announces the news to the rest of the Greeks, sub-group 3 (Figure 11a). But they do not respond as expected; some members feel that their cities are degraded, some others cannot believe their ears. The European Commission sub-group is surprised: “Why are those people complaining?”. Disagreements expand among participants and between the subgroups (Figure 11b).

   

   Figure 11. Moments from “Role 3.”

2. B. “How much do you know about Eleusis?”—the quietest subsections In this subsection, the participants are directed to move to a distance from Eleusis (represented by a book placed in the center of the room) depending on how much they know about the city. The more they know the closer to the city they can move. Then, the sociodramatist asks each participant to express briefly what they would like to know about Eleusis. One after the other participants respond to this call. Some participants become very emotional when it’s their turn. For some of them Eleusis was their birthplace, for others the city of residence, but not for all of them.