Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-745bb68f8f-f46jp Total loading time: 0 Render date: 2025-01-18T14:08:44.903Z Has data issue: false hasContentIssue false

10 - Embodiment and expressive communication on the internet

Published online by Cambridge University Press:  05 June 2012

By
Elisabeth Oberzaucher ,
Karl Grammer  and
Susanne Schmehl
Edited by
Arvid Kappas  and
Nicole C. Krämer
Elisabeth Oberzaucher
Affiliation:
University of Vienna, Austria
Karl Grammer
Affiliation:
University of Vienna, Austria
Susanne Schmehl
Affiliation:
University of Vienna, Austria
Arvid Kappas
Affiliation:
Jacobs University Bremen
Nicole C. Krämer
Affiliation:
Universität Duisburg–Essen
Get access

Summary

Overview: Human brains are basically social, and use communication mechanisms that have evolved during our evolutionary past. Thus, we suggest that even in communication with and by machines, humans will tend to react socially and use communication mechanisms that are primarily social and embodied. One of these mechanisms is communicative feedback, which refers to unobtrusive (usually short) vocal or bodily expressions, whereby a recipient of information can inform a contributor of information about whether he or she is able and willing to communicate, perceive the information, and understand the information. We will show how feedback can be modeled in virtual agents on facial expressions of a virtual agent or verbot and thus contribute to human–human communication over the internet. We will present a simple model based on a pleasure, arousal, and dominance space, which allows a complex stimulus generation program to be driven with only a few parameters.

Humans are social – but what about human–machine communication?

Internet communication consists of two major domains: communication with a machine and human–human communication through a machine. Both processes involve different but comparable elements in order to be efficient, as we will outline here.

In its early years, the internet was used by a rather small group of scientists for communication via email and bulletin boards. As compared to phone calls and direct face-to-face communication, it seemed to be missing a social component, thus leading to the introduction of emoticons such as the well-known smiley, which constituted a first attempt to fill this gap.

Type
Chapter
Information
Face-to-Face Communication over the Internet
Emotions in a Web of Culture, Language, and Technology
 , pp. 237 - 279
Publisher: Cambridge University Press
Print publication year: 2011

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Allwood, J., Nivre, J., and Ahlsén, E. (1992). On the semantics and pragmatics of linguistic feedback. Journal of Semantics, 9(1), 1–26.CrossRefGoogle Scholar
Allwood, J., Kopp, S., Grammer, K., Ahlsén, E., Oberzaucher, E., and Koppensteiner, M. (2008). The analysis of embodied communicative feedback in multimodal corpora – a prerequisite for behavior simulation. Journal of Language Resources and Evaluation (Special Issue on Multimodal Corpora), 41(3–4), 255–272.CrossRefGoogle Scholar
Becker, C., Kopp, S., and Wachsmuth, I. (2004). Simulating the emotion dynamics of a multimodal conversational agent. In André, E. (ed.), Proceedings of Affective Dialogue Systems Conference (ADS '04), LNAI 3068 (pp. 154–165). Berlin: Springer.CrossRefGoogle Scholar
Beun, R. J. and Eijk, R. M. (2004). A cooperative dialogue game for resolving ontological discrepancies. In Dignum, F., , F. (ed.), Advances in Agent Communication (pp. 349–363). Berlin: Springer.CrossRefGoogle Scholar
Bulbulia, J. (2004). Religious costs as adaptations that signal altruistic intention. Evolution and Cognition, 10(1), 19–42.Google Scholar
Cassell, J. (2004). Towards a model of technology and literacy development: story listening systems. Journal of Applied Developmental Psychology, 25(1), 75–105.CrossRefGoogle Scholar
Cassell, J. and Thórisson, K. R. (1999). The power of a nod and a glance: envelope vs. emotional feedback in animated conversational agents. Applied Artificial Intelligence, 13, 519–538.CrossRefGoogle Scholar
Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., and Yan, H. (1999). Embodiment in conversational interfaces: Rea. In CHI '99 Conference Proceedings (pp. 520–527). New York: ACM Press.Google Scholar
Cassell, J., Vilhjálmsson, H., and Bickmore, T. (2001). BEAT: the Behavior Expression Animation Toolkit. In Proceedings of SIGGRAPH '01 (pp. 477–486). New York: ACM Press.CrossRefGoogle Scholar
Cleveland, W. S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74, 829–836.CrossRefGoogle Scholar
Cleveland, W. S. and Devlin, S. J. (1988). Locally weighted regression: an approach to regression analysis by local fitting. Journal of the American Statistical Association, 83, 596–610.CrossRefGoogle Scholar
Cosmides, L. and Tooby, J. (1992). Cognitive adaptations for social exchange. In Barkow, J., Cosmides, L., and Tooby, J. (eds), The Adapted Mind: Evolutionary Psychology and the Generation of Culture (pp. 163–228). New York: Oxford University Press.Google Scholar
Cosmides, L., Tooby, J., and Barkow, J. H. (1992). Evolutionary Psychology and Conceptual Integration. Oxford University Press.Google Scholar
Coulson, M. (2004). Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior, 28, 117–139.CrossRefGoogle Scholar
Darwin, C. (1872). The Expression of the Emotions in Man and Animals. New York: D. Appleton and Company.CrossRefGoogle Scholar
Dawkins, R. and Krebs, J. R. (1978). Animal signals: information or manipulation? In Krebs, J. R. and Davies, N. B. (eds), Behavioural Ecology: An Evolutionary Approach (pp. 282–309). Sunderland, MA: Sinauer.Google Scholar
Rosis, F., Pelachaud, C., Poggi, I., Carofiglio, V., and Carolis, B. (2003). From Greta's mind to her face: modeling the dynamics of affective states in a conversational embodied agent. International Journal of Human–Computer Studies, 59, 81–118.CrossRefGoogle Scholar
Ehlich, K. (1986). Interjektionen. Tübingen: Niemeyer.CrossRefGoogle Scholar
Eibl-Eibesfeldt, I. and Hass, H. (1967). Neue Wege der Humanethologie[New Ways in Human Ethology], Homo, 18, 13–23.Google Scholar
Ekman, P. (1969). The repertoire of nonverbal behavior – categories, origins, usage and coding. Semiotica, 1, 49–98.CrossRefGoogle Scholar
Ekman, P. (1971). Universals and cultural differences in facial expressions of emotion. Nebraska Symposium on Motivation, 19, 207–283.Google Scholar
Ekman, P. (1980). Biological and cultural contributions to body and facial movement in the expression of emotions. In Rorty, A. O. (ed.), Explaining Emotions (pp. 73–101). Berkeley, CA: University of California Press.Google Scholar
Ekman, P. (1984). Expression and the nature of emotion. In Scherer, K. and Ekman, P. (eds), Approaches to Emotion (pp. 319–343). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Ekman, P. (1994). Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. Psychological Bulletin, 115, 268–287.CrossRefGoogle ScholarPubMed
Ekman, P. and Friesen, W. V. (1978). Investigator's Guide: Facial Action Coding System. Palo Alto, CA: Consulting Psychologists Press.Google Scholar
Fahlmann, S. E. (2007). Smiley lore :-). www.cs.cmu.edu/~sef/sefSmiley.htm.
Fridlund, A. J. (1991). Sociality and solitary smiling: potentiation by an implicit audience. Journal of Personality and Social Psychology, 60, 229–240.CrossRefGoogle Scholar
Fridlund, A. J. (1994). Human facial expression: An Evolutionary View. San Diego, CA: Academic Press.Google Scholar
Fried, I. (2007). Warning sounded over ‘flirting robots.’ www.news.com/beyond-binary/8301-13860_3-9831133-56.html.
Frijda, N. H. (1986). The Emotions. Cambridge University Press.Google Scholar
Fujie, S., Fukushima, K., and Kobayashi, T. (2004). A conversation robot with back-channel feedback function based on linguistic and nonlinguistic information. In Proceedings of the International Conference on Autonomous Robots and Agents (pp. 379–384).Google Scholar
Gallese, V. and Goldmann, A. (1998). Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Science, 2, 493–501.CrossRefGoogle ScholarPubMed
Gillenson, M. L. (1974). The Interactive Generation of Facial Images on a CRT Using a Heuristic Strategy. Ohio State University, Computer Graphics Research Group.Google Scholar
Grammer, K., Schiefenhovel, W., Schleidt, M., Lorenz, B., and Eibl-Eibesfeldt, I. (1988). Patterns on the face: the eyebrow flash in crosscultural comparison. Ethology, 77, 279–299.CrossRefGoogle Scholar
Grammer, K. and Fieder, M. (1997). A neural network approach for the classification of body movements (Abstract). In Schmitt, A., Atzwanger, K., Grammer, K., and Schäfer, K. (eds), New Aspects of Human Ethology (pp. 202–203). New York: Plenum Press.Google Scholar
Grammer, K., Filova, V., and Fieder, M. (1997). The communication paradox and possible solutions: towards a radical empiricism. In Schmitt, A., Atzwanger, K., Grammer, K., and Schäfer, K. (eds), New Aspects of Human Ethology (pp. 91–120). New York: Plenum Press.CrossRefGoogle Scholar
Grammer, K., Fink, B., and Renninger, L. (2002). Dynamic systems and inferential information processing in human communication. Neuro Endocrinological Letters (Special Issue), 23(4), 15–22.Google ScholarPubMed
Grammer, K., Fink, B., Møller, A. P., and Thornhill, R. (2003). Darwinian aesthetics: sexual selection and the biology of beauty. Biological Reviews, 78(3), 385–340.CrossRefGoogle ScholarPubMed
Grammer, K., Fink, B., Oberzaucher, E., Atzmueller, M., Blantar, I., and Mitteroecker, P. (2004). The representation of self-reported affect in body posture and body posture simulation. Collegium Anthropologicum, 28(2), 159–173.Google ScholarPubMed
Grammer, K. and Oberzaucher, E. (2006). The reconstruction of facial expressions in embodied systems: new approaches to an old problem. ZIF Mitteilungen, 2, 14–31.Google Scholar
Griffiths, P. E. (1990). Modularity and the psychoevolutionary theory of emotion. Biology & Philosophy, 5, 175–196.CrossRefGoogle Scholar
Guthrie, S. (1993). Faces in the Clouds: A New Theory of Religion. Oxford University Press.Google Scholar
Hess, U., Adams, R. S., and Kleck, R. E. (2004). Facial appearance, gender, and emotion expression. Emotion, 4, 378–388.CrossRefGoogle ScholarPubMed
Houck, N. and Gass, S. M. (1997). Cross-cultural back channels in English refusals: a source of trouble. In Jaworski, A. (ed.), Silence – Interdisciplinary Perspectives (pp. 285–308). Berlin: Mouton de Gruyter.Google Scholar
Izard, C. E. (1971). The Face of Emotion. New York: Appleton-Century-Crofts.Google Scholar
Izard, C. E. (1977). Human Emotions. New York: Plenum.CrossRefGoogle Scholar
Izard, C. E. (1991). The Psychology of Emotions. New York: Plenum.CrossRefGoogle Scholar
James, W. (1950 [1890]). The Principles of Psychology. New York: Dover.Google Scholar
Johansson, G. (1973). Visual perception of biological motion and a model of its analysis. Perception & Psychophysics, 14, 201–211.CrossRefGoogle Scholar
Johansson, G. (1976). Spatio-temporal differentiation and integration in visual motion perception. Psychological Research, 38, 379–398.CrossRefGoogle ScholarPubMed
Kipp, M. (2001). Anvil – a generic annotation tool for multimodal dialogue. In Proceedings of Eurospeech 2001, Aalborg (pp. 1367–1370).Google Scholar
Kleinginna, P. R. and Kleinginna, A. M. (1981). A categorized list of emotion definitions with suggestions for a consensual definition. Motivation and Emotion, 5, 345–379.CrossRefGoogle Scholar
Kopp, S., Gesellensetter, L., Krämer, N.C., and Wachsmuth, I. (2005). A conversational agent as a museum guide. Design and evaluation of a real-world application. In Panayiotopoulos, T.et al. (eds), Intelligent Virtual Agents 2005 (pp. 329–343). Hamburg: Springer.CrossRefGoogle Scholar
Kopp, S., Allwood, J., Ahlsén, E., Grammer, K., and Stocksmeier, T. (2008). Modeling embodied feedback with virtual humans. In Wachsmuth, I. and Knoblich, G. (eds), Modeling Communication with Robots and Virtual Humans (pp. 18–37). Berlin: Springer.CrossRefGoogle Scholar
Krämer, N. C. (2001). Bewegende Bewegung. Sozio-emotionale Wirkungen nonverbalen Verhaltens und deren experimentelle Untersuchung mittels Computeranimation. [Moving Movements. Socio-emotional Effects of Nonverbal Behavior and Their Experimental Analysis by Means of Computer Animation] Lengerich: Pabst Science Publishers.Google Scholar
Krebs, J. R. and Dawkins, R. (1984). Animal signals: mind reading and manipulation. In Krebs, J. R. and Davies, N. B. (eds), Behavioural Ecology: An Evolutionary Approach (pp. 380–402). Oxford: Blackwell Scientific.Google Scholar
Krumhuber, E. and Kappas, A. (2005). Moving smiles: the role of dynamic components for the perception of the genuineness of smiles. Journal of Nonverbal Behavior, 29, 3–24.CrossRefGoogle Scholar
Lang, P. J. (1995). The emotion probe. American Psychologist, 50, 372–385.CrossRefGoogle ScholarPubMed
Lazarus, R. S. (1991). Emotion and Adaption. New York: Oxford University Press.Google Scholar
Lee, E., Kang, J. I., Park, I. H., Kim, J.-J., and An, S. K. (2008). Is a neutral face really evaluated as being emotionally neutral?Psychiatry Research, 157, 77–85.CrossRefGoogle ScholarPubMed
Mehrabian, A. and Russell, J. A. (1974). An Approach to Environmental Psychology. Cambridge, MA: MIT Press.Google Scholar
,Microsoft (2003). Microsoft Agent product information. (www.microsoft.com/msagent/prodinfo/).
Moser, E., Derntl, B., Robinson, S., Fink, B., Gur, R. C., and Grammer, K. (2006). Amygdala activation at 3T in response to human and avatar facial expressions of emotions. Journal of Neuroscience Methods, 161(1), 126–133.CrossRefGoogle ScholarPubMed
Ortony, A. and Turner, T. J. (1990). What's basic about basic emotions?Psychological Review, 97, 315–331.CrossRefGoogle ScholarPubMed
Osgood, C. E. (1966). Dimensionality of the semantic space for communication via facial expressions. Scandinavian Journal of Psychology, 7, 1–30.CrossRefGoogle ScholarPubMed
Panksepp, J. (1992). A critical role for “affective neuroscience” in resolving what is basic about basic emotions. Psychological Review, 99(3), 554–560.CrossRefGoogle ScholarPubMed
Parke, F. I. (1972). Computer-generated animation of faces. Proceedings of ACM National Conference (vol. I, pp. 451–457).CrossRef
Petta, P., Staller, A., Trappl, R., Mantler, S., Szalavari, Z., Psik, T., and Gervautz, M. (1999). Towards engaging full-body interaction. In Bullinger, H.-J. and Vossen, P. H. (eds), Adjunct Conference Proceedings, HCI International '99, 8th International Conference on Human–Computer Interaction (pp. 280–281). Stuttgart: Fraunhofer IRB Verlag.Google Scholar
Platt, S. M. and Badler, N. (1981). Animating facial expression. Computer Graphics, 15(3), 245–252.CrossRefGoogle Scholar
Poggi, I., Pelachaud, C., Rosis, F., Carofiglio, V., and Carolis, B. (2005). GRETA. A believable embodied conversational agent. In Stock, O. and Zancarano, M. (eds), Multimodal Intelligent Information Presentation. (pp. 3–25). Dordrecht: Kluwer.CrossRefGoogle Scholar
Reeves, B. and Nass, C. (1996). The Media Equation. How People Treat Computers, Television, and New Media like Real People and Places. New York: Cambridge University Press.Google Scholar
Russell, J. A. (1978). Evidence of convergent validity on the dimensions of affect. Journal of Personality and Social Psychology, 36, 1152–1168.CrossRefGoogle Scholar
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178.CrossRefGoogle Scholar
Russell, J. A. (1991). In defense of a prototype approach to emotion concepts. Journal of Personality and Social Psychology, 60, 425–438.CrossRefGoogle Scholar
Russell, J. A. (1995). Facial expression of emotion: what lies beyond minimal universality?Psychological Bulletin, 118, 379–391.CrossRefGoogle ScholarPubMed
Russell, J. A. and Mehrabian, A. (1977). Evidence for a three-factor theory of emotions. Journal of Research in Personality, 11, 273–294.CrossRefGoogle Scholar
Scherer, K. R. (1984). On the nature and function of emotion: a component process approach. In Scherer, K. R. and Ekman, P. (eds), Approaches to Emotion (pp. 293–317). Hillsdale, NJ: Erlbaum.Google Scholar
Scherer, K. R. (1994). Affect bursts. In Goozen, S., Poll, N. E., and Sergeant, J. A. (eds), Emotions: Essays on Emotion Theory (pp. 161–193). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Scherer, K. R. (1997). Profiles of emotion-antecedent appraisal: testing theoretical predictions across cultures. Cognition and Emotion, 11, 113–150.CrossRefGoogle Scholar
Scherer, K. R. (1999). Appraisal theory. In Dalgleish, T. and Power, M. J. (eds), Handbook of Emotion and Cognition (pp. 637–663). New York: Wiley.Google Scholar
Scherer, K. R. (2001). Appraisal considered as a process of multi-level sequential checking. In Scherer, R., Schorr, A., and Johnstone, T. (eds), Appraisal Processes in Emotion: Theory, Methods, Research (pp. 92–120). New York: Oxford University Press.Google Scholar
Schmidt, K. L. and Cohen, J. F. (2002). Human facial expressions as adaptations: evolutionary questions in facial expression research. Yearbook of Physical Anthropology, 44, 3–24.Google Scholar
Shneiderman, B. (1989). Social and individual impact. Educational Media International, 26(2), 101–106.CrossRefGoogle Scholar
Smith, C. A. and Scott, H. S. (1997). Spontaneous facial behavior during intense emotional episodes: artistic truth and optical truth. In Russell, J. A. and Fernández-Dols, J. M. (eds), The Psychology of Facial Expression (pp. 229–254). Cambridge University Press.CrossRefGoogle Scholar
Snodgrass, J. (1992). Judgment of Feeling States from Facial Behavior: A Bottom-up Approach. Unpublished doctoral dissertation, University of British Columbia.Google Scholar
Spencer-Smith, J., Wild, H., Innes-Ker, A. H., Townsend, J. T., Duffy, C., Edwards, C., Ervin, K., Merritt, N., and Paik, J. W. (2001). Making faces: creating three-dimensional parameterized models of facial expression. Behavior Research Methods, Instruments and Computers, 33, 115–123.CrossRefGoogle ScholarPubMed
Staller, A. and Petta, P. (1998). Towards a tractable appraisal-based architecture for situated cognizers. In Canamero, D., Numaoka, C., and Petta, P. (eds), Grounding Emotions in Adaptive Systems, Workshop Notes, 5th International Conference of the Society for Adaptive Behavior (pp. 56–61). Zurich: Society for Adaptive Behavior.Google Scholar
Takeuchi, M., Kitaoka, N., and Nakagawa, S. (2004). Timing detection for realtime dialog systems using prosodic and linguistic information. In Proceedings of the International Conference on Speech Prosody (pp. 529–532).
Thórisson, K. R. (1996). Communicative Humanoids – A Computational Model of Psychological Dialogue Skills. PhD thesis, School of Architecture and Planning, Massachusetts Institute of Technology.Google Scholar
Tomkins, S. S. (1962). Affect Imagery Consciousness: Vol. I. The Positive Affects. New York: Springer.Google Scholar
Ward, N. and Tsukahara, W. (2000). Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics, 32(8), 1177–1207.CrossRefGoogle Scholar
Watson, D., Clark, L. A., and Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063–1070.CrossRefGoogle ScholarPubMed
Wehrle, T. and Scherer, K. R. (2001). Towards computational modeling of appraisal theories. In Scherer, K. R., Schorr, A., and Johnstone, T. (eds), Appraisal Processes in Emotion: Theory, Methods, Research (pp. 350–365). New York: Oxford University Press.Google Scholar
Wiener, N. (1984). Cybernetics and Control and Communication in the Animal and the Machine. Cambridge, MA: MIT Press.Google Scholar
Woodworth, R. S. (1938). Experimental Psychology. New York: Holt.Google Scholar
Wundt, W. (1924 [1912]). An Introduction to Psychology (Pinter, R., Trans.). London: Allen and Unwin.Google Scholar
Xiao, J. (2001). Understanding the Use and Utility of Anthropomorphic Interface Agents. Student Poster at the CHI 2001 in Seattle, WA, USA.CrossRefGoogle Scholar
Yngve, V. H. (1970). On getting a word in edgewise. In Papers from the Sixth Regional Meeting of the Chicago Linguistics Society (pp. 567–578). Chicago: Chicago Linguistics Society.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×