Several other experiments have been conducted in fMRI scanners while subjects were shown videos of real and digital expressions revealing minor differences in certain types of brain activity. The study by Lorena C Kegel et al (fig. 1) suggest that different levels of face realism elicit different neural responses. The more real, the stronger the effect. Avatars deliver smaller brain responses, however the researchers speculate that “we should consider that not empathy but rather processes related to emotional intelligence may account for the observed variance in brain responses and rating differences between human and avatar faces. It is possible that our lack of expertise with computer-generated faces is associated with lower measures of emotional intelligence regarding facial expressions of avatars.” This implies that once we get to know more avatars we will learn how to read their emotions.
Amy Baylor has conducted experiments showing that the presence of facial expressions facilitated learning when it was not associated with gesture. Adding further complication, “gesture deteriorated the effectiveness of attitudinal instruction while enhancing the effectiveness of the procedural instruction.” Meanwhile, Tze Wei Liew observed that an avatar’s smile can be perceived as masking negative emotions. Adding, “the social interpretation of an agent’s smile as “dominant” and “fake” produces negative impacts on emotion and motivation in virtual learning systems.” I would argue that these contradictory results derive from the complex way in which we process non-verbal visual cues. The subject knows (as shown by the variation in brain activity in the fMRI scans) that this entity is virtual and that its expressions differ from those of a real human, so why is it smiling at me? What does that smile mean? At this point we find ourselves in the uncanny valley; the complication of the emotional response to an avatar generated by subconscious psychological projections.
Race, gender, coolness
Baylor and others have explored preferences for and the effectiveness of avatars of different apparent ethnicity and gender. Referencing multiple studies, she shows that subjects are more influenced by an agent of the same gender and same ethnicity/race.
It is not difficult to understand circumstances in which people would prefer an avatar that reflects their own ethnicity. A Hispanic American, for example, finding that a medical information system has an avatar with Hispanic features, may be reassured that the information is relevant to them. Indeed, if we see the avatar as representing a form of power (the teacher, the doctor, the librarian, the state, the status quo, conformity) then the user may suspect it of being a form of oppression or control that may reproduce and reinforce the social/political hierarchy. Giving the avatar an appearance similar to the user may be at least a nod towards attempting to recognise and reflect the user.
Partnering in multiple experiments around race and gender, Baylor’s work reveals that subjects bring their prior stereotypes and expectations to bare on how they value avatars. Her results are sometimes contradictory and I suspect this is because they reflect the contradictions in our own prejudices and stereotypes.
A 2009 experiment by Rosenberg and Kima, used two separate sets of undergraduate students enrolled in an introductory technology course; 80 African American and 39 white. The role of the avatar was to encourage these young women to pursue careers in engineering by targeting women’s negative beliefs about engineering and their lack of confidence in their own abilities. A previous study by the same researchers found that female agents were the most effective for influencing young women’s stereotypes about engineering. Similarly, in this experiment, for all students the female avatar of the same skin colour was most effective. However, for all students the second most effective was the white male avatar. This led the researchers to suspect that the students may hold a stereotype of engineers as white and male. This is confirmed in another experiment by Rosenberg and Kima where students selected “male, older, uncool” avatars (from a selection of 16 avatars) as “most like engineers.” Baylor writes, “They tended to choose to ‘learn about engineering’ from agents who were male and attractive, but uncool.”
This is where I bring my experience from a former life as a writer, producer and director in television. The students here are being asked to do their own casting. The experiment says, “cast me an engineering teacher.” If they enrolled in an engineering class they probably wouldn’t get to choose the teacher. They have fallen back on a stereotype of authority in engineering. This is not surprising, we all carry many deeply ingrained social stereotypes. I think film and TV producers can show us the way here. Those same students would probably have no problem in believing in the authority of: Kate Winslet as a CDC epidemiologist in Contagion, Natalie Portman playing a cellular-biology professor in Annihilation or Sandra Bullock as space-faring biomedical engineer almost lost in space in Gravity. None of those films would have been improved by the casting of uncool older men.
How should designers of relational agents decide on the gender and ethnicity of the avatar? My argument is to do what is best for the role. Your avatar is not just a fact deliverer it is a fictional character. Let’s imagine that we are going to create an avatar to train engineering students in electronics. If you have the budget to let the students customise their avatar’s appearance, that is fine, they may, as Amy Baylor suggests, choose one that looks like them. And the research suggests this choice will improve their results in class. But if you can’t afford to offer multiple avatars your task is to create a plausible character. My real-life electronics lecture at college was an older, uncool, white man in his 50’s with a thick Bristolian accent. He knew his stuff (I got a distinction) but he was strict and a hard task master. My colleague at UWE who teaches hardware hacking (aka electronics) is white, in her late 20’s with a brown bob and a hipster vibe. If I were making an electronics training system with an avatar, even if the majority of the students were male I’d probably make it closer to her as she knows as much as my college tutor, but she is more creative and more emotionally engaged with her students. As we’ve seen people respond well to avatars that are similar to themselves. For electronics students, I believe, an avatar based on my colleague would be more relatable and aspirational. This female character will demonstrate her authority through her knowledge and command of a classroom. As a writer or when I am casting for a drama, I sometimes want to go with a stereotype and sometimes I want to go against it. This decision may be politically or socially motivated. Making my young female avatar black, Asian or any ethnicity doesn’t undermine the role because I don’t find any race more or less plausible as an electronics tutor. The demographics of the audience would help inform a decision on race, but, like a film or tv producer, I may have a more positive agenda in mind. I not only want to avoid reinforcing existing prejudices, but I may want to create an appropriate virtual role model.
Baylor A. L. (2009). Promoting motivation with virtual agents and avatars: role of visual presence and appearance. Philosophical transactions of the Royal Society of London. Series B, Biological sciences. 364(1535), 3559–3565. https://doi.org/10.1098/rstb.2009.0148
Boker, S.M., Cohn, J.F., Theobald, B., Matthews, I., Brick, T.R. & Spies, J.R. (2009) Effects of damping head movement and facial expression in dyadic conversation using real–time facial expression tracking and synthesized avatars. Philosophical Transactions of the Royal Society B: Biological Sciences. Vol. 364, no. 1535, pp. 3485-3495.
Heeter, C. (1992) Being There: The subjective experience of presence. Presence: Teleoperators and Virtual Environments. MIT Press.
Heeter C. (eds Biocca F., Levy M.) (1995) Communication research on consumer VR. In Communication in the age of virtual reality. Hillsdale, NJ, USA: Lawrence Erlbaum Associates
Kegel, L. Brugger, P. Frühholz, S. Grunwald, T. Hilfiker, P. Kohnen, O. Loertscher, M. Mersch, M. Rey, A. Sollfrank, T. Steiger, B. Sternagel, J. Weber, M. Jokeit, H. (2020) Dynamic human and avatar facial expressions elicit differential brain responses, Social Cognitive and Affective Neuroscience. Volume 15, Issue 3, March 2020, Pages 303–317, https://doi.org/10.1093/scan/nsaa039
Nam, Chang & Shu, Joseph & Chung, Donghun. (2008). The Roles of Sensory Modalities in Collaborative Virtual Environments (CVEs). Computers in Human Behavior. 24. 1404-1417. 10.1016/j.chb.2007.07.014.
Tze, W.L., Nor Azan, M.Z., Sahari, N. & Su-Mae, T. (2016) The Effects of a Pedagogical Agent’s Smiling Expression on the Learner’s Emotions and Motivation in a Virtual Learning Environment. International Review of Research in Open and Distributed Learning, vol. 17, no. 5.