Matias Volonte

Human Centered Computing PhD candidate, Clemson University

Download PDF

Virtual humans have the potential to revolutionize human-computer interaction since they could be used as interfaces for social or collaborative scenarios. It is predicted that in the near future, virtual humans will possess a degree of animation and look that will make them almost indistinguishable from humans. Proof of this statement is the work presented by Facebook’s codec Avatars (Lombardi et al., 2018) or the BabyX project (Sagar et al., 2015). These virtual humans show a behavioral and appearance fidelity that is close to overpass the uncanny feeling that sometimes photorealistic virtual humans can elicit in users (Mori, M., et al, 1970).

Despite these technological breakthroughs, I believe that is important to answer, what is the impact of animation and rendering fidelity on users’ emotions and visual attention? Is high fidelity animation and photorealistic appearance mandatory for virtual agents during interaction with humans? Understanding the emotional impact on users during interaction with digital humans it is primordial in learning systems since emotion influences learning results (Qi, Dunsworth. Et al. 2007). Memory and learning outcomes are improved if users are emotionally affected by the stimulus. Furthermore, understanding users’ gaze mechanisms during interaction with conversational virtual agents can provide useful behavioral data. This is important especially in learning systems since studies reported that humans are mostly visual learners (Felder, R. M., 2022). If we want to have engaging and effective virtual humans in simulated inter-personal settings, we need to provide information and understand to the best of our capabilities, how a virtual human’s behavior and appearance affects the users’ emotions and visual attention.

For my DPA 307 Studio Methods for Digital Production class, I taught 3D Pipeline to undergraduate students. This course was taught entirely in Blender and exposed students to the basic’s methods and practices that a small-scale 3D production contains. These include basic organic and non-organic modeling, rigging and animation, rendering, and editing. For my CPSC 8070 - 3D Modeling and Animation course, I taught intermediate to advanced 3D concepts to graduate students. The class included advanced methods for producing a 3D short film focused on character animation. These were advanced character modeling, rigging, animation, camera and film theory, lighting, UV unwrapping, and rendering.

System description

For studying the impact of animation and rendering fidelities on users’ emotions and their visual attention, I utilized and co-developed a medical interactive virtual training simulator named Rapid Response Training system (RRTS). It was designed to train nurses in recognizing the signs and symptoms of rapid deterioration in patients (Dukes et al., 2012). This system is currently deployed and actively used at the St. Francis Hospital, Greenville South Carolina. The RRTS simulates the daily duties of a nurse including visiting patients typically four times a day, gathering their vital signs, and reporting them in a simulated electronic health record system. Pineapple In the RRTS, the virtual human’s health gradually declines and exhibits a noticeable affective negative behavior over the course of four time-steps. The patient’s health declination is shown in his verbal and non-verbal behaviors (facial expressions, body posture, gaze, etc.), vital signs, cognition, and appearance. The virtual human in the RRTS was modeled after real life patients who have undergone rapid deterioration over the course of a nurse’s shift. Based on this repository of real patient behaviors, the signs and symptoms of deterioration have been carefully modeled and animated in our virtual patients in a data driven manner, and evaluated by partnering medical experts. Finally, users can interact with the patient via a graphical user interface containing a predefined set of questions or by accessing medical tools in the simulation environment.

Virtual Human Animation and Rendering Fidelity Research

I specialized in studying how a virtual human’s animation and appearance fidelity affected users’ emotions and visual attention. To study this, I manipulated the rendering styles of a virtual patient that exhibited negative affective emotions, facial expressions, conversation and non-conversational behavioral cues.

On the one hand, I examined how the virtual patient’s negative behaviors affected users’ emotions and attentional gaze. Overall results revealed that the behaviors of the virtual patient (Bob) elicited higher visual attention from the users when he did not show acute symptoms of distress. However, when Bob’s negative affective behavior was very intense, users’ gaze shifted from the virtual human towards the medical tools and the graphical user interface. This type of selective attention has been reported in studies since visual attention can be used as a mechanism for coping with a stressful situation (Dirkin, G.R. et al., 1983). Also, we compared users’ gaze behavior during interaction with a virtual human that produced conversational behaviors with life-like animations and found that users gaze scores were higher when the virtual human did not engage in conversational processes. This suggests that users intended to extract further information from the virtual human gazing the non-verbal communication information provided from his body posture and facial expressions.

On the other hand, we compared the effect of the photo-realistic vs non-photorealistic rendering styles of a virtual patient on users’ emotion contagion and visual attention. The photorealistic virtual human possessed a realistic skin shader while the non-photorealistic conditions encompassed a cartoon and pencil shader style. Overall, findings suggest that users that interacted with a photorealistic virtual patient tended to notice facial expression cues to a greater extent. Interestingly this condition elicited emotions tied to social constructs such as shame and shyness (Leeming2003, Campbell, 2003). Also, during simulated dialogue, the realistic virtual humans elicited greater gaze towards them than the non-photorealistic conditions. This information suggests that photorealistic virtual humans should be implemented if users are expected to interact with the virtual humans in social simulated conversations.

Furthermore, we also found that high salience or foreground and background segregation could be used as an effective tool to elicit higher visual attention and higher rate of tool usage during interaction with virtual humans. Findings showed that users spent less time in the simulation but gazed more at the virtual human in conditions where it was segregated from the background via differences in the shading algorithms (Figure 2: PS, CT and HL). However, visual attention scores were lower when the virtual human blended with the background (FIG 2: APS and ACT). This information is important in order to manipulate users’ gaze towards certain objects in the scene, in this case, a virtual human.

Pineapple

Moreover, based on literature that states that emotion and attention are closely linked (Yamaguchi, S. et al., 2012), we conducted a pathway study analyzing the interplay between emotion and attention during interaction with Bob. Results showed that attention was affecting users’ emotions during the interaction with the patient. Results showed that attention and emotion were not significantly correlated when Bob showed high negative affective behaviors. In the pathway analysis, we discovered that when the virtual patient is normal, emotional reactions to virtual humans in the different fidelity conditions is mediated by visual attention. Certain rendering conditions of the virtual human attract attention, which then causes a change in emotional reactions of the users.

Finally, I also believe that the information gathered in my studies can clarify, to some extent, the relationship between visual attention and emotion during human-virtual human interaction. Moreover, this information can be used to control and develop efficient simulations. Additionally, virtual reality developers could use the guidelines discussed in my investigations for creating compelling and effective virtual training simulations that includes interaction with virtual humans.

My research interest is expanding to several areas. First, I am working towards my dissertation focused on investigating how human emotions and task performance are affected during interaction with emotional virtual agents in an immersive virtual crowd scenario. This study is novel because most of the examinations of the emotional effect is the result of an interaction between human and virtual humans is examined in a micro setup, this means a dyadic human-virtual human conversation. However, in my dissertation study we analyze emotion contagion in a macro setup since users will be interacting with a crowd of 70 emotional virtual humans. Additionally, the locomotion behavioral component of the virtual humans relies in artificial Intelligent (AI) algorithms.

Second, I am contributing to a study focused on collaboration between a human and a virtual human in an immersive virtual environment. In this study, in a joint task setup, a user embodied in an avatar system transports a box to a designated target area in collaboration with a virtual agent. In a within study design, in one of the phases, the virtual agent will be leading the task (agent is proactive), while in another, the roles will be inverted, and the human will guide the task and the agent reacts accordingly (agent is reactive). Finally, the behavioral reactions of the virtual agent collaborator are driven by AI. My dissertation and this investigation are in collaboration between National Chiao Tung University Taiwan and Clemson University.

I am currently organizing studies with Universities from Argentina, South America. I started contacting professors and arranging future meetings for collaborative research between me and institutions from this country. My initial project will examine and compare human-virtual human interaction in different cultural scenarios. Also, I will create an education system to teach users on how to speak English while collecting data to produce investigations. Finally, I intend to recruit and mentor students to join my research efforts and, increase minorities diversity and inclusion in the computer science field.

Funding philosophy

I plan to submit proposals to NSF and NSF Cyber Humans Division among other available research funding resources. My first goal is aimed to explore how virtual humans can be used as an interface for human-robot interaction. Overall, I plan to investigate how virtual humans can interface the communication between humans and robots. This project is planned to be conducted over multiple phases in a longitudinal period and includes both virtual reality and augmented reality (AR) platforms. My secondary goal is to contribute to the virtual reality and robotics community by providing findings that can significantly advance the knowledge in this topic and have a deep and broad societal impact.

Sincerely,

Matias Volonte.

References

Campbell, C.E., 2003. Sociology of Shyness: A Self Introduction. Human Architecture: Journal of the Sociology of Self-Knowledge, 2(1), p.9.

C. Dukes, J. Bertrand, M. Gupta, R. Armstrong, T. Fasolino, S. V.Babu, and L. F. Hodges. A multitasking healthcare simulation for training nurses to recognize patient deterioration. In Proceedings of Carolina Women in Computing 2012 (CWIC 2012), 2012.

Dirkin, G.R., 1983. Cognitive tunneling: Use of visual information under stress. Perceptual and Motor Skills, 56(1), pp.191-198.

elder, R.M. and Soloman, B.A., 2000. Learning styles and strategies.

Jin, W., Fan, J., Gromala, D. and Pasquier, P., 2018, August. Automatic Prediction of Cybersickness for Virtual Reality Games. In 2018 IEEE Games, Entertainment, Media Conference (GEM) (pp. 1-9). IEEE.

Kasahara, S., Konno, K., Owaki, R., Nishi, T., Takeshita, A., Ito, T., Kasuga, S. and Ushiba, J., 2017, May. Malleable embodiment: Changing sense of embodiment by spatial-temporal deformation of virtual human body. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 6438-6448). ACM.

Leeming, D. and Boyle, M., 2003, August. Shame as a social phenomenon. In Proceedings of the British psychological society (Vol. 11, No. 2, p. 295). The British Psychological Society.

Lombardi, S., Saragih, J., Simon, T. and Sheikh, Y., 2018. Deep appearance models for face rendering. ACM Transactions on Graphics (TOG), 37(4), p.68.

Mori, M., MacDorman, K.F. and Kageki, N., 2012. The uncanny valley [from the field]. IEEE Robotics & Automation Magazine, 19(2), pp.98-100.

Moser, L., Williams, M., Hendler, D. and Roble, D., 2018, August. High-quality, cost-effective facial motion capture pipeline with 3D regression. In ACM SIGGRAPH 2018 Talks (p. 59). ACM.

Qi Dunsworth and Robert K Atkinson. 2007. Fostering multimedia learning of science: Exploring the role of an animated agent’s image. Computers & Education 49, 3 (2007), 677–690.

Sagar, M., 2015, July. BabyX. In ACM SIGGRAPH 2015 Computer Animation Festival (pp. 184-184). ACM.

Todd, R., Cunningham, W.A., Anderson, A.K. and Thompson, E., 2017. Affect-biased attention as emotion regulation.

Volante, M., Babu, S.V., Chaturvedi, H., Newsome, N., Ebrahimi, E., Roy, T., Daily, S.B. and Fasolino, T., 2016. Effects of virtual human appearance fidelity on emotion contagion in affective inter-personal simulations. IEEE transactions on visualization and computer graphics, 22(4), pp.1326-1335.

Volonte, M., Robb, A., Duchowski, A.T. and Babu, S.V., 2018, March. Empirical Evaluation of Virtual Human Conversational and Affective Animations on Visual Attention in Inter-Personal Simulations. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 25-32). IEEE.

Volonte, M., Duchowski, A.T. and Babu, S.V., 2019, July. Effects of a Virtual Human Appearance Fidelity Continuum on Visual Attention in Virtual Reality. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (pp. 141-147). ACM.

Volonte, M., Anaraky, R. G., Duchowski, Knijnenburg, B. P., A.T. and Babu, S.V., 2019 (Accepted to SAP 2019, in publishing process).

Weech, S., Kenny, S. and Barnett-Cowan, M., 2019. Presence and cybersickness in virtual reality are negatively related: a review. Frontiers in psychology, 10, p.158.

Yamaguchi, S. and Onoda, K., 2012. Interaction between emotion and attention systems. Frontiers in neuroscience, 6, p.139.