A Systematic Review on the Socio-affective Perception of IVAs' Multi-modal behaviour

In this section, the research works investigating the perception of IVA's emotions through their multimodal behaviour are reported. Initially, the literature reveals that emotional dimensions are explored through different affective concepts such as mood and emotions [1]. To compare the research results, the analysis of the research works on the emotional perception of IVA behaviour is proposed in light of the valence and arousal dimensions.

These dimensions are expressed and perceived in various ways, for example, through posture and facial expressions [24]. According to the authors, the SAM questionnaire, which studies the dimensions of valence, arousal, and dominance, is a good tool to assess the perception of these dimensions. However, dominance, which could also be considered as a personality trait, is discussed in Section 3.2.

According to [6], the notion of valence refers to the IVA's opinion, i.e., the positive or negative feelings it has towards the user. As shown in several research works [6, 9, 14, 18, 28, 38], the emotional state, and more specifically valence, is conveyed by facial expressions to display basic emotions such as anger, happiness, and sadness [28], but also more subtle emotional states such as stress [9], amusement, or politeness [38]. Even if facial expressions are a strong cue of emotional state, the way they are displayed in IVAs can vary considerably from one study to another, leading to misinterpretation depending on the considered representation, the intensity of the expression, and the combination of modalities [14, 28]. Although it seems that the frown conveys a negative valence [9, 13], the way it is represented in the IVA could vary, particularly in terms of intensity and of the parameters used to display facial expression. For example, in the study of [14], a smiling facial expression is perceived as neutral, while it is generally a positive valence signal, indicating a potential confusion between a fake smile and a genuine smile [14, 38]. Valence is mainly conveyed by the combination of facial expressions and head movements. Indeed, researchers consistently associate smiling and nodding with positive valence, and frowning and shaking the head with negative valence [6, 9, 14, 18]. However, some signals may be predominant in the perception of valence. Indeed, as shown in [14], head shake is the most negative modality identified and is always judged negatively, regardless of the other modality it is associated with. Regarding the other signals, the literature is not consensual. For example, posture, such as crossed arms, may convey a negative valence as shown in [6, 14], while [18] does not find no impact of posture on the assessment of valence. To express the valence, some modalities may be more important than others. For instance, as highlighted in [6], to assess the IVA's valence, the users generally first consider the head movements, then the posture, the gaze direction, and finally the facial expressions, as this signal is more subtle in VR. Additionally, research on the perception of behaviour on the valence highlights the importance of combining non-verbal behavioral modalities. For example, nodding is mainly a sign of positive valence but is sometimes considered as neutral based on the associated signals [14]. In the same way, an IVA with the head tilted, its elbow on the table and its torso leaning forward appears to be positive, while separately, these signals convey a neutral or negative valence [14]. The perception of the non-verbal modality may also vary according to the associated verbal behaviour, as highlighted in [38]. In addition, vocal non-verbal immediacy, corresponding to the pitch level and speech rate, may also have an impact on the assessment of participants’ affect towards the content or the virtual model. It also impacts the likelihood of following the same virtual instructor again for other similar videos in the future [16]. Indeed, stronger vocal immediacy, characterised by an average pitch of 260 Hz and a speech rate of 133 words per minute (wpm), enhance affective learning compared with a virtual model that uses weaker vocal immediacy, with an average pitch of 115 Hz and a speech rate of 119 wpm [16]. Some factors may influence the perception of valence, such as the appearance of the IVA itself. Although few studies examine this factor, some research shows that a female IVA is perceived as more positive than a male IVA when smiling [38]. Another study also shows that for a given facial expression, such as raising eyebrows, a virtual female IVA appears to be more stressed if it uses a frontal body with direct gaze than if it used an averted body with an averted gaze, whereas no difference is found between these behaviours for the male IVA [9].

The dimension of arousal describes the excitement of the event for a person and ranges from low to high alertness [51]. It takes the form of an IVA interested or attentive to what people are saying and is characterised by two types of non-verbal behaviour, namely proximity and body movements [51]. Consistently, all the authors who study this dimension find a relationship between the evaluation of arousal and the direction of gaze, the frequency of movements, and the proximity of posture [6, 14, 18]. A high level of arousal or engagement is associated with an IVA looking at the speaker, with a closer posture, i.e. a torso leaning forward and frequent head movements and facial expressions. On the contrary, an IVA that looks away with a more distant and relaxed posture is perceived as disconnected [6, 18]. It is also interesting that eyebrow raising is judged to be neutral in terms of arousal and that head shaking is always associated with high arousal, regardless of other signals [14]. Another interesting point is the relationship between perception of valence and arousal. The combination of the valence-arousal pair can convey a specific attitude, and, as indicated by [18], users are able to perceive different social attitudes based on these dimensions (indifferent, critical, and enthusiastic). However, several authors report that users do not distinguish between different levels of valence (positive and negative) for low arousal [6, 14, 18].

In this section, three personality traits are considered: the dominance, the extroversion, and the friendliness. The dominance is part of the PAD (Pleasure-Arousal-Dominance) model, described by [44]. The dominance corresponds to a scale ranging from the absence of control or impact on the event to the feeling of influence or control on the situation [51]. Extroversion and friendliness are two components of the "Big Five" personality traits model [21] that identifies Openness, Conscientiousness, Extroversion, Agreeableness, and Neuroticism. As explained in [11], agreeableness is an indicator of friendliness. Warmth may also refer to friendliness [27]. Similarly, the concept of politeness involves the dimension of friendliness, as explained in [52, 53]. For the sake of clarity, only the term friendliness is used in this section. According to [1], extroversion corresponds to the sociability of IVA perceived by users.

The choice of only keeping dominance, extroversion and friendliness to express personality is explained by the interrelationships between them. Indeed, [3] shows that dominance and extroversion are positively correlated, while dominance and friendliness are negatively correlated. In addition, friendliness and dominance are two dimensions that appear in the Interpersonal Circumplex model [27], which further indicates their strong relationship.

To express dominance, an IVA can look at the user in various ways: by looking from below, which is perceived as less dominant, from above, or by aligning its eyes close to those of the participant [3]. Furthermore, as shown by [43, 47], dominant non-verbal behaviours can be displayed using akimbo posture, crossing arms, sagittal head up, gesture with large radius. On the contrary, submissive non-verbal behaviour includes neck-adapter (self-touch), arms open, sagittal head down, gesture with small radius.

Regarding non-verbal signals, the more dominant position is adopted, the more dominant the IVA is perceived [43]. Furthermore, [47] shows that, concerning the perception of dominance, crossing arms and the akimbo position are the most effective gestures, but taking up more space (gesture with a large radius) is not perceived as more dominant than the other gestures. Furthermore, [11] and [10] demonstrate that by using more (or less) dominant cues, an increase (or decrease) in perceived dominance can indeed be implied. As shown by [3], the more linguistic friendly cues are used, the less dominant is the IVA perceived. It is further established by [5] when using verbal modalities of friendliness. Moreover, regarding vocal modalities, [17] shows that the longer the interruption handling time, the more dominant the IVA was perceived.

To express extroversion, IVAs can exhibit positive emotional states and medium arousal levels (see Section 3.1), and high dominance behaviours (closer interaction distances, expansive gestures, rapid gesture execution, sustained eye contact, and prolonged gaze duration). In contrast, introverted agents display negative emotional states, low arousal, more distant interactions, reduced spatial extent in gestures, slower gesture speeds, and averted gazes [7, 45].

As shown by [3], the more dominant gaze cues are used, the less extroverted the IVA is perceived. Generally, it is possible to distinguish extroversion from introversion by voice or facial expression alone [46]. However, if the voice is combined with body movements, it is the most informative signal to judge the extroversion of the virtual agent [46]. More specifically, [45] shows that a virtual agent is considered to be extroverted if it has a positive emotional state, a medium level of arousal (see Section 3.1), and a positive dominance value. On the contrary, an introverted virtual agent has a negative emotional state, a low level of arousal, and a low dominance value. At the behavioural level, extroverted personality translates into a closer distance during the interaction (e.g., the torso leaning forward), a higher value of spatial extent during the execution of gestures, a higher speed of execution gestures with greater eye contact, and a longer duration of gaze for an extroverted virtual agent. On the contrary, an introverted virtual agent is more distant during interaction (leaning back) and adopts a lower value of spatial extent and speed of gesture execution, with an averted gaze [7].

To express friendliness, regarding verbal modalities, an IVA can use fewer synonyms and negations, shorter sentences, more pronouns, verbs, negations, informal language, positive content, and references to the speaker [4].

Regarding non-verbal modalities, the more dominant positions are adopted, the less friendly the IVA is perceived [43]. As expected, IVA friendliness can be perceived through positive facial expression (e.g., smile [38]) but specific gestures, such as commanding the user through arm gestures (using finger pointing) can have quite the reverse effect [52]. However, [5] shows that non-verbal behaviour or verbal modalities of friendliness alone are not sufficient to render friendliness, suggesting that verbal modalities must be added to ensure the right perception of it. Furthermore, [52] shows that, to maintain a certain degree of friendliness in the IVA, it is preferable to use suggestion rather than command. In addition, one may prefer stronger vocal immediacy (i.e. higher pitch and faster speech rate) to depict a more friendly IVA [3]. Lastly, regarding vocal modalities, [17] shows that the longer the interruption handling time, the less friendly the IVA is perceived. Moreover, [10] and [11] show that using more (respectively less) dominant cues and/or less (respectively more) friendly cues implies an increase (or decrease) of perceived friendliness. Similarly, [4] shows that the model that uses adaptive algorithm (Reinforcement Learning) to adapt to user impressions can indeed increase the perceived degree of friendliness compared to when IVA does not adapt its behaviour to user reactions.

Trustworthiness is a complex dimension, closely related to underlying concepts, such as the performance of the IVA and its inclination to collaborate [20]. In this literature review, the perception of several sub-dimensions of trust is investigated, such as the IVA's competence, intelligence, autonomy and helpfulness, reflecting its performance, and its cooperativeness and persuasiveness, displaying a certain predisposition for collaboration with the user.

In the literature, research highlights the impact of IVA's emotional non-verbal behaviour on its attributed trust level. In [23], the IVA displays an emotional non-verbal behaviour, either positive (i.e. smile, head nod, head nod plus smile) or negative (i.e. sad face, head down, dropping the arms plus sad face), while respectively announcing good or bad news. Compared to an IVA that remains neutral in its behaviour, its perceived trustworthiness increases under emotional conditions. This relationship is further studied in [47], more specifically by investigating the impact of emotional behaviours on perceived cooperativeness. The hypothesis assumed by the authors, justifying the adoption of such behaviours to depict cooperativeness, states that this sub-dimension is closely related to IVAs’ expressiveness. This hypothesis is confirmed, therefore strengthening the already established link between an IVA's emotions and trustworthiness. The results of the considered study include the preeminence of behaviours combining expressive gestures and mimic to provide the IVA with a cooperative attitude. The impact of lateral head tilts, whatever the side, is positive as well. Naturally, opposite results are found for neutral and non-expressive behaviours, which are associated to lower cooperativeness perception levels, however not as low as averting gazes.

The personality of the IVA, either depicted by its friendly or dominant behaviour towards the user, is also recognised as primordial in shaping perceived competence, trustworthiness, and cooperativeness [4, 10, 35, 43, 47]. A model commonly used in the literature is the Warmth and Competence model [15], which links the IVA's friendliness (see 3.2) with its competence. This model is adopted by [4] and [35] to assign specific non-verbal behaviours to IVAs depending on the desired competence level. High competence perception is successfully obtained by synchronising the IVA's gestures with the semantic content of the speech [35]. In contrast, IVAs convey low competency through desynchronised gestures [35]. This manipulation effectively transmits the expected competence signal, regardless of the IVA warmth level, which is characterised by an open (high warmth) or closed gesture (low warmth). However, it is worth mentioning that IVA's warmth, and by extension its friendliness, also has an impact on competence perception, with a positive correlation between these two variables. The accuracy of the warmth and Competence model to design IVAs perceived as such is confirmed in a second study [4].

As previously stated, studies also investigate the interrelation between dominant behaviours and trust sub-dimensions, more specifically considering the impact on competence, intelligence and persuasion [43], as well as on helpfulness [10] and cooperativeness [10, 43, 47]. The effect of dominance is, however, not confirmed for all sub-dimensions. In [43], classical dominant non-verbal signals (i.e. akimbo posture, arms crossed, gestures with large radius, sagittal head up) have no effect on competence, autonomy, intelligence, and persuasion ratings. On the contrary, the helpfulness of the IVA appears to be negatively impacted [10]. When it comes to cooperativeness perception, contradictory results can, however, be observed. Despite [43] additionally confirms that no relationship can be established between these two variables, [47] finds a negative effect of dominance on cooperativeness, although its effect remains quite small. The latter study additionally identifies specific dominant signals, particularly perceived as uncooperative, such as keeping the arms crossed, which corresponded to the lowest cooperativeness rating among several dominant behaviours, including keeping the head up or making large radius movements. Conversely, the authors identify IVA’ behaviours, associated to submission, interpreted as cooperative signals. More specifically, an IVA adopting small radius gestures or keeping its arms open improves its attributed cooperativeness level.

The persuasiveness of IVAs’ multimodal behaviour is also investigated. Two studies firstly compare the effectiveness of several strategies in order to persuade the user to join a group of IVAs, either using monitor [52] or a VR headset [53]. In both conditions, similar results are obtained. Strategies adopting a direct approach, explicitly formalising the request, are the most persuasive ones. Actually, when the IVA directly commands the user to move to a specific location, while pointing it with its index, the level of associated persuasiveness is the highest among all possible strategies [52]. In comparison, less commanding approaches, such as politely asking or proposing to the user to join the group, are less persuasive but yet still effective [52]. The authors further highlights the primordial importance of clear formulation of the demand to maximise IVAs’ force of persuasion. In addition to the influence of the strategy adopted, communication modalities also play a role. Indeed, multimodal modalities, combining verbal (i.e. proposition or command) and non-verbal communication (i.e. gaze and hand movements), reach the highest impact levels on persuasiveness [53]. Nevertheless, in [39], results indicate that verbal modality only should be preferred to maximise the IVA's force of persuasion, reducing distractions for the user, who can therefore be focused on the command itself. An important difference between this study and previous ones that should be considered is that the non-verbal behaviour in this case is not directly related to the command asked of the user, thereby justifying its perceived uselessness.

Generally speaking, the IVA's verbal behaviour also plays a significant role in trust perception. The importance of a realistic voice is highlighted, with IVAs eliciting higher level of trust when endowed with a human voice rather than a synthetic one [39]. Verbal behaviour also turns out to be important in eliciting competence. An interesting signal used to reflect high competence and proved to be relevant is the formulation of sentences using "We" or “You” pronouns, rather than sentences formulated in the first person singular [4]. Similarly, IVAs disclosing personal information are not perceived as more competent [48], further reinforcing the previous statement regarding the irrelevance of “I” pronouns to elicit competence. Although the words used are important, the number of words used has no influence on competence perception [48].

Valuable insights for the design of trustworthy IVAs are provided, but the design of their behaviours should still be carefully considered. Indeed, specific aspects can considerably impact perception, leading to undesired effects. Such negative effects are observed in [49], with mimicking IVAs, evoking a low level of trust and of helpfulness among users when reproducing their behaviour, being even perceived as creepy. The immediacy of the reaction further reinforces the identified negative effect. Another surprising result from the considered papers. [43] identifies that guiding behaviours reduce the perceived level of competence. Such behaviours are represented with deistic gestures and gaze, meaning that IVAs pointing to specific elements are not seen as competent.

In this section, the research works exploring users’ perception of IVAs’ social capabilities are presented. The articles considered in the systematic review identify the following main dimensions related to social capabilities: the mutual understanding or comprehension, the mutual agreement, the intimacy or self-disclosure, the interpersonal and dyadic stances and finally, the social status. These dimensions are strongly related to the notion of virtual rapport [19].

The mutual understanding, attention, agreement, interest and pleasantness are explored in [42] in terms of users’ perception of an interaction between two IVAs. In social interaction, intimacy corresponds to "a reciprocal expression of personal or emotional contents, and the perception of positive feelings and comprehension." [41]. In [25], intimacy refers to the perceived self-disclosure of the IVA, which corresponds to the capacity to share personal information with the user to create social connections. The interpersonal stances in [38] correspond to the perceived relationship expressed by the IVA toward its interlocutor, as for instance cold or warmth. Finally, the dyadic stance results from a behaviour alignment or nonalignment between the two agents reflecting for instance an agreement or hostility [38].

The perception of these social capabilities dimensions described above can be significantly influenced by the multimodal behaviour of the IVA. For instance, as shown in [42] and [38], the mutual reinforcement of smiles between two IVAs or between an IVA and a user, enhances the perception of the mutual understanding and impacts the perception of the interpersonal and dyadic stances [38].

Concerning intimacy, in [40, 41], the authors show that a high level of perceived intimacy leads to a longer interaction with the user. Moreover, the behaviour of the agent may influence its perceived honesty and genuineness: users rated the IVA as more honest and authentic when it displays behaviours associated with intimacy such as emotional facial expressions (e.g., smiling), open-arm gestures, self-directed motions, head nods and tilts, and eye contact. Moreover, [48] investigates the perception of the talkative and self-disclosure dimensions. Results show that an IVA is perceived as more talkative and open, when it uses more words and shares personals information with the user.

A last sub-dimension is the influence of its non-verbal behaviours on its perceived social status. Among the selected papers in this literature review (see Table 2), a unique study, [36] aims to investigate the effects of different gaze behaviours, reflective of either high or low social status, during a job interview scenario with an IVA. The study finds that an IVA perceived as belonging to a higher social status displays longer duration of eye contact and prolongs stares following the user's responses. Furthermore, an IVA raising his head posture and with a body leaned toward the user is also associated with a higher social status, highlighting the significance of non-verbal modalities in shaping perceptions of social hierarchies within virtual interactions.

The user's perception may be strongly impacted by the believability of the IVAs influenced by the quality of the animations and of the voice. In this literature review, believability is assessed across several dimensions. In [16], the authors refer to the dimensions of animacy, anthropomorphism and liveliness which correspond to dimensions of the Godspeed questionnaire [2]. The notion of believability corresponds also to the perception of credibility, for instance in [5] to explore how credible the IVA is in its role of recruiter. In [26], the believability of the IVA's behaviour is evaluated through the notion of plausibility and naturalness focusing on both the behaviour and the appearance.

Several articles in our literature review have shown the influence of IVAs’ behaviours on the users’ perception of believability. First of all, concerning the animations, [39] and [26] emphasise that an animated IVA, regardless of the animation mode, is perceived as more natural and more plausible than a static one, indicating that the movements are a key factor in the perception of believability. Moreover, [39] shows that a more varied and nuanced behaviour leads to a better perception of the realism. [26] also focuses on the naturalness and plausibility of facial animation behaviour, showing that synthesised expressions (i.e. facial expressions generated from data such as audio, head movements, and tagged gaze targets to correspond with expected facial expressions in specific situations) are evaluated as more natural and plausible than tracked expressions (i.e. facial expressions captured in real-time from facial movements), both for verbal and non-verbal behaviour. In [29], the authors highlight the importance of the appropriateness of the expressed signals showing that inappropriate nods are perceived as less natural than those adhering to established norms.

In [16], the authors investigate the impact of the verbal behaviour and in particular the vocal immediacy on the animacy, anthropomorphism and liveliness. They manipulate the virtual agent's vocal parameters, such as pitch and speech rate. Results indicate that participants exposed to the condition with stronger vocal immediacy (high pitch and fast speech rate) perceive the agent as more anthropomorphic, animated, and likeable compared to those in the weaker vocal immediacy condition.