Validation of a Multi-level Self-Report Rapport Scale and its Impact on Multimodal Modeling of Small Group Interaction

Rapport, defined as a sense of mutual connection and harmony between individuals, has been considered a key component of both social and task interactions [3, 57, 61]. It emerges from the fine-grained temporal coordination of behavioral signals [46]. While its positive impact on dyadic communication is well-established [6, 12, 42], for humans and conversational agents [33], its role in small groups remains underexplored, overlooking the multi-leveled structure of group dynamics where individuals may address another member or the whole group [25]. As conversational agents transition from personal assistants into multi-party environments, developing robust, interpretable measures of group rapport becomes as important as it is for single-user dialogue systems [13]. Group settings introduce additional complexities as they may emerge through intra-group dyadic and whole-group relationships, while the group composition itself (i.e., the different levels of personality traits within a tetrad) may play a role. This multi-leveled structure raises fundamental questions about how to measure rapport.

We therefore investigate self-reported rapport in groups of four participants (tetrads) engaged in collaborative conversations, based on tasks used in [31, 48]. To this end, building on existing rapport questionnaires [6, 22], we design a post-task questionnaire that captures both dyadic and group-level perceptions. To round out the picture, we collect responses to the Inclusion-of-Other-in-Self scale [2], and ask about task satisfaction. Following work by [31], we also collect self-reported personality traits before the participants gather, via the IPIP-NEO-60 OCEAN inventory online [26, 44, 45] allowing us to see how these individual differences relate to rapport both at dyadic and group levels.

Our analysis proceeds to: (1) a psychometric validation using factor analysis and multi-level modeling; (2) an exploration of the links between personality tests and rapport; and (3) a multimodal exploration using nested model comparisons (likelihood-ratio tests) to assess the respective predictive power of personality and audio-visual features.

The study of rapport and, more broadly, the pragmatics of social interaction, spans psychology, linguistics, and computational science, yet there exists a gap in how these constructs translate from dyads to groups.

Rapport and Synchrony in Multimodal Group Modeling. Interpersonal rapport is indexed by ways of speaking, nonverbal devices, and behavioral synchrony [6, 14, 57, 61]. The OCEAN personality traits of interlocutors (namely, Openness to Experience, Conscientiousness, Extraversion, Agreeableness, and Neuroticism) have also been shown to play a role [30], with extraversion and agreeableness emerging as the strongest predictors in human-agent conversation [13, 16]. However, the vast majority of this research has focused on dyadic settings [6, 66, 71], and has often focused on either nonverbal behavior or spoken language (both content and acoustic features), but not both. Additionally, prior work demonstrates that processes observed between two people do not necessarily generalize to multi-party settings [25, 38]. On the other hand, numerous studies have explored various other aspects of small group collaboration, such as creative problem solving [53], influential statements [48], curiosity in learning contexts [51], group engagement [49, 55], or group cohesion [34]. However, other phenomena, such as group cohesion, often defined in terms of alignment in attitudes or opinions, do not fully capture rapport, which instead reflects the moment-to-moment signaling of interpersonal connection and harmony during interaction. It therefore remains important to model group rapport.

In this context, [22] links rapport to interactional synchrony and group performance, highlighting its functional role in collaborative settings, and proposes a group rapport scale based on earlier dyadic measures [8]; however, she does not explore the relationship between dyadic and group rapport in groups, nor the impact of personality traits. Building on [22], who found that non-verbal behavioral synchrony was predictive of group rapport, our work adapts a compact subset of her scale's group rapport items to suit the analysis of a dataset that also examines the impact of personality (see section 4).

From another angle, [47] has demonstrated that multimodal behavioral features can detect low dyadic rapport in small groups. Collecting self-reported rapport at the dyadic level, they found the best predictor to be facial expressions. However, their work does not examine the relationship between dyadic and group rapport, nor does it investigate how psychological traits such as personality influence rapport signaling.

Evaluation of Rapport. How rapport is measured differs widely. Some studies rely on self-report questionnaires pertaining to an entire interaction [8, 13]. Others have relied on “thin-slice" observer judgments [1, 29, 41], where video is divided into short segments, and each is annotated by a naive observer.

The 18-item scale of Bernieri [6], originally developed for inter-human interaction, has served as a common reference point for computational models that have relied on self-report [28, 30, 47, 68]. Within these different perspectives, scales tailored to specific interaction contexts have also been proposed, including for teacher-student [69], investigative interviewing [18], and therapy [43]. In human-machine interaction research, dedicated rapport scales have further been introduced [13, 39, 40, 67].

These self-report scales are motivated by work that suggests that external observers may fail to capture the subjective experience of interactants [8, 13, 50], and highlights a mismatch between ‘how participants describe their feelings’ and ‘how observers perceive them’ [17, 21, 50]. For example, [8, 13] reported low consistency between self-reported and observer assessment of rapport, as well as different behavioral correlates across the two perspectives. However, an alternative perspective argues that people may access their internal states largely through their behavior, and that such access can fade by the end of an interaction [5], with end-of-task judgments disproportionately shaped by initial and final impressions [20, 58].

Rapport and Personality in Groups. Much research has examined the role of the five OCEAN personality traits in group outcomes. In fact, a meta-analysis has shown that team personality composition influences performance — often assessed through aggregated supervisor ratings of team effectiveness across task-related dimensions — with higher and more homogeneous levels of agreeableness and Conscientiousness predicting better team outcomes [52]. [9] also highlights the importance of group personality composition for both individual- and group-level outcomes, as coded by observers who rated group behaviors such as conflict, cooperation, communication, and coordination. However, personality remains insufficiently integrated into frameworks that jointly consider multimodal social signals and multi-level perceptions of rapport. In addition, existing approaches often treat personality as a holistic predictor, rather than modeling the different traits and how they interact with behavioral signals to shape emergent group processes such as rapport.

A Multi-Level Approach. Our work addresses the above limitations by examining self-reported dyadic and group-level rapport within the same small-group setting, and by focusing on the impact of observable behaviors as well as the underlying psychological factors of personality traits.

To the best of our knowledge, this is the first scale that proposes to integrate this multi-level approach of rapport into a relatively short questionnaire. By then integrating personality traits and multimodal behavioral synchrony into our analysis, we can investigate how mechanisms that have been shown to underlie dyadic rapport may extend to the group-level —an important extension to current literature. This approach allows us to validate self-report scales while bridging the gap between fine-grained temporal behavioral signals and the underlying psychological traits of personality and rapport.

The dataset¹ consists of groups of four native or near-native French-speaking participants (tetrads) engaging in three verbal tasks designed to elicit collaborative conversations. It is designed to be comparable to the MATRICS corpus recorded in Japanese [31, 48], in order to allow future cross-cultural/cross-linguistic comparisons (small changes were made in order to capture the upper bodies on video). Tetrads participated in three 20-minute tasks: (1) Guest Selection: choosing and scheduling five celebrity guests for a festival and outlining their stage performances; (2) Booth Planning: designing a plan for six food booths distributed across key festival areas; and (3) Travel Planning: organizing a two-day sightseeing itinerary in Paris for a friend, including visits and rest periods.

Recording procedure and setup. At the beginning of each task, written instructions were distributed, then also given verbally by the experimenters, allowing participants to ask questions. Participants were asked to reach a consensus on each task. Before and after each task, participants engaged in 7 minutes of free talk. Video was recorded on four GoPro cameras, and audio was collected with four wireless microphones mounted on headsets. Participants sat around a small table in a quiet room with cameras facing each participant, placed at chest level to ensure they could freely move their upper bodies. Participants received €25 as compensation.

Demographics and subset. 30 tetrads constitute the data for Studies 1 and 2, with a subset of 16 tetrads for Study 3. All groups were same-gender, with an equal number of male and female tetrads. Participants were recruited through a local online research site. Mean participant age was 28 (σ = 9.94, min = 18). Participants were not acquainted beforehand.

The goal here is to construct a scale that measures both group and dyadic rapport between every pair of participants within a group. The dyadic rapport scale was developed based on [8]. However, the original 18-item scale was not suitable here, where each participant evaluated their level of rapport with every other group member as well as for the group. To limit fatigue while preserving reliability [19], we therefore initially reduced the dyadic rapport scale to 6 items, selecting two for each core component of rapport as conceptualized in [61]—(positivity: Q1, Q2; coordination: Q3–Q4; attentiveness: Q5–Q6). Preliminary analyses revealed insufficient internal consistency (see Study 1 in section 5), and thus the scale was extended with one additional item per component from the original scale, resulting in a 9-item dyadic rapport scale. For group rapport, we relied on prior work by Fultz [22], who proposed a 19-item scale for groups, grounded in rapport theory and the [8] dyadic scale. The validation of the scale in [22] identified two main components through PCA: rapport and compliance with group wishes. We retained a subset of 6 items, with strong loadings on the rapport component (Positivity, Coordination, and Attention). More questions were retained for attentiveness than for coordination to avoid inadvertently capturing group-task functioning instead of rapport, as per [22]. Additionally, we incorporated 3 items to assess task satisfaction, as strong loadings between task satisfaction items and group rapport were reported in [22]. This resulted in a final 9-item group-level scale (6 group rapport and 3 task satisfaction items). To account for potential acquiescence effects [32] in responses, as well as for subsequent validity checks [37, 65], we reverse-coded Q2 and Q4 in the dyadic rapport scale, and Q3 and Q5 in the group scale. All items were rated on a 7-point Likert scale.

To assess convergent validity, we also collected measures of interpersonal closeness using the Inclusion-of-Other-in-Self (IOS) scale [2], which has been shown to correlate with constructs such as liking and intimacy [22, 56], as well as rapport [23]. Participants completed the scale for each partner and then an adapted group-level version, the Inclusion of Ingroup in the Self (IIS) scale for group evaluations [22, 63].

The items from the dyadic and group rapport scales were translated into French by the authors and subsequently reviewed by two native French speakers for clarity and linguistic accuracy (see Appendix A for full questionnaires). Participants recorded responses to all scales on computers right after the interaction.

Following established procedures in psychometric scale validation, we assess the internal consistency of the dyadic and group rapport scales and their convergent validity with task satisfaction, the Inclusion of Other in the Self (IOS) and Inclusion of Ingroup in the Self (IIS) measures². Dyadic and group scales were analyzed separately within a multi-level framework. Sample sizes are consistent with respondent-to-item heuristics for both scales at their respective units of analysis [10] (see footnote 3). An exploratory factor analysis (EFA) was conducted separately for the two scales to examine their underlying structure. Internal consistency was assessed using Cronbach's α and McDonald's ω [54, 59]. In order to ensure comparability, internal consistency was subsequently re-evaluated.³ Convergent validity was assessed by estimating linear mixed-effects models predicting Dyadic IOS, Group IIS, and Task Satisfaction from rapport scores. Spearman correlation coefficients were computed to examine associations among study variables, after averaging item-level scores into construct-level means for each tetrad and individual at both the group and dyadic levels, merging these aggregates, and using pairwise complete observations. To further investigate interpersonal dynamics, dyadic reciprocity was calculated using Spearman correlations between mutual rapport ratings (rater-to-target and target-to-rater) across the 180 dyads within the 30 tetrads. Finally, to account for the nested structure of the data, a linear mixed-effects model with tetrad as a random intercept was used to examine the effect of gender on dyadic rapport.

Building on the validated dyadic and group rapport measures, this study investigated whether individual differences in personality traits from the Big Five (OCEAN) predict self-reported rapport. Personality traits were assessed using a 60-item adaptation drawn from the Canadian-French IPIP-NEO-300 [27]. See Appendix B for details and the full questionnaire. Internal consistency estimates for the five personality scales are reported in Appendix C.2.

Building on the validated rapport measures established in Study 1 and the personality effects identified in Study 2, we examine how non-verbal behavioral synchrony and individual personality differences might jointly contribute to rapport, and whether these associations generalize to unseen groups. Specifically, we asked three questions: (Q1) whether behavioral synchrony explains variance in rapport beyond personality traits, (Q2) whether personality traits retain predictive value once synchrony is considered, and (Q3) whether personality moderates the relationship between behavioral synchrony and rapport. The sample comprises 16 tetrads (64 participants, constituting 192 dyads), the subset of the full sample for which synchronized audio-visual recordings were processed.

We introduced and validated the Self-reported Group and Dyadic Rapport (SGDR) scale, a compact instrument measuring rapport at both the dyadic and the group level in small-group interaction. Across three studies (30 tetrads, with 16 in study 3’s multimodal subset), the scale showed solid reliability and meaningful links to interpersonal closeness and task satisfaction at both levels, supporting its use in multimodal research. Findings suggest that dyadic and group rapport may function differently. Dyadic rapport was associated with personality traits and behavioral synchrony: higher target Agreeableness and lower target Conscientiousness predicted higher rapport; when behavioral synchrony was added, pitch synchrony became the strongest predictor. Moreover, these two classes of predictors interact rather than being merely additive: the association between pitch synchrony and rapport was roughly four times stronger for raters high in Openness, a moderation supported by the likelihood-ratio test. Group rapport, by contrast, was explained neither by a rater's own traits nor by tetrad-level mean personality traits. It was instead explained by the extremes of the rater's dyadic experiences (the best and the worst dyadic interactions independently predicted the group rating), while tetrad-wide behavioral coordination showed a suggestive, not-yet-conclusive association that nonetheless outperformed personality both in explained variance and in generalization to unseen groups. Notably, Conscientiousness was significantly negative for dyadic rapport, and trended positive at the group level — a further indication that the two levels may be influenced by different mechanisms. Taken together, these findings caution against treating group rapport as a scaled-up version of dyadic rapport and support modeling the two levels separately. The SGDR provides a practical tool for doing so, and a step toward multimodal systems able to track rapport in task-oriented multi-party interaction.

The present findings should be interpreted in light of several limitations. While studies 1 and 2 included 30 tetrads, Study 3 relied on only 16, limiting the ability to detect more than moderate effects. Group-level findings that include nonverbal and paraverbal behaviors should thus be considered exploratory. The Openness moderation identified in Study 3 should also be interpreted with caution, as it was observed only in the 16-tetrad multimodal subset and not in the full 30-tetrad personality analysis. This discrepancy, together with the comparatively modest reliability of the Openness scale (see Appendix C), suggests that this finding is preliminary and requires replication in larger multimodal datasets.

Results presented within these three studies, moreover, rely on self-report completed at the end of the sessions, while externally assessed rapport annotations, currently underway, will allow self-perceptions to be triangulated with observer judgments. Finally, rapport ratings were generally high with restricted variance, and the dataset covers a single linguistic and cultural context; the planned comparison with the MATRICS corpus [31, 48] will help establish cross-cultural generalizability, as will a planned comparison with our Korean colleagues, also collecting an identical dataset.

Generative AI tools (ChatGPT, Claude) were used for editing and polishing the manuscript. All scientific content, experimental design, and results were produced by the authors.

This work follows ethical and responsible research practices for multimodal interaction. In compliance with GDPR, data are securely stored, pseudonymized, and anonymized where possible, with risks such as personal disclosure mitigated through screening and controlled access. The study protocol, recruitment, and materials were approved by a local Institutional Review Board, and all participants provided informed consent after full information. We support transparent, privacy-preserving, and socially beneficial human–agent interaction while minimizing misuse risks. We also consider bias, inclusivity, and cross-cultural validity in the dataset design.