FaceExpressions-70k: A Dataset of Perceived Expression Differences

Several datasets that include facial expressions exist. The HUMBI dataset [Yu et al. 2020] captured multi-view data of body and face expressions of 772 unique subjects. However, the resolution of the scans is insufficient to allow viewing subtle facial details. Similarly, the FaceWarehouse dataset [Cao et al. 2014] offers 3D facial expression data of 150 subjects, but relies on low-resolution RGB-D sensors, restricting geometric precision. Finally, the MultiFace dataset [Wuu et al. 2022] provides synchronized recordings of 13 individuals performing over 60 diverse facial expressions and head poses. Their multi-camera system includes 2048 × 1334 resolution images, tracked 3D meshes, unwrapped textures, and metadata, which enables more precision in the modeling of fine facial details for applications like virtual reality and telepresence. Due to its high resolution, geometric accuracy, and expression diversity, we selected MultiFace as the baseline dataset for this work, creating a comprehensive set of subjective labels of expression similarity.

Recent efforts have advanced facial behavior understanding by learning face-specific representations. Zheng et al. [2021] proposed FaRL, a framework that learns transferable facial representations by combining contrastive image-text pretraining with masked image modeling. Ma et al. [2023] introduced MAE-Face, a unified approach for facial affect analysis that leverages masked autoencoder pretraining and multi-modal fusion. Building on face-specific learning, several works further disentangle identity to enhance expression analysis. Ning et al. [2024] proposed a framework that combines representation learning with identity-adversarial training to disentangle identity information and improve facial expression analysis. Liu et al. [2024] introduced NorFace, which enhances facial action unit (AU) analysis and facial emotion recognition (FER) by normalizing identity features to reduce subject bias. Zhang et al. [2021] proposed a Deviation Learning Network, modeling expressions as deviations from identity to learn a compact, identity-invariant embedding.

Subjective data on faces has been leveraged to optimize digital human rendering quality. Wolski et al. [2022] quantified the visibility of geometry distortions on untextured human faces. Although sufficient geometric detail and quality is needed to perceive expressions, this study only focused on neutral faces. FaceMap [Jiang et al. 2024] created a saliency map of the face, focusing on the visibility of texture and geometry distortions. While localized saliency information can correlate with expressivity, this was not part of the study.

Perceptual evaluations of expression have been conducted. Wallraven et al. [2008] studied the accuracy of animation techniques in reproducing expressions. Zibrek et al. [2019] studied the impact of photorealism for three expression types: friendly, unfriendly, and sad, finding that more photorealistic imagery enhances expressiveness. Tessier et al. [2019] studied the realism of pain expressions in avatars, concluding that the order of action unit activation impacts the perceived realism of the expression. Treal et al. [2021] studied the impact of body and postural movements on the empathic response to expressions of pain, concluding that it has a significant impact. While all these studies examine facial expressions, they focus on specific aspects of the problem (animation accuracy, photorealism, pain expressions), and as a consequence, are not suitable for a general quantitative model of expression accuracy.

Mean Squared Error (MSE) and Mean Absolute Error (MAE) are commonly used in image processing tasks but lack perceptual alignment, focusing only on pixel differences. Structural Similarity Index (SSIM) [Wang et al. 2004] improves upon this by considering luminance, contrast, and structure. Learned Perceptual Image Patch Similarity (LPIPS) [Zhang et al. 2018] is a neural network-based metric that compares feature activations for perceptual similarity. Both SSIM and LPIPS are widely used in tasks like image quality assessment and image similarity due to their more substantial alignment with human perception. Recently, activations from pretrained models [Chen et al. 2020; He et al. 2015; Radford et al. 2021] have also been shown to be useful for various tasks involving human perception. While these methods can present good accuracy for general content, they are not tuned to facial expressions. As a result, their performances at predicting the perceived differences of expression changes are limited (see Sec. 4).

Specific to expressions, McDonnell et al. [2021] examine the perception of facial action units (AUs) in virtual human models. They examined 12 expressions, each at five activation levels for six digital characters. Analyzing the data, the authors draw conclusions on perceptual salience across facial regions, and study the influence of factors like race and sex. Similarly,  Cipiloglu Yildiz [2023] developed a perceptual distance metric for 3D blendshape models using crowdsourced evaluations of 2,905 intra and inter-expression triplets, obtained from a single blendshape model with five expressions, 173 raters, and metric learning techniques that incorporate saliency and curvature. Both works focus on expressions but employ only virtual characters animated via blendshapes. In contrast, our study employs filmed captures of human subjects, thus avoiding the influence of the rendering and animation methods on the result. Furthermore, our dataset, comprising 70.5k intra- and inter-expression pairs from eight human faces and 61 expressions, annotated by 1021 raters over more than 8000 hours, is significantly more extensive, which makes it suitable for fine-tuning general-purpose visual computing algorithms, such as neural networks or metrics.

We utilize the MultiFace dataset [Wuu et al. 2022] as the source of the images used to run our study. This repository features a diverse range of facial expressions, including distinct and highly prominent expressions, as well as nuanced variations. Moreover, the controlled imaging conditions in MultiFace ensured that our dataset provided consistent and representative data for studying facial expression differences, forming a robust basis for investigating perceptual variations and developing objective evaluation metrics.

For this study, we sample frames from eight of the thirteen identities in the MultiFace dataset to maintain a feasible study size. This set was down-selected in a way that maintained as much demographic diversity as possible. See Appendix A.1 for an extended discussion of the identity selection process. We focus exclusively on front-facing camera captures, as this perspective minimizes pose variability, enhances the prominence of facial features, and provides a clear, uniform view, making the dataset particularly suitable for analyzing perceptual nuances in expression differences.

Facial expressions are transitional and often involve complex dynamics; in this work, we chose frame-based analysis as it provides more granular control, allowing us to examine different expressions at varying levels of activation.

The MultiFace dataset provides video sequences capturing facial expression transitions from peak intensity to neutrality. In a video, many frames display imperceptible differences. To maintain a reasonable size, we want to ensure that the images down-selected for the study are reasonably perceptually spaced. Expression transitions occur non-linearly, with peak and neutral expressions often sustained at the beginning and end of each sequence. To ensure key points in the transition are captured, we sampled one frame early in the transition (e.g., frames 1–5), where the expression remained at its peak intensity, and another near the end (e.g., frames n − 5 to n), where the expression approached neutrality. Next, we ran one iteration of QUEST procedure [Watson and Pelli 1983] with 50 pairwise comparisons (See Appendix A.2 for more details). to identify another two frames: one that is 1 Just Noticeable Difference (JND) away from the peak expression, and another that is 1 JND away from the neutral expression as shown in Fig. 2. In total, 462 sequences across eight actors (53-60 per actor as shown in Fig. 3) were sampled in this way. To ensure a balanced representation of the transition, we also selected a midpoint frame between these two JND-determined frames. This sampling strategy resulted in 5 frames, emphasizing perceptually meaningful changes while offering a comprehensive and accurate depiction of expression dynamics. Figure 2 illustrates the sampling process. Five frames were deemed sufficient for our purposes based on informal trials using the QUEST method applied to expressions in this dataset, where we observed approximately 5-6 JNDs from peak to neutral.

Following frame sampling from the expression transition sequences, paired image comparisons are created to prepare the dataset for subjective annotation. These are categorized in two sets:

• Intra-Expression Comparisons: Paired images were drawn from the same expression transition sequence, capturing variations between different intensities of the same expression. For each transition sequence, we generated $ \binom{5}{2} = 10$ comparisons, resulting in a total of 4,620 comparisons across 462 sequences.
  
• Inter-Expression Comparisons: To limit the number of comparisons, we executed two steps. First, we excluded the neutral images from both sequences and randomly selected 3 out of the 4 non-neutral images from each sequence, resulting in 3 × 3 = 9 comparisons across two expression transition sequences. Second, to further reduce the total number of comparisons, we used only half of all possible combinations of expression pairs ($ 0.5 \times \binom{61}{2} = 915$) per actor. Since not all expression sequences were available for every actor, we managed the allocation of inter-expression pairs to ensure each pair appeared four times in the final dataset while maintaining 915 pairs per actor. To accomplish this, we identified 1,830 inter-expression pairs and determined how many of the eight actors shared both expressions within each pair. Next, we sorted the list based on the number of actors. Starting with pairs involving the fewest actors, we allocated each expression pair to four actors using a round-robin approach. Consequently, this segment comprised a total of 915 × 8 (faces) × 9 = 65, 880 comparisons.
  

This results in a total of 70,500 comparisons, between both varying levels of the same expression and across distinct expressions.

Given the dataset's size and the requirement for a large number of ratings per sample, an in-lab study was not feasible. Instead, we conducted an online study using AMT to gather ground-truth human opinion scores on facial expression differences. AMT has the advantage of enabling data collection from a diverse demographic, providing more representative ratings compared to in-lab studies, which are often restricted to homogenous groups, such as university students. To ensure data reliability, a rigorous methodology of quality control measures was employed to exclude unreliable participants (Sec. 3.4.3). Before conducting the large-scale study on AMT, we carried out a small in-lab pilot study with reliable subjects to validate our overall protocol.

3.4.1 Data Organization and HIT. We organized all inter-expression and intra-expression sequences into 354 batches, each batch containing approximately 22 sequences. This resulted in around 200 image pair comparisons per batch for annotation. Each batch was carefully designed to include at least one sequence from one of the eight actors, as well as at least one sequence of intra-expression comparisons. These batches were then presented as Human Intelligence Tasks (HITs) on AMT for annotation. At the beginning of each rating session, all comparisons in the batch were randomly shuffled, and the two images in each pair were randomly assigned to appear on the left or right side of the screen. This was done to minimize any bias in the rating process across subjects related to the order of presentation or the positioning of the image pairs. Before raters could participate in the annotation process, each batch began with a training session designed to familiarize them with the rating process. This included example ratings of expression differences, helping raters understand the diversity of expression differences they would encounter during the task. A screenshot of the exemplar rating screen is shown in Fig. 4. The rating bar is initially set to zero (indicating ‘Identical’) for each trial, and users were required to adjust the bar based on their perceived expression difference.

3.4.7 Processing of Raw Scores. Using the subjective methodology described above, we obtained a total of over 2.2 million subjective opinions from 1,021 unique subjects, with 32 ratings of expression differences per image pair. The ratings obtained in this manner were then processed to obtain a single expression difference score label per image pair in the dataset. The simplest method for calculating this is the Mean Opinion Score (MOS), determined by averaging the subjective ratings given to each pair of images.:

where d_i is the expression difference rating provided by subject i for a specific image pair, and n is the total number of ratings collected for that pair, which is 32 in our case.

A more refined method for determining labels, which we adopt in this work, is presented in the SUREAL framework [Li et al. 2020], and uses Maximum Likelihood Estimates, grounded in the following subject rating model:

where D_p corresponds to the “true expression difference” for image pair p, $ \mathcal {N}(0, 1)$ is a standard Gaussian random variable, b_s denotes “subject bias,” and σ_s denotes “subject variability.” The Alternating Projection solver described by Li et al. [2020] is used to estimate the model parameters. The use of this approach offers several advantages, such as reduced vulnerability to subject bias, the generation of tighter confidence intervals, effective handling of missing data, the ability to deliver comprehensive insights into both the dataset samples and the test subjects, and has recently emerged as the preferred methodology for obtaining mean opinion scores in newer psychometric studies on video quality [Chen et al. 2024; Saha et al. 2023; Shang et al. 2022; Venkataramanan and Bovik 2024].

The distribution of the estimated SUREAL scores of face expression differences for all image pairs in the FaceExpressions-70k database is shown in Fig. 5, revealing a broad range of magnitudes of expression differences all falling within the range [1.88, 87.50]. The intra-expression pairs are shown in blue, while the inter-expression pairs are in red. The histogram for intra-expression differences is almost uniform, while the histogram for inter-expression differences is left-skewed. This is expected, as image pairs resulting from inter-expression differences, typically exhibit greater expression differences than those originating from intra-expression differences. The average standard deviation of the obtained expression difference scores was 2.8, in line with previous psychometric studies on video quality assessment using the SUREAL framework [Chen et al. 2024; Saha et al. 2023]. More discussions in Appendix A.3.

3.4.8 Inter-subject Correlation. To assess the reliability of subject ratings, we analyzed inter-subject correlation by dividing the ratings for each sample in the dataset into two random equal-size subsets. SUREAL scores were independently calculated for each subset, resulting in two quality labels per sample. The Spearman's Rank Order Correlation Coefficient (SROCC) was computed between these labels, and the process was repeated 50 times. The average SROCC across all iterations was used to quantify inter-subject correlation, with higher values reflecting greater agreement among subjects. The resulting inter-subject correlation was 0.914, typical of large-scale psychometric studies on AMT [Venkataramanan and Bovik 2024].

In this section, we analyze subjective facial expression difference scores, exploring patterns of perceived expression similarity and distinctiveness to uncover key trends in human perceptual judgments. The findings show how subtle and dynamic facial changes contribute to the perception of expression differences.

3.5.1 Variation in Peak to Neutral Expression Difference across Expressions. In Fig. 6, we plot the subjective expression differences between peak and neutral frames across eight actors for 41 expressions. The blue lines indicate the range of differences, showing the minimum-to-maximum variation across actors, while the red dots represent the median values for each expression and black ‘x’ for individual actor values. The plot reveals notable trends, with expressions such as “Lips Together Pushed Forward” and “Jaw Back” exhibiting large variability, indicating that the same expressions were perceived as being highly different across actors. In contrast, expressions like “Raise Upper Lip Scrunch Nose” and “Jaw Open Huge Smile” show narrow ranges, indicating a more uniform portrayal and perception across actors. Fig. 7 illustrates this phenomenon. High-intensity expressions, such as “Show All Teeth” and “Open Mouth Wide Tongue Up and Back,” exhibit higher median values, reflecting significant perceptual differences from neutral. By contrast, subtle expressions like “Raise Inner Eyebrows” show smaller differences, suggesting they are less perceptually distinct. Thus, subjective perception of facial expressions varies significantly across actors, with some expressions showing large variability and others being more consistently perceived. These observations highlight the complexity of subjective expression perception and the need to account for actor-specific variability when building robust facial expression differencing models.

3.5.2 Variability in Expression Transitions. Expression transitions and differences from neutral are not uniform, varying across different actors and expressions for the same actor. Analyzing intra-sequence images reveals that expression differences from neutral vary significantly, both between actors and within the same actor across different expressions. Fig. 9 illustrates these trends by plotting the normalized expression differences from neutral: in the two plots on the left, actors are kept constant to analyze variability across expressions for the same actors, while in the two plots on the right, expressions are kept constant to examine variation across different actors for the same expressions. The transition back to neutral after reaching the full expression shows variability in timing and slope during the recovery phase. This highlights that individuals not only require different amounts of time to exhibit and recover from the same expression, but also show unique temporal patterns across different expressions, emphasizing the complex nature of expression dynamics, shaped by individual differences and expression-specific factors. More visualizations are in Appendix A.4.

In Sec. 4, we benchmarked the performance of existing methods using the FaceExpressions-70k dataset. However, the dataset can also be utilized to train improved facial expression difference predictors. To demonstrate this, we extracted features from the best-performing method in Table 2, FaRL, and trained a support vector regressor (SVR) model using a linear kernel to regress from differences in embeddings to expression difference scores. The training was conducted using a 6:2 actor train-test split across 28 iterations, and the median performance was recorded. Following previous studies [Chen et al. 2024; Saha et al. 2023], this split was done along the “actor” variable to prevent content bias from influencing the test data. The SVR model achieved a relatively stronger performance, with an SROCC of 0.811 and a PLCC of 0.823. These results indicate that even simple regression models using features extracted from pre-trained models can improve expression difference prediction. This suggests that custom methods to measure expression differences could achieve even better performance.

In virtual reality applications, HMD (head-mounted display) cameras continuously track users’ facial expressions to ensure accurate face avatar generation. High tracking accuracy often requires dense sampling, which increases computational load. We demonstrate that the data from our crowdsourced study can be used to determine sampling intervals that avoid perceptible lag between the user's facial expressions and avatar counterpart while balancing efficiency. Another closely related application is determining the acceptable latency for driving face avatars in Virtual Reality, considering rendering, transmission, and other processing delays. Both applications require selecting time intervals to ensure no perceptible difference in expressions between the two sampled time stamps.

To achieve this, we first determine the expression difference magnitude corresponding to 1 JND, which we will use as a threshold. Leveraging our dataset design (see Fig. 2), we have an abundance of image pairs with annotated expression differences sampled 1 JND. We compute the median of these annotations, yielding an expression difference of 20.39. Next, using a linear expression transition approximation (valid only for small differences) between consecutive pairs of the three intermediate intra-sequence samples, we computed the transition time required for a 1 JND expression difference (20.39 MOS). This was repeated for each expression, and the per-actor statistics using box plots are shown in Fig. 11 (left). The first and last intra-expression samples were excluded, as the peak and neutral expressions were held constant and do not reflect expression transition dynamics reliably. From the figure, we observe that an approximate 11 ms (shown in red) interval, or 90 frames per second, ensured no perceptible expression difference across expression and actors. However, if we customized the selection for each actor, higher intervals could be used—for example, for actor 7, the interval could be as high as 22 ms. We further demonstrate the alignment of the predicted JND with visual perception for two expressions in Fig. 11 (right).