Inferring Age from Linguistic and Verbal Cues in Celebrity Interviews

Both VoxCeleb1 and VoxCeleb2 provide gender information of the speaker as well as the YouTube URL from which the video is extracted. Only VoxCeleb1 provides celebrity names, while VoxCeleb2 is anonymous. To add speaker name to VoxCeleb2, we request the YouTube video title through YouTube API, and run a named entity recognition algorithm on each video title. After extracting PERSON entities, we look at all the names corresponding to the same speaker id. If the count of the most common name is greater than double the count of the second most common name, we select the most common name as the celebrity name. Speakers for which we are not confident about their names are removed from the dataset.

Age can be annotated given speaker names and video URLs. Since YouTube videos do not have recording dates, we use the video publish date of each interview as a proxy for its recording date. Video publish dates are requested through YouTube API, and birth dates of celebrities are scraped from Wikipedia. Age at the time of video is calculated as the difference between video publish date and celebrity birth date. We remove the video clips from the dataset when it falls into one of the following cases: 1) YouTube URLs are no longer available 2) multiple people may have the same name so we cannot determine who is the speaker 3) birth date is not public on Wikipedia. Celebrities who have passed away are also removed from the dataset, because some of the videos are published after their death date.

After these steps, 730,281 utterances from 4,432 speakers remain in the dataset. Of the remaining speakers, 34.7% are female, and 65.7% are male. The dataset is then partitioned into non-overlapping training set and test set. Number of clips for each integer age is shown in Figure 2 (left), and a plot of number of celebrities vs. number of appearances in the dataset is shown in Figure 2 (right).

Data and code will be available on GitHub after publication of the final article (https://github.com/sbu-dsl).

We experimented with multiple audio embeddings including i-vector, x-vector, and VGGVox embedding, among which VGGVox embedding outperforms the rest. VGGVox is a neural embedding system based on VGG-M CNN, which is known for good classification performance. The model takes spectrograms computed with 16,000 sampling rate and frame length 0.025 as input, extracts and pools frame-level features into utterance-level speaker features, and outputs 1024-dimensional embedding. The model architecture is shown in Figure 3.

Sentence embeddings capture semantic information and represent sentences as feature vectors. We first obtain transcripts of the audio clips using the Speech Recognition library (https://pypi.org/project/SpeechRecognition), and then use the SentenceTransformers library [18] to extract sentence embeddings from the following pre-trained models:

• MPNet: A model trained on a large-scale dataset (over 160GB text corpora) and fine-tuned on a variety of down-streaming tasks (GLUE, SQuAD, etc). The model maps sentences into 768-dimensional vector space capturing semantic information [19].
  
• Multilingual Universal Sentence Encoder: A retrieval focused multilingual sentence encoding model that maps sentences to cross-lingual representations [20].
  
• ALBERT: A lightweight variant of BERT that models inter-sentence coherence and uses parameter-reduction techniques to lower memory consumption and increase the training speed of BERT [21].
  
• MiniLM: A variant of BERT that uses task-agnostic model compression and conducts deep self-attention distillation. In its teacher student model, the student is trained by deeply mimicking the self-attention module of the teacher. It also uses the technique of distilling the self-attention module of the last Transformer layer of the teacher, as well as using scaled dot-product between values in the self-attention module as the new deep self-attention knowledge [22].
  

We experiment with these sentence embeddings on the age regression task. The annotated dataset is randomly splitted into training set (60%) and test set (40%), in which there are no overlapping celebrities. Age regression accuracy based on sentence embedding alone is shown in Table 1. Principal component analysis is performed on each type of embedding to observe accuracy with reduced dimensions.

For age estimation using combined audio and text features, we use the same training set and test set split as that in the previous section. We experiment with three regression models and different combinations of the features. The mean absolute error measured in years is shown in Table 2. The audio embedding alone performs best when its original dimension is preserved, and the accuracy of each model improves as text embedding is concatenated to audio embedding. Overall, ridge regression using combined VGGVox and MPNet embeddings performs best and achieves an error of 8.22 years, which means our prediction is about 8 years off on average for all the celebrities in the dataset.

For the age classification task, celebrity ages are divided into five equal buckets so that each bucket has the same amount of data. We divide the data in this way to mitigate the problem of imbalanced data since most of the celebrities fall into middle age groups. We apply a variety of machine learning models including logistic regression, support vector classifier, decision trees, random forest, KNeighbors classifier, AdaBoost [23], multi-layer perceptron, and LSTM to perform multi-class classification. Description of the classification models used are as follows:

• Logistic regression with one-vs-rest scheme using L2 penalty is used for our experiments, which calculates the argmax of probabilities obtained by each model to make predictions.
  
• Support Vector Classifier finds a hyperplane that best divides a dataset into separate classes and maximizes the margin between the hyperplane and points closest to the hyperplane.
  
• Decision trees infer decision rules from training data and fork in tree structures until a prediction decision is made for a given record.
  
• Random forest is an ensemble method consisting of an uncorrelated forest of decision trees to make overall combined predictions.
  
• KNeighbors classifier calculates the distances between test data and all training data, selects a specified number of points closest to the test data, and votes for the most frequent label.
  
• AdaBoost is a boosting technique that combines weak learners and adjusts error metric for each iteration to find the best split. The algorithm is able to identify outliers, i.e., examples that are either mislabeled in the training data, or which are inherently ambiguous and hard to categorize [24].
  
• Multi-layer perceptron is a neural network structure that optimizes the log-loss function using stochastic gradient descent.
  
• LSTM is a recurrent neural network that recognizes long-term dependencies in sequential data.
  

The classification accuracies are higher for female celebrities, although there are fewer of them in the dataset. Logistic regression achieves the best classification accuracy of 46.1% on female subjects and 40.0% on male subjects. The overall accuracies of these models are shown in Table 3.

Normalized confusion matrices in Figure 4 show classification accuracy for each age group. Numbers shown on the diagonal represent the percentage of correctly classified subjects for each age bucket. The youngest and oldest age groups have the highest classification accuracy, while the middle age groups are more difficult to distinguish, which is similar to human performance.

According to accuracies shown in Table 2, all prediction models improve when the sentence embeddings are added. We perform text length analysis to see if the length of text affects age prediction. We expect the error to be reduced for clips with longer transcripts. The length distribution of each audio clip measured in number of words is shown in Figure 5. The text lengths form a normal distribution centered around 15 words per audio clip with outliers clustered around 5 words per audio clip.

We further examine the clips with word count lower than 40, since most of the clips fall in this range. Figure 6 shows the dot plot of average prediction error for each word count. The age prediction error is reduced when there are more words spoken in the interview clip as expected. There is a negative correlation of -0.252 between average prediction error and number of words.

Meaningful age prediction results should reflect each individual process of aging. We examine this process using our best regression model for the two age groups we find easily identifiable – the youngest age group containing children and young adults, and the oldest age group containing older adults. The number of points that need to be deleted to make these polylines monotonically increasing are calculated as a longest increasing subsequence problem. The average number of points to be removed per speaker is calculated for comparison. Based on the result, aging in the youngest age group is best captured. To make the polylines monotonically increasing, 1.25 and 1.57 points need to be removed for female and male respectively. To visualize how our estimated age changes as the real age of speaker increases, dots belonging to the same speaker in the youngest age group are connected and drawn as a polyline in Figure 7. Asa Butterfield in the male celebrity plot is one of the celebrities showing good aging prediction. He has 4 monotonically increasing data points ranging from 14 years old to 19 years ago. On the other hand, the age prediction result for Dakota Fanning in the female celebrity plot does not reflect the process of aging, as her predicted age decreases slightly when she ages from 16 years old to 22 years old.

Permutation tests are performed to check if our result is statistically significant. Random permutations of estimated age are generated, and the number of points to be removed to satisfy monotonicity is calculated for each permutation. Table 4 shows our calculated statistics for each age group. The p-value for female and male celebrities younger than 30 are 0.006 and 0.019 respectively, which indicates that observations in the youngest age group are not likely to be obtained by chance, since they have a p-value close to zero.