Social Media for Predicting the Mexican Presidential Election: Beyond X and Facebook

Describing Mexico´s electoral process, presidential elections are held every six years. In 2024, they will take place on June 2. The party in power, MORENA, has already defined its candidate with anticipation: former Mexico City´s state chief Claudia Sheinbaum. In the same sense, the opposition side has made official their trust in federal senator Xóchitl Gálvez to represent their 3-party alliance: PRI, PAN, and PRD. In a sudden turn of events, the other major party that remained unrepresented, Movimiento Ciudadano, originally decided to give Nuevo Leon´s governor Samuel García their support. Still, after some state-related conflicts around García´s current role, Movimiento Ciudadano opted for federal senator Jorge Álvarez Maynez.

The race between Gálvez and Sheinbaum, the two main candidates in the election, was officially announced on September 7 (when Sheinbaum was declared the candidate for MORENA), the major and most trusted news sites in Mexico in accordance to the 2022 Digital News Report by Reuters had Sheinbaum in a far-off first place with around 50% of the preference, Gálvez reaching 25% of the preference, and García with around 10%, except for unique cases of polls not ranked by Reuters which had Movimiento Ciudadano's candidate with more than 20% of the voting preference and Gálvez with close to 15% [23], noting the case for this study.

Mexico´s official electoral authority, INE (Instituto Nacional Electoral), has approved 19 polling institutions [10] to conduct polls for the 2024 electoral process. However, results from other organizations are used regardless of their credited status by the current candidates in campaigns. It is key in this case to identify the source of each result shown.

In the current study, we will refer to the analyzed polls as those published by Oraculus. mx [[19] which is a credited and independent organization in charge of ’polling the polls’ or contrasting their results to form a single indicator from all of them. Their public and documented selection process to include a poll or not is based on credibility by experts in the subject. Their output list will be selected to perform all the experiments for the next sections in this work. In the same sense, its constructed prediction will be used to contrast the set of predicted results by the current study. Their founders, Jorge Buendía, Juan Ricardo Pérez Escamilla, Javier Márquez, and Leonardo Zuckerman are renamed and well-acknowledged journalists in Mexico and created Oraculus to propose a new methodology to traditional polling in the country.

Around the polls considered, this proposed a major challenge. In the Mexican election context, there are no predefined guidelines to assign a trust level to each organization that may perform a polling process and publish results. For this, we considered only those that Oraculus.mx [19] published on its official site, as previously discussed. Since Oraculus does not provide a concise day for each published poll but rather only the month in which they were performed, we assigned the collected polls to be conducted starting on the 15th of each month. This is not a particular detail of interest, but rather a reference guideline to start counting the predefined set of days. e.g. the first poll was labeled on the 15th, the second on the 16th, and so on.

As previously discussed, the general trust in the polling system has encountered a decline. In the same sense, recent sanctions and predictive errors have also contributed to damage the collective trust in the polls. For this, the current work takes Oraculus as a solution: while we take polling results as a part of the modeling and predictive processes to prove machine learning models along with real-world indicators (polls in this case) can outperform traditional methodologies, we only consider highly evaluated and studied pollsters that were previously analyzed and filtered by the Oraculus.mx team based on historic results. Since we praise the inclusion of offline features to contribute to the model, and not only consider those from social media or online platforms, the inclusion of polls in this case is needed. We support this in the evaluation section also.

As the final consideration for the current work, the linear regression and the multi-layer perceptron algorithms are used to model polling results, like in [4], and following the depicted set of hyper-parameters described in the research to be latterly described. Although various regression models can be used to try to predict the result of the 2024 Mexican presidential election in further research, the principal focus of the present work is to prove polling behavior via regression analysis can be both replicated and predicted, while also analyzing which features of the described dataset may be more important for this process and why it may be the case.

In addition to the modeling selection, the incoming data will be modeled with the usage of scaling. We will be referring to standard scaling as detailed in the Scikit-learn documentation [21]. Principal Component Analysis (PCA) was discussed in [4] as an alternative to avoid dimensionality problems as exposed in [27]. However, since we can not ensure a linear relationship between all the features included and the objective of the experiment is the analysis of the importance each social media platform may have in expected predictions, PCA will be discarded as a dimensionality-reduction technique, as it has been stated multiple platforms should be considered for predictions.

Finally, we´ll discuss the integration of all the predictions made for each time frame (1...7,14,21, and 28 days). An ensemble of predictors was created for this set of experiments. Each model (LR and MLP) is trained on a different dataset consisting of each selected period before the poll is considered. This results in 10 different predictions for each candidate. These results are then averaged to form a single, predicted result. This was based on the methodology in [4].

Moreover, the predictions and the overall process can be described as follows:

1. Build a database that outputs the number of features before each time frame for each poll. The number of rows in each dataset represents a poll from September 2023 to January 2024 (see Figure 3 for an extract)
   
2. 10 different datasets were considered for each candidate (since we count with 10 different time-frames)
   
3. Both ten LR models and ten MLP models are trained with each of the databases mentioned above.
   
4. We construct two instances of each model since we consider both all features and only X, and Facebook features
   

Sample dataset constructed for YouTube and a single-day period

To evaluate RQ1, we calculated the Pearson correlation coefficient r between the social media metrics selected and the corresponding voting preference depicted by the polls. In this case, since we still don´t count on a unique reference to guide our evaluations (as a final election result would be), we considered the average percentage of preference for each month for each candidate. This is, we obtained the monthly mean of all the included polling results, compared it with the social media performance of every candidate, and analyzed the correlation between those metrics up to the last day of January. The coefficient r may vary from -1 to 1. A value of 1 would mean that as each variable increases the other one does too, depicting a linear correlation. A value of -1 means there exists a perfect negative correlation between the two variables and a zero value translates into not being correlated at all. More specifically, we considered a value for r to be greater than 0.7 to conclude a high correlation exists between social media performance and polling results, following the commonly accepted margin.

For RQ2, two predictions for each model are performed: i) one considering a dataset constructed by features only from X and Facebook and ii) a prediction based on datasets formed by features from all social media applications. Moreover, Mean Absolute Error (MAE) is reported for each model in contrast with Oraculus´s results.

To evaluate the importance of the included features and their role in a predicted result, we analyzed the feature importance for each model after the first round of experiments that included only features belonging to X and Facebook. After this, we performed the feature importance evaluation for the models including all features from all social media platforms including X, Facebook, Instagram, and YouTube. We´ll count the number of features from the second round of experiments that may appear above the first set of features, providing a set of the features from the newly included apps that now appear above the features from X and Facebook.

Secondly, each prediction was contrasted with the single prediction that Oraculus provides, this is, the predicted result of the experiments will be analyzed with the ’poll of polls’ indicator provided by Oraculus in its "Presidential tracker". This is because we consider the same polls as them and the results of this research can be categorized as a "poll of polls" on its own, but with another set of processing methodologies, now based on social media. This evaluation metric is used since official election results are still unknown and will remain so until June 2.

There is no defined date on which Oraculus bases its monthly prediction. For this, we assigned the features to be summed up before the last day of the testing month (January in this case), e.g. all features are considered and summed up to be used as testing metrics as if Oraculus made its final prediction for January on the 31st.

We do not intend to compete directly with Oraculus.mx. Moreover, we intend to create a completely different methodology to what they do, based now on social media metrics. However, we do consider Oraculus.mx as the guideline to include one poll over another. This is because we believe they provide a necessary filter to know which polls should be considered by the analysis. The extraction of their aggregated polls is used as a way for this paper to save the process of manually looking for polls to include without the proper, unbiased study of each one of them. To the best of our knowledge, this is an innovative approach in the field of election predictions in Latin American countries, as the literature provides no guideline on why a poll may have been considered or not.

To state that a set of features is ’significant’ enough to influence the result of a prediction, we considered the work presented by [14], which took the historic MAE error of 2.7 points to categorize predictions as successful. This metric was measured along the 2 resulting predictions for each model and Oraculus´s result. To conclude significance, we evaluated if a 2.7-point differential existed in the set of results produced by the models which only varied in the networks they were trained upon. If there was such a difference, we declared the inclusion of Instagram and YouTube (apps to be evaluated) significant enough to influence the state of a prediction.

The set of experiments conducted and the retrieval of polls for this work are considered to be around the polls finalized before February 1st.

Initially, it is crucial to contrast the polls considered throughout the current work. Table 2 presents a summary of the metadata from the included polls in the research. In contrast, Table 3 indicates the number of polls each institution performed until the last round of experiments was performed. High variation is expected from these as electoral preferences change over time.

With this, we graph the preference for each candidate as reported by the polls throughout the period we established in previous sections and their reported difference. It is important to add the following figures which cluster together the polls included throughout the analyzed period, as it is easier to observe the number of outliers and the difference that may exist between different values of predictions for polls, see Figure 4:

Point difference between Claudia Sheinbaum and Xóchitl Gálvez by the polls (from September to January)

As pictured in Figure 4 b, the polls seem to agree more on the voting preference for Gálvez, as narrower boxes are depicted in the graph. However, for Sheinbaum, Figure 4 a, presents a wider set of records, especially in January where the polls do not seem to have a concise margin of predictions. It is important to also mention that outliers occur for all the presented pictures, reiterating the fact polls may mark a somewhat clear pattern but not a concise record of results. Similarly, we must note the size of some boxes included and the presence of outliers in the same month.

We can observe a narrower set of records in September concerning the difference between the candidates (Figure 4 c), which indicated a shorter margin between Sheinbaum and Gálvez initially. However, the distance progressively increased in both the margin of predictions and the number of outliers, providing an even less clear image of the real distance between them. As mentioned in previous sections, a wide distance of 25 points between candidates in Estado de México resulted in an eight-point victory for one of them. With this set of results, we can conclude that great margins may not represent the real results of the election, especially with such significant outliers present in the polls, which are completely unaligned from the majority of included ones.

We can observe a growing tendency for the voting preference for Claudia Sheinbaum. The same can be mentioned for the gap reported by the polls between the two candidates. Gálvez on the other hand, appears to have had her highest preference peaking in September of 2023, which can also be referenced in Table 4.

The second phase consisted of examining a given set of metrics across social media for both candidates and aimed to find a correlation between what is observed above depicted by the polls and the online performance of both Sheinbaum and Gálvez. For this, we considered the Pearson correlation between the average polling results (Table 4) and the performance of the candidates in social media with the distinct set of metrics mentioned above.

We must also obtain a general overview of the collected data for each candidate, as seen in Table 5 and Table 6.

The crucial factor was then to find any correlation that may describe the behavior of the polls based on the social media metrics listed in Tables 5 and 6. To do so, we analyzed the work in [8] in which social media performance (Twitter mentions for different candidates) is correlated to the number of votes these received across their disputes.

For this, we attempted to find a correlation between the metrics mentioned above through their monthly occurrences and the averaged polling results in Table 4. We analyzed which were the features with the most correlation. We sought to visualize them in a correlation matrix, evaluating our results based on the Pearson correlation coefficient r to assess which of those influenced the average polling performances the most.

The results of the correlation experiments are depicted in Figure 5. We can observe the different Pearson correlation coefficients for each feature with the electoral performance of each candidate across time. The features with the highest correlation with the electoral performance mentioned above appear with the darkest tones and at the top of the matrix. We can ignore the Share as this simply corresponds to the that we are aiming to find a correlation with. This coefficient r provides insightful information about how features from commonly ignored social media apps like Instagram can better correlate to the voting performance of a candidate, e.g. the likes per post on Gálvez´s Instagram profile (Figure 5 b) is the single most positively correlated feature that leads and ’tracks’ her polling average. It is the only feature surpassing the r value of 0.7, concluding it would be critically wounding the predicting performance of a model if excluded. As for Claudia Sheinbaum, we observe all engagement metrics for Facebook are features above the r value of 0.7, while the likes of Instagram also equal r to 0.7 (Figure 5 a).

Feature correlation with polling performance for Xóchitl Gálvez

In this way, we evidenced the correlation that features from other social media apps, like Instagram especially, may have on the polling performance of a candidate. Interestingly, none of the features examined from X (Twitter) provided a high Pearson correlation value r to justify its highly adopted usage across research in the field.

For RQ2, we took the lessons from RQ1 to confirm the importance of the features we obtained to be the most correlated with the importance these may have had in modeling experiments.

Both linear regression and the Multi-Layer Perceptron algorithm were tested as described in [4]. The LR algorithm was implemented without any set of parameter tuning. On the other hand, the MLP model included the following parameter setup: one hidden layer with three neurons, L-FBGS as a solver, an alpha set to 0.05, a constant learning rate, and a logistic activation.

As previously discussed, two rounds of experiments for each model were performed: one with only X and Facebook features and another one with features from all platforms mentioned.

The results can be summarized as referenced in Table 7.

Before describing them, we can also observe the importance each feature takes in the Linear regression algorithm, as feature importance is clear to visualize. Figure 6 a refers to the feature importance for Sheinbaum coming only from X and FB, while Figure 6 b refers to all features from Sheinbaum. Figure 6 c refers to the feature importance for Gálvez coming only from X and FB, and Figure 6 d refers to all features from Gálvez.

Feature importance for Linear Regression for both candidates

For the Multi-Layer Perceptron model, we will solely use the MAE as the evaluation metric when contrasted with Oraculus´s reported preferences. As evaluated with the Linear Regression results, Table 8 includes the complete set of metrics.

As shown, the worst set of results was reported by the MLP model implemented with only those features belonging to X and Facebook solely. More importantly, the inclusion of features from Instagram and YouTube changes the status of a failed prediction (from 9 MAE to 6 MAE) to a successful voting reported preference as described by the MLP (ALL) model. This is an important factor for the development of a predictor, as features from social media applications other than X and Facebook can be a determining factor when evaluating different machine learning models. Concerning the linear regression algorithm, the inclusion of features from Instagram and YouTube only appears to make the predictor 1.5 points worse. This difference is less than the 2.7-point historic MAE result for election margin and is considered, therefore, not significant. Moreover, this does not affect the ’successful’ state of the LR prediction.

Moreover, it is also clear that the Linear Regression model considering only X and Facebook performed better than the iteration considering all social media apps mentioned in this paper. In this case, it is important to point out that while this precise example may argue against the inclusion of other social media features in predicting models, we argue the possible differences in other sets of model experiments (3.5 point difference in MLP for instance) is sufficient to support this inclusion of YouTube and Instagram or their addition into future work in the field over their complete omission. Also, this claim references the different Pearson correlation results we analyzed in previous sections, in which features from Instagram and YouTube had a greater correlation with electoral performance than those of X and Facebook.

When considering the results for Sheinbaum (Figures 6 a and 6 b), we can observe that most of the newly added features from Instagram and YouTube appear to have negative importance to the prediction result. This is, the more negative a feature is the more likely it can be omitted and not impact the result. However, this negative impact is nowhere close to most of the features Facebook already has on the model. In the same sense, the number of Instagram posts appears to be the most influential factor of all the features considered in the second iteration of the model.

Xóchitl Galvez´s results (Figures 12 and 13) are quite in the same direction as Sheinbaum´s: most of the features included even contribute positively to the outcome of the model. While the most important features remain to FB and X, none of the newly considered ones other than the views from YouTube affect the model in a significant way.