Trust in AI is dynamically updated based on users' expectations

Three hundred participants from the United States self-selected into the current study via the online participant recruitment platform Prolific [37]. Sample size was determined a priori using a power analysis, which indicated that 80% power would be achieved for detecting a small-to-medium effect size for interaction terms in our design. After complete data was returned, a final set of 285 participants (mean age = 34.84 ± 12.26; 102 males, 172 females, 11 unknown) were retained. Participants were reimbursed at £6.60/hr and provided informed consent prior to undertaking the study. The experiment was approved by CSIRO's Social and Interdisciplinary Science Human Research Ethics Committee (245/23).

Stimuli were presented using jsPsych v7.3 [8] and were delivered to participants via an in-house server using JATOS [26]. Participants performed a deepfake classification task, where a face was presented, and participants were asked to decide if the face was ‘real’ or ‘fake’ (Fig. 1A). Fifty genuine faces were sourced from the Flickr-Faces-HQ (FFHQ) dataset [22] and 50 artificially generated faces from the StyleGAN2 dataset [23] from Nvidia Labs. Critically, the deepfakes were created after being trained on the FFHQ database. Author AL selected an ethnically and gender diverse set of faces. In a separate pilot study, human classification accuracy for this set of selected faces was 66.45%. To aid in their decision, we presented advice from an AI assistant which suggested that the face was likely to be real or fake. Note, this AI was simply an algorithm that provided the correct image identity ∼75% of the time, a level deemed appropriate from results of a pilot version of the task, used in previous cognitive forcing work [4], and similar to that shown to facilitate human-AI collaboration [48]. In practice, AI accuracy was slightly less than 75% due to sampling stratification and the exclusion of training trials from analyses. After deciding the nature of faces, participants rated their confidence in their own detection ability, confidence in the AI's detection ability and trust in the AI assistant. Finally, feedback about whether their decision was correct was presented. Feedback was deliberately presented after confidence ratings to reduce participants calibrating their subjective ratings toward the objective feedback of their performance [2].

Figure 1. Illustration of the AI-assisted deepfake detection task with one-step (A) and two-step (B) decision processes. In (A), participants see a face alongside an AI label (e.g., “AI thinks this face is REAL”), then choose ‘REAL’ or ‘FAKE.’ Next, they rate confidence in their own detection, confidence in the AI, and trust in the AI, followed by a feedback screen showing whether their decision was correct. In (B), participants first decide on ‘REAL’ or ‘FAKE’ before seeing the AI's suggestion, then complete the same rating and feedback steps.

The study used a between-subjects design. Participants were randomly assigned into equal sized groups (n=150) associated with the different decision step combinations¹ (Fig. 1B). First, for the one-step condition, AI advice and the face image were presented contemporaneously, allowing participants to incorporate AI advice and their own judgment when making their decision. In contrast, for the two-step condition, participants first made their classification choice without AI advice and then were presented the same image alongside AI advice requiring a second decision. This two-step decision served as our cognitive forcing manipulation.

Prior to the experiment, participants were given brief instructions and provided three practice trials to familiarize themselves with the task. For the experiment, on each trial, participants were first presented with a preparation screen for 750 ms, with a prompt to ‘Get Ready’ and a reminder of the response key mapping. Next, a blank screen was presented for 250 ms, replaced with the appropriate decision stage screen dependent on the AI advice condition. Real vs. fake decisions were made with key presses ‘z’ and ‘/’ respectively. After the decision was recorded, participants used their mouse to rate their 1) own confidence “Confidence in your detection ability for that face?”, 2) confidence in the AI “Confidence in the AI's detection ability for that face?” and 3) trust in the AI “I can trust the AI assistant.” For confidence, participants rated on a 4-point scale, going from 1 = “Complete Guess” to 4 = “Absolutely Certain” and for trust participants responded with a 5-point scale: 1 = “Not at all” to 5 = “Completely.” After ratings, feedback about the participant's decision was presented, with an accompanying confetti animation for correct answers and a sad robot face for incorrect answers.

Only valid trials from the experiment were included in analyses. Valid trials comprised those with RTs > 200 ms and RTs < mean RT + 3 SD. An average of 1.47 ± 0.93 trials were removed per participant from the overall analyses. Analyses were undertaken using the R programming language [41]. To assess the influence of AI assistance on deepfake detection, we compared performance without AI advice against the one-step and two-step conditions. As we did not include an explicit group without AI, we used the decisions in the two-step condition when participants did not receive AI advice (i.e., their first response).

Next, we used mixed effects modelling to assess the predictors of deepfake detection ability and trust within our design. This was due to the design of our cognitive forcing manipulation, where some participants would see identical stimuli (i.e., the two-step condition) [34]. Specifically, we used the lme4 package [1] to perform logistic mixed effects modelling to predict detection accuracy and agreement as a function of assistance type and trust. Agreement rates were defined as the proportion of trials in which the participants made the same decision as the AI. All models included Participant and Stimulus as random effects and Trial and the respective variable of interest as fixed effects.

Finally, to examine how trust developed, we employed a computational model that estimated how participants updated their trust scores based on the veracity of AI advice. For these updates, we implemented a reinforcement learning framework based on prediction errors (i.e., a Rescorla-Wagner learning rule [43]). We estimated a participant's current belief in the expected veracity of AI advice, according to:

Next, to determine the relationship between trust and these violations in expected AI veracity (i.e., prediction errors), we used linear mixed-effects modelling to predict observed trust scores from the model estimates of prediction errors.

General fitness of models was assessed using a likelihood-ratio test and where appropriate we used Akaike Information Criterion (AIC) fit estimates to compare models.

Overall, participants were able to detect if a face was real or deepfake successfully on 67.86% ± 2.41 of trials, significantly greater than chance (χ² = 1646, p < .001)² . This is comparable to the AI assistant's mean accuracy of 72.76% ± 2.3 but does not realize collaborative gains by exceeding it. We next compared accuracy as a function of AI assistance type (i.e., None, One-step, Two-step) to determine whether this lack of collaborative improvement was driven by heuristic-based decision-making, and consequently rectified by cognitive forcing. Given the lack of a distinct baseline group, and that participants saw different combinations of stimuli (which may have varied in classification difficulty) we used a logistic mixed effects model that included Assistance Type as a factor to control for such confounds:

A likelihood-ratio test demonstrated that the model including Assistance Type was a better fit for the data than one without (χ²(2) = 15.9, p < .001). Model estimates revealed that having any AI assistance improved accuracy on the task, with both one-step (β = .033, SE = .012, t(502) = 2.86, p = .005) and two-step (β = .041, SE = .011, t(1586) = 3.8, p < .001) more accurate than the decisions made without AI support. However, two-step decision making (reflecting cognitive forcing) did not lead to significantly greater accuracy than one-step decision making (β = 6.8×10^-4, SE = .012, t(502) =.58, p = .56).

We similarly assessed how rates of agreement with AI recommendations were influenced by AI assistance type. Here, we applied a similar model:

Next, we tested whether trust in the AI assistant was related to performance on the task. First, we examined if classification accuracy was associated with trust:

A likelihood-ratio test demonstrated the model including Trust was a better fit for the data than one without (χ²(1) = 152.1, p < .001). Model estimates revealed that increasing trust in the AI was associated with increased human decision accuracy, and that at the highest levels of human trust, collaborative performance exceeded the accuracy of the AI system alone (β = .064, SE = 5.041 ×10^-3, t(2047) = 12.67, p < .001, see Fig. 2).

Figure 2. Two panels displaying model outcomes as a function of trust in the AI. Left panel shows a rising probability of making the correct decision (y-axis) as trust in the AI (x-axis) increases. Right panel shows a similar positive relationship between trust in the AI (x-axis) and likelihood of agreeing with the AI (y-axis). In both panels, the blue line represents the modeled effect and the points represent observed data.

These findings suggest that trust may be driving underperformance of human-AI pairs, that is, participants with low trust in the AI recommendations are not agreeing with the system despite its higher accuracy. To examine this, we tested whether trust predicted an increase in the likelihood of agreeing with the AI's recommendation. Here we ran two models:

The model including Trust was a better fit for the data than the null model (χ²(1) = 1741.3, p < .001), with greater trust in the AI increasing the probability of agreeing with AI (β = .247, SE = 5.391 ×10^-3, t(7043) = 45.84, p < .001, see Fig. 2). Taken together, higher levels of AI trust were associated with both increased agreement with AI suggestions and increased accuracy.

Given that trust in the AI assistant was a strong predictor of successful deepfake classification, we explored how trust was established and updated over the course of the task. Although participants were informed a priori about the AI's typical accuracy, trust was not a stable value throughout the task, suggesting factors beyond explicit performance metrics of the AI influenced trust. Instead, participants might dynamically adjust their trust - and their willingness to rely on AI advice - based on their recent experiences with the AI's performance. Such adjustments could occur gradually or, as suggested in previous work [9,24], rapidly in response to recent experience. To examine this possibility, we compared two mechanisms of trust adjustment:

• Reactive updating (AI advice model): Trust increases or decreases based solely on the accuracy of AI advice in the previous trial, independent of prior expectations. For example, if the AI makes an error, trust is reduced.
  
• Expectation-based updating (prediction error model): Trust dynamically aligns with participants' internal expectations of the AI's performance. In this case, deviations from expected accuracy - termed prediction errors - drive changes in trust. For example, if the AI is expected to make an error, and it does, trust may not be negatively impacted.
  

To formalise this second mechanism, we adapted the Rescorla-Wagner learning rule as detailed in Equations 1 and 2.

Because prediction errors are signed, this model captures both the magnitude and direction (valence) of discrepancies: larger δ values indicate greater surprise and drive larger updates in trust, either positively or negatively, depending on whether the AI performed better or worse than expected.

To determine how these factors influenced trust in AI, we ran mixed-effects linear regressions with the format:

First, we evaluated the two models separately. For the Reactive updating AI advice model, we found that trust ratings significantly increased when AI advice was accurate on the previous trial (Table 1). This model provided a significantly better fit than a null model with no predictors (χ²(1) = 18.217, p < .001) and demonstrated strong correspondence between observed and predicted (i.e., derived from modelling) trust (r² = .472).

Next, for the prediction error model, we found including prediction error was a significant improvement on modelling vs. the null model (χ²(1) = 175, p < .001). Our modelling revealed trust ratings significantly increased with larger prediction errors (Table 1). That is, reported trust in AI increased when AI performed better than expected. Predicted trust values from this model were strongly correlated with observed trust values (r² = .482).

Finally, we compared the AI Advice model and the Prediction Error model directly. The Prediction Error model provided a better fit to the data than the AI Advice model (ΔAIC=161.68), suggesting trial-by-trial prediction errors better explain changes in trust than reactive updates based solely on previous advice accuracy.

Our findings support growing evidence that trust in an AI assistant critically influences human decision making. Trust calibration has been argued to be essential for appropriate reliance on AI, with mis-calibration leading to over- and/or under-reliance on the decision support system [4,32,48,49]. However, to date many trust measures are temporally disconnected from the AI advice, e.g., after a block of trials (e.g., [4]), at the start/conclusion of the experiment [33] or derived from behavioural/physical measures [36,49]. Trust in AI appears to develop differently than trust in humans, with people tending to have a biased view of AI capabilities prior to use and then rapidly distrusting AI upon observation of mistakes [11,33]. We found that these rapid changes in trust ratings impact performance, including both task accuracy and agreement with the AI. Computational modelling approaches, like those performed here, allow individual differences in trust trajectories to be examined. While we did not examine what subjective factors influenced these individual differences, future research may consider exploring the nature of variable trust setting.

Despite AI advice improving task performance, our results did not support the idea that cognitive forcing improves reliance on AI. Given previous work has shown mixed results for cognitive forcing, task context may be a factor in whether cognitive forcing changes AI-supported decision making. Our task was a perceptual decision-making task that relied on forced choice (i.e., participants could only choose between real and fake). In this scenario, using heuristics may be beneficial to task performance. For instance, if participants have a set of features they consider markers of a deepfake, then making the decision about real or fake could be based on detecting the presence of such features. Indeed, previous work has shown that human deepfake detection often relies on particular local features in a face (e.g., smooth texture on skin, eye gaze direction [3]). Therefore, participants may be motivated to see if the AI agreed with their successful detection of the feature space they searched, rather than looking to the AI for advice per se. Indeed, this would fit with our pattern of results revealing high but not ceiling agreement rates with AI. In such a case, cognitive forcing may not be appropriate to influence decision making.

Additionally, our task was a difficult task for humans. Agreeing with an AI that has accuracy greater than the typical human respondent would have been an effective strategy to employ. Cognitive forcing, when found to be effective, is often shown to limit over-reliance on the AI advice [4]. While over-relying on the AI would have led to accuracy on par with the AI support, this still would be an improvement on typical performance. As such, it may be that there is an ideal context where the discrepancy between human and AI capabilities is at a vital level that over-relying becomes a problematic strategy and, in this situation, cognitive forcing may provide benefits.