Multimodal Behavioral Typicality as a Training-Free Screening Signal for Dementia^✱

The DementiaBank Pitt Corpus [4] is the most widely used dataset for speech-based dementia detection, providing Cookie Theft picture descriptions from elderly controls and individuals with dementia (primarily probable Alzheimer's disease). The ADReSS [26] and ADReSSo [25] challenges provided demographically balanced subsets, with top systems reaching 80–90% classification accuracy under supervised learning, chiefly by fine-tuning pretrained language models on ADReSS transcripts [3, 42]. A potential limitation of such supervised classifiers is that they may learn surface artifacts—filler-word rates, recording conditions, speaker demographics—rather than genuine cognitive markers, a concern we revisit empirically in Section 5 through a filler-removal analysis. Recent paired perplexity methods [13, 23, 40] contrast transcript likelihood under control-adapted and impaired language models—the latter obtained by training on dementia transcripts or by deliberately degrading the control model—and report further gains, but each relies on in-domain dementia or control transcripts. Beyond speech-only pipelines, Fraser et al. [12] combined linguistic and acoustic features to identify AD from Cookie Theft narratives, and König et al. [22] showed that automatic vocal markers can distinguish healthy controls, MCI, and early AD; both rely on supervised learning over hand-crafted feature sets, and neither includes a visual or gaze modality. Foundation models for medical domains [18, 29] have shown the promise of pretrained representations for clinical tasks, though medical imaging has dominated the focus, with little work on spontaneous behavior. Our work departs along a single axis: we use pretrained models directly, with no fine-tuning at any stage—distinguishing it from transfer learning [35], domain adaptation [5], and few-shot approaches [39], all of which rely on target-domain data. To our knowledge no prior dementia-detection work adopts this fully pretrained-only stance, a property valuable for rare conditions, underrepresented populations, and resource-limited settings.

Eye tracking has long been used to assess cognitive function in dementia through fixation analysis, saccade dynamics, and visual-exploration statistics [8, 28, 31]. Existing pipelines rely on either low-level oculomotor features or hand-labeled regions of interest defined by clinicians or annotators. Findings on scene exploration in AD are mixed, but several report a suggestive dissociation: patients overlook informative or incongruous scene regions while basic saccade measures remain comparable to controls [28]. Our gaze results give this clinical observation a computational form. Gaussian Mixture Models [27] are a standard tool for density estimation over spatial distributions, and EyeFormer [21] predicts personalized scanpaths with transformer-guided reinforcement learning. Vision-language models such as CLIP [32] learn joint image-text representations that enable zero-shot transfer, and have been applied successfully to medical imaging tasks [18, 44]. To our knowledge, no prior work uses a VLM to approximate a semantically meaningful expected-gaze distribution for AD assessment. We formulate an automated semantic annotation pipeline using CLIP scores that replaces hand-labeled regions-of-interest for gaze analysis.

Across modalities, we compute NLL under pretrained models as a measure of behavioral typicality. Pretrained models encode distributional expectations about “normal” behavior—normal gaze patterns, normal language, normal speech. Dementia-related behavioral changes should produce higher NLL (more surprising behavior) without any model fine-tuning. Once z-normalized, NLL becomes a common currency across modalities. Beyond scoring, we also compare two classes of pretrained model within each modality. Semantic models (CLIP for gaze, language models for text, self-supervised speech models for audio) are trained to represent content rather than low-level signal structure. Mechanical models (bottom-up saliency, free-viewing temporal dynamics, an acoustic codec) are trained to represent low-level sensory or temporal structure independent of content. Holding this axis constant where possible (gaze and audio) lets the cross-modal pattern (Section 5) surface cleanly: a mechanical baseline that fails where its semantic counterpart succeeds is hard to attribute to a pipeline artifact.

Each participant's real recorded gaze is scored against a golden target: a distribution over expected fixation locations derived from the stimulus image alone, identical for every participant and estimated without reference to any gaze data. Group separation therefore arises from how real gaze scores against a shared reference—no gaze is simulated. A central challenge in gaze-based cognitive screening is the absence of ground truth: there is no “correct” way to look at the Cookie Theft picture. Constructing the golden target is therefore the central design question, and we systematically tested ten strategies organized into two categories. Our experimental design is hypothesis-driven: the ten gaze strategies test a specific theoretical prediction (semantic vs. mechanical contrast), not a search for optimal configurations.

Hand-annotated semantic targets (8 strategies): We annotated the Cookie Theft image with pixel-level labels for 11 semantic objects (characters, key action zones, scene context). Eight clinically motivated configurations were constructed encoding different hypotheses about which scene elements matter. Automated targets (2 strategies): A bottom-up saliency strategy (GradCAM) uses ResNet50 [15] and GradCAM [36] activation maps to identify regions of high visual contrast and edge density—where low-level features draw the eye regardless of meaning. A vision-language grounding strategy (CLIP) uses CLIP (ViT-B/16) [32] as an automatic annotator rather than a classifier. The Cookie Theft image (1302 × 943 pixels) is scanned with 64 × 64 patches at stride 32, yielding 1,092 overlapping patches. We define seven descriptive prompts capturing the Cookie Theft narrative (e.g., “a boy standing on a stool stealing cookies from a jar,” “water overflowing from the sink onto the floor”); these prompts were selected a priori based on the narrative elements of the picture and were not modified after initial evaluation. Each image patch and each prompt are encoded by CLIP's vision and text encoders into a shared 512-dimensional space. For each patch p_i, we average its cosine similarity across the seven descriptive prompts:

Prompt-specificity ablation for CLIP. Replacing the Cookie Theft-specific prompts with task-unrelated random prompts (e.g., “a mountain landscape”) tests whether the effect depends on task semantics rather than on CLIP's vision encoder alone; results are reported in Section 5.2. We additionally evaluated EyeFormer [21], a transformer-based scanpath model, as a temporal-dynamics baseline; per-participant temporal NLL was computed over non-overlapping 7-second windows (mean 13.4 per participant), matching the free-viewing clip duration EyeFormer was trained on.

For causal models, NLL is the mean per-token cross-entropy loss. For masked language models (BERT [11], RoBERTa [24]), we compute pseudo-log-likelihood [37]. We evaluate eleven models spanning three generations: GPT-2 (small: 117M, medium: 345M, large: 774M) [34], XLNet [41], BERT, and RoBERTa (2019); GPT-Neo 1.3B [7], GPT-J 6B [38], and OPT-1.3B [43] (2021–2022); Phi-2 (2.7B) [19] and Mistral-7B [20] (2023). Three transcript conditions test whether the NLL signal resides in surface disfluency (filler words such as “uh,” “um,” and pauses) or deeper semantic content: (1) paragraph (full transcript), (2) paragraph with filler words removed, (3) utterance-by-utterance (transcripts split into sentence-level units scored independently, removing cross-utterance discourse context).

We distinguish two categories of audio models. Linguistic speech models: HuBERT [17] and wav2vec2 [2] are self-supervised models pretrained on raw speech; they encode phonemic and lexical structure. We note these representations are not purely linguistic: they also carry prosodic, speaker-dependent, and recording-condition information. The audio arm is therefore a weaker test of the content-vs-mechanics contrast than gaze, where CLIP and GradCAM operate on identical input, and we interpret it accordingly (Section 5.7). Whisper [33], weakly rather than self-supervised, is reported alongside them for comparison. Acoustic codec (control condition): EnCodec [10] provides a content-agnostic baseline, trained to compress and reconstruct any audio signal without modeling linguistic content. Reconstruction MSE—the Gaussian NLL up to an affine constant, so g and AUC are unchanged—therefore measures purely acoustic typicality: voice quality and spectral characteristics independent of what is being said. This contrast between linguistic and acoustic models parallels CLIP-vs-GradCAM in gaze: if EnCodec separates groups, the signal lies at least partly in how people sound.

Per-participant NLL scores are standardized to z-scores within each modality. Late fusion is computed as the arithmetic mean of z-scores:

As a robustness audit for our technique, we conducted an exhaustive grid search over all 1,320 combinations of gaze strategies (n = 10)  ×  language models and transcript conditions (n = 33)  ×  audio models (n = 4), using three-way fusion effect size as the evaluation criterion. Rather than selecting a “winner,” this audit asks whether the content-mechanics pattern holds across the entire configuration space—i.e., whether high-performing combinations consistently pair semantic models while mechanical baselines consistently fail. Results are reported in Section 5.6.

Because the dementia group is small (n = 14), we triangulate across three statistical test families to evaluate the separation of control and dementia participant scores: Welch's t-test (robust to unequal variance), Mann-Whitney U (robust to non-normality), and permutation tests (10,000 two-sided iterations, assumption-free). Effect sizes are reported as Hedges’ g [16] with small-sample bias correction and 95% bootstrap confidence intervals; positive g indicates higher NLL in dementia than control. Benjamini-Hochberg FDR correction [6] is applied within each modality, since our hypotheses concern within-modality semantic-vs-mechanical contrasts. Hedges’ g (sample-size invariant) is our primary effect-size metric; AUC (threshold-independent) is the primary classification metric. Per-modality analyses use n = 39 (see Section 3 for exclusions); three-way fusion uses n = 38.

The comparison takes two forms (Table 1, Figure 2). In gaze and audio, we pair each semantic model with a mechanically-matched baseline that scores a different kind of structure within the same modality: CLIP vs. GradCAM and EyeFormer in gaze, wav2vec2 vs. EnCodec in audio. In gaze, the semantic model reaches FDR significance while both mechanical baselines do not; in audio, the semantic model shows a positive but nominal effect (Table 4) while the acoustic baseline is flat. In text, we cannot construct a fully content-free baseline of the same model family, so we instead test whether the signal depends on the most obvious surface production feature: filler words. Removing them strengthens rather than weakens the effect (Mistral-7B: g = 1.24 → 1.31), indicating the signal does not reduce to surface-level disfluency.

Bar chart grouped by modality. Gaze shows CLIP tall blue bar, GradCAM and EyeFormer negative/flat red. Text shows Mistral paragraph and Mistral no-fillers both tall blue, with no-fillers slightly stronger. Audio shows wav2vec2 positive but marked nominal, HuBERT and Whisper positive and hatched, EnCodec flat.

The pattern supports the interpretation: dementia impairs the content of behavioral engagement—what people attend to and say—while the production-level signals we can measure (low-level gaze statistics, acoustic reconstruction, transcript length) do not reach significance. We detail the per-modality results below.

CLIP achieved the strongest effect (g = 1.04, p_perm = .002, p_FDR = .021, AUC = 0.73), outperforming all eight hand-annotated alternatives by effect size. The direct contrast between CLIP (semantic) and GradCAM (bottom-up saliency, g = −0.28) constitutes a controlled comparison: both strategies produce probability distributions over the same image and feed an identical GMM-NLL pipeline, differing in whether the target encodes semantic content or low-level saliency. The thresholds also differ (CLIP 60th, GradCAM 70th percentile), but in the direction that disfavors CLIP: its broader region set flattens the target density and reduces discriminative power. The prompt ablation isolates that semantic difference further. Replacing task-relevant prompts with unrelated ones (g ≈ 0.67) preserves CLIP's architecture and the entire downstream pipeline while removing task semantics; pure saliency removes semantics altogether (g = −0.28). The ordering task-relevant > random ≫ saliency is a dose-response on semantic specificity. Among hand-crafted strategies, narrower configurations limited to single aspects of the scene—characters alone (g ≈ 0.00), scene context (g = 0.02), specific action zones (g ≈ 0 for action/danger and kids’ mischief)—fail to separate groups. The Full narrative / All annotated configuration, which covers the complete set of narrative-relevant objects, reaches g = 0.93 and FDR significance. CLIP's advantage over careful full-narrative labeling is modest (Δg = 0.11), but CLIP-derived annotation recovers that effect automatically, without the task-specific labor.

EyeFormer temporal NLL showed no group separation (g = −0.02, p_perm = .944). EyeFormer was trained and evaluated on free-viewing clips of roughly seven seconds, so we segmented each recording into non-overlapping 7-second windows (mean 13.4 per participant); the null persists with duration matched. The remaining mismatch is task rather than duration—free-viewing versus narrative description—and our framework predicts that a description-trained temporal model would recover effects EyeFormer does not, a falsifiable prediction testable once such a model exists. The null, combined with GradCAM's null and CLIP's positive effect, is consistent with localizing the difference to top-down, meaning-driven attention allocation, though the present design cannot establish this mechanism directly. A null in low-level gaze metrics alongside impaired content selection has precedent in the clinical scene-exploration literature [28].

Ten of eleven models achieved significance (Table 3), with effect sizes ranging from g = 0.95 to g = 1.24 for the significant models. This consistency across architectures, sizes (110M to 7B parameters), and release years (2019–2023) indicates that the signal is model-independent rather than an artifact of any single architecture.

Content vs. disfluency. For every autoregressive model, removing filler words strengthened the NLL-based group separation. Specifically, Mistral-7B's effect size rose from g = 1.24 (with fillers) to g = 1.31 (fillers removed)—a pattern also dominant in our grid-search audit (Section 5.6). While filler-word rate does differ between groups (dementia participants produced more; d = 0.94, p < .001), other surface features do not: word count, vocabulary size, and type-token ratio all fail to reach significance.

Context-dependence of the signal. Scoring transcripts utterance-by-utterance—which strips cross-utterance discourse context—yields a weaker but still significant effect for Mistral-7B (g = 0.91, p_perm = .011). The attenuation from g = 1.24 (paragraph) to g = 0.91 (utterance) is consistent with the content-integration interpretation: some of the signal lies in how utterances cohere across the full description, not only within individual sentences. The method is not uniformly significant. Broad agreement across language models reflects their shared large-scale English pretraining rather than an indiscriminate positive: most hand-crafted gaze targets fail to separate groups (Table 2), the mechanical baselines do not reach significance in either gaze or audio (Table 1), XLNet does not reach significance (Table 3), and BERT and RoBERTa reverse direction on the external corpus (Section 5.7).

The linguistic speech model wav2vec2 shows a positive effect (Table 4; g = 0.66, p_perm = .046), as does Whisper (g = 0.69, p_perm = .060); neither survives FDR correction across the four audio models (p_FDR = .119), and the three test families disagree at the margin (wav2vec2: Welch p = .073, Mann-Whitney p = .132; Whisper: Welch p = .043, Mann-Whitney p = .045). HuBERT shows a positive but weaker trend (g = 0.44). EnCodec, which measures acoustic reconstruction independent of linguistic content, shows no group difference (g = −0.03, p_perm = .945), directly paralleling the CLIP-vs-GradCAM contrast in gaze. The audio arm therefore supports the content-vs-mechanics pattern in direction, but not with the statistical strength of gaze and text (Section 4.4).

Gaze+text reaches g = 1.69 (AUC = 0.88); adding audio yields g = 1.79 (AUC = 0.94). The remaining pairs are gaze+audio (g = 1.21, AUC = 0.84) and text+audio (g = 1.37, AUC = 0.82); all fusion analyses use n = 38. Fusion balances complementary error profiles (Figure 3). We report the paragraph condition (not the marginally stronger no-fillers condition, g = 1.31) since it preserves the transcript as naturally produced; as the grid-search audit shows, the result holds either way. An exhaustive audit of all 1,320 configurations (10 gaze strategies × 33 text model-condition pairs × 4 audio models) establishes that the content-mechanics pattern is not configuration-dependent. The grid covers spatial strategies only, so EyeFormer is excluded; GradCAM and EnCodec are its mechanical baselines. All top-10 configurations pair semantic gaze targets (CLIP or full-narrative annotation), large language models (Mistral-7B, BERT, or OPT-1.3B), and linguistic audio models (wav2vec2, Whisper, or HuBERT). No mechanical baseline (GradCAM, EnCodec) appears anywhere in the top 10, and neither does XLNet—the one language model that failed to reach individual significance. Seven of the top-10 use filler-removed transcripts, corroborating the “content, not disfluency” claim. The effect-size spread among the top-10 is narrow (Δg = 0.14 from rank #1 to #10), and the configuration we report throughout this paper (CLIP + Mistral-7B paragraph + wav2vec2) ranks #3 of 1,320—broad robustness rather than a single lucky combination.

Left: scatter plot with gaze NLL z-score on the x-axis and text NLL z-score on the y-axis; blue circles (control) cluster lower-left, red squares (dementia) upper-right, with a dashed diagonal fusion threshold. Right: horizontal bar chart, one bar per participant, sorted by three-way fusion z-score; blue bars negative, red bars positive, with dotted lines marking group means.

We applied the text pipeline to 549 DementiaBank Pitt Corpus transcripts (243 control, 306 dementia; ∼ 14 × as many as our sample) and the audio pipeline to the 292 with usable recordings. We report Cohen's d for consistency with prior DementiaBank literature, and note that at this sample size Hedges’ g and Cohen's d are numerically equivalent up to two decimals. Text generalizes robustly. Of the eleven text models from our primary analysis, ten are reported here (XLNet omitted due to individual non-significance). All 8 autoregressive models replicated the direction (dementia > control); 7/8 reached significance (mean d = 0.35, range 0.16–0.45), with the large-model cluster at d = 0.41–0.45. BERT and RoBERTa replicated in the opposite direction, which we attribute to pseudo-log-likelihood scoring interacting with Pitt's transcript format—consistent with our emphasis on autoregressive NLL as the more robust metric. The reduced magnitude (d ≈ 0.35 vs. our g ≈ 1.0+) is a boundary condition rather than a contradiction: with severity and etiology unavailable for both samples (Section 3) we cannot attribute the attenuation to composition, but Pitt's decades-long, multi-equipment collection is a heterogeneity source known to attenuate rather than invert effects. The central claim is direction rather than magnitude. Pitt is longitudinal: these 549 transcripts derive from 291 unique individuals, so repeat sessions are not independent and significance levels should be read as approximate. Audio marks the boundary of training-free generalization. Audio models applied to Pitt showed effects in the opposite direction from our primary sample (wav2vec2 d = −0.38, HuBERT d = −0.29; both p_FDR < .05), while EnCodec remained null as designed (d ≈ 0). Decades of equipment heterogeneity disproportionately affects acoustic features exposed to speech models trained on modern recording standards. The text pipeline, invariant to recording conditions, generalizes; the audio pipeline requires standardized protocols.

Our training-free framework recovers, without any supervision, the clinically-observed gradient in which semantic and top-down control processes are disrupted more consistently than the basic machinery of production [8, 9, 28]. CLIP, Mistral-7B, and wav2vec2 have no knowledge of cortical anatomy, yet collectively they mark the same boundary between affected semantic and comparatively spared production processes in AD.

The filler-removal result bears directly on the surface-artifact concern of Section 2, which motivated the demographically balanced ADReSS benchmark [26]: classifiers may learn filler rate rather than cognitive markers. That explicitly encoding pause structure measurably improves supervised accuracy [42] makes the concern concrete—disfluency is exploitable signal. Filler rate differs between our groups (d = 0.94), so any classifier with a flexible objective could exploit it as a shortcut; a model trained on in-domain transcripts would have no reason not to. Our result runs the other way: removing fillers strengthens the effect. A training-free NLL approach instead scores transcripts under a pretrained language model whose surprise is shaped by general English—a distribution in which filler words are common—so the signal reflects content coherence, not fluency.

The fusion gains (Section 5.6) suggest that gaze, language, and linguistic speech probe at least partially separable cognitive sub-processes: narrative visual integration, linguistic semantic coherence, and lexical-phonemic retrieval. The errors are partially non-overlapping: cases that three-way fusion catches but single modalities miss account for the AUC gain from 0.780 (text alone, n = 38) to 0.938 (three-way).

The primary sample is single-site and small (n = 39). The gaze strategy carried into fusion was the strongest of ten in this same sample, so fusion effect sizes should be read as optimistic; the grid-search audit (Section 5.6) bounds how much this matters. The Pitt validation extends this for text but not audio, which reversed direction under recording heterogeneity (Section 5.7); no external gaze corpus exists. Control participants self-reported the absence of a dementia diagnosis and were not formally screened (Section 3), so undetected preclinical impairment in the control group cannot be excluded and would attenuate rather than inflate the reported separation. Our sample is demographically homogeneous (59% female, 73% White) and the pipeline is English-only, so cross-linguistic deployment requires per-population validation. Finally, per-participant severity and etiology labels were unavailable for both samples (Section 3), precluding analyses relating the NLL signal to disease stage or subtype. Gaze sampling density was lower in the dementia group at the margin of significance (p = .055); because fewer points yield a noisier rather than a shifted NLL estimate, this would attenuate rather than inflate the reported separation.

The content-vs-mechanics pattern reframes what multimodal interaction systems should measure. Much of current HCI assumes behavioral production—fluency of speech, smoothness of gaze, voice quality—carries the signal of cognitive state. Our results suggest this assumption is misplaced for dementia. Instead, the signal is in what users engage with, not only how they produce behavior. The concrete move for conversational and assistive systems is a shift from disfluency-based to coherence-based adaptation. Coherence-based flagging—whether verbal descriptions integrate across a scene, whether gaze samples narratively relevant regions—captures the actual locus of cognitive change.