Behavioral Fingerprints for LLM Endpoint Stability and Identity

The software deployment intuition “if it's up and fast, it's fine” is insufficient for AI-native systems: an endpoint can pass standard health checks while its outputs drift, shifting formatting, tone, or tool behavior in ways that break downstream parsers and guardrails. Stability failures are particularly impactful in multi-step agentic workflows, where small variance in early steps can compound into large workflow variance. This motivates stability as a distinct operational metric: behavioral consistency of a model endpoint under a fixed interface contract.

Even if users fix visible settings (e.g., temperature), endpoint behavior can vary due to system-level nondeterminism “beneath the surface”: variance in inference engines, kernels, caching, batch sizes, and hardware—and providers may route requests across heterogeneous environments. The values of these factors at run time may be a function of overall system load, which the user cannot control, thus making the endpoint nondeterministic from the user's perspective [4].

These system-level factors, which can result in nondeterministic behavior even at temperature 0, must be considered on top of discrete, deliberate updates such as a model version change, undermining reproducibility in both production and evaluation environments. Evaluation instability and provider effects are now recognized as a major source of variance in benchmarking results [1].

We introduce two components: Stability Monitor, which continuously fingerprints endpoints and detects behavioral change events, and Stability Arena, a web application that publishes and visualizes the data that Stability Monitor produces. Together, the goal is to operationalize continuous stability monitoring under realistic constraints:

• Black-box: observe only public API I/O; no privileged access to weights or provider internals.
  
• Lightweight and frequent: monitor at high cadence versus heavy periodic evals (motivating the notions of stability periods and change events).
  
• Actionable: produce audit trails and change event alerts usable by engineering, security, and compliance.
  

We contribute:

1. A practical fingerprinting and change-detection pipeline based on energy-distance permutation testing and sequential evidence aggregation.
   
2. Controlled validation across multiple real-world change types.
   
3. Real-world provider comparisons for the same nominal model showing large cross-provider and within-provider stability differences, motivating “same model, different behavior” as an operational reality.
   

A fingerprint is constructed by selecting a fixed set of prompts and sampling multiple responses per prompt; the first token of each response is embedded as a real-valued vector using a fixed subword embedding.¹ A fingerprint thus consists of a set of sets of vectors—one sample set per prompt. Our implementation requires a total of 800 inference requests, each consisting of a few tokens, to be made to an endpoint to generate a fingerprint.

Given fingerprints X and Y, we compute an energy distance statistic per prompt between the prompt-specific embedding samples, and sum across prompts to produce an aggregate distance E = E(X, Y).

To test the null hypothesis that two fingerprints are drawn from the same underlying distribution, we compute a p-value via a permutation test on the aggregate statistic E. To compute the p-value, the embedding vectors for each prompt are pooled and randomly re-split into two groups of the original sizes according to a permutation π on the pooled set of vectors, yielding a permuted statistic E_π. The p-value is the fraction of permutations for which E_π ≥ E when this comparison is done across a set of randomly selected permutations. Lower p-values indicate stronger evidence of change.

Monitoring of an endpoint begins by generating a baseline fingerprint F₀. At a regular cadence we generate new fingerprints F_i at time t_i and compute the aforementioned p-value p_i = p(F₀, F_i). Each time a new fingerprint F_i is generated, using the e-values methods from [7, 10] applied to the sequence of p-values p₁,..., p_i, we assess the cumulative evidence for a change in the endpoint's effective output distribution since the time of the baseline fingerprint. If the evidence exceeds a predetermined threshold, we declare a change event and set the most recent fingerprint as the new baseline, to which subsequent fingerprints are compared.

Since only the first token of each response is used for fingerprinting, the output token limit can typically be set low (e.g., 10 tokens), substantially reducing per-request cost. However, this must be configured per endpoint, as some models return empty responses when the token limit is too low for the model to produce a useful answer.

Fingerprint generation parallelizes across prompts, with each thread handling all repeated samples for a single prompt. We chose levels of concurrency based on explicit endpoint rate limits and observed latency. We found that anywhere from one to four concurrent threads enabled us to generate fingerprints quickly enough without overloading the target endpoint. A typical fingerprint could be generated within ten minutes, although outlier cases with aggressive rate limiting or abnormally high latency could take up to several hours.

At the live demonstration, attendees will inspect current stability data for endpoints serving the most widely deployed models. The demo will walk through the provider matrix to show recent change event trends, drill into a per-endpoint view to examine stability periods and change events in context, and use the cross-provider comparison to highlight providers whose behavior diverges from the consensus for the same model. Because Stability Arena runs continuously against live endpoints, the data shown will reflect the latest real behavior rather than a static snapshot.

Validation experiments were run by hosting a model at a local endpoint. In an independent process, we ran Stability Monitor, generating fingerprints via API calls to the local endpoint at an hourly cadence. Once Stability Monitor had generated a few fingerprints, we made a discrete change to the model hosted at the local endpoint without modifying Stability Monitor. Stability Monitor continued to run, generating fingerprints, unaware of the change that had been made to the endpoint.

We tested five change types, with examples of each shown in Table 1. Each change was evaluated on its own, holding the other change types constant.

With one exception, each endpoint change produced a change event immediately on the next fingerprint; the exception was a small temperature change (0.7 → 0.6), which took 18 fingerprints after the change was made to trigger a change event.

Moreover, Stability Monitor recorded exactly one change event per intervention: the periods both before and after the change detection were stable. That is, there were no false positives—no instances in which a change event was incorrectly triggered in the absence of an actual change.

In the case of (a), we consider multiple providers serving the same nominal model over a shared time period and perform pairwise fingerprint comparisons using energy distance to form a distance matrix. One application is provider identification: given an endpoint serving a model known to be among a set of candidate providers, we can determine which provider is serving it by fingerprinting the endpoint and comparing against the candidates. In Figure 2, the diagonal entries are the minimum in each row and column, confirming that a provider's fingerprints are most similar to its own—and therefore that nearest-neighbor comparison against known-provider fingerprints reliably identifies the serving provider.³ We can likewise perform the above analysis over a sequence of consecutive shared time windows, enabling us to measure pairwise provider similarity over time.

For the case of (b), we propose a “provider vs the pack” deviation score in which we compare one provider's fingerprints to the pooled distribution of all other providers’ fingerprints for the same model over matched time windows. We normalize each provider-vs-pack distance by the median across all providers to produce a divergence ratio. For example, a provider with a divergence ratio of 2 is twice as different from the pooled distribution as the median provider is.

In the Stability Arena “Provider Comparisons” tab we compute this divergence ratio daily and plot its values over time, enabling us to identify which providers diverge from the consensus and when. Because the metric is normalized by the median, it is sensitive to individual outliers rather than uniform shifts; a rolling model update that propagates unevenly across providers would appear as a sequence of outlier spikes.