Qualification by Calibration: A Readable Benchmark for Admitting Language Models to Human-Computation Tasks

Human computation increasingly runs through language models that pass work to one another: one model drafts and another checks, a router sends each item to whichever model is most likely to be right, an arbitrator breaks ties, and an escalation rule decides when to hand a case to a person. Every one of these mechanisms consumes a model's expressed confidence as a control signal; an escalation rule keyed to confidence never escalates a model that is confidently mistaken. The property that makes these mechanisms safe is trust calibration: the alignment between how sure a model says it is and how often it is actually correct [7, 11, 20].

Calibration is well studied for neural networks in the aggregate [6, 16] and for language models [9, 26]. What a team builder lacks is not another large corpus but a different kind of object: a small, auditable instrument that can be run on a candidate model before wiring it into a pipeline, read in its entirety, and reasoned about, the model-worker counterpart of the gold-question screens crowdsourcing has long used to admit human contributors [3, 17]. We present such an instrument, the Trust-Calibration Benchmark (TCB): sixty yes-or-no questions across five everyday domains, each answered with a stated confidence, scored for accuracy, calibration error, and Brier score, with every gold label independently audited.

The deepest lesson of this paper, however, is about the screen rather than the screened. Crowdsourcing learned long ago that a badly designed qualification task rejects good workers [3]; we show the same failure arises, mechanically and at scale, when the worker is a model. A qualification harness must parse free-form model output and must allot a token budget, and both choices can manufacture verdicts. In our own initial runs, a naive harness scored one model at 65 percent accuracy with a 95 percent yes-bias and confidence inversely related to correctness; an audit of the per-trial raw responses traced every one of those pathologies to the harness, which had been force-scoring budget-truncated reasoning fragments and misreading quoted instruction templates as answers. Under the audited protocol the same model answers all sixty items correctly with nearly perfectly calibrated stated confidences. We therefore treat the administration protocol as part of the instrument, ship it with an audited parser, escalation rule, and per-trial flags, and report a three-protocol comparison quantifying the harness effect.

Our evaluation asks four questions. Q1: what verdicts does the instrument return for nine contemporary models under a validated protocol? Q2: how sensitive are those verdicts to the qualification harness itself? Q3: does the calibration axis carry information that accuracy does not? Q4: do the screen's verdicts predict behavior on tasks beyond the benchmark? In brief: properly administered, all nine models are admissible on accuracy (0.91 to 1.00, no significant pairwise differences), and the durable differentiators are interface properties, calibration of stated confidence, format discipline, confidence-field discipline, and availability under budget (Q1); the harness effect is large, up to a 35-point apparent accuracy error with fabricated answer bias and fabricated confidence inversion (Q2); a controlled manipulation dissociates calibration from accuracy, steering calibration error while answers stay fixed, and below-ceiling stated confidences mark trials at elevated error, so the calibration axis is real and dissociable (Q3); and the interface verdicts transfer, with calibration error and availability predicting their downstream counterparts on two disjoint tasks while accuracy saturates (Q4).

This paper makes four contributions. (1) A compact, tier-decomposed trust-calibration benchmark for language-model workers, with an independent audit of all sixty gold labels (one error found and corrected) and a release designed to be read and re-derived in full. (2) A harness-validity study showing that qualification verdicts for model workers are protocol-dependent: a naive harness manufactures inaccuracy, answer bias, confidence inversion, and instability that an audited protocol eliminates, and we operationalize the audited protocol as part of the instrument. (3) A nine-model panel under the validated protocol, showing contemporary models are uniformly admissible on accuracy while differing in interface properties, together with a controlled manipulation showing calibration is dissociable from accuracy. (4) A criterion study on two disjoint downstream tasks showing that the surviving differentiators, calibration and availability, are the ones that transfer, plus a release of all per-trial responses under every protocol so that both the verdicts and the harness itself can be audited with one command. We scope these claims accordingly: the load-bearing contribution is the harness-validity lesson, and the instrument's validated role for this model generation is a compact, auditable preflight on parser behavior, confidence reporting, format, and availability; a general accuracy-based admission screen awaits the larger, non-binary validation outlined as future work.

Requesters screen contributors with gold-standard questions [17], aggregate redundant labels to manage noise [23, 24], and estimate worker quality to decide whom to trust [3, 8]. That literature also documents the converse hazard: a defective qualification task, an ambiguous gold answer, a broken interface, misgrades capable workers. As language models increasingly serve as the workers, in some annotation settings matching or exceeding human contributors [5], both lessons transfer: the screen must extend to model workers, and the screen itself must be validated. The LLM-evaluation literature has documented that benchmark scores are sensitive to prompt formatting and harness implementation [2, 22], and mature evaluation frameworks such as the LM Evaluation Harness [2] and HELM [12] standardize prompts, parsers, and metrics for exactly this reason; our contribution is not the existence of harness sensitivity but its consequence for worker qualification in human computation, where we show it manufactures precisely the failure profiles, inaccuracy, response bias, inverted confidence, that admission decisions key on, and we fold the remedy into the qualification instrument itself. Those frameworks administer large evaluations reproducibly; ours makes one admission decision auditable end to end, down to the raw response behind every scored trial.

The expected calibration error and the reliability diagram come from calibrating probabilistic classifiers [16] and modern neural networks [6], where accuracy and calibration proved to be distinct properties. For language models, asking for a confidence yields usable calibration [9, 26], models can be trained to verbalize uncertainty [13], and elicitation quality varies across models [28]. We use the plain ask-for-a-confidence elicitation, a stated confidence on a final line, because it is the signal deployed pipelines read.

The trust-calibration construct we measure originates in the human-factors literature on trust in automation, where appropriate reliance depends on trust tracking true reliability [7, 11, 20], and miscalibrated confidence is known to drive over-reliance and under-reliance and to degrade joint outcomes [1, 27]. In a multi-LLM pipeline the consumer of the confidence is often another model, but the requirement is identical: the consumer can only act well if the upstream confidence is calibrated.

As language-model agents are composed into teams [15, 18, 19, 21], what to measure and report becomes central [1, 27]. Datasheets for datasets [4] argue that an artifact should travel with a precise account of itself; we adopt that stance and treat the items, the gold-label audit, the administration protocol, and the scoring code as parts of the contribution rather than an appendix.

Our instrument likewise complements large calibration corpora: small enough to read, decomposed enough to localize a failure, audited down to individual labels and the harness itself, and cheap enough to run as a routine preflight on every new model and endpoint version.

With accuracy saturated and stated confidence concentrated at 100, one might ask whether ECE here simply restates accuracy. Three lines of evidence, observational, comparative, and causal, show the axis carries signal of its own.

Observationally, the confidence field, degenerate as its distribution is, still ranks trials by risk: trials stated at exactly 100 are 0.986 accurate, the 91 below-ceiling trials 0.890, and stated confidence predicts correctness with an AUROC of 0.75 even though 82 percent of confidences are the identical value 100. The two heaviest users of the below-ceiling range show it individually: gptoss-120b is 1.000 accurate at confidence 100 but 0.917 below (36 trials), qwen3-omni-30b-a3b 1.000 against 0.826 (23 trials). When these models say they are less than certain they are wrong more often, a property of the confidence, not the accuracy column, and precisely what a confidence-keyed router needs.

Comparatively, calibration ranks models that accuracy cannot separate: accuracy and ECE ranks are near-independent (Spearman ρ = −0.25); qwen3.6-35b-a3b-fp8 and deepseek-v4-pro tie at 0.983 accuracy with ECE 0.051 against 0.014, qwen3.5-9b and glm-5.1-fp8 at 1.000 with blended ECE 0.100 against 0.021 (the confidence-availability gap of Section 4); and calibration error transfers downstream where accuracy ranks do not (Section 4.6).

The causal evidence is a controlled manipulation: we re-elicited three top-accuracy models, gemma-4-31b, gptoss-120b, and deepseek-v4-pro, under prompt variants that change only how confidence is reported, not what is answered. Answers, and thus accuracy near 0.98, barely move (answer agreement with the default elicitation 0.95 to 1.00), establishing the dissociation in principle. The size of the effect depends on how hard one steers. The realistic nudges move calibration only a little: a “be bold” instruction is the largest, raising gptoss-120b's ECE from 0.001 to 0.031, and across models the “hedged” and “bold” variants move ECE by at most 0.03, within the bootstrap interval widths of the panel's own ECE estimates. The dramatic swing comes only from a mechanical worst case that bounds the manipulation rather than modeling a realistic elicitation: forcing a flat mid-scale confidence drives every model's ECE from 0.015 or below to roughly 0.48 at essentially identical answers. A model can therefore be top-accuracy and still fail a calibration gate, the property an accuracy-only screen cannot see; the large swing is an artifact of the forced-flat bound, and that gentle nudges barely move these models means their default calibration is robust to prompting, a property to measure rather than assume. (All comparisons are within the manipulation's own default-elicitation baseline run.)

The evaluation answers the four questions. Under a validated protocol, contemporary models are uniformly admissible on accuracy for common-knowledge items, and the durable verdicts concern interface properties: stated-confidence calibration, format discipline, confidence-field discipline, availability (Q1). Those verdicts are radically harness-dependent (Q2). The calibration axis carries signal distinct from correctness, observationally, comparatively, and causally (Q3). And the interface verdicts, not the saturated accuracy ranks, transfer downstream, an exploratory result we read as suggestive rather than confirmatory (Q4).

What the calibration axis measures deserves precision: the instrument scores elicited, verbalized confidence, not intrinsic model uncertainty, which token probabilities, predictive entropy, or semantic entropy [10] estimate and from which verbalized confidence can diverge [26, 28]. We measure the stated signal deliberately: it is what deployed routing and escalation mechanisms consume, it is available even from closed endpoints exposing no token probabilities, and it is the analogue of asking a worker how sure they are [7, 11]. A model could carry sound intrinsic uncertainty and still fail the screen by verbalizing it badly; that interface failure is what a confidence-consuming pipeline needs exposed. Scoring logprob-derived confidence beside the stated number, where available, is a natural extension of the battery.

As a qualification test, a team builder runs the benchmark on a candidate model exactly as a requester runs a gold-question screen on a prospective worker [3, 17]; for this model generation the operative gates are on calibration error, format adherence, and availability, since accuracy no longer discriminates, and Section 4.6 shows those gates carry to tasks beyond the benchmark. The per-gate remedies (the deployment guide below) are the model-worker counterparts of the redundancy and quality-control responses crowdsourcing applies to noisy human labelers [8, 23]. For harness validation, the three-protocol comparison is itself a template: run a candidate harness against the shipped per-trial raws and verify it reproduces the validated verdicts before trusting it on a new model. For version-drift and availability monitoring, re-running the benchmark when an endpoint updates turns a rise in calibration error or a drop in adherence into an early warning, before the downstream pipeline degrades.

The parser rules generalize beyond this benchmark into a short reporting standard any model-worker qualification should meet and a reviewer can check: retain the raw model output for every trial; separate parseability from correctness and report availability alongside accuracy; report the rate of missing confidences and flag every imputed value; never force-score a truncated or unparseable response, escalating the budget or recording it unavailable instead; and test the parser against adversarial and template-quoting responses before trusting its verdicts. Each rule corresponds to a way Section 4.4 shows a naive harness fabricates a verdict.

Could better output engineering prevent these failures? Partly: JSON mode or constrained decoding, where supported, would eliminate the template-echo misparse class, and structured elicitation is a worthwhile fourth protocol for the next version. But it narrows the parser problem, not the audit problem: truncation still yields malformed objects that must be escalated rather than force-scored, missing confidence fields still need imputation and flagging, constrained decoding is not uniformly available and can perturb the distribution being measured, and a strict format does not make a harness correct. Whatever the format, a qualification result should ship its protocol and raw responses so the screen can be audited.

To admit a model in practice, a team builder reads four columns: accuracy on parseable items, calibration error (the ECE interval with the Brier score beside it), strict-format adherence, and availability. We suggest gates of accuracy at least 0.90 and ECE at most 0.05, but these are illustrative defaults, not validated or pre-registered thresholds, and the split is threshold-sensitive: at ECE at most 0.03, 0.05, and 0.07 the panel passes three, five, and seven of nine. How tight the gates should be is a deployment decision, as a crowdsourcing qualification threshold has always been a requester's cost judgment: tighten the calibration gate with how heavily the pipeline keys routing and escalation on confidence, the availability gate with the cost of a dropped item, and the accuracy gate with the harm of a wrong label and the redundancy available to absorb it, with high-stakes settings requiring verdicts robust across the interval. We ship the full columns with intervals precisely so a deployer can set gates to their own risk level rather than inherit ours. When a model's ECE interval straddles the threshold, as two of our nine do, the gate is inconclusive and the decision should fall back to the binning-free Brier score or a repeat run. Availability and calibration should be gated separately, because a missing confidence is an interface failure, not miscalibration: gate confidence availability on its own and calibration on the stated-confidence-only ECE, reporting both rather than blending them; the one model that fails the blended ECE gate here does so on confidence availability, not on the calibration of the confidences it states. The remedy follows the failed gate: a calibration failure bars a model from confidence-keyed routing or escalation, an availability failure calls for redundancy or a fallback worker, and a format failure calls for a tolerant parser or exclusion from strict-parse stages. The name qualification by calibration is shorthand: once accuracy saturates, the instrument qualifies a worker by the cluster of interface properties calibration anchors, not by calibration alone.

The items probe widely known facts, so the absolute scores measure calibration and interface behavior on knowledge the models plausibly have, the right target for a screening instrument, and saturation is itself a finding with a shelf life: this instrument no longer ranks frontier accuracy, not that it never did or that harder item banks would not. The gate we pre-set (accuracy 0.90, ECE 0.05) splits the panel five to four entirely on the calibration criterion, and one model (qwen3.6-35b-a3b-fp8, ECE 0.051) misses it by 0.001; gates near a cluster boundary are necessarily sensitive, and that model and the passing model nearest the line both have bootstrap intervals including 0.05, so two of the nine gate verdicts are within sampling noise. This is why we report the full table with intervals rather than only the verdicts, and state the imputation-counting rule explicitly alongside the stated-confidence-only sensitivity numbers.

Admitting language models to roles once held by crowd workers also raises governance questions this instrument informs but does not settle: who is accountable when an admitted model errs, how a screen's verdicts are disclosed, and when a human must remain in the loop. Passing the screen certifies interface reliability, not that substituting a model for a human worker is appropriate; the instrument's role is to make the interface evidence legible to that labor and accountability judgment, not to replace it.

Every design choice trades coverage for inspectability: a deployer can read all sixty items, re-check every gold label, re-derive every number from the shipped responses with one command, and, because raw responses under every protocol are released, re-audit the harness itself. The events of Section 4.4 are the argument: had we shipped only scored trials, the manufactured verdicts would have been unfalsifiable.

Sixty common-knowledge items no longer separate contemporary models on accuracy; the instrument's accuracy axis has saturated for this model generation, so until a harder item bank exists the instrument functions as a calibration, format, and availability preflight rather than an accuracy screen. The items probe widely known facts almost certainly within every model's training distribution, so the benchmark cannot speak to task competence on novel material, consistent with the saturation we report. Harder item banks would restore an accuracy gradient and are a natural next version, as is a multiple-choice extension beyond binary items; and because both downstream transfer tasks share the benchmark's binary format, a non-binary downstream criterion task is the priority next test. The items are author-written; the tiers differ in gold base rate (0.53 versus 0.67 yes) and show no consistent difficulty gradient, so we treat them as a diagnostic axis rather than a validated scale; one pair of arithmetic items probes the same product under different proposed answers, a redundancy the next version will remove. The harness study covers the two defects we found and fixed, parsing and budget; other harness choices (system prompts, sampling parameters, retry policies) deserve the same treatment, and all models here are served through a single inference stack, so the protocol comparison should be replicated across serving stacks before its magnitudes are generalized. The panel comprises nine open-weight models; closed-source frontier systems are absent, and since they would very likely also saturate the accuracy axis, their inclusion would reinforce the conclusion that discrimination must rest on the interface axes, though their calibration, format, and availability behavior remains open and worth measuring. The downstream criterion study spans the same nine models, so its rank correlations are low-powered; broadening the panel and the downstream battery is the highest-value replication of this work. The protocol forces a yes-or-no answer, so a model that abstains or hedges produces no parseable answer and is recorded unavailable; abstention therefore surfaces as an availability signal rather than a first-class response, and an explicit abstain option is a natural protocol extension. The benchmark is a versioned artifact rather than a frozen one: the item bank is meant to be hardened and re-released, with the administration protocol, the part our evidence shows is load-bearing, carried forward unchanged. Finally, our evaluation covers the model-to-model consumer of the confidence signal; studying how human operators of multi-LLM pipelines read these signals, and whether passing models support appropriately calibrated human reliance, is the natural human-subjects continuation of this work.

A team builder admitting language models to a human-computation pipeline needs what a requester admitting human workers has long had: a small, readable, audited qualification test. We built one, and in validating it found that the most dangerous miscalibration in the room was the harness's, not the models’: a naive administration protocol manufactured a failing worker out of a competent one, fabricating a 35-point accuracy deficit (on this serving stack), answer bias, and inverted confidence that vanished under an audited protocol. Properly administered, nine contemporary models are uniformly admissible on accuracy for common-knowledge items, and the verdicts that matter, and that transfer downstream, concern stated-confidence calibration, format discipline, and availability. Observational, comparative, and causal evidence together confirm the calibration axis carries signal distinct from accuracy. We release the items, the audited gold labels, the administration protocol, and every per-trial response under every protocol, so that the check, and the audit of the check, are each one command. The instrument is offered as a small, auditable, extensible starting point for qualification testing of model workers, and its central lesson is one human computation already knew: validate the screen before trusting what it says about the worker.