# Measuring Autonomous Healthcare Administration

### How Omni Health evaluates the Omni Agent Hub on HealthAdminBench

**Version 1.0 · July 2026 · Omni Health**

---

## Abstract

The Omni Agent Hub performs healthcare administrative work — prior authorization, claim-denial appeals, durable medical equipment (DME) fulfillment — end to end, in a browser, the way a human operator does. Claims like that are easy to make and hard to trust. This paper describes the discipline behind our numbers: an independent public benchmark; execution by the production system; a structurally enforced blindness firewall between the system and the grader; scoring computed exclusively from the environment's own recorded end state; pass semantics that award nothing for near-misses; pinning of corpus, build, and environment; explicit statistical treatment of every stochastic component; and a regression discipline under which no aggregate gain can offset any per-task regression.

On the July 2026 evaluation, the production system completed **91 of 135 tasks (67.4%) of the HealthAdminBench v2 corpus with full credit** — every graded point earned — in a **single attempt per task**, with a **mean task score of 0.931**. This paper explains precisely what those numbers mean and the controls behind them.

---

## 1. The measurement problem

Autonomous agents demo well. Measuring them honestly is harder, for reasons specific to this domain:

**Browser agents fail silently.** A form can look filled and never persist. A click can land on the wrong patient with the right name. A submission can appear to succeed while the receiving system recorded nothing. An evaluation that trusts the agent's account of its own actions — or even a screenshot of an intermediate state — will systematically overcount success.

**Averages hide the failures that matter.** Administrative work is not partial-credit work. An appeal packet that is 96% correct is an appeal that was not correctly filed. A mean score can rise steadily while the fraction of tasks a system can actually be trusted to finish stays flat.

**Wrong action is worse than inaction.** In healthcare administration, the costliest failure is not an unfinished task; it is a consequential action taken when it shouldn't have been — a submission filed against inactive coverage, a record cleared that needed escalation. An evaluation that only rewards task completion selects for exactly the wrong instinct.

**Contamination is the default.** If any representation of the grader — rubrics, expected values, expected outcomes — can reach the system under test through any channel, benchmark results measure leakage, not capability.

Our methodology is the set of controls we adopted against each of these failure modes. This paper describes how we *measure* the system, not how the system works. The task corpus and grading specifications are public and independently authored; the execution and enforcement layer is ours, and §11 states exactly what a reader can check and where trust in us is still required.

---

## 2. The benchmark

We evaluate on **HealthAdminBench**, an independent benchmark authored by the Shah Lab at the Stanford School of Medicine and released under the Apache-2.0 license. We chose an external, publicly available benchmark deliberately: the task corpus, grading specifications, and environments were authored by researchers with no stake in our results, and anyone can run their own system against the identical corpus.

The v2 corpus contains **135 tasks** across three families of revenue-cycle work:

| Family | Tasks | The work |
|---|---|---|
| Prior authorization | 60 | Verify coverage, codes, and auth requirements against a referral; submit authorization requests to payer portals; document determinations |
| Claim denials & appeals | 60 | Triage denials from remittance evidence; file electronic and fax appeals; verify eligibility; correct and rebill claims; track follow-ups |
| DME order fulfillment | 15 | Assemble exact supporting documentation from the chart; transmit to the correct supplier; document and close the order |

Each family spans three difficulty tiers:

- **Easy** — single-system work: open one referral or denial, inspect the chart, make one determination, document it, take one worklist action.
- **Medium** — multi-system execution: leave the EMR for a second system (a payer portal, a fax portal), transcribe clinical and insurance data accurately across the boundary, capture returned identifiers, and document the round trip.
- **Hard** — adversarial and judgment variants: tasks where the correct behavior is to *detect a blocker and refuse to proceed*, deadline-driven triage, calculations, batch analysis across records, and dispositions that are genuinely ambiguous.

The benchmark runs against **hosted, stateful web applications** — an EMR with worklists, charts, documents, and notes; payer portals with distinct navigation and form topologies, including a multi-step wizard; a fax transmission portal. These are real web applications designed to reproduce the failure modes production systems exhibit accidentally: controls that only respond to genuine interaction, look-alike fields, decoy records, validation gates, and surfaces without stable element identifiers.

**Grading** is specified per task as a weighted set of checks of two kinds. **Deterministic checks** — 69.5% of total graded weight across the corpus — are machine-evaluated queries over the environment's recorded end state, compared against expected values fixed by the benchmark's authors. **Rubric-judged checks** — the remaining 30.5% — grade free-text artifacts the system was required to produce (typically clinical or administrative notes) with a language-model judge against the benchmark's rubric. The rubric-judged share varies by family: 16.8% of graded weight in prior authorization, 38.9% in DME, 46.0% in denials & appeals. Several checks are **negative**: they award points only if the system correctly did *not* act — did not submit, did not clear, did not attach the decoy document.

We evaluate against a **vendored snapshot** of the corpus pinned to upstream commit `e71a8f4d69`. The snapshot is hashed, and the hash is recorded in every run's metadata; tasks are never fetched at evaluation time, so corpus mutation — accidental or otherwise — is detectable from the evaluation record.

---

## 3. Execution: the production path

The system under test is the production system. Precisely scoped, that claim means:

- **The acting stack is the production stack.** The models, decision-making, and execution machinery that perform benchmark tasks are the same ones that perform customer work, in the same configuration, pinned by source revision and container-image digest, with the configuration recorded in run metadata. The benchmark environments are onboarded the way any customer site is onboarded. Benchmark-specific code exists only in the **measurement harness** — task loading, the blindness firewall, and scoring — which never plans or acts.
- **The lifecycle is the production lifecycle.** Every task is dispatched through the same job pipeline production work uses and executed in an isolated, freshly provisioned environment; results are collected only after the run terminates.
- **The interface is a real browser.** The system operates a live browser session against the deployed benchmark applications. There are no HTTP-level shortcuts and no synthetic DOM: graded state in the benchmark environments persists only through genuine interaction with the page, and several surfaces expose no stable element identifiers, so controls must be located from visible structure. What is measured is actual operation of a user interface, not API calls that approximate one.
- **Isolation is per task.** Each task receives a fresh browser context and a freshly reseeded environment state. Nothing carries over between tasks.
- **One attempt.** Reported results are single-attempt (pass@1). No retries, no sampling multiple candidate runs and reporting the best. Where we make claims about *reliability* we run replicates and report the minimum, not the maximum (§7).

A run that stalls or dies is not quietly re-scored; it is accounted for under the fail-closed rules in §5.

---

## 4. Blindness: the system never sees the grader

The benchmark ships each task with its grading specification attached. A credible evaluation must guarantee the system under test cannot read it. Our firewall makes this a structural property rather than a policy:

- **Projection.** The only representation of a task that reaches the running system is a minimal projection of the benchmark's public task definition: the natural-language work instruction, the task's family, and the benchmark's short scenario label. Every grading field — checks, rubrics, point values, expected values, the expected-outcome field, and authoring walkthroughs — is excluded from that projection, and from job records, prompts, and every other runtime surface, by construction. No tier, difficulty, or point information reaches the runtime. One disclosure we make rather than overclaim: the benchmark's scenario labels name the scenario class, and for some tasks the name is outcome-suggestive (a task asking whether coverage is valid may carry a label naming the inactive-coverage scenario). These labels are descriptive fields of the published task schema — identical for any system evaluated on this corpus — and carry no grading content, but a projection that includes them is not entirely outcome-free, and we say so.
- **Verified, not assumed.** Scored evaluation runs execute in a strict verification mode, recorded in run metadata, under which the harness mechanically checks, for every task, that the working representation the system received is independent of the grading specification. A failed check fails the task closed — blocked before execution, counted as zero, never scored. The firewall admits no exceptions during scored runs.
- **The environment carries no answers.** The benchmark applications expose chart content only. There is no grader data in the environment for the system to scrape.
- **Computation is not retrieval.** Tasks requiring calculation are presented with expected answers stripped; the system must compute results from chart-derived facts.
- **Grader content stays out of the runtime.** Grading specifications are examined only after runs complete, and no grader content is available to any runtime surface at any time.

---

## 5. Scoring: only the environment testifies

**All scoring input comes from the benchmark environment's own recorded end state**, read back after the run terminates. The system's account of what it did — logs, messages, claimed completions — is never a scoring input. The grader is decoupled from the path the system took; it sees only what the environment says happened.

From that end state:

- **Deterministic checks** earn full points if and only if the queried state equals the expected value. All-or-nothing per check.
- **Rubric-judged checks** earn fractional credit from the judge's score. A judge that errors or is unavailable *deflates and flags* the task score — ungraded points stay in the denominator — and can never convert into an ungraded pass.
- **Task score** = points earned ÷ total weight of all checks, with the denominator derived from the checks themselves rather than from any declared total.
- **Pass means full credit.** A task passes only at score 1.0 — every point, including negative checks. Mean score is reported alongside as a separate, explicitly partial-credit metric.
- **Fail closed.** A task whose end state cannot be read is *unmeasurable*: tracked separately, never silently dropped, and — by our accounting policy — counted as zero in headline metrics. An unmeasurable task is treated as worse than a failed one, because it is unaccountable. The July 2026 evaluation had zero unmeasurable tasks.

---

## 6. Consequential actions: evidence, and the asymmetric standard

Two rules govern everything in the benchmark that resembles an irreversible act — filing an appeal, submitting an authorization, transmitting a fax, clearing a work item:

**Credit requires captured evidence.** A consequential action earns points only through evidence recorded by the receiving system — a confirmation identifier read back from the confirmation surface, a transmission record, a state change in the target application. "The system says it submitted" is worth nothing; "the payer portal's recorded state contains the confirmation number the portal issued" is what scores.

**Correct inaction is a scored outcome.** Hard-tier tasks embed blockers — inactive coverage, missing prerequisites, decoy confirmations — where the correct behavior is to stop, document why, and escalate. The benchmark grades this with negative checks, and we treat the distinction as load-bearing: a system that files everything it touches will outscore a careful one on completion metrics and fail exactly where it matters.

As a statement of internal practice — a policy, not a property of this evaluation: safety-relevant behavior in our release process is permitted to become more conservative over time and forbidden to become less so.

---

## 7. Statistics: what one run does and does not prove

Two components of the evaluation are stochastic, and we treat both explicitly rather than averaging them away.

**Judge variance.** Rubric-judged points can flip between identical runs of an identical build; we have internally measured a single rubric point moving across repeated runs with everything pinned. Consequences: an individual task score carries an uncertainty of roughly one rubric point's weight; tier means carry internally observed noise bands of approximately ±0.02–0.05 depending on cell size; and no per-task claim is ever founded on a single judge flip. The judge's identity and version are pinned and recorded in the run record.

**Environment nondeterminism.** Real browsers against real web applications exhibit timing- and rendering-dependent variation. Some task families are measurably noisier than others; for those, single runs are not accepted as evidence of improvement or regression in either direction.

The discipline that follows:

- Headline numbers are **single-attempt** and say so, with a binomial confidence interval on the completion rate.
- **Reliability claims use minimum-over-replicates:** a task is counted as *reliably* passing only if it earns full credit in **every one** of N ≥ 5 replicate runs. We do not report best-of-N as capability anywhere. This report makes single-attempt claims only.
- Comparisons on noisy families are made with matched replicate sets on identical pinned builds, not single paired runs.

---

## 8. Regression discipline: aggregate gains never offset per-task regressions

Every score we publish sits on a protocol designed so that backsliding surfaces in the per-task record rather than being absorbed into an aggregate:

- **Frozen per-task baselines.** Reference runs are frozen as read-only, per-task score records, pinned to an exact source revision and image digest.
- **Any-task regression invalidates a comparison.** Runs are compared task by task against the frozen baseline. A single task dropping beyond tolerance means the newer result is not reported as an improvement — a large aggregate gain cannot buy back one regressed task. Judge-point flips are the sole recognized exception, and they must be identified explicitly, never absorbed silently.
- **A lost score is a regression.** A task that scored before and produced nothing after counts as regressed unless the failure is affirmatively attributable to infrastructure — and the attribution rule is fail-safe: ambiguous evidence is charged to the system, not the infrastructure.
- **Builds are verified and recorded.** Before any scored run, the harness verifies whether the deployed build matches the exact source revision under evaluation, records the verdict and the running image digest in the run manifest, and aborts on any mismatch between the pinned image and the running build.
- **The environment is fingerprinted.** Hosted environments drift. We maintain frozen structural fingerprints of the benchmark environments and assert them before scoring runs, so environment drift is caught before it silently zeroes — or silently inflates — a result.

---

## 9. Results

Evaluation executed July 13, 2026 (18:13–19:31 UTC): production build, single attempt per task, all 135 tasks scored. These are the most recent full-corpus results for the release build at the time of reporting; no selection among runs occurred.

| Metric | Result |
|---|---|
| **Full-credit completions** | **91 / 135 (67.4%)** — 95% CI [59.1%, 74.7%] |
| **Mean task score** (partial credit) | **0.931** |
| Unmeasurable tasks | 0 |

| Family | Tasks | Full credit | Mean |
|---|---|---|---|
| Prior authorization | 60 | 52 (86.7%) | 0.988 |
| Denials & appeals | 60 | 25 (41.7%) | 0.859 |
| DME orders | 15 | 14 (93.3%) | 0.993 |

| Tier | Tasks | Full credit | Mean |
|---|---|---|---|
| Easy | 45 | 40 (88.9%) | 0.973 |
| Medium | 45 | 31 (68.9%) | 0.936 |
| Hard | 45 | 20 (44.4%) | 0.884 |

Family × tier detail, mean score (full credit / tasks):

| | Easy | Medium | Hard |
|---|---|---|---|
| **Prior authorization** | 1.000 (20/20) | 0.996 (19/20) | 0.968 (13/20) |
| **Denials & appeals** | 0.940 (15/20) | 0.860 (7/20) | 0.776 (3/20) |
| **DME orders** | 1.000 (5/5) | 1.000 (5/5) | 0.980 (4/5) |

Reading the table: prior authorization and DME are near ceiling on this corpus, including perfect easy-tier sweeps under full-credit semantics. Denials and appeals is where full-credit completion is hardest — its hard tier concentrates the corpus's multi-source investigation and ambiguous-disposition judgment, and the family also carries the highest rubric-judged share of points (46.0%), so its scores bear the widest judge-related uncertainty. The gap between the family's mean (0.859) and its completion rate (41.7%) is the partial-credit gap described in §1: many near-complete tasks, strictly counted as failures.

---

## 10. Limitations

- **Simulated environments.** The benchmark's applications are adversarial simulations of revenue-cycle systems, not live payer or EHR systems. They reproduce the *failure modes* of production surfaces, but benchmark performance is not a guarantee of production outcomes.
- **A fixed, public corpus.** The 135 tasks are public and fixed, and our engineering organization was not blind to them: results should be read as performance on a known task distribution, not on a held-out set. The blindness firewall (§4) governs what the running system can see; it does not make the corpus unseen by the people who build the system. Held-out evaluation requires new tasks, which only the benchmark's authors or a third party can supply.
- **Self-run evaluation.** This evaluation was executed and scored by Omni Health on our own harness. The corpus and grading specifications are independent and public; the enforcement layer is ours, and no third party has audited this run. §11 states what an external reader can and cannot check.
- **Vendor-operated judge.** Rubric points are graded by a language-model judge that we operate, configured against the benchmark's rubrics, with its identity pinned in the run record. Judge variance is quantified (§7) and judge failure deflates rather than inflates scores, but rubric grading is not deterministic and not independently adjudicated.
- **Corpus size.** 135 tasks supports family-level conclusions; at family × tier granularity (cells of 5–20 tasks), confidence intervals are wide and small count differences are not meaningful.
- **Point-in-time.** Results describe the stated pinned build on the stated pinned corpus. The benchmark's authors may revise the corpus; revised corpora are new evaluations, not comparable continuations.

---

## 11. Verifying and comparing

The benchmark is public. Any team can obtain the identical 135-task corpus and environments and evaluate their own system. For results to be comparable to ours, a report should state, at minimum:

1. Corpus version and pin, and whether tasks were vendored or fetched live
2. Attempts per task, and whether any candidate selection occurred between attempt and report
3. Pass semantics (full credit vs. thresholded), and the headline metric's denominator
4. Whether the system could read any part of the grading specification, and how that is *enforced*
5. What the scoring input was (environment end state vs. any form of self-report)
6. Accounting for unmeasurable tasks
7. State isolation between tasks
8. Build identification (source revision, artifact digest) for the system under test
9. Treatment of judge variance, and the replication protocol behind any reliability claim

We record and publish all nine for every result we release. The results report accompanying this paper prints them in-line — including the source revision, container-image digest, and corpus snapshot hash — and publishes the full per-task score record alongside. Baselines for this corpus have already been reported — by the benchmark's authors in their paper (models under a standardized harness, plus native computer-use systems) and in at least one vendor system card — and we are aware of no other result reported for a full production system under the end-to-end, full-credit, single-attempt protocol and disclosure items above. We invite other teams to publish comparable runs under these items.

Our evaluation harness — the firewall enforcement, drift checks, regression gates, and replication tooling described above — is proprietary; the benchmark is not.

---

## Attribution

HealthAdminBench was created by the Shah Lab at the Stanford School of Medicine and is available under the Apache-2.0 license at `github.com/som-shahlab/health-admin-bench`, together with the project's accompanying materials linked from that repository. Omni Health is not affiliated with, and this evaluation is not endorsed by, the benchmark's authors. All patient data in the benchmark is synthetic; no PHI is involved at any stage of evaluation.

*© 2026 Omni Health. For questions about this methodology, contact the Omni Health evaluation team.*