# HealthAdminBench Results — Omni Agent Hub **Report date:** July 2026 · **Evaluation executed:** July 13, 2026, 18:13–19:31 UTC **System under test:** Omni Agent Hub (production build) **Benchmark:** HealthAdminBench v2 — 135 tasks (Stanford School of Medicine, Shah Lab; Apache-2.0) **Protocol:** Single attempt per task · full-credit pass semantics · blind execution · environment-state scoring --- ## Headline results | Metric | Result | |---|---| | **Full-credit completions** (score = 1.0, every graded point earned) | **91 / 135 (67.4%)** — 95% CI [59.1%, 74.7%] | | **Mean task score** (partial credit; secondary metric) | **0.931** | | Attempts per task | 1 (no retries, no candidate selection) | | Unmeasurable tasks | 0 / 135 (every task produced a scoreable end state) | Full-credit completion is the primary metric: a task counts only when the environment's own recorded end state earns **every graded point** — including points awarded for correctly *declining* to act when a task's preconditions fail. A task scoring 0.96 counts as a failure in the completion rate. The mean score is reported as an explicitly partial-credit secondary metric. These results are from the most recent full-corpus run of the release build at the time of reporting; no selection among runs occurred. ## Results by task family | Family | Tasks | Full credit | Mean score | Rubric-judged share of points | |---|---|---|---|---| | Prior authorization | 60 | 52 / 60 (86.7%) | 0.988 | 16.8% | | Claim denials & appeals | 60 | 25 / 60 (41.7%) | 0.859 | 46.0% | | DME order fulfillment | 15 | 14 / 15 (93.3%) | 0.993 | 38.9% | Across the corpus, 69.5% of graded weight attaches to deterministic checks of environment state and 30.5% to rubric-judged free text (a property of the benchmark, not of our protocol). Note that the weakest family, denials & appeals, is also the most rubric-dependent; its scores carry the widest judge-related uncertainty. ## Results by difficulty tier | Tier | Tasks | Full credit | Mean score | |---|---|---|---| | Easy — single-system verify & document | 45 | 40 / 45 (88.9%) | 0.973 | | Medium — multi-system execution | 45 | 31 / 45 (68.9%) | 0.936 | | Hard — adversarial & judgment | 45 | 20 / 45 (44.4%) | 0.884 | ## Family × tier matrix | | Easy | Medium | Hard | |---|---|---|---| | **Prior authorization** | 1.000 (20/20) | 0.996 (19/20) | 0.968 (13/20) | | **Denials & appeals** | 0.940 (15/20) | 0.860 (7/20) | 0.776 (3/20) | | **DME orders** | 1.000 (5/5) | 1.000 (5/5) | 0.980 (4/5) | Cells show mean score (full-credit completions / tasks). Cells contain 5–20 tasks; small count differences between cells are not statistically meaningful. ## Run record The nine disclosure items our methodology paper commits to (*Measuring Autonomous Healthcare Administration*, §11): 1. **Corpus:** HealthAdminBench v2, 135 tasks, evaluated from a vendored snapshot pinned to upstream commit `e71a8f4d6923037805b7f51fbbf608d12ea56cf5`; tasks are never fetched at evaluation time. The snapshot hash recorded by every one of this run's 135 task records is `b2bafeda3388b0d08c27646ad1950ff89e16dbb05ffe2cb3e1c27eee9fe30b63` (SHA-256), and recomputing the hash over the vendored corpus reproduces it exactly. 2. **Attempts:** 1 per task; no candidate selection between attempt and report. 3. **Pass semantics:** full credit only (score 1.0); task score denominator recomputed as the sum of the task's check weights — declared point totals are not trusted. 4. **Blindness:** the system receives the task's natural-language work instruction plus the benchmark's family and scenario labels — descriptive fields of the public task schema, identical for any evaluated system. Every grading field (checks, rubrics, point values, expected values, the expected-outcome field, authoring walkthroughs) is excluded from every runtime surface; no tier, difficulty, or point information reaches the runtime. Disclosed for precision: some upstream scenario labels name the scenario class and are therefore outcome-suggestive; we report this rather than claim an entirely outcome-free projection (methodology §4). Scored runs execute in a strict verification mode, recorded in run metadata, under which a detected leak of grading content fails the task closed. 5. **Scoring input:** the benchmark environment's recorded end state only, read after the run terminates; no self-report of any kind is graded. Consequential actions (filing an appeal, sending a fax, submitting an authorization) earn credit only through evidence captured from the receiving system — e.g., a confirmation identifier read back from its confirmation surface. 6. **Unmeasurable-task accounting:** a task whose end state cannot be read is classed unmeasurable and, by our accounting policy, counts as zero in headline metrics — never silently dropped. This run: 0 of 135. 7. **Isolation:** fresh browser context and freshly provisioned, reseeded environment state per task; nothing carries over between tasks. 8. **Build identification:** source revision `13d371c5ceeb3a428325e49ea455aad624e1a2ae`; container-image digest `sha256:5d46c666baee0a038f032bf9daecdb41cdc5972e29b48592f8f1e9827da36468`. Freshness was verified at dispatch — the container image tagged with this exact source revision carries this digest (image published minutes before the run started) — and re-verified against the registry during report preparation. Before any scored run the harness verifies whether the deployed build matches the source revision under evaluation, records the verdict and the running image digest in the run manifest, and aborts on any mismatch between the pinned image and the running build. 9. **Judge treatment:** rubric points are graded by a language-model judge against the benchmark's rubrics; the judge's identity and version are pinned in the run record. Internally observed run-to-run judge variance is about one rubric point per task; observed tier-mean noise bands are roughly ±0.02–0.05 depending on cell size. Reliability claims (a task counted as *reliably* passing) require full credit across N ≥ 5 replicate runs; this report makes single-attempt claims only. The full per-task score record for this run, with the pins above, is published alongside this report as `omni-hab-run-record-2026-07-13.json`. ## Context Baselines for this corpus have already been reported — by the benchmark's authors in their paper (models run under a standardized harness, plus native computer-use systems) and in at least one vendor system card. We are aware of no other result reported for a full production system under the end-to-end, full-credit, single-attempt semantics and disclosure items above. The corpus and environments are public; we invite other teams to publish comparable runs under those items. ## Attribution HealthAdminBench is an independent benchmark authored by the Shah Lab at the Stanford School of Medicine and released under the Apache-2.0 license (`github.com/som-shahlab/health-admin-bench`). Omni Health is not affiliated with, and this evaluation is not endorsed by, the benchmark's authors. All patient data in the benchmark is synthetic; no PHI is involved at any stage of evaluation. Evaluation methodology is described in the accompanying whitepaper, *Measuring Autonomous Healthcare Administration*. --- © 2026 Omni Health. Results apply to the pinned build and corpus stated above.