Before expanding ambient AI scribes, measure each specialty separately | Verilian Health
Before expanding ambient AI scribes, measure each specialty separately
A 2026 study found one in five oncology clinicians tried an enterprise scribe and about one in eight kept using it. CMIOs expanding scribes should give each specialty its own baselines and thresholds.
Par Verilian Health Editorial10 min de lecture
Cet article est en anglais.
Few oncology clinicians kept using the enterprise scribe
An enterprise rollout of an ambient AI scribe in oncology was used by 190 of 958 clinicians, or 19.8 percent, according to a study published in JCO Oncology Practice on 17 October 2026. That figure counts any use. Sustained adoption, defined as three or more weeks of use, reached 124 clinicians, about 13 percent; another 66 tried the tool for one to two weeks. Roughly four in five clinicians never used it.
The scribe was implemented from June to September 2025 and assessed with surveys, EHR data and tool-use data in a prospective mixed-methods design. The authors are direct: "Despite enterprise-wide implementation and training, adoption was low and varied by clinician and visit characteristics." Use was most common in new, video and in-person visits, and users were more often physicians from medicine and surgery.
The burden results point in two directions. Among the 45 clinicians who completed both pre- and post-rollout surveys, the share reporting high or excessive note-writing time fell from 78 percent to 60 percent (P = .043). In the EHR data, the likelihood of closing notes on the visit day held at 70 percent for never-users and fell slightly for users: from 73 to 70 percent among attempted adopters and from 56 to 54 percent among adopters.
A result in one specialty does not establish results in another
The study cannot tell you whether low uptake is a fact about oncology. It grouped clinicians by observed use rather than assigning them, so specialty is bundled with how the tool was implemented and who took it up. The groups also differed before rollout: adopters closed 56 percent of notes on the visit day, against 70 percent for never-users and 73 percent for attempted adopters. Later differences may partly reflect the people rather than the tool.
A business case should keep three results separate: low adoption, lower reported note-writing time among 45 survey respondents who may not represent their colleagues, and no improvement in same-day note closure in any use group. A board paper that quotes only one of them tells a partial story.
Two earlier studies relied on volunteers. A University of Iowa Health Care study (Applied Clinical Informatics, 2025) enrolled 38 volunteer physicians and advanced practice providers, and a 2025 pilot at a New York City cancer center recruited voluntary participants from clinicians who had worked on care innovation projects. Our judgment is that early volunteers' results may overstate what later adopters will see. Treat that as a reason for caution, not a measured ceiling.
Recevez la prochaine analyse dans The Signal.
Les nouvelles analyses sont incluses dans The Signal du vendredi. Un courriel par semaine, désabonnement en un clic.
Randomization isolates the product, not the specialty
The one randomized trial in our review, a pragmatic trial published in NEJM AI on 26 November 2025, randomized 238 outpatient physicians from 14 specialties to Microsoft Dragon Ambient eXperience (DAX) Copilot, to Nabla or to usual care between 4 November 2024 and 3 January 2025. Its abstract reports results by product, with specialties combined. Nabla users cut time-in-note by 9.5 percent against usual care (95 percent CI, −17.2 to −1.8 percent). DAX users showed no significant change (−1.7 percent; 95 percent CI, −9.4 to +5.9 percent).
A significant result for one product and a nonsignificant result for another does not, by itself, establish a difference between the products. The final paper's abstract reports each product against usual care. An earlier preprint of the trial, not peer reviewed, reported a direct comparison: Nabla's time-in-note was 7.8 percent lower than DAX's (95 percent CI, −15.5 to −0.1 percent; P = .05). That figure comes from the preprint alone and may differ from the final paper, so treat it as provisional.
The trial also says little about any single specialty. Physicians used the tools in a minority of visits: DAX in 33.5 percent of 24,696 visits and Nabla in 29.5 percent of 23,653, according to the trial's PubMed record. The arms held 79, 79 and 80 physicians across 14 specialties, which by our arithmetic averages fewer than six physicians per specialty in each arm. That is too few to show how either product performs in a given specialty.
Published time savings do not add up to a range
Other studies measure different things against different comparators. A 2025 JAMIA study of 45 physicians in eight ambulatory disciplines at one large academic medical center found that median time per note fell by 0.57 minutes against baseline over three months. A 2026 retrospective cohort study in JMIR AI in one tertiary academic emergency department found 72.6 seconds less on-shift documentation time per encounter, compared with encounters where physicians chose not to use the scribe. A 12-week study of 39 clinicians in a pediatric primary care network (Applied Clinical Informatics, 2025) found no significant change in documentation time, after-hours or total EHR time, or visit closure rates.
Minutes per note against a baseline, seconds per encounter against self-selected encounters and a relative change against randomized controls do not form a range. Quote any such figure only with its measure, comparator and setting. In the expansion business case, replace published time-savings assumptions with your own measured results when available. Before then, use clearly labeled scenarios rather than promised savings.
What holds everywhere, and what depends on your system
One point holds for every health system: adoption, reported burden, measured efficiency and documentation quality can move independently, as the first three did in the oncology study, so each needs its own measure. The rest depends on your setting. The NEJM AI trial took time-in-note from a routinely collected Epic Signal metric, so a system on another EHR must define an equivalent before it can collect a baseline.
Analytics staffing sets how many specialties you can measure at once. A system with a first wave already live can re-cut existing data by use tier, while one starting fresh needs baselines first. In our view, the more voluntary the use, the more adopters and non-adopters will differ in ways that cloud the result.
Give each specialty its own evaluation and decision thresholds
Every new specialty needs its own evaluation, with baselines, adoption targets and decision thresholds agreed before go-live. Specialties can go live at the same time if each has the support and measurement it needs; what they cannot share is a baseline or a borrowed success figure. We would take four steps, in this order.
Fix the measures so results compare
Adoption is the share of eligible clinicians using the tool and the share of their visits. Burden is a short survey plus time-in-note, after-hours EHR time and same-day note closure, defined the same way in every specialty. Documentation quality comes from a structured review of sampled notes for accuracy and clinically important omissions. Correction burden is the time clinicians spend revising drafts and the share of drafts that need substantial changes.
Collect baselines before go-live
Collect pre-go-live baselines for burden and documentation quality. Define adoption denominators, usage categories and post-launch targets before activation; a new tool's adoption starts at zero, so it needs a target rather than a history. We suggest 12 weeks of baseline as a practical starting point, not a published standard, and would extend it for low-volume clinics, seasonal swings and clinicians whose numbers vary widely from week to week.
Write thresholds before activation
Write the thresholds that trigger a fix (training, templates or workflow changes), a pause and a stop, for benefit and for harm. Set them before activation, because afterward you are negotiating with sunk cost and with your keenest adopters. Set workflow thresholds against each specialty's own baseline variation: a slip like the oncology adopters' 56 to 54 percent could be ordinary variation, so decide in advance how large and how sustained a change must be to count as deterioration.
Report by use tier
Report heavy, occasional and never-users separately, each against its own pre-rollout baseline, with adoption beside time saved. In the JAMIA study the scribe was used in 9,629 of 17,428 encounters (55.25 percent) with significant variation between users, and the authors called for further work to identify which users benefit most. An average across all users hides where the value sits.
The order matters: thresholds without a baseline are guesses, and tiered reporting without agreed measures becomes an argument. The trade-off is capacity. Each specialty adds a cohort for informatics and analytics staff to support and measure, and that capacity, not the contract, should set how many specialties go live together.
Measure accuracy and correction burden, not only time
Workflow measures do not establish safety. Same-day note closure records when a note was signed, not whether it is right. In the NEJM AI trial, clinicians reported clinically significant inaccuracies "occasionally", with ratings of 2.7 for DAX and 2.8 for Nabla on a five-point scale, a clinician rating rather than an audited error rate.
Reviews point the same way. A 2025 systematic review in Applied Clinical Informatics found that scribe accuracy varied, "frequently requiring manual edits and raising occasional concerns about errors." A January 2026 narrative review reports "frequent documentation omissions and occasional clinically significant hallucinations" across studies. Physicians interviewed for a 2025 JAMA Network Open study were largely negative about accuracy and style, particularly note length and editing requirements.
The harm guard therefore sits beside the workflow measures: note accuracy and clinically important omissions against the pre-go-live sample, plus correction burden against a threshold fixed in advance. Ask your patient safety and risk leaders to decide before go-live how omissions are classified and routed, so that they reach review rather than disappear into an average.
The strongest case for a broad rollout
The best advocate for a broad rollout would start from burnout. In the NEJM AI trial, burnout, task load and work exhaustion scores suggested improvement with both tools, and the Applied Clinical Informatics review reports that seven of 10 studies highlighted positive effects on clinician wellness and burnout. Per-specialty evaluation, the advocate would say, delays that relief, adds measurement work most systems cannot staff and forfeits enterprise pricing. The oncology result may simply reflect a weak rollout that better training would fix.
Part of this is right. The burnout signal is encouraging, though the trial's authors call these secondary findings that "need confirmation in larger, multicenter trials." Delayed access has a human cost, which is why we do not ask specialties to queue behind one another. The training argument is weaker: the oncology program already included enterprise-wide training, so more training is a hypothesis to test, not a fix to assume.
The cost objections carry the most weight. Put the pricing question to each vendor in writing, and where analytics capacity is thin, shrink to a core set rather than dropping baselines: the burden survey and note-accuracy sample with pre-go-live baselines, a defined adoption denominator and target, and thresholds agreed before activation. A broad rollout that keeps that core for each specialty is our recommendation under another name. Where a new specialty shares clinics, templates and visit types with one you have already measured, that experience can inform its thresholds and sampling plan, but the new specialty still needs its own pre-go-live baseline.
What to do Monday
Ask your EHR analytics lead for time-in-note, after-hours EHR time and same-day note closure by clinician for each specialty on your expansion list, covering at least the last 12 weeks and longer where volumes are low, seasonal or variable.
Have the CMIO draft a one-page charter per specialty naming the adoption denominator and target, the baseline and correction-burden measures, and fix, pause and stop thresholds, for the AI governance committee to approve before activation.
Ask a lead physician informaticist to design each specialty's structured note review with a pre-go-live sample, and ask your patient safety and risk leaders to decide how clinically important omissions are classified and routed.
If a first wave is live, ask your ambient program lead to re-cut its results by heavy, occasional and never-users against each group's pre-rollout metrics.
Ask the executive sponsor to replace published time-savings assumptions in the business case with your own measured results when available, and with clearly labeled scenarios until then.
Ask your contracts lead to ask each vendor in writing whether activation and pricing can be set by specialty.