SHAKED in the ER: Accuracy Held, Adoption Fell

30-second summary

• What — In a single hospital in Haifa, Israel, researchers ran a 4-week, non-randomized, prospective quasi-experiment (n = 1,138 emergency department patients, propensity-score/IPTW-adjusted comparison across two parallel ED wings) of SHAKED, a multi-LLM clinical decision-support tool built on AWS Bedrock. The study is reported under the DECIDE-AI stage-1 standard — an early feasibility-and-safety evaluation, explicitly not a randomized efficacy trial.
• So what — The tool itself did not fail: 0 adverse events, and 99 of 100 expert-reviewed outputs were judged clinically appropriate. What failed was sustained use. Weekly adoption fell from 68% to 30%, and each additional hour on shift lowered the odds of continued use (OR 0.72, 95% CI 0.62–0.83) — workload-sensitive disengagement, not a rejection of the tool’s competence. Clinicians did not abandon it evenly: they kept preferring it for radiology-consult tasks (OR 2.98, 95% CI 1.58–5.63). The trial’s primary clinical outcome, ED length of stay, was identical in both wings (4.9 hours, P = 0.99); a secondary measure, consultation cycle time, showed a non-significant trend toward shortening (–9.4 minutes, P = 0.077).
• Now what — The authors’ own abstract states the results “do not justify clinical deployment of AI clinical decision support at this stage,” and they call for a randomized trial next. Read plainly, that is not a failure story — it is a stage-1 feasibility study doing exactly the job that standard was designed to do. AWS provided Partner funding and $20,000 in cloud credits for the pilot; the paper does not evaluate AWS’s technical performance, and neither do we. The open question this study leaves behind is about clinician engagement design, not model accuracy.

5-minute read

Two curves, opposite directions

The most useful way to read this study is as two separate storylines that happen to share a timeline. One storyline is about whether the tool is safe and correct. The other is about whether clinicians keep using it. In SHAKED’s four weeks, those two lines moved in opposite directions.

On the safety side: zero adverse events across the pilot, and in an expert-reviewed sample of 100 SHAKED outputs, 99 were judged clinically appropriate. On the adoption side: usage fell from 68% of eligible cases in week one to 30% by week four, and the odds of a clinician continuing to use the tool dropped by 28% for every additional hour worked in a shift (OR 0.72). The authors describe this as workload-sensitive disengagement — a specific, measured pattern, not a stand-in for “burnout” or “distrust,” neither of which this study measured.

Not a blanket rejection

The decline was not uniform across use cases. Clinicians were significantly more likely to keep using SHAKED specifically for radiology-consultation questions (OR 2.98, 95% CI 1.58–5.63) than for other tasks. That is a meaningfully different finding from “doctors stopped trusting the AI.” It looks more like task-specific triage of when a tool is worth the extra step during a busy shift — selective reliance, not wholesale abandonment.

The endpoint that did not move

ED length of stay — the trial’s primary clinical outcome — was 4.9 hours in both the SHAKED wing and the control wing (P = 0.99). Consultation cycle time trended shorter by 9.4 minutes in the intention-to-treat analysis but did not reach significance (P = 0.077). None of this is dressed up in the abstract: the authors state directly that the evidence does not justify deploying this kind of tool clinically yet.

Deep dive

1. Background

Clinical decision-support (CDS) tools built on large language models are being piloted inside real hospital workflows faster than the evaluation science around them is maturing. DECIDE-AI (Vasey et al. 2022) was written to close part of that gap: it is a reporting standard for the early stage of clinical AI evaluation — feasibility and safety in real settings — before a system is put through a randomized efficacy trial. Conceptually it borrows the logic of phased drug and device development: confirm safety and workability first, test for benefit second. This study is one of the more complete public examples of a hospital following that standard in the emergency department.

SHAKED itself is described in the paper as an event-driven pipeline: clinical events in the electronic medical record trigger a pipeline coordinated by an event server and scheduler, which pulls patient data, generates structured history and automated clinical insights, and stores them in a central database that clinicians access through a chat interface backed by a retrieval-augmented generation (RAG) module. The paper describes SHAKED as built on “multiple large language models” served through the AWS Bedrock platform; it does not specify which underlying model family is used for which task, and Bedrock itself is a multi-vendor model marketplace, so no single model can be identified from the available material. That detail remains unverified.

2. What this study newly shows

Three findings, reported together, are the actual contribution: (1) the system was safe and largely accurate in real clinical use over four weeks (0 adverse events; 99/100 expert-reviewed outputs appropriate); (2) adoption nonetheless collapsed, and the mechanism identified was workload, not accuracy — each additional shift-hour lowered the odds of continued use (OR 0.72, 95% CI 0.62–0.83); (3) that decline was not uniform: clinicians selectively kept using the tool for radiology-consult tasks (OR 2.98, 95% CI 1.58–5.63), suggesting task-specific triage of when the tool earns its cost in time rather than a general loss of confidence. The trial’s primary clinical outcome, ED length of stay, showed no difference between wings (4.9 h vs. 4.9 h, P = 0.99).

Put together, the paper’s own framing is the right one: the bottleneck this pilot surfaces is sustained clinician engagement, not model competence. That is a narrower and more specific claim than “AI adoption is hard,” and it is falsifiable in a way that matters for the next trial design.

3. Strengths and limits of the method

This is a single-institution, non-randomized, 4-week quasi-experiment in 1,138 patients (DECIDE-AI stage-1 feasibility and safety evaluation), not a randomized efficacy trial, and its primary clinical outcome was null. The two ED wings ran in parallel rather than being randomly assigned, and the comparison was adjusted using inverse-probability-of-treatment weighting (IPTW) with a lasso-logistic propensity model, stabilized average-treatment-effect weights truncated at the 1st/99th percentiles. Reported covariate balance after weighting was good — all standardized mean differences within ±0.10, maximum 0.032 — which is a transparent and reasonably rigorous way to run a quasi-experiment. It does not, however, rule out unmeasured confounding (for example, staffing patterns or case-mix differences between wings that were not captured in the observed covariates).

On the reproducibility side, this pilot is unusually well-instrumented for a clinical AI study: the analysis and figure-generation code is released under an MIT license on GitHub with a permanent Zenodo archive (DOI 10.5281/zenodo.20736931), and the trial is registered (ClinicalTrials.gov NCT06902675). Patient-level data itself is under controlled access per Israeli health-privacy law and institutional review board approval — a standard, not unusual, tradeoff for clinical data. What remains behind Nature Medicine’s paywall, and is therefore marked unverified in our own review, is the full statistical methods narrative and any qualitative account of why clinicians disengaged as shifts got longer.

4. Connection to a neighboring domain

DECIDE-AI’s structure — safety and feasibility first, efficacy later — is not a metaphor borrowed loosely from drug development; it was deliberately modeled on phased clinical trial design (Vasey et al. 2022 is explicit about this). That gives this study a clean cross-domain hook: the same “prove it doesn’t hurt anyone and clinicians can actually use it before you prove it helps” sequencing that governs Phase 1/2/3 drug and device trials is being asked, here, of an AI system. Firms and investors used to reading TRL ladders for hardware or clinical-trial phases for therapeutics can read DECIDE-AI stage 1 the same way: an early, necessary, deliberately unglamorous checkpoint.

Two other pieces of clinical-LLM research surfaced in the same period are worth naming without conflating them with this one. A separate, unrelated-authorship preprint (not peer-reviewed) evaluated how several frontier LLMs’ accuracy on nephrology questions degrades under answer-choice perturbation — a robustness question, not an adoption question, and its evidence base and authorship are entirely distinct from this study. A different Nature Medicine brief communication from a separate NYU-affiliated group compared general-purpose frontier LLMs against specialized clinical tools on a benchmark basis. Both are useful company for this study — they sit at different layers of “how good is clinical AI, really” — but neither verifies nor is verified by this pilot; the three simply happened to publish in the same season, and we are not treating them as a mutually-confirming set.

5. Commercialization and investment angle

TRL 4 — a single-institution initial-feasibility pilot. By the authors’ own conclusion, moving to TRL 5 (deployment-justifying evidence) requires a multi-site randomized trial and, specifically, evidence that the adoption-decay pattern found here can be engineered around.

On funding: the study’s Funding section discloses that AWS (a subsidiary of Amazon, NASDAQ: AMZN) provided Partner funding and $20,000 in cloud-computing credits, covering use of the AWS Bedrock inference platform underlying SHAKED. The authors state the funder had no role in study design, data collection, analysis, or the decision to publish, and all seven authors declare no competing financial interests. We want to be precise about scope here: this result does not reflect on AWS Bedrock’s technical performance, and the paper does not make any such claim either. The finding is about clinical workflow integration and clinician time-cost under workload, not about model or infrastructure quality. Because this is nonetheless a publicly traded company’s funded system producing a declining-adoption, null-primary-outcome result, this post is flagged for Principal review under our red-zone policy before any public issuance; no security-level judgment is expressed here or implied.

6. The counterargument

Our independent review (skeptic gate) flagged one framing risk worth stating directly: it would be easy, but wrong, to summarize this paper as either “LLM clinical decision support is safe” (true but incomplete — it also failed to move the primary outcome and lost most of its users) or as a failure story (also wrong — the authors’ own conclusion is measured, not defensive). The corrective language we are required to carry into this piece:

This is a single-institution, non-randomized quasi-experiment (DECIDE-AI stage-1 feasibility-and-safety evaluation) conducted over four weeks in 1,138 patients, not a randomized efficacy trial. The primary clinical outcome, length of stay, was identical in both wings (4.9 hours, P = 0.99), and the authors state directly in their abstract that the results “do not justify clinical deployment of AI clinical decision support at this stage.” The disengagement observed is described as workload-sensitive; constructs like “fatigue” or “distrust” were not measured in this study. Cloud infrastructure and credits ($20,000) were funded by AWS, but the paper evaluates no vendor’s product.

A second, structural limitation: because the wings were not randomized, IPTW-based balance on observed covariates does not fully rule out unmeasured confounding (staffing, acuity mix) as a partial explanation for any of the reported associations, including the length-of-stay null result itself.

7. Metrics to watch

• Whether the authors’ promised follow-up randomized trial materializes, and whether it targets clinician engagement design (not just model accuracy) as its primary lever.
• Whether any public description of SHAKED’s deployment status outpaces the trial’s own conclusion. We note, without independently adjudicating it, that public communications associated with the hospital have at times described a related system in more affirmative, already-in-use terms than this trial’s concluding sentence; we have not independently confirmed which system version or deployment scope those communications refer to, and flag the gap rather than resolve it.
• Whether the underlying LLM vendor/model mix behind SHAKED is disclosed in any future publication or code update — currently unverified.
• Any workflow or UX redesign work aimed specifically at workload-sensitive disengagement, since that is the bottleneck the authors themselves identify.

References

• Leibovitch, Liron, Adi Ahituv, Alon Gorenshtein, Dvir Aran, Moran Sorka, Keren Miron, and Shahar Shelly. 2026. “Prospective Evaluation of a Large Language Model Clinical Decision Support System in the Emergency Department.” Nature Medicine, August 19, 2026. https://doi.org/10.1038/s41591-026-04601-5.
• ClinicalTrials.gov. Trial registration NCT06902675, “Prospective Evaluation of an LLM-Based Clinical Decision Support System (SHAKED) in the Emergency Department.” https://clinicaltrials.gov/study/NCT06902675.
• Leibovitch, Liron, et al. 2026. SHAKED-analysis (analysis and figure-generation code), GitHub, MIT license; permanent archive at Zenodo, https://doi.org/10.5281/zenodo.20736931.
• Vasey, Baptiste, Myura Nagendran, Bruce Campbell, et al. 2022. “Reporting Guideline for the Early-Stage Clinical Evaluation of Decision Support Systems Driven by Artificial Intelligence: DECIDE-AI.” Nature Medicine 28: 924–933. DOI: 10.1038/s41591-022-01772-9.

Disclosure

• This post is for informational purposes only and does not constitute investment advice.
• The author discloses no positions in the securities or companies mentioned.
• COI note: All seven authors of the underlying study declare no financial competing interests (per the journal’s published Ethics declarations). The study’s Funding section discloses that AWS, a subsidiary of Amazon (NASDAQ: AMZN), provided Partner funding and $20,000 in cloud-computing credits used for the AWS Bedrock inference platform underlying SHAKED; the authors state the funder had no role in study design, data collection, analysis, or the decision to publish. All quantitative results in this post — adoption rates, odds ratios, length-of-stay figures, expert-review appropriateness — are the study authors’ own reported findings, attributed to the paper, not vendor claims.