Doctors, AI, And The Measure In Microsoft’s Science Paper
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Doctors, AI, And The Measure In Microsoft’s Science Paper on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get everyday essentials delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Microsoft’s MAI-DxO diagnostic system scored 83.2% on 304 complex New England Journal of Medicine cases in results published in Science on October 22, 2026. The comparison involved individual generalists and historical, text-based cases, not specialists or real patients; a prospective study is planned at Beth Israel Deaconess Medical Center.

Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 complex medical cases, compared with 36.1% for generalist physicians using textbooks and the internet, according to a paper published in Science on October 22, 2026. The results show strong performance on selected historical cases, but do not establish how the system would perform in routine care or with real patients.

The paper, titled Sequential diagnosis with language models, also reports results from 325 emergency department cases. MAI-DxO scored 80.3% on that set, compared with 46.8% for physicians. The figures are Microsoft’s results for its system, and the article was peer-reviewed. Peer review evaluates the study’s methods and reporting; it does not by itself mean researchers outside Microsoft have reproduced the findings.

The benchmark presents a brief case summary first. The system or physician can request additional information, such as a test result, from a model holding the full case file. Those requests carry estimated costs, so the benchmark considers both the diagnosis and the process used to reach it. MAI-DxO uses a language model in a multi-agent setup: virtual physician roles propose diagnoses, request questions or tests, and critique one another over several rounds.

Microsoft’s chart also reports that GPT-5 alone scored 78.9% on the complex-case set, without the MAI-DxO orchestration layer. The source account says the layer added about four percentage points and reduced estimated test costs. That distinction matters: the result reflects both a capable underlying model and a method for managing diagnostic steps, rather than showing that the orchestration system alone produced the high score.

At a glance
reportWhen: Published October 22, 2026; prospective…
The developmentMicrosoft published a peer-reviewed Science paper reporting that its MAI-DxO system outscored generalist physicians on two sets of historical diagnostic cases.
Thorsten Meyer AI
Science · 22 Oct 2026

Infographic 01 · The scoreboard

83% vs. 36%: the number, and who it was measured against.

Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.

83.2%MAI-DxO on 304 hard NEJM cases
36.1%generalist physicians with textbooks and internet
22.3%generalist physicians without references
80.3 / 46.8emergency cases: MAI-DxO vs. physicians, in %

Hard cases (NEJM)

Diagnostic accuracy, 304 case records

  • MAI-DxOno test budget83.2%
  • GPT-5 alonesame benchmark, no orchestration78.9%
  • Physicians, open booktextbooks and internet36.1%
  • Physicians, closed bookno references22.3%

Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.

Who was compared

The fine print behind “beats doctors”

  • Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
  • Working alone: no colleagues; in the preprint also no search engines or language models.
  • Text-only, historical cases: no real patients, no physical exam, no imaging.
  • Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
  • Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.

Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI

What the Scores Mean for Care

The results matter because they test a task closer to clinical reasoning than a conventional multiple-choice examination: deciding what information to seek next, and when to stop. A system that can ask useful questions and weigh tests against their costs could, if validated in practice, affect how diagnostic support tools are built and assessed. The study, however, measures performance on a benchmark rather than patient outcomes.

The comparison also does not show that MAI-DxO is ready to replace clinicians or that it outperforms the teams most likely to handle difficult cases. The physicians were primary care doctors and hospital generalists working individually; specialists and collaborative clinical teams were not included. According to the source account, the cases were selected for difficulty and would often be referred to specialists in practice. The gap is informative about this test setup, but not a direct estimate of a contest between the AI and a hospital’s full diagnostic process.

For hospitals, insurers and patients, the immediate takeaway is narrower: Microsoft has reported high accuracy in a controlled, simulated setting, and a peer-reviewed paper now describes the work. Whether the system can improve care safely, efficiently and fairly depends on evidence from real clinical use, including how it handles common conditions, incomplete information and the consequences of incorrect answers.

Products Worth Considering

Samsung Galaxy Tab S10 Lite, 10.9-inch LCD, 6GB RAM, 128GB Storage, Gray

Samsung Galaxy Tab S10 Lite, 10.9-inch LCD, 6GB RAM, 128GB Storage, Gray

Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite

Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite

As an affiliate, we earn on qualifying purchases.

How the Benchmark Was Built

The study uses Microsoft’s Sequential Diagnosis Benchmark, or SDBench. Instead of giving participants every detail at once, it starts with a short presentation and requires them to request more information. This imitates one part of diagnosis: choosing which questions or tests to pursue. The benchmark scores the final diagnosis and assigns estimated costs to requested tests.

The source account describes additional methodological details from a June 2025 preprint, and cautions that these may differ from the final Science paper. In that earlier version, a model graded diagnoses against the published case answer, while physicians re-graded a sample with good but imperfect agreement. If a requested test result was absent from the original file, a “gatekeeper” model generated a synthetic result. Test prices were estimates based on billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge.

The earlier preprint also described 21 U.S. and U.K. generalists, with a median of 12 years’ experience. They were assessed on a subset of held-out cases, while the AI was assessed on all 304. The final paper reports physician scores of 22.3% without reference materials and 36.1% with textbooks and internet access. The source material notes that the final article’s full text is paywalled, so not every preprint detail can be assumed to match the published version.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft CEO Mustafa Suleyman, speaking to TechCrunch

Limits the Study Leaves Open

The study does not establish how MAI-DxO performs with live patients, physical examinations or imaging. The cases were historical text records delivered through a simulator. The source account also says the NEJM set contained no healthy patients and no harmless explanations, limiting what the study can say about false alarms. A system tested only on difficult cases may not be calibrated for the range of patients seen in everyday care.

Other questions remain about independent replication, the final paper’s detailed methods, and how performance would compare with specialists or multidisciplinary teams. The reported costs are estimates, not a demonstration of actual savings to a hospital or insurer. The results also do not show whether AI assistance changes treatment decisions, patient outcomes, clinician workload or the risk of missed diagnoses.

The Planned Patient Study

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence about how the tool works in a clinical workflow, though the study’s design, timing, measures and results have not been supplied here.

Until those findings are available, the Science paper supports a claim about performance on specified historical benchmarks, not a conclusion about clinical readiness. Further evidence would need to show how the system behaves with real patients and whether its diagnostic suggestions improve care without creating unacceptable risks or costs.

Key Questions

What did Microsoft’s MAI-DxO score on the complex cases?

Microsoft reported 83.2% accuracy on 304 complex cases from the New England Journal of Medicine. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without those resources, according to the published results described in the source material.

Was MAI-DxO tested on real patients?

No. The reported benchmark used historical, text-based case records in a simulator. The source material says a prospective study involving consenting patients is planned at Beth Israel Deaconess Medical Center.

Did the AI outperform specialists?

The reported comparison was with generalist physicians working individually. Specialists and physician teams were not included, so the study does not establish how MAI-DxO compares with them.

Did the MAI-DxO method account for all of the accuracy result?

Microsoft’s chart also gives GPT-5 alone a 78.9% score on the complex cases. The source account says MAI-DxO’s orchestration added about four percentage points and lowered estimated test costs, indicating that the underlying model contributed substantially to the result.

Source: ThorstenMeyerAI.com

You May Also Like

The Real Limits of AI in Medical Diagnostics

Just when AI seems poised to transform diagnostics, critical limitations raise questions about its true potential in healthcare’s future.

Wellness Signal Monitor: Coroner Weighs In On Suspected Measles Death

Pennsylvania coroner reports unvaccinated woman likely died from measles complications, highlighting ongoing vaccination concerns and public health risks.

FDA’s Landmark Approval: What It Means For Consumer Safety And Cancer Therapy Trends

The FDA has approved a groundbreaking targeted therapy for metastatic pancreatic cancer, marking a significant step in cancer treatment and consumer safety.

6 Effective Natural Remedies for Diagnosis and Treatment Using Medical AI

AIThis post was created with the assistance of artificial intelligence (AI). Welcome…