LEGACY & INSIGHTS: HEALTH & TECHNOLOGY
Microsoft Says Its AI Out-Diagnosed Physicians. Here Is What the Test Did Not Cover.
Microsoft MAI-DxO outscored physicians on written medical cases in a new Science paper, but Microsoft has not yet tested it on real patients.
Microsoft MAI-DxO out-diagnosed physicians on paper, and the paper is the catch. On 304 hard New England Journal of Medicine cases it named the right diagnosis 83% of the time, while physicians in the same test managed 22% to 36%, Microsoft reports. Its team published the work in Science on October 22, and the gap is hard to wave away. But those cases were written files, not people in a waiting room. So the better question is what 83% proves, and what it cannot yet. Microsoft says it plans a study with real patients, and it has given no timeline.
What Microsoft MAI-DxO Actually Scored
On the 304 journal cases, MAI-DxO reached 83%, according to Microsoft’s announcement. Physicians working without textbooks or the internet reached 22%, and those with outside references reached 36%. Microsoft calls that nearly four times the accuracy of unaided physicians. On a second set of emergency-room cases from a database called MIMIC-IV-ED, it also scored 80% against 47%. The announcement does not say who those physicians were. Microsoft’s November 2025 preprint does: 21 closed-book and 19 open-book doctors, all primary care physicians or hospital generalists.
Who Built the Benchmark
Microsoft built the benchmark, called SDBench, and it built the system tested on it. Its MAI Health Team also wrote the announcement, so the company is describing its own work. Licensing exams, Microsoft says, assess physicians’ knowledge, “not the realities of making a clinical diagnosis,” which is why it built SDBench around step-by-step diagnosis. Its 2025 preprint calls MAI-DxO an orchestrator that simulates a panel of physicians and picks tests with cost in mind, so it works more like a panel than a single model. None of that makes the result wrong, but it does make outside replication important.
What Microsoft MAI-DxO Says About Cost
Microsoft says MAI-DxO delivered its results “at half or even one third of the cost.” The preprint shows why that wording needs care. Its unrestricted setup cost about $5,430 a case, above the $4,725 for physicians with reference tools. Two cost-limited setups ran $2,241 and $1,302 and stayed near 80% accuracy. Cost matters because needless tests are a real expense in medicine. However, those figures come from a November 2025 draft, so the Science paper’s final numbers may differ.
How This Test Differs From Past AI Exams
Licensing exams have often set the bar for medical AI, but Microsoft says they measure knowledge more than clinical practice. MAI-DxO instead had to work step by step, asking for information and ordering tests before it committed to an answer. That is closer to a real clinic, though not the same thing. Still, every case was a written record. So the study could not show how the system handles a patient who hesitates or contradicts a chart. A benchmark can show skill at a task, but it cannot show how a tool behaves on a busy ward.
WATCH / WION NEWS REPORT
WION examines Microsoft’s MAI-DxO diagnostic AI, its step-by-step clinical reasoning, research benchmarks and planned emergency-department evaluation. Source: WION News.
FROM KMOB1003
What Microsoft MAI-DxO Could Mean for Patients
If the results hold up, a tool like this could help doctors work through hard cases and order fewer needless tests. But those are possibilities, not findings. Microsoft says clinical teams will work alongside the system in its planned study, to see where it succeeds and where it falls short. Patients may also want to know who is responsible when an AI suggestion is wrong. Yet the announcement we reviewed does not address that.
Who Checks Microsoft MAI-DxO Next?
The next test is the one that counts. Microsoft plans a prospective study with Beth Israel Deaconess Medical Center in Boston, where patients interact with the system directly. Researchers will compare its performance with physicians’ independent assessments. However, the post gives no timeline or patient count. Until then, 83% is a strong score on written cases. What it means for a person in the room is the one question this research has not answered. The planned Boston study is built to ask it.
Signal Breakdown
Not every claim around Microsoft MAI-DxO carries the same weight.
Confirmed Records
- The announcement. Microsoft’s MAI Health Team posted its results on October 22, 2026.
- The scores. Microsoft reports 83% for MAI-DxO against 22% and 36% for physicians on 304 journal cases.
- Emergency cases. Microsoft reports 80% against 47% on cases from MIMIC-IV-ED.
- The physicians. Microsoft’s November 2025 preprint lists 21 closed-book and 19 open-book doctors, all primary care physicians or hospital generalists.
- The benchmark. Microsoft built SDBench to test step-by-step diagnosis.
- Next step. Microsoft plans a prospective study with Beth Israel Deaconess Medical Center.
Attributed Claims and Analysis
- Relative accuracy. Microsoft calls MAI-DxO nearly four times as accurate as unaided physicians.
- Cost. Microsoft says results came at half or even one third of the cost, but the preprint shows its unrestricted setup cost more than physicians did.
- Design. The 2025 preprint says the system simulates a panel of physicians.
- Exams. Microsoft says licensing exams test knowledge, not clinical diagnosis.
- Independence. Microsoft built both the system and the test, so outside review matters.
Not Established
- who the 16 physicians in the emergency-case comparison were
- which emergency cases the 80% covers, since the preprint lists 325 admissions and holds 71 back as a test set
- whether the tables in the Science paper match the announcement and the preprint, because we have not opened the paper
- how the system performs on real patients, and when the planned study starts
- who is responsible when an AI suggestion is wrong
Sourcing: this report draws on Microsoft AI’s October 22, 2026 announcement and on Microsoft’s preprints, including Sequential Diagnosis with Language Models. science.org blocked our access, so we did not open the Science paper. Physician and cost detail comes from a November 2025 preprint draft, which may differ from the published version. We found no outside clinician comment in the sources we reviewed. The Signal Breakdown above separates confirmed records from attributed claims and open questions.
MAI-DXO / KEY FACTS
RADAR / WHAT TO WATCH
KMOB1003 SELECTS
Music, style, tools, art and objects worth your attention.
