AI gets Choosing Wisely right, but stumbles on a few low-value tests

Clinical takeaway: Current commercial large language models (LLMs) generally aligned with Choosing Wisely recommendations and rarely failed to recommend indicated care, but clinicians should be cautious when using AI guidance for low-value testing scenarios, particularly incidental thyroid nodules and routine ECGs in low-risk adults.

AI tools are increasingly being used by both clinicians and patients, yet most evaluations have focused on diagnostic accuracy rather than whether these systems promote high-value care. Overuse remains a major challenge in US healthcare, costing an estimated tens of billions of dollars annually, making it important to understand whether AI reinforces or reduces unnecessary testing and treatment.

In this benchmark study, investigators evaluated 6 commercial LLMs from OpenAI, Anthropic, and Google using 225 clinician-facing scenarios derived from 15 primary care Choosing Wisely recommendations. The dataset included 150 scenarios in which restraint was appropriate and 75 matched scenarios in which the same intervention was clinically indicated. Across 4,050 responses, models correctly recommended appropriate action 98.2% of the time and correctly advised restraint in 90.7% of low-value scenarios.

Overall, LLMs recommended unnecessary care in 5.4% of low-value scenarios and failed to recommend appropriate care in just 0.6% of action-appropriate scenarios. Eleven of the 15 recommendations produced no false-restraint errors, and 8 recommendations generated no overuse at all.

Most overuse was concentrated in just two recommendations. LLMs recommended thyroid ultrasound for incidental subcentimeter nodules in 42.8% of cases despite guidance favoring restraint, and they recommended routine ECGs in low-risk adults in 28.3% of scenarios. By contrast, overuse rates for the remaining recommendations were generally 3% or less.

Differences among models were modest overall, with overuse rates ranging from 2.4% to 9.6%, while false-restraint rates remained below 2% for every model tested. Notably, performance differences were greatest in areas where professional societies disagree, such as management of small incidental thyroid nodules.

The accompanying editorial called the study "an important step toward measuring the impact of AI on value in terms of both overuse and underuse" and noted that, unlike human clinicians who often gravitate toward either more-intensive or less-intensive care, the models showed little evidence of a tradeoff between overuse and underuse. However, the editorialists cautioned that future research must move beyond standardized scenarios and evaluate AI performance in the "messier middle zone" of real-world patient care.

“For a clinician choosing among these tools, the models behaved alike on settled recommendations and differed most where the guidance itself is unsettled,” the study authors wrote.

Source: Ariel D, Endelman S, Laban G. (2026 Oct 20) Ann Intern Med. Overuse and False Restraint in Large Language Model Recommendations for Choosing Wisely Scenarios