Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices

Background

I found an interesting article co-authored by Dr. Topol on the imperative for regulatory oversight of LLMs/genAI in healthcare that links to a database of FDA-cleared medical devices that use AI.

“As of April, 2023, 521 devices are included in that database… As other papers have pointed out, only a few of these devices were tested in randomized controlled trials (RCTs) trials; and only a limited number of studies have used external validation, prospective evaluation and diverse metrics to explore the full impact of AI in real clinical settings, and the range of assessed use cases has been relatively narrow with no or very little transparency10.”

This echoes issues raised in a 2022 article co-authored by Dr. Topol around transparency and lack of adequate evaluation, which Dr. Topol confirmed as of our interview 3/27 are still unresolved concerns:

“It is important to point out that the datasets used for these regulatory clearances are often made up of retrospective, single-institution data that are mostly unpublished and considered proprietary. To build trust in medical AI systems, stronger standards for reporting transparency and validation will be required, including demonstrations of impact on clinical outcomes.”

FDA Database of AI-enabled Medical Devices

Below I use the database of FDA-cleared medical devices that use AI to visualize the total number of AI-enabled medical devices cleared by FDA over time since 1995, categorized by area of medical application. Click series to remove, or invert selection and click series to add. Mouse over datapoints to see series and data labels.

We can also look at the data in terms of the number of new devices cleared each year:

Finally, we can look at how the total number of devices by category changes over time with a racing bars plot. To watch the animation across time press play.