AI-Enabled Clinical Decision Support Tools for Mental Healthcare: A Product Review
Abstract
Introduction
While AI has transformed several fields of medicine, its application in mental healthcare is still in the early stages due to challenges such as the subjective nature of psychiatric diagnoses and the complexity of mental health disorders. Nevertheless, advances are being made, with AI tools beginning to assist in tasks like detection of mental health conditions through speech and text analysis, and personalized treatment recommendation. The implementation of AI into mental healthcare holds the promise of a more accurate, efficient and personalized approach to the treatment of mental disorders (Shen et al., 2021).
One of the key challenges in designing AI systems for human-AI interaction is the uncertainty surrounding AI’s capabilities and the complexity of generating its output (Yang et al., 2020). This poses unique design challenges for AI systems, particularly in the context of healthcare, where accurate and reliable diagnoses and treatment recommendations are crucial (Shen et al., 2021). Regulatory agencies, such as the US Food and Drug Administration (FDA), have undertaken efforts to address the challenges associated with regulating AI software (Carolan et al., 2022; Gerke et al., 2020). However, the implementation of these regulatory reforms is still pending and no tangible solutions to address the complexities involved in regulating AI software is currently available, leading to a significant shortage of regulated AI-CDSS for healthcare (Mashar et al., 2023; US Food and Drug Administration, 2019, 2021).
Given the complex nature of mental health conditions, the lack of standardized, high-quality data, rigorous regulatory processes, and the ethical implications in data use, the development and regulation of AI-CDSS for mental healthcare present considerable challenges. These hurdles often prove too resource-intensive for small firms and start-ups to overcome. By mapping out the regulatory status and scientific efficacy of each product, our review provides insights into the current landscape of regulated AI-CDSS in mental healthcare, thereby identifying potential avenues for future innovation, application, and regulation.
Materials and methods
Artificial intelligence (AI)-enabled clinical decision support products
Artificial intelligence refers to “the capability of an engineered system to acquire, process and apply knowledge and skills” (International Organization for Standardization, 2020). Medical devices are intended for diagnostic, treatment, or prevention of mental disorders (European Commission, 2022; United States Code, 1964; World Health Organization, 2017). AI software that fulfills the purpose of a medical device may be considered a medical device itself (European Commission, 2019; US Food and Drug Administration, 2013). In this context, we focus on regulated diagnostic and treatment recommendation products that may be used for clinical decision support.
Search strategies and inclusion criteria
We followed four main approaches to generate an overview of AI software products for mental health. Patenting is a common practice to protect intellectual property rights and encourage innovation. As AI technologies continue to advance, patents are expected to increase, particularly for AI-enabled medical devices (ResearchandMarkets, 2020). Accordingly, first, we used information collected as part of a patent review on AI-enabled devices for mental healthcare (Kleine et al., 2023). We checked the assignee and inventor details of the retrieved patents to identify relevant companies. Detailed information about the search strategy and selection criteria is described in Kleine et al. (2023).
Second, exhibitor lists from the American Psychological Association Congress, the American Psychiatric Association Congress, the Association for Behavioral and Cognitive Therapies, the Anxiety and Depression Association of America, and the German Psychotherapy Congress were searched for relevant companies. Third, we searched the FDA Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices database (Drug Administration, 2022), the Medical AI Evaluation Database (Wu et al., 2021), the Medical Futurist AI-based algorithms database (The Medical Futurist, 2023), and the HealthSkouts FDA/CE Certified Health App database (HealthSkouts, 2023).
We applied four inclusion criteria. First, the product should be a medical device. Thus, we excluded products targeting diseases unrelated to mental health (e.g., Medtronic), wellness apps (e.g., the Sonde Mental Fitness App), products used for organizing and managing patient information (e.g., Telemynd), and educational and research products or services (e.g., ABA Technologies).
Second, the product should include AI software in its core functionality. This criterion led to the exclusion of products that did not clearly employ AI algorithms (e.g., Autism Analytica).
Third, we included products specifically aimed at diagnosing or generating treatment recommendations for mental health practitioners. This led to the exclusion of diagnosis or treatment apps to be used by patients (e.g., Endeavorrx by Akili Interactive), and psychopharmaceutical and technical products (e.g., Neumora, Otsuka, and the SAINT Neuromodulation System by Magnus Medical).
The U.S. Food and Drug Administration (FDA) and the European Union (EU), via its CE marking, are the leading medical device regulatory bodies globally. The FDA and the EU’s CE marking assure high-quality standards in safety and efficacy, guaranteeing that the device has been thoroughly reviewed and tested (Muehlematter et al., 2021). Accordingly, fourth, we included products that were CE-marked or cleared by the FDA. Information about CE certificates and FDA clearance or approval was confirmed using the EUDAMED and the FDA databases. In addition, all vendors were contacted to verify and supplement the collected information.
We coded information about the targeted disorder, data input, data output, the method of deployment, integration possibilities, and date to market (Leeuwen et al., 2021). We maintain an overview of products on XYZ.com. Discrepancies between the products considered in the review and those displayed on XYZ.com may be caused by updates of the products’ regulatory status, the inclusion of new products, or refusal of companies to appear on the website.
Scientific evidence
We searched for scientific evidence proving the efficacy of the included products. First, the database PubMed was searched by vendor and product name for peer-reviewed articles published between Jan 1, 2015, and July, X 2023 (Leeuwen et al., 2021). Queries are provided in Table XYZ. Second, a manual search was performed by inspecting the vendor’s websites for listings of papers and requesting vendors to provide peer-reviewed papers. No date restriction was applied for the manual search.
Included articles were original, peer-reviewed, and in English, and aimed to demonstrate the efficacy of the AI software. Papers were included when the product name (including known former names) and/or company name were mentioned, the tool was applied on in vivo human data, and efficacy of the product was reported on an independent dataset (data on which the algorithm was not trained). Letters, commentaries, reviews, study protocols, white papers, and case reports were excluded. Papers were assessed by two of the authors who independently screened the title, abstracts, and full paper for inclusion criteria. Cases of disagreement were resolved by the reviewers in a consensus meeting.
Evaluation of scientific evidence
We used the CONSORT-AI (Consolidated Standards of Reporting Trials Artificial Intelligence) checklist to evaluate the scientific evidence (Liu et al., 2020). The CONSORT-AI extension of the original CONSORT checklist (Moher et al., 2012) includes 14 new items requiring a clear description of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention is integrated, the handling of inputs and outputs of the AI intervention, and provision of an analysis of error cases (Liu et al., 2020; Tornero-Costa et al., 2023; zhou_etal21is?). Information regarding the scientific evidence was coded independently by two of the authors. Any discrepancies were documented and resolved in consensus meetings.
Data analysis
The data were analyzed using R (Version 4.2.2, rcoreteam22b?). First, we display information on the inclusion and exclusion of products found through our search strategies. Second, we provide information on the targeted disorder(s), data input, output, and the time passed between company foundation, product certification, and the publication of scientific evidence addressing product efficiency. In addition, we provide information on CE marks and FDA clearance. Finally, we evaluate the scientific evidence addressing product efficiency. We obtained all relevant information from vendors directly, through vendor websites, online press releases, and online articles.
Results
Inclusion and exclusion of products based on inclusion criteria
[1] 240The PRISMA diagram is shown in Figure Figure 1.
Exclude reason
NO MH MEDICAL DEVICE NOT AI SOFTWARE NOT_CDSS
44 2 15 Product overview
The product search resulted in three products that were cleared by the FDA, five that were CE marked, and one that was UKCA marked. One company offered two products. Most products were cloud-based stand-alone third party applications. For two tools, information about integration possibilities was missing and one tool was compatible with leading polysomnography devices.
Figure 2 displays the disorders targeted by the products.
Figure 3 shows an overview of the data sources used to derive recommendations.
Figure 4 shows an overview of the outputs generated by the products.
Figure 5 displays the type of output generated by the mental disorder targeted.
Figure 6 displays the time passed between company foundation, product certification, and the publication of scientific evidence addressing product efficiency. Please note that we excluded the foundation date of Medibio (MEBSLEEP), which was founded in 1987.
Figure 7 displays the certification the products received. De Novo is a regulatory pathway for marketing approval of novel medical devices in the US that do not have a predicate device for comparison. It is used for low to moderate risk devices that are not substantially equivalent to any existing device, enabling them to enter the market without requiring the traditional pre-market approval process. 510(k) is a pre-market submission to the FDA demonstrating that a medical device to be marketed is at least as safe and effective, or “substantially equivalent,” to a legally marketed device (predicate device) that is not subject to pre-market approval. Predicate devices serve as a benchmark for comparative purpose allowing more rapid approval of new devices. They help provide assurance for safety and effectiveness of the new device through comparison. The predicate device for the EarliPoint System is the Cognoa ASD Diagnosis Aid and the predicate device of EnsoSleep is a similar previous version of EnsoSleep.
Evidence
Discussion
Summary of Key Findings
The aim of this study was to increase transparency in the field of commercial AI-CDSS software in mental healthcare and its scientific evidence. Our results demonstrate that the market is still in its very infancy. We could identify only nine products that obtained FDA clearance or CE marking. The fact that all of the products received regulatory approval in the past three years (November 2020 - January 2023) may be an explanation for this. That is, despite the large number of solutions having been developed to aid clinicians in deriving diagnoses and selecting appropriate treatment approaches (Aafjes-van Doorn et al., 2021; Chekroud et al., 2021; Cho et al., 2019; Graham et al., 2019; Shatte et al., 2019), for most products we still await to discover the impact they may have in clinical practice.
- discussion of scientific evidence
- discussion of device type by mental disorder
- discussion of time passed between company founding, product certification, publication of scientific evidence
Implications for Current AI Landscape in Mental Healthcare
- What does the presence of regulated products mean for start-ups and small firms?
- Do the regulated products shed light on effective strategies for overcoming regulatory and data-related challenges?
- What could other firms glean from those successful products?
Intersection of Regulatory Practices and AI-CDSS Development
- Reflect on the role of FDA clearance, CE marking, de novo regulations, and 510(k) benchmarking in the development of AI-CDSS
- How do these regulatory practices impact the AI development process?
- How do they shape the adoption and implementation of AI applications in mental healthcare?
- Discussion of AI-CDSS for mental healthcare that are unregulated - reasons for lack of regulation (documented for each single product)
- Discussion of options to get around regulation (Wellness app, research use only, “selling a platform”)
- Discussion of currently available unregulated products (e.g., treatment apps)
Necessary Adaptations of Regulatory Frameworks for the Regulation of AI-CDSS medical devices in Mental Health
- what needs to be done to make the regulatory process more efficient?
Study Limitations and Future Directions
- Review limitations
- Sources of bias
- Unavailability of data
- Lack of responses from vendors
- What products or companies may have been excluded and why?
- Future directions for regulated AI-CDSS for mental health (which areas, how approach the regulation process)