The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision

Cresswell, Kathrin; Rigby, Michael; Magrabi, Farah; Scott, Philip; Brender, Jytte; Craven, Catherine K.; Wong, Zoie Shui-Yee; Kukhareva, Polina; Ammenwerth, Elske; Georgiou, Andrew; Medlock, Stephanie; De Keizer, Nicolette F.; Nykänen, Pirkko; Prgomet, Mirela; Williams, Robin

Abstract

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology.

We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes.

Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

Citation

Cresswell, K., Rigby, M., Magrabi, F., Scott, P., Brender, J., Craven, C. K., …Williams, R. (2023). The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision. Health Policy, 136, Article 104889. https://doi.org/10.1016/j.healthpol.2023.104889

• Journal Article Type: Article
• Acceptance Date: Aug 4, 2023
• Online Publication Date: Aug 12, 2023
• Publication Date: 2023-10
• Deposit Date: Jan 24, 2024
• Journal: Health Policy
• Print ISSN: 0168-8510
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 136
• Article Number: 104889
• DOI: https://doi.org/10.1016/j.healthpol.2023.104889
• Keywords: Health Policy
• Publisher URL: https://www.sciencedirect.com/science/article/pii/S0168851023001744?via%3Dihub
• Additional Information: This article is maintained by: Elsevier; Article Title: The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision; Journal Title: Health Policy; CrossRef DOI link to publisher maintained version: https://doi.org/10.1016/j.healthpol.2023.104889; Content Type: article; Copyright: © 2023 The Author(s). Published by Elsevier B.V.