Usability of an artificial intelligence clinical decision support system for trauma: A pilot study

Abstract text (incl. references and figure legends)

Introduction

Trauma-induced coagulopathy (TIC) affects 25% of severely injured patients, and mortality approaches 50%. An Artificial Intelligence (AI) model was developed and validated to predict TIC, and integrated into a clinical decision support system (CDSS). This pilot study aimed to evaluate our usability study design, determining the usability, usefulness, and potential barriers to adoption of our AI CDSS.

Material & Methods

This was a prospective, mixed-methods, pilot usability study. Participants (n=5) tested the CDSS by inputting information into its interface, generating a risk prediction output (Fig 1). Usability issues were captured using "think aloud" methodology, categorised using Nielsen"s Heuristics (NH), and graded for severity (1=cosmetic, 2=minor, 3=major, 4=catastrophic). A validated questionnaire was administered based on the System Usability Scale (SUS), Technology Acceptance Model (TAM), and nonadoption, abandonment, scale-up, sustainability and spread (NASSS) adoption framework.

Results

Participants identified 18 study design problems: 13 user testing, 5 questionnaire-related. Usability issues were identified in 9/10 NH categories (Fig 2): 40 input- and 11 output-related (average severity 2.1/4 and 1.8/4, respectively). The most common issue was "consistency and standards" (n=14). The CDSS achieved an SUS score of 60 (standard is ≥68). Participants said the CDSS would be useful in their clinical practice (60%), improve their clinical performance (60%), and enhance their clinical effectiveness (40%). Potential barriers were related to the adopters, organisation and external environment.

Conclusions

This pilot study identified issues with study design, and usability of the AI CDSS. After iterative improvements, a prospective study will evaluate the usability, clinician trust, and impact on decision-making of an updated AI CDSS in trauma.

Figure 1. CDSS user interface

Figure 2. NH usability issues

Disclosure: Do you have a significant financial interest, consultancy or other relationship with products, manufacturer(s) of products or providers of services related to this abstract? (If not, please enter "No" in the text field.)

JMW, RSS, EP, EK, WM, ZP, and NT have received research funding from the United States Department of Defense. RSS is also funded by the Royal College of Surgeons of Edinburgh and Orthopaedic Research UK. JMW has received funding from the Royal College of Surgeons of England. For the remaining authors none were declared.