Improving event prediction using general practitioner clinical judgement in a digital risk stratification model: a pilot study

Parry, Emma; Ahmed, Kamran; Guest, Elizabeth; Klaire, Vijay; Koodaruth, Abdool; Labutale, Prasadika; Matthews, Dawn; Lampitt, Jonathan; Nevill, Alan; Pickavance, Gillian; Sidhu, Mona; Warren, Kate; Singh, Baldev M.

Abstract

Background: Numerous tools based on electronic health record (EHR) data that predict risk of unscheduled care and mortality exist. These are often criticised due to lack of external validation, potential for low predictive ability and the use of thresholds that can lead to large numbers being escalated for assessment that would not have an adverse outcome leading to unsuccessful active case management. Evidence supports the importance of clinical judgement in risk prediction particularly when ruling out disease. The aim of this pilot study was to explore performance analysis of a digitally driven risk stratification model combined with GP clinical judgement to identify patients with escalating urgent care and mortality events. Methods: Clinically risk stratified cohort study of 6 GP practices in a deprived, multi-ethnic UK city. Initial digital driven risk stratification into Escalated and Non-escalated groups used 7 risk factors. The Escalated group underwent stratification using GP global clinical judgement (GCJ) into Concern and No concern groupings. Results: 3968 out of 31,392 patients were data stratified into the Escalated group and further categorised into No concern (n = 3450 (10.9%)) or Concern (n = 518 (1.7%)) by GPs. The 30-day combined event rate (unscheduled care or death) per 1,000 was 19.0 in the whole population, 67.8 in the Escalated group and 168.0 in the Concern group (p < 0.001). The de-escalation effect of GP assessment into No Concern versus Concern was strongly negatively predictive (OR 0.25 (95%CI 0.19–0.33; p < 0.001)). The whole population ROC for the global approach (Non-escalated, GP No Concern, GP Concern) was 0.614 (0.592—0.637), p < 0.001, and the increase in the ROC area under the curve for 30-day events was all focused here (+ 0.4% (0.3–0.6%, p < 0.001), translating into a specific ROC c-statistic for GP GCJ of 0.603 ((0.565—0.642), p < 0.001). Conclusions: The digital only component of the model performed well but adding GP clinical judgement significantly improved risk prediction, particularly by adding negative predictive value.

Citation

Parry, E., Ahmed, K., Guest, E., Klaire, V., Koodaruth, A., Labutale, P., Matthews, D., Lampitt, J., Nevill, A., Pickavance, G., Sidhu, M., Warren, K., & Singh, B. M. (2024). Improving event prediction using general practitioner clinical judgement in a digital risk stratification model: a pilot study. BMC Medical Informatics and Decision Making, 24(1), Article 382. https://doi.org/10.1186/s12911-024-02797-5

• Journal Article Type: Article
• Acceptance Date: Nov 29, 2024
• Online Publication Date: Dec 18, 2024
• Publication Date: Dec 18, 2024
• Deposit Date: Dec 19, 2024
• Publicly Available Date: Jan 8, 2025
• Journal: BMC Medical Informatics and Decision Making
• Electronic ISSN: 1472-6947
• Publisher: Springer Verlag
• Peer Reviewed: Peer Reviewed
• Volume: 24
• Issue: 1
• Article Number: 382
• DOI: https://doi.org/10.1186/s12911-024-02797-5
• Keywords: Urgent care, Global clinical judgement, General practitioner, Mortality, Risk prediction
• Public URL: https://keele-repository.worktribe.com/output/1019635
• Additional Information: Received: 7 December 2023; Accepted: 29 November 2024; First Online: 18 December 2024; : ; : Ethics approval and informed consent was not deemed necessary for this study according to our Institutional Review Board (Royal Wolverhampton NHS Trust Research and Development Department).The systems were designed and the data accrued for a wider programme relating to service reconfiguration in our local health economy and as such the GP assessments were part of routine care.No selection or randomisation was applied, interventions were part of indicated clinical care and thus research ethical approval was not deemed necessary as confirmed within local governance processes. All methods were carried out in accordance with relevant guidelines and regulations and in accordance with the Declaration of Helsinki.; : Not applicable.; : The authors declare no competing interests.