Inter-school variations in the standard of examiners’ graduation-level OSCE judgements

Yeates, Peter; Maluf, Adriano; McCray, Gareth; Kinston, Ruth; Cope, Natalie; Cullen, Kathy; O’Neill, Vikki; Cole, Aidan; Chung, Ching-wa; Goodfellow, Rhian; Vallender, Rebecca; Ensaff, Sue; Goddard-Fuller, Rikki; McKinley, Robert

Abstract

Introduction
Ensuring equivalence in high-stakes performance exams is important for patient safety and candidate fairness. We compared inter-school examiner differences within a shared OSCE and resulting impact on students’ pass/fail categorisation.

Methods
The same 6 station formative OSCE ran asynchronously in 4 medical schools, with 2 parallel circuits/school. We compared examiners’ judgements using Video-based Examiner Score Comparison and Adjustment (VESCA): examiners scored station-specific comparator videos in addition to ‘live’ student performances, enabling 1/controlled score comparisons by a/examiner-cohorts and b/schools and 2/data linkage to adjust for the influence of examiner-cohorts. We calculated score impact and change in pass/fail categorisation by school.

Results
On controlled video-based comparisons, inter-school variations in examiners’ scoring (16.3%) were nearly double within-school variations (8.8%). Students’ scores received a median adjustment of 5.26% (IQR 2.87–7.17%). The impact of adjusting for examiner differences on students’ pass/fail categorisation varied by school, with adjustment reducing failure rate from 39.13% to 8.70% (school 2) whilst increasing failure from 0.00% to 21.74% (school 4).

Discussion
Whilst the formative context may partly account for differences, these findings query whether variations may exist between medical schools in examiners’ judgements. This may benefit from systematic appraisal to safeguard equivalence. VESCA provided a viable method for comparisons.

Citation

Yeates, P., Maluf, A., McCray, G., Kinston, R., Cope, N., Cullen, K., …McKinley, R. (2024). Inter-school variations in the standard of examiners’ graduation-level OSCE judgements. Medical Teacher, https://doi.org/10.1080/0142159X.2024.2372087

• Journal Article Type: Article
• Acceptance Date: Jun 20, 2024
• Online Publication Date: Jul 8, 2024
• Publication Date: Jul 8, 2024
• Deposit Date: Jul 15, 2024
• Journal: Medical Teacher
• Print ISSN: 0142-159X
• Electronic ISSN: 1466-187X
• Publisher: Taylor and Francis
• Peer Reviewed: Peer Reviewed
• DOI: https://doi.org/10.1080/0142159X.2024.2372087
• Keywords: Assessment; equivalence; medical education; OSCE
• Public URL: https://keele-repository.worktribe.com/output/875667