Video-based benchmarking: exploring an intervention that aims to increase examiner alignment in objective structured clinical exams (OSCEs)

Edwards, Rebecca Jane

Abstract

Introduction
Examiner variability poses a challenge to OSCE fairness, with existing training methods shown as largely ineffective. Assessors, however, are influenced by recent experiences. Thus, I hypothesised that a targeted, near-in-time intervention could improve examiner alignment. Video-Based Benchmarking (VBB) is a novel approach aimed at reducing examiner variability in OSCEs. It involves examiners evaluating a station-specific OSCE performance and receiving feedback shortly before scoring the target station, enabling mental calibration and adjustment to expectations.

Methods
Study 1 (S1): Using realist evaluation, I explored how, why, and under what conditions VBB influences scoring and its potential development. An initial program theory was formulated via literature review and tested through 16 interviews with OSCE examiners. Interviews were analysed using Context-Mechanism-Outcome configurations (CMOc’s) to test and modify this theory and develop the intervention. Study 2 (S2): This internet-based experiment tested whether VBB reduced score variability among OSCE examiners. The intervention group underwent VBB before assessing two target performances. The control group assessed the same performances without VBB and later watched the same benchmark video without feedback. Group assignment and video order were randomized. Data were analysed using a Generalized Linear Model (GLiM).

Findings
S1 indicated that viewing a station-specific OSCE performance close to scoring could reduce variability by refining examiners’ internal marking framework, minimizing the need to “get their eye in,” and providing a direct performance comparator. In contrast, S2’s primary analysis found no significant impact of VBB on examiner variability. Exploratory findings suggested that VBB may benefit examiners with greater initial variance from the group mean.

Conclusions
These results imply that additional refinements to VBB might better resolve discrepancies, or that VBB may improve perceived confidence without actual alignment. While VBB shows promise in influencing examiner judgments, its effect on score variability requires further research and procedural refinement to establish its efficacy.

Citation

Edwards, R. J. (2025). Video-based benchmarking: exploring an intervention that aims to increase examiner alignment in objective structured clinical exams (OSCEs). (Thesis). Keele University. https://keele-repository.worktribe.com/output/1279929

• Thesis Type: Thesis
• Online Publication Date: Jun 26, 2025
• Deposit Date: Jun 17, 2025
• Publicly Available Date: Jun 27, 2025
• Public URL: https://keele-repository.worktribe.com/output/1279929
• Award Date: 2025-06