Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites in a distributed objective structured clinical exam (OSCE)

Yeates, Peter; Maluf, Adriano; Cope, Natalie; McCray, Gareth; McBain, Stuart; Beardow, Dominic; Fuller, Richard; McKinley, Robert Bob

Abstract

Purpose: Ensuring equivalence of examiners’ judgements within distributed objective structured clinical exams (OSCEs) is key to both fairness and validity but is hampered by lack of cross-over in the performances which different groups of examiners observe. This study develops a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA) using it to compare examiners scoring from different OSCE sites for the first time. Materials/ methods: Within a summative 16 station OSCE, volunteer students were videoed on each station and all examiners invited to score station-specific comparator videos in addition to usual student scoring. Linkage provided through the video-scores enabled use of Many Facet Rasch Modelling (MFRM) to compare 1/ examiner-cohort and 2/ site effects on students’ scores. Results: Examiner-cohorts varied by 6.9% in the overall score allocated to students of the same ability. Whilst only a tiny difference was apparent between sites, examiner-cohort variability was greater in one site than the other. Adjusting student scores produced a median change in rank position of 6 places (0.48 deciles), however 26.9% of students changed their rank position by at least 1 decile. By contrast, only 1 student’s pass/fail classification was altered by score adjustment. Conclusions: Whilst comparatively limited examiner participation rates may limit interpretation of score adjustment in this instance, this study demonstrates the feasibility of using VESCA for quality assurance purposes in large scale distributed OSCEs.

Citation

Yeates, P., Maluf, A., Cope, N., McCray, G., McBain, S., Beardow, D., …McKinley, R. B. (2023). Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites in a distributed objective structured clinical exam (OSCE). BMC Medical Education, 23(1), Article 803. https://doi.org/10.1186/s12909-023-04774-4

• Journal Article Type: Article
• Acceptance Date: Oct 14, 2023
• Online Publication Date: Oct 26, 2023
• Publication Date: Oct 26, 2023
• Deposit Date: Oct 30, 2023
• Publicly Available Date: Oct 30, 2023
• Journal: BMC Medical Education
• Publisher: Springer Verlag
• Peer Reviewed: Peer Reviewed
• Volume: 23
• Issue: 1
• Article Number: 803
• DOI: https://doi.org/10.1186/s12909-023-04774-4
• Keywords: Examiner-Cohorts, Equivalence, Assessment, Distributed Assessment, OSCE
• Public URL: https://keele-repository.worktribe.com/output/613675