Skip to main content

Research Repository

Advanced Search

Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites in a distributed objective structured clinical exam (OSCE)

Yeates, Peter; Maluf, Adriano; Cope, Natalie; McCray, Gareth; McBain, Stuart; Beardow, Dominic; Fuller, Richard; McKinley, Robert Bob

Authors

Adriano Maluf

Stuart McBain

Dominic Beardow

Richard Fuller

Robert Bob McKinley



Abstract

Purpose: Ensuring equivalence of examiners’ judgements within distributed objective structured clinical exams (OSCEs) is key to both fairness and validity but is hampered by lack of cross-over in the performances which different groups of examiners observe. This study develops a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA) using it to compare examiners scoring from different OSCE sites for the first time. Materials/ methods: Within a summative 16 station OSCE, volunteer students were videoed on each station and all examiners invited to score station-specific comparator videos in addition to usual student scoring. Linkage provided through the video-scores enabled use of Many Facet Rasch Modelling (MFRM) to compare 1/ examiner-cohort and 2/ site effects on students’ scores. Results: Examiner-cohorts varied by 6.9% in the overall score allocated to students of the same ability. Whilst only a tiny difference was apparent between sites, examiner-cohort variability was greater in one site than the other. Adjusting student scores produced a median change in rank position of 6 places (0.48 deciles), however 26.9% of students changed their rank position by at least 1 decile. By contrast, only 1 student’s pass/fail classification was altered by score adjustment. Conclusions: Whilst comparatively limited examiner participation rates may limit interpretation of score adjustment in this instance, this study demonstrates the feasibility of using VESCA for quality assurance purposes in large scale distributed OSCEs.

Journal Article Type Article
Acceptance Date Oct 14, 2023
Online Publication Date Oct 26, 2023
Deposit Date Oct 30, 2023
Publicly Available Date Oct 30, 2023
Journal BMC Medical Education
Publisher Springer Verlag
Peer Reviewed Peer Reviewed
Volume 23
Issue 1
Pages 803
DOI https://doi.org/10.1186/s12909-023-04774-4
Keywords Examiner-Cohorts, Equivalence, Assessment, Distributed Assessment, OSCE

Files

Yeates_published version (1.5 Mb)
PDF

Publisher Licence URL
https://creativecommons.org/licenses/by/4.0/

Copyright Statement
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.




You might also like



Downloadable Citations