Peter Yeates p.yeates@keele.ac.uk
Exploring differences in individual and group judgements in standard setting
Yeates, P; Cope, N; Luksaite, E; Hassell, A; Dikomitis, L
Authors
Natalie Cope n.a.cope@keele.ac.uk
Dr Eva Fiks e.fiks@keele.ac.uk
Andrew Hassell a.hassell@keele.ac.uk
L Dikomitis
Abstract
Background: Standard setting is critically important to assessment decisions in medical education. Recent research has demonstrated variations between medical schools in the standards set for shared items. Despite the centrality of judgement to criterion referenced standard setting methods, little is known about the individual or group processes which underpin them. This study aimed to explore the operation and interaction of these processes in order to illuminate potential sources of variability.
Methods: Using qualitative research, we purposively sampled across UK medical schools who set either a low, medium, or high standard on nationally-shared items, collecting data by observation of graduation-level standard setting meetings and semi-structured interviews with standard setting judges. Data were analysed using thematic analysis based on the principles of grounded theory.
Results: Standard setting occurred through the complex interaction of: institutional context; judges’ individual perspectives; and group interactions. Schools’ procedures, panel members and atmosphere produced unique contexts. Individual judges formed varied understandings of the clinical and technical features of each question, relating these to their differing (sometimes contradictory) conceptions of minimally-competent students, by balancing information and making suppositions. Conceptions of minimal competence variously comprised: limited attendance, limited knowledge, poor knowledge application, emotional responses to questions, “test-savviness”, or a strategic focus on safety. Judges experienced tensions trying to situate these abstract conceptions in reality, revealing uncertainty. Groups constructively revised scores through debate, sharing information, often constructing detailed clinical representations of cases. Groups frequently displayed conformity, illustrating a belief that outlying judges were likely to be incorrect. Less frequently judges resisted change, using emphatic language, bargaining or rarely “polarization” to influence colleagues.
Conclusions: Despite careful conduct through well-established procedures, standard setting is judgementally complex and involves uncertainty. Understanding whether or how these varied processes produce the previously observed variations in outcomes may offer routes to enhance equivalence of criterion-referenced standards.
Citation
Yeates, P., Cope, N., Luksaite, E., Hassell, A., & Dikomitis, L. (2019). Exploring differences in individual and group judgements in standard setting. Medical Education, https://doi.org/10.1111/medu.13915
Journal Article Type | Article |
---|---|
Acceptance Date | Apr 25, 2019 |
Publication Date | Jul 2, 2019 |
Journal | Medical Education |
Print ISSN | 0308-0110 |
Publisher | Wiley |
DOI | https://doi.org/10.1111/medu.13915 |
Publisher URL | https://doi.org/10.1111/medu.13915 |
Files
How absolutes vary accepted manuscript.docx
(102 Kb)
Document
Publisher Licence URL
https://creativecommons.org/licenses/by-nc/4.0/
You might also like
Downloadable Citations
About Keele Repository
Administrator e-mail: research.openaccess@keele.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search