Relatively speaking: contrast effects influence assessors’ scores and narrative feedback

Yeates, Peter; Cardell, Jenna; Byrne, Gerard; Eva, Kevin W

Abstract

Context
In prior research, the scores assessors assign can be biased away from the standard of preceding performances (i.e. ‘contrast effects’ occur).

Objectives
This study examines the mechanism and robustness of these findings to advance understanding of assessor cognition. We test the influence of the immediately preceding performance relative to that of a series of prior performances. Further, we examine whether assessors’ narrative comments are similarly influenced by contrast effects.

Methods
Clinicians (n = 61) were randomised to three groups in a blinded, Internet-based experiment. Participants viewed identical videos of good, borderline and poor performances by first-year doctors in varied orders. They provided scores and written feedback after each video. Narrative comments were blindly content-analysed to generate measures of valence and content. Variability of narrative comments and scores was compared between groups.

Results
Comparisons indicated contrast effects after a single performance. When a good performance was preceded by a poor performance, ratings were higher (mean 5.01, 95% confidence interval [CI] 4.79–5.24) than when observation of the good performance was unbiased (mean 4.36, 95% CI 4.14–4.60; p < 0.05, d = 1.3). Similarly, borderline performance was rated lower when preceded by good performance (mean 2.96, 95% CI 2.56–3.37) than when viewed without preceding bias (mean 3.55, 95% CI 3.17–3.92; p < 0.05, d = 0.7). The series of ratings participants assigned suggested that the magnitude of contrast effects is determined by an averaging of recent experiences. The valence (but not content) of narrative comments showed contrast effects similar to those found in numerical scores.

Conclusions
These findings are consistent with research from behavioural economics and psychology that suggests judgement tends to be relative in nature. Observing that the valence of narrative comments is similarly influenced suggests these effects represent more than difficulty in translating impressions into a number. The extent to which such factors impact upon assessment in practice remains to be determined as the influence is likely to depend on context.

Citation

Yeates, P., Cardell, J., Byrne, G., & Eva, K. W. (2015). Relatively speaking: contrast effects influence assessors’ scores and narrative feedback. Medical Education, 49(9), 909-919. https://doi.org/10.1111/medu.12777

• Journal Article Type: Article
• Acceptance Date: Apr 27, 2015
• Online Publication Date: Aug 22, 2015
• Publication Date: 2015-09
• Deposit Date: Nov 28, 2023
• Journal: Medical Education
• Print ISSN: 0308-0110
• Publisher: Wiley
• Peer Reviewed: Peer Reviewed
• Volume: 49
• Issue: 9
• Pages: 909-919
• DOI: https://doi.org/10.1111/medu.12777
• Keywords: Education; General Medicine