Problems with Statistical Practice in Software Engineering Research

Kitchenham, Barbara; Madeyski, Lech; Brereton, Pearl

Abstract

Background
Examples of questionable statistical practice, when published in high quality software engineering (SE) journals, may lead to novice researchers adopting incorrect statistical practices.

Objective
Our goal is to highlight issues contributing to poor statistical practice in human-centric SE experiments.

Method
We reviewed the statistical analysis practices used in the 13 papers that reported families of human-centric SE experiments and were published in high quality journals.

Results
Reviewed papers related to 45 experiments and involved a total of 1303 human participants. We searched for issues that were related to questionable statistical practice that were found in more than one paper. We observed three types of bad practice: incorrect use of terminology, incorrect analysis of repeated measures designs, and post-hoc power testing. We also found two analysis practices (i.e., multiple testing and pre-testing for normality) where statisticians disagree about good practice.

Conclusions
Identified issues pose a problem because readers may expect the statistical methods used in papers published in
top quality, peer-reviewed journals to be correct. We explain why the practices are problematic and provide recommendations for improved practice.

Citation

Kitchenham, B., Madeyski, L., & Brereton, P. (2019). Problems with Statistical Practice in Software Engineering Research. In EASE '19 Proceedings of the Evaluation and Assessment on Software Engineering (134-143). https://doi.org/10.1145/3319008.3319009

• Conference Name: EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering
• Conference Location: Copenhagen, Denmark
• Start Date: Apr 15, 2019
• End Date: Apr 17, 2019
• Acceptance Date: Feb 28, 2019
• Online Publication Date: Apr 15, 2019
• Publication Date: Feb 28, 2019
• Publicly Available Date: May 26, 2023
• Pages: 134-143
• Series Title: Evaluation and Assessment in Software Engineering
• Book Title: EASE '19 Proceedings of the Evaluation and Assessment on Software Engineering
• ISBN: 978-1-4503-7145-2
• DOI: https://doi.org/10.1145/3319008.3319009
• Publisher URL: http://doi.org/10.1145/3319008.3319009