Emotion Classification on Software Engineering Q&A Websites

Awovi Ahavi-Tete, Didi; Sangeeta, Sangeeta

Abstract

Background: With the rapid proliferation of question-and-answer websites for software developers like Stack Overflow, there is an increasing need to discern developers’ emotions from their posts to assess the influence of these emotions on their productivity such as efficiency in bug fixing.

Aim: We aimed to develop a reliable emotion classification tool capable of accurately categorizing emotions in Software Engineering (SE) websites using data augmentation techniques to address the data scarcity problem because previous research has shown that tools trained on other domains can perform poorly when applied to SE domain directly.

Method: We utilized four machine learning techniques, namely BERT, CodeBERT, RFC (Random Forest Classifier), and LSTM. Taking an innovative approach to dataset augmentation, we employed word substitution, back translation, and easy data augmentation methods. Using these we developed sixteen unique emotion classification models: EmoClassBERT–Original, EmoClassRFC-Original, EmoClassLSTMOriginal, EmoClass-CodeBERT-Original EmoClassLSTM-Substitution, EmoClassBERT-Substitution, EmoClassRFC-Substitution, EmoClassCodeBERT-Substitution, Emo-ClassBERT-Translation, EmoClassLSTM-Translation, EmoClassRFC-Translation, EmoClassCodeBERT-Translation, EmoClassBERT-EDA, EmoClass-LSTM-EDA, EmoClassCodeBERT-EDA, and EmoClassRFC-EDA. We compared the performance of this model on a gold standard state-of-the-art database and techniques (Multi-label SO BERT and EmoTxt).

Results: An initial investigation of models trained on the augmented datasets demonstrated superior performance to those trained on the original dataset. EmoClassLSTM-Substitution, EmoClassBERT-Substitution, EmoClassCodeBERT-Substitution, and EmoClassRFC-Substitution models show improvements of 13%, 5%, 5%, and 10% as compared to EmoClass-LSTM-Original, EmoClassBERT-Original, EmoClassCodeBERT-Original, and EmoClassRFC-Original, respectively, in average F1 score. The Emo-ClassCodeBERT-Substitution performed the best and outperformed the Multi-label SO BERT and Emotxt by 2.37% and 21.17%, respectively, in average F1-score. A detailed investigation of the models on 100 runs of the dataset shows that BERT-based and CodeBERT-based models gave the best performance. This detailed investigation reveals no significant differences in the performance of models trained on augmented datasets and the original dataset on multiple runs of the dataset.

Conclusion: This research not only underlines the strengths and weaknesses of each architecture but also highlights the pivotal role of data augmentation in refining model performance, especially in the software engineering domain.

Citation

Awovi Ahavi-Tete, D., & Sangeeta, S. (2025). Emotion Classification on Software Engineering Q&A Websites. e-Informatica Software Engineering Journal (EISEJ), 19(1), Article 250104. https://doi.org/10.37190/e-inf250104

• Journal Article Type: Article
• Acceptance Date: Oct 6, 2024
• Online Publication Date: Nov 19, 2024
• Publication Date: 2025
• Deposit Date: Jan 13, 2025
• Journal: e-Informatica Software Engineering Journal
• Print ISSN: 1897-7979
• Publisher: Software Engineering Section of the Committee on Informatics of the Polish Academy of Sciences
• Peer Reviewed: Peer Reviewed
• Volume: 19
• Issue: 1
• Article Number: 250104
• DOI: https://doi.org/10.37190/e-inf250104
• Keywords: empirical and experimental studies in software engineering, data mining in software engineering, prediction models in software engineering, AI and knowledge based software engineering
• Public URL: https://keele-repository.worktribe.com/output/1021804
• Publisher URL: https://www.e-informatyka.pl/EISEJ/papers/2025/1/4/