A novel attention model across heterogeneous features for stuttering event detection

Al-Banna, Abedal-Kareem; Fang, Hui; Edirisinghe, Eran

Abstract

Stuttering is a prevalent speech disorder affecting millions worldwide. To provide an automatic and objective stuttering assessment tool, Stuttering Event Detection (SED) is under extensive investigation for advanced speech research and applications. Despite significant progress achieved by various machine learning and deep learning models, SED directly from speech signal is still challenging due to stuttering speech's heterogeneous and overlapped nature. This paper presents a novel SED approach using multi-feature fusion and attention mechanisms. The model utilises multiple acoustic features extracted based on different pitch, time-domain, frequency domain, and automatic speech recognition feature to detect stuttering core behaviours more accurately and reliably. In addition, we exploit both spatial and temporal attention mechanisms as well as Bidirectional Long Short-Term Memory (BI-LSTM) modules to learn better representations to improve the SED performance. The experimental evaluation and analysis convincingly demonstrate that our proposed model surpasses the state-of-the-art models on two popular stuttering datasets, with 4% and 3% overall F1 scores, respectively. The superior results indicate the consistency of our proposed method, supported by both multi-feature and attention mechanisms in different stuttering events datasets.

• Journal Article Type: Article
• Acceptance Date: Dec 13, 2023
• Online Publication Date: Dec 28, 2023
• Publication Date: Jun 15, 2024
• Deposit Date: Mar 12, 2024
• Journal: Expert Systems with Applications
• Print ISSN: 0957-4174
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 244
• Article Number: 122967
• DOI: https://doi.org/10.1016/j.eswa.2023.122967
• Keywords: Stuttering; Stuttering events detection; Multi-feature attention model; Stuttering severity systems; Dysfluency; Deep learning
• Additional Information: This article is maintained by: Elsevier; Article Title: A novel attention model across heterogeneous features for stuttering event detection; Journal Title: Expert Systems with Applications; CrossRef DOI link to publisher maintained version: https://doi.org/10.1016/j.eswa.2023.122967; Content Type: article; Copyright: © 2023 The Author(s). Published by Elsevier Ltd.