HGOT-SER: Hybrid genetic optimization for multi-granularity Transformer-based speech emotion recognition

Speech emotion recognition (SER) has wide applications in many real-world areas such as customer service, healthcare, and intelligent agents. Recently, Transformer-based models have been used in SER. However, training these models predominantly relies on gradient-based optimization methods, which may become trapped in local optima, affecting recognition accuracy. To address this issue, we propose a hybrid genetic optimization model for multi-granularity Transformer-based speech emotion recognition (HGOT-SER). HGOT-SER employs a gradient-genetic synergistic optimization mechanism, including (i) a stochastic gradient descent for rapid convergence to a near-optimal region, and then (ii) a genetic algorithm with an adaptive cyclic mutation strategy to fine-tune parameters and escape the local optima. To enhance emotional feature extraction, HGOT-SER introduces a multi-granularity Transformer as its backbone, which extracts acoustic features through parallel temporal windows of different sizes, enabling more effective hierarchical temporal modeling. Experiments on the IEMOCAP, CASIA, EMODB, and RAVDESS datasets show that HGOT-SER outperforms the multi-granularity Transformer, achieving improvements exceeding 1.2% in weighted accuracy and 1.1% in unweighted accuracy on all datasets. HGOT-SER also outperforms state-of-the-art methods on the CASIA and EMODB datasets. These results demonstrate the effectiveness and robustness of HGOT-SER on multiple SER datasets.