Parameter-Efficient Personalized Speech Synthesis via EMD-based Speaker Modeling

Personalized speech synthesis has attracted increasing attention in the field of robotics. Compared to traditional speech synthesis, it faces two primary challenges: the limited availability of adaptation data and the necessity for highly efficient adaptation method using compact parameters to reduce both training time and memory consumption. To address both the challenges, this paper proposes a personalized speech synthesis approach that incorporates an Empirical Mode Decomposition (EMD)-based speaker modeling method alongside a novel decoder structure with masked inputs, which improves the model's ability to extract speaker-specific features accurately. Furthermore, we introduce a parameter-efficient fine-tuning technique, Attention-based Speaker-Text Scaling and Shifting Feature (AST-SSF), to enhance adaptation efficiency. We validate our approach using the MAGICDATA Corpus. The results indicate that our proposed approach outperforms the baseline in both naturalness and similarity, demonstrating its effectiveness. Moreover, although the proposed adaptation method substantially reduces the number of parameters, it exhibits only minimal performance degradation compared to full and partial fine-tuning strategies.