FabasedVC: Voice Conversion with Text Modality Fusion and Phoneme-Level SSL Feature

Contents

• Abstract
• Features introduction
• Demos -- VC Experiment
• Demos -- Duration Experiment
• Demos -- Ablation

1. Abstract

In voice conversion (VC), it is crucial to preserve complete semantic information while accurately modeling the target speaker's timbre and prosody.

This paper proposes FabasedVC to achieve VC with enhanced similarity in timbre, prosody, and duration to the target speaker, as well as improved content integrity. It is an end-to-end VITS-based VC system that integrates relevant textual information, phoneme-level self-supervised learning (SSL) features, and a duration predictor.

Specifically, we employ a textual feature encoder to encode attributes such as text, phonemes, and tones. We then process the frame-level SSL features into phoneme-level features using two methods: average pooling based on each phoneme's duration and an attention mechanism. Moreover, a duration predictor is incorporated to better align the speech rate and prosody of the target speaker.

Experimental results demonstrate that our method outperforms competing systems in terms of naturalness, similarity, and content integrity.

2. Features introduction

Words, Phonemes, and Tones are obtained through ASR models and related annotation tools, representing the textual content, phonetic information, and tonal characteristics of speech, respectively. Tonal information refers to the pitch variation patterns of a syllable during pronunciation; since Chinese is a tonal language, this feature is not applicable when processing non-tonal languages such as English. Some speech voice synthesis systems employ BERT features of text to enhance semantic information. Similarly, we derive WordsBERT features from the Words feature using BERT to further enrich the information.

A forced alignment annotation model can generate timestamps at the phoneme level of speech. By leveraging existing forced alignment annotation models and integrating them with speech data, as well as the Words and Phones features, we can extract duration information for each element of Words and Phonemes in the speech. By aligning this duration information with the frames of the spectrogram, we obtain Pframe and Wframe features, which represent the frame lengths corresponding to each character and character, respectively. The W2P feature indicates the number of phonemes associated with each character in the Words. The Speaker feature conveys speaker-related information linked to the speech, while the Contentvec feature refers to the disentangled SSL features extracted from the speech.

Due to certain limitations, we will not be demonstrating the WordsBERT and Contentvec features here; both are represented as Tensor vectors.

3. Demos -- VC Experiment

1. aishell to aishell

2. databaker to aishell

4. Demos -- Duration Experiment

The average phoneme duration (phones/s) of the source and target speaker's speech before and after conversion.

Please note that the demo showcases sampled data from the experiment, while the average phoneme duration is the mean value obtained from all audio data throughout the entire experiment.

5. Demos -- Ablation