VALL-E X: Multilingual Text-to-Speech Synthesis and Voice Cloning 🔊


VALL-E X is an amazing multilingual text-to-speech (TTS) model proposed by Microsoft. While Microsoft initially publish in their research paper, they did not release any code or pretrained models. Recognizing the potential and value of this technology, our team took on the challenge to reproduce the results and train our own model. We are glad to share our trained VALL-E X model with the community, allowing everyone to experience the power next-generation TTS! 🎧

More details about the model are presented in model card.

📖 Quick Index

• 🚀 Updates
• 📢 Features
• 💻 Installation
• 🎧 Demos
• 🐍 Usage
• ❓ FAQ
• 🧠 TODO

🚀 Updates

• Added AR decoder batch decoding for more stable generation result.

• Replaced EnCodec decoder with Vocos decoder, improved audio quality. (Thanks to @v0xie)

• Added long text generation.

• Added Chinese README.

• Pretrained VALL-E X checkpoint is now released. Download it here

💻 Installation

Install with pip, Python 3.10, CUDA 11.7 ~ 12.0, PyTorch 2.0+

git clone https://github.com/Plachtaa/VALL-E-X.git
cd VALL-E-X
pip install -r requirements.txt

Note: If you want to make prompt, you need to install ffmpeg and add its folder to the environment variable PATH.

When you run the program for the first time, it will automatically download the corresponding model.

If the download fails and reports an error, please follow the steps below to manually download the model.

(Please pay attention to the capitalization of folders)

1. Check whether there is a checkpoints folder in the installation directory. If not, manually create a checkpoints folder (./checkpoints/) in the installation directory.

2. Check whether there is a vallex-checkpoint.pt file in the checkpoints folder. If not, please manually download the vallex-checkpoint.pt file from here and put it in the checkpoints folder.

3. Check whether there is a whisper folder in the installation directory. If not, manually create a whisper folder (./whisper/) in the installation directory.

4. Check whether there is a medium.pt file in the whisper folder. If not, please manually download the medium.pt file from here and put it in the whisper folder.

🎧 Demos

📢 Features

VALL-E X comes packed with cutting-edge functionalities:

1. Multilingual TTS: Speak in three languages - English, Chinese, and Japanese - with natural and expressive speech synthesis.

2. Zero-shot Voice Cloning: Enroll a short 3~10 seconds recording of an unseen speaker, and watch VALL-E X create personalized, high-quality speech that sounds just like them!

3. Speech Emotion Control: Experience the power of emotions! VALL-E X can synthesize speech with the same emotion as the acoustic prompt provided, adding an extra layer of expressiveness to your audio.

4. Zero-shot Cross-Lingual Speech Synthesis: Take monolingual speakers on a linguistic journey! VALL-E X can produce personalized speech in another language without compromising on fluency or accent. Below is a Japanese speaker talk in Chinese & English. 🇯🇵 🗣

5. Accent Control: Get creative with accents! VALL-E X allows you to experiment with different accents, like speaking Chinese with an English accent or vice versa. 🇨🇳 💬

6. Acoustic Environment Maintenance: No need for perfectly clean audio prompts! VALL-E X adapts to the acoustic environment of the input, making speech generation feel natural and immersive.

Explore our demo page for a lot more examples!

🐍 Usage in Python

from utils.generation import SAMPLE_RATE, generate_audio, preload_models
from scipy.io.wavfile import write as write_wav
from IPython.display import Audio
download and load all models
preload_models()
generate audio from text
text_prompt = """
Hello, my name is Nose. And uh, and I like hamburger. Hahaha... But I also have other interests such as playing tactic toast.
"""
audio_array = generate_audio(text_prompt)
save audio to disk
write_wav("vallex_generation.wav", SAMPLE_RATE, audio_array)
play text in notebook
Audio(audio_array, rate=SAMPLE_RATE)

text_prompt = """
    チュソクは私のお気に入りの祭りです。 私は数日間休んで、友人や家族との時間を過ごすことができます。
"""
audio_array = generate_audio(text_prompt)

text_prompt = """
    [EN]The Thirty Years' War was a devastating conflict that had a profound impact on Europe.[EN]
    [ZH]这是历史的开始。 如果您想听更多，请继续。[ZH]
"""
audio_array = generate_audio(text_prompt, language='mix')

text_prompt = """
I am an innocent boy with a smoky voice. It is a great honor for me to speak at the United Nations today.
"""
audio_array = generate_audio(text_prompt, prompt="dingzhen")

from utils.prompt_making import make_prompt
Use given transcript
make_prompt(name="paimon", audio_prompt_path="paimon_prompt.wav",
                transcript="Just, what was that? Paimon thought we were gonna get eaten.")
Alternatively, use whisper
make_prompt(name="paimon", audio_prompt_path="paimon_prompt.wav")

from utils.generation import SAMPLE_RATE, generate_audio, preload_models
from scipy.io.wavfile import write as write_wav
download and load all models
preload_models()
text_prompt = """
Hey, Traveler, Listen to this, This machine has taken my voice, and now it can talk just like me!
"""
audio_array = generate_audio(text_prompt, prompt="paimon")
write_wav("paimon_cloned.wav", SAMPLE_RATE, audio_array)

python -X utf8 launch-ui.py

🛠️ Hardware and Inference Speed

VALL-E X works well on both CPU and GPU (pytorch 2.0+, CUDA 11.7 and CUDA 12.0).

A GPU VRAM of 6GB is enough for running VALL-E X without offloading.

⚙️ Details

VALL-E X is similar to Bark, VALL-E and AudioLM, which generates audio in GPT-style by predicting audio tokens quantized by EnCodec.
Comparing to Bark:

• ✔ Light-weighted: 3️⃣ ✖ smaller,
• ✔ Efficient: 4️⃣ ✖ faster,
• ✔ Better quality on Chinese & Japanese
• ✔ Cross-lingual speech without foreign accent
• ✔ Easy voice-cloning
• ❌ Less languages
• ❌ No special tokens for music / sound effects

Supported Languages

| Language | Status | | --- | :---: | | English (en) | ✅ | | Japanese (ja) | ✅ | | Chinese, simplified (zh) | ✅ |

❓ FAQ

Where is code for training?

• lifeiteng's vall-e has almost everything. There is no plan to release our training code because there is no difference between lifeiteng's implementation.

Where can I download the model checkpoint?

• We use wget to download the model to directory ./checkpoints/ when you run the program for the first time.
• If the download fails on the first run, please manually download from this link, and put the file under directory ./checkpoints/.

How much VRAM do I need?

• 6GB GPU VRAM - Almost all NVIDIA GPUs satisfy the requirement.

Why the model fails to generate long text?

• Transformer's computation complexity increases quadratically while the sequence length increases. Hence, all training

MORE TO BE ADDED...

🧠 TODO

• [x] Add Chinese README
• [x] Long text generation
• [x] Replace Encodec decoder with Vocos decoder
• [ ] Fine-tuning for better voice adaptation
• [ ] .bat scripts for non-python users
• [ ] To be added...

🙏 Appreciation

• VALL-E X paper for the brilliant idea
• lifeiteng's vall-e for related training code
• bark for the amazing pioneering work in neuro-codec TTS model

⭐️ Show Your Support

If you find VALL-E X interesting and useful, give us a star on GitHub! ⭐️ It encourages us to keep improving the model and adding exciting features.

📜 License

VALL-E X is licensed under the MIT License.

---

Have questions or need assistance? Feel free to open an issue or join our Discord

Happy voice cloning! 🎤