Abstract. We present RALL-E, a robust language modeling method for text-to-speech (TTS) synthesis. While previous work based on large language models (LLMs) shows impressive performance on zero-shot TTS, such methods often suffer from poor robustness, such as unstable prosody (weird pitch and rhythm/duration) and a high word error rate (WER), due to the autoregressive prediction style of language models. The core idea behind RALL-E is chain-of-thought (CoT) prompting, which decomposes the task into simpler steps to enhance the robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts prosody features (pitch and duration) of the input text and uses them as intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E utilizes the predicted duration prompt to guide the computing of self-attention weights in Transformer to enforce the model to focus on the corresponding phonemes and prosody features when predicting speech tokens. Results of comprehensive objective and subjective evaluations demonstrate that, compared to a powerful baseline method VALL-E, RALL-E significantly improves the WER of zero-shot TTS from 5.6% (without reranking) and 1.7% (with reranking) to 2.8% and 1.0%, respectively. Furthermore, we demonstrate that RALL-E correctly synthesizes sentences that are hard for VALL-E and reduces the error rate from 68% to 4%.

This page is for research demonstration purposes only.

Zero-Shot TTS Samples

We show the audio samples of GT, RALL-E, ELLA-V, and VALL-E selected from LibriSpeech test-clean set. We cannot show the samples of VALL-T since it is tested on LibriTTS. The WER% and transcriptions recognized by the ASR model used in the paper are also provided, where mispronunciation/repetition/hallucination is denoted in UPPER case, word omission is denoted by *.

---

Prompt text: I thought few men would follow me for I was young

Prompt audio:

Synthesis text: But young sharp tongue, now that we have caught you, we will put you into a trap that you cannot get out of.

Prompt text: using every possible contraction a quarter of an hour, not less.

Prompt audio:

Synthesis text: When we were out in the darkness of the quadrangle, we again looked up at the windows.

Prompt text: of this long separation will wear away.

Prompt audio:

Synthesis text: She, a Tory and clergyman's daughter, was always in a minority of one in our house of violent dissent and radicalism.

Prompt text: Far from its sire, your majesty haven't given

Prompt audio:

Synthesis text: Quick, quick then, among the high-reed grass said Montalais. Stoop, Athenais, you are so tall.

Prompt text: drew back from John Yago as he approached the empty chair next

Prompt audio:

Synthesis text: Soft heart, he said gently to her. Then to Thorkel. Well, let him go, Thorkel.

Samples of hard setences

We show samples of the hard sentences mentioned in Section 4.4 of the paper. These sentences have no GT utterance. We describe the errors (mispronunciation, omission, repetition, hallucination) existing in the samples.

---

Synthesis text: b

Synthesis text: 22222222 hello 22222222

Synthesis text: c five eight zero three three nine a zero bf eight FALSE zero zero zero bba3add2 - c229 - 4cdb -

Synthesis text: You can call me directly at four two five seven zero three seven three four four or my cell four two five four four four seven four seven four or send me a meeting request with all the appropriate information.

Synthesis text: Thanks J RGR Are you using the LDDM driver for this system or the in the build XDDM driver ? 12