今日论文合集：cs.SD语音3篇，eess.AS音频处理4篇。
【1】 Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion
标题：莫赛：长语境潜伏扩散的文本到音乐世代
链接：https://arxiv.org/abs/2301.11757
作者：Flavio Schneider,Zhijing Jin,Bernhard Schölkopf机构：Switzerland 2Max Planck Institute for Intelli-gent Systems备注：Music samples for this paper: this https URL ; all music samples for all models: this https URL ; codes: this https URL摘要：近来用于图像生成的扩散模型的流行度的激增已经引起了对这些模型在媒体合成的其它领域中的潜力的新的关注。一个尚未充分探索的领域是扩散模型在音乐生成中的应用。音乐生成需要处理多个方面，包括时间维度、长期结构、多层重叠声音以及只有训练有素的听众才能察觉的细微差别。在我们的工作中，我们研究了文本条件音乐生成的扩散模型的潜力。我们开发了一种级联的潜在扩散方法，可以从文本描述中生成多分钟的48 kHz高质量立体声音乐。对于每个模型，我们都努力保持合理的推理速度，目标是在单个消费级GPU上实现实时性。除了经过训练的模型之外，我们还提供了一个开源库的集合，希望能够促进该领域未来的工作。 我们开源了以下内容：- 本文的音乐样本：https://bit.ly/anonymous-mousai-适用于所有型号的全部音乐样本：网址：https://bit.ly/audio-diffusion-代码：https://github.com/archinetai/audio-diffusion-pytorch摘要：The recent surge in popularity of diffusion models for image generation has brought new attention to the potential of these models in other areas of media synthesis. One area that has yet to be fully explored is the application of diffusion models to music generation. Music generation requires to handle multiple aspects, including the temporal dimension, long-term structure, multiple layers of overlapping sounds, and nuances that only trained listeners can detect. In our work, we investigate the potential of diffusion models for text-conditional music generation. We develop a cascading latent diffusion approach that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions. For each model, we make an effort to maintain reasonable inference speed, targeting real-time on a single consumer GPU. In addition to trained models, we provide a collection of open-source libraries with the hope of facilitating future work in the field. We open-source the following: - Music samples for this paper: https://bit.ly/anonymous-mousai - All music samples for all models: https://bit.ly/audio-diffusion - Codes: https://github.com/archinetai/audio-diffusion-pytorch
【2】 Pre-training for Speech Translation: CTC Meets Optimal Transport
标题：语音翻译的前期训练：CTC满足最佳传输
链接：https://arxiv.org/abs/2301.11716
作者：Phuong-Hang Le,Hongyu Gong,Changhan Wang,Juan Pino,Benjamin Lecouteux,Didier Schwab摘要：语篇模态和语音模态之间的差异是语篇翻译面临的主要挑战。人们提出了不同的方法来减少这种差距，但大多数方法都需要改变ST训练的结构。在这项工作中，我们建议在预训练阶段缓解这个问题，不需要改变ST模型。首先，我们证明了连接主义时间分类（CTC）的损失可以通过设计来减少模态差距。我们提供了与更常见的交叉熵损失的定量比较，表明使用CTC的预训练一致地实现了更好的最终ST准确性。然而，CTC只是一个部分的解决方案，因此，在我们的第二个贡献，我们提出了一个新的预训练方法结合CTC和最佳运输，以进一步减少这一差距。我们的方法预先训练了一个由两个编码器组成的类似暹罗的模型，一个用于声音输入，另一个用于文本输入，这样它们产生在Wasserstein空间中彼此接近的表示。在标准CoVoST-2和MuST-C数据集上的大量实验表明，应用于vanilla编解码器Transformer的预训练方法在无外部数据设置下达到了最先进的性能，并且与最近使用外部数据训练的强多任务学习系统的性能相当。最后，我们的方法也可以应用在这些多任务系统之上，从而进一步改进这些模型。摘要：The gap between speech and text modalities is a major challenge in speech-to-text translation (ST). Different methods have been proposed for reducing this gap, but most of them require architectural changes in ST training. In this work, we propose to mitigate this issue at the pre-training stage, requiring no change in the ST model. First, we show that the connectionist temporal classification (CTC) loss can reduce the modality gap by design. We provide a quantitative comparison with the more common cross-entropy loss, showing that pre-training with CTC consistently achieves better final ST accuracy. Nevertheless, CTC is only a partial solution and thus, in our second contribution, we propose a novel pre-training method combining CTC and optimal transport to further reduce this gap. Our method pre-trains a Siamese-like model composed of two encoders, one for acoustic inputs and the other for textual inputs, such that they produce representations that are close to each other in the Wasserstein space. Extensive experiments on the standard CoVoST-2 and MuST-C datasets show that our pre-training method applied to the vanilla encoder-decoder Transformer achieves state-of-the-art performance under the no-external-data setting, and performs on par with recent strong multi-task learning systems trained with external data. Finally, our method can also be applied on top of these multi-task systems, leading to further improvements for these models.
【3】 On granularity of prosodic representations in expressive text-to-speech
标题：表现力文语转换中韵律表征的粒度研究
链接：https://arxiv.org/abs/2301.11446
作者：Mikolaj Babianski,Kamil Pokora,Raahil Shah,Rafal Sienkiewicz,Daniel Korzekwa,Viacheslav Klimkov机构：Amazon Text-to-Speech Research备注：Accepted to IEEE SLT 2022摘要：在表达性语音合成中，广泛采用潜在韵律表示来处理训练过程中数据的可变性。同一个文本可能对应着不同的声学实现，这就是文本到语音的一对多映射问题。在自动编码设置中从目标信号中提取话语、单词或音素级表示，以补充语音输入并简化映射。本文比较了不同粒度下的韵律嵌入，并检验了它们对文本的预测。我们发现话语级嵌入容量不足，而音素级嵌入在预测文本时容易引入不稳定性。词级表示在容量和可预测性之间施加平衡。因此，我们将合成语音和LibriTTS数据集上的录音之间的自然度差距缩小了90%，而不牺牲可懂度。摘要：In expressive speech synthesis it is widely adopted to use latent prosody representations to deal with variability of the data during training. Same text may correspond to various acoustic realizations, which is known as a one-to-many mapping problem in text-to-speech. Utterance, word, or phoneme-level representations are extracted from target signal in an auto-encoding setup, to complement phonetic input and simplify that mapping. This paper compares prosodic embeddings at different levels of granularity and examines their prediction from text. We show that utterance-level embeddings have insufficient capacity and phoneme-level tend to introduce instabilities when predicted from text. Word-level representations impose balance between capacity and predictability. As a result, we close the gap in naturalness by 90% between synthetic speech and recordings on LibriTTS dataset, without sacrificing intelligibility.
【1】 On granularity of prosodic representations in expressive text-to-speech
标题：表现力文语转换中韵律表征的粒度研究
链接：https://arxiv.org/abs/2301.11446
*与cs.SD语音【3】为同一篇
作者：Mikolaj Babianski,Kamil Pokora,Raahil Shah,Rafal Sienkiewicz,Daniel Korzekwa,Viacheslav Klimkov机构：Amazon Text-to-Speech Research备注：Accepted to IEEE SLT 2022摘要：在表达性语音合成中，广泛采用潜在韵律表示来处理训练过程中数据的可变性。同一个文本可能对应着不同的声学实现，这就是文本到语音的一对多映射问题。在自动编码设置中从目标信号中提取话语、单词或音素级表示，以补充语音输入并简化映射。本文比较了不同粒度下的韵律嵌入，并检验了它们对文本的预测。我们发现话语级嵌入容量不足，而音素级嵌入在预测文本时容易引入不稳定性。词级表示在容量和可预测性之间施加平衡。因此，我们将合成语音和LibriTTS数据集上的录音之间的自然度差距缩小了90%，而不牺牲可懂度。摘要：In expressive speech synthesis it is widely adopted to use latent prosody representations to deal with variability of the data during training. Same text may correspond to various acoustic realizations, which is known as a one-to-many mapping problem in text-to-speech. Utterance, word, or phoneme-level representations are extracted from target signal in an auto-encoding setup, to complement phonetic input and simplify that mapping. This paper compares prosodic embeddings at different levels of granularity and examines their prediction from text. We show that utterance-level embeddings have insufficient capacity and phoneme-level tend to introduce instabilities when predicted from text. Word-level representations impose balance between capacity and predictability. As a result, we close the gap in naturalness by 90% between synthetic speech and recordings on LibriTTS dataset, without sacrificing intelligibility.
【2】 Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion
标题：莫赛：长语境潜伏扩散的文本到音乐世代
链接：https://arxiv.org/abs/2301.11757
*与cs.SD语音【1】为同一篇
作者：Flavio Schneider,Zhijing Jin,Bernhard Schölkopf机构：Switzerland 2Max Planck Institute for Intelli-gent Systems备注：Music samples for this paper: this https URL ; all music samples for all models: this https URL ; codes: this https URL摘要：近来用于图像生成的扩散模型的流行度的激增已经引起了对这些模型在媒体合成的其它领域中的潜力的新的关注。一个尚未充分探索的领域是扩散模型在音乐生成中的应用。音乐生成需要处理多个方面，包括时间维度、长期结构、多层重叠声音以及只有训练有素的听众才能察觉的细微差别。在我们的工作中，我们研究了文本条件音乐生成的扩散模型的潜力。我们开发了一种级联的潜在扩散方法，可以从文本描述中生成多分钟的48 kHz高质量立体声音乐。对于每个模型，我们都努力保持合理的推理速度，目标是在单个消费级GPU上实现实时性。除了经过训练的模型之外，我们还提供了一个开源库的集合，希望能够促进该领域未来的工作。 我们开源了以下内容：- 本文的音乐样本：https://bit.ly/anonymous-mousai-适用于所有型号的全部音乐样本：网址：https://bit.ly/audio-diffusion-代码：https://github.com/archinetai/audio-diffusion-pytorch摘要：The recent surge in popularity of diffusion models for image generation has brought new attention to the potential of these models in other areas of media synthesis. One area that has yet to be fully explored is the application of diffusion models to music generation. Music generation requires to handle multiple aspects, including the temporal dimension, long-term structure, multiple layers of overlapping sounds, and nuances that only trained listeners can detect. In our work, we investigate the potential of diffusion models for text-conditional music generation. We develop a cascading latent diffusion approach that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions. For each model, we make an effort to maintain reasonable inference speed, targeting real-time on a single consumer GPU. In addition to trained models, we provide a collection of open-source libraries with the hope of facilitating future work in the field. We open-source the following: - Music samples for this paper: https://bit.ly/anonymous-mousai - All music samples for all models: https://bit.ly/audio-diffusion - Codes: https://github.com/archinetai/audio-diffusion-pytorch
【3】 Pre-training for Speech Translation: CTC Meets Optimal Transport
标题：语音翻译的前期训练：CTC满足最佳传输
链接：https://arxiv.org/abs/2301.11716
*与cs.SD语音【2】为同一篇
作者：Phuong-Hang Le,Hongyu Gong,Changhan Wang,Juan Pino,Benjamin Lecouteux,Didier Schwab摘要：语篇模态和语音模态之间的差异是语篇翻译面临的主要挑战。人们提出了不同的方法来减少这种差距，但大多数方法都需要改变ST训练的结构。在这项工作中，我们建议在预训练阶段缓解这个问题，不需要改变ST模型。首先，我们证明了连接主义时间分类（CTC）的损失可以通过设计来减少模态差距。我们提供了与更常见的交叉熵损失的定量比较，表明使用CTC的预训练一致地实现了更好的最终ST准确性。然而，CTC只是一个部分的解决方案，因此，在我们的第二个贡献，我们提出了一个新的预训练方法结合CTC和最佳运输，以进一步减少这一差距。我们的方法预先训练了一个由两个编码器组成的类似暹罗的模型，一个用于声音输入，另一个用于文本输入，这样它们产生在Wasserstein空间中彼此接近的表示。在标准CoVoST-2和MuST-C数据集上的大量实验表明，应用于vanilla编解码器Transformer的预训练方法在无外部数据设置下达到了最先进的性能，并且与最近使用外部数据训练的强多任务学习系统的性能相当。最后，我们的方法也可以应用在这些多任务系统之上，从而进一步改进这些模型。摘要：The gap between speech and text modalities is a major challenge in speech-to-text translation (ST). Different methods have been proposed for reducing this gap, but most of them require architectural changes in ST training. In this work, we propose to mitigate this issue at the pre-training stage, requiring no change in the ST model. First, we show that the connectionist temporal classification (CTC) loss can reduce the modality gap by design. We provide a quantitative comparison with the more common cross-entropy loss, showing that pre-training with CTC consistently achieves better final ST accuracy. Nevertheless, CTC is only a partial solution and thus, in our second contribution, we propose a novel pre-training method combining CTC and optimal transport to further reduce this gap. Our method pre-trains a Siamese-like model composed of two encoders, one for acoustic inputs and the other for textual inputs, such that they produce representations that are close to each other in the Wasserstein space. Extensive experiments on the standard CoVoST-2 and MuST-C datasets show that our pre-training method applied to the vanilla encoder-decoder Transformer achieves state-of-the-art performance under the no-external-data setting, and performs on par with recent strong multi-task learning systems trained with external data. Finally, our method can also be applied on top of these multi-task systems, leading to further improvements for these models.
【4】 Four ways to recover the symbol of a non-binary localization operator
标题：恢复非二进制局部化运算符的四种方法
链接：https://arxiv.org/abs/2301.11618
摘要：我们提出了一组结果如何符号的定位算子可以恢复从频谱信息，图像的白噪声或图像的正交基。这扩展了以前的结果，这些结果仅限于符号是二进制掩码的情况。此外，我们给出了不同方法的一些数值方面，并讨论了它们的性能。摘要：We present a set of results on how the symbol of a localization operator can be recovered from spectral information, the image of white noise or the image of an orthonormal basis. This extends earlier results which have been limited to the case where the symbol is a binary mask. Moreover, we present some numerical aspects of the different methods and discuss their performance.
机器翻译，仅供参考
每日语音速递