今日论文合集：cs.SD语音5篇，eess.AS音频处理6篇。
【1】 Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors
标题：基于概率视听扩散先验的话头生成
链接：https://arxiv.org/abs/2212.04248
作者：Zhentao Yu,Zixin Yin,Deyu Zhou,Duomin Wang,Finn Wong,Baoyuan Wang机构：Xiaobing.AI, Identity , Sample , Driving Audio摘要：本文介绍了一种简单新颖的单次音频驱动的说话头部生成框架。与需要用于以确定性方式进行受控合成的附加驱动源的现有工作不同，我们改为概率性地采样所有与嘴唇无关的整体面部运动（即姿势、表情、眨眼、凝视等）。以在语义上匹配输入音频，同时仍然保持音频-嘴唇同步的照片真实感和整体自然度。这是通过我们新提出的在音频和解纠缠的非嘴唇面部表征之间的映射之上训练的视听扩散来实现的。由于扩散先验的概率性质，我们的框架的一个很大的优势是它可以合成不同的面部运动序列给定相同的音频剪辑，这是非常用户友好的许多实际应用。通过对公共基准数据的综合评估，我们得出以下结论：（1）在几乎所有相关指标上，我们的扩散先验都显著优于自回归先验;（2）我们的整个系统在音频-嘴唇同步方面与现有的作品有竞争力，但是可以有效地采样丰富且看起来自然的与嘴唇无关的面部动作，同时仍然与音频输入在语义上协调。摘要：In this paper, we introduce a simple and novel framework for one-shot audio-driven talking head generation. Unlike prior works that require additional driving sources for controlled synthesis in a deterministic manner, we instead probabilistically sample all the holistic lip-irrelevant facial motions (i.e. pose, expression, blink, gaze, etc.) to semantically match the input audio while still maintaining both the photo-realism of audio-lip synchronization and the overall naturalness. This is achieved by our newly proposed audio-to-visual diffusion prior trained on top of the mapping between audio and disentangled non-lip facial representations. Thanks to the probabilistic nature of the diffusion prior, one big advantage of our framework is it can synthesize diverse facial motion sequences given the same audio clip, which is quite user-friendly for many real applications. Through comprehensive evaluations on public benchmarks, we conclude that (1) our diffusion prior outperforms auto-regressive prior significantly on almost all the concerned metrics; (2) our overall system is competitive with prior works in terms of audio-lip synchronization but can effectively sample rich and natural-looking lip-irrelevant facial motions while still semantically harmonized with the audio input.
【2】 Learning to Dub Movies via Hierarchical Prosody Models
标题：通过层次化韵律模型学习配音电影
链接：https://arxiv.org/abs/2212.04054
作者：Gaoxiang Cong,Liang Li,Yuankai Qi,Zhengjun Zha,Qi Wu,Wenyu Wang,Bin Jiang,Ming-Hsuan Yang,Qingming Huang机构：Shandong University, Institute of Computing Technology, Chinese Academy of Sciences, University of Adelaide, University of Science and Technology of China, University of California摘要：给定一段文本、一个视频剪辑和一个参考音频，电影配音（也称为可视语音克隆V2C）任务旨在使用期望的说话者语音作为参考来生成与视频中呈现的说话者情感相匹配的语音。V2C比传统的文本到语音任务更具挑战性，因为它还需要生成的语音与视频中呈现的变化的情绪和说话速度精确匹配。与以往的工作不同，本文提出了一种新颖的电影配音体系结构，通过层次化的韵律建模来解决这些问题，该模型从三个方面将视觉信息与相应的语音韵律联系起来：唇、脸和场景。具体而言，我们将嘴唇运动与言语时长相匹配，并通过基于效价和唤醒表征的注意机制将面部表情传递给言语能量和音高。此外，我们还设计了一个情感助推器来捕捉全球视频场景中的气氛。所有这些嵌入一起被用于生成Mel谱图，然后通过现有的声码器转换成语音波形。在Chem和V2C基准数据集上的实验结果表明，该方法具有良好的性能.源代码和经过训练的模型将向公众发布。摘要：Given a piece of text, a video clip and a reference audio, the movie dubbing (also known as visual voice clone V2C) task aims to generate speeches that match the speaker's emotion presented in the video using the desired speaker voice as reference. V2C is more challenging than conventional text-to-speech tasks as it additionally requires the generated speech to exactly match the varying emotions and speaking speed presented in the video. Unlike previous works, we propose a novel movie dubbing architecture to tackle these problems via hierarchical prosody modelling, which bridges the visual information to corresponding speech prosody from three aspects: lip, face, and scene. Specifically, we align lip movement to the speech duration, and convey facial expression to speech energy and pitch via attention mechanism based on valence and arousal representations inspired by recent psychology findings. Moreover, we design an emotion booster to capture the atmosphere from global video scenes. All these embeddings together are used to generate mel-spectrogram and then convert to speech waves via existing vocoder. Extensive experimental results on the Chem and V2C benchmark datasets demonstrate the favorable performance of the proposed method. The source code and trained models will be released to the public.
【3】 Robust Speech Recognition via Large-Scale Weak Supervision
标题：基于大规模弱监督的稳健语音识别
链接：https://arxiv.org/abs/2212.04356
作者：Alec Radford,Jong Wook Kim,Tao Xu,Greg Brockman,Christine McLeavey,Ilya Sutskever摘要：我们研究了语音处理系统的能力，这些系统被训练来预测互联网上大量的音频转录本。当扩展到680，000小时的多语言和多任务监控时，生成的模型可以很好地推广到标准基准，并且通常可以与之前的完全监控结果竞争，但在zero-shot传输设置中，不需要任何微调。当与人类相比时，模型接近其准确性和鲁棒性。我们正在发布模型和推理代码，作为鲁棒语音处理进一步工作的基础。摘要：We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
【4】 Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers
标题：基于音素的神经传感器的无格子序列辨别训练
链接：https://arxiv.org/abs/2212.04325
作者：Zijian Yang,Wei Zhou,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：submitted to ICASSP 2023摘要：近年来，RNN-转换器在各种自动语音识别任务中取得了显著的效果。然而，在混合模式下获得优越性能的无格序列鉴别训练方法在RNN换能器中的研究很少。在本文中，我们提出了三个无格训练目标，即无格最大互信息、无格段级最小贝叶斯风险和无格最小贝叶斯风险，它们被用于有限上下文依赖的基于音素的神经传感器的最终后验输出.与使用N-最佳列表的准则相比，无格方法消除了训练期间用于假设生成的解码步骤，这导致更有效的训练。实验结果表明，与序列级交叉熵训练模型相比，无格方法在单词错误率方面获得了高达6.5%的相对改善。与基于N-best-list的最小贝叶斯风险目标相比，无格方法在性能略有下降的情况下，相对训练时间提高了40% ~ 70%。摘要：Recently, RNN-Transducers have achieved remarkable results on various automatic speech recognition tasks. However, lattice-free sequence discriminative training methods, which obtain superior performance in hybrid modes, are rarely investigated in RNN-Transducers. In this work, we propose three lattice-free training objectives, namely lattice-free maximum mutual information, lattice-free segment-level minimum Bayes risk, and lattice-free minimum Bayes risk, which are used for the final posterior output of the phoneme-based neural transducer with a limited context dependency. Compared to criteria using N-best lists, lattice-free methods eliminate the decoding step for hypotheses generation during training, which leads to more efficient training. Experimental results show that lattice-free methods gain up to 6.5% relative improvement in word error rate compared to a sequence-level cross-entropy trained model. Compared to the N-best-list based minimum Bayes risk objectives, lattice-free methods gain 40% - 70% relative training time speedup with a small degradation in performance.
【5】 On The Relevance Of The Differences Between HRTF Measurement Setups For Machine Learning
标题：关于HRTF测量设置之间的差异与机器学习的相关性
链接：https://arxiv.org/abs/2212.04283
作者：Johan Pauwels,Lorenzo Picinali机构：Queen Mary University of London, Imperial College London摘要：随着空间音频的流行，数据驱动的机器学习技术越来越多地被用于处理头部相关传递函数测量，这些技术在其他领域已经被证明是成功的。然而，这些技术需要大量的数据，而现有的数据集范围从几十个到几百个数据点。因此，尽管这些数据集是在不同的条件下测量的，但将它们组合起来是有吸引力的。本文首先建立了多个数据集之间的共同点，然后研究了混合数据集的潜在缺陷。我们执行一个简单的实验来测试在应用机器学习技术时数据集之间剩余差异的相关性。最后，我们指出最相关的差异。摘要：As spatial audio is enjoying a surge in popularity, data-driven machine learning techniques that have been proven successful in other domains are increasingly used to process head-related transfer function measurements. However, these techniques require much data, whereas the existing datasets are ranging from tens to the low hundreds of datapoints. It therefore becomes attractive to combine multiple of these datasets, although they are measured under different conditions. In this paper, we first establish the common ground between a number of datasets, then we investigate potential pitfalls of mixing datasets. We perform a simple experiment to test the relevance of the remaining differences between datasets when applying machine learning techniques. Finally, we pinpoint the most relevant differences.
【1】 Low-Bitrate Redundancy Coding of Speech Using a Rate-Distortion-Optimized Variational Autoencoder
标题：使用率失真优化的变分自动编码器的语音低码率冗余编码
链接：https://arxiv.org/abs/2212.04453
作者：Jean-Marc Valin,Jan Büthe,Ahmed Mustafa机构：Amazon Web Services, Palo Alto, USA备注：Submitted to ICASSP 2023, 5 pages摘要：对分组丢失的鲁棒性是实时语音通信中的主要挑战之一。深度分组丢失隐藏（PLC）技术最近已经证明了与传统PLC相比改进的质量。尽管如此，当丢失太多声学信息时，所有PLC技术都会遇到基本限制。首先，为了减少损失，通常使用各种冗余机制多次发送数据。我们提出了一种神经语音编码器，它能以非常低的比特率传输大量的重叠冗余，最高可达50倍的冗余，使用的比特率小于32 kb/s。结果表明，提出的冗余比现有的Opus编解码器冗余更有效，并且可以将两者结合以获得更大的鲁棒性。摘要：Robustness to packet loss is one of the main ongoing challenges in real-time speech communication. Deep packet loss concealment (PLC) techniques have recently demonstrated improved quality compared to traditional PLC. Despite that, all PLC techniques hit fundamental limitations when too much acoustic information is lost. To reduce losses in the first place, data is commonly sent multiple times using various redundancy mechanisms. We propose a neural speech coder specifically optimized to transmit a large amount of overlapping redundancy at a very low bitrate, up to 50x redundancy using less than 32~kb/s. Results show that the proposed redundancy is more effective than the existing Opus codec redundancy, and that the two can be combined for even greater robustness.
【2】 Robust Speech Recognition via Large-Scale Weak Supervision
标题：基于大规模弱监督的稳健语音识别
链接：https://arxiv.org/abs/2212.04356
*与cs.SD语音【3】为同一篇
作者：Alec Radford,Jong Wook Kim,Tao Xu,Greg Brockman,Christine McLeavey,Ilya Sutskever摘要：我们研究了语音处理系统的能力，这些系统被训练来预测互联网上大量的音频转录本。当扩展到680，000小时的多语言和多任务监控时，生成的模型可以很好地推广到标准基准，并且通常可以与之前的完全监控结果竞争，但在zero-shot传输设置中，不需要任何微调。当与人类相比时，模型接近其准确性和鲁棒性。我们正在发布模型和推理代码，作为鲁棒语音处理进一步工作的基础。摘要：We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
【3】 Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers
标题：基于音素的神经传感器的无格子序列辨别训练
链接：https://arxiv.org/abs/2212.04325
*与cs.SD语音【4】为同一篇
作者：Zijian Yang,Wei Zhou,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：submitted to ICASSP 2023摘要：Recently, RNN-Transducers have achieved remarkable results on various automatic speech recognition tasks. However, lattice-free sequence discriminative training methods, which obtain superior performance in hybrid modes, are rarely investigated in RNN-Transducers. In this work, we propose three lattice-free training objectives, namely lattice-free maximum mutual information, lattice-free segment-level minimum Bayes risk, and lattice-free minimum Bayes risk, which are used for the final posterior output of the phoneme-based neural transducer with a limited context dependency. Compared to criteria using N-best lists, lattice-free methods eliminate the decoding step for hypotheses generation during training, which leads to more efficient training. Experimental results show that lattice-free methods gain up to 6.5% relative improvement in word error rate compared to a sequence-level cross-entropy trained model. Compared to the N-best-list based minimum Bayes risk objectives, lattice-free methods gain 40% - 70% relative training time speedup with a small degradation in performance.
【4】 On The Relevance Of The Differences Between HRTF Measurement Setups For Machine Learning
标题：关于HRTF测量设置之间的差异与机器学习的相关性
链接：https://arxiv.org/abs/2212.04283
*与cs.SD语音【5】为同一篇
作者：Johan Pauwels,Lorenzo Picinali机构：Queen Mary University of London, Imperial College London摘要：As spatial audio is enjoying a surge in popularity, data-driven machine learning techniques that have been proven successful in other domains are increasingly used to process head-related transfer function measurements. However, these techniques require much data, whereas the existing datasets are ranging from tens to the low hundreds of datapoints. It therefore becomes attractive to combine multiple of these datasets, although they are measured under different conditions. In this paper, we first establish the common ground between a number of datasets, then we investigate potential pitfalls of mixing datasets. We perform a simple experiment to test the relevance of the remaining differences between datasets when applying machine learning techniques. Finally, we pinpoint the most relevant differences.
【5】 Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors
标题：基于概率视听扩散先验的话头生成
链接：https://arxiv.org/abs/2212.04248
*与cs.SD语音【1】为同一篇
作者：Zhentao Yu,Zixin Yin,Deyu Zhou,Duomin Wang,Finn Wong,Baoyuan Wang机构：Xiaobing.AI, Identity , Sample , Driving Audio摘要：本文介绍了一种简单新颖的单次音频驱动的说话头部生成框架。与需要用于以确定性方式进行受控合成的附加驱动源的现有工作不同，我们改为概率性地采样所有整体嘴唇无关的面部运动（即姿势、表情、眨眼、凝视等）。以在语义上匹配输入音频，同时仍然保持音频-嘴唇同步的照片真实感和整体自然度。这是通过我们新提出的在音频和解纠缠的非嘴唇面部表征之间的映射之上训练的视听扩散来实现的。由于扩散先验的概率性质，我们的框架的一个很大的优势是它可以合成不同的面部运动序列给定相同的音频剪辑，这是非常用户友好的许多实际应用。通过对公共基准数据的综合评估，我们得出以下结论：（1）在几乎所有相关指标上，我们的扩散先验都显著优于自回归先验;（2）我们的整个系统在音频-嘴唇同步方面与现有的作品有竞争力，但是可以有效地采样丰富且看起来自然的与嘴唇无关的面部动作，同时仍然与音频输入在语义上协调。摘要：In this paper, we introduce a simple and novel framework for one-shot audio-driven talking head generation. Unlike prior works that require additional driving sources for controlled synthesis in a deterministic manner, we instead probabilistically sample all the holistic lip-irrelevant facial motions (i.e. pose, expression, blink, gaze, etc.) to semantically match the input audio while still maintaining both the photo-realism of audio-lip synchronization and the overall naturalness. This is achieved by our newly proposed audio-to-visual diffusion prior trained on top of the mapping between audio and disentangled non-lip facial representations. Thanks to the probabilistic nature of the diffusion prior, one big advantage of our framework is it can synthesize diverse facial motion sequences given the same audio clip, which is quite user-friendly for many real applications. Through comprehensive evaluations on public benchmarks, we conclude that (1) our diffusion prior outperforms auto-regressive prior significantly on almost all the concerned metrics; (2) our overall system is competitive with prior works in terms of audio-lip synchronization but can effectively sample rich and natural-looking lip-irrelevant facial motions while still semantically harmonized with the audio input.
【6】 Learning to Dub Movies via Hierarchical Prosody Models
标题：通过层次化韵律模型学习配音电影
链接：https://arxiv.org/abs/2212.04054
*与cs.SD语音【2】为同一篇
作者：Gaoxiang Cong,Liang Li,Yuankai Qi,Zhengjun Zha,Qi Wu,Wenyu Wang,Bin Jiang,Ming-Hsuan Yang,Qingming Huang机构：Shandong University, Institute of Computing Technology, Chinese Academy of Sciences, University of Adelaide, University of Science and Technology of China, University of California摘要：Given a piece of text, a video clip and a reference audio, the movie dubbing (also known as visual voice clone V2C) task aims to generate speeches that match the speaker's emotion presented in the video using the desired speaker voice as reference. V2C is more challenging than conventional text-to-speech tasks as it additionally requires the generated speech to exactly match the varying emotions and speaking speed presented in the video. Unlike previous works, we propose a novel movie dubbing architecture to tackle these problems via hierarchical prosody modelling, which bridges the visual information to corresponding speech prosody from three aspects: lip, face, and scene. Specifically, we align lip movement to the speech duration, and convey facial expression to speech energy and pitch via attention mechanism based on valence and arousal representations inspired by recent psychology findings. Moreover, we design an emotion booster to capture the atmosphere from global video scenes. All these embeddings together are used to generate mel-spectrogram and then convert to speech waves via existing vocoder. Extensive experimental results on the Chem and V2C benchmark datasets demonstrate the favorable performance of the proposed method. The source code and trained models will be released to the public.
机器翻译，仅供参考
每日语音速递