今日论文合集：cs.SD语音5篇，eess.AS音频处理6篇。

cs.SD语音

【1】 WuYun: Exploring hierarchical skeleton-guided melody generation using  knowledge-enhanced deep learning

标题：乌云：利用知识增强型深度学习探索层次化骨架引导旋律生成

链接：https://arxiv.org/abs/2301.04488

作者：Kejun Zhang,Xinda Wu,Tieyao Zhang,Zhijie Huang,Xu Tan,Qihao Liang,Songruoyao Wu,Lingyun Sun
机构：College of Computer Science and Technology, Zhejiang University, China., Alibaba-Zhejiang University Joint Institute of Frontier Technologies, China., Innovation Center of Yangtze River Delta, China., Microsoft Research Asia
摘要：尽管深度学习已经彻底改变了音乐生成，但现有的结构化旋律生成方法遵循端到端、从左到右、逐个音符的生成范式，并平等地对待每个音符。在此，我们介绍WuYun，一个用于改进生成旋律结构的知识增强型深度学习架构，它首先生成结构上最重要的音符来构建旋律骨架，然后用动态装饰音符填充到完整的旋律中。具体而言，利用音乐领域知识提取旋律骨架，并采用序列学习重构旋律骨架，作为附加知识为旋律生成过程提供辅助指导。我们证明了五云算法能够生成具有更好的长期结构和音乐性的旋律，并且在所有主观评价指标上平均比其他最先进的方法高出0.51。我们的研究提供了一个多学科的视角来设计旋律层次结构，并在众多音乐生成任务中弥合数据驱动和基于知识的方法之间的差距。
摘要：Although deep learning has revolutionized music generation, existing methods for structured melody generation follow an end-to-end left-to-right note-by-note generative paradigm and treat each note equally. Here, we present WuYun, a knowledge-enhanced deep learning architecture for improving the structure of generated melodies, which first generates the most structurally important notes to construct a melodic skeleton and subsequently infills it with dynamically decorative notes into a full-fledged melody. Specifically, we use music domain knowledge to extract melodic skeletons and employ sequence learning to reconstruct them, which serve as additional knowledge to provide auxiliary guidance for the melody generation process. We demonstrate that WuYun can generate melodies with better long-term structure and musicality and outperforms other state-of-the-art methods by 0.51 on average on all subjective evaluation metrics. Our study provides a multidisciplinary lens to design melodic hierarchical structures and bridge the gap between data-driven and knowledge-based approaches for numerous music generation tasks.


【2】 Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

标题：基于音频条件扩散模型的语音驱动视频编辑

链接：https://arxiv.org/abs/2301.04474

作者：Dan Bigioi,Shubhajit Basak,Hugh Jordan,Rachel McDonnell,Peter Corcoran
机构：University of Galway, Trinity College Dublin
备注：8 Pages
摘要：本文提出了一种基于去噪扩散模型的端到端语音驱动视频编辑方法。给定一个人说话的视频，我们的目标是响应于单独的听觉语音记录来重新同步该人的嘴唇和下巴运动，而不依赖于诸如面部界标或3D面部模型之类的中间结构表示。我们证明了这是可能的条件去噪扩散模型与音频频谱特征，以产生同步的面部运动。我们在非结构化的单说话人视频编辑任务上取得了令人信服的结果，使用现成的唇读模型实现了45%的单词错误率。我们进一步展示了如何将我们的方法扩展到多说话人领域。据我们所知，这是第一个探索将去噪扩散模型应用于音频驱动的视频编辑任务的可行性的工作。
摘要：In this paper we propose a method for end-to-end speech driven video editing using a denoising diffusion model. Given a video of a person speaking, we aim to re-synchronise the lip and jaw motion of the person in response to a separate auditory speech recording without relying on intermediate structural representations such as facial landmarks or a 3D face model. We show this is possible by conditioning a denoising diffusion model with audio spectral features to generate synchronised facial motion. We achieve convincing results on the task of unstructured single-speaker video editing, achieving a word error rate of 45% using an off the shelf lip reading model. We further demonstrate how our approach can be extended to the multi-speaker domain. To our knowledge, this is the first work to explore the feasibility of applying denoising diffusion models to the task of audio-driven video editing.


【3】 Perceive and predict: self-supervised speech representation based loss  functions for speech enhancement

标题：感知和预测：基于自监督语音表示的损失函数语音增强

链接：https://arxiv.org/abs/2301.04388

作者：George Close,William Ravenscroft,Thomas Hain,Stefan Goetze
机构：Department of Computer Science, University of Sheffield
备注：4 pages, submitted to ICASSP 2023
摘要：在语音增强领域中的最近工作已经探索了使用自监督语音表示来帮助训练神经语音增强模型。然而，这些工作的大部分集中于使用自监督语音表示模型的最深或最终输出，而不是早期的特征编码。以这样的方式使用自监督表示通常不是完全有动机的。在这项工作中，表明干净语音和含噪语音的特征编码之间的距离与语音质量和可懂度的心理声学动机测量以及人类平均意见得分（MOS）评级密切相关。执行使用该距离作为损失函数的实验，并且使用诸如语音质量的感知评估（PESQ）和短时客观可懂度（STOI）的客观测量来证明相对于使用基于STFT谱图距离的损失以及来自语音增强文献的其他常见损失函数的改进性能。
摘要：Recent work in the domain of speech enhancement has explored the use of self-supervised speech representations to aid in the training of neural speech enhancement models. However, much of this work focuses on using the deepest or final outputs of self supervised speech representation models, rather than the earlier feature encodings. The use of self supervised representations in such a way is often not fully motivated. In this work it is shown that the distance between the feature encodings of clean and noisy speech correlate strongly with psychoacoustically motivated measures of speech quality and intelligibility, as well as with human Mean Opinion Score (MOS) ratings. Experiments using this distance as a loss function are performed and improved performance over the use of STFT spectrogram distance based loss as well as other common loss functions from speech enhancement literature is demonstrated using objective measures such as perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI).


【4】 Rethinking complex-valued deep neural networks for monaural speech  enhancement

标题：复值深度神经网络用于单声道语音增强的再思考

链接：https://arxiv.org/abs/2301.04320

作者：Haibin Wu,Ke Tan,Buye Xu,Anurag Kumar,Daniel Wong
机构：Meta Reality Labs Research, USA
摘要：尽管为采用复值深度神经网络（DNN）进行了多种努力，但对于单耳语音增强，复值DNN是否通常比实值DNN更有效仍然是一个开放的问题。这项工作致力于提出一个关键的评估，通过系统地审查复值DNN对他们的实际对应。具体来说，我们研究了复值DNN原子单元，包括线性层、卷积层、长短时记忆（LSTM）和门控线性单元。通过比较最近开发的门控卷积递归网络（GCRN）中基本构建块的复数和实值版本，我们展示了基本块的不同机制如何影响性能。我们还发现，当模型规模小时，复值运算的使用阻碍了模型的容量。此外，我们还研究了两种最新的复值DNN，即深度复卷积递归网络（DCCRN）和深度复U-Net（DCUNET）。仿真结果表明，两种DNN的性能与实值DNN相当，但计算量大得多。基于这些全面的比较，我们得出结论，对于单耳语音增强，复值DNN并不提供超过其实值对应物的性能增益，因此由于其较高的计算成本而不太理想。
摘要：Despite multiple efforts made towards adopting complex-valued deep neural networks (DNNs), it remains an open question whether complex-valued DNNs are generally more effective than real-valued DNNs for monaural speech enhancement. This work is devoted to presenting a critical assessment by systematically examining complex-valued DNNs against their real-valued counterparts. Specifically, we investigate complex-valued DNN atomic units, including linear layers, convolutional layers, long short-term memory (LSTM), and gated linear units. By comparing complex- and real-valued versions of fundamental building blocks in the recently developed gated convolutional recurrent network (GCRN), we show how different mechanisms for basic blocks affect the performance. We also find that the use of complex-valued operations hinders the model capacity when the model size is small. In addition, we examine two recent complex-valued DNNs, i.e. deep complex convolutional recurrent network (DCCRN) and deep complex U-Net (DCUNET). Evaluation results show that both DNNs produce identical performance to their real-valued counterparts while requiring much more computation. Based on these comprehensive comparisons, we conclude that complex-valued DNNs do not provide a performance gain over their real-valued counterparts for monaural speech enhancement, and thus are less desirable due to their higher computational costs.


【5】 Modelling low-resource accents without accent-specific TTS frontend

标题：对没有特定于重音的TTS前端的低资源重音进行建模

链接：https://arxiv.org/abs/2301.04606

作者：Georgi Tinchev,Marta Czarnowska,Kamil Deja,Kayoko Yanagisawa,Marius Cotescu
机构：Amazon Research, United Kingdom ,Warsaw University of Technology, Poland
备注：The first two authors contributed equally to this work. In Review. Samples available on this https URL
摘要：这项工作的重点是建模说话人的口音，没有一个专门的文本到语音（TTS）前端，包括一个字形到音素（G2P）模块。对口音建模的先前工作假设语音标音可用于目标口音，这对于低资源、地区口音可能不是这种情况。在我们的工作中，我们提出了一种方法，首先通过语音转换将目标口音数据增强到听起来像供体语音，然后结合录音和合成数据训练多说话者多口音TTS模型，以生成以目标口音说话的供体语音。在整个过程中，我们使用为同一种语言开发的TTS前端，但重音不同。我们显示定性和定量分析，其中提出的战略取得了最先进的结果相比，其他生成模型。我们的工作表明，低资源重音可以用相对较少的数据建模，而无需开发特定重音的TTS前端。我们的模型转换为多种口音的音频样本可在我们的网页上。
摘要：This work focuses on modelling a speaker's accent that does not have a dedicated text-to-speech (TTS) frontend, including a grapheme-to-phoneme (G2P) module. Prior work on modelling accents assumes a phonetic transcription is available for the target accent, which might not be the case for low-resource, regional accents. In our work, we propose an approach whereby we first augment the target accent data to sound like the donor voice via voice conversion, then train a multi-speaker multi-accent TTS model on the combination of recordings and synthetic data, to generate the donor's voice speaking in the target accent. Throughout the procedure, we use a TTS frontend developed for the same language but a different accent. We show qualitative and quantitative analysis where the proposed strategy achieves state-of-the-art results compared to other generative models. Our work demonstrates that low resource accents can be modelled with relatively little data and without developing an accent-specific TTS frontend. Audio samples of our model converting to multiple accents are available on our web page.
eess.AS音频处理

【1】 Modelling low-resource accents without accent-specific TTS frontend

标题：对没有特定于重音的TTS前端的低资源重音进行建模

链接：https://arxiv.org/abs/2301.04606

*与cs.SD语音【5】为同一篇

作者：Georgi Tinchev,Marta Czarnowska,Kamil Deja,Kayoko Yanagisawa,Marius Cotescu
机构：Amazon Research, United Kingdom ,Warsaw University of Technology, Poland
备注：The first two authors contributed equally to this work. In Review. Samples available on this https URL
摘要：这项工作的重点是建模说话人的口音，没有一个专门的文本到语音（TTS）前端，包括一个字形到音素（G2P）模块。对口音建模的先前工作假设语音标音可用于目标口音，这对于低资源、地区口音可能不是这种情况。在我们的工作中，我们提出了一种方法，首先通过语音转换将目标口音数据增强到听起来像供体语音，然后结合录音和合成数据训练多说话者多口音TTS模型，以生成以目标口音说话的供体语音。在整个过程中，我们使用为同一种语言开发的TTS前端，但重音不同。我们显示定性和定量分析，其中提出的战略取得了最先进的结果相比，其他生成模型。我们的工作表明，低资源重音可以用相对较少的数据建模，而无需开发特定重音的TTS前端。我们的模型转换为多种口音的音频样本可在我们的网页上。
摘要：This work focuses on modelling a speaker's accent that does not have a dedicated text-to-speech (TTS) frontend, including a grapheme-to-phoneme (G2P) module. Prior work on modelling accents assumes a phonetic transcription is available for the target accent, which might not be the case for low-resource, regional accents. In our work, we propose an approach whereby we first augment the target accent data to sound like the donor voice via voice conversion, then train a multi-speaker multi-accent TTS model on the combination of recordings and synthetic data, to generate the donor's voice speaking in the target accent. Throughout the procedure, we use a TTS frontend developed for the same language but a different accent. We show qualitative and quantitative analysis where the proposed strategy achieves state-of-the-art results compared to other generative models. Our work demonstrates that low resource accents can be modelled with relatively little data and without developing an accent-specific TTS frontend. Audio samples of our model converting to multiple accents are available on our web page.


【2】 Improving And Analyzing Neural Speaker Embeddings for ASR

标题：ASR中神经扬声器嵌入的改进与分析

链接：https://arxiv.org/abs/2301.04571

作者：Christoph Lüscher,Jingjing Xu,Mohammad Zeineldeen,Ralf Schlüter,Hermann Ney
机构：Human Language Technology and Pattern Recognition Group, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany
备注：Submitted to ICASSP 2023
摘要：神经说话人嵌入通过DNN模型对说话人的语音特征进行编码，在说话人确认任务中非常流行。然而，很少有研究探讨神经说话人嵌入在ASR系统中的应用。在这篇论文中，我们提出了我们的努力，将神经说话人嵌入整合到一个基于构象器的混合HMM ASR系统中。对于ASR，我们改进的嵌入提取流水线与加权简单相加积分方法相结合，使x向量和c向量达到与i向量相同的性能。我们进一步比较和分析了不同的说话人嵌入。我们介绍了我们的声学模型改进，通过从newbob学习速率计划切换到一个周期学习计划，使Switchboard上的相对WER减少约3%，另外将总体训练时间减少17%。通过进一步添加神经扬声器嵌入，我们在Hub5'00上获得了额外的约3%的相对WER改善。我们最好的基于Conformer的混合ASR系统（带有扬声器嵌入）在Hub5'00和Hub5'01上实现了9.0%的WER，并在SWB 300h上进行了训练。
摘要：Neural speaker embeddings encode the speaker's speech characteristics through a DNN model and are prevalent for speaker verification tasks. However, few studies have investigated the usage of neural speaker embeddings for an ASR system. In this work, we present our efforts w.r.t integrating neural speaker embeddings into a conformer based hybrid HMM ASR system. For ASR, our improved embedding extraction pipeline in combination with the Weighted-Simple-Add integration method results in x-vector and c-vector reaching on par performance with i-vectors. We further compare and analyze different speaker embeddings. We present our acoustic model improvements obtained by switching from newbob learning rate schedule to one cycle learning schedule resulting in a ~3% relative WER reduction on Switchboard, additionally reducing the overall training time by 17%. By further adding neural speaker embeddings, we gain additional ~3% relative WER improvement on Hub5'00. Our best Conformer-based hybrid ASR system with speaker embeddings achieves 9.0% WER on Hub5'00 and Hub5'01 with training on SWB 300h.


【3】 WuYun: Exploring hierarchical skeleton-guided melody generation using  knowledge-enhanced deep learning

标题：乌云：利用知识增强型深度学习探索层次化骨架引导旋律生成

链接：https://arxiv.org/abs/2301.04488

*与cs.SD语音【1】为同一篇

作者：Kejun Zhang,Xinda Wu,Tieyao Zhang,Zhijie Huang,Xu Tan,Qihao Liang,Songruoyao Wu,Lingyun Sun
机构：College of Computer Science and Technology, Zhejiang University, China., Alibaba-Zhejiang University Joint Institute of Frontier Technologies, China., Innovation Center of Yangtze River Delta, China., Microsoft Research Asia
摘要：尽管深度学习已经彻底改变了音乐生成，但现有的结构化旋律生成方法遵循端到端、从左到右、逐个音符的生成范式，并平等地对待每个音符。在此，我们介绍WuYun，一个用于改进生成旋律结构的知识增强型深度学习架构，它首先生成结构上最重要的音符来构建旋律骨架，然后用动态装饰音符填充到完整的旋律中。具体而言，利用音乐领域知识提取旋律骨架，并采用序列学习重构旋律骨架，作为附加知识为旋律生成过程提供辅助指导。我们证明了五云算法能够生成具有更好的长期结构和音乐性的旋律，并且在所有主观评价指标上平均比其他最先进的方法高出0.51。我们的研究提供了一个多学科的视角来设计旋律层次结构，并在众多音乐生成任务中弥合数据驱动和基于知识的方法之间的差距。
摘要：Although deep learning has revolutionized music generation, existing methods for structured melody generation follow an end-to-end left-to-right note-by-note generative paradigm and treat each note equally. Here, we present WuYun, a knowledge-enhanced deep learning architecture for improving the structure of generated melodies, which first generates the most structurally important notes to construct a melodic skeleton and subsequently infills it with dynamically decorative notes into a full-fledged melody. Specifically, we use music domain knowledge to extract melodic skeletons and employ sequence learning to reconstruct them, which serve as additional knowledge to provide auxiliary guidance for the melody generation process. We demonstrate that WuYun can generate melodies with better long-term structure and musicality and outperforms other state-of-the-art methods by 0.51 on average on all subjective evaluation metrics. Our study provides a multidisciplinary lens to design melodic hierarchical structures and bridge the gap between data-driven and knowledge-based approaches for numerous music generation tasks.


【4】 Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

标题：基于音频条件扩散模型的语音驱动视频编辑

链接：https://arxiv.org/abs/2301.04474

*与cs.SD语音【2】为同一篇

作者：Dan Bigioi,Shubhajit Basak,Hugh Jordan,Rachel McDonnell,Peter Corcoran
机构：University of Galway, Trinity College Dublin
备注：8 Pages
摘要：本文提出了一种基于去噪扩散模型的端到端语音驱动视频编辑方法。给定一个人说话的视频，我们的目标是响应于单独的听觉语音记录来重新同步该人的嘴唇和下巴运动，而不依赖于诸如面部界标或3D面部模型之类的中间结构表示。我们证明了这是可能的条件去噪扩散模型与音频频谱特征，以产生同步的面部运动。我们在非结构化的单说话人视频编辑任务上取得了令人信服的结果，使用现成的唇读模型实现了45%的单词错误率。我们进一步展示了如何将我们的方法扩展到多说话人领域。据我们所知，这是第一个探索将去噪扩散模型应用于音频驱动的视频编辑任务的可行性的工作。
摘要：In this paper we propose a method for end-to-end speech driven video editing using a denoising diffusion model. Given a video of a person speaking, we aim to re-synchronise the lip and jaw motion of the person in response to a separate auditory speech recording without relying on intermediate structural representations such as facial landmarks or a 3D face model. We show this is possible by conditioning a denoising diffusion model with audio spectral features to generate synchronised facial motion. We achieve convincing results on the task of unstructured single-speaker video editing, achieving a word error rate of 45% using an off the shelf lip reading model. We further demonstrate how our approach can be extended to the multi-speaker domain. To our knowledge, this is the first work to explore the feasibility of applying denoising diffusion models to the task of audio-driven video editing.


【5】 Perceive and predict: self-supervised speech representation based loss  functions for speech enhancement

标题：感知和预测：基于自监督语音表示的损失函数语音增强

链接：https://arxiv.org/abs/2301.04388

*与cs.SD语音【3】为同一篇

作者：George Close,William Ravenscroft,Thomas Hain,Stefan Goetze
机构：Department of Computer Science, University of Sheffield
备注：4 pages, submitted to ICASSP 2023
摘要：在语音增强领域中的最近工作已经探索了使用自监督语音表示来帮助训练神经语音增强模型。然而，这些工作的大部分集中于使用自监督语音表示模型的最深或最终输出，而不是早期的特征编码。以这样的方式使用自监督表示通常不是完全有动机的。在这项工作中，表明干净语音和含噪语音的特征编码之间的距离与语音质量和可懂度的心理声学动机测量以及人类平均意见得分（MOS）评级密切相关。执行使用该距离作为损失函数的实验，并且使用诸如语音质量的感知评估（PESQ）和短时客观可懂度（STOI）的客观测量来证明相对于使用基于STFT谱图距离的损失以及来自语音增强文献的其他常见损失函数的改进性能。
摘要：Recent work in the domain of speech enhancement has explored the use of self-supervised speech representations to aid in the training of neural speech enhancement models. However, much of this work focuses on using the deepest or final outputs of self supervised speech representation models, rather than the earlier feature encodings. The use of self supervised representations in such a way is often not fully motivated. In this work it is shown that the distance between the feature encodings of clean and noisy speech correlate strongly with psychoacoustically motivated measures of speech quality and intelligibility, as well as with human Mean Opinion Score (MOS) ratings. Experiments using this distance as a loss function are performed and improved performance over the use of STFT spectrogram distance based loss as well as other common loss functions from speech enhancement literature is demonstrated using objective measures such as perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI).


【6】 Rethinking complex-valued deep neural networks for monaural speech  enhancement

标题：复值深度神经网络用于单声道语音增强的再思考

链接：https://arxiv.org/abs/2301.04320

*与cs.SD语音【4】为同一篇

作者：Haibin Wu,Ke Tan,Buye Xu,Anurag Kumar,Daniel Wong
机构：Meta Reality Labs Research, USA
摘要：尽管为采用复值深度神经网络（DNN）进行了多种努力，但对于单耳语音增强，复值DNN是否通常比实值DNN更有效仍然是一个开放的问题。这项工作致力于提出一个关键的评估，通过系统地审查复值DNN对他们的实际对应。具体来说，我们研究了复值DNN原子单元，包括线性层、卷积层、长短时记忆（LSTM）和门控线性单元。通过比较最近开发的门控卷积递归网络（GCRN）中基本构建块的复数和实值版本，我们展示了基本块的不同机制如何影响性能。我们还发现，当模型规模小时，复值运算的使用阻碍了模型的容量。此外，我们还研究了两种最新的复值DNN，即深度复卷积递归网络（DCCRN）和深度复U-Net（DCUNET）。仿真结果表明，两种DNN的性能与实值DNN相当，但计算量大得多。基于这些全面的比较，我们得出结论，对于单耳语音增强，复值DNN并不提供超过其实值对应物的性能增益，因此由于其较高的计算成本而不太理想。
摘要：Despite multiple efforts made towards adopting complex-valued deep neural networks (DNNs), it remains an open question whether complex-valued DNNs are generally more effective than real-valued DNNs for monaural speech enhancement. This work is devoted to presenting a critical assessment by systematically examining complex-valued DNNs against their real-valued counterparts. Specifically, we investigate complex-valued DNN atomic units, including linear layers, convolutional layers, long short-term memory (LSTM), and gated linear units. By comparing complex- and real-valued versions of fundamental building blocks in the recently developed gated convolutional recurrent network (GCRN), we show how different mechanisms for basic blocks affect the performance. We also find that the use of complex-valued operations hinders the model capacity when the model size is small. In addition, we examine two recent complex-valued DNNs, i.e. deep complex convolutional recurrent network (DCCRN) and deep complex U-Net (DCUNET). Evaluation results show that both DNNs produce identical performance to their real-valued counterparts while requiring much more computation. Based on these comprehensive comparisons, we conclude that complex-valued DNNs do not provide a performance gain over their real-valued counterparts for monaural speech enhancement, and thus are less desirable due to their higher computational costs.

机器翻译，仅供参考