今日论文合集：cs.SD语音4篇，eess.AS音频处理4篇。
【1】 A Self-Adjusting Fusion Representation Learning Model for Unaligned Text-Audio Sequences
标题：一种自调整的非对齐文本-音频序列融合表征学习模型
链接：https://arxiv.org/abs/2212.11772
作者：Kaicheng Yang,Ruxuan Zhang,Hua Xu,Kai Gao机构：State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science, and Technology, Tsinghua University, Beijing , China, School of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang , China摘要：模态间交互在多模态情感分析中起着不可或缺的作用。由于不同模态序列通常是不对齐的，如何整合各模态的相关信息以学习融合表示一直是多模态学习的核心挑战之一。本文提出了一种自调整融合表示学习模型（SA-FRLM），直接从未对齐的文本和音频序列中学习鲁棒的跨模态融合表示。与以往的研究不同，该模型不仅充分利用了不同模态之间的相互作用，而且最大限度地保护了模态的单模态特性。具体地说，我们首先采用跨模态对准模块将不同模态特征投影到同一维度。然后采用跨模态协作注意力对文本和音频序列的跨模态交互进行建模，并初始化融合表示。在此基础上，提出了以交叉模式调整Transformer作为SA-FRLM的核心单元，以保护原有的单峰特性。它可以通过使用单模态流来动态地适应融合表示。我们在公开的多模态情感分析数据集CMU-MOSI和CMU-MOSEI上评估了我们的方法。实验结果表明，该模型在未对齐文本-音频序列上的性能有了显著提高。摘要：Inter-modal interaction plays an indispensable role in multimodal sentiment analysis. Due to different modalities sequences are usually non-alignment, how to integrate relevant information of each modality to learn fusion representations has been one of the central challenges in multimodal learning. In this paper, a Self-Adjusting Fusion Representation Learning Model (SA-FRLM) is proposed to learn robust crossmodal fusion representations directly from the unaligned text and audio sequences. Different from previous works, our model not only makes full use of the interaction between different modalities but also maximizes the protection of the unimodal characteristics. Specifically, we first employ a crossmodal alignment module to project different modalities features to the same dimension. The crossmodal collaboration attention is then adopted to model the inter-modal interaction between text and audio sequences and initialize the fusion representations. After that, as the core unit of the SA-FRLM, the crossmodal adjustment transformer is proposed to protect original unimodal characteristics. It can dynamically adapt the fusion representations by using single modal streams. We evaluate our approach on the public multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results show that our model has significantly improved the performance of all the metrics on the unaligned text-audio sequences.
【2】 Audio Denoising for Robust Audio Fingerprinting
标题：用于稳健音频指纹识别的音频去噪方法
链接：https://arxiv.org/abs/2212.11277
机构：Tutors, Benjamin Martin, Dorian Desblancs, Referent teacher, Gaël Richard, arXiv:,.,v, [cs.SD] , Dec备注：63 pages, master thesis摘要：音乐发现服务允许用户从短的移动记录中识别歌曲。这些解决方案通常基于音频指纹识别，并且更具体地依赖于频谱峰值的提取以便对许多失真具有鲁棒性。很少有人研究这些算法对真实环境中捕获的背景噪声的鲁棒性。特别地，当信噪比低时，即当背景噪声强时，AFP系统仍然很困难。在这个项目中，我们通过深度学习来解决这个问题。我们测试了一种新的混合策略，该策略包括在基于峰值的AFP算法之前插入去噪DL模型。我们使用一个真实的数据增强管道来模拟有噪声的音乐录音，并训练一个DL模型来对它们进行降噪。去噪模型限制了背景噪声对AFP系统提取峰的影响，提高了其对噪声的鲁棒性。我们进一步提出了一个新的损失函数，以适应DL模型考虑AFP系统，提高其精度方面的检索光谱峰值。据我们所知，这种混合战略以前没有经过测试。摘要：Music discovery services let users identify songs from short mobile recordings. These solutions are often based on Audio Fingerprinting, and rely more specifically on the extraction of spectral peaks in order to be robust to a number of distortions. Few works have been done to study the robustness of these algorithms to background noise captured in real environments. In particular, AFP systems still struggle when the signal to noise ratio is low, i.e when the background noise is strong. In this project, we tackle this problematic with Deep Learning. We test a new hybrid strategy which consists of inserting a denoising DL model in front of a peak-based AFP algorithm. We simulate noisy music recordings using a realistic data augmentation pipeline, and train a DL model to denoise them. The denoising model limits the impact of background noise on the AFP system's extracted peaks, improving its robustness to noise. We further propose a novel loss function to adapt the DL model to the considered AFP system, increasing its precision in terms of retrieved spectral peaks. To the best of our knowledge, this hybrid strategy has not been tested before.
【3】 StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
标题：STORM：一种基于扩散的语音增强和去混响随机再生模型
链接：https://arxiv.org/abs/2212.11851
作者：Jean-Marie Lemercier,Julius Richter,Simon Welker,Timo Gerkmann机构：Student Member, IEEE, Julius, Richter, Senior Member, IEEE备注：This work has been submitted to the IEEE for publication. Copyright may be transferred without notice摘要：扩散模型在语音增强的预测方法和生成方法之间的性能差距上表现出了很大的能力。我们已经证明，对于非累加性损坏类型，或者在不匹配的条件下进行评估时，它们甚至可能优于其预测性对应物。然而，扩散模型遭受高计算负担，主要是因为它们需要为每个反扩散步骤运行神经网络，而预测方法仅需要一遍。由于扩散模型是生成方法，它们也可能在不利条件下产生发声和呼吸伪影。相比之下，在这种困难的场景中，预测模型通常不产生这种伪像，而是倾向于使目标语音失真，从而降低语音质量。在这项工作中，我们提出了一种随机再生方法，其中预测模型给出的估计值作为进一步扩散的指导。我们表明，所提出的方法使用预测模型来消除发声和呼吸伪影，同时由于扩散模型而产生非常高质量的样本，即使在不利条件下。我们进一步表明，这种方法能够使用更轻的采样方案与更少的扩散步骤，而不牺牲质量，从而提高了一个数量级的计算负担。源代码和音频示例可在线获取（https：//uhh.de/inf-sp-storm）。摘要：Diffusion models have shown a great ability at bridging the performance gap between predictive and generative approaches for speech enhancement. We have shown that they may even outperform their predictive counterparts for non-additive corruption types or when they are evaluated on mismatched conditions. However, diffusion models suffer from a high computational burden, mainly as they require to run a neural network for each reverse diffusion step, whereas predictive approaches only require one pass. As diffusion models are generative approaches they may also produce vocalizing and breathing artifacts in adverse conditions. In comparison, in such difficult scenarios, predictive models typically do not produce such artifacts but tend to distort the target speech instead, thereby degrading the speech quality. In this work, we present a stochastic regeneration approach where an estimate given by a predictive model is provided as a guide for further diffusion. We show that the proposed approach uses the predictive model to remove the vocalizing and breathing artifacts while producing very high quality samples thanks to the diffusion model, even in adverse conditions. We further show that this approach enables to use lighter sampling schemes with fewer diffusion steps without sacrificing quality, thus lifting the computational burden by an order of magnitude. Source code and audio examples are available online (https://uhh.de/inf-sp-storm).
【4】 ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
标题：REVISE：具有视觉输入的自监督语音重新合成，用于通用和通用语音增强
链接：https://arxiv.org/abs/2212.11377
作者：Wei-Ning Hsu,Tal Remez,Bowen Shi,Jacob Donley,Yossi Adi机构：FAIR, Meta AI Research, Meta Reality Labs Research, Toyota Technological Institute at Chicago, The Hebrew University of Jerusalem摘要：关于利用视觉输入改善语音质量的现有工作通常单独研究每种类型的听觉失真（例如，分离、修补、视频到语音）并呈现定制的算法。本文提出将这些学科结合起来，研究广义语音增强，其目标不是重建精确的参考干净信号，而是集中于改善语音的某些方面。本文特别关注可懂度、质量和视频同步。我们把这个问题描述为音视频语音再合成，它由两个步骤组成：伪音视频语音识别（P-AVSR）和伪文本到语音合成（P-TTS）。P-AVSR和P-TTS由从自监督语音模型导出的离散单元连接。此外，我们利用自监督音视频语音模型来初始化P-AVSR。所提出的模型被称为ReVISE。ReVISE是第一款用于野外视频语音合成的高质量模型，可通过单一模型在所有LRS 3视听增强任务中实现卓越性能。为了证明其在现实世界中的适用性，还在EasyCom上对ReVISE进行了评估，EasyCom是一个在具有挑战性的声学条件下收集的视听基准，只有1.6小时的训练数据。类似地，ReVISE极大地抑制了噪声并提高了质量。项目页面：https://wnhsu.github.io/ReVISE。摘要：Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present tailored algorithms. This paper proposes to unify these subjects and study Generalized Speech Enhancement, where the goal is not to reconstruct the exact reference clean signal, but to focus on improving certain aspects of speech. In particular, this paper concerns intelligibility, quality, and video synchronization. We cast the problem as audio-visual speech resynthesis, which is composed of two steps: pseudo audio-visual speech recognition (P-AVSR) and pseudo text-to-speech synthesis (P-TTS). P-AVSR and P-TTS are connected by discrete units derived from a self-supervised speech model. Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR. The proposed model is coined ReVISE. ReVISE is the first high-quality model for in-the-wild video-to-speech synthesis and achieves superior performance on all LRS3 audio-visual enhancement tasks with a single model. To demonstrates its applicability in the real world, ReVISE is also evaluated on EasyCom, an audio-visual benchmark collected under challenging acoustic conditions with only 1.6 hours of training data. Similarly, ReVISE greatly suppresses noise and improves quality. Project page: https://wnhsu.github.io/ReVISE.
【1】 HMM-based data augmentation for E2E systems for building conversational speech synthesis systems
标题：基于隐马尔可夫模型的E2E系统数据扩充以构建会话语音合成系统
链接：https://arxiv.org/abs/2212.11982
作者：Ishika Gupta,Anusha Prakash,Jom Kuriakose,Hema A. Murthy机构：Department of Computer Science & Engineering, Indian Institute of Technology Madras, India, Department of Electrical Engineering, Indian Institute of Technology Madras, India备注：6 pages, 7 figures, 33 references摘要：本文提出了一种利用数据增强技术为技术领域构建高质量文本到语音（TTS）系统的方法。端到端（E2E）系统在基于隐马尔可夫模型（HMM）的合成语音上被训练，并且进一步用录音室记录的TTS数据来微调以改善合成语音的音色。这项工作背后的动机是，由于HMM系统能够准确地模拟音素的持续时间分布，因此通常不存在单词跳过和重复的问题。上下文相关的五声建模，以及基于树的聚类和状态绑定，处理看不见的上下文和词汇表外的单词。为了进一步减少合成误差，还采用了语言模型。主观评估表明，当组合来自HMM和E2E框架的补充属性时，使用所提出的系统产生的语音在可懂度方面优于基线E2E合成方法。进一步的分析突出了所提议的方法在低资源情景中的功效。摘要：This paper proposes an approach to build a high-quality text-to-speech (TTS) system for technical domains using data augmentation. An end-to-end (E2E) system is trained on hidden Markov model (HMM) based synthesized speech and further fine-tuned with studio-recorded TTS data to improve the timbre of the synthesized voice. The motivation behind the work is that issues of word skips and repetitions are usually absent in HMM systems due to their ability to model the duration distribution of phonemes accurately. Context-dependent pentaphone modeling, along with tree-based clustering and state-tying, takes care of unseen context and out-of-vocabulary words. A language model is also employed to reduce synthesis errors further. Subjective evaluations indicate that speech produced using the proposed system is superior to the baseline E2E synthesis approach in terms of intelligibility when combining complementing attributes from HMM and E2E frameworks. The further analysis highlights the proposed approach's efficacy in low-resource scenarios.
【2】 StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
标题：STORM：一种基于扩散的语音增强和去混响随机再生模型
链接：https://arxiv.org/abs/2212.11851
*与cs.SD语音【3】为同一篇
作者：Jean-Marie Lemercier,Julius Richter,Simon Welker,Timo Gerkmann机构：Student Member, IEEE, Julius, Richter, Senior Member, IEEE备注：This work has been submitted to the IEEE for publication. Copyright may be transferred without notice摘要：扩散模型在语音增强的预测方法和生成方法之间的性能差距上表现出了很大的能力。我们已经证明，对于非累加性损坏类型，或者在不匹配的条件下进行评估时，它们甚至可能优于其预测性对应物。然而，扩散模型遭受高计算负担，主要是因为它们需要为每个反扩散步骤运行神经网络，而预测方法仅需要一遍。由于扩散模型是生成方法，它们也可能在不利条件下产生发声和呼吸伪影。相比之下，在这种困难的场景中，预测模型通常不产生这种伪像，而是倾向于使目标语音失真，从而降低语音质量。在这项工作中，我们提出了一种随机再生方法，其中预测模型给出的估计值作为进一步扩散的指导。我们表明，所提出的方法使用预测模型来消除发声和呼吸伪影，同时由于扩散模型而产生非常高质量的样本，即使在不利条件下。我们进一步表明，这种方法能够使用更轻的采样方案与更少的扩散步骤，而不牺牲质量，从而提高了一个数量级的计算负担。源代码和音频示例可在线获取（https：//uhh.de/inf-sp-storm）。摘要：Diffusion models have shown a great ability at bridging the performance gap between predictive and generative approaches for speech enhancement. We have shown that they may even outperform their predictive counterparts for non-additive corruption types or when they are evaluated on mismatched conditions. However, diffusion models suffer from a high computational burden, mainly as they require to run a neural network for each reverse diffusion step, whereas predictive approaches only require one pass. As diffusion models are generative approaches they may also produce vocalizing and breathing artifacts in adverse conditions. In comparison, in such difficult scenarios, predictive models typically do not produce such artifacts but tend to distort the target speech instead, thereby degrading the speech quality. In this work, we present a stochastic regeneration approach where an estimate given by a predictive model is provided as a guide for further diffusion. We show that the proposed approach uses the predictive model to remove the vocalizing and breathing artifacts while producing very high quality samples thanks to the diffusion model, even in adverse conditions. We further show that this approach enables to use lighter sampling schemes with fewer diffusion steps without sacrificing quality, thus lifting the computational burden by an order of magnitude. Source code and audio examples are available online (https://uhh.de/inf-sp-storm).
【3】 ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
标题：REVISE：具有视觉输入的自监督语音重新合成，用于通用和通用语音增强
链接：https://arxiv.org/abs/2212.11377
*与cs.SD语音【4】为同一篇
作者：Wei-Ning Hsu,Tal Remez,Bowen Shi,Jacob Donley,Yossi Adi机构：FAIR, Meta AI Research, Meta Reality Labs Research, Toyota Technological Institute at Chicago, The Hebrew University of Jerusalem摘要：关于利用视觉输入改善语音质量的现有工作通常单独研究每种类型的听觉失真（例如，分离、修补、视频到语音）并呈现定制的算法。本文提出将这些学科结合起来，研究广义语音增强，其目标不是重建精确的参考干净信号，而是集中于改善语音的某些方面。本文特别关注可懂度、质量和视频同步。我们把这个问题描述为音视频语音再合成，它由两个步骤组成：伪音视频语音识别（P-AVSR）和伪文本到语音合成（P-TTS）。P-AVSR和P-TTS由从自监督语音模型导出的离散单元连接。此外，我们利用自监督音视频语音模型来初始化P-AVSR。所提出的模型被称为ReVISE。ReVISE是第一款用于野外视频语音合成的高质量模型，可通过单一模型在所有LRS 3视听增强任务中实现卓越性能。为了证明其在现实世界中的适用性，还在EasyCom上对ReVISE进行了评估，EasyCom是一个在具有挑战性的声学条件下收集的视听基准，只有1.6小时的训练数据。类似地，ReVISE极大地抑制了噪声并提高了质量。项目页面：https://wnhsu.github.io/ReVISE。摘要：Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present tailored algorithms. This paper proposes to unify these subjects and study Generalized Speech Enhancement, where the goal is not to reconstruct the exact reference clean signal, but to focus on improving certain aspects of speech. In particular, this paper concerns intelligibility, quality, and video synchronization. We cast the problem as audio-visual speech resynthesis, which is composed of two steps: pseudo audio-visual speech recognition (P-AVSR) and pseudo text-to-speech synthesis (P-TTS). P-AVSR and P-TTS are connected by discrete units derived from a self-supervised speech model. Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR. The proposed model is coined ReVISE. ReVISE is the first high-quality model for in-the-wild video-to-speech synthesis and achieves superior performance on all LRS3 audio-visual enhancement tasks with a single model. To demonstrates its applicability in the real world, ReVISE is also evaluated on EasyCom, an audio-visual benchmark collected under challenging acoustic conditions with only 1.6 hours of training data. Similarly, ReVISE greatly suppresses noise and improves quality. Project page: https://wnhsu.github.io/ReVISE.
【4】 A Self-Adjusting Fusion Representation Learning Model for Unaligned Text-Audio Sequences
标题：一种自调整的非对齐文本-音频序列融合表征学习模型
链接：https://arxiv.org/abs/2212.11772
*与cs.SD语音【1】为同一篇
作者：Kaicheng Yang,Ruxuan Zhang,Hua Xu,Kai Gao机构：State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science, and Technology, Tsinghua University, Beijing , China, School of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang , China摘要：模态间交互在多模态情感分析中起着不可或缺的作用。由于不同模态序列通常是不对齐的，如何整合各模态的相关信息以学习融合表示一直是多模态学习的核心挑战之一。本文提出了一种自调整融合表示学习模型（SA-FRLM），直接从未对齐的文本和音频序列中学习鲁棒的跨模态融合表示。与以往的研究不同，该模型不仅充分利用了不同模态之间的相互作用，而且最大限度地保护了模态的单模态特性。具体地说，我们首先采用跨模态对准模块将不同模态特征投影到同一维度。然后采用跨模态协作注意力对文本和音频序列的跨模态交互进行建模，并初始化融合表示。在此基础上，提出了以交叉模式调整Transformer作为SA-FRLM的核心单元，以保护原有的单峰特性。它可以通过使用单模态流来动态地适应融合表示。我们在公开的多模态情感分析数据集CMU-MOSI和CMU-MOSEI上评估了我们的方法。实验结果表明，该模型在未对齐文本-音频序列上的性能有了显著提高。摘要：Inter-modal interaction plays an indispensable role in multimodal sentiment analysis. Due to different modalities sequences are usually non-alignment, how to integrate relevant information of each modality to learn fusion representations has been one of the central challenges in multimodal learning. In this paper, a Self-Adjusting Fusion Representation Learning Model (SA-FRLM) is proposed to learn robust crossmodal fusion representations directly from the unaligned text and audio sequences. Different from previous works, our model not only makes full use of the interaction between different modalities but also maximizes the protection of the unimodal characteristics. Specifically, we first employ a crossmodal alignment module to project different modalities features to the same dimension. The crossmodal collaboration attention is then adopted to model the inter-modal interaction between text and audio sequences and initialize the fusion representations. After that, as the core unit of the SA-FRLM, the crossmodal adjustment transformer is proposed to protect original unimodal characteristics. It can dynamically adapt the fusion representations by using single modal streams. We evaluate our approach on the public multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results show that our model has significantly improved the performance of all the metrics on the unaligned text-audio sequences.
机器翻译，仅供参考
每日语音速递