【1】 Detecting Synthetic Speech Manipulation in Real Audio Recordings
标题：检测真实录音中的合成语音操作
链接：https://arxiv.org/abs/2209.07498
作者：Md Hafizur Rahman,Martin Graciarena,Diego Castan,Chris Cobo-Kroenke,Mitchell McLaren,Aaron Lawson机构：STAR Lab, SRI International备注：Submitted to IEEE International Workshop on Information Forensics and Security (WIFS)摘要：人工语音和音频技术的最新进展提高了深度伪造运营商伪造媒体和传播恶意错误信息的能力。任何编码技能有限的人都可以使用免费提供的语音合成工具来创建有影响力的说话者的声音的令人信服的模拟，其恶意意图是扭曲原始信息。利用最新的技术，恶意操作者不必生成整个音频剪辑;相反，他们可以将部分操作或一段合成语音插入到真正的音频记录中，以改变原始消息的整个上下文和含义。检测这些插入尤其具有挑战性，因为部分处理的音频比完全伪造的消息更容易避开合成语音检测器。本文描述了一个潜在的部分合成语音检测系统，该系统基于x-ResNet架构，具有概率线性判别分析（PLDA）后端和交织感知评分处理。实验结果表明，与非PLDA基线相比，PLDA后端导致部分合成数据集之间的平均错误减少25%。摘要：Recent advances in artificial speech and audio technologies have improved the abilities of deep-fake operators to falsify media and spread malicious misinformation. Anyone with limited coding skills can use freely available speech synthesis tools to create convincing simulations of influential speakers' voices with the malicious intent to distort the original message. With the latest technology, malicious operators do not have to generate an entire audio clip; instead, they can insert a partial manipulation or a segment of synthetic speech into a genuine audio recording to change the entire context and meaning of the original message. Detecting these insertions is especially challenging because partially manipulated audio can more easily avoid synthetic speech detectors than entirely fake messages can. This paper describes a potential partial synthetic speech detection system based on the x-ResNet architecture with a probabilistic linear discriminant analysis (PLDA) backend and interleaved aware score processing. Experimental results suggest that the PLDA backend results in a 25% average error reduction among partially synthesized datasets over a non-PLDA baseline.
【2】 CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations
标题：CMSBERT-CLR：语境驱动的情态转换BERT与语言、视觉、声音表征的对比学习
链接：https://arxiv.org/abs/2209.07424
机构：Department of Artificial Intelligence, Dongguk University, Seoul, Korea备注：Accepted by IJCNN 2022摘要：随着对多模态在线内容需求的不断增长，多模态情感分析已经成为一个日益流行的研究领域。在多模态情感分析中，词语会因语境和非言语信息的不同而具有不同的含义，因此理解词语的含义至关重要。此外，词义的解释还应放在包括非语言信息在内的整个话语语境中进行。本文提出了一种基于对比学习的语境驱动的语言、视觉、听觉表征的模态转换BERT（CMSBERT-CLR）算法，该算法综合了整个语境中的非言语和言语信息，通过对比学习更有效地实现模态转换。首先，我们引入了语境驱动的情态转换（CMS），将非言语和言语信息整合到句子话语的整个语境中。然后，为了改善不同模态在公共嵌入空间内的对齐，我们应用对比学习。此外，采用指数滑动平均参数和标签平滑作为优化策略，使网络的收敛更加稳定，增加了比对的灵活性。在我们的实验中，我们证明了我们的方法达到了最先进的结果。摘要：Multimodal sentiment analysis has become an increasingly popular research area as the demand for multimodal online content is growing. For multimodal sentiment analysis, words can have different meanings depending on the linguistic context and non-verbal information, so it is crucial to understand the meaning of the words accordingly. In addition, the word meanings should be interpreted within the whole utterance context that includes nonverbal information. In this paper, we present a Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations (CMSBERT-CLR), which incorporates the whole context's non-verbal and verbal information and aligns modalities more effectively through contrastive learning. First, we introduce a Context-driven Modality Shifting (CMS) to incorporate the non-verbal and verbal information within the whole context of the sentence utterance. Then, for improving the alignment of different modalities within a common embedding space, we apply contrastive learning. Furthermore, we use an exponential moving average parameter and label smoothing as optimization strategies, which can make the convergence of the network more stable and increase the flexibility of the alignment. In our experiments, we demonstrate that our approach achieves state-of-the-art results.
【3】 Self-Supervised Attention Networks and Uncertainty Loss Weighting for Multi-Task Emotion Recognition on Vocal Bursts
标题：基于自监督注意网络和不确定性损失加权的突发语音多任务情绪识别
链接：https://arxiv.org/abs/2209.07384
作者：Vincent Karas,Andreas Triantafyllopoulos,Meishu Song,Björn W. Schuller机构：University of Augsburg BMW Group, University of Augsburg University of Tokyo, Augsburg, Germany Tokyo, Japan, th Bj¨orn W. Schuller, Chair EIHW & ZIG GLAM, Unversity of Augsburg Imperial College London, Augsburg, Germany London, UK备注：4 pages, 1 figure, submitted to The 2022 ACII Affective Vocal Burst Workshop & Challenge (A-VB)摘要：突发性发声在情感交流中起着重要作用，对提高言语情感识别具有重要价值。在这里，我们介绍了我们的方法，分类发声爆发和预测他们的情感意义的ACII情感发声爆发研讨会和挑战2022（A-VB）。我们使用一个大型的自监督音频模型作为共享特征提取器，并比较了基于分类器链和注意力网络的多种结构，结合不确定性损失加权策略。在所有四项任务中，我们的方法都远远超过了挑战基线。摘要：Vocal bursts play an important role in communicating affect, making them valuable for improving speech emotion recognition. Here, we present our approach for classifying vocal bursts and predicting their emotional significance in the ACII Affective Vocal Burst Workshop & Challenge 2022 (A-VB). We use a large self-supervised audio model as shared feature extractor and compare multiple architectures built on classifier chains and attention networks, combined with uncertainty loss weighting strategies. Our approach surpasses the challenge baseline by a wide margin on all four tasks.
【4】 MVNet: Memory Assistance and Vocal Reinforcement Network for Speech Enhancement
标题：MVNet：语音增强的记忆辅助和语音增强网络
链接：https://arxiv.org/abs/2209.07302
作者：Jianrong Wang,Xiaomin Li,Xuewei Li,Mei Yu,Qiang Fang,Li Liu机构：College of Intelligence and Computing, Tianjin University, Tianjin, China, Institute of Linguistics, Chinese Academy of Social Sciences, Beijing, China, Shenzhen Research Institute of Big Data, the Chinese University of Hong Kong, Shenzhen, China摘要：语音增强提高了语音质量，并促进了各种下游任务的性能。然而，目前的语音增强技术主要致力于提高下游自动语音识别（ASR）的性能，只有相对少量的工作集中在自动说话人确认（ASV）任务上。本文提出了一个MVNet，该MVNet由记忆辅助模块和语音增强模块组成，记忆辅助模块提高了下游ASR的性能，语音增强模块提高了ASV的性能。此外，我们设计了一个新的损失函数来提高说话人声音的相似度。在Libri2mix数据集上的实验结果表明，该方法在语音质量、可懂度、说话人声音相似度等多个指标上均优于基线方法。摘要：Speech enhancement improves speech quality and promotes the performance of various downstream tasks. However, most current speech enhancement work was mainly devoted to improving the performance of downstream automatic speech recognition (ASR), only a relatively small amount of work focused on the automatic speaker verification (ASV) task. In this work, we propose a MVNet consisted of a memory assistance module which improves the performance of downstream ASR and a vocal reinforcement module which boosts the performance of ASV. In addition, we design a new loss function to improve speaker vocal similarity. Experimental results on the Libri2mix dataset show that our method outperforms baseline methods in several metrics, including speech quality, intelligibility, and speaker vocal similarity et al.
【5】 Domain Adversarial Training on Conditional Variational Auto-Encoder for Controllable Music Generation
标题：用于可控音乐生成的条件变分自动编码器的领域对抗性训练
链接：https://arxiv.org/abs/2209.07144
作者：Jingwei Zhao,Gus Xia,Ye Wang机构：School of Computing, NUS, Institute of Data Science, NUS, Music X Lab, NYU Shanghai, Integrative Sciences and Engineering Programme, NUS Graduate School, MBZUAI备注：Accepted by ISMIR 2022摘要：变分自动编码器已经成为符号音乐生成的主要框架，如何有效地控制符号音乐的生成过程是当前的研究热点。一种直接的方法是在推理期间使用不同的条件来控制模型。然而，在音乐实践中，条件通常是连续的（而不是简单的分类标签），涉及与学习的表征重叠的丰富信息。因此，解码器对于是“监听”潜在表示还是该条件感到困惑，并且有时只是忽略该条件。为了解决这个问题，我们利用领域对抗训练来将表征与条件线索分离，以便更好地控制。具体地，我们提出了一种条件破坏目标，其使用该表示来对破坏的条件进行去噪。通过鉴别器最小化并且通过VAE编码器最大化，该目标不利地诱导条件不变表示。本文以旋律和声任务为例来说明我们的思想，同时我们的方法也可以推广到其他可控的生成任务。演示和实验表明，该方法不仅有利于条件不变的表示学习，而且与基线相比具有更高质量的可控性.摘要：The variational auto-encoder has become a leading framework for symbolic music generation, and a popular research direction is to study how to effectively control the generation process. A straightforward way is to control a model using different conditions during inference. However, in music practice, conditions are usually sequential (rather than simple categorical labels), involving rich information that overlaps with the learned representation. Consequently, the decoder gets confused about whether to "listen to" the latent representation or the condition, and sometimes just ignores the condition. To solve this problem, we leverage domain adversarial training to disentangle the representation from condition cues for better control. Specifically, we propose a condition corruption objective that uses the representation to denoise a corrupted condition. Minimized by a discriminator and maximized by the VAE encoder, this objective adversarially induces a condition-invariant representation. In this paper, we focus on the task of melody harmonization to illustrate our idea, while our methodology can be generalized to other controllable generative tasks. Demos and experiments show that our methodology facilitates not only condition-invariant representation learning but also higher-quality controllability compared to baselines.
【6】 Beat Transformer: Demixed Beat and Downbeat Tracking with Dilated Self-Attention
标题：节拍Transformer：分流节拍和消沉节拍跟踪和自我关注
链接：https://arxiv.org/abs/2209.07140
作者：Jingwei Zhao,Gus Xia,Ye Wang机构：School of Computing, NUS, Institute of Data Science, NUS, Music X Lab, NYU Shanghai, Integrative Sciences and Engineering Programme, NUS Graduate School, MBZUAI备注：Accepted by ISMIR 2022摘要：提出了一种新的Transformer编码器结构,, Beat Transformer，实现了差拍和下拍的联合跟踪。与以往仅基于混合音频的声谱图来跟踪节拍的模型不同，本文的模型处理的是多乐器声道的去混合声谱图。这是因为人类从更丰富的音乐背景中感知韵律结构，如和弦进行和乐器。为此，我们开发了一个Transformer模型，同时考虑了时间注意和乐器注意，以捕捉深埋的韵律线索。此外，该模型采用了一种新颖的扩展自注意机制，以线性复杂度实现了强有力的分层建模.实验表明，与未去混的版本相比，去混的节拍跟踪有了显著的改善。此外，Beat Transformer在下拍跟踪精度方面比TCN架构提高了4%。我们进一步发现了一种可解释的注意模式，它反映了我们对等级韵律结构的理解。摘要：We propose Beat Transformer, a novel Transformer encoder architecture for joint beat and downbeat tracking. Different from previous models that track beats solely based on the spectrogram of an audio mixture, our model deals with demixed spectrograms with multiple instrument channels. This is inspired by the fact that humans perceive metrical structures from richer musical contexts, such as chord progression and instrumentation. To this end, we develop a Transformer model with both time-wise attention and instrument-wise attention to capture deep-buried metrical cues. Moreover, our model adopts a novel dilated self-attention mechanism, which achieves powerful hierarchical modelling with only linear complexity. Experiments demonstrate a significant improvement in demixed beat tracking over the non-demixed version. Also, Beat Transformer achieves up to 4% point improvement in downbeat tracking accuracy over the TCN architectures. We further discover an interpretable attention pattern that mirrors our understanding of hierarchical metrical structures.
【7】 Non-Parallel Voice Conversion for ASR Augmentation
标题：用于ASR增强的非并行语音转换
链接：https://arxiv.org/abs/2209.06987
作者：Gary Wang,Andrew Rosenberg,Bhuvana Ramabhadran,Fadi Biadsy,Yinghui Huang,Jesse Emond,Pedro Moreno Mengibar备注：Accepted by Interspeech 2022摘要：自动语音识别（ASR）需要对说话者差异具有鲁棒性。语音转换（VC）修改输入语音的说话者特性。这是ASR数据扩充的一个有吸引力的特性。在本文中，我们证明了语音转换可以作为一种数据增强技术来提高ASR性能，即使是在包含2，456个扬声器的LibriSpeech上。为了增强ASR，VC模型必须对大范围的输入语音具有鲁棒性。这激发了非自回归、非并行VC模型的使用，以及VC模型内的预训练ASR编码器的使用。这项工作表明，尽管包括许多说话者，说话者多样性可能仍然是ASR质量的一个限制。最后，对风险投资绩效的调查为风险投资质量的客观评价提供了有用的指标。摘要：Automatic speech recognition (ASR) needs to be robust to speaker differences. Voice Conversion (VC) modifies speaker characteristics of input speech. This is an attractive feature for ASR data augmentation. In this paper, we demonstrate that voice conversion can be used as a data augmentation technique to improve ASR performance, even on LibriSpeech, which contains 2,456 speakers. For ASR augmentation, it is necessary that the VC model be robust to a wide range of input speech. This motivates the use of a non-autoregressive, non-parallel VC model, and the use of a pretrained ASR encoder within the VC model. This work suggests that despite including many speakers, speaker diversity may remain a limitation to ASR quality. Finally, interrogation of our VC performance has provided useful metrics for objective evaluation of VC quality.
【8】 Environment Classification via Blind Roomprints Estimation
标题：基于盲Roomprint估计的环境分类
链接：https://arxiv.org/abs/2209.07196
作者：Malte Baum,Luca Cuccovillo,Artem Yaroshchuk,Patrick Aichroth机构：∗Fraunhofer Institute for Digital Media Technology, Ilmenau, Germany, †Ilmenau University of Technology, Ilmenau, Germany备注：To appear in: IEEE International Workshop on Information Forensics and Security (WIFS), December 12-16, 2022, Shanghai, China摘要：本文提出了一种新的语音环境分类方法，该方法不需要选择衰减的混响尾。它基于盲信道估计的多频带RT60分析，并且在从ACE语料库导出的测试记录上实现高达93.6%的准确度。摘要：In this paper we present a novel approach for environment classification for speech recordings, which does not require the selection of decaying reverberation tails. It is based on a multi-band RT60 analysis of blind channel estimates and achieves an accuracy of up to 93.6% on test recordings derived from the ACE corpus.
【9】 Open Challenges in Synthetic Speech Detection
标题：人工合成语音检测面临的挑战
链接：https://arxiv.org/abs/2209.07180
作者：Luca Cuccovillo,Christoforos Papastergiopoulos,Anastasios Vafeiadis,Artem Yaroshchuk,Patrick Aichroth,Konstantinos Votis,Dimitrios Tzovaras机构：†Fraunhofer Institute for Digital Media Technology, Ilmenau, Germany, ∗Centre for Research and Technology Hellas, Thessaloniki, Greece备注：To appear in: IEEE International Workshop on Information Forensics and Security (WIFS), December 12-16, 2022, Shanghai, China摘要：本文讨论了合成语音检测的现状和面临的挑战。这项工作包括对现有开放数据集和现有检测方法的初步分析，对符合法规并更好地代表实际情况的新研究数据集的要求的描述，以及从功能和非功能要求两个方面讨论未来值得信赖的检测方法的预期特征。与其他基于特定检测方案或提供单一合成语音数据集的研究相比，本文旨在为该领域未来的研究提供方向，以快速缩小合成和检测方法之间的差距。摘要：In this paper the current status and open challenges of synthetic speech detection are addressed. The work comprises an initial analysis of available open datasets and of existing detection methods, a description of the requirements for new research datasets compliant with regulations and better representing real-case scenarios, and a discussion of the desired characteristics of future trustworthy detection methods in terms of both functional and non-functional requirements. Compared to other works, based on specific detection solutions or presenting single dataset of synthetic speeches, our paper is meant to orient future state-of-the-art research in the domain, to quickly lessen the current gap between synthesis and detection approaches.
【10】 ESSumm: Extractive Speech Summarization from Untranscribed Meeting
标题：ESSumm：非文字记录会议的摘录演讲摘要
链接：https://arxiv.org/abs/2209.06913
机构：University of Maryland, College Park, USA摘要：本文提出了一种新的直接抽取式语音摘要模型ESAmmm，它是一种不依赖于中间转录文本的无监督模型。与以往的文本表示方法不同，本文的目标是直接从语音中生成摘要，而不需要转录。首先，基于语音信号的声学特征提取一组较小的语音片段。对于每个候选语音片段，设计了基于距离的摘要置信度作为潜在语音表示度量。具体地说，我们利用现成的自监督卷积神经网络从原始音频中提取深度语音特征。我们的方法自动预测最佳的语音片段序列，其捕获具有目标摘要长度的关键信息。在两个著名的会议数据集（AMI和ICSI语料库）上的大量实验表明，该方法能够有效地提高非转录数据的摘要质量.我们还观察到，我们的无监督的基于语音的方法甚至与最近的基于转录本的摘要方法一样执行，其中需要额外的语音识别。摘要：In this paper, we propose a novel architecture for direct extractive speech-to-speech summarization, ESSumm, which is an unsupervised model without dependence on intermediate transcribed text. Different from previous methods with text presentation, we are aimed at generating a summary directly from speech without transcription. First, a set of smaller speech segments are extracted based on speech signal's acoustic features. For each candidate speech segment, a distance-based summarization confidence score is designed for latent speech representation measure. Specifically, we leverage the off-the-shelf self-supervised convolutional neural network to extract the deep speech features from raw audio. Our approach automatically predicts the optimal sequence of speech segments that capture the key information with a target summary length. Extensive results on two well-known meeting datasets (AMI and ICSI corpora) show the effectiveness of our direct speech-based method to improve the summarization quality with untranscribed data. We also observe that our unsupervised speech-based method even performs on par with recent transcript-based summarization approaches, where extra speech recognition is required.
【1】 Environment Classification via Blind Roomprints Estimation
标题：基于盲Roomprint估计的环境分类
链接：https://arxiv.org/abs/2209.07196
* 与cs.SD语音【8】为同一篇
作者：Malte Baum,Luca Cuccovillo,Artem Yaroshchuk,Patrick Aichroth机构：∗Fraunhofer Institute for Digital Media Technology, Ilmenau, Germany, †Ilmenau University of Technology, Ilmenau, Germany备注：To appear in: IEEE International Workshop on Information Forensics and Security (WIFS), December 12-16, 2022, Shanghai, China摘要：本文提出了一种新的语音环境分类方法，该方法不需要选择衰减的混响尾。它基于盲信道估计的多频带RT60分析，并且在从ACE语料库导出的测试记录上实现高达93.6%的准确度。摘要：In this paper we present a novel approach for environment classification for speech recordings, which does not require the selection of decaying reverberation tails. It is based on a multi-band RT60 analysis of blind channel estimates and achieves an accuracy of up to 93.6% on test recordings derived from the ACE corpus.
【2】 Open Challenges in Synthetic Speech Detection
标题：人工合成语音检测面临的挑战
链接：https://arxiv.org/abs/2209.07180
* 与cs.SD语音【9】为同一篇
作者：Luca Cuccovillo,Christoforos Papastergiopoulos,Anastasios Vafeiadis,Artem Yaroshchuk,Patrick Aichroth,Konstantinos Votis,Dimitrios Tzovaras机构：†Fraunhofer Institute for Digital Media Technology, Ilmenau, Germany, ∗Centre for Research and Technology Hellas, Thessaloniki, Greece备注：To appear in: IEEE International Workshop on Information Forensics and Security (WIFS), December 12-16, 2022, Shanghai, China摘要：本文讨论了合成语音检测的现状和面临的挑战。这项工作包括对现有开放数据集和现有检测方法的初步分析，对符合法规并更好地代表实际情况的新研究数据集的要求的描述，以及从功能和非功能要求两个方面讨论未来值得信赖的检测方法的预期特征。与其他基于特定检测方案或提供单一合成语音数据集的研究相比，本文旨在为该领域未来的研究提供方向，以快速缩小合成和检测方法之间的差距。摘要：In this paper the current status and open challenges of synthetic speech detection are addressed. The work comprises an initial analysis of available open datasets and of existing detection methods, a description of the requirements for new research datasets compliant with regulations and better representing real-case scenarios, and a discussion of the desired characteristics of future trustworthy detection methods in terms of both functional and non-functional requirements. Compared to other works, based on specific detection solutions or presenting single dataset of synthetic speeches, our paper is meant to orient future state-of-the-art research in the domain, to quickly lessen the current gap between synthesis and detection approaches.
【3】 ESSumm: Extractive Speech Summarization from Untranscribed Meeting
标题：ESSumm：非文字记录会议的摘录演讲摘要
链接：https://arxiv.org/abs/2209.06913
* 与cs.SD语音【10】为同一篇
机构：University of Maryland, College Park, USA摘要：本文提出了一种新的直接抽取式语音摘要模型ESAmmm，它是一种不依赖于中间转录文本的无监督模型。与以往的文本表示方法不同，本文的目标是直接从语音中生成摘要，而不需要转录。首先，基于语音信号的声学特征提取一组较小的语音片段。对于每个候选语音片段，设计了基于距离的摘要置信度作为潜在语音表示度量。具体地说，我们利用现成的自监督卷积神经网络从原始音频中提取深度语音特征。我们的方法自动预测最佳的语音片段序列，其捕获具有目标摘要长度的关键信息。在两个著名的会议数据集（AMI和ICSI语料库）上的大量实验表明，该方法能够有效地提高非转录数据的摘要质量.我们还观察到，我们的无监督的基于语音的方法甚至与最近的基于转录本的摘要方法一样执行，其中需要额外的语音识别。摘要：In this paper, we propose a novel architecture for direct extractive speech-to-speech summarization, ESSumm, which is an unsupervised model without dependence on intermediate transcribed text. Different from previous methods with text presentation, we are aimed at generating a summary directly from speech without transcription. First, a set of smaller speech segments are extracted based on speech signal's acoustic features. For each candidate speech segment, a distance-based summarization confidence score is designed for latent speech representation measure. Specifically, we leverage the off-the-shelf self-supervised convolutional neural network to extract the deep speech features from raw audio. Our approach automatically predicts the optimal sequence of speech segments that capture the key information with a target summary length. Extensive results on two well-known meeting datasets (AMI and ICSI corpora) show the effectiveness of our direct speech-based method to improve the summarization quality with untranscribed data. We also observe that our unsupervised speech-based method even performs on par with recent transcript-based summarization approaches, where extra speech recognition is required.
【4】 CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations
标题：CMSBERT-CLR：语境驱动的情态转换BERT与语言、视觉、声音表征的对比学习
链接：https://arxiv.org/abs/2209.07424
* 与cs.SD语音【2】为同一篇
机构：Department of Artificial Intelligence, Dongguk University, Seoul, Korea备注：Accepted by IJCNN 2022摘要：随着对多模态在线内容需求的不断增长，多模态情感分析已经成为一个日益流行的研究领域。在多模态情感分析中，词语会因语境和非言语信息的不同而具有不同的含义，因此理解词语的含义至关重要。此外，词义的解释还应放在包括非语言信息在内的整个话语语境中进行。本文提出了一种基于对比学习的语境驱动的语言、视觉、听觉表征的模态转换BERT（CMSBERT-CLR）算法，该算法综合了整个语境中的非言语和言语信息，通过对比学习更有效地实现模态转换。首先，我们引入了语境驱动的情态转换（CMS），将非言语和言语信息整合到句子话语的整个语境中。然后，为了改善不同模态在公共嵌入空间内的对齐，我们应用对比学习。此外，采用指数滑动平均参数和标签平滑作为优化策略，使网络的收敛更加稳定，增加了比对的灵活性。在我们的实验中，我们证明了我们的方法达到了最先进的结果。摘要：Multimodal sentiment analysis has become an increasingly popular research area as the demand for multimodal online content is growing. For multimodal sentiment analysis, words can have different meanings depending on the linguistic context and non-verbal information, so it is crucial to understand the meaning of the words accordingly. In addition, the word meanings should be interpreted within the whole utterance context that includes nonverbal information. In this paper, we present a Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations (CMSBERT-CLR), which incorporates the whole context's non-verbal and verbal information and aligns modalities more effectively through contrastive learning. First, we introduce a Context-driven Modality Shifting (CMS) to incorporate the non-verbal and verbal information within the whole context of the sentence utterance. Then, for improving the alignment of different modalities within a common embedding space, we apply contrastive learning. Furthermore, we use an exponential moving average parameter and label smoothing as optimization strategies, which can make the convergence of the network more stable and increase the flexibility of the alignment. In our experiments, we demonstrate that our approach achieves state-of-the-art results.
【5】 Self-Supervised Attention Networks and Uncertainty Loss Weighting for Multi-Task Emotion Recognition on Vocal Bursts
标题：基于自监督注意网络和不确定性损失加权的突发语音多任务情绪识别
链接：https://arxiv.org/abs/2209.07384
* 与cs.SD语音【3】为同一篇
作者：Vincent Karas,Andreas Triantafyllopoulos,Meishu Song,Björn W. Schuller机构：University of Augsburg BMW Group, University of Augsburg University of Tokyo, Augsburg, Germany Tokyo, Japan, th Bj¨orn W. Schuller, Chair EIHW & ZIG GLAM, Unversity of Augsburg Imperial College London, Augsburg, Germany London, UK备注：4 pages, 1 figure, submitted to The 2022 ACII Affective Vocal Burst Workshop & Challenge (A-VB)摘要：突发性发声在情感交流中起着重要作用，对提高言语情感识别具有重要价值。在这里，我们介绍了我们的方法，分类发声爆发和预测他们的情感意义的ACII情感发声爆发研讨会和挑战2022（A-VB）。我们使用一个大型的自监督音频模型作为共享特征提取器，并比较了基于分类器链和注意力网络的多种结构，结合不确定性损失加权策略。在所有四项任务中，我们的方法都远远超过了挑战基线。摘要：Vocal bursts play an important role in communicating affect, making them valuable for improving speech emotion recognition. Here, we present our approach for classifying vocal bursts and predicting their emotional significance in the ACII Affective Vocal Burst Workshop & Challenge 2022 (A-VB). We use a large self-supervised audio model as shared feature extractor and compare multiple architectures built on classifier chains and attention networks, combined with uncertainty loss weighting strategies. Our approach surpasses the challenge baseline by a wide margin on all four tasks.
【6】 MVNet: Memory Assistance and Vocal Reinforcement Network for Speech Enhancement
标题：MVNet：语音增强的记忆辅助和语音增强网络
链接：https://arxiv.org/abs/2209.07302
* 与cs.SD语音【4】为同一篇
作者：Jianrong Wang,Xiaomin Li,Xuewei Li,Mei Yu,Qiang Fang,Li Liu机构：College of Intelligence and Computing, Tianjin University, Tianjin, China, Institute of Linguistics, Chinese Academy of Social Sciences, Beijing, China, Shenzhen Research Institute of Big Data, the Chinese University of Hong Kong, Shenzhen, China摘要：语音增强提高了语音质量，并促进了各种下游任务的性能。然而，目前的语音增强技术主要致力于提高下游自动语音识别（ASR）的性能，只有相对少量的工作集中在自动说话人确认（ASV）任务上。本文提出了一个MVNet，该MVNet由记忆辅助模块和语音增强模块组成，记忆辅助模块提高了下游ASR的性能，语音增强模块提高了ASV的性能。此外，我们设计了一个新的损失函数来提高说话人声音的相似度。在Libri2mix数据集上的实验结果表明，该方法在语音质量、可懂度、说话人声音相似度等多个指标上均优于基线方法。摘要：Speech enhancement improves speech quality and promotes the performance of various downstream tasks. However, most current speech enhancement work was mainly devoted to improving the performance of downstream automatic speech recognition (ASR), only a relatively small amount of work focused on the automatic speaker verification (ASV) task. In this work, we propose a MVNet consisted of a memory assistance module which improves the performance of downstream ASR and a vocal reinforcement module which boosts the performance of ASV. In addition, we design a new loss function to improve speaker vocal similarity. Experimental results on the Libri2mix dataset show that our method outperforms baseline methods in several metrics, including speech quality, intelligibility, and speaker vocal similarity et al.
【7】 Domain Adversarial Training on Conditional Variational Auto-Encoder for Controllable Music Generation
标题：用于可控音乐生成的条件变分自动编码器的领域对抗性训练
链接：https://arxiv.org/abs/2209.07144
* 与cs.SD语音【5】为同一篇
作者：Jingwei Zhao,Gus Xia,Ye Wang机构：School of Computing, NUS, Institute of Data Science, NUS, Music X Lab, NYU Shanghai, Integrative Sciences and Engineering Programme, NUS Graduate School, MBZUAI备注：Accepted by ISMIR 2022摘要：变分自动编码器已经成为符号音乐生成的主要框架，如何有效地控制符号音乐的生成过程是当前的研究热点。一种直接的方法是在推理期间使用不同的条件来控制模型。然而，在音乐实践中，条件通常是连续的（而不是简单的分类标签），涉及与学习的表征重叠的丰富信息。因此，解码器对于是“监听”潜在表示还是该条件感到困惑，并且有时只是忽略该条件。为了解决这个问题，我们利用领域对抗训练来将表征与条件线索分离，以便更好地控制。具体地，我们提出了一种条件破坏目标，其使用该表示来对破坏的条件进行去噪。通过鉴别器最小化并且通过VAE编码器最大化，该目标不利地诱导条件不变表示。本文以旋律和声任务为例来说明我们的思想，同时我们的方法也可以推广到其他可控的生成任务。演示和实验表明，该方法不仅有利于条件不变的表示学习，而且与基线相比具有更高质量的可控性.摘要：The variational auto-encoder has become a leading framework for symbolic music generation, and a popular research direction is to study how to effectively control the generation process. A straightforward way is to control a model using different conditions during inference. However, in music practice, conditions are usually sequential (rather than simple categorical labels), involving rich information that overlaps with the learned representation. Consequently, the decoder gets confused about whether to "listen to" the latent representation or the condition, and sometimes just ignores the condition. To solve this problem, we leverage domain adversarial training to disentangle the representation from condition cues for better control. Specifically, we propose a condition corruption objective that uses the representation to denoise a corrupted condition. Minimized by a discriminator and maximized by the VAE encoder, this objective adversarially induces a condition-invariant representation. In this paper, we focus on the task of melody harmonization to illustrate our idea, while our methodology can be generalized to other controllable generative tasks. Demos and experiments show that our methodology facilitates not only condition-invariant representation learning but also higher-quality controllability compared to baselines.
【8】 Beat Transformer: Demixed Beat and Downbeat Tracking with Dilated Self-Attention
标题：节拍Transformer：分流节拍和消沉节拍跟踪和自我关注
链接：https://arxiv.org/abs/2209.07140
* 与cs.SD语音【6】为同一篇
作者：Jingwei Zhao,Gus Xia,Ye Wang机构：School of Computing, NUS, Institute of Data Science, NUS, Music X Lab, NYU Shanghai, Integrative Sciences and Engineering Programme, NUS Graduate School, MBZUAI备注：Accepted by ISMIR 2022摘要：提出了一种新的Transformer编码器结构,, Beat Transformer，实现了差拍和下拍的联合跟踪。与以往仅基于混合音频的声谱图来跟踪节拍的模型不同，本文的模型处理的是多乐器声道的去混合声谱图。这是因为人类从更丰富的音乐背景中感知韵律结构，如和弦进行和乐器。为此，我们开发了一个Transformer模型，同时考虑了时间注意和乐器注意，以捕捉深埋的韵律线索。此外，该模型采用了一种新颖的扩展自注意机制，以线性复杂度实现了强有力的分层建模.实验表明，与未去混的版本相比，去混的节拍跟踪有了显著的改善。此外，Beat Transformer在下拍跟踪精度方面比TCN架构提高了4%。我们进一步发现了一种可解释的注意模式，它反映了我们对等级韵律结构的理解。摘要：We propose Beat Transformer, a novel Transformer encoder architecture for joint beat and downbeat tracking. Different from previous models that track beats solely based on the spectrogram of an audio mixture, our model deals with demixed spectrograms with multiple instrument channels. This is inspired by the fact that humans perceive metrical structures from richer musical contexts, such as chord progression and instrumentation. To this end, we develop a Transformer model with both time-wise attention and instrument-wise attention to capture deep-buried metrical cues. Moreover, our model adopts a novel dilated self-attention mechanism, which achieves powerful hierarchical modelling with only linear complexity. Experiments demonstrate a significant improvement in demixed beat tracking over the non-demixed version. Also, Beat Transformer achieves up to 4% point improvement in downbeat tracking accuracy over the TCN architectures. We further discover an interpretable attention pattern that mirrors our understanding of hierarchical metrical structures.
【9】 Non-Parallel Voice Conversion for ASR Augmentation
标题：用于ASR增强的非并行语音转换
链接：https://arxiv.org/abs/2209.06987
* 与cs.SD语音【7】为同一篇
作者：Gary Wang,Andrew Rosenberg,Bhuvana Ramabhadran,Fadi Biadsy,Yinghui Huang,Jesse Emond,Pedro Moreno Mengibar备注：Accepted by Interspeech 2022摘要：自动语音识别（ASR）需要对说话者差异具有鲁棒性。语音转换（VC）修改输入语音的说话者特性。这是ASR数据扩充的一个有吸引力的特性。在本文中，我们证明了语音转换可以作为一种数据增强技术来提高ASR性能，即使是在包含2，456个扬声器的LibriSpeech上。为了增强ASR，VC模型必须对大范围的输入语音具有鲁棒性。这激发了非自回归、非并行VC模型的使用，以及VC模型内的预训练ASR编码器的使用。这项工作表明，尽管包括许多说话者，说话者多样性可能仍然是ASR质量的一个限制。最后，对风险投资绩效的调查为风险投资质量的客观评价提供了有用的指标。摘要：Automatic speech recognition (ASR) needs to be robust to speaker differences. Voice Conversion (VC) modifies speaker characteristics of input speech. This is an attractive feature for ASR data augmentation. In this paper, we demonstrate that voice conversion can be used as a data augmentation technique to improve ASR performance, even on LibriSpeech, which contains 2,456 speakers. For ASR augmentation, it is necessary that the VC model be robust to a wide range of input speech. This motivates the use of a non-autoregressive, non-parallel VC model, and the use of a pretrained ASR encoder within the VC model. This work suggests that despite including many speakers, speaker diversity may remain a limitation to ASR quality. Finally, interrogation of our VC performance has provided useful metrics for objective evaluation of VC quality.
机器翻译，仅供参考
每日语音速递