今日论文合集：cs.SD语音7篇，eess.AS音频处理9篇。
cs.SD语音

【1】 Learning Cross-lingual Visual Speech Representations

标题：学习跨语言的视觉语音表征链接：http://arxiv.org/pdf/2303.09455v1

作者：Andreas Zinonos,Alexandros Haliassos,Pingchuan Ma,Stavros Petridis,Maja Pantic机构：Department of Computing, Imperial College London, UK摘要：跨语言自监督学习是近几年来研究的热点。然而，目前的工作只探索了使用音频信号来创建表示。在这篇论文中，我们研究了跨语言的自监督视觉表征学习。我们使用最近提出的原始视听语音编码器（RAVEn）框架来使用未标记的多语言数据预训练视听模型，然后在标记的转录上微调视觉模型。我们的实验表明：（1）数据量较大的多语言模型的性能优于单语言模型，但在数据量一定的情况下，单语言模型的性能更好;（2）多语种训练优于纯英语训练;（3）使用更相似的语言产生更好的结果;以及（4）对看不见的语言的微调与使用预训练集中的目标语言是有竞争力的。我们希望我们的研究能对非英语语言表征学习的研究有所启发。摘要：Cross-lingual self-supervised learning has been a growing research topic in the last few years. However, current works only explored the use of audio signals to create representations. In this work, we study cross-lingual self-supervised visual representation learning. We use the recently-proposed Raw Audio-Visual Speech Encoders (RAVEn) framework to pre-train an audio-visual model with unlabelled multilingual data, and then fine-tune the visual model on labelled transcriptions. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance; (2) multi-lingual outperforms English-only pre-training; (3) using languages which are more similar yields better results; and (4) fine-tuning on unseen languages is competitive to using the target language in the pre-training set. We hope our study inspires future research on non-English-only speech representation learning.

【2】 Trustera: A Live Conversation Redaction System标题：Trustera：一个实时对话编辑系统链接：http://arxiv.org/pdf/2303.09438v1

作者：Evandro Gouvêa,Ali Dadgar,Shahab Jalalvand,Rathi Chengalvarayan,Badrinath Jayakumar,Ryan Price,Nicholas Ruiz,Jennifer McGovern,Srinivas Bangalore,Ben Stern备注：5机构：Interactions LLC, New Providence, New Jersey, USA摘要：Trustera是第一个在实时对话中编辑个人身份信息（PII）的功能系统，可消除代理人听到敏感信息的需要，同时保持客户与代理人实时对话的自然性。与呼叫后编辑相反，音频屏蔽在客户开始与PII实体交谈时立即开始。这显著降低了PII被拦截或存储在不安全数据存储中的风险。Trustera的架构由自动语音识别、自然语言理解和现场音频编辑器模块组成。该系统的目标有三：编辑作为PII的实体，屏蔽发送给代理的音频，同时捕获该实体，以便捕获的PII可用于支付交易或呼叫者识别。Trustera目前正被成千上万的代理商用来保护客户的敏感信息。摘要：Trustera, the first functional system that redacts personally identifiable information (PII) in real-time spoken conversations to remove agents' need to hear sensitive information while preserving the naturalness of live customer-agent conversations. As opposed to post-call redaction, audio masking starts as soon as the customer begins speaking to a PII entity. This significantly reduces the risk of PII being intercepted or stored in insecure data storage. Trustera's architecture consists of a pipeline of automatic speech recognition, natural language understanding, and a live audio redactor module. The system's goal is three-fold: redact entities that are PII, mask the audio that goes to the agent, and at the same time capture the entity, so that the captured PII can be used for a payment transaction or caller identification. Trustera is currently being used by thousands of agents to secure customers' sensitive information.

【3】 Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation标题：用于音频驱动的协同语音手势生成的驯服扩散模型链接：http://arxiv.org/pdf/2303.09119v1

作者：Lingting Zhu,Xian Liu,Xuanyu Liu,Rui Qian,Ziwei Liu,Lequan Yu机构：The University of Hong Kong, The Chinese University of Hong Kong, Nanyang Technological University备注：Accepted by IEEECVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023. 10 pages, 3 figures摘要：动画化虚拟化身以做出共语音姿势促进人机交互中的各种应用。现有方法主要依赖于生成式对抗网络（generative adversarial networks，GANs），GANs存在模式崩溃和训练不稳定等问题，难以获得准确的音频-手势联合分布.本文提出了一种新的基于扩散的框架，称为扩散共语手势（DiffGesture），以有效地捕获跨模态的音频-手势关联并保持时间相干性，从而实现高保真音频驱动的共语手势生成。具体来说，我们首先在骨架序列和音频的片段上建立扩散条件生成过程，以使整个框架成为可能。然后，设计了一种新的扩散音频-手势Transformer，以更好地处理来自多个模态的信息，并对长期时间依赖性进行建模。此外，为了消除时间上的不一致性，我们提出了一个有效的扩散手势稳定器与退火噪声采样策略。受益于扩散模型的架构优势，我们进一步引入隐式无分类器指导来权衡多样性和手势质量。大量的实验表明，DiffGesture算法具有最佳的性能，能以更好的模式覆盖率和更强的音频相关性呈现连贯的手势。代码可从www.example.com获取https: github.com Advocate99 DiffGesture。摘要：Animating virtual avatars to make co-speech gestures facilitates various applications in human-machine interaction. The existing methods mainly rely on generative adversarial networks (GANs), which typically suffer from notorious mode collapse and unstable training, thus making it difficult to learn accurate audio-gesture joint distributions. In this work, we propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to effectively capture the cross-modal audio-to-gesture associations and preserve temporal coherence for high-fidelity audio-driven co-speech gesture generation. Specifically, we first establish the diffusion-conditional generation process on clips of skeleton sequences and audio to enable the whole framework. Then, a novel Diffusion Audio-Gesture Transformer is devised to better attend to the information from multiple modalities and model the long-term temporal dependency. Moreover, to eliminate temporal inconsistency, we propose an effective Diffusion Gesture Stabilizer with an annealed noise sampling strategy. Benefiting from the architectural advantages of diffusion models, we further incorporate implicit classifier-free guidance to trade off between diversity and gesture quality. Extensive experiments demonstrate that DiffGesture achieves state-of-theart performance, which renders coherent gestures with better mode coverage and stronger audio correlations. Code is available at https: github.com Advocate99 DiffGesture.

【4】 Improving Perceptual Quality, Intelligibility, and Acoustics on VoIP Platforms标题：提高VoIP平台的感知质量、清晰度和音质链接：http://arxiv.org/pdf/2303.09048v1

作者：Joseph Konan,Ojas Bhargave,Shikhar Agnihotri,Hojeong Lee,Ankit Shah,Shuo Han,Yunyang Zeng,Amanda Shu,Haohui Liu,Xuankai Chang,Hamza Khalid,Minseon Gwak,Kawon Lee,Minjeong Kim,Bhiksha Raj机构：†Carnegie Mellon University, Pittsburgh备注：Under review at European Association for Signal Processing. 5 pages摘要：本文提出了一种微调在深度噪声抑制（DNS）2020挑战赛中训练的模型的方法，以提高其在互联网协议语音（VoIP）应用中的性能。我们的方法涉及调整DNS 2020模型以适应VoIP通信的特定声学特征，包括压缩、传输和平台特定处理导致的失真和伪像。为此，我们提出了一个多任务学习框架，为VoIP DNS的联合优化噪声抑制和VoIP特定的声学语音增强。我们评估了我们的方法在不同的VoIP场景，并表明它优于行业性能和国家的最先进的方法语音增强VoIP应用。我们的结果证明了在DNS-2020上训练的模型的潜力，以改善和定制到使用VoIP-DNS的不同VoIP平台，其发现在语音识别，语音助手和电信等领域有重要的应用。摘要：In this paper, we present a method for fine-tuning models trained on the Deep Noise Suppression (DNS) 2020 Challenge to improve their performance on Voice over Internet Protocol (VoIP) applications. Our approach involves adapting the DNS 2020 models to the specific acoustic characteristics of VoIP communications, which includes distortion and artifacts caused by compression, transmission, and platform-specific processing. To this end, we propose a multi-task learning framework for VoIP-DNS that jointly optimizes noise suppression and VoIP-specific acoustics for speech enhancement. We evaluate our approach on a diverse VoIP scenarios and show that it outperforms both industry performance and state-of-the-art methods for speech enhancement on VoIP applications. Our results demonstrate the potential of models trained on DNS-2020 to be improved and tailored to different VoIP platforms using VoIP-DNS, whose findings have important applications in areas such as speech recognition, voice assistants, and telecommunication.

【5】 Speech Modeling with a Hierarchical Transformer Dynamical VAE标题：基于分层Transformer的语音建模动态VAE链接：http://arxiv.org/pdf/2303.09404v1

作者：Xiaoyu Lin,Xiaoyu Bie,Simon Leglaive,Laurent Girin,Xavier Alameda-Pineda机构：Inria Grenoble Rhˆone-Alpes, Univ. Grenoble Alpes, France, CentraleSup´elec, IETR (UMR CNRS ,), France, Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France摘要：动态变分自动编码器（DVAE）是一类隐变量深度生成模型，它将VAE扩展到对观测数据序列和相应的隐向量序列进行建模。在几乎所有的DVAE文献中，每个序列内和两个序列之间的时间依赖性都用递归神经网络建模。本文提出了一种分层Transformer DVAE（HiT-DVAE）语音信号建模方法，该方法是一种具有两级隐变量（序列级和帧级）的DVAE，其时间依赖性由Transformer结构实现。我们证明HiT-DVAE在语音谱图建模方面优于其他几种DVAE，同时实现了更简单的训练过程，揭示了其在语音增强等下游低级别语音处理任务中的巨大潜力。摘要：The dynamical variational autoencoders (DVAEs) are a family of latent-variable deep generative models that extends the VAE to model a sequence of observed data and a corresponding sequence of latent vectors. In almost all the DVAEs of the literature, the temporal dependencies within each sequence and across the two sequences are modeled with recurrent neural networks. In this paper, we propose to model speech signals with the Hierarchical Transformer DVAE (HiT-DVAE), which is a DVAE with two levels of latent variable (sequence-wise and frame-wise) and in which the temporal dependencies are implemented with the Transformer architecture. We show that HiT-DVAE outperforms several other DVAEs for speech spectrogram modeling, while enabling a simpler training procedure, revealing its high potential for downstream low-level speech processing tasks such as speech enhancement.

【6】DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model标题：DistillW2V2：一种基于Wav2vec 2.0的小型流媒体ASR模型链接：http://arxiv.org/pdf/2303.09278v1

作者：Yanzhe Fu,Yueteng Kang,Songjun Cao,Long Ma机构：Tencent Cloud Xiaowei, Beijing, China摘要：Wav 2 vec 2.0（W2 V2）在自动语音识别（ASR）中表现出了令人印象深刻的性能。然而，大的模型尺寸和非流架构使得它难以在低资源或流场景下使用。在本研究中，我们提出一个两阶段的知识蒸馏方法来解决这两个问题：第一步是将大的非流教师模型变小，第二步是使其流。特别地，我们采用MSE损失提取隐藏层，采用改进的LF-MMI损失提取预测层。实验在Gigaspeech、Librispeech和内部数据集上进行。结果表明，我们最终得到的精简学生模型（DistillW 2 V2）比原始教师模型快8倍，小12倍。对于480 ms的延迟设置，DistillW 2 V2的相对字错误率（WER）在测试集上下降了9%到23.4%，这为扩展W2 V2的应用范围提供了一种有前景的方法。摘要：Wav2vec 2.0 (W2V2) has shown impressive performance in automatic speech recognition (ASR). However, the large model size and the non-streaming architecture make it hard to be used under low-resource or streaming scenarios. In this work, we propose a two-stage knowledge distillation method to solve these two problems: the first step is to make the big and non-streaming teacher model smaller, and the second step is to make it streaming. Specially, we adopt the MSE loss for the distillation of hidden layers and the modified LF-MMI loss for the distillation of the prediction layer. Experiments are conducted on Gigaspeech, Librispeech, and an in-house dataset. The results show that the distilled student model (DistillW2V2) we finally get is 8x faster and 12x smaller than the original teacher model. For the 480ms latency setup, the DistillW2V2's relative word error rate (WER) degradation varies from 9% to 23.4% on test sets, which reveals a promising way to extend the W2V2's application scope.

【7】 TriAAN-VC: Triple Adaptive Attention Normalization for Any-to-Any Voice Conversion标题：TriAAN-VC：任意语音到任意语音转换的三重自适应注意力标准化链接：http://arxiv.org/pdf/2303.09057v1

作者：Hyun Joon Park,Seok Woo Yang,Jin Sob Kim,Wooseok Shin,Sung Won Han机构：School of Industrial and Management Engineering, Korea University, Seoul, Republic of Korea备注：To appear in ICASSP 2023摘要：语音转换（VC）必须在保持源语音内容的同时，表现目标说话人的特征。现有方法不能同时满足VC的上述两个方面，并且它们的转换输出遭受保持源内容和目标特性之间的折衷问题。在本研究中，我们提出三重适应性注意力正规化VC（TriAAN-VC），包含一个编码器-解码器和一个基于注意力的适应性正规化区块，可应用于非平行的任意对任意VC。该自适应归一化模块提取目标说话人表示并实现转换，同时最小化源内容的损失与连体损失。我们在VCTK数据集上评估了TriAAN-VC对源内容和目标说话人相似度的维护。单次VC的实验结果表明，TriAAN-VC实现了最先进的性能，同时缓解了现有VC方法中遇到的权衡问题。摘要：Voice Conversion (VC) must be achieved while maintaining the content of the source speech and representing the characteristics of the target speaker. The existing methods do not simultaneously satisfy the above two aspects of VC, and their conversion outputs suffer from a trade-off problem between maintaining source contents and target characteristics. In this study, we propose Triple Adaptive Attention Normalization VC (TriAAN-VC), comprising an encoder-decoder and an attention-based adaptive normalization block, that can be applied to non-parallel any-to-any VC. The proposed adaptive normalization block extracts target speaker representations and achieves conversion while minimizing the loss of the source content with siamese loss. We evaluated TriAAN-VC on the VCTK dataset in terms of the maintenance of the source content and target speaker similarity. Experimental results for one-shot VC suggest that TriAAN-VC achieves state-of-the-art performance while mitigating the trade-off problem encountered in the existing VC methods.
eess.AS音频处理

【1】 Controlling High-Dimensional Data With Sparse Input标题：使用稀疏输入控制高维数据链接：http://arxiv.org/pdf/2303.09446v1

作者：Dan Andrei Iliescu,Devang Savita Ram Mohan,Tian Huey Teh,Zack Hodari备注：11 pages摘要：我们解决了生成高度结构化数据的人在环控制问题。这项任务是具有挑战性的，因为现有的生成式模型缺乏有效的接口，通过它用户可以修改输出。用户可以选择手动探索不可解释的潜在空间，或者费力地用条件标签注释数据。为了解决这个问题，我们引入了一个新的框架，其中编码器映射稀疏的，人类可解释的控制空间到生成模型的潜在空间。我们将此架构应用于文语合成中韵律控制的任务。我们提出了一种多实例CVAE模型（MICVAE），专门用于编码稀疏的韵律特征并输出完整的波形。我们的经验表明，MICVAE显示了稀疏人在环控制机制的理想品质：效率、健壮性和忠实性。即使输入值非常少（~4），MICVAE也能使用户在收听者偏好方面（4：1）显著提高输出质量。摘要：We address the problem of human-in-the-loop control for generating highly-structured data. This task is challenging because existing generative models lack an efficient interface through which users can modify the output. Users have the option to either manually explore a non-interpretable latent space, or to laboriously annotate the data with conditioning labels. To solve this, we introduce a novel framework whereby an encoder maps a sparse, human interpretable control space onto the latent space of a generative model. We apply this framework to the task of controlling prosody in text-to-speech synthesis. We propose a model, called Multiple-Instance CVAE (MICVAE), that is specifically designed to encode sparse prosodic features and output complete waveforms. We show empirically that MICVAE displays desirable qualities of a sparse human-in-the-loop control mechanism: efficiency, robustness, and faithfulness. With even a very small number of input values (~4), MICVAE enables users to improve the quality of the output significantly, in terms of listener preference (4:1).

【2】 Speech Modeling with a Hierarchical Transformer Dynamical VAE标题：基于分层Transformer的语音建模动态VAE链接：http://arxiv.org/pdf/2303.09404v1

*与cs.SD语音【5】为同一篇

作者：Xiaoyu Lin,Xiaoyu Bie,Simon Leglaive,Laurent Girin,Xavier Alameda-Pineda机构：Inria Grenoble Rhˆone-Alpes, Univ. Grenoble Alpes, France, CentraleSup´elec, IETR (UMR CNRS ,), France, Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France摘要：动态变分自动编码器（DVAE）是一类隐变量深度生成模型，它将VAE扩展到对观测数据序列和相应的隐向量序列进行建模。在几乎所有的DVAE文献中，每个序列内和两个序列之间的时间依赖性都用递归神经网络建模。本文提出了一种分层Transformer DVAE（HiT-DVAE）语音信号建模方法，该方法是一种具有两级隐变量（序列级和帧级）的DVAE，其时间依赖性由Transformer结构实现。我们证明HiT-DVAE在语音谱图建模方面优于其他几种DVAE，同时实现了更简单的训练过程，揭示了其在语音增强等下游低级别语音处理任务中的巨大潜力。摘要：The dynamical variational autoencoders (DVAEs) are a family of latent-variable deep generative models that extends the VAE to model a sequence of observed data and a corresponding sequence of latent vectors. In almost all the DVAEs of the literature, the temporal dependencies within each sequence and across the two sequences are modeled with recurrent neural networks. In this paper, we propose to model speech signals with the Hierarchical Transformer DVAE (HiT-DVAE), which is a DVAE with two levels of latent variable (sequence-wise and frame-wise) and in which the temporal dependencies are implemented with the Transformer architecture. We show that HiT-DVAE outperforms several other DVAEs for speech spectrogram modeling, while enabling a simpler training procedure, revealing its high potential for downstream low-level speech processing tasks such as speech enhancement.

【3】 DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model标题：DistillW2V2：一种基于Wav2vec 2.0的小型流媒体ASR模型链接：http://arxiv.org/pdf/2303.09278v1

*与cs.SD语音【6】为同一篇

作者：Yanzhe Fu,Yueteng Kang,Songjun Cao,Long Ma机构：Tencent Cloud Xiaowei, Beijing, China摘要：Wav 2 vec 2.0（W2 V2）在自动语音识别（ASR）中表现出了令人印象深刻的性能。然而，大的模型尺寸和非流架构使得它难以在低资源或流场景下使用。在本研究中，我们提出一个两阶段的知识蒸馏方法来解决这两个问题：第一步是将大的非流教师模型变小，第二步是使其流。特别地，我们采用MSE损失提取隐藏层，采用改进的LF-MMI损失提取预测层。实验在Gigaspeech、Librispeech和内部数据集上进行。结果表明，我们最终得到的精简学生模型（DistillW 2 V2）比原始教师模型快8倍，小12倍。对于480 ms的延迟设置，DistillW 2 V2的相对字错误率（WER）在测试集上下降了9%到23.4%，这为扩展W2 V2的应用范围提供了一种有前景的方法。摘要：Wav2vec 2.0 (W2V2) has shown impressive performance in automatic speech recognition (ASR). However, the large model size and the non-streaming architecture make it hard to be used under low-resource or streaming scenarios. In this work, we propose a two-stage knowledge distillation method to solve these two problems: the first step is to make the big and non-streaming teacher model smaller, and the second step is to make it streaming. Specially, we adopt the MSE loss for the distillation of hidden layers and the modified LF-MMI loss for the distillation of the prediction layer. Experiments are conducted on Gigaspeech, Librispeech, and an in-house dataset. The results show that the distilled student model (DistillW2V2) we finally get is 8x faster and 12x smaller than the original teacher model. For the 480ms latency setup, the DistillW2V2's relative word error rate (WER) degradation varies from 9% to 23.4% on test sets, which reveals a promising way to extend the W2V2's application scope.

【4】 TriAAN-VC: Triple Adaptive Attention Normalization for Any-to-Any Voice Conversion标题：TriAAN-VC：任意语音到任意语音转换的三重自适应注意力标准化链接：http://arxiv.org/pdf/2303.09057v1

*与cs.SD语音【7】为同一篇

作者：Hyun Joon Park,Seok Woo Yang,Jin Sob Kim,Wooseok Shin,Sung Won Han机构：School of Industrial and Management Engineering, Korea University, Seoul, Republic of Korea备注：To appear in ICASSP 2023摘要：语音转换（VC）必须在保持源语音内容的同时，表现目标说话人的特征。现有方法不能同时满足VC的上述两个方面，并且它们的转换输出遭受保持源内容和目标特性之间的折衷问题。在本研究中，我们提出三重适应性注意力正规化VC（TriAAN-VC），包含一个编码器-解码器和一个基于注意力的适应性正规化区块，可应用于非平行的任意对任意VC。该自适应归一化模块提取目标说话人表示并实现转换，同时最小化源内容的损失与连体损失。我们在VCTK数据集上评估了TriAAN-VC对源内容和目标说话人相似度的维护。单次VC的实验结果表明，TriAAN-VC实现了最先进的性能，同时缓解了现有VC方法中遇到的权衡问题。摘要：Voice Conversion (VC) must be achieved while maintaining the content of the source speech and representing the characteristics of the target speaker. The existing methods do not simultaneously satisfy the above two aspects of VC, and their conversion outputs suffer from a trade-off problem between maintaining source contents and target characteristics. In this study, we propose Triple Adaptive Attention Normalization VC (TriAAN-VC), comprising an encoder-decoder and an attention-based adaptive normalization block, that can be applied to non-parallel any-to-any VC. The proposed adaptive normalization block extracts target speaker representations and achieves conversion while minimizing the loss of the source content with siamese loss. We evaluated TriAAN-VC on the VCTK dataset in terms of the maintenance of the source content and target speaker similarity. Experimental results for one-shot VC suggest that TriAAN-VC achieves state-of-the-art performance while mitigating the trade-off problem encountered in the existing VC methods.

【5】 Subspace Hybrid Beamforming for Head-worn Microphone Arrays标题：用于头戴式麦克风阵列的子空间混合波束形成链接：http://arxiv.org/pdf/2303.08967v1

作者：Sina Hafezi,Alastair H. Moore,Pierre Guiraud,Patrick A. Naylor,Jacob Donley,Vladimir Tourbabin,Thomas Lunner机构：† Electrical and Electronic Engineering, Imperial College London, London, UK, ‡ Meta Reality Labs Research, Redmond, Washington, USA备注：5 pages, 4 figures, accepted for ICASSP 2023摘要：提出了一种两阶段多通道语音增强方法，该方法包括一种新的自适应波束形成器、混合最小方差无失真响应（MVDR）、各向同性MVDR（Iso）和一种新的多通道谱主成分分析（PCA）去噪。在第一阶段，Hybrid-MVDR使用预定义的噪声场模型字典执行多个MVDR，并选取最小功率结果，这得益于信号无关波束形成的鲁棒性和自适应波束形成的性能。在第二阶段，将Hybrid和Iso的结果联合用于基于PCA的双通道去噪，以去除Hybrid波束形成器产生的“音乐噪声”。在真实的“鸡尾酒会”头戴式阵列录音数据集上，该方法在噪声抑制（fwSegSNR，SDR，SIR，SAR）和语音可懂度（STOI）方面优于基线超定向波束形成器，并具有相似的语音质量（PESQ）改善。摘要：A two-stage multi-channel speech enhancement method is proposed which consists of a novel adaptive beamformer, Hybrid Minimum Variance Distortionless Response (MVDR), Isotropic-MVDR (Iso), and a novel multi-channel spectral Principal Components Analysis (PCA) denoising. In the first stage, the Hybrid-MVDR performs multiple MVDRs using a dictionary of pre-defined noise field models and picks the minimum-power outcome, which benefits from the robustness of signal-independent beamforming and the performance of adaptive beamforming. In the second stage, the outcomes of Hybrid and Iso are jointly used in a two-channel PCA-based denoising to remove the 'musical noise' produced by Hybrid beamformer. On a dataset of real 'cocktail-party' recordings with head-worn array, the proposed method outperforms the baseline superdirective beamformer in noise suppression (fwSegSNR, SDR, SIR, SAR) and speech intelligibility (STOI) with similar speech quality (PESQ) improvement.

【6】 Learning Cross-lingual Visual Speech Representations

标题：学习跨语言的视觉语音表征链接：http://arxiv.org/pdf/2303.09455v1

*与cs.SD语音【1】为同一篇

作者：Andreas Zinonos,Alexandros Haliassos,Pingchuan Ma,Stavros Petridis,Maja Pantic机构：Department of Computing, Imperial College London, UK摘要：跨语言自监督学习是近几年来研究的热点。然而，目前的工作只探索了使用音频信号来创建表示。在这篇论文中，我们研究了跨语言的自监督视觉表征学习。我们使用最近提出的原始视听语音编码器（RAVEn）框架来使用未标记的多语言数据预训练视听模型，然后在标记的转录上微调视觉模型。我们的实验表明：（1）数据量较大的多语言模型的性能优于单语言模型，但在数据量一定的情况下，单语言模型的性能更好;（2）多语种训练优于纯英语训练;（3）使用更相似的语言产生更好的结果;以及（4）对看不见的语言的微调与使用预训练集中的目标语言是有竞争力的。我们希望我们的研究能对非英语语言表征学习的研究有所启发。摘要：Cross-lingual self-supervised learning has been a growing research topic in the last few years. However, current works only explored the use of audio signals to create representations. In this work, we study cross-lingual self-supervised visual representation learning. We use the recently-proposed Raw Audio-Visual Speech Encoders (RAVEn) framework to pre-train an audio-visual model with unlabelled multilingual data, and then fine-tune the visual model on labelled transcriptions. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance; (2) multi-lingual outperforms English-only pre-training; (3) using languages which are more similar yields better results; and (4) fine-tuning on unseen languages is competitive to using the target language in the pre-training set. We hope our study inspires future research on non-English-only speech representation learning.

【7】 Trustera: A Live Conversation Redaction System标题：Trustera：一个实时对话编辑系统链接：http://arxiv.org/pdf/2303.09438v1

*与cs.SD语音【2】为同一篇

作者：Evandro Gouvêa,Ali Dadgar,Shahab Jalalvand,Rathi Chengalvarayan,Badrinath Jayakumar,Ryan Price,Nicholas Ruiz,Jennifer McGovern,Srinivas Bangalore,Ben Stern备注：5机构：Interactions LLC, New Providence, New Jersey, USA摘要：Trustera是第一个在实时对话中编辑个人身份信息（PII）的功能系统，可消除代理人听到敏感信息的需要，同时保持客户与代理人实时对话的自然性。与呼叫后编辑相反，音频屏蔽在客户开始与PII实体交谈时立即开始。这显著降低了PII被拦截或存储在不安全数据存储中的风险。Trustera的架构由自动语音识别、自然语言理解和现场音频编辑器模块组成。该系统的目标有三：编辑作为PII的实体，屏蔽发送给代理的音频，同时捕获该实体，以便捕获的PII可用于支付交易或呼叫者识别。Trustera目前正被成千上万的代理商用来保护客户的敏感信息。摘要：Trustera, the first functional system that redacts personally identifiable information (PII) in real-time spoken conversations to remove agents' need to hear sensitive information while preserving the naturalness of live customer-agent conversations. As opposed to post-call redaction, audio masking starts as soon as the customer begins speaking to a PII entity. This significantly reduces the risk of PII being intercepted or stored in insecure data storage. Trustera's architecture consists of a pipeline of automatic speech recognition, natural language understanding, and a live audio redactor module. The system's goal is three-fold: redact entities that are PII, mask the audio that goes to the agent, and at the same time capture the entity, so that the captured PII can be used for a payment transaction or caller identification. Trustera is currently being used by thousands of agents to secure customers' sensitive information.

【8】 Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation标题：用于音频驱动的协同语音手势生成的驯服扩散模型链接：http://arxiv.org/pdf/2303.09119v1

*与cs.SD语音【3】为同一篇

作者：Lingting Zhu,Xian Liu,Xuanyu Liu,Rui Qian,Ziwei Liu,Lequan Yu机构：The University of Hong Kong, The Chinese University of Hong Kong, Nanyang Technological University备注：Accepted by IEEECVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023. 10 pages, 3 figures摘要：动画化虚拟化身以做出共语音姿势促进人机交互中的各种应用。现有方法主要依赖于生成式对抗网络（generative adversarial networks，GANs），GANs存在模式崩溃和训练不稳定等问题，难以获得准确的音频-手势联合分布.本文提出了一种新的基于扩散的框架，称为扩散共语手势（DiffGesture），以有效地捕获跨模态的音频-手势关联并保持时间相干性，从而实现高保真音频驱动的共语手势生成。具体来说，我们首先在骨架序列和音频的片段上建立扩散条件生成过程，以使整个框架成为可能。然后，设计了一种新的扩散音频-手势Transformer，以更好地处理来自多个模态的信息，并对长期时间依赖性进行建模。此外，为了消除时间上的不一致性，我们提出了一个有效的扩散手势稳定器与退火噪声采样策略。受益于扩散模型的架构优势，我们进一步引入隐式无分类器指导来权衡多样性和手势质量。大量的实验表明，DiffGesture算法具有最佳的性能，能以更好的模式覆盖率和更强的音频相关性呈现连贯的手势。代码可从www.example.com获取https: github.com Advocate99 DiffGesture。摘要：Animating virtual avatars to make co-speech gestures facilitates various applications in human-machine interaction. The existing methods mainly rely on generative adversarial networks (GANs), which typically suffer from notorious mode collapse and unstable training, thus making it difficult to learn accurate audio-gesture joint distributions. In this work, we propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to effectively capture the cross-modal audio-to-gesture associations and preserve temporal coherence for high-fidelity audio-driven co-speech gesture generation. Specifically, we first establish the diffusion-conditional generation process on clips of skeleton sequences and audio to enable the whole framework. Then, a novel Diffusion Audio-Gesture Transformer is devised to better attend to the information from multiple modalities and model the long-term temporal dependency. Moreover, to eliminate temporal inconsistency, we propose an effective Diffusion Gesture Stabilizer with an annealed noise sampling strategy. Benefiting from the architectural advantages of diffusion models, we further incorporate implicit classifier-free guidance to trade off between diversity and gesture quality. Extensive experiments demonstrate that DiffGesture achieves state-of-theart performance, which renders coherent gestures with better mode coverage and stronger audio correlations. Code is available at https: github.com Advocate99 DiffGesture.

【9】 Improving Perceptual Quality, Intelligibility, and Acoustics on VoIP Platforms标题：提高VoIP平台的感知质量、清晰度和音质链接：http://arxiv.org/pdf/2303.09048v1

*与cs.SD语音【4】为同一篇

作者：Joseph Konan,Ojas Bhargave,Shikhar Agnihotri,Hojeong Lee,Ankit Shah,Shuo Han,Yunyang Zeng,Amanda Shu,Haohui Liu,Xuankai Chang,Hamza Khalid,Minseon Gwak,Kawon Lee,Minjeong Kim,Bhiksha Raj机构：†Carnegie Mellon University, Pittsburgh备注：Under review at European Association for Signal Processing. 5 pages摘要：本文提出了一种微调在深度噪声抑制（DNS）2020挑战赛中训练的模型的方法，以提高其在互联网协议语音（VoIP）应用中的性能。我们的方法涉及调整DNS 2020模型以适应VoIP通信的特定声学特征，包括压缩、传输和平台特定处理导致的失真和伪像。为此，我们提出了一个多任务学习框架，为VoIP DNS的联合优化噪声抑制和VoIP特定的声学语音增强。我们评估了我们的方法在不同的VoIP场景，并表明它优于行业性能和国家的最先进的方法语音增强VoIP应用。我们的结果证明了在DNS-2020上训练的模型的潜力，以改善和定制到使用VoIP-DNS的不同VoIP平台，其发现在语音识别，语音助手和电信等领域有重要的应用。摘要：In this paper, we present a method for fine-tuning models trained on the Deep Noise Suppression (DNS) 2020 Challenge to improve their performance on Voice over Internet Protocol (VoIP) applications. Our approach involves adapting the DNS 2020 models to the specific acoustic characteristics of VoIP communications, which includes distortion and artifacts caused by compression, transmission, and platform-specific processing. To this end, we propose a multi-task learning framework for VoIP-DNS that jointly optimizes noise suppression and VoIP-specific acoustics for speech enhancement. We evaluate our approach on a diverse VoIP scenarios and show that it outperforms both industry performance and state-of-the-art methods for speech enhancement on VoIP applications. Our results demonstrate the potential of models trained on DNS-2020 to be improved and tailored to different VoIP platforms using VoIP-DNS, whose findings have important applications in areas such as speech recognition, voice assistants, and telecommunication.

机器翻译，仅供参考