今日论文合集：cs.SD语音13篇，eess.AS音频处理14篇。【1】 Efficient Speech Translation with Pre-trained Models
标题：基于预训练模型的高效语音翻译
链接：https://arxiv.org/abs/2211.04939
作者：Zhaolin Li,Jan Niehues机构：Department of Data Science and Knowledge Engineering, Maastricht University, The Netherlands, Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany摘要：当建立最先进的语音翻译模型时，由于训练数据量大和模型复杂，需要大量的计算资源是一个重大障碍。预训练模型的可用性是有效构建强大语音翻译系统的一个有希望的机会。在第一步，我们研究有效的策略，以建立级联和端到端的语音翻译系统的基础上，预先训练的模型。使用此策略，我们可以在单个GPU上训练和应用模型。虽然端到端模型显示出优于级联模型的翻译性能，但该技术的应用在需要额外的端到端训练数据方面存在限制。在第二步中，我们提出了一个额外的相似性损失，以鼓励模型生成相似的隐藏表示的语音和副本。在端到端训练数据有限的情况下，使用该技术可以提高数据效率，并将翻译质量提高6个BLEU点。摘要：When building state-of-the-art speech translation models, the need for large computational resources is a significant obstacle due to the large training data size and complex models. The availability of pre-trained models is a promising opportunity to build strong speech translation systems efficiently. In a first step, we investigate efficient strategies to build cascaded and end-to-end speech translation systems based on pre-trained models. Using this strategy, we can train and apply the models on a single GPU. While the end-to-end models show superior translation performance to cascaded ones, the application of this technology has a limitation on the need for additional end-to-end training data. In a second step, we proposed an additional similarity loss to encourage the model to generate similar hidden representations for speech and transcript. Using this technique, we can increase the data efficiency and improve the translation quality by 6 BLEU points in scenarios with limited end-to-end training data.
【2】 Global, and Local Optimization Beamforming for Acoustic Broadband Sources
标题：声学宽带信源的全局和局部优化波束形成
链接：https://arxiv.org/abs/2211.04921
机构：German Aerospace Center (DLR), G¨ottingen, Germany备注：Submitted to Journal of Sound and Vibration摘要：本文提出了一种扩展的宽带声源全局优化波束形成算法。假定诸如源位置、空间形状、多极旋转或流动性质的性质可以在频率上参数化，则可以同时对所有频率执行CSM拟合。数值分析表明，标准全局优化问题的非线性误差函数类似于点扩散函数，并包含局部极小值，但可以通过所提出的宽带优化来改善。不仅增加了宽带优化过程中方程与未知变量的比率，而且还平滑了成本函数。它还简化了识别源和根据结果重建其光谱的过程。结果表明，在合成单极子上，该方法优于标准全局优化方法和CLEAN-SC方法，对于真实数据，宽带全局优化方法、标准全局优化方法和CLEAN-SC方法的结果相似。然而，所提出的方法不需要识别和整合感兴趣区域。结果表明，通过合理选取初始值，全局优化问题可转化为局部优化问题，且具有相似的结果。进一步地，该方法能够识别具有不同极点幅值和未知极点旋转的多极子。摘要：This paper presents an extension to global optimization beamforming for acoustic broadband sources. Given, that properties such as the source location, spatial shape, multipole rotation, or flow properties can be parameterized over the frequency, a CSM-fitting can be performed for all frequencies at the same time. A numerical analysis shows that the non-linear error function for the standard global optimization problem is similar to a Point Spread Function and contains local minima, but can be improved with the proposed broadband optimization. Not only increases the broadband optimization process the ratio of equations to unknown variables, but it also smooths out the cost function. It also simplifies the process of identifying sources and reconstructing their spectra from the results. The paper shows that the method is superior on synthetic monopoles compared to standard global optimization and CLEAN-SC. For real-world data the results of broadband global optimization, standard global optimization, and CLEAN-SC are similar. However, the proposed method does not require the identification and integration of Regions Of Interest. It is shown, that by using reasonable initial values the global optimization problem reduces to a local optimization problem with similar results. Further, it is shown that the proposed method is able to identify multipoles with different pole amplitudes and unknown pole rotations.
【3】 Distribution-based Emotion Recognition in Conversation
标题：基于分布的会话情感识别
链接：https://arxiv.org/abs/2211.04834
作者：Wen Wu,Chao Zhang,Philip C. Woodland机构：Department of Engineering, University of Cambridge, Trumpington St., Cambridge, UK.摘要：会话中的自动情感识别（ERC）是实现情感感知的会话人工智能的关键。本文提出了一种基于分布的框架，将ERC描述为序列到序列的情感分布估计问题。由于情感本身的模糊性和人类感知的主观性，导致了情感标签的不一致性，本文从情感分布不确定性估计的角度对情感标签进行了处理。通过引入贝叶斯训练损失，将每个情感状态置于特定话语的狄利克雷先验分布下，以改善不确定性估计。在IEMOCAP数据集上的实验结果表明，ERC的分类性能优于基于单话语的分类系统，所提出的基于分布的ERC方法不仅具有更好的分类精度，而且具有更好的不确定性估计能力.摘要：Automatic emotion recognition in conversation (ERC) is crucial for emotion-aware conversational artificial intelligence. This paper proposes a distribution-based framework that formulates ERC as a sequence-to-sequence problem for emotion distribution estimation. The inherent ambiguity of emotions and the subjectivity of human perception lead to disagreements in emotion labels, which is handled naturally in our framework from the perspective of uncertainty estimation in emotion distributions. A Bayesian training loss is introduced to improve the uncertainty estimation by conditioning each emotional state on an utterance-specific Dirichlet prior distribution. Experimental results on the IEMOCAP dataset show that ERC outperformed the single-utterance-based system, and the proposed distribution-based ERC methods have not only better classification accuracy, but also show improved uncertainty estimation.
【4】 Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation
标题：基于Transformer-to-CNN知识蒸馏的高效大规模音频标注
链接：https://arxiv.org/abs/2211.04772
作者：Florian Schmid,Khaled Koutini,Gerhard Widmer机构：Institute of Computational Perception (CP-JKU),LIT Artificial Intelligence Lab, Johannes Kepler University Linz, Austria备注：Submitted to ICASSP 2023. Source Code available at: this https URL摘要：音频谱图Transformer模型统治着音频标记领域，超过了以前占主导地位的卷积神经网络（CNNs）。它们的优势在于能够纵向扩展和利用大规模数据集（如AudioSet）。然而，与CNN相比，Transformers在模型大小和计算要求方面要求更高。提出了一种基于离线知识提取的高效神经网络训练方法。所提出的训练方案和基于MobileNetV3的高效CNN设计使得模型在参数、计算效率和预测性能方面优于先前的解决方案。我们提供不同复杂度级别的模型，从低复杂度模型扩展到AudioSet上最新的.483 mAP性能。源代码位于：https://github.com/fschmid56/EfficientAT摘要：Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of .483 mAP on AudioSet. Source Code available at: https://github.com/fschmid56/EfficientAT
【5】 Improving Noisy Student Training on Non-target Domain Data for Automatic Speech Recognition
标题：自动语音识别中非目标域数据噪声训练的改进
链接：https://arxiv.org/abs/2211.04717
作者：Yu Chen,Wen Ding,Junjie Lai机构：The University of Hong Kong, Hong Kong China, NVIDIA, Shanghai China摘要：噪声学生训练（NST）最近在自动语音识别（ASR）中表现出了极强的性能。本文提出了一种LM Filter数据选择策略，以提高NST对ASR任务中非目标域数据的选择性能。生成带有和不带有语言模型的假设，并利用它们之间的CER差异作为过滤阈值。结果显示，与无数据过滤基线相比，显著改善了10.4%。在AISHELL-1测试集上，我们可以得到3.31%的CER，这是我们在没有任何其他监督数据的情况下所得到的最好结果。我们还对有监督的1000小时AISHELL-2数据集进行了评估，并获得了4.72% CER的竞争结果。摘要：Noisy Student Training (NST) has recently demonstrated extremely strong performance in Automatic Speech Recognition (ASR). In this paper, we propose a data selection strategy named LM Filter to improve the performances of NST on non-target domain data in ASR tasks. Hypothesis with and without Language Model are generated and CER differences between them are utilized as a filter threshold. Results reveal that significant improvements of 10.4% compared with no data filtering baselines. We can achieve 3.31% CER in AISHELL-1 test set, which is best result from our knowledge without any other supervised data. We also perform evaluations on supervised 1000 hour AISHELL-2 dataset and competitive results of 4.72% CER can be achieved.
【6】 SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations
标题：SpeechMatrix：一个大规模挖掘的多语种语音翻译语料库
链接：https://arxiv.org/abs/2211.04508
作者：Paul-Ambroise Duquenne,Hongyu Gong,Ning Dong,Jingfei Du,Ann Lee,Vedanuj Goswani,Changhan Wang,Juan Pino,Benoît Sagot,Holger Schwenk机构：Meta AI Research, Inria摘要：我们介绍了SpeechMatrix，这是一个从欧洲议会的真实演讲录音中挖掘出来的大规模多语言语音到语音翻译语料库。它包含136种语言对的语音对齐，总共有41.8万个小时的语音。为了评估这种并行语音的质量，我们仅在挖掘的数据上训练双语语音到语音翻译模型，并在EuroParl-ST，VoxPopuli和FLEURS测试集上建立广泛的基线结果。由于SpeechMatrix的多语言性，我们还探讨了多语言语音到语音的翻译，这是一个很少有其他作品涉及的主题。我们还证明了模型预训练和使用混合专家的稀疏缩放带来了巨大的翻译性能增益。挖掘的数据和模型是免费提供的。摘要：We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.
【7】 Towards Improved Room Impulse Response Estimation for Speech Recognition
标题：语音识别中改进的房间脉冲响应估计方法
链接：https://arxiv.org/abs/2211.04473
作者：Anton Ratnarajah,Ishwarya Ananthabhotla,Vamsi Krishna Ithapu,Pablo Hoffmann,Dinesh Manocha,Paul Calamia机构：University of Maryland, College Park, USA, Reality Labs Research at Meta, Redmond, WA USA摘要：我们提出在下游应用场景远场自动语音识别（ASR）的背景下表征和改善盲室脉冲响应（RIR）估计系统的性能。我们首先得出改进的RIR估计和改进的ASR性能之间的联系，作为评估神经RIR估计器的一种手段。然后，我们提出了一种基于GAN的结构，该结构对混响语音的RIR特征进行编码，并根据编码后的特征构建RIR，并使用一种新的能量衰减减轻损失来优化捕获输入混响语音的基于能量的属性。我们证明了我们的模型在声学基准测试（能量衰减缓解72%，早期反射能量度量22%）以及ASR评估任务（单词错误率6.9%）方面优于最先进的基线。摘要：We propose to characterize and improve the performance of blind room impulse response (RIR) estimation systems in the context of a downstream application scenario, far-field automatic speech recognition (ASR). We first draw the connection between improved RIR estimation and improved ASR performance, as a means of evaluating neural RIR estimators. We then propose a GAN-based architecture that encodes RIR features from reverberant speech and constructs an RIR from the encoded features, and uses a novel energy decay relief loss to optimize for capturing energy-based properties of the input reverberant speech. We show that our model outperforms the state-of-the-art baselines on acoustic benchmarks (by 72% on the energy decay relief and 22% on an early-reflection energy metric), as well as in an ASR evaluation task (by 6.9% in word error rate).
【8】 Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models
标题：偶然学习者：多语言自我监督模型中的口语识别
链接：https://arxiv.org/abs/2211.05103
作者：Travis M. Bartley,Fei Jia,Krishna C. Puvvada,Samuel Kriman,Boris Ginsburg机构：NVIDIA Corporation†, City University of New York⋆备注：Submitted to ICASSP 2023摘要：在这篇文章中，我们通过在多语言预训练范例中试验基于Conformer的架构来扩展先前用于语言识别的自监督方法。我们发现，预先训练的语音模型在较低层中最佳地编码语言辨别信息。进一步地，我们证明了从这些层获得的嵌入对于分类不可见的语言和不同的声学环境具有显著的鲁棒性，而不需要额外的训练。在VoxLingua107数据集上对预先训练的Conformer模型进行微调后，我们获得了与当前最先进的语言识别系统相似的结果。此外，我们的模型实现这一点所需的参数减少了5倍。我们通过NVIDIA NeMo工具包开源该模型。摘要：In this paper, we extend previous self-supervised approaches for language identification by experimenting with Conformer based architecture in a multilingual pre-training paradigm. We find that pre-trained speech models optimally encode language discriminatory information in lower layers. Further, we demonstrate that the embeddings obtained from these layers are significantly robust to classify unseen languages and different acoustic environments without additional training. After fine-tuning a pre-trained Conformer model on the VoxLingua107 dataset, we achieve results similar to current state-of-the-art systems for language identification. More, our model accomplishes this with 5x less parameters. We open-source the model through the NVIDIA NeMo toolkit.
【9】 A Diffeomorphic Flow-based Variational Framework for Multi-speaker Emotion Conversion
标题：一种基于微分同构流的多说话人情感转换变分框架
链接：https://arxiv.org/abs/2211.05071
作者：Ravi Shankar,Hsi-Wei Hsieh,Nicolas Charon,Archana Venkataraman机构：Hsi-Wei Hsieh and Nicolas Charon are with the Department of AppliedMath and Statistics备注：Accepted in IEEE Transactions on Audio, Speech and Language Processing摘要：本文提出了一种新的语音非并行情感转换框架。我们的框架基于两个关键贡献。首先，我们提出了一个流行的CycleGAN模型的随机版本。我们修改的损失函数引入了一个Kullback Leibler（KL）散度项，该散度项将生成器学习到的源数据和目标数据分布对齐，从而克服了样本生成的局限性。通过对这个随机损失函数进行变分近似，我们证明了我们的KL散度项可以通过成对密度鉴别器来实现。我们称这种新结构为变分CycleGAN（VCGAN）。其次，我们将目标情感的韵律特征建模为源韵律特征的平滑且可学习的变形。该方法提供了隐式正则化，其在对未看到的和分布之外的说话者的更好的范围对准方面提供了关键优点。我们进行了严格的实验和对比研究，以证明我们提出的框架是相当稳健的，与几个最先进的基线高性能。摘要：This paper introduces a new framework for non-parallel emotion conversion in speech. Our framework is based on two key contributions. First, we propose a stochastic version of the popular CycleGAN model. Our modified loss function introduces a Kullback Leibler (KL) divergence term that aligns the source and target data distributions learned by the generators, thus overcoming the limitations of sample wise generation. By using a variational approximation to this stochastic loss function, we show that our KL divergence term can be implemented via a paired density discriminator. We term this new architecture a variational CycleGAN (VCGAN). Second, we model the prosodic features of target emotion as a smooth and learnable deformation of the source prosodic features. This approach provides implicit regularization that offers key advantages in terms of better range alignment to unseen and out of distribution speakers. We conduct rigorous experiments and comparative studies to demonstrate that our proposed framework is fairly robust with high performance against several state-of-the-art baselines.
【10】 A Comparative Study of Data Augmentation Techniques for Deep Learning Based Emotion Recognition
标题：基于深度学习的情感识别数据增强技术比较研究
链接：https://arxiv.org/abs/2211.05047
作者：Ravi Shankar,Abdouh Harouna Kenfack,Arjun Somayazulu,Archana Venkataraman机构：Department of Electrical and Computer Engineering, Johns Hopkins University, USA, Department of Applied Mathematics and Statistics, Johns Hopkins University, USA, Department of Computer Science, Johns Hopkins University, USA摘要：语音中的自动情感识别是一个长期存在的问题。虽然早期的情绪识别工作依赖于手工制作的特征和简单的分类器，但该领域现在已经接受了使用深度神经网络的端到端特征学习和分类。与这些模型并行的是，研究人员提出了几种数据扩充技术，以增加现有标记数据集的大小和可变性。尽管在该领域有许多开创性的贡献，但我们仍然对网络架构和数据增强选择之间的相互作用缺乏了解。此外，只有少数研究证明了特定模型在多个数据集上的推广性，这是稳健的真实世界性能的先决条件。本文对当前流行的情感识别深度学习方法进行了综合评价。为了消除偏倚，我们使用VESUS数据集修复模型架构和优化超参数，然后使用重复的5重交叉验证来评估IEMOCAP和CREMA-D数据集的性能。我们的结果表明，语音信号中的长程相关性对于情感识别至关重要，并且速度/速率增强提供了跨模型的最鲁棒的性能增益。摘要：Automated emotion recognition in speech is a long-standing problem. While early work on emotion recognition relied on hand-crafted features and simple classifiers, the field has now embraced end-to-end feature learning and classification using deep neural networks. In parallel to these models, researchers have proposed several data augmentation techniques to increase the size and variability of existing labeled datasets. Despite many seminal contributions in the field, we still have a poor understanding of the interplay between the network architecture and the choice of data augmentation. Moreover, only a handful of studies demonstrate the generalizability of a particular model across multiple datasets, which is a prerequisite for robust real-world performance. In this paper, we conduct a comprehensive evaluation of popular deep learning approaches for emotion recognition. To eliminate bias, we fix the model architectures and optimization hyperparameters using the VESUS dataset and then use repeated 5-fold cross validation to evaluate the performance on the IEMOCAP and CREMA-D datasets. Our results demonstrate that long-range dependencies in the speech signal are critical for emotion recognition and that speed/rate augmentation offers the most robust performance gain across models.
【11】 Absolute decision corrupts absolutely: conservative online speaker diarisation
标题：绝对决定绝对腐败：保守的在线演讲者腹泻
链接：https://arxiv.org/abs/2211.04768
作者：Youngki Kwon,Hee-Soo Heo,Bong-Jin Lee,You Jin Kim,Jee-weon Jung机构：Naver Corporation, South Korea备注：5pages, 2 figure, 4 tables, submitted to ICASSP摘要：我们的重点在于开发一个在线演讲者日记框架，该框架在不同领域表现出稳健的性能。在在线演讲者日记中，实时生成的输出是不可逆的，在输入会话的早期阶段的一些错误判断可能导致灾难性的结果。我们假设谨慎地增加估计说话者的数量在许多其他因素中是最重要的。因此，我们提出的框架包括当系统判断过去的增加是错误的时，将发言者的数目减少一个。采用双缓冲区、检查点和质心的方法，将检查点和轮廓系数相结合来估计说话人的个数，质心代表说话人。同样，我们相信可以从一个扬声器产生多于一个的质心。因此，我们设计了一种基于聚类的标签匹配技术来实时分配标签。所得到的系统是轻质的，但令人惊讶地有效。该系统在DIHARD 2和3数据集上表现出了最先进的性能，在AMI和VoxConverse测试集上也具有竞争力。摘要：Our focus lies in developing an online speaker diarisation framework which demonstrates robust performance across diverse domains. In online speaker diarisation, outputs generated in real-time are irreversible, and a few misjudgements in the early phase of an input session can lead to catastrophic results. We hypothesise that cautiously increasing the number of estimated speakers is of paramount importance among many other factors. Thus, our proposed framework includes decreasing the number of speakers by one when the system judges that an increase in the past was faulty. We also adopt dual buffers, checkpoints and centroids, where checkpoints are combined with silhouette coefficients to estimate the number of speakers and centroids represent speakers. Again, we believe that more than one centroid can be generated from one speaker. Thus we design a clustering-based label matching technique to assign labels in real-time. The resulting system is lightweight yet surprisingly effective. The system demonstrates state-of-the-art performance on DIHARD 2 and 3 datasets, where it is also competitive in AMI and VoxConverse test sets.
【12】 Expressive-VC: Highly Expressive Voice Conversion with Attention Fusion of Bottleneck and Perturbation Features
标题：Expressitive-VC：具有瓶颈和扰动特征注意力融合的高表现力语音转换
链接：https://arxiv.org/abs/2211.04710
作者：Ziqian Ning,Qicong Xie,Pengcheng Zhu,Zhichao Wang,Liumeng Xue,Jixun Yao,Lei Xie,Mengxiao Bi机构：Northwestern Polytechnical University, Xi’an, China, Fuxi AI Lab, NetEase Inc., Hangzhou, China摘要：高表达语音的语音转换具有挑战性。当前的方法在说话者相似性、可理解性和表达性之间的平衡上挣扎。针对这一问题，提出了一种新的端到端语音转换框架Expressive-VC，该框架综合了神经瓶颈特征（BNF）方法和信息扰动方法的优点.具体而言，我们使用BNF编码器和Perturbed-Wav编码器来形成内容提取器，分别学习语言和副语言特征，其中BNF来自鲁棒的预训练ASR模型，并且扰动波在信号扰动之后变为说话人无关的。该算法通过引入注意机制融合语言特征和副语言特征，将说话人相关的韵律特征作为注意查询，韵律编码器嵌入目标说话人，以源语音的归一化基音和能量作为输入。最后，解码器使用综合特征和说话者相关韵律特征来生成转换后的语音。实验结果表明，Expressive-VC系统比现有的几种系统具有更好的性能，既能获得高的源语音表达能力，又能获得与目标说话人的高相似度;同时很好地保持了可懂度。摘要：Voice conversion for highly expressive speech is challenging. Current approaches struggle with the balancing between speaker similarity, intelligibility and expressiveness. To address this problem, we propose Expressive-VC, a novel end-to-end voice conversion framework that leverages advantages from both neural bottleneck feature (BNF) approach and information perturbation approach. Specifically, we use a BNF encoder and a Perturbed-Wav encoder to form a content extractor to learn linguistic and para-linguistic features respectively, where BNFs come from a robust pre-trained ASR model and the perturbed wave becomes speaker-irrelevant after signal perturbation. We further fuse the linguistic and para-linguistic features through an attention mechanism, where speaker-dependent prosody features are adopted as the attention query, which result from a prosody encoder with target speaker embedding and normalized pitch and energy of source speech as input. Finally the decoder consumes the integrated features and the speaker-dependent prosody feature to generate the converted speech. Experiments demonstrate that Expressive-VC is superior to several state-of-the-art systems, achieving both high expressiveness captured from the source speech and high speaker similarity with the target speaker; meanwhile intelligibility is well maintained.
【13】 PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping
标题：PhaseAug：一种模拟一对多映射的语音合成可区分增强算法
链接：https://arxiv.org/abs/2211.04610
作者：Junhyeok Lee,Seungu Han,Hyunjae Cho,Wonbin Jung机构：MINDsLab Inc., Republic of Korea, Seoul National University (SNU), Republic of Korea, Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea备注：Submitted to ICASSP 2023摘要：现有的基于生成对抗网络（GAN）的神经声码器仅能从成对的Mel谱图中重构出准确的真实波形，没有考虑语音合成中的一对多关系。这种常规训练导致鉴别器和生成器两者的过拟合，从而导致所生成的音频信号中的周期性伪像。在本文中，我们提出了PhaseAug，它是第一个用于语音合成的可微增强算法，通过旋转每个频率单元的相位来模拟一对多映射。通过我们提出的方法，我们在不修改任何体系结构的情况下，性能优于基线。代码和音频样本将在www.example.com上提供https://github.com/mindslab-ai/phaseaug。摘要：Previous generative adversarial network (GAN)-based neural vocoders are trained to reconstruct the exact ground truth waveform from the paired mel-spectrogram and do not consider the one-to-many relationship of speech synthesis. This conventional training causes overfitting for both the discriminators and the generator, leading to the periodicity artifacts in the generated audio signal. In this work, we present PhaseAug, the first differentiable augmentation for speech synthesis that rotates the phase of each frequency bin to simulate one-to-many mapping. With our proposed method, we outperform baselines without any architecture modification. Code and audio samples will be available at https://github.com/mindslab-ai/phaseaug.
【1】 Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models
标题：偶然学习者：多语言自我监督模型中的口语识别
链接：https://arxiv.org/abs/2211.05103
* 与cs.SD语音【8】为同一篇
作者：Travis M. Bartley,Fei Jia,Krishna C. Puvvada,Samuel Kriman,Boris Ginsburg机构：NVIDIA Corporation†, City University of New York⋆备注：Submitted to ICASSP 2023摘要：在这篇文章中，我们通过在多语言预训练范例中试验基于Conformer的架构来扩展先前用于语言识别的自监督方法。我们发现，预先训练的语音模型在较低层中最佳地编码语言辨别信息。进一步地，我们证明了从这些层获得的嵌入对于分类不可见的语言和不同的声学环境具有显著的鲁棒性，而不需要额外的训练。在VoxLingua107数据集上对预先训练的Conformer模型进行微调后，我们获得了与当前最先进的语言识别系统相似的结果。此外，我们的模型实现这一点所需的参数减少了5倍。我们通过NVIDIA NeMo工具包开源该模型。摘要：In this paper, we extend previous self-supervised approaches for language identification by experimenting with Conformer based architecture in a multilingual pre-training paradigm. We find that pre-trained speech models optimally encode language discriminatory information in lower layers. Further, we demonstrate that the embeddings obtained from these layers are significantly robust to classify unseen languages and different acoustic environments without additional training. After fine-tuning a pre-trained Conformer model on the VoxLingua107 dataset, we achieve results similar to current state-of-the-art systems for language identification. More, our model accomplishes this with 5x less parameters. We open-source the model through the NVIDIA NeMo toolkit.
【2】 A Diffeomorphic Flow-based Variational Framework for Multi-speaker Emotion Conversion
标题：一种基于微分同构流的多说话人情感转换变分框架
链接：https://arxiv.org/abs/2211.05071
* 与cs.SD语音【9】为同一篇
作者：Ravi Shankar,Hsi-Wei Hsieh,Nicolas Charon,Archana Venkataraman机构：Hsi-Wei Hsieh and Nicolas Charon are with the Department of AppliedMath and Statistics备注：Accepted in IEEE Transactions on Audio, Speech and Language Processing摘要：本文提出了一种新的语音非并行情感转换框架。我们的框架基于两个关键贡献。首先，我们提出了一个流行的CycleGAN模型的随机版本。我们修改的损失函数引入了一个Kullback Leibler（KL）散度项，该散度项将生成器学习到的源数据和目标数据分布对齐，从而克服了样本生成的局限性。通过对这个随机损失函数进行变分近似，我们证明了我们的KL散度项可以通过成对密度鉴别器来实现。我们称这种新结构为变分CycleGAN（VCGAN）。其次，我们将目标情感的韵律特征建模为源韵律特征的平滑且可学习的变形。该方法提供了隐式正则化，其在对未看到的和分布之外的说话者的更好的范围对准方面提供了关键优点。我们进行了严格的实验和对比研究，以证明我们提出的框架是相当稳健的，与几个最先进的基线高性能。摘要：This paper introduces a new framework for non-parallel emotion conversion in speech. Our framework is based on two key contributions. First, we propose a stochastic version of the popular CycleGAN model. Our modified loss function introduces a Kullback Leibler (KL) divergence term that aligns the source and target data distributions learned by the generators, thus overcoming the limitations of sample wise generation. By using a variational approximation to this stochastic loss function, we show that our KL divergence term can be implemented via a paired density discriminator. We term this new architecture a variational CycleGAN (VCGAN). Second, we model the prosodic features of target emotion as a smooth and learnable deformation of the source prosodic features. This approach provides implicit regularization that offers key advantages in terms of better range alignment to unseen and out of distribution speakers. We conduct rigorous experiments and comparative studies to demonstrate that our proposed framework is fairly robust with high performance against several state-of-the-art baselines.
【3】 A Comparative Study of Data Augmentation Techniques for Deep Learning Based Emotion Recognition
标题：基于深度学习的情感识别数据增强技术比较研究
链接：https://arxiv.org/abs/2211.05047
* 与cs.SD语音【10】为同一篇
作者：Ravi Shankar,Abdouh Harouna Kenfack,Arjun Somayazulu,Archana Venkataraman机构：Department of Electrical and Computer Engineering, Johns Hopkins University, USA, Department of Applied Mathematics and Statistics, Johns Hopkins University, USA, Department of Computer Science, Johns Hopkins University, USA摘要：语音中的自动情感识别是一个长期存在的问题。虽然早期的情绪识别工作依赖于手工制作的特征和简单的分类器，但该领域现在已经接受了使用深度神经网络的端到端特征学习和分类。与这些模型并行的是，研究人员提出了几种数据扩充技术，以增加现有标记数据集的大小和可变性。尽管在该领域有许多开创性的贡献，但我们仍然对网络架构和数据增强选择之间的相互作用缺乏了解。此外，只有少数研究证明了特定模型在多个数据集上的推广性，这是稳健的真实世界性能的先决条件。本文对当前流行的情感识别深度学习方法进行了综合评价。为了消除偏倚，我们使用VESUS数据集修复模型架构和优化超参数，然后使用重复的5重交叉验证来评估IEMOCAP和CREMA-D数据集的性能。我们的结果表明，语音信号中的长程相关性对于情感识别至关重要，并且速度/速率增强提供了跨模型的最鲁棒的性能增益。摘要：Automated emotion recognition in speech is a long-standing problem. While early work on emotion recognition relied on hand-crafted features and simple classifiers, the field has now embraced end-to-end feature learning and classification using deep neural networks. In parallel to these models, researchers have proposed several data augmentation techniques to increase the size and variability of existing labeled datasets. Despite many seminal contributions in the field, we still have a poor understanding of the interplay between the network architecture and the choice of data augmentation. Moreover, only a handful of studies demonstrate the generalizability of a particular model across multiple datasets, which is a prerequisite for robust real-world performance. In this paper, we conduct a comprehensive evaluation of popular deep learning approaches for emotion recognition. To eliminate bias, we fix the model architectures and optimization hyperparameters using the VESUS dataset and then use repeated 5-fold cross validation to evaluate the performance on the IEMOCAP and CREMA-D datasets. Our results demonstrate that long-range dependencies in the speech signal are critical for emotion recognition and that speed/rate augmentation offers the most robust performance gain across models.
【4】 Absolute decision corrupts absolutely: conservative online speaker diarisation
标题：绝对决定绝对腐败：保守的在线演讲者腹泻
链接：https://arxiv.org/abs/2211.04768
* 与cs.SD语音【11】为同一篇
作者：Youngki Kwon,Hee-Soo Heo,Bong-Jin Lee,You Jin Kim,Jee-weon Jung机构：Naver Corporation, South Korea备注：5pages, 2 figure, 4 tables, submitted to ICASSP摘要：我们的重点在于开发一个在线演讲者日记框架，该框架在不同领域表现出稳健的性能。在在线演讲者日记中，实时生成的输出是不可逆的，在输入会话的早期阶段的一些错误判断可能导致灾难性的结果。我们假设谨慎地增加估计说话者的数量在许多其他因素中是最重要的。因此，我们提出的框架包括当系统判断过去的增加是错误的时，将发言者的数目减少一个。采用双缓冲区、检查点和质心的方法，将检查点和轮廓系数相结合来估计说话人的个数，质心代表说话人。同样，我们相信可以从一个扬声器产生多于一个的质心。因此，我们设计了一种基于聚类的标签匹配技术来实时分配标签。所得到的系统是轻质的，但令人惊讶地有效。该系统在DIHARD 2和3数据集上表现出了最先进的性能，在AMI和VoxConverse测试集上也具有竞争力。摘要：Our focus lies in developing an online speaker diarisation framework which demonstrates robust performance across diverse domains. In online speaker diarisation, outputs generated in real-time are irreversible, and a few misjudgements in the early phase of an input session can lead to catastrophic results. We hypothesise that cautiously increasing the number of estimated speakers is of paramount importance among many other factors. Thus, our proposed framework includes decreasing the number of speakers by one when the system judges that an increase in the past was faulty. We also adopt dual buffers, checkpoints and centroids, where checkpoints are combined with silhouette coefficients to estimate the number of speakers and centroids represent speakers. Again, we believe that more than one centroid can be generated from one speaker. Thus we design a clustering-based label matching technique to assign labels in real-time. The resulting system is lightweight yet surprisingly effective. The system demonstrates state-of-the-art performance on DIHARD 2 and 3 datasets, where it is also competitive in AMI and VoxConverse test sets.
【5】 Expressive-VC: Highly Expressive Voice Conversion with Attention Fusion of Bottleneck and Perturbation Features
标题：Expressitive-VC：具有瓶颈和扰动特征注意力融合的高表现力语音转换
链接：https://arxiv.org/abs/2211.04710
* 与cs.SD语音【12】为同一篇
作者：Ziqian Ning,Qicong Xie,Pengcheng Zhu,Zhichao Wang,Liumeng Xue,Jixun Yao,Lei Xie,Mengxiao Bi机构：Northwestern Polytechnical University, Xi’an, China, Fuxi AI Lab, NetEase Inc., Hangzhou, China摘要：高表达语音的语音转换具有挑战性。当前的方法在说话者相似性、可理解性和表达性之间的平衡上挣扎。针对这一问题，提出了一种新的端到端语音转换框架Expressive-VC，该框架综合了神经瓶颈特征（BNF）方法和信息扰动方法的优点.具体而言，我们使用BNF编码器和Perturbed-Wav编码器来形成内容提取器，分别学习语言和副语言特征，其中BNF来自鲁棒的预训练ASR模型，并且扰动波在信号扰动之后变为说话人无关的。该算法通过引入注意机制融合语言特征和副语言特征，将说话人相关的韵律特征作为注意查询，韵律编码器嵌入目标说话人，以源语音的归一化基音和能量作为输入。最后，解码器使用综合特征和说话者相关韵律特征来生成转换后的语音。实验结果表明，Expressive-VC系统比现有的几种系统具有更好的性能，既能获得高的源语音表达能力，又能获得与目标说话人的高相似度;同时很好地保持了可懂度。摘要：Voice conversion for highly expressive speech is challenging. Current approaches struggle with the balancing between speaker similarity, intelligibility and expressiveness. To address this problem, we propose Expressive-VC, a novel end-to-end voice conversion framework that leverages advantages from both neural bottleneck feature (BNF) approach and information perturbation approach. Specifically, we use a BNF encoder and a Perturbed-Wav encoder to form a content extractor to learn linguistic and para-linguistic features respectively, where BNFs come from a robust pre-trained ASR model and the perturbed wave becomes speaker-irrelevant after signal perturbation. We further fuse the linguistic and para-linguistic features through an attention mechanism, where speaker-dependent prosody features are adopted as the attention query, which result from a prosody encoder with target speaker embedding and normalized pitch and energy of source speech as input. Finally the decoder consumes the integrated features and the speaker-dependent prosody feature to generate the converted speech. Experiments demonstrate that Expressive-VC is superior to several state-of-the-art systems, achieving both high expressiveness captured from the source speech and high speaker similarity with the target speaker; meanwhile intelligibility is well maintained.
【6】 PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping
标题：PhaseAug：一种模拟一对多映射的语音合成可区分增强算法
链接：https://arxiv.org/abs/2211.04610
* 与cs.SD语音【13】为同一篇
作者：Junhyeok Lee,Seungu Han,Hyunjae Cho,Wonbin Jung机构：MINDsLab Inc., Republic of Korea, Seoul National University (SNU), Republic of Korea, Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea备注：Submitted to ICASSP 2023摘要：现有的基于生成对抗网络（GAN）的神经声码器仅能从成对的Mel谱图中重构出准确的真实波形，没有考虑语音合成中的一对多关系。这种常规训练导致鉴别器和生成器两者的过拟合，从而导致所生成的音频信号中的周期性伪像。在本文中，我们提出了PhaseAug，它是第一个用于语音合成的可微增强算法，通过旋转每个频率单元的相位来模拟一对多映射。通过我们提出的方法，我们在不修改任何体系结构的情况下，性能优于基线。代码和音频样本将在www.example.com上提供https://github.com/mindslab-ai/phaseaug。摘要：Previous generative adversarial network (GAN)-based neural vocoders are trained to reconstruct the exact ground truth waveform from the paired mel-spectrogram and do not consider the one-to-many relationship of speech synthesis. This conventional training causes overfitting for both the discriminators and the generator, leading to the periodicity artifacts in the generated audio signal. In this work, we present PhaseAug, the first differentiable augmentation for speech synthesis that rotates the phase of each frequency bin to simulate one-to-many mapping. With our proposed method, we outperform baselines without any architecture modification. Code and audio samples will be available at https://github.com/mindslab-ai/phaseaug.
【7】 Efficient Speech Translation with Pre-trained Models
标题：基于预训练模型的高效语音翻译
链接：https://arxiv.org/abs/2211.04939
* 与cs.SD语音【1】为同一篇
作者：Zhaolin Li,Jan Niehues机构：Department of Data Science and Knowledge Engineering, Maastricht University, The Netherlands, Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany摘要：当建立最先进的语音翻译模型时，由于训练数据量大和模型复杂，需要大量的计算资源是一个重大障碍。预训练模型的可用性是有效构建强大语音翻译系统的一个有希望的机会。在第一步，我们研究有效的策略，以建立级联和端到端的语音翻译系统的基础上，预先训练的模型。使用此策略，我们可以在单个GPU上训练和应用模型。虽然端到端模型显示出优于级联模型的翻译性能，但该技术的应用在需要额外的端到端训练数据方面存在限制。在第二步中，我们提出了一个额外的相似性损失，以鼓励模型生成相似的隐藏表示的语音和副本。在端到端训练数据有限的情况下，使用该技术可以提高数据效率，并将翻译质量提高6个BLEU点。摘要：When building state-of-the-art speech translation models, the need for large computational resources is a significant obstacle due to the large training data size and complex models. The availability of pre-trained models is a promising opportunity to build strong speech translation systems efficiently. In a first step, we investigate efficient strategies to build cascaded and end-to-end speech translation systems based on pre-trained models. Using this strategy, we can train and apply the models on a single GPU. While the end-to-end models show superior translation performance to cascaded ones, the application of this technology has a limitation on the need for additional end-to-end training data. In a second step, we proposed an additional similarity loss to encourage the model to generate similar hidden representations for speech and transcript. Using this technique, we can increase the data efficiency and improve the translation quality by 6 BLEU points in scenarios with limited end-to-end training data.
【8】 Global, and Local Optimization Beamforming for Acoustic Broadband Sources
标题：声学宽带信源的全局和局部优化波束形成
链接：https://arxiv.org/abs/2211.04921
* 与cs.SD语音【2】为同一篇
机构：German Aerospace Center (DLR), G¨ottingen, Germany备注：Submitted to Journal of Sound and Vibration摘要：本文提出了一种扩展的宽带声源全局优化波束形成算法。假定诸如源位置、空间形状、多极旋转或流动性质的性质可以在频率上参数化，则可以同时对所有频率执行CSM拟合。数值分析表明，标准全局优化问题的非线性误差函数类似于点扩散函数，并包含局部极小值，但可以通过所提出的宽带优化来改善。不仅增加了宽带优化过程中方程与未知变量的比率，而且还平滑了成本函数。它还简化了识别源和根据结果重建其光谱的过程。结果表明，在合成单极子上，该方法优于标准全局优化方法和CLEAN-SC方法，对于真实数据，宽带全局优化方法、标准全局优化方法和CLEAN-SC方法的结果相似。然而，所提出的方法不需要识别和整合感兴趣区域。结果表明，通过合理选取初始值，全局优化问题可转化为局部优化问题，且具有相似的结果。进一步地，该方法能够识别具有不同极点幅值和未知极点旋转的多极子。摘要：This paper presents an extension to global optimization beamforming for acoustic broadband sources. Given, that properties such as the source location, spatial shape, multipole rotation, or flow properties can be parameterized over the frequency, a CSM-fitting can be performed for all frequencies at the same time. A numerical analysis shows that the non-linear error function for the standard global optimization problem is similar to a Point Spread Function and contains local minima, but can be improved with the proposed broadband optimization. Not only increases the broadband optimization process the ratio of equations to unknown variables, but it also smooths out the cost function. It also simplifies the process of identifying sources and reconstructing their spectra from the results. The paper shows that the method is superior on synthetic monopoles compared to standard global optimization and CLEAN-SC. For real-world data the results of broadband global optimization, standard global optimization, and CLEAN-SC are similar. However, the proposed method does not require the identification and integration of Regions Of Interest. It is shown, that by using reasonable initial values the global optimization problem reduces to a local optimization problem with similar results. Further, it is shown that the proposed method is able to identify multipoles with different pole amplitudes and unknown pole rotations.
【9】 Distribution-based Emotion Recognition in Conversation
标题：基于分布的会话情感识别
链接：https://arxiv.org/abs/2211.04834
* 与cs.SD语音【3】为同一篇
作者：Wen Wu,Chao Zhang,Philip C. Woodland机构：Department of Engineering, University of Cambridge, Trumpington St., Cambridge, UK.摘要：会话中的自动情感识别（ERC）是实现情感感知的会话人工智能的关键。本文提出了一种基于分布的框架，将ERC描述为序列到序列的情感分布估计问题。由于情感本身的模糊性和人类感知的主观性，导致了情感标签的不一致性，本文从情感分布不确定性估计的角度对情感标签进行了处理。通过引入贝叶斯训练损失，将每个情感状态置于特定话语的狄利克雷先验分布下，以改善不确定性估计。在IEMOCAP数据集上的实验结果表明，ERC的分类性能优于基于单话语的分类系统，所提出的基于分布的ERC方法不仅具有更好的分类精度，而且具有更好的不确定性估计能力.摘要：Automatic emotion recognition in conversation (ERC) is crucial for emotion-aware conversational artificial intelligence. This paper proposes a distribution-based framework that formulates ERC as a sequence-to-sequence problem for emotion distribution estimation. The inherent ambiguity of emotions and the subjectivity of human perception lead to disagreements in emotion labels, which is handled naturally in our framework from the perspective of uncertainty estimation in emotion distributions. A Bayesian training loss is introduced to improve the uncertainty estimation by conditioning each emotional state on an utterance-specific Dirichlet prior distribution. Experimental results on the IEMOCAP dataset show that ERC outperformed the single-utterance-based system, and the proposed distribution-based ERC methods have not only better classification accuracy, but also show improved uncertainty estimation.
【10】 Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation
标题：基于Transformer-to-CNN知识蒸馏的高效大规模音频标注
链接：https://arxiv.org/abs/2211.04772
* 与cs.SD语音【4】为同一篇
作者：Florian Schmid,Khaled Koutini,Gerhard Widmer机构：Institute of Computational Perception (CP-JKU),LIT Artificial Intelligence Lab, Johannes Kepler University Linz, Austria备注：Submitted to ICASSP 2023. Source Code available at: this https URL摘要：音频谱图Transformer模型统治着音频标记领域，超过了以前占主导地位的卷积神经网络（CNNs）。它们的优势在于能够纵向扩展和利用大规模数据集（如AudioSet）。然而，与CNN相比，Transformers在模型大小和计算要求方面要求更高。提出了一种基于离线知识提取的高效神经网络训练方法。所提出的训练方案和基于MobileNetV3的高效CNN设计使得模型在参数、计算效率和预测性能方面优于先前的解决方案。我们提供不同复杂度级别的模型，从低复杂度模型扩展到AudioSet上最新的.483 mAP性能。源代码位于：https://github.com/fschmid56/EfficientAT摘要：Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of .483 mAP on AudioSet. Source Code available at: https://github.com/fschmid56/EfficientAT
【11】 Improving Noisy Student Training on Non-target Domain Data for Automatic Speech Recognition
标题：自动语音识别中非目标域数据噪声训练的改进
链接：https://arxiv.org/abs/2211.04717
* 与cs.SD语音【5】为同一篇
作者：Yu Chen,Wen Ding,Junjie Lai机构：The University of Hong Kong, Hong Kong China, NVIDIA, Shanghai China摘要：噪声学生训练（NST）最近在自动语音识别（ASR）中表现出了极强的性能。本文提出了一种LM Filter数据选择策略，以提高NST对ASR任务中非目标域数据的选择性能。生成带有和不带有语言模型的假设，并利用它们之间的CER差异作为过滤阈值。结果显示，与无数据过滤基线相比，显著改善了10.4%。在AISHELL-1测试集上，我们可以得到3.31%的CER，这是我们在没有任何其他监督数据的情况下所得到的最好结果。我们还对有监督的1000小时AISHELL-2数据集进行了评估，并获得了4.72% CER的竞争结果。摘要：Noisy Student Training (NST) has recently demonstrated extremely strong performance in Automatic Speech Recognition (ASR). In this paper, we propose a data selection strategy named LM Filter to improve the performances of NST on non-target domain data in ASR tasks. Hypothesis with and without Language Model are generated and CER differences between them are utilized as a filter threshold. Results reveal that significant improvements of 10.4% compared with no data filtering baselines. We can achieve 3.31% CER in AISHELL-1 test set, which is best result from our knowledge without any other supervised data. We also perform evaluations on supervised 1000 hour AISHELL-2 dataset and competitive results of 4.72% CER can be achieved.
【12】 LiCo-Net: Linearized Convolution Network for Hardware-efficient Keyword Spotting
标题：LICO-Net：用于硬件高效关键词检测的线性化卷积网络
链接：https://arxiv.org/abs/2211.04635
作者：Haichuan Yang,Zhaojun Yang,Li Wan,Biqiao Zhang,Yangyang Shi,Yiteng Huang,Ivaylo Enchev,Limin Tang,Raziel Alvarez,Ming Sun,Xin Lei,Raghuraman Krishnamoorthi,Vikas Chandra摘要：本文提出了一种硬件高效的关键词识别结构--线性卷积网络（LiCo-Net）。它专门针对微控制器等低功耗处理器单元进行了优化。ML运算符在高能效硬件上表现出不同的效率特性。给定精确的理论计算成本，int 8运算符比浮点运算符计算效率更高，并且线性层通常比其他层效率更高。所提出的LiCo-Net是一个双阶段系统，在推理阶段使用高效的int 8线性算子，在训练阶段应用流式卷积以保持高模型容量。实验结果表明，LiCo-Net在硬件效率上优于单值分解滤波器（SVDF），检测性能与单值分解滤波器相当。与SVDF相比，LiCo-Net在HiFi 4 DSP上的周期减少了40%。摘要：This paper proposes a hardware-efficient architecture, Linearized Convolution Network (LiCo-Net) for keyword spotting. It is optimized specifically for low-power processor units like microcontrollers. ML operators exhibit heterogeneous efficiency profiles on power-efficient hardware. Given the exact theoretical computation cost, int8 operators are more computation-effective than float operators, and linear layers are often more efficient than other layers. The proposed LiCo-Net is a dual-phase system that uses the efficient int8 linear operators at the inference phase and applies streaming convolutions at the training phase to maintain a high model capacity. The experimental results show that LiCo-Net outperforms single-value decomposition filter (SVDF) on hardware efficiency with on-par detection performance. Compared to SVDF, LiCo-Net reduces cycles by 40% on HiFi4 DSP.
【13】 SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations
标题：SpeechMatrix：一个大规模挖掘的多语种语音翻译语料库
链接：https://arxiv.org/abs/2211.04508
* 与cs.SD语音【6】为同一篇
作者：Paul-Ambroise Duquenne,Hongyu Gong,Ning Dong,Jingfei Du,Ann Lee,Vedanuj Goswani,Changhan Wang,Juan Pino,Benoît Sagot,Holger Schwenk机构：Meta AI Research, Inria摘要：我们介绍了SpeechMatrix，这是一个从欧洲议会的真实演讲录音中挖掘出来的大规模多语言语音到语音翻译语料库。它包含136种语言对的语音对齐，总共有41.8万个小时的语音。为了评估这种并行语音的质量，我们仅在挖掘的数据上训练双语语音到语音翻译模型，并在EuroParl-ST，VoxPopuli和FLEURS测试集上建立广泛的基线结果。由于SpeechMatrix的多语言性，我们还探讨了多语言语音到语音的翻译，这是一个很少有其他作品涉及的主题。我们还证明了模型预训练和使用混合专家的稀疏缩放带来了巨大的翻译性能增益。挖掘的数据和模型是免费提供的。摘要：We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.
【14】 Towards Improved Room Impulse Response Estimation for Speech Recognition
标题：语音识别中改进的房间脉冲响应估计方法
链接：https://arxiv.org/abs/2211.04473
* 与cs.SD语音【7】为同一篇
作者：Anton Ratnarajah,Ishwarya Ananthabhotla,Vamsi Krishna Ithapu,Pablo Hoffmann,Dinesh Manocha,Paul Calamia机构：University of Maryland, College Park, USA, Reality Labs Research at Meta, Redmond, WA USA摘要：我们提出在下游应用场景远场自动语音识别（ASR）的背景下表征和改善盲室脉冲响应（RIR）估计系统的性能。我们首先得出改进的RIR估计和改进的ASR性能之间的联系，作为评估神经RIR估计器的一种手段。然后，我们提出了一种基于GAN的结构，该结构对混响语音的RIR特征进行编码，并根据编码后的特征构建RIR，并使用一种新的能量衰减减轻损失来优化捕获输入混响语音的基于能量的属性。我们证明了我们的模型在声学基准测试（能量衰减缓解72%，早期反射能量度量22%）以及ASR评估任务（单词错误率6.9%）方面优于最先进的基线。摘要：We propose to characterize and improve the performance of blind room impulse response (RIR) estimation systems in the context of a downstream application scenario, far-field automatic speech recognition (ASR). We first draw the connection between improved RIR estimation and improved ASR performance, as a means of evaluating neural RIR estimators. We then propose a GAN-based architecture that encodes RIR features from reverberant speech and constructs an RIR from the encoded features, and uses a novel energy decay relief loss to optimize for capturing energy-based properties of the input reverberant speech. We show that our model outperforms the state-of-the-art baselines on acoustic benchmarks (by 72% on the energy decay relief and 22% on an early-reflection energy metric), as well as in an ASR evaluation task (by 6.9% in word error rate).
机器翻译，仅供参考
每日语音速递