今日论文合集：cs.SD语音7篇，eess.AS音频处理9篇。
cs.SD语音

【1】 Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss标题：基于全和损失的带噪声训练标签RNN-T模型的稳健知识提取链接：http://arxiv.org/pdf/2303.05958v1

作者：Mohammad Zeineldeen,Kartik Audhkhasi,Murali Karthick Baskar,Bhuvana Ramabhadran机构：Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Aachen, Germany, Google LLC, New York备注：Accepted at ICASSP 2023摘要：研究了知识提取（KD）问题，并解决了知识提取对递归神经网络传感器（RNN-T）模型的约束问题。在硬蒸馏中，教师模型转录大量未标记语音以训练学生模型。软蒸馏是另一种流行的KD方法，它蒸馏教师模型的输出logit。由于RNN-T比对的性质，在具有不同后验分布的RNN-T架构之间应用软蒸馏是具有挑战性的。此外，高错别字率的不良教师降低了KD的效能感。我们调查如何有效地提取知识，从可变质量的ASR教师，这还没有研究之前，以我们的知识。我们证明了序列水平KD，全和蒸馏，优于其他蒸馏方法的RNN-T模型，特别是对坏教师。我们还提出了一种全和蒸馏的变体，它提取了教师的序列判别知识，从而进一步提高了WER。我们在公共数据集SpeechStew和LibriSpeech以及内部生产数据上进行了实验。摘要：This work studies knowledge distillation (KD) and addresses its constraints for recurrent neural network transducer (RNN-T) models. In hard distillation, a teacher model transcribes large amounts of unlabelled speech to train a student model. Soft distillation is another popular KD method that distills the output logits of the teacher model. Due to the nature of RNN-T alignments, applying soft distillation between RNN-T architectures having different posterior distributions is challenging. In addition, bad teachers having high word-error-rate (WER) reduce the efficacy of KD. We investigate how to effectively distill knowledge from variable quality ASR teachers, which has not been studied before to the best of our knowledge. We show that a sequence-level KD, full-sum distillation, outperforms other distillation methods for RNN-T models, especially for bad teachers. We also propose a variant of full-sum distillation that distills the sequence discriminative knowledge of the teacher leading to further improvement in WER. We conduct experiments on public datasets namely SpeechStew and LibriSpeech, and on in-house production data.

【2】 MIXPGD: Hybrid Adversarial Training for Speech Recognition Systems标题：MIXPGD：语音识别系统的混合对抗性训练链接：http://arxiv.org/pdf/2303.05758v1

作者：Aminul Huq,Weiyi Zhang,Xiaolin Hu机构：Department of Computer Science and Technology, State Key Laboratory of Intelligent Technology and Systems, Tsinghua Laboratory of Brain and Intelligence (THBI), IDGMcGovern Institute of Brain Research, Tsinghua University, Beijing ,China摘要：基于深度神经网络的自动语音识别系统抗干扰能力较弱。为了提高ASR系统模型的鲁棒性，提出了混合PGD对抗训练方法。在标准的对抗训练中，对抗样本通过利用监督或非监督方法来生成。我们在我们的方法中融合了监督和非监督方法的能力来生成新的对抗样本，这有助于提高模型的鲁棒性。对各种最先进的防御方法和对抗性攻击进行了广泛的实验和比较，结果表明，在白盒对抗性攻击设置下，mixPGD比以前的最佳性能模型获得了4.1%的WER。我们针对白盒攻击和基于传输的黑盒攻击设置测试了我们提出的防御方法，以确保我们的防御策略对各种类型的攻击都是鲁棒的。针对几种对抗性攻击的实验结果验证了该方法的有效性。摘要：Automatic speech recognition (ASR) systems based on deep neural networks are weak against adversarial perturbations. We propose mixPGD adversarial training method to improve the robustness of the model for ASR systems. In standard adversarial training, adversarial samples are generated by leveraging supervised or unsupervised methods. We merge the capabilities of both supervised and unsupervised approaches in our method to generate new adversarial samples which aid in improving model robustness. Extensive experiments and comparison across various state-of-the-art defense methods and adversarial attacks have been performed to show that mixPGD gains 4.1% WER of better performance than previous best performing models under white-box adversarial attack setting. We tested our proposed defense method against both white-box and transfer based black-box attack settings to ensure that our defense strategy is robust against various types of attacks. Empirical results on several adversarial attacks validate the effectiveness of our proposed approach.

【3】 Improving Text-Audio Retrieval by Text-aware Attention Pooling and Prior Matrix Revised Loss标题：利用文本感知注意力集中和先验矩阵修正损失改进文本-音频检索链接：http://arxiv.org/pdf/2303.05681v1

作者：Yifei Xin,Dongchao Yang,Yuexian Zou机构：School of ECE, Peking University, Shenzhen, China备注：4 pages. Accepted by ICASSP 2023摘要：在文本-音频检索（TAR）任务中，由于文本和音频内容的异质性，文本中包含的语义信息只与音频中的某些帧相似。然而，现有的作品聚合整个音频而不考虑文本，例如帧上的均值池，这很可能编码给定文本中没有描述的误导性音频信息。在这篇文章中，我们提出了一个用于TAR的文本感知注意力池（TAP）模块，它本质上是一个缩放的点积注意力，用于文本关注其语义上最相似的框架。此外，以往的方法只对每一个单侧检索进行softmax，忽略了潜在的交叉检索信息。通过挖掘每个文本-音频对的固有先验知识，引入先验矩阵修正（PMR）损失来过滤文本-音频相似度高（或低）但音频-文本相似度低（或高）的硬情况，从而获得双重最优匹配。实验结果表明，TAP的性能明显优于各种文本无关池函数.此外，我们的PMR损失也显示了在多个数据集上的稳定性能增益。摘要：In text-audio retrieval (TAR) tasks, due to the heterogeneity of contents between text and audio, the semantic information contained in the text is only similar to certain frames within the audio. Yet, existing works aggregate the entire audio without considering the text, such as mean-pooling over the frames, which is likely to encode misleading audio information not described in the given text. In this paper, we present a text-aware attention pooling (TAP) module for TAR, which is essentially a scaled dot product attention for a text to attend to its most semantically similar frames. Furthermore, previous methods only conduct the softmax for every single-side retrieval, ignoring the potential cross-retrieval information. By exploring the intrinsic prior of each text-audio pair, we introduce a prior matrix revised (PMR) loss to filter the hard case with high (or low) text-to-audio but low (or high) audio-to-text similarity scores, thus achieving the dual optimal match. Experiments show that our TAP significantly outperforms various text-agnostic pooling functions. Moreover, our PMR loss also shows stable performance gains on multiple datasets.

【4】 Improving Weakly Supervised Sound Event Detection with Causal Intervention标题：利用因果干预改进弱监督声音事件检测链接：http://arxiv.org/pdf/2303.05678v1

作者：Yifei Xin,Dongchao Yang,Fan Cui,Yujun Wang,Yuexian Zou机构：School of ECE, Peking University, Shenzhen, China, Xiaomi Corporation, Beijing, China备注：Accepted by ICASSP2023摘要：现有的弱监督声音事件检测（WSSED）工作尚未同时探索两种类型的同现，即，一些声音事件经常同时发生，并且它们的发生通常伴随着特定的背景声音，因此它们将不可避免地纠缠在一起，从而导致错误分类和仅利用剪辑级监督的有偏差的定位结果。为了解决这个问题，我们首先建立了一个结构因果模型（SCM），揭示了上下文是共现混杂因素的主要原因，这些混杂因素误导模型学习帧和剪辑级别标签之间的虚假相关性。在因果分析的基础上，提出一种WSSED因果干预方法，通过迭代累积每类可能的上下文，然后将上下文重新投影到帧级特征上，使事件边界更加清晰，从而消除共现混杂因素的负面影响。实验结果表明，该方法有效地提高了多数据集上的性能，并能推广到各种基线模型。摘要：Existing weakly supervised sound event detection (WSSED) work has not explored both types of co-occurrences simultaneously, i.e., some sound events often co-occur, and their occurrences are usually accompanied by specific background sounds, so they would be inevitably entangled, causing misclassification and biased localization results with only clip-level supervision. To tackle this issue, we first establish a structural causal model (SCM) to reveal that the context is the main cause of co-occurrence confounders that mislead the model to learn spurious correlations between frames and clip-level labels. Based on the causal analysis, we propose a causal intervention (CI) method for WSSED to remove the negative impact of co-occurrence confounders by iteratively accumulating every possible context of each class and then re-projecting the contexts to the frame-level features for making the event boundary clearer. Experiments show that our method effectively improves the performance on multiple datasets and can generalize to various baseline models.

【5】 wav2vec and its current potential to Automatic Speech Recognition in German for the usage in Digital History: A comparative assessment of available ASR-technologies for the use in cultural heritage contexts标题：Wave2vec及其目前在数字历史中使用的德语自动语音识别的潜力：可用于文化遗产上下文的ASR技术的比较评估链接：http://arxiv.org/pdf/2303.06026v1

作者：Michael Fleck,Wolfgang Göderle备注：11 pages, 2 tables摘要：在本案例研究中，我们为德语自动语音识别（ASR）培训并发布了一个最先进的开源模型，以评估该技术在数字人文学科和文化遗产索引化这一更大背景下的使用潜力。除了本文，我们还发布了基于wav2vec2的语音到文本模型，同时我们在我们收集的历史录音语料库上评估了它的性能，并与基于云的商业服务和专有服务进行了比较。虽然我们的模型取得了中等的结果，但我们看到专有云服务的表现要好得多。正如我们的结果所示，目前可以达到90%以上的识别率，然而，一旦录音的音频质量有限或使用非日常语言或陈腐语言，这些数字就会迅速下降。一个大问题是德语中各种不同的方言和口音。尽管如此，本文强调，目前可用的识别质量足够高，足以解决数字人文学科中的各种用例。我们认为ASR将成为记录和分析视听资源的关键技术，并确定了DH社区和文化遗产利益相关者在不久的将来必须解决的一系列重要问题。摘要：In this case study we trained and published a state-of-the-art open-source model for Automatic Speech Recognition (ASR) for German to evaluate the current potential of this technology for the use in the larger context of Digital Humanities and cultural heritage indexation. Along with this paper we publish our wav2vec2 based speech to text model while we evaluate its performance on a corpus of historical recordings we assembled compared against commercial cloud-based and proprietary services. While our model achieves moderate results, we see that proprietary cloud services fare significantly better. As our results show, recognition rates over 90 percent can currently be achieved, however, these numbers drop quickly once the recordings feature limited audio quality or use of non-every day or outworn language. A big issue is the high variety of different dialects and accents in the German language. Nevertheless, this paper highlights that the currently available quality of recognition is high enough to address various use cases in the Digital Humanities. We argue that ASR will become a key technology for the documentation and analysis of audio-visual sources and identify an array of important questions that the DH community and cultural heritage stakeholders will have to address in the near future.

【6】Distribution Preserving Source Separation With Time Frequency Predictive Models标题：基于时频预测模型的分布保持信源分离链接：http://arxiv.org/pdf/2303.05896v1

作者：Pedro J. Villasana T.,Janusz Klejsa,Lars Villemoes,Per Hedelin机构：Dolby Sweden AB, Stockholm, Sweden备注：5 pages, 4 figures, pre-review version submitted to EUSIPCO 2023摘要：我们提供了一个分布保持源分离方法的例子，其目的在于解决现有技术方法的感知缺陷。我们的方法使用信号源的无条件生成模型。通过从以混合实现为条件的分布进行混合一致采样来实现重构。分离的信号遵循它们各自的源分布，这在收听测试中评估分离结果时提供了优点。摘要：We provide an example of a distribution preserving source separation method, which aims at addressing perceptual shortcomings of state-of-the-art methods. Our approach uses unconditioned generative models of signal sources. Reconstruction is achieved by means of mix-consistent sampling from a distribution conditioned on a realization of a mix. The separated signals follow their respective source distributions, which provides an advantage when separation results are evaluated in a listening test.

【7】 Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings标题：临床BERTScore：临床环境中自动语音识别性能的改进测量链接：http://arxiv.org/pdf/2303.05737v1

作者：Joel Shor,Ruyue Agnes Bi,Subhashini Venugopalan,Steven Ibara,Roman Goldenberg,Ehud Rivlen机构：Verily Life Sciences, USA,Massachusetts Institute of Technology, USA, Google, USA,Verily Life Sciences, Israel摘要：自动语音识别（ASR）在医疗环境中具有节省时间、降低成本、提高报告准确性和减少医生疲劳的潜力。然而，医疗保健行业采用这种技术的速度较慢，部分原因是避免医学相关转录错误的重要性。在这项工作中，我们提出了临床BERTScore（CBERTScore），这是一个ASR指标，比其他指标更能惩罚临床相关错误。我们证明，与其他指标（WER、BLUE、METEOR等）相比，该指标与临床医生对医疗句子的偏好更接近，有时差距很大。我们收集了149个真实医学句子的13个临床医生偏好的基准，称为临床医生转录偏好基准（CTP），证明CBERTScore更接近临床医生的偏好，并发布该基准供社区进一步开发临床感知ASR指标。摘要：Automatic Speech Recognition (ASR) in medical contexts has the potential to save time, cut costs, increase report accuracy, and reduce physician burnout. However, the healthcare industry has been slower to adopt this technology, in part due to the importance of avoiding medically-relevant transcription mistakes. In this work, we present the Clinical BERTScore (CBERTScore), an ASR metric that penalizes clinically-relevant mistakes more than others. We demonstrate that this metric more closely aligns with clinician preferences on medical sentences as compared to other metrics (WER, BLUE, METEOR, etc), sometimes by wide margins. We collect a benchmark of 13 clinician preferences on 149 realistic medical sentences called the Clinician Transcript Preference benchmark (CTP), demonstrate that CBERTScore more closely matches what clinicians prefer, and release the benchmark for the community to further develop clinically-aware ASR metrics.
eess.AS音频处理

【1】 An End-to-End Neural Network for Image-to-Audio Transformation标题：一种用于音像转换的端到端神经网络链接：http://arxiv.org/pdf/2303.06078v1

作者：Liu Chen,Michael Deisher,Munir Georges机构：Oregon Health and Science University, Oregon USA, Intel Corporation, Hillsboro, Oregon USA, Intel Labs, Munich, Germany, Technische Hochschule Ingolstadt备注：5 pages, 3 figures, 2023 IEEE Conference on Acoustics, Speech, and Signal Processing摘要：本文描述了一种端到端（E2E）神经网络架构，用于在低资源个人计算设备上对显示内容的小部分进行音频渲染。其目的是在硬件层面解决视觉受损或视觉分散用户的可访问性问题。回顾了神经图像到文本（ITT）和文本到语音（TTS）方法，并引入了一种新技术来以高效且可反向传播的方式有效地集成它们，从而产生高效且可训练的非自回归E2E图像到语音（ITS）神经网络。实验结果表明，与非E2E方法相比，所提出的E2E系统速度提高了29%，使用的参数减少了19%，通话准确率降低了2%。提出了解决精度问题的未来方向。摘要：This paper describes an end-to-end (E2E) neural architecture for the audio rendering of small portions of display content on low resource personal computing devices. It is intended to address the problem of accessibility for vision-impaired or vision-distracted users at the hardware level. Neural image-to-text (ITT) and text-to-speech (TTS) approaches are reviewed and a new technique is introduced to efficiently integrate them in a way that is both efficient and back-propagate-able, leading to a non-autoregressive E2E image-to-speech (ITS) neural network that is efficient and trainable. Experimental results are presented showing that, compared with the non-E2E approach, the proposed E2E system is 29% faster and uses 19% fewer parameters with a 2% reduction in phone accuracy. A future direction to address accuracy is presented.

【2】 wav2vec and its current potential to Automatic Speech Recognition in German for the usage in Digital History: A comparative assessment of available ASR-technologies for the use in cultural heritage contexts标题：Wave2vec及其目前在数字历史中使用的德语自动语音识别的潜力：可用于文化遗产上下文的ASR技术的比较评估链接：http://arxiv.org/pdf/2303.06026v1

*与cs.SD语音【5】为同一篇

作者：Michael Fleck,Wolfgang Göderle备注：11 pages, 2 tables摘要：在本案例研究中，我们为德语自动语音识别（ASR）培训并发布了一个最先进的开源模型，以评估该技术在数字人文学科和文化遗产索引化这一更大背景下的使用潜力。除了本文，我们还发布了基于wav2vec2的语音到文本模型，同时我们在我们收集的历史录音语料库上评估了它的性能，并与基于云的商业服务和专有服务进行了比较。虽然我们的模型取得了中等的结果，但我们看到专有云服务的表现要好得多。正如我们的结果所示，目前可以达到90%以上的识别率，然而，一旦录音的音频质量有限或使用非日常语言或陈腐语言，这些数字就会迅速下降。一个大问题是德语中各种不同的方言和口音。尽管如此，本文强调，目前可用的识别质量足够高，足以解决数字人文学科中的各种用例。我们认为ASR将成为记录和分析视听资源的关键技术，并确定了DH社区和文化遗产利益相关者在不久的将来必须解决的一系列重要问题。摘要：In this case study we trained and published a state-of-the-art open-source model for Automatic Speech Recognition (ASR) for German to evaluate the current potential of this technology for the use in the larger context of Digital Humanities and cultural heritage indexation. Along with this paper we publish our wav2vec2 based speech to text model while we evaluate its performance on a corpus of historical recordings we assembled compared against commercial cloud-based and proprietary services. While our model achieves moderate results, we see that proprietary cloud services fare significantly better. As our results show, recognition rates over 90 percent can currently be achieved, however, these numbers drop quickly once the recordings feature limited audio quality or use of non-every day or outworn language. A big issue is the high variety of different dialects and accents in the German language. Nevertheless, this paper highlights that the currently available quality of recognition is high enough to address various use cases in the Digital Humanities. We argue that ASR will become a key technology for the documentation and analysis of audio-visual sources and identify an array of important questions that the DH community and cultural heritage stakeholders will have to address in the near future.

【3】 Distribution Preserving Source Separation With Time Frequency Predictive Models标题：基于时频预测模型的分布保持信源分离链接：http://arxiv.org/pdf/2303.05896v1

*与cs.SD语音【6】为同一篇

作者：Pedro J. Villasana T.,Janusz Klejsa,Lars Villemoes,Per Hedelin机构：Dolby Sweden AB, Stockholm, Sweden备注：5 pages, 4 figures, pre-review version submitted to EUSIPCO 2023摘要：我们提供了一个分布保持源分离方法的例子，其目的在于解决现有技术方法的感知缺陷。我们的方法使用信号源的无条件生成模型。通过从以混合实现为条件的分布进行混合一致采样来实现重构。分离的信号遵循它们各自的源分布，这在收听测试中评估分离结果时提供了优点。摘要：We provide an example of a distribution preserving source separation method, which aims at addressing perceptual shortcomings of state-of-the-art methods. Our approach uses unconditioned generative models of signal sources. Reconstruction is achieved by means of mix-consistent sampling from a distribution conditioned on a realization of a mix. The separated signals follow their respective source distributions, which provides an advantage when separation results are evaluated in a listening test.

【4】 Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings标题：临床BERTScore：临床环境中自动语音识别性能的改进测量链接：http://arxiv.org/pdf/2303.05737v1

*与cs.SD语音【7】为同一篇

作者：Joel Shor,Ruyue Agnes Bi,Subhashini Venugopalan,Steven Ibara,Roman Goldenberg,Ehud Rivlen机构：Verily Life Sciences, USA,Massachusetts Institute of Technology, USA, Google, USA,Verily Life Sciences, Israel摘要：自动语音识别（ASR）在医疗环境中具有节省时间、降低成本、提高报告准确性和减少医生疲劳的潜力。然而，医疗保健行业采用这种技术的速度较慢，部分原因是避免医学相关转录错误的重要性。在这项工作中，我们提出了临床BERTScore（CBERTScore），这是一个ASR指标，比其他指标更能惩罚临床相关错误。我们证明，与其他指标（WER、BLUE、METEOR等）相比，该指标与临床医生对医疗句子的偏好更接近，有时差距很大。我们收集了149个真实医学句子的13个临床医生偏好的基准，称为临床医生转录偏好基准（CTP），证明CBERTScore更接近临床医生的偏好，并发布该基准供社区进一步开发临床感知ASR指标。摘要：Automatic Speech Recognition (ASR) in medical contexts has the potential to save time, cut costs, increase report accuracy, and reduce physician burnout. However, the healthcare industry has been slower to adopt this technology, in part due to the importance of avoiding medically-relevant transcription mistakes. In this work, we present the Clinical BERTScore (CBERTScore), an ASR metric that penalizes clinically-relevant mistakes more than others. We demonstrate that this metric more closely aligns with clinician preferences on medical sentences as compared to other metrics (WER, BLUE, METEOR, etc), sometimes by wide margins. We collect a benchmark of 13 clinician preferences on 149 realistic medical sentences called the Clinician Transcript Preference benchmark (CTP), demonstrate that CBERTScore more closely matches what clinicians prefer, and release the benchmark for the community to further develop clinically-aware ASR metrics.

【5】 UNFUSED: UNsupervised Finetuning Using SElf supervised Distillation标题：未融合：使用自监督蒸馏的无监督微调链接：http://arxiv.org/pdf/2303.05668v1

作者：Ashish Seth,Sreyan Ghosh,S. Umesh,Dinesh Manocha机构：University of Maryland, College Park, USA, Speech Lab, Department of Electrical Engineering, IIT Madras, Chennai, India备注：Under review at ICASSP 2023 SASB Workshop摘要：None摘要：In this paper, we introduce UnFuSeD, a novel approach to leverage self-supervised learning and reduce the need for large amounts of labeled data for audio classification. Unlike prior works, which directly fine-tune a self-supervised pre-trained encoder on a target dataset, we use the encoder to generate pseudo-labels for unsupervised fine-tuning before the actual fine-tuning step. We first train an encoder using a novel self-supervised learning algorithm (SSL) on an unlabeled audio dataset. Then, we use that encoder to generate pseudo-labels on our target task dataset via clustering the extracted representations. These pseudo-labels are then used to guide self-distillation on a randomly initialized model, which we call unsupervised fine-tuning. Finally, the resultant encoder is then fine-tuned on our target task dataset. Through UnFuSeD, we propose the first system that moves away from generic SSL paradigms in literature, which pre-train and fine-tune the same encoder, and present a novel self-distillation-based system to leverage SSL pre-training for low-resource audio classification. In practice, UnFuSeD achieves state-of-the-art results on the LAPE Benchmark, significantly outperforming all our baselines. Additionally, UnFuSeD allows us to achieve this at a 40% reduction in the number of parameters over the previous state-of-the-art system. We make all our codes publicly available.

【6】 Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss标题：基于全和损失的带噪声训练标签RNN-T模型的稳健知识提取链接：http://arxiv.org/pdf/2303.05958v1

*与cs.SD语音【1】为同一篇

作者：Mohammad Zeineldeen,Kartik Audhkhasi,Murali Karthick Baskar,Bhuvana Ramabhadran机构：Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Aachen, Germany, Google LLC, New York备注：Accepted at ICASSP 2023摘要：研究了知识提取（KD）问题，并解决了知识提取对递归神经网络传感器（RNN-T）模型的约束问题。在硬蒸馏中，教师模型转录大量未标记语音以训练学生模型。软蒸馏是另一种流行的KD方法，它蒸馏教师模型的输出logit。由于RNN-T比对的性质，在具有不同后验分布的RNN-T架构之间应用软蒸馏是具有挑战性的。此外，高错别字率的不良教师降低了KD的效能感。我们调查如何有效地提取知识，从可变质量的ASR教师，这还没有研究之前，以我们的知识。我们证明了序列水平KD，全和蒸馏，优于其他蒸馏方法的RNN-T模型，特别是对坏教师。我们还提出了一种全和蒸馏的变体，它提取了教师的序列判别知识，从而进一步提高了WER。我们在公共数据集SpeechStew和LibriSpeech以及内部生产数据上进行了实验。摘要：This work studies knowledge distillation (KD) and addresses its constraints for recurrent neural network transducer (RNN-T) models. In hard distillation, a teacher model transcribes large amounts of unlabelled speech to train a student model. Soft distillation is another popular KD method that distills the output logits of the teacher model. Due to the nature of RNN-T alignments, applying soft distillation between RNN-T architectures having different posterior distributions is challenging. In addition, bad teachers having high word-error-rate (WER) reduce the efficacy of KD. We investigate how to effectively distill knowledge from variable quality ASR teachers, which has not been studied before to the best of our knowledge. We show that a sequence-level KD, full-sum distillation, outperforms other distillation methods for RNN-T models, especially for bad teachers. We also propose a variant of full-sum distillation that distills the sequence discriminative knowledge of the teacher leading to further improvement in WER. We conduct experiments on public datasets namely SpeechStew and LibriSpeech, and on in-house production data.

【7】 MIXPGD: Hybrid Adversarial Training for Speech Recognition Systems标题：MIXPGD：语音识别系统的混合对抗性训练链接：http://arxiv.org/pdf/2303.05758v1

*与cs.SD语音【2】为同一篇

作者：Aminul Huq,Weiyi Zhang,Xiaolin Hu机构：Department of Computer Science and Technology, State Key Laboratory of Intelligent Technology and Systems, Tsinghua Laboratory of Brain and Intelligence (THBI), IDGMcGovern Institute of Brain Research, Tsinghua University, Beijing ,China摘要：基于深度神经网络的自动语音识别系统抗干扰能力较弱。为了提高ASR系统模型的鲁棒性，提出了混合PGD对抗训练方法。在标准的对抗训练中，对抗样本通过利用监督或非监督方法来生成。我们在我们的方法中融合了监督和非监督方法的能力来生成新的对抗样本，这有助于提高模型的鲁棒性。对各种最先进的防御方法和对抗性攻击进行了广泛的实验和比较，结果表明，在白盒对抗性攻击设置下，mixPGD比以前的最佳性能模型获得了4.1%的WER。我们针对白盒攻击和基于传输的黑盒攻击设置测试了我们提出的防御方法，以确保我们的防御策略对各种类型的攻击都是鲁棒的。针对几种对抗性攻击的实验结果验证了该方法的有效性。摘要：Automatic speech recognition (ASR) systems based on deep neural networks are weak against adversarial perturbations. We propose mixPGD adversarial training method to improve the robustness of the model for ASR systems. In standard adversarial training, adversarial samples are generated by leveraging supervised or unsupervised methods. We merge the capabilities of both supervised and unsupervised approaches in our method to generate new adversarial samples which aid in improving model robustness. Extensive experiments and comparison across various state-of-the-art defense methods and adversarial attacks have been performed to show that mixPGD gains 4.1% WER of better performance than previous best performing models under white-box adversarial attack setting. We tested our proposed defense method against both white-box and transfer based black-box attack settings to ensure that our defense strategy is robust against various types of attacks. Empirical results on several adversarial attacks validate the effectiveness of our proposed approach.

【8】 Improving Text-Audio Retrieval by Text-aware Attention Pooling and Prior Matrix Revised Loss标题：利用文本感知注意力集中和先验矩阵修正损失改进文本-音频检索链接：http://arxiv.org/pdf/2303.05681v1

*与cs.SD语音【3】为同一篇

作者：Yifei Xin,Dongchao Yang,Yuexian Zou机构：School of ECE, Peking University, Shenzhen, China备注：4 pages. Accepted by ICASSP 2023摘要：在文本-音频检索（TAR）任务中，由于文本和音频内容的异质性，文本中包含的语义信息只与音频中的某些帧相似。然而，现有的作品聚合整个音频而不考虑文本，例如帧上的均值池，这很可能编码给定文本中没有描述的误导性音频信息。在这篇文章中，我们提出了一个用于TAR的文本感知注意力池（TAP）模块，它本质上是一个缩放的点积注意力，用于文本关注其语义上最相似的框架。此外，以往的方法只对每一个单侧检索进行softmax，忽略了潜在的交叉检索信息。通过挖掘每个文本-音频对的固有先验知识，引入先验矩阵修正（PMR）损失来过滤文本-音频相似度高（或低）但音频-文本相似度低（或高）的硬情况，从而获得双重最优匹配。实验结果表明，TAP的性能明显优于各种文本无关池函数.此外，我们的PMR损失也显示了在多个数据集上的稳定性能增益。摘要：In text-audio retrieval (TAR) tasks, due to the heterogeneity of contents between text and audio, the semantic information contained in the text is only similar to certain frames within the audio. Yet, existing works aggregate the entire audio without considering the text, such as mean-pooling over the frames, which is likely to encode misleading audio information not described in the given text. In this paper, we present a text-aware attention pooling (TAP) module for TAR, which is essentially a scaled dot product attention for a text to attend to its most semantically similar frames. Furthermore, previous methods only conduct the softmax for every single-side retrieval, ignoring the potential cross-retrieval information. By exploring the intrinsic prior of each text-audio pair, we introduce a prior matrix revised (PMR) loss to filter the hard case with high (or low) text-to-audio but low (or high) audio-to-text similarity scores, thus achieving the dual optimal match. Experiments show that our TAP significantly outperforms various text-agnostic pooling functions. Moreover, our PMR loss also shows stable performance gains on multiple datasets.

【9】 Improving Weakly Supervised Sound Event Detection with Causal Intervention标题：利用因果干预改进弱监督声音事件检测链接：http://arxiv.org/pdf/2303.05678v1

*与cs.SD语音【4】为同一篇

作者：Yifei Xin,Dongchao Yang,Fan Cui,Yujun Wang,Yuexian Zou机构：School of ECE, Peking University, Shenzhen, China, Xiaomi Corporation, Beijing, China备注：Accepted by ICASSP2023摘要：现有的弱监督声音事件检测（WSSED）工作尚未同时探索两种类型的同现，即，一些声音事件经常同时发生，并且它们的发生通常伴随着特定的背景声音，因此它们将不可避免地纠缠在一起，从而导致错误分类和仅利用剪辑级监督的有偏差的定位结果。为了解决这个问题，我们首先建立了一个结构因果模型（SCM），揭示了上下文是共现混杂因素的主要原因，这些混杂因素误导模型学习帧和剪辑级别标签之间的虚假相关性。在因果分析的基础上，提出一种WSSED因果干预方法，通过迭代累积每类可能的上下文，然后将上下文重新投影到帧级特征上，使事件边界更加清晰，从而消除共现混杂因素的负面影响。实验结果表明，该方法有效地提高了多数据集上的性能，并能推广到各种基线模型。摘要：Existing weakly supervised sound event detection (WSSED) work has not explored both types of co-occurrences simultaneously, i.e., some sound events often co-occur, and their occurrences are usually accompanied by specific background sounds, so they would be inevitably entangled, causing misclassification and biased localization results with only clip-level supervision. To tackle this issue, we first establish a structural causal model (SCM) to reveal that the context is the main cause of co-occurrence confounders that mislead the model to learn spurious correlations between frames and clip-level labels. Based on the causal analysis, we propose a causal intervention (CI) method for WSSED to remove the negative impact of co-occurrence confounders by iteratively accumulating every possible context of each class and then re-projecting the contexts to the frame-level features for making the event boundary clearer. Experiments show that our method effectively improves the performance on multiple datasets and can generalize to various baseline models.

机器翻译，仅供参考