今日论文合集：cs.SD语音12篇，eess.AS音频处理12篇。【1】 Improving Speech Enhancement via Event-based Query
标题：通过基于事件的查询改进语音增强
链接：https://arxiv.org/abs/2302.11558
作者：Yifei Xin,Xiulian Peng,Yan Lu机构：Peking University, Beijing, China, Microsoft Research Asia, Beijing, China摘要：现有的基于深度学习的语音增强（SE）方法要么使用盲端到端训练，要么显式地将说话者嵌入或语音信息结合到SE网络中以增强语音质量。本文将语音和噪声看作是不同类型的声音事件，提出了一种基于事件的SE查询方法。具体地，首先利用声音事件检测（SED）任务来预训练能够区分语音与噪声的代表性语音嵌入。嵌入然后被聚集成固定的黄金语音查询，以帮助SE网络增强来自噪声音频的语音。黄金语音查询可以离线获得并且可推广到不同的SE数据集和网络。因此，几乎不引入额外的复杂性，并且不需要为每个发言者登记。实验结果表明，与基线相比，该方法具有显著的性能提升，且黄金查询能很好地推广到不同的数据集。摘要：Existing deep learning based speech enhancement (SE) methods either use blind end-to-end training or explicitly incorporate speaker embedding or phonetic information into the SE network to enhance speech quality. In this paper, we perceive speech and noises as different types of sound events and propose an event-based query method for SE. Specifically, representative speech embeddings that can discriminate speech with noises are first pre-trained with the sound event detection (SED) task. The embeddings are then clustered into fixed golden speech queries to assist the SE network to enhance the speech from noisy audio. The golden speech queries can be obtained offline and generalizable to different SE datasets and networks. Therefore, little extra complexity is introduced and no enrollment is needed for each speaker. Experimental results show that the proposed method yields significant gains compared with baselines and the golden queries are well generalized to different datasets.【2】 Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning
标题：基于数据增强、类均衡丢失和多语境深度学习的口吃检测
链接：https://arxiv.org/abs/2302.11343
作者：Shakeel A. Sheikh,Md Sahidullah,Fabrice Hirsch,Slim Ouni备注：Accepted in IEEE Journal of Biomedical Health Informatics 2023摘要：口吃是一种神经发育性言语障碍，其特征是不受控制的言语（感叹词）和核心行为（阻塞、重复和延长），是由言语感觉运动障碍引起的。由于其复杂的性质，口吃检测（SD）是一项困难的任务。如果在早期阶段发现，它可以帮助言语治疗师观察和纠正口吃者（PWS）的言语模式。PWS的口吃语音通常是有限的，并且高度不平衡。为此，我们通过多分支（MB）方案并通过加权总体损失函数中的类别贡献来解决SD域中的类别不平衡问题，从而使SEP-28 k数据集上的口吃类别比基线（StutterNet）有了巨大改善。为了解决数据不足的问题，我们研究了基于多分支训练方案的数据扩充的有效性。在宏观F1评分（F1）方面，增强训练优于MB StutterNet（clean），相对优势为4.18%。此外，我们还提出了一种多上下文（MC）StutterNet，它利用口吃语音的不同上下文，使F1的总成绩比基于单上下文的MB StutterNet提高了4.48%。最后，我们已经表明，在跨语料库场景中应用数据扩充可以提高总体SD性能，在F1中相对于干净训练的相对幅度为13.23%。摘要：Stuttering is a neuro-developmental speech impairment characterized by uncontrolled utterances (interjections) and core behaviors (blocks, repetitions, and prolongations), and is caused by the failure of speech sensorimotors. Due to its complex nature, stuttering detection (SD) is a difficult task. If detected at an early stage, it could facilitate speech therapists to observe and rectify the speech patterns of persons who stutter (PWS). The stuttered speech of PWS is usually available in limited amounts and is highly imbalanced. To this end, we address the class imbalance problem in the SD domain via a multibranching (MB) scheme and by weighting the contribution of classes in the overall loss function, resulting in a huge improvement in stuttering classes on the SEP-28k dataset over the baseline (StutterNet). To tackle data scarcity, we investigate the effectiveness of data augmentation on top of a multi-branched training scheme. The augmented training outperforms the MB StutterNet (clean) by a relative margin of 4.18% in macro F1-score (F1). In addition, we propose a multi-contextual (MC) StutterNet, which exploits different contexts of the stuttered speech, resulting in an overall improvement of 4.48% in F 1 over the single context based MB StutterNet. Finally, we have shown that applying data augmentation in the cross-corpora scenario can improve the overall SD performance by a relative margin of 13.23% in F1 over the clean training.【3】 Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification
标题：用于文本无关说话人确认的跨模式视听协同学习
链接：https://arxiv.org/abs/2302.11254
作者：Meng Liu,Kong Aik Lee,Longbiao Wang,Hanyi Zhang,Chang Zeng,Jianwu Dang机构：College of Intelligence and Computing, Tianjin University, China, Institute for Infocomm Research, A⋆STAR, Singapore, National Institute of Informatics, Tokyo, Japan摘要：视觉语音（即，嘴唇运动）由于语音产生中的同现和同步而与听觉语音高度相关。本文研究了这种相关性，并提出了一种跨模态语音协同学习模式。我们的跨通道协同学习方法的主要动机是通过利用来自另一个通道的知识来辅助对一个通道进行建模。具体而言，基于视听伪连体结构引入两个跨模态增强器来学习模态变换后的相关性。在每个增强器内部，提出了一个最大特征映射嵌入式Transformer变体，用于模态对齐和增强的特征生成。该网络是从头开始和与预先训练的模型共同学习的。在LRSLip 3、GridLip、LomGridLip和VoxLip数据集上的实验结果表明，与独立训练的纯音频/纯视频融合系统和基线融合系统相比，该方法分别获得了60%和20%的平均相对性能提升.摘要：Visual speech (i.e., lip motion) is highly related to auditory speech due to the co-occurrence and synchronization in speech production. This paper investigates this correlation and proposes a cross-modal speech co-learning paradigm. The primary motivation of our cross-modal co-learning method is modeling one modality aided by exploiting knowledge from another modality. Specifically, two cross-modal boosters are introduced based on an audio-visual pseudo-siamese structure to learn the modality-transformed correlation. Inside each booster, a max-feature-map embedded Transformer variant is proposed for modality alignment and enhanced feature generation. The network is co-learned both from scratch and with pretrained models. Experimental results on the LRSLip3, GridLip, LomGridLip, and VoxLip datasets demonstrate that our proposed method achieves 60% and 20% average relative performance improvement over independently trained audio-only/visual-only and baseline fusion systems, respectively.【4】 MADI: Inter-domain Matching and Intra-domain Discrimination for Cross-domain Speech Recognition
标题：MADI：跨域语音识别的域间匹配和域内区分
链接：https://arxiv.org/abs/2302.11224
作者：Jiaming Zhou,Shiwan Zhao,Ning Jiang,Guoqing Zhao,Yong Qin机构：Nankai University, Tianjin, China, Mashang Consumer Finance Co., Ltd.备注：Accepted to ICASSP 2023摘要：端到端自动语音识别（ASR）在应用到新的领域时，往往会由于领域迁移而导致性能下降。无监督领域自适应（Unsupervised Domain Adaptation，UDA）旨在通过将知识从源域转移到目标域来提高未标记目标域的性能。为了提高模型的可移植性，现有的UDA方法主要关注于全局和/或局部匹配源域和目标域的分布，而忽略了模型的可区分性.本文提出了一种新的基于域间匹配和域内区分的ASR UDA方法，通过细粒度的域间匹配提高模型的可移植性，同时通过域内对比区分提高模型的可区分性。在Libri-Adapt数据集上的评价证明了该方法的有效性。MADI使跨设备和跨环境ASR的相对误字率（WER）分别降低了17.7%和22.8%。摘要：End-to-end automatic speech recognition (ASR) usually suffers from performance degradation when applied to a new domain due to domain shift. Unsupervised domain adaptation (UDA) aims to improve the performance on the unlabeled target domain by transferring knowledge from the source to the target domain. To improve transferability, existing UDA approaches mainly focus on matching the distributions of the source and target domains globally and/or locally, while ignoring the model discriminability. In this paper, we propose a novel UDA approach for ASR via inter-domain MAtching and intra-domain DIscrimination (MADI), which improves the model transferability by fine-grained inter-domain matching and discriminability by intra-domain contrastive discrimination simultaneously. Evaluations on the Libri-Adapt dataset demonstrate the effectiveness of our approach. MADI reduces the relative word error rate (WER) on cross-device and cross-environment ASR by 17.7% and 22.8%, respectively.【5】 Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation
标题：通过外部声学注意和语义感知数据增强提高上下文拼写纠正
链接：https://arxiv.org/abs/2302.11192
作者：Xiaoqiang Wang,Yanqing Liu,Jinyu Li,Sheng Zhao机构：Microsoft Corporation, Redmond, WA, US备注：Accepted by ICASSP 2023摘要：背景拼写校正（contextual spelling correction，CSC）是利用姓名、地点等背景信息对端到端（end-to-end，E2 E）自动语音识别（automatic speech recognition，ASR）模型的输出进行校正的一种方法。虽然CSC在偏误问题上取得了一定的改进，但仍存在两个不足，有待进一步提高准确率。首先，由于纯文本假设中的信息限制或ASR模型在稀有领域上的弱性能，CSC模型可能无法校正具有相似发音的短语或其中所有偏向短语不存在于话语中的反上下文情况。第二，CSC的训练和推理存在差异。训练中的偏差列表是随机选择的，但是在推断中，在地面真实短语和其他短语之间可能存在更多相似性。为了解决上述问题，本文提出了一种改进的非自回归（NAR）拼写校正模型，用于基于E2 E神经传感器的ASR系统中的上下文偏差，以从两个方面改进先前的CSC模型：首先，我们将声学信息与外部注意力以及文本假设结合到CSC中，以更好地将目标短语与不相似或不相关的短语区分开来。其次，在训练短语中设计语义感知的数据扩充模式，减少训练和推理之间的不匹配，进一步提高偏置准确率。实验结果表明，改进后的方法比基线ASR+Biasing系统的相对姓名召回率提高了20.3%，并且在不同的偏好列表姓名覆盖率下，与传统CSC方法相比取得了稳定的提高。摘要：We previously proposed contextual spelling correction (CSC) to correct the output of end-to-end (E2E) automatic speech recognition (ASR) models with contextual information such as name, place, etc. Although CSC has achieved reasonable improvement in the biasing problem, there are still two drawbacks for further accuracy improvement. First, due to information limitation in text only hypothesis or weak performance of ASR model on rare domains, the CSC model may fail to correct phrases with similar pronunciation or anti-context cases where all biasing phrases are not present in the utterance. Second, there is a discrepancy between the training and inference of CSC. The bias list in training is randomly selected but in inference there may be more similarity between ground truth phrase and other phrases. To solve above limitations, in this paper we propose an improved non-autoregressive (NAR) spelling correction model for contextual biasing in E2E neural transducer-based ASR systems to improve the previous CSC model from two perspectives: Firstly, we incorporate acoustics information with an external attention as well as text hypotheses into CSC to better distinguish target phrase from dissimilar or irrelevant phrases. Secondly, we design a semantic aware data augmentation schema in training phrase to reduce the mismatch between training and inference to further boost the biasing accuracy. Experiments show that the improved method outperforms the baseline ASR+Biasing system by as much as 20.3% relative name recall gain and achieves stable improvement compared to the previous CSC method over different bias list name coverage ratio.【6】 Do Orcas Have Semantic Language? Machine Learning to Predict Orca Behaviors Using Partially Labeled Vocalization Data
标题：奥卡有语义语言吗？使用部分标记的发声数据预测虎鲸行为的机器学习
链接：https://arxiv.org/abs/2302.10983
摘要：虎鲸（逆戟鲸）发出复杂的叫声。它们只持续一秒钟。在一次呼叫中，虎鲸通常同时使用多个频率，改变频率，改变音量。由于虎鲸生活在水下，行动迅速，因此很难获得行为数据。声音数据相对容易捕获。作为一个科学目标，我们想知道虎鲸的发声是否构成了一种语义语言。我们通过研究机器学习是否能从发声预测行为来做到这一点。这样的预测还将有助于科学研究和安全应用，因为人们希望预测行为而仅必须捕获声音。这一过程中的一个重大挑战是缺乏标记数据。我们使用麦克默多湾逆戟鲸的最新记录[Wellard et al. 2020]，其中每个记录都标有记录期间观察到的行为。这就产生了一个数据集，其中录音中的声音片段--可以被认为是呼叫序列或更一般结构的连续发声--被标记为多余的行为。尽管如此，通过仔细结合最新的机器学习技术，我们实现了96. 4%的分类准确率。这表明虎鲸确实使用语义语言。具有良好的研究和应用前景。摘要：Orcinus orca (killer whales) exhibit complex calls. They last about a second. In a call, an orca typically uses multiple frequencies simultaneously, varies the frequencies, and varies their volumes. Behavior data is hard to obtain because orcas live under water and travel quickly. Sound data is relatively easy to capture. As a science goal, we would like to know whether orca vocalizations constitute a semantic language. We do this by studying whether machine learning can predict behavior from vocalizations. Such prediction would also help scientific research and safety applications because one would like to predict behavior while only having to capture sound. A significant challenge in this process is lack of labeled data. We work with recent recordings of McMurdo Sound orcas [Wellard et al. 2020] where each recording is labeled with the behaviors observed during the recording. This yields a dataset where sound segments - continuous vocalizations that can be thought of as call sequences or more general structures - within the recordings are labeled with superfluous behaviors. Despite that, with a careful combination of recent machine learning techniques, we achieve 96.4% classification accuracy. This suggests that orcas do use a semantic language. It is also promising for research and applications.【7】 A Reinforcement Learning Framework for Online Speaker Diarization
标题：一种在线说话人二值化的强化学习框架
链接：https://arxiv.org/abs/2302.10924
作者：Baihan Lin,Xinxin Zhang机构：Columbia University, New York, NY , USA, New York University摘要：说话人日记化是在每个给定的时间戳用说话人的身份来标记音频或视频记录的任务。在这项工作中，我们提出了一个新的机器学习框架，以进行实时多说话人日记化和识别，而无需事先注册和预训练，在一个完全在线和强化学习设置。我们的框架将嵌入提取、聚类和再分割结合到同一个问题中，作为在线决策问题。我们讨论了一些实用的考虑因素和先进的技术，如离线强化学习、半监督和域自适应，以解决有限的训练数据和非分布环境的挑战。该方法将说话人日记化视为说话人识别任务的一个完全在线学习问题，其中Agent在部署前不接受任何训练集的预训练，而是通过奖励反馈动态学习说话人身份检测.说话人日记化的强化学习方法的范例提出了一种自适应的、轻量级的和可推广的系统，该系统对于多用户电话会议是有用的，其中许多人可能来来去去而没有提前广泛的预注册。最后，我们提供了一个桌面应用程序，它使用我们提出的方法作为概念验证。据我们所知，这是第一种将强化学习方法应用于说话人日记化任务的方法。摘要：Speaker diarization is a task to label an audio or video recording with the identity of the speaker at each given time stamp. In this work, we propose a novel machine learning framework to conduct real-time multi-speaker diarization and recognition without prior registration and pretraining in a fully online and reinforcement learning setting. Our framework combines embedding extraction, clustering, and resegmentation into the same problem as an online decision-making problem. We discuss practical considerations and advanced techniques such as the offline reinforcement learning, semi-supervision, and domain adaptation to address the challenges of limited training data and out-of-distribution environments. Our approach considers speaker diarization as a fully online learning problem of the speaker recognition task, where the agent receives no pretraining from any training set before deployment, and learns to detect speaker identity on the fly through reward feedbacks. The paradigm of the reinforcement learning approach to speaker diarization presents an adaptive, lightweight, and generalizable system that is useful for multi-user teleconferences, where many people might come and go without extensive pre-registration ahead of time. Lastly, we provide a desktop application that uses our proposed approach as a proof of concept. To the best of our knowledge, this is the first approach to apply a reinforcement learning approach to the speaker diarization task.【8】 Conformers are All You Need for Visual Speech Recogntion
标题：仿真器是视觉语音识别所需的全部
链接：https://arxiv.org/abs/2302.10915
作者：Oscar Chang,Hank Liao,Dmitriy Serdyuk,Ankit Shah,Olivier Siohan摘要：视觉语音识别模型以分层方式提取视觉特征。在较低的层次上，有一个视觉前端，它有一个有限的时间感受野，处理描绘嘴唇或面部的原始像素。在更高的层次上，有一个编码器，它处理前端在一个大的时间感受野上产生的嵌入。先前的工作集中在改进模型的视觉前端，以提取更多对语音识别有用的特征。令人惊讶的是，我们的工作表明复杂的视觉前端是不必要的。我们发现，与将资源分配给复杂的视觉前端不同，线性视觉前端与较大的Conformer编码器配对会导致更低的延迟、更高效的内存使用和更好的WER性能。我们在TED LRS 3数据集的视觉语音识别方面取得了12.8 $ WER的最新技术水平，其性能可与四年前的纯音频模型相媲美。摘要：Visual speech recognition models extract visual features in a hierarchical manner. At the lower level, there is a visual front-end with a limited temporal receptive field that processes the raw pixels depicting the lips or faces. At the higher level, there is an encoder that attends to the embeddings produced by the front-end over a large temporal receptive field. Previous work has focused on improving the visual front-end of the model to extract more useful features for speech recognition. Surprisingly, our work shows that complex visual front-ends are not necessary. Instead of allocating resources to a sophisticated visual front-end, we find that a linear visual front-end paired with a larger Conformer encoder results in lower latency, more efficient memory usage, and improved WER performance. We achieve a new state-of-the-art of $12.8\%$ WER for visual speech recognition on the TED LRS3 dataset, which rivals the performance of audio-only models from just four years ago.【9】 Gradient Remedy for Multi-Task Learning in End-to-End Noise-Robust Speech Recognition
标题：端到端抗噪语音识别中多任务学习的梯度补救算法
链接：https://arxiv.org/abs/2302.11362
作者：Yuchen Hu,Chen Chen,Ruizhe Li,Qiushi Zhu,Eng Siong Chng机构：Nanyang Technological University, Singapore, University of Aberdeen, UK, University of Science and Technology of China, China备注：5 pages, 5 figures, Accepted by ICASSP 2023摘要：语音增强（SE）技术被证明可以有效地降低下游自动语音识别（ASR）中的噪声，其中采用多任务学习策略来联合优化这两个任务。然而，通过SE目标学习的增强语音可能不总是产生良好的ASR结果。从优化的角度来看，搜索引擎和ASR任务的梯度之间有时会存在干扰，这会阻碍多任务学习，最终导致ASR性能的次优。本文提出了一种简单而有效的梯度修正方法（GR），从角度和幅度两个方面解决了噪声鲁棒语音识别中任务梯度之间的干扰。具体地说，我们首先将SE任务的梯度投影到与ASR梯度成锐角的动态曲面上，以消除它们之间的冲突并辅助ASR优化。此外，我们自适应地重新调整两个梯度的大小，以防止主导ASR任务被SE梯度误导。实验结果表明，该方法能够很好地解决梯度干扰问题，在RATS和CHiME-4数据集上，相对于多任务学习基线，相对误词率（WER）分别降低了9.3%和11.1%。我们的代码可以在GitHub上找到。摘要：Speech enhancement (SE) is proved effective in reducing noise from noisy speech signals for downstream automatic speech recognition (ASR), where multi-task learning strategy is employed to jointly optimize these two tasks. However, the enhanced speech learned by SE objective may not always yield good ASR results. From the optimization view, there sometimes exists interference between the gradients of SE and ASR tasks, which could hinder the multi-task learning and finally lead to sub-optimal ASR performance. In this paper, we propose a simple yet effective approach called gradient remedy (GR) to solve interference between task gradients in noise-robust speech recognition, from perspectives of both angle and magnitude. Specifically, we first project the SE task's gradient onto a dynamic surface that is at acute angle to ASR gradient, in order to remove the conflict between them and assist in ASR optimization. Furthermore, we adaptively rescale the magnitude of two gradients to prevent the dominant ASR task from being misled by SE gradient. Experimental results show that the proposed approach well resolves the gradient interference and achieves relative word error rate (WER) reductions of 9.3% and 11.1% over multi-task learning baseline, on RATS and CHiME-4 datasets, respectively. Our code is available at GitHub.【10】 Contrastive Representation Learning for Acoustic Parameter Estimation
标题：声学参数估计的对比表示学习
链接：https://arxiv.org/abs/2302.11205
作者：Philipp Götz,Cagdas Tuna,Andreas Walther,Emanuël A. P. Habets机构：International Audio Laboratories Erlangen†, Germany., Fraunhofer Institute for Integrated Circuits IIS, Erlangen, Germany.备注：Accepted for ICASSP 2023摘要：提出了一种对比学习方法，用于从单通道混响语音信号中提取低维的声学环境表示。房间脉冲响应（RIR）与无回声源信号的卷积被用作数据扩增技术，其在上游任务的设计中提供相当大的灵活性。我们评估了三个不同下游任务的嵌入，包括声学参数混响时间RT60和清晰度指数C50的回归，以及小房间和大房间的分类。我们证明了学习的表示很好地推广到看不见的数据，并取得了类似的性能相比，完全监督的基线。摘要：A study is presented in which a contrastive learning approach is used to extract low-dimensional representations of the acoustic environment from single-channel, reverberant speech signals. Convolution of room impulse responses (RIRs) with anechoic source signals is leveraged as a data augmentation technique that offers considerable flexibility in the design of the upstream task. We evaluate the embeddings across three different downstream tasks, which include the regression of acoustic parameters reverberation time RT60 and clarity index C50, and the classification into small and large rooms. We demonstrate that the learned representations generalize well to unseen data and achieve similar performance compared to a fully supervised baseline.【11】 UML: A Universal Monolingual Output Layer for Multilingual ASR
标题：统一建模语言：支持多语言ASR的通用单语输出层
链接：https://arxiv.org/abs/2302.11186
作者：Chao Zhang,Bo Li,Tara N. Sainath,Trevor Strohman,Shuo-yiin Chang备注：Published as a conference paper at ICASSP 2023摘要：词块模型（WPM）是当前端到端自动语音识别（ASR）系统中常用的子词单元。对于多语言ASR，由于不同语言的书面脚本存在差异，多语言WPM带来了输出层过大和扩展到更多语言的挑战。在这项工作中，我们提出了一个通用的单语输出层（UML）来解决这些问题。UML将每个输出节点与多个WPM（每种语言一个WPM）重新关联，而不是仅为一个WPM提供一个输出节点，并产生跨语言共享的较小的单语言输出层。因此，UML能够根据输入语音的语言切换每个输出节点的解释。在11种语言语音搜索任务上的实验结果证明了UML用于高质量、高效率的多语言流ASR的可行性。摘要：Word-piece models (WPMs) are commonly used subword units in state-of-the-art end-to-end automatic speech recognition (ASR) systems. For multilingual ASR, due to the differences in written scripts across languages, multilingual WPMs bring the challenges of having overly large output layers and scaling to more languages. In this work, we propose a universal monolingual output layer (UML) to address such problems. Instead of one output node for only one WPM, UML re-associates each output node with multiple WPMs, one for each language, and results in a smaller monolingual output layer shared across languages. Consequently, the UML enables to switch in the interpretation of each output node depending on the language of the input speech. Experimental results on an 11-language voice search task demonstrated the feasibility of using UML for high-quality and high-efficiency multilingual streaming ASR.【12】 Unifying Speech Enhancement and Separation with Gradient Modulation for End-to-End Noise-Robust Speech Separation
标题：语音增强和分离与梯度调制相结合的端到端抗噪语音分离
链接：https://arxiv.org/abs/2302.11131
作者：Yuchen Hu,Chen Chen,Heqing Zou,Xionghu Zhong,Eng Siong Chng机构：School of Computer Science and Engineering, Nanyang Technological University, Singapore, College of Computer Science and Electronic Engineering, Hunan University, China备注：5 pages, 5 figures, Accepted by ICASSP 2023摘要：近年来，由于对长序列建模能力的提高，基于神经网络的单耳语音分离（SS）研究取得了显著的成功。然而，当置于实际噪声条件下时，它们会显著退化，因为背景噪声可能被误认为是说话者的语音，从而干扰分离的源。为了解决这个问题，我们提出了一个新的网络，统一语音增强和分离与梯度调制，以提高噪声的鲁棒性。具体而言，我们首先通过结合语音增强（SE）和分离模块构建统一网络，并使用多任务学习进行优化，其中SE由并行清洁混合监督以降低下游语音分离的噪声。此外，为了避免在降噪过程中抑制有效说话人信息，本文提出了一种梯度调制（GM）策略，从优化的角度协调SE和SS任务。实验结果表明，该方法在大规模Libri 2 Mix和Libri 3 Mix噪声数据集上达到了最优水平，SI-SNRi分别为16.0dB和15.8dB.我们的代码可以在GitHub上找到。摘要：Recent studies in neural network-based monaural speech separation (SS) have achieved a remarkable success thanks to increasing ability of long sequence modeling. However, they would degrade significantly when put under realistic noisy conditions, as the background noise could be mistaken for speaker's speech and thus interfere with the separated sources. To alleviate this problem, we propose a novel network to unify speech enhancement and separation with gradient modulation to improve noise-robustness. Specifically, we first build a unified network by combining speech enhancement (SE) and separation modules, with multi-task learning for optimization, where SE is supervised by parallel clean mixture to reduce noise for downstream speech separation. Furthermore, in order to avoid suppressing valid speaker information when reducing noise, we propose a gradient modulation (GM) strategy to harmonize the SE and SS tasks from optimization view. Experimental results show that our approach achieves the state-of-the-art on large-scale Libri2Mix- and Libri3Mix-noisy datasets, with SI-SNRi results of 16.0 dB and 15.8 dB respectively. Our code is available at GitHub.【1】 Gradient Remedy for Multi-Task Learning in End-to-End Noise-Robust Speech Recognition
标题：端到端抗噪语音识别中多任务学习的梯度补救算法
链接：https://arxiv.org/abs/2302.11362
*与cs.SD语音【9】为同一篇
作者：Yuchen Hu,Chen Chen,Ruizhe Li,Qiushi Zhu,Eng Siong Chng机构：Nanyang Technological University, Singapore, University of Aberdeen, UK, University of Science and Technology of China, China备注：5 pages, 5 figures, Accepted by ICASSP 2023摘要：语音增强（SE）技术被证明可以有效地降低下游自动语音识别（ASR）中的噪声，其中采用多任务学习策略来联合优化这两个任务。然而，通过SE目标学习的增强语音可能不总是产生良好的ASR结果。从优化的角度来看，搜索引擎和ASR任务的梯度之间有时会存在干扰，这会阻碍多任务学习，最终导致ASR性能的次优。本文提出了一种简单而有效的梯度修正方法（GR），从角度和幅度两个方面解决了噪声鲁棒语音识别中任务梯度之间的干扰。具体地说，我们首先将SE任务的梯度投影到与ASR梯度成锐角的动态曲面上，以消除它们之间的冲突并辅助ASR优化。此外，我们自适应地重新调整两个梯度的大小，以防止主导ASR任务被SE梯度误导。实验结果表明，该方法能够很好地解决梯度干扰问题，在RATS和CHiME-4数据集上，相对于多任务学习基线，相对误词率（WER）分别降低了9.3%和11.1%。我们的代码可以在GitHub上找到。摘要：Speech enhancement (SE) is proved effective in reducing noise from noisy speech signals for downstream automatic speech recognition (ASR), where multi-task learning strategy is employed to jointly optimize these two tasks. However, the enhanced speech learned by SE objective may not always yield good ASR results. From the optimization view, there sometimes exists interference between the gradients of SE and ASR tasks, which could hinder the multi-task learning and finally lead to sub-optimal ASR performance. In this paper, we propose a simple yet effective approach called gradient remedy (GR) to solve interference between task gradients in noise-robust speech recognition, from perspectives of both angle and magnitude. Specifically, we first project the SE task's gradient onto a dynamic surface that is at acute angle to ASR gradient, in order to remove the conflict between them and assist in ASR optimization. Furthermore, we adaptively rescale the magnitude of two gradients to prevent the dominant ASR task from being misled by SE gradient. Experimental results show that the proposed approach well resolves the gradient interference and achieves relative word error rate (WER) reductions of 9.3% and 11.1% over multi-task learning baseline, on RATS and CHiME-4 datasets, respectively. Our code is available at GitHub.【2】 Contrastive Representation Learning for Acoustic Parameter Estimation
标题：声学参数估计的对比表示学习
链接：https://arxiv.org/abs/2302.11205
*与cs.SD语音【10】为同一篇
作者：Philipp Götz,Cagdas Tuna,Andreas Walther,Emanuël A. P. Habets机构：International Audio Laboratories Erlangen†, Germany., Fraunhofer Institute for Integrated Circuits IIS, Erlangen, Germany.备注：Accepted for ICASSP 2023摘要：提出了一种对比学习方法，用于从单通道混响语音信号中提取低维的声学环境表示。房间脉冲响应（RIR）与无回声源信号的卷积被用作数据扩增技术，其在上游任务的设计中提供相当大的灵活性。我们评估了三个不同下游任务的嵌入，包括声学参数混响时间RT60和清晰度指数C50的回归，以及小房间和大房间的分类。我们证明了学习的表示很好地推广到看不见的数据，并取得了类似的性能相比，完全监督的基线。摘要：A study is presented in which a contrastive learning approach is used to extract low-dimensional representations of the acoustic environment from single-channel, reverberant speech signals. Convolution of room impulse responses (RIRs) with anechoic source signals is leveraged as a data augmentation technique that offers considerable flexibility in the design of the upstream task. We evaluate the embeddings across three different downstream tasks, which include the regression of acoustic parameters reverberation time RT60 and clarity index C50, and the classification into small and large rooms. We demonstrate that the learned representations generalize well to unseen data and achieve similar performance compared to a fully supervised baseline.【3】 UML: A Universal Monolingual Output Layer for Multilingual ASR
标题：统一建模语言：支持多语言ASR的通用单语输出层
链接：https://arxiv.org/abs/2302.11186
*与cs.SD语音【11】为同一篇
作者：Chao Zhang,Bo Li,Tara N. Sainath,Trevor Strohman,Shuo-yiin Chang备注：Published as a conference paper at ICASSP 2023摘要：词块模型（WPM）是当前端到端自动语音识别（ASR）系统中常用的子词单元。对于多语言ASR，由于不同语言的书面脚本存在差异，多语言WPM带来了输出层过大和扩展到更多语言的挑战。在这项工作中，我们提出了一个通用的单语输出层（UML）来解决这些问题。UML将每个输出节点与多个WPM（每种语言一个WPM）重新关联，而不是仅为一个WPM提供一个输出节点，并产生跨语言共享的较小的单语言输出层。因此，UML能够根据输入语音的语言切换每个输出节点的解释。在11种语言语音搜索任务上的实验结果证明了UML用于高质量、高效率的多语言流ASR的可行性。摘要：Word-piece models (WPMs) are commonly used subword units in state-of-the-art end-to-end automatic speech recognition (ASR) systems. For multilingual ASR, due to the differences in written scripts across languages, multilingual WPMs bring the challenges of having overly large output layers and scaling to more languages. In this work, we propose a universal monolingual output layer (UML) to address such problems. Instead of one output node for only one WPM, UML re-associates each output node with multiple WPMs, one for each language, and results in a smaller monolingual output layer shared across languages. Consequently, the UML enables to switch in the interpretation of each output node depending on the language of the input speech. Experimental results on an 11-language voice search task demonstrated the feasibility of using UML for high-quality and high-efficiency multilingual streaming ASR.【4】 Unifying Speech Enhancement and Separation with Gradient Modulation for End-to-End Noise-Robust Speech Separation
标题：语音增强和分离与梯度调制相结合的端到端抗噪语音分离
链接：https://arxiv.org/abs/2302.11131
*与cs.SD语音【12】为同一篇
作者：Yuchen Hu,Chen Chen,Heqing Zou,Xionghu Zhong,Eng Siong Chng机构：School of Computer Science and Engineering, Nanyang Technological University, Singapore, College of Computer Science and Electronic Engineering, Hunan University, China备注：5 pages, 5 figures, Accepted by ICASSP 2023摘要：近年来，由于对长序列建模能力的提高，基于神经网络的单耳语音分离（SS）研究取得了显著的成功。然而，当置于实际噪声条件下时，它们会显著退化，因为背景噪声可能被误认为是说话者的语音，从而干扰分离的源。为了解决这个问题，我们提出了一个新的网络，统一语音增强和分离与梯度调制，以提高噪声的鲁棒性。具体而言，我们首先通过结合语音增强（SE）和分离模块构建统一网络，并使用多任务学习进行优化，其中SE由并行清洁混合监督以降低下游语音分离的噪声。此外，为了避免在降噪过程中抑制有效说话人信息，本文提出了一种梯度调制（GM）策略，从优化的角度协调SE和SS任务。实验结果表明，该方法在大规模Libri 2 Mix和Libri 3 Mix噪声数据集上达到了最优水平，SI-SNRi分别为16.0dB和15.8dB.我们的代码可以在GitHub上找到。摘要：Recent studies in neural network-based monaural speech separation (SS) have achieved a remarkable success thanks to increasing ability of long sequence modeling. However, they would degrade significantly when put under realistic noisy conditions, as the background noise could be mistaken for speaker's speech and thus interfere with the separated sources. To alleviate this problem, we propose a novel network to unify speech enhancement and separation with gradient modulation to improve noise-robustness. Specifically, we first build a unified network by combining speech enhancement (SE) and separation modules, with multi-task learning for optimization, where SE is supervised by parallel clean mixture to reduce noise for downstream speech separation. Furthermore, in order to avoid suppressing valid speaker information when reducing noise, we propose a gradient modulation (GM) strategy to harmonize the SE and SS tasks from optimization view. Experimental results show that our approach achieves the state-of-the-art on large-scale Libri2Mix- and Libri3Mix-noisy datasets, with SI-SNRi results of 16.0 dB and 15.8 dB respectively. Our code is available at GitHub.【5】 Improving Speech Enhancement via Event-based Query
标题：通过基于事件的查询改进语音增强
链接：https://arxiv.org/abs/2302.11558
*与cs.SD语音【1】为同一篇
作者：Yifei Xin,Xiulian Peng,Yan Lu机构：Peking University, Beijing, China, Microsoft Research Asia, Beijing, China摘要：Existing deep learning based speech enhancement (SE) methods either use blind end-to-end training or explicitly incorporate speaker embedding or phonetic information into the SE network to enhance speech quality. In this paper, we perceive speech and noises as different types of sound events and propose an event-based query method for SE. Specifically, representative speech embeddings that can discriminate speech with noises are first pre-trained with the sound event detection (SED) task. The embeddings are then clustered into fixed golden speech queries to assist the SE network to enhance the speech from noisy audio. The golden speech queries can be obtained offline and generalizable to different SE datasets and networks. Therefore, little extra complexity is introduced and no enrollment is needed for each speaker. Experimental results show that the proposed method yields significant gains compared with baselines and the golden queries are well generalized to different datasets.【6】 Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning
标题：基于数据增强、类均衡丢失和多语境深度学习的口吃检测
链接：https://arxiv.org/abs/2302.11343
*与cs.SD语音【2】为同一篇
作者：Shakeel A. Sheikh,Md Sahidullah,Fabrice Hirsch,Slim Ouni备注：Accepted in IEEE Journal of Biomedical Health Informatics 2023摘要：口吃是一种神经发育性言语障碍，其特征是不受控制的言语（感叹词）和核心行为（阻塞、重复和延长），是由言语感觉运动障碍引起的。由于其复杂的性质，口吃检测（SD）是一项困难的任务。如果在早期阶段发现，它可以帮助言语治疗师观察和纠正口吃者（PWS）的言语模式。PWS的口吃语音通常是有限的，并且高度不平衡。为此，我们通过多分支（MB）方案并通过加权总体损失函数中的类别贡献来解决SD域中的类别不平衡问题，从而使SEP-28 k数据集上的口吃类别比基线（StutterNet）有了巨大改善。为了解决数据不足的问题，我们研究了基于多分支训练方案的数据扩充的有效性。在宏观F1评分（F1）方面，增强训练优于MB StutterNet（clean），相对优势为4.18%。此外，我们还提出了一种多上下文（MC）StutterNet，它利用口吃语音的不同上下文，使F1的总成绩比基于单上下文的MB StutterNet提高了4.48%。最后，我们已经表明，在跨语料库场景中应用数据扩充可以提高总体SD性能，在F1中相对于干净训练的相对幅度为13.23%。摘要：Stuttering is a neuro-developmental speech impairment characterized by uncontrolled utterances (interjections) and core behaviors (blocks, repetitions, and prolongations), and is caused by the failure of speech sensorimotors. Due to its complex nature, stuttering detection (SD) is a difficult task. If detected at an early stage, it could facilitate speech therapists to observe and rectify the speech patterns of persons who stutter (PWS). The stuttered speech of PWS is usually available in limited amounts and is highly imbalanced. To this end, we address the class imbalance problem in the SD domain via a multibranching (MB) scheme and by weighting the contribution of classes in the overall loss function, resulting in a huge improvement in stuttering classes on the SEP-28k dataset over the baseline (StutterNet). To tackle data scarcity, we investigate the effectiveness of data augmentation on top of a multi-branched training scheme. The augmented training outperforms the MB StutterNet (clean) by a relative margin of 4.18% in macro F1-score (F1). In addition, we propose a multi-contextual (MC) StutterNet, which exploits different contexts of the stuttered speech, resulting in an overall improvement of 4.48% in F 1 over the single context based MB StutterNet. Finally, we have shown that applying data augmentation in the cross-corpora scenario can improve the overall SD performance by a relative margin of 13.23% in F1 over the clean training.【7】 Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification
标题：用于文本无关说话人确认的跨模式视听协同学习
链接：https://arxiv.org/abs/2302.11254
*与cs.SD语音【3】为同一篇
作者：Meng Liu,Kong Aik Lee,Longbiao Wang,Hanyi Zhang,Chang Zeng,Jianwu Dang机构：College of Intelligence and Computing, Tianjin University, China, Institute for Infocomm Research, A⋆STAR, Singapore, National Institute of Informatics, Tokyo, Japan摘要：视觉语音（即，嘴唇运动）由于语音产生中的同现和同步而与听觉语音高度相关。本文研究了这种相关性，并提出了一种跨模态语音协同学习模式。我们的跨通道协同学习方法的主要动机是通过利用来自另一个通道的知识来辅助对一个通道进行建模。具体而言，基于视听伪连体结构引入两个跨模态增强器来学习模态变换后的相关性。在每个增强器内部，提出了一个最大特征映射嵌入式Transformer变体，用于模态对齐和增强的特征生成。该网络是从头开始和与预先训练的模型共同学习的。在LRSLip 3、GridLip、LomGridLip和VoxLip数据集上的实验结果表明，与独立训练的纯音频/纯视频融合系统和基线融合系统相比，该方法分别获得了60%和20%的平均相对性能提升.摘要：Visual speech (i.e., lip motion) is highly related to auditory speech due to the co-occurrence and synchronization in speech production. This paper investigates this correlation and proposes a cross-modal speech co-learning paradigm. The primary motivation of our cross-modal co-learning method is modeling one modality aided by exploiting knowledge from another modality. Specifically, two cross-modal boosters are introduced based on an audio-visual pseudo-siamese structure to learn the modality-transformed correlation. Inside each booster, a max-feature-map embedded Transformer variant is proposed for modality alignment and enhanced feature generation. The network is co-learned both from scratch and with pretrained models. Experimental results on the LRSLip3, GridLip, LomGridLip, and VoxLip datasets demonstrate that our proposed method achieves 60% and 20% average relative performance improvement over independently trained audio-only/visual-only and baseline fusion systems, respectively.【8】 MADI: Inter-domain Matching and Intra-domain Discrimination for Cross-domain Speech Recognition
标题：MADI：跨域语音识别的域间匹配和域内区分
链接：https://arxiv.org/abs/2302.11224
*与cs.SD语音【4】为同一篇
作者：Jiaming Zhou,Shiwan Zhao,Ning Jiang,Guoqing Zhao,Yong Qin机构：Nankai University, Tianjin, China, Mashang Consumer Finance Co., Ltd.备注：Accepted to ICASSP 2023摘要：End-to-end automatic speech recognition (ASR) usually suffers from performance degradation when applied to a new domain due to domain shift. Unsupervised domain adaptation (UDA) aims to improve the performance on the unlabeled target domain by transferring knowledge from the source to the target domain. To improve transferability, existing UDA approaches mainly focus on matching the distributions of the source and target domains globally and/or locally, while ignoring the model discriminability. In this paper, we propose a novel UDA approach for ASR via inter-domain MAtching and intra-domain DIscrimination (MADI), which improves the model transferability by fine-grained inter-domain matching and discriminability by intra-domain contrastive discrimination simultaneously. Evaluations on the Libri-Adapt dataset demonstrate the effectiveness of our approach. MADI reduces the relative word error rate (WER) on cross-device and cross-environment ASR by 17.7% and 22.8%, respectively.【9】 Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation
标题：通过外部声学注意和语义感知数据增强提高上下文拼写纠正
链接：https://arxiv.org/abs/2302.11192
*与cs.SD语音【5】为同一篇
作者：Xiaoqiang Wang,Yanqing Liu,Jinyu Li,Sheng Zhao机构：Microsoft Corporation, Redmond, WA, US备注：Accepted by ICASSP 2023摘要：背景拼写校正（contextual spelling correction，CSC）是利用姓名、地点等背景信息对端到端（end-to-end，E2 E）自动语音识别（automatic speech recognition，ASR）模型的输出进行校正的一种方法。虽然CSC在偏误问题上取得了一定的改进，但仍存在两个不足，有待进一步提高准确率。首先，由于纯文本假设中的信息限制或ASR模型在稀有领域上的弱性能，CSC模型可能无法校正具有相似发音的短语或其中所有偏向短语不存在于话语中的反上下文情况。第二，CSC的训练和推理存在差异。训练中的偏差列表是随机选择的，但是在推断中，在地面真实短语和其他短语之间可能存在更多相似性。为了解决上述问题，本文提出了一种改进的非自回归（NAR）拼写校正模型，用于基于E2 E神经传感器的ASR系统中的上下文偏差，以从两个方面改进先前的CSC模型：首先，我们将声学信息与外部注意力以及文本假设结合到CSC中，以更好地将目标短语与不相似或不相关的短语区分开来。其次，在训练短语中设计语义感知的数据扩充模式，减少训练和推理之间的不匹配，进一步提高偏置准确率。实验结果表明，改进后的方法比基线ASR+Biasing系统的相对姓名召回率提高了20.3%，并且在不同的偏好列表姓名覆盖率下，与传统CSC方法相比取得了稳定的提高。摘要：We previously proposed contextual spelling correction (CSC) to correct the output of end-to-end (E2E) automatic speech recognition (ASR) models with contextual information such as name, place, etc. Although CSC has achieved reasonable improvement in the biasing problem, there are still two drawbacks for further accuracy improvement. First, due to information limitation in text only hypothesis or weak performance of ASR model on rare domains, the CSC model may fail to correct phrases with similar pronunciation or anti-context cases where all biasing phrases are not present in the utterance. Second, there is a discrepancy between the training and inference of CSC. The bias list in training is randomly selected but in inference there may be more similarity between ground truth phrase and other phrases. To solve above limitations, in this paper we propose an improved non-autoregressive (NAR) spelling correction model for contextual biasing in E2E neural transducer-based ASR systems to improve the previous CSC model from two perspectives: Firstly, we incorporate acoustics information with an external attention as well as text hypotheses into CSC to better distinguish target phrase from dissimilar or irrelevant phrases. Secondly, we design a semantic aware data augmentation schema in training phrase to reduce the mismatch between training and inference to further boost the biasing accuracy. Experiments show that the improved method outperforms the baseline ASR+Biasing system by as much as 20.3% relative name recall gain and achieves stable improvement compared to the previous CSC method over different bias list name coverage ratio.【10】 Do Orcas Have Semantic Language? Machine Learning to Predict Orca Behaviors Using Partially Labeled Vocalization Data
标题：奥卡有语义语言吗？使用部分标记的发声数据预测虎鲸行为的机器学习
链接：https://arxiv.org/abs/2302.10983
*与cs.SD语音【6】为同一篇
摘要：Orcinus orca (killer whales) exhibit complex calls. They last about a second. In a call, an orca typically uses multiple frequencies simultaneously, varies the frequencies, and varies their volumes. Behavior data is hard to obtain because orcas live under water and travel quickly. Sound data is relatively easy to capture. As a science goal, we would like to know whether orca vocalizations constitute a semantic language. We do this by studying whether machine learning can predict behavior from vocalizations. Such prediction would also help scientific research and safety applications because one would like to predict behavior while only having to capture sound. A significant challenge in this process is lack of labeled data. We work with recent recordings of McMurdo Sound orcas [Wellard et al. 2020] where each recording is labeled with the behaviors observed during the recording. This yields a dataset where sound segments - continuous vocalizations that can be thought of as call sequences or more general structures - within the recordings are labeled with superfluous behaviors. Despite that, with a careful combination of recent machine learning techniques, we achieve 96.4% classification accuracy. This suggests that orcas do use a semantic language. It is also promising for research and applications.【11】 A Reinforcement Learning Framework for Online Speaker Diarization
标题：一种在线说话人二值化的强化学习框架
链接：https://arxiv.org/abs/2302.10924
*与cs.SD语音【7】为同一篇
作者：Baihan Lin,Xinxin Zhang机构：Columbia University, New York, NY , USA, New York University摘要：Speaker diarization is a task to label an audio or video recording with the identity of the speaker at each given time stamp. In this work, we propose a novel machine learning framework to conduct real-time multi-speaker diarization and recognition without prior registration and pretraining in a fully online and reinforcement learning setting. Our framework combines embedding extraction, clustering, and resegmentation into the same problem as an online decision-making problem. We discuss practical considerations and advanced techniques such as the offline reinforcement learning, semi-supervision, and domain adaptation to address the challenges of limited training data and out-of-distribution environments. Our approach considers speaker diarization as a fully online learning problem of the speaker recognition task, where the agent receives no pretraining from any training set before deployment, and learns to detect speaker identity on the fly through reward feedbacks. The paradigm of the reinforcement learning approach to speaker diarization presents an adaptive, lightweight, and generalizable system that is useful for multi-user teleconferences, where many people might come and go without extensive pre-registration ahead of time. Lastly, we provide a desktop application that uses our proposed approach as a proof of concept. To the best of our knowledge, this is the first approach to apply a reinforcement learning approach to the speaker diarization task.【12】 Conformers are All You Need for Visual Speech Recogntion
标题：仿真器是视觉语音识别所需的全部
链接：https://arxiv.org/abs/2302.10915
*与cs.SD语音【8】为同一篇
作者：Oscar Chang,Hank Liao,Dmitriy Serdyuk,Ankit Shah,Olivier Siohan摘要：视觉语音识别模型以分层方式提取视觉特征。在较低的层次上，有一个视觉前端，它有一个有限的时间感受野，处理描绘嘴唇或面部的原始像素。在更高的层次上，有一个编码器，它处理前端在一个大的时间感受野上产生的嵌入。先前的工作集中在改进模型的视觉前端，以提取更多对语音识别有用的特征。令人惊讶的是，我们的工作表明复杂的视觉前端是不必要的。我们发现，与将资源分配给复杂的视觉前端不同，线性视觉前端与较大的Conformer编码器配对会导致更低的延迟、更高效的内存使用和更好的WER性能。我们在TED LRS 3数据集的视觉语音识别方面取得了12.8 $ WER的最新技术水平，其性能可与四年前的纯音频模型相媲美。摘要：Visual speech recognition models extract visual features in a hierarchical manner. At the lower level, there is a visual front-end with a limited temporal receptive field that processes the raw pixels depicting the lips or faces. At the higher level, there is an encoder that attends to the embeddings produced by the front-end over a large temporal receptive field. Previous work has focused on improving the visual front-end of the model to extract more useful features for speech recognition. Surprisingly, our work shows that complex visual front-ends are not necessary. Instead of allocating resources to a sophisticated visual front-end, we find that a linear visual front-end paired with a larger Conformer encoder results in lower latency, more efficient memory usage, and improved WER performance. We achieve a new state-of-the-art of $12.8\%$ WER for visual speech recognition on the TED LRS3 dataset, which rivals the performance of audio-only models from just four years ago.机器翻译，仅供参考
每日语音速递