今日论文合集：cs.SD语音11篇，eess.AS音频处理11篇。【1】 Handling the Alignment for Wake Word Detection: A Comparison Between Alignment-Based, Alignment-Free and Hybrid Approaches
标题：处理尾迹词检测的对齐：基于对齐、无对齐和混合方法的比较
链接：https://arxiv.org/abs/2302.08950
作者：Vinicius Ribeiro,Yiteng Huang,Yuan Shangguan,Zhaojun Yang,Li Wan,Ming Sun机构：Meta AI, Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：唤醒词检测存在于大多数智能家庭和便携式设备中。它为这些设备提供了在召唤时“唤醒”的能力，同时降低了电力和计算成本。本文着重于理解对齐在开发一个回答通用短语的唤醒词系统中的作用。我们讨论三种方法。第一种是基于对齐的，其中使用帧方式交叉熵来训练模型。第二种是无对齐的，其中使用CTC训练模型。第三种是我们提出的一种混合解决方案，即先用一小部分对齐的数据集训练模型，然后用一个相当大的未对齐的数据集调整模型。我们比较了这三种方法，并评估了不同对齐与未对齐比率对混合训练的影响。我们的结果表明，免对准系统对于目标操作点执行更好的基于对准的系统，并且利用一小部分数据（20%），我们可以训练符合我们的初始约束的模型。摘要：Wake word detection exists in most intelligent homes and portable devices. It offers these devices the ability to "wake up" when summoned at a low cost of power and computing. This paper focuses on understanding alignment's role in developing a wake-word system that answers a generic phrase. We discuss three approaches. The first is alignment-based, where the model is trained with frame-wise cross-entropy. The second is alignment-free, where the model is trained with CTC. The third, proposed by us, is a hybrid solution in which the model is trained with a small set of aligned data and then tuned with a sizeable unaligned dataset. We compare the three approaches and evaluate the impact of the different aligned-to-unaligned ratios for hybrid training. Our results show that the alignment-free system performs better alignment-based for the target operating point, and with a small fraction of the data (20%), we can train a model that complies with our initial constraints.【2】 Deep Implicit Distribution Alignment Networks for Cross-Corpus Speech Emotion Recognition
标题：用于跨语料库语音情感识别的深度隐式分布对齐网络
链接：https://arxiv.org/abs/2302.08921
作者：Yan Zhao,Jincen Wang,Yuan Zong,Wenming Zheng,Hailun Lian,Li Zhao机构：Key Laboratory of Child Development and Learning Science of Ministry of Education, Southeast University, Nanjing , China摘要：本文提出了一种新的深度迁移学习方法--深度隐式分布对齐网络（DIDAN），用于处理带标记训练（源）语音信号和未标记测试（目标）语音信号来自不同语料库的跨语料库语音情感识别（SER）问题。具体地说，DIDAN首先采用一个由一组卷积和全连通层组成的简单深度回归网络，将源语音频谱直接回归为情感标签，从而使DIDAN具有情感辨别能力.然后，通过求助于被称为隐式分布对准（IDA）的良好设计的正则化项，这种能力被转移到也可应用于目标语音样本，而不管语料库方差如何。与广泛使用的最大均值差异（MMD）及其变体不同，该算法吸收了样本重构的思想来隐式对齐分布间隙，从而使DIDAN能够从语音频谱中学习情感判别特征和语料库不变特征。为了评估所提出的DIDAN，在广泛使用的语音情感语料库上进行了跨语料库SER实验。实验结果表明，该方法在处理跨语料库SER任务时优于许多现有方法.摘要：In this paper, we propose a novel deep transfer learning method called deep implicit distribution alignment networks (DIDAN) to deal with cross-corpus speech emotion recognition (SER) problem, in which the labeled training (source) and unlabeled testing (target) speech signals come from different corpora. Specifically, DIDAN first adopts a simple deep regression network consisting of a set of convolutional and fully connected layers to directly regress the source speech spectrums into the emotional labels such that the proposed DIDAN can own the emotion discriminative ability. Then, such ability is transferred to be also applicable to the target speech samples regardless of corpus variance by resorting to a well-designed regularization term called implicit distribution alignment (IDA). Unlike widely-used maximum mean discrepancy (MMD) and its variants, the proposed IDA absorbs the idea of sample reconstruction to implicitly align the distribution gap, which enables DIDAN to learn both emotion discriminative and corpus invariant features from speech spectrums. To evaluate the proposed DIDAN, extensive cross-corpus SER experiments on widely-used speech emotion corpora are carried out. Experimental results show that the proposed DIDAN can outperform lots of recent state-of-the-art methods in coping with the cross-corpus SER tasks.【3】 Lip-to-Speech Synthesis in the Wild with Multi-task Learning
标题：基于多任务学习的野外唇语合成
链接：https://arxiv.org/abs/2302.08841
作者：Minsu Kim,Joanna Hong,Yong Man Ro机构：School of Electrical Engineering, KAIST, South Korea备注：Accepted at ICASSP 2023. Demo available: this https URL摘要：最近的研究表明，唇到语音合成的目标是重建语音从视觉信息单独的令人印象深刻的性能。但由于指导模型推断正确内容的监督不足，他们一直苦于在野外合成准确的语音。与以往的方法不同，本文提出了一种功能强大的Lip2Speech方法，即使在野外环境下，也能从输入的嘴唇运动中重构出内容正确的语音。为此，我们设计了多任务学习，使用多模态监督指导模型，即：文本和音频，以补充声学特征重建损失的词表示不足。因此，所提出的框架带来了合成包含多个说话者的正确内容的语音与不受约束的句子的优点。我们使用LRS2、LRS3和LRW数据集验证了所提方法的有效性。摘要：Recent studies have shown impressive performance in Lip-to-speech synthesis that aims to reconstruct speech from visual information alone. However, they have been suffering from synthesizing accurate speech in the wild, due to insufficient supervision for guiding the model to infer the correct content. Distinct from the previous methods, in this paper, we develop a powerful Lip2Speech method that can reconstruct speech with correct contents from the input lip movements, even in a wild environment. To this end, we design multi-task learning that guides the model using multimodal supervision, i.e., text and audio, to complement the insufficient word representations of acoustic feature reconstruction loss. Thus, the proposed framework brings the advantage of synthesizing speech containing the right content of multiple speakers with unconstrained sentences. We verify the effectiveness of the proposed method using LRS2, LRS3, and LRW datasets.【4】 Build a training interface to install the bat's echolocation skills in humans
标题：建立一个训练界面，将蝙蝠的回声定位技能安装在人类身上
链接：https://arxiv.org/abs/2302.08794
作者：Miyoko Tsumaki,Yu Teshima,Takao Tsuchiya,Kaoru Ashihara,Kohta I. Kobayasi,Shizuko Hiryu机构：Graduate School of Life and Medical Sciences, Doshisha University, Japan, Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and, Technology (AIST), Japan摘要：蝙蝠使用一种叫做回声定位的复杂的超声波传感方法来识别环境。最近，据报道，没有回声定位经验的有视力的人类参与者可以通过训练收听回声来提高他们感知各种环境的空间布局的能力（Norman等人，2021年）。在本研究中，我们发展了一套新的使用眼动追踪器的人体回声定位训练系统。利用波动方程时域有限差分法，模拟了回声定位蝙蝠取食策略激发的连续向下线性调频脉冲的双耳回声。虚拟回波响应于监视器上他或她的眼球运动而呈现给视力正常的对象。从眼睛注视到回声出现的潜伏期不是听觉延迟。在一个初步实验中，参与者被要求识别隐藏目标的形状，他们发现参与者的目光集中在监视器上隐藏目标的边缘。我们将进行一个心理声学实验来检验人类在形状识别任务中回声定位的学习过程，这将导致福利工程领域的设备开发。摘要：Bats use a sophisticated ultrasonic sensing method called echolocation to recognize the environment. Recently, it has been reported that sighted human participants with no prior experience in echolocation can improve their ability to perceive the spatial layout of various environments through training to listen to echoes (Norman, et al., 2021). In this study, we developed the new training system for human echolocation using the eye-tracker. Binaural echoes of consecutive downward linear FM pulses that were inspired by feeding strategies of echolocating bats were simulated using the wave equation finite difference time domain method. The virtual echoes were presented to the sighted subject in response to his or her eye movements on the monitor. The latency from eye gazing to the echo presentation wasn't audible delay to perceive. In a preliminary experiment in which the participants were asked to identify the shapes of the hidden target, the participants were found to concentrate their gaze on the edges of the hidden target on the monitor. We will conduct a psycho-acoustical experiment to examine the learning process of human echolocation in a shape-identification task, which will lead to device development in the field of welfare engineering.【5】 Gaussian-smoothed Imbalance Data Improves Speech Emotion Recognition
标题：高斯平滑的不平衡数据改善了语音情感识别
链接：https://arxiv.org/abs/2302.08650
作者：Xuefeng Liang,Hexin Jiang,Wenxin Xu,Ying Zhou机构：⋆ School of Artificial Intelligence, Xidian University, China, †Pazhou Lab, Huangpu, China摘要：在语音情感识别任务中，模型从数据集中学习情感表示。我们发现IEMOCAP数据集中的数据分布非常不平衡，这可能会影响模型学习更好的表示。针对这一问题，提出了一种新的成对情感数据分布平滑（PDDS）方法。PDDS算法考虑到情感数据在现实中的分布应该是平滑的，因此对情感对进行高斯平滑，构造出分布更平滑的新训练集。所需的新数据使用混合扩充来补充。由于PDDS与模型和模态无关，因此使用IEMOCAP数据集上的三个SOTA模型对其进行评价。实验结果表明，该模型在WA和UA方面分别提高了0.2% ~ 4.8%和1.5% ~ 5.9%。此外，一项消融研究表明，PDDS的关键优势是合理的数据分布，而不是简单的数据扩增。摘要：In speech emotion recognition tasks, models learn emotional representations from datasets. We find the data distribution in the IEMOCAP dataset is very imbalanced, which may harm models to learn a better representation. To address this issue, we propose a novel Pairwise-emotion Data Distribution Smoothing (PDDS) method. PDDS considers that the distribution of emotional data should be smooth in reality, then applies Gaussian smoothing to emotion-pairs for constructing a new training set with a smoother distribution. The required new data are complemented using the mixup augmentation. As PDDS is model and modality agnostic, it is evaluated with three SOTA models on the IEMOCAP dataset. The experimental results show that these models are improved by 0.2\% - 4.8\% and 1.5\% - 5.9\% in terms of WA and UA. In addition, an ablation study demonstrates that the key advantage of PDDS is the reasonable data distribution rather than a simple data augmentation.【6】 jazznet: A Dataset of Fundamental Piano Patterns for Music Audio Machine Learning Research
标题：Jazznet：用于音乐音频机器学习研究的钢琴基本模式数据集
链接：https://arxiv.org/abs/2302.08632
机构：Department of Electrical and Computer Engineering, University of Arizona, Tucson AZ, USA备注：To Appear at IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2023摘要：本文介绍了jazznet数据集，这是一个用于开发音乐信息检索（MIR）中的机器学习（ML）算法的基本爵士钢琴音乐模式数据集。该数据集包含162520个标记的钢琴模式，包括和弦、琶音、音阶和和弦进行及其转位，导致超过26k小时的音频和95GB的总大小。本文解释了数据集的组成、创建和生成，并介绍了一个使用基于距离的模式结构（DBPS）方法的开源模式生成器，它允许研究人员通过定义音乐模式中音高之间的距离来轻松生成新的钢琴模式。我们证明了数据集可以帮助研究人员使用卷积递归神经网络（CRNN）和深度卷积神经网络对新模型进行基准测试，以完成具有挑战性的MIR任务。数据集和代码可通过以下方式获得：www.example.com网站。https://github.com/tosiron/jazznet.摘要：This paper introduces the jazznet Dataset, a dataset of fundamental jazz piano music patterns for developing machine learning (ML) algorithms in music information retrieval (MIR). The dataset contains 162520 labeled piano patterns, including chords, arpeggios, scales, and chord progressions with their inversions, resulting in more than 26k hours of audio and a total size of 95GB. The paper explains the dataset's composition, creation, and generation, and presents an open-source Pattern Generator using a method called Distance-Based Pattern Structures (DBPS), which allows researchers to easily generate new piano patterns simply by defining the distances between pitches within the musical patterns. We demonstrate that the dataset can help researchers benchmark new models for challenging MIR tasks, using a convolutional recurrent neural network (CRNN) and a deep convolutional neural network. The dataset and code are available via: https://github.com/tosiron/jazznet.【7】 Adaptive Axonal Delays in feedforward spiking neural networks for accurate spoken word recognition
标题：用于准确语音识别的前馈棘波神经网络中的自适应轴突延迟
链接：https://arxiv.org/abs/2302.08607
作者：Pengfei Sun,Ehsan Eqlimi,Yansong Chua,Paul Devos,Dick Botteldooren机构：WAVES Research Group, Ghent University, Belgium, China Nanhu Academy of Electronics and Information Technology, China备注：Accepted by ICASSP 2023摘要：脉冲神经网络（SNN）是建立准确、高效的自动语音识别系统的一条有前途的研究途径。近年来，音频-棘波编码和训练算法的发展使神经网络能够应用于实际任务。受生物启发的SNN使用稀疏异步事件进行通信。因此，尖峰定时对SNN的性能至关重要。在这方面，大多数的研究都集中在训练突触权重上，很少考虑事件传递的延迟，即轴突延迟。在这项工作中，我们考虑了一个可学习的轴突延迟上限的最大值，这可以适应根据轴突延迟分布在每一个网络层。实验结果表明，该方法在SHD数据集（92.45%）和NTIDIGITS数据集（95.09%）上取得了最好的分类效果.我们的工作说明了在复杂的时间结构任务中训练轴突延迟的潜力。摘要：Spiking neural networks (SNN) are a promising research avenue for building accurate and efficient automatic speech recognition systems. Recent advances in audio-to-spike encoding and training algorithms enable SNN to be applied in practical tasks. Biologically-inspired SNN communicates using sparse asynchronous events. Therefore, spike-timing is critical to SNN performance. In this aspect, most works focus on training synaptic weights and few have considered delays in event transmission, namely axonal delay. In this work, we consider a learnable axonal delay capped at a maximum value, which can be adapted according to the axonal delay distribution in each network layer. We show that our proposed method achieves the best classification results reported on the SHD dataset (92.45%) and NTIDIGITS dataset (95.09%). Our work illustrates the potential of training axonal delays for tasks with complex temporal structures.【8】 Improving Transformer-based Networks With Locality For Automatic Speaker Verification
标题：用于说话人自动确认的局部性改进Transformer网络
链接：https://arxiv.org/abs/2302.08639
作者：Mufan Sang,Yong Zhao,Gang Liu,John H. L. Hansen,Jian Wu机构：The University of Texas at Dallas, TX, USA, Microsoft Corporation, One Microsoft Way, Redmond, WA , USA备注：Accepted to ICASSP 2023摘要：最近，基于变换器的体系结构已经被探索用于说话人嵌入提取。虽然Transformer采用了自注意机制来有效地对令牌嵌入之间的全局交互进行建模，但它不足以捕获短范围的局部上下文，而这对于说话人信息的准确提取至关重要。在本研究中，我们在两个方向上使用局部性模型来增强Transformer。首先，我们通过在Conformer块中引入深度方向卷积和通道方向注意，提出了局部增强Conformer（LE-Confomer）。其次，我们将原本针对视觉任务所提出的Swin转换器应用到说话人嵌入网路中，提出了说话人Swin Transformer（SST）。我们在VoxCeleb数据集和大规模微软内部多语言（MS内部）数据集上评估了所提出的方法。所提出的模型在VoxCeleb 1测试集上实现了0.75%的EER，优于先前提出的基于Transformer的模型和基于CNN的模型，例如ResNet 34和ECAPA-TDNN。当在MS内部数据集上训练时，所提出的模型获得了有希望的结果，与Res 2Net 50模型相比，EER相对降低了14.6%。摘要：Recently, Transformer-based architectures have been explored for speaker embedding extraction. Although the Transformer employs the self-attention mechanism to efficiently model the global interaction between token embeddings, it is inadequate for capturing short-range local context, which is essential for the accurate extraction of speaker information. In this study, we enhance the Transformer with the locality modeling in two directions. First, we propose the Locality-Enhanced Conformer (LE-Confomer) by introducing depth-wise convolution and channel-wise attention into the Conformer blocks. Second, we present the Speaker Swin Transformer (SST) by adapting the Swin Transformer, originally proposed for vision tasks, into speaker embedding network. We evaluate the proposed approaches on the VoxCeleb datasets and a large-scale Microsoft internal multilingual (MS-internal) dataset. The proposed models achieve 0.75% EER on VoxCeleb 1 test set, outperforming the previously proposed Transformer-based models and CNN-based models, such as ResNet34 and ECAPA-TDNN. When trained on the MS-internal dataset, the proposed models achieve promising results with 14.6% relative reduction in EER over the Res2Net50 model.【9】 JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognition
标题：JEIT：用于语音识别的联合端到端模型和内部语言模型训练
链接：https://arxiv.org/abs/2302.08583
作者：Zhong Meng,Weiran Wang,Rohit Prabhavalkar,Tara N. Sainath,Tongzhou Chen,Ehsan Variani,Yu Zhang,Bo Li,Andrew Rosenberg,Bhuvana Ramabhadran摘要：提出了一种端到端（E2 E）模型和内部语言模型（ILM）联合训练方法JEIT，在E2 E训练过程中将大规模的不成对文本注入到ILM中，提高了生僻词语音识别率。对于JEIT，E2 E模型计算音频-抄本对的E2 E损失，而ILM估计非配对文本的交叉熵损失。训练E2 E模型以最小化E2 E和ILM损失的加权和。在JEIT过程中，ILM从不配对文本中吸收知识，而E2 E训练则起到正则化的作用。与ILM自适应方法不同，JEIT不需要单独的自适应步骤，并且避免了对ILM的Kullback-Leibler散度正则化的需要。我们还证明了模块化混合自回归转换器（MHAT）在JEIT框架中比HAT性能更好，并且在ILM自适应过程中比HAT鲁棒得多。为了突破不成对文本注入的限制，我们进一步提出了一种结合JEIT和JOIST训练（CJJT），它受益于模态匹配、编码器文本注入和ILM训练。JEIT和CJJT都可以促进更有效的LM融合。对于100 B个不成对的句子，JEIT/CJJT将稀有单词识别准确率提高了16.4%，超过了没有不成对文本的模型。摘要：We propose JEIT, a joint end-to-end (E2E) model and internal language model (ILM) training method to inject large-scale unpaired text into ILM during E2E training which improves rare-word speech recognition. With JEIT, the E2E model computes an E2E loss on audio-transcript pairs while its ILM estimates a cross-entropy loss on unpaired text. The E2E model is trained to minimize a weighted sum of E2E and ILM losses. During JEIT, ILM absorbs knowledge from unpaired text while the E2E training serves as regularization. Unlike ILM adaptation methods, JEIT does not require a separate adaptation step and avoids the need for Kullback-Leibler divergence regularization of ILM. We also show that modular hybrid autoregressive transducer (MHAT) performs better than HAT in the JEIT framework, and is much more robust than HAT during ILM adaptation. To push the limit of unpaired text injection, we further propose a combined JEIT and JOIST training (CJJT) that benefits from modality matching, encoder text injection and ILM training. Both JEIT and CJJT can foster a more effective LM fusion. With 100B unpaired sentences, JEIT/CJJT improves rare-word recognition accuracy by up to 16.4% over a model trained without unpaired text.【10】 Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmax
标题：使用可更换的内部LMS和剩余Softmax的可适应的端到端ASR模型
链接：https://arxiv.org/abs/2302.08579
作者：Keqi Deng,Philip C. Woodland机构：Department of Engineering, University of Cambridge, Trumpington St., Cambridge, UK.备注：Accepted by ICASSP2023摘要：端到端（E2 E）自动语音识别（ASR）隐式学习成对音频记录训练数据的标记序列分布。然而，它仍然遭受从训练到测试的领域转移，并且领域适应仍然具有挑战性。针对这一问题，设计了一种可替换内部语言模型（RILM）方法，使得E2 E ASR模型在解码阶段遇到域转移时，可以直接用目标域LM替换内部语言模型（LM）。此外，本文还提出了一种针对基于CTC的E2 E ASR模型的残差软最大（R-softmax）算法，使其在推理过程中无需重新训练即可适应目标域。实验表明，对于在LibriSpeech语料库上训练的E2 E ASR模型，所提方法在保持域内ASR结果的同时，对Switchboard数据和AESRC 2020语料库分别降低了2.6%和1.0%的绝对WER.摘要：End-to-end (E2E) automatic speech recognition (ASR) implicitly learns the token sequence distribution of paired audio-transcript training data. However, it still suffers from domain shifts from training to testing, and domain adaptation is still challenging. To alleviate this problem, this paper designs a replaceable internal language model (RILM) method, which makes it feasible to directly replace the internal language model (LM) of E2E ASR models with a target-domain LM in the decoding stage when a domain shift is encountered. Furthermore, this paper proposes a residual softmax (R-softmax) that is designed for CTC-based E2E ASR models to adapt to the target domain without re-training during inference. For E2E ASR models trained on the LibriSpeech corpus, experiments showed that the proposed methods gave a 2.6% absolute WER reduction on the Switchboard data and a 1.0% WER reduction on the AESRC2020 corpus while maintaining intra-domain ASR results.【11】 Speaker Change Detection for Transformer Transducer ASR
标题：TransformerASR中的扬声器更换检测
链接：https://arxiv.org/abs/2302.08549
作者：Jian Wu,Zhuo Chen,Min Hu,Xiong Xiao,Jinyu Li机构：Microsoft, One Microsoft Way, Redmond, WA, USA备注：5 pages, 1 figure, accepted by ICASSP 2023摘要：说话人变化检测（SCD）是一个重要的特征，它通过在说话人变化点处将单词序列分解成段落来提高来自自动语音识别（ASR）系统的识别单词的可读性。现有的SCD解决方案或者需要用于基于时间的决策和识别的字序列的附加集合，或者实现ASR和SCD之间的紧密集成，从而限制了两个任务的潜在最佳性能。为了解决这些问题，我们提出了一个新的SCD任务框架，其中一个额外的SCD模块建立在现有的Transformer换能器ASR（TT-ASR）网络之上。在该框架中探索了SCD网络的两个变体，其自然地估计每个词的说话人改变概率，同时允许ASR和SCD具有独立的优化方案以获得最佳性能。实验表明，与SCD和ASR联合基线相比，该方法在不降低ASR性能的情况下，显著提高了LibriCSS和Microsoft呼叫中心数据集的F1评分。摘要：Speaker change detection (SCD) is an important feature that improves the readability of the recognized words from an automatic speech recognition (ASR) system by breaking the word sequence into paragraphs at speaker change points. Existing SCD solutions either require additional ensemble for the time based decisions and recognized word sequences, or implement a tight integration between ASR and SCD, limiting the potential optimum performance for both tasks. To address these issues, we propose a novel framework for the SCD task, where an additional SCD module is built on top of an existing Transformer Transducer ASR (TT-ASR) network. Two variants of the SCD network are explored in this framework that naturally estimate speaker change probability for each word, while allowing the ASR and SCD to have independent optimization scheme for the best performance. Experiments show that our methods can significantly improve the F1 score on LibriCSS and Microsoft call center data sets without ASR degradation, compared with a joint SCD and ASR baseline.【1】 Improving Transformer-based Networks With Locality For Automatic Speaker Verification
标题：用于说话人自动确认的局部性改进Transformer网络
链接：https://arxiv.org/abs/2302.08639
*与cs.SD语音【8】为同一篇
作者：Mufan Sang,Yong Zhao,Gang Liu,John H. L. Hansen,Jian Wu机构：The University of Texas at Dallas, TX, USA, Microsoft Corporation, One Microsoft Way, Redmond, WA , USA备注：Accepted to ICASSP 2023摘要：最近，基于变换器的体系结构已经被探索用于说话人嵌入提取。虽然Transformer采用了自注意机制来有效地对令牌嵌入之间的全局交互进行建模，但它不足以捕获短范围的局部上下文，而这对于说话人信息的准确提取至关重要。在本研究中，我们在两个方向上使用局部性模型来增强Transformer。首先，我们通过在Conformer块中引入深度方向卷积和通道方向注意，提出了局部增强Conformer（LE-Confomer）。其次，我们将原本针对视觉任务所提出的Swin转换器应用到说话人嵌入网路中，提出了说话人Swin Transformer（SST）。我们在VoxCeleb数据集和大规模微软内部多语言（MS内部）数据集上评估了所提出的方法。所提出的模型在VoxCeleb 1测试集上实现了0.75%的EER，优于先前提出的基于Transformer的模型和基于CNN的模型，例如ResNet 34和ECAPA-TDNN。当在MS内部数据集上训练时，所提出的模型获得了有希望的结果，与Res 2Net 50模型相比，EER相对降低了14.6%。摘要：Recently, Transformer-based architectures have been explored for speaker embedding extraction. Although the Transformer employs the self-attention mechanism to efficiently model the global interaction between token embeddings, it is inadequate for capturing short-range local context, which is essential for the accurate extraction of speaker information. In this study, we enhance the Transformer with the locality modeling in two directions. First, we propose the Locality-Enhanced Conformer (LE-Confomer) by introducing depth-wise convolution and channel-wise attention into the Conformer blocks. Second, we present the Speaker Swin Transformer (SST) by adapting the Swin Transformer, originally proposed for vision tasks, into speaker embedding network. We evaluate the proposed approaches on the VoxCeleb datasets and a large-scale Microsoft internal multilingual (MS-internal) dataset. The proposed models achieve 0.75% EER on VoxCeleb 1 test set, outperforming the previously proposed Transformer-based models and CNN-based models, such as ResNet34 and ECAPA-TDNN. When trained on the MS-internal dataset, the proposed models achieve promising results with 14.6% relative reduction in EER over the Res2Net50 model.【2】 JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognition
标题：JEIT：用于语音识别的联合端到端模型和内部语言模型训练
链接：https://arxiv.org/abs/2302.08583
*与cs.SD语音【9】为同一篇
作者：Zhong Meng,Weiran Wang,Rohit Prabhavalkar,Tara N. Sainath,Tongzhou Chen,Ehsan Variani,Yu Zhang,Bo Li,Andrew Rosenberg,Bhuvana Ramabhadran摘要：We propose JEIT, a joint end-to-end (E2E) model and internal language model (ILM) training method to inject large-scale unpaired text into ILM during E2E training which improves rare-word speech recognition. With JEIT, the E2E model computes an E2E loss on audio-transcript pairs while its ILM estimates a cross-entropy loss on unpaired text. The E2E model is trained to minimize a weighted sum of E2E and ILM losses. During JEIT, ILM absorbs knowledge from unpaired text while the E2E training serves as regularization. Unlike ILM adaptation methods, JEIT does not require a separate adaptation step and avoids the need for Kullback-Leibler divergence regularization of ILM. We also show that modular hybrid autoregressive transducer (MHAT) performs better than HAT in the JEIT framework, and is much more robust than HAT during ILM adaptation. To push the limit of unpaired text injection, we further propose a combined JEIT and JOIST training (CJJT) that benefits from modality matching, encoder text injection and ILM training. Both JEIT and CJJT can foster a more effective LM fusion. With 100B unpaired sentences, JEIT/CJJT improves rare-word recognition accuracy by up to 16.4% over a model trained without unpaired text.【3】 Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmax
标题：使用可更换的内部LMS和剩余Softmax的可适应的端到端ASR模型
链接：https://arxiv.org/abs/2302.08579
*与cs.SD语音【10】为同一篇
作者：Keqi Deng,Philip C. Woodland机构：Department of Engineering, University of Cambridge, Trumpington St., Cambridge, UK.备注：Accepted by ICASSP2023摘要：端到端（E2 E）自动语音识别（ASR）隐式学习成对音频记录训练数据的标记序列分布。然而，它仍然遭受从训练到测试的领域转移，并且领域适应仍然具有挑战性。针对这一问题，设计了一种可替换内部语言模型（RILM）方法，使得E2 E ASR模型在解码阶段遇到域转移时，可以直接用目标域LM替换内部语言模型（LM）。此外，本文还提出了一种针对基于CTC的E2 E ASR模型的残差软最大（R-softmax）算法，使其在推理过程中无需重新训练即可适应目标域。实验表明，对于在LibriSpeech语料库上训练的E2 E ASR模型，所提方法在保持域内ASR结果的同时，对Switchboard数据和AESRC 2020语料库分别降低了2.6%和1.0%的绝对WER.摘要：End-to-end (E2E) automatic speech recognition (ASR) implicitly learns the token sequence distribution of paired audio-transcript training data. However, it still suffers from domain shifts from training to testing, and domain adaptation is still challenging. To alleviate this problem, this paper designs a replaceable internal language model (RILM) method, which makes it feasible to directly replace the internal language model (LM) of E2E ASR models with a target-domain LM in the decoding stage when a domain shift is encountered. Furthermore, this paper proposes a residual softmax (R-softmax) that is designed for CTC-based E2E ASR models to adapt to the target domain without re-training during inference. For E2E ASR models trained on the LibriSpeech corpus, experiments showed that the proposed methods gave a 2.6% absolute WER reduction on the Switchboard data and a 1.0% WER reduction on the AESRC2020 corpus while maintaining intra-domain ASR results.【4】 Speaker Change Detection for Transformer Transducer ASR
标题：TransformerASR中的扬声器更换检测
链接：https://arxiv.org/abs/2302.08549
*与cs.SD语音【11】为同一篇
作者：Jian Wu,Zhuo Chen,Min Hu,Xiong Xiao,Jinyu Li机构：Microsoft, One Microsoft Way, Redmond, WA, USA备注：5 pages, 1 figure, accepted by ICASSP 2023摘要：说话人变化检测（SCD）是一个重要的特征，它通过在说话人变化点处将单词序列分解成段落来提高来自自动语音识别（ASR）系统的识别单词的可读性。现有的SCD解决方案或者需要用于基于时间的决策和识别的字序列的附加集合，或者实现ASR和SCD之间的紧密集成，从而限制了两个任务的潜在最佳性能。为了解决这些问题，我们提出了一个新的SCD任务框架，其中一个额外的SCD模块建立在现有的Transformer换能器ASR（TT-ASR）网络之上。在该框架中探索了SCD网络的两个变体，其自然地估计每个词的说话人改变概率，同时允许ASR和SCD具有独立的优化方案以获得最佳性能。实验表明，与SCD和ASR联合基线相比，该方法在不降低ASR性能的情况下，显著提高了LibriCSS和Microsoft呼叫中心数据集的F1评分。摘要：Speaker change detection (SCD) is an important feature that improves the readability of the recognized words from an automatic speech recognition (ASR) system by breaking the word sequence into paragraphs at speaker change points. Existing SCD solutions either require additional ensemble for the time based decisions and recognized word sequences, or implement a tight integration between ASR and SCD, limiting the potential optimum performance for both tasks. To address these issues, we propose a novel framework for the SCD task, where an additional SCD module is built on top of an existing Transformer Transducer ASR (TT-ASR) network. Two variants of the SCD network are explored in this framework that naturally estimate speaker change probability for each word, while allowing the ASR and SCD to have independent optimization scheme for the best performance. Experiments show that our methods can significantly improve the F1 score on LibriCSS and Microsoft call center data sets without ASR degradation, compared with a joint SCD and ASR baseline.【5】 Handling the Alignment for Wake Word Detection: A Comparison Between Alignment-Based, Alignment-Free and Hybrid Approaches
标题：处理尾迹词检测的对齐：基于对齐、无对齐和混合方法的比较
链接：https://arxiv.org/abs/2302.08950
*与cs.SD语音【1】为同一篇
作者：Vinicius Ribeiro,Yiteng Huang,Yuan Shangguan,Zhaojun Yang,Li Wan,Ming Sun机构：Meta AI, Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：唤醒词检测存在于大多数智能家庭和便携式设备中。它为这些设备提供了在召唤时“唤醒”的能力，同时降低了电力和计算成本。本文着重于理解对齐在开发一个回答通用短语的唤醒词系统中的作用。我们讨论三种方法。第一种是基于对齐的，其中使用帧方式交叉熵来训练模型。第二种是无对齐的，其中使用CTC训练模型。第三种是我们提出的一种混合解决方案，即先用一小部分对齐的数据集训练模型，然后用一个相当大的未对齐的数据集调整模型。我们比较了这三种方法，并评估了不同对齐与未对齐比率对混合训练的影响。我们的结果表明，免对准系统对于目标操作点执行更好的基于对准的系统，并且利用一小部分数据（20%），我们可以训练符合我们的初始约束的模型。摘要：Wake word detection exists in most intelligent homes and portable devices. It offers these devices the ability to "wake up" when summoned at a low cost of power and computing. This paper focuses on understanding alignment's role in developing a wake-word system that answers a generic phrase. We discuss three approaches. The first is alignment-based, where the model is trained with frame-wise cross-entropy. The second is alignment-free, where the model is trained with CTC. The third, proposed by us, is a hybrid solution in which the model is trained with a small set of aligned data and then tuned with a sizeable unaligned dataset. We compare the three approaches and evaluate the impact of the different aligned-to-unaligned ratios for hybrid training. Our results show that the alignment-free system performs better alignment-based for the target operating point, and with a small fraction of the data (20%), we can train a model that complies with our initial constraints. 【6】 Deep Implicit Distribution Alignment Networks for Cross-Corpus Speech Emotion Recognition
标题：用于跨语料库语音情感识别的深度隐式分布对齐网络
链接：https://arxiv.org/abs/2302.08921
*与cs.SD语音【2】为同一篇
作者：Yan Zhao,Jincen Wang,Yuan Zong,Wenming Zheng,Hailun Lian,Li Zhao机构：Key Laboratory of Child Development and Learning Science of Ministry of Education, Southeast University, Nanjing , China摘要：本文提出了一种新的深度迁移学习方法--深度隐式分布对齐网络（DIDAN），用于处理带标记训练（源）语音信号和未标记测试（目标）语音信号来自不同语料库的跨语料库语音情感识别（SER）问题。具体地说，DIDAN首先采用一个由一组卷积和全连通层组成的简单深度回归网络，将源语音频谱直接回归为情感标签，从而使DIDAN具有情感辨别能力.然后，通过求助于被称为隐式分布对准（IDA）的良好设计的正则化项，这种能力被转移到也可应用于目标语音样本，而不管语料库方差如何。与广泛使用的最大均值差异（MMD）及其变体不同，该算法吸收了样本重构的思想来隐式对齐分布间隙，从而使DIDAN能够从语音频谱中学习情感判别特征和语料库不变特征。为了评估所提出的DIDAN，在广泛使用的语音情感语料库上进行了跨语料库SER实验。实验结果表明，该方法在处理跨语料库SER任务时优于许多现有方法.摘要：In this paper, we propose a novel deep transfer learning method called deep implicit distribution alignment networks (DIDAN) to deal with cross-corpus speech emotion recognition (SER) problem, in which the labeled training (source) and unlabeled testing (target) speech signals come from different corpora. Specifically, DIDAN first adopts a simple deep regression network consisting of a set of convolutional and fully connected layers to directly regress the source speech spectrums into the emotional labels such that the proposed DIDAN can own the emotion discriminative ability. Then, such ability is transferred to be also applicable to the target speech samples regardless of corpus variance by resorting to a well-designed regularization term called implicit distribution alignment (IDA). Unlike widely-used maximum mean discrepancy (MMD) and its variants, the proposed IDA absorbs the idea of sample reconstruction to implicitly align the distribution gap, which enables DIDAN to learn both emotion discriminative and corpus invariant features from speech spectrums. To evaluate the proposed DIDAN, extensive cross-corpus SER experiments on widely-used speech emotion corpora are carried out. Experimental results show that the proposed DIDAN can outperform lots of recent state-of-the-art methods in coping with the cross-corpus SER tasks.【7】 Lip-to-Speech Synthesis in the Wild with Multi-task Learning
标题：基于多任务学习的野外唇语合成
链接：https://arxiv.org/abs/2302.08841
*与cs.SD语音【3】为同一篇
作者：Minsu Kim,Joanna Hong,Yong Man Ro机构：School of Electrical Engineering, KAIST, South Korea备注：Accepted at ICASSP 2023. Demo available: this https URL摘要：最近的研究表明，唇到语音合成的目标是重建语音从视觉信息单独的令人印象深刻的性能。但由于指导模型推断正确内容的监督不足，他们一直苦于在野外合成准确的语音。与以往的方法不同，本文提出了一种功能强大的Lip2Speech方法，即使在野外环境下，也能从输入的嘴唇运动中重构出内容正确的语音。为此，我们设计了多任务学习，使用多模态监督指导模型，即：文本和音频，以补充声学特征重建损失的词表示不足。因此，所提出的框架带来了合成包含多个说话者的正确内容的语音与不受约束的句子的优点。我们使用LRS2、LRS3和LRW数据集验证了所提方法的有效性。摘要：Recent studies have shown impressive performance in Lip-to-speech synthesis that aims to reconstruct speech from visual information alone. However, they have been suffering from synthesizing accurate speech in the wild, due to insufficient supervision for guiding the model to infer the correct content. Distinct from the previous methods, in this paper, we develop a powerful Lip2Speech method that can reconstruct speech with correct contents from the input lip movements, even in a wild environment. To this end, we design multi-task learning that guides the model using multimodal supervision, i.e., text and audio, to complement the insufficient word representations of acoustic feature reconstruction loss. Thus, the proposed framework brings the advantage of synthesizing speech containing the right content of multiple speakers with unconstrained sentences. We verify the effectiveness of the proposed method using LRS2, LRS3, and LRW datasets.【8】 Build a training interface to install the bat's echolocation skills in humans
标题：建立一个训练界面，将蝙蝠的回声定位技能安装在人类身上
链接：https://arxiv.org/abs/2302.08794
*与cs.SD语音【4】为同一篇
作者：Miyoko Tsumaki,Yu Teshima,Takao Tsuchiya,Kaoru Ashihara,Kohta I. Kobayasi,Shizuko Hiryu机构：Graduate School of Life and Medical Sciences, Doshisha University, Japan, Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and, Technology (AIST), Japan摘要：蝙蝠使用一种叫做回声定位的复杂的超声波传感方法来识别环境。最近，据报道，没有回声定位经验的有视力的人类参与者可以通过训练收听回声来提高他们感知各种环境的空间布局的能力（Norman等人，2021年）。在本研究中，我们发展了一套新的使用眼动追踪器的人体回声定位训练系统。利用波动方程时域有限差分法，模拟了回声定位蝙蝠取食策略激发的连续向下线性调频脉冲的双耳回声。虚拟回波响应于监视器上他或她的眼球运动而呈现给视力正常的对象。从眼睛注视到回声出现的潜伏期不是听觉延迟。在一个初步实验中，参与者被要求识别隐藏目标的形状，他们发现参与者的目光集中在监视器上隐藏目标的边缘。我们将进行一个心理声学实验来检验人类在形状识别任务中回声定位的学习过程，这将导致福利工程领域的设备开发。摘要：Bats use a sophisticated ultrasonic sensing method called echolocation to recognize the environment. Recently, it has been reported that sighted human participants with no prior experience in echolocation can improve their ability to perceive the spatial layout of various environments through training to listen to echoes (Norman, et al., 2021). In this study, we developed the new training system for human echolocation using the eye-tracker. Binaural echoes of consecutive downward linear FM pulses that were inspired by feeding strategies of echolocating bats were simulated using the wave equation finite difference time domain method. The virtual echoes were presented to the sighted subject in response to his or her eye movements on the monitor. The latency from eye gazing to the echo presentation wasn't audible delay to perceive. In a preliminary experiment in which the participants were asked to identify the shapes of the hidden target, the participants were found to concentrate their gaze on the edges of the hidden target on the monitor. We will conduct a psycho-acoustical experiment to examine the learning process of human echolocation in a shape-identification task, which will lead to device development in the field of welfare engineering.【9】 Gaussian-smoothed Imbalance Data Improves Speech Emotion Recognition
标题：高斯平滑的不平衡数据改善了语音情感识别
链接：https://arxiv.org/abs/2302.08650
*与cs.SD语音【5】为同一篇
作者：Xuefeng Liang,Hexin Jiang,Wenxin Xu,Ying Zhou机构：⋆ School of Artificial Intelligence, Xidian University, China, †Pazhou Lab, Huangpu, China摘要：在语音情感识别任务中，模型从数据集中学习情感表示。我们发现IEMOCAP数据集中的数据分布非常不平衡，这可能会影响模型学习更好的表示。针对这一问题，提出了一种新的成对情感数据分布平滑（PDDS）方法。PDDS算法考虑到情感数据在现实中的分布应该是平滑的，因此对情感对进行高斯平滑，构造出分布更平滑的新训练集。所需的新数据使用混合扩充来补充。由于PDDS与模型和模态无关，因此使用IEMOCAP数据集上的三个SOTA模型对其进行评价。实验结果表明，该模型在WA和UA方面分别提高了0.2% ~ 4.8%和1.5% ~ 5.9%。此外，一项消融研究表明，PDDS的关键优势是合理的数据分布，而不是简单的数据扩增。摘要：In speech emotion recognition tasks, models learn emotional representations from datasets. We find the data distribution in the IEMOCAP dataset is very imbalanced, which may harm models to learn a better representation. To address this issue, we propose a novel Pairwise-emotion Data Distribution Smoothing (PDDS) method. PDDS considers that the distribution of emotional data should be smooth in reality, then applies Gaussian smoothing to emotion-pairs for constructing a new training set with a smoother distribution. The required new data are complemented using the mixup augmentation. As PDDS is model and modality agnostic, it is evaluated with three SOTA models on the IEMOCAP dataset. The experimental results show that these models are improved by 0.2\% - 4.8\% and 1.5\% - 5.9\% in terms of WA and UA. In addition, an ablation study demonstrates that the key advantage of PDDS is the reasonable data distribution rather than a simple data augmentation.【10】 jazznet: A Dataset of Fundamental Piano Patterns for Music Audio Machine Learning Research
标题：Jazznet：用于音乐音频机器学习研究的钢琴基本模式数据集
链接：https://arxiv.org/abs/2302.08632
*与cs.SD语音【6】为同一篇
机构：Department of Electrical and Computer Engineering, University of Arizona, Tucson AZ, USA备注：To Appear at IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2023摘要：本文介绍了jazznet数据集，这是一个用于开发音乐信息检索（MIR）中的机器学习（ML）算法的基本爵士钢琴音乐模式数据集。该数据集包含162520个标记的钢琴模式，包括和弦、琶音、音阶和和弦进行及其转位，导致超过26k小时的音频和95GB的总大小。本文解释了数据集的组成、创建和生成，并介绍了一个使用基于距离的模式结构（DBPS）方法的开源模式生成器，它允许研究人员通过定义音乐模式中音高之间的距离来轻松生成新的钢琴模式。我们证明了数据集可以帮助研究人员使用卷积递归神经网络（CRNN）和深度卷积神经网络对新模型进行基准测试，以完成具有挑战性的MIR任务。数据集和代码可通过以下方式获得：www.example.com网站。https://github.com/tosiron/jazznet.摘要：This paper introduces the jazznet Dataset, a dataset of fundamental jazz piano music patterns for developing machine learning (ML) algorithms in music information retrieval (MIR). The dataset contains 162520 labeled piano patterns, including chords, arpeggios, scales, and chord progressions with their inversions, resulting in more than 26k hours of audio and a total size of 95GB. The paper explains the dataset's composition, creation, and generation, and presents an open-source Pattern Generator using a method called Distance-Based Pattern Structures (DBPS), which allows researchers to easily generate new piano patterns simply by defining the distances between pitches within the musical patterns. We demonstrate that the dataset can help researchers benchmark new models for challenging MIR tasks, using a convolutional recurrent neural network (CRNN) and a deep convolutional neural network. The dataset and code are available via: https://github.com/tosiron/jazznet.【11】 Adaptive Axonal Delays in feedforward spiking neural networks for accurate spoken word recognition
标题：用于准确语音识别的前馈棘波神经网络中的自适应轴突延迟
链接：https://arxiv.org/abs/2302.08607
*与cs.SD语音【7】为同一篇
作者：Pengfei Sun,Ehsan Eqlimi,Yansong Chua,Paul Devos,Dick Botteldooren机构：WAVES Research Group, Ghent University, Belgium, China Nanhu Academy of Electronics and Information Technology, China备注：Accepted by ICASSP 2023摘要：脉冲神经网络（SNN）是建立准确、高效的自动语音识别系统的一条有前途的研究途径。近年来，音频-棘波编码和训练算法的发展使神经网络能够应用于实际任务。受生物启发的SNN使用稀疏异步事件进行通信。因此，尖峰定时对SNN的性能至关重要。在这方面，大多数的研究都集中在训练突触权重上，很少考虑事件传递的延迟，即轴突延迟。在这项工作中，我们考虑了一个可学习的轴突延迟上限的最大值，这可以适应根据轴突延迟分布在每一个网络层。实验结果表明，该方法在SHD数据集（92.45%）和NTIDIGITS数据集（95.09%）上取得了最好的分类效果.我们的工作说明了在复杂的时间结构任务中训练轴突延迟的潜力。摘要：Spiking neural networks (SNN) are a promising research avenue for building accurate and efficient automatic speech recognition systems. Recent advances in audio-to-spike encoding and training algorithms enable SNN to be applied in practical tasks. Biologically-inspired SNN communicates using sparse asynchronous events. Therefore, spike-timing is critical to SNN performance. In this aspect, most works focus on training synaptic weights and few have considered delays in event transmission, namely axonal delay. In this work, we consider a learnable axonal delay capped at a maximum value, which can be adapted according to the axonal delay distribution in each network layer. We show that our proposed method achieves the best classification results reported on the SHD dataset (92.45%) and NTIDIGITS dataset (95.09%). Our work illustrates the potential of training axonal delays for tasks with complex temporal structures.机器翻译，仅供参考
每日语音速递