今天跟大家分享一篇语音相关的论文合集：cs.SD语音9篇，eess.AS音频处理12篇。
【1】 Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
标题：让声音和视觉提示变得重要：CH-SIMS v2.0数据集和AV-Mixup一致性模块
链接：https://arxiv.org/abs/2209.02604
作者：Yihe Liu,Ziqi Yuan,Huisheng Mao,Zhiyun Liang,Wanqiuyue Yang,Yuanzhe Qiu,Tie Cheng,Xiaoteng Li,Hua Xu,Kai Gao机构：WANQIUYUE, YANG, State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University, China,School of Information Science and Engineering, Hebei University of Science and Tech-备注：16pages, 7 figures, accepted by ICMI 2022摘要：多模态情感分析（Multimodal Sentiment Analysis，MSA）是一个新兴的研究领域，其主要目的是将听觉和视觉模态结合起来，以提高基于文本的情感分析的效率。然而，现有的研究发现，听觉和视觉模态的贡献远远小于语篇模态，被称为语篇主导。在这种情况下，在这项工作中，我们强调使非语言线索的MSA任务的重要性。首先，从资源角度出发，我们提出了CH-SIMS v2.0数据集，它是CH-SIMS的扩展和增强。与原始数据集相比，CH-SIMS v2.0的大小增加了一倍，另外增加了2121个具有单峰和多峰标注的细化视频片段，并收集了10161个具有丰富的听觉和视觉情感承载上下文的未标记原始视频片段，以突出非语言线索进行情感预测。其次，从模型角度出发，借鉴CH-SIMS v2. 0的单峰标注和无监督数据，提出了声视混合一致性（AV-MC）框架。所设计的模态混合模块可以被看作是一个扩展，它混合了来自不同视频的声学和视觉模态。通过将未观察到的多模态语境与文本一起提取，该模型可以学习感知不同的非言语语境，从而进行情感预测。我们的评估表明，CH-SIMS v2.0和AV-MC框架都支持进一步的研究，以发现承载情感的声音和视觉线索，并为现实世界场景中可解释的端到端HCI应用铺平道路。摘要：Multimodal sentiment analysis (MSA), which supposes to improve text-based sentiment analysis with associated acoustic and visual modalities, is an emerging research area due to its potential applications in Human-Computer Interaction (HCI). However, the existing researches observe that the acoustic and visual modalities contribute much less than the textual modality, termed as text-predominant. Under such circumstances, in this work, we emphasize making non-verbal cues matter for the MSA task. Firstly, from the resource perspective, we present the CH-SIMS v2.0 dataset, an extension and enhancement of the CH-SIMS. Compared with the original dataset, the CH-SIMS v2.0 doubles its size with another 2121 refined video segments with both unimodal and multimodal annotations and collects 10161 unlabelled raw video segments with rich acoustic and visual emotion-bearing context to highlight non-verbal cues for sentiment prediction. Secondly, from the model perspective, benefiting from the unimodal annotations and the unsupervised data in the CH-SIMS v2.0, the Acoustic Visual Mixup Consistent (AV-MC) framework is proposed. The designed modality mixup module can be regarded as an augmentation, which mixes the acoustic and visual modalities from different videos. Through drawing unobserved multimodal context along with the text, the model can learn to be aware of different non-verbal contexts for sentiment prediction. Our evaluations demonstrate that both CH-SIMS v2.0 and AV-MC framework enables further research for discovering emotion-bearing acoustic and visual cues and paves the path to interpretable end-to-end HCI applications for real-world scenarios.
【2】 Distilling the Knowledge of BERT for CTC-based ASR
标题：基于CTC的ASR中BERT知识的提取
链接：https://arxiv.org/abs/2209.02030
作者：Hayato Futami,Hirofumi Inaguma,Masato Mimura,Shinsuke Sakai,Tatsuya Kawahara机构：Graduate School of Informatics, Kyoto University, Sakyo-ku, Kyoto, Japan摘要：基于连接主义时态分类（CTC）的模型在自动语音识别（ASR）中具有快速推理的优点。浅融合、重评分等语言模型集成方法可以充分利用文本语料库中的知识，提高基于CTC的ASR的识别准确率。然而，它们显著地减慢了CTC的推断。在本研究中，我们提出了提取BERT知识用于基于CTC的ASR，扩展了我们先前的基于注意力的ASR研究。基于CTC的ASR在训练阶段学习BERT知识，在测试阶段不使用BERT知识，保持了CTC的快速推理能力。与基于注意力的模型不同，基于CTC的模型进行帧级预测，因此它们需要与BERT的令牌级预测对齐以进行蒸馏。我们建议通过计算最合理的CTC路径来获得比对。在自发日语语料库（CSJ）和TED-LIUM2上的实验结果表明，该方法在不影响推理速度的前提下，提高了基于CTC的ASR的性能。摘要：Connectionist temporal classification (CTC) -based models are attractive because of their fast inference in automatic speech recognition (ASR). Language model (LM) integration approaches such as shallow fusion and rescoring can improve the recognition accuracy of CTC-based ASR by taking advantage of the knowledge in text corpora. However, they significantly slow down the inference of CTC. In this study, we propose to distill the knowledge of BERT for CTC-based ASR, extending our previous study for attention-based ASR. CTC-based ASR learns the knowledge of BERT during training and does not use BERT during testing, which maintains the fast inference of CTC. Different from attention-based models, CTC-based models make frame-level predictions, so they need to be aligned with token-level predictions of BERT for distillation. We propose to obtain alignments by calculating the most plausible CTC paths. Experimental evaluations on the Corpus of Spontaneous Japanese (CSJ) and TED-LIUM2 show that our method improves the performance of CTC-based ASR without the cost of inference speed.
【3】 Bridging Music and Text with Crowdsourced Music Comments: A Sequence-to-Sequence Framework for Thematic Music Comments Generation
标题：用众包音乐评论连接音乐和文本：一种主题音乐评论生成的序列到序列框架
链接：https://arxiv.org/abs/2209.01996
作者：Peining Zhang,Junliang Guo,Linli Xu,Mu You,Junming Yin机构：Received: date Accepted: date摘要：我们考虑自动生成音乐的文本描述的新颖任务。与其他成熟的文本生成任务（如图像字幕）相比，缺乏配对良好的音乐和文本数据集使其成为一个更具挑战性的任务。本文利用众包音乐评论构建新的数据集，提出一种基于序列到序列的音乐文本描述生成模型。更具体地说，我们使用膨胀卷积层作为编码器的基本组件，并使用基于记忆的递归神经网络作为解码器。为了提高生成文本的真实性和主题性，我们进一步提出了一个新的主题鉴别器和一个新的主题评价器来微调模型。为了衡量生成文本的质量，我们还提出了两个新的评价指标，它们比传统的指标如BLEU更符合人类的评价。实验结果表明，该模型能够生成流畅、有意义的评论，同时包含了原音乐的主题和内容信息。摘要：We consider a novel task of automatically generating text descriptions of music. Compared with other well-established text generation tasks such as image caption, the scarcity of well-paired music and text datasets makes it a much more challenging task. In this paper, we exploit the crowd-sourced music comments to construct a new dataset and propose a sequence-to-sequence model to generate text descriptions of music. More concretely, we use the dilated convolutional layer as the basic component of the encoder and a memory based recurrent neural network as the decoder. To enhance the authenticity and thematicity of generated texts, we further propose to fine-tune the model with a discriminator as well as a novel topic evaluator. To measure the quality of generated texts, we also propose two new evaluation metrics, which are more aligned with human evaluation than traditional metrics such as BLEU. Experimental results verify that our model is capable of generating fluent and meaningful comments while containing thematic and content information of the original music.
【4】 Predict-and-Update Network: Audio-Visual Speech Recognition Inspired by Human Speech Perception
标题：预测与更新网络：受人类语音感知启发的视听语音识别
链接：https://arxiv.org/abs/2209.01768
作者：Jiadong Wang,Xinyuan Qian,Haizhou Li摘要：在人类语音感知中，音频和视频信号是相辅相成的，在语音识别中也是如此。就语音感知而言，视觉提示不如声学提示明显，但在复杂的声学环境中更鲁棒。如何有效地利用音频和视频信号之间的相互作用进行自动语音识别仍然是一个挑战。已经有研究以同步方式利用视觉信号作为音频输入的冗余或补充信息。对人类的研究表明，视觉信号会提前让听者准备好什么时候和用什么频率去听。我们提出一个预测更新网络（P&U网），来模拟这种视觉线索机制用于音频—视频语音识别（AVSR）。具体地说，我们首先基于视觉信号预测说出的单词的字符后验，即视觉嵌入。然后通过一个新颖的跨模态Conformer对音频信号进行视觉嵌入调节，该Conformer更新字符后验。通过大量的实验验证了视觉线索机制的有效性。在LRS2-BBC和LRS3-BBC数据集上，所提出的P&U网的性能优于现有的AVSR方法，在干净和噪声条件下，相对降低的误字率（WER）分别超过10%和40%.摘要：Audio and visual signals complement each other in human speech perception, so do they in speech recognition. The visual hint is less evident than the acoustic hint, but more robust in a complex acoustic environment, as far as speech perception is concerned. It remains a challenge how we effectively exploit the interaction between audio and visual signals for automatic speech recognition. There have been studies to exploit visual signals as redundant or complementary information to audio input in a synchronous manner. Human studies suggest that visual signal primes the listener in advance as to when and on which frequency to attend to. We propose a Predict-and-Update Network (P&U net), to simulate such a visual cueing mechanism for Audio-Visual Speech Recognition (AVSR). In particular, we first predict the character posteriors of the spoken words, i.e. the visual embedding, based on the visual signals. The audio signal is then conditioned on the visual embedding via a novel cross-modal Conformer, that updates the character posteriors. We validate the effectiveness of the visual cueing mechanism through extensive experiments. The proposed P&U net outperforms the state-of-the-art AVSR methods on both LRS2-BBC and LRS3-BBC datasets, with the relative reduced Word Error Rate (WER)s exceeding 10% and 40% under clean and noisy conditions, respectively.
【5】 Exploiting Pre-trained Feature Networks for Generative Adversarial Networks in Audio-domain Loop Generation
标题：基于预训练特征网络的生成性对抗性网络在音频域环路生成中的应用
链接：https://arxiv.org/abs/2209.01751
作者：Yen-Tung Yeh,Bo-Yu Chen,Yi-Hsuan Yang机构：Academia Sinica, National Taiwan University, Taiwan AI Labs备注：Accepted at ISMIR 2022摘要：生成式对抗网络（generative adversarial networks，GANs）已被广泛应用于音频生成的研究中，但GAN模型的训练存在不稳定、耗时和数据效率低等缺点.在改进GANs的训练过程的尝试中，投影GAN的思想作为基于GAN的图像生成的有效解决方案而出现，在不同的图像应用中建立了最先进的技术。其核心思想是使用预先训练好的分类器约束鉴别器的特征空间，以稳定和改进GAN训练。本文通过评估基于StyleGAN2的音频域循环生成模型在鉴别器中使用和不使用预先训练的特征空间的情况下的性能，研究了投影GAN是否可以类似地改善音频生成。此外，我们还比较了使用通用分类器和特定领域分类器作为预训练音频分类器的性能。通过对鼓循环和合成器循环生成的实验，我们表明通用音频分类器工作得更好，而使用投影GAN，我们的循环生成模型可以在不降低性能的情况下以大约5倍的速度收敛。摘要：While generative adversarial networks (GANs) have been widely used in research on audio generation, the training of a GAN model is known to be unstable, time consuming, and data inefficient. Among the attempts to ameliorate the training process of GANs, the idea of Projected GAN emerges as an effective solution for GAN-based image generation, establishing the state-of-the-art in different image applications. The core idea is to use a pre-trained classifier to constrain the feature space of the discriminator to stabilize and improve GAN training. This paper investigates whether Projected GAN can similarly improve audio generation, by evaluating the performance of a StyleGAN2-based audio-domain loop generation model with and without using a pre-trained feature space in the discriminator. Moreover, we compare the performance of using a general versus domain-specific classifier as the pre-trained audio classifier. With experiments on both drum loop and synth loop generation, we show that a general audio classifier works better, and that with Projected GAN our loop generation models can converge around 5 times faster without performance degradation.
【6】 Equivariant Self-Supervision for Musical Tempo Estimation
标题：用于音乐节奏估计的等变自监督算法
链接：https://arxiv.org/abs/2209.01478
备注：Accepted at ISMIR 2022摘要：近年来，由于自监督方法减少了对稀缺且昂贵的标记数据集的需求，因此成为表征学习的一种有前途的方法。对比方法是音频域中用于自监督的流行选择，并且通常通过迫使模型对于输入的一些变换是不变的来提供学习信号。然而，这些方法需要采取诸如负采样或某种形式的正则化等措施，以防止模型在平凡解上崩溃。在这项工作中，我们建议使用等方差作为自监督信号，以代替不变性，从而从未标记数据中学习音频节奏表示。我们推导出一个简单的损失函数，它可以防止网络在训练过程中崩溃在一个平凡的解上，而不需要任何形式的正则化或负采样。实验结果表明，该方法可以通过等变自监督学习有意义的表征，在多个基准测试上获得与监督方法相当的性能.作为一个额外的好处，我们的方法只需要适度的计算资源，因此仍然可以为广泛的研究团体所访问。摘要：Self-supervised methods have emerged as a promising avenue for representation learning in the recent years since they alleviate the need for labeled datasets, which are scarce and expensive to acquire. Contrastive methods are a popular choice for self-supervision in the audio domain, and typically provide a learning signal by forcing the model to be invariant to some transformations of the input. These methods, however, require measures such as negative sampling or some form of regularisation to be taken to prevent the model from collapsing on trivial solutions. In this work, instead of invariance, we propose to use equivariance as a self-supervision signal to learn audio tempo representations from unlabelled data. We derive a simple loss function that prevents the network from collapsing on a trivial solution during training, without requiring any form of regularisation or negative sampling. Our experiments show that it is possible to learn meaningful representations for tempo estimation by solely relying on equivariant self-supervision, achieving performance comparable with supervised methods on several benchmarks. As an added benefit, our method only requires moderate compute resources and therefore remains accessible to a wide research community.
【7】 Identify The Beehive Sound Using Deep Learning
标题：基于深度学习的蜂窝声音识别
链接：https://arxiv.org/abs/2209.01374
作者：Shah Jafor Sadeek Quaderi,Sadia Afrin Labonno,Sadia Mostafa,Shamim Akhter机构：Technology, University of Malaya, Kuala Lumpur, Malaysia, AISIP Lab, Department of Computer Science and Engineering, International, University of Business Agriculture and Technology, Dhaka, Bangladesh, Stamford University Bangladesh, Dhaka, Bangladesh摘要：花在消除环境中的沉闷发挥着重要作用。开花植物的生命周期包括授粉、受精、开花、种子形成、散布和萌发。蜜蜂为大约75%的开花植物授粉。环境污染、气候变化、自然景观破坏等，威胁着自然栖息地，从而不断减少蜜蜂的数量。因此，一些研究人员正在试图解决这个问题。将声学分类应用于蜂窝声音的记录可能是一种检测其中变化的方法。在本研究中，我们使用深度学习技术，即序列神经网络、卷积神经网络和递归神经网络，对记录的声音进行分类，将蜜蜂声音与非蜂窝噪声区分开来。此外，本文还对支持向量机、决策树、随机森林和朴素贝叶斯等非深度学习技术与深度学习技术进行了对比研究。还在组合的记录声音（25-75%噪声）上验证了这些技术。摘要：Flowers play an essential role in removing the duller from the environment. The life cycle of the flowering plants involves pollination, fertilization, flowering, seed-formation, dispersion, and germination. Honeybees pollinate approximately 75% of all flowering plants. Environmental pollution, climate change, natural landscape demolition, and so on, threaten the natural habitats, thus continuously reducing the number of honeybees. As a result, several researchers are attempting to resolve this issue. Applying acoustic classification to recordings of beehive sounds may be a way of detecting changes within them. In this research, we use deep learning techniques, namely Sequential Neural Network, Convolutional Neural Network, and Recurrent Neural Network, on the recorded sounds to classify bee sounds from the nonbeehive noises. In addition, we perform a comparative study among some popular non-deep learning techniques, namely Support Vector Machine, Decision Tree, Random Forest, and Na\"ive Bayes, with the deep learning techniques. The techniques are also verified on the combined recorded sounds (25-75% noises).
【8】 Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model
标题：基于交替拼写预测模型的生词语境识别
链接：https://arxiv.org/abs/2209.01250
作者：Jennifer Drexler Fox,Natalie Delworth摘要：随着ASR的使用变得越来越广泛，将一系列偏见术语与音频一起作为输入的上下文ASR最近引起了人们的兴趣。我们发布了与Earnings21数据集相配套的上下文偏倚列表，为这项任务创建了一个公共基准。我们使用WeNet工具包中预先训练好的端到端ASR模型，展示了该基准测试的基线结果。我们示出了应用于两种不同解码算法的浅融合上下文偏置的结果。我们的基线结果证实了这样的观察，即端到端模型尤其与在训练期间很少或从未看到的单词作斗争，并且现有的浅融合技术不足以解决这个问题。我们提出了一个替代拼写预测模型，与没有替代拼写的上下文偏差相比，该模型将稀有单词的召回率相对提高了34.7%，将超出词汇表的单词的召回率相对提高了97.2%。该模型在概念上类似于在先前工作中使用的模型，但是由于其不依赖于发音字典或现有的文本到语音系统，因此实现起来更简单。摘要：Contextual ASR, which takes a list of bias terms as input along with audio, has drawn recent interest as ASR use becomes more widespread. We are releasing contextual biasing lists to accompany the Earnings21 dataset, creating a public benchmark for this task. We present baseline results on this benchmark using a pretrained end-to-end ASR model from the WeNet toolkit. We show results for shallow fusion contextual biasing applied to two different decoding algorithms. Our baseline results confirm observations that end-to-end models struggle in particular with words that are rarely or never seen during training, and that existing shallow fusion techniques do not adequately address this problem. We propose an alternate spelling prediction model that improves recall of rare words by 34.7% relative and of out-of-vocabulary words by 97.2% relative, compared to contextual biasing without alternate spellings. This model is conceptually similar to ones used in prior work, but is simpler to implement as it does not rely on either a pronunciation dictionary or an existing text-to-speech system.
【9】 Sound Event Localization and Detection for Real Spatial Sound Scenes: Event-Independent Network and Data Augmentation Chains
标题：真实空间声音场景的声音事件定位与检测：事件无关的网络和数据增强链
链接：https://arxiv.org/abs/2209.01802
作者：Jinbo Hu,Yin Cao,Ming Wu,Qiuqiang Kong,Feiran Yang,Mark D. Plumbley,Jun Yang机构：Key Laboratory of Noise and Vibration Research, Institute of Acoustics, University of Chinese Academy of Sciences, Beijing, China, Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK备注：Submitted to DCASE 2022 Workshop. Code is available at this https URL arXiv admin note: substantial text overlap with arXiv:2203.10228摘要：复调声音事件定位与检测（SELD）旨在检测具有相应时间活动和空间位置的声音事件类型。在DCASE 2022任务3中，数据类型从计算生成的空间记录转换为真实声音场景的记录。我们提交给DCASE 2022任务3的系统是基于我们先前提出的事件无关网络V2（EINV2）和一种新颖的数据扩充方法。为了检测具有不同位置的相同类型的不同声音事件，我们的方法采用EINV2，结合了逐声道输出格式、排列不变训练和软参数共享。EINV2也被扩展为使用构象异构体结构来学习局部和全局模式。为了提高模型的泛化能力，我们使用了一种包含多个数据扩充链的数据扩充方法，这些数据扩充链是由多个不同的数据扩充操作的随机组合而成。为了缓解开发数据集中缺乏真实场景记录和声音事件不平衡的问题，我们利用FSD50K、AudioSet和TAU空间房间脉冲响应数据库（TAU-SRIR DB）生成用于训练的模拟数据集。我们详细介绍了Sony-TAu Realistic Spatial Soundscapes 2022（STARSS22）验证集的结果。实验结果表明，泛化能力和类间性能不平衡是两个主要挑战。我们在DCASE 2022挑战的任务3中评估了我们提出的方法，并在团队排名中获得了第二名。源代码已发布。摘要：Polyphonic sound event localization and detection (SELD) aims at detecting types of sound events with corresponding temporal activities and spatial locations. In DCASE 2022 Task 3, types of data transform from computationally generated spatial recordings to recordings of real-sound scenes. Our system submitted to the DCASE 2022 Task 3 is based on our previous proposed Event-Independent Network V2 (EINV2) and a novel data augmentation method. To detect different sound events of the same type with different locations, our method employs EINV2, combining a track-wise output format, permutation-invariant training, and soft parameter-sharing. EINV2 is also extended to use conformer structures to learn local and global patterns. To improve the generalization ability of the model, we use a data augmentation method containing several data augmentation chains, which are composed of random combinations of several different data augmentation operations. To mitigate the lack of real-scene recordings in the development dataset and the presence of sound events being unbalanced, we exploit FSD50K, AudioSet, and TAU Spatial Room Impulse Response Database (TAU-SRIR DB) to generate simulated datasets for training. We present results on the validation set of Sony-TAu Realistic Spatial Soundscapes 2022 (STARSS22) in detail. Experimental results indicate that the ability to generalize to different environments and unbalanced performance among different classes are two main challenges. We evaluate our proposed method in Task 3 of DCASE 2022 challenge and obtain the second rank in the teams ranking. Source code is released.
【1】 Investigation into Target Speaking Rate Adaptation for Voice Conversion
标题：用于语音转换的目标语速自适应研究
链接：https://arxiv.org/abs/2209.01978
作者：Michael Kuhlmann,Fritz Seebauer,Janek Ebbers,Petra Wagner,Reinhold Haeb-Umbach机构：Paderborn University, Germany, Bielefeld University, Germany备注：Accepted to INTERSPEECH 2022摘要：将语音信号的说话者和内容属性解缠成单独的潜在表示，随后用交换的说话者表示来解码内容是用于话音转换的流行方法，其可以用非并行且未标记的语音数据来训练。然而，先前的方法仅通过某种信息瓶颈或归一化隐式地执行解纠缠，其中通常难以在语音转换和内容重构之间找到良好的折衷。此外，以前的工作通常不考虑说话速率对目标说话者的适应，或者它们对数据或用例施加一些主要限制。因此，这项工作的贡献是双重的。首先，我们采用了一种显式的和完全无监督的解纠缠方法，该方法以前仅用于表示学习，并表明它允许获得优异的语音转换和内容重构。第二，我们研究简单和通用的方法，以线性地调整语音信号的长度，从而调整讲话速率，以适应目标说话人，并且示出所提出的调整允许增加相对于目标说话人的讲话速率相似性。摘要：Disentangling speaker and content attributes of a speech signal into separate latent representations followed by decoding the content with an exchanged speaker representation is a popular approach for voice conversion, which can be trained with non-parallel and unlabeled speech data. However, previous approaches perform disentanglement only implicitly via some sort of information bottleneck or normalization, where it is usually hard to find a good trade-off between voice conversion and content reconstruction. Further, previous works usually do not consider an adaptation of the speaking rate to the target speaker or they put some major restrictions to the data or use case. Therefore, the contribution of this work is two-fold. First, we employ an explicit and fully unsupervised disentanglement approach, which has previously only been used for representation learning, and show that it allows to obtain both superior voice conversion and content reconstruction. Second, we investigate simple and generic approaches to linearly adapt the length of a speech signal, and hence the speaking rate, to a target speaker and show that the proposed adaptation allows to increase the speaking rate similarity with respect to the target speaker.
【2】 Sound Event Localization and Detection for Real Spatial Sound Scenes: Event-Independent Network and Data Augmentation Chains
标题：真实空间声音场景的声音事件定位与检测：事件无关的网络和数据增强链
链接：https://arxiv.org/abs/2209.01802
* 与cs.SD语音【9】为同一篇
作者：Jinbo Hu,Yin Cao,Ming Wu,Qiuqiang Kong,Feiran Yang,Mark D. Plumbley,Jun Yang机构：Key Laboratory of Noise and Vibration Research, Institute of Acoustics, University of Chinese Academy of Sciences, Beijing, China, Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK备注：Submitted to DCASE 2022 Workshop. Code is available at this https URL arXiv admin note: substantial text overlap with arXiv:2203.10228摘要：复调声音事件定位与检测（SELD）旨在检测具有相应时间活动和空间位置的声音事件类型。在DCASE 2022任务3中，数据类型从计算生成的空间记录转换为真实声音场景的记录。我们提交给DCASE 2022任务3的系统是基于我们先前提出的事件无关网络V2（EINV2）和一种新颖的数据扩充方法。为了检测具有不同位置的相同类型的不同声音事件，我们的方法采用EINV2，结合了逐声道输出格式、排列不变训练和软参数共享。EINV2也被扩展为使用构象异构体结构来学习局部和全局模式。为了提高模型的泛化能力，我们使用了一种包含多个数据扩充链的数据扩充方法，这些数据扩充链是由多个不同的数据扩充操作的随机组合而成。为了缓解开发数据集中缺乏真实场景记录和声音事件不平衡的问题，我们利用FSD50K、AudioSet和TAU空间房间脉冲响应数据库（TAU-SRIR DB）生成用于训练的模拟数据集。我们详细介绍了Sony-TAu Realistic Spatial Soundscapes 2022（STARSS22）验证集的结果。实验结果表明，泛化能力和类间性能不平衡是两个主要挑战。我们在DCASE 2022挑战的任务3中评估了我们提出的方法，并在团队排名中获得了第二名。源代码已发布。摘要：Polyphonic sound event localization and detection (SELD) aims at detecting types of sound events with corresponding temporal activities and spatial locations. In DCASE 2022 Task 3, types of data transform from computationally generated spatial recordings to recordings of real-sound scenes. Our system submitted to the DCASE 2022 Task 3 is based on our previous proposed Event-Independent Network V2 (EINV2) and a novel data augmentation method. To detect different sound events of the same type with different locations, our method employs EINV2, combining a track-wise output format, permutation-invariant training, and soft parameter-sharing. EINV2 is also extended to use conformer structures to learn local and global patterns. To improve the generalization ability of the model, we use a data augmentation method containing several data augmentation chains, which are composed of random combinations of several different data augmentation operations. To mitigate the lack of real-scene recordings in the development dataset and the presence of sound events being unbalanced, we exploit FSD50K, AudioSet, and TAU Spatial Room Impulse Response Database (TAU-SRIR DB) to generate simulated datasets for training. We present results on the validation set of Sony-TAu Realistic Spatial Soundscapes 2022 (STARSS22) in detail. Experimental results indicate that the ability to generalize to different environments and unbalanced performance among different classes are two main challenges. We evaluate our proposed method in Task 3 of DCASE 2022 challenge and obtain the second rank in the teams ranking. Source code is released.
【3】 Movement Detection of Tongue and Related Body Parts Using IR-UWB Radar
标题：基于IR-UWB雷达的舌及相关身体部位运动检测
链接：https://arxiv.org/abs/2209.01762
作者：Sunghwa Lee,Younghoon Shin机构：School of Integrated Technology, Yonsei University, Incheon, Korea, Robotics Lab, Hyundai Motor Company, Uiwang, Korea, ∗ Corresponding author备注：Submitted to the 13th International Conference on ICT Convergence (ICTC)摘要：由于脉冲无线电超宽带（IR-UWB）雷达能够高精度地探测目标、穿过遮挡材料并且无接触地工作，因此对于构建无声语音接口（非基于音频的语音通信设备）而言，它是一种有吸引力的硬件解决方案。由于舌头运动与发音密切相关，因此检测舌头运动对于开发无声语音界面至关重要。在这项研究中，我们尝试使用IR-UWB雷达（其天线指向受试者的下巴）来分类看不见的舌头及其相关身体部位的静止和运动状态。利用所提出的特征提取算法和高斯混合模型—隐马尔可夫模型，我们对四个个体参与者的两种状态的不可见舌进行了分类，最低准确率为90%.摘要：Because an impulse radio ultra-wideband (IR-UWB) radar can detect targets with high accuracy, work through occluding materials, and operate without contact, it is an attractive hardware solution for building silent speech interfaces, which are non-audio-based speech communication devices. As tongue movement is strongly engaged in pronunciation, detecting its movement is crucial for developing silent speech interfaces. In this study, we attempted to classify the motionless and moving states of an invisible tongue and its related body parts using an IR-UWB radar whose antennas were pointed toward the participant's chin. Using the proposed feature extraction algorithm and a Gaussian mixture model - hidden Markov model, we classified two states of the invisible tongue of four individual participants with a minimum accuracy of 90%.
【4】 Time-domain speech super-resolution with GAN based modeling for telephony speaker verification
标题：用于电话说话人确认的基于GAN的时域语音超分辨率建模
链接：https://arxiv.org/abs/2209.01702
作者：Saurabh Kataria,Jesús Villalba,Laureano Moro-Velázquez,Piotr Żelasko,Najim Dehak机构：All authors are associated with the Department of Electrical and ComputerEngineering and Center for Language and Speech Processing (CLSP), JohnsHopkins University备注：Submit to IEEE/ACM Transactions on Audio, Speech, and Language Processing摘要：自动说话人确认（ASV）技术已经在虚拟助手中变得普遍。然而，当训练域和测试域之间不匹配时，其性能受到影响。混合带宽培训，即：汇集来自两个域的训练数据是用于开发对窄带和宽带域都起作用的通用模型的优选选择。我们建议通过执行窄带信号的神经上采样（也称为带宽扩展）来补充该技术。我们的主要目标是发现和分析高性能的时域生成对抗网络（GAN）为基础的模型，以改善我们的下游国家的最先进的ASV系统。我们选择GANs是因为它们（1）对于学习条件分布是强大的，并且（2）允许在具有数据扩充的下游任务（ASV）的训练期间作为预处理器灵活地使用插件。现有工作主要集中在特征域带宽扩展和有限的实验设置。我们通过1）使用时域扩展模型，2）报告三个真实测试集的结果，2）扩展训练数据，以及3）设计新的测试时间方案来解决这些限制。我们比较了有监督的（条件GAN）和无监督的GANs（CycleGAN），并分别证明了在等错误率方面的平均相对改进为8.6%和7.7%。为了进一步分析，我们研究了频谱图视觉质量、音频感知质量、t-SNE嵌入和ASV分数分布的变化。我们表明，我们的带宽扩展导致了诸如电话（测试）嵌入向宽带（训练）信号的转移、感知质量与下游性能的负相关性以及条件无关的分数校准等现象。摘要：Automatic Speaker Verification (ASV) technology has become commonplace in virtual assistants. However, its performance suffers when there is a mismatch between the train and test domains. Mixed bandwidth training, i.e., pooling training data from both domains, is a preferred choice for developing a universal model that works for both narrowband and wideband domains. We propose complementing this technique by performing neural upsampling of narrowband signals, also known as bandwidth extension. Our main goal is to discover and analyze high-performing time-domain Generative Adversarial Network (GAN) based models to improve our downstream state-of-the-art ASV system. We choose GANs since they (1) are powerful for learning conditional distribution and (2) allow flexible plug-in usage as a pre-processor during the training of downstream task (ASV) with data augmentation. Prior works mainly focus on feature-domain bandwidth extension and limited experimental setups. We address these limitations by 1) using time-domain extension models, 2) reporting results on three real test sets, 2) extending training data, and 3) devising new test-time schemes. We compare supervised (conditional GAN) and unsupervised GANs (CycleGAN) and demonstrate average relative improvement in Equal Error Rate of 8.6% and 7.7%, respectively. For further analysis, we study changes in spectrogram visual quality, audio perceptual quality, t-SNE embeddings, and ASV score distributions. We show that our bandwidth extension leads to phenomena such as a shift of telephone (test) embeddings towards wideband (train) signals, a negative correlation of perceptual quality with downstream performance, and condition-independent score calibration.
【5】 Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
标题：让声音和视觉提示变得重要：CH-SIMS v2.0数据集和AV-Mixup一致性模块
链接：https://arxiv.org/abs/2209.02604
* 与cs.SD语音【1】为同一篇
作者：Yihe Liu,Ziqi Yuan,Huisheng Mao,Zhiyun Liang,Wanqiuyue Yang,Yuanzhe Qiu,Tie Cheng,Xiaoteng Li,Hua Xu,Kai Gao机构：WANQIUYUE, YANG, State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University, China,School of Information Science and Engineering, Hebei University of Science and Tech-备注：16pages, 7 figures, accepted by ICMI 2022摘要：多模态情感分析（Multimodal Sentiment Analysis，MSA）是一个新兴的研究领域，其主要目的是将听觉和视觉模态结合起来，以提高基于文本的情感分析的效率。然而，现有的研究发现，听觉和视觉模态的贡献远远小于语篇模态，被称为语篇主导。在这种情况下，在这项工作中，我们强调使非语言线索的MSA任务的重要性。首先，从资源角度出发，我们提出了CH-SIMS v2.0数据集，它是CH-SIMS的扩展和增强。与原始数据集相比，CH-SIMS v2.0的大小增加了一倍，另外增加了2121个具有单峰和多峰标注的细化视频片段，并收集了10161个具有丰富的听觉和视觉情感承载上下文的未标记原始视频片段，以突出非语言线索进行情感预测。其次，从模型角度出发，借鉴CH-SIMS v2. 0的单峰标注和无监督数据，提出了声视混合一致性（AV-MC）框架。所设计的模态混合模块可以被看作是一个扩展，它混合了来自不同视频的声学和视觉模态。通过将未观察到的多模态语境与文本一起提取，该模型可以学习感知不同的非言语语境，从而进行情感预测。我们的评估表明，CH-SIMS v2.0和AV-MC框架都支持进一步的研究，以发现承载情感的声音和视觉线索，并为现实世界场景中可解释的端到端HCI应用铺平道路。摘要：Multimodal sentiment analysis (MSA), which supposes to improve text-based sentiment analysis with associated acoustic and visual modalities, is an emerging research area due to its potential applications in Human-Computer Interaction (HCI). However, the existing researches observe that the acoustic and visual modalities contribute much less than the textual modality, termed as text-predominant. Under such circumstances, in this work, we emphasize making non-verbal cues matter for the MSA task. Firstly, from the resource perspective, we present the CH-SIMS v2.0 dataset, an extension and enhancement of the CH-SIMS. Compared with the original dataset, the CH-SIMS v2.0 doubles its size with another 2121 refined video segments with both unimodal and multimodal annotations and collects 10161 unlabelled raw video segments with rich acoustic and visual emotion-bearing context to highlight non-verbal cues for sentiment prediction. Secondly, from the model perspective, benefiting from the unimodal annotations and the unsupervised data in the CH-SIMS v2.0, the Acoustic Visual Mixup Consistent (AV-MC) framework is proposed. The designed modality mixup module can be regarded as an augmentation, which mixes the acoustic and visual modalities from different videos. Through drawing unobserved multimodal context along with the text, the model can learn to be aware of different non-verbal contexts for sentiment prediction. Our evaluations demonstrate that both CH-SIMS v2.0 and AV-MC framework enables further research for discovering emotion-bearing acoustic and visual cues and paves the path to interpretable end-to-end HCI applications for real-world scenarios.
【6】 Distilling the Knowledge of BERT for CTC-based ASR
标题：基于CTC的ASR中BERT知识的提取
链接：https://arxiv.org/abs/2209.02030
* 与cs.SD语音【2】为同一篇
作者：Hayato Futami,Hirofumi Inaguma,Masato Mimura,Shinsuke Sakai,Tatsuya Kawahara机构：Graduate School of Informatics, Kyoto University, Sakyo-ku, Kyoto, Japan摘要：基于连接主义时态分类（CTC）的模型在自动语音识别（ASR）中具有快速推理的优点。浅融合、重评分等语言模型集成方法可以充分利用文本语料库中的知识，提高基于CTC的ASR的识别准确率。然而，它们显著地减慢了CTC的推断。在本研究中，我们提出了提取BERT知识用于基于CTC的ASR，扩展了我们先前的基于注意力的ASR研究。基于CTC的ASR在训练阶段学习BERT知识，在测试阶段不使用BERT知识，保持了CTC的快速推理能力。与基于注意力的模型不同，基于CTC的模型进行帧级预测，因此它们需要与BERT的令牌级预测对齐以进行蒸馏。我们建议通过计算最合理的CTC路径来获得比对。在自发日语语料库（CSJ）和TED-LIUM2上的实验结果表明，该方法在不影响推理速度的前提下，提高了基于CTC的ASR的性能。摘要：Connectionist temporal classification (CTC) -based models are attractive because of their fast inference in automatic speech recognition (ASR). Language model (LM) integration approaches such as shallow fusion and rescoring can improve the recognition accuracy of CTC-based ASR by taking advantage of the knowledge in text corpora. However, they significantly slow down the inference of CTC. In this study, we propose to distill the knowledge of BERT for CTC-based ASR, extending our previous study for attention-based ASR. CTC-based ASR learns the knowledge of BERT during training and does not use BERT during testing, which maintains the fast inference of CTC. Different from attention-based models, CTC-based models make frame-level predictions, so they need to be aligned with token-level predictions of BERT for distillation. We propose to obtain alignments by calculating the most plausible CTC paths. Experimental evaluations on the Corpus of Spontaneous Japanese (CSJ) and TED-LIUM2 show that our method improves the performance of CTC-based ASR without the cost of inference speed.
【7】 Bridging Music and Text with Crowdsourced Music Comments: A Sequence-to-Sequence Framework for Thematic Music Comments Generation
标题：用众包音乐评论连接音乐和文本：一种主题音乐评论生成的序列到序列框架
链接：https://arxiv.org/abs/2209.01996
* 与cs.SD语音【3】为同一篇
作者：Peining Zhang,Junliang Guo,Linli Xu,Mu You,Junming Yin机构：Received: date Accepted: date摘要：我们考虑自动生成音乐的文本描述的新颖任务。与其他成熟的文本生成任务（如图像字幕）相比，缺乏配对良好的音乐和文本数据集使其成为一个更具挑战性的任务。本文利用众包音乐评论构建新的数据集，提出一种基于序列到序列的音乐文本描述生成模型。更具体地说，我们使用膨胀卷积层作为编码器的基本组件，并使用基于记忆的递归神经网络作为解码器。为了提高生成文本的真实性和主题性，我们进一步提出了一个新的主题鉴别器和一个新的主题评价器来微调模型。为了衡量生成文本的质量，我们还提出了两个新的评价指标，它们比传统的指标如BLEU更符合人类的评价。实验结果表明，该模型能够生成流畅、有意义的评论，同时包含了原音乐的主题和内容信息。摘要：We consider a novel task of automatically generating text descriptions of music. Compared with other well-established text generation tasks such as image caption, the scarcity of well-paired music and text datasets makes it a much more challenging task. In this paper, we exploit the crowd-sourced music comments to construct a new dataset and propose a sequence-to-sequence model to generate text descriptions of music. More concretely, we use the dilated convolutional layer as the basic component of the encoder and a memory based recurrent neural network as the decoder. To enhance the authenticity and thematicity of generated texts, we further propose to fine-tune the model with a discriminator as well as a novel topic evaluator. To measure the quality of generated texts, we also propose two new evaluation metrics, which are more aligned with human evaluation than traditional metrics such as BLEU. Experimental results verify that our model is capable of generating fluent and meaningful comments while containing thematic and content information of the original music.
【8】 Predict-and-Update Network: Audio-Visual Speech Recognition Inspired by Human Speech Perception
标题：预测与更新网络：受人类语音感知启发的视听语音识别
链接：https://arxiv.org/abs/2209.01768
* 与cs.SD语音【4】为同一篇
作者：Jiadong Wang,Xinyuan Qian,Haizhou Li摘要：在人类语音感知中，音频和视频信号是相辅相成的，在语音识别中也是如此。就语音感知而言，视觉提示不如声学提示明显，但在复杂的声学环境中更鲁棒。如何有效地利用音频和视频信号之间的相互作用进行自动语音识别仍然是一个挑战。已经有研究以同步方式利用视觉信号作为音频输入的冗余或补充信息。对人类的研究表明，视觉信号会提前让听者准备好什么时候和用什么频率去听。我们提出一个预测更新网络（P&U网），来模拟这种视觉线索机制用于音频—视频语音识别（AVSR）。具体地说，我们首先基于视觉信号预测说出的单词的字符后验，即视觉嵌入。然后通过一个新颖的跨模态Conformer对音频信号进行视觉嵌入调节，该Conformer更新字符后验。通过大量的实验验证了视觉线索机制的有效性。在LRS2-BBC和LRS3-BBC数据集上，所提出的P&U网的性能优于现有的AVSR方法，在干净和噪声条件下，相对降低的误字率（WER）分别超过10%和40%.摘要：Audio and visual signals complement each other in human speech perception, so do they in speech recognition. The visual hint is less evident than the acoustic hint, but more robust in a complex acoustic environment, as far as speech perception is concerned. It remains a challenge how we effectively exploit the interaction between audio and visual signals for automatic speech recognition. There have been studies to exploit visual signals as redundant or complementary information to audio input in a synchronous manner. Human studies suggest that visual signal primes the listener in advance as to when and on which frequency to attend to. We propose a Predict-and-Update Network (P&U net), to simulate such a visual cueing mechanism for Audio-Visual Speech Recognition (AVSR). In particular, we first predict the character posteriors of the spoken words, i.e. the visual embedding, based on the visual signals. The audio signal is then conditioned on the visual embedding via a novel cross-modal Conformer, that updates the character posteriors. We validate the effectiveness of the visual cueing mechanism through extensive experiments. The proposed P&U net outperforms the state-of-the-art AVSR methods on both LRS2-BBC and LRS3-BBC datasets, with the relative reduced Word Error Rate (WER)s exceeding 10% and 40% under clean and noisy conditions, respectively.
【9】 Exploiting Pre-trained Feature Networks for Generative Adversarial Networks in Audio-domain Loop Generation
标题：基于预训练特征网络的生成性对抗性网络在音频域环路生成中的应用
链接：https://arxiv.org/abs/2209.01751
* 与cs.SD语音【5】为同一篇
作者：Yen-Tung Yeh,Bo-Yu Chen,Yi-Hsuan Yang机构：Academia Sinica, National Taiwan University, Taiwan AI Labs备注：Accepted at ISMIR 2022摘要：生成式对抗网络（generative adversarial networks，GANs）已被广泛应用于音频生成的研究中，但GAN模型的训练存在不稳定、耗时和数据效率低等缺点.在改进GANs的训练过程的尝试中，投影GAN的思想作为基于GAN的图像生成的有效解决方案而出现，在不同的图像应用中建立了最先进的技术。其核心思想是使用预先训练好的分类器约束鉴别器的特征空间，以稳定和改进GAN训练。本文通过评估基于StyleGAN2的音频域循环生成模型在鉴别器中使用和不使用预先训练的特征空间的情况下的性能，研究了投影GAN是否可以类似地改善音频生成。此外，我们还比较了使用通用分类器和特定领域分类器作为预训练音频分类器的性能。通过对鼓循环和合成器循环生成的实验，我们表明通用音频分类器工作得更好，而使用投影GAN，我们的循环生成模型可以在不降低性能的情况下以大约5倍的速度收敛。摘要：While generative adversarial networks (GANs) have been widely used in research on audio generation, the training of a GAN model is known to be unstable, time consuming, and data inefficient. Among the attempts to ameliorate the training process of GANs, the idea of Projected GAN emerges as an effective solution for GAN-based image generation, establishing the state-of-the-art in different image applications. The core idea is to use a pre-trained classifier to constrain the feature space of the discriminator to stabilize and improve GAN training. This paper investigates whether Projected GAN can similarly improve audio generation, by evaluating the performance of a StyleGAN2-based audio-domain loop generation model with and without using a pre-trained feature space in the discriminator. Moreover, we compare the performance of using a general versus domain-specific classifier as the pre-trained audio classifier. With experiments on both drum loop and synth loop generation, we show that a general audio classifier works better, and that with Projected GAN our loop generation models can converge around 5 times faster without performance degradation.
【10】 Equivariant Self-Supervision for Musical Tempo Estimation
标题：用于音乐节奏估计的等变自监督算法
链接：https://arxiv.org/abs/2209.01478
* 与cs.SD语音【6】为同一篇
备注：Accepted at ISMIR 2022摘要：近年来，由于自监督方法减少了对稀缺且昂贵的标记数据集的需求，因此成为表征学习的一种有前途的方法。对比方法是音频域中用于自监督的流行选择，并且通常通过迫使模型对于输入的一些变换是不变的来提供学习信号。然而，这些方法需要采取诸如负采样或某种形式的正则化等措施，以防止模型在平凡解上崩溃。在这项工作中，我们建议使用等方差作为自监督信号，以代替不变性，从而从未标记数据中学习音频节奏表示。我们推导出一个简单的损失函数，它可以防止网络在训练过程中崩溃在一个平凡的解上，而不需要任何形式的正则化或负采样。实验结果表明，该方法可以通过等变自监督学习有意义的表征，在多个基准测试上获得与监督方法相当的性能.作为一个额外的好处，我们的方法只需要适度的计算资源，因此仍然可以为广泛的研究团体所访问。摘要：Self-supervised methods have emerged as a promising avenue for representation learning in the recent years since they alleviate the need for labeled datasets, which are scarce and expensive to acquire. Contrastive methods are a popular choice for self-supervision in the audio domain, and typically provide a learning signal by forcing the model to be invariant to some transformations of the input. These methods, however, require measures such as negative sampling or some form of regularisation to be taken to prevent the model from collapsing on trivial solutions. In this work, instead of invariance, we propose to use equivariance as a self-supervision signal to learn audio tempo representations from unlabelled data. We derive a simple loss function that prevents the network from collapsing on a trivial solution during training, without requiring any form of regularisation or negative sampling. Our experiments show that it is possible to learn meaningful representations for tempo estimation by solely relying on equivariant self-supervision, achieving performance comparable with supervised methods on several benchmarks. As an added benefit, our method only requires moderate compute resources and therefore remains accessible to a wide research community.
【11】 Identify The Beehive Sound Using Deep Learning
标题：基于深度学习的蜂窝声音识别
链接：https://arxiv.org/abs/2209.01374
* 与cs.SD语音【7】为同一篇
作者：Shah Jafor Sadeek Quaderi,Sadia Afrin Labonno,Sadia Mostafa,Shamim Akhter机构：Technology, University of Malaya, Kuala Lumpur, Malaysia, AISIP Lab, Department of Computer Science and Engineering, International, University of Business Agriculture and Technology, Dhaka, Bangladesh, Stamford University Bangladesh, Dhaka, Bangladesh摘要：花在消除环境中的沉闷发挥着重要作用。开花植物的生命周期包括授粉、受精、开花、种子形成、散布和萌发。蜜蜂为大约75%的开花植物授粉。环境污染、气候变化、自然景观破坏等，威胁着自然栖息地，从而不断减少蜜蜂的数量。因此，一些研究人员正在试图解决这个问题。将声学分类应用于蜂窝声音的记录可能是一种检测其中变化的方法。在本研究中，我们使用深度学习技术，即序列神经网络、卷积神经网络和递归神经网络，对记录的声音进行分类，将蜜蜂声音与非蜂窝噪声区分开来。此外，本文还对支持向量机、决策树、随机森林和朴素贝叶斯等非深度学习技术与深度学习技术进行了对比研究。还在组合的记录声音（25-75%噪声）上验证了这些技术。摘要：Flowers play an essential role in removing the duller from the environment. The life cycle of the flowering plants involves pollination, fertilization, flowering, seed-formation, dispersion, and germination. Honeybees pollinate approximately 75% of all flowering plants. Environmental pollution, climate change, natural landscape demolition, and so on, threaten the natural habitats, thus continuously reducing the number of honeybees. As a result, several researchers are attempting to resolve this issue. Applying acoustic classification to recordings of beehive sounds may be a way of detecting changes within them. In this research, we use deep learning techniques, namely Sequential Neural Network, Convolutional Neural Network, and Recurrent Neural Network, on the recorded sounds to classify bee sounds from the nonbeehive noises. In addition, we perform a comparative study among some popular non-deep learning techniques, namely Support Vector Machine, Decision Tree, Random Forest, and Na\"ive Bayes, with the deep learning techniques. The techniques are also verified on the combined recorded sounds (25-75% noises).
【12】 Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model
标题：基于交替拼写预测模型的生词语境识别
链接：https://arxiv.org/abs/2209.01250
* 与cs.SD语音【8】为同一篇
作者：Jennifer Drexler Fox,Natalie Delworth摘要：随着ASR的使用变得越来越广泛，将一系列偏见术语与音频一起作为输入的上下文ASR最近引起了人们的兴趣。我们发布了与Earnings21数据集相配套的上下文偏倚列表，为这项任务创建了一个公共基准。我们使用WeNet工具包中预先训练好的端到端ASR模型，展示了该基准测试的基线结果。我们示出了应用于两种不同解码算法的浅融合上下文偏置的结果。我们的基线结果证实了这样的观察，即端到端模型尤其与在训练期间很少或从未看到的单词作斗争，并且现有的浅融合技术不足以解决这个问题。我们提出了一个替代拼写预测模型，与没有替代拼写的上下文偏差相比，该模型将稀有单词的召回率相对提高了34.7%，将超出词汇表的单词的召回率相对提高了97.2%。该模型在概念上类似于在先前工作中使用的模型，但是由于其不依赖于发音字典或现有的文本到语音系统，因此实现起来更简单。摘要：Contextual ASR, which takes a list of bias terms as input along with audio, has drawn recent interest as ASR use becomes more widespread. We are releasing contextual biasing lists to accompany the Earnings21 dataset, creating a public benchmark for this task. We present baseline results on this benchmark using a pretrained end-to-end ASR model from the WeNet toolkit. We show results for shallow fusion contextual biasing applied to two different decoding algorithms. Our baseline results confirm observations that end-to-end models struggle in particular with words that are rarely or never seen during training, and that existing shallow fusion techniques do not adequately address this problem. We propose an alternate spelling prediction model that improves recall of rare words by 34.7% relative and of out-of-vocabulary words by 97.2% relative, compared to contextual biasing without alternate spellings. This model is conceptually similar to ones used in prior work, but is simpler to implement as it does not rely on either a pronunciation dictionary or an existing text-to-speech system.
机器翻译，仅供参考
每日语音速递