今日论文合集：cs.SD语音12篇，eess.AS音频处理11篇。
cs.SD语音
【1】 Audio-to-Intent Using Acoustic-Textual Subword Representations from End-to-End ASR
标题：使用来自端到端ASR的声文本子词表示的音频到意图
链接：https://arxiv.org/abs/2210.12134
作者：Pranay Dighe,Prateeth Nayak,Oggi Rudovic,Erik Marchi,Xiaochuan Niu,Ahmed Tewfik摘要：准确预测用户与设备上（例如，电话上）的语音助手（VA）交互的意图对于实现与VA的自然的、有吸引力的和以隐私为中心的交互是至关重要的。为此，我们提出了一种新颖的方法，用于直接根据在子词标记处编码的声学和文本信息来预测用户的意图（用户是否对着设备说话），所述子词标记经由端到端ASR模型获得。与音素和/或完整单词的建模相比，直接对子单词标记进行建模具有至少两个优点：（i）它提供了唯一的词汇表表示，其中，与音素级表示相比，每个标记具有语义含义，（i i）每个子词标记具有可重用的“子”词声学模式（其可用于构造多个完整词），从而导致比完整词大大减少的词汇表空间。为了学习用于音频到意图分类的子词表示，我们提取：（i）来自E2 E-ASR模型的声学信息，其提供子词标记的帧级CTC后验概率，以及（ii）来自预先训练的连续词袋模型的文本信息，其捕获子词标记的语义含义。我们方法的关键在于它使用位置编码的概念将声学子词级后验与文本信息相结合，以便同时考虑多个ASR假设。我们表明，我们的方法为音频到意图分类提供了更鲁棒和更丰富的表示，并且具有高准确性，以99%的真阳性率正确地减少了93.3%的非预期用户音频调用智能助手。摘要：Accurate prediction of the user intent to interact with a voice assistant (VA) on a device (e.g. on the phone) is critical for achieving naturalistic, engaging, and privacy-centric interactions with the VA. To this end, we present a novel approach to predict the user's intent (the user speaking to the device or not) directly from acoustic and textual information encoded at subword tokens which are obtained via an end-to-end ASR model. Modeling directly the subword tokens, compared to modeling of the phonemes and/or full words, has at least two advantages: (i) it provides a unique vocabulary representation, where each token has a semantic meaning, in contrast to the phoneme-level representations, (ii) each subword token has a reusable "sub"-word acoustic pattern (that can be used to construct multiple full words), resulting in a largely reduced vocabulary space than of the full words. To learn the subword representations for the audio-to-intent classification, we extract: (i) acoustic information from an E2E-ASR model, which provides frame-level CTC posterior probabilities for the subword tokens, and (ii) textual information from a pre-trained continuous bag-of-words model capturing the semantic meaning of the subword tokens. The key to our approach is the way it combines acoustic subword-level posteriors with text information using the notion of positional-encoding in order to account for multiple ASR hypotheses simultaneously. We show that our approach provides more robust and richer representations for audio-to-intent classification, and is highly accurate with correctly mitigating 93.3% of unintended user audio from invoking the smart assistant at 99% true positive rate.
【2】 Adversarial Permutation Invariant Training for Universal Sound Separation
标题：通用声音分离的对抗性排列不变训练
链接：https://arxiv.org/abs/2210.12108
作者：Emilian Postolache,Jordi Pons,Santiago Pascual,Joan Serrà机构：Dolby Laboratories, Sapienza University of Rome备注：Demo page: this http URL摘要：通用声音分离包括分离具有不同类型的任意声音的混合，并且使用排列不变训练（PIT）来训练这样做的源不可知模型。在这项工作中，我们补充PIT与对抗损失，但发现它的挑战与标准公式使用的语音源分离。我们通过一种新颖的基于I-替换上下文的对抗性损失，并通过使用多个鉴别器进行训练来克服这一挑战。我们的实验表明，通过简单地改进损耗（保持相同的模型和数据集），我们在混响FUSS数据集中获得了1.4dB SI-SNRi的不可忽略的改进。我们还发现，对抗性PIT在减少谱洞方面是有效的，谱洞在基于掩蔽的分离模型中普遍存在，这突出了对抗性损失对于源分离的潜在相关性。摘要：Universal sound separation consists of separating mixes with arbitrary sounds of different types, and permutation invariant training (PIT) is used to train source agnostic models that do so. In this work, we complement PIT with adversarial losses but find it challenging with the standard formulation used in speech source separation. We overcome this challenge with a novel I-replacement context-based adversarial loss, and by training with multiple discriminators. Our experiments show that by simply improving the loss (keeping the same model and dataset) we obtain a non-negligible improvement of 1.4 dB SI-SNRi in the reverberant FUSS dataset. We also find adversarial PIT to be effective at reducing spectral holes, ubiquitous in mask-based separation models, which highlights the potential relevance of adversarial losses for source separation.
【3】 Spoken Term Detection and Relevance Score Estimation using Dot-Product of Pronunciation Embeddings
标题：基于发音嵌入点积的口语词检测和相关度得分估计
链接：https://arxiv.org/abs/2210.11895
作者：Jan Švec,Luboš Šmídl,Josef V. Psutka,Aleš Pražák机构：Department of Cybernetics, University of West Bohemia, Pilsen, Czech Republic摘要：本文提出了一种基于深度LSTM网络的大型口语语料库中的口语词检测方法。该工作基于先前使用暹罗神经网络进行STD的方法，并自然地将其扩展到直接定位口语术语并估计其相关性得分。由音素识别器生成的音素混淆网络由深度LSTM网络处理，该深度LSTM网络将混淆网络的每个片段投影到嵌入空间中。使用另一个深度LSTM网络将搜索项投影到相同的嵌入空间中。然后使用嵌入空间中的简单点积来计算相关性分数，并使用S形函数来校准以预测发生概率。然后从输出概率序列估计搜索项的位置。深度LSTM网络是以自监督的方式从词和音素水平上的成对识别假设训练的。在英语和捷克语的MALACH数据上对该方法进行了实验评估。摘要：The paper describes a novel approach to Spoken Term Detection (STD) in large spoken archives using deep LSTM networks. The work is based on the previous approach of using Siamese neural networks for STD and naturally extends it to directly localize a spoken term and estimate its relevance score. The phoneme confusion network generated by a phoneme recognizer is processed by the deep LSTM network which projects each segment of the confusion network into an embedding space. The searched term is projected into the same embedding space using another deep LSTM network. The relevance score is then computed using a simple dot-product in the embedding space and calibrated using a sigmoid function to predict the probability of occurrence. The location of the searched term is then estimated from the sequence of output probabilities. The deep LSTM networks are trained in a self-supervised manner from paired recognition hypotheses on word and phoneme levels. The method is experimentally evaluated on MALACH data in English and Czech languages.
【4】 Deep LSTM Spoken Term Detection using Wav2Vec 2.0 Recognizer
标题：基于Wav2Vec 2.0识别器的深度LSTM语音检测
链接：https://arxiv.org/abs/2210.11885
作者：Jan Švec,Jan Lehečka,Luboš Šmídl机构：Luboˇs ˇSm´ıdlDepartment of Cybernetics, University of West Bohemia摘要：近年来，端到端语音识别系统的性能已经超过了标准的混合DNN-HMM语音识别器。其中一个很有前途的方法是字素Wav2Vec2.0模型，它使用自监督预训练方法结合微调语音识别器的迁移学习。由于缺乏发音词汇和语言模型，该方法适用于不容易或几乎不可能获得这些模型的任务。 本文使用Wav 2 Vec语音识别器对大量的口语文档进行口语术语检测。该方法采用深度LSTM网络，该网络将识别的假设和搜索的术语映射到共享发音嵌入空间，在该空间中术语出现和分配的分数容易计算。 本文描述了一种自举方法，该方法允许将DNN-HMM混合ASR的传统发音词汇中包含的知识转移到基于字形的Wav 2 Vec的上下文中。在英语和捷克语的MALACH数据上，该方法的性能明显优于先前发表的基于DNN-HMM混合ASR和音素识别器相结合的系统。摘要：In recent years, the standard hybrid DNN-HMM speech recognizers are outperformed by the end-to-end speech recognition systems. One of the very promising approaches is the grapheme Wav2Vec 2.0 model, which uses the self-supervised pretraining approach combined with transfer learning of the fine-tuned speech recognizer. Since it lacks the pronunciation vocabulary and language model, the approach is suitable for tasks where obtaining such models is not easy or almost impossible. In this paper, we use the Wav2Vec speech recognizer in the task of spoken term detection over a large set of spoken documents. The method employs a deep LSTM network which maps the recognized hypothesis and the searched term into a shared pronunciation embedding space in which the term occurrences and the assigned scores are easily computed. The paper describes a bootstrapping approach that allows the transfer of the knowledge contained in traditional pronunciation vocabulary of DNN-HMM hybrid ASR into the context of grapheme-based Wav2Vec. The proposed method outperforms the previously published system based on the combination of the DNN-HMM hybrid ASR and phoneme recognizer by a large margin on the MALACH data in both English and Czech languages.
【5】 A Textless Metric for Speech-to-Speech Comparison
标题：一种用于语音到语音比对的无文本度量
链接：https://arxiv.org/abs/2210.11835
作者：Laurent Besacier,Swen Ribeiro,Olivier Galibert,Ioan Calapodescu机构：Naver Labs Europe, France, LNE, France摘要：本文提出了一种无文本的语音-语音比较度量，该度量允许将语音假设与语音参考进行比较，而无需退回到它们的文本转录本。我们利用最近提出的speech 2unit编码器（如HuBERT）将语音话语伪转录为离散的声学单元，并提出了一种简单的神经结构，该结构学习基于语音的度量，该度量与基于文本的度量很好地相关。这样的无文本度量最终可能对于语音到语音翻译评估（对于口头语言或没有可靠ASR系统可用的语言）是感兴趣的。摘要：This paper proposes a textless speech-to-speech comparison metric that allows comparing a speech hypothesis with a speech reference without falling-back to their text transcripts. We leverage recently proposed speech2unit encoders (such as HuBERT) to pseudo-transcribe the speech utterances into discrete acoustic units and propose a simple neural architecture that learns a speech-based metric which correlates well with its text-based counterpart. Such a textless metric could ultimately be interesting for speech-to-speech translation evaluation (for oral languages or languages with no reliable ASR system available).
【6】 Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
标题：语音自我监督学习中声道清晰度的证据
链接：https://arxiv.org/abs/2210.11723
作者：Cheol Jun Cho,Peter Wu,Abdelrahman Mohamed,Gopala K. Anumanchipalli机构：UC Berkeley, EECS, CA, Meta AI摘要：许多用于语音的自监督学习（SSL）模型已经被提出用于语音表示的预训练模型，并且最近的SSL模型在各种下游任务中非常成功。为了理解这种效用，先前的工作探索语音模型的表示，以揭示哪些和如何语音相关的信息被编码在学习的表示中。虽然编码特性已经从声学、语音学和语义学的角度得到了广泛的探讨，但语音产生的物理基础尚未得到充分的关注。为了弥补这一差距，我们进行了一项综合分析，将言语表征与电磁发音描记术（EMA）测量的发音轨迹联系起来。我们的分析是基于线性探测方法，我们测量发音评分作为线性映射到EMA的平均相关性。我们分析了从SU-PERB基准测试的排行榜中选出的一组SSL模型，并对两个主要模型Wav 2 Vec 2.0和HuBERT进行了进一步的详细分析。令人惊讶的是，来自最近的语音SSL模型的表示与EMA迹线高度相关（最佳：r = 0.81），并且仅5分钟就足以训练具有高性能的线性模型（r = 0.77）。我们的研究结果表明，SSL模型学习与连续发音紧密结合，并提供了一个新的见解语音SSL。摘要：Numerous self-supervised learning (SSL) models for speech have been proposed for pre-training models of speech representations, and recent SSL models are very successful in diverse downstream tasks. To understand such utilities, previous works probe representations of speech models to reveal which & how speech related information is encoded in the learned representations. While encoding properties have been extensively explored from the perspective of acoustics, phonetics, and semantics, the physical grounding by speech production has not yet received full attention. To bridge this gap, we conduct a comprehensive analysis to link speech representations to articulatory trajectories measured by electromagnetic articulography (EMA). Our analysis is based on a linear probing approach where we measure articulatory score as an average correlation of linear mapping to EMA. We analyze a set of SSL models selected from the leaderboard of the SU- PERB benchmark and perform further detailed analyses on two major models, Wav2Vec 2.0 and HuBERT. Surprisingly, representations from the recent speech SSL models are highly correlated with EMA traces (best: r = 0.81), and only 5 minutes were sufficient to train a linear model with high performance (r = 0.77). Our findings suggest that SSL models learn to closely align with continuous articulations and provide a novel insight into speech SSL.
【7】 Adaptive re-calibration of channel-wise features for Adversarial Audio Classification
标题：用于对抗性音频分类的通道特征的自适应重新校准
链接：https://arxiv.org/abs/2210.11722
作者：Vardhan Dongre,Abhinav Thimma Reddy,Nikhitha Reddeddy机构：[,] University of Illinois Urbana-Champaign备注：7 pages, 8 figures, 4 tables摘要：与DeepFake图像和视频不同，DeepFake音频从检测角度来说相对较少被探索，并且存在的用于合成语音分类的解决方案或者使用复杂网络或者不推广到使用不同的生成和基于优化的方法获得的不同种类的合成语音。通过这项工作，我们提出了一种使用注意力特征融合的通道式特征重新校准方法，用于合成语音检测，并比较了其与不同检测方法（包括End2End模型和基于Resnet的模型）在使用文语转换和声码器系统（如WaveNet、WaveRNN、Tactotron和WaveGlow）生成的合成语音上的性能。我们还在我们的Resnet模型中实验了挤压激励（SE）模块，发现这种组合能够获得更好的性能。除了分析之外，我们还证明了使用注意特征融合技术的线性频率倒谱系数（LFCC）和梅尔频率倒谱系数（MFCC）的组合创建了更好的输入特征表示，这可以帮助甚至更简单的模型在合成语音分类任务上很好地推广。我们的模型（使用特征融合的基于Resnet的）在Fake or Real（FoR）数据集上训练，并且能够在FoR数据上实现95%的测试准确度，并且在适应该框架后，对于我们使用不同的生成模型生成的样本，能够实现平均90%的准确度。摘要：DeepFake Audio, unlike DeepFake images and videos, has been relatively less explored from detection perspective, and the solutions which exist for the synthetic speech classification either use complex networks or dont generalize to different varieties of synthetic speech obtained using different generative and optimization-based methods. Through this work, we propose a channel-wise recalibration of features using attention feature fusion for synthetic speech detection and compare its performance against different detection methods including End2End models and Resnet-based models on synthetic speech generated using Text to Speech and Vocoder systems like WaveNet, WaveRNN, Tactotron, and WaveGlow. We also experiment with Squeeze Excitation (SE) blocks in our Resnet models and found that the combination was able to get better performance. In addition to the analysis, we also demonstrate that the combination of Linear frequency cepstral coefficients (LFCC) and Mel Frequency cepstral coefficients (MFCC) using the attentional feature fusion technique creates better input features representations which can help even simpler models generalize well on synthetic speech classification tasks. Our models (Resnet based using feature fusion) trained on Fake or Real (FoR) dataset and were able to achieve 95% test accuracy with the FoR data, and an average of 90% accuracy with samples we generated using different generative models after adapting this framework.
【8】 Improving Semi-supervised End-to-end Automatic Speech Recognition using CycleGAN and Inter-domain Losses
标题：利用CycleGAN和域间损失改进半监督端到端自动语音识别
链接：https://arxiv.org/abs/2210.11642
作者：Chia-Yu Li,Ngoc Thang Vu机构：Institute for Natural Language Processing (IMS), University of Stuttgart, Germany备注：6 pages + 2 references, 6 figures, accepted by SLT2022摘要：提出了一种结合CycleGAN和域间损失的半监督端到端自动语音识别方法。域间丢失的目标是使用共享网络提取语音和文本输入的中间共享表示。CycleGAN使用循环一致性损失和身份映射损失来在从一个域转换到另一个域之后保持输入特征的相关特征。因此，两种方法都适合于在不成对的语音-文本输入上训练端到端模型。本文利用域间丢失和CycleGAN的优点，实现非成对语音和文本输入的更好共享表示，从而改进语音到文本的映射。在WSJ eval 92和Voxforge（非英语）上的实验结果表明，与基线相比，该算法的字符错误率降低了8~ 8. 5%，在LibriSpeech test_clean上的实验结果也显示了明显的改善。摘要：We propose a novel method that combines CycleGAN and inter-domain losses for semi-supervised end-to-end automatic speech recognition. Inter-domain loss targets the extraction of an intermediate shared representation of speech and text inputs using a shared network. CycleGAN uses cycle-consistent loss and the identity mapping loss to preserve relevant characteristics of the input feature after converting from one domain to another. As such, both approaches are suitable to train end-to-end models on unpaired speech-text inputs. In this paper, we exploit the advantages from both inter-domain loss and CycleGAN to achieve better shared representation of unpaired speech and text inputs and thus improve the speech-to-text mapping. Our experimental results on the WSJ eval92 and Voxforge (non English) show 8~8.5% character error rate reduction over the baseline, and the results on LibriSpeech test_clean also show noticeable improvement.
【9】 Improved Normalizing Flow-Based Speech Enhancement using an All-pole Gammatone Filterbank for Conditional Input Representation
标题：用于条件输入表示的全极点Gammatone滤波器组改进的归一化流语音增强
链接：https://arxiv.org/abs/2210.11654
作者：Martin Strauss,Matteo Torcoli,Bernd Edler机构：International Audio Laboratories Erlangen, a joint institution of the Friedrich-Alexander-Universit¨at, Erlangen-N¨urnberg (FAU) and Fraunhofer IIS, Germany备注：Accepted for Presentation at IEEE SLT 2022摘要：近年来，语音增强的深度生成模型受到了越来越多的关注。最突出的例子是生成式对抗网络（GANs），而规范化流（NF）尽管具有潜力，但受到的关注较少。在以前工作的基础上，提出了体系结构的修改，以及对不同条件输入表示的研究。尽管Mel谱图是相关工作中的常见选择，但对于给定的场景，Mel谱图被证明是不充分的。另外，提出了一种新的具有高时间分辨率的全极点Gammatone滤波器组（APG）。虽然计算评估度量结果将表明基于GAN的现有技术的方法性能最好，但是通过听音测试的感知评估表明所提出的NF方法（基于时域和APG）性能最好，特别是在较低SNR的情况下。平均而言，APG输出被评为具有良好的质量，这是其他方法（包括GAN）无法比拟的。摘要：Deep generative models for Speech Enhancement (SE) received increasing attention in recent years. The most prominent example are Generative Adversarial Networks (GANs), while normalizing flows (NF) received less attention despite their potential. Building on previous work, architectural modifications are proposed, along with an investigation of different conditional input representations. Despite being a common choice in related works, Mel-spectrograms demonstrate to be inadequate for the given scenario. Alternatively, a novel All-Pole Gammatone filterbank (APG) with high temporal resolution is proposed. Although computational evaluation metric results would suggest that state-of-the-art GAN-based methods perform best, a perceptual evaluation via a listening test indicates that the presented NF approach (based on time domain and APG) performs best, especially at lower SNRs. On average, APG outputs are rated as having good quality, which is unmatched by the other methods, including GAN.
【10】 Anchored Speech Recognition with Neural Transducers
标题：基于神经传感器的锚定语音识别
链接：https://arxiv.org/abs/2210.11588
作者：Desh Raj,Junteng Jia,Jay Mahadeokar,Chunyang Wu,Niko Moritz,Xiaohui Zhang,Ozlem Kalinli机构：Center for Language and Speech Processing, Johns Hopkins University, USA,Meta AI, USA备注：Submitted to IEEE ICASSP 2023摘要：神经传感器在生产ASR系统中越来越受欢迎，在标准基准数据集上实现了人类级别的识别准确度。然而，在存在串扰的情况下，尤其是当背景语音/噪声与主语音相比不可忽略时（即，低信噪比），它们的性能显著降低。锚定语音识别指的是使用来自锚定段的信息（例如，唤醒字）来识别设备引导的语音，同时忽略干扰的背景语音/噪声。在这篇文章中，我们研究了神经传感器环境下的锚定语音识别。我们使用一个微小的辅助网络从锚段中提取上下文信息，并利用编码器偏置和连接器门控来引导换能器朝向目标语音。此外，为了提高上下文嵌入提取的鲁棒性，我们提出了辅助训练目标来分离词汇内容和说话风格。我们提出的方法在合成的基于LibriSpeech的混合物上进行了评估，与背景增强基线相比，它们将单词错误率提高了36%。摘要：Neural transducers have gained popularity in production ASR systems, achieving human level recognition accuracy on standard benchmark datasets. However, their performance significantly degrades in the presence of crosstalks, especially when the background speech/noise is non-negligible as compared to the primary speech (i.e. low signal-to-noise ratio). Anchored speech recognition refers to a class of methods that use information from an anchor segment (e.g., wake-words) to recognize device-directed speech while ignoring interfering background speech/noise. In this paper, we investigate anchored speech recognition in the context of neural transducers. We use a tiny auxiliary network to extract context information from the anchor segment, and explore encoder biasing and joiner gating to guide the transducer towards the target speech. Moreover, to improve the robustness of context embedding extraction, we propose auxiliary training objectives to disentagle lexical content from speaking style. Our proposed methods are evaluated on synthetic LibriSpeech-based mixtures, where they improve word error rates by up to 36% compared to a background augmentation baseline.
【11】 Discriminatory and orthogonal feature learning for noise robust keyword spotting
标题：基于区分和正交特征学习的抗噪关键词检测
链接：https://arxiv.org/abs/2210.11519
作者：Donghyeon Kim,Kyungdeuk Ko,David K. Han,Hanseok Ko摘要：关键字识别（Keyword Spotting，KWS）是智能设备中的一个重要组件，用于在用户使用命令提示系统时向系统发出警报。由于这些设备通常受到计算和能源资源的限制，因此KWS模型应设计为占用空间小。在我们以前的工作中，我们开发了轻量级的动态滤波器，它在噪声环境中提取鲁棒的特征图。利用交叉熵损失对动态滤波器的学习变量和KWS权值进行联合优化。然而，当SNR较低时，单独的CE损耗不足以实现高性能。为了训练网络在噪声环境中具有更强的鲁棒性，我们引入了LOw变量正交（LOVO）损失。LOVO损耗由应用于动态滤波器输出端的三重态损耗、基于谱范数的正交损耗和应用于KWS模型的内部类距离损耗组成。这些损失在鼓励网络在看不见的噪声环境中提取区别特征方面特别有用。摘要：Keyword Spotting (KWS) is an essential component in a smart device for alerting the system when a user prompts it with a command. As these devices are typically constrained by computational and energy resources, the KWS model should be designed with a small footprint. In our previous work, we developed lightweight dynamic filters which extract a robust feature map within a noisy environment. The learning variables of the dynamic filter are jointly optimized with KWS weights by using Cross-Entropy (CE) loss. CE loss alone, however, is not sufficient for high performance when the SNR is low. In order to train the network for more robust performance in noisy environments, we introduce the LOw Variant Orthogonal (LOVO) loss. The LOVO loss is composed of a triplet loss applied on the output of the dynamic filter, a spectral norm-based orthogonal loss, and an inner class distance loss applied in the KWS model. These losses are particularly useful in encouraging the network to extract discriminatory features in unseen noise environments.
【12】 Are E2E ASR models ready for an industrial usage?
标题：E2E ASR型号是否已准备好投入工业使用？
链接：https://arxiv.org/abs/2112.12572
作者：Valentin Vielzeuf,Grigory Antipov机构：Orange, rue du Clos Courtel, Cesson-S´evign´e, France摘要：随着全神经（端到端，E2E）方法的兴起，自动语音识别（ASR）社区经历了一个重大转折点。同时，传统的混合模型仍然是ASR实际应用的标准选择。根据以前的研究，E2E ASR在实际应用中的采用受到两个主要限制的阻碍：它们对看不见的领域进行归纳的能力以及它们的高操作成本。在本文中，我们通过对几个当代E2E模型和一个混合基线进行全面的多领域基准测试，研究了上述两个缺点。实验表明，E2E模型是混合方法的可行替代方案，甚至在准确性和运行效率上都优于基线。因此，我们的研究表明，通用性和复杂性问题不再是行业集成的主要障碍，并提请社区注意E2E方法在某些特定用例中的其他潜在局限性。摘要：The Automated Speech Recognition (ASR) community experiences a major turning point with the rise of the fully-neural (End-to-End, E2E) approaches. At the same time, the conventional hybrid model remains the standard choice for the practical usage of ASR. According to previous studies, the adoption of E2E ASR in real-world applications was hindered by two main limitations: their ability to generalize on unseen domains and their high operational cost. In this paper, we investigate both above-mentioned drawbacks by performing a comprehensive multi-domain benchmark of several contemporary E2E models and a hybrid baseline. Our experiments demonstrate that E2E models are viable alternatives for the hybrid approach, and even outperform the baseline both in accuracy and in operational efficiency. As a result, our study shows that the generalization and complexity issues are no longer the major obstacle for industrial integration, and draws the community's attention to other potential limitations of the E2E approaches in some specific use-cases.
【1】 Improved Normalizing Flow-Based Speech Enhancement using an All-pole Gammatone Filterbank for Conditional Input Representation
标题：用于条件输入表示的全极点Gammatone滤波器组改进的归一化流语音增强
链接：https://arxiv.org/abs/2210.11654
* 与cs.SD语音【9】为同一篇
作者：Martin Strauss,Matteo Torcoli,Bernd Edler机构：International Audio Laboratories Erlangen, a joint institution of the Friedrich-Alexander-Universit¨at, Erlangen-N¨urnberg (FAU) and Fraunhofer IIS, Germany备注：Accepted for Presentation at IEEE SLT 2022摘要：近年来，语音增强的深度生成模型受到了越来越多的关注。最突出的例子是生成式对抗网络（GANs），而规范化流（NF）尽管具有潜力，但受到的关注较少。在以前工作的基础上，提出了体系结构的修改，以及对不同条件输入表示的研究。尽管Mel谱图是相关工作中的常见选择，但对于给定的场景，Mel谱图被证明是不充分的。另外，提出了一种新的具有高时间分辨率的全极点Gammatone滤波器组（APG）。虽然计算评估度量结果将表明基于GAN的现有技术的方法性能最好，但是通过听音测试的感知评估表明所提出的NF方法（基于时域和APG）性能最好，特别是在较低SNR的情况下。平均而言，APG输出被评为具有良好的质量，这是其他方法（包括GAN）无法比拟的。摘要：Deep generative models for Speech Enhancement (SE) received increasing attention in recent years. The most prominent example are Generative Adversarial Networks (GANs), while normalizing flows (NF) received less attention despite their potential. Building on previous work, architectural modifications are proposed, along with an investigation of different conditional input representations. Despite being a common choice in related works, Mel-spectrograms demonstrate to be inadequate for the given scenario. Alternatively, a novel All-Pole Gammatone filterbank (APG) with high temporal resolution is proposed. Although computational evaluation metric results would suggest that state-of-the-art GAN-based methods perform best, a perceptual evaluation via a listening test indicates that the presented NF approach (based on time domain and APG) performs best, especially at lower SNRs. On average, APG outputs are rated as having good quality, which is unmatched by the other methods, including GAN.
【2】 Anchored Speech Recognition with Neural Transducers
标题：基于神经传感器的锚定语音识别
链接：https://arxiv.org/abs/2210.11588
* 与cs.SD语音【10】为同一篇
作者：Desh Raj,Junteng Jia,Jay Mahadeokar,Chunyang Wu,Niko Moritz,Xiaohui Zhang,Ozlem Kalinli机构：Center for Language and Speech Processing, Johns Hopkins University, USA,Meta AI, USA备注：Submitted to IEEE ICASSP 2023摘要：神经传感器在生产ASR系统中越来越受欢迎，在标准基准数据集上实现了人类级别的识别准确度。然而，在存在串扰的情况下，尤其是当背景语音/噪声与主语音相比不可忽略时（即，低信噪比），它们的性能显著降低。锚定语音识别指的是使用来自锚定段的信息（例如，唤醒字）来识别设备引导的语音，同时忽略干扰的背景语音/噪声。在这篇文章中，我们研究了神经传感器环境下的锚定语音识别。我们使用一个微小的辅助网络从锚段中提取上下文信息，并利用编码器偏置和连接器门控来引导换能器朝向目标语音。此外，为了提高上下文嵌入提取的鲁棒性，我们提出了辅助训练目标来分离词汇内容和说话风格。我们提出的方法在合成的基于LibriSpeech的混合物上进行了评估，与背景增强基线相比，它们将单词错误率提高了36%。摘要：Neural transducers have gained popularity in production ASR systems, achieving human level recognition accuracy on standard benchmark datasets. However, their performance significantly degrades in the presence of crosstalks, especially when the background speech/noise is non-negligible as compared to the primary speech (i.e. low signal-to-noise ratio). Anchored speech recognition refers to a class of methods that use information from an anchor segment (e.g., wake-words) to recognize device-directed speech while ignoring interfering background speech/noise. In this paper, we investigate anchored speech recognition in the context of neural transducers. We use a tiny auxiliary network to extract context information from the anchor segment, and explore encoder biasing and joiner gating to guide the transducer towards the target speech. Moreover, to improve the robustness of context embedding extraction, we propose auxiliary training objectives to disentagle lexical content from speaking style. Our proposed methods are evaluated on synthetic LibriSpeech-based mixtures, where they improve word error rates by up to 36% compared to a background augmentation baseline.
【3】 Discriminatory and orthogonal feature learning for noise robust keyword spotting
标题：基于区分和正交特征学习的抗噪关键词检测
链接：https://arxiv.org/abs/2210.11519
* 与cs.SD语音【11】为同一篇
作者：Donghyeon Kim,Kyungdeuk Ko,David K. Han,Hanseok Ko摘要：关键字识别（Keyword Spotting，KWS）是智能设备中的一个重要组件，用于在用户使用命令提示系统时向系统发出警报。由于这些设备通常受到计算和能源资源的限制，因此KWS模型应设计为占用空间小。在我们以前的工作中，我们开发了轻量级的动态滤波器，它在噪声环境中提取鲁棒的特征图。利用交叉熵损失对动态滤波器的学习变量和KWS权值进行联合优化。然而，当SNR较低时，单独的CE损耗不足以实现高性能。为了训练网络在噪声环境中具有更强的鲁棒性，我们引入了LOw变量正交（LOVO）损失。LOVO损耗由应用于动态滤波器输出端的三重态损耗、基于谱范数的正交损耗和应用于KWS模型的内部类距离损耗组成。这些损失在鼓励网络在看不见的噪声环境中提取区别特征方面特别有用。摘要：Keyword Spotting (KWS) is an essential component in a smart device for alerting the system when a user prompts it with a command. As these devices are typically constrained by computational and energy resources, the KWS model should be designed with a small footprint. In our previous work, we developed lightweight dynamic filters which extract a robust feature map within a noisy environment. The learning variables of the dynamic filter are jointly optimized with KWS weights by using Cross-Entropy (CE) loss. CE loss alone, however, is not sufficient for high performance when the SNR is low. In order to train the network for more robust performance in noisy environments, we introduce the LOw Variant Orthogonal (LOVO) loss. The LOVO loss is composed of a triplet loss applied on the output of the dynamic filter, a spectral norm-based orthogonal loss, and an inner class distance loss applied in the KWS model. These losses are particularly useful in encouraging the network to extract discriminatory features in unseen noise environments.
【4】 Audio-to-Intent Using Acoustic-Textual Subword Representations from End-to-End ASR
标题：使用来自端到端ASR的声文本子词表示的音频到意图
链接：https://arxiv.org/abs/2210.12134
* 与cs.SD语音【1】为同一篇
作者：Pranay Dighe,Prateeth Nayak,Oggi Rudovic,Erik Marchi,Xiaochuan Niu,Ahmed Tewfik摘要：准确预测用户与设备上（例如，电话上）的语音助手（VA）交互的意图对于实现与VA的自然的、有吸引力的和以隐私为中心的交互是至关重要的。为此，我们提出了一种新颖的方法，用于直接根据在子词标记处编码的声学和文本信息来预测用户的意图（用户是否对着设备说话），所述子词标记经由端到端ASR模型获得。与音素和/或完整单词的建模相比，直接对子单词标记进行建模具有至少两个优点：（i）它提供了唯一的词汇表表示，其中，与音素级表示相比，每个标记具有语义含义，（i i）每个子词标记具有可重用的“子”词声学模式（其可用于构造多个完整词），从而导致比完整词大大减少的词汇表空间。为了学习用于音频到意图分类的子词表示，我们提取：（i）来自E2 E-ASR模型的声学信息，其提供子词标记的帧级CTC后验概率，以及（ii）来自预先训练的连续词袋模型的文本信息，其捕获子词标记的语义含义。我们方法的关键在于它使用位置编码的概念将声学子词级后验与文本信息相结合，以便同时考虑多个ASR假设。我们表明，我们的方法为音频到意图分类提供了更鲁棒和更丰富的表示，并且具有高准确性，以99%的真阳性率正确地减少了93.3%的非预期用户音频调用智能助手。摘要：Accurate prediction of the user intent to interact with a voice assistant (VA) on a device (e.g. on the phone) is critical for achieving naturalistic, engaging, and privacy-centric interactions with the VA. To this end, we present a novel approach to predict the user's intent (the user speaking to the device or not) directly from acoustic and textual information encoded at subword tokens which are obtained via an end-to-end ASR model. Modeling directly the subword tokens, compared to modeling of the phonemes and/or full words, has at least two advantages: (i) it provides a unique vocabulary representation, where each token has a semantic meaning, in contrast to the phoneme-level representations, (ii) each subword token has a reusable "sub"-word acoustic pattern (that can be used to construct multiple full words), resulting in a largely reduced vocabulary space than of the full words. To learn the subword representations for the audio-to-intent classification, we extract: (i) acoustic information from an E2E-ASR model, which provides frame-level CTC posterior probabilities for the subword tokens, and (ii) textual information from a pre-trained continuous bag-of-words model capturing the semantic meaning of the subword tokens. The key to our approach is the way it combines acoustic subword-level posteriors with text information using the notion of positional-encoding in order to account for multiple ASR hypotheses simultaneously. We show that our approach provides more robust and richer representations for audio-to-intent classification, and is highly accurate with correctly mitigating 93.3% of unintended user audio from invoking the smart assistant at 99% true positive rate.
【5】 Adversarial Permutation Invariant Training for Universal Sound Separation
标题：通用声音分离的对抗性排列不变训练
链接：https://arxiv.org/abs/2210.12108
* 与cs.SD语音【2】为同一篇
作者：Emilian Postolache,Jordi Pons,Santiago Pascual,Joan Serrà机构：Dolby Laboratories, Sapienza University of Rome备注：Demo page: this http URL摘要：通用声音分离包括分离具有不同类型的任意声音的混合，并且使用排列不变训练（PIT）来训练这样做的源不可知模型。在这项工作中，我们补充PIT与对抗损失，但发现它的挑战与标准公式使用的语音源分离。我们通过一种新颖的基于I-替换上下文的对抗性损失，并通过使用多个鉴别器进行训练来克服这一挑战。我们的实验表明，通过简单地改进损耗（保持相同的模型和数据集），我们在混响FUSS数据集中获得了1.4dB SI-SNRi的不可忽略的改进。我们还发现，对抗性PIT在减少谱洞方面是有效的，谱洞在基于掩蔽的分离模型中普遍存在，这突出了对抗性损失对于源分离的潜在相关性。摘要：Universal sound separation consists of separating mixes with arbitrary sounds of different types, and permutation invariant training (PIT) is used to train source agnostic models that do so. In this work, we complement PIT with adversarial losses but find it challenging with the standard formulation used in speech source separation. We overcome this challenge with a novel I-replacement context-based adversarial loss, and by training with multiple discriminators. Our experiments show that by simply improving the loss (keeping the same model and dataset) we obtain a non-negligible improvement of 1.4 dB SI-SNRi in the reverberant FUSS dataset. We also find adversarial PIT to be effective at reducing spectral holes, ubiquitous in mask-based separation models, which highlights the potential relevance of adversarial losses for source separation.
【6】 Spoken Term Detection and Relevance Score Estimation using Dot-Product of Pronunciation Embeddings
标题：基于发音嵌入点积的口语词检测和相关度得分估计
链接：https://arxiv.org/abs/2210.11895
* 与cs.SD语音【3】为同一篇
作者：Jan Švec,Luboš Šmídl,Josef V. Psutka,Aleš Pražák机构：Department of Cybernetics, University of West Bohemia, Pilsen, Czech Republic摘要：本文提出了一种基于深度LSTM网络的大型口语语料库中的口语词检测方法。该工作基于先前使用暹罗神经网络进行STD的方法，并自然地将其扩展到直接定位口语术语并估计其相关性得分。由音素识别器生成的音素混淆网络由深度LSTM网络处理，该深度LSTM网络将混淆网络的每个片段投影到嵌入空间中。使用另一个深度LSTM网络将搜索项投影到相同的嵌入空间中。然后使用嵌入空间中的简单点积来计算相关性分数，并使用S形函数来校准以预测发生概率。然后从输出概率序列估计搜索项的位置。深度LSTM网络是以自监督的方式从词和音素水平上的成对识别假设训练的。在英语和捷克语的MALACH数据上对该方法进行了实验评估。摘要：The paper describes a novel approach to Spoken Term Detection (STD) in large spoken archives using deep LSTM networks. The work is based on the previous approach of using Siamese neural networks for STD and naturally extends it to directly localize a spoken term and estimate its relevance score. The phoneme confusion network generated by a phoneme recognizer is processed by the deep LSTM network which projects each segment of the confusion network into an embedding space. The searched term is projected into the same embedding space using another deep LSTM network. The relevance score is then computed using a simple dot-product in the embedding space and calibrated using a sigmoid function to predict the probability of occurrence. The location of the searched term is then estimated from the sequence of output probabilities. The deep LSTM networks are trained in a self-supervised manner from paired recognition hypotheses on word and phoneme levels. The method is experimentally evaluated on MALACH data in English and Czech languages.
【7】 Deep LSTM Spoken Term Detection using Wav2Vec 2.0 Recognizer
标题：基于Wav2Vec 2.0识别器的深度LSTM语音检测
链接：https://arxiv.org/abs/2210.11885
* 与cs.SD语音【4】为同一篇
作者：Jan Švec,Jan Lehečka,Luboš Šmídl机构：Luboˇs ˇSm´ıdlDepartment of Cybernetics, University of West Bohemia摘要：近年来，端到端语音识别系统的性能已经超过了标准的混合DNN-HMM语音识别器。其中一个很有前途的方法是字素Wav2Vec2.0模型，它使用自监督预训练方法结合微调语音识别器的迁移学习。由于缺乏发音词汇和语言模型，该方法适用于不容易或几乎不可能获得这些模型的任务。 本文使用Wav 2 Vec语音识别器对大量的口语文档进行口语术语检测。该方法采用深度LSTM网络，该网络将识别的假设和搜索的术语映射到共享发音嵌入空间，在该空间中术语出现和分配的分数容易计算。 本文描述了一种自举方法，该方法允许将DNN-HMM混合ASR的传统发音词汇中包含的知识转移到基于字形的Wav 2 Vec的上下文中。在英语和捷克语的MALACH数据上，该方法的性能明显优于先前发表的基于DNN-HMM混合ASR和音素识别器相结合的系统。摘要：In recent years, the standard hybrid DNN-HMM speech recognizers are outperformed by the end-to-end speech recognition systems. One of the very promising approaches is the grapheme Wav2Vec 2.0 model, which uses the self-supervised pretraining approach combined with transfer learning of the fine-tuned speech recognizer. Since it lacks the pronunciation vocabulary and language model, the approach is suitable for tasks where obtaining such models is not easy or almost impossible. In this paper, we use the Wav2Vec speech recognizer in the task of spoken term detection over a large set of spoken documents. The method employs a deep LSTM network which maps the recognized hypothesis and the searched term into a shared pronunciation embedding space in which the term occurrences and the assigned scores are easily computed. The paper describes a bootstrapping approach that allows the transfer of the knowledge contained in traditional pronunciation vocabulary of DNN-HMM hybrid ASR into the context of grapheme-based Wav2Vec. The proposed method outperforms the previously published system based on the combination of the DNN-HMM hybrid ASR and phoneme recognizer by a large margin on the MALACH data in both English and Czech languages.
【8】 A Textless Metric for Speech-to-Speech Comparison
标题：一种用于语音到语音比对的无文本度量
链接：https://arxiv.org/abs/2210.11835
* 与cs.SD语音【5】为同一篇
作者：Laurent Besacier,Swen Ribeiro,Olivier Galibert,Ioan Calapodescu机构：Naver Labs Europe, France, LNE, France摘要：本文提出了一种无文本的语音-语音比较度量，该度量允许将语音假设与语音参考进行比较，而无需退回到它们的文本转录本。我们利用最近提出的speech 2unit编码器（如HuBERT）将语音话语伪转录为离散的声学单元，并提出了一种简单的神经结构，该结构学习基于语音的度量，该度量与基于文本的度量很好地相关。这样的无文本度量最终可能对于语音到语音翻译评估（对于口头语言或没有可靠ASR系统可用的语言）是感兴趣的。摘要：This paper proposes a textless speech-to-speech comparison metric that allows comparing a speech hypothesis with a speech reference without falling-back to their text transcripts. We leverage recently proposed speech2unit encoders (such as HuBERT) to pseudo-transcribe the speech utterances into discrete acoustic units and propose a simple neural architecture that learns a speech-based metric which correlates well with its text-based counterpart. Such a textless metric could ultimately be interesting for speech-to-speech translation evaluation (for oral languages or languages with no reliable ASR system available).
【9】 Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
标题：语音自我监督学习中声道清晰度的证据
链接：https://arxiv.org/abs/2210.11723
* 与cs.SD语音【6】为同一篇
作者：Cheol Jun Cho,Peter Wu,Abdelrahman Mohamed,Gopala K. Anumanchipalli机构：UC Berkeley, EECS, CA, Meta AI摘要：许多用于语音的自监督学习（SSL）模型已经被提出用于语音表示的预训练模型，并且最近的SSL模型在各种下游任务中非常成功。为了理解这种效用，先前的工作探索语音模型的表示，以揭示哪些和如何语音相关的信息被编码在学习的表示中。虽然编码特性已经从声学、语音学和语义学的角度得到了广泛的探讨，但语音产生的物理基础尚未得到充分的关注。为了弥补这一差距，我们进行了一项综合分析，将言语表征与电磁发音描记术（EMA）测量的发音轨迹联系起来。我们的分析是基于线性探测方法，我们测量发音评分作为线性映射到EMA的平均相关性。我们分析了从SU-PERB基准测试的排行榜中选出的一组SSL模型，并对两个主要模型Wav 2 Vec 2.0和HuBERT进行了进一步的详细分析。令人惊讶的是，来自最近的语音SSL模型的表示与EMA迹线高度相关（最佳：r = 0.81），并且仅5分钟就足以训练具有高性能的线性模型（r = 0.77）。我们的研究结果表明，SSL模型学习与连续发音紧密结合，并提供了一个新的见解语音SSL。摘要：Numerous self-supervised learning (SSL) models for speech have been proposed for pre-training models of speech representations, and recent SSL models are very successful in diverse downstream tasks. To understand such utilities, previous works probe representations of speech models to reveal which & how speech related information is encoded in the learned representations. While encoding properties have been extensively explored from the perspective of acoustics, phonetics, and semantics, the physical grounding by speech production has not yet received full attention. To bridge this gap, we conduct a comprehensive analysis to link speech representations to articulatory trajectories measured by electromagnetic articulography (EMA). Our analysis is based on a linear probing approach where we measure articulatory score as an average correlation of linear mapping to EMA. We analyze a set of SSL models selected from the leaderboard of the SU- PERB benchmark and perform further detailed analyses on two major models, Wav2Vec 2.0 and HuBERT. Surprisingly, representations from the recent speech SSL models are highly correlated with EMA traces (best: r = 0.81), and only 5 minutes were sufficient to train a linear model with high performance (r = 0.77). Our findings suggest that SSL models learn to closely align with continuous articulations and provide a novel insight into speech SSL.
【10】 Adaptive re-calibration of channel-wise features for Adversarial Audio Classification
标题：用于对抗性音频分类的通道特征的自适应重新校准
链接：https://arxiv.org/abs/2210.11722
* 与cs.SD语音【7】为同一篇
作者：Vardhan Dongre,Abhinav Thimma Reddy,Nikhitha Reddeddy机构：[,] University of Illinois Urbana-Champaign备注：7 pages, 8 figures, 4 tables摘要：与DeepFake图像和视频不同，DeepFake音频从检测角度来说相对较少被探索，并且存在的用于合成语音分类的解决方案或者使用复杂网络或者不推广到使用不同的生成和基于优化的方法获得的不同种类的合成语音。通过这项工作，我们提出了一种使用注意力特征融合的通道式特征重新校准方法，用于合成语音检测，并比较了其与不同检测方法（包括End2End模型和基于Resnet的模型）在使用文语转换和声码器系统（如WaveNet、WaveRNN、Tactotron和WaveGlow）生成的合成语音上的性能。我们还在我们的Resnet模型中实验了挤压激励（SE）模块，发现这种组合能够获得更好的性能。除了分析之外，我们还证明了使用注意特征融合技术的线性频率倒谱系数（LFCC）和梅尔频率倒谱系数（MFCC）的组合创建了更好的输入特征表示，这可以帮助甚至更简单的模型在合成语音分类任务上很好地推广。我们的模型（使用特征融合的基于Resnet的）在Fake or Real（FoR）数据集上训练，并且能够在FoR数据上实现95%的测试准确度，并且在适应该框架后，对于我们使用不同的生成模型生成的样本，能够实现平均90%的准确度。摘要：DeepFake Audio, unlike DeepFake images and videos, has been relatively less explored from detection perspective, and the solutions which exist for the synthetic speech classification either use complex networks or dont generalize to different varieties of synthetic speech obtained using different generative and optimization-based methods. Through this work, we propose a channel-wise recalibration of features using attention feature fusion for synthetic speech detection and compare its performance against different detection methods including End2End models and Resnet-based models on synthetic speech generated using Text to Speech and Vocoder systems like WaveNet, WaveRNN, Tactotron, and WaveGlow. We also experiment with Squeeze Excitation (SE) blocks in our Resnet models and found that the combination was able to get better performance. In addition to the analysis, we also demonstrate that the combination of Linear frequency cepstral coefficients (LFCC) and Mel Frequency cepstral coefficients (MFCC) using the attentional feature fusion technique creates better input features representations which can help even simpler models generalize well on synthetic speech classification tasks. Our models (Resnet based using feature fusion) trained on Fake or Real (FoR) dataset and were able to achieve 95% test accuracy with the FoR data, and an average of 90% accuracy with samples we generated using different generative models after adapting this framework.
【11】 Improving Semi-supervised End-to-end Automatic Speech Recognition using CycleGAN and Inter-domain Losses
标题：利用CycleGAN和域间损失改进半监督端到端自动语音识别
链接：https://arxiv.org/abs/2210.11642
* 与cs.SD语音【8】为同一篇
作者：Chia-Yu Li,Ngoc Thang Vu机构：Institute for Natural Language Processing (IMS), University of Stuttgart, Germany备注：6 pages + 2 references, 6 figures, accepted by SLT2022摘要：提出了一种结合CycleGAN和域间损失的半监督端到端自动语音识别方法。域间丢失的目标是使用共享网络提取语音和文本输入的中间共享表示。CycleGAN使用循环一致性损失和身份映射损失来在从一个域转换到另一个域之后保持输入特征的相关特征。因此，两种方法都适合于在不成对的语音-文本输入上训练端到端模型。本文利用域间丢失和CycleGAN的优点，实现非成对语音和文本输入的更好共享表示，从而改进语音到文本的映射。在WSJ eval 92和Voxforge（非英语）上的实验结果表明，与基线相比，该算法的字符错误率降低了8~ 8. 5%，在LibriSpeech test_clean上的实验结果也显示了明显的改善。摘要：We propose a novel method that combines CycleGAN and inter-domain losses for semi-supervised end-to-end automatic speech recognition. Inter-domain loss targets the extraction of an intermediate shared representation of speech and text inputs using a shared network. CycleGAN uses cycle-consistent loss and the identity mapping loss to preserve relevant characteristics of the input feature after converting from one domain to another. As such, both approaches are suitable to train end-to-end models on unpaired speech-text inputs. In this paper, we exploit the advantages from both inter-domain loss and CycleGAN to achieve better shared representation of unpaired speech and text inputs and thus improve the speech-to-text mapping. Our experimental results on the WSJ eval92 and Voxforge (non English) show 8~8.5% character error rate reduction over the baseline, and the results on LibriSpeech test_clean also show noticeable improvement.
机器翻译，仅供参考
每日语音速递