今日论文合集：cs.SD语音16篇，eess.AS音频处理18篇。
cs.SD语音
【1】 An objective evaluation of Hearing Aids and DNN-based speech enhancement  in complex acoustic scenes标题：复杂声学场景中助听器和DNN语音增强的客观评价链接：https://arxiv.org/abs/2307.12888
作者：Enric Gusó,Joanna Luberadzka,Martí Baig,Umut Sayin Saraç,Xavier Serra备注：Accepted to WASPAA23摘要：我们调查的客观性能的五个高端商用助听器（HA）设备相比，DNN为基础的语音增强算法在复杂的声学环境。为此，我们测量单个HA设备的HRTF以合成双耳数据集，用于训练两个最先进的因果和非因果DNN增强模型。然后，我们生成一个评估集的现实的语音在噪声的情况下使用高保真度混响扬声器设置和记录与KU100假人头佩戴的HA设备，无论是和没有传统的HA算法，应用DNN增强器后者。我们发现，基于DNN的增强优于HA算法的噪声抑制和客观的清晰度指标。摘要：We investigate the objective performance of five high-end commercially available Hearing Aid (HA) devices compared to DNN-based speech enhancement algorithms in complex acoustic environments. To this end, we measure the HRTFs of a single HA device to synthesize a binaural dataset for training two state-of-the-art causal and non-causal DNN enhancement models. We then generate an evaluation set of realistic speech-in-noise situations using an Ambisonics loudspeaker setup and record with a KU100 dummy head wearing each of the HA devices, both with and without the conventional HA algorithms, applying the DNN enhancers to the latter. We find that the DNN-based enhancement outperforms the HA algorithms in terms of noise suppression and objective intelligibility metrics.
【2】 Code-Switched Urdu ASR for Noisy Telephonic Environment using Data  Centric Approach with Hybrid HMM and CNN-TDNN标题：基于混合隐马尔可夫模型和CNN-TDNN的数据中心编码交换URDU ASR链接：https://arxiv.org/abs/2307.12759
作者：Muhammad Danyal Khan,Raheem Ali,Arshad Aziz备注：32 pages, 19 figures, 2 tables, preprint摘要：呼叫中心有大量的音频数据，可用于实现有价值的业务洞察和电话的转录是手动繁琐的任务.有效的自动语音识别系统可以准确地转录这些呼叫，以便通过呼叫历史中的特定上下文和内容进行轻松搜索，从而允许自动呼叫监控，通过关键词搜索和情感分析提高QoS。呼叫中心ASR要求更强的鲁棒性，因为电话环境通常是嘈杂的。此外，有许多濒临灭绝的低资源语言可以通过自动语音识别技术得以保存。乌尔都语是世界上使用最广泛的语言，全球有231，295，440个乌尔都语仍然是ASR中资源有限的语言。区域呼叫中心的对话使用当地语言，混合了英语数字和专业术语，通常会导致“代码转换”问题。因此，本文描述了一个资源高效的自动语音识别/语音转文本系统在嘈杂的呼叫中心环境中使用链混合HMM和CNN-TDNN代码交换乌尔都语语言的实现框架。使用混合HMM-DNN方法使我们能够利用神经网络的优势，具有较少的标记数据。将CNN与TDNN一起添加已经显示出在噪声环境中更好地工作，这是由于CNN的额外频率维度从噪声语音中捕获额外信息，从而提高准确性。我们从各种开放来源收集数据，并在分析其一般背景和乌尔都语内容以及其他语言（主要是英语）的常用单词后标记了一些未标记的数据，并且能够在嘈杂和干净的环境中以孤立的单词或数字以及连续的自发语音实现5.2%的WER。摘要：Call Centers have huge amount of audio data which can be used for achieving valuable business insights and transcription of phone calls is manually tedious task. An effective Automated Speech Recognition system can accurately transcribe these calls for easy search through call history for specific context and content allowing automatic call monitoring, improving QoS through keyword search and sentiment analysis. ASR for Call Center requires more robustness as telephonic environment are generally noisy. Moreover, there are many low-resourced languages that are on verge of extinction which can be preserved with help of Automatic Speech Recognition Technology. Urdu is the $10^{th}$ most widely spoken language in the world, with 231,295,440 worldwide still remains a resource constrained language in ASR. Regional call-center conversations operate in local language, with a mix of English numbers and technical terms generally causing a "code-switching" problem. Hence, this paper describes an implementation framework of a resource efficient Automatic Speech Recognition/ Speech to Text System in a noisy call-center environment using Chain Hybrid HMM and CNN-TDNN for Code-Switched Urdu Language. Using Hybrid HMM-DNN approach allowed us to utilize the advantages of Neural Network with less labelled data. Adding CNN with TDNN has shown to work better in noisy environment due to CNN's additional frequency dimension which captures extra information from noisy speech, thus improving accuracy. We collected data from various open sources and labelled some of the unlabelled data after analysing its general context and content from Urdu language as well as from commonly used words from other languages, primarily English and were able to achieve WER of 5.2% with noisy as well as clean environment in isolated words or numbers as well as in continuous spontaneous speech.
【3】 Online Continual Learning in Keyword Spotting for Low-Resource Devices  via Pooling High-Order Temporal Statistics标题：汇聚高阶时间统计量的低资源设备关键字在线连续学习链接：https://arxiv.org/abs/2307.12660
作者：Umberto Michieli,Pablo Peso Parada,Mete Ozay备注：INTERSPEECH 2023摘要：嵌入式设备上的关键字识别（KWS）模型应该快速适应新的用户定义的单词，而不会忘记以前的单词。嵌入式设备具有有限的存储和计算资源，因此，它们不能保存样本或更新大型模型。我们考虑了嵌入式在线持续学习（EOCL）的设置，其中具有冻结主干的KWS模型被训练为从非重复的样本流中增量地识别新词，每次看到一个。为此，我们提出了时间感知池（TAP），它构建了一个丰富的特征空间计算高阶矩的语音特征提取的预训练的骨干。我们的方法，TAP-SLDA，更新高斯模型，为每个类的丰富的特征空间，以有效地使用音频表示。在实验分析中，TAP-SLDA在几个设置，主干和基线上优于竞争对手，在GSC数据集上带来了11.3%的相对平均增益。摘要：Keyword Spotting (KWS) models on embedded devices should adapt fast to new user-defined words without forgetting previous ones. Embedded devices have limited storage and computational resources, thus, they cannot save samples or update large models. We consider the setup of embedded online continual learning (EOCL), where KWS models with frozen backbone are trained to incrementally recognize new words from a non-repeated stream of samples, seen one at a time. To this end, we propose Temporal Aware Pooling (TAP) which constructs an enriched feature space computing high-order moments of speech features extracted by a pre-trained backbone. Our method, TAP-SLDA, updates a Gaussian model for each class on the enriched feature space to effectively use audio representations. In experimental analyses, TAP-SLDA outperforms competitors on several setups, backbones, and baselines, bringing a relative average gain of 11.3% on the GSC dataset.
【4】 A Model for Every User and Budget: Label-Free and Personalized  Mixed-Precision Quantization标题：一种面向每个用户和预算的模型：无标签和个性化的混合精度量化链接：https://arxiv.org/abs/2307.12659
作者：Edward Fish,Umberto Michieli,Mete Ozay备注：INTERSPEECH 2023摘要：自动语音识别（ASR）的最新进展已经产生了大型AI模型，这些模型对于在移动设备中的部署变得不切实际。模型量化对于产生压缩的通用模型是有效的，然而这样的模型可以仅部署到感兴趣的受限子域。我们表明，ASR模型可以在量化过程中个性化，而依赖于只是一小组未标记的样本从目标域。为此，我们提出了myQASR，一种混合精度的量化方法，生成量身定制的量化方案，为不同的用户在任何内存的要求，没有微调。myQASR通过分析全精度激活值自动评估网络层的量化敏感性。然后，我们能够为任何预定的存储器预算生成个性化的混合精度量化方案。大规模ASR模型的结果显示了myQASR如何提高特定性别、语言和说话者的性能。摘要：Recent advancement in Automatic Speech Recognition (ASR) has produced large AI models, which become impractical for deployment in mobile devices. Model quantization is effective to produce compressed general-purpose models, however such models may only be deployed to a restricted sub-domain of interest. We show that ASR models can be personalized during quantization while relying on just a small set of unlabelled samples from the target domain. To this end, we propose myQASR, a mixed-precision quantization method that generates tailored quantization schemes for diverse users under any memory requirement with no fine-tuning. myQASR automatically evaluates the quantization sensitivity of network layers by analysing the full-precision activation values. We are then able to generate a personalised mixed-precision quantization scheme for any pre-determined memory budget. Results for large-scale ASR models show how myQASR improves performance for specific genders, languages, and speakers.
【5】 Robust Automatic Speech Recognition via WavAugment Guided Phoneme  Adversarial Training标题：基于WavAugment引导的音素对抗训练的稳健自动语音识别链接：https://arxiv.org/abs/2307.12498
作者：Gege Qi,Yuefeng Chen,Xiaofeng Mao,Xiaojun Jia,Ranjie Duan,Rong Zhang,Hui Xue摘要：开发一个实用的鲁棒的自动语音识别（ASR）是一个挑战，因为模型不仅应该保持原始的性能干净的样本，但也实现一致的功效下，小的体积扰动和大的域移位。为了解决这个问题，我们提出了一种新的WavAugment引导音素对抗训练（wapat）。WAPAT使用音素空间中的对抗示例作为增强，以使模型对音素表示中的微小波动不变，并保持干净样本上的性能。此外，wapat利用增强样本的音素表示来指导对手的生成，这有助于找到更稳定和多样化的梯度方向，从而提高泛化能力。大量的实验证明了wapat在端到端语音挑战基准（ESB）上的有效性。值得注意的是，SpeechLM-wapat在ESB上的WER降低了6.28%，超过了原始模型，达到了新的最先进水平。摘要：Developing a practically-robust automatic speech recognition (ASR) is challenging since the model should not only maintain the original performance on clean samples, but also achieve consistent efficacy under small volume perturbations and large domain shifts. To address this problem, we propose a novel WavAugment Guided Phoneme Adversarial Training (wapat). wapat use adversarial examples in phoneme space as augmentation to make the model invariant to minor fluctuations in phoneme representation and preserve the performance on clean samples. In addition, wapat utilizes the phoneme representation of augmented samples to guide the generation of adversaries, which helps to find more stable and diverse gradient-directions, resulting in improved generalization. Extensive experiments demonstrate the effectiveness of wapat on End-to-end Speech Challenge Benchmark (ESB). Notably, SpeechLM-wapat outperforms the original model by 6.28% WER reduction on ESB, achieving the new state-of-the-art.
【6】 SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic  Spaces标题：残片：语音和语音空间的语音对比表示链接：https://arxiv.org/abs/2307.12445
作者：Ivan Vallés-Pérez,Grzegorz Beringer,Piotr Bilinski,Gary Cook,Roberto Barra-Chicote备注：In proceedings of the 26th European Conference on Artificial Intelligence ECAI 2023. 8 pages + 1 appendix page摘要：文献中的许多例子证明，深度学习模型能够很好地处理多模态数据。最近，CLIP已经使深度学习系统能够学习图像和文本描述之间的共享潜在空间，在下游任务中具有出色的零或Few-Shot结果。在本文中，我们探讨了同样的想法，提出了CLIP，但适用于语音域，语音和声学空间通常共存。我们训练了一个基于CLIP的模型，目的是学习语音和声学空间的共享表示。结果表明，该模型是敏感的语音变化，91%的分数下降时，随机替换20%的音素，同时提供了大量的鲁棒性对不同种类的噪声，与10%的性能下降时，混合音频与75%的高斯噪声。我们还提供了经验证据，表明所得到的嵌入对于各种下游应用是有用的，例如可懂度评估和在语音生成任务中利用丰富的预训练语音嵌入的能力。最后，我们讨论了潜在的应用与有趣的影响，语音生成和识别领域。摘要：Numerous examples in the literature proved that deep learning models have the ability to work well with multimodal data. Recently, CLIP has enabled deep learning systems to learn shared latent spaces between images and text descriptions, with outstanding zero- or few-shot results in downstream tasks. In this paper we explore the same idea proposed by CLIP but applied to the speech domain, where the phonetic and acoustic spaces usually coexist. We train a CLIP-based model with the aim to learn shared representations of phonetic and acoustic spaces. The results show that the proposed model is sensible to phonetic changes, with a 91% of score drops when replacing 20% of the phonemes at random, while providing substantial robustness against different kinds of noise, with a 10% performance drop when mixing the audio with 75% of Gaussian noise. We also provide empirical evidence showing that the resulting embeddings are useful for a variety of downstream applications, such as intelligibility evaluation and the ability to leverage rich pre-trained phonetic embeddings in speech generation task. Finally, we discuss potential applications with interesting implications for the speech generation and recognition fields.
【7】 Self-Supervised Learning for Audio-Based Emotion Recognition标题：基于音频的情感识别中的自监督学习链接：https://arxiv.org/abs/2307.12343
作者：Peranut Nimitsurachat,Peter Washington备注：8 pages, 9 figures, submitted to IEEE Transactions on Affective Computing摘要：使用音频输入数据的情绪识别模型可以实现在精神保健、营销、游戏和社交媒体分析中应用的交互系统的开发。虽然使用音频数据的情感计算领域很丰富，但实现一致高性能模型的主要障碍是缺乏可用的训练标签。自监督学习（SSL）是一系列方法，可以通过预测数据本身的属性来学习，尽管缺乏监督标签。为了理解自监督学习对于基于音频的情感识别的效用，我们已经将自监督学习预训练应用于来自CMU-MOSEI的声学模态的情感分类。不像以前的论文，实验与原始的声学数据，我们的技术已被应用到编码的声学数据。我们的模型首先被预训练，以揭示随机掩蔽的时间戳的声学数据。然后使用注释数据的小样本对预训练模型进行微调。然后，通过几个评估指标对具有相同骨干架构的基线深度学习模型评估最终模型的性能。我们发现，自我监督学习一致地提高了模型在所有指标的性能。这项工作显示了自我监督学习在情感计算中的效用，证明了自我监督学习在训练样本的数量很少时是最有用的，并且对于更容易分类的情绪，如快乐，悲伤和愤怒的效果最明显。这项工作进一步证明了自监督学习在应用于嵌入式特征表示时有效，而不是在原始输入空间上进行预训练的传统方法。摘要：Emotion recognition models using audio input data can enable the development of interactive systems with applications in mental healthcare, marketing, gaming, and social media analysis. While the field of affective computing using audio data is rich, a major barrier to achieve consistently high-performance models is the paucity of available training labels. Self-supervised learning (SSL) is a family of methods which can learn despite a scarcity of supervised labels by predicting properties of the data itself. To understand the utility of self-supervised learning for audio-based emotion recognition, we have applied self-supervised learning pre-training to the classification of emotions from the CMU- MOSEI's acoustic modality. Unlike prior papers that have experimented with raw acoustic data, our technique has been applied to encoded acoustic data. Our model is first pretrained to uncover the randomly-masked timestamps of the acoustic data. The pre-trained model is then fine-tuned using a small sample of annotated data. The performance of the final model is then evaluated via several evaluation metrics against a baseline deep learning model with an identical backbone architecture. We find that self-supervised learning consistently improves the performance of the model across all metrics. This work shows the utility of self-supervised learning for affective computing, demonstrating that self-supervised learning is most useful when the number of training examples is small, and that the effect is most pronounced for emotions which are easier to classify such as happy, sad and anger. This work further demonstrates that self-supervised learning works when applied to embedded feature representations rather than the traditional approach of pre-training on the raw input space.
【8】 A meta learning scheme for fast accent domain expansion in Mandarin  speech recognition标题：汉语语音识别中一种快速扩展重音域的元学习方法链接：https://arxiv.org/abs/2307.12262
作者：Ziwei Zhu,Changhao Shan,Bihong Zhang,Jian Yu摘要：口语在普通话和口音方面表现出显著的差异。尽管普通话自动语音识别（ASR）已经取得了很高的性能，但口音识别仍然是一个具有挑战性的任务。在本文中，我们介绍了元学习技术的快速口音域扩展在普通话语音识别，这扩大了口音的领域，而不会恶化的性能的普通话ASR。元学习或lear-to-learning可以学习多个领域中的一般关系，而不仅仅是为了过拟合某个特定领域。因此，我们选择了元学习作为领域扩展任务。这种更重要的学习将提高重音域扩展任务的性能。我们结合了Meta学习和模型参数冻结的方法，使得识别性能在不同情况下更加稳定，训练速度提高了20%左右。我们的方法显着优于其他方法约3%，相对于重音域扩展任务。在普通话测试集不变的情况下，与基线模型相比，该模型提高了37%。此外，它也证明了该方法是有效的，在口音测试集的相对性能提高了4%。摘要：Spoken languages show significant variation across mandarin and accent. Despite the high performance of mandarin automatic speech recognition (ASR), accent ASR is still a challenge task. In this paper, we introduce meta-learning techniques for fast accent domain expansion in mandarin speech recognition, which expands the field of accents without deteriorating the performance of mandarin ASR. Meta-learning or learn-to-learn can learn general relation in multi domains not only for over-fitting a specific domain. So we select meta-learning in the domain expansion task. This more essential learning will cause improved performance on accent domain extension tasks. We combine the methods of meta learning and freeze of model parameters, which makes the recognition performance more stable in different cases and the training faster about 20%. Our approach significantly outperforms other methods about 3% relatively in the accent domain expansion task. Compared to the baseline model, it improves relatively 37% under the condition that the mandarin test set remains unchanged. In addition, it also proved this method to be effective on a large amount of data with a relative performance improvement of 4% on the accent test set.
【9】 Signal Reconstruction from Mel-spectrogram Based on Bi-level Consistency  of Full-band Magnitude and Phase标题：基于全带幅相两级一致性的Mel谱图信号重构链接：https://arxiv.org/abs/2307.12232
作者：Yoshiki Masuyama,Natsuki Ueno,Nobutaka Ono备注：Accepted to IEEE WASPAA 2023摘要：我们提出了一种基于优化的方法，用于从低维频谱表示，如梅尔频谱图重建时域信号。相位重建已经被研究以从全频带短时傅里叶变换（STFT）幅度重建时域信号。Griffin-Lim算法（GLA）由于只依赖于STFT的冗余性，适用于各种音频信号，因而得到了广泛的应用。在本文中，我们联合重建的全频带幅度和相位，考虑到时域信号，它的STFT系数，和它的梅尔频谱图之间的双电平关系。所提出的方法被配制成一个严格的优化问题，并估计全频带幅度的基础上使用的GLA的标准。我们的实验证明了所提出的方法对语音，音乐和环境信号的有效性。摘要：We propose an optimization-based method for reconstructing a time-domain signal from a low-dimensional spectral representation such as a mel-spectrogram. Phase reconstruction has been studied to reconstruct a time-domain signal from the full-band short-time Fourier transform (STFT) magnitude. The Griffin-Lim algorithm (GLA) has been widely used because it relies only on the redundancy of STFT and is applicable to various audio signals. In this paper, we jointly reconstruct the full-band magnitude and phase by considering the bi-level relationships among the time-domain signal, its STFT coefficients, and its mel-spectrogram. The proposed method is formulated as a rigorous optimization problem and estimates the full-band magnitude based on the criterion used in GLA. Our experiments demonstrate the effectiveness of the proposed method on speech, music, and environmental signals.
【10】 Exploring the Integration of Speech Separation and Recognition with  Self-Supervised Learning Representation标题：语音分离和识别与自监督学习表示相结合的探索链接：https://arxiv.org/abs/2307.12231
作者：Yoshiki Masuyama,Xuankai Chang,Wangyou Zhang,Samuele Cornell,Zhong-Qiu Wang,Nobutaka Ono,Yanmin Qian,Shinji Watanabe备注：Accepted to IEEE WASPAA 2023摘要：神经语音分离技术已经取得了显著的进展，将其与自动语音识别技术相结合是实现多说话人自动语音识别的重要方向。这项工作提供了一个有见地的调查语音分离混响和噪声混响的情况下作为ASR前端。详细地说，我们探讨了多通道分离方法，基于掩模的波束成形和复杂的频谱映射，以及在ASR后端模型中使用的最佳功能。我们采用最近的自监督学习表示（SSLR）作为一个功能，并提高了识别性能的情况下，过滤器组功能。为了进一步提高多说话人识别的性能，我们提出了一个精心设计的训练策略，集成语音分离和识别SSLR。所提出的集成使用TF-GridNet为基础的复杂的频谱映射和WavLM为基础的SSLR实现了2.5%的字错误率混响WHAMR！测试集，显著优于现有的基于掩模的MVDR波束成形和滤波器组集成（28.9%）。摘要：Neural speech separation has made remarkable progress and its integration with automatic speech recognition (ASR) is an important direction towards realizing multi-speaker ASR. This work provides an insightful investigation of speech separation in reverberant and noisy-reverberant scenarios as an ASR front-end. In detail, we explore multi-channel separation methods, mask-based beamforming and complex spectral mapping, as well as the best features to use in the ASR back-end model. We employ the recent self-supervised learning representation (SSLR) as a feature and improve the recognition performance from the case with filterbank features. To further improve multi-speaker recognition performance, we present a carefully designed training strategy for integrating speech separation and recognition with SSLR. The proposed integration using TF-GridNet-based complex spectral mapping and WavLM-based SSLR achieves a 2.5% word error rate in reverberant WHAMR! test set, significantly outperforming an existing mask-based MVDR beamforming and filterbank integration (28.9%).
【11】 Adversarial Agents For Attacking Inaudible Voice Activated Devices标题：攻击听不见的声控设备的对抗性代理链接：https://arxiv.org/abs/2307.12204
作者：Forrest McKee,David Noever摘要：我们对语音激活设备上的听不见攻击的分析证实了7.6的惊人风险系数（满分10分），强调了NIST国家漏洞数据库（NVD）独立评分的重大安全漏洞。我们的基线网络模型展示了一个场景，其中攻击者使用听不见的语音命令来获得未经授权的访问机密信息的安全笔记本电脑。我们在此基线网络模型上模拟了许多攻击场景，揭示了大规模利用互联设备通过物理访问发现和拥有特权信息的潜力，而无需添加新硬件或增强设备技能。使用微软的CyberBattleSim框架，我们评估了六种强化学习算法，发现Deep-Q学习和利用被证明是最佳的，可以在更少的步骤中快速拥有所有节点。我们的研究结果强调了在不断扩大的数字环境中理解非传统网络和新的网络安全措施的迫切需要，特别是那些以移动设备，语音激活和非线性麦克风为特征的网络，这些网络容易受到恶意行为者在近超声波或听不见的范围内进行隐形攻击的影响。到2024年，这种新的攻击面可能涵盖比地球上的人更多的数字语音助理，但提供的补救措施比传统的修补或固件修复更少，因为听不见的攻击固有地来自麦克风设计和数字信号处理。摘要：Our analysis of inaudible attacks on voice-activated devices confirms the alarming risk factor of 7.6 out of 10, underlining significant security vulnerabilities scored independently by NIST National Vulnerability Database (NVD). Our baseline network model showcases a scenario in which an attacker uses inaudible voice commands to gain unauthorized access to confidential information on a secured laptop. We simulated many attack scenarios on this baseline network model, revealing the potential for mass exploitation of interconnected devices to discover and own privileged information through physical access without adding new hardware or amplifying device skills. Using Microsoft's CyberBattleSim framework, we evaluated six reinforcement learning algorithms and found that Deep-Q learning with exploitation proved optimal, leading to rapid ownership of all nodes in fewer steps. Our findings underscore the critical need for understanding non-conventional networks and new cybersecurity measures in an ever-expanding digital landscape, particularly those characterized by mobile devices, voice activation, and non-linear microphones susceptible to malicious actors operating stealth attacks in the near-ultrasound or inaudible ranges. By 2024, this new attack surface might encompass more digital voice assistants than people on the planet yet offer fewer remedies than conventional patching or firmware fixes since the inaudible attacks arise inherently from the microphone design and digital signal processing.
【12】 Modality Confidence Aware Training for Robust End-to-End Spoken Language  Understanding标题：用于强健的端到端口语理解的通道置信度感知训练链接：https://arxiv.org/abs/2307.12134
作者：Suyoun Kim,Akshat Shrivastava,Duc Le,Ju Lin,Ozlem Kalinli,Michael L. Seltzer备注：INTERSPEECH 2023摘要：最近，从语音生成语义解析的端到端（E2E）口语理解（SLU）系统变得越来越有前途。这种方法使用单个模型，该模型利用来自预训练语音识别模型（ASR）的音频和文本表示，并且在设备上的流媒体场景中优于传统的流水线SLU系统。然而，E2E SLU系统仍然表现出弱点时，由于ASR转录错误的文本表示质量低。为了克服这个问题，我们提出了一种新的E2E SLU系统，通过融合音频和文本表示的基础上估计的模态置信度的ASR假设，增强了对ASR错误的鲁棒性。我们介绍两种新技术：1）编码ASR假设的质量的有效方法和2）将它们集成到E2E SLU模型中的有效方法。我们展示了STOP数据集的准确性改进，并分享了分析结果，以证明我们方法的有效性。摘要：End-to-end (E2E) spoken language understanding (SLU) systems that generate a semantic parse from speech have become more promising recently. This approach uses a single model that utilizes audio and text representations from pre-trained speech recognition models (ASR), and outperforms traditional pipeline SLU systems in on-device streaming scenarios. However, E2E SLU systems still show weakness when text representation quality is low due to ASR transcription errors. To overcome this issue, we propose a novel E2E SLU system that enhances robustness to ASR errors by fusing audio and text representations based on the estimated modality confidence of ASR hypotheses. We introduce two novel techniques: 1) an effective method to encode the quality of ASR hypotheses and 2) an effective approach to integrate them into E2E SLU models. We show accuracy improvements on STOP dataset and share the analysis to demonstrate the effectiveness of our approach.
【13】 Estimating speaker direction on a humanoid robot with binaural acoustic  signals标题：基于双耳声学信号的仿人机器人说话人方向估计链接：https://arxiv.org/abs/2307.12129
作者：Pranav Barot,Katja Mombaur,Ewen MacDonald摘要：为了在语音交互过程中实现类似人类的行为，人形机器人有必要估计人类说话者的位置。在这里，我们提出了一种方法来优化用于到达方向（DOA）估计的参数，同时也考虑到实时应用的人机交互场景。该方法被应用于双耳声源定位框架的仿人机器人头部。真实的数据收集和注释这项工作。通过蛮力方法和基于贝叶斯模型的方法进行优化，验证和讨论结果，并探讨对实时使用的延迟的影响。摘要：To achieve human-like behaviour during speech interactions, it is necessary for a humanoid robot to estimate the location of a human talker. Here, we present a method to optimize the parameters used for the direction of arrival (DOA) estimation, while also considering real-time applications for human-robot interaction scenarios. This method is applied to binaural sound source localization framework on a humanoid robotic head. Real data is collected and annotated for this work. Optimizations are performed via a brute force method and a Bayesian model based method, results are validated and discussed, and effects on latency for real-time use are also explored.
【14】 Transsion TSUP's speech recognition system for ASRU 2023 MADASR  Challenge标题：Transsion Tsup用于ASRU 2023 MADASR挑战赛的语音识别系统链接：https://arxiv.org/abs/2307.11778
作者：Xiaoxiao Li,Gaosheng Zhang,An Zhu,Weiyong Li,Shuming Fang,Xiaoyue Yang,Jianchao Zhu摘要：本文介绍了由Transsion语音理解处理团队（TSUP）为ASRU 2023 MADASR挑战赛开发的语音识别系统。该系统的重点是适应低资源印度语言的ASR模型，并涵盖了所有四个轨道的挑战。对于轨道1和2，声学模型利用具有联合CTC注意力训练损失的squeezeformer编码器和双向Transformer解码器。此外，在TLG波束搜索解码期间使用外部KenLM语言模型。对于轨道3和4，采用了预训练的IndicWhisper模型，并在挑战数据集和公开可用的数据集上进行了微调。还修改了耳语波束搜索解码，以支持外部KenLM语言模型，从而更好地利用挑战提供的附加文本。所提出的方法实现了字错误率（WER）的24.17%，24.43%，15.97%和15.97%的孟加拉语在四个轨道，和WER的19.61%，19.54%，15.48%和15.48%的Bhojpuri语言在四个轨道。这些结果证明了所提出的方法的有效性。摘要：This paper presents a speech recognition system developed by the Transsion Speech Understanding Processing Team (TSUP) for the ASRU 2023 MADASR Challenge. The system focuses on adapting ASR models for low-resource Indian languages and covers all four tracks of the challenge. For tracks 1 and 2, the acoustic model utilized a squeezeformer encoder and bidirectional transformer decoder with joint CTC-Attention training loss. Additionally, an external KenLM language model was used during TLG beam search decoding. For tracks 3 and 4, pretrained IndicWhisper models were employed and finetuned on both the challenge dataset and publicly available datasets. The whisper beam search decoding was also modified to support an external KenLM language model, which enabled better utilization of the additional text provided by the challenge. The proposed method achieved word error rates (WER) of 24.17%, 24.43%, 15.97%, and 15.97% for Bengali language in the four tracks, and WER of 19.61%, 19.54%, 15.48%, and 15.48% for Bhojpuri language in the four tracks. These results demonstrate the effectiveness of the proposed method.
【15】 Integration of Frame- and Label-synchronous Beam Search for Streaming  Encoder-decoder Speech Recognition标题：帧与标签同步波束搜索相结合的流编解码语音识别链接：https://arxiv.org/abs/2307.12767
作者：Emiru Tsunoo,Hayato Futami,Yosuke Kashiwagi,Siddhant Arora,Shinji Watanabe备注：Accepted for Interspeech 2023摘要：虽然基于帧的模型，如CTC和换能器，有亲和力的流式自动语音识别，其解码不使用未来的知识，这可能导致不正确的修剪。相反，基于标签的注意力编码器-解码器使用对输入的软注意力来缓解这个问题，而它倾向于高估偏向其训练域的标签，不像CTC。我们利用这些互补的属性，并建议集成帧和标签同步（F-/L-同步）解码交替执行一个单一的波束搜索方案。F-Sync解码领先用于逐块处理的解码，而L-Sync解码使用块内的前瞻未来帧来提供优先化假设。我们保持从两种解码方法的假设，以执行有效的修剪。实验表明，该算法实现了较低的错误率相比，其他搜索方法，同时对域外的情况下是强大的。摘要：Although frame-based models, such as CTC and transducers, have an affinity for streaming automatic speech recognition, their decoding uses no future knowledge, which could lead to incorrect pruning. Conversely, label-based attention encoder-decoder mitigates this issue using soft attention to the input, while it tends to overestimate labels biased towards its training domain, unlike CTC. We exploit these complementary attributes and propose to integrate the frame- and label-synchronous (F-/L-Sync) decoding alternately performed within a single beam-search scheme. F-Sync decoding leads the decoding for block-wise processing, while L-Sync decoding provides the prioritized hypotheses using look-ahead future frames within a block. We maintain the hypotheses from both decoding methods to perform effective pruning. Experiments demonstrate that the proposed search algorithm achieves lower error rates compared to the other search methods, while being robust against out-of-domain situations.
【16】 Self-refining of Pseudo Labels for Music Source Separation with Noisy  Labeled Data标题：带噪声标签数据的伪标签源分离自精化算法链接：https://arxiv.org/abs/2307.12576
作者：Junghyun Koo,Yunkee Chae,Chang-Bin Jeon,Kyogu Lee备注：24th International Society for Music Information Retrieval Conference (ISMIR 2023)摘要：音乐源分离（MSS）面临的挑战，由于有限的可用性，正确标记的个人乐器轨道。随着获取更大数据集以提高MSS性能的推动，遇到错误标记的单个仪器轨迹的不可避免性成为需要解决的重大挑战。本文介绍了一种自动化技术，用于细化部分错误标记的数据集中的标签。我们提出的自精炼技术，采用噪声标记的数据集，结果只有1%的精度下降，在多标签仪器识别相比，一个干净的标签数据集训练的分类器。该研究证明了在MSS模型训练中改进噪声标记数据的重要性，并表明利用改进的数据集可以从干净标记的数据集获得可比的结果。值得注意的是，在仅访问噪声数据集时，在自细化数据集上训练的MSS模型甚至优于在用在干净标签上训练的分类器细化的数据集上训练的MSS模型。摘要：Music source separation (MSS) faces challenges due to the limited availability of correctly-labeled individual instrument tracks. With the push to acquire larger datasets to improve MSS performance, the inevitability of encountering mislabeled individual instrument tracks becomes a significant challenge to address. This paper introduces an automated technique for refining the labels in a partially mislabeled dataset. Our proposed self-refining technique, employed with a noisy-labeled dataset, results in only a 1% accuracy degradation in multi-label instrument recognition compared to a classifier trained on a clean-labeled dataset. The study demonstrates the importance of refining noisy-labeled data in MSS model training and shows that utilizing the refined dataset leads to comparable results derived from a clean-labeled dataset. Notably, upon only access to a noisy dataset, MSS models trained on a self-refined dataset even outperform those trained on a dataset refined with a classifier trained on clean labels.
eess.AS音频处理
【1】 Integration of Frame- and Label-synchronous Beam Search for Streaming  Encoder-decoder Speech Recognition标题：帧与标签同步波束搜索相结合的流编解码语音识别链接：https://arxiv.org/abs/2307.12767
作者：Emiru Tsunoo,Hayato Futami,Yosuke Kashiwagi,Siddhant Arora,Shinji Watanabe备注：Accepted for Interspeech 2023摘要：虽然基于帧的模型，如CTC和换能器，有亲和力的流式自动语音识别，其解码不使用未来的知识，这可能导致不正确的修剪。相反，基于标签的注意力编码器-解码器使用对输入的软注意力来缓解这个问题，而它倾向于高估偏向其训练域的标签，不像CTC。我们利用这些互补的属性，并建议集成帧和标签同步（F-/L-同步）解码交替执行一个单一的波束搜索方案。F-Sync解码领先用于逐块处理的解码，而L-Sync解码使用块内的前瞻未来帧来提供优先化假设。我们保持从两种解码方法的假设，以执行有效的修剪。实验表明，该算法实现了较低的错误率相比，其他搜索方法，同时对域外的情况下是强大的。摘要：Although frame-based models, such as CTC and transducers, have an affinity for streaming automatic speech recognition, their decoding uses no future knowledge, which could lead to incorrect pruning. Conversely, label-based attention encoder-decoder mitigates this issue using soft attention to the input, while it tends to overestimate labels biased towards its training domain, unlike CTC. We exploit these complementary attributes and propose to integrate the frame- and label-synchronous (F-/L-Sync) decoding alternately performed within a single beam-search scheme. F-Sync decoding leads the decoding for block-wise processing, while L-Sync decoding provides the prioritized hypotheses using look-ahead future frames within a block. We maintain the hypotheses from both decoding methods to perform effective pruning. Experiments demonstrate that the proposed search algorithm achieves lower error rates compared to the other search methods, while being robust against out-of-domain situations.
【2】 Self-refining of Pseudo Labels for Music Source Separation with Noisy  Labeled Data标题：带噪声标签数据的伪标签源分离自精化算法链接：https://arxiv.org/abs/2307.12576
作者：Junghyun Koo,Yunkee Chae,Chang-Bin Jeon,Kyogu Lee备注：24th International Society for Music Information Retrieval Conference (ISMIR 2023)摘要：音乐源分离（MSS）面临的挑战，由于有限的可用性，正确标记的个人乐器轨道。随着获取更大数据集以提高MSS性能的推动，遇到错误标记的单个仪器轨迹的不可避免性成为需要解决的重大挑战。本文介绍了一种自动化技术，用于细化部分错误标记的数据集中的标签。我们提出的自精炼技术，采用噪声标记的数据集，结果只有1%的精度下降，在多标签仪器识别相比，一个干净的标签数据集训练的分类器。该研究证明了在MSS模型训练中改进噪声标记数据的重要性，并表明利用改进的数据集可以从干净标记的数据集获得可比的结果。值得注意的是，在仅访问噪声数据集时，在自细化数据集上训练的MSS模型甚至优于在用在干净标签上训练的分类器细化的数据集上训练的MSS模型。摘要：Music source separation (MSS) faces challenges due to the limited availability of correctly-labeled individual instrument tracks. With the push to acquire larger datasets to improve MSS performance, the inevitability of encountering mislabeled individual instrument tracks becomes a significant challenge to address. This paper introduces an automated technique for refining the labels in a partially mislabeled dataset. Our proposed self-refining technique, employed with a noisy-labeled dataset, results in only a 1% accuracy degradation in multi-label instrument recognition compared to a classifier trained on a clean-labeled dataset. The study demonstrates the importance of refining noisy-labeled data in MSS model training and shows that utilizing the refined dataset leads to comparable results derived from a clean-labeled dataset. Notably, upon only access to a noisy dataset, MSS models trained on a self-refined dataset even outperform those trained on a dataset refined with a classifier trained on clean labels.
【3】 Prompting Large Language Models with Speech Recognition Abilities标题：提示具有语音识别能力的大型语言模型链接：https://arxiv.org/abs/2307.11795
作者：Yassir Fathullah,Chunyang Wu,Egor Lakomkin,Junteng Jia,Yuan Shangguan,Ke Li,Jinxi Guo,Wenhan Xiong,Jay Mahadeokar,Ozlem Kalinli,Christian Fuegen,Mike Seltzer摘要：大型语言模型已经证明了自己的高度灵活性，能够解决广泛的生成任务，例如抽象摘要和开放式问答。在本文中，我们扩展的能力的LLM直接附加一个小的音频编码器，允许它执行语音识别。通过将音频嵌入序列直接前置到文本标记嵌入，LLM可以被转换为自动语音识别（ASR）系统，并且以与其文本对应物完全相同的方式使用。多语言LibriSpeech（MLS）的实验表明，将构象编码器纳入开源的LLaMA-7 B中，使其性能优于单语基线18%，并执行多语言语音识别，尽管LLaMA绝大多数是在英语文本上训练的。此外，我们执行消融研究，以调查LLM是否可以在训练期间完全冻结以保持其原始能力，按比例放大音频编码器，并增加音频编码器跨步以生成更少的嵌入。这些研究的结果表明，即使LLM被冻结或在音频编码器中使用几乎1秒的步幅时，多语言ASR也是可能的，这为LLM在长格式音频上操作开辟了可能性。摘要：Large language models have proven themselves highly flexible, able to solve a wide range of generative tasks, such as abstractive summarization and open-ended question answering. In this paper we extend the capabilities of LLMs by directly attaching a small audio encoder allowing it to perform speech recognition. By directly prepending a sequence of audial embeddings to the text token embeddings, the LLM can be converted to an automatic speech recognition (ASR) system, and be used in the exact same manner as its textual counterpart. Experiments on Multilingual LibriSpeech (MLS) show that incorporating a conformer encoder into the open sourced LLaMA-7B allows it to outperform monolingual baselines by 18% and perform multilingual speech recognition despite LLaMA being trained overwhelmingly on English text. Furthermore, we perform ablation studies to investigate whether the LLM can be completely frozen during training to maintain its original capabilities, scaling up the audio encoder, and increasing the audio encoder striding to generate fewer embeddings. The results from these studies show that multilingual ASR is possible even when the LLM is frozen or when strides of almost 1 second are used in the audio encoder opening up the possibility for LLMs to operate on long-form audio.
【4】 An objective evaluation of Hearing Aids and DNN-based speech enhancement  in complex acoustic scenes标题：复杂声学场景中助听器和DNN语音增强的客观评价链接：https://arxiv.org/abs/2307.12888
作者：Enric Gusó,Joanna Luberadzka,Martí Baig,Umut Sayin Saraç,Xavier Serra备注：Accepted to WASPAA23摘要：我们调查的客观性能的五个高端商用助听器（HA）设备相比，DNN为基础的语音增强算法在复杂的声学环境。为此，我们测量单个HA设备的HRTF以合成双耳数据集，用于训练两个最先进的因果和非因果DNN增强模型。然后，我们生成一个评估集的现实的语音在噪声的情况下使用高保真度混响扬声器设置和记录与KU100假人头佩戴的HA设备，无论是和没有传统的HA算法，应用DNN增强器后者。我们发现，基于DNN的增强优于HA算法的噪声抑制和客观的清晰度指标。摘要：We investigate the objective performance of five high-end commercially available Hearing Aid (HA) devices compared to DNN-based speech enhancement algorithms in complex acoustic environments. To this end, we measure the HRTFs of a single HA device to synthesize a binaural dataset for training two state-of-the-art causal and non-causal DNN enhancement models. We then generate an evaluation set of realistic speech-in-noise situations using an Ambisonics loudspeaker setup and record with a KU100 dummy head wearing each of the HA devices, both with and without the conventional HA algorithms, applying the DNN enhancers to the latter. We find that the DNN-based enhancement outperforms the HA algorithms in terms of noise suppression and objective intelligibility metrics.
【5】 Code-Switched Urdu ASR for Noisy Telephonic Environment using Data  Centric Approach with Hybrid HMM and CNN-TDNN标题：基于混合隐马尔可夫模型和CNN-TDNN的数据中心编码交换URDU ASR链接：https://arxiv.org/abs/2307.12759
作者：Muhammad Danyal Khan,Raheem Ali,Arshad Aziz备注：32 pages, 19 figures, 2 tables, preprint摘要：呼叫中心有大量的音频数据，可用于实现有价值的业务洞察和电话的转录是手动繁琐的任务.有效的自动语音识别系统可以准确地转录这些呼叫，以便通过呼叫历史中的特定上下文和内容进行轻松搜索，从而允许自动呼叫监控，通过关键词搜索和情感分析提高QoS。呼叫中心ASR要求更强的鲁棒性，因为电话环境通常是嘈杂的。此外，有许多濒临灭绝的低资源语言可以通过自动语音识别技术得以保存。乌尔都语是世界上使用最广泛的语言，全球有231，295，440个乌尔都语仍然是ASR中资源有限的语言。区域呼叫中心的对话使用当地语言，混合了英语数字和专业术语，通常会导致“代码转换”问题。因此，本文描述了一个资源高效的自动语音识别/语音转文本系统在嘈杂的呼叫中心环境中使用链混合HMM和CNN-TDNN代码交换乌尔都语语言的实现框架。使用混合HMM-DNN方法使我们能够利用神经网络的优势，具有较少的标记数据。将CNN与TDNN一起添加已经显示出在噪声环境中更好地工作，这是由于CNN的额外频率维度从噪声语音中捕获额外信息，从而提高准确性。我们从各种开放来源收集数据，并在分析其一般背景和乌尔都语内容以及其他语言（主要是英语）的常用单词后标记了一些未标记的数据，并且能够在嘈杂和干净的环境中以孤立的单词或数字以及连续的自发语音实现5.2%的WER。摘要：Call Centers have huge amount of audio data which can be used for achieving valuable business insights and transcription of phone calls is manually tedious task. An effective Automated Speech Recognition system can accurately transcribe these calls for easy search through call history for specific context and content allowing automatic call monitoring, improving QoS through keyword search and sentiment analysis. ASR for Call Center requires more robustness as telephonic environment are generally noisy. Moreover, there are many low-resourced languages that are on verge of extinction which can be preserved with help of Automatic Speech Recognition Technology. Urdu is the $10^{th}$ most widely spoken language in the world, with 231,295,440 worldwide still remains a resource constrained language in ASR. Regional call-center conversations operate in local language, with a mix of English numbers and technical terms generally causing a "code-switching" problem. Hence, this paper describes an implementation framework of a resource efficient Automatic Speech Recognition/ Speech to Text System in a noisy call-center environment using Chain Hybrid HMM and CNN-TDNN for Code-Switched Urdu Language. Using Hybrid HMM-DNN approach allowed us to utilize the advantages of Neural Network with less labelled data. Adding CNN with TDNN has shown to work better in noisy environment due to CNN's additional frequency dimension which captures extra information from noisy speech, thus improving accuracy. We collected data from various open sources and labelled some of the unlabelled data after analysing its general context and content from Urdu language as well as from commonly used words from other languages, primarily English and were able to achieve WER of 5.2% with noisy as well as clean environment in isolated words or numbers as well as in continuous spontaneous speech.
【6】 Online Continual Learning in Keyword Spotting for Low-Resource Devices  via Pooling High-Order Temporal Statistics标题：汇聚高阶时间统计量的低资源设备关键字在线连续学习链接：https://arxiv.org/abs/2307.12660
作者：Umberto Michieli,Pablo Peso Parada,Mete Ozay备注：INTERSPEECH 2023摘要：嵌入式设备上的关键字识别（KWS）模型应该快速适应新的用户定义的单词，而不会忘记以前的单词。嵌入式设备具有有限的存储和计算资源，因此，它们不能保存样本或更新大型模型。我们考虑了嵌入式在线持续学习（EOCL）的设置，其中具有冻结主干的KWS模型被训练为从非重复的样本流中增量地识别新词，每次看到一个。为此，我们提出了时间感知池（TAP），它构建了一个丰富的特征空间计算高阶矩的语音特征提取的预训练的骨干。我们的方法，TAP-SLDA，更新高斯模型，为每个类的丰富的特征空间，以有效地使用音频表示。在实验分析中，TAP-SLDA在几个设置，主干和基线上优于竞争对手，在GSC数据集上带来了11.3%的相对平均增益。摘要：Keyword Spotting (KWS) models on embedded devices should adapt fast to new user-defined words without forgetting previous ones. Embedded devices have limited storage and computational resources, thus, they cannot save samples or update large models. We consider the setup of embedded online continual learning (EOCL), where KWS models with frozen backbone are trained to incrementally recognize new words from a non-repeated stream of samples, seen one at a time. To this end, we propose Temporal Aware Pooling (TAP) which constructs an enriched feature space computing high-order moments of speech features extracted by a pre-trained backbone. Our method, TAP-SLDA, updates a Gaussian model for each class on the enriched feature space to effectively use audio representations. In experimental analyses, TAP-SLDA outperforms competitors on several setups, backbones, and baselines, bringing a relative average gain of 11.3% on the GSC dataset.
【7】 A Model for Every User and Budget: Label-Free and Personalized  Mixed-Precision Quantization标题：一种面向每个用户和预算的模型：无标签和个性化的混合精度量化链接：https://arxiv.org/abs/2307.12659
作者：Edward Fish,Umberto Michieli,Mete Ozay备注：INTERSPEECH 2023摘要：自动语音识别（ASR）的最新进展已经产生了大型AI模型，这些模型对于在移动设备中的部署变得不切实际。模型量化对于产生压缩的通用模型是有效的，然而这样的模型可以仅部署到感兴趣的受限子域。我们表明，ASR模型可以在量化过程中个性化，而依赖于只是一小组未标记的样本从目标域。为此，我们提出了myQASR，一种混合精度的量化方法，生成量身定制的量化方案，为不同的用户在任何内存的要求，没有微调。myQASR通过分析全精度激活值自动评估网络层的量化敏感性。然后，我们能够为任何预定的存储器预算生成个性化的混合精度量化方案。大规模ASR模型的结果显示了myQASR如何提高特定性别、语言和说话者的性能。摘要：Recent advancement in Automatic Speech Recognition (ASR) has produced large AI models, which become impractical for deployment in mobile devices. Model quantization is effective to produce compressed general-purpose models, however such models may only be deployed to a restricted sub-domain of interest. We show that ASR models can be personalized during quantization while relying on just a small set of unlabelled samples from the target domain. To this end, we propose myQASR, a mixed-precision quantization method that generates tailored quantization schemes for diverse users under any memory requirement with no fine-tuning. myQASR automatically evaluates the quantization sensitivity of network layers by analysing the full-precision activation values. We are then able to generate a personalised mixed-precision quantization scheme for any pre-determined memory budget. Results for large-scale ASR models show how myQASR improves performance for specific genders, languages, and speakers.
【8】 Robust Automatic Speech Recognition via WavAugment Guided Phoneme  Adversarial Training标题：基于WavAugment引导的音素对抗训练的稳健自动语音识别链接：https://arxiv.org/abs/2307.12498
作者：Gege Qi,Yuefeng Chen,Xiaofeng Mao,Xiaojun Jia,Ranjie Duan,Rong Zhang,Hui Xue摘要：开发一个实用的鲁棒的自动语音识别（ASR）是一个挑战，因为模型不仅应该保持原始的性能干净的样本，但也实现一致的功效下，小的体积扰动和大的域移位。为了解决这个问题，我们提出了一种新的WavAugment引导音素对抗训练（wapat）。WAPAT使用音素空间中的对抗示例作为增强，以使模型对音素表示中的微小波动不变，并保持干净样本上的性能。此外，wapat利用增强样本的音素表示来指导对手的生成，这有助于找到更稳定和多样化的梯度方向，从而提高泛化能力。大量的实验证明了wapat在端到端语音挑战基准（ESB）上的有效性。值得注意的是，SpeechLM-wapat在ESB上的WER降低了6.28%，超过了原始模型，达到了新的最先进水平。摘要：Developing a practically-robust automatic speech recognition (ASR) is challenging since the model should not only maintain the original performance on clean samples, but also achieve consistent efficacy under small volume perturbations and large domain shifts. To address this problem, we propose a novel WavAugment Guided Phoneme Adversarial Training (wapat). wapat use adversarial examples in phoneme space as augmentation to make the model invariant to minor fluctuations in phoneme representation and preserve the performance on clean samples. In addition, wapat utilizes the phoneme representation of augmented samples to guide the generation of adversaries, which helps to find more stable and diverse gradient-directions, resulting in improved generalization. Extensive experiments demonstrate the effectiveness of wapat on End-to-end Speech Challenge Benchmark (ESB). Notably, SpeechLM-wapat outperforms the original model by 6.28% WER reduction on ESB, achieving the new state-of-the-art.
【9】 SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic  Spaces标题：残片：语音和语音空间的语音对比表示链接：https://arxiv.org/abs/2307.12445
作者：Ivan Vallés-Pérez,Grzegorz Beringer,Piotr Bilinski,Gary Cook,Roberto Barra-Chicote备注：In proceedings of the 26th European Conference on Artificial Intelligence ECAI 2023. 8 pages + 1 appendix page摘要：文献中的许多例子证明，深度学习模型能够很好地处理多模态数据。最近，CLIP已经使深度学习系统能够学习图像和文本描述之间的共享潜在空间，在下游任务中具有出色的零或Few-Shot结果。在本文中，我们探讨了同样的想法，提出了CLIP，但适用于语音域，语音和声学空间通常共存。我们训练了一个基于CLIP的模型，目的是学习语音和声学空间的共享表示。结果表明，该模型是敏感的语音变化，91%的分数下降时，随机替换20%的音素，同时提供了大量的鲁棒性对不同种类的噪声，与10%的性能下降时，混合音频与75%的高斯噪声。我们还提供了经验证据，表明所得到的嵌入对于各种下游应用是有用的，例如可懂度评估和在语音生成任务中利用丰富的预训练语音嵌入的能力。最后，我们讨论了潜在的应用与有趣的影响，语音生成和识别领域。摘要：Numerous examples in the literature proved that deep learning models have the ability to work well with multimodal data. Recently, CLIP has enabled deep learning systems to learn shared latent spaces between images and text descriptions, with outstanding zero- or few-shot results in downstream tasks. In this paper we explore the same idea proposed by CLIP but applied to the speech domain, where the phonetic and acoustic spaces usually coexist. We train a CLIP-based model with the aim to learn shared representations of phonetic and acoustic spaces. The results show that the proposed model is sensible to phonetic changes, with a 91% of score drops when replacing 20% of the phonemes at random, while providing substantial robustness against different kinds of noise, with a 10% performance drop when mixing the audio with 75% of Gaussian noise. We also provide empirical evidence showing that the resulting embeddings are useful for a variety of downstream applications, such as intelligibility evaluation and the ability to leverage rich pre-trained phonetic embeddings in speech generation task. Finally, we discuss potential applications with interesting implications for the speech generation and recognition fields.
【10】 Performance Comparison Between VoLTE and non-VoLTE Voice Calls During  Mobility in Commercial Deployment: A Drive Test-Based Analysis标题：商用部署移动过程中VoLTE和非VoLTE语音呼叫的性能比较：基于路测的分析链接：https://arxiv.org/abs/2307.12397
作者：Rashed Hasan Ratul,Muhammad Iqbal,Jen-Yi Pan,Mohammad Mahadi Al Deen,Mohammad Tawhid Kawser,Mohammad Masum Billah备注：Accepted for presentation and Publication on the IEEE 10th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI 2023)摘要：网络性能的优化对于使用用于移动通信的标准蜂窝技术来递送服务是至关重要的。呼叫建立延迟和用户设备（UE）电池节省显著影响网络性能。改善这些因素对于确保提供最佳服务至关重要。与传统的电路交换语音呼叫相比，VoLTE（LTE语音）技术提供了更快的呼叫建立持续时间和更好的电池节省性能。为了验证这些声明，使用XCAL路测工具进行了路测，以收集VoLTE和非VoLTE语音呼叫中的实时网络参数详细信息。研究结果突出了实时网络特性的分析，如呼叫建立延迟计算，节省电池的性能，和DRX机制。该研究有助于网络优化策略的理解，并为提高移动通信网络的服务质量（QoS）提供了见解。通过研究VoLTE和非VoLTE运营，这项研究强调了VoLTE所获得的大量能源节约。具体地，VoLTE在服务请求之前节省大约60.76%的能量，并且在服务请求之后节省大约38.97%的能量。此外，由于所需的信令消息较少，VoLTE到VoLTE呼叫的呼叫建立延迟比非基于VoLTE的LTE到LTE呼叫快72.6%。此外，与非VoLTE到非VoLTE呼叫相比，VoLTE到非VoLTE呼叫的呼叫建立延迟快18.6%。这些结果展示了VoLTE的性能优势，并增强了其在无线通信网络中提供更好服务的潜力。摘要：The optimization of network performance is vital for the delivery of services using standard cellular technologies for mobile communications. Call setup delay and User Equipment (UE) battery savings significantly influence network performance. Improving these factors is vital for ensuring optimal service delivery. In comparison to traditional circuit-switched voice calls, VoLTE (Voice over LTE) technology offers faster call setup durations and better battery-saving performance. To validate these claims, a drive test was carried out using the XCAL drive test tool to collect real-time network parameter details in VoLTE and non-VoLTE voice calls. The findings highlight the analysis of real-time network characteristics, such as the call setup delay calculation, battery-saving performance, and DRX mechanism. The study contributes to the understanding of network optimization strategies and provides insights for enhancing the quality of service (QoS) in mobile communication networks. Examining VoLTE and non-VoLTE operations, this research highlights the substantial energy savings obtained by VoLTE. Specifically, VoLTE saves approximately 60.76% of energy before the Service Request and approximately 38.97% of energy after the Service Request. Moreover, VoLTE to VoLTE calls have a 72.6% faster call setup delay than non-VoLTE-based LTE to LTE calls, because of fewer signaling messages required. Furthermore, as compared to non-VoLTE to non-VoLTE calls, VoLTE to non-VoLTE calls offer an 18.6% faster call setup delay. These results showcase the performance advantages of VoLTE and reinforce its potential for offering better services in wireless communication networks.
【11】 Self-Supervised Learning for Audio-Based Emotion Recognition标题：基于音频的情感识别中的自监督学习链接：https://arxiv.org/abs/2307.12343
作者：Peranut Nimitsurachat,Peter Washington备注：8 pages, 9 figures, submitted to IEEE Transactions on Affective Computing摘要：使用音频输入数据的情绪识别模型可以实现在精神保健、营销、游戏和社交媒体分析中应用的交互系统的开发。虽然使用音频数据的情感计算领域很丰富，但实现一致高性能模型的主要障碍是缺乏可用的训练标签。自监督学习（SSL）是一系列方法，可以通过预测数据本身的属性来学习，尽管缺乏监督标签。为了理解自监督学习对于基于音频的情感识别的效用，我们已经将自监督学习预训练应用于来自CMU-MOSEI的声学模态的情感分类。不像以前的论文，实验与原始的声学数据，我们的技术已被应用到编码的声学数据。我们的模型首先被预训练，以揭示随机掩蔽的时间戳的声学数据。然后使用注释数据的小样本对预训练模型进行微调。然后，通过几个评估指标对具有相同骨干架构的基线深度学习模型评估最终模型的性能。我们发现，自我监督学习一致地提高了模型在所有指标的性能。这项工作显示了自我监督学习在情感计算中的效用，证明了自我监督学习在训练样本的数量很少时是最有用的，并且对于更容易分类的情绪，如快乐，悲伤和愤怒的效果最明显。这项工作进一步证明了自监督学习在应用于嵌入式特征表示时有效，而不是在原始输入空间上进行预训练的传统方法。摘要：Emotion recognition models using audio input data can enable the development of interactive systems with applications in mental healthcare, marketing, gaming, and social media analysis. While the field of affective computing using audio data is rich, a major barrier to achieve consistently high-performance models is the paucity of available training labels. Self-supervised learning (SSL) is a family of methods which can learn despite a scarcity of supervised labels by predicting properties of the data itself. To understand the utility of self-supervised learning for audio-based emotion recognition, we have applied self-supervised learning pre-training to the classification of emotions from the CMU- MOSEI's acoustic modality. Unlike prior papers that have experimented with raw acoustic data, our technique has been applied to encoded acoustic data. Our model is first pretrained to uncover the randomly-masked timestamps of the acoustic data. The pre-trained model is then fine-tuned using a small sample of annotated data. The performance of the final model is then evaluated via several evaluation metrics against a baseline deep learning model with an identical backbone architecture. We find that self-supervised learning consistently improves the performance of the model across all metrics. This work shows the utility of self-supervised learning for affective computing, demonstrating that self-supervised learning is most useful when the number of training examples is small, and that the effect is most pronounced for emotions which are easier to classify such as happy, sad and anger. This work further demonstrates that self-supervised learning works when applied to embedded feature representations rather than the traditional approach of pre-training on the raw input space.
【12】 A meta learning scheme for fast accent domain expansion in Mandarin  speech recognition标题：汉语语音识别中一种快速扩展重音域的元学习方法链接：https://arxiv.org/abs/2307.12262
作者：Ziwei Zhu,Changhao Shan,Bihong Zhang,Jian Yu摘要：口语在普通话和口音方面表现出显著的差异。尽管普通话自动语音识别（ASR）已经取得了很高的性能，但口音识别仍然是一个具有挑战性的任务。在本文中，我们介绍了元学习技术的快速口音域扩展在普通话语音识别，这扩大了口音的领域，而不会恶化的性能的普通话ASR。元学习或lear-to-learning可以学习多个领域中的一般关系，而不仅仅是为了过拟合某个特定领域。因此，我们选择了元学习作为领域扩展任务。这种更重要的学习将提高重音域扩展任务的性能。我们结合了Meta学习和模型参数冻结的方法，使得识别性能在不同情况下更加稳定，训练速度提高了20%左右。我们的方法显着优于其他方法约3%，相对于重音域扩展任务。在普通话测试集不变的情况下，与基线模型相比，该模型提高了37%。此外，它也证明了该方法是有效的，在口音测试集的相对性能提高了4%。摘要：Spoken languages show significant variation across mandarin and accent. Despite the high performance of mandarin automatic speech recognition (ASR), accent ASR is still a challenge task. In this paper, we introduce meta-learning techniques for fast accent domain expansion in mandarin speech recognition, which expands the field of accents without deteriorating the performance of mandarin ASR. Meta-learning or learn-to-learn can learn general relation in multi domains not only for over-fitting a specific domain. So we select meta-learning in the domain expansion task. This more essential learning will cause improved performance on accent domain extension tasks. We combine the methods of meta learning and freeze of model parameters, which makes the recognition performance more stable in different cases and the training faster about 20%. Our approach significantly outperforms other methods about 3% relatively in the accent domain expansion task. Compared to the baseline model, it improves relatively 37% under the condition that the mandarin test set remains unchanged. In addition, it also proved this method to be effective on a large amount of data with a relative performance improvement of 4% on the accent test set.
【13】 Signal Reconstruction from Mel-spectrogram Based on Bi-level Consistency  of Full-band Magnitude and Phase标题：基于全带幅相两级一致性的Mel谱图信号重构链接：https://arxiv.org/abs/2307.12232
作者：Yoshiki Masuyama,Natsuki Ueno,Nobutaka Ono备注：Accepted to IEEE WASPAA 2023摘要：我们提出了一种基于优化的方法，用于从低维频谱表示，如梅尔频谱图重建时域信号。相位重建已经被研究以从全频带短时傅里叶变换（STFT）幅度重建时域信号。Griffin-Lim算法（GLA）由于只依赖于STFT的冗余性，适用于各种音频信号，因而得到了广泛的应用。在本文中，我们联合重建的全频带幅度和相位，考虑到时域信号，它的STFT系数，和它的梅尔频谱图之间的双电平关系。所提出的方法被配制成一个严格的优化问题，并估计全频带幅度的基础上使用的GLA的标准。我们的实验证明了所提出的方法对语音，音乐和环境信号的有效性。摘要：We propose an optimization-based method for reconstructing a time-domain signal from a low-dimensional spectral representation such as a mel-spectrogram. Phase reconstruction has been studied to reconstruct a time-domain signal from the full-band short-time Fourier transform (STFT) magnitude. The Griffin-Lim algorithm (GLA) has been widely used because it relies only on the redundancy of STFT and is applicable to various audio signals. In this paper, we jointly reconstruct the full-band magnitude and phase by considering the bi-level relationships among the time-domain signal, its STFT coefficients, and its mel-spectrogram. The proposed method is formulated as a rigorous optimization problem and estimates the full-band magnitude based on the criterion used in GLA. Our experiments demonstrate the effectiveness of the proposed method on speech, music, and environmental signals.
【14】 Exploring the Integration of Speech Separation and Recognition with  Self-Supervised Learning Representation标题：语音分离和识别与自监督学习表示相结合的探索链接：https://arxiv.org/abs/2307.12231
作者：Yoshiki Masuyama,Xuankai Chang,Wangyou Zhang,Samuele Cornell,Zhong-Qiu Wang,Nobutaka Ono,Yanmin Qian,Shinji Watanabe备注：Accepted to IEEE WASPAA 2023摘要：神经语音分离技术已经取得了显著的进展，将其与自动语音识别技术相结合是实现多说话人自动语音识别的重要方向。这项工作提供了一个有见地的调查语音分离混响和噪声混响的情况下作为ASR前端。详细地说，我们探讨了多通道分离方法，基于掩模的波束成形和复杂的频谱映射，以及在ASR后端模型中使用的最佳功能。我们采用最近的自监督学习表示（SSLR）作为一个功能，并提高了识别性能的情况下，过滤器组功能。为了进一步提高多说话人识别的性能，我们提出了一个精心设计的训练策略，集成语音分离和识别SSLR。所提出的集成使用TF-GridNet为基础的复杂的频谱映射和WavLM为基础的SSLR实现了2.5%的字错误率混响WHAMR！测试集，显著优于现有的基于掩模的MVDR波束成形和滤波器组集成（28.9%）。摘要：Neural speech separation has made remarkable progress and its integration with automatic speech recognition (ASR) is an important direction towards realizing multi-speaker ASR. This work provides an insightful investigation of speech separation in reverberant and noisy-reverberant scenarios as an ASR front-end. In detail, we explore multi-channel separation methods, mask-based beamforming and complex spectral mapping, as well as the best features to use in the ASR back-end model. We employ the recent self-supervised learning representation (SSLR) as a feature and improve the recognition performance from the case with filterbank features. To further improve multi-speaker recognition performance, we present a carefully designed training strategy for integrating speech separation and recognition with SSLR. The proposed integration using TF-GridNet-based complex spectral mapping and WavLM-based SSLR achieves a 2.5% word error rate in reverberant WHAMR! test set, significantly outperforming an existing mask-based MVDR beamforming and filterbank integration (28.9%).
【15】 Adversarial Agents For Attacking Inaudible Voice Activated Devices标题：攻击听不见的声控设备的对抗性代理链接：https://arxiv.org/abs/2307.12204
作者：Forrest McKee,David Noever摘要：我们对语音激活设备上的听不见攻击的分析证实了7.6的惊人风险系数（满分10分），强调了NIST国家漏洞数据库（NVD）独立评分的重大安全漏洞。我们的基线网络模型展示了一个场景，其中攻击者使用听不见的语音命令来获得未经授权的访问机密信息的安全笔记本电脑。我们在此基线网络模型上模拟了许多攻击场景，揭示了大规模利用互联设备通过物理访问发现和拥有特权信息的潜力，而无需添加新硬件或增强设备技能。使用微软的CyberBattleSim框架，我们评估了六种强化学习算法，发现Deep-Q学习和利用被证明是最佳的，可以在更少的步骤中快速拥有所有节点。我们的研究结果强调了在不断扩大的数字环境中理解非传统网络和新的网络安全措施的迫切需要，特别是那些以移动设备，语音激活和非线性麦克风为特征的网络，这些网络容易受到恶意行为者在近超声波或听不见的范围内进行隐形攻击的影响。到2024年，这种新的攻击面可能涵盖比地球上的人更多的数字语音助理，但提供的补救措施比传统的修补或固件修复更少，因为听不见的攻击固有地来自麦克风设计和数字信号处理。摘要：Our analysis of inaudible attacks on voice-activated devices confirms the alarming risk factor of 7.6 out of 10, underlining significant security vulnerabilities scored independently by NIST National Vulnerability Database (NVD). Our baseline network model showcases a scenario in which an attacker uses inaudible voice commands to gain unauthorized access to confidential information on a secured laptop. We simulated many attack scenarios on this baseline network model, revealing the potential for mass exploitation of interconnected devices to discover and own privileged information through physical access without adding new hardware or amplifying device skills. Using Microsoft's CyberBattleSim framework, we evaluated six reinforcement learning algorithms and found that Deep-Q learning with exploitation proved optimal, leading to rapid ownership of all nodes in fewer steps. Our findings underscore the critical need for understanding non-conventional networks and new cybersecurity measures in an ever-expanding digital landscape, particularly those characterized by mobile devices, voice activation, and non-linear microphones susceptible to malicious actors operating stealth attacks in the near-ultrasound or inaudible ranges. By 2024, this new attack surface might encompass more digital voice assistants than people on the planet yet offer fewer remedies than conventional patching or firmware fixes since the inaudible attacks arise inherently from the microphone design and digital signal processing.
【16】 Modality Confidence Aware Training for Robust End-to-End Spoken Language  Understanding标题：用于强健的端到端口语理解的通道置信度感知训练链接：https://arxiv.org/abs/2307.12134
作者：Suyoun Kim,Akshat Shrivastava,Duc Le,Ju Lin,Ozlem Kalinli,Michael L. Seltzer备注：INTERSPEECH 2023摘要：最近，从语音生成语义解析的端到端（E2E）口语理解（SLU）系统变得越来越有前途。这种方法使用单个模型，该模型利用来自预训练语音识别模型（ASR）的音频和文本表示，并且在设备上的流媒体场景中优于传统的流水线SLU系统。然而，E2E SLU系统仍然表现出弱点时，由于ASR转录错误的文本表示质量低。为了克服这个问题，我们提出了一种新的E2E SLU系统，通过融合音频和文本表示的基础上估计的模态置信度的ASR假设，增强了对ASR错误的鲁棒性。我们介绍两种新技术：1）编码ASR假设的质量的有效方法和2）将它们集成到E2E SLU模型中的有效方法。我们展示了STOP数据集的准确性改进，并分享了分析结果，以证明我们方法的有效性。摘要：End-to-end (E2E) spoken language understanding (SLU) systems that generate a semantic parse from speech have become more promising recently. This approach uses a single model that utilizes audio and text representations from pre-trained speech recognition models (ASR), and outperforms traditional pipeline SLU systems in on-device streaming scenarios. However, E2E SLU systems still show weakness when text representation quality is low due to ASR transcription errors. To overcome this issue, we propose a novel E2E SLU system that enhances robustness to ASR errors by fusing audio and text representations based on the estimated modality confidence of ASR hypotheses. We introduce two novel techniques: 1) an effective method to encode the quality of ASR hypotheses and 2) an effective approach to integrate them into E2E SLU models. We show accuracy improvements on STOP dataset and share the analysis to demonstrate the effectiveness of our approach.
【17】 Estimating speaker direction on a humanoid robot with binaural acoustic  signals标题：基于双耳声学信号的仿人机器人说话人方向估计链接：https://arxiv.org/abs/2307.12129
作者：Pranav Barot,Katja Mombaur,Ewen MacDonald摘要：为了在语音交互过程中实现类似人类的行为，人形机器人有必要估计人类说话者的位置。在这里，我们提出了一种方法来优化用于到达方向（DOA）估计的参数，同时也考虑到实时应用的人机交互场景。该方法被应用于双耳声源定位框架的仿人机器人头部。真实的数据收集和注释这项工作。通过蛮力方法和基于贝叶斯模型的方法进行优化，验证和讨论结果，并探讨对实时使用的延迟的影响。摘要：To achieve human-like behaviour during speech interactions, it is necessary for a humanoid robot to estimate the location of a human talker. Here, we present a method to optimize the parameters used for the direction of arrival (DOA) estimation, while also considering real-time applications for human-robot interaction scenarios. This method is applied to binaural sound source localization framework on a humanoid robotic head. Real data is collected and annotated for this work. Optimizations are performed via a brute force method and a Bayesian model based method, results are validated and discussed, and effects on latency for real-time use are also explored.
【18】 Transsion TSUP's speech recognition system for ASRU 2023 MADASR  Challenge标题：Transsion Tsup用于ASRU 2023 MADASR挑战赛的语音识别系统链接：https://arxiv.org/abs/2307.11778
作者：Xiaoxiao Li,Gaosheng Zhang,An Zhu,Weiyong Li,Shuming Fang,Xiaoyue Yang,Jianchao Zhu摘要：本文介绍了由Transsion语音理解处理团队（TSUP）为ASRU 2023 MADASR挑战赛开发的语音识别系统。该系统的重点是适应低资源印度语言的ASR模型，并涵盖了所有四个轨道的挑战。对于轨道1和2，声学模型利用具有联合CTC注意力训练损失的squeezeformer编码器和双向Transformer解码器。此外，在TLG波束搜索解码期间使用外部KenLM语言模型。对于轨道3和4，采用了预训练的IndicWhisper模型，并在挑战数据集和公开可用的数据集上进行了微调。还修改了耳语波束搜索解码，以支持外部KenLM语言模型，从而更好地利用挑战提供的附加文本。所提出的方法实现了字错误率（WER）的24.17%，24.43%，15.97%和15.97%的孟加拉语在四个轨道，和WER的19.61%，19.54%，15.48%和15.48%的Bhojpuri语言在四个轨道。这些结果证明了所提出的方法的有效性。摘要：This paper presents a speech recognition system developed by the Transsion Speech Understanding Processing Team (TSUP) for the ASRU 2023 MADASR Challenge. The system focuses on adapting ASR models for low-resource Indian languages and covers all four tracks of the challenge. For tracks 1 and 2, the acoustic model utilized a squeezeformer encoder and bidirectional transformer decoder with joint CTC-Attention training loss. Additionally, an external KenLM language model was used during TLG beam search decoding. For tracks 3 and 4, pretrained IndicWhisper models were employed and finetuned on both the challenge dataset and publicly available datasets. The whisper beam search decoding was also modified to support an external KenLM language model, which enabled better utilization of the additional text provided by the challenge. The proposed method achieved word error rates (WER) of 24.17%, 24.43%, 15.97%, and 15.97% for Bengali language in the four tracks, and WER of 19.61%, 19.54%, 15.48%, and 15.48% for Bhojpuri language in the four tracks. These results demonstrate the effectiveness of the proposed method.
机器翻译，仅供参考