今日论文合集：cs.SD语音13篇，eess.AS音频处理11篇。
cs.SD语音
【1】 Trustworthy Sensor Fusion against Inaudible Command Attacks in Advanced  Driver-Assistance System标题：高级驾驶员辅助系统中抗隐形指令攻击的可信传感器融合链接：https://arxiv.org/abs/2306.05358
作者：Jiwei Guan,Lei Pan,Chen Wang,Shui Yu,Longxiang Gao,Xi Zheng机构：Lei Pan is with the School of Information Technology摘要：人们越来越担心自动驾驶汽车受到恶意攻击。特别地，随着语音命令在自动驾驶系统中变得可用，听不见的语音命令攻击构成了重大威胁。如何根据经验防御这些听不见的攻击仍然是一个悬而未决的问题。以前的研究调查了利用基于深度学习的多模态融合进行防御，而没有考虑模型可信度的不确定性。随着深度学习被应用于越来越敏感的任务，不确定性测量在帮助提高模型鲁棒性方面至关重要，特别是在关键任务场景中。在本文中，我们提出了多模态融合框架（MFF）作为一个智能安全系统，以抵御听不见的语音命令攻击。MFF融合异构视听模态使用VGG族神经网络，并实现了检测准确率为92.25%的比较融合方法的实证研究。此外，大量的视听任务的实验揭示了模型的不确定性。使用预期校准误差，我们测量校准误差和蒙特卡罗丢弃估计所提出的模型的预测分布。我们的研究结果表明，经验训练强大的多模态模型，提高标准的准确性，并提供了进一步的可解释性。最后，我们讨论了我们的方法的优点和缺点，其适用于先进的驾驶辅助系统。摘要：There are increasing concerns about malicious attacks on autonomous vehicles. In particular, inaudible voice command attacks pose a significant threat as voice commands become available in autonomous driving systems. How to empirically defend against these inaudible attacks remains an open question. Previous research investigates utilizing deep learning-based multimodal fusion for defense, without considering the model uncertainty in trustworthiness. As deep learning has been applied to increasingly sensitive tasks, uncertainty measurement is crucial in helping improve model robustness, especially in mission-critical scenarios. In this paper, we propose the Multimodal Fusion Framework (MFF) as an intelligent security system to defend against inaudible voice command attacks. MFF fuses heterogeneous audio-vision modalities using VGG family neural networks and achieves the detection accuracy of 92.25% in the comparative fusion method empirical study. Additionally, extensive experiments on audio-vision tasks reveal the model's uncertainty. Using Expected Calibration Errors, we measure calibration errors and Monte-Carlo Dropout to estimate the predictive distribution for the proposed models. Our findings show empirically to train robust multimodal models, improve standard accuracy and provide a further step toward interpretability. Finally, we discuss the pros and cons of our approach and its applicability for Advanced Driver Assistance Systems.
【2】 PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches  For Speech Emotion Recognition Using Pre-trained Speech Models标题：PEFT-SER：参数高效转移学习方法在基于预训练语音模型的语音情感识别中的应用链接：https://arxiv.org/abs/2306.05350
作者：Tiantian Feng,Shrikanth Narayanan机构：Department of Computer Science, University of Southern California, Los Angeles, USA, Department of Electrical and Computer Engineering备注：This work was accepted to the 11th International Conference on Affective Computing and Intelligent Interaction (ACII), 2023摘要：最近的许多研究都集中在微调预训练模型的语音情感识别（SER），导致有前途的性能相比，传统的方法，主要依赖于低层次的，知识启发的声学特征。这些预先训练的语音模型使用来自大规模数据集的自监督或弱监督学习目标来学习通用语音表示。尽管通过使用预训练架构在SER中取得了重大进展，但针对不同数据集微调这些大型预训练模型需要保存整个权重参数的副本，这使得它们在现实环境中部署不切实际。作为一种替代方案，这项工作探讨了参数有效的微调（PEFT）方法，用于适应情绪识别的预训练语音模型。具体来说，我们评估的适配器调谐，嵌入提示调谐，和LoRa（低秩近似）的四个流行的SER测试平台的功效。我们的研究结果表明，LoRa在情感识别方面实现了最佳的微调性能，同时提高了公平性，并且只需要最少的额外权重参数。此外，我们的研究结果提供了新的见解SER未来的研究方向，不同于现有的方法专注于直接微调模型架构。我们的代码可通过以下方式公开获取：https://github.com/usc-sail/peft-ser。摘要：Many recent studies have focused on fine-tuning pre-trained models for speech emotion recognition (SER), resulting in promising performance compared to traditional methods that rely largely on low-level, knowledge-inspired acoustic features. These pre-trained speech models learn general-purpose speech representations using self-supervised or weakly-supervised learning objectives from large-scale datasets. Despite the significant advances made in SER through the use of pre-trained architecture, fine-tuning these large pre-trained models for different datasets requires saving copies of entire weight parameters, rendering them impractical to deploy in real-world settings. As an alternative, this work explores parameter-efficient fine-tuning (PEFT) approaches for adapting pre-trained speech models for emotion recognition. Specifically, we evaluate the efficacy of adapter tuning, embedding prompt tuning, and LoRa (Low-rank approximation) on four popular SER testbeds. Our results reveal that LoRa achieves the best fine-tuning performance in emotion recognition while enhancing fairness and requiring only a minimal extra amount of weight parameters. Furthermore, our findings offer novel insights into future research directions in SER, distinct from existing approaches focusing on directly fine-tuning the model architecture. Our code is publicly available under: https://github.com/usc-sail/peft-ser.
【3】 KIT's Multilingual Speech Translation System for IWSLT 2023标题：KIT为IWSLT 2023提供的多语言语音翻译系统链接：https://arxiv.org/abs/2306.05320
作者：Danni Liu,Thai Binh Nguyen,Sai Koneru,Enes Yavuz Ugan,Ngoc-Quan Pham,Tuan-Nam Nguyen,Tu Anh Dinh,Carlos Mullov,Alexander Waibel,Jan Niehues机构：Karlsruhe Institute of Technology备注：IWSLT 2023摘要：许多现有的语音翻译基准专注于高质量记录条件下的母语英语语音，这通常与现实生活中的用例条件不匹配。在本文中，我们描述了我们的语音翻译系统的多语言轨道IWSLT 2023，重点是翻译的科学会议讲话。测试条件的特点是重音输入语音和术语密集的内容。这些任务需要翻译成10种语言，所需资源不一。在没有来自目标域的训练数据的情况下，我们使用基于检索的方法（kNN-MT）进行有效的自适应（语音翻译的BLEU为+0.8）。我们还使用适配器来轻松地集成来自数据增强的增量训练数据，并表明它与重新训练的性能相匹配。我们观察到，级联系统更容易适应特定的目标领域，由于其单独的模块。我们的级联语音系统大大优于其端到端的科学谈话翻译，虽然他们的表现仍然类似于TED演讲。摘要：Many existing speech translation benchmarks focus on native-English speech in high-quality recording conditions, which often do not match the conditions in real-life use-cases. In this paper, we describe our speech translation system for the multilingual track of IWSLT 2023, which focuses on the translation of scientific conference talks. The test condition features accented input speech and terminology-dense contents. The tasks requires translation into 10 languages of varying amounts of resources. In absence of training data from the target domain, we use a retrieval-based approach (kNN-MT) for effective adaptation (+0.8 BLEU for speech translation). We also use adapters to easily integrate incremental training data from data augmentation, and show that it matches the performance of re-training. We observe that cascaded systems are more easily adaptable towards specific target domains, due to their separate modules. Our cascaded speech system substantially outperforms its end-to-end counterpart on scientific talk translation, although their performance remains similar on TED talks.
【4】 Simple and Controllable Music Generation标题：简单可控的音乐生成链接：https://arxiv.org/abs/2306.05284
作者：Jade Copet,Felix Kreuk,Itai Gat,Tal Remez,David Kant,Gabriel Synnaeve,Yossi Adi,Alexandre Défossez机构：♠: equal contributions, ♢: core team, Meta AI摘要：我们解决了条件音乐生成的任务。我们引入MusicGen，一个单一的语言模型（LM），它在几个压缩的离散音乐表示流上操作，即，代币与先前的工作不同，MusicGen由单级Transformer LM以及高效的令牌交织模式组成，这消除了对级联几个模型的需要，例如，分层或上采样。按照这种方法，我们演示了MusicGen如何生成高质量的样本，同时以文本描述或旋律特征为条件，可以更好地控制所生成的输出。我们进行了广泛的实证评估，考虑自动和人类的研究，显示所提出的方法是优于标准的文本到音乐基准的评估基线。通过消融研究，我们揭示了组成MusicGen的每个组件的重要性。音乐样本、代码和模型可在www.example.com上获得https://github.com/facebookresearch/audiocraft。摘要：We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen. Music samples, code, and models are available at https://github.com/facebookresearch/audiocraft.
【5】 Language-specific Acoustic Boundary Learning for Mandarin-English  Code-switching Speech Recognition标题：汉语-英语码型转换语音识别中的特定语言声学边界学习链接：https://arxiv.org/abs/2306.05279
作者：Zhiyun Fan,Linhao Dong,Chen Shen,Zhenlin Liang,Jun Zhang,Lu Lu,Zejun Ma机构：Bytedance Research摘要：代码切换语音识别（CSSR）转录在单个句子中在多种语言或方言之间切换的语音。这项任务的主要挑战是，不同的语言通常具有相似的发音，这使得模型很难区分它们。在本文中，我们提出了一种方法来解决CSSR任务的角度来看，特定语言的声学边界学习。我们引入语言特定的权重估计器（LSWE）分别在不同的语言模型声学边界学习。另外，采用非自回归（NAR）解码器和语言变化检测（LCD）模块来辅助训练。在SEAME语料库上进行评估，我们的方法在test_man和test_sge集上实现了16.29%和22.81%的最先进的混合错误率（MER）。我们还证明了我们的方法在9000小时的内部会议代码转换数据集上的有效性，其中我们的方法实现了相对7.9%的MER减少。摘要：Code-switching speech recognition (CSSR) transcribes speech that switches between multiple languages or dialects within a single sentence. The main challenge in this task is that different languages often have similar pronunciations, making it difficult for models to distinguish between them. In this paper, we propose a method for solving the CSSR task from the perspective of language-specific acoustic boundary learning. We introduce language-specific weight estimators (LSWE) to model acoustic boundary learning in different languages separately. Additionally, a non-autoregressive (NAR) decoder and a language change detection (LCD) module are employed to assist in training. Evaluated on the SEAME corpus, our method achieves a state-of-the-art mixed error rate (MER) of 16.29% and 22.81% on the test_man and test_sge sets. We also demonstrate the effectiveness of our method on a 9000-hour in-house meeting code-switching dataset, where our method achieves a relatively 7.9% MER reduction.
【6】 The ART of Conversation: Measuring Phonetic Convergence and Deliberate  Imitation in L2-Speech with a Siamese RNN标题：会话艺术：用暹罗语RNN测量二语语音中的语音融合和刻意模仿链接：https://arxiv.org/abs/2306.05088
作者：Zheng Yuan,Aldo Pastore,Dorina de Jong,Hao Xu,Luciano Fadiga,Alessandro D'Ausilio机构：Alessandro D’Ausilio, Istituto Italiano di Tecnologia, Italy, Universita degli Studi di Ferrara, Italy,  University of California San Diego, USA备注：Accepted at INTERSPEECH 2023摘要：语音趋同是指会话中两个对话者自动的、无意识的言语适应。本文提出了一种连体递归神经网络（RNN）架构来衡量收敛的整体频谱特性的语音在L2-L2的互动。我们扩展了交替阅读任务（ART）数据集，增加了20个母语斯洛伐克L2英语。我们训练和测试暹罗RNN模型，以测量来自三个不同母语组的L2英语语音的语音收敛：意大利语（9对）、法语（10对）和斯洛伐克语（10对）。我们的研究结果表明，Siamese RNN模型有效地捕捉到语音收敛和说话人的模仿能力的动态。此外，这种文本无关的模型是可扩展的，能够处理L1引起的扬声器的变化。摘要：Phonetic convergence describes the automatic and unconscious speech adaptation of two interlocutors in a conversation. This paper proposes a Siamese recurrent neural network (RNN) architecture to measure the convergence of the holistic spectral characteristics of speech sounds in an L2-L2 interaction. We extend an alternating reading task (the ART) dataset by adding 20 native Slovak L2 English speakers. We train and test the Siamese RNN model to measure phonetic convergence of L2 English speech from three different native language groups: Italian (9 dyads), French (10 dyads) and Slovak (10 dyads). Our results indicate that the Siamese RNN model effectively captures the dynamics of phonetic convergence and the speaker's imitation ability. Moreover, this text-independent model is scalable and capable of handling L1-induced speaker variability.
【7】 Assessing Phrase Break of ESL Speech with Pre-trained Language Models  and Large Language Models标题：基于预训练语言模型和大语言模型的ESL语音断词评估链接：https://arxiv.org/abs/2306.04980
作者：Zhiyi Wang,Shaoguang Mao,Wenshan Wu,Yan Xia,Yan Deng,Jonathan Tien机构：Peking University, Beijing, China, Microsoft Research Asia, Beijing, China备注：Accepted by InterSpeech 2023. arXiv admin note: substantial text overlap with arXiv:2210.16029摘要：本文介绍了使用预训练语言模型（PLM）和大语言模型（LLM）评估ESL学习者语音中短语中断的方法。有两个任务：对语音剪辑的短语中断的总体评估和对每个可能的短语中断位置的细粒度评估。为了利用NLP模型，语音输入首先与文本强制对齐，然后预处理为令牌序列，包括单词和短语中断信息。为了利用PLM，我们提出了一个预训练和微调管道与处理的令牌。该过程包括使用替换的中断标记检测模块进行预训练，以及使用文本分类和序列标记进行微调。为了使用LLM，我们为ChatGPT设计了提示符。实验结果表明，与PLM，对标记的训练数据的依赖已大大减少，并提高了性能。同时，我们验证了ChatGPT，一个著名的法学硕士，在这方面有进一步发展的潜力。摘要：This work introduces approaches to assessing phrase breaks in ESL learners' speech using pre-trained language models (PLMs) and large language models (LLMs). There are two tasks: overall assessment of phrase break for a speech clip and fine-grained assessment of every possible phrase break position. To leverage NLP models, speech input is first force-aligned with texts, and then pre-processed into a token sequence, including words and phrase break information. To utilize PLMs, we propose a pre-training and fine-tuning pipeline with the processed tokens. This process includes pre-training with a replaced break token detection module and fine-tuning with text classification and sequence labeling. To employ LLMs, we design prompts for ChatGPT. The experiments show that with the PLMs, the dependence on labeled training data has been greatly reduced, and the performance has improved. Meanwhile, we verify that ChatGPT, a renowned LLM, has potential for further advancement in this area.
【8】 Adaptive Fake Audio Detection with Low-Rank Model Squeezing标题：基于低阶模型压缩的自适应伪音频检测链接：https://arxiv.org/abs/2306.04956
作者：Xiaohui Zhang,Jiangyan Yi,Jianhua Tao,Chenlong Wang,Le Xu,Ruibo Fu机构：State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences, School of Computer and Information Technology, Beijing Jiaotong University, University of Science and Technology of China摘要：欺骗算法的快速发展需要开发能够准确识别新兴虚假音频的鲁棒检测方法。传统的方法，例如对包含这些新颖的欺骗算法的新数据集进行微调，是计算密集型的，并且造成损害已知的假音频类型的所获取的知识的风险。为了应对这些挑战，本文提出了一种创新的方法，减轻了与微调相关的限制。我们引入了专门针对新出现的假音频类型的训练低秩自适应矩阵的概念。在推断阶段期间，这些自适应矩阵与现有模型组合以生成最终预测输出。进行了大量的实验，以评估所提出的方法的有效性。结果表明，我们的方法有效地保留了已知的假音频类型的现有模型的预测精度。此外，我们的方法提供了几个优点，包括减少存储内存的要求和较低的等错误率相比，传统的微调方法，特别是在特定的欺骗算法。摘要：The rapid advancement of spoofing algorithms necessitates the development of robust detection methods capable of accurately identifying emerging fake audio. Traditional approaches, such as finetuning on new datasets containing these novel spoofing algorithms, are computationally intensive and pose a risk of impairing the acquired knowledge of known fake audio types. To address these challenges, this paper proposes an innovative approach that mitigates the limitations associated with finetuning. We introduce the concept of training low-rank adaptation matrices tailored specifically to the newly emerging fake audio types. During the inference stage, these adaptation matrices are combined with the existing model to generate the final prediction output. Extensive experimentation is conducted to evaluate the efficacy of the proposed method. The results demonstrate that our approach effectively preserves the prediction accuracy of the existing model for known fake audio types. Furthermore, our approach offers several advantages, including reduced storage memory requirements and lower equal error rates compared to conventional finetuning methods, particularly on specific spoofing algorithms.
【9】 Towards Ultrasound Tongue Image prediction from EEG during speech  production标题：语音产生过程中基于脑电的超声舌图像预测链接：https://arxiv.org/abs/2306.05374
作者：Tamás Gábor Csapó,Frigyes Viktor Arthur,Péter Nagy,Ádám Boncz机构：Department of Telecommunications and Media Informatics, Budapest University of Technology and Economics (BME), Budapest, Hungary, Department of Measurement and Information Systems, BME, Budapest, Hungary备注：accepted at Interspeech 2023摘要：已经进行了先前的初始研究以提出使用脑信号（例如~非侵入性EEG和侵入性sEEG / ECoG），但是缺乏将非侵入性脑、发音和语音信号一起研究并分析脑中的认知过程、发音运动的运动学和所得语音信号的组合方法。在本文中，我们描述了我们的多模式（脑电图，超声舌成像和语音）的分析和合成实验，作为一个可行性研究。我们扩展了基于超声的清晰度数据的语音生产过程中记录的大脑信号的分析。根据EEG测量的大脑信号，我们用完全连接的深度神经网络预测舌头的超声图像。结果表明，脑电和超声舌象之间存在一种微弱但明显的关系，即脑电舌象与超声舌象之间的关系。该网络可以区分发音语音和中性舌头位置。摘要：Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g.~non-invasive EEG and invasive sEEG / ECoG), but there is a lack of combined methods that investigate non-invasive brain, articulation, and speech signals together and analyze the cognitive processes in the brain, the kinematics of the articulatory movement and the resulting speech signal. In this paper, we describe our multimodal (electroencephalography, ultrasound tongue imaging, and speech) analysis and synthesis experiments, as a feasibility study. We extend the analysis of brain signals recorded during speech production with ultrasound-based articulation data. From the brain signal measured with EEG, we predict ultrasound images of the tongue with a fully connected deep neural network. The results show that there is a weak but noticeable relationship between EEG and ultrasound tongue images, i.e. the network can differentiate articulated speech and neutral tongue position.
【10】 Matching Latent Encoding for Audio-Text based Keyword Spotting标题：基于音文的匹配隐含编码关键词检测链接：https://arxiv.org/abs/2306.05245
作者：Kumari Nishu,Minsik Cho,Devang Naik机构：Apple摘要：联合使用音频和文本嵌入进行关键词识别（KWS）已经显示出高质量的结果，但是如何在语义上对齐不同序列长度的多词关键词的两个嵌入的关键挑战在很大程度上仍然没有解决。在本文中，我们提出了一个基于音频文本的端到端的模型架构，灵活的关键字定位（KWS），它建立在学习音频和文本嵌入。我们的架构使用了一种新的动态编程为基础的算法，动态序列分割（DSP），以最佳分割的音频序列到相同的长度为基础的文字序列使用单调对齐的口语内容。我们提出的模型包括一个编码器块，以获得音频和文本嵌入，一个投影块，以项目个别的嵌入到一个共同的潜在空间，和一个音频文本对齐包含一个新的DSP算法，它对齐的音频和文本嵌入，以确定是否口语内容是相同的文本。实验结果表明，我们的DSP是更有效的比其他分区方案，和所提出的架构优于国家的最先进的结果在公共数据集上的ROC曲线下面积（AUC）和等错误率（EER）的14.4%和28.9%，分别。摘要：Using audio and text embeddings jointly for Keyword Spotting (KWS) has shown high-quality results, but the key challenge of how to semantically align two embeddings for multi-word keywords of different sequence lengths remains largely unsolved. In this paper, we propose an audio-text-based end-to-end model architecture for flexible keyword spotting (KWS), which builds upon learned audio and text embeddings. Our architecture uses a novel dynamic programming-based algorithm, Dynamic Sequence Partitioning (DSP), to optimally partition the audio sequence into the same length as the word-based text sequence using the monotonic alignment of spoken content. Our proposed model consists of an encoder block to get audio and text embeddings, a projector block to project individual embeddings to a common latent space, and an audio-text aligner containing a novel DSP algorithm, which aligns the audio and text embeddings to determine if the spoken content is the same as the text. Experimental results show that our DSP is more effective than other partitioning schemes, and the proposed architecture outperformed the state-of-the-art results on the public dataset in terms of Area Under the ROC Curve (AUC) and Equal-Error-Rate (EER) by 14.4 % and 28.9%, respectively.
【11】 VIFS: An End-to-End Variational Inference for Foley Sound Synthesis标题：VIFS：Foley声音合成的端到端变分推理链接：https://arxiv.org/abs/2306.05004
作者：Junhyeok Lee,Hyeonuk Nam,Yong-Hwa Park机构：maum.ai Inc., Republic of Korea,  Korea Advanced Institute of Science and Technology, Republic of Korea备注：DCASE 2023 Challenge Task 7摘要：DCASE 2023挑战任务7的目标是通过“类别到声音”的方法为Foley声音合成（FSS）生成各种声音片段。“类”由单一的索引表示，而相应的“音”则涵盖了多样性和不同的音例。为了生成不同的声音为一个给定的类别，我们采用VITS，一个文本到语音（TTS）模型与变分推理。此外，我们应用了语音合成的各种技术，包括PhaseAug和Avocodo。与从音素和说话者身份生成短发音的TTS模型不同，类别到声音的问题需要仅从类别索引生成不同的声音。为了补偿差异，同时保持每个音频片段内的一致性，我们对先前的编码器进行了大量修改，以增强与后验潜在变量的一致性。这在现有编码器上引入了额外的高斯，这促进了类别内的方差。通过这些修改，我们提出了VIFS，变分推断端到端的福利声音合成，产生不同的高品质的声音。摘要：The goal of DCASE 2023 Challenge Task 7 is to generate various sound clips for Foley sound synthesis (FSS) by "category-to-sound" approach. "Category" is expressed by a single index while corresponding "sound" covers diverse and different sound examples. To generate diverse sounds for a given category, we adopt VITS, a text-to-speech (TTS) model with variational inference. In addition, we apply various techniques from speech synthesis including PhaseAug and Avocodo. Different from TTS models which generate short pronunciation from phonemes and speaker identity, the category-to-sound problem requires generating diverse sounds just from a category index. To compensate for the difference while maintaining consistency within each audio clip, we heavily modified the prior encoder to enhance consistency with posterior latent variables. This introduced additional Gaussian on the prior encoder which promotes variance within the category. With these modifications, we propose VIFS, variational inference for end-to-end Foley sound synthesis, which generates diverse high-quality sounds.
【12】 Two-stage Autoencoder Neural Network for 3D Speech Enhancement标题：用于3D语音增强的两级自编码神经网络链接：https://arxiv.org/abs/2306.04987
作者：Han Yin,Jisheng Bai,Siwei Huang,Mou Wang,Yafei Jia,Jianfeng Chen机构：Joint Laboratory of Environmental Sound Sensing, School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, China备注：5 pages,5 figures摘要：近年来，随着增强现实技术的发展，三维语音增强技术引起了人们的广泛关注。传统的去噪卷积自动编码器在提取动态语音信息方面存在局限性。在本文中，我们提出了一个两阶段的自动编码器神经网络的三维语音增强。我们将双路径递归神经网络块集成到卷积自动编码器中，以交替方式迭代应用时域和频域建模。提出了一种融合高维特征的注意机制。我们还引入了一个损失函数，同时优化网络的时频域和时域。实验结果表明，我们的系统优于国家的最先进的系统上的数据集ICASSP L3DAS23挑战。摘要：3D speech enhancement has attracted much attention in recent years with the development of augmented reality technology. Traditional denoising convolutional autoencoders have limitations in extracting dynamic voice information. In this paper, we propose a two-stage autoencoder neural network for 3D speech enhancement. We incorporate a dual-path recurrent neural network block into the convolutional autoencoder to iteratively apply time-domain and frequency-domain modeling in an alternate fashion. And an attention mechanism for fusing the high-dimension features is proposed. We also introduce a loss function to simultaneously optimize the network in the time-frequency and time domains. Experimental results show that our system outperforms the state-of-the-art systems on the dataset of ICASSP L3DAS23 challenge.
【13】 Modulation Classification Through Deep Learning Using Resolution  Transformed Spectrograms标题：基于分辨率变换谱图的深度学习调制分类链接：https://arxiv.org/abs/2306.04655
作者：Muhammad Waqas,Muhammad Ashraf,Muhammad Zakwan机构：Department of Avionics Engineering, Air University, Islamabad备注：15 pages, 12 figures摘要：调制方式识别是信号处理的重要环节，在通信领域有着广泛的应用。由于频率相对于时间的变化在具有不同调制格式的无线电信号之间保持重要的区别，因此这些变化可以通过将1-D无线电信号转换到频域来用于特征提取。在本文中，我们提出了一个自动调制分类（AMC）的方案，使用卷积神经网络（CNN）的现代架构，通过生成11个不同的调制类型的频谱图像。此外，我们执行频谱图的分辨率转换，结果高达99.61%的计算负载减少和8倍的速度从接收到的I/Q数据的转换。提出的AMC在CPU和GPU上实现，以识别信号上的数字和模拟信号调制方案。在现有的CNN模型上评估性能，包括SqueezeNet，Resnet-50，InceptionResnet-V2，Inception-V3，VGG-16和Densenet-201。在信号中存在AWGN和其他噪声损伤的情况下，实现了91.2%的最佳结果，说明变换后的基于频谱图的AMC具有良好的分类准确性，因为频谱特征具有高度的判别性，并且基于CNN的模型具有提取这些高维特征的能力。在5 ~ 30 db的不同信噪比下，以5db的步长生成频谱图，观察不同信噪比下的实验结果。所提出的方法是有效的应用在无线通信网络中的实时应用。摘要：Modulation classification is an essential step of signal processing and has been regularly applied in the field of tele-communication. Since variations of frequency with respect to time remains a vital distinction among radio signals having different modulation formats, these variations can be used for feature extraction by converting 1-D radio signals into frequency domain. In this paper, we propose a scheme for Automatic Modulation Classification (AMC) using modern architectures of Convolutional Neural Networks (CNN), through generating spectrum images of eleven different modulation types. Additionally, we perform resolution transformation of spectrograms that results up to 99.61% of computational load reduction and 8x faster conversion from the received I/Q data. This proposed AMC is implemented on CPU and GPU, to recognize digital as well as analogue signal modulation schemes on signals. The performance is evaluated on existing CNN models including SqueezeNet, Resnet-50, InceptionResnet-V2, Inception-V3, VGG-16 and Densenet-201. Best results of 91.2% are achieved in presence of AWGN and other noise impairments in the signals, stating that the transformed spectrogram-based AMC has good classification accuracy as the spectral features are highly discriminant, and CNN based models have capability to extract these high-dimensional features. The spectrograms were created under different SNRs ranging from 5 to 30db with a step size of 5db to observe the experimental results at various SNR levels. The proposed methodology is efficient to be applied in wireless communication networks for real-time applications.
eess.AS音频处理
【1】 Matching Latent Encoding for Audio-Text based Keyword Spotting标题：基于音文的匹配隐含编码关键词检测链接：https://arxiv.org/abs/2306.05245
作者：Kumari Nishu,Minsik Cho,Devang Naik机构：Apple摘要：联合使用音频和文本嵌入进行关键词识别（KWS）已经显示出高质量的结果，但是如何在语义上对齐不同序列长度的多词关键词的两个嵌入的关键挑战在很大程度上仍然没有解决。在本文中，我们提出了一个基于音频文本的端到端的模型架构，灵活的关键字定位（KWS），它建立在学习音频和文本嵌入。我们的架构使用了一种新的动态编程为基础的算法，动态序列分割（DSP），以最佳分割的音频序列到相同的长度为基础的文字序列使用单调对齐的口语内容。我们提出的模型包括一个编码器块，以获得音频和文本嵌入，一个投影块，以项目个别的嵌入到一个共同的潜在空间，和一个音频文本对齐包含一个新的DSP算法，它对齐的音频和文本嵌入，以确定是否口语内容是相同的文本。实验结果表明，我们的DSP是更有效的比其他分区方案，和所提出的架构优于国家的最先进的结果在公共数据集上的ROC曲线下面积（AUC）和等错误率（EER）的14.4%和28.9%，分别。摘要：Using audio and text embeddings jointly for Keyword Spotting (KWS) has shown high-quality results, but the key challenge of how to semantically align two embeddings for multi-word keywords of different sequence lengths remains largely unsolved. In this paper, we propose an audio-text-based end-to-end model architecture for flexible keyword spotting (KWS), which builds upon learned audio and text embeddings. Our architecture uses a novel dynamic programming-based algorithm, Dynamic Sequence Partitioning (DSP), to optimally partition the audio sequence into the same length as the word-based text sequence using the monotonic alignment of spoken content. Our proposed model consists of an encoder block to get audio and text embeddings, a projector block to project individual embeddings to a common latent space, and an audio-text aligner containing a novel DSP algorithm, which aligns the audio and text embeddings to determine if the spoken content is the same as the text. Experimental results show that our DSP is more effective than other partitioning schemes, and the proposed architecture outperformed the state-of-the-art results on the public dataset in terms of Area Under the ROC Curve (AUC) and Equal-Error-Rate (EER) by 14.4 % and 28.9%, respectively.
【2】 VIFS: An End-to-End Variational Inference for Foley Sound Synthesis标题：VIFS：Foley声音合成的端到端变分推理链接：https://arxiv.org/abs/2306.05004
作者：Junhyeok Lee,Hyeonuk Nam,Yong-Hwa Park机构：maum.ai Inc., Republic of Korea,  Korea Advanced Institute of Science and Technology, Republic of Korea备注：DCASE 2023 Challenge Task 7摘要：DCASE 2023挑战任务7的目标是通过“类别到声音”的方法为Foley声音合成（FSS）生成各种声音片段。“类”由单一的索引表示，而相应的“音”则涵盖了多样性和不同的音例。为了生成不同的声音为一个给定的类别，我们采用VITS，一个文本到语音（TTS）模型与变分推理。此外，我们应用了语音合成的各种技术，包括PhaseAug和Avocodo。与从音素和说话者身份生成短发音的TTS模型不同，类别到声音的问题需要仅从类别索引生成不同的声音。为了补偿差异，同时保持每个音频片段内的一致性，我们对先前的编码器进行了大量修改，以增强与后验潜在变量的一致性。这在现有编码器上引入了额外的高斯，这促进了类别内的方差。通过这些修改，我们提出了VIFS，变分推断端到端的福利声音合成，产生不同的高品质的声音。摘要：The goal of DCASE 2023 Challenge Task 7 is to generate various sound clips for Foley sound synthesis (FSS) by "category-to-sound" approach. "Category" is expressed by a single index while corresponding "sound" covers diverse and different sound examples. To generate diverse sounds for a given category, we adopt VITS, a text-to-speech (TTS) model with variational inference. In addition, we apply various techniques from speech synthesis including PhaseAug and Avocodo. Different from TTS models which generate short pronunciation from phonemes and speaker identity, the category-to-sound problem requires generating diverse sounds just from a category index. To compensate for the difference while maintaining consistency within each audio clip, we heavily modified the prior encoder to enhance consistency with posterior latent variables. This introduced additional Gaussian on the prior encoder which promotes variance within the category. With these modifications, we propose VIFS, variational inference for end-to-end Foley sound synthesis, which generates diverse high-quality sounds.
【3】 Two-stage Autoencoder Neural Network for 3D Speech Enhancement标题：用于3D语音增强的两级自编码神经网络链接：https://arxiv.org/abs/2306.04987
作者：Han Yin,Jisheng Bai,Siwei Huang,Mou Wang,Yafei Jia,Jianfeng Chen机构：Joint Laboratory of Environmental Sound Sensing, School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, China备注：5 pages,5 figures摘要：近年来，随着增强现实技术的发展，三维语音增强技术引起了人们的广泛关注。传统的去噪卷积自动编码器在提取动态语音信息方面存在局限性。在本文中，我们提出了一个两阶段的自动编码器神经网络的三维语音增强。我们将双路径递归神经网络块集成到卷积自动编码器中，以交替方式迭代应用时域和频域建模。提出了一种融合高维特征的注意机制。我们还引入了一个损失函数，同时优化网络的时频域和时域。实验结果表明，我们的系统优于国家的最先进的系统上的数据集ICASSP L3DAS23挑战。摘要：3D speech enhancement has attracted much attention in recent years with the development of augmented reality technology. Traditional denoising convolutional autoencoders have limitations in extracting dynamic voice information. In this paper, we propose a two-stage autoencoder neural network for 3D speech enhancement. We incorporate a dual-path recurrent neural network block into the convolutional autoencoder to iteratively apply time-domain and frequency-domain modeling in an alternate fashion. And an attention mechanism for fusing the high-dimension features is proposed. We also introduce a loss function to simultaneously optimize the network in the time-frequency and time domains. Experimental results show that our system outperforms the state-of-the-art systems on the dataset of ICASSP L3DAS23 challenge.
【4】 Modulation Classification Through Deep Learning Using Resolution  Transformed Spectrograms标题：基于分辨率变换谱图的深度学习调制分类链接：https://arxiv.org/abs/2306.04655
作者：Muhammad Waqas,Muhammad Ashraf,Muhammad Zakwan机构：Department of Avionics Engineering, Air University, Islamabad备注：15 pages, 12 figures摘要：调制方式识别是信号处理的重要环节，在通信领域有着广泛的应用。由于频率相对于时间的变化在具有不同调制格式的无线电信号之间保持重要的区别，因此这些变化可以通过将1-D无线电信号转换到频域来用于特征提取。在本文中，我们提出了一个自动调制分类（AMC）的方案，使用卷积神经网络（CNN）的现代架构，通过生成11个不同的调制类型的频谱图像。此外，我们执行频谱图的分辨率转换，结果高达99.61%的计算负载减少和8倍的速度从接收到的I/Q数据的转换。提出的AMC在CPU和GPU上实现，以识别信号上的数字和模拟信号调制方案。在现有的CNN模型上评估性能，包括SqueezeNet，Resnet-50，InceptionResnet-V2，Inception-V3，VGG-16和Densenet-201。在信号中存在AWGN和其他噪声损伤的情况下，实现了91.2%的最佳结果，说明变换后的基于频谱图的AMC具有良好的分类准确性，因为频谱特征具有高度的判别性，并且基于CNN的模型具有提取这些高维特征的能力。在5 ~ 30 db的不同信噪比下，以5db的步长生成频谱图，观察不同信噪比下的实验结果。所提出的方法是有效的应用在无线通信网络中的实时应用。摘要：Modulation classification is an essential step of signal processing and has been regularly applied in the field of tele-communication. Since variations of frequency with respect to time remains a vital distinction among radio signals having different modulation formats, these variations can be used for feature extraction by converting 1-D radio signals into frequency domain. In this paper, we propose a scheme for Automatic Modulation Classification (AMC) using modern architectures of Convolutional Neural Networks (CNN), through generating spectrum images of eleven different modulation types. Additionally, we perform resolution transformation of spectrograms that results up to 99.61% of computational load reduction and 8x faster conversion from the received I/Q data. This proposed AMC is implemented on CPU and GPU, to recognize digital as well as analogue signal modulation schemes on signals. The performance is evaluated on existing CNN models including SqueezeNet, Resnet-50, InceptionResnet-V2, Inception-V3, VGG-16 and Densenet-201. Best results of 91.2% are achieved in presence of AWGN and other noise impairments in the signals, stating that the transformed spectrogram-based AMC has good classification accuracy as the spectral features are highly discriminant, and CNN based models have capability to extract these high-dimensional features. The spectrograms were created under different SNRs ranging from 5 to 30db with a step size of 5db to observe the experimental results at various SNR levels. The proposed methodology is efficient to be applied in wireless communication networks for real-time applications.
【5】 Towards Ultrasound Tongue Image prediction from EEG during speech  production标题：语音产生过程中基于脑电的超声舌图像预测链接：https://arxiv.org/abs/2306.05374
作者：Tamás Gábor Csapó,Frigyes Viktor Arthur,Péter Nagy,Ádám Boncz机构：Department of Telecommunications and Media Informatics, Budapest University of Technology and Economics (BME), Budapest, Hungary, Department of Measurement and Information Systems, BME, Budapest, Hungary备注：accepted at Interspeech 2023摘要：已经进行了先前的初始研究以提出使用脑信号（例如~非侵入性EEG和侵入性sEEG / ECoG），但是缺乏将非侵入性脑、发音和语音信号一起研究并分析脑中的认知过程、发音运动的运动学和所得语音信号的组合方法。在本文中，我们描述了我们的多模式（脑电图，超声舌成像和语音）的分析和合成实验，作为一个可行性研究。我们扩展了基于超声的清晰度数据的语音生产过程中记录的大脑信号的分析。根据EEG测量的大脑信号，我们用完全连接的深度神经网络预测舌头的超声图像。结果表明，脑电和超声舌象之间存在一种微弱但明显的关系，即脑电舌象与超声舌象之间的关系。该网络可以区分发音语音和中性舌头位置。摘要：Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g.~non-invasive EEG and invasive sEEG / ECoG), but there is a lack of combined methods that investigate non-invasive brain, articulation, and speech signals together and analyze the cognitive processes in the brain, the kinematics of the articulatory movement and the resulting speech signal. In this paper, we describe our multimodal (electroencephalography, ultrasound tongue imaging, and speech) analysis and synthesis experiments, as a feasibility study. We extend the analysis of brain signals recorded during speech production with ultrasound-based articulation data. From the brain signal measured with EEG, we predict ultrasound images of the tongue with a fully connected deep neural network. The results show that there is a weak but noticeable relationship between EEG and ultrasound tongue images, i.e. the network can differentiate articulated speech and neutral tongue position.
【6】 Trustworthy Sensor Fusion against Inaudible Command Attacks in Advanced  Driver-Assistance System标题：高级驾驶员辅助系统中抗隐形指令攻击的可信传感器融合链接：https://arxiv.org/abs/2306.05358
作者：Jiwei Guan,Lei Pan,Chen Wang,Shui Yu,Longxiang Gao,Xi Zheng机构：Lei Pan is with the School of Information Technology摘要：人们越来越担心自动驾驶汽车受到恶意攻击。特别地，随着语音命令在自动驾驶系统中变得可用，听不见的语音命令攻击构成了重大威胁。如何根据经验防御这些听不见的攻击仍然是一个悬而未决的问题。以前的研究调查了利用基于深度学习的多模态融合进行防御，而没有考虑模型可信度的不确定性。随着深度学习被应用于越来越敏感的任务，不确定性测量在帮助提高模型鲁棒性方面至关重要，特别是在关键任务场景中。在本文中，我们提出了多模态融合框架（MFF）作为一个智能安全系统，以抵御听不见的语音命令攻击。MFF融合异构视听模态使用VGG族神经网络，并实现了检测准确率为92.25%的比较融合方法的实证研究。此外，大量的视听任务的实验揭示了模型的不确定性。使用预期校准误差，我们测量校准误差和蒙特卡罗丢弃估计所提出的模型的预测分布。我们的研究结果表明，经验训练强大的多模态模型，提高标准的准确性，并提供了进一步的可解释性。最后，我们讨论了我们的方法的优点和缺点，其适用于先进的驾驶辅助系统。摘要：There are increasing concerns about malicious attacks on autonomous vehicles. In particular, inaudible voice command attacks pose a significant threat as voice commands become available in autonomous driving systems. How to empirically defend against these inaudible attacks remains an open question. Previous research investigates utilizing deep learning-based multimodal fusion for defense, without considering the model uncertainty in trustworthiness. As deep learning has been applied to increasingly sensitive tasks, uncertainty measurement is crucial in helping improve model robustness, especially in mission-critical scenarios. In this paper, we propose the Multimodal Fusion Framework (MFF) as an intelligent security system to defend against inaudible voice command attacks. MFF fuses heterogeneous audio-vision modalities using VGG family neural networks and achieves the detection accuracy of 92.25% in the comparative fusion method empirical study. Additionally, extensive experiments on audio-vision tasks reveal the model's uncertainty. Using Expected Calibration Errors, we measure calibration errors and Monte-Carlo Dropout to estimate the predictive distribution for the proposed models. Our findings show empirically to train robust multimodal models, improve standard accuracy and provide a further step toward interpretability. Finally, we discuss the pros and cons of our approach and its applicability for Advanced Driver Assistance Systems.
【7】 PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches  For Speech Emotion Recognition Using Pre-trained Speech Models标题：PEFT-SER：参数高效转移学习方法在基于预训练语音模型的语音情感识别中的应用链接：https://arxiv.org/abs/2306.05350
作者：Tiantian Feng,Shrikanth Narayanan机构：Department of Computer Science, University of Southern California, Los Angeles, USA, Department of Electrical and Computer Engineering备注：This work was accepted to the 11th International Conference on Affective Computing and Intelligent Interaction (ACII), 2023摘要：最近的许多研究都集中在微调预训练模型的语音情感识别（SER），导致有前途的性能相比，传统的方法，主要依赖于低层次的，知识启发的声学特征。这些预先训练的语音模型使用来自大规模数据集的自监督或弱监督学习目标来学习通用语音表示。尽管通过使用预训练架构在SER中取得了重大进展，但针对不同数据集微调这些大型预训练模型需要保存整个权重参数的副本，这使得它们在现实环境中部署不切实际。作为一种替代方案，这项工作探讨了参数有效的微调（PEFT）方法，用于适应情绪识别的预训练语音模型。具体来说，我们评估的适配器调谐，嵌入提示调谐，和LoRa（低秩近似）的四个流行的SER测试平台的功效。我们的研究结果表明，LoRa在情感识别方面实现了最佳的微调性能，同时提高了公平性，并且只需要最少的额外权重参数。此外，我们的研究结果提供了新的见解SER未来的研究方向，不同于现有的方法专注于直接微调模型架构。我们的代码可通过以下方式公开获取：https://github.com/usc-sail/peft-ser。摘要：Many recent studies have focused on fine-tuning pre-trained models for speech emotion recognition (SER), resulting in promising performance compared to traditional methods that rely largely on low-level, knowledge-inspired acoustic features. These pre-trained speech models learn general-purpose speech representations using self-supervised or weakly-supervised learning objectives from large-scale datasets. Despite the significant advances made in SER through the use of pre-trained architecture, fine-tuning these large pre-trained models for different datasets requires saving copies of entire weight parameters, rendering them impractical to deploy in real-world settings. As an alternative, this work explores parameter-efficient fine-tuning (PEFT) approaches for adapting pre-trained speech models for emotion recognition. Specifically, we evaluate the efficacy of adapter tuning, embedding prompt tuning, and LoRa (Low-rank approximation) on four popular SER testbeds. Our results reveal that LoRa achieves the best fine-tuning performance in emotion recognition while enhancing fairness and requiring only a minimal extra amount of weight parameters. Furthermore, our findings offer novel insights into future research directions in SER, distinct from existing approaches focusing on directly fine-tuning the model architecture. Our code is publicly available under: https://github.com/usc-sail/peft-ser.
【8】 Simple and Controllable Music Generation标题：简单可控的音乐生成链接：https://arxiv.org/abs/2306.05284
作者：Jade Copet,Felix Kreuk,Itai Gat,Tal Remez,David Kant,Gabriel Synnaeve,Yossi Adi,Alexandre Défossez机构：♠: equal contributions, ♢: core team, Meta AI摘要：我们解决了条件音乐生成的任务。我们引入MusicGen，一个单一的语言模型（LM），它在几个压缩的离散音乐表示流上操作，即，代币与先前的工作不同，MusicGen由单级Transformer LM以及高效的令牌交织模式组成，这消除了对级联几个模型的需要，例如，分层或上采样。按照这种方法，我们演示了MusicGen如何生成高质量的样本，同时以文本描述或旋律特征为条件，可以更好地控制所生成的输出。我们进行了广泛的实证评估，考虑自动和人类的研究，显示所提出的方法是优于标准的文本到音乐基准的评估基线。通过消融研究，我们揭示了组成MusicGen的每个组件的重要性。音乐样本、代码和模型可在www.example.com上获得https://github.com/facebookresearch/audiocraft。摘要：We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen. Music samples, code, and models are available at https://github.com/facebookresearch/audiocraft.
【9】 The ART of Conversation: Measuring Phonetic Convergence and Deliberate  Imitation in L2-Speech with a Siamese RNN标题：会话艺术：用暹罗语RNN测量二语语音中的语音融合和刻意模仿链接：https://arxiv.org/abs/2306.05088
作者：Zheng Yuan,Aldo Pastore,Dorina de Jong,Hao Xu,Luciano Fadiga,Alessandro D'Ausilio机构：Alessandro D’Ausilio, Istituto Italiano di Tecnologia, Italy, Universita degli Studi di Ferrara, Italy,  University of California San Diego, USA备注：Accepted at INTERSPEECH 2023摘要：语音趋同是指会话中两个对话者自动的、无意识的言语适应。本文提出了一种连体递归神经网络（RNN）架构来衡量收敛的整体频谱特性的语音在L2-L2的互动。我们扩展了交替阅读任务（ART）数据集，增加了20个母语斯洛伐克L2英语。我们训练和测试暹罗RNN模型，以测量来自三个不同母语组的L2英语语音的语音收敛：意大利语（9对）、法语（10对）和斯洛伐克语（10对）。我们的研究结果表明，Siamese RNN模型有效地捕捉到语音收敛和说话人的模仿能力的动态。此外，这种文本无关的模型是可扩展的，能够处理L1引起的扬声器的变化。摘要：Phonetic convergence describes the automatic and unconscious speech adaptation of two interlocutors in a conversation. This paper proposes a Siamese recurrent neural network (RNN) architecture to measure the convergence of the holistic spectral characteristics of speech sounds in an L2-L2 interaction. We extend an alternating reading task (the ART) dataset by adding 20 native Slovak L2 English speakers. We train and test the Siamese RNN model to measure phonetic convergence of L2 English speech from three different native language groups: Italian (9 dyads), French (10 dyads) and Slovak (10 dyads). Our results indicate that the Siamese RNN model effectively captures the dynamics of phonetic convergence and the speaker's imitation ability. Moreover, this text-independent model is scalable and capable of handling L1-induced speaker variability.
【10】 Assessing Phrase Break of ESL Speech with Pre-trained Language Models  and Large Language Models标题：基于预训练语言模型和大语言模型的ESL语音断词评估链接：https://arxiv.org/abs/2306.04980
作者：Zhiyi Wang,Shaoguang Mao,Wenshan Wu,Yan Xia,Yan Deng,Jonathan Tien机构：Peking University, Beijing, China, Microsoft Research Asia, Beijing, China备注：Accepted by InterSpeech 2023. arXiv admin note: substantial text overlap with arXiv:2210.16029摘要：本文介绍了使用预训练语言模型（PLM）和大语言模型（LLM）评估ESL学习者语音中短语中断的方法。有两个任务：对语音剪辑的短语中断的总体评估和对每个可能的短语中断位置的细粒度评估。为了利用NLP模型，语音输入首先与文本强制对齐，然后预处理为令牌序列，包括单词和短语中断信息。为了利用PLM，我们提出了一个预训练和微调管道与处理的令牌。该过程包括使用替换的中断标记检测模块进行预训练，以及使用文本分类和序列标记进行微调。为了使用LLM，我们为ChatGPT设计了提示符。实验结果表明，与PLM，对标记的训练数据的依赖已大大减少，并提高了性能。同时，我们验证了ChatGPT，一个著名的法学硕士，在这方面有进一步发展的潜力。摘要：This work introduces approaches to assessing phrase breaks in ESL learners' speech using pre-trained language models (PLMs) and large language models (LLMs). There are two tasks: overall assessment of phrase break for a speech clip and fine-grained assessment of every possible phrase break position. To leverage NLP models, speech input is first force-aligned with texts, and then pre-processed into a token sequence, including words and phrase break information. To utilize PLMs, we propose a pre-training and fine-tuning pipeline with the processed tokens. This process includes pre-training with a replaced break token detection module and fine-tuning with text classification and sequence labeling. To employ LLMs, we design prompts for ChatGPT. The experiments show that with the PLMs, the dependence on labeled training data has been greatly reduced, and the performance has improved. Meanwhile, we verify that ChatGPT, a renowned LLM, has potential for further advancement in this area.
【11】 Adaptive Fake Audio Detection with Low-Rank Model Squeezing标题：基于低阶模型压缩的自适应伪音频检测链接：https://arxiv.org/abs/2306.04956
作者：Xiaohui Zhang,Jiangyan Yi,Jianhua Tao,Chenlong Wang,Le Xu,Ruibo Fu机构：State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences, School of Computer and Information Technology, Beijing Jiaotong University, University of Science and Technology of China摘要：欺骗算法的快速发展需要开发能够准确识别新兴虚假音频的鲁棒检测方法。传统的方法，例如对包含这些新颖的欺骗算法的新数据集进行微调，是计算密集型的，并且造成损害已知的假音频类型的所获取的知识的风险。为了应对这些挑战，本文提出了一种创新的方法，减轻了与微调相关的限制。我们引入了专门针对新出现的假音频类型的训练低秩自适应矩阵的概念。在推断阶段期间，这些自适应矩阵与现有模型组合以生成最终预测输出。进行了大量的实验，以评估所提出的方法的有效性。结果表明，我们的方法有效地保留了已知的假音频类型的现有模型的预测精度。此外，我们的方法提供了几个优点，包括减少存储内存的要求和较低的等错误率相比，传统的微调方法，特别是在特定的欺骗算法。摘要：The rapid advancement of spoofing algorithms necessitates the development of robust detection methods capable of accurately identifying emerging fake audio. Traditional approaches, such as finetuning on new datasets containing these novel spoofing algorithms, are computationally intensive and pose a risk of impairing the acquired knowledge of known fake audio types. To address these challenges, this paper proposes an innovative approach that mitigates the limitations associated with finetuning. We introduce the concept of training low-rank adaptation matrices tailored specifically to the newly emerging fake audio types. During the inference stage, these adaptation matrices are combined with the existing model to generate the final prediction output. Extensive experimentation is conducted to evaluate the efficacy of the proposed method. The results demonstrate that our approach effectively preserves the prediction accuracy of the existing model for known fake audio types. Furthermore, our approach offers several advantages, including reduced storage memory requirements and lower equal error rates compared to conventional finetuning methods, particularly on specific spoofing algorithms.
机器翻译，仅供参考