今日论文合集：cs.SD语音15篇，eess.AS音频处理16篇。
cs.SD语音
【1】 LocSelect: Target Speaker Localization with an Auditory Selective  Hearing Mechanism标题：LocSelect：具有听觉选择性听力机制的目标说话人定位链接：https://arxiv.org/abs/2310.10497
作者：Yu Chen,Xinyuan Qian,Zexu Pan,Kainan Chen,Haizhou Li机构：School of Computer and Communication Engineering, University of Science and Technology Beijing (USTB), China, School of Data Science, The Chinese University of Hong Kong, Shenzhen, China, Institute of Data Science, National University of Singapore, Singapore备注：Submitted to ICASSP 2024摘要：针对现有的抗噪声和抗混响的定位算法主要强调在多说话人环境中分离出每个说话人并为其提供定向输出，而不考虑说话人身份的问题，提出了一种基于选择性听觉机制的目标说话人定位算法。给定目标说话人的参考语音，我们首先产生一个说话人相关的频谱图掩模，以消除干扰说话人的语音。随后，采用长短期记忆（LSTM）网络从过滤后的频谱图中提取目标说话人的位置。实验验证了该方法在不同尺度不变信噪比（SNR）条件下的优越性。具体而言，在SNR = -10 dB时，我们提出的网络LocSelect实现了3.55的平均绝对误差（MAE）和87.40%的准确度（ACC）。摘要：The prevailing noise-resistant and reverberation-resistant localization algorithms primarily emphasize separating and providing directional output for each speaker in multi-speaker scenarios, without association with the identity of speakers.In this paper, we present a target speaker localization algorithm with a selective hearing mechanism. Given a reference speech of the target speaker, we first produce a speaker-dependent spectrogram mask to eliminate interfering speakers' speech. Subsequently, a Long short-term memory (LSTM) network is employed to extract the target speaker's location from the filtered spectrogram. Experiments validate the superiority of our proposed method over the existing algorithms for different scale invariant signal-to-noise ratios (SNR) conditions. Specifically, at SNR = -10 dB, our proposed network LocSelect achieves a mean absolute error (MAE) of 3.55 and an accuracy (ACC) of 87.40%.
【2】 BeatDance: A Beat-Based Model-Agnostic Contrastive Learning Framework  for Music-Dance Retrieval标题：BeatDance：一种基于节拍的模型不可知性音乐舞蹈检索对比学习框架链接：https://arxiv.org/abs/2310.10300
作者：Kaixing Yang,Xukun Zhou,Xulong Tang,Ran Diao,Hongyan Liu,Jun He,Zhaoxin Fan机构：Renmin University of China, The University of Texas at Dallas, Tsinghua University, Psyche AI Inc摘要：舞蹈和音乐是密切相关的表达形式，舞蹈视频和音乐之间的相互检索是教育，艺术和体育等各个领域的基本任务。然而，现有的方法往往遭受不自然的生成效果或未能充分探索音乐和舞蹈之间的相关性。为了克服这些挑战，我们提出了BeatDance，一个新的基于节拍的模型不可知的对比学习框架。BeatDance集成了节拍感知音乐舞蹈信息提取器，跨时间节拍混合器和节拍增强型集线器还原器，通过利用音乐节拍和舞蹈动作之间的对齐来提高舞蹈音乐检索性能。我们还介绍了音乐舞蹈（MD）数据集，这是一个大规模的音乐舞蹈视频对集合，用于训练和测试。MD数据集上的实验结果表明，我们的方法优于现有的基线，达到最先进的性能。代码和数据集将在接受后公开提供。摘要：Dance and music are closely related forms of expression, with mutual retrieval between dance videos and music being a fundamental task in various fields like education, art, and sports. However, existing methods often suffer from unnatural generation effects or fail to fully explore the correlation between music and dance. To overcome these challenges, we propose BeatDance, a novel beat-based model-agnostic contrastive learning framework. BeatDance incorporates a Beat-Aware Music-Dance InfoExtractor, a Trans-Temporal Beat Blender, and a Beat-Enhanced Hubness Reducer to improve dance-music retrieval performance by utilizing the alignment between music beats and dance movements. We also introduce the Music-Dance (MD) dataset, a large-scale collection of over 10,000 music-dance video pairs for training and testing. Experimental results on the MD dataset demonstrate the superiority of our method over existing baselines, achieving state-of-the-art performance. The code and dataset will be made public available upon acceptance.
【3】 Joint Music and Language Attention Models for Zero-shot Music Tagging标题：零拍音乐标注中的音乐和语言联合注意模型链接：https://arxiv.org/abs/2310.10159
作者：Xingjian Du,Zhesong Yu,Jiaju Lin,Bilei Zhu,Qiuqiang Kong机构：Bytedance ,The Chinese University of Hongkong备注：\begin{keywords} Music tagging, joint music and language attention models, Music Foundation Model. \end{keywords}摘要：音乐标注是一项预测音乐唱片标签的任务。然而，以往的音乐标注研究主要集中在封闭的音乐标注任务，不能推广到新的标签。在这项工作中，我们提出了一个zero-shot音乐标记系统建模的联合音乐和语言注意力（JMLA）模型，以解决开放集的音乐标记问题。JMLA模型由一个由预训练的掩码自动编码器建模的音频编码器和一个由Falcon 7 B建模的解码器组成。我们引入preceiver重采样器将任意长度的音频转换为固定长度的嵌入。我们在编码器和解码器层之间引入密集注意力连接，以改善编码器和解码器层之间的信息流。我们从互联网上收集了一个大规模的音乐和描述数据集。我们建议使用ChatGPT将原始描述转换为形式化和多样化的描述，以训练JMLA模型。我们提出的JMLA系统在GTZAN数据集上实现了64.82\% $的zero-shot音频标记精度，优于以前的zero-shot系统，并在FMA和MagnaTagATune数据集上实现了与以前的系统相当的结果。摘要：Music tagging is a task to predict the tags of music recordings. However, previous music tagging research primarily focuses on close-set music tagging tasks which can not be generalized to new tags. In this work, we propose a zero-shot music tagging system modeled by a joint music and language attention (JMLA) model to address the open-set music tagging problem. The JMLA model consists of an audio encoder modeled by a pretrained masked autoencoder and a decoder modeled by a Falcon7B. We introduce preceiver resampler to convert arbitrary length audio into fixed length embeddings. We introduce dense attention connections between encoder and decoder layers to improve the information flow between the encoder and decoder layers. We collect a large-scale music and description dataset from the internet. We propose to use ChatGPT to convert the raw descriptions into formalized and diverse descriptions to train the JMLA models. Our proposed JMLA system achieves a zero-shot audio tagging accuracy of $ 64.82\% $ on the GTZAN dataset, outperforming previous zero-shot systems and achieves comparable results to previous systems on the FMA and the MagnaTagATune datasets.
【4】 End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder  and Input Feature Analysis标题：端到端多声道说话人ASR：说话人引导译码及输入特征分析链接：https://arxiv.org/abs/2310.10106
作者：Can Cui,Imran Ahamad Sheikh,Mostafa Sadeghi,Emmanuel Vincent机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France, Vivoka, Metz, France备注：None摘要：我们提出了一个端到端的多通道说话人属性的自动语音识别（MC-SA-ASR）系统，它结合了一个基于一致性的编码器与多帧跨通道的注意和一个说话人属性的转换器为基础的解码器。据我们所知，这是第一个在多通道环境中有效集成ASR和说话人识别模块的模型。在LibriSpeech数据的模拟混合物上，与先前提出的单通道和多通道方法相比，我们的系统分别将字错误率（WER）降低了12%和16%。此外，我们研究了不同的输入功能，包括多通道的幅度和相位信息，对ASR性能的影响。最后，我们在AMI语料库上的实验证实了我们的系统对真实世界的多通道会议转录的有效性。摘要：We present an end-to-end multichannel speaker-attributed automatic speech recognition (MC-SA-ASR) system that combines a Conformer-based encoder with multi-frame crosschannel attention and a speaker-attributed Transformer-based decoder. To the best of our knowledge, this is the first model that efficiently integrates ASR and speaker identification modules in a multichannel setting. On simulated mixtures of LibriSpeech data, our system reduces the word error rate (WER) by up to 12% and 16% relative compared to previously proposed single-channel and multichannel approaches, respectively. Furthermore, we investigate the impact of different input features, including multichannel magnitude and phase information, on the ASR performance. Finally, our experiments on the AMI corpus confirm the effectiveness of our system for real-world multichannel meeting transcription.
【5】 MERTech: Instrument Playing Technique Detection Using Self-Supervised  Pretrained Model With Multi-Task Finetuning标题：MERTech：基于多任务精调的自监督预训练模型乐器演奏技巧检测链接：https://arxiv.org/abs/2310.09853
作者：Dichucheng Li,Yinghao Ma,Weixing Wei,Qiuqiang Kong,Yulun Wu,Mingjin Che,Fan Xia,Emmanouil Benetos,Wei Li机构：School of Computer Science and Technology, Fudan University, Shanghai, China,  Centre for Digital Music, Queen Mary University of London, UK, Multimodal Art Projection Research Community,  The Chinese University of Hong Kong (CUHK)备注：submitted to ICASSP 2024摘要：乐器演奏技巧是音乐表现力的重要组成部分。然而，自动IPT检测方法的发展受到有限的标记数据和固有的类不平衡问题的影响。在本文中，我们提出应用一个在大规模未标记音乐数据上预训练的自监督学习模型，并在IPT检测任务中对其进行微调。这种方法解决了数据稀缺和类别不平衡的挑战。认识到音高在捕捉IPT的细微差别和定位IPT事件的重要性，我们研究了多任务微调与音高和起始检测作为辅助任务。此外，我们应用后处理方法进行事件级预测，其中IPT激活仅在发作输出确认该帧中的发作时才启动事件。我们的方法在多个IPT基准数据集上的帧级和事件级指标上都优于以前的方法。进一步的实验证明了多任务微调对每个IPT类的有效性。摘要：Instrument playing techniques (IPTs) constitute a pivotal component of musical expression. However, the development of automatic IPT detection methods suffers from limited labeled data and inherent class imbalance issues. In this paper, we propose to apply a self-supervised learning model pre-trained on large-scale unlabeled music data and finetune it on IPT detection tasks. This approach addresses data scarcity and class imbalance challenges. Recognizing the significance of pitch in capturing the nuances of IPTs and the importance of onset in locating IPT events, we investigate multi-task finetuning with pitch and onset detection as auxiliary tasks. Additionally, we apply a post-processing approach for event-level prediction, where an IPT activation initiates an event only if the onset output confirms an onset in that frame. Our method outperforms prior approaches in both frame-level and event-level metrics across multiple IPT benchmark datasets. Further experiments demonstrate the efficacy of multi-task finetuning on each IPT class.
【6】 CoCoFormer: A controllable feature-rich polyphonic music generation  method标题：CoCoFormer：一种可控的特征丰富的复调音乐生成方法链接：https://arxiv.org/abs/2310.09843
作者：Jiuyang Zhou,Tengfei Niu,Hong Zhu,Xingping Wang机构：Xi’an University of Technology, Xi’an, China摘要：探讨了复调音乐序列的建模方法。由于Transformer模型在音乐生成方面的巨大潜力，可控音乐生成正受到更多关注。在复调音乐任务中，目前的可控生成研究主要集中在和弦的可控生成上，而对合唱音乐织体的可控生成缺乏精确的调节。本文提出了条件Choir Transformer（CoCoFormer），它通过在细粒度水平上控制和弦和节奏输入来控制模型的输出。本文提出的自监督方法改进了损失函数，通过条件控制输入和无条件输入训练进行联合训练。为了缓解教师强迫训练导致生成样本缺乏多样性的问题，本文增加了对抗训练方法。CoCoFormer通过对和弦和节奏的显式和隐式输入来增强模型性能。本文通过实验证明，CoCoFormer已经达到了目前比现有机型更好的水平。在规定复调音乐织体的前提下，同样的旋律也可以多种方式生成。摘要：This paper explores the modeling method of polyphonic music sequence. Due to the great potential of Transformer models in music generation, controllable music generation is receiving more attention. In the task of polyphonic music, current controllable generation research focuses on controlling the generation of chords, but lacks precise adjustment for the controllable generation of choral music textures. This paper proposed Condition Choir Transformer (CoCoFormer) which controls the output of the model by controlling the chord and rhythm inputs at a fine-grained level. In this paper, the self-supervised method improves the loss function and performs joint training through conditional control input and unconditional input training. In order to alleviate the lack of diversity on generated samples caused by the teacher forcing training, this paper added an adversarial training method. CoCoFormer enhances model performance with explicit and implicit inputs to chords and rhythms. In this paper, the experiments proves that CoCoFormer has reached the current better level than current models. On the premise of specifying the polyphonic music texture, the same melody can also be generated in a variety of ways.
【7】 Improved Contextual Recognition In Automatic Speech Recognition Systems  By Semantic Lattice Rescoring标题：语义格检索改进自动语音识别系统中的上下文识别链接：https://arxiv.org/abs/2310.09680
作者：Ankitha Sudarshan,Vinay Samuel,Parth Patwa,Ibtihel Amara,Aman Chadha机构：Purdue University ,Carnegie Mellon University ,University of California Los Angeles, McGill University ,Stanford University摘要：自动语音识别（ASR）是语音识别领域的一个研究热点。最近的突破为ASR系统带来了不同的前景，例如忠实地转录口语，这是构建会话代理的关键进步。然而，准确识别上下文相关的单词和短语仍然是一个迫在眉睫的挑战。在这项工作中，我们提出了一种新的方法，用于通过语义网格处理来增强ASR系统中的上下文识别，该方法利用深度学习模型的力量，在各种词汇和说话风格中准确地提供即时的翻译。我们的解决方案包括使用隐马尔可夫模型和高斯混合模型（HMM-GMM）以及深度神经网络（DNN）模型，该模型集成了语言和声学建模，以提高准确性。我们通过使用基于transformer的模型来注入我们的网络，以正确地重新划分单词网格，从而实现显着的功能，并显著降低单词错误率（WER）。我们证明了我们提出的框架的LibriSpeech数据集的实证分析的有效性。摘要：Automatic Speech Recognition (ASR) has witnessed a profound research interest. Recent breakthroughs have given ASR systems different prospects such as faithfully transcribing spoken language, which is a pivotal advancement in building conversational agents. However, there is still an imminent challenge of accurately discerning context-dependent words and phrases. In this work, we propose a novel approach for enhancing contextual recognition within ASR systems via semantic lattice processing leveraging the power of deep learning models in accurately delivering spot-on transcriptions across a wide variety of vocabularies and speaking styles. Our solution consists of using Hidden Markov Models and Gaussian Mixture Models (HMM-GMM) along with Deep Neural Networks (DNN) models integrating both language and acoustic modeling for better accuracy. We infused our network with the use of a transformer-based model to properly rescore the word lattice achieving remarkable capabilities with a palpable reduction in Word Error Rate (WER). We demonstrate the effectiveness of our proposed framework on the LibriSpeech dataset with empirical analyses.
【8】 SelfVC: Voice Conversion With Iterative Refinement using Self  Transformations标题：SelfVC：使用自变换进行迭代精化的语音转换链接：https://arxiv.org/abs/2310.09653
作者：Paarth Neekhara,Shehzeen Hussain,Rafael Valle,Boris Ginsburg,Rishabh Ranjan,Shlomo Dubnov,Farinaz Koushanfar,Julian McAuley机构：Boris Gingsburg, NVIDIA,UC San Diego, Equal Contribution摘要：我们提出了SelfVC，这是一种训练策略，可以用自合成的例子迭代地改进语音转换模型。以前的语音转换工作集中在明确地解开语音表示，分别编码扬声器的特点和语言内容。然而，使用特定于任务的损失项来解开语音表示以捕获这样的属性可能通过丢弃原始信号的更细微差别而导致信息损失。在这项工作中，而不是明确地解开与损失条款的属性，我们提出了一个框架，训练一个可控的语音转换模型纠缠的语音表示来自自监督学习和说话人验证模型。首先，我们开发的技术，以获得韵律信息的音频信号和SSL表示训练预测子模块的合成模型。接下来，我们提出了一种训练策略，通过使用自合成的示例创建具有挑战性的训练目标来迭代地改进语音转换的合成模型。在这种训练方法中，合成模型的当前状态用于生成话语的语音转换的变化，其用作重建任务的输入，确保模型的连续和有目的的细化。我们证明，在训练过程中纳入这样的自我合成的例子，提高了扬声器的相似性生成的语音相比，一个基线的语音转换模型只训练的扰动输入。SelfVC在没有任何文本的情况下进行训练，适用于一系列任务，如zero-shot语音转换，跨语言语音转换以及具有音高和节奏修改的可控语音合成。SelfVC在zero-shot语音转换中实现了最先进的结果，这些结果基于评估合成音频的自然度、扬声器相似性和可懂度的指标。摘要：We propose SelfVC, a training strategy to iteratively improve a voice conversion model with self-synthesized examples. Previous efforts on voice conversion focus on explicitly disentangling speech representations to separately encode speaker characteristics and linguistic content. However, disentangling speech representations to capture such attributes using task-specific loss terms can lead to information loss by discarding finer nuances of the original signal. In this work, instead of explicitly disentangling attributes with loss terms, we present a framework to train a controllable voice conversion model on entangled speech representations derived from self-supervised learning and speaker verification models. First, we develop techniques to derive prosodic information from the audio signal and SSL representations to train predictive submodules in the synthesis model. Next, we propose a training strategy to iteratively improve the synthesis model for voice conversion, by creating a challenging training objective using self-synthesized examples. In this training approach, the current state of the synthesis model is used to generate voice-converted variations of an utterance, which serve as inputs for the reconstruction task, ensuring a continuous and purposeful refinement of the model. We demonstrate that incorporating such self-synthesized examples during training improves the speaker similarity of generated speech as compared to a baseline voice conversion model trained solely on heuristically perturbed inputs. SelfVC is trained without any text and is applicable to a range of tasks such as zero-shot voice conversion, cross-lingual voice conversion, and controllable speech synthesis with pitch and pace modifications. SelfVC achieves state-of-the-art results in zero-shot voice conversion on metrics evaluating naturalness, speaker similarity, and intelligibility of synthesized audio.
【9】 Dynamic Prediction of Full-Ocean Depth SSP by Hierarchical LSTM: An  Experimental Result标题：基于分层LSTM的全海洋深度SSP动态预报：实验结果链接：https://arxiv.org/abs/2310.09522
作者：Jiajun Lu,Wei Huang,Hao Zhang机构：Ocean University of China摘要：SSP分布是水下定位、导航和定时（PNT）的一个重要参数，因为它影响着水声信号的传播模式。为了准确预测未来声速分布，提出了一种用于未来声速预测的分层长短期记忆（H-LSTM）神经网络，探索声速在时间维度上的分布规律。为了验证其可行性和有效性，我们进行了仿真和实际实验。海洋实验于2023年4月在南海举行。结果表明，该方法的准确性优于国家-的-的-最先进的方法。摘要：SSP distribution is an important parameter for underwater positioning, navigation and timing (PNT) because it affects the propagation mode of underwater acoustic signals. To accurate predict future sound speed distribution, we propose a hierarchical long short--term memory (H--LSTM) neural network for future sound speed prediction, which explore the distribution pattern of sound velocity in the time dimension. To verify the feasibility and effectiveness, we conducted both simulations and real experiments. The ocean experiment was held in the South China Sea in April, 2023. Results show that the accuracy of the proposed method outperforms the state--of--the--art methods.
【10】 Advancing Test-Time Adaptation for Acoustic Foundation Models in  Open-World Shifts标题：开放世界移动中声学基础模型的超前测试时间自适应链接：https://arxiv.org/abs/2310.09505
作者：Hongfu Liu,Hengguan Huang,Ye Wang机构：School of Computing, National University of Singapore摘要：测试时自适应（TTA）是解决推理过程中分布变化的一个重要范式，特别是在视觉识别任务中。然而，虽然声学模型面临着类似的挑战，由于分布变化的测试时间语音，TTA技术专门设计的声学建模的背景下，开放世界的数据变化仍然很少。考虑到声学基础模型的独特特性，这一差距进一步加剧：1）它们主要建立在具有层归一化的Transformer架构上，以及2）它们以非平稳方式处理不同长度的测试时间语音数据。这些方面使得视觉聚焦TTA方法的直接应用变得不可行，该方法主要依赖于批量归一化并假设独立样本。在本文中，我们深入研究了面向开放世界数据迁移的预训练声学模型的TTA。我们发现，嘈杂的，高熵的语音帧，往往是非无声的，进行关键的语义内容。传统的TTA方法可能会使用潜在的有缺陷的分析法无意中过滤掉这些信息。作为回应，我们引入了一个无经验的，基于学习的适应性增强信心。考虑到语音信号的短时一致性，我们在测试时间优化中也应用了一致性正则化。我们在合成和真实世界数据集上的实验证实了我们的方法优于现有基线。摘要：Test-Time Adaptation (TTA) is a critical paradigm for tackling distribution shifts during inference, especially in visual recognition tasks. However, while acoustic models face similar challenges due to distribution shifts in test-time speech, TTA techniques specifically designed for acoustic modeling in the context of open-world data shifts remain scarce. This gap is further exacerbated when considering the unique characteristics of acoustic foundation models: 1) they are primarily built on transformer architectures with layer normalization and 2) they deal with test-time speech data of varying lengths in a non-stationary manner. These aspects make the direct application of vision-focused TTA methods, which are mostly reliant on batch normalization and assume independent samples, infeasible. In this paper, we delve into TTA for pre-trained acoustic models facing open-world data shifts. We find that noisy, high-entropy speech frames, often non-silent, carry key semantic content. Traditional TTA methods might inadvertently filter out this information using potentially flawed heuristics. In response, we introduce a heuristic-free, learning-based adaptation enriched by confidence enhancement. Noting that speech signals' short-term consistency, we also apply consistency regularization during test-time optimization. Our experiments on synthetic and real-world datasets affirm our method's superiority over existing baselines.
【11】 SALM: Speech-augmented Language Model with In-context Learning for  Speech Recognition and Translation标题：SALM：支持语境学习的语音识别与翻译语音增强语言模型链接：https://arxiv.org/abs/2310.09424
作者：Zhehuai Chen,He Huang,Andrei Andrusenko,Oleksii Hrinchuk,Krishna C. Puvvada,Jason Li,Subhankar Ghosh,Jagadeesh Balam,Boris Ginsburg机构：NVIDIA, USA备注：submit to ICASSP 2024摘要：我们提出了一种新的语音增强语言模型（SALM）与{\em多任务}和{\em在上下文}学习能力。SALM包括冻结文本LLM、音频编码器、模态适配器模块和LoRA层，以适应语音输入和相关联的任务指令。统一的SALM不仅实现了与自动语音识别（ASR）和语音翻译（AST）的特定任务Conformer基线相当的性能，而且还表现出zero-shot上下文学习能力，通过ASR和AST的关键字提升任务证明了这一点。此外，提出了{\em speech supervised in-context training}来弥合LLM训练和下游语音任务之间的差距，这进一步增强了语音到文本模型的上下文学习能力。建议的模型是通过NeMo工具包开源的。摘要：We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, {\em speech supervised in-context training} is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.
【12】 Generation or Replication: Auscultating Audio Latent Diffusion Models标题：生成还是复制：Ausculating音频潜伏扩散模型链接：https://arxiv.org/abs/2310.10604
作者：Dimitrios Bralios,Gordon Wichern,François G. Germain,Zexu Pan,Sameer Khurana,Chiori Hori,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, University of Illinois Urbana-Champaign, Urbana, IL, USA备注：Submitted to ICASSP 2024摘要：音频潜在扩散模型的引入具有从文本描述按需生成逼真声音剪辑的能力，有可能彻底改变我们使用音频的方式。在这项工作中，我们通过研究音频输出如何与训练数据进行比较来初步尝试了解音频潜在扩散模型的内部工作，类似于医生如何通过倾听器官的声音来听诊患者。使用在AudioCaps数据集上训练的文本到音频潜在扩散模型，我们系统地分析了记忆行为作为训练集大小的函数。我们还评估了不同的检索指标的证据的训练数据记忆，发现梅尔频谱图之间的相似性是更强大的检测匹配比学习嵌入向量。在分析音频潜在扩散模型中的记忆过程中，我们还发现了AudioCaps数据库中大量重复的音频片段。摘要：The introduction of audio latent diffusion models possessing the ability to generate realistic sound clips on demand from a text description has the potential to revolutionize how we work with audio. In this work, we make an initial attempt at understanding the inner workings of audio latent diffusion models by investigating how their audio outputs compare with the training data, similar to how a doctor auscultates a patient by listening to the sounds of their organs. Using text-to-audio latent diffusion models trained on the AudioCaps dataset, we systematically analyze memorization behavior as a function of training set size. We also evaluate different retrieval metrics for evidence of training data memorization, finding the similarity between mel spectrograms to be more robust in detecting matches than learned embedding vectors. In the process of analyzing memorization in audio latent diffusion models, we also discover a large amount of duplicated audio clips within the AudioCaps database.
【13】 Advancing Audio Emotion and Intent Recognition with Large Pre-Trained  Models and Bayesian Inference标题：基于大预训练模型和贝叶斯推理的音频情感和意图识别链接：https://arxiv.org/abs/2310.10179
作者：Dejan Porjazovski,Yaroslav Getman,Tamás Grósz,Mikko Kurimo机构：Aalto University, Espoo, Finland备注：Accepted at ACMM 2023摘要：大型预训练模型在语言系统中至关重要，在情感识别和口吃检测等任务中表现出有效性。在本文中，我们使用大型预训练模型进行ACM多媒体计算副语言挑战，解决请求和情感共享任务。我们探索利用音频和文本模态的纯音频和混合解决方案。我们的实证结果一致显示的优越性的混合方法的音频模型。此外，我们引入贝叶斯层作为标准线性输出层的替代。多模态融合方法在HC请求上实现了85.4%UAR，在HC投诉上实现了60.2%。情感分享任务的集成模型产生的最佳rho值为0.614。在这项研究中探索的贝叶斯wav2vec2方法，使我们能够很容易地建立合奏，在微调只有一个模型的成本。此外，我们可以有可用的置信度值，而不是通常的过度自信的后验概率。摘要：Large pre-trained models are essential in paralinguistic systems, demonstrating effectiveness in tasks like emotion recognition and stuttering detection. In this paper, we employ large pre-trained models for the ACM Multimedia Computational Paralinguistics Challenge, addressing the Requests and Emotion Share tasks. We explore audio-only and hybrid solutions leveraging audio and text modalities. Our empirical results consistently show the superiority of the hybrid approaches over the audio-only models. Moreover, we introduce a Bayesian layer as an alternative to the standard linear output layer. The multimodal fusion approach achieves an 85.4% UAR on HC-Requests and 60.2% on HC-Complaints. The ensemble model for the Emotion Share task yields the best rho value of .614. The Bayesian wav2vec2 approach, explored in this study, allows us to easily build ensembles, at the cost of fine-tuning only one model. Moreover, we can have usable confidence values instead of the usual overconfident posterior probabilities.
【14】 Real-time Speech Enhancement and Separation with a Unified Deep Neural  Network for Single/Dual Talker Scenarios标题：基于统一深度神经网络的单/双人实时语音增强与分离链接：https://arxiv.org/abs/2310.10026
作者：Kashyap Patel,Anton Kovalyov,Issa Panahi机构：Electrical and Computer Engineering, University of Texas at Dallas, Richardson, USA备注：6 Pages, Accepted at IEEE Asilomar摘要：本文介绍了一种实用的方法，利用实时深度学习模型在语音增强和联合语音增强和分离之间交替，具体取决于输入混合物是否包含一个或两个活跃的说话人。尺度不变信号失真比（SI-SDR）已被证明是一个非常有效的训练措施，在时域语音分离。然而，SI-SDR度量对于零能量目标信号是不好定义的，这在使用具有不同数量的说话者的话语来训练语音分离模型时是一个问题。与专注于修改损失函数以适应零能量目标信号的现有解决方案不同，所提出的方法通过训练模型在其两个输出通道上提取语音来避免这个问题，而不管输入是单个还是双说话者混合。一个轻量级的扬声器重叠检测（SOD）模块也被引入到区分单和双说话人段实时。所提出的模块利用新的公式，直接对分离模型给出的分离掩模而不是原始混合物进行操作，从而有效地简化了检测任务。实验结果表明，该方法优于现有的解决方案，SOD模块表现出较高的准确性。摘要：This paper introduces a practical approach for leveraging a real-time deep learning model to alternate between speech enhancement and joint speech enhancement and separation depending on whether the input mixture contains one or two active speakers. Scale-invariant signal-to-distortion ratio (SI-SDR) has shown to be a highly effective training measure in time-domain speech separation. However, the SI-SDR metric is ill-defined for zero-energy target signals, which is a problem when training a speech separation model using utterances with varying numbers of talkers. Unlike existing solutions that focus on modifying the loss function to accommodate zero-energy target signals, the proposed approach circumvents this problem by training the model to extract speech on both its output channels regardless if the input is a single or dual-talker mixture. A lightweight speaker overlap detection (SOD) module is also introduced to differentiate between single and dual-talker segments in real-time. The proposed module takes advantage of the new formulation by operating directly on the separated masks, given by the separation model, instead of the original mixture, thus effectively simplifying the detection task. Experimental results show that the proposed training approach outperforms existing solutions, and the SOD module exhibits high accuracy.
【15】 CORN: Co-Trained Full-Reference And No-Reference Audio Metrics标题：COMM：联合训练的全参考和无参考音频指标链接：https://arxiv.org/abs/2310.09388
作者：Pranay Manocha,Donald Williamson,Adam Finkelstein机构：Department of Computer Science, Princeton University, Princeton, NJ, USA, Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA摘要：感知评价是各种音频处理任务的一个重要方面。完全参考（FR）或基于相似性的度量依赖于高质量的参考记录，可以将记录的较低质量或损坏版本与其进行比较以进行评估。相比之下，无参考（NR）度量在不依赖于参考的情况下评估记录。FR和NR方法都表现出相对于彼此的优点和缺点。在本文中，我们提出了一个名为CORN的新框架，它融合了这些双重方法，同时训练FR和NR模型。训练后，模型可以独立应用。我们通过预测几个常见的客观指标和两种不同的架构来评估CORN。使用CORN训练的NR模型在训练期间可以访问参考记录，因此，正如人们所期望的那样，它始终优于独立训练的基线NR模型。也许更值得注意的是，CORN FR模型也优于其基线对应模型，即使它依赖于相同的训练数据和相同的模型架构。因此，单个训练方案产生两个独立有用的模型，每个模型都优于独立训练的模型。摘要：Perceptual evaluation constitutes a crucial aspect of various audio-processing tasks. Full reference (FR) or similarity-based metrics rely on high-quality reference recordings, to which lower-quality or corrupted versions of the recording may be compared for evaluation. In contrast, no-reference (NR) metrics evaluate a recording without relying on a reference. Both the FR and NR approaches exhibit advantages and drawbacks relative to each other. In this paper, we present a novel framework called CORN that amalgamates these dual approaches, concurrently training both FR and NR models together. After training, the models can be applied independently. We evaluate CORN by predicting several common objective metrics and across two different architectures. The NR model trained using CORN has access to a reference recording during training, and thus, as one would expect, it consistently outperforms baseline NR models trained independently. Perhaps even more remarkable is that the CORN FR model also outperforms its baseline counterpart, even though it relies on the same training data and the same model architecture. Thus, a single training regime produces two independently useful models, each outperforming independently trained models.
eess.AS音频处理
【1】 Generation or Replication: Auscultating Audio Latent Diffusion Models标题：生成还是复制：Ausculating音频潜伏扩散模型链接：https://arxiv.org/abs/2310.10604
作者：Dimitrios Bralios,Gordon Wichern,François G. Germain,Zexu Pan,Sameer Khurana,Chiori Hori,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, University of Illinois Urbana-Champaign, Urbana, IL, USA备注：Submitted to ICASSP 2024摘要：音频潜在扩散模型的引入具有从文本描述按需生成逼真声音剪辑的能力，有可能彻底改变我们使用音频的方式。在这项工作中，我们通过研究音频输出如何与训练数据进行比较来初步尝试了解音频潜在扩散模型的内部工作，类似于医生如何通过倾听器官的声音来听诊患者。使用在AudioCaps数据集上训练的文本到音频潜在扩散模型，我们系统地分析了记忆行为作为训练集大小的函数。我们还评估了不同的检索指标的证据的训练数据记忆，发现梅尔频谱图之间的相似性是更强大的检测匹配比学习嵌入向量。在分析音频潜在扩散模型中的记忆过程中，我们还发现了AudioCaps数据库中大量重复的音频片段。摘要：The introduction of audio latent diffusion models possessing the ability to generate realistic sound clips on demand from a text description has the potential to revolutionize how we work with audio. In this work, we make an initial attempt at understanding the inner workings of audio latent diffusion models by investigating how their audio outputs compare with the training data, similar to how a doctor auscultates a patient by listening to the sounds of their organs. Using text-to-audio latent diffusion models trained on the AudioCaps dataset, we systematically analyze memorization behavior as a function of training set size. We also evaluate different retrieval metrics for evidence of training data memorization, finding the similarity between mel spectrograms to be more robust in detecting matches than learned embedding vectors. In the process of analyzing memorization in audio latent diffusion models, we also discover a large amount of duplicated audio clips within the AudioCaps database.
【2】 Advancing Audio Emotion and Intent Recognition with Large Pre-Trained  Models and Bayesian Inference标题：基于大预训练模型和贝叶斯推理的音频情感和意图识别链接：https://arxiv.org/abs/2310.10179
作者：Dejan Porjazovski,Yaroslav Getman,Tamás Grósz,Mikko Kurimo机构：Aalto University, Espoo, Finland备注：Accepted at ACMM 2023摘要：大型预训练模型在语言系统中至关重要，在情感识别和口吃检测等任务中表现出有效性。在本文中，我们使用大型预训练模型进行ACM多媒体计算副语言挑战，解决请求和情感共享任务。我们探索利用音频和文本模态的纯音频和混合解决方案。我们的实证结果一致显示的优越性的混合方法的音频模型。此外，我们引入贝叶斯层作为标准线性输出层的替代。多模态融合方法在HC请求上实现了85.4%UAR，在HC投诉上实现了60.2%。情感分享任务的集成模型产生的最佳rho值为0.614。在这项研究中探索的贝叶斯wav2vec2方法，使我们能够很容易地建立合奏，在微调只有一个模型的成本。此外，我们可以有可用的置信度值，而不是通常的过度自信的后验概率。摘要：Large pre-trained models are essential in paralinguistic systems, demonstrating effectiveness in tasks like emotion recognition and stuttering detection. In this paper, we employ large pre-trained models for the ACM Multimedia Computational Paralinguistics Challenge, addressing the Requests and Emotion Share tasks. We explore audio-only and hybrid solutions leveraging audio and text modalities. Our empirical results consistently show the superiority of the hybrid approaches over the audio-only models. Moreover, we introduce a Bayesian layer as an alternative to the standard linear output layer. The multimodal fusion approach achieves an 85.4% UAR on HC-Requests and 60.2% on HC-Complaints. The ensemble model for the Emotion Share task yields the best rho value of .614. The Bayesian wav2vec2 approach, explored in this study, allows us to easily build ensembles, at the cost of fine-tuning only one model. Moreover, we can have usable confidence values instead of the usual overconfident posterior probabilities.
【3】 Real-time Speech Enhancement and Separation with a Unified Deep Neural  Network for Single/Dual Talker Scenarios标题：基于统一深度神经网络的单/双人实时语音增强与分离链接：https://arxiv.org/abs/2310.10026
作者：Kashyap Patel,Anton Kovalyov,Issa Panahi机构：Electrical and Computer Engineering, University of Texas at Dallas, Richardson, USA备注：6 Pages, Accepted at IEEE Asilomar摘要：None摘要：This paper introduces a practical approach for leveraging a real-time deep learning model to alternate between speech enhancement and joint speech enhancement and separation depending on whether the input mixture contains one or two active speakers. Scale-invariant signal-to-distortion ratio (SI-SDR) has shown to be a highly effective training measure in time-domain speech separation. However, the SI-SDR metric is ill-defined for zero-energy target signals, which is a problem when training a speech separation model using utterances with varying numbers of talkers. Unlike existing solutions that focus on modifying the loss function to accommodate zero-energy target signals, the proposed approach circumvents this problem by training the model to extract speech on both its output channels regardless if the input is a single or dual-talker mixture. A lightweight speaker overlap detection (SOD) module is also introduced to differentiate between single and dual-talker segments in real-time. The proposed module takes advantage of the new formulation by operating directly on the separated masks, given by the separation model, instead of the original mixture, thus effectively simplifying the detection task. Experimental results show that the proposed training approach outperforms existing solutions, and the SOD module exhibits high accuracy.
【4】 Protecting Voice-Controlled Devices against LASER Injection Attacks标题：保护声控设备免受激光注入攻击链接：https://arxiv.org/abs/2310.09404
作者：Hashim Ali,Dhimant Khuttan,Rafi Ud Daula Refat,Hafiz Malik机构：Department of Electrical EngineeringUniversity of Michigan-Dearborn, Dearborn, MI USA备注：6 pages, 7 figures摘要：语音可控设备（VCD）由于MEMS麦克风的外形尺寸小且易于集成到现代小工具中，因此其采用率呈上升趋势。最近的研究表明，MEMS麦克风容易受到音频调制激光注入攻击。本文的目的是制定对策，以检测和防止激光注入攻击的MEMS麦克风。采用基于离散小波变换（DWT）的时频分解将麦克风输出音频信号分解为n + 1个频率子带以捕获光声相关伪影。由子带音频信号的前四个矩组成的高阶统计特征，例如，方差、偏斜和峰度用于区分声学响应和光声响应。SVM分类器用于学习区分MEMS麦克风中的声学和激光诱导（光声）响应的基础模型。建议的框架进行评估的数据集190个音频，包括19个扬声器。实验结果表明，该框架能够正确分类$98 $的声学和激光诱导的音频在随机数据分区设置和$100 $的音频在扬声器独立和文本独立的数据分区设置。摘要：Voice-Controllable Devices (VCDs) have seen an increasing trend towards their adoption due to the small form factor of the MEMS microphones and their easy integration into modern gadgets. Recent studies have revealed that MEMS microphones are vulnerable to audio-modulated laser injection attacks. This paper aims to develop countermeasures to detect and prevent laser injection attacks on MEMS microphones. A time-frequency decomposition based on discrete wavelet transform (DWT) is employed to decompose microphone output audio signal into n + 1 frequency subbands to capture photo-acoustic related artifacts. Higher-order statistical features consisting of the first four moments of subband audio signals, e.g., variance, skew, and kurtosis are used to distinguish between acoustic and photo-acoustic responses. An SVM classifier is used to learn the underlying model that differentiates between an acoustic- and laser-induced (photo-acoustic) response in the MEMS microphone. The proposed framework is evaluated on a data set of 190 audios, consisting of 19 speakers. The experimental results indicate that the proposed framework is able to correctly classify $98\%$ of the acoustic- and laser-induced audio in a random data partition setting and $100\%$ of the audio in speaker-independent and text-independent data partition settings.
【5】 CORN: Co-Trained Full-Reference And No-Reference Audio Metrics标题：COMM：联合训练的全参考和无参考音频指标链接：https://arxiv.org/abs/2310.09388
作者：Pranay Manocha,Donald Williamson,Adam Finkelstein机构：Department of Computer Science, Princeton University, Princeton, NJ, USA, Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA摘要：感知评价是各种音频处理任务的一个重要方面。完全参考（FR）或基于相似性的度量依赖于高质量的参考记录，可以将记录的较低质量或损坏版本与其进行比较以进行评估。相比之下，无参考（NR）度量在不依赖于参考的情况下评估记录。FR和NR方法都表现出相对于彼此的优点和缺点。在本文中，我们提出了一个名为CORN的新框架，它融合了这些双重方法，同时训练FR和NR模型。训练后，模型可以独立应用。我们通过预测几个常见的客观指标和两种不同的架构来评估CORN。使用CORN训练的NR模型在训练期间可以访问参考记录，因此，正如人们所期望的那样，它始终优于独立训练的基线NR模型。也许更值得注意的是，CORN FR模型也优于其基线对应模型，即使它依赖于相同的训练数据和相同的模型架构。因此，单个训练方案产生两个独立有用的模型，每个模型都优于独立训练的模型。摘要：Perceptual evaluation constitutes a crucial aspect of various audio-processing tasks. Full reference (FR) or similarity-based metrics rely on high-quality reference recordings, to which lower-quality or corrupted versions of the recording may be compared for evaluation. In contrast, no-reference (NR) metrics evaluate a recording without relying on a reference. Both the FR and NR approaches exhibit advantages and drawbacks relative to each other. In this paper, we present a novel framework called CORN that amalgamates these dual approaches, concurrently training both FR and NR models together. After training, the models can be applied independently. We evaluate CORN by predicting several common objective metrics and across two different architectures. The NR model trained using CORN has access to a reference recording during training, and thus, as one would expect, it consistently outperforms baseline NR models trained independently. Perhaps even more remarkable is that the CORN FR model also outperforms its baseline counterpart, even though it relies on the same training data and the same model architecture. Thus, a single training regime produces two independently useful models, each outperforming independently trained models.
【6】 LocSelect: Target Speaker Localization with an Auditory Selective  Hearing Mechanism标题：LocSelect：具有听觉选择性听力机制的目标说话人定位链接：https://arxiv.org/abs/2310.10497
作者：Yu Chen,Xinyuan Qian,Zexu Pan,Kainan Chen,Haizhou Li机构：School of Computer and Communication Engineering, University of Science and Technology Beijing (USTB), China, School of Data Science, The Chinese University of Hong Kong, Shenzhen, China, Institute of Data Science, National University of Singapore, Singapore备注：Submitted to ICASSP 2024摘要：针对现有的抗噪声和抗混响的定位算法主要强调在多说话人环境中分离出每个说话人并为其提供定向输出，而不考虑说话人身份的问题，提出了一种基于选择性听觉机制的目标说话人定位算法。给定目标说话人的参考语音，我们首先产生一个说话人相关的频谱图掩模，以消除干扰说话人的语音。随后，采用长短期记忆（LSTM）网络从过滤后的频谱图中提取目标说话人的位置。实验验证了该方法在不同尺度不变信噪比（SNR）条件下的优越性。具体而言，在SNR = -10 dB时，我们提出的网络LocSelect实现了3.55的平均绝对误差（MAE）和87.40%的准确度（ACC）。摘要：The prevailing noise-resistant and reverberation-resistant localization algorithms primarily emphasize separating and providing directional output for each speaker in multi-speaker scenarios, without association with the identity of speakers.In this paper, we present a target speaker localization algorithm with a selective hearing mechanism. Given a reference speech of the target speaker, we first produce a speaker-dependent spectrogram mask to eliminate interfering speakers' speech. Subsequently, a Long short-term memory (LSTM) network is employed to extract the target speaker's location from the filtered spectrogram. Experiments validate the superiority of our proposed method over the existing algorithms for different scale invariant signal-to-noise ratios (SNR) conditions. Specifically, at SNR = -10 dB, our proposed network LocSelect achieves a mean absolute error (MAE) of 3.55 and an accuracy (ACC) of 87.40%.
【7】 BeatDance: A Beat-Based Model-Agnostic Contrastive Learning Framework  for Music-Dance Retrieval标题：BeatDance：一种基于节拍的模型不可知性音乐舞蹈检索对比学习框架链接：https://arxiv.org/abs/2310.10300
作者：Kaixing Yang,Xukun Zhou,Xulong Tang,Ran Diao,Hongyan Liu,Jun He,Zhaoxin Fan机构：Renmin University of China, The University of Texas at Dallas, Tsinghua University, Psyche AI Inc摘要：舞蹈和音乐是密切相关的表达形式，舞蹈视频和音乐之间的相互检索是教育，艺术和体育等各个领域的基本任务。然而，现有的方法往往遭受不自然的生成效果或未能充分探索音乐和舞蹈之间的相关性。为了克服这些挑战，我们提出了BeatDance，一个新的基于节拍的模型不可知的对比学习框架。BeatDance集成了节拍感知音乐舞蹈信息提取器，跨时间节拍混合器和节拍增强型集线器还原器，通过利用音乐节拍和舞蹈动作之间的对齐来提高舞蹈音乐检索性能。我们还介绍了音乐舞蹈（MD）数据集，这是一个大规模的音乐舞蹈视频对集合，用于训练和测试。MD数据集上的实验结果表明，我们的方法优于现有的基线，达到最先进的性能。代码和数据集将在接受后公开提供。摘要：Dance and music are closely related forms of expression, with mutual retrieval between dance videos and music being a fundamental task in various fields like education, art, and sports. However, existing methods often suffer from unnatural generation effects or fail to fully explore the correlation between music and dance. To overcome these challenges, we propose BeatDance, a novel beat-based model-agnostic contrastive learning framework. BeatDance incorporates a Beat-Aware Music-Dance InfoExtractor, a Trans-Temporal Beat Blender, and a Beat-Enhanced Hubness Reducer to improve dance-music retrieval performance by utilizing the alignment between music beats and dance movements. We also introduce the Music-Dance (MD) dataset, a large-scale collection of over 10,000 music-dance video pairs for training and testing. Experimental results on the MD dataset demonstrate the superiority of our method over existing baselines, achieving state-of-the-art performance. The code and dataset will be made public available upon acceptance.
【8】 Joint Music and Language Attention Models for Zero-shot Music Tagging标题：零拍音乐标注中的音乐和语言联合注意模型链接：https://arxiv.org/abs/2310.10159
作者：Xingjian Du,Zhesong Yu,Jiaju Lin,Bilei Zhu,Qiuqiang Kong机构：Bytedance ,The Chinese University of Hongkong备注：\begin{keywords} Music tagging, joint music and language attention models, Music Foundation Model. \end{keywords}摘要：音乐标注是一项预测音乐唱片标签的任务。然而，以往的音乐标注研究主要集中在封闭的音乐标注任务，不能推广到新的标签。在这项工作中，我们提出了一个zero-shot音乐标记系统建模的联合音乐和语言注意力（JMLA）模型，以解决开放集的音乐标记问题。JMLA模型由一个由预训练的掩码自动编码器建模的音频编码器和一个由Falcon 7 B建模的解码器组成。我们引入preceiver重采样器将任意长度的音频转换为固定长度的嵌入。我们在编码器和解码器层之间引入密集注意力连接，以改善编码器和解码器层之间的信息流。我们从互联网上收集了一个大规模的音乐和描述数据集。我们建议使用ChatGPT将原始描述转换为形式化和多样化的描述，以训练JMLA模型。我们提出的JMLA系统在GTZAN数据集上实现了64.82\% $的zero-shot音频标记精度，优于以前的zero-shot系统，并在FMA和MagnaTagATune数据集上实现了与以前的系统相当的结果。摘要：Music tagging is a task to predict the tags of music recordings. However, previous music tagging research primarily focuses on close-set music tagging tasks which can not be generalized to new tags. In this work, we propose a zero-shot music tagging system modeled by a joint music and language attention (JMLA) model to address the open-set music tagging problem. The JMLA model consists of an audio encoder modeled by a pretrained masked autoencoder and a decoder modeled by a Falcon7B. We introduce preceiver resampler to convert arbitrary length audio into fixed length embeddings. We introduce dense attention connections between encoder and decoder layers to improve the information flow between the encoder and decoder layers. We collect a large-scale music and description dataset from the internet. We propose to use ChatGPT to convert the raw descriptions into formalized and diverse descriptions to train the JMLA models. Our proposed JMLA system achieves a zero-shot audio tagging accuracy of $ 64.82\% $ on the GTZAN dataset, outperforming previous zero-shot systems and achieves comparable results to previous systems on the FMA and the MagnaTagATune datasets.
【9】 End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder  and Input Feature Analysis标题：端到端多声道说话人ASR：说话人引导译码及输入特征分析链接：https://arxiv.org/abs/2310.10106
作者：Can Cui,Imran Ahamad Sheikh,Mostafa Sadeghi,Emmanuel Vincent机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France, Vivoka, Metz, France备注：None摘要：我们提出了一个端到端的多通道说话人属性的自动语音识别（MC-SA-ASR）系统，它结合了一个基于一致性的编码器与多帧跨通道的注意和一个说话人属性的转换器为基础的解码器。据我们所知，这是第一个在多通道环境中有效集成ASR和说话人识别模块的模型。在LibriSpeech数据的模拟混合物上，与先前提出的单通道和多通道方法相比，我们的系统分别将字错误率（WER）降低了12%和16%。此外，我们研究了不同的输入功能，包括多通道的幅度和相位信息，对ASR性能的影响。最后，我们在AMI语料库上的实验证实了我们的系统对真实世界的多通道会议转录的有效性。摘要：We present an end-to-end multichannel speaker-attributed automatic speech recognition (MC-SA-ASR) system that combines a Conformer-based encoder with multi-frame crosschannel attention and a speaker-attributed Transformer-based decoder. To the best of our knowledge, this is the first model that efficiently integrates ASR and speaker identification modules in a multichannel setting. On simulated mixtures of LibriSpeech data, our system reduces the word error rate (WER) by up to 12% and 16% relative compared to previously proposed single-channel and multichannel approaches, respectively. Furthermore, we investigate the impact of different input features, including multichannel magnitude and phase information, on the ASR performance. Finally, our experiments on the AMI corpus confirm the effectiveness of our system for real-world multichannel meeting transcription.
【10】 MERTech: Instrument Playing Technique Detection Using Self-Supervised  Pretrained Model With Multi-Task Finetuning标题：MERTech：基于多任务精调的自监督预训练模型乐器演奏技巧检测链接：https://arxiv.org/abs/2310.09853
作者：Dichucheng Li,Yinghao Ma,Weixing Wei,Qiuqiang Kong,Yulun Wu,Mingjin Che,Fan Xia,Emmanouil Benetos,Wei Li机构：School of Computer Science and Technology, Fudan University, Shanghai, China,  Centre for Digital Music, Queen Mary University of London, UK, Multimodal Art Projection Research Community,  The Chinese University of Hong Kong (CUHK)备注：submitted to ICASSP 2024摘要：乐器演奏技巧是音乐表现力的重要组成部分。然而，自动IPT检测方法的发展受到有限的标记数据和固有的类不平衡问题的影响。在本文中，我们提出应用一个在大规模未标记音乐数据上预训练的自监督学习模型，并在IPT检测任务中对其进行微调。这种方法解决了数据稀缺和类别不平衡的挑战。认识到音高在捕捉IPT的细微差别和定位IPT事件的重要性，我们研究了多任务微调与音高和起始检测作为辅助任务。此外，我们应用后处理方法进行事件级预测，其中IPT激活仅在发作输出确认该帧中的发作时才启动事件。我们的方法在多个IPT基准数据集上的帧级和事件级指标上都优于以前的方法。进一步的实验证明了多任务微调对每个IPT类的有效性。摘要：Instrument playing techniques (IPTs) constitute a pivotal component of musical expression. However, the development of automatic IPT detection methods suffers from limited labeled data and inherent class imbalance issues. In this paper, we propose to apply a self-supervised learning model pre-trained on large-scale unlabeled music data and finetune it on IPT detection tasks. This approach addresses data scarcity and class imbalance challenges. Recognizing the significance of pitch in capturing the nuances of IPTs and the importance of onset in locating IPT events, we investigate multi-task finetuning with pitch and onset detection as auxiliary tasks. Additionally, we apply a post-processing approach for event-level prediction, where an IPT activation initiates an event only if the onset output confirms an onset in that frame. Our method outperforms prior approaches in both frame-level and event-level metrics across multiple IPT benchmark datasets. Further experiments demonstrate the efficacy of multi-task finetuning on each IPT class.
【11】 CoCoFormer: A controllable feature-rich polyphonic music generation  method标题：CoCoFormer：一种可控的特征丰富的复调音乐生成方法链接：https://arxiv.org/abs/2310.09843
作者：Jiuyang Zhou,Tengfei Niu,Hong Zhu,Xingping Wang机构：Xi’an University of Technology, Xi’an, China摘要：探讨了复调音乐序列的建模方法。由于Transformer模型在音乐生成方面的巨大潜力，可控音乐生成正受到更多关注。在复调音乐任务中，目前的可控生成研究主要集中在和弦的可控生成上，而对合唱音乐织体的可控生成缺乏精确的调节。本文提出了条件Choir Transformer（CoCoFormer），它通过在细粒度水平上控制和弦和节奏输入来控制模型的输出。本文提出的自监督方法改进了损失函数，通过条件控制输入和无条件输入训练进行联合训练。为了缓解教师强迫训练导致生成样本缺乏多样性的问题，本文增加了对抗训练方法。CoCoFormer通过对和弦和节奏的显式和隐式输入来增强模型性能。本文通过实验证明，CoCoFormer已经达到了目前比现有机型更好的水平。在规定复调音乐织体的前提下，同样的旋律也可以多种方式生成。摘要：This paper explores the modeling method of polyphonic music sequence. Due to the great potential of Transformer models in music generation, controllable music generation is receiving more attention. In the task of polyphonic music, current controllable generation research focuses on controlling the generation of chords, but lacks precise adjustment for the controllable generation of choral music textures. This paper proposed Condition Choir Transformer (CoCoFormer) which controls the output of the model by controlling the chord and rhythm inputs at a fine-grained level. In this paper, the self-supervised method improves the loss function and performs joint training through conditional control input and unconditional input training. In order to alleviate the lack of diversity on generated samples caused by the teacher forcing training, this paper added an adversarial training method. CoCoFormer enhances model performance with explicit and implicit inputs to chords and rhythms. In this paper, the experiments proves that CoCoFormer has reached the current better level than current models. On the premise of specifying the polyphonic music texture, the same melody can also be generated in a variety of ways.
【12】 Improved Contextual Recognition In Automatic Speech Recognition Systems  By Semantic Lattice Rescoring标题：语义格检索改进自动语音识别系统中的上下文识别链接：https://arxiv.org/abs/2310.09680
作者：Ankitha Sudarshan,Vinay Samuel,Parth Patwa,Ibtihel Amara,Aman Chadha机构：Purdue University ,Carnegie Mellon University ,University of California Los Angeles, McGill University ,Stanford University摘要：自动语音识别（ASR）是语音识别领域的一个研究热点。最近的突破为ASR系统带来了不同的前景，例如忠实地转录口语，这是构建会话代理的关键进步。然而，准确识别上下文相关的单词和短语仍然是一个迫在眉睫的挑战。在这项工作中，我们提出了一种新的方法，用于通过语义网格处理来增强ASR系统中的上下文识别，该方法利用深度学习模型的力量，在各种词汇和说话风格中准确地提供即时的翻译。我们的解决方案包括使用隐马尔可夫模型和高斯混合模型（HMM-GMM）以及深度神经网络（DNN）模型，该模型集成了语言和声学建模，以提高准确性。我们通过使用基于transformer的模型来注入我们的网络，以正确地重新划分单词网格，从而实现显着的功能，并显著降低单词错误率（WER）。我们证明了我们提出的框架的LibriSpeech数据集的实证分析的有效性。摘要：Automatic Speech Recognition (ASR) has witnessed a profound research interest. Recent breakthroughs have given ASR systems different prospects such as faithfully transcribing spoken language, which is a pivotal advancement in building conversational agents. However, there is still an imminent challenge of accurately discerning context-dependent words and phrases. In this work, we propose a novel approach for enhancing contextual recognition within ASR systems via semantic lattice processing leveraging the power of deep learning models in accurately delivering spot-on transcriptions across a wide variety of vocabularies and speaking styles. Our solution consists of using Hidden Markov Models and Gaussian Mixture Models (HMM-GMM) along with Deep Neural Networks (DNN) models integrating both language and acoustic modeling for better accuracy. We infused our network with the use of a transformer-based model to properly rescore the word lattice achieving remarkable capabilities with a palpable reduction in Word Error Rate (WER). We demonstrate the effectiveness of our proposed framework on the LibriSpeech dataset with empirical analyses.
【13】 SelfVC: Voice Conversion With Iterative Refinement using Self  Transformations标题：SelfVC：使用自变换进行迭代精化的语音转换链接：https://arxiv.org/abs/2310.09653
作者：Paarth Neekhara,Shehzeen Hussain,Rafael Valle,Boris Ginsburg,Rishabh Ranjan,Shlomo Dubnov,Farinaz Koushanfar,Julian McAuley机构：Boris Gingsburg, NVIDIA,UC San Diego, Equal Contribution摘要：我们提出了SelfVC，这是一种训练策略，可以用自合成的例子迭代地改进语音转换模型。以前的语音转换工作集中在明确地解开语音表示，分别编码扬声器的特点和语言内容。然而，使用特定于任务的损失项来解开语音表示以捕获这样的属性可能通过丢弃原始信号的更细微差别而导致信息损失。在这项工作中，而不是明确地解开与损失条款的属性，我们提出了一个框架，训练一个可控的语音转换模型纠缠的语音表示来自自监督学习和说话人验证模型。首先，我们开发的技术，以获得韵律信息的音频信号和SSL表示训练预测子模块的合成模型。接下来，我们提出了一种训练策略，通过使用自合成的示例创建具有挑战性的训练目标来迭代地改进语音转换的合成模型。在这种训练方法中，合成模型的当前状态用于生成话语的语音转换的变化，其用作重建任务的输入，确保模型的连续和有目的的细化。我们证明，在训练过程中纳入这样的自我合成的例子，提高了扬声器的相似性生成的语音相比，一个基线的语音转换模型只训练的扰动输入。SelfVC在没有任何文本的情况下进行训练，适用于一系列任务，如zero-shot语音转换，跨语言语音转换以及具有音高和节奏修改的可控语音合成。SelfVC在zero-shot语音转换中实现了最先进的结果，这些结果基于评估合成音频的自然度、扬声器相似性和可懂度的指标。摘要：We propose SelfVC, a training strategy to iteratively improve a voice conversion model with self-synthesized examples. Previous efforts on voice conversion focus on explicitly disentangling speech representations to separately encode speaker characteristics and linguistic content. However, disentangling speech representations to capture such attributes using task-specific loss terms can lead to information loss by discarding finer nuances of the original signal. In this work, instead of explicitly disentangling attributes with loss terms, we present a framework to train a controllable voice conversion model on entangled speech representations derived from self-supervised learning and speaker verification models. First, we develop techniques to derive prosodic information from the audio signal and SSL representations to train predictive submodules in the synthesis model. Next, we propose a training strategy to iteratively improve the synthesis model for voice conversion, by creating a challenging training objective using self-synthesized examples. In this training approach, the current state of the synthesis model is used to generate voice-converted variations of an utterance, which serve as inputs for the reconstruction task, ensuring a continuous and purposeful refinement of the model. We demonstrate that incorporating such self-synthesized examples during training improves the speaker similarity of generated speech as compared to a baseline voice conversion model trained solely on heuristically perturbed inputs. SelfVC is trained without any text and is applicable to a range of tasks such as zero-shot voice conversion, cross-lingual voice conversion, and controllable speech synthesis with pitch and pace modifications. SelfVC achieves state-of-the-art results in zero-shot voice conversion on metrics evaluating naturalness, speaker similarity, and intelligibility of synthesized audio.
【14】 Dynamic Prediction of Full-Ocean Depth SSP by Hierarchical LSTM: An  Experimental Result标题：基于分层LSTM的全海洋深度SSP动态预报：实验结果链接：https://arxiv.org/abs/2310.09522
作者：Jiajun Lu,Wei Huang,Hao Zhang机构：Ocean University of China摘要：SSP分布是水下定位、导航和定时（PNT）的一个重要参数，因为它影响着水声信号的传播模式。为了准确预测未来声速分布，提出了一种用于未来声速预测的分层长短期记忆（H-LSTM）神经网络，探索声速在时间维度上的分布规律。为了验证其可行性和有效性，我们进行了仿真和实际实验。海洋实验于2023年4月在南海举行。结果表明，该方法的准确性优于国家-的-的-最先进的方法。摘要：SSP distribution is an important parameter for underwater positioning, navigation and timing (PNT) because it affects the propagation mode of underwater acoustic signals. To accurate predict future sound speed distribution, we propose a hierarchical long short--term memory (H--LSTM) neural network for future sound speed prediction, which explore the distribution pattern of sound velocity in the time dimension. To verify the feasibility and effectiveness, we conducted both simulations and real experiments. The ocean experiment was held in the South China Sea in April, 2023. Results show that the accuracy of the proposed method outperforms the state--of--the--art methods.
【15】 Advancing Test-Time Adaptation for Acoustic Foundation Models in  Open-World Shifts标题：开放世界移动中声学基础模型的超前测试时间自适应链接：https://arxiv.org/abs/2310.09505
作者：Hongfu Liu,Hengguan Huang,Ye Wang机构：School of Computing, National University of Singapore摘要：测试时自适应（TTA）是解决推理过程中分布变化的一个重要范式，特别是在视觉识别任务中。然而，虽然声学模型面临着类似的挑战，由于分布变化的测试时间语音，TTA技术专门设计的声学建模的背景下，开放世界的数据变化仍然很少。考虑到声学基础模型的独特特性，这一差距进一步加剧：1）它们主要建立在具有层归一化的Transformer架构上，以及2）它们以非平稳方式处理不同长度的测试时间语音数据。这些方面使得视觉聚焦TTA方法的直接应用变得不可行，该方法主要依赖于批量归一化并假设独立样本。在本文中，我们深入研究了面向开放世界数据迁移的预训练声学模型的TTA。我们发现，嘈杂的，高熵的语音帧，往往是非无声的，进行关键的语义内容。传统的TTA方法可能会使用潜在的有缺陷的分析法无意中过滤掉这些信息。作为回应，我们引入了一个无经验的，基于学习的适应性增强信心。考虑到语音信号的短时一致性，我们在测试时间优化中也应用了一致性正则化。我们在合成和真实世界数据集上的实验证实了我们的方法优于现有基线。摘要：Test-Time Adaptation (TTA) is a critical paradigm for tackling distribution shifts during inference, especially in visual recognition tasks. However, while acoustic models face similar challenges due to distribution shifts in test-time speech, TTA techniques specifically designed for acoustic modeling in the context of open-world data shifts remain scarce. This gap is further exacerbated when considering the unique characteristics of acoustic foundation models: 1) they are primarily built on transformer architectures with layer normalization and 2) they deal with test-time speech data of varying lengths in a non-stationary manner. These aspects make the direct application of vision-focused TTA methods, which are mostly reliant on batch normalization and assume independent samples, infeasible. In this paper, we delve into TTA for pre-trained acoustic models facing open-world data shifts. We find that noisy, high-entropy speech frames, often non-silent, carry key semantic content. Traditional TTA methods might inadvertently filter out this information using potentially flawed heuristics. In response, we introduce a heuristic-free, learning-based adaptation enriched by confidence enhancement. Noting that speech signals' short-term consistency, we also apply consistency regularization during test-time optimization. Our experiments on synthetic and real-world datasets affirm our method's superiority over existing baselines.
【16】 SALM: Speech-augmented Language Model with In-context Learning for  Speech Recognition and Translation标题：SALM：支持语境学习的语音识别与翻译语音增强语言模型链接：https://arxiv.org/abs/2310.09424
作者：Zhehuai Chen,He Huang,Andrei Andrusenko,Oleksii Hrinchuk,Krishna C. Puvvada,Jason Li,Subhankar Ghosh,Jagadeesh Balam,Boris Ginsburg机构：NVIDIA, USA备注：submit to ICASSP 2024摘要：我们提出了一种新的语音增强语言模型（SALM）与{\em多任务}和{\em在上下文}学习能力。SALM包括冻结文本LLM、音频编码器、模态适配器模块和LoRA层，以适应语音输入和相关联的任务指令。统一的SALM不仅实现了与自动语音识别（ASR）和语音翻译（AST）的特定任务Conformer基线相当的性能，而且还表现出zero-shot上下文学习能力，通过ASR和AST的关键字提升任务证明了这一点。此外，提出了{\em speech supervised in-context training}来弥合LLM训练和下游语音任务之间的差距，这进一步增强了语音到文本模型的上下文学习能力。建议的模型是通过NeMo工具包开源的。摘要：We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, {\em speech supervised in-context training} is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.
机器翻译，仅供参考