语音/音频处理学术速递[10.23]

今日论文合集：cs.SD语音9篇，eess.AS音频处理13篇。

cs.SD语音

【1】 Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval标题：视听检索课程强化的两阶段三联体失分训练链接：https://arxiv.org/abs/2310.13451

作者：Donghuo Zeng,Kazushi Ikeda机构：KDDI Research, Inc., Saitama, Japan备注：8 pages, 6 figures摘要：跨模态检索模型利用三重损失优化的潜力来学习鲁棒的嵌入空间。然而，现有的方法往往训练这些模型在一个单一的通过，忽略了半硬和硬三元组之间的区别，在优化过程中。不区分半硬和硬三元组的疏忽导致次优模型性能。在本文中，我们介绍了一种新的方法植根于课程学习来解决这个问题。我们提出了一个两阶段的训练范式，指导模型的学习过程从半硬到硬三元组。在第一阶段，模型从低损失的基础开始，用一组半硬三元组进行训练。随后，在第二阶段，我们使用插值技术来增强嵌入。这个过程识别了潜在的硬否定，缓解了由于缺乏硬三元组而导致的高损失函数所带来的问题。然后，我们的方法在增强的嵌入空间中应用硬三元组挖掘，以进一步优化模型。在两个视听数据集上进行的大量实验结果表明，在AVE数据集上的视听跨模态检索（AV-CMR）任务中，与当前最先进的方法MSNSCA相比，平均平均精度（MAP）显著提高了约9.8%，表明了我们所提出的方法的有效性。摘要：The cross-modal retrieval model leverages the potential of triple loss optimization to learn robust embedding spaces. However, existing methods often train these models in a singular pass, overlooking the distinction between semi-hard and hard triples in the optimization process. The oversight of not distinguishing between semi-hard and hard triples leads to suboptimal model performance. In this paper, we introduce a novel approach rooted in curriculum learning to address this problem. We propose a two-stage training paradigm that guides the model's learning process from semi-hard to hard triplets. In the first stage, the model is trained with a set of semi-hard triplets, starting from a low-loss base. Subsequently, in the second stage, we augment the embeddings using an interpolation technique. This process identifies potential hard negatives, alleviating issues arising from high-loss functions due to a scarcity of hard triples. Our approach then applies hard triplet mining in the augmented embedding space to further optimize the model. Extensive experimental results conducted on two audio-visual datasets show a significant improvement of approximately 9.8% in terms of average Mean Average Precision (MAP) over the current state-of-the-art method, MSNSCA, for the Audio-Visual Cross-Modal Retrieval (AV-CMR) task on the AVE dataset, indicating the effectiveness of our proposed method.

【2】 Definition-independent Formalization of Soundscapes: Towards a Formal Methodology标题：独立于定义的音景形式化：走向形式化方法论链接：https://arxiv.org/abs/2310.13404

作者：Mikel D. Jedrusiak,Thomas Harweg,Timo Haselhoff,Bryce T. Lawrence,Susanne Moebus,Frank Weichert机构：Department of Computer Science, Computergraphics, Technical University of Dortmund, Otto-Hahn-Straße , Dortmund, Germany, Institute for Urban Public Health, University Hospital Essen, University Duisburg-Essen, Hufelandstraße , Essen, Germany摘要：声景已经被来自不同学科的研究人员研究过，每个学科都有不同的视角，目标，方法和术语。因此，根据不同的领域，声景的组成部分的概念会发生变化，从而改变基本定义。这导致跨学科交流和结果比较复杂化。特别是当涉及到与音景无关的研究领域时。出于这个原因，我们提出了一个潜在的形式化，是独立的基本声景定义，能够捕捉的异构结构的数据，以及在一个模型中的不同的意识形态的目标。在一个示例性的分析频率相关矩阵的土地利用类型检测作为替代功能，如MFCC，我们展示了我们提出的形式化的实际应用。摘要：Soundscapes have been studied by researchers from various disciplines, each with different perspectives, goals, approaches, and terminologies. Accordingly, depending on the field, the concept of a soundscape's components changes, consequently changing the basic definition. This results in complicating interdisciplinary communication and comparison of results. Especially when soundscape-unrelated research areas are involved. For this reason, we present a potential formalization that is independent of the underlying soundscape definition, with the goal of being able to capture the heterogeneous structure of the data as well as the different ideologies in one model. In an exemplary analysis of frequency correlation matrices for land use type detection as an alternative to features like MFCCs, we show a practical application of our presented formalization.

【3】 Music Augmentation and Denoising For Peak-Based Audio Fingerprinting标题：基于峰值的音频指纹识别中的音乐增强和去噪链接：https://arxiv.org/abs/2310.13388

作者：Kamil Akesbi,Dorian Desblancs,Benjamin Martin机构：Deezer摘要：音频指纹识别是一种成熟的解决方案，用于从简短的录音摘录中识别歌曲。流行的方法依赖于稀疏表示的提取，通常是谱峰，并且已被证明是准确的，快速的，可扩展到大型集合。然而，音频识别的实际应用经常发生在嘈杂的环境中，这可能导致这些系统失败。在这项工作中，我们通过引入和发布一个新的音频增强管道来解决这个问题，该管道通过随机模仿现实世界的场景，以逼真的方式向音乐片段添加噪声。然后，我们提出并发布了一个深度学习模型，该模型可以从频谱图中去除噪声成分，以提高基于峰值的指纹识别系统的准确性。我们表明，除了我们的模型提高了常用的音频指纹识别系统的识别性能，即使在嘈杂的条件下。摘要：Audio fingerprinting is a well-established solution for song identification from short recording excerpts. Popular methods rely on the extraction of sparse representations, generally spectral peaks, and have proven to be accurate, fast, and scalable to large collections. However, real-world applications of audio identification often happen in noisy environments, which can cause these systems to fail. In this work, we tackle this problem by introducing and releasing a new audio augmentation pipeline that adds noise to music snippets in a realistic way, by stochastically mimicking real-world scenarios. We then propose and release a deep learning model that removes noisy components from spectrograms in order to improve peak-based fingerprinting systems' accuracy. We show that the addition of our model improves the identification performance of commonly used audio fingerprinting systems, even under noisy conditions.

【4】 SALMONN: Towards Generic Hearing Abilities for Large Language Models标题：SALMONN：面向大型语言模型的通用听力链接：https://arxiv.org/abs/2310.13289

作者：Changli Tang,Wenyi Yu,Guangzhi Sun,Xianzhao Chen,Tian Tan,Wei Li,Lu Lu,Zejun Ma,Chao Zhang机构：Department of Electronic Engineering, Tsinghua University, ByteDance摘要：听觉可以说是物理世界中人工智能（AI）代理的基本能力，它是指对至少由三种类型的声音组成的一般听觉信息的感知和理解：语音、音频事件和音乐。在本文中，我们提出了SALMONN，一个语音音频语言音乐开放神经网络，通过将预训练的基于文本的大型语言模型（LLM）与语音和音频编码器集成到一个单一的多模态模型中。SALMONN使LLM能够直接处理和理解一般的音频输入，并在训练中使用的许多语音和音频任务上实现有竞争力的性能，例如自动语音识别和翻译，基于语音信息的问题回答，情感识别，说话人验证以及音乐和音频字幕等。SALMONN还具有多种在训练中看不到的新兴能力，包括但不限于语音翻译到未经训练的语言，基于语音的插槽填充，基于口语查询的问题回答，基于音频的故事讲述和语音音频协同推理等。研究了跨通道涌现能力的存在性，提出了一种新的Few-Shot激活调谐方法来激活SALMONN的跨通道涌现能力。据我们所知，SALMONN是其类型的第一个模型，可以被视为向具有通用听力能力的AI迈出的一步。SALMONN的交互式演示可在\texttt{\url{https：//github.com/bytedance/SALMONN}}获得，培训代码和模型检查点将在接受后发布。摘要：Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory information consisting of at least three types of sounds: speech, audio events, and music. In this paper, we propose SALMONN, a speech audio language music open neural network, built by integrating a pre-trained text-based large language model (LLM) with speech and audio encoders into a single multimodal model. SALMONN enables the LLM to directly process and understand general audio inputs and achieve competitive performances on a number of speech and audio tasks used in training, such as automatic speech recognition and translation, auditory-information-based question answering, emotion recognition, speaker verification, and music and audio captioning \textit{etc.} SALMONN also has a diverse set of emergent abilities unseen in the training, which includes but is not limited to speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning \textit{etc}. The presence of the cross-modal emergent abilities is studied, and a novel few-shot activation tuning approach is proposed to activate such abilities of SALMONN. To our knowledge, SALMONN is the first model of its type and can be regarded as a step towards AI with generic hearing abilities. An interactive demo of SALMONN is available at \texttt{\url{https://github.com/bytedance/SALMONN}}, and the training code and model checkpoints will be released upon acceptance.

【5】 On the Language Encoder of Contrastive Cross-modal Models标题：对比跨通道模型的语言编码器研究链接：https://arxiv.org/abs/2310.13267

作者：Mengjie Zhao,Junya Ono,Zhi Zhong,Chieh-Hsin Lai,Yuhta Takida,Naoki Murata,Wei-Hsiang Liao,Takashi Shibuya,Hiromi Wakaki,Yuki Mitsufuji机构：†Sony Group Corporation, ‡Sony AI摘要：CLIP和CLAP等对比跨模态模型有助于各种视觉语言（VL）和听觉语言（AL）任务。然而，他们的语言编码器，这是将图像/音频的自然语言描述编码为矢量表示的核心组件，其研究和改进有限。我们广泛评估了无监督和有监督的句子嵌入训练如何影响语言编码器质量和跨模态任务性能。在VL预训练中，我们发现句子嵌入训练语言编码器质量，并有助于跨模态任务，改善对比VL模型，如CyCLIP。相比之下，AL预训练从句子嵌入训练中获益较少，这可能是由于预训练数据量有限。我们分析了表示空间，以了解句子嵌入训练的优势，并发现它提高了文本空间的均匀性，但代价是降低了跨模态对齐。摘要：Contrastive cross-modal models such as CLIP and CLAP aid various vision-language (VL) and audio-language (AL) tasks. However, there has been limited investigation of and improvement in their language encoder, which is the central component of encoding natural language descriptions of image/audio into vector representations. We extensively evaluate how unsupervised and supervised sentence embedding training affect language encoder quality and cross-modal task performance. In VL pretraining, we found that sentence embedding training language encoder quality and aids in cross-modal tasks, improving contrastive VL models such as CyCLIP. In contrast, AL pretraining benefits less from sentence embedding training, which may result from the limited amount of pretraining data. We analyze the representation spaces to understand the strengths of sentence embedding training, and find that it improves text-space uniformity, at the cost of decreased cross-modal alignment.

【6】 AVTENet: Audio-Visual Transformer-based Ensemble Network Exploiting Multiple Experts for Video Deepfake Detection标题：AVTENet：基于视听变换的集成网络，利用多专家进行视频深伪检测链接：https://arxiv.org/abs/2310.13103

作者：Ammarah Hashmi,Sahibzada Adil Shahzad,Chia-Wen Lin,Yu Tsao,Hsin-Min Wang机构：and also with the Department of ComputerScience, National Chengchi University摘要：在社交媒体平台上广泛分享的伪造内容是一个重大的社会问题，需要加强监管，并对研究界提出了新的挑战。最近超现实的deepfake视频的激增引起了人们对音频和视觉伪造威胁的关注。以前大多数关于检测人工智能生成的假视频的工作都只使用视觉模态或音频模态。虽然文献中有一些利用音频和视觉模态来检测伪造视频的方法，但它们尚未在涉及声学和视觉操作的deepfake视频的多模态数据集上进行全面评估。此外，这些现有方法大多基于CNN，并且遭受低检测精度。受Transformer最近在各个领域取得的成功的启发，为了应对deepfake技术带来的挑战，在本文中，我们提出了一种基于视听转换器的包围网（AVTENet）框架，该框架考虑了声学操纵和视觉操纵，以实现有效的视频伪造检测。具体而言，该模型集成了几个纯粹基于transformer的变体，这些变体捕获视频、音频和视听显著线索，以在预测中达成共识。为了进行评估，我们使用了最近发布的基准多模态音频-视频FakeAVCeleb数据集。为了进行详细分析，我们在FakeAVCeleb数据集的多个测试集上评估了AVTENet及其变体和几种现有方法。实验结果表明，我们的最佳模型优于所有现有的方法，并在FakeAVCeleb数据集的Testset-I和Testset-II上实现了最先进的性能。摘要：Forged content shared widely on social media platforms is a major social problem that requires increased regulation and poses new challenges to the research community. The recent proliferation of hyper-realistic deepfake videos has drawn attention to the threat of audio and visual forgeries. Most previous work on detecting AI-generated fake videos only utilizes visual modality or audio modality. While there are some methods in the literature that exploit audio and visual modalities to detect forged videos, they have not been comprehensively evaluated on multi-modal datasets of deepfake videos involving acoustic and visual manipulations. Moreover, these existing methods are mostly based on CNN and suffer from low detection accuracy. Inspired by the recent success of Transformer in various fields, to address the challenges posed by deepfake technology, in this paper, we propose an Audio-Visual Transformer-based Ensemble Network (AVTENet) framework that considers both acoustic manipulation and visual manipulation to achieve effective video forgery detection. Specifically, the proposed model integrates several purely transformer-based variants that capture video, audio, and audio-visual salient cues to reach a consensus in prediction. For evaluation, we use the recently released benchmark multi-modal audio-video FakeAVCeleb dataset. For a detailed analysis, we evaluate AVTENet, its variants, and several existing methods on multiple test sets of the FakeAVCeleb dataset. Experimental results show that our best model outperforms all existing methods and achieves state-of-the-art performance on Testset-I and Testset-II of the FakeAVCeleb dataset.

【7】 Powerset multi-class cross entropy loss for neural speaker diarization标题：神经说话人二值化的功率集多类交叉熵损失链接：https://arxiv.org/abs/2310.13025

作者：Alexis Plaquet,Hervé Bredin机构：IRIT, Universit´e de Toulouse, CNRS, Toulouse INP, UT, Toulouse, France备注：None摘要：自2019年推出以来，整个端到端神经日志化（EEND）工作线一直在将说话者日志化作为一种具有置换不变训练的逐帧多标签分类问题来解决。尽管EEND显示出很大的希望，最近的一些作品退了一步，研究了（局部）监督EEND日记化与（全局）无监督聚类的可能组合。然而，这些混合贡献并没有质疑最初的多标记制剂。我们建议从多标签（其中任何两个扬声器可以同时处于活动状态）切换到powerset多类分类（其中专用类被分配给重叠的扬声器对）。通过对9个不同的基准测试进行广泛的实验，我们表明，这种配方导致显着更好的性能（主要是重叠语音）和鲁棒性域失配，同时消除检测阈值超参数，多标签配方的关键。摘要：Since its introduction in 2019, the whole end-to-end neural diarization (EEND) line of work has been addressing speaker diarization as a frame-wise multi-label classification problem with permutation-invariant training. Despite EEND showing great promise, a few recent works took a step back and studied the possible combination of (local) supervised EEND diarization with (global) unsupervised clustering. Yet, these hybrid contributions did not question the original multi-label formulation. We propose to switch from multi-label (where any two speakers can be active at the same time) to powerset multi-class classification (where dedicated classes are assigned to pairs of overlapping speakers). Through extensive experiments on 9 different benchmarks, we show that this formulation leads to significantly better performance (mostly on overlapping speech) and robustness to domain mismatch, while eliminating the detection threshold hyperparameter, critical for the multi-label formulation.

【8】 Generative error correction for code-switching speech recognition using large language models标题：使用大语言模型的代码转换语音识别的生成性纠错链接：https://arxiv.org/abs/2310.13013

作者：Chen Chen,Yuchen Hu,Chao-Han Huck Yang,Hexin Liu,Sabato Marco Siniscalchi,Eng Siong Chng机构：Nanyang Technological University, Singapore; ,Georgia Institute of Technology, USA, Norwegian University of Science and Technology, Norway备注：Submitted to ICASSP2024摘要：语码转换是指两种或两种以上语言在同一句子中混合使用的现象。尽管近年来自动语音识别（ASR）取得了很大的进展，但由于CS-ASR现象语法结构的复杂性和特定训练语料的缺乏，CS-ASR仍然是一项具有挑战性的任务。在这项工作中，我们建议利用大型语言模型（LLM）和ASR生成的假设列表来解决CS问题。具体来说，我们首先采用多个训练有素的ASR模型来生成N个最佳假设，目的是增加假设集中的多样性和信息量。接下来，我们利用LLM通过添加可训练的低秩适配器来学习假设到转录（H2 T）映射。这种生成式纠错（GER）方法根据其专家语言学知识和N-best假设直接预测准确的转录，从而从传统的语言模型重新评分或纠错技术实现了范式转变。实验证据表明，GER显着提高CS-ASR的准确性，在减少混合错误率（MER）。此外，LLM在H2 T学习中表现出显着的数据效率，为CS-ASR在低资源语言中的数据稀缺问题提供了潜在的解决方案。摘要：Code-switching (CS) speech refers to the phenomenon of mixing two or more languages within the same sentence. Despite the recent advances in automatic speech recognition (ASR), CS-ASR is still a challenging task ought to the grammatical structure complexity of the phenomenon and the data scarcity of specific training corpus. In this work, we propose to leverage large language models (LLMs) and lists of hypotheses generated by an ASR to address the CS problem. Specifically, we first employ multiple well-trained ASR models for N-best hypotheses generation, with the aim of increasing the diverse and informative elements in the set of hypotheses. Next, we utilize the LLMs to learn the hypotheses-to-transcription (H2T) mapping by adding a trainable low-rank adapter. Such a generative error correction (GER) method directly predicts the accurate transcription according to its expert linguistic knowledge and N-best hypotheses, resulting in a paradigm shift from the traditional language model rescoring or error correction techniques. Experimental evidence demonstrates that GER significantly enhances CS-ASR accuracy, in terms of reduced mixed error rate (MER). Furthermore, LLMs show remarkable data efficiency for H2T learning, providing a potential solution to the data scarcity problem of CS-ASR in low-resource languages.

【9】 Neural domain alignment for spoken language recognition based on optimal transport标题：基于最优传输的口语识别神经域对齐链接：https://arxiv.org/abs/2310.13471

作者：Xugang Lu,Peng Shen,Yu Tsao,Hisashi Kawai机构：National Institute of Information and Communications Technology摘要：领域移位降低了跨领域口语识别的有效性，是跨领域口语识别的一个重要挑战。无监督域自适应（UDA）算法已被探索，以解决在SLR域的变化，而不依赖于目标域中的类标签。一种成功的UDA方法专注于学习域不变表示，以对齐域之间的特征分布。然而，在域不变表示的学习过程中忽略类结构可能会导致过度对齐，对分类任务产生负面影响。为了克服这一限制，我们提出了一个基于最优传输（OT）的UDA算法的跨域SLR，利用分布几何结构感知属性的OT。在基于OT的UDA中，在域对齐期间考虑特征和标签信息的联合分布上的基于OT的差异度量。我们之前的研究发现，完全对齐源域和目标域之间的分布可能会引入负迁移，在分布对齐过程中，来自源域的类或不相关的类映射到目标域中的不同类。这种负传递降低了自适应模型的性能。为了缓解这个问题，我们在我们的UDA框架内引入耦合加权部分最优传输（POT）SLR，其中基于传输成本的OT耦合的软权重在域对齐期间自适应设置。一个跨域的SLR任务中使用的实验，以评估建议的UDA。结果表明，我们提出的UDA算法显着提高了跨通道SLR任务中现有的UDA算法的性能。摘要：Domain shift poses a significant challenge in cross-domain spoken language recognition (SLR) by reducing its effectiveness. Unsupervised domain adaptation (UDA) algorithms have been explored to address domain shifts in SLR without relying on class labels in the target domain. One successful UDA approach focuses on learning domain-invariant representations to align feature distributions between domains. However, disregarding the class structure during the learning process of domain-invariant representations can result in over-alignment, negatively impacting the classification task. To overcome this limitation, we propose an optimal transport (OT)-based UDA algorithm for a cross-domain SLR, leveraging the distribution geometry structure-aware property of OT. An OT-based discrepancy measure on a joint distribution over feature and label information is considered during domain alignment in OT-based UDA. Our previous study discovered that completely aligning the distributions between the source and target domains can introduce a negative transfer, where classes or irrelevant classes from the source domain map to a different class in the target domain during distribution alignment. This negative transfer degrades the performance of the adaptive model. To mitigate this issue, we introduce coupling-weighted partial optimal transport (POT) within our UDA framework for SLR, where soft weighting on the OT coupling based on transport cost is adaptively set during domain alignment. A cross-domain SLR task was used in the experiments to evaluate the proposed UDA. The results demonstrated that our proposed UDA algorithm significantly improved the performance over existing UDA algorithms in a cross-channel SLR task.

eess.AS音频处理

【1】 Neural domain alignment for spoken language recognition based on optimal transport标题：基于最优传输的口语识别神经域对齐链接：https://arxiv.org/abs/2310.13471

【2】 HRTF Interpolation using a Spherical Neural Process Meta-Learner标题：基于球面神经过程元学习器的HRTF插补链接：https://arxiv.org/abs/2310.13430

作者：Etienne Thuillier,Craig Jin,Vesa Välimäki机构：Jin is with the School of Electrical and Computer Engineering备注：12 pages. 11 figures. Submitted for publication in IEEE/ACM Transactions on Audio, Speech and Language Processing (T-ASL)摘要：最近已经提出了几种个性化的方法来估计一个主题的头部相关的传递函数（HRTF），使用方便的输入方式，如人体测量或pinnacle照片。需要使用来自对象的HRTF的几个数据点样本自适应地校正由这样的方法造成的估计误差，所述数据点样本使用声学测量或感知反馈获取。为此，我们引入了一个卷积条件神经过程元学习器，专门用于HRTF误差插值。特别地，该模型包括球面卷积神经网络组件以适应HRTF数据的球面几何形状。它还利用了HRTF的左声道和右声道之间关于中轴的潜在对称性。在这项工作中，我们评估所提出的模型的性能纯粹的时间对齐的频谱插值的理由下，一个简化的设置，一个通用的人口平均HRTF形成的初始估计之前，而不是个性化的校正。与最先进的插值方法相比，训练模型实现了高达3 dB的相对误差降低，尽管只使用85个受试者进行训练。这一改进转化为实现相当精度所需的数据点计数的近一半，特别是从50到28个点，以达到每个内插特征的平均-20 dB相对误差。此外，我们表明，训练模型提供了校准良好的不确定性估计。因此，这样的估计可以通知顺序决策问题，即获取所需的尽可能少的校正HRTF数据点以满足HRTF个性化准确度的期望水平。摘要：Several individualization methods have recently been proposed to estimate a subject's Head-Related Transfer Function (HRTF) using convenient input modalities such as anthropometric measurements or pinnae photographs. There exists a need for adaptively correcting the estimation error committed by such methods using a few data point samples from the subject's HRTF, acquired using acoustic measurements or perceptual feedback. To this end, we introduce a Convolutional Conditional Neural Process meta-learner specialized in HRTF error interpolation. In particular, the model includes a Spherical Convolutional Neural Network component to accommodate the spherical geometry of HRTF data. It also exploits potential symmetries between the HRTF's left and right channels about the median axis. In this work, we evaluate the proposed model's performance purely on time-aligned spectrum interpolation grounds under a simplified setup where a generic population-mean HRTF forms the initial estimates prior to corrections instead of individualized ones. The trained model achieves up to 3 dB relative error reduction compared to state-of-the-art interpolation methods despite being trained using only 85 subjects. This improvement translates up to nearly a halving of the data point count required to achieve comparable accuracy, in particular from 50 to 28 points to reach an average of -20 dB relative error per interpolated feature. Moreover, we show that the trained model provides well-calibrated uncertainty estimates. Accordingly, such estimates can inform the sequential decision problem of acquiring as few correcting HRTF data points as needed to meet a desired level of HRTF individualization accuracy.

【3】 GenDistiller: Distilling Pre-trained Language Models based on Generative Models标题：GenDistiller：基于产生式模型的预训练语言模型提取链接：https://arxiv.org/abs/2310.13418

作者：Yingying Gao,Shilei Zhang,Zihao Cui,Yanhan Xu,Chao Deng,Junlan Feng机构：China Mobile Research摘要：HuBERT和WavLM等自监督预训练模型利用未标记的语音数据进行表示学习，并为许多下游任务提供显着改进。尽管这些方法取得了成功，但它们的大存储器和强计算需求阻碍了它们在资源受限设备上的应用。因此，本文介绍了GenDistiller，一个新的知识蒸馏框架提取隐藏表示的教师网络的基础上生成语言模型。生成结构使所提出的模型能够自回归地生成目标教师隐藏层，考虑隐藏层之间的相互作用，而不引入额外的输入。一个二维的注意力机制，以确保隐藏层的因果关系，同时保持在时间维的双向注意力。实验表明，生成式提取器的优势，基线系统，直接预测教师网络的隐藏层没有一个generatvie模型。摘要：Self-supervised pre-trained models such as HuBERT and WavLM leverage unlabeled speech data for representation learning and offer significantly improve for numerous downstream tasks. Despite the success of these methods, their large memory and strong computational requirements hinder their application on resource restricted devices. Therefore, this paper introduces GenDistiller, a novel knowledge distillation framework to distill hidden representations from teacher network based on generative language model. The generative structure enables the proposed model to generate the target teacher hidden layers autoregressively, considering the interactions between hidden layers without instroducing additional inputs. A two-dimensional attention mechanism is implemented to ensure the causality of hidden layers, while preserving bidirectional attention in the time dimension. Experiments reveal the advantage of the generative distiller over the baseline system that predicts the hidden layers of teacher network directly without a generatvie model.

【4】 Detecting Speech Abnormalities with a Perceiver-based Sequence Classifier that Leverages a Universal Speech Model标题：利用利用通用语音模型的基于感知器的序列分类器检测语音异常链接：https://arxiv.org/abs/2310.13010

作者：Hagen Soltau,Izhak Shafran,Alex Ottenwess,Joseph R. JR Duffy,Rene L. Utianski,Leland R. Barnard,John L. Stricker,Daniela Wiepert,David T. Jones,Hugo Botha机构：Google DeepMind, Google备注：None摘要：我们提出了一种基于感知器的序列分类器来检测反映几种神经系统疾病的语音异常。我们将此分类器与通用语音模型（USM）相结合，该模型在1200万小时的不同音频记录上进行了训练（无监督）。我们的模型压缩长序列到一个小的类特定的潜在表示和因式分解投影用于预测不同属性的无序输入语音。我们的方法的好处是，它允许我们为不同的类建模输入的不同区域，同时数据效率高。我们在Mayo Clinic的精选语料库上广泛评估了所提出的模型。我们的模型优于标准的Transformer（80.9%）和感知器（81.8%）模型，并达到83.1%的平均准确率。在有限的特定任务数据的情况下，我们发现预训练很重要，令人惊讶的是，使用不相关的自动语音识别（ASR）任务进行预训练也是有益的。与仅使用最终层编码相比，中间层编码提供了声学和语音信息的混合，并实现了最佳预测结果（83.1% vs. 79.6%）。结果是有希望的，进一步的改进可能有助于临床医生检测语音异常，而不需要访问高度专业的语音语言病理学家。摘要：We propose a Perceiver-based sequence classifier to detect abnormalities in speech reflective of several neurological disorders. We combine this classifier with a Universal Speech Model (USM) that is trained (unsupervised) on 12 million hours of diverse audio recordings. Our model compresses long sequences into a small set of class-specific latent representations and a factorized projection is used to predict different attributes of the disordered input speech. The benefit of our approach is that it allows us to model different regions of the input for different classes and is at the same time data efficient. We evaluated the proposed model extensively on a curated corpus from the Mayo Clinic. Our model outperforms standard transformer (80.9%) and perceiver (81.8%) models and achieves an average accuracy of 83.1%. With limited task-specific data, we find that pretraining is important and surprisingly pretraining with the unrelated automatic speech recognition (ASR) task is also beneficial. Encodings from the middle layers provide a mix of both acoustic and phonetic information and achieve best prediction results compared to just using the final layer encodings (83.1% vs. 79.6%). The results are promising and with further refinements may help clinicians detect speech abnormalities without needing access to highly specialized speech-language pathologists.

【5】 Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval标题：视听检索课程强化的两阶段三联体失分训练链接：https://arxiv.org/abs/2310.13451

【6】 Definition-independent Formalization of Soundscapes: Towards a Formal Methodology标题：独立于定义的音景形式化：走向形式化方法论链接：https://arxiv.org/abs/2310.13404

【7】 Music Augmentation and Denoising For Peak-Based Audio Fingerprinting标题：基于峰值的音频指纹识别中的音乐增强和去噪链接：https://arxiv.org/abs/2310.13388

【8】 SALMONN: Towards Generic Hearing Abilities for Large Language Models标题：SALMONN：面向大型语言模型的通用听力链接：https://arxiv.org/abs/2310.13289

作者：Changli Tang,Wenyi Yu,Guangzhi Sun,Xianzhao Chen,Tian Tan,Wei Li,Lu Lu,Zejun Ma,Chao Zhang机构：Department of Electronic Engineering, Tsinghua University, ByteDance摘要：None摘要：Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory information consisting of at least three types of sounds: speech, audio events, and music. In this paper, we propose SALMONN, a speech audio language music open neural network, built by integrating a pre-trained text-based large language model (LLM) with speech and audio encoders into a single multimodal model. SALMONN enables the LLM to directly process and understand general audio inputs and achieve competitive performances on a number of speech and audio tasks used in training, such as automatic speech recognition and translation, auditory-information-based question answering, emotion recognition, speaker verification, and music and audio captioning \textit{etc.} SALMONN also has a diverse set of emergent abilities unseen in the training, which includes but is not limited to speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning \textit{etc}. The presence of the cross-modal emergent abilities is studied, and a novel few-shot activation tuning approach is proposed to activate such abilities of SALMONN. To our knowledge, SALMONN is the first model of its type and can be regarded as a step towards AI with generic hearing abilities. An interactive demo of SALMONN is available at \texttt{\url{https://github.com/bytedance/SALMONN}}, and the training code and model checkpoints will be released upon acceptance.

【9】 On the Language Encoder of Contrastive Cross-modal Models标题：对比跨通道模型的语言编码器研究链接：https://arxiv.org/abs/2310.13267

【10】 AVTENet: Audio-Visual Transformer-based Ensemble Network Exploiting Multiple Experts for Video Deepfake Detection标题：AVTENet：基于视听变换的集成网络，利用多专家进行视频深伪检测链接：https://arxiv.org/abs/2310.13103

【11】 Powerset multi-class cross entropy loss for neural speaker diarization标题：神经说话人二值化的功率集多类交叉熵损失链接：https://arxiv.org/abs/2310.13025

【12】 Audio-AdapterFusion: A Task-ID-free Approach for Efficient and Non-Destructive Multi-task Speech Recognition标题：音频适配器融合：一种无任务ID的高效无损多任务语音识别方法链接：https://arxiv.org/abs/2310.13015

作者：Hillary Ngai,Rohan Agrawal,Neeraj Gaur,Ronny Huang,Parisa Haghani,Pedro Moreno Mengibar机构：Google LLC备注：2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Proceedings摘要：适配器是对预训练模型进行全面微调的一种高效、可组合的替代方案，有助于将大型ASR模型的部署扩展到许多任务。实际上，在推理过程中，任务ID通常被预先添加到输入中，以路由到指定任务的单任务适配器。然而，这种方法的一个主要限制是在推断期间可能不知道任务ID，从而使其不适合大多数多任务设置。为了解决这个问题，我们提出了三种新的无任务ID的方法来结合多任务ASR中的单任务适配器，并研究了两种用于训练的学习算法。我们从4个不同的ASR任务的10个测试集上评估我们的方法，并表明我们的方法是非破坏性的和参数有效的。虽然只更新17%的模型参数，我们的方法可以实现8%的平均WER改善相对于全面微调，并与任务ID适配器路由。摘要：Adapters are an efficient, composable alternative to full fine-tuning of pre-trained models and help scale the deployment of large ASR models to many tasks. In practice, a task ID is commonly prepended to the input during inference to route to single-task adapters for the specified task. However, one major limitation of this approach is that the task ID may not be known during inference, rendering it unsuitable for most multi-task settings. To address this, we propose three novel task-ID-free methods to combine single-task adapters in multi-task ASR and investigate two learning algorithms for training. We evaluate our methods on 10 test sets from 4 diverse ASR tasks and show that our methods are non-destructive and parameter-efficient. While only updating 17% of the model parameters, our methods can achieve an 8% mean WER improvement relative to full fine-tuning and are on-par with task-ID adapter routing.

【13】 Generative error correction for code-switching speech recognition using large language models标题：使用大语言模型的代码转换语音识别的生成性纠错链接：https://arxiv.org/abs/2310.13013

机器翻译，仅供参考

语音/音频处理学术速递[10.23]

评论 0