今日论文合集：cs.SD语音11篇，eess.AS音频处理13篇。
cs.SD语音
【1】 Towards an AI to Win Ghana's National Science and Maths Quiz标题：走向人工智能赢得加纳国家科学和数学测验链接：https://arxiv.org/abs/2308.04333
作者：George Boateng,Jonathan Abrefah Mensah,Kevin Takyi Yeboah,William Edor,Andrew Kojo Mensah-Onumah,Naafi Dasana Ibrahim,Nana Sam Yeboah备注：7 pages. Under review at Deep Learning Indaba and Black in AI Workshop @NeurIPS 2023摘要：人工智能能赢得加纳的国家科学和数学测验吗？这就是我们在MSMQ AI项目中寻求答案的问题，这是一个开源项目，它正在构建AI以在MSMQ中竞争并获胜。MSMQ是加纳高中生的年度现场科学和数学竞赛，其中3队2名学生通过回答生物，化学，物理和数学问题进行5轮5个渐进阶段的竞争，直到获胜的团队获得该年度的冠军。MSMQ是一个令人兴奋的现场问答比赛，在语音到文本、文本到语音、问答和人机交互方面都有有趣的技术挑战。在2023年1月开始的这项正在进行的工作中，我们概述了该项目，描述了每个团队，到目前为止取得的进展，以及我们计划在10月份为MSMQ 2023推出和首次推出AI的下一步。克服这一巨大挑战的人工智能可以对教育产生现实世界的影响，例如使非洲数百万学生能够从这种人工智能中获得一对一的学习支持。摘要：Can an AI win Ghana's National Science and Maths Quiz (NSMQ)? That is the question we seek to answer in the NSMQ AI project, an open-source project that is building AI to compete live in the NSMQ and win. The NSMQ is an annual live science and mathematics competition for senior secondary school students in Ghana in which 3 teams of 2 students compete by answering questions across biology, chemistry, physics, and math in 5 rounds over 5 progressive stages until a winning team is crowned for that year. The NSMQ is an exciting live quiz competition with interesting technical challenges across speech-to-text, text-to-speech, question-answering, and human-computer interaction. In this ongoing work that began in January 2023, we give an overview of the project, describe each of the teams, progress made thus far, and the next steps toward our planned launch and debut of the AI in October for NSMQ 2023. An AI that conquers this grand challenge can have real-world impact on education such as enabling millions of students across Africa to have one-on-one learning support from this AI.
【2】 Auditory Attention Decoding with Task-Related Multi-View Contrastive  Learning标题：基于任务相关多视点对比学习的听觉注意解码链接：https://arxiv.org/abs/2308.04244
作者：Xiaoyu Chen,Changde Du,Qiongyi Zhou,Huiguang He摘要：在鸡尾酒会等场合，人类的大脑可以很容易地专注于一个演讲者，而压制其他人。最近，研究人员发现听觉注意可以从脑电图（EEG）数据解码。然而，大多数现有的深度学习方法难以利用不同视图的先验知识（即参与语音和EEG都是任务相关的视图），提取出令人不满意的表示。Broadbent的过滤器模型的启发下，我们解码听觉注意在多视图的范例，并提取最相关的和重要的信息，利用丢失的视图。具体来说，我们提出了一个听觉注意解码（AAD）方法的基础上，多视图VAE与任务相关的多视图对比（TMC）学习。在多视图VAE中采用TMC学习可以利用缺失的视图将不同视图的先验知识累积到表示的融合中，并提取近似的任务相关表示。我们检查我们的方法在两个流行的AAD数据集，并证明我们的方法的优越性，通过比较它的国家的最先进的方法。摘要：The human brain can easily focus on one speaker and suppress others in scenarios such as a cocktail party. Recently, researchers found that auditory attention can be decoded from the electroencephalogram (EEG) data. However, most existing deep learning methods are difficult to use prior knowledge of different views (that is attended speech and EEG are task-related views) and extract an unsatisfactory representation. Inspired by Broadbent's filter model, we decode auditory attention in a multi-view paradigm and extract the most relevant and important information utilizing the missing view. Specifically, we propose an auditory attention decoding (AAD) method based on multi-view VAE with task-related multi-view contrastive (TMC) learning. Employing TMC learning in multi-view VAE can utilize the missing view to accumulate prior knowledge of different views into the fusion of representation, and extract the approximate task-related representation. We examine our method on two popular AAD datasets, and demonstrate the superiority of our method by comparing it to the state-of-the-art method.
【3】 Evil Operation: Breaking Speaker Recognition with PaddingBack标题：邪恶操作：用PaddingBack破解说话人识别链接：https://arxiv.org/abs/2308.04179
作者：Zhe Ye,Diqun Yan,Li Dong,Kailai Shen摘要：机器学习即服务（MLaaS）由于机器学习的进步而获得了普及。然而，不受信任的第三方平台引发了人们对AI安全性的担忧，特别是在后门攻击方面。最近的研究表明，语音后门可以利用转换作为触发器，类似于图像后门。然而，人类的耳朵很容易察觉到这些变化，导致怀疑。在本文中，我们介绍了PaddingBack，一种听不见的后门攻击，利用恶意操作，使中毒样本无法区分干净的。而不是使用外部扰动作为触发器，我们利用广泛使用的语音信号操作，填充，打破说话人识别系统。我们的实验结果表明，所提出的方法的有效性，实现了显着高的攻击成功率，同时保持了较高的良性准确率。此外，PaddingBack展示了抵抗防御方法的能力，同时保持其对人类感知的隐身性。隐蔽性实验的结果已经在https://nbufabio25.github.io/paddingback/上提供。摘要：Machine Learning as a Service (MLaaS) has gained popularity due to advancements in machine learning. However, untrusted third-party platforms have raised concerns about AI security, particularly in backdoor attacks. Recent research has shown that speech backdoors can utilize transformations as triggers, similar to image backdoors. However, human ears easily detect these transformations, leading to suspicion. In this paper, we introduce PaddingBack, an inaudible backdoor attack that utilizes malicious operations to make poisoned samples indistinguishable from clean ones. Instead of using external perturbations as triggers, we exploit the widely used speech signal operation, padding, to break speaker recognition systems. Our experimental results demonstrate the effectiveness of the proposed approach, achieving a significantly high attack success rate while maintaining a high rate of benign accuracy. Furthermore, PaddingBack demonstrates the ability to resist defense methods while maintaining its stealthiness against human perception. The results of the stealthiness experiment have been made available at https://nbufabio25.github.io/paddingback/.
【4】 Dual input neural networks for positional sound source localization标题：用于位置声源定位的双输入神经网络链接：https://arxiv.org/abs/2308.04169
作者：Eric Grinstein,Vincent W. Neo,Patrick A. Naylor摘要：在许多信号处理应用中，元数据可以有利地与高维信号结合使用以产生期望的输出。在经典声源定位（SSL）算法的情况下，来自由许多分布式麦克风接收的高维多声道音频信号的信息与描述场景的声学特性的信息（诸如麦克风在空间中的坐标）组合，以估计声源的位置。我们引入双输入神经网络（DI-NNs）作为在神经网络中对这两种数据类型进行建模的简单有效的方法。我们训练和评估我们提出的DI-NN在不同的难度和现实的情况下，并将其与替代架构，经典的最小二乘（LS）方法以及经典的卷积递归神经网络（CRNN）进行比较。我们的研究结果表明，DI-NN显着优于基线，实现了5倍低于LS方法的定位误差和2倍低于CRNN在测试数据集的真实记录。摘要：In many signal processing applications, metadata may be advantageously used in conjunction with a high dimensional signal to produce a desired output. In the case of classical Sound Source Localization (SSL) algorithms, information from a high dimensional, multichannel audio signals received by many distributed microphones is combined with information describing acoustic properties of the scene, such as the microphones' coordinates in space, to estimate the position of a sound source. We introduce Dual Input Neural Networks (DI-NNs) as a simple and effective way to model these two data types in a neural network. We train and evaluate our proposed DI-NN on scenarios of varying difficulty and realism and compare it against an alternative architecture, a classical Least-Squares (LS) method as well as a classical Convolutional Recurrent Neural Network (CRNN). Our results show that the DI-NN significantly outperforms the baselines, achieving a five times lower localization error than the LS method and two times lower than the CRNN in a test dataset of real recordings.
【5】 OmniDataComposer: A Unified Data Structure for Multimodal Data Fusion  and Infinite Data Generation标题：OmniDataComposer：一种支持多通道数据融合和无限数据生成的统一数据结构链接：https://arxiv.org/abs/2308.04126
作者：Dongyang Yu,Shihao Wang,Yuan Fang,Wangpeng An摘要：本文介绍了OmniDataComposer，多模态数据融合和无限的数据生成的创新方法，旨在完善和简化不同的数据形态之间的相互作用。在核心突破方面，它引入了一种内聚数据结构，擅长处理和合并多模式数据输入，包括视频，音频和文本。我们精心设计的算法利用了多个操作的进步，例如视频/图像字幕提取，密集字幕提取，自动语音识别（ASR），光学字符识别（OCR），识别任何模型（RAM）和对象跟踪。OmniDataComposer能够识别超过6400个类别的对象，大大拓宽了视觉信息的范围。它合并了这些不同的模式，促进模式之间的相互增强，并促进跨模式的数据校正。\textbf{最终输出将每个视频输入变形为一个精心制作的顺序文档}，实际上将视频转化为完整的叙述，使它们更容易被大型语言模型处理。未来的前景包括优化每种模式的数据集，以鼓励无限的数据生成。这个强大的基础将为ChatGPT等模型提供无价的见解，使他们能够为视频字幕创建更高质量的数据集，并简化基于视频内容的问答任务。OmniDataComposer开创了多模态学习的新阶段，为增强AI对复杂真实世界数据的理解和生成提供了巨大的潜力。摘要：This paper presents OmniDataComposer, an innovative approach for multimodal data fusion and unlimited data generation with an intent to refine and uncomplicate interplay among diverse data modalities. Coming to the core breakthrough, it introduces a cohesive data structure proficient in processing and merging multimodal data inputs, which include video, audio, and text. Our crafted algorithm leverages advancements across multiple operations such as video/image caption extraction, dense caption extraction, Automatic Speech Recognition (ASR), Optical Character Recognition (OCR), Recognize Anything Model(RAM), and object tracking. OmniDataComposer is capable of identifying over 6400 categories of objects, substantially broadening the spectrum of visual information. It amalgamates these diverse modalities, promoting reciprocal enhancement among modalities and facilitating cross-modal data correction. \textbf{The final output metamorphoses each video input into an elaborate sequential document}, virtually transmuting videos into thorough narratives, making them easier to be processed by large language models. Future prospects include optimizing datasets for each modality to encourage unlimited data generation. This robust base will offer priceless insights to models like ChatGPT, enabling them to create higher quality datasets for video captioning and easing question-answering tasks based on video content. OmniDataComposer inaugurates a new stage in multimodal learning, imparting enormous potential for augmenting AI's understanding and generation of complex, real-world data.
【6】 MSAC: Multiple Speech Attribute Control Method for Speech Emotion  Recognition标题：MSAC：面向语音情感识别的多语音属性控制方法链接：https://arxiv.org/abs/2308.04025
作者：Yu Pan备注：5 pages摘要：尽管取得了重大进展，语音情感识别（SER）仍然具有挑战性，由于固有的复杂性和模糊性的情感属性，特别是在野生世界。而目前的研究主要集中在识别和泛化能力，这项工作的先驱探索SER方法的可靠性，并探讨如何从数据分布方面的语音情感在不同的语音属性。具体来说，我们首先建立了一个新的基于CNN的SER模型，该模型采用加性余量softmax损失来扩大不同类别特征之间的距离，从而提高它们的区分度。其次，提出了一种新的多语音属性控制方法MSAC显式控制语音属性，使模型受到情感不可知属性的影响较小，并捕获更多的细粒度情感相关的功能。第三，我们第一次尝试测试和分析所提出的SER工作流程的可靠性，使用的分布检测方法。在单一和跨语料库SER场景上的广泛实验表明，我们提出的统一SER工作流程在识别，泛化和可靠性性能方面始终优于基线。此外，在IEMOCAP语料库上，本文提出的SER工作流程在单语料库SER中取得了较好的识别效果，WAR为72.97%，UAR为71.76%.摘要：Despite significant progress, speech emotion recognition (SER) remains challenging due to inherent complexity and ambiguity of the emotion attribute, particularly in wild world. Whereas current studies primarily focus on recognition and generalization capabilities, this work pioneers an exploration into the reliability of SER methods and investigates how to model the speech emotion from the aspect of data distribution across various speech attributes. Specifically, we first build a novel CNN-based SER model which adopts additive margin softmax loss to expand the distance between features of different classes, thereby enhancing their discrimination. Second, a novel multiple speech attribute control method MSAC is proposed to explicitly control speech attributes, enabling the model to be less affected by emotion-agnostic attributes and capture more fine-grained emotion-related features. Third, we make a first attempt to test and analyze the reliability of the proposed SER workflow using the out-of-distribution detection method. Extensive experiments on both single and cross-corpus SER scenarios show that our proposed unified SER workflow consistently outperforms the baseline in terms of recognition, generalization, and reliability performance. Besides, in single-corpus SER, the proposed SER workflow achieves superior recognition results with a WAR of 72.97\% and a UAR of 71.76\% on the IEMOCAP corpus.
【7】 Universal Automatic Phonetic Transcription into the International  Phonetic Alphabet标题：国际音标通用自动拼音链接：https://arxiv.org/abs/2308.03917
作者：Chihiro Taguchi,Yusuke Sakai,Parisa Haghani,David Chiang备注：5 pages, 7 tables摘要：本文提出了一个国家的最先进的模型转录语音在任何语言到国际音标（IPA）。将口语转录成IPA是语言文档中一个必不可少但耗时的过程，即使部分自动化这一过程也有可能大大加快濒危语言的文档编制。像以前最好的语音到IPA模型（Wav2Vec2Phoneme）一样，我们的模型基于wav2vec 2.0，并经过微调以预测来自音频输入的IPA。我们使用来自CommonVoice 11.0的七种语言的训练数据，半自动地转录成IPA。虽然这个训练数据集比Wav2Vec2Phoneme的要小得多，但它更高的质量让我们的模型获得了相当或更好的结果。此外，我们表明，我们的通用语音到IPA模型的质量接近人类注释。摘要：This paper presents a state-of-the-art model for transcribing speech in any language into the International Phonetic Alphabet (IPA). Transcription of spoken languages into IPA is an essential yet time-consuming process in language documentation, and even partially automating this process has the potential to drastically speed up the documentation of endangered languages. Like the previous best speech-to-IPA model (Wav2Vec2Phoneme), our model is based on wav2vec 2.0 and is fine-tuned to predict IPA from audio input. We use training data from seven languages from CommonVoice 11.0, transcribed into IPA semi-automatically. Although this training dataset is much smaller than Wav2Vec2Phoneme's, its higher quality lets our model achieve comparable or better results. Furthermore, we show that the quality of our universal speech-to-IPA models is close to that of human annotators.
【8】 SoK: Acoustic Side Channels标题：SOK：声学侧声道链接：https://arxiv.org/abs/2308.03806
作者：Ping Wang,Shishir Nagaraja,Aurélien Bourquard,Haichang Gao,Jeff Yan备注：16 pages摘要：我们提供最先进的声学侧通道分析，涵盖该领域所有重要的学术研究，讨论其安全影响和对策，并确定未来研究的领域。我们还试图桥接边通道和逆问题，这两个领域似乎是完全孤立的彼此，但有很深的联系。摘要：We provide a state-of-the-art analysis of acoustic side channels, cover all the significant academic research in the area, discuss their security implications and countermeasures, and identify areas for future research. We also make an attempt to bridge side channels and inverse problems, two fields that appear to be completely isolated from each other but have deep connections.
【9】 Comparative Analysis of the wav2vec 2.0 Feature Extractor标题：Wave2vec 2.0特征提取算法的比较分析链接：https://arxiv.org/abs/2308.04286
作者：Peter Vieting,Ralf Schlüter,Hermann Ney备注：Accepted at ITG 2023摘要：自动语音识别（ASR）系统通常使用手工特征提取流水线。为了避免其固有的信息丢失，并实现从语音到转录文本的更一致的建模，神经原始波形特征提取器（FE）是一种有吸引力的方法。此外，wav2vec 2.0模型，最近获得了很大的普及，使用卷积FE直接对语音波形进行操作。然而，在文献中尚未对其进行广泛研究。在这项工作中，我们研究了它的能力，以取代标准的特征提取方法的连接主义时间分类（CTC）ASR模型，并比较它的替代神经FE。我们表明，这两个都是传统的自由电子商务的LibriSpeech基准和分析的影响的各个组成部分的竞争力。此外，我们分析的学习滤波器，并表明，最重要的信息，为ASR系统获得的一组带通滤波器。摘要：Automatic speech recognition (ASR) systems typically use handcrafted feature extraction pipelines. To avoid their inherent information loss and to achieve more consistent modeling from speech to transcribed text, neural raw waveform feature extractors (FEs) are an appealing approach. Also the wav2vec 2.0 model, which has recently gained large popularity, uses a convolutional FE which operates directly on the speech waveform. However, it is not yet studied extensively in the literature. In this work, we study its capability to replace the standard feature extraction methods in a connectionist temporal classification (CTC) ASR model and compare it to an alternative neural FE. We show that both are competitive with traditional FEs on the LibriSpeech benchmark and analyze the effect of the individual components. Furthermore, we analyze the learned filters and show that the most important information for the ASR system is obtained by a set of bandpass filters.
【10】 Advancing Natural-Language Based Audio Retrieval with PaSST and Large  Audio-Caption Data Sets标题：利用PASST和大型音频字幕数据集改进基于自然语言的音频检索链接：https://arxiv.org/abs/2308.04258
作者：Paul Primus,Khaled Koutini,Gerhard Widmer备注：submitted to DCASE Workshop 2023摘要：本文提出了一种基于预训练文本和声谱图Transformers的文本到音频检索系统。我们的方法项目的录音和文字描述到一个共享的音频字幕空间，其中相关的例子从不同的方式接近。通过系统的分析，我们研究了系统的每个组成部分如何影响检索性能。因此，我们确定了两个在推动性能方面发挥关键作用的关键组件：用于音频嵌入的基于自注意的音频编码器以及在预训练期间利用附加的人类生成的和合成的数据集。我们进一步实验用可用的关键字来增强ClothoV 2字幕以增加它们的多样性;然而，这只带来了微不足道的改进。我们的系统在2023年的DCASE挑战赛中排名第一，它在ClothoV 2基准测试中的表现超过了目前的最先进水平5.6 pp。mAP@10。摘要：This work presents a text-to-audio-retrieval system based on pre-trained text and spectrogram transformers. Our method projects recordings and textual descriptions into a shared audio-caption space in which related examples from different modalities are close. Through a systematic analysis, we examine how each component of the system influences retrieval performance. As a result, we identify two key components that play a crucial role in driving performance: the self-attention-based audio encoder for audio embedding and the utilization of additional human-generated and synthetic data sets during pre-training. We further experimented with augmenting ClothoV2 captions with available keywords to increase their variety; however, this only led to marginal improvements. Our system ranked first in the 2023's DCASE Challenge, and it outperforms the current state of the art on the ClothoV2 benchmark by 5.6 pp. mAP@10.
【11】 Target Speech Extraction with Conditional Diffusion Model标题：基于条件扩散模型的目标语音提取链接：https://arxiv.org/abs/2308.03987
作者：Naoyuki Kamo,Marc Delcroix,Tomohiro Nakatan备注：5 pages, 4 figures, Interspeech2023摘要：基于扩散模型的语音增强由于能够产生非常自然的增强信号并且能够很好地推广到看不见的情况而受到越来越多的关注。扩散模型已经被探索用于语音增强的若干子任务，诸如语音去噪、去混响和源分离。在本文中，我们调查他们使用的目标语音提取（TSE），其中包括估计的清洁语音信号的目标扬声器的混合物中的多个扬声器。TSE是通过在识别目标说话人的线索上调节提取过程来实现的。我们表明，我们可以实现TSE使用条件扩散模型的线索。此外，我们引入集成推理，以减少潜在的提取误差所造成的扩散过程。在Libri2mix语料库上的实验中，我们表明，所提出的基于扩散模型的TSE与集成推理相结合，优于一个可比的TSE系统训练的区别。摘要：Diffusion model-based speech enhancement has received increased attention since it can generate very natural enhanced signals and generalizes well to unseen conditions. Diffusion models have been explored for several sub-tasks of speech enhancement, such as speech denoising, dereverberation, and source separation. In this paper, we investigate their use for target speech extraction (TSE), which consists of estimating the clean speech signal of a target speaker in a mixture of multi-talkers. TSE is realized by conditioning the extraction process on a clue identifying the target speaker. We show we can realize TSE using a conditional diffusion model conditioned on the clue. Besides, we introduce ensemble inference to reduce potential extraction errors caused by the diffusion process. In experiments on Libri2mix corpus, we show that the proposed diffusion model-based TSE combined with ensemble inference outperforms a comparable TSE system trained discriminatively.
eess.AS音频处理
【1】 Comparative Analysis of the wav2vec 2.0 Feature Extractor标题：Wave2vec 2.0特征提取算法的比较分析链接：https://arxiv.org/abs/2308.04286
作者：Peter Vieting,Ralf Schlüter,Hermann Ney备注：Accepted at ITG 2023摘要：自动语音识别（ASR）系统通常使用手工特征提取流水线。为了避免其固有的信息丢失，并实现从语音到转录文本的更一致的建模，神经原始波形特征提取器（FE）是一种有吸引力的方法。此外，wav2vec 2.0模型，最近获得了很大的普及，使用卷积FE直接对语音波形进行操作。然而，在文献中尚未对其进行广泛研究。在这项工作中，我们研究了它的能力，以取代标准的特征提取方法的连接主义时间分类（CTC）ASR模型，并比较它的替代神经FE。我们表明，这两个都是传统的自由电子商务的LibriSpeech基准和分析的影响的各个组成部分的竞争力。此外，我们分析的学习滤波器，并表明，最重要的信息，为ASR系统获得的一组带通滤波器。摘要：Automatic speech recognition (ASR) systems typically use handcrafted feature extraction pipelines. To avoid their inherent information loss and to achieve more consistent modeling from speech to transcribed text, neural raw waveform feature extractors (FEs) are an appealing approach. Also the wav2vec 2.0 model, which has recently gained large popularity, uses a convolutional FE which operates directly on the speech waveform. However, it is not yet studied extensively in the literature. In this work, we study its capability to replace the standard feature extraction methods in a connectionist temporal classification (CTC) ASR model and compare it to an alternative neural FE. We show that both are competitive with traditional FEs on the LibriSpeech benchmark and analyze the effect of the individual components. Furthermore, we analyze the learned filters and show that the most important information for the ASR system is obtained by a set of bandpass filters.
【2】 Advancing Natural-Language Based Audio Retrieval with PaSST and Large  Audio-Caption Data Sets标题：利用PASST和大型音频字幕数据集改进基于自然语言的音频检索链接：https://arxiv.org/abs/2308.04258
作者：Paul Primus,Khaled Koutini,Gerhard Widmer备注：submitted to DCASE Workshop 2023摘要：本文提出了一种基于预训练文本和声谱图Transformers的文本到音频检索系统。我们的方法项目的录音和文字描述到一个共享的音频字幕空间，其中相关的例子从不同的方式接近。通过系统的分析，我们研究了系统的每个组成部分如何影响检索性能。因此，我们确定了两个在推动性能方面发挥关键作用的关键组件：用于音频嵌入的基于自注意的音频编码器以及在预训练期间利用附加的人类生成的和合成的数据集。我们进一步实验用可用的关键字来增强ClothoV 2字幕以增加它们的多样性;然而，这只带来了微不足道的改进。我们的系统在2023年的DCASE挑战赛中排名第一，它在ClothoV 2基准测试中的表现超过了目前的最先进水平5.6 pp。mAP@10。摘要：This work presents a text-to-audio-retrieval system based on pre-trained text and spectrogram transformers. Our method projects recordings and textual descriptions into a shared audio-caption space in which related examples from different modalities are close. Through a systematic analysis, we examine how each component of the system influences retrieval performance. As a result, we identify two key components that play a crucial role in driving performance: the self-attention-based audio encoder for audio embedding and the utilization of additional human-generated and synthetic data sets during pre-training. We further experimented with augmenting ClothoV2 captions with available keywords to increase their variety; however, this only led to marginal improvements. Our system ranked first in the 2023's DCASE Challenge, and it outperforms the current state of the art on the ClothoV2 benchmark by 5.6 pp. mAP@10.
【3】 Investigating Speaker Embedding Disentanglement on Natural Read Speech标题：自然阅读语音中嵌入解缠的说话人研究链接：https://arxiv.org/abs/2308.04225
作者：Michael Kuhlmann,Adrian Meise,Fritz Seebauer,Petra Wagner,Reinhold Haeb-Umbach备注：To be published at 15th ITG conference on speech communication摘要：解纠缠是学习表征的任务，这些表征识别和分离解释数据中观察到的变化的因素。解纠缠表示对于提高数据驱动模型的可概括性、可解释性和公平性是有用的。人们对这种解纠缠对于语音表征的效果知之甚少。解决语音表示的解纠缠时的一个主要挑战是语音信号的未知生成因素。在这项工作中，我们调查到什么程度的语音表示编码扬声器的身份可以解开。为了量化解纠缠，我们确定的声学特征是高度说话者变化，可以作为代理的变化的因素的语音。我们发现，当用标准目标训练时，说话人嵌入的解纠缠是有限的，但可以在一定程度上改善香草表示学习。摘要：Disentanglement is the task of learning representations that identify and separate factors that explain the variation observed in data. Disentangled representations are useful to increase the generalizability, explainability, and fairness of data-driven models. Only little is known about how well such disentanglement works for speech representations. A major challenge when tackling disentanglement for speech representations are the unknown generative factors underlying the speech signal. In this work, we investigate to what degree speech representations encoding speaker identity can be disentangled. To quantify disentanglement, we identify acoustic features that are highly speaker-variant and can serve as proxies for the factors of variation underlying speech. We find that disentanglement of the speaker embedding is limited when trained with standard objectives promoting disentanglement but can be improved over vanilla representation learning to some extent.
【4】 Target Speech Extraction with Conditional Diffusion Model标题：基于条件扩散模型的目标语音提取链接：https://arxiv.org/abs/2308.03987
作者：Naoyuki Kamo,Marc Delcroix,Tomohiro Nakatan备注：5 pages, 4 figures, Interspeech2023摘要：基于扩散模型的语音增强由于能够产生非常自然的增强信号并且能够很好地推广到看不见的情况而受到越来越多的关注。扩散模型已经被探索用于语音增强的若干子任务，诸如语音去噪、去混响和源分离。在本文中，我们调查他们使用的目标语音提取（TSE），其中包括估计的清洁语音信号的目标扬声器的混合物中的多个扬声器。TSE是通过在识别目标说话人的线索上调节提取过程来实现的。我们表明，我们可以实现TSE使用条件扩散模型的线索。此外，我们引入集成推理，以减少潜在的提取误差所造成的扩散过程。在Libri2mix语料库上的实验中，我们表明，所提出的基于扩散模型的TSE与集成推理相结合，优于一个可比的TSE系统训练的区别。摘要：Diffusion model-based speech enhancement has received increased attention since it can generate very natural enhanced signals and generalizes well to unseen conditions. Diffusion models have been explored for several sub-tasks of speech enhancement, such as speech denoising, dereverberation, and source separation. In this paper, we investigate their use for target speech extraction (TSE), which consists of estimating the clean speech signal of a target speaker in a mixture of multi-talkers. TSE is realized by conditioning the extraction process on a clue identifying the target speaker. We show we can realize TSE using a conditional diffusion model conditioned on the clue. Besides, we introduce ensemble inference to reduce potential extraction errors caused by the diffusion process. In experiments on Libri2mix corpus, we show that the proposed diffusion model-based TSE combined with ensemble inference outperforms a comparable TSE system trained discriminatively.
【5】 Towards an AI to Win Ghana's National Science and Maths Quiz标题：走向人工智能赢得加纳国家科学和数学测验链接：https://arxiv.org/abs/2308.04333
作者：George Boateng,Jonathan Abrefah Mensah,Kevin Takyi Yeboah,William Edor,Andrew Kojo Mensah-Onumah,Naafi Dasana Ibrahim,Nana Sam Yeboah备注：7 pages. Under review at Deep Learning Indaba and Black in AI Workshop @NeurIPS 2023摘要：人工智能能赢得加纳的国家科学和数学测验吗？这就是我们在MSMQ AI项目中寻求答案的问题，这是一个开源项目，它正在构建AI以在MSMQ中竞争并获胜。MSMQ是加纳高中生的年度现场科学和数学竞赛，其中3队2名学生通过回答生物，化学，物理和数学问题进行5轮5个渐进阶段的竞争，直到获胜的团队获得该年度的冠军。MSMQ是一个令人兴奋的现场问答比赛，在语音到文本、文本到语音、问答和人机交互方面都有有趣的技术挑战。在2023年1月开始的这项正在进行的工作中，我们概述了该项目，描述了每个团队，到目前为止取得的进展，以及我们计划在10月份为MSMQ 2023推出和首次推出AI的下一步。克服这一巨大挑战的人工智能可以对教育产生现实世界的影响，例如使非洲数百万学生能够从这种人工智能中获得一对一的学习支持。摘要：Can an AI win Ghana's National Science and Maths Quiz (NSMQ)? That is the question we seek to answer in the NSMQ AI project, an open-source project that is building AI to compete live in the NSMQ and win. The NSMQ is an annual live science and mathematics competition for senior secondary school students in Ghana in which 3 teams of 2 students compete by answering questions across biology, chemistry, physics, and math in 5 rounds over 5 progressive stages until a winning team is crowned for that year. The NSMQ is an exciting live quiz competition with interesting technical challenges across speech-to-text, text-to-speech, question-answering, and human-computer interaction. In this ongoing work that began in January 2023, we give an overview of the project, describe each of the teams, progress made thus far, and the next steps toward our planned launch and debut of the AI in October for NSMQ 2023. An AI that conquers this grand challenge can have real-world impact on education such as enabling millions of students across Africa to have one-on-one learning support from this AI.
【6】 Auditory Attention Decoding with Task-Related Multi-View Contrastive  Learning标题：基于任务相关多视点对比学习的听觉注意解码链接：https://arxiv.org/abs/2308.04244
作者：Xiaoyu Chen,Changde Du,Qiongyi Zhou,Huiguang He摘要：在鸡尾酒会等场合，人类的大脑可以很容易地专注于一个演讲者，而压制其他人。最近，研究人员发现听觉注意可以从脑电图（EEG）数据解码。然而，大多数现有的深度学习方法难以利用不同视图的先验知识（即参与语音和EEG都是任务相关的视图），提取出令人不满意的表示。Broadbent的过滤器模型的启发下，我们解码听觉注意在多视图的范例，并提取最相关的和重要的信息，利用丢失的视图。具体来说，我们提出了一个听觉注意解码（AAD）方法的基础上，多视图VAE与任务相关的多视图对比（TMC）学习。在多视图VAE中采用TMC学习可以利用缺失的视图将不同视图的先验知识累积到表示的融合中，并提取近似的任务相关表示。我们检查我们的方法在两个流行的AAD数据集，并证明我们的方法的优越性，通过比较它的国家的最先进的方法。摘要：The human brain can easily focus on one speaker and suppress others in scenarios such as a cocktail party. Recently, researchers found that auditory attention can be decoded from the electroencephalogram (EEG) data. However, most existing deep learning methods are difficult to use prior knowledge of different views (that is attended speech and EEG are task-related views) and extract an unsatisfactory representation. Inspired by Broadbent's filter model, we decode auditory attention in a multi-view paradigm and extract the most relevant and important information utilizing the missing view. Specifically, we propose an auditory attention decoding (AAD) method based on multi-view VAE with task-related multi-view contrastive (TMC) learning. Employing TMC learning in multi-view VAE can utilize the missing view to accumulate prior knowledge of different views into the fusion of representation, and extract the approximate task-related representation. We examine our method on two popular AAD datasets, and demonstrate the superiority of our method by comparing it to the state-of-the-art method.
【7】 Evil Operation: Breaking Speaker Recognition with PaddingBack标题：邪恶操作：用PaddingBack破解说话人识别链接：https://arxiv.org/abs/2308.04179
作者：Zhe Ye,Diqun Yan,Li Dong,Kailai Shen摘要：机器学习即服务（MLaaS）由于机器学习的进步而获得了普及。然而，不受信任的第三方平台引发了人们对AI安全性的担忧，特别是在后门攻击方面。最近的研究表明，语音后门可以利用转换作为触发器，类似于图像后门。然而，人类的耳朵很容易察觉到这些变化，导致怀疑。在本文中，我们介绍了PaddingBack，一种听不见的后门攻击，利用恶意操作，使中毒样本无法区分干净的。而不是使用外部扰动作为触发器，我们利用广泛使用的语音信号操作，填充，打破说话人识别系统。我们的实验结果表明，所提出的方法的有效性，实现了显着高的攻击成功率，同时保持了较高的良性准确率。此外，PaddingBack展示了抵抗防御方法的能力，同时保持其对人类感知的隐身性。隐蔽性实验的结果已经在https://nbufabio25.github.io/paddingback/上提供。摘要：Machine Learning as a Service (MLaaS) has gained popularity due to advancements in machine learning. However, untrusted third-party platforms have raised concerns about AI security, particularly in backdoor attacks. Recent research has shown that speech backdoors can utilize transformations as triggers, similar to image backdoors. However, human ears easily detect these transformations, leading to suspicion. In this paper, we introduce PaddingBack, an inaudible backdoor attack that utilizes malicious operations to make poisoned samples indistinguishable from clean ones. Instead of using external perturbations as triggers, we exploit the widely used speech signal operation, padding, to break speaker recognition systems. Our experimental results demonstrate the effectiveness of the proposed approach, achieving a significantly high attack success rate while maintaining a high rate of benign accuracy. Furthermore, PaddingBack demonstrates the ability to resist defense methods while maintaining its stealthiness against human perception. The results of the stealthiness experiment have been made available at https://nbufabio25.github.io/paddingback/.
【8】 Dual input neural networks for positional sound source localization标题：用于位置声源定位的双输入神经网络链接：https://arxiv.org/abs/2308.04169
作者：Eric Grinstein,Vincent W. Neo,Patrick A. Naylor摘要：在许多信号处理应用中，元数据可以有利地与高维信号结合使用以产生期望的输出。在经典声源定位（SSL）算法的情况下，来自由许多分布式麦克风接收的高维多声道音频信号的信息与描述场景的声学特性的信息（诸如麦克风在空间中的坐标）组合，以估计声源的位置。我们引入双输入神经网络（DI-NNs）作为在神经网络中对这两种数据类型进行建模的简单有效的方法。我们训练和评估我们提出的DI-NN在不同的难度和现实的情况下，并将其与替代架构，经典的最小二乘（LS）方法以及经典的卷积递归神经网络（CRNN）进行比较。我们的研究结果表明，DI-NN显着优于基线，实现了5倍低于LS方法的定位误差和2倍低于CRNN在测试数据集的真实记录。摘要：In many signal processing applications, metadata may be advantageously used in conjunction with a high dimensional signal to produce a desired output. In the case of classical Sound Source Localization (SSL) algorithms, information from a high dimensional, multichannel audio signals received by many distributed microphones is combined with information describing acoustic properties of the scene, such as the microphones' coordinates in space, to estimate the position of a sound source. We introduce Dual Input Neural Networks (DI-NNs) as a simple and effective way to model these two data types in a neural network. We train and evaluate our proposed DI-NN on scenarios of varying difficulty and realism and compare it against an alternative architecture, a classical Least-Squares (LS) method as well as a classical Convolutional Recurrent Neural Network (CRNN). Our results show that the DI-NN significantly outperforms the baselines, achieving a five times lower localization error than the LS method and two times lower than the CRNN in a test dataset of real recordings.
【9】 EPCFormer: Expression Prompt Collaboration Transformer for Universal  Referring Video Object Segmentation标题：EPCFormer：通用参考视频对象分割的表情提示协作转换器链接：https://arxiv.org/abs/2308.04162
作者：Jiajun Chen,Jiacheng Lin,Zhiqiang Xiao,Haolong Fu,Ke Nai,Kailun Yang,Zhiyong Li备注：The source code will be made publicly available at this https URL摘要：音频引导的视频对象分割（A-VOS）和参考视频对象分割（R-VOS）是两个高度相关的任务，它们都旨在根据用户提供的表达提示从视频序列中分割特定对象。然而，由于在不同模态的建模表示中的挑战，当代方法努力在交互灵活性与高精度定位和分割之间取得平衡。在本文中，我们从两个角度来解决这个问题：音频和文本的对齐表示以及音频、文本和视觉特征之间的深度交互。首先，我们提出了一个通用的架构，表达式提示协作Transformer，这里EPCFormer。接下来，我们提出了一个表达式对齐（EA）的机制，音频和文本表达。通过引入音频和文本表达的对比学习，建议EPCFormer实现的音频和文本表达表示相同的对象之间的语义等价的理解。然后，为了促进音频，文本和视频功能之间的深度交互，我们引入了表达视觉注意力（EVA）机制。在表达提示方面的视频对象分割的知识可以通过深入探索文本和音频之间的互补线索在两个任务之间无缝地转移。在公认的基准测试上的实验表明，我们的通用EPCFormer在这两项任务上都达到了最先进的结果。EPCFormer的源代码将在https://github.com/lab206/EPCFormer上公开。摘要：Audio-guided Video Object Segmentation (A-VOS) and Referring Video Object Segmentation (R-VOS) are two highly-related tasks, which both aim to segment specific objects from video sequences according to user-provided expression prompts. However, due to the challenges in modeling representations for different modalities, contemporary methods struggle to strike a balance between interaction flexibility and high-precision localization and segmentation. In this paper, we address this problem from two perspectives: the alignment representation of audio and text and the deep interaction among audio, text, and visual features. First, we propose a universal architecture, the Expression Prompt Collaboration Transformer, herein EPCFormer. Next, we propose an Expression Alignment (EA) mechanism for audio and text expressions. By introducing contrastive learning for audio and text expressions, the proposed EPCFormer realizes comprehension of the semantic equivalence between audio and text expressions denoting the same objects. Then, to facilitate deep interactions among audio, text, and video features, we introduce an Expression-Visual Attention (EVA) mechanism. The knowledge of video object segmentation in terms of the expression prompts can seamlessly transfer between the two tasks by deeply exploring complementary cues between text and audio. Experiments on well-recognized benchmarks demonstrate that our universal EPCFormer attains state-of-the-art results on both tasks. The source code of EPCFormer will be made publicly available at https://github.com/lab206/EPCFormer.
【10】 OmniDataComposer: A Unified Data Structure for Multimodal Data Fusion  and Infinite Data Generation标题：OmniDataComposer：一种支持多通道数据融合和无限数据生成的统一数据结构链接：https://arxiv.org/abs/2308.04126
作者：Dongyang Yu,Shihao Wang,Yuan Fang,Wangpeng An摘要：本文介绍了OmniDataComposer，多模态数据融合和无限的数据生成的创新方法，旨在完善和简化不同的数据形态之间的相互作用。在核心突破方面，它引入了一种内聚数据结构，擅长处理和合并多模式数据输入，包括视频，音频和文本。我们精心设计的算法利用了多个操作的进步，例如视频/图像字幕提取，密集字幕提取，自动语音识别（ASR），光学字符识别（OCR），识别任何模型（RAM）和对象跟踪。OmniDataComposer能够识别超过6400个类别的对象，大大拓宽了视觉信息的范围。它合并了这些不同的模式，促进模式之间的相互增强，并促进跨模式的数据校正。\textbf{最终输出将每个视频输入变形为一个精心制作的顺序文档}，实际上将视频转化为完整的叙述，使它们更容易被大型语言模型处理。未来的前景包括优化每种模式的数据集，以鼓励无限的数据生成。这个强大的基础将为ChatGPT等模型提供无价的见解，使他们能够为视频字幕创建更高质量的数据集，并简化基于视频内容的问答任务。OmniDataComposer开创了多模态学习的新阶段，为增强AI对复杂真实世界数据的理解和生成提供了巨大的潜力。摘要：This paper presents OmniDataComposer, an innovative approach for multimodal data fusion and unlimited data generation with an intent to refine and uncomplicate interplay among diverse data modalities. Coming to the core breakthrough, it introduces a cohesive data structure proficient in processing and merging multimodal data inputs, which include video, audio, and text. Our crafted algorithm leverages advancements across multiple operations such as video/image caption extraction, dense caption extraction, Automatic Speech Recognition (ASR), Optical Character Recognition (OCR), Recognize Anything Model(RAM), and object tracking. OmniDataComposer is capable of identifying over 6400 categories of objects, substantially broadening the spectrum of visual information. It amalgamates these diverse modalities, promoting reciprocal enhancement among modalities and facilitating cross-modal data correction. \textbf{The final output metamorphoses each video input into an elaborate sequential document}, virtually transmuting videos into thorough narratives, making them easier to be processed by large language models. Future prospects include optimizing datasets for each modality to encourage unlimited data generation. This robust base will offer priceless insights to models like ChatGPT, enabling them to create higher quality datasets for video captioning and easing question-answering tasks based on video content. OmniDataComposer inaugurates a new stage in multimodal learning, imparting enormous potential for augmenting AI's understanding and generation of complex, real-world data.
【11】 MSAC: Multiple Speech Attribute Control Method for Speech Emotion  Recognition标题：MSAC：面向语音情感识别的多语音属性控制方法链接：https://arxiv.org/abs/2308.04025
作者：Yu Pan备注：5 pages摘要：尽管取得了重大进展，语音情感识别（SER）仍然具有挑战性，由于固有的复杂性和模糊性的情感属性，特别是在野生世界。而目前的研究主要集中在识别和泛化能力，这项工作的先驱探索SER方法的可靠性，并探讨如何从数据分布方面的语音情感在不同的语音属性。具体来说，我们首先建立了一个新的基于CNN的SER模型，该模型采用加性余量softmax损失来扩大不同类别特征之间的距离，从而提高它们的区分度。其次，提出了一种新的多语音属性控制方法MSAC显式控制语音属性，使模型受到情感不可知属性的影响较小，并捕获更多的细粒度情感相关的功能。第三，我们第一次尝试测试和分析所提出的SER工作流程的可靠性，使用的分布检测方法。在单一和跨语料库SER场景上的广泛实验表明，我们提出的统一SER工作流程在识别，泛化和可靠性性能方面始终优于基线。此外，在IEMOCAP语料库上，本文提出的SER工作流程在单语料库SER中取得了较好的识别效果，WAR为72.97%，UAR为71.76%.摘要：Despite significant progress, speech emotion recognition (SER) remains challenging due to inherent complexity and ambiguity of the emotion attribute, particularly in wild world. Whereas current studies primarily focus on recognition and generalization capabilities, this work pioneers an exploration into the reliability of SER methods and investigates how to model the speech emotion from the aspect of data distribution across various speech attributes. Specifically, we first build a novel CNN-based SER model which adopts additive margin softmax loss to expand the distance between features of different classes, thereby enhancing their discrimination. Second, a novel multiple speech attribute control method MSAC is proposed to explicitly control speech attributes, enabling the model to be less affected by emotion-agnostic attributes and capture more fine-grained emotion-related features. Third, we make a first attempt to test and analyze the reliability of the proposed SER workflow using the out-of-distribution detection method. Extensive experiments on both single and cross-corpus SER scenarios show that our proposed unified SER workflow consistently outperforms the baseline in terms of recognition, generalization, and reliability performance. Besides, in single-corpus SER, the proposed SER workflow achieves superior recognition results with a WAR of 72.97\% and a UAR of 71.76\% on the IEMOCAP corpus.
【12】 Universal Automatic Phonetic Transcription into the International  Phonetic Alphabet标题：国际音标通用自动拼音链接：https://arxiv.org/abs/2308.03917
作者：Chihiro Taguchi,Yusuke Sakai,Parisa Haghani,David Chiang备注：5 pages, 7 tables摘要：本文提出了一个国家的最先进的模型转录语音在任何语言到国际音标（IPA）。将口语转录成IPA是语言文档中一个必不可少但耗时的过程，即使部分自动化这一过程也有可能大大加快濒危语言的文档编制。像以前最好的语音到IPA模型（Wav2Vec2Phoneme）一样，我们的模型基于wav2vec 2.0，并经过微调以预测来自音频输入的IPA。我们使用来自CommonVoice 11.0的七种语言的训练数据，半自动地转录成IPA。虽然这个训练数据集比Wav2Vec2Phoneme的要小得多，但它更高的质量让我们的模型获得了相当或更好的结果。此外，我们表明，我们的通用语音到IPA模型的质量接近人类注释。摘要：This paper presents a state-of-the-art model for transcribing speech in any language into the International Phonetic Alphabet (IPA). Transcription of spoken languages into IPA is an essential yet time-consuming process in language documentation, and even partially automating this process has the potential to drastically speed up the documentation of endangered languages. Like the previous best speech-to-IPA model (Wav2Vec2Phoneme), our model is based on wav2vec 2.0 and is fine-tuned to predict IPA from audio input. We use training data from seven languages from CommonVoice 11.0, transcribed into IPA semi-automatically. Although this training dataset is much smaller than Wav2Vec2Phoneme's, its higher quality lets our model achieve comparable or better results. Furthermore, we show that the quality of our universal speech-to-IPA models is close to that of human annotators.
【13】 SoK: Acoustic Side Channels标题：SOK：声学侧声道链接：https://arxiv.org/abs/2308.03806
作者：Ping Wang,Shishir Nagaraja,Aurélien Bourquard,Haichang Gao,Jeff Yan备注：16 pages摘要：我们提供最先进的声学侧通道分析，涵盖该领域所有重要的学术研究，讨论其安全影响和对策，并确定未来研究的领域。我们还试图桥接边通道和逆问题，这两个领域似乎是完全孤立的彼此，但有很深的联系。摘要：We provide a state-of-the-art analysis of acoustic side channels, cover all the significant academic research in the area, discuss their security implications and countermeasures, and identify areas for future research. We also make an attempt to bridge side channels and inverse problems, two fields that appear to be completely isolated from each other but have deep connections.
机器翻译，仅供参考