语音/音频处理学术速递[10.24]

今日论文合集：cs.SD语音20篇，eess.AS音频处理27篇。

cs.SD语音

【1】 Novel-View Acoustic Synthesis from 3D Reconstructed Rooms标题：基于三维重建房间的新视图声合成链接：https://arxiv.org/abs/2310.15130

作者：Byeongjoo Ahn,Karren Yang,Brian Hamilton,Jonathan Sheaffer,Anurag Ranjan,Miguel Sarabia,Oncel Tuzel,Jen-Hao Rick Chang摘要：我们调查的好处相结合的盲音频记录与3D场景信息的新颖视图声学合成。给定来自2-4个麦克风的音频记录以及包含多个未知声源的场景的3D几何形状和材料，我们估计场景中任何地方的声音。我们确定了主要的挑战，新颖的视图声学合成声源定位，分离和去混响。虽然单纯地训练端到端网络无法产生高质量的结果，但我们发现，将来自3D重建房间的房间脉冲响应（RIR）合并在一起，使同一个网络能够共同处理这些任务。我们的方法优于现有的方法设计的个人任务，证明其有效性，利用三维视觉信息。在对Matterport 3D-NVAS数据集的模拟研究中，我们的模型在源定位方面实现了近乎完美的精度，源分离和去混响的PSNR为26.44 dB，SDR为14.23 dB，从而在新视图声学合成上实现了25.55 dB的PSNR和14.20 dB的SDR。代码、预训练模型和视频结果可在项目网页上获得（https：//github.com/apple/ml-nvas3d）。摘要：We investigate the benefit of combining blind audio recordings with 3D scene information for novel-view acoustic synthesis. Given audio recordings from 2-4 microphones and the 3D geometry and material of a scene containing multiple unknown sound sources, we estimate the sound anywhere in the scene. We identify the main challenges of novel-view acoustic synthesis as sound source localization, separation, and dereverberation. While naively training an end-to-end network fails to produce high-quality results, we show that incorporating room impulse responses (RIRs) derived from 3D reconstructed rooms enables the same network to jointly tackle these tasks. Our method outperforms existing methods designed for the individual tasks, demonstrating its effectiveness at utilizing 3D visual information. In a simulated study on the Matterport3D-NVAS dataset, our model achieves near-perfect accuracy on source localization, a PSNR of 26.44 dB and a SDR of 14.23 dB for source separation and dereverberation, resulting in a PSNR of 25.55 dB and a SDR of 14.20 dB on novel-view acoustic synthesis. Code, pretrained model, and video results are available on the project webpage (https://github.com/apple/ml-nvas3d).

【2】 Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognition标题：一种基于整形器的高效端到端语音识别关键帧机制链接：https://arxiv.org/abs/2310.14954

作者：Peng Fan,Changhao Shan,Jianwei Zhang,Sining Sun,Qing Yang备注：This manuscript has been accepted by IEEE Signal Processing Letters for publication摘要：最近，Conformer作为端到端自动语音识别的骨干网络实现了最先进的性能。Conformer块利用自注意机制来捕获全局信息，以及卷积神经网络来捕获局部信息，从而提高性能。然而，基于Conformer的模型遇到了自注意机制的问题，因为计算复杂度随着输入序列的长度呈二次方增长。受先前的连接主义时间分类（CTC）在解码期间引导的空白跳过的启发，我们将中间CTC输出作为指导引入Conformer编码器的下采样过程。我们将非空白输出的帧定义为关键帧。具体来说，我们介绍了基于关键帧的自注意（KFSA）机制，一种新的方法，以减少使用关键帧的自注意机制的计算。我们提出的方法的结构包括两个编码器。在初始编码器之后，我们引入了一个中间CTC损失函数来计算标签帧，使我们能够为KFSA提取关键帧和空白帧。此外，我们引入了基于关键帧的下采样（KFDS）机制，直接对高维声学特征进行操作，并丢弃与空白标签对应的帧，从而产生新的声学特征序列作为第二编码器的输入。通过使用所提出的方法，它实现了可比或更高的性能比香草构象和其他类似的工作，如有效的构象。同时，该方法在模型训练和推理过程中可以丢弃60%以上的无用帧，大大提高了推理速度。此工作代码可在{https：//github.com/scufan1990/Key-Frame-Mechanism-For-Efficient-Conformer}中找到摘要：Recently, Conformer as a backbone network for end-to-end automatic speech recognition achieved state-of-the-art performance. The Conformer block leverages a self-attention mechanism to capture global information, along with a convolutional neural network to capture local information, resulting in improved performance. However, the Conformer-based model encounters an issue with the self-attention mechanism, as computational complexity grows quadratically with the length of the input sequence. Inspired by previous Connectionist Temporal Classification (CTC) guided blank skipping during decoding, we introduce intermediate CTC outputs as guidance into the downsampling procedure of the Conformer encoder. We define the frame with non-blank output as key frame. Specifically, we introduce the key frame-based self-attention (KFSA) mechanism, a novel method to reduce the computation of the self-attention mechanism using key frames. The structure of our proposed approach comprises two encoders. Following the initial encoder, we introduce an intermediate CTC loss function to compute the label frame, enabling us to extract the key frames and blank frames for KFSA. Furthermore, we introduce the key frame-based downsampling (KFDS) mechanism to operate on high-dimensional acoustic features directly and drop the frames corresponding to blank labels, which results in new acoustic feature sequences as input to the second encoder. By using the proposed method, which achieves comparable or higher performance than vanilla Conformer and other similar work such as Efficient Conformer. Meantime, our proposed method can discard more than 60\% useless frames during model training and inference, which will accelerate the inference speed significantly. This work code is available in {https://github.com/scufan1990/Key-Frame-Mechanism-For-Efficient-Conformer}

【3】 8+8=4: Formalizing Time Units to Handle Symbolic Music Durations标题：8+8=4：将时间单位正式化以处理符号音乐持续时间链接：https://arxiv.org/abs/2310.14952

作者：Emmanouil Karystinaios,Francesco Foscarin,Florent Jacquemard,Masahiko Sakai,Satoshi Tojo,Gerhard Widmer备注：In Proceedings of the International Symposium on Computer Music Multidisciplinary Research (CMMR 2023), Tokyo, Japan摘要：本文着重讨论符号乐谱中音乐事件（音符和休止符）的标称持续时间，以及如何在计算机应用中方便地处理这些问题。我们提出了一个时间单位，这是直接相关的乐谱中的图形符号的使用和配对这与一组操作，涵盖典型的计算在音乐应用程序。我们正式这个时间单位和更常用的方法在一个单一的数学框架，半环，代数结构，使算法/处理管道的抽象描述。然后，我们讨论了一些实际的用例，并强调了我们的系统何时可以通过使它们在使用的数据类型和计算数量方面更有效来改进这些管道。摘要：This paper focuses on the nominal durations of musical events (notes and rests) in a symbolic musical score, and on how to conveniently handle these in computer applications. We propose the usage of a temporal unit that is directly related to the graphical symbols in musical scores and pair this with a set of operations that cover typical computations in music applications. We formalize this time unit and the more commonly used approach in a single mathematical framework, as semirings, algebraic structures that enable an abstract description of algorithms/processing pipelines. We then discuss some practical use cases and highlight when our system can improve such pipelines by making them more efficient in terms of data type used and the number of computations.

【4】 Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model标题：使用单一训练模型的直观多语言视听语音识别链接：https://arxiv.org/abs/2310.14946

作者：Joanna Hong,Se Jin Park,Yong Man Ro备注：EMNLP 2023 Findings摘要：我们提出了一种新的方法，多语言的视听语音识别任务，通过引入一个单一的多语言数据集的模型。受人类认知系统的启发，人类可以直观地区分不同的语言，没有任何有意识的努力或指导，我们提出了一个模型，可以捕捉哪种语言是作为输入语音，通过区分语言之间的内在相似性和差异。为此，我们设计了一种快速微调技术，用于基本上预先训练的视听表示模型，以便网络可以识别语言类以及相应语言的语音。我们的工作有助于开发强大而高效的多语言视听语音识别系统，减少对特定语言模型的需求。摘要：We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.

【5】 SpeakEasy: A Conversational Intelligence Chatbot for Enhancing College Students' Communication Skills标题：Speakeasy：一种提高大学生沟通能力的对话智能聊天机器人链接：https://arxiv.org/abs/2310.14891

作者：Hyunbae Jeon,Rhea Ramachandran,Victoria Ploerer,Yella Diekmann,Max Bagga摘要：社会交往和谈话技巧将成功者与其他人区分开来，将自信者与害羞者区分开来。特别是对于大学生来说，交谈的能力可以是日常生活中所经历的压力和焦虑的一个出口，也是所有重要职业技能的基础。有鉴于此，我们设计了SpeakEasy：一个具有一定程度智能的聊天机器人，可以向用户提供反馈，以了解他们与聊天机器人进行自由形式对话的能力。SpeakEasy试图通过与用户进行七分钟的口语对话来帮助大学生提高他们的沟通技巧，使用基于先前心理学和语言学研究设计的指标来分析用户的反应，并向用户提供如何提高他们的对话能力的反馈。为了模拟自然对话，SpeakEasy可以与用户就两个人第一次见面时可能讨论的各种话题进行对话：旅行，体育和娱乐。与大多数以提高对话技能为目标的其他聊天机器人不同，SpeakEasy实际上记录了用户的讲话，将音频转录为令牌，并使用宏-例如，计算语音节奏的序列，确定用户是否过度依赖某些单词，并识别尴尬的过渡，以评估对话的质量。根据评估结果，SpeakEasy会提供详细的反馈，指导用户如何改进他们的对话。反过来，SpeakEasy会根据用户对SpeakEasy性能的一系列问题的回答来更新其算法。摘要：Social interactions and conversation skills separate the successful from the rest and the confident from the shy. For college students in particular, the ability to converse can be an outlet for the stress and anxiety experienced on a daily basis along with a foundation for all-important career skills. In light of this, we designed SpeakEasy: a chatbot with some degree of intelligence that provides feedback to the user on their ability to engage in free-form conversations with the chatbot. SpeakEasy attempts to help college students improve their communication skills by engaging in a seven-minute spoken conversation with the user, analyzing the user's responses with metrics designed based on previous psychology and linguistics research, and providing feedback to the user on how they can improve their conversational ability. To simulate natural conversation, SpeakEasy converses with the user on a wide assortment of topics that two people meeting for the first time might discuss: travel, sports, and entertainment. Unlike most other chatbots with the goal of improving conversation skills, SpeakEasy actually records the user speaking, transcribes the audio into tokens, and uses macros-e.g., sequences that calculate the pace of speech, determine if the user has an over-reliance on certain words, and identifies awkward transitions-to evaluate the quality of the conversation. Based on the evaluation, SpeakEasy provides elaborate feedback on how the user can improve their conversations. In turn, SpeakEasy updates its algorithms based on a series of questions that the user responds to regarding SpeakEasy's performance.

【6】 Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translation标题：利用时间戳信息进行串行联合流识别和翻译链接：https://arxiv.org/abs/2310.14806

作者：Sara Papi,Peidong Wang,Junkun Chen,Jian Xue,Naoyuki Kanda,Jinyu Li,Yashesh Gaur备注：\c{opyright} 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：对即时口语转录和翻译的需求日益增长，这是由于全球交流和跨语言互动的增加。这使得提供多种语言的翻译对于用户应用程序至关重要。传统的自动语音识别（ASR）和语音翻译（ST）方法通常依赖于单独的系统，导致计算资源效率低下，并增加了实时同步的复杂性。在本文中，我们提出了一个流式转换器-转换器（T-T）模型，能够使用单个解码器联合产生多对一和一对多的转录和翻译。我们介绍了一种新的方法，联合令牌级串行化输出训练的基础上，时间戳信息，有效地产生ASR和ST输出流设置。在{it，es，de}->en上的实验证明了该方法的有效性，首次实现了用单个解码器生成一对多联合输出。摘要：The growing need for instant spoken language transcription and translation is driven by increased global communication and cross-lingual interactions. This has made offering translations in multiple languages essential for user applications. Traditional approaches to automatic speech recognition (ASR) and speech translation (ST) have often relied on separate systems, leading to inefficiencies in computational resources, and increased synchronization complexity in real time. In this paper, we propose a streaming Transformer-Transducer (T-T) model able to jointly produce many-to-one and one-to-many transcription and translation using a single decoder. We introduce a novel method for joint token-level serialized output training based on timestamp information to effectively produce ASR and ST outputs in the streaming setting. Experiments on {it,es,de}->en prove the effectiveness of our approach, enabling the generation of one-to-many joint outputs with a single decoder for the first time.

【7】 A Novel Transfer Learning Method Utilizing Acoustic and Vibration Signals for Rotating Machinery Fault Diagnosis标题：一种基于声学和振动信号的旋转机械故障诊断传递学习新方法链接：https://arxiv.org/abs/2310.14796

作者：Zhongliang Chen,Zhuofei Huang,Wenxiong Kang摘要：旋转机械故障诊断对现代工业系统的安全稳定运行起着重要作用。然而，训练数据与实际操作场景数据之间存在分布差异，导致现有系统性能下降。本文提出了一种基于迁移学习的方法，利用声学和振动信号来解决这种分布差异。为了提供更丰富、更可靠的故障信息，设计了声振特征融合MAVgram，并与基于DNN的分类器相配合，获得更有效的诊断表示。对骨干进行预训练，然后进行微调，以获得目标任务的优异性能。实验结果证明了该方法的有效性，与STgram-MFN方法相比，其性能得到了提高。摘要：Fault diagnosis of rotating machinery plays a important role for the safety and stability of modern industrial systems. However, there is a distribution discrepancy between training data and data of real-world operation scenarios, which causing the decrease of performance of existing systems. This paper proposed a transfer learning based method utilizing acoustic and vibration signal to address this distribution discrepancy. We designed the acoustic and vibration feature fusion MAVgram to offer richer and more reliable information of faults, coordinating with a DNN-based classifier to obtain more effective diagnosis representation. The backbone was pre-trained and then fine-tuned to obtained excellent performance of the target task. Experimental results demonstrate the effectiveness of the proposed method, and achieved improved performance compared to STgram-MFN.

【8】 Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directions标题：视听说话人跟踪：进展、挑战和未来方向链接：https://arxiv.org/abs/2310.14778

作者：Jinzheng Zhao,Yong Xu,Xinyuan Qian,Davide Berghi,Peipei Wu,Meng Cui,Jianyuan Sun,Philip J. B. Jackson,Wenwu Wang摘要：视听说话人跟踪由于其学术价值和广泛的应用，近年来受到越来越多的关注。音频和视觉模态可以为定位和跟踪提供补充信息。贝叶斯滤波器利用音频和视频信息，可以解决数据关联、音频和视频融合以及航迹管理等问题。在本文中，我们进行了全面的概述视听说话人跟踪。据我们所知，这是过去五年来首次进行的广泛调查。我们介绍了家庭的贝叶斯滤波器，并总结了获得视听测量的方法。此外，总结了现有的跟踪器及其在AV16.3数据集上的性能。在过去的几年里，深度学习技术蓬勃发展，这也促进了视听扬声器跟踪的发展。还讨论了深度学习技术在测量提取和状态估计方面的影响。最后，我们讨论了视听说话人跟踪与语音分离、分布式说话人跟踪等领域的联系。摘要：Audio-visual speaker tracking has drawn increasing attention over the past few years due to its academic values and wide application. Audio and visual modalities can provide complementary information for localization and tracking. With audio and visual information, the Bayesian-based filter can solve the problem of data association, audio-visual fusion and track management. In this paper, we conduct a comprehensive overview of audio-visual speaker tracking. To our knowledge, this is the first extensive survey over the past five years. We introduce the family of Bayesian filters and summarize the methods for obtaining audio-visual measurements. In addition, the existing trackers and their performance on AV16.3 dataset are summarized. In the past few years, deep learning techniques have thrived, which also boosts the development of audio visual speaker tracking. The influence of deep learning techniques in terms of measurement extraction and state estimation is also discussed. At last, we discuss the connections between audio-visual speaker tracking and other areas such as speech separation and distributed speaker tracking.

【9】 Acoustic BPE for Speech Generation with Discrete Tokens标题：用于离散令牌语音生成的声学BPE链接：https://arxiv.org/abs/2310.14580

作者：Feiyu Shen,Yiwei Guo,Chenpeng Du,Xie Chen,Kai Yu备注：5 pages, 2 figures; submitted to ICASSP 2024摘要：从自监督学习模型中得到的离散音频标记在语音生成中得到了广泛的使用。然而，由于令牌序列的长度，直接利用音频令牌的当前实践对序列建模提出了挑战。此外，这种方法给模型带来了在令牌之间建立相关性的负担，进一步使建模过程复杂化。为了解决这个问题，我们提出了声学BPE编码频繁的音频令牌模式，利用字节对编码。声学BPE有效地减少了序列长度，并利用了令牌序列中存在的先验形态信息，这消除了令牌相关性建模的挑战。通过对声学BPE训练的语音语言模型的全面调查，我们确认了它提供的显着优势，包括更快的推理和改进的语法捕获能力。此外，我们提出了一种新的rescore方法来选择最佳的合成语音中产生的丰富多样性的TTS系统的多个候选人。实验证明，rescore选择与人类的喜好，这突出了声学BPE的潜力，其他语音生成任务。摘要：Discrete audio tokens derived from self-supervised learning models have gained widespread usage in speech generation. However, current practice of directly utilizing audio tokens poses challenges for sequence modeling due to the length of the token sequence. Additionally, this approach places the burden on the model to establish correlations between tokens, further complicating the modeling process. To address this issue, we propose acoustic BPE which encodes frequent audio token patterns by utilizing byte-pair encoding. Acoustic BPE effectively reduces the sequence length and leverages the prior morphological information present in token sequence, which alleviates the modeling challenges of token correlation. Through comprehensive investigations on a speech language model trained with acoustic BPE, we confirm the notable advantages it offers, including faster inference and improved syntax capturing capabilities. In addition, we propose a novel rescore method to select the optimal synthetic speech among multiple candidates generated by rich-diversity TTS system. Experiments prove that rescore selection aligns closely with human preference, which highlights acoustic BPE's potential to other speech generation tasks.

【10】 Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation标题：基于学习音文跨模式语境表征的对话式语音识别链接：https://arxiv.org/abs/2310.14278

作者：Kun Wei,Bei Li,Hang Lv,Quan Lu,Ning Jiang,Lei Xie备注：Submitted to TASLP摘要：会话环境中的自动语音识别（ASR）提出了独特的挑战，包括从先前的会话回合中提取相关的上下文信息。由于不相关的内容，错误传播和冗余，现有的方法难以提取更长和更有效的上下文。为了解决这个问题，我们引入了一种新的会话ASR系统，扩展了Conformer编码器-解码器模型与跨模态会话表示。我们的方法利用了一个跨模态提取器，它通过一个专门的编码器和一个模态级掩码输入将预训练的语音和文本模型结合起来。这使得能够在没有显式错误传播的情况下提取更丰富的历史语音上下文。我们还将有条件的潜在变分模块学习会话级属性，如角色偏好和主题连贯性。通过在解码器中引入跨模态和会话表示，我们的模型在没有信息丢失的情况下保留了较长句子的上下文，与标准的Conformer模型相比，在普通话会话数据集HKUST和MagicData-RAMC上分别实现了8.8%和23%的相对准确度提高。摘要：Automatic Speech Recognition (ASR) in conversational settings presents unique challenges, including extracting relevant contextual information from previous conversational turns. Due to irrelevant content, error propagation, and redundancy, existing methods struggle to extract longer and more effective contexts. To address this issue, we introduce a novel Conversational ASR system, extending the Conformer encoder-decoder model with cross-modal conversational representation. Our approach leverages a cross-modal extractor that combines pre-trained speech and text models through a specialized encoder and a modal-level mask input. This enables the extraction of richer historical speech context without explicit error propagation. We also incorporate conditional latent variational modules to learn conversational level attributes such as role preference and topic coherence. By introducing both cross-modal and conversational representations into the decoder, our model retains context over longer sentences without information loss, achieving relative accuracy improvements of 8.8% and 23% on Mandarin conversation datasets HKUST and MagicData-RAMC, respectively, compared to the standard Conformer model.

【11】 First-Shot Unsupervised Anomalous Sound Detection With Unknown Anomalies Estimated by Metadata-Assisted Audio Generation标题：元数据辅助音频生成估计未知异常的首发非监督异常声检测链接：https://arxiv.org/abs/2310.14173

作者：Hejing Zhang,Qiaoxi Zhu,Jian Guan,Haohe Liu,Feiyang Xiao,Jiantong Tian,Xinhao Mei,Xubo Liu,Wenwu Wang备注：Submitted to ICASSP 2024摘要：首次（FS）无监督异常声音检测（ASD）是DCASE 2023挑战任务2中引入的一项全新任务，其中目标机器类型的异常声音在训练中不可见。现有的方法通常依赖于来自目标机器的正常和异常声音数据的可用性。然而，由于缺乏异常的声音数据的目标机器类型，它变得具有挑战性时，适应现有的ASD方法的第一次拍摄任务。在本文中，我们提出了一个新的框架，用于第一次无监督ASD，其中元数据辅助音频生成用于估计未知异常，通过利用可用的机器信息（即，元数据和声音数据）来微调用于生成异常声音的文本到音频生成模型，所述异常声音包含说明每种不同机器类型的唯一声学特性。然后，我们使用的方法，时间加权频域音频表示与高斯混合模型（TWFR-GMM）作为骨干，以实现第一枪无监督ASD。我们提出的FS-TWFR-GMM方法在DCASE 2023挑战任务2中的顶级系统中实现了具有竞争力的性能，同时只需要1%的模型参数进行检测，正如我们的实验所验证的那样。摘要：First-shot (FS) unsupervised anomalous sound detection (ASD) is a brand-new task introduced in DCASE 2023 Challenge Task 2, where the anomalous sounds for the target machine types are unseen in training. Existing methods often rely on the availability of normal and abnormal sound data from the target machines. However, due to the lack of anomalous sound data for the target machine types, it becomes challenging when adapting the existing ASD methods to the first-shot task. In this paper, we propose a new framework for the first-shot unsupervised ASD, where metadata-assisted audio generation is used to estimate unknown anomalies, by utilising the available machine information (i.e., metadata and sound data) to fine-tune a text-to-audio generation model for generating the anomalous sounds that contain unique acoustic characteristics accounting for each different machine types. We then use the method of Time-Weighted Frequency domain audio Representation with Gaussian Mixture Model (TWFR-GMM) as the backbone to achieve the first-shot unsupervised ASD. Our proposed FS-TWFR-GMM method achieves competitive performance amongst top systems in DCASE 2023 Challenge Task 2, while requiring only 1% model parameters for detection, as validated in our experiments.

【12】 Composer Style-specific Symbolic Music Generation Using Vector Quantized Discrete Diffusion Models标题：基于矢量量化离散扩散模型的作曲家风格符号音乐生成链接：https://arxiv.org/abs/2310.14044

作者：Jincheng Zhang,Jingjing Tang,Charalampos Saitis,György Fazekas摘要：新兴的去噪扩散概率模型（DDPM）已经越来越多地使用，因为他们已经取得了可喜的成果，在不同的生成任务与连续的数据，如图像和声音合成。尽管如此，扩散模型的成功还没有完全扩展到离散的符号音乐。我们建议结合矢量量化变分自编码器（VQ-VAE）和离散扩散模型生成所需的作曲家风格的符号音乐。训练的VQ-VAE可以将符号音乐表示为与学习的码本中的特定条目相对应的索引序列。随后，一个离散的扩散模型被用来模拟VQ-VAE的离散的潜在空间。扩散模型被训练以生成由码本索引组成的中间音乐序列，然后使用VQ-VAE的解码器将其解码为符号音乐。实验结果表明，该模型可以生成符合给定条件的目标作曲家风格的符号音乐，准确率高达72.36%。摘要：Emerging Denoising Diffusion Probabilistic Models (DDPM) have become increasingly utilised because of promising results they have achieved in diverse generative tasks with continuous data, such as image and sound synthesis. Nonetheless, the success of diffusion models has not been fully extended to discrete symbolic music. We propose to combine a vector quantized variational autoencoder (VQ-VAE) and discrete diffusion models for the generation of symbolic music with desired composer styles. The trained VQ-VAE can represent symbolic music as a sequence of indexes that correspond to specific entries in a learned codebook. Subsequently, a discrete diffusion model is used to model the VQ-VAE's discrete latent space. The diffusion model is trained to generate intermediate music sequences consisting of codebook indexes, which are then decoded to symbolic music using the VQ-VAE's decoder. The results demonstrate our model can generate symbolic music with target composer styles that meet the given conditions with a high accuracy of 72.36%.

【13】 Fast Diffusion GAN Model for Symbolic Music Generation Controlled by Emotions标题：情绪控制的符号音乐生成的快速扩散GAN模型链接：https://arxiv.org/abs/2310.14040

作者：Jincheng Zhang,György Fazekas,Charalampos Saitis摘要：扩散模型已经显示出有前途的结果，为广泛的生成任务与连续数据，如图像和音频合成。然而，几乎没有取得进展，使用扩散模型来生成离散的符号音乐，因为这类新的生成模型不太适合离散数据，而其迭代采样过程是计算昂贵的。在这项工作中，我们提出了一个与生成对抗网络相结合的扩散模型，旨在（i）减轻算法音乐生成中剩余的挑战之一，即对目标情感的生成控制，以及（ii）减轻扩散模型应用于符号音乐生成的缓慢采样缺点。我们首先使用一个经过训练的变分自动编码器来获取带有情感标签的符号音乐数据集的嵌入，然后使用这些嵌入来训练扩散模型。我们的研究结果表明，成功地控制我们的扩散模型，以产生具有所需情感的象征性音乐。我们的模型实现了几个数量级的计算成本的改善，只需要四个时间步骤去噪，而目前的国家的最先进的扩散模型所需的步骤，符号音乐生成是在数千的顺序。摘要：Diffusion models have shown promising results for a wide range of generative tasks with continuous data, such as image and audio synthesis. However, little progress has been made on using diffusion models to generate discrete symbolic music because this new class of generative models are not well suited for discrete data while its iterative sampling process is computationally expensive. In this work, we propose a diffusion model combined with a Generative Adversarial Network, aiming to (i) alleviate one of the remaining challenges in algorithmic music generation which is the control of generation towards a target emotion, and (ii) mitigate the slow sampling drawback of diffusion models applied to symbolic music generation. We first used a trained Variational Autoencoder to obtain embeddings of a symbolic music dataset with emotion labels and then used those to train a diffusion model. Our results demonstrate the successful control of our diffusion model to generate symbolic music with a desired emotion. Our model achieves several orders of magnitude improvement in computational cost, requiring merely four time steps to denoise while the steps required by current state-of-the-art diffusion models for symbolic music generation is in the order of thousands.

【14】 Temporal convolutional neural networks to generate a head-related impulse response from one direction to another标题：用于产生从一个方向到另一个方向的头部相关脉冲响应的时间卷积神经网络链接：https://arxiv.org/abs/2310.14018

作者：Tatsuki Kobayashi,Yoshiko Maruyama,Isao Nambu,Shohei Yano,Yasuhiro Wada摘要：虚拟声音合成是一种允许用户通过头戴式耳机或耳塞感知空间声音的技术。然而，准确的虚拟声音需要单独的头部相关传递函数（HRTF），由于需要专门的环境，这可能难以测量。在这项研究中，我们提出了一种方法来产生HRTF从一个方向到另一个。为此，我们使用时间卷积神经网络（TCN）来生成头部相关脉冲响应（HRIR）。为了训练TCN，使用了水平面中公开可用的数据集。使用经过训练的网络，我们成功地为数据集中的前向以外的方向生成了HRIR。我们发现，所提出的方法成功地生成HRIR公开可用的数据集。为了测试该方法的泛化能力，我们测量了一个新数据集的HRIR，并测试了训练好的网络是否可以用于这个新数据集。虽然光谱失真评估的相似性略有下降，与人类参与者的行为实验表明，所产生的HRIR相当于测量的。这些结果表明，所提出的TCN可以用于从一个方向到另一个方向生成个性化HRIR，这可能有助于虚拟声音的个性化。摘要：Virtual sound synthesis is a technology that allows users to perceive spatial sound through headphones or earphones. However, accurate virtual sound requires an individual head-related transfer function (HRTF), which can be difficult to measure due to the need for a specialized environment. In this study, we proposed a method to generate HRTFs from one direction to the other. To this end, we used temporal convolutional neural networks (TCNs) to generate head-related impulse responses (HRIRs). To train the TCNs, publicly available datasets in the horizontal plane were used. Using the trained networks, we successfully generated HRIRs for directions other than the front direction in the dataset. We found that the proposed method successfully generated HRIRs for publicly available datasets. To test the generalization of the method, we measured the HRIRs of a new dataset and tested whether the trained networks could be used for this new dataset. Although the similarity evaluated by spectral distortion was slightly degraded, behavioral experiments with human participants showed that the generated HRIRs were equivalent to the measured ones. These results suggest that the proposed TCNs can be used to generate personalized HRIRs from one direction to another, which could contribute to the personalization of virtual sound.

【15】 Automatic Pronunciation Assessment -- A Review标题：语音自动测评--综述链接：https://arxiv.org/abs/2310.13974

作者：Yassine El Kheir,Ahmed Ali,Shammur Absar Chowdhury备注：9 pages, accepted to EMNLP Findings摘要：近年来，语音评估及其在计算机辅助语音训练中的应用取得了令人瞩目的进展。随着过去几年语言处理和深度学习的快速发展，有必要进行更新审查。在本文中，我们回顾了语音评估的音素和韵律的方法。我们对突出的研究趋势中观察到的主要挑战进行了分类，并强调了现有的限制和可用的资源。随后讨论了剩余的挑战和今后工作的可能方向。摘要：Pronunciation assessment and its application in computer-aided pronunciation training (CAPT) have seen impressive progress in recent years. With the rapid growth in language processing and deep learning over the past few years, there is a need for an updated review. In this paper, we review methods employed in pronunciation assessment for both phonemic and prosodic. We categorize the main challenges observed in prominent research trends, and highlight existing limitations, and available resources. This is followed by a discussion of the remaining challenges and possible directions for future work.

【16】 Yet Another Model for Arabic Dialect Identification标题：另一种阿拉伯方言识别模型链接：https://arxiv.org/abs/2310.13812

作者：Ajinkya Kulkarni,Hanan Aldarmaki备注：ACCEPTED AT ArabicNLP 2023摘要：在本文中，我们描述了一个阿拉伯语口语方言识别（ADI）模型，该模型在两个基准数据集ADI-5和ADI-17上的性能始终优于先前发表的结果。我们探索了两种架构变体：ResNet和ECAPA-TDNN，以及两种类型的声学特征：MFCC和从预训练的自监督模型UniSpeech-SAT Large中提取的特征，以及所有四种变体的融合。我们发现，ECAPA-TDNN网络的性能优于ResNet，具有UniSpeech-SAT特征的模型的性能大大优于具有MFCC的模型。此外，所有四种变体的融合始终优于单个模型。我们的最佳模型在这两个数据集上的表现优于先前报道的结果，在ADI-5和ADI-17上的准确率分别为84.7%和96.9%。摘要：In this paper, we describe a spoken Arabic dialect identification (ADI) model for Arabic that consistently outperforms previously published results on two benchmark datasets: ADI-5 and ADI-17. We explore two architectural variations: ResNet and ECAPA-TDNN, coupled with two types of acoustic features: MFCCs and features exratected from the pre-trained self-supervised model UniSpeech-SAT Large, as well as a fusion of all four variants. We find that individually, ECAPA-TDNN network outperforms ResNet, and models with UniSpeech-SAT features outperform models with MFCCs by a large margin. Furthermore, a fusion of all four variants consistently outperforms individual models. Our best models outperform previously reported results on both datasets, with accuracies of 84.7% and 96.9% on ADI-5 and ADI-17, respectively.

【17】 Multi-label Open-set Audio Classification标题：多标签开放集音频分类链接：https://arxiv.org/abs/2310.13759

作者：Sripathi Sridhar,Mark Cartwright备注：Published at the Workshop on Detection and Classification of Acoustic Scenes and Events, 2023 (DCASE 2023)摘要：当前的音频分类模型相对于现实世界中感兴趣的大量声音事件类别具有小的类别词汇表。因此，它们提供的世界视图有限，可能会错过重要但意外或未知的声音事件。为了解决这个问题，已经开发了开集音频分类技术来检测来自未知类别的声音事件。虽然这些方法已经应用于音频中的多类别上下文，例如声音场景分类，但是它们还没有被研究用于其中声音事件重叠的复调音频，这需要使用多标签模型。在这项研究中，我们通过创建一个具有不同未知类分布的数据集并评估基于现有技术的基线方法来建立多标签开集音频分类问题。摘要：Current audio classification models have small class vocabularies relative to the large number of sound event classes of interest in the real world. Thus, they provide a limited view of the world that may miss important yet unexpected or unknown sound events. To address this issue, open-set audio classification techniques have been developed to detect sound events from unknown classes. Although these methods have been applied to a multi-class context in audio, such as sound scene classification, they have yet to be investigated for polyphonic audio in which sound events overlap, requiring the use of multi-label models. In this study, we establish the problem of multi-label open-set audio classification by creating a dataset with varying unknown class distributions and evaluating baseline approaches built upon existing techniques.

【18】 An overview of text-to-speech systems and media applications标题：文本到语音转换系统和媒体应用概述链接：https://arxiv.org/abs/2310.14301

作者：Mohammad Reza Hasanabadi备注：Accepted in ABU Technical Review journal 2023/6摘要：产生类似于人类声音的合成语音是现代交互式媒体系统的新兴事物。文本到语音（TTS）系统试图通过文本输入生成合成和真实的声音。此外，众所周知的和熟悉的配音，宣布和叙述的声音，作为任何媒体机构的宝贵财产，可以永远保留通过使用TTS和语音转换（VC）算法。深度学习方法的出现使这种TTS系统更加准确和可访问。为了更好地理解TTS系统，本文研究了这种系统的关键组成部分，包括文本分析，声学建模和语音编码。然后，本文详细介绍了基于深度学习的重要的最先进的TTS系统。最后，比较了最近发布的系统在骨干结构，输入和转换的类型，声码器和主观评价（MOS）。因此，Tacotron 2，Transformer TTS，WaveNet和FastSpeech 1是有史以来最成功的TTS系统之一。在讨论部分，提出了一些建议，开发一个TTS系统方面的预期应用。摘要：Producing synthetic voice, similar to human-like sound, is an emerging novelty of modern interactive media systems. Text-To-Speech (TTS) systems try to generate synthetic and authentic voices via text input. Besides, well known and familiar dubbing, announcing and narrating voices, as valuable possessions of any media organization, can be kept forever by utilizing TTS and Voice Conversion (VC) algorithms . The emergence of deep learning approaches has made such TTS systems more accurate and accessible. To understand TTS systems better, this paper investigates the key components of such systems including text analysis, acoustic modelling and vocoding. The paper then provides details of important state-of-the-art TTS systems based on deep learning. Finally, a comparison is made between recently released systems in term of backbone architecture, type of input and conversion, vocoder used and subjective assessment (MOS). Accordingly, Tacotron 2, Transformer TTS, WaveNet and FastSpeech 1 are among the most successful TTS systems ever released. In the discussion section, some suggestions are made to develop a TTS system with regard to the intended application.

【19】 MFCC-GAN Codec: A New AI-based Audio Coding标题：MFCC-GAN编解码器：一种新的基于人工智能的音频编码链接：https://arxiv.org/abs/2310.14300

作者：Mohammad Reza Hasanabadi备注：Accepted in ABU Technical Review journal 2023/3摘要：在本文中，我们提出了在对抗环境中使用MFCC特征的基于AI的音频编码。我们将传统编码器与对抗学习解码器相结合，以更好地重建原始波形。由于GAN给出了隐式密度估计，因此，这样的模型不太容易过拟合。我们将我们的工作与五种知名的编解码器进行了比较，即AAC，AC 3，Opus，Vorbis和Speex，其比特率从2kbps到128 kbps。MFCCGAN_36k在SNR方面达到了最先进的结果，尽管与AC3_128k、AAC_112k、Vorbis_48k、Opus_48k和Speex_48K相比，其比特率较低。另一方面，MFCCGAN_13k还实现了与AC3_128k和AAC_112k相等的高SNR=27，同时具有显著更低的比特率（13 kbps）。与AAC_48k相比，MFCCGAN_36k实现了更高的NISQA-MOS结果，同时具有低20%的比特率。此外，MFCCGAN_13k获得了NISQAMOS= 3.9，其远高于AAC_24k、AAC_32k、AC3_32k和AAC_48k。对于未来的工作，我们最后建议采用损失函数优化可懂度和感知指标的MFCCGAN结构，以提高质量和可懂度同时。摘要：In this paper, we proposed AI-based audio coding using MFCC features in an adversarial setting. We combined a conventional encoder with an adversarial learning decoder to better reconstruct the original waveform. Since GAN gives implicit density estimation, therefore, such models are less prone to overfitting. We compared our work with five well-known codecs namely AAC, AC3, Opus, Vorbis, and Speex, performing on bitrates from 2kbps to 128kbps. MFCCGAN_36k achieved the state-of-the-art result in terms of SNR despite a lower bitrate in comparison to AC3_128k, AAC_112k, Vorbis_48k, Opus_48k, and Speex_48K. On the other hand, MFCCGAN_13k also achieved high SNR=27 which is equal to that of AC3_128k, and AAC_112k while having a significantly lower bitrate (13 kbps). MFCCGAN_36k achieved higher NISQA-MOS results compared to AAC_48k while having a 20% lower bitrate. Furthermore, MFCCGAN_13k obtained NISQAMOS= 3.9 which is much higher than AAC_24k, AAC_32k, AC3_32k, and AAC_48k. For future work, we finally suggest adopting loss functions optimizing intelligibility and perceptual metrics in the MFCCGAN structure to improve quality and intelligibility simultaneously.

【20】 Diffusion-Based Adversarial Purification for Speaker Verification标题：基于扩散的对抗性净化说话人确认链接：https://arxiv.org/abs/2310.14270

作者：Yibo Bai,Xiao-Lei Zhang摘要：近年来，基于深度学习的自动说话人确认（ASV）容易受到对抗性攻击的污染，对抗性攻击是一种新型的攻击方式，通过对音频信号注入不可感知的扰动，使ASV产生错误的决策。这对ASV系统的安全性和可靠性构成了重大威胁。为了解决这个问题，我们提出了一种基于扩散的对抗性净化（DAP）方法，该方法增强了ASV系统对这种对抗性攻击的鲁棒性。我们的方法利用条件去噪扩散概率模型来有效地净化对抗性示例并减轻扰动的影响。DAP首先将受控噪声引入对抗性示例，然后执行反向去噪过程以重建干净的音频。实验结果表明，提出的DAP在提高ASV的安全性，同时最大限度地减少净化音频信号的失真的有效性。摘要：Recently, automatic speaker verification (ASV) based on deep learning is easily contaminated by adversarial attacks, which is a new type of attack that injects imperceptible perturbations to audio signals so as to make ASV produce wrong decisions. This poses a significant threat to the security and reliability of ASV systems. To address this issue, we propose a Diffusion-Based Adversarial Purification (DAP) method that enhances the robustness of ASV systems against such adversarial attacks. Our method leverages a conditional denoising diffusion probabilistic model to effectively purify the adversarial examples and mitigate the impact of perturbations. DAP first introduces controlled noise into adversarial examples, and then performs a reverse denoising process to reconstruct clean audio. Experimental results demonstrate the efficacy of the proposed DAP in enhancing the security of ASV and meanwhile minimizing the distortion of the purified audio signals.

eess.AS音频处理

【1】 Prompt-driven Target Speech Diarization标题：语音驱动的目标语音日记链接：https://arxiv.org/abs/2310.14823

作者：Yidi Jiang,Zhengyang Chen,Ruijie Tao,Liqun Deng,Yanmin Qian,Haizhou Li备注：submitted to ICASSP 2024摘要：我们介绍了一种新的任务命名为“目标语音diarization”，旨在确定“当目标事件发生”的音频信号。我们设计了一个神经架构，称为神经元驱动的目标语音日记（PTSD），它与不同的提示，指定感兴趣的目标语音事件。我们训练和评估PTSD使用sim2spk，sim3spk和sim4spk数据集，这是来自Librispeech。我们表明，该框架准确地定位目标语音事件。此外，我们的框架表现出多功能性，通过其令人印象深刻的表现在三个diarization相关的任务：目标说话人的语音活动检测，重叠语音检测和性别diarization。特别是，PTSD在真实和模拟数据上的这些任务中达到了与专业模型相当的性能。这项工作作为一个参考基准，并提供了有价值的见解，语音驱动的目标语音处理。摘要：We introduce a novel task named `target speech diarization', which seeks to determine `when target event occurred' within an audio signal. We devise a neural architecture called Prompt-driven Target Speech Diarization (PTSD), that works with diverse prompts that specify the target speech events of interest. We train and evaluate PTSD using sim2spk, sim3spk and sim4spk datasets, which are derived from the Librispeech. We show that the proposed framework accurately localizes target speech events. Furthermore, our framework exhibits versatility through its impressive performance in three diarization-related tasks: target speaker voice activity detection, overlapped speech detection and gender diarization. In particular, PTSD achieves comparable performance to specialized models across these tasks on both real and simulated data. This work serves as a reference benchmark and provides valuable insights into prompt-driven target speech processing.

【2】 DPP-TTS: Diversifying prosodic features of speech via determinantal point processes标题：DPP-TTS：通过行列式点过程实现语音韵律特征的多样化链接：https://arxiv.org/abs/2310.14663

作者：Seongho Joo,Hyukhun Koh,Kyomin Jung备注：EMNLP 2023摘要：随着深度生成模型的快速发展，最近的神经文本到语音（TTS）模型已经成功地合成了类似人类的语音。除了单调的韵律模式之外，已经有一些努力来生成具有各种韵律的语音。然而，以前的作品有几个局限性。首先，典型的TTS模型依赖于缩放的采样温度来提高韵律的多样性。在高采样温度下生成的语音样本通常缺乏感知韵律多样性，这会对语音的自然度产生不利影响。第二，样本之间的多样性被忽略，因为采样过程往往集中在单个语音样本，而不是多个。在本文中，我们提出了DPP-TTS：一个文本到语音模型的基础上确定性点过程（DPPs）的韵律多样化模块。我们的TTS模型是能够产生的语音样本，同时考虑在每个样本和多个样本之间的感知多样性。我们证明，DPP-TTS生成的语音样本更多样化的韵律比基线在并排比较测试，同时考虑到语音的自然性。摘要：With the rapid advancement in deep generative models, recent neural Text-To-Speech(TTS) models have succeeded in synthesizing human-like speech. There have been some efforts to generate speech with various prosody beyond monotonous prosody patterns. However, previous works have several limitations. First, typical TTS models depend on the scaled sampling temperature for boosting the diversity of prosody. Speech samples generated at high sampling temperatures often lack perceptual prosodic diversity, which can adversely affect the naturalness of the speech. Second, the diversity among samples is neglected since the sampling procedure often focuses on a single speech sample rather than multiple ones. In this paper, we propose DPP-TTS: a text-to-speech model based on Determinantal Point Processes (DPPs) with a prosody diversifying module. Our TTS model is capable of generating speech samples that simultaneously consider perceptual diversity in each sample and among multiple samples. We demonstrate that DPP-TTS generates speech samples with more diversified prosody than baselines in the side-by-side comparison test considering the naturalness of speech at the same time.

【3】 An overview of text-to-speech systems and media applications标题：文本到语音转换系统和媒体应用概述链接：https://arxiv.org/abs/2310.14301

【4】 MFCC-GAN Codec: A New AI-based Audio Coding标题：MFCC-GAN编解码器：一种新的基于人工智能的音频编码链接：https://arxiv.org/abs/2310.14300

【5】 Diffusion-Based Adversarial Purification for Speaker Verification标题：基于扩散的对抗性净化说话人确认链接：https://arxiv.org/abs/2310.14270

【6】 A Study on Prosodic Entrainment in Relation to Therapist Empathy in Counseling Conversation标题：心理咨询对话中韵律训练与治疗师共情的关系研究链接：https://arxiv.org/abs/2310.14181

作者：Dehua Tao,Tan Lee,Harold Chui,Sarah Luk备注：Accepted by INTERSPEECH 2023摘要：咨询是以治疗师和客户之间的口头对话进行的。治疗师表达的共情水平被认为是咨询质量的重要指标，通常由观察者或客户评估。本研究探讨言语韵律对主观移情的影响。实验结果表明，在来访者与治疗师的互动中，强度的夹带比音高或语速的夹带对共情观察的影响更大。观察者和来访者对治疗师共情有不同的感知，但在音高和强度上有相同的夹带现象。来访者对言语的音高变化和强度的调整意图被认为是来访者对咨询质量感知的一个指标。摘要：Counseling is carried out as spoken conversation between a therapist and a client. The empathy level expressed by the therapist is considered an important index of the quality of counseling and often assessed by an observer or the client. This research investigates the entrainment of speech prosody in relation to subjectively rated empathy. Experimental results show that the entrainment of intensity is more influential to empathy observation than that of pitch or speech rate in client-therapist interaction. The observer and the client have different perceptions of therapist empathy with the same entrained phenomena in pitch and intensity. The client's intention to make adjustment on pitch variation and intensity of speech is considered an indicator of the client's perception of counseling quality.

【7】 Modeling Intrapersonal and Interpersonal Influences for Automatic Estimation of Therapist Empathy in Counseling Conversation标题：在咨询谈话中为治疗师移情自动估计建模的个人内和人际影响链接：https://arxiv.org/abs/2310.14178

作者：Dehua Tao,Tan Lee,Harold Chui,Sarah Luk摘要：咨询通常通过治疗师和客户之间的口头对话进行。治疗师的同理心水平是治疗效果的一个关键指标。假设治疗师的共情表达是由他们过去的行为和他们对客户的行为的看法，我们提出了一个模型来估计治疗师共情，同时考虑个人和人际的影响。这些动态的影响被捕获通过应用注意机制治疗师转向和治疗师和客户的历史转折。我们的研究结果表明，整合的动态影响，提高共情水平的估计。影响衍生嵌入应构成一个小部分，在目标轮表示最佳的移情估计。客户的回合（人际影响）似乎略超过治疗师自己的回合（自我影响）在共情估计的有效性。值得注意的是，专注于最近的历史转折可以显着影响治疗师共情的估计。摘要：Counseling is usually conducted through spoken conversation between a therapist and a client. The empathy level of therapist is a key indicator of outcomes. Presuming that therapist's empathy expression is shaped by their past behavior and their perception of the client's behavior, we propose a model to estimate the therapist empathy by considering both intrapersonal and interpersonal influences. These dynamic influences are captured by applying an attention mechanism to the therapist turn and the historical turns of both therapist and client. Our findings suggest that the integration of dynamic influences enhances empathy level estimation. The influence-derived embedding should constitute a minor portion in the target turn representation for optimal empathy estimation. The client's turns (interpersonal influence) appear to slightly surpass the therapist's own turns (intrapersonal influence) in empathy estimation effectiveness. It is noted that concentrating exclusively on recent historical turns can significantly impact the estimation of therapist empathy.

【8】 SwG-former: Sliding-window Graph Convolutional Network Integrated with Conformer for Sound Event Localization and Detection标题：SWG-FORFER：滑动窗图卷积网络与整形器相结合的声事件定位与检测链接：https://arxiv.org/abs/2310.14016

作者：Weiming Huang,Qinghua Huang,Liyan Ma,Zhengyu Chen,Chuan Wang摘要：声事件定位与检测（SELD）是声事件检测（SED）和波达方向（DoA）估计的联合任务。SED主要依赖于时间相关性来区分不同的声音类别，而DoA估计依赖于空间相关性来估计源方向。为了联合优化两个子任务，SELD系统应该同时提取空间相关性和建模时间依赖性。然而，许多模型主要提取空间相关性和模型的时间依赖性分开。本文利用音频信号中时空信息的相互依赖性进行同步提取，以提高模型的性能。作为响应，一种新的图表示利用图卷积网络（GCN）在非欧空间开发的时空信息提取并发。在此基础上，设计了滑动窗口图（SwG）模块。它利用不同大小的滑动窗口来学习时间上下文信息，并在频率信道（F-C）域中动态地构造图顶点来捕获空间相关性。此外，作为消息传递的基石，一个强大的Conv 2dAgg函数被提出并嵌入到SwG模块中，以聚合相邻顶点的特征。为了提高SELD在自然空间声环境中的性能，本文提出了一种通用的、高效的SWG-former模型。它表现出优越的性能相比，最近先进的SELD模型。为了进一步验证SwG-former的通用性和效率，它被无缝集成到称为SwG-EINV 2的事件无关网络版本2（EINV 2）中。在相同的声学环境下，SwG-EINV 2超越了最先进的（SOTA）方法。摘要：Sound event localization and detection (SELD) is a joint task of sound event detection (SED) and direction of arrival (DoA) estimation. SED mainly relies on temporal dependencies to distinguish different sound classes, while DoA estimation depends on spatial correlations to estimate source directions. To jointly optimize two subtasks, the SELD system should extract spatial correlations and model temporal dependencies simultaneously. However, numerous models mainly extract spatial correlations and model temporal dependencies separately. In this paper, the interdependence of spatial-temporal information in audio signals is exploited for simultaneous extraction to enhance the model performance. In response, a novel graph representation leveraging graph convolutional network (GCN) in non-Euclidean space is developed to extract spatial-temporal information concurrently. A sliding-window graph (SwG) module is designed based on the graph representation. It exploits sliding-windows with different sizes to learn temporal context information and dynamically constructs graph vertices in the frequency-channel (F-C) domain to capture spatial correlations. Furthermore, as the cornerstone of message passing, a robust Conv2dAgg function is proposed and embedded into the SwG module to aggregate the features of neighbor vertices. To improve the performance of SELD in a natural spatial acoustic environment, a general and efficient SwG-former model is proposed by integrating the SwG module with the Conformer. It exhibits superior performance in comparison to recent advanced SELD models. To further validate the generality and efficiency of the SwG-former, it is seamlessly integrated into the event-independent network version 2 (EINV2) called SwG-EINV2. The SwG-EINV2 surpasses the state-of-the-art (SOTA) methods under the same acoustic environment.

【9】 Novel-View Acoustic Synthesis from 3D Reconstructed Rooms标题：基于三维重建房间的新视图声合成链接：https://arxiv.org/abs/2310.15130

【10】 Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate标题：延迟存储单元：通过延迟门建立时间依赖模型链接：https://arxiv.org/abs/2310.14982

作者：Pengfei Sun,Jibin Wu,Malu Zhang,Paul Devos,Dick Botteldooren摘要：递归神经网络（RNN）以其在建模时间依赖性方面的能力而闻名，这一特性推动了它们在顺序数据处理中的广泛采用。然而，普通RNN面临着众所周知的梯度消失和爆炸问题，这对学习和建立长期依赖关系构成了重大挑战。此外，门控RNN往往是过度参数化的，导致网络泛化能力差。为了解决这些挑战，本文提出了一种新的延迟存储单元（DMU），其中引入了延迟线结构，与延迟门耦合，以促进时间交互和时间信用分配，从而增强香草RNN的时间建模能力。特别地，DMU被设计为直接将输入信息分配到未来的最佳时刻，而不是通过复杂的网络动态来聚合和重新分配。我们提出的DMU在广泛的顺序建模任务中表现出卓越的时间建模能力，在语音识别，雷达手势识别，ECG波形分割和排列顺序图像分类等应用中使用的参数比其他最先进的门控RNN模型少得多。摘要：Recurrent Neural Networks (RNNs) are renowned for their adeptness in modeling temporal dependencies, a trait that has driven their widespread adoption for sequential data processing. Nevertheless, vanilla RNNs are confronted with the well-known issue of gradient vanishing and exploding, posing a significant challenge for learning and establishing long-range dependencies. Additionally, gated RNNs tend to be over-parameterized, resulting in poor network generalization. To address these challenges, we propose a novel Delayed Memory Unit (DMU) in this paper, wherein a delay line structure, coupled with delay gates, is introduced to facilitate temporal interaction and temporal credit assignment, so as to enhance the temporal modeling capabilities of vanilla RNNs. Particularly, the DMU is designed to directly distribute the input information to the optimal time instant in the future, rather than aggregating and redistributing it over time through intricate network dynamics. Our proposed DMU demonstrates superior temporal modeling capabilities across a broad range of sequential modeling tasks, utilizing considerably fewer parameters than other state-of-the-art gated RNN models in applications such as speech recognition, radar gesture recognition, ECG waveform segmentation, and permuted sequential image classification.

【11】 Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognition标题：一种基于整形器的高效端到端语音识别关键帧机制链接：https://arxiv.org/abs/2310.14954

【12】 8+8=4: Formalizing Time Units to Handle Symbolic Music Durations标题：8+8=4：将时间单位正式化以处理符号音乐持续时间链接：https://arxiv.org/abs/2310.14952

【13】 Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model标题：使用单一训练模型的直观多语言视听语音识别链接：https://arxiv.org/abs/2310.14946

【14】 SpeakEasy: A Conversational Intelligence Chatbot for Enhancing College Students' Communication Skills标题：Speakeasy：一种提高大学生沟通能力的对话智能聊天机器人链接：https://arxiv.org/abs/2310.14891

【15】 Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translation标题：利用时间戳信息进行串行化联合流识别和翻译链接：https://arxiv.org/abs/2310.14806

【16】 A Novel Transfer Learning Method Utilizing Acoustic and Vibration Signals for Rotating Machinery Fault Diagnosis标题：一种基于声学和振动信号的旋转机械故障诊断传递学习新方法链接：https://arxiv.org/abs/2310.14796

【17】 Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directions标题：视听说话人跟踪：进展、挑战和未来方向链接：https://arxiv.org/abs/2310.14778

【18】 SPRING-INX: A Multilingual Indian Language Speech Corpus by SPRING Lab, IIT Madras标题：Spring-INX：IIT Madras春季实验室开发的多语种印度语言语音语料库链接：https://arxiv.org/abs/2310.14654

作者：Nithya R,Malavika S,Jordan F,Arjun Gangwar,Metilda N J,S Umesh,Rithik Sarab,Akhilesh Kumar Dubey,Govind Divakaran,Samudra Vijaya K,Suryakanth V Gangashetty备注：3 pages, About SPRING-INX Data摘要：印度拥有多种语言，其中22种语言被印度宪法承认为官方语言。由于数据有限以及需要容纳的语言和口音的数量，为印度人口构建基于语音的应用程序是一个困难的问题。为了鼓励语言技术社区以印度语言构建基于语音的应用程序，我们正在开源SPRING-INX数据，其中包含约2000小时的合法来源和手动转录的语音数据，用于构建阿萨姆语，孟加拉语，古吉拉特语，印地语，卡纳达语，马拉雅拉姆语，马拉地语，Odia，旁遮普语和泰米尔语的ASR系统。这项工作由印度马德拉斯理工学院的SPRING实验室负责，是印度政府电子和信息技术部（MeitY）资助的国家语言翻译任务（NLTM）的一部分。我们描述了数据收集和数据清洗过程以及数据统计。摘要：India is home to a multitude of languages of which 22 languages are recognised by the Indian Constitution as official. Building speech based applications for the Indian population is a difficult problem owing to limited data and the number of languages and accents to accommodate. To encourage the language technology community to build speech based applications in Indian languages, we are open sourcing SPRING-INX data which has about 2000 hours of legally sourced and manually transcribed speech data for ASR system building in Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi and Tamil. This endeavor is by SPRING Lab , Indian Institute of Technology Madras and is a part of National Language Translation Mission (NLTM), funded by the Indian Ministry of Electronics and Information Technology (MeitY), Government of India. We describe the data collection and data cleaning process along with the data statistics in this paper.

【19】 Acoustic BPE for Speech Generation with Discrete Tokens标题：用于离散令牌语音生成的声学BPE链接：https://arxiv.org/abs/2310.14580

【20】 Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation标题：基于学习音文跨模式语境表征的对话式语音识别链接：https://arxiv.org/abs/2310.14278

【21】 First-Shot Unsupervised Anomalous Sound Detection With Unknown Anomalies Estimated by Metadata-Assisted Audio Generation标题：元数据辅助音频生成估计未知异常的首发非监督异常声检测链接：https://arxiv.org/abs/2310.14173

【22】 Composer Style-specific Symbolic Music Generation Using Vector Quantized Discrete Diffusion Models标题：基于矢量量化离散扩散模型的作曲家风格符号音乐生成链接：https://arxiv.org/abs/2310.14044

【23】 Fast Diffusion GAN Model for Symbolic Music Generation Controlled by Emotions标题：情绪控制的符号音乐生成的快速扩散GAN模型链接：https://arxiv.org/abs/2310.14040

【24】 Temporal convolutional neural networks to generate a head-related impulse response from one direction to another标题：用于产生从一个方向到另一个方向的头部相关脉冲响应的时间卷积神经网络链接：https://arxiv.org/abs/2310.14018

【25】 Automatic Pronunciation Assessment -- A Review标题：语音自动测评--综述链接：https://arxiv.org/abs/2310.13974

【26】 Yet Another Model for Arabic Dialect Identification标题：另一种阿拉伯方言识别模型链接：https://arxiv.org/abs/2310.13812

【27】 Multi-label Open-set Audio Classification标题：多标签开放集音频分类链接：https://arxiv.org/abs/2310.13759

机器翻译，仅供参考

语音/音频处理学术速递[10.24]

评论 0