每日语音速递本文经arXiv每日学术速递授权转载
【1】 pTSE-T: Presentation Target Speaker Extraction using Unaligned Text Cues
链接：https://arxiv.org/abs/2411.03109
摘要：TSE旨在从音频混合中提取目标说话者的清晰语音，从而消除不相关的背景噪音和语音。虽然先前的工作已经探索了各种辅助线索，包括预先录制的语音、视觉信息（例如，嘴唇运动和姿势）以及空间信息，但是在许多实际情况下，获取和选择这样的强线索是不可行的。与所有现有的工作，在本文中，我们的条件TSE算法从有限的和不对齐的文本内容，如浓缩点从演示幻灯片中提取的语义线索。这种方法在会议、海报会议或讲座演示等场景中特别有用，因为在这些场景中，实时获取其他线索具有挑战性。为此，我们设计了两个不同的网络。具体来说，我们提出的TPE融合音频功能与基于内容的语义线索，以方便时频掩码生成过滤掉外来噪声，而另一个建议，即TSR，采用对比学习技术，将盲分离的语音信号与语义线索。实验结果表明，该方法能够有效地利用有限文本和未对齐文本中的语义线索准确识别目标说话人，其SI-SDRi为12.16 dB，SDRi为12.66 dB，PESQi为0.830，STOIi为0.150。数据集和源代码将公开提供。项目演示页面：https://slideTSE.github.io/。
摘要：TSE aims to extract the clean speech of the target speaker in an audiomixture, thus eliminating irrelevant background noise and speech. While priorwork has explored various auxiliary cues including pre-recorded speech, visualinformation (e.g., lip motions and gestures), and spatial information, theacquisition and selection of such strong cues are infeasible in many practicalscenarios. Unlike all existing work, in this paper, we condition the TSEalgorithm on semantic cues extracted from limited and unaligned text content,such as condensed points from a presentation slide. This method is particularlyuseful in scenarios like meetings, poster sessions, or lecture presentations,where acquiring other cues in real-time is challenging. To this end, we designtwo different networks. Specifically, our proposed TPE fuses audio featureswith content-based semantic cues to facilitate time-frequency mask generationto filter out extraneous noise, while another proposal, namely TSR, employs thecontrastive learning technique to associate blindly separated speech signalswith semantic cues. The experimental results show the efficacy in accuratelyidentifying the target speaker by utilizing semantic cues derived from limitedand unaligned text, resulting in SI-SDRi of 12.16 dB, SDRi of 12.66 dB, PESQiof 0.830 and STOIi of 0.150, respectively. Dataset and source code will bepublicly available. Project demo page: https://slideTSE.github.io/.
标题：使用预训练前端进行语音分离以最大限度地减少域不匹配
链接：https://arxiv.org/abs/2411.03085
备注：IEEE/ACM Transactions on Audio, Speech, and Language Processing
摘要：语音分离试图从语音混合物中分离单独的语音信号。通常情况下，大多数分离模型都是在合成数据上训练的，因为在现实世界的鸡尾酒会场景中无法获得目标参考。因此，在实际应用中部署语音分离模型时，真实数据和合成数据之间存在域差距。在本文中，我们提出了一个自监督的域不变预训练（DIP）前端，它暴露于混合数据，而不需要目标参考语音。DIP前端利用具有两个创新借口任务的Siamese网络，混合预测编码（MPC）和混合不变编码（MIC），以捕获真实和合成未标记混合物之间的共享上下文线索。随后，当在合成数据上训练下游语音分离模型时，我们将DIP前端冻结为特征提取器。通过使用上下文线索预训练DIP前端，我们希望从合成数据中学习的语音分离技能可以有效地转移到真实数据中。为了受益于DIP前端，我们引入了一种新的分离管道来对齐分离模型的特征分辨率。我们在标准基准和真实世界数据集上评估语音分离质量。结果证实了我们的DIP前端优于现有的语音分离模型。这项研究强调了大规模预训练的潜力，以提高语音分离在现实世界中的应用的质量和可懂度。
摘要：Speech separation seeks to separate individual speech signals from a speechmixture. Typically, most separation models are trained on synthetic data due tothe unavailability of target reference in real-world cocktail party scenarios.As a result, there exists a domain gap between real and synthetic data whendeploying speech separation models in real-world applications. In this paper,we propose a self-supervised domain-invariant pretrained (DIP) frontend that isexposed to mixture data without the need for target reference speech. The DIPfrontend utilizes a Siamese network with two innovative pretext tasks, mixturepredictive coding (MPC) and mixture invariant coding (MIC), to capture sharedcontextual cues between real and synthetic unlabeled mixtures. Subsequently, wefreeze the DIP frontend as a feature extractor when training the downstreamspeech separation models on synthetic data. By pretraining the DIP frontendwith the contextual cues, we expect that the speech separation skills learnedfrom synthetic data can be effectively transferred to real data. To benefitfrom the DIP frontend, we introduce a novel separation pipeline to align thefeature resolution of the separation models. We evaluate the speech separationquality on standard benchmarks and real-world datasets. The results confirm thesuperiority of our DIP frontend over existing speech separation models. Thisstudy underscores the potential of large-scale pretraining to enhance thequality and intelligibility of speech separation in real-world applications.
标题：建筑工地工人安全的实时尖叫检测和位置估计
链接：https://arxiv.org/abs/2411.03016
备注：12 pages, 14 figures, 1 table, submitted to AIRISE conference
摘要：由于事故频发，建筑业面临着高风险，往往使工人处于危险的境地，快速反应至关重要。传统的安全监测方法，包括可穿戴传感器和GPS，通常在障碍物或室内条件下失效。本研究介绍一种新的实时尖叫声检测和定位系统，为建筑工地量身定制，特别是在低资源环境中。集成Wav 2 Vec 2和增强型ConvNet模型以实现准确的尖叫检测，再加上GCC-PHAT算法以实现混响条件下的鲁棒时间延迟估计，然后采用基于梯度下降的方法来实现噪声环境中的精确位置估计。我们的方法结合了这些概念，以实现高检测精度和快速定位，从而最大限度地减少误报和优化应急响应。初步结果表明，该系统不仅可以准确地检测建筑噪音中的求救信号，而且可以可靠地识别呼叫者的位置。该解决方案代表了工人安全的重大改善，具有在高风险职业环境中广泛应用的潜力。用于训练、尖叫检测评估、位置估计和集成框架的脚本将在https://github.com/Anmol2059/construction_safety上发布。
摘要：The construction industry faces high risks due to frequent accidents, oftenleaving workers in perilous situations where rapid response is critical.Traditional safety monitoring methods, including wearable sensors and GPS,often fail under obstructive or indoor conditions. This research introduces anovel real-time scream detection and localization system tailored forconstruction sites, especially in low-resource environments. IntegratingWav2Vec2 and Enhanced ConvNet models for accurate scream detection, coupledwith the GCC-PHAT algorithm for robust time delay estimation under reverberantconditions, followed by a gradient descent-based approach to achieve preciseposition estimation in noisy environments. Our approach combines these conceptsto achieve high detection accuracy and rapid localization, thereby minimizingfalse alarms and optimizing emergency response. Preliminary results demonstratethat the system not only accurately detects distress calls amidst constructionnoise but also reliably identifies the caller's location. This solutionrepresents a substantial improvement in worker safety, with the potential forwidespread application across high-risk occupational environments. The scriptsused for training, evaluation of scream detection, position estimation, andintegrated framework will be released at:https://github.com/Anmol2059/construction_safety.
标题：说话者情绪识别：利用自监督模型使用Wav2 Vec 2和HuBERT进行特征提取
链接：https://arxiv.org/abs/2411.02964
摘要：语言是人类表达自我最自然的方式。从语音中识别情感是一项重要的任务，因为情感本身的定义模糊不清。说话人情感识别是理解人类情感行为的关键。由于说话者的多样性、背景噪音、情绪的复杂性和说话风格，SER任务具有挑战性。它在教育，医疗保健，客户服务和人机交互（HCI）中有许多应用。以前，传统的机器学习方法，如SVM，HMM和KNN已被用于SER任务。近年来，深度学习方法变得越来越流行，卷积神经网络和递归神经网络被用于SER任务。这些方法的输入主要是频谱图和手工制作的特征。在这项工作中，我们研究了使用自我监督的基于变换的模型，Wav 2 Vec 2和HuBERT，从他们的声音来确定说话者的情绪。这些模型自动从原始音频信号中提取特征，然后用于分类任务。建议的解决方案在信誉良好的数据集上进行评估，包括RAVDESS，SHEMO，SAVEE，AESDD和ESD-DB。实验结果表明了该方法在不同数据集上的有效性。此外，该模型已被用于呼叫中心对话等现实世界的应用，结果表明，该模型准确地预测情绪。
摘要：Speech is the most natural way of expressing ourselves as humans. Identifyingemotion from speech is a nontrivial task due to the ambiguous definition ofemotion itself. Speaker Emotion Recognition (SER) is essential forunderstanding human emotional behavior. The SER task is challenging due to thevariety of speakers, background noise, complexity of emotions, and speakingstyles. It has many applications in education, healthcare, customer service,and Human-Computer Interaction (HCI). Previously, conventional machine learningmethods such as SVM, HMM, and KNN have been used for the SER task. In recentyears, deep learning methods have become popular, with convolutional neuralnetworks and recurrent neural networks being used for SER tasks. The input ofthese methods is mostly spectrograms and hand-crafted features. In this work,we study the use of self-supervised transformer-based models, Wav2Vec2 andHuBERT, to determine the emotion of speakers from their voice. The modelsautomatically extract features from raw audio signals, which are then used forthe classification task. The proposed solution is evaluated on reputabledatasets, including RAVDESS, SHEMO, SAVEE, AESDD, and Emo-DB. The results showthe effectiveness of the proposed method on different datasets. Moreover, themodel has been used for real-world applications like call center conversations,and the results demonstrate that the model accurately predicts emotions.
标题：连续视听声音分离
链接：https://arxiv.org/abs/2411.02860
备注：NeurIPS 2024
摘要：在本文中，我们介绍了一种新的连续的视听声音分离任务，目的是不断分离的声源为新的类，同时保留性能上以前学习的类，在视觉指导的帮助下。这个问题对于实际的视觉引导听觉感知至关重要，因为它可以显着增强视听声音分离模型的适应性和鲁棒性，使它们更适用于遇到新声源的现实世界场景。这项任务本身就具有挑战性，因为我们的模型不仅必须有效地利用当前任务中两种模态的信息，而且还必须在旧任务中保留它们的跨模态关联，以减轻视听持续学习过程中的灾难性遗忘。为了应对这些挑战，我们提出了一种新的方法，名为ContAV-Sep（\textbf{Cont}inual \textbf{A}udio-\textbf{V} issual Sound \textbf{Sep}aration）。ContAV-Sep提出了一种新的跨模态相似性蒸馏约束（CrossSDC），通过增量任务来维护跨模态语义相似性，并保留旧模型中先前获得的语义相似性知识，从而降低灾难性遗忘的风险。CrossSDC可以无缝集成到不同视听声音分离框架的训练过程中。实验表明，ContAV-Sep可以有效地减轻灾难性遗忘，并取得显着更好的性能相比，其他持续学习基线的视听声音分离。代码可在：\url{https：//github.com/weiguoPian/ContAV-Sep_NeurIPS2024}获得。
摘要：In this paper, we introduce a novel continual audio-visual sound separationtask, aiming to continuously separate sound sources for new classes whilepreserving performance on previously learned classes, with the aid of visualguidance. This problem is crucial for practical visually guided auditoryperception as it can significantly enhance the adaptability and robustness ofaudio-visual sound separation models, making them more applicable forreal-world scenarios where encountering new sound sources is commonplace. Thetask is inherently challenging as our models must not only effectively utilizeinformation from both modalities in current tasks but also preserve theircross-modal association in old tasks to mitigate catastrophic forgetting duringaudio-visual continual learning. To address these challenges, we propose anovel approach named ContAV-Sep (\textbf{Cont}inual\textbf{A}udio-\textbf{V}isual Sound \textbf{Sep}aration). ContAV-Sep presentsa novel Cross-modal Similarity Distillation Constraint (CrossSDC) to uphold thecross-modal semantic similarity through incremental tasks and retain previouslyacquired knowledge of semantic similarity in old models, mitigating the risk ofcatastrophic forgetting. The CrossSDC can seamlessly integrate into thetraining process of different audio-visual sound separation frameworks.Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophicforgetting and achieve significantly better performance compared to othercontinual learning baselines for audio-visual sound separation. Code isavailable at: \url{https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024}.
标题：对抗性多任务水下声学目标识别：针对各种影响因素的鲁棒性
链接：https://arxiv.org/abs/2411.02848
摘要：基于被动声纳的水声目标识别在实际应用中面临着许多挑战。主要挑战之一在于信号特征对不同环境条件和数据采集配置的敏感性，这可能导致识别系统的不稳定性。虽然在水声学的其他领域中，人们一直致力于解决这些影响因素，但在水声目标识别领域中，它们往往被忽视。为了克服这一局限性，本研究设计了辅助任务，模拟影响因素（例如，源范围、水柱深度或风速），并采用多任务框架来将这些因素连接到识别任务。此外，我们将对抗性学习机制集成到多任务框架中，以促使模型提取对影响因素具有鲁棒性的表示。通过对ShipsEar数据集的广泛实验和分析，我们提出的对抗性多任务模型证明了其有效建模影响因素并在12类识别任务上实现最先进性能的能力。
摘要：Underwater acoustic target recognition based on passive sonar faces numerouschallenges in practical maritime applications. One of the main challenges liesin the susceptibility of signal characteristics to diverse environmentalconditions and data acquisition configurations, which can lead to instabilityin recognition systems. While significant efforts have been dedicated toaddressing these influential factors in other domains of underwater acoustics,they are often neglected in the field of underwater acoustic targetrecognition. To overcome this limitation, this study designs auxiliary tasksthat model influential factors (e.g., source range, water column depth, or windspeed) based on available annotations and adopts a multi-task framework toconnect these factors to the recognition task. Furthermore, we integrate anadversarial learning mechanism into the multi-task framework to prompt themodel to extract representations that are robust against influential factors.Through extensive experiments and analyses on the ShipsEar dataset, ourproposed adversarial multi-task model demonstrates its capacity to effectivelymodel the influential factors and achieve state-of-the-art performance on the12-class recognition task.
标题：通过多任务学习和多门混合专家推进稳健的水下声学目标识别
链接：https://arxiv.org/abs/2411.02787
摘要：水声目标识别是水声领域的一个重要研究方向。然而，真实的水下声信号记录的当前可用性仍然有限，这阻碍了数据驱动的声学识别模型从有限的一组复杂的水下信号中学习鲁棒的目标模式，从而损害了它们在实际应用中的稳定性。为了克服这些局限性，本研究提出了一个识别框架称为M3（多任务，多门，多专家），以提高模型的能力，通过使其了解目标的固有属性，捕捉强大的模式。在这个框架中，一个辅助任务，侧重于目标的属性，如估计目标的大小，被设计。辅助任务与识别任务共享参数，实现多任务学习。这种范式允许模型集中于跨任务的共享信息，并以正则化的方式识别目标的鲁棒模式，从而增强模型的泛化能力。此外，M3采用了多专家和多门机制，允许分配不同的参数空间，以各种水下信号。这使得模型能够以细粒度和差异化的方式处理复杂的信号模式。为了评估M3的有效性，在ShipsEar水下船舶辐射噪声数据集上进行了广泛的实验。实验结果表明，M3具有超越最先进的单任务识别模型的能力，从而实现了最先进的性能。
摘要：Underwater acoustic target recognition has emerged as a prominent researcharea within the field of underwater acoustics. However, the currentavailability of authentic underwater acoustic signal recordings remainslimited, which hinders data-driven acoustic recognition models from learningrobust patterns of targets from a limited set of intricate underwater signals,thereby compromising their stability in practical applications. To overcomethese limitations, this study proposes a recognition framework called M3(Multi-task, Multi-gate, Multi-expert) to enhance the model's ability tocapture robust patterns by making it aware of the inherent properties oftargets. In this framework, an auxiliary task that focuses on targetproperties, such as estimating target size, is designed. The auxiliary taskthen shares parameters with the recognition task to realize multi-tasklearning. This paradigm allows the model to concentrate on shared informationacross tasks and identify robust patterns of targets in a regularized manner,thereby enhancing the model's generalization ability. Moreover, M3 incorporatesmulti-expert and multi-gate mechanisms, allowing for the allocation of distinctparameter spaces to various underwater signals. This enables the model toprocess intricate signal patterns in a fine-grained and differentiated manner.To evaluate the effectiveness of M3, extensive experiments were implemented onthe ShipsEar underwater ship-radiated noise dataset. The results substantiatethat M3 has the ability to outperform the most advanced single-task recognitionmodels, thereby achieving the state-of-the-art performance.
标题：DEMONet：基于多专家网络和跨时间变分自动编码器的水下声学目标识别
链接：https://arxiv.org/abs/2411.02758
摘要：由于水下环境的复杂性和目标的动态运动状态，在现实场景中建立鲁棒的水下声学识别系统具有挑战性。一种有前途的优化方法是利用目标的固有物理特性，无论环境条件如何，这些特性都保持不变，以提供强大的见解。然而，我们的研究表明，虽然物理特征表现出强大的特性，但它们可能缺乏特定于类别的区分模式。因此，直接将物理特性纳入模型训练可能会引入意想不到的归纳偏差，导致性能下降。为了利用物理特性的好处，同时减轻可能的不利影响，我们提出了DEMONet在这项研究中，它利用检测噪声的包络调制（DEMON）提供强大的洞察到轴频率或叶片计数的目标。DEMONet是一个多专家网络，它根据DEMON谱将各种水下信号分配到最匹配的专家层，进行细粒度的信号处理。其中，DEMON谱只负责提供隐含的物理特征，而不与目标类别建立映射关系。此外，为了减轻DEMON特征中的噪声和杂散调制谱，我们引入了跨时间对齐策略，并采用变分自动编码器（VAE）来重建抗噪声的DEMON谱，以取代原始的DEMON特征。所提出的具有跨时间VAE的DEMONet的有效性主要在DeepShip数据集和我们的专有数据集上进行了评估。实验结果表明，我们的方法可以在两个数据集上实现最先进的性能。
摘要：Building a robust underwater acoustic recognition system in real-worldscenarios is challenging due to the complex underwater environment and thedynamic motion states of targets. A promising optimization approach is toleverage the intrinsic physical characteristics of targets, which remaininvariable regardless of environmental conditions, to provide robust insights.However, our study reveals that while physical characteristics exhibit robustproperties, they may lack class-specific discriminative patterns. Consequently,directly incorporating physical characteristics into model training canpotentially introduce unintended inductive biases, leading to performancedegradation. To utilize the benefits of physical characteristics whilemitigating possible detrimental effects, we propose DEMONet in this study,which utilizes the detection of envelope modulation on noise (DEMON) to providerobust insights into the shaft frequency or blade counts of targets. DEMONet isa multi-expert network that allocates various underwater signals to theirbest-matched expert layer based on DEMON spectra for fine-grained signalprocessing. Thereinto, DEMON spectra are solely responsible for providingimplicit physical characteristics without establishing a mapping relationshipwith the target category. Furthermore, to mitigate noise and spuriousmodulation spectra in DEMON features, we introduce a cross-temporal alignmentstrategy and employ a variational autoencoder (VAE) to reconstructnoise-resistant DEMON spectra to replace the raw DEMON features. Theeffectiveness of the proposed DEMONet with cross-temporal VAE was primarilyevaluated on the DeepShip dataset and our proprietary datasets. Experimentalresults demonstrated that our approach could achieve state-of-the-artperformance on both datasets.
标题：用于分解音乐音频表示的自我监督多视图学习
链接：https://arxiv.org/abs/2411.02711
备注：Late Breaking Demo at ISMIR 2024. this https URL
摘要：自监督学习（SSL）提供了一种强大的方法来学习鲁棒的，可推广的表示，而无需标记数据。在音乐中，标记数据很少，现有的SSL方法通常使用生成的监督和多视图冗余来创建借口任务。然而，这些方法通常会产生纠缠表示，并丢失特定于视图的信息。我们提出了一种新的自监督多视图学习框架，旨在激励私人和共享表示空间之间的分离。一个在受控环境中的音频解缠的案例研究证明了我们的方法的有效性。
摘要：Self-supervised learning (SSL) offers a powerful way to learn robust,generalizable representations without labeled data. In music, where labeleddata is scarce, existing SSL methods typically use generated supervision andmulti-view redundancy to create pretext tasks. However, these approaches oftenproduce entangled representations and lose view-specific information. Wepropose a novel self-supervised multi-view learning framework for audiodesigned to incentivize separation between private and shared representationspaces. A case study on audio disentanglement in a controlled settingdemonstrates the effectiveness of our method.
标题：DeliverGlobe ++：通过描述自适应球形载体的描述可控的Zero-Shot文本到语音
链接：https://arxiv.org/abs/2411.02625
摘要：近年来，情感语音合成（TTS）技术取得了长足的进步，但由于情感本身的复杂性以及现有情感语音数据集和模型的局限性，其面临的挑战依然存在。以前的研究通常依赖于有限的情感语音数据集或需要大量的手动注释，限制了他们对不同说话者和情感风格进行概括的能力。在本文中，我们提出了一个情感可控的zero-shot TTS模型，可以控制情感的风格和强度，以模仿自然的人类语音。我们引入了一种新的情感自适应球形向量模型的情感风格和强度没有人的注释。此外，我们提出了一个多层次的风格编码器，可以确保有效的推广，看到和看不见的扬声器。我们还引入了额外的损失函数，以提高情感转移性能的zero-shot场景。我们采用了一个条件流匹配为基础的解码器，以实现高质量和表达情感的TTS在几个采样步骤。实验结果证明了该框架的有效性。
摘要：Emotional text-to-speech (TTS) technology has achieved significant progressin recent years; however, challenges remain owing to the inherent complexity ofemotions and limitations of the available emotional speech datasets and models.Previous studies typically relied on limited emotional speech datasets orrequired extensive manual annotations, restricting their ability to generalizeacross different speakers and emotional styles. In this paper, we presentEmoSphere++, an emotion-controllable zero-shot TTS model that can controlemotional style and intensity to resemble natural human speech. We introduce anovel emotion-adaptive spherical vector that models emotional style andintensity without human annotation. Moreover, we propose a multi-level styleencoder that can ensure effective generalization for both seen and unseenspeakers. We also introduce additional loss functions to enhance the emotiontransfer performance for zero-shot scenarios. We employ a conditional flowmatching-based decoder to achieve high-quality and expressive emotional TTS ina few sampling steps. Experimental results demonstrate the effectiveness of theproposed framework.
标题：利用深度学习估计回响环境中边界的数量和位置
链接：https://arxiv.org/abs/2411.02609
摘要：水声环境估计是遥感领域的一项重要而又具有挑战性的任务。当前的估计方法需要高信号强度和对脆弱回波标记问题的有效解决方案。在以前的出版物中，我们提出了一种基于深度学习的二维环境估计方法，无论是在模拟还是在现实实验环境中，该方法都优于最先进的方法。这种方法的局限性在于，用户必须提供关于反射边界的数量和位置的一些先验信息，并且其神经网络必须针对不同的环境进行相应的重新训练。利用更先进的神经网络和时间延迟估计技术，所提出的改进方法不再需要先验知识的边界的数量或其位置，并能够估计二维环境的一个或两个边界。未来的工作将扩展所提出的方法，更多的边界和更大规模的环境。
摘要：Underwater acoustic environment estimation is a challenging but importanttask for remote sensing scenarios. Current estimation methods require highsignal strength and a solution to the fragile echo labeling problem to beeffective. In previous publications, we proposed a general deep learning-basedmethod for two-dimensional environment estimation which outperformed thestate-of-the-art, both in simulation and in real-life experimental settings. Alimitation of this method was that some prior information had to be provided bythe user on the number and locations of the reflective boundaries, and that itsneural networks had to be re-trained accordingly for different environments.Utilizing more advanced neural network and time delay estimation techniques,the proposed improved method no longer requires prior knowledge the number ofboundaries or their locations, and is able to estimate two-dimensionalenvironments with one or two boundaries. Future work will extend the proposedmethod to more boundaries and larger-scale environments.
标题：PIAST：包含音频、符号和文本的多模式钢琴数据集
链接：https://arxiv.org/abs/2411.02551
备注：Accepted for publication at the 3rd Workshop on NLP for Music and Audio (NLP4MusA 2024)
摘要：虽然钢琴音乐已经成为音乐信息检索（MIR）的一个重要研究领域，但显着缺乏带有文本标签的钢琴独奏音乐数据集。为了解决这一差距，我们提出了PIAST（具有音频，符号和文本的钢琴数据集），一个钢琴音乐数据集。利用钢琴特定的语义标签分类，我们从YouTube上收集了9，673首曲目，并为音乐专家的2，023首曲目添加了人工注释，产生了两个子集：PIAST-YT和PIAST-AT。两者都包括音频，文本，标签注释，并转录利用国家的最先进的钢琴转录和节拍跟踪模型。在多模态数据集的许多可能的任务中，我们使用音频和音频数据进行音乐标记和检索，并报告基线性能，以证明其作为MIR研究的宝贵资源的潜力。
摘要：While piano music has become a significant area of study in Music InformationRetrieval (MIR), there is a notable lack of datasets for piano solo music withtext labels. To address this gap, we present PIAST (PIano dataset with Audio,Symbolic, and Text), a piano music dataset. Utilizing a piano-specific taxonomyof semantic tags, we collected 9,673 tracks from YouTube and added humanannotations for 2,023 tracks by music experts, resulting in two subsets:PIAST-YT and PIAST-AT. Both include audio, text, tag annotations, andtranscribed MIDI utilizing state-of-the-art piano transcription and beattracking models. Among many possible tasks with the multi-modal dataset, weconduct music tagging and retrieval using both audio and MIDI data and reportbaseline performances to demonstrate its potential as a valuable resource forMIR research.
标题：最佳交通地图是良好的语音转换器
链接：https://arxiv.org/abs/2411.02402
摘要：最近，基于神经网络的方法来计算最优运输地图已被有效地应用到风格转移问题。然而，这些方法在语音转换中的应用还没有得到充分的研究。在我们的论文中，我们填补了这一空白，调查最佳运输作为语音转换的框架。我们提出了各种最佳的传输算法，设计用于不同的数据表示，如梅尔频谱图和潜在的自我监督语音模型的表示。对于梅尔频谱图数据表示，我们取得了很好的结果，在弗雷歇音频距离（FAD）。这种性能与我们的理论分析是一致的，这表明我们的方法提供了目标和生成的分布之间的FAD上界。在WavLM编码器的潜在空间内，我们实现了最先进的结果，即使在参考扬声器数据有限的情况下也优于现有方法。
摘要：Recently, neural network-based methods for computing optimal transport mapshave been effectively applied to style transfer problems. However, theapplication of these methods to voice conversion is underexplored. In ourpaper, we fill this gap by investigating optimal transport as a framework forvoice conversion. We present a variety of optimal transport algorithms designedfor different data representations, such as mel-spectrograms and latentrepresentation of self-supervised speech models. For the mel-spectogram datarepresentation, we achieve strong results in terms of Frechet Audio Distance(FAD). This performance is consistent with our theoretical analysis, whichsuggests that our method provides an upper bound on the FAD between the targetand generated distributions. Within the latent space of the WavLM encoder, weachived state-of-the-art results and outperformed existing methods even withlimited reference speaker data.
标题：利用谱空间协方差特征盲估计立体声声响声记录的子带声学参数
链接：https://arxiv.org/abs/2411.03172
备注：Submitted to ICASSP2025
摘要：频率变化的声学参数的估计是必不可少的，以增强沉浸式感知在现实的空间音频创作。在本文中，我们提出了一个统一的框架，盲估计混响时间（T60），直接混响比（DRR），和清晰度（C50）在10个频带使用一阶高保真度立体声（FOA）语音录音作为输入。该框架利用了一种新的功能命名为频谱空间协方差向量（SSCV），有效地表示时间，频谱以及空间信息的FOA信号。我们的模型显着优于现有的单通道方法，只有频谱信息，减少了一半以上的所有三个声学参数的估计误差。此外，我们还介绍了FOA-Conv 3D，这是一种新型的后端网络，用于有效地利用3D卷积编码器的SSCV功能。FOA-Conv 3D的性能优于卷积神经网络（CNN）和递归卷积神经网络（CRNN）后端，实现了更低的估计误差，并为所有3个声学参数提供了更高的方差比例（PoV）。
摘要：Estimating frequency-varying acoustic parameters is essential for enhancingimmersive perception in realistic spatial audio creation. In this paper, wepropose a unified framework that blindly estimates reverberation time (T60),direct-to-reverberant ratio (DRR), and clarity (C50) across 10 frequency bandsusing first-order Ambisonics (FOA) speech recordings as inputs. The proposedframework utilizes a novel feature named Spectro-Spatial Covariance Vector(SSCV), efficiently representing temporal, spectral as well as spatialinformation of the FOA signal. Our models significantly outperform existingsingle-channel methods with only spectral information, reducing estimationerrors by more than half for all three acoustic parameters. Additionally, weintroduce FOA-Conv3D, a novel back-end network for effectively utilising theSSCV feature with a 3D convolutional encoder. FOA-Conv3D outperforms theconvolutional neural network (CNN) and recurrent convolutional neural network(CRNN) backends, achieving lower estimation errors and accounting for a higherproportion of variance (PoV) for all 3 acoustic parameters.
标题：利用谱空间协方差特征盲估计立体声声响声记录的子带声学参数
链接：https://arxiv.org/abs/2411.03172
备注：Submitted to ICASSP2025
摘要：频率变化的声学参数的估计是必不可少的，以增强沉浸式感知在现实的空间音频创作。在本文中，我们提出了一个统一的框架，盲估计混响时间（T60），直接混响比（DRR），和清晰度（C50）在10个频带使用一阶高保真度立体声（FOA）语音录音作为输入。该框架利用了一种新的功能命名为频谱空间协方差向量（SSCV），有效地表示时间，频谱以及空间信息的FOA信号。我们的模型显着优于现有的单通道方法，只有频谱信息，减少了一半以上的所有三个声学参数的估计误差。此外，我们还介绍了FOA-Conv 3D，这是一种新型的后端网络，用于有效地利用3D卷积编码器的SSCV功能。FOA-Conv 3D的性能优于卷积神经网络（CNN）和递归卷积神经网络（CRNN）后端，实现了更低的估计误差，并为所有3个声学参数提供了更高的方差比例（PoV）。
摘要：Estimating frequency-varying acoustic parameters is essential for enhancingimmersive perception in realistic spatial audio creation. In this paper, wepropose a unified framework that blindly estimates reverberation time (T60),direct-to-reverberant ratio (DRR), and clarity (C50) across 10 frequency bandsusing first-order Ambisonics (FOA) speech recordings as inputs. The proposedframework utilizes a novel feature named Spectro-Spatial Covariance Vector(SSCV), efficiently representing temporal, spectral as well as spatialinformation of the FOA signal. Our models significantly outperform existingsingle-channel methods with only spectral information, reducing estimationerrors by more than half for all three acoustic parameters. Additionally, weintroduce FOA-Conv3D, a novel back-end network for effectively utilising theSSCV feature with a 3D convolutional encoder. FOA-Conv3D outperforms theconvolutional neural network (CNN) and recurrent convolutional neural network(CRNN) backends, achieving lower estimation errors and accounting for a higherproportion of variance (PoV) for all 3 acoustic parameters.
标题：使用标准化L-p规范的加权预测误差算法的参考麦克风选择
链接：https://arxiv.org/abs/2411.03168
摘要：混响可能严重降低使用房间中的麦克风记录的语音信号的质量。对于紧凑型麦克风阵列，用于多麦克风去混响的参考麦克风的选择通常对去混响性能没有大的影响。相反，当麦克风在空间上分布时，参考麦克风的选择可以显著地有助于去混响性能。在本文中，我们提出了执行参考麦克风选择加权预测误差（WPE）去混响算法的基础上去混响输出信号的归一化$\ell_p$-范数。混响实验室中不同声源位置的实验结果表明，该方法比基于前后混响比或信号功率的参考传声器选择方法具有更好的去混响效果。
摘要：Reverberation may severely degrade the quality of speech signals recordedusing microphones in a room. For compact microphone arrays, the choice of thereference microphone for multi-microphone dereverberation typically does nothave a large influence on the dereverberation performance. In contrast, whenthe microphones are spatially distributed, the choice of the referencemicrophone may significantly contribute to the dereverberation performance. Inthis paper, we propose to perform reference microphone selection for theweighted prediction error (WPE) dereverberation algorithm based on thenormalized $\ell_p$-norm of the dereverberated output signal. Experimentalresults for different source positions in a reverberant laboratory show thatthe proposed method yields a better dereverberation performance than referencemicrophone selection based on the early-to-late reverberation ratio or signalpower.
标题：降噪助听器语音控制
链接：https://arxiv.org/abs/2411.03150
备注：Submitted to IEEE Signal Processing Letters
摘要：提高助听器（HA）语音控制的设计对于提高重听人群的HA使用率以及改善HA用户体验至关重要。在这项工作中，我们通过以下方式为实现这一目标做出贡献：首先，提出一种新型的HA语音数据集，由2个耳后式（BTE）和1个耳道式（IEC）麦克风捕获的嘈杂的自己的声音组成。第二，我们提供了基线HA语音控制结果的评估轻，国家的最先进的关键字定位模型，利用不同的组合HA麦克风信号。实验结果表明，利用带宽有限的骨传导语音（BCS）从IEC麦克风实现噪声鲁棒HA语音控制的好处。此外，结果还表明，语音控制性能可以提高通过辅助BCS的带宽BTE麦克风信号。为了设定科学界可以继续进步的基线，HA噪声语音数据集已公开提供。
摘要：Advancing the design of robust hearing aid (HA) voice control is crucial toincrease the HA use rate among hard of hearing people as well as to improve HAusers' experience. In this work, we contribute towards this goal by, first,presenting a novel HA speech dataset consisting of noisy own voice captured by2 behind-the-ear (BTE) and 1 in-ear-canal (IEC) microphones. Second, we providebaseline HA voice control results from the evaluation of light,state-of-the-art keyword spotting models utilizing different combinations of HAmicrophone signals. Experimental results show the benefits of exploitingbandwidth-limited bone-conducted speech (BCS) from the IEC microphone toachieve noise-robust HA voice control. Furthermore, results also demonstratethat voice control performance can be boosted by assisting BCS by thebroader-bandwidth BTE microphone signals. Aiming at setting a baseline uponwhich the scientific community can continue to progress, the HA noisy speechdataset has been made publicly available.
标题：使用PCG信号中的相异性矩阵对心跳进行无监督检测和分类
链接：https://arxiv.org/abs/2411.03061
备注：None
摘要：该系统由两级级联组成。第一阶段执行粗略的心跳检测，而第二阶段细化先前的心跳检测，从而改进时间定位并且还将心跳分类为类型S1和S2。第一个贡献是一种新的方法，它结合了相异性矩阵与帧级频谱发散定位心跳使用的重复性所示的心音和事件S1/S2和非S1/S2（收缩期和舒张期）定义的间隔之间的时间关系。第二个贡献是基于滑动窗口的验证-校正-分类过程，该滑动窗口允许保存心动周期的时间结构，以便应用于心音分类。所提出的方法进行了评估，使用开放存取数据库PASCAL，CirCor DigiScope Phonocardiogram和一个额外的声音混合过程，考虑到加性高斯白噪声（AWGN）和不同种类的临床环境噪声从商业数据库。所提出的方法提供了最佳的检测/分类性能在现实的情况下，心脏异常的存在，以及不同类型的临床环境噪声是活跃的PCG信号。值得注意的是，由相异性矩阵提供的心脏的时间结构的有前途的建模连同帧级谱发散，以及去除大量的假心脏事件和恢复丢失的心脏事件，两者都由所提出的验证-校正-分类算法校正，表明我们的提议是应用于心脏分割的成功工具。
摘要：The proposed system consists of a two-stage cascade. The first stage performsa rough heartbeat detection while the second stage refines the previous one,improving the temporal localization and also classifying the heartbeats intotypes S1 and S2. The first contribution is a novel approach that combines thedissimilarity matrix with the frame-level spectral divergence to locateheartbeats using the repetitiveness shown by the heart sounds and the temporalrelationships between the intervals defined by the events S1/S2 and non-S1/S2(systole and diastole). The second contribution is averification-correction-classification process based on a sliding window thatallows the preservation of the temporal structure of the cardiac cycle in orderto be applied in the heart sound classification. The proposed method has beenassessed using the open access databases PASCAL, CirCor DigiScopePhonocardiogram and an additional sound mixing procedure considering bothAdditive White Gaussian Noise (AWGN) and different kinds of clinical ambientnoises from a commercial database. The proposed method provides the bestdetection/classification performance in realistic scenarios where the presenceof cardiac anomalies as well as different types of clinical environmentalnoises are active in the PCG signal. Of note, the promising modelling of thetemporal structures of the heart provided by the dissimilarity matrix togetherwith the frame-level spectral divergence, as well as the removal of asignificant number of spurious heart events and recovery of missing heartevents, both corrected by the proposed verification-correction-classificationalgorithm, suggest that our proposal is a successful tool to be applied inheart segmentation.
标题：建筑工地工人安全的实时尖叫检测和位置估计
链接：https://arxiv.org/abs/2411.03016
备注：12 pages, 14 figures, 1 table, submitted to AIRISE conference
摘要：由于事故频发，建筑业面临着高风险，往往使工人处于危险的境地，快速反应至关重要。传统的安全监测方法，包括可穿戴传感器和GPS，通常在障碍物或室内条件下失效。本研究介绍一种新的实时尖叫声检测和定位系统，为建筑工地量身定制，特别是在低资源环境中。集成Wav 2 Vec 2和增强型ConvNet模型以实现准确的尖叫检测，再加上GCC-PHAT算法以实现混响条件下的鲁棒时间延迟估计，然后采用基于梯度下降的方法来实现噪声环境中的精确位置估计。我们的方法结合了这些概念，以实现高检测精度和快速定位，从而最大限度地减少误报和优化应急响应。初步结果表明，该系统不仅可以准确地检测建筑噪音中的求救信号，而且可以可靠地识别呼叫者的位置。该解决方案代表了工人安全的重大改善，具有在高风险职业环境中广泛应用的潜力。用于训练、尖叫检测评估、位置估计和集成框架的脚本将在https://github.com/Anmol2059/construction_safety上发布。
摘要：The construction industry faces high risks due to frequent accidents, oftenleaving workers in perilous situations where rapid response is critical.Traditional safety monitoring methods, including wearable sensors and GPS,often fail under obstructive or indoor conditions. This research introduces anovel real-time scream detection and localization system tailored forconstruction sites, especially in low-resource environments. IntegratingWav2Vec2 and Enhanced ConvNet models for accurate scream detection, coupledwith the GCC-PHAT algorithm for robust time delay estimation under reverberantconditions, followed by a gradient descent-based approach to achieve preciseposition estimation in noisy environments. Our approach combines these conceptsto achieve high detection accuracy and rapid localization, thereby minimizingfalse alarms and optimizing emergency response. Preliminary results demonstratethat the system not only accurately detects distress calls amidst constructionnoise but also reliably identifies the caller's location. This solutionrepresents a substantial improvement in worker safety, with the potential forwidespread application across high-risk occupational environments. The scriptsused for training, evaluation of scream detection, position estimation, andintegrated framework will be released at:https://github.com/Anmol2059/construction_safety.
标题：DeliverGlobe ++：通过描述自适应球形载体的描述可控的Zero-Shot文本到语音
链接：https://arxiv.org/abs/2411.02625
摘要：近年来，情感语音合成（TTS）技术取得了长足的进步，但由于情感本身的复杂性以及现有情感语音数据集和模型的局限性，其面临的挑战依然存在。以前的研究通常依赖于有限的情感语音数据集或需要大量的手动注释，限制了他们对不同说话者和情感风格进行概括的能力。在本文中，我们提出了一个情感可控的zero-shot TTS模型，可以控制情感的风格和强度，以模仿自然的人类语音。我们引入了一种新的情感自适应球形向量模型的情感风格和强度没有人的注释。此外，我们提出了一个多层次的风格编码器，可以确保有效的推广，看到和看不见的扬声器。我们还引入了额外的损失函数，以提高情感转移性能的zero-shot场景。我们采用了一个条件流匹配为基础的解码器，以实现高质量和表达情感的TTS在几个采样步骤。实验结果证明了该框架的有效性。
摘要：Emotional text-to-speech (TTS) technology has achieved significant progressin recent years; however, challenges remain owing to the inherent complexity ofemotions and limitations of the available emotional speech datasets and models.Previous studies typically relied on limited emotional speech datasets orrequired extensive manual annotations, restricting their ability to generalizeacross different speakers and emotional styles. In this paper, we presentEmoSphere++, an emotion-controllable zero-shot TTS model that can controlemotional style and intensity to resemble natural human speech. We introduce anovel emotion-adaptive spherical vector that models emotional style andintensity without human annotation. Moreover, we propose a multi-level styleencoder that can ensure effective generalization for both seen and unseenspeakers. We also introduce additional loss functions to enhance the emotiontransfer performance for zero-shot scenarios. We employ a conditional flowmatching-based decoder to achieve high-quality and expressive emotional TTS ina few sampling steps. Experimental results demonstrate the effectiveness of theproposed framework.
标题：利用深度学习估计回响环境中边界的数量和位置
链接：https://arxiv.org/abs/2411.02609
摘要：水声环境估计是遥感领域的一项重要而又具有挑战性的任务。当前的估计方法需要高信号强度和对脆弱回波标记问题的有效解决方案。在以前的出版物中，我们提出了一种基于深度学习的二维环境估计方法，无论是在模拟还是在现实实验环境中，该方法都优于最先进的方法。这种方法的局限性在于，用户必须提供关于反射边界的数量和位置的一些先验信息，并且其神经网络必须针对不同的环境进行相应的重新训练。利用更先进的神经网络和时间延迟估计技术，所提出的改进方法不再需要先验知识的边界的数量或其位置，并能够估计二维环境的一个或两个边界。未来的工作将扩展所提出的方法，更多的边界和更大规模的环境。
摘要：Underwater acoustic environment estimation is a challenging but importanttask for remote sensing scenarios. Current estimation methods require highsignal strength and a solution to the fragile echo labeling problem to beeffective. In previous publications, we proposed a general deep learning-basedmethod for two-dimensional environment estimation which outperformed thestate-of-the-art, both in simulation and in real-life experimental settings. Alimitation of this method was that some prior information had to be provided bythe user on the number and locations of the reflective boundaries, and that itsneural networks had to be re-trained accordingly for different environments.Utilizing more advanced neural network and time delay estimation techniques,the proposed improved method no longer requires prior knowledge the number ofboundaries or their locations, and is able to estimate two-dimensionalenvironments with one or two boundaries. Future work will extend the proposedmethod to more boundaries and larger-scale environments.
标题：最佳交通地图是良好的语音转换器
链接：https://arxiv.org/abs/2411.02402
摘要：最近，基于神经网络的方法来计算最优运输地图已被有效地应用到风格转移问题。然而，这些方法在语音转换中的应用还没有得到充分的研究。在我们的论文中，我们填补了这一空白，调查最佳运输作为语音转换的框架。我们提出了各种最佳的传输算法，设计用于不同的数据表示，如梅尔频谱图和潜在的自我监督语音模型的表示。对于梅尔频谱图数据表示，我们取得了很好的结果，在弗雷歇音频距离（FAD）。这种性能与我们的理论分析是一致的，这表明我们的方法提供了目标和生成的分布之间的FAD上界。在WavLM编码器的潜在空间内，我们实现了最先进的结果，即使在参考扬声器数据有限的情况下也优于现有方法。
摘要：Recently, neural network-based methods for computing optimal transport mapshave been effectively applied to style transfer problems. However, theapplication of these methods to voice conversion is underexplored. In ourpaper, we fill this gap by investigating optimal transport as a framework forvoice conversion. We present a variety of optimal transport algorithms designedfor different data representations, such as mel-spectrograms and latentrepresentation of self-supervised speech models. For the mel-spectogram datarepresentation, we achieve strong results in terms of Frechet Audio Distance(FAD). This performance is consistent with our theoretical analysis, whichsuggests that our method provides an upper bound on the FAD between the targetand generated distributions. Within the latent space of the WavLM encoder, weachived state-of-the-art results and outperformed existing methods even withlimited reference speaker data.
