今日论文合集：cs.SD语音14篇，eess.AS音频处理15篇。【1】 An Inception-Residual-Based Architecture with Multi-Objective Loss for Detecting Respiratory Anomalies
标题：一种基于初始残差的多目标损失呼吸异常检测框架
链接：http://arxiv.org/pdf/2303.04104v1
作者：Dat Ngo,Lam Pham,Huy Phan,Minh Tran,Delaram Jarchi,Sefki Kolozali机构：. School of Computer Science and Electronic Engineering, University of Essex, UK.摘要：本文提出了一个用于检测呼吸音记录异常的深度学习系统。最初，我们的系统从使用Gammatone和连续小波变换的音频特征提取开始。该步骤旨在将呼吸声输入转换为二维声谱图，其中呈现了频谱和时间特征。然后，我们所提出的系统整合了多个头部注意和多目标损失的基于吸气残差的骨干模型来分类呼吸异常。在这项工作中，我们在IEEE BioCAS 2022挑战赛提出的SPRSound（开放源代码SJTU儿科呼吸音）基准数据集上进行了实验。关于由平均得分和谐波得分之间的平均值计算的得分，与挑战基线系统相比，我们提出的系统在任务1-1、任务1-2、任务2-1和任务2-2中获得了9.7%、15.8%、17.0%和9.4%的显著改善。值得注意的是，我们在任务2-1中取得了最高分73.7%的前1名绩效。摘要：This paper presents a deep learning system applied for detecting anomalies from respiratory sound recordings. Initially, our system begins with audio feature extraction using Gammatone and Continuous Wavelet transformation. This step aims to transform the respiratory sound input into a two-dimensional spectrogram where both spectral and temporal features are presented. Then, our proposed system integrates Inception-residual-based backbone models combined with multi-head attention and multi-objective loss to classify respiratory anomalies. In this work, we conducted experiments over the benchmark dataset of SPRSound (The Open-Source SJTU Paediatric Respiratory Sound) proposed by the IEEE BioCAS 2022 challenge. As regards the Score computed by an average between the average score and harmonic score, our proposed system gained significant improvements of 9.7%, 15.8%, 17.0%, and 9.4% in Task 1-1, Task 1-2, Task 2-1, and Task 2-2 compared to the challenge baseline system. Notably, we achieved the Top-1 performance in Task 2-1 with the highest Score of 73.7%.【2】 VOCALExplore: Pay-as-You-Go Video Data Exploration and Model Building
标题：VOCALExplore：现收现付视频数据探索和模型构建
链接：http://arxiv.org/pdf/2303.04068v1
作者：Maureen Daum,Enhao Zhang,Dong He,Stephen Mussmann,Brandon Haynes,Ranjay Krishna,Magdalena Balazinska机构：University of Washington, Microsoft Gray Systems Lab摘要：我们介绍了VOCALExplore，一个设计用于支持用户在视频数据集上构建特定领域模型的系统。VOCALExplore支持交互式标签会话并使用用户提供的标签训练模型。VOCALExplore通过自动决定如何根据所收集标签中观察到的偏差选择样本，最大限度地提高模型质量。它还通过将特征选择铸造为上升强盗问题来选择训练模型时使用的最佳视频表示。最后，VOCALExplore实现了优化，以实现低延迟而不牺牲模型性能。我们证明了VOCALExplore在给定候选采集函数和特征提取器的情况下实现了接近最佳可能模型质量的目标，并且具有较低的可见延迟（每次迭代约1秒），无需昂贵的预处理。摘要：We introduce VOCALExplore, a system designed to support users in building domain-specific models over video datasets. VOCALExplore supports interactive labeling sessions and trains models using user-supplied labels. VOCALExplore maximizes model quality by automatically deciding how to select samples based on observed skew in the collected labels. It also selects the optimal video representations to use when training models by casting feature selection as a rising bandit problem. Finally, VOCALExplore implements optimizations to achieve low latency without sacrificing model performance. We demonstrate that VOCALExplore achieves close to the best possible model quality given candidate acquisition functions and feature extractors, and it does so with low visible latency (~1 second per iteration) and no expensive preprocessing.【3】 Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
标题：用自己的声音说外语：跨语言神经编解码器语言建模
链接：http://arxiv.org/pdf/2303.03926v1
作者：Ziqiang Zhang,Long Zhou,Chengyi Wang,Sanyuan Chen,Yu Wu,Shujie Liu,Zhuo Chen,Yanqing Liu,Huaming Wang,Jinyu Li,Lei He,Sheng Zhao,Furu Wei备注：We encourage readers to listen to the audio samples on our demo page: url{this https URL}摘要：提出了一种用于跨语言语音合成的跨语言神经编解码语言模型VALL-EX。具体而言，我们扩展了VALL-E并训练了一个多语言条件编解码器语言模型，以源语言语音和目标语言文本作为提示来预测目标语言语音的声学标记序列。VALL-E X继承了强大的上下文学习能力，可以应用于zero-shot跨语言的文本到语音合成和零拍语音到语音翻译任务。实验结果表明，该方法能够以源语言中的一个语音作为提示，生成高质量的目标语言语音，同时保留了未被看见的说话人的声音、情感和声学环境。此外，VALL-E X有效地缓解了外国口音问题，这可以通过语言ID来控制。音频样本可在 url{https： aka.ms vallex}上获得。摘要：We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis. Specifically, we extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at url{https: aka.ms vallex}.【4】 DanceAnyWay: Synthesizing Mixed-Genre 3D Dance Movements Through Beat Disentanglement标题：DanceAnyWay：通过节拍解缠合成混合体裁的3D舞蹈动作
链接：http://arxiv.org/pdf/2303.03870v1
作者：Aneesh Bhattacharya,Uttaran Bhattacharya,Aniket Bera机构：International Institute of Information Technology, Naya Raipur, Chhattisgarh, India, Adobe Research, San Jose, California, USA, Purdue University, West Lafayette, Indiana, USA摘要：提出了一种分层生成对抗式学习方法DanceAnyWay，用于合成与音乐同步的三维人物角色的混合风格舞蹈动作。我们的方法学习通过在两个层次上操作来将节拍帧处的舞蹈动作与所有剩余帧处的舞蹈动作分开。在较粗糙的“节拍”级别，它仅在节拍帧处通过专用特征表示对输入音乐的节奏、音高和旋律信息进行编码。它利用序列到序列的学习框架来综合目标舞蹈的节拍姿势。在更精细的“重复”级别，我们的方法通过专用特征表示对来自输入音乐的所有帧的相似节奏、音高和旋律信息进行编码，并将它们与来自粗略级别的合成节拍姿态耦合，以使用对抗性学习框架合成完整的目标舞蹈序列。通过将较粗层次上的更广泛的舞蹈风格与较细层次上的特定舞蹈动作分离，我们的方法可以有效地合成由任意混合的流派和风格组成的舞蹈。我们通过在混合类型TikTok舞蹈数据集和单类型AIST++数据集上的大量实验来评估我们的方法的性能，并且观察到在两个数据集中，运动质量度量和运动多样性度量分别比当前基线提高了约2%和1.6% - 5.9%。我们还进行了一项用户研究，以评估我们的合成舞蹈的视觉质量。我们注意到，在运动质量和多样性方面，通过我们的方法生成的样本平均比当前可用的最佳基线高出约9%，并且具有高出12%的五点Likert量表评分。摘要：We present DanceAnyWay, a hierarchical generative adversarial learning method to synthesize mixed-genre dance movements of 3D human characters synchronized with music. Our method learns to disentangle the dance movements at the beat frames from the dance movements at all the remaining frames by operating at two hierarchical levels. At the coarser "beat" level, it encodes the rhythm, pitch, and melody information of the input music via dedicated feature representations only at the beat frames. It leverages them to synthesize the beat poses of the target dance using a sequence-to-sequence learning framework. At the finer "repletion" level, our method encodes similar rhythm, pitch, and melody information from all the frames of the input music via dedicated feature representations and couples them with the synthesized beat poses from the coarser level to synthesize the full target dance sequence using an adversarial learning framework. By disentangling the broader dancing styles at the coarser level from the specific dance movements at the finer level, our method can efficiently synthesize dances composed of arbitrarily mixed genres and styles. We evaluate the performance of our approach through extensive experiments on both the mixed-genre TikTok dance dataset and the single-genre AIST++ dataset and observe improvements of about 2% in motion quality metrics and 1.6% - 5.9% in motion diversity metrics over the current baselines in the two datasets respectively. We also conducted a user study to evaluate the visual quality of our synthesized dances. We noted that, on average, the samples generated by our method were about 9% more preferred by the participants and had a 12% better five-point Likert-scale score over the best available current baseline in terms of motion quality and diversity.【5】 Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Study
标题：利用预先训练的AudioLDM生成声音：一项基准研究
链接：http://arxiv.org/pdf/2303.03857v1
作者：Yi Yuan,Haohe Liu,Jinhua Liang,Xubo Liu,Mark D. Plumbley,Wenwu Wang机构：Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK, Centre for Digital Music (C,DM), Queen Mary University of London摘要：深度神经网络最近在声音生成方面取得了突破。尽管具有突出的样本质量，但是当前的声音生成模型在小规模数据集上面临问题（例如，声音类别的过拟合和低覆盖），这显著地限制了性能。在本文中，我们首次尝试以音频生成的前沿模型AudioLDM为主干，研究预训练对声音生成的益处。我们的研究证明了预训练AudioLDM的优势，尤其是在数据稀缺的场景中。此外，发声系统的基线和评价协议不够一致，无法直接比较不同的研究。为了便于声音生成任务的进一步研究，我们在各种常用数据集上对声音生成任务进行了基准测试。希望本文的迁移学习和测试结果能为条件声生成的进一步研究提供参考。摘要：Deep neural networks have recently achieved breakthroughs in sound generation. Despite the outstanding sample quality, current sound generation models face issues on small-scale datasets (e.g., overfitting and low coverage of sound classes), significantly limiting performance. In this paper, we make the first attempt to investigate the benefits of pre-training on sound generation with AudioLDM, the cutting-edge model for audio generation, as the backbone. Our study demonstrates the advantages of the pre-trained AudioLDM, especially in data-scarcity scenarios. In addition, the baselines and evaluation protocol for sound generation systems are not consistent enough to compare different studies directly. Aiming to facilitate further study on sound generation tasks, we benchmark the sound generation task on various frequently-used datasets. We hope our results on transfer learning and benchmarks can provide references for further research on conditional sound generation.【6】 Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning
标题：区分学习优化语音分离的多维多尺度建模
链接：http://arxiv.org/pdf/2303.03737v1
作者：Zhaoxi Mu,Xinyu Yang,Wenjing Zhu机构：⋆ School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China, † Du Xiaoman, Beijing, China备注：Accepted by ICASSP 2023摘要：Transformer在语音分离方面表现出了先进的性能，这得益于它捕捉全局特征的能力。然而，在语音分离中，捕获音频序列的局部特征和信道信息同样重要。本文提出了一种新的语音信号分离方法--内变换-内变换（ISCIT）法。具体而言，我们设计了一个新的网络SE-Conformer，它可以在多个维度和尺度上对音频序列进行建模，并将其应用到双路语音分离框架中。此外，我们提出了多块特征聚合，通过选择性地利用来自分离网络中间块的信息来改善分离效果。同时，针对说话人语音相似时语音分离效果不佳的问题，提出了一种说话人相似度判别损失来优化语音分离模型。在基准数据集WSJ 0 - 2 mix和WHAM！表明ISCIT可以取得最先进结果。摘要：Transformer has shown advanced performance in speech separation, benefiting from its ability to capture global features. However, capturing local features and channel information of audio sequences in speech separation is equally important. In this paper, we present a novel approach named Intra-SE-Conformer and Inter-Transformer (ISCIT) for speech separation. Specifically, we design a new network SE-Conformer that can model audio sequences in multiple dimensions and scales, and apply it to the dual-path speech separation framework. Furthermore, we propose Multi-Block Feature Aggregation to improve the separation effect by selectively utilizing information from the intermediate blocks of the separation network. Meanwhile, we propose a speaker similarity discriminative loss to optimize the speech separation model to address the problem of poor performance when speakers have similar voices. Experimental results on the benchmark datasets WSJ0-2mix and WHAM! show that ISCIT can achieve state-of-the-art results.【7】 A Multi-Stage Triple-Path Method for Speech Separation in Noisy and Reverberant Environments
标题：噪声和混响环境下的多级三径语音分离方法
链接：http://arxiv.org/pdf/2303.03732v1
作者：Zhaoxi Mu,Xinyu Yang,Xiangyuan Yang,Wenjing Zhu机构：⋆ School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China, † Du Xiaoman, Beijing, China备注：Accepted by ICASSP 2023摘要：在噪声和混响环境中，基于深度学习的语音分离方法的性能急剧下降，因为以前的方法没有针对这种情况进行设计和优化。为了解决这一问题，我们提出了一种多阶段端到端学习方法，该方法将噪声和混响环境中困难的语音分离问题解耦为三个子问题：语音去噪、分离和去混响。通过缩小解空间，提高了语音分离模型搜索最优解的概率和速度。此外，由于时域中音频序列的信道信息对于语音分离至关重要，因此我们提出了一种能够对音频序列的信道维度进行建模的三路径结构。实验结果表明，该方法能够以较小的模型参数增量为代价提高语音分离模型的性能。摘要：In noisy and reverberant environments, the performance of deep learning-based speech separation methods drops dramatically because previous methods are not designed and optimized for such situations. To address this issue, we propose a multi-stage end-to-end learning method that decouples the difficult speech separation problem in noisy and reverberant environments into three sub-problems: speech denoising, separation, and de-reverberation. The probability and speed of searching for the optimal solution of the speech separation model are improved by reducing the solution space. Moreover, since the channel information of the audio sequence in the time domain is crucial for speech separation, we propose a triple-path structure capable of modeling the channel dimension of audio sequences. Experimental results show that the proposed multi-stage triple-path method can improve the performance of speech separation models at the cost of little model parameter increment.【8】 Improving Self-Supervised Learning for Audio Representations by Feature Diversity and Decorrelation
标题：利用特征分集和去相关改进音频表示的自监督学习
链接：http://arxiv.org/pdf/2303.03717v1
作者：Bac Nguyen,Stefan Uhlich,Fabien Cardinaux机构：Sony Europe B.V., R&D Center, Stuttgart Laboratory , Germany摘要：自监督学习（SSL）最近在缩小监督学习和无监督学习之间的差距方面显示出显著的效果。其思想是学习对输入数据的失真不变的鲁棒特征。尽管取得了成功，但这个想法可能会遇到崩溃的问题，即网络产生恒定的表示。为此，我们引入了SELFIE，一种新的自监督学习方法，通过特征多样性和解相关来表示音频。SELFIE通过确保表示（i）在嵌入之间保持高多样性以及（ii）解相关维度之间的依赖性来避免塌陷问题。SELFIE在大规模AudioSet数据集上进行了预训练，其嵌入在9个音频下游任务上进行了验证，包括语音、音乐和声音事件识别。实验结果表明，SELFIE在多个任务中的性能优于现有SSL方法。摘要：Self-supervised learning (SSL) has recently shown remarkable results in closing the gap between supervised and unsupervised learning. The idea is to learn robust features that are invariant to distortions of the input data. Despite its success, this idea can suffer from a collapsing issue where the network produces a constant representation. To this end, we introduce SELFIE, a novel Self-supervised Learning approach for audio representation via Feature Diversity and Decorrelation. SELFIE avoids the collapsing issue by ensuring that the representation (i) maintains a high diversity among embeddings and (ii) decorrelates the dependencies between dimensions. SELFIE is pre-trained on the large-scale AudioSet dataset and its embeddings are validated on nine audio downstream tasks, including speech, music, and sound event recognition. Experimental results show that SELFIE outperforms existing SSL methods in several tasks.【9】 Face: Fast, Accurate and Context-Aware Audio Annotation and Classification
标题：人脸：快速、准确和上下文感知的音频标注和分类
链接：http://arxiv.org/pdf/2303.03666v1
作者：M. Mehrdad Morsali,Hoda Mohammadzade,Saeed Bagheri Shouraki机构：Department of Electrical Engineering, Sharif University of Technology摘要：提出了一种上下文感知框架下的特征选择和分类程序，实现了快速准确的音频事件标注和分类。上下文感知设计从探索特征提取技术开始，以找到合适的组合来选择集合，从而以最小的计算量获得显著的分类准确率。对特征选择的探索还包括对音频速度表示的研究，这是在环境音频分类研究范围中先前工作所遗漏的有利特征提取方法。该标注方法考虑了离群点、内点和难以预测的数据样本，实现了上下文感知的主动学习，在只有15%的数据具有初始标注的情况下，平均准确率达到90%。该算法在UrbanSound8K数据集上的平均预测准确率为98.05%。包含源代码和实现结果的笔记本可在www.example.com上获得https: github.com gitmehrdad FACE。摘要：This paper presents a context-aware framework for feature selection and classification procedures to realize a fast and accurate audio event annotation and classification. The context-aware design starts with exploring feature extraction techniques to find an appropriate combination to select a set resulting in remarkable classification accuracy with minimal computational effort. The exploration for feature selection also embraces an investigation of audio Tempo representation, an advantageous feature extraction method missed by previous works in the environmental audio classification research scope. The proposed annotation method considers outlier, inlier, and hard-to-predict data samples to realize context-aware Active Learning, leading to the average accuracy of 90% when only 15% of data possess initial annotation. Our proposed algorithm for sound classification obtained average prediction accuracy of 98.05% on the UrbanSound8K dataset. The notebooks containing our source codes and implementation results are available at https: github.com gitmehrdad FACE.【10】 Adaptive Knowledge Distillation between Text and Speech Pre-trained Models
标题：文本和语音预训练模型之间的自适应知识提取
链接：http://arxiv.org/pdf/2303.03600v1
作者：Jinjie Ni,Yukun Ma,Wen Wang,Qian Chen,Dianwen Ng,Han Lei,Trung Hieu Nguyen,Chong Zhang,Bin Ma,Erik Cambria机构：Alibaba Group, School of Computer Science and Engineering, Nanyang Technological University摘要：近年来，大量语音语料库的学习导致了许多自监督语音模型的成功。通过知识提炼，这些模型还可以受益于由语言模型编码的知识，所述语言模型是在丰富的文本源上预先训练的。然而，由于文本和语音嵌入空间之间的模态差异，提取过程具有挑战性。研究了基于度量的提取算法，在不修改模型结构的情况下，仅用少量数据对齐文本和语音的嵌入空间。由于文献中忽略了文本和语音之间的语义和粒度差距，这影响了提取，因此我们提出了先验信息自适应知识提取（PAD），该方法自适应地利用可变粒度和先验分布的文本 语音单元，以实现文本和语音预训练模型之间更好的全局和局部对齐。我们对三个口语理解基准进行了评估，结果表明PAD比其他基于度量的提炼方法更有效地传递语言知识。摘要：Learning on a massive amount of speech corpus leads to the recent success of many self-supervised speech models. With knowledge distillation, these models may also benefit from the knowledge encoded by language models that are pre-trained on rich sources of texts. The distillation process, however, is challenging due to the modal disparity between textual and speech embedding spaces. This paper studies metric-based distillation to align the embedding space of text and speech with only a small amount of data without modifying the model structure. Since the semantic and granularity gap between text and speech has been omitted in literature, which impairs the distillation, we propose the Prior-informed Adaptive knowledge Distillation (PAD) that adaptively leverages text speech units of variable granularity and prior distributions to achieve better global and local alignments between text and speech pre-trained models. We evaluate on three spoken language understanding benchmarks to show that PAD is more effective in transferring linguistic knowledge than other metric-based distillation approaches.【11】Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms
标题：批量嵌入协方差正则化和常量-Q变换学习广义音频表示法
链接：http://arxiv.org/pdf/2303.03591v1
作者：Ankit Shah,Shuyi Chen,Kejun Zhou,Yue Chen,Bhiksha Raj机构：Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA , Heinz Information College, Carnegie Mellon Institute备注：Technical report, 10 pages摘要：通用嵌入对于包括音频任务在内的许多应用场景中的少镜头甚至零镜头学习是非常期望的。为了更好地理解表述，我们对HEAR 2021投稿结果进行了彻底的错误分析和可视化。受此启发，本文对恒Q变换（CQT）和短时傅立叶变换（STFT）等前端音频预处理方法进行了实验，并提出了一种批量嵌入协方差正则化（BECR）项，以揭示更全面的人类听觉系统接收到的频率信息的模拟。我们在HEAR 2021任务套件上测试了这些模型，这些任务涵盖了广泛的任务类别。实验结果表明：（1）BECR算法能够在测试集上产生更分散的嵌入;（2）BECR算法在不增加计算复杂度的情况下改进了PaSST模型;（3）STFT预处理在所有测试任务中均优于CQT.网站：www.example.comhttps: github.com ankitshah009 general_audio_embedding_hear_2021摘要：General-purpose embedding is highly desirable for few-shot even zero-shot learning in many application scenarios, including audio tasks. In order to understand representations better, we conducted a thorough error analysis and visualization of HEAR 2021 submission results. Inspired by the analysis, this work experiments with different front-end audio preprocessing methods, including Constant-Q Transform (CQT) and Short-time Fourier transform (STFT), and proposes a Batch Embedding Covariance Regularization (BECR) term to uncover a more holistic simulation of the frequency information received by the human auditory system. We tested the models on the suite of HEAR 2021 tasks, which encompass a broad category of tasks. Preliminary results show (1) the proposed BECR can incur a more dispersed embedding on the test set, (2) BECR improves the PaSST model without extra computation complexity, and (3) STFT preprocessing outperforms CQT in all tasks we tested. Github:https: github.com ankitshah009 general_audio_embedding_hear_2021【12】Kernel interpolation of acoustic transfer functions with adaptive kernel for directed and residual reverberations
标题：用于定向混响和残余混响的自适应核函数的核内插
链接：http://arxiv.org/pdf/2303.03869v1
作者：Juliano G. C. Ribeiro,Shoichi Koyama,Hiroshi Saruwatari机构：The University of Tokyo,-,-, Hongo, Bunkyo-ku, Tokyo ,-, JapanJournal-ref：2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：提出了一种基于自适应核岭回归的区域间声传递函数插值方法。大多数当前的ATF插值方法不包括对其执行测量的声学特性。我们提出的方法基于方向加权函数对定向和残余混响的单独自适应，其用于自适应核函数。因此，该方法不仅可以对基本声学特性进行约束，而且可以适应声学环境。数值实验结果表明，该方法在插值精度方面优于现有方法，尤其是在高频区域.摘要：An interpolation method for region-to-region acoustic transfer functions (ATFs) based on kernel ridge regression with an adaptive kernel is proposed. Most current ATF interpolation methods do not incorporate the acoustic properties for which measurements are performed. Our proposed method is based on a separate adaptation of directional weighting functions to directed and residual reverberations, which are used for adapting kernel functions. Thus, the proposed method can not only impose constraints on fundamental acoustic properties, but can also adapt to the acoustic environment. Numerical experimental results indicated that our proposed method outperforms the current methods in terms of interpolation accuracy, especially at high frequencies.【13】 TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings
标题：TS-SEP：基于估计说话人嵌入条件的联合二值化和分离
链接：http://arxiv.org/pdf/2303.03849v1
作者：Christoph Boeddeker,Aswin Shanmugam Subramanian,Gordon Wichern,Reinhold Haeb-Umbach,Jonathan Le Roux机构：Haeb-Umbach are with Paderborn University备注：Submitted to IEEEACM TASLP摘要：由于会议数据的日记化和源分离是密切相关的任务，因此我们在这里提出一种方法来联合执行这两个目标。它建立在目标说话人语音活动检测（TS-VAD）二值化方法的基础上，该方法假设初始说话人嵌入是可用的。我们用以时间-频率分辨率产生说话者活动估计的网络来替换TS-VAD的最终组合的说话者活动估计网络。它们充当用于源提取的掩模，或者经由掩模或者经由波束形成。该技术可应用于单通道和多通道输入，在这两种情况下，LibriCSS会议数据识别任务均实现了最新的字错误率（WER）。我们进一步计算说话人感知和说话人不可知的WER，以分离出日记化错误对整体WER性能的贡献。摘要：Since diarization and source separation of meeting data are closely related tasks, we here propose an approach to perform the two objectives jointly. It builds upon the target-speaker voice activity detection (TS-VAD) diarization approach, which assumes that initial speaker embeddings are available. We replace the final combined speaker activity estimation network of TS-VAD with a network that produces speaker activity estimates at a time-frequency resolution. Those act as masks for source extraction, either via masking or via beamforming. The technique can be applied both for single-channel and multi-channel input and, in both cases, achieves a new state-of-the-art word error rate (WER) on the LibriCSS meeting data recognition task. We further compute speaker-aware and speaker-agnostic WERs to isolate the contribution of diarization errors to the overall WER performance.【14】 AST-SED: An Effective Sound Event Detection Method Based on Audio Spectrogram Transformer
标题：AST-SED：一种有效的基于音频频谱变换的声音事件检测方法
链接：http://arxiv.org/pdf/2303.03689v1
作者：Kang Li,Yan Song,Li-Rong Dai,Ian McLoughlin,Xin Fang,Lin Liu机构：National Engineering Research Centre of Speech and Language Information Processing, University Of Science And Technology Of China, Hefei, China., ICT Cluster, Singapore Institute of Technology, Singapore., FLYTEK Research, iFLYTEK Co. Ltd, Hefei, China.备注：accepted to ICASSP 2023摘要：本文提出了一种有效的声音事件检测（SED）方法AST-SED，该方法基于在大规模AudioSet上预训练的音频谱图Transformer（AST）模型，用于音频标注（AT）任务。预训练的AST模型最近在DCASE 2022挑战任务4中表现出了希望，它们有助于缓解缺乏足够的真实注释数据的问题。然而，主要由于AT和SED任务之间的差异，直接利用来自预训练AST模型的输出是次优的。因此，所提出的AST-SED采用编码器-解码器架构来实现有效和高效的微调，而不需要重新设计或重新训练AST模型。具体而言，频率式Transformer编码器（FTE）由沿频率轴具有自关注的Transformers组成，以解决单个剪辑中的多个重叠音频事件问题。局部门控递归单元解码器（LGD）由最近邻插值（NNI）和双向门控递归单元（Bi-GRU）组成，用于补偿预训练AST模型输出中的时间分辨率损失。在DCASE 2022 task 4开发集上的实验结果验证了所提出的AST-SED和FTE-LGD结构的优越性。具体而言，59.60%的基于事件的F1评分（EB-F1）和0.5140的复调声音检测评分场景1（PSDS 1）评分显著优于CRNN和其他预训练的基于AST的系统。摘要：In this paper, we propose an effective sound event detection (SED) method based on the audio spectrogram transformer (AST) model, pretrained on the large-scale AudioSet for audio tagging (AT) task, termed AST-SED. Pretrained AST models have recently shown promise on DCASE2022 challenge task4 where they help mitigate a lack of sufficient real annotated data. However, mainly due to differences between the AT and SED tasks, it is suboptimal to directly utilize outputs from a pretrained AST model. Hence the proposed AST-SED adopts an encoder-decoder architecture to enable effective and efficient fine-tuning without needing to redesign or retrain the AST model. Specifically, the Frequency-wise Transformer Encoder (FTE) consists of transformers with self attention along the frequency axis to address multiple overlapped audio events issue in a single clip. The Local Gated Recurrent Units Decoder (LGD) consists of nearest-neighbor interpolation (NNI) and Bidirectional Gated Recurrent Units (Bi-GRU) to compensate for temporal resolution loss in the pretrained AST model output. Experimental results on DCASE2022 task4 development set have demonstrated the superiority of the proposed AST-SED with FTE-LGD architecture. Specifically, the Event-Based F1-score (EB-F1) of 59.60% and Polyphonic Sound detection Score scenario1 (PSDS1) score of 0.5140 significantly outperform CRNN and other pretrained AST-based systems.【1】 Kernel interpolation of acoustic transfer functions with adaptive kernel for directed and residual reverberations
标题：用于定向混响和残余混响的自适应核函数的核内插
链接：http://arxiv.org/pdf/2303.03869v1
*与cs.SD语音【12】为同一篇
作者：Juliano G. C. Ribeiro,Shoichi Koyama,Hiroshi Saruwatari机构：The University of Tokyo,-,-, Hongo, Bunkyo-ku, Tokyo ,-, JapanJournal-ref：2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：提出了一种基于自适应核岭回归的区域间声传递函数插值方法。大多数当前的ATF插值方法不包括对其执行测量的声学特性。我们提出的方法基于方向加权函数对定向和残余混响的单独自适应，其用于自适应核函数。因此，该方法不仅可以对基本声学特性进行约束，而且可以适应声学环境。数值实验结果表明，该方法在插值精度方面优于现有方法，尤其是在高频区域.摘要：An interpolation method for region-to-region acoustic transfer functions (ATFs) based on kernel ridge regression with an adaptive kernel is proposed. Most current ATF interpolation methods do not incorporate the acoustic properties for which measurements are performed. Our proposed method is based on a separate adaptation of directional weighting functions to directed and residual reverberations, which are used for adapting kernel functions. Thus, the proposed method can not only impose constraints on fundamental acoustic properties, but can also adapt to the acoustic environment. Numerical experimental results indicated that our proposed method outperforms the current methods in terms of interpolation accuracy, especially at high frequencies.【2】 TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings
标题：TS-SEP：基于估计说话人嵌入条件的联合二值化和分离
链接：http://arxiv.org/pdf/2303.03849v1
*与cs.SD语音【13】为同一篇
作者：Christoph Boeddeker,Aswin Shanmugam Subramanian,Gordon Wichern,Reinhold Haeb-Umbach,Jonathan Le Roux机构：Haeb-Umbach are with Paderborn University备注：Submitted to IEEEACM TASLP摘要：由于会议数据的日记化和源分离是密切相关的任务，因此我们在这里提出一种方法来联合执行这两个目标。它建立在目标说话人语音活动检测（TS-VAD）二值化方法的基础上，该方法假设初始说话人嵌入是可用的。我们用以时间-频率分辨率产生说话者活动估计的网络来替换TS-VAD的最终组合的说话者活动估计网络。它们充当用于源提取的掩模，或者经由掩模或者经由波束形成。该技术可应用于单通道和多通道输入，在这两种情况下，LibriCSS会议数据识别任务均实现了最新的字错误率（WER）。我们进一步计算说话人感知和说话人不可知的WER，以分离出日记化错误对整体WER性能的贡献。摘要：Since diarization and source separation of meeting data are closely related tasks, we here propose an approach to perform the two objectives jointly. It builds upon the target-speaker voice activity detection (TS-VAD) diarization approach, which assumes that initial speaker embeddings are available. We replace the final combined speaker activity estimation network of TS-VAD with a network that produces speaker activity estimates at a time-frequency resolution. Those act as masks for source extraction, either via masking or via beamforming. The technique can be applied both for single-channel and multi-channel input and, in both cases, achieves a new state-of-the-art word error rate (WER) on the LibriCSS meeting data recognition task. We further compute speaker-aware and speaker-agnostic WERs to isolate the contribution of diarization errors to the overall WER performance.【3】 AST-SED: An Effective Sound Event Detection Method Based on Audio Spectrogram Transformer
标题：AST-SED：一种有效的基于音频频谱变换的声音事件检测方法
链接：http://arxiv.org/pdf/2303.03689v1
*与cs.SD语音【14】为同一篇
作者：Kang Li,Yan Song,Li-Rong Dai,Ian McLoughlin,Xin Fang,Lin Liu机构：National Engineering Research Centre of Speech and Language Information Processing, University Of Science And Technology Of China, Hefei, China., ICT Cluster, Singapore Institute of Technology, Singapore., FLYTEK Research, iFLYTEK Co. Ltd, Hefei, China.备注：accepted to ICASSP 2023摘要：本文提出了一种有效的声音事件检测（SED）方法AST-SED，该方法基于在大规模AudioSet上预训练的音频谱图Transformer（AST）模型，用于音频标注（AT）任务。预训练的AST模型最近在DCASE 2022挑战任务4中表现出了希望，它们有助于缓解缺乏足够的真实注释数据的问题。然而，主要由于AT和SED任务之间的差异，直接利用来自预训练AST模型的输出是次优的。因此，所提出的AST-SED采用编码器-解码器架构来实现有效和高效的微调，而不需要重新设计或重新训练AST模型。具体而言，频率式Transformer编码器（FTE）由沿频率轴具有自关注的Transformers组成，以解决单个剪辑中的多个重叠音频事件问题。局部门控递归单元解码器（LGD）由最近邻插值（NNI）和双向门控递归单元（Bi-GRU）组成，用于补偿预训练AST模型输出中的时间分辨率损失。在DCASE 2022 task 4开发集上的实验结果验证了所提出的AST-SED和FTE-LGD结构的优越性。具体而言，59.60%的基于事件的F1评分（EB-F1）和0.5140的复调声音检测评分场景1（PSDS 1）评分显著优于CRNN和其他预训练的基于AST的系统。摘要：In this paper, we propose an effective sound event detection (SED) method based on the audio spectrogram transformer (AST) model, pretrained on the large-scale AudioSet for audio tagging (AT) task, termed AST-SED. Pretrained AST models have recently shown promise on DCASE2022 challenge task4 where they help mitigate a lack of sufficient real annotated data. However, mainly due to differences between the AT and SED tasks, it is suboptimal to directly utilize outputs from a pretrained AST model. Hence the proposed AST-SED adopts an encoder-decoder architecture to enable effective and efficient fine-tuning without needing to redesign or retrain the AST model. Specifically, the Frequency-wise Transformer Encoder (FTE) consists of transformers with self attention along the frequency axis to address multiple overlapped audio events issue in a single clip. The Local Gated Recurrent Units Decoder (LGD) consists of nearest-neighbor interpolation (NNI) and Bidirectional Gated Recurrent Units (Bi-GRU) to compensate for temporal resolution loss in the pretrained AST model output. Experimental results on DCASE2022 task4 development set have demonstrated the superiority of the proposed AST-SED with FTE-LGD architecture. Specifically, the Event-Based F1-score (EB-F1) of 59.60% and Polyphonic Sound detection Score scenario1 (PSDS1) score of 0.5140 significantly outperform CRNN and other pretrained AST-based systems.【4】 Self-FiLM: Conditioning GANs with self-supervised representations for bandwidth extension based speaker recognition
标题：自拍：用于基于带宽扩展的说话人识别的自监督表示条件Gans
链接：http://arxiv.org/pdf/2303.03657v1
作者：Saurabh Kataria,Jesús Villalba,Laureano Moro-Velázquez,Thomas Thebaud,Najim Dehak机构：Center for Language and Speech Processing,Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA摘要：Speech super-resolution Bandwidth Extension (BWE) can improve downstream tasks like Automatic Speaker Verification (ASV). We introduce a simple novel technique called Self-FiLM to inject self-supervision into existing BWE models via Feature-wise Linear Modulation. We hypothesize that such information captures domain environment information, which can give zero-shot generalization. Self-FiLM Conditional GAN (CGAN) gives 18% relative improvement in Equal Error Rate and 8.5% in minimum Decision Cost Function using state-of-the-art ASV system on SRE21 test. We further by 1) deep feature loss from time-domain models and 2) re-training of data2vec 2.0 models on naturalistic wideband (VoxCeleb) and telephone data (SRE Superset etc.). Lastly, we integrate self-supervision with CycleGAN to present a completely unsupervised solution that matches the semi-supervised performance.【5】 An Inception-Residual-Based Architecture with Multi-Objective Loss for Detecting Respiratory Anomalies
标题：一种基于初始残差的多目标损失呼吸异常检测框架
链接：http://arxiv.org/pdf/2303.04104v1
*与cs.SD语音【1】为同一篇
作者：Dat Ngo,Lam Pham,Huy Phan,Minh Tran,Delaram Jarchi,Sefki Kolozali机构：. School of Computer Science and Electronic Engineering, University of Essex, UK.摘要：本文提出了一个用于检测呼吸音记录异常的深度学习系统。最初，我们的系统从使用Gammatone和连续小波变换的音频特征提取开始。该步骤旨在将呼吸声输入转换为二维声谱图，其中呈现了频谱和时间特征。然后，我们所提出的系统整合了多个头部注意和多目标损失的基于吸气残差的骨干模型来分类呼吸异常。在这项工作中，我们在IEEE BioCAS 2022挑战赛提出的SPRSound（开放源代码SJTU儿科呼吸音）基准数据集上进行了实验。关于由平均得分和谐波得分之间的平均值计算的得分，与挑战基线系统相比，我们提出的系统在任务1-1、任务1-2、任务2-1和任务2-2中获得了9.7%、15.8%、17.0%和9.4%的显著改善。值得注意的是，我们在任务2-1中取得了最高分73.7%的前1名绩效。摘要：This paper presents a deep learning system applied for detecting anomalies from respiratory sound recordings. Initially, our system begins with audio feature extraction using Gammatone and Continuous Wavelet transformation. This step aims to transform the respiratory sound input into a two-dimensional spectrogram where both spectral and temporal features are presented. Then, our proposed system integrates Inception-residual-based backbone models combined with multi-head attention and multi-objective loss to classify respiratory anomalies. In this work, we conducted experiments over the benchmark dataset of SPRSound (The Open-Source SJTU Paediatric Respiratory Sound) proposed by the IEEE BioCAS 2022 challenge. As regards the Score computed by an average between the average score and harmonic score, our proposed system gained significant improvements of 9.7%, 15.8%, 17.0%, and 9.4% in Task 1-1, Task 1-2, Task 2-1, and Task 2-2 compared to the challenge baseline system. Notably, we achieved the Top-1 performance in Task 2-1 with the highest Score of 73.7%.【6】 VOCALExplore: Pay-as-You-Go Video Data Exploration and Model Building
标题：VOCALExplore：现收现付视频数据探索和模型构建
链接：http://arxiv.org/pdf/2303.04068v1
*与cs.SD语音【2】为同一篇
作者：Maureen Daum,Enhao Zhang,Dong He,Stephen Mussmann,Brandon Haynes,Ranjay Krishna,Magdalena Balazinska机构：University of Washington, Microsoft Gray Systems Lab摘要：我们介绍了VOCALExplore，一个设计用于支持用户在视频数据集上构建特定领域模型的系统。VOCALExplore支持交互式标签会话并使用用户提供的标签训练模型。VOCALExplore通过自动决定如何根据所收集标签中观察到的偏差选择样本，最大限度地提高模型质量。它还通过将特征选择铸造为上升强盗问题来选择训练模型时使用的最佳视频表示。最后，VOCALExplore实现了优化，以实现低延迟而不牺牲模型性能。我们证明了VOCALExplore在给定候选采集函数和特征提取器的情况下实现了接近最佳可能模型质量的目标，并且具有较低的可见延迟（每次迭代约1秒），无需昂贵的预处理。摘要：We introduce VOCALExplore, a system designed to support users in building domain-specific models over video datasets. VOCALExplore supports interactive labeling sessions and trains models using user-supplied labels. VOCALExplore maximizes model quality by automatically deciding how to select samples based on observed skew in the collected labels. It also selects the optimal video representations to use when training models by casting feature selection as a rising bandit problem. Finally, VOCALExplore implements optimizations to achieve low latency without sacrificing model performance. We demonstrate that VOCALExplore achieves close to the best possible model quality given candidate acquisition functions and feature extractors, and it does so with low visible latency (~1 second per iteration) and no expensive preprocessing.【7】 Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
标题：用自己的声音说外语：跨语言神经编解码器语言建模
链接：http://arxiv.org/pdf/2303.03926v1
*与cs.SD语音【3】为同一篇
作者：Ziqiang Zhang,Long Zhou,Chengyi Wang,Sanyuan Chen,Yu Wu,Shujie Liu,Zhuo Chen,Yanqing Liu,Huaming Wang,Jinyu Li,Lei He,Sheng Zhao,Furu Wei备注：We encourage readers to listen to the audio samples on our demo page: url{this https URL}摘要：提出了一种用于跨语言语音合成的跨语言神经编解码语言模型VALL-EX。具体而言，我们扩展了VALL-E并训练了一个多语言条件编解码器语言模型，以源语言语音和目标语言文本作为提示来预测目标语言语音的声学标记序列。VALL-E X继承了强大的上下文学习能力，可以应用于zero-shot跨语言的文本到语音合成和零拍语音到语音翻译任务。实验结果表明，该方法能够以源语言中的一个语音作为提示，生成高质量的目标语言语音，同时保留了未被看见的说话人的声音、情感和声学环境。此外，VALL-E X有效地缓解了外国口音问题，这可以通过语言ID来控制。音频样本可在 url{https： aka.ms vallex}上获得。摘要：We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis. Specifically, we extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at url{https: aka.ms vallex}.【8】 DanceAnyWay: Synthesizing Mixed-Genre 3D Dance Movements Through Beat Disentanglement
标题：DanceAnyWay：通过节拍解缠合成混合体裁的3D舞蹈动作
链接：http://arxiv.org/pdf/2303.03870v1
*与cs.SD语音【4】为同一篇
作者：Aneesh Bhattacharya,Uttaran Bhattacharya,Aniket Bera机构：International Institute of Information Technology, Naya Raipur, Chhattisgarh, India, Adobe Research, San Jose, California, USA, Purdue University, West Lafayette, Indiana, USA摘要：提出了一种分层生成对抗式学习方法DanceAnyWay，用于合成与音乐同步的三维人物角色的混合风格舞蹈动作。我们的方法学习通过在两个层次上操作来将节拍帧处的舞蹈动作与所有剩余帧处的舞蹈动作分开。在较粗糙的“节拍”级别，它仅在节拍帧处通过专用特征表示对输入音乐的节奏、音高和旋律信息进行编码。它利用序列到序列的学习框架来综合目标舞蹈的节拍姿势。在更精细的“重复”级别，我们的方法通过专用特征表示对来自输入音乐的所有帧的相似节奏、音高和旋律信息进行编码，并将它们与来自粗略级别的合成节拍姿态耦合，以使用对抗性学习框架合成完整的目标舞蹈序列。通过将较粗层次上的更广泛的舞蹈风格与较细层次上的特定舞蹈动作分离，我们的方法可以有效地合成由任意混合的流派和风格组成的舞蹈。我们通过在混合类型TikTok舞蹈数据集和单类型AIST++数据集上的大量实验来评估我们的方法的性能，并且观察到在两个数据集中，运动质量度量和运动多样性度量分别比当前基线提高了约2%和1.6% - 5.9%。我们还进行了一项用户研究，以评估我们的合成舞蹈的视觉质量。我们注意到，在运动质量和多样性方面，通过我们的方法生成的样本平均比当前可用的最佳基线高出约9%，并且具有高出12%的五点Likert量表评分。摘要：We present DanceAnyWay, a hierarchical generative adversarial learning method to synthesize mixed-genre dance movements of 3D human characters synchronized with music. Our method learns to disentangle the dance movements at the beat frames from the dance movements at all the remaining frames by operating at two hierarchical levels. At the coarser "beat" level, it encodes the rhythm, pitch, and melody information of the input music via dedicated feature representations only at the beat frames. It leverages them to synthesize the beat poses of the target dance using a sequence-to-sequence learning framework. At the finer "repletion" level, our method encodes similar rhythm, pitch, and melody information from all the frames of the input music via dedicated feature representations and couples them with the synthesized beat poses from the coarser level to synthesize the full target dance sequence using an adversarial learning framework. By disentangling the broader dancing styles at the coarser level from the specific dance movements at the finer level, our method can efficiently synthesize dances composed of arbitrarily mixed genres and styles. We evaluate the performance of our approach through extensive experiments on both the mixed-genre TikTok dance dataset and the single-genre AIST++ dataset and observe improvements of about 2% in motion quality metrics and 1.6% - 5.9% in motion diversity metrics over the current baselines in the two datasets respectively. We also conducted a user study to evaluate the visual quality of our synthesized dances. We noted that, on average, the samples generated by our method were about 9% more preferred by the participants and had a 12% better five-point Likert-scale score over the best available current baseline in terms of motion quality and diversity.【9】 Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Study
标题：利用预先训练的AudioLDM生成声音：一项基准研究
链接：http://arxiv.org/pdf/2303.03857v1
*与cs.SD语音【5】为同一篇
作者：Yi Yuan,Haohe Liu,Jinhua Liang,Xubo Liu,Mark D. Plumbley,Wenwu Wang机构：Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK, Centre for Digital Music (C,DM), Queen Mary University of London摘要：深度神经网络最近在声音生成方面取得了突破。尽管具有突出的样本质量，但是当前的声音生成模型在小规模数据集上面临问题（例如，声音类别的过拟合和低覆盖），这显著地限制了性能。在本文中，我们首次尝试以音频生成的前沿模型AudioLDM为主干，研究预训练对声音生成的益处。我们的研究证明了预训练AudioLDM的优势，尤其是在数据稀缺的场景中。此外，发声系统的基线和评价协议不够一致，无法直接比较不同的研究。为了便于声音生成任务的进一步研究，我们在各种常用数据集上对声音生成任务进行了基准测试。希望本文的迁移学习和测试结果能为条件声生成的进一步研究提供参考。摘要：Deep neural networks have recently achieved breakthroughs in sound generation. Despite the outstanding sample quality, current sound generation models face issues on small-scale datasets (e.g., overfitting and low coverage of sound classes), significantly limiting performance. In this paper, we make the first attempt to investigate the benefits of pre-training on sound generation with AudioLDM, the cutting-edge model for audio generation, as the backbone. Our study demonstrates the advantages of the pre-trained AudioLDM, especially in data-scarcity scenarios. In addition, the baselines and evaluation protocol for sound generation systems are not consistent enough to compare different studies directly. Aiming to facilitate further study on sound generation tasks, we benchmark the sound generation task on various frequently-used datasets. We hope our results on transfer learning and benchmarks can provide references for further research on conditional sound generation.【10】 Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning
标题：区分学习优化语音分离的多维多尺度建模
链接：http://arxiv.org/pdf/2303.03737v1
*与cs.SD语音【6】为同一篇
作者：Zhaoxi Mu,Xinyu Yang,Wenjing Zhu机构：⋆ School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China, † Du Xiaoman, Beijing, China备注：Accepted by ICASSP 2023摘要：Transformer在语音分离方面表现出了先进的性能，这得益于它捕捉全局特征的能力。然而，在语音分离中，捕获音频序列的局部特征和信道信息同样重要。本文提出了一种新的语音信号分离方法--内变换-内变换（ISCIT）法。具体而言，我们设计了一个新的网络SE-Conformer，它可以在多个维度和尺度上对音频序列进行建模，并将其应用到双路语音分离框架中。此外，我们提出了多块特征聚合，通过选择性地利用来自分离网络中间块的信息来改善分离效果。同时，针对说话人语音相似时语音分离效果不佳的问题，提出了一种说话人相似度判别损失来优化语音分离模型。在基准数据集WSJ 0 - 2 mix和WHAM！表明ISCIT可以取得最先进结果。摘要：Transformer has shown advanced performance in speech separation, benefiting from its ability to capture global features. However, capturing local features and channel information of audio sequences in speech separation is equally important. In this paper, we present a novel approach named Intra-SE-Conformer and Inter-Transformer (ISCIT) for speech separation. Specifically, we design a new network SE-Conformer that can model audio sequences in multiple dimensions and scales, and apply it to the dual-path speech separation framework. Furthermore, we propose Multi-Block Feature Aggregation to improve the separation effect by selectively utilizing information from the intermediate blocks of the separation network. Meanwhile, we propose a speaker similarity discriminative loss to optimize the speech separation model to address the problem of poor performance when speakers have similar voices. Experimental results on the benchmark datasets WSJ0-2mix and WHAM! show that ISCIT can achieve state-of-the-art results.【11】 A Multi-Stage Triple-Path Method for Speech Separation in Noisy and Reverberant Environments
标题：噪声和混响环境下的多级三径语音分离方法
链接：http://arxiv.org/pdf/2303.03732v1
*与cs.SD语音【7】为同一篇
作者：Zhaoxi Mu,Xinyu Yang,Xiangyuan Yang,Wenjing Zhu机构：⋆ School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China, † Du Xiaoman, Beijing, China备注：Accepted by ICASSP 2023摘要：在噪声和混响环境中，基于深度学习的语音分离方法的性能急剧下降，因为以前的方法没有针对这种情况进行设计和优化。为了解决这一问题，我们提出了一种多阶段端到端学习方法，该方法将噪声和混响环境中困难的语音分离问题解耦为三个子问题：语音去噪、分离和去混响。通过缩小解空间，提高了语音分离模型搜索最优解的概率和速度。此外，由于时域中音频序列的信道信息对于语音分离至关重要，因此我们提出了一种能够对音频序列的信道维度进行建模的三路径结构。实验结果表明，该方法能够以较小的模型参数增量为代价提高语音分离模型的性能。摘要：In noisy and reverberant environments, the performance of deep learning-based speech separation methods drops dramatically because previous methods are not designed and optimized for such situations. To address this issue, we propose a multi-stage end-to-end learning method that decouples the difficult speech separation problem in noisy and reverberant environments into three sub-problems: speech denoising, separation, and de-reverberation. The probability and speed of searching for the optimal solution of the speech separation model are improved by reducing the solution space. Moreover, since the channel information of the audio sequence in the time domain is crucial for speech separation, we propose a triple-path structure capable of modeling the channel dimension of audio sequences. Experimental results show that the proposed multi-stage triple-path method can improve the performance of speech separation models at the cost of little model parameter increment.【12】 Improving Self-Supervised Learning for Audio Representations by Feature Diversity and Decorrelation
标题：利用特征分集和去相关改进音频表示的自监督学习
链接：http://arxiv.org/pdf/2303.03717v1
*与cs.SD语音【8】为同一篇
作者：Bac Nguyen,Stefan Uhlich,Fabien Cardinaux机构：Sony Europe B.V., R&D Center, Stuttgart Laboratory , Germany摘要：自监督学习（SSL）最近在缩小监督学习和无监督学习之间的差距方面显示出显著的效果。其思想是学习对输入数据的失真不变的鲁棒特征。尽管取得了成功，但这个想法可能会遇到崩溃的问题，即网络产生恒定的表示。为此，我们引入了SELFIE，一种新的自监督学习方法，通过特征多样性和解相关来表示音频。SELFIE通过确保表示（i）在嵌入之间保持高多样性以及（ii）解相关维度之间的依赖性来避免塌陷问题。SELFIE在大规模AudioSet数据集上进行了预训练，其嵌入在9个音频下游任务上进行了验证，包括语音、音乐和声音事件识别。实验结果表明，SELFIE在多个任务中的性能优于现有SSL方法。摘要：Self-supervised learning (SSL) has recently shown remarkable results in closing the gap between supervised and unsupervised learning. The idea is to learn robust features that are invariant to distortions of the input data. Despite its success, this idea can suffer from a collapsing issue where the network produces a constant representation. To this end, we introduce SELFIE, a novel Self-supervised Learning approach for audio representation via Feature Diversity and Decorrelation. SELFIE avoids the collapsing issue by ensuring that the representation (i) maintains a high diversity among embeddings and (ii) decorrelates the dependencies between dimensions. SELFIE is pre-trained on the large-scale AudioSet dataset and its embeddings are validated on nine audio downstream tasks, including speech, music, and sound event recognition. Experimental results show that SELFIE outperforms existing SSL methods in several tasks.【13】 Face: Fast, Accurate and Context-Aware Audio Annotation and Classification
标题：人脸：快速、准确和上下文感知的音频标注和分类
链接：http://arxiv.org/pdf/2303.03666v1
*与cs.SD语音【9】为同一篇
作者：M. Mehrdad Morsali,Hoda Mohammadzade,Saeed Bagheri Shouraki机构：Department of Electrical Engineering, Sharif University of Technology摘要：提出了一种上下文感知框架下的特征选择和分类程序，实现了快速准确的音频事件标注和分类。上下文感知设计从探索特征提取技术开始，以找到合适的组合来选择集合，从而以最小的计算量获得显著的分类准确率。对特征选择的探索还包括对音频速度表示的研究，这是在环境音频分类研究范围中先前工作所遗漏的有利特征提取方法。该标注方法考虑了离群点、内点和难以预测的数据样本，实现了上下文感知的主动学习，在只有15%的数据具有初始标注的情况下，平均准确率达到90%。该算法在UrbanSound8K数据集上的平均预测准确率为98.05%。包含源代码和实现结果的笔记本可在www.example.com上获得https: github.com gitmehrdad FACE。摘要：This paper presents a context-aware framework for feature selection and classification procedures to realize a fast and accurate audio event annotation and classification. The context-aware design starts with exploring feature extraction techniques to find an appropriate combination to select a set resulting in remarkable classification accuracy with minimal computational effort. The exploration for feature selection also embraces an investigation of audio Tempo representation, an advantageous feature extraction method missed by previous works in the environmental audio classification research scope. The proposed annotation method considers outlier, inlier, and hard-to-predict data samples to realize context-aware Active Learning, leading to the average accuracy of 90% when only 15% of data possess initial annotation. Our proposed algorithm for sound classification obtained average prediction accuracy of 98.05% on the UrbanSound8K dataset. The notebooks containing our source codes and implementation results are available at https: github.com gitmehrdad FACE.【14】 Adaptive Knowledge Distillation between Text and Speech Pre-trained Models
标题：文本和语音预训练模型之间的自适应知识提取
链接：http://arxiv.org/pdf/2303.03600v1
*与cs.SD语音【10】为同一篇
作者：Jinjie Ni,Yukun Ma,Wen Wang,Qian Chen,Dianwen Ng,Han Lei,Trung Hieu Nguyen,Chong Zhang,Bin Ma,Erik Cambria机构：Alibaba Group, School of Computer Science and Engineering, Nanyang Technological University摘要：近年来，大量语音语料库的学习导致了许多自监督语音模型的成功。通过知识提炼，这些模型还可以受益于由语言模型编码的知识，所述语言模型是在丰富的文本源上预先训练的。然而，由于文本和语音嵌入空间之间的模态差异，提取过程具有挑战性。研究了基于度量的提取算法，在不修改模型结构的情况下，仅用少量数据对齐文本和语音的嵌入空间。由于文献中忽略了文本和语音之间的语义和粒度差距，这影响了提取，因此我们提出了先验信息自适应知识提取（PAD），该方法自适应地利用可变粒度和先验分布的文本 语音单元，以实现文本和语音预训练模型之间更好的全局和局部对齐。我们对三个口语理解基准进行了评估，结果表明PAD比其他基于度量的提炼方法更有效地传递语言知识。摘要：Learning on a massive amount of speech corpus leads to the recent success of many self-supervised speech models. With knowledge distillation, these models may also benefit from the knowledge encoded by language models that are pre-trained on rich sources of texts. The distillation process, however, is challenging due to the modal disparity between textual and speech embedding spaces. This paper studies metric-based distillation to align the embedding space of text and speech with only a small amount of data without modifying the model structure. Since the semantic and granularity gap between text and speech has been omitted in literature, which impairs the distillation, we propose the Prior-informed Adaptive knowledge Distillation (PAD) that adaptively leverages text speech units of variable granularity and prior distributions to achieve better global and local alignments between text and speech pre-trained models. We evaluate on three spoken language understanding benchmarks to show that PAD is more effective in transferring linguistic knowledge than other metric-based distillation approaches.【15】Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms
标题：批量嵌入协方差正则化和常量-Q变换学习广义音频表示法
链接：http://arxiv.org/pdf/2303.03591v1
*与cs.SD语音【11】为同一篇
作者：Ankit Shah,Shuyi Chen,Kejun Zhou,Yue Chen,Bhiksha Raj机构：Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA , Heinz Information College, Carnegie Mellon Institute备注：Technical report, 10 pages摘要：通用嵌入对于包括音频任务在内的许多应用场景中的少镜头甚至零镜头学习是非常期望的。为了更好地理解表述，我们对HEAR 2021投稿结果进行了彻底的错误分析和可视化。受此启发，本文对恒Q变换（CQT）和短时傅立叶变换（STFT）等前端音频预处理方法进行了实验，并提出了一种批量嵌入协方差正则化（BECR）项，以揭示更全面的人类听觉系统接收到的频率信息的模拟。我们在HEAR 2021任务套件上测试了这些模型，这些任务涵盖了广泛的任务类别。实验结果表明：（1）BECR算法能够在测试集上产生更分散的嵌入;（2）BECR算法在不增加计算复杂度的情况下改进了PaSST模型;（3）STFT预处理在所有测试任务中均优于CQT.网站：www.example.comhttps: github.com ankitshah009 general_audio_embedding_hear_2021摘要：General-purpose embedding is highly desirable for few-shot even zero-shot learning in many application scenarios, including audio tasks. In order to understand representations better, we conducted a thorough error analysis and visualization of HEAR 2021 submission results. Inspired by the analysis, this work experiments with different front-end audio preprocessing methods, including Constant-Q Transform (CQT) and Short-time Fourier transform (STFT), and proposes a Batch Embedding Covariance Regularization (BECR) term to uncover a more holistic simulation of the frequency information received by the human auditory system. We tested the models on the suite of HEAR 2021 tasks, which encompass a broad category of tasks. Preliminary results show (1) the proposed BECR can incur a more dispersed embedding on the test set, (2) BECR improves the PaSST model without extra computation complexity, and (3) STFT preprocessing outperforms CQT in all tasks we tested. Github:https: github.com ankitshah009 general_audio_embedding_hear_2021机器翻译，仅供参考
每日语音速递