今日论文合集：cs.SD语音15篇，eess.AS音频处理16篇。
cs.SD语音

【1】Pre-Training Strategies Using Contrastive Learning and Playlist Information for Music Classification and Similarity标题：基于对比学习和播放列表信息的音乐分类相似度预训练策略链接：http://arxiv.org/pdf/2304.12257v1

作者：Pablo Alonso-Jiménez,Xavier Favory,Hadrien Foroughmand,Grigoris Bourdalas,Xavier Serra,Thomas Lidy,Dmitry Bogdanov机构：Utopia Music, Switzerland, Music Technology Group, Universitat Pompeu Fabra, Spain备注：Accepted at the 2023 International Conference on Acoustics, Speech, and Signal Processing (ICASSP'23)摘要：在这项工作中，我们研究了一种方法，依赖于对比学习和音乐元数据作为一个薄弱的监督来源，以训练音乐表示模型。最近的研究表明，对比学习可以与编辑元数据（例如，艺术家或专辑名称）来学习对不同分类任务有用的音频表示。在本文中，我们扩展这个想法，使用播放列表数据作为音乐相似性信息的来源，并研究三种方法来生成锚和积极的轨道对。我们通过微调音乐多标签分类任务（流派，情绪和乐器标记）和音乐相似性的预训练模型来评估这些方法。我们发现，通过依赖于播放列表中的同现创建锚和积极的轨道对提供了更好的音乐相似性和竞争力的分类结果相比，从相同的艺术家在以前的作品中选择的轨道。此外，我们基于播放列表的最佳预训练方法为大多数数据集提供了卓越的分类性能。摘要：In this work, we investigate an approach that relies on contrastive learning and music metadata as a weak source of supervision to train music representation models. Recent studies show that contrastive learning can be used with editorial metadata (e.g., artist or album name) to learn audio representations that are useful for different classification tasks. In this paper, we extend this idea to using playlist data as a source of music similarity information and investigate three approaches to generate anchor and positive track pairs. We evaluate these approaches by fine-tuning the pre-trained models for music multi-label classification tasks (genre, mood, and instrument tagging) and music similarity. We find that creating anchor and positive track pairs by relying on co-occurrences in playlists provides better music similarity and competitive classification results compared to choosing tracks from the same artist as in previous works. Additionally, our best pre-training approach based on playlists provides superior classification performance for most datasets.

【2】Small-footprint slimmable networks for keyword spotting标题：用于关键词识别的占地面积小的可精简网络链接：http://arxiv.org/pdf/2304.12183v1

作者：Zuhaib Akhtar,Mohammad Omar Khursheed,Dongsu Du,Yuzong Liu机构：Alexa Perceptual Technologies, Amazon, New York University, NY摘要：在这项工作中，我们提出了应用于小足迹关键字定位问题的Slimmable神经网络。我们表明，可瘦身的神经网络允许我们从卷积神经网络和Transformers中创建超网，从中可以提取不同大小的子网络。我们在内部Alexa数据和Google语音命令上展示了这些模型的有用性，并将我们的努力集中在设备上用例的模型上，将我们的参数限制在250k以内。我们表明，可精简模型可以匹配（在某些情况下，优于）从头开始训练的模型。因此，可瘦身神经网络是一类特别有用的模型，当相同的功能要在不同的存储器和计算预算下复制时，具有不同的精度要求。摘要：In this work, we present Slimmable Neural Networks applied to the problem of small-footprint keyword spotting. We show that slimmable neural networks allow us to create super-nets from Convolutioanl Neural Networks and Transformers, from which sub-networks of different sizes can be extracted. We demonstrate the usefulness of these models on in-house Alexa data and Google Speech Commands, and focus our efforts on models for the on-device use case, limiting ourselves to less than 250k parameters. We show that slimmable models can match (and in some cases, outperform) models trained from scratch. Slimmable neural networks are therefore a class of models particularly useful when the same functionality is to be replicated at different memory and compute budgets, with different accuracy requirements.

【3】Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models标题：基于自监督学习模型的深度视听演唱配音链接：http://arxiv.org/pdf/2304.12082v1

作者：Xiangming Gu,Wei Zeng,Jianan Zhang,Longshen Ou,Ye Wang机构：Ye Wang are withSchool of Computing, National University of Singapore摘要：歌唱语音转录将录制的歌唱音频转换为乐谱。声音污染（如伴奏）和注释数据的缺乏使歌唱声音转录成为一项极其困难的任务。我们采取两种方法来应对上述挑战：1）引入多模态学习用于歌唱声音转录以及新的多模态歌唱数据集N20 EMv 2，通过利用视频信息（嘴唇运动来预测音符的开始 结束）来增强噪声鲁棒性，以及2）将语音域的自监督学习模型适应于歌唱声音转录任务，在保留预训练特征的同时显着减少注释数据要求。我们建立了一个基于自监督学习的纯音频歌唱语音转录系统，它不仅优于当前最先进的技术作为一个强大的基线，但也很好地推广到域外歌唱数据。然后，我们开发了一个基于自我监督学习的纯视频歌唱语音转录系统，检测注意的起始和偏移的准确率约为80%。最后，基于上述两个系统提取的强大的声学和视觉表示以及特征融合设计，我们创建了一个视听歌唱语音转录系统，提高了噪声鲁棒性显着不同的声学环境下相比，只有音频系统。摘要：Singing voice transcription converts recorded singing audio to musical notation. Sound contamination (such as accompaniment) and lack of annotated data make singing voice transcription an extremely difficult task. We take two approaches to tackle the above challenges: 1) introducing multimodal learning for singing voice transcription together with a new multimodal singing dataset, N20EMv2, enhancing noise robustness by utilizing video information (lip movements to predict the onset offset of notes), and 2) adapting self-supervised learning models from the speech domain to the singing voice transcription task, significantly reducing annotated data requirements while preserving pretrained features. We build a self-supervised learning based audio-only singing voice transcription system, which not only outperforms current state-of-the-art technologies as a strong baseline, but also generalizes well to out-of-domain singing data. We then develop a self-supervised learning based video-only singing voice transcription system that detects note onsets and offsets with an accuracy of about 80 %. Finally, based on the powerful acoustic and visual representations extracted by the above two systems as well as the feature fusion design, we create an audio-visual singing voice transcription system that improves the noise robustness significantly under different acoustic environments compared to the audio-only systems.

【4】Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model标题：基于自监督语音表示模型的零发声文语合成链接：http://arxiv.org/pdf/2304.11976v1

作者：Kenichi Fujita,Takanori Ashihara,Hiroki Kanagawa,Takafumi Moriya,Yusuke Ijima机构：NTT Corporation, Japan备注：5 pages,3 figures, Accepted to IEEE ICASSP 2023 workshop Self-supervision in Audio, Speech and Beyond摘要：本文提出了一种zero-shot文本到语音（TTS）的条件下，通过自监督学习（SSL）获得的自监督语音表示模型。具有来自X向量或全局样式令牌的嵌入向量的常规方法在再现未看见的说话者的说话者特性方面仍然具有差距。所提出的方法的一个新颖点是直接使用SSL模型从用大量数据训练的语音表示中获得嵌入向量。我们还引入了单独的条件的声学特征和音素持续时间预测器，以获得基于节奏的扬声器特性和基于声学特征的之间的解开嵌入。解开的嵌入将使我们能够实现更好的再现性能为看不见的发言者和节奏转移的不同的讲话。客观和主观评价表明，该方法可以合成语音的相似性提高，实现语音节奏转移。摘要：This paper proposes a zero-shot text-to-speech (TTS) conditioned by a self-supervised speech-representation model acquired through self-supervised learning (SSL). Conventional methods with embedding vectors from x-vector or global style tokens still have a gap in reproducing the speaker characteristics of unseen speakers. A novel point of the proposed method is the direct use of the SSL model to obtain embedding vectors from speech representations trained with a large amount of data. We also introduce the separate conditioning of acoustic features and a phoneme duration predictor to obtain the disentangled embeddings between rhythm-based speaker characteristics and acoustic-feature-based ones. The disentangled embeddings will enable us to achieve better reproduction performance for unseen speakers and rhythm transfer conditioned by different speeches. Objective and subjective evaluations showed that the proposed method can synthesize speech with improved similarity and achieve speech-rhythm transfer.

【5】Advancing underwater acoustic target recognition via adaptive data pruning and smoothness-inducing regularization标题：基于自适应数据剪枝和平滑诱导正则化的水声目标识别链接：http://arxiv.org/pdf/2304.11907v1

作者：Yuan Xie,Tianyu Chen,Ji Xu机构：Key laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, BDBC, Beihang University, China ,SKLSDE, Beihang University摘要：舰船辐射信号的水声识别具有识别非视距目标的能力，具有很高的实际应用价值。但由于数据采集的困难，采集到的信号数量稀少，且主要由机械周期性噪声组成。根据实验，我们观察到，周期信号的重复性导致双下降现象，这表明一个显着的局部偏向重复的样本。为了解决这个问题，我们提出了一种基于交叉熵的策略，以修剪训练数据中过于相似的片段。此外，为了补偿训练数据的减少，我们生成噪声样本，并应用基于KL散度的平滑诱导正则化来减轻过拟合。实验表明，我们提出的数据修剪和正则化策略可以带来稳定的利益，我们的框架显着优于国家的最先进的低资源的情况下。摘要：Underwater acoustic recognition for ship-radiated signals has high practical application value due to the ability to recognize non-line-of-sight targets. However, due to the difficulty of data acquisition, the collected signals are scarce in quantity and mainly composed of mechanical periodic noise. According to the experiments, we observe that the repeatability of periodic signals leads to a double-descent phenomenon, which indicates a significant local bias toward repeated samples. To address this issue, we propose a strategy based on cross-entropy to prune excessively similar segments in training data. Furthermore, to compensate for the reduction of training data, we generate noisy samples and apply smoothness-inducing regularization based on KL divergence to mitigate overfitting. Experiments show that our proposed data pruning and regularization strategy can bring stable benefits and our framework significantly outperforms the state-of-the-art in low-resource scenarios.

【6】Sound-based drone fault classification using multitask learning标题：基于多任务学习的声学无人机故障分类链接：http://arxiv.org/pdf/2304.11708v1

作者：Wonjun Yi,Jung-Woo Choi,Jae-Woo Lee机构：Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea备注：Accepted at 29th International Congress on Sound and Vibration (ICSV29). Dataset available: this https URL摘要：无人机已被用于各种目的，包括军事应用，空中摄影和农药喷洒。然而，无人机容易受到外部干扰，并且螺旋桨和电机容易发生故障。为了提高无人机操作的安全性，应该实时检测无人机的机械故障。本文提出了一种基于声音的深度神经网络（DNN）故障分类器和无人机声音数据集。该数据集是通过在消声室中从安装在三个不同无人机上的麦克风收集无人机的操作声音来构建的。该数据集包括无人机的各种操作条件，例如飞行方向（前，后，右，左，顺时针，逆时针）以及螺旋桨和电机上的故障。然后，无人机的声音与大学校园内五个不同地点记录的噪音混合，信噪比（SNR）从10 dB到15 dB不等。使用所获得的数据集，我们训练了DNN分类器1DCNN-ResNet，该分类器从短时间输入波形中分类机械故障的类型及其位置。我们采用多任务学习（MTL），并将方向分类任务作为辅助任务，使分类器学习更一般的音频特征。对未知数据的测试表明，所提出的多任务模型可以成功地对无人机中的故障进行分类，即使训练数据较少，也优于单任务模型。摘要：The drone has been used for various purposes, including military applications, aerial photography, and pesticide spraying. However, the drone is vulnerable to external disturbances, and malfunction in propellers and motors can easily occur. To improve the safety of drone operations, one should detect the mechanical faults of drones in real-time. This paper proposes a sound-based deep neural network (DNN) fault classifier and drone sound dataset. The dataset was constructed by collecting the operating sounds of drones from microphones mounted on three different drones in an anechoic chamber. The dataset includes various operating conditions of drones, such as flight directions (front, back, right, left, clockwise, counterclockwise) and faults on propellers and motors. The drone sounds were then mixed with noises recorded in five different spots on the university campus, with a signal-to-noise ratio (SNR) varying from 10 dB to 15 dB. Using the acquired dataset, we train a DNN classifier, 1DCNN-ResNet, that classifies the types of mechanical faults and their locations from short-time input waveforms. We employ multitask learning (MTL) and incorporate the direction classification task as an auxiliary task to make the classifier learn more general audio features. The test over unseen data reveals that the proposed multitask model can successfully classify faults in drones and outperforms single-task models even with less training data.

【7】SAR: Self-Supervised Anti-Distortion Representation for End-To-End Speech Model标题：SAR：端到端语音模型的自监督抗失真表示链接：http://arxiv.org/pdf/2304.11547v1

作者：Jianzong Wang,Xulong Zhang,Haobin Tang,Aolan Sun,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd., University of Science and Technology of China备注：Accepted by IJCNN2023. 2023 International Joint Conference on Neural Networks (IJCNN2023)摘要：在最近的文本到语音（TTS）系统中，神经声码器通常通过仅调节从声学模型预测的声学特征来生成语音样本。然而，与地面实况的那些相比，预测的声学特征中总是存在失真，特别是在由于低质量训练数据而导致的声学建模不佳的常见情况下。为了克服这些限制，我们提出了一个自监督学习框架来学习抗失真声学表示（SAR），以取代人工制作的声学特征，通过在自动编码器预训练过程之前引入失真。从所提出的框架学习的声学表示被证明是抗失真相比，最常用的梅尔频谱图，通过客观和主观评价。摘要：In recent Text-to-Speech (TTS) systems, a neural vocoder often generates speech samples by solely conditioning on acoustic features predicted from an acoustic model. However, there are always distortions existing in the predicted acoustic features, compared to those of the groundtruth, especially in the common case of poor acoustic modeling due to low-quality training data. To overcome such limits, we propose a Self-supervised learning framework to learn an Anti-distortion acoustic Representation (SAR) to replace human-crafted acoustic features by introducing distortion prior to an auto-encoder pre-training process. The learned acoustic representation from the proposed framework is proved anti-distortion compared to the most commonly used mel-spectrogram through both objective and subjective evaluation.

【8】An Order-Complexity Model for Aesthetic Quality Assessment of Homophony Music Performance标题：同音音乐演奏审美质量评价的有序复杂性模型链接：http://arxiv.org/pdf/2304.11521v1

作者：Xin Jin,Wu Zhou,Jinyu Wang,Duo Xu,Yiqing Rong,Jialin Sun机构：Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China, Tongji University, Shanghai, China ,Tianjin Conservatory of Music, Tianjin, ChinaJournal-ref：AIART 2023 ICME Workshop摘要：虽然计算美学评价在很多领域都取得了一定的成果，但其在音乐表演方面的研究还有待探索。目前，主观评价仍是音乐美学研究的一种终极方法，但它会耗费大量的人力物力。此外，AI产生的音乐演奏仍然是机械的，单调的，缺乏美感。为了指导人工智能音乐表演的生成任务，提高人类表演者的表演效果，本文利用Birkhoff的美学度量，提出了一种客观的美的度量方法。本文的主要贡献如下：首先，我们提出了一种客观的审美评价方法来衡量音乐表演审美;其次，提出了10个基本音乐特征和4个审美音乐特征。实验结果表明，该方法具有较好的性能评估效果。摘要：Although computational aesthetics evaluation has made certain achievements in many fields, its research of music performance remains to be explored. At present, subjective evaluation is still a ultimate method of music aesthetics research, but it will consume a lot of human and material resources. In addition, the music performance generated by AI is still mechanical, monotonous and lacking in beauty. In order to guide the generation task of AI music performance, and to improve the performance effect of human performers, this paper uses Birkhoff's aesthetic measure to propose a method of objective measurement of beauty. The main contributions of this paper are as follows: Firstly, we put forward an objective aesthetic evaluation method to measure the music performance aesthetic; Secondly, we propose 10 basic music features and 4 aesthetic music features. Experiments show that our method performs well on performance assessment.

【9】Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey标题：递归神经网络和长短期记忆网络：教程和综述链接：http://arxiv.org/pdf/2304.11461v1

作者：Benyamin Ghojogh,Ali Ghodsi机构：Department of Statistics and Actuarial Science & David R. Cheriton School of Computer Science, Data Analytics Laboratory, University of Waterloo, Waterloo, ON, Canada备注：To appear as a part of an upcoming textbook on deep learning摘要：这是一篇关于递归神经网络（RNN），长短期记忆网络（LSTM）及其变体的教程论文。我们从一个动态系统和RNN的时间反向传播开始。然后，我们讨论了梯度消失和爆炸问题的长期依赖。我们解释接近身份的权重矩阵，长延迟，泄漏的单位，和回声状态网络解决这个问题。然后，我们介绍了LSTM门和单元，LSTM的历史和变体，以及门控递归单元（GRU）。最后，我们介绍了双向RNN，双向LSTM和语言模型嵌入（ELMo）网络，用于在两个方向上处理序列。摘要：This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), and their variants. We start with a dynamical system and backpropagation through time for RNN. Then, we discuss the problems of gradient vanishing and explosion in long-term dependencies. We explain close-to-identity weight matrix, long delays, leaky units, and echo state networks for solving this problem. Then, we introduce LSTM gates and cells, history and variants of LSTM, and Gated Recurrent Units (GRU). Finally, we introduce bidirectional RNN, bidirectional LSTM, and the Embeddings from Language Model (ELMo) network, for processing a sequence in both directions.

【10】Lightweight Toxicity Detection in Spoken Language: A Transformer-based Approach for Edge Devices标题：口语中的轻量级毒性检测：一种基于Transformer的边缘设备方法链接：http://arxiv.org/pdf/2304.11408v1

作者：Ahlam Husni Abu Nada,Siddique Latif,Junaid Qadir备注：Under Rewiew摘要：毒性是一种普遍的社会行为，涉及使用仇恨言论，攻击性语言，欺凌和辱骂性言论。虽然用于毒性检测的基于文本的方法是常见的，但在物理世界中处理语音信号的研究有限。在物理世界中检测毒性具有挑战性，因为将具有AI能力的计算机集成到环境中是困难的。我们提出了一个轻量级的Transformer模型的基础上wav2vec2.0，并优化它使用的技术，如量化和知识蒸馏。我们的模型使用多任务学习，平均宏观F1得分为90.3%，加权准确率为88%，在DeToxy-B和公共数据集上优于最先进的方法。我们的研究结果表明，量化减少了近4倍的模型大小和RAM的使用量的3.3%，只有1%的F1分数下降。知识提取使模型规模减少了3.7倍，RAM使用量减少了1.9倍，推理时间减少了2倍，但准确率降低了8%。结合这两种技术，模型大小减少了14.6倍，RAM使用量减少了约4.3倍，推理时间提高了两倍。我们的紧凑型模型是第一个端到端基于语音的毒性检测模型，该模型基于适合在物理空间中部署的轻量级Transformer模型。结果表明，其可行性的毒性检测边缘设备在现实世界的环境。摘要：Toxicity is a prevalent social behavior that involves the use of hate speech, offensive language, bullying, and abusive speech. While text-based approaches for toxicity detection are common, there is limited research on processing speech signals in the physical world. Detecting toxicity in the physical world is challenging due to the difficulty of integrating AI-capable computers into the environment. We propose a lightweight transformer model based on wav2vec2.0 and optimize it using techniques such as quantization and knowledge distillation. Our model uses multitask learning and achieves an average macro F1-score of 90.3 % and a weighted accuracy of 88 %, outperforming state-of-the-art methods on DeToxy-B and a public dataset. Our results show that quantization reduces the model size by almost 4 times and RAM usage by 3.3 %, with only a 1 % F1 score decrease. Knowledge distillation reduces the model size by 3.7 times, RAM usage by 1.9, and inference time by 2 times, but decreases accuracy by 8 %. Combining both techniques reduces the model size by 14.6 times and RAM usage by around 4.3 times, with a two-fold inference time improvement. Our compact model is the first end-to-end speech-based toxicity detection model based on a lightweight transformer model suitable for deployment in physical spaces. The results show its feasibility for toxicity detection on edge devices in real-world environments.

【11】DisImmersion in Mindfulness Meditation with a Wandering Voice Assistant标题：漫游的语音助手在正念冥想中的分散沉浸链接：http://arxiv.org/pdf/2304.11286v1

作者：Bonhee Ku,Tatsuya Itagaki,Katie Seaborn机构：Korea Advanced Institute of Science and Technology (KAIST), Daejon, South Korea, Tokyo Institute of Technology, Tokyo, JapanJournal-ref：In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23). Association for Computing Machinery, New York, NY, USA, Article 87, 1-6摘要：正念冥想是一种有效的帮助人们管理压力的方法。智能扬声器、智能手机和智能环境中基于语音的虚拟助理（VA）可以帮助人们通过引导体验进行正念冥想。然而，VA的常见固定位置实施例使得难以提供直观的支持。在这项工作中，我们探索了一个“流浪的声音”，这是共同定位与用户和“移动”的任务的新的体现。我们开发了一种嵌入瑜伽垫的多扬声器VA，它可以根据冥想体验改变身体的位置。我们在两个会话中进行了定性用户研究，将典型的固定智能扬声器与漫游VA实施例进行比较。从12人的采访中，厚厚的描述揭示了有时同时发生的沉浸和反沉浸的经历。我们提供了设计的影响“流浪的声音”和VA实施例，可以扩展到在其他情况下的指导任务的一个新的范例。摘要：Mindfulness meditation is a validated means of helping people manage stress. Voice-based virtual assistants (VAs) in smart speakers, smartphones, and smart environments can assist people in carrying out mindfulness meditation through guided experiences. However, the common fixed location embodiment of VAs makes it difficult to provide intuitive support. In this work, we explored the novel embodiment of a "wandering voice" that is co-located with the user and "moves" with the task. We developed a multi-speaker VA embedded in a yoga mat that changes location along the body according to the meditation experience. We conducted a qualitative user study in two sessions, comparing a typical fixed smart speaker to the wandering VA embodiment. Thick descriptions from interviews with twelve people revealed sometimes simultaneous experiences of immersion and dis-immersion. We offer design implications for "wandering voices" and a new paradigm for VA embodiment that may extend to guidance tasks in other contexts.

【12】Multi-channel Speech Separation Using Spatially Selective Deep Non-linear Filters标题：基于空间选择性深度非线性滤波器的多通道语音分离链接：http://arxiv.org/pdf/2304.12023v1

作者：Kristina Tesch,Timo Gerkmann机构：Senior Member, IEEE备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：在多个扬声器的多通道分离任务中，我们的目标是从混合物中恢复所有单独的语音信号。与依赖于语音信号的不同频谱-时间特性的单通道方法相比，多通道方法应当另外利用源的不同空间位置以用于更强大的分离，尤其是当源的数量增加时。为了增强多通道源分离场景中的空间处理，在这项工作中，我们提出了一种基于深度神经网络（DNN）的空间选择性滤波器（SSF），可以通过用目标方向初始化递归神经网络层来空间引导以提取感兴趣的说话者。我们比较建议的SSF与一个共同的端到端的直接分离（DS）的方法训练使用话语明智的排列不变的训练（PIT），它只隐含地学习执行空间滤波。我们表明，SSF有一个明显的优势，DS方法具有相同的底层网络架构时，有两个以上的扬声器的混合物，这可以归因于更好地利用空间信息。此外，我们发现SSF更好地推广到训练期间未看到的其他噪声源。摘要：In a multi-channel separation task with multiple speakers, we aim to recover all individual speech signals from the mixture. In contrast to single-channel approaches, which rely on the different spectro-temporal characteristics of the speech signals, multi-channel approaches should additionally utilize the different spatial locations of the sources for a more powerful separation especially when the number of sources increases. To enhance the spatial processing in a multi-channel source separation scenario, in this work, we propose a deep neural network (DNN) based spatially selective filter (SSF) that can be spatially steered to extract the speaker of interest by initializing a recurrent neural network layer with the target direction. We compare the proposed SSF with a common end-to-end direct separation (DS) approach trained using utterance-wise permutation invariant training (PIT), which only implicitly learns to perform spatial filtering. We show that the SSF has a clear advantage over a DS approach with the same underlying network architecture when there are more than two speakers in the mixture, which can be attributed to a better use of the spatial information. Furthermore, we find that the SSF generalizes much better to additional noise sources that were not seen during training.

【13】Self-regularised Minimum Latency Training for Streaming Transformer-based Speech Recognition标题：基于流Transformer的语音识别的自正则最小潜伏期训练链接：http://arxiv.org/pdf/2304.11985v1

作者：Mohan Li,Rama Doddipatla,Catalin Zorila机构：Cambridge Research Laboratory, Toshiba Europe Ltd, Cambridge, UK备注：5 pages, 2 figures, accepted at Interspeech2022摘要：本文提出了一种自正则化的最小延迟训练（SR-MLT）的方法，流转换器为基础的自动语音识别（ASR）系统。在先前的工作中，通过基于从常规ASR模型获得的硬比对截断在线注意力权重来优化延迟，而不考虑ASR准确性的潜在损失。相反，在这里，我们提出了一种策略，以获得对齐作为模型训练的一部分，而无需外部监督。由所提出的方法产生的对齐在训练数据上被动态地正则化，使得延迟减少不会导致ASR准确性的损失。SR-MLT被应用为预训练的Transformer模型上的微调步骤，该预训练的Transformer模型基于用于在线解码的单调分块注意（MoChA）或累积注意（CA）算法。在AIShell-1和Librispeech数据集上的ASR实验表明，当应用于体面的预训练MoChA或CA基线模型时，SR-MLT可以有效地减少延迟，相对增益范围从11.8%到39.5%。此外，我们还证明，在一定的准确性水平下，与使用外部硬对齐的监督相比，使用SR-MLT训练的模型可以实现更低的延迟。摘要：This paper proposes a self-regularised minimum latency training (SR-MLT) method for streaming Transformer-based automatic speech recognition (ASR) systems. In previous works, latency was optimised by truncating the online attention weights based on the hard alignments obtained from conventional ASR models, without taking into account the potential loss of ASR accuracy. On the contrary, here we present a strategy to obtain the alignments as a part of the model training without external supervision. The alignments produced by the proposed method are dynamically regularised on the training data, such that the latency reduction does not result in the loss of ASR accuracy. SR-MLT is applied as a fine-tuning step on the pre-trained Transformer models that are based on either monotonic chunkwise attention (MoChA) or cumulative attention (CA) algorithms for online decoding. ASR experiments on the AIShell-1 and Librispeech datasets show that when applied on a decent pre-trained MoChA or CA baseline model, SR-MLT can effectively reduce the latency with the relative gains ranging from 11.8% to 39.5%. Furthermore, we also demonstrate that under certain accuracy levels, the models trained with SR-MLT can achieve lower latency when compared to those supervised using external hard alignments.

【14】DiffVoice: Text-to-Speech with Latent Diffusion标题：DiffVoice：潜在扩散的文语转换链接：http://arxiv.org/pdf/2304.11750v1

作者：Zhijun Liu,Yiwei Guo,Kai Yu机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted to ICASSP2023摘要：在这项工作中，我们提出了DiffVoice，一种新的文本到语音模型的基础上潜在的扩散。我们建议首先将语音信号编码为音素率潜在表示，其中变分自动编码器通过对抗训练增强，然后用扩散模型联合建模持续时间和潜在表示。LJSpeech和LibriTTS数据集上的主观评估表明，我们的方法在自然性方面击败了最好的公开系统。通过采用最新的生成逆问题求解算法的扩散模型，DiffVoice实现了最先进的性能，在基于文本的语音编辑，和zero-shot适应。摘要：In this work, we present DiffVoice, a novel text-to-speech model based on latent diffusion. We propose to first encode speech signals into a phoneme-rate latent representation with a variational autoencoder enhanced by adversarial training, and then jointly model the duration and the latent representation with a diffusion model. Subjective evaluations on LJSpeech and LibriTTS datasets demonstrate that our method beats the best publicly available systems in naturalness. By adopting recent generative inverse problem solving algorithms for diffusion models, DiffVoice achieves the state-of-the-art performance in text-based speech editing, and zero-shot adaptation.

【15】Towards Controllable Audio Texture Morphing标题：走向可控的音频纹理变形链接：http://arxiv.org/pdf/2304.11648v1

作者：Chitralekha Gupta,Purnima Kamath,Yize Wei,Zhuoyao Li,Suranga Nanayakkara,Lonce WyseScepanovic,Zoran Radivojevic,Cecilia Mascolo机构：†National University of Singapore, Singapore, ‡Universitat Pompeu Fabra, Barcelona, Spain, ∗equal contribution备注：accepted to ICASSP 2023摘要：在本文中，我们提出了一种数据驱动的方法来训练一个生成对抗网络（GAN）的条件“软标签”从倒数第二层的音频分类器训练的目标集的音频纹理类。我们证明了这样的条件或控制向量之间的插值提供了所生成的音频纹理之间的平滑变形，并显示出类似或更好的音频纹理变形能力相比，国家的最先进的方法。所提出的方法导致在一个组织良好的潜在空间，产生新的音频输出，同时保持与语义的条件参数一致。这是朝着设计具有定制控件的生成音频模型的通用数据驱动方法迈出的一步，该定制控件能够遍历分布外区域以用于新颖的声音合成。摘要：In this paper, we propose a data-driven approach to train a Generative Adversarial Network (GAN) conditioned on "soft-labels" distilled from the penultimate layer of an audio classifier trained on a target set of audio texture classes. We demonstrate that interpolation between such conditions or control vectors provides smooth morphing between the generated audio textures, and shows similar or better audio texture morphing capability compared to the state-of-the-art methods. The proposed approach results in a well-organized latent space that generates novel audio outputs while remaining consistent with the semantics of the conditioning parameters. This is a step towards a general data-driven approach to designing generative audio models with customized controls capable of traversing out-of-distribution regions for novel sound synthesis.
eess.AS音频处理

【1】Multi-channel Speech Separation Using Spatially Selective Deep Non-linear Filters标题：基于空间选择性深度非线性滤波器的多通道语音分离链接：http://arxiv.org/pdf/2304.12023v1

*与cs.SD语音【12】为同一篇

作者：Kristina Tesch,Timo Gerkmann机构：Senior Member, IEEE备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：在多个扬声器的多通道分离任务中，我们的目标是从混合物中恢复所有单独的语音信号。与依赖于语音信号的不同频谱-时间特性的单通道方法相比，多通道方法应当另外利用源的不同空间位置以用于更强大的分离，尤其是当源的数量增加时。为了增强多通道源分离场景中的空间处理，在这项工作中，我们提出了一种基于深度神经网络（DNN）的空间选择性滤波器（SSF），可以通过用目标方向初始化递归神经网络层来空间引导以提取感兴趣的说话者。我们比较建议的SSF与一个共同的端到端的直接分离（DS）的方法训练使用话语明智的排列不变的训练（PIT），它只隐含地学习执行空间滤波。我们表明，SSF有一个明显的优势，DS方法具有相同的底层网络架构时，有两个以上的扬声器的混合物，这可以归因于更好地利用空间信息。此外，我们发现SSF更好地推广到训练期间未看到的其他噪声源。摘要：In a multi-channel separation task with multiple speakers, we aim to recover all individual speech signals from the mixture. In contrast to single-channel approaches, which rely on the different spectro-temporal characteristics of the speech signals, multi-channel approaches should additionally utilize the different spatial locations of the sources for a more powerful separation especially when the number of sources increases. To enhance the spatial processing in a multi-channel source separation scenario, in this work, we propose a deep neural network (DNN) based spatially selective filter (SSF) that can be spatially steered to extract the speaker of interest by initializing a recurrent neural network layer with the target direction. We compare the proposed SSF with a common end-to-end direct separation (DS) approach trained using utterance-wise permutation invariant training (PIT), which only implicitly learns to perform spatial filtering. We show that the SSF has a clear advantage over a DS approach with the same underlying network architecture when there are more than two speakers in the mixture, which can be attributed to a better use of the spatial information. Furthermore, we find that the SSF generalizes much better to additional noise sources that were not seen during training.

【2】Self-regularised Minimum Latency Training for Streaming Transformer-based Speech Recognition标题：基于流Transformer的语音识别的自正则最小潜伏期训练链接：http://arxiv.org/pdf/2304.11985v1

*与cs.SD语音【13】为同一篇

作者：Mohan Li,Rama Doddipatla,Catalin Zorila机构：Cambridge Research Laboratory, Toshiba Europe Ltd, Cambridge, UK备注：5 pages, 2 figures, accepted at Interspeech2022摘要：本文提出了一种自正则化的最小延迟训练（SR-MLT）的方法，流转换器为基础的自动语音识别（ASR）系统。在先前的工作中，通过基于从常规ASR模型获得的硬比对截断在线注意力权重来优化延迟，而不考虑ASR准确性的潜在损失。相反，在这里，我们提出了一种策略，以获得对齐作为模型训练的一部分，而无需外部监督。由所提出的方法产生的对齐在训练数据上被动态地正则化，使得延迟减少不会导致ASR准确性的损失。SR-MLT被应用为预训练的Transformer模型上的微调步骤，该预训练的Transformer模型基于用于在线解码的单调分块注意（MoChA）或累积注意（CA）算法。在AIShell-1和Librispeech数据集上的ASR实验表明，当应用于体面的预训练MoChA或CA基线模型时，SR-MLT可以有效地减少延迟，相对增益范围从11.8%到39.5%。此外，我们还证明，在一定的准确性水平下，与使用外部硬对齐的监督相比，使用SR-MLT训练的模型可以实现更低的延迟。摘要：This paper proposes a self-regularised minimum latency training (SR-MLT) method for streaming Transformer-based automatic speech recognition (ASR) systems. In previous works, latency was optimised by truncating the online attention weights based on the hard alignments obtained from conventional ASR models, without taking into account the potential loss of ASR accuracy. On the contrary, here we present a strategy to obtain the alignments as a part of the model training without external supervision. The alignments produced by the proposed method are dynamically regularised on the training data, such that the latency reduction does not result in the loss of ASR accuracy. SR-MLT is applied as a fine-tuning step on the pre-trained Transformer models that are based on either monotonic chunkwise attention (MoChA) or cumulative attention (CA) algorithms for online decoding. ASR experiments on the AIShell-1 and Librispeech datasets show that when applied on a decent pre-trained MoChA or CA baseline model, SR-MLT can effectively reduce the latency with the relative gains ranging from 11.8% to 39.5%. Furthermore, we also demonstrate that under certain accuracy levels, the models trained with SR-MLT can achieve lower latency when compared to those supervised using external hard alignments.

【3】DiffVoice: Text-to-Speech with Latent Diffusion标题：DiffVoice：潜在扩散的文语转换链接：http://arxiv.org/pdf/2304.11750v1

*与cs.SD语音【14】为同一篇

作者：Zhijun Liu,Yiwei Guo,Kai Yu机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted to ICASSP2023摘要：在这项工作中，我们提出了DiffVoice，一种新的文本到语音模型的基础上潜在的扩散。我们建议首先将语音信号编码为音素率潜在表示，其中变分自动编码器通过对抗训练增强，然后用扩散模型联合建模持续时间和潜在表示。LJSpeech和LibriTTS数据集上的主观评估表明，我们的方法在自然性方面击败了最好的公开系统。通过采用最新的生成逆问题求解算法的扩散模型，DiffVoice实现了最先进的性能，在基于文本的语音编辑，和zero-shot适应。摘要：In this work, we present DiffVoice, a novel text-to-speech model based on latent diffusion. We propose to first encode speech signals into a phoneme-rate latent representation with a variational autoencoder enhanced by adversarial training, and then jointly model the duration and the latent representation with a diffusion model. Subjective evaluations on LJSpeech and LibriTTS datasets demonstrate that our method beats the best publicly available systems in naturalness. By adopting recent generative inverse problem solving algorithms for diffusion models, DiffVoice achieves the state-of-the-art performance in text-based speech editing, and zero-shot adaptation.

【4】Towards Controllable Audio Texture Morphing标题：走向可控的音频纹理变形链接：http://arxiv.org/pdf/2304.11648v1

*与cs.SD语音【15】为同一篇

作者：Chitralekha Gupta,Purnima Kamath,Yize Wei,Zhuoyao Li,Suranga Nanayakkara,Lonce WyseScepanovic,Zoran Radivojevic,Cecilia Mascolo机构：†National University of Singapore, Singapore, ‡Universitat Pompeu Fabra, Barcelona, Spain, ∗equal contribution备注：accepted to ICASSP 2023摘要：在本文中，我们提出了一种数据驱动的方法来训练一个生成对抗网络（GAN）的条件“软标签”从倒数第二层的音频分类器训练的目标集的音频纹理类。我们证明了这样的条件或控制向量之间的插值提供了所生成的音频纹理之间的平滑变形，并显示出类似或更好的音频纹理变形能力相比，国家的最先进的方法。所提出的方法导致在一个组织良好的潜在空间，产生新的音频输出，同时保持与语义的条件参数一致。这是朝着设计具有定制控件的生成音频模型的通用数据驱动方法迈出的一步，该定制控件能够遍历分布外区域以用于新颖的声音合成。摘要：In this paper, we propose a data-driven approach to train a Generative Adversarial Network (GAN) conditioned on "soft-labels" distilled from the penultimate layer of an audio classifier trained on a target set of audio texture classes. We demonstrate that interpolation between such conditions or control vectors provides smooth morphing between the generated audio textures, and shows similar or better audio texture morphing capability compared to the state-of-the-art methods. The proposed approach results in a well-organized latent space that generates novel audio outputs while remaining consistent with the semantics of the conditioning parameters. This is a step towards a general data-driven approach to designing generative audio models with customized controls capable of traversing out-of-distribution regions for novel sound synthesis.

【5】A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognition标题：用于语音情感识别的预训练语音和音频嵌入的比较研究链接：http://arxiv.org/pdf/2304.11472v1

作者：Orchid Chetia Phukan,Arun Balaji Buduru,Rajesh Sharma机构：Dept. of CSE, IIIT Delhi, India, Institute of Computer Science, University of Tartu, Estonia摘要：预训练模型（PTM）在语音和音频领域已经显示出巨大的前景。利用这些模型的嵌入作为学习算法的输入，在各种下游任务中的应用程序。其中一个关键任务是语音情感识别（SER），其具有广泛的应用，包括客户呼叫的动态分析、心理健康评估和个性化语言学习。PTM嵌入有助于提高SER，然而，这些PTM嵌入的综合比较考虑了多个方面，如嵌入模型架构，用于预训练的数据，以及遵循的预训练过程。对PTM嵌入的全面比较将有助于更快、更有效地开发模型，并使其能够在现实世界的场景中部署。在这项工作中，我们利用这一研究差距，并进行了比较分析的嵌入提取的八个语音和音频PTM（wav 2 vec 2.0，data 2 vec，wavLM，UniSpeech-SAT，wav 2clip，YAMNet，x向量，ECAPA）。我们进行了广泛的实证分析与四个语音情感数据集（CREMA-D，TESS，SAVEE，Emo-DB）通过训练三种算法（XGBoost，随机森林，FCN）的派生嵌入。我们的研究结果表明，最好的性能是由算法训练的嵌入来自PTM训练的说话人识别，其次是wav 2clip和UniSpeech-SAT。这可以表明，来自说话者识别PTM的嵌入的最佳性能最有可能是由于模型在其说话者识别训练期间占用了关于许多语音特征（诸如音调、口音、音高等）的信息。这项工作的见解将有助于未来的研究在他们的选择嵌入的应用程序相关的SER。摘要：Pre-trained models (PTMs) have shown great promise in the speech and audio domain. Embeddings leveraged from these models serve as inputs for learning algorithms with applications in various downstream tasks. One such crucial task is Speech Emotion Recognition (SER) which has a wide range of applications, including dynamic analysis of customer calls, mental health assessment, and personalized language learning. PTM embeddings have helped advance SER, however, a comprehensive comparison of these PTM embeddings that consider multiple facets such as embedding model architecture, data used for pre-training, and the pre-training procedure being followed is missing. A thorough comparison of PTM embeddings will aid in the faster and more efficient development of models and enable their deployment in real-world scenarios. In this work, we exploit this research gap and perform a comparative analysis of embeddings extracted from eight speech and audio PTMs (wav2vec 2.0, data2vec, wavLM, UniSpeech-SAT, wav2clip, YAMNet, x-vector, ECAPA). We perform an extensive empirical analysis with four speech emotion datasets (CREMA-D, TESS, SAVEE, Emo-DB) by training three algorithms (XGBoost, Random Forest, FCN) on the derived embeddings. The results of our study indicate that the best performance is achieved by algorithms trained on embeddings derived from PTMs trained for speaker recognition followed by wav2clip and UniSpeech-SAT. This can relay that the top performance by embeddings from speaker recognition PTMs is most likely due to the model taking up information about numerous speech features such as tone, accent, pitch, and so on during its speaker recognition training. Insights from this work will assist future studies in their selection of embeddings for applications related to SER.

【6】Pre-Training Strategies Using Contrastive Learning and Playlist Information for Music Classification and Similarity标题：基于对比学习和播放列表信息的音乐分类相似度预训练策略链接：http://arxiv.org/pdf/2304.12257v1

【1】为同一篇

作者：Pablo Alonso-Jiménez,Xavier Favory,Hadrien Foroughmand,Grigoris Bourdalas,Xavier Serra,Thomas Lidy,Dmitry Bogdanov机构：Utopia Music, Switzerland, Music Technology Group, Universitat Pompeu Fabra, Spain备注：Accepted at the 2023 International Conference on Acoustics, Speech, and Signal Processing (ICASSP'23)摘要：在这项工作中，我们研究了一种方法，依赖于对比学习和音乐元数据作为一个薄弱的监督来源，以训练音乐表示模型。最近的研究表明，对比学习可以与编辑元数据（例如，艺术家或专辑名称）来学习对不同分类任务有用的音频表示。在本文中，我们扩展这个想法，使用播放列表数据作为音乐相似性信息的来源，并研究三种方法来生成锚和积极的轨道对。我们通过微调音乐多标签分类任务（流派，情绪和乐器标记）和音乐相似性的预训练模型来评估这些方法。我们发现，通过依赖于播放列表中的同现创建锚和积极的轨道对提供了更好的音乐相似性和竞争力的分类结果相比，从相同的艺术家在以前的作品中选择的轨道。此外，我们基于播放列表的最佳预训练方法为大多数数据集提供了卓越的分类性能。摘要：In this work, we investigate an approach that relies on contrastive learning and music metadata as a weak source of supervision to train music representation models. Recent studies show that contrastive learning can be used with editorial metadata (e.g., artist or album name) to learn audio representations that are useful for different classification tasks. In this paper, we extend this idea to using playlist data as a source of music similarity information and investigate three approaches to generate anchor and positive track pairs. We evaluate these approaches by fine-tuning the pre-trained models for music multi-label classification tasks (genre, mood, and instrument tagging) and music similarity. We find that creating anchor and positive track pairs by relying on co-occurrences in playlists provides better music similarity and competitive classification results compared to choosing tracks from the same artist as in previous works. Additionally, our best pre-training approach based on playlists provides superior classification performance for most datasets.

【7】Small-footprint slimmable networks for keyword spotting标题：用于关键词识别的占地面积小的可精简网络链接：http://arxiv.org/pdf/2304.12183v1

*与cs.SD语音【2】为同一篇

作者：Zuhaib Akhtar,Mohammad Omar Khursheed,Dongsu Du,Yuzong Liu机构：Alexa Perceptual Technologies, Amazon, New York University, NY摘要：在这项工作中，我们提出了应用于小足迹关键字定位问题的Slimmable神经网络。我们表明，可瘦身的神经网络允许我们从卷积神经网络和Transformers中创建超网，从中可以提取不同大小的子网络。我们在内部Alexa数据和Google语音命令上展示了这些模型的有用性，并将我们的努力集中在设备上用例的模型上，将我们的参数限制在250k以内。我们表明，可精简模型可以匹配（在某些情况下，优于）从头开始训练的模型。因此，可瘦身神经网络是一类特别有用的模型，当相同的功能要在不同的存储器和计算预算下复制时，具有不同的精度要求。摘要：In this work, we present Slimmable Neural Networks applied to the problem of small-footprint keyword spotting. We show that slimmable neural networks allow us to create super-nets from Convolutioanl Neural Networks and Transformers, from which sub-networks of different sizes can be extracted. We demonstrate the usefulness of these models on in-house Alexa data and Google Speech Commands, and focus our efforts on models for the on-device use case, limiting ourselves to less than 250k parameters. We show that slimmable models can match (and in some cases, outperform) models trained from scratch. Slimmable neural networks are therefore a class of models particularly useful when the same functionality is to be replicated at different memory and compute budgets, with different accuracy requirements.

【8】Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models标题：基于自监督学习模型的深度视听演唱配音链接：http://arxiv.org/pdf/2304.12082v1

*与cs.SD语音【3】为同一篇

作者：Xiangming Gu,Wei Zeng,Jianan Zhang,Longshen Ou,Ye Wang机构：Ye Wang are withSchool of Computing, National University of Singapore摘要：歌唱语音转录将录制的歌唱音频转换为乐谱。声音污染（如伴奏）和注释数据的缺乏使歌唱声音转录成为一项极其困难的任务。我们采取两种方法来应对上述挑战：1）引入多模态学习用于歌唱声音转录以及新的多模态歌唱数据集N20 EMv 2，通过利用视频信息（嘴唇运动来预测音符的开始 结束）来增强噪声鲁棒性，以及2）将语音域的自监督学习模型适应于歌唱声音转录任务，在保留预训练特征的同时显着减少注释数据要求。我们建立了一个基于自监督学习的纯音频歌唱语音转录系统，它不仅优于当前最先进的技术作为一个强大的基线，但也很好地推广到域外歌唱数据。然后，我们开发了一个基于自我监督学习的纯视频歌唱语音转录系统，检测注意的起始和偏移的准确率约为80%。最后，基于上述两个系统提取的强大的声学和视觉表示以及特征融合设计，我们创建了一个视听歌唱语音转录系统，提高了噪声鲁棒性显着不同的声学环境下相比，只有音频系统。摘要：Singing voice transcription converts recorded singing audio to musical notation. Sound contamination (such as accompaniment) and lack of annotated data make singing voice transcription an extremely difficult task. We take two approaches to tackle the above challenges: 1) introducing multimodal learning for singing voice transcription together with a new multimodal singing dataset, N20EMv2, enhancing noise robustness by utilizing video information (lip movements to predict the onset offset of notes), and 2) adapting self-supervised learning models from the speech domain to the singing voice transcription task, significantly reducing annotated data requirements while preserving pretrained features. We build a self-supervised learning based audio-only singing voice transcription system, which not only outperforms current state-of-the-art technologies as a strong baseline, but also generalizes well to out-of-domain singing data. We then develop a self-supervised learning based video-only singing voice transcription system that detects note onsets and offsets with an accuracy of about 80 %. Finally, based on the powerful acoustic and visual representations extracted by the above two systems as well as the feature fusion design, we create an audio-visual singing voice transcription system that improves the noise robustness significantly under different acoustic environments compared to the audio-only systems.

【9】Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model标题：基于自监督语音表示模型的零发声文语合成链接：http://arxiv.org/pdf/2304.11976v1

*与cs.SD语音【4】为同一篇

作者：Kenichi Fujita,Takanori Ashihara,Hiroki Kanagawa,Takafumi Moriya,Yusuke Ijima机构：NTT Corporation, Japan备注：5 pages,3 figures, Accepted to IEEE ICASSP 2023 workshop Self-supervision in Audio, Speech and Beyond摘要：本文提出了一种zero-shot文本到语音（TTS）的条件下，通过自监督学习（SSL）获得的自监督语音表示模型。具有来自X向量或全局样式令牌的嵌入向量的常规方法在再现未看见的说话者的说话者特性方面仍然具有差距。所提出的方法的一个新颖点是直接使用SSL模型从用大量数据训练的语音表示中获得嵌入向量。我们还引入了单独的条件的声学特征和音素持续时间预测器，以获得基于节奏的扬声器特性和基于声学特征的之间的解开嵌入。解开的嵌入将使我们能够实现更好的再现性能为看不见的发言者和节奏转移的不同的讲话。客观和主观评价表明，该方法可以合成语音的相似性提高，实现语音节奏转移。摘要：This paper proposes a zero-shot text-to-speech (TTS) conditioned by a self-supervised speech-representation model acquired through self-supervised learning (SSL). Conventional methods with embedding vectors from x-vector or global style tokens still have a gap in reproducing the speaker characteristics of unseen speakers. A novel point of the proposed method is the direct use of the SSL model to obtain embedding vectors from speech representations trained with a large amount of data. We also introduce the separate conditioning of acoustic features and a phoneme duration predictor to obtain the disentangled embeddings between rhythm-based speaker characteristics and acoustic-feature-based ones. The disentangled embeddings will enable us to achieve better reproduction performance for unseen speakers and rhythm transfer conditioned by different speeches. Objective and subjective evaluations showed that the proposed method can synthesize speech with improved similarity and achieve speech-rhythm transfer.

【10】Advancing underwater acoustic target recognition via adaptive data pruning and smoothness-inducing regularization标题：基于自适应数据剪枝和平滑诱导正则化的水声目标识别链接：http://arxiv.org/pdf/2304.11907v1

*与cs.SD语音【5】为同一篇

作者：Yuan Xie,Tianyu Chen,Ji Xu机构：Key laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences, BDBC, Beihang University, China ,SKLSDE, Beihang University摘要：舰船辐射信号的水声识别具有识别非视距目标的能力，具有很高的实际应用价值。但由于数据采集的困难，采集到的信号数量稀少，且主要由机械周期性噪声组成。根据实验，我们观察到，周期信号的重复性导致双下降现象，这表明一个显着的局部偏向重复的样本。为了解决这个问题，我们提出了一种基于交叉熵的策略，以修剪训练数据中过于相似的片段。此外，为了补偿训练数据的减少，我们生成噪声样本，并应用基于KL散度的平滑诱导正则化来减轻过拟合。实验表明，我们提出的数据修剪和正则化策略可以带来稳定的利益，我们的框架显着优于国家的最先进的低资源的情况下。摘要：Underwater acoustic recognition for ship-radiated signals has high practical application value due to the ability to recognize non-line-of-sight targets. However, due to the difficulty of data acquisition, the collected signals are scarce in quantity and mainly composed of mechanical periodic noise. According to the experiments, we observe that the repeatability of periodic signals leads to a double-descent phenomenon, which indicates a significant local bias toward repeated samples. To address this issue, we propose a strategy based on cross-entropy to prune excessively similar segments in training data. Furthermore, to compensate for the reduction of training data, we generate noisy samples and apply smoothness-inducing regularization based on KL divergence to mitigate overfitting. Experiments show that our proposed data pruning and regularization strategy can bring stable benefits and our framework significantly outperforms the state-of-the-art in low-resource scenarios.

【11】Sound-based drone fault classification using multitask learning标题：基于多任务学习的声学无人机故障分类链接：http://arxiv.org/pdf/2304.11708v1

*与cs.SD语音【6】为同一篇

作者：Wonjun Yi,Jung-Woo Choi,Jae-Woo Lee机构：Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea备注：Accepted at 29th International Congress on Sound and Vibration (ICSV29). Dataset available: this https URL摘要：无人机已被用于各种目的，包括军事应用，空中摄影和农药喷洒。然而，无人机容易受到外部干扰，并且螺旋桨和电机容易发生故障。为了提高无人机操作的安全性，应该实时检测无人机的机械故障。本文提出了一种基于声音的深度神经网络（DNN）故障分类器和无人机声音数据集。该数据集是通过在消声室中从安装在三个不同无人机上的麦克风收集无人机的操作声音来构建的。该数据集包括无人机的各种操作条件，例如飞行方向（前，后，右，左，顺时针，逆时针）以及螺旋桨和电机上的故障。然后，无人机的声音与大学校园内五个不同地点记录的噪音混合，信噪比（SNR）从10 dB到15 dB不等。使用所获得的数据集，我们训练了DNN分类器1DCNN-ResNet，该分类器从短时间输入波形中分类机械故障的类型及其位置。我们采用多任务学习（MTL），并将方向分类任务作为辅助任务，使分类器学习更一般的音频特征。对未知数据的测试表明，所提出的多任务模型可以成功地对无人机中的故障进行分类，即使训练数据较少，也优于单任务模型。摘要：The drone has been used for various purposes, including military applications, aerial photography, and pesticide spraying. However, the drone is vulnerable to external disturbances, and malfunction in propellers and motors can easily occur. To improve the safety of drone operations, one should detect the mechanical faults of drones in real-time. This paper proposes a sound-based deep neural network (DNN) fault classifier and drone sound dataset. The dataset was constructed by collecting the operating sounds of drones from microphones mounted on three different drones in an anechoic chamber. The dataset includes various operating conditions of drones, such as flight directions (front, back, right, left, clockwise, counterclockwise) and faults on propellers and motors. The drone sounds were then mixed with noises recorded in five different spots on the university campus, with a signal-to-noise ratio (SNR) varying from 10 dB to 15 dB. Using the acquired dataset, we train a DNN classifier, 1DCNN-ResNet, that classifies the types of mechanical faults and their locations from short-time input waveforms. We employ multitask learning (MTL) and incorporate the direction classification task as an auxiliary task to make the classifier learn more general audio features. The test over unseen data reveals that the proposed multitask model can successfully classify faults in drones and outperforms single-task models even with less training data.

【12】SAR: Self-Supervised Anti-Distortion Representation for End-To-End Speech Model标题：SAR：端到端语音模型的自监督抗失真表示链接：http://arxiv.org/pdf/2304.11547v1

*与cs.SD语音【7】为同一篇

作者：Jianzong Wang,Xulong Zhang,Haobin Tang,Aolan Sun,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd., University of Science and Technology of China备注：Accepted by IJCNN2023. 2023 International Joint Conference on Neural Networks (IJCNN2023)摘要：在最近的文本到语音（TTS）系统中，神经声码器通常通过仅调节从声学模型预测的声学特征来生成语音样本。然而，与地面实况的那些相比，预测的声学特征中总是存在失真，特别是在由于低质量训练数据而导致的声学建模不佳的常见情况下。为了克服这些限制，我们提出了一个自监督学习框架来学习抗失真声学表示（SAR），以取代人工制作的声学特征，通过在自动编码器预训练过程之前引入失真。从所提出的框架学习的声学表示被证明是抗失真相比，最常用的梅尔频谱图，通过客观和主观评价。摘要：In recent Text-to-Speech (TTS) systems, a neural vocoder often generates speech samples by solely conditioning on acoustic features predicted from an acoustic model. However, there are always distortions existing in the predicted acoustic features, compared to those of the groundtruth, especially in the common case of poor acoustic modeling due to low-quality training data. To overcome such limits, we propose a Self-supervised learning framework to learn an Anti-distortion acoustic Representation (SAR) to replace human-crafted acoustic features by introducing distortion prior to an auto-encoder pre-training process. The learned acoustic representation from the proposed framework is proved anti-distortion compared to the most commonly used mel-spectrogram through both objective and subjective evaluation.

【13】An Order-Complexity Model for Aesthetic Quality Assessment of Homophony Music Performance标题：同音音乐演奏审美质量评价的有序复杂性模型链接：http://arxiv.org/pdf/2304.11521v1

*与cs.SD语音【8】为同一篇

作者：Xin Jin,Wu Zhou,Jinyu Wang,Duo Xu,Yiqing Rong,Jialin Sun机构：Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China, Tongji University, Shanghai, China ,Tianjin Conservatory of Music, Tianjin, ChinaJournal-ref：AIART 2023 ICME Workshop摘要：虽然计算美学评价在很多领域都取得了一定的成果，但其在音乐表演方面的研究还有待探索。目前，主观评价仍是音乐美学研究的一种终极方法，但它会耗费大量的人力物力。此外，AI产生的音乐演奏仍然是机械的，单调的，缺乏美感。为了指导人工智能音乐表演的生成任务，提高人类表演者的表演效果，本文利用Birkhoff的美学度量，提出了一种客观的美的度量方法。本文的主要贡献如下：首先，我们提出了一种客观的审美评价方法来衡量音乐表演审美;其次，提出了10个基本音乐特征和4个审美音乐特征。实验结果表明，该方法具有较好的性能评估效果。摘要：Although computational aesthetics evaluation has made certain achievements in many fields, its research of music performance remains to be explored. At present, subjective evaluation is still a ultimate method of music aesthetics research, but it will consume a lot of human and material resources. In addition, the music performance generated by AI is still mechanical, monotonous and lacking in beauty. In order to guide the generation task of AI music performance, and to improve the performance effect of human performers, this paper uses Birkhoff's aesthetic measure to propose a method of objective measurement of beauty. The main contributions of this paper are as follows: Firstly, we put forward an objective aesthetic evaluation method to measure the music performance aesthetic; Secondly, we propose 10 basic music features and 4 aesthetic music features. Experiments show that our method performs well on performance assessment.

【14】Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey标题：递归神经网络和长短期记忆网络：教程和综述链接：http://arxiv.org/pdf/2304.11461v1

*与cs.SD语音【9】为同一篇

作者：Benyamin Ghojogh,Ali Ghodsi机构：Department of Statistics and Actuarial Science & David R. Cheriton School of Computer Science, Data Analytics Laboratory, University of Waterloo, Waterloo, ON, Canada备注：To appear as a part of an upcoming textbook on deep learning摘要：这是一篇关于递归神经网络（RNN），长短期记忆网络（LSTM）及其变体的教程论文。我们从一个动态系统和RNN的时间反向传播开始。然后，我们讨论了梯度消失和爆炸问题的长期依赖。我们解释接近身份的权重矩阵，长延迟，泄漏的单位，和回声状态网络解决这个问题。然后，我们介绍了LSTM门和单元，LSTM的历史和变体，以及门控递归单元（GRU）。最后，我们介绍了双向RNN，双向LSTM和语言模型嵌入（ELMo）网络，用于在两个方向上处理序列。摘要：This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), and their variants. We start with a dynamical system and backpropagation through time for RNN. Then, we discuss the problems of gradient vanishing and explosion in long-term dependencies. We explain close-to-identity weight matrix, long delays, leaky units, and echo state networks for solving this problem. Then, we introduce LSTM gates and cells, history and variants of LSTM, and Gated Recurrent Units (GRU). Finally, we introduce bidirectional RNN, bidirectional LSTM, and the Embeddings from Language Model (ELMo) network, for processing a sequence in both directions.

【15】Lightweight Toxicity Detection in Spoken Language: A Transformer-based Approach for Edge Devices标题：口语中的轻量级毒性检测：一种基于Transformer的边缘设备方法链接：http://arxiv.org/pdf/2304.11408v1

*与cs.SD语音【10】为同一篇

作者：Ahlam Husni Abu Nada,Siddique Latif,Junaid Qadir备注：Under Rewiew摘要：毒性是一种普遍的社会行为，涉及使用仇恨言论，攻击性语言，欺凌和辱骂性言论。虽然用于毒性检测的基于文本的方法是常见的，但在物理世界中处理语音信号的研究有限。在物理世界中检测毒性具有挑战性，因为将具有AI能力的计算机集成到环境中是困难的。我们提出了一个轻量级的Transformer模型的基础上wav2vec2.0，并优化它使用的技术，如量化和知识蒸馏。我们的模型使用多任务学习，平均宏观F1得分为90.3%，加权准确率为88%，在DeToxy-B和公共数据集上优于最先进的方法。我们的研究结果表明，量化减少了近4倍的模型大小和RAM的使用量的3.3%，只有1%的F1分数下降。知识提取使模型规模减少了3.7倍，RAM使用量减少了1.9倍，推理时间减少了2倍，但准确率降低了8%。结合这两种技术，模型大小减少了14.6倍，RAM使用量减少了约4.3倍，推理时间提高了两倍。我们的紧凑型模型是第一个端到端基于语音的毒性检测模型，该模型基于适合在物理空间中部署的轻量级Transformer模型。结果表明，其可行性的毒性检测边缘设备在现实世界的环境。摘要：Toxicity is a prevalent social behavior that involves the use of hate speech, offensive language, bullying, and abusive speech. While text-based approaches for toxicity detection are common, there is limited research on processing speech signals in the physical world. Detecting toxicity in the physical world is challenging due to the difficulty of integrating AI-capable computers into the environment. We propose a lightweight transformer model based on wav2vec2.0 and optimize it using techniques such as quantization and knowledge distillation. Our model uses multitask learning and achieves an average macro F1-score of 90.3 % and a weighted accuracy of 88 %, outperforming state-of-the-art methods on DeToxy-B and a public dataset. Our results show that quantization reduces the model size by almost 4 times and RAM usage by 3.3 %, with only a 1 % F1 score decrease. Knowledge distillation reduces the model size by 3.7 times, RAM usage by 1.9, and inference time by 2 times, but decreases accuracy by 8 %. Combining both techniques reduces the model size by 14.6 times and RAM usage by around 4.3 times, with a two-fold inference time improvement. Our compact model is the first end-to-end speech-based toxicity detection model based on a lightweight transformer model suitable for deployment in physical spaces. The results show its feasibility for toxicity detection on edge devices in real-world environments.

【16】DisImmersion in Mindfulness Meditation with a Wandering Voice Assistant标题：漫游的语音助手在正念冥想中的分散沉浸链接：http://arxiv.org/pdf/2304.11286v1

*与cs.SD语音【11】为同一篇

作者：Bonhee Ku,Tatsuya Itagaki,Katie Seaborn机构：Korea Advanced Institute of Science and Technology (KAIST), Daejon, South Korea, Tokyo Institute of Technology, Tokyo, JapanJournal-ref：In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23). Association for Computing Machinery, New York, NY, USA, Article 87, 1-6摘要：正念冥想是一种有效的帮助人们管理压力的方法。智能扬声器、智能手机和智能环境中基于语音的虚拟助理（VA）可以帮助人们通过引导体验进行正念冥想。然而，VA的常见固定位置实施例使得难以提供直观的支持。在这项工作中，我们探索了一个“流浪的声音”，这是共同定位与用户和“移动”的任务的新的体现。我们开发了一种嵌入瑜伽垫的多扬声器VA，它可以根据冥想体验改变身体的位置。我们在两个会话中进行了定性用户研究，将典型的固定智能扬声器与漫游VA实施例进行比较。从12人的采访中，厚厚的描述揭示了有时同时发生的沉浸和反沉浸的经历。我们提供了设计的影响“流浪的声音”和VA实施例，可以扩展到在其他情况下的指导任务的一个新的范例。摘要：Mindfulness meditation is a validated means of helping people manage stress. Voice-based virtual assistants (VAs) in smart speakers, smartphones, and smart environments can assist people in carrying out mindfulness meditation through guided experiences. However, the common fixed location embodiment of VAs makes it difficult to provide intuitive support. In this work, we explored the novel embodiment of a "wandering voice" that is co-located with the user and "moves" with the task. We developed a multi-speaker VA embedded in a yoga mat that changes location along the body according to the meditation experience. We conducted a qualitative user study in two sessions, comparing a typical fixed smart speaker to the wandering VA embodiment. Thick descriptions from interviews with twelve people revealed sometimes simultaneous experiences of immersion and dis-immersion. We offer design implications for "wandering voices" and a new paradigm for VA embodiment that may extend to guidance tasks in other contexts.

机器翻译，仅供参考