今日论文合集：cs.SD语音15篇，eess.AS音频处理16篇。
cs.SD语音
【1】 Integrating Pretrained ASR and LM to Perform Sequence Generation for  Spoken Language Understanding标题：集成预先训练的ASR和LM进行口语理解序列生成链接：https://arxiv.org/abs/2307.11005
作者：Siddhant Arora,Hayato Futami,Yosuke Kashiwagi,Emiru Tsunoo,Brian Yan,Shinji Watanabe机构：Carnegie Mellon University, U.S.A., Sony Group Corporation, Japan备注：Accepted at INTERSPEECH 2023摘要：将预训练语音识别（ASR）和语言模型（LM）集成到SLU框架中的兴趣越来越大。然而，先前的方法经常与预训练模型之间的词汇不匹配作斗争，并且LM不能直接使用，因为它们与其NLU公式不同。在这项研究中，我们提出了一个三通端到端（E2E）SLU系统，有效地将ASR和LM子网络到SLU制定序列生成任务。在第一遍中，我们的体系结构使用ASR子网络预测ASR成绩单。随后是LM子网络，其进行初始SLU预测。最后，在第三遍中，审议子网络对来自ASR和LM子网络的表示进行条件化以做出最终预测。我们提出的三遍SLU系统显示出改进的性能级联和E2E SLU模型上的两个基准SLU数据集，SLURP和SLUE，特别是在声学上具有挑战性的话语。摘要：There has been an increased interest in the integration of pretrained speech recognition (ASR) and language models (LM) into the SLU framework. However, prior methods often struggle with a vocabulary mismatch between pretrained models, and LM cannot be directly utilized as they diverge from its NLU formulation. In this study, we propose a three-pass end-to-end (E2E) SLU system that effectively integrates ASR and LM subnetworks into the SLU formulation for sequence generation tasks. In the first pass, our architecture predicts ASR transcripts using the ASR subnetwork. This is followed by the LM subnetwork, which makes an initial SLU prediction. Finally, in the third pass, the deliberation subnetwork conditions on representations from the ASR and LM subnetworks to make the final prediction. Our proposed three-pass SLU system shows improved performance over cascaded and E2E SLU models on two benchmark SLU datasets, SLURP and SLUE, especially on acoustically challenging utterances.
【2】 Progressive distillation diffusion for raw music generation标题：用于原创音乐生成的逐级蒸馏扩散链接：https://arxiv.org/abs/2307.10994
作者：Svetlana Pavlova机构：Skolkovo Institute of Science and Technology, Moscow, Russia备注：9 pages摘要：本文旨在将一种新的深度学习方法应用于生成原始音频文件的任务。它基于扩散模型，这是一种新的深度生成模型。这种新型的方法最近显示出出色的结果与图像生成。计算机视觉社区对这些模型给予了很多关注。另一方面，对于其他类型的应用，例如波形域中的音乐生成，实际上很少给出。  本文实现了应用于音乐的无条件生成模型：使用1D U-Net的渐进蒸馏扩散。然后，不同的扩散参数和它们的值在一个完整的结果的比较。通过这项工作实现的方法的一个很大的优点是，该模型能够处理渐进的音频处理和生成，使用从1声道128 x 384到3声道128 x 128梅尔频谱图的转换和循环生成。在不同的自我收集的数据集之间实现的实证比较。摘要：This paper aims to apply a new deep learning approach to the task of generating raw audio files. It is based on diffusion models, a recent type of deep generative model. This new type of method has recently shown outstanding results with image generation. A lot of focus has been given to those models by the computer vision community. On the other hand, really few have been given for other types of applications such as music generation in waveform domain.  In this paper the model for unconditional generating applied to music is implemented: Progressive distillation diffusion with 1D U-Net. Then, a comparison of different parameters of diffusion and their value in a full result is presented. One big advantage of the methods implemented through this work is the fact that the model is able to deal with progressing audio processing and generating , using transformation from 1-channel 128 x 384 to 3-channel 128 x 128 mel-spectrograms and looped generation. The empirical comparisons are realized across different self-collected datasets.
【3】 MASR: Metadata Aware Speech Representation标题：MASR：元数据感知语音表示链接：https://arxiv.org/abs/2307.10982
作者：Anjali Raj,Shikhar Bharadwaj,Sriram Ganapathy,Min Ma,Shikhar Vashishth机构：Google Research摘要：近年来，语音表示学习主要被构建为一个自我监督学习（SSL）任务，仅使用原始音频信号，而忽略了边信息，这是经常可用于给定的语音记录。在本文中，我们提出了MASR，元数据感知语音表示学习框架，它解决了上述限制。MASR允许包含多个外部知识源以增强元数据信息的利用。外部知识源被纳入的样本级成对的相似性矩阵的形式是有用的硬挖掘损失。MASR框架的一个关键优势是它可以与任何SSL方法组合。使用MASR表示，我们进行评估的几个下游任务，如语言识别，语音识别和其他非语义任务，如扬声器和情感识别。在这些实验中，我们说明了显着的性能改进的MASR比其他已建立的基准。我们对语言识别任务进行了详细的分析，以提供有关建议的损失函数如何使表示分离密切相关的语言的见解。摘要：In the recent years, speech representation learning is constructed primarily as a self-supervised learning (SSL) task, using the raw audio signal alone, while ignoring the side-information that is often available for a given speech recording. In this paper, we propose MASR, a Metadata Aware Speech Representation learning framework, which addresses the aforementioned limitations. MASR enables the inclusion of multiple external knowledge sources to enhance the utilization of meta-data information. The external knowledge sources are incorporated in the form of sample-level pair-wise similarity matrices that are useful in a hard-mining loss. A key advantage of the MASR framework is that it can be combined with any choice of SSL method. Using MASR representations, we perform evaluations on several downstream tasks such as language identification, speech recognition and other non-semantic tasks such as speaker and emotion recognition. In these experiments, we illustrate significant performance improvements for the MASR over other established benchmarks. We perform a detailed analysis on the language identification task to provide insights on how the proposed loss function enables the representations to separate closely related languages.
【4】 Cross-Corpus Multilingual Speech Emotion Recognition: Amharic vs. Other  Languages标题：跨语料库多语言语音情感识别：阿姆哈拉语与其他语言链接：https://arxiv.org/abs/2307.10814
作者：Ephrem Afele Retta,Richard Sutcliffe,Jabar Mahmood,Michael Abebe Berwo,Eiad Almekhlafi,Sajjad Ahmed Khan,Shehzad Ashraf Chaudhry,Mustafa Mhamed,Jun Feng机构：School of Information Science and Technology, School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe Park, Colchester CO,SQ, UK, Northwest University, Xi’an , China, University of Sialkot, Punjab, Pakistan备注：16 pages, 9 tables, 5 figures摘要：在传统的语音情感识别（SER）任务中，针对给定语言的分类器是在针对该相同语言的预先存在的数据集上训练的。然而，在不存在用于语言的训练数据的情况下，可以替代地使用来自其他语言的数据。我们尝试跨语言和多语言SER，与阿姆哈拉语，英语，德语和URDU合作。对于阿姆哈拉语，我们使用我们自己的公开可用的阿姆哈拉语语音情感数据集（ASED）。对于英语，德语和乌尔都语，我们使用现有的RAVDESS，EMO-DB和URDU数据集。我们遵循之前的研究，将所有数据集的标签映射到两个类别，正面和负面。因此，我们可以直接比较不同语言的性能，并结合语言进行训练和测试。在实验1中，单语SER试验进行了使用三个分类器，AlexNet，VGGE（VGG的一个拟议的变体），和ResNet 50。ASED和RAVDESS的三种模型的平均结果非常相似，这表明阿姆哈拉语和英语SER同样困难。同样，德语SER更难，乌尔都语SER更容易。在实验2中，我们在一种语言上进行训练，并在另一种语言上进行测试，每对都是双向的：阿姆哈拉<->德语、阿姆哈拉<->英语和阿姆哈拉<->乌尔都语。以阿姆哈拉语为目标的结果表明，使用英语或德语作为源将得到最好的结果。在实验3中，我们训练了几种非阿姆哈拉语语言，然后测试阿姆哈拉语。获得的最佳准确度比实验2中的最佳准确度高出几个百分点，这表明使用两种或三种非阿姆哈拉语进行训练时可以获得比仅使用一种非阿姆哈拉语时更好的结果。总的来说，结果表明，跨语言和多语言的培训可以是一个有效的策略，用于训练SER分类器时，资源的语言是稀缺的。摘要：In a conventional Speech emotion recognition (SER) task, a classifier for a given language is trained on a pre-existing dataset for that same language. However, where training data for a language does not exist, data from other languages can be used instead. We experiment with cross-lingual and multilingual SER, working with Amharic, English, German and URDU. For Amharic, we use our own publicly-available Amharic Speech Emotion Dataset (ASED). For English, German and Urdu we use the existing RAVDESS, EMO-DB and URDU datasets. We followed previous research in mapping labels for all datasets to just two classes, positive and negative. Thus we can compare performance on different languages directly, and combine languages for training and testing. In Experiment 1, monolingual SER trials were carried out using three classifiers, AlexNet, VGGE (a proposed variant of VGG), and ResNet50. Results averaged for the three models were very similar for ASED and RAVDESS, suggesting that Amharic and English SER are equally difficult. Similarly, German SER is more difficult, and Urdu SER is easier. In Experiment 2, we trained on one language and tested on another, in both directions for each pair: Amharic<->German, Amharic<->English, and Amharic<->Urdu. Results with Amharic as target suggested that using English or German as source will give the best result. In Experiment 3, we trained on several non-Amharic languages and then tested on Amharic. The best accuracy obtained was several percent greater than the best accuracy in Experiment 2, suggesting that a better result can be obtained when using two or three non-Amharic languages for training than when using just one non-Amharic language. Overall, the results suggest that cross-lingual and multilingual training can be an effective strategy for training a SER classifier when resources for a language are scarce.
【5】 Perceptual Quality Assessment of Omnidirectional Audio-visual Signals标题：全方位视听信号的感知质量评价链接：https://arxiv.org/abs/2307.10813
作者：Xilei Zhu,Huiyu Duan,Yuqin Cao,Yuxin Zhu,Yucheng Zhu,Jing Liu,Li Chen,Xiongkuo Min,Guangtao Zhai机构：Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University备注：12 pages, 5 figures, to be published in CICAI2023摘要：全向视频（ODV）在医疗、教育、广告、旅游等应用领域发挥着越来越重要的作用。评估ODV的质量对于服务提供商改善用户的体验质量（QoE）是重要的。然而，大多数现有的ODV质量评估研究只关注视频的视觉失真，而忽略了整体QoE也取决于伴随的音频信号。在本文中，我们首先建立了一个大规模的视听质量评估数据集的全向视频，其中包括375失真的全向视听（A/V）序列产生的15个高质量的原始全向A/V内容，以及相应的感知视听质量分数。然后，我们设计了三个基准方法全参考全向视听质量评估（OAVQA），结合现有的国家的最先进的单模式音频和视频QA模型，通过多模式融合策略。我们验证了我们的数据集上的OAVQA的A/V多模态融合方法的有效性，这为全向QoE评估提供了一个新的基准。我们的数据集可在https://github.com/iamazxl/OAVQA上获得。摘要：Omnidirectional videos (ODVs) play an increasingly important role in the application fields of medical, education, advertising, tourism, etc. Assessing the quality of ODVs is significant for service-providers to improve the user's Quality of Experience (QoE). However, most existing quality assessment studies for ODVs only focus on the visual distortions of videos, while ignoring that the overall QoE also depends on the accompanying audio signals. In this paper, we first establish a large-scale audio-visual quality assessment dataset for omnidirectional videos, which includes 375 distorted omnidirectional audio-visual (A/V) sequences generated from 15 high-quality pristine omnidirectional A/V contents, and the corresponding perceptual audio-visual quality scores. Then, we design three baseline methods for full-reference omnidirectional audio-visual quality assessment (OAVQA), which combine existing state-of-the-art single-mode audio and video QA models via multimodal fusion strategies. We validate the effectiveness of the A/V multimodal fusion method for OAVQA on our dataset, which provides a new benchmark for omnidirectional QoE evaluation. Our dataset is available at https://github.com/iamazxl/OAVQA.
【6】 Music Genre Classification with ResNet and Bi-GRU Using Visual  Spectrograms标题：基于ResNet和BiGRU的视觉谱图音乐类型分类链接：https://arxiv.org/abs/2307.10773
作者：Junfei Zhang机构：potentially improving the genre classification•The University of MelbourneE-mail摘要：音乐推荐系统已经成为一个重要的组成部分，以提高用户的体验和满足的音乐流媒体服务，主导音乐消费。改进这些推荐系统的关键挑战在于理解音乐数据的复杂性，特别是对于基础音乐流派分类。手动流派分类的局限性突出了对更高级系统的需要，即自动音乐流派分类（AMGC）系统。虽然传统的机器学习技术在流派分类方面表现出了潜力，但它们严重依赖于手动设计的特征和特征选择，无法捕获音乐数据的全部复杂性。另一方面，传统卷积神经网络（CNN）等深度学习分类架构在捕捉空间层次结构方面是有效的，但难以捕捉音乐数据固有的时间动态。为了解决这些挑战，本研究提出了一种使用视觉频谱图作为输入的新方法，并提出了一种混合模型，该模型结合了残差神经网络（ResNet）和门控递归单元（GRU）的强度。该模型被设计为提供对音乐数据的更全面的分析，通过实现对音乐数据的更全面的分析并因此潜在地更准确的流派分类来提供改进音乐推荐系统的潜力。摘要：Music recommendation systems have emerged as a vital component to enhance user experience and satisfaction for the music streaming services, which dominates music consumption. The key challenge in improving these recommender systems lies in comprehending the complexity of music data, specifically for the underpinning music genre classification. The limitations of manual genre classification have highlighted the need for a more advanced system, namely the Automatic Music Genre Classification (AMGC) system. While traditional machine learning techniques have shown potential in genre classification, they heavily rely on manually engineered features and feature selection, failing to capture the full complexity of music data. On the other hand, deep learning classification architectures like the traditional Convolutional Neural Networks (CNN) are effective in capturing the spatial hierarchies but struggle to capture the temporal dynamics inherent in music data. To address these challenges, this study proposes a novel approach using visual spectrograms as input, and propose a hybrid model that combines the strength of the Residual neural Network (ResNet) and the Gated Recurrent Unit (GRU). This model is designed to provide a more comprehensive analysis of music data, offering the potential to improve the music recommender systems through achieving a more comprehensive analysis of music data and hence potentially more accurate genre classification.
【7】 Vesper: A Compact and Effective Pretrained Model for Speech Emotion  Recognition标题：Vesper：一种紧凑有效的语音情感识别预训练模型链接：https://arxiv.org/abs/2307.10757
作者：Weidong Chen,Xiaofen Xing,Peihao Chen,Xiangmin Xu机构：PeihaoCheniswiththeSchoolofSoftwareEngineering, SouthChinaUniversityofTechnology备注：13 pages, 5 figures, 8 tables摘要：本文提出了一种范式，适应一般的大规模预训练模型（PTM）的语音情感识别任务。尽管PTM为人工通用智能提供了新的视角，但它们是在考虑到一般任务的情况下构建的，因此，它们对特定任务的功效可以进一步提高。另外，在实际应用中采用PTM可能由于其相当大的尺寸而具有挑战性。上述限制催生了另一个研究方向，即针对特定任务优化大规模PTM，以生成紧凑且有效的特定任务PTM。在本文中，我们专注于语音情感识别任务，并提出了一种改进的情感特定的预训练编码器Vesper。Vesper在基于WavLM的语音数据集上进行预训练，并考虑了情感特征。为了提高对情感信息的敏感性，Vesper采用了情感引导的掩蔽策略来识别需要掩蔽的区域。随后，Vesper采用分层和跨层的自我监督，以提高其捕获声学和语义表示的能力，这两者都是情感识别的关键。在IEMOCAP、MELD和CREMA-D数据集上的实验结果表明，Vesper的4层性能优于WavLM Base的12层，Vesper的12层性能优于WavLM Large的24层。摘要：This paper presents a paradigm that adapts general large-scale pretrained models (PTMs) to speech emotion recognition task. Although PTMs shed new light on artificial general intelligence, they are constructed with general tasks in mind, and thus, their efficacy for specific tasks can be further improved. Additionally, employing PTMs in practical applications can be challenging due to their considerable size. Above limitations spawn another research direction, namely, optimizing large-scale PTMs for specific tasks to generate task-specific PTMs that are both compact and effective. In this paper, we focus on the speech emotion recognition task and propose an improved emotion-specific pretrained encoder called Vesper. Vesper is pretrained on a speech dataset based on WavLM and takes into account emotional characteristics. To enhance sensitivity to emotional information, Vesper employs an emotion-guided masking strategy to identify the regions that need masking. Subsequently, Vesper employs hierarchical and cross-layer self-supervision to improve its ability to capture acoustic and semantic representations, both of which are crucial for emotion recognition. Experimental results on the IEMOCAP, MELD, and CREMA-D datasets demonstrate that Vesper with 4 layers outperforms WavLM Base with 12 layers, and the performance of Vesper with 12 layers surpasses that of WavLM Large with 24 layers.
【8】 SC VALL-E: Style-Controllable Zero-Shot Text to Speech Synthesizer标题：SC VALL-E：风格可控的零发声文语合成器链接：https://arxiv.org/abs/2307.10550
作者：Daegyeom Kim,Seongho Hong,Yong-Hoon Choi机构：School of Robotics, Kwangwoon University, Seoul , South Korea, This work was supported in part by the National Research Foundation of Korea (NRF) Grant funded by the Korea Government MSIT under Grant, R,F,A,.摘要：表达性语音合成模型通过将具有不同说话者、各种情绪和不同说话风格的语料库添加到数据集来训练，以便控制语音的各种特征并生成期望的语音。在本文中，我们提出了一种基于神经编解码器语言模型（称为VALL-E）的风格控制（SC）VALL-E模型，该模型遵循生成式预训练Transformer 3（GPT-3）的结构。建议的SC VALL-E从文本句子和提示音频输入，并被设计为通过不简单地模仿提示音频的特性，而是通过控制属性产生不同的声音来生成可控的语音。我们确定令牌的风格嵌入矩阵的新设计的风格网络，代表属性，如情感，语速，音高和声音强度，并设计一个模型，可以控制这些属性。为了评估SC VALL-E的性能，我们用三个代表性的表达性语音合成模型进行比较实验：全局样式令牌（GST）Tacotron 2、变分自动编码器（VAE）Tacotron 2和原始VALL-E。我们衡量字错误率（WER），F0浊音错误（FVE），和F0总音高错误（F0 GPE）作为评估指标，以评估生成的句子的准确性。为了比较合成语音的质量，我们测量比较平均选项得分（CMOS）和相似性平均选项得分（SMOS）。为了评估生成的语音的风格控制能力，我们通过修改训练的标记来观察F0和Mel谱图的变化。当使用训练数据中不存在的提示音频时，SC VALL-E会生成各种富有表现力的声音，并与现有模型相比表现出具有竞争力的性能。我们的实现，预训练模型和音频样本位于GitHub上。摘要：Expressive speech synthesis models are trained by adding corpora with diverse speakers, various emotions, and different speaking styles to the dataset, in order to control various characteristics of speech and generate the desired voice. In this paper, we propose a style control (SC) VALL-E model based on the neural codec language model (called VALL-E), which follows the structure of the generative pretrained transformer 3 (GPT-3). The proposed SC VALL-E takes input from text sentences and prompt audio and is designed to generate controllable speech by not simply mimicking the characteristics of the prompt audio but by controlling the attributes to produce diverse voices. We identify tokens in the style embedding matrix of the newly designed style network that represent attributes such as emotion, speaking rate, pitch, and voice intensity, and design a model that can control these attributes. To evaluate the performance of SC VALL-E, we conduct comparative experiments with three representative expressive speech synthesis models: global style token (GST) Tacotron2, variational autoencoder (VAE) Tacotron2, and original VALL-E. We measure word error rate (WER), F0 voiced error (FVE), and F0 gross pitch error (F0GPE) as evaluation metrics to assess the accuracy of generated sentences. For comparing the quality of synthesized speech, we measure comparative mean option score (CMOS) and similarity mean option score (SMOS). To evaluate the style control ability of the generated speech, we observe the changes in F0 and mel-spectrogram by modifying the trained tokens. When using prompt audio that is not present in the training data, SC VALL-E generates a variety of expressive sounds and demonstrates competitive performance compared to the existing models. Our implementation, pretrained models, and audio samples are located on GitHub.
【9】 Mood Classification of Bangla Songs Based on Lyrics标题：基于歌词的孟加拉歌曲情绪分类链接：https://arxiv.org/abs/2307.10314
作者：Maliha Mahajebin,Mohammad Rifat Ahmmad Rashid,Nafees Mansoor机构：University of Liberal Arts Bangladesh, Dhaka, Bangladesh., East West University, Bangladesh, Dhaka, Bangladesh.备注：Presented at International Conference on. Inventive Communication and Computational Technologies 2023摘要：音乐可以唤起各种情感，随着技术的进步，它变得更加容易为人们所接受。描绘不同人类情感的孟加拉音乐缺乏足够的研究。本文的作者旨在分析孟加拉歌曲，并根据歌词对其情绪进行分类。为了实现这一目标，本研究编制了一个包含4000首孟加拉歌曲歌词和流派的数据集，并使用自然语言处理和BERT算法来分析数据。在4000首歌曲中，1513首歌曲代表悲伤情绪，1362首代表浪漫情绪，886首代表快乐，其余239首被归类为放松。通过嵌入歌曲的歌词，作者将歌曲分为四种情绪：快乐，悲伤，浪漫，放松。这项研究至关重要，因为它可以对歌曲的情绪进行多类分类，使音乐与人们的情绪更加相关。本文给出了从歌词中准确推导出的四种情绪的自动化结果。摘要：Music can evoke various emotions, and with the advancement of technology, it has become more accessible to people. Bangla music, which portrays different human emotions, lacks sufficient research. The authors of this article aim to analyze Bangla songs and classify their moods based on the lyrics. To achieve this, this research has compiled a dataset of 4000 Bangla song lyrics, genres, and used Natural Language Processing and the Bert Algorithm to analyze the data. Among the 4000 songs, 1513 songs are represented for the sad mood, 1362 for the romantic mood, 886 for happiness, and the rest 239 are classified as relaxation. By embedding the lyrics of the songs, the authors have classified the songs into four moods: Happy, Sad, Romantic, and Relaxed. This research is crucial as it enables a multi-class classification of songs' moods, making the music more relatable to people's emotions. The article presents the automated result of the four moods accurately derived from the song lyrics.
【10】 Polyffusion: A Diffusion Model for Polyphonic Score Generation with  Internal and External Controls标题：扩散：具有内外控制的复调乐谱生成扩散模型链接：https://arxiv.org/abs/2307.10304
作者：Lejun Min,Junyan Jiang,Gus Xia,Jingwei Zhao机构：Music X Lab, Computer Science Department, NYU Shanghai,  MBZUAI,  Institute of Data Science, NUS,  Zhiyuan College, Shanghai Jiao Tong University备注：In Proceedings of the 24th Conference of the International Society for Music Information Retrieval (ISMIR 2023), Milan, Italy摘要：我们提出了Polyffusion，扩散模型，产生复调乐谱，把音乐作为图像般的钢琴卷表示。该模型能够以两种范例进行可控音乐生成：内部控制和外部控制。内部控制是指用户预先定义音乐的一部分，然后让模型填充其余部分的过程，类似于蒙面音乐生成（或音乐修复）的任务。外部控制通过交叉注意机制用外部但相关的信息（诸如和弦、纹理或其他特征）来调节模型。我们表明，通过使用内部和外部控制，Polyffusion统一了广泛的音乐创作任务，包括旋律生成给定的伴奏，伴奏生成给定的旋律，任意的音乐片段修复，和音乐安排给定的和弦或纹理。实验结果表明，我们的模型显着优于现有的Transformer和基于采样的基线，并使用预先训练的解纠缠表示作为外部条件产生更有效的控制。摘要：We propose Polyffusion, a diffusion model that generates polyphonic music scores by regarding music as image-like piano roll representations. The model is capable of controllable music generation with two paradigms: internal control and external control. Internal control refers to the process in which users pre-define a part of the music and then let the model infill the rest, similar to the task of masked music generation (or music inpainting). External control conditions the model with external yet related information, such as chord, texture, or other features, via the cross-attention mechanism. We show that by using internal and external controls, Polyffusion unifies a wide range of music creation tasks, including melody generation given accompaniment, accompaniment generation given melody, arbitrary music segment inpainting, and music arrangement given chords or textures. Experimental results show that our model significantly outperforms existing Transformer and sampling-based baselines, and using pre-trained disentangled representations as external conditions yields more effective controls.
【11】 Interpretable Timbre Synthesis using Variational Autoencoders  Regularized on Timbre Descriptors标题：基于音色描述符正则的变分自动编码器的可解释音色合成链接：https://arxiv.org/abs/2307.10283
作者：Anastasia Natsiou,Luca Longo,Sean O'Leary机构：Technological University Dublin, Dublin, Ireland摘要：可控音色合成是几十年来的研究课题，深度神经网络在这一领域最为成功。诸如变分自动编码器（VAE）的深度生成模型具有生成音频的高级表示的能力，同时提供结构化的潜在空间。尽管它们的优点，这些潜在的空间在人类感知方面的可解释性往往是有限的。为了解决这一限制，并加强对音色生成的控制，我们提出了一个正则化的基于VAE的潜在空间，采用音色描述符。此外，我们建议一个更简洁的声音表示，利用其谐波含量，以尽量减少潜在空间的维数。摘要：Controllable timbre synthesis has been a subject of research for several decades, and deep neural networks have been the most successful in this area. Deep generative models such as Variational Autoencoders (VAEs) have the ability to generate a high-level representation of audio while providing a structured latent space. Despite their advantages, the interpretability of these latent spaces in terms of human perception is often limited. To address this limitation and enhance the control over timbre generation, we propose a regularized VAE-based latent space that incorporates timbre descriptors. Moreover, we suggest a more concise representation of sound by utilizing its harmonic content, in order to minimize the dimensionality of the latent space.
【12】 Brain2Music: Reconstructing Music from Human Brain Activity标题：Brain2Music：从人脑活动重构音乐链接：https://arxiv.org/abs/2307.11078
作者：Timo I. Denk,Yu Takagi,Takuya Matsuyama,Andrea Agostinelli,Tomoya Nakai,Christian Frank,Shinji Nishimoto机构：Google,  Osaka University, Japan,  NICT, Japan,  Araya Inc., Japan备注：Preprint; 21 pages; supplementary material: this https URL摘要：从人类大脑活动中重建经验的过程为大脑如何解释和代表世界提供了一个独特的视角。在本文中，我们介绍了一种方法，用于重建音乐从大脑活动，捕获使用功能磁共振成像（fMRI）。我们的方法使用音乐检索或MusicLM音乐生成模型的条件嵌入来自功能磁共振成像数据。所生成的音乐类似于人类受试者经历的音乐刺激，相对于语义属性，如流派，乐器和情绪。我们调查的MusicLM和大脑活动的不同组成部分之间的关系，通过逐体素编码建模分析。此外，我们将讨论哪些大脑区域代表来自纯文本描述的音乐刺激的信息。我们在https://google-research.github.io/seanet/brain2music上提供补充材料，包括重建音乐的示例摘要：The process of reconstructing experiences from human brain activity offers a unique lens into how the brain interprets and represents the world. In this paper, we introduce a method for reconstructing music from brain activity, captured using functional magnetic resonance imaging (fMRI). Our approach uses either music retrieval or the MusicLM music generation model conditioned on embeddings derived from fMRI data. The generated music resembles the musical stimuli that human subjects experienced, with respect to semantic properties like genre, instrumentation, and mood. We investigate the relationship between different components of MusicLM and brain activity through a voxel-wise encoding modeling analysis. Furthermore, we discuss which brain regions represent information derived from purely textual descriptions of music stimuli. We provide supplementary material including examples of the reconstructed music at https://google-research.github.io/seanet/brain2music
【13】 Globally Normalising the Transducer for Streaming Speech Recognition标题：对流语音识别的换能器进行全局标准化链接：https://arxiv.org/abs/2307.10975
作者：Rogier van Dalen备注：9 pages plus references and appendices摘要：传感器（例如RNN-转换器或Conformer-Transducer）在遍历输入序列时生成输出标签序列。它可以直接用于流模式，在流模式下，它在看到完整的输入之前生成部分假设。这使得它在语音识别中很受欢迎。然而，在流式模式下，传感器有一个数学缺陷，简单地说，限制了模型改变主意的能力。修复方法是替换局部归一化（例如softmax），但是随后损失函数变得不可能精确地评估。最近的一篇论文提出通过近似模型来解决这个问题，严重降低了性能。相反，本文提出了近似的损失函数，允许全球归一化应用到一个国家的最先进的流模型。全局标准化将其字错误率相对降低了9-11%，几乎缩小了流模式和前瞻模式之间的一半差距。摘要：The Transducer (e.g. RNN-Transducer or Conformer-Transducer) generates an output label sequence as it traverses the input sequence. It is straightforward to use in streaming mode, where it generates partial hypotheses before the complete input has been seen. This makes it popular in speech recognition. However, in streaming mode the Transducer has a mathematical flaw which, simply put, restricts the model's ability to change its mind. The fix is to replace local normalisation (e.g. a softmax) with global normalisation, but then the loss function becomes impossible to evaluate exactly. A recent paper proposes to solve this by approximating the model, severely degrading performance. Instead, this paper proposes to approximate the loss function, allowing global normalisation to apply to a state-of-the-art streaming model. Global normalisation reduces its word error rate by 9-11% relative, closing almost half the gap between streaming and lookahead mode.
【14】 Transfer Learning and Bias Correction with Pre-trained Audio Embeddings标题：基于预训练音频嵌入的迁移学习和偏差校正链接：https://arxiv.org/abs/2307.10834
作者：Changhong Wang,Gaël Richard,Brian McFee机构：LTCI, Télécom Paris, Institut Polytechnique de Paris, France, Music and Audio Research Laboratory, New York University, USA备注：7 pages, 3 figures, accepted to the conference of the International Society for Music Information Retrieval (ISMIR 2023)摘要：深度神经网络模型已经成为音乐信息检索（MIR）中各种任务的主要方法。这些模型通常需要大量的（注释的）训练数据来实现高精度。由于并非MIR中的所有应用程序都有足够的训练数据，因此跨域传输模型变得越来越普遍。这种方法允许将针对一个任务导出的表示应用于另一个任务，并且可以导致高准确度，而对下游任务的训练数据要求不那么严格。然而，预训练的音频嵌入的属性尚未完全理解。具体而言，与传统的工程特征不同，从预先训练的深度网络中提取的表示可以嵌入和传播来自模型训练机制的偏差。这项工作调查的背景下，预先训练的音频表示的文书识别的任务的偏差传播的现象。我们首先证明了三种不同的预训练表示（VGGish，OpenL3和YAMNet）在约束于单个数据集时表现出相当的性能，但在跨数据集（OpenMIC和IRMAS）的泛化能力方面有所不同。然后，我们调查数据集的身份和体裁分布的潜在来源的偏见。最后，我们提出并评估后处理对策，以减轻偏见的影响，并提高跨数据集的泛化。摘要：Deep neural network models have become the dominant approach to a large variety of tasks within music information retrieval (MIR). These models generally require large amounts of (annotated) training data to achieve high accuracy. Because not all applications in MIR have sufficient quantities of training data, it is becoming increasingly common to transfer models across domains. This approach allows representations derived for one task to be applied to another, and can result in high accuracy with less stringent training data requirements for the downstream task. However, the properties of pre-trained audio embeddings are not fully understood. Specifically, and unlike traditionally engineered features, the representations extracted from pre-trained deep networks may embed and propagate biases from the model's training regime. This work investigates the phenomenon of bias propagation in the context of pre-trained audio representations for the task of instrument recognition. We first demonstrate that three different pre-trained representations (VGGish, OpenL3, and YAMNet) exhibit comparable performance when constrained to a single dataset, but differ in their ability to generalize across datasets (OpenMIC and IRMAS). We then investigate dataset identity and genre distribution as potential sources of bias. Finally, we propose and evaluate post-processing countermeasures to mitigate the effects of bias, and improve generalization across datasets.
【15】 PAS: Partial Additive Speech Data Augmentation Method for Noise Robust  Speaker Verification标题：PAS：用于抗噪说话人确认的部分加性语音数据增强方法链接：https://arxiv.org/abs/2307.10628
作者：Wonbin Kim,Hyun-seo Shin,Ju-ho Kim,Jungwoo Heo,Chan-yeong Lim,Ha-Jin Yu机构：Chan-yeong Lim 1 and Ha-Jin Yu 1 1 School of Computer Science,  University of Seoul备注：5 pages, 2 figures, 1 table, accepted to CKAIA2023 as a conference paper摘要：背景噪声降低了语音的可懂度和质量，使得噪声环境中的说话人确认（SV）成为一项具有挑战性的任务。为了提高SV系统对噪声的鲁棒性，通常采用加性噪声数据增强方法。在本文中，我们提出了一种新的加性噪声方法，部分加性语音（PAS），其目的是训练SV系统受噪声环境的影响较小。实验结果表明，PAS优于传统的加性噪声的等错误率（EER），与SE-ResNet 34和ECAPA-TDNN观察到的4.64%和5.01%的相对改善。我们还通过分析注意力模块和可视化的说话人嵌入显示所提出的方法的有效性。摘要：Background noise reduces speech intelligibility and quality, making speaker verification (SV) in noisy environments a challenging task. To improve the noise robustness of SV systems, additive noise data augmentation method has been commonly used. In this paper, we propose a new additive noise method, partial additive speech (PAS), which aims to train SV systems to be less affected by noisy environments. The experimental results demonstrate that PAS outperforms traditional additive noise in terms of equal error rates (EER), with relative improvements of 4.64% and 5.01% observed in SE-ResNet34 and ECAPA-TDNN. We also show the effectiveness of proposed method by analyzing attention modules and visualizing speaker embeddings.
eess.AS音频处理
【1】 Globally Normalising the Transducer for Streaming Speech Recognition标题：对流语音识别的换能器进行全局标准化链接：https://arxiv.org/abs/2307.10975
作者：Rogier van Dalen备注：9 pages plus references and appendices摘要：传感器（例如RNN-转换器或Conformer-Transducer）在遍历输入序列时生成输出标签序列。它可以直接用于流模式，在流模式下，它在看到完整的输入之前生成部分假设。这使得它在语音识别中很受欢迎。然而，在流式模式下，传感器有一个数学缺陷，简单地说，限制了模型改变主意的能力。修复方法是替换局部归一化（例如softmax），但是随后损失函数变得不可能精确地评估。最近的一篇论文提出通过近似模型来解决这个问题，严重降低了性能。相反，本文提出了近似的损失函数，允许全球归一化应用到一个国家的最先进的流模型。全局标准化将其字错误率相对降低了9-11%，几乎缩小了流模式和前瞻模式之间的一半差距。摘要：The Transducer (e.g. RNN-Transducer or Conformer-Transducer) generates an output label sequence as it traverses the input sequence. It is straightforward to use in streaming mode, where it generates partial hypotheses before the complete input has been seen. This makes it popular in speech recognition. However, in streaming mode the Transducer has a mathematical flaw which, simply put, restricts the model's ability to change its mind. The fix is to replace local normalisation (e.g. a softmax) with global normalisation, but then the loss function becomes impossible to evaluate exactly. A recent paper proposes to solve this by approximating the model, severely degrading performance. Instead, this paper proposes to approximate the loss function, allowing global normalisation to apply to a state-of-the-art streaming model. Global normalisation reduces its word error rate by 9-11% relative, closing almost half the gap between streaming and lookahead mode.
【2】 Transfer Learning and Bias Correction with Pre-trained Audio Embeddings标题：基于预训练音频嵌入的迁移学习和偏差校正链接：https://arxiv.org/abs/2307.10834
作者：Changhong Wang,Gaël Richard,Brian McFee机构：LTCI, Télécom Paris, Institut Polytechnique de Paris, France, Music and Audio Research Laboratory, New York University, USA备注：7 pages, 3 figures, accepted to the conference of the International Society for Music Information Retrieval (ISMIR 2023)摘要：深度神经网络模型已经成为音乐信息检索（MIR）中各种任务的主要方法。这些模型通常需要大量的（注释的）训练数据来实现高精度。由于并非MIR中的所有应用程序都有足够的训练数据，因此跨域传输模型变得越来越普遍。这种方法允许将针对一个任务导出的表示应用于另一个任务，并且可以导致高准确度，而对下游任务的训练数据要求不那么严格。然而，预训练的音频嵌入的属性尚未完全理解。具体而言，与传统的工程特征不同，从预先训练的深度网络中提取的表示可以嵌入和传播来自模型训练机制的偏差。这项工作调查的背景下，预先训练的音频表示的文书识别的任务的偏差传播的现象。我们首先证明了三种不同的预训练表示（VGGish，OpenL3和YAMNet）在约束于单个数据集时表现出相当的性能，但在跨数据集（OpenMIC和IRMAS）的泛化能力方面有所不同。然后，我们调查数据集的身份和体裁分布的潜在来源的偏见。最后，我们提出并评估后处理对策，以减轻偏见的影响，并提高跨数据集的泛化。摘要：Deep neural network models have become the dominant approach to a large variety of tasks within music information retrieval (MIR). These models generally require large amounts of (annotated) training data to achieve high accuracy. Because not all applications in MIR have sufficient quantities of training data, it is becoming increasingly common to transfer models across domains. This approach allows representations derived for one task to be applied to another, and can result in high accuracy with less stringent training data requirements for the downstream task. However, the properties of pre-trained audio embeddings are not fully understood. Specifically, and unlike traditionally engineered features, the representations extracted from pre-trained deep networks may embed and propagate biases from the model's training regime. This work investigates the phenomenon of bias propagation in the context of pre-trained audio representations for the task of instrument recognition. We first demonstrate that three different pre-trained representations (VGGish, OpenL3, and YAMNet) exhibit comparable performance when constrained to a single dataset, but differ in their ability to generalize across datasets (OpenMIC and IRMAS). We then investigate dataset identity and genre distribution as potential sources of bias. Finally, we propose and evaluate post-processing countermeasures to mitigate the effects of bias, and improve generalization across datasets.
【3】 PAS: Partial Additive Speech Data Augmentation Method for Noise Robust  Speaker Verification标题：PAS：用于抗噪说话人确认的部分加性语音数据增强方法链接：https://arxiv.org/abs/2307.10628
作者：Wonbin Kim,Hyun-seo Shin,Ju-ho Kim,Jungwoo Heo,Chan-yeong Lim,Ha-Jin Yu机构：Chan-yeong Lim 1 and Ha-Jin Yu 1 1 School of Computer Science,  University of Seoul备注：5 pages, 2 figures, 1 table, accepted to CKAIA2023 as a conference paper摘要：背景噪声降低了语音的可懂度和质量，使得噪声环境中的说话人确认（SV）成为一项具有挑战性的任务。为了提高SV系统对噪声的鲁棒性，通常采用加性噪声数据增强方法。在本文中，我们提出了一种新的加性噪声方法，部分加性语音（PAS），其目的是训练SV系统受噪声环境的影响较小。实验结果表明，PAS优于传统的加性噪声的等错误率（EER），与SE-ResNet 34和ECAPA-TDNN观察到的4.64%和5.01%的相对改善。我们还通过分析注意力模块和可视化的说话人嵌入显示所提出的方法的有效性。摘要：Background noise reduces speech intelligibility and quality, making speaker verification (SV) in noisy environments a challenging task. To improve the noise robustness of SV systems, additive noise data augmentation method has been commonly used. In this paper, we propose a new additive noise method, partial additive speech (PAS), which aims to train SV systems to be less affected by noisy environments. The experimental results demonstrate that PAS outperforms traditional additive noise in terms of equal error rates (EER), with relative improvements of 4.64% and 5.01% observed in SE-ResNet34 and ECAPA-TDNN. We also show the effectiveness of proposed method by analyzing attention modules and visualizing speaker embeddings.
【4】 Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning  Fine-tuning标题：具有提示条件微调的零击点域敏感语音识别链接：https://arxiv.org/abs/2307.10274
作者：Feng-Ting Liao,Yung-Chieh Chan,Yi-Chang Chen,Chan-Jan Hsu,Da-shan Shiu机构：MediaTek Research备注：F-T Liao and Y-C Chan contributed equally摘要：在这项工作中，我们提出了一种方法来创建域敏感的语音识别模型，利用文本域信息的条件下产生一个给定的文本提示。这是通过微调预训练的端到端模型（Whisper）来完成的，以从具有提示示例的演示中学习。我们表明，这种能力可以推广到不同的领域，甚至各种提示上下文，我们的模型在来自各个领域的看不见的数据集上获得了高达33%的字错误率（WER）降低，例如医疗对话，空中交通管制通信和金融会议。考虑到有限的音频转录对数据的可用性，我们进一步扩展我们的方法，以文本微调，以实现域敏感性以及域适应。我们证明，我们的纯文本微调模型也可以参加各种提示的情况下，该模型达到最WER减少29%的医疗会话数据集。摘要：In this work, we propose a method to create domain-sensitive speech recognition models that utilize textual domain information by conditioning its generation on a given text prompt. This is accomplished by fine-tuning a pre-trained, end-to-end model (Whisper) to learn from demonstrations with prompt examples. We show that this ability can be generalized to different domains and even various prompt contexts, with our model gaining a Word Error Rate (WER) reduction of up to 33% on unseen datasets from various domains, such as medical conversation, air traffic control communication, and financial meetings. Considering the limited availability of audio-transcript pair data, we further extend our method to text-only fine-tuning to achieve domain sensitivity as well as domain adaptation. We demonstrate that our text-only fine-tuned model can also attend to various prompt contexts, with the model reaching the most WER reduction of 29% on the medical conversation dataset.
【5】 Brain2Music: Reconstructing Music from Human Brain Activity标题：Brain2Music：从人脑活动重构音乐链接：https://arxiv.org/abs/2307.11078
作者：Timo I. Denk,Yu Takagi,Takuya Matsuyama,Andrea Agostinelli,Tomoya Nakai,Christian Frank,Shinji Nishimoto机构：Google,  Osaka University, Japan,  NICT, Japan,  Araya Inc., Japan备注：Preprint; 21 pages; supplementary material: this https URL摘要：从人类大脑活动中重建经验的过程为大脑如何解释和代表世界提供了一个独特的视角。在本文中，我们介绍了一种方法，用于重建音乐从大脑活动，捕获使用功能磁共振成像（fMRI）。我们的方法使用音乐检索或MusicLM音乐生成模型的条件嵌入来自功能磁共振成像数据。所生成的音乐类似于人类受试者经历的音乐刺激，相对于语义属性，如流派，乐器和情绪。我们调查的MusicLM和大脑活动的不同组成部分之间的关系，通过逐体素编码建模分析。此外，我们将讨论哪些大脑区域代表来自纯文本描述的音乐刺激的信息。我们在https://google-research.github.io/seanet/brain2music上提供补充材料，包括重建音乐的示例摘要：The process of reconstructing experiences from human brain activity offers a unique lens into how the brain interprets and represents the world. In this paper, we introduce a method for reconstructing music from brain activity, captured using functional magnetic resonance imaging (fMRI). Our approach uses either music retrieval or the MusicLM music generation model conditioned on embeddings derived from fMRI data. The generated music resembles the musical stimuli that human subjects experienced, with respect to semantic properties like genre, instrumentation, and mood. We investigate the relationship between different components of MusicLM and brain activity through a voxel-wise encoding modeling analysis. Furthermore, we discuss which brain regions represent information derived from purely textual descriptions of music stimuli. We provide supplementary material including examples of the reconstructed music at https://google-research.github.io/seanet/brain2music
【6】 Integrating Pretrained ASR and LM to Perform Sequence Generation for  Spoken Language Understanding标题：集成预先训练的ASR和LM进行口语理解序列生成链接：https://arxiv.org/abs/2307.11005
作者：Siddhant Arora,Hayato Futami,Yosuke Kashiwagi,Emiru Tsunoo,Brian Yan,Shinji Watanabe机构：Carnegie Mellon University, U.S.A., Sony Group Corporation, Japan备注：Accepted at INTERSPEECH 2023摘要：将预训练语音识别（ASR）和语言模型（LM）集成到SLU框架中的兴趣越来越大。然而，先前的方法经常与预训练模型之间的词汇不匹配作斗争，并且LM不能直接使用，因为它们与其NLU公式不同。在这项研究中，我们提出了一个三通端到端（E2E）SLU系统，有效地将ASR和LM子网络到SLU制定序列生成任务。在第一遍中，我们的体系结构使用ASR子网络预测ASR成绩单。随后是LM子网络，其进行初始SLU预测。最后，在第三遍中，审议子网络对来自ASR和LM子网络的表示进行条件化以做出最终预测。我们提出的三遍SLU系统显示出改进的性能级联和E2E SLU模型上的两个基准SLU数据集，SLURP和SLUE，特别是在声学上具有挑战性的话语。摘要：There has been an increased interest in the integration of pretrained speech recognition (ASR) and language models (LM) into the SLU framework. However, prior methods often struggle with a vocabulary mismatch between pretrained models, and LM cannot be directly utilized as they diverge from its NLU formulation. In this study, we propose a three-pass end-to-end (E2E) SLU system that effectively integrates ASR and LM subnetworks into the SLU formulation for sequence generation tasks. In the first pass, our architecture predicts ASR transcripts using the ASR subnetwork. This is followed by the LM subnetwork, which makes an initial SLU prediction. Finally, in the third pass, the deliberation subnetwork conditions on representations from the ASR and LM subnetworks to make the final prediction. Our proposed three-pass SLU system shows improved performance over cascaded and E2E SLU models on two benchmark SLU datasets, SLURP and SLUE, especially on acoustically challenging utterances.
【7】 Progressive distillation diffusion for raw music generation标题：用于原创音乐生成的逐级蒸馏扩散链接：https://arxiv.org/abs/2307.10994
作者：Svetlana Pavlova机构：Skolkovo Institute of Science and Technology, Moscow, Russia备注：9 pages摘要：本文旨在将一种新的深度学习方法应用于生成原始音频文件的任务。它基于扩散模型，这是一种新的深度生成模型。这种新型的方法最近显示出出色的结果与图像生成。计算机视觉社区对这些模型给予了很多关注。另一方面，对于其他类型的应用，例如波形域中的音乐生成，实际上很少给出。  本文实现了应用于音乐的无条件生成模型：使用1D U-Net的渐进蒸馏扩散。然后，不同的扩散参数和它们的值在一个完整的结果的比较。通过这项工作实现的方法的一个很大的优点是，该模型能够处理渐进的音频处理和生成，使用从1声道128 x 384到3声道128 x 128梅尔频谱图的转换和循环生成。在不同的自我收集的数据集之间实现的实证比较。摘要：This paper aims to apply a new deep learning approach to the task of generating raw audio files. It is based on diffusion models, a recent type of deep generative model. This new type of method has recently shown outstanding results with image generation. A lot of focus has been given to those models by the computer vision community. On the other hand, really few have been given for other types of applications such as music generation in waveform domain.  In this paper the model for unconditional generating applied to music is implemented: Progressive distillation diffusion with 1D U-Net. Then, a comparison of different parameters of diffusion and their value in a full result is presented. One big advantage of the methods implemented through this work is the fact that the model is able to deal with progressing audio processing and generating , using transformation from 1-channel 128 x 384 to 3-channel 128 x 128 mel-spectrograms and looped generation. The empirical comparisons are realized across different self-collected datasets.
【8】 MASR: Metadata Aware Speech Representation标题：MASR：元数据感知语音表示链接：https://arxiv.org/abs/2307.10982
作者：Anjali Raj,Shikhar Bharadwaj,Sriram Ganapathy,Min Ma,Shikhar Vashishth机构：Google Research摘要：近年来，语音表示学习主要被构建为一个自我监督学习（SSL）任务，仅使用原始音频信号，而忽略了边信息，这是经常可用于给定的语音记录。在本文中，我们提出了MASR，元数据感知语音表示学习框架，它解决了上述限制。MASR允许包含多个外部知识源以增强元数据信息的利用。外部知识源被纳入的样本级成对的相似性矩阵的形式是有用的硬挖掘损失。MASR框架的一个关键优势是它可以与任何SSL方法组合。使用MASR表示，我们进行评估的几个下游任务，如语言识别，语音识别和其他非语义任务，如扬声器和情感识别。在这些实验中，我们说明了显着的性能改进的MASR比其他已建立的基准。我们对语言识别任务进行了详细的分析，以提供有关建议的损失函数如何使表示分离密切相关的语言的见解。摘要：In the recent years, speech representation learning is constructed primarily as a self-supervised learning (SSL) task, using the raw audio signal alone, while ignoring the side-information that is often available for a given speech recording. In this paper, we propose MASR, a Metadata Aware Speech Representation learning framework, which addresses the aforementioned limitations. MASR enables the inclusion of multiple external knowledge sources to enhance the utilization of meta-data information. The external knowledge sources are incorporated in the form of sample-level pair-wise similarity matrices that are useful in a hard-mining loss. A key advantage of the MASR framework is that it can be combined with any choice of SSL method. Using MASR representations, we perform evaluations on several downstream tasks such as language identification, speech recognition and other non-semantic tasks such as speaker and emotion recognition. In these experiments, we illustrate significant performance improvements for the MASR over other established benchmarks. We perform a detailed analysis on the language identification task to provide insights on how the proposed loss function enables the representations to separate closely related languages.
【9】 Cross-Corpus Multilingual Speech Emotion Recognition: Amharic vs. Other  Languages标题：跨语料库多语言语音情感识别：阿姆哈拉语与其他语言链接：https://arxiv.org/abs/2307.10814
作者：Ephrem Afele Retta,Richard Sutcliffe,Jabar Mahmood,Michael Abebe Berwo,Eiad Almekhlafi,Sajjad Ahmed Khan,Shehzad Ashraf Chaudhry,Mustafa Mhamed,Jun Feng机构：School of Information Science and Technology, School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe Park, Colchester CO,SQ, UK, Northwest University, Xi’an , China, University of Sialkot, Punjab, Pakistan备注：16 pages, 9 tables, 5 figures摘要：在传统的语音情感识别（SER）任务中，针对给定语言的分类器是在针对该相同语言的预先存在的数据集上训练的。然而，在不存在用于语言的训练数据的情况下，可以替代地使用来自其他语言的数据。我们尝试跨语言和多语言SER，与阿姆哈拉语，英语，德语和URDU合作。对于阿姆哈拉语，我们使用我们自己的公开可用的阿姆哈拉语语音情感数据集（ASED）。对于英语，德语和乌尔都语，我们使用现有的RAVDESS，EMO-DB和URDU数据集。我们遵循之前的研究，将所有数据集的标签映射到两个类别，正面和负面。因此，我们可以直接比较不同语言的性能，并结合语言进行训练和测试。在实验1中，单语SER试验进行了使用三个分类器，AlexNet，VGGE（VGG的一个拟议的变体），和ResNet 50。ASED和RAVDESS的三种模型的平均结果非常相似，这表明阿姆哈拉语和英语SER同样困难。同样，德语SER更难，乌尔都语SER更容易。在实验2中，我们在一种语言上进行训练，并在另一种语言上进行测试，每对都是双向的：阿姆哈拉<->德语、阿姆哈拉<->英语和阿姆哈拉<->乌尔都语。以阿姆哈拉语为目标的结果表明，使用英语或德语作为源将得到最好的结果。在实验3中，我们训练了几种非阿姆哈拉语语言，然后测试阿姆哈拉语。获得的最佳准确度比实验2中的最佳准确度高出几个百分点，这表明使用两种或三种非阿姆哈拉语进行训练时可以获得比仅使用一种非阿姆哈拉语时更好的结果。总的来说，结果表明，跨语言和多语言的培训可以是一个有效的策略，用于训练SER分类器时，资源的语言是稀缺的。摘要：In a conventional Speech emotion recognition (SER) task, a classifier for a given language is trained on a pre-existing dataset for that same language. However, where training data for a language does not exist, data from other languages can be used instead. We experiment with cross-lingual and multilingual SER, working with Amharic, English, German and URDU. For Amharic, we use our own publicly-available Amharic Speech Emotion Dataset (ASED). For English, German and Urdu we use the existing RAVDESS, EMO-DB and URDU datasets. We followed previous research in mapping labels for all datasets to just two classes, positive and negative. Thus we can compare performance on different languages directly, and combine languages for training and testing. In Experiment 1, monolingual SER trials were carried out using three classifiers, AlexNet, VGGE (a proposed variant of VGG), and ResNet50. Results averaged for the three models were very similar for ASED and RAVDESS, suggesting that Amharic and English SER are equally difficult. Similarly, German SER is more difficult, and Urdu SER is easier. In Experiment 2, we trained on one language and tested on another, in both directions for each pair: Amharic<->German, Amharic<->English, and Amharic<->Urdu. Results with Amharic as target suggested that using English or German as source will give the best result. In Experiment 3, we trained on several non-Amharic languages and then tested on Amharic. The best accuracy obtained was several percent greater than the best accuracy in Experiment 2, suggesting that a better result can be obtained when using two or three non-Amharic languages for training than when using just one non-Amharic language. Overall, the results suggest that cross-lingual and multilingual training can be an effective strategy for training a SER classifier when resources for a language are scarce.
【10】 Perceptual Quality Assessment of Omnidirectional Audio-visual Signals标题：全方位视听信号的感知质量评价链接：https://arxiv.org/abs/2307.10813
作者：Xilei Zhu,Huiyu Duan,Yuqin Cao,Yuxin Zhu,Yucheng Zhu,Jing Liu,Li Chen,Xiongkuo Min,Guangtao Zhai机构：Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University备注：12 pages, 5 figures, to be published in CICAI2023摘要：全向视频（ODV）在医疗、教育、广告、旅游等应用领域发挥着越来越重要的作用。评估ODV的质量对于服务提供商改善用户的体验质量（QoE）是重要的。然而，大多数现有的ODV质量评估研究只关注视频的视觉失真，而忽略了整体QoE也取决于伴随的音频信号。在本文中，我们首先建立了一个大规模的视听质量评估数据集的全向视频，其中包括375失真的全向视听（A/V）序列产生的15个高质量的原始全向A/V内容，以及相应的感知视听质量分数。然后，我们设计了三个基准方法全参考全向视听质量评估（OAVQA），结合现有的国家的最先进的单模式音频和视频QA模型，通过多模式融合策略。我们验证了我们的数据集上的OAVQA的A/V多模态融合方法的有效性，这为全向QoE评估提供了一个新的基准。我们的数据集可在https://github.com/iamazxl/OAVQA上获得。摘要：Omnidirectional videos (ODVs) play an increasingly important role in the application fields of medical, education, advertising, tourism, etc. Assessing the quality of ODVs is significant for service-providers to improve the user's Quality of Experience (QoE). However, most existing quality assessment studies for ODVs only focus on the visual distortions of videos, while ignoring that the overall QoE also depends on the accompanying audio signals. In this paper, we first establish a large-scale audio-visual quality assessment dataset for omnidirectional videos, which includes 375 distorted omnidirectional audio-visual (A/V) sequences generated from 15 high-quality pristine omnidirectional A/V contents, and the corresponding perceptual audio-visual quality scores. Then, we design three baseline methods for full-reference omnidirectional audio-visual quality assessment (OAVQA), which combine existing state-of-the-art single-mode audio and video QA models via multimodal fusion strategies. We validate the effectiveness of the A/V multimodal fusion method for OAVQA on our dataset, which provides a new benchmark for omnidirectional QoE evaluation. Our dataset is available at https://github.com/iamazxl/OAVQA.
【11】 Music Genre Classification with ResNet and Bi-GRU Using Visual  Spectrograms标题：基于ResNet和BiGRU的视觉谱图音乐类型分类链接：https://arxiv.org/abs/2307.10773
作者：Junfei Zhang机构：potentially improving the genre classification•The University of MelbourneE-mail摘要：音乐推荐系统已经成为一个重要的组成部分，以提高用户的体验和满足的音乐流媒体服务，主导音乐消费。改进这些推荐系统的关键挑战在于理解音乐数据的复杂性，特别是对于基础音乐流派分类。手动流派分类的局限性突出了对更高级系统的需要，即自动音乐流派分类（AMGC）系统。虽然传统的机器学习技术在流派分类方面表现出了潜力，但它们严重依赖于手动设计的特征和特征选择，无法捕获音乐数据的全部复杂性。另一方面，传统卷积神经网络（CNN）等深度学习分类架构在捕捉空间层次结构方面是有效的，但难以捕捉音乐数据固有的时间动态。为了解决这些挑战，本研究提出了一种使用视觉频谱图作为输入的新方法，并提出了一种混合模型，该模型结合了残差神经网络（ResNet）和门控递归单元（GRU）的强度。该模型被设计为提供对音乐数据的更全面的分析，通过实现对音乐数据的更全面的分析并因此潜在地更准确的流派分类来提供改进音乐推荐系统的潜力。摘要：Music recommendation systems have emerged as a vital component to enhance user experience and satisfaction for the music streaming services, which dominates music consumption. The key challenge in improving these recommender systems lies in comprehending the complexity of music data, specifically for the underpinning music genre classification. The limitations of manual genre classification have highlighted the need for a more advanced system, namely the Automatic Music Genre Classification (AMGC) system. While traditional machine learning techniques have shown potential in genre classification, they heavily rely on manually engineered features and feature selection, failing to capture the full complexity of music data. On the other hand, deep learning classification architectures like the traditional Convolutional Neural Networks (CNN) are effective in capturing the spatial hierarchies but struggle to capture the temporal dynamics inherent in music data. To address these challenges, this study proposes a novel approach using visual spectrograms as input, and propose a hybrid model that combines the strength of the Residual neural Network (ResNet) and the Gated Recurrent Unit (GRU). This model is designed to provide a more comprehensive analysis of music data, offering the potential to improve the music recommender systems through achieving a more comprehensive analysis of music data and hence potentially more accurate genre classification.
【12】 Vesper: A Compact and Effective Pretrained Model for Speech Emotion  Recognition标题：Vesper：一种紧凑有效的语音情感识别预训练模型链接：https://arxiv.org/abs/2307.10757
作者：Weidong Chen,Xiaofen Xing,Peihao Chen,Xiangmin Xu机构：PeihaoCheniswiththeSchoolofSoftwareEngineering, SouthChinaUniversityofTechnology备注：13 pages, 5 figures, 8 tables摘要：本文提出了一种范式，适应一般的大规模预训练模型（PTM）的语音情感识别任务。尽管PTM为人工通用智能提供了新的视角，但它们是在考虑到一般任务的情况下构建的，因此，它们对特定任务的功效可以进一步提高。另外，在实际应用中采用PTM可能由于其相当大的尺寸而具有挑战性。上述限制催生了另一个研究方向，即针对特定任务优化大规模PTM，以生成紧凑且有效的特定任务PTM。在本文中，我们专注于语音情感识别任务，并提出了一种改进的情感特定的预训练编码器Vesper。Vesper在基于WavLM的语音数据集上进行预训练，并考虑了情感特征。为了提高对情感信息的敏感性，Vesper采用了情感引导的掩蔽策略来识别需要掩蔽的区域。随后，Vesper采用分层和跨层的自我监督，以提高其捕获声学和语义表示的能力，这两者都是情感识别的关键。在IEMOCAP、MELD和CREMA-D数据集上的实验结果表明，Vesper的4层性能优于WavLM Base的12层，Vesper的12层性能优于WavLM Large的24层。摘要：This paper presents a paradigm that adapts general large-scale pretrained models (PTMs) to speech emotion recognition task. Although PTMs shed new light on artificial general intelligence, they are constructed with general tasks in mind, and thus, their efficacy for specific tasks can be further improved. Additionally, employing PTMs in practical applications can be challenging due to their considerable size. Above limitations spawn another research direction, namely, optimizing large-scale PTMs for specific tasks to generate task-specific PTMs that are both compact and effective. In this paper, we focus on the speech emotion recognition task and propose an improved emotion-specific pretrained encoder called Vesper. Vesper is pretrained on a speech dataset based on WavLM and takes into account emotional characteristics. To enhance sensitivity to emotional information, Vesper employs an emotion-guided masking strategy to identify the regions that need masking. Subsequently, Vesper employs hierarchical and cross-layer self-supervision to improve its ability to capture acoustic and semantic representations, both of which are crucial for emotion recognition. Experimental results on the IEMOCAP, MELD, and CREMA-D datasets demonstrate that Vesper with 4 layers outperforms WavLM Base with 12 layers, and the performance of Vesper with 12 layers surpasses that of WavLM Large with 24 layers.
【13】 SC VALL-E: Style-Controllable Zero-Shot Text to Speech Synthesizer标题：SC VALL-E：风格可控的零发声文语合成器链接：https://arxiv.org/abs/2307.10550
作者：Daegyeom Kim,Seongho Hong,Yong-Hoon Choi机构：School of Robotics, Kwangwoon University, Seoul , South Korea, This work was supported in part by the National Research Foundation of Korea (NRF) Grant funded by the Korea Government MSIT under Grant, R,F,A,.摘要：表达性语音合成模型通过将具有不同说话者、各种情绪和不同说话风格的语料库添加到数据集来训练，以便控制语音的各种特征并生成期望的语音。在本文中，我们提出了一种基于神经编解码器语言模型（称为VALL-E）的风格控制（SC）VALL-E模型，该模型遵循生成式预训练Transformer 3（GPT-3）的结构。建议的SC VALL-E从文本句子和提示音频输入，并被设计为通过不简单地模仿提示音频的特性，而是通过控制属性产生不同的声音来生成可控的语音。我们确定令牌的风格嵌入矩阵的新设计的风格网络，代表属性，如情感，语速，音高和声音强度，并设计一个模型，可以控制这些属性。为了评估SC VALL-E的性能，我们用三个代表性的表达性语音合成模型进行比较实验：全局样式令牌（GST）Tacotron 2、变分自动编码器（VAE）Tacotron 2和原始VALL-E。我们衡量字错误率（WER），F0浊音错误（FVE），和F0总音高错误（F0 GPE）作为评估指标，以评估生成的句子的准确性。为了比较合成语音的质量，我们测量比较平均选项得分（CMOS）和相似性平均选项得分（SMOS）。为了评估生成的语音的风格控制能力，我们通过修改训练的标记来观察F0和Mel谱图的变化。当使用训练数据中不存在的提示音频时，SC VALL-E会生成各种富有表现力的声音，并与现有模型相比表现出具有竞争力的性能。我们的实现，预训练模型和音频样本位于GitHub上。摘要：Expressive speech synthesis models are trained by adding corpora with diverse speakers, various emotions, and different speaking styles to the dataset, in order to control various characteristics of speech and generate the desired voice. In this paper, we propose a style control (SC) VALL-E model based on the neural codec language model (called VALL-E), which follows the structure of the generative pretrained transformer 3 (GPT-3). The proposed SC VALL-E takes input from text sentences and prompt audio and is designed to generate controllable speech by not simply mimicking the characteristics of the prompt audio but by controlling the attributes to produce diverse voices. We identify tokens in the style embedding matrix of the newly designed style network that represent attributes such as emotion, speaking rate, pitch, and voice intensity, and design a model that can control these attributes. To evaluate the performance of SC VALL-E, we conduct comparative experiments with three representative expressive speech synthesis models: global style token (GST) Tacotron2, variational autoencoder (VAE) Tacotron2, and original VALL-E. We measure word error rate (WER), F0 voiced error (FVE), and F0 gross pitch error (F0GPE) as evaluation metrics to assess the accuracy of generated sentences. For comparing the quality of synthesized speech, we measure comparative mean option score (CMOS) and similarity mean option score (SMOS). To evaluate the style control ability of the generated speech, we observe the changes in F0 and mel-spectrogram by modifying the trained tokens. When using prompt audio that is not present in the training data, SC VALL-E generates a variety of expressive sounds and demonstrates competitive performance compared to the existing models. Our implementation, pretrained models, and audio samples are located on GitHub.
【14】 Mood Classification of Bangla Songs Based on Lyrics标题：基于歌词的孟加拉歌曲情绪分类链接：https://arxiv.org/abs/2307.10314
作者：Maliha Mahajebin,Mohammad Rifat Ahmmad Rashid,Nafees Mansoor机构：University of Liberal Arts Bangladesh, Dhaka, Bangladesh., East West University, Bangladesh, Dhaka, Bangladesh.备注：Presented at International Conference on. Inventive Communication and Computational Technologies 2023摘要：音乐可以唤起各种情感，随着技术的进步，它变得更加容易为人们所接受。描绘不同人类情感的孟加拉音乐缺乏足够的研究。本文的作者旨在分析孟加拉歌曲，并根据歌词对其情绪进行分类。为了实现这一目标，本研究编制了一个包含4000首孟加拉歌曲歌词和流派的数据集，并使用自然语言处理和BERT算法来分析数据。在4000首歌曲中，1513首歌曲代表悲伤情绪，1362首代表浪漫情绪，886首代表快乐，其余239首被归类为放松。通过嵌入歌曲的歌词，作者将歌曲分为四种情绪：快乐，悲伤，浪漫，放松。这项研究至关重要，因为它可以对歌曲的情绪进行多类分类，使音乐与人们的情绪更加相关。本文给出了从歌词中准确推导出的四种情绪的自动化结果。摘要：Music can evoke various emotions, and with the advancement of technology, it has become more accessible to people. Bangla music, which portrays different human emotions, lacks sufficient research. The authors of this article aim to analyze Bangla songs and classify their moods based on the lyrics. To achieve this, this research has compiled a dataset of 4000 Bangla song lyrics, genres, and used Natural Language Processing and the Bert Algorithm to analyze the data. Among the 4000 songs, 1513 songs are represented for the sad mood, 1362 for the romantic mood, 886 for happiness, and the rest 239 are classified as relaxation. By embedding the lyrics of the songs, the authors have classified the songs into four moods: Happy, Sad, Romantic, and Relaxed. This research is crucial as it enables a multi-class classification of songs' moods, making the music more relatable to people's emotions. The article presents the automated result of the four moods accurately derived from the song lyrics.
【15】 Polyffusion: A Diffusion Model for Polyphonic Score Generation with  Internal and External Controls标题：扩散：具有内外控制的复调乐谱生成扩散模型链接：https://arxiv.org/abs/2307.10304
作者：Lejun Min,Junyan Jiang,Gus Xia,Jingwei Zhao机构：Music X Lab, Computer Science Department, NYU Shanghai,  MBZUAI,  Institute of Data Science, NUS,  Zhiyuan College, Shanghai Jiao Tong University备注：In Proceedings of the 24th Conference of the International Society for Music Information Retrieval (ISMIR 2023), Milan, Italy摘要：我们提出了Polyffusion，扩散模型，产生复调乐谱，把音乐作为图像般的钢琴卷表示。该模型能够以两种范例进行可控音乐生成：内部控制和外部控制。内部控制是指用户预先定义音乐的一部分，然后让模型填充其余部分的过程，类似于蒙面音乐生成（或音乐修复）的任务。外部控制通过交叉注意机制用外部但相关的信息（诸如和弦、纹理或其他特征）来调节模型。我们表明，通过使用内部和外部控制，Polyffusion统一了广泛的音乐创作任务，包括旋律生成给定的伴奏，伴奏生成给定的旋律，任意的音乐片段修复，和音乐安排给定的和弦或纹理。实验结果表明，我们的模型显着优于现有的Transformer和基于采样的基线，并使用预先训练的解纠缠表示作为外部条件产生更有效的控制。摘要：We propose Polyffusion, a diffusion model that generates polyphonic music scores by regarding music as image-like piano roll representations. The model is capable of controllable music generation with two paradigms: internal control and external control. Internal control refers to the process in which users pre-define a part of the music and then let the model infill the rest, similar to the task of masked music generation (or music inpainting). External control conditions the model with external yet related information, such as chord, texture, or other features, via the cross-attention mechanism. We show that by using internal and external controls, Polyffusion unifies a wide range of music creation tasks, including melody generation given accompaniment, accompaniment generation given melody, arbitrary music segment inpainting, and music arrangement given chords or textures. Experimental results show that our model significantly outperforms existing Transformer and sampling-based baselines, and using pre-trained disentangled representations as external conditions yields more effective controls.
【16】 Interpretable Timbre Synthesis using Variational Autoencoders  Regularized on Timbre Descriptors标题：基于音色描述符正则的变分自动编码器的可解释音色合成链接：https://arxiv.org/abs/2307.10283
作者：Anastasia Natsiou,Luca Longo,Sean O'Leary机构：Technological University Dublin, Dublin, Ireland摘要：可控音色合成是几十年来的研究课题，深度神经网络在这一领域最为成功。诸如变分自动编码器（VAE）的深度生成模型具有生成音频的高级表示的能力，同时提供结构化的潜在空间。尽管它们的优点，这些潜在的空间在人类感知方面的可解释性往往是有限的。为了解决这一限制，并加强对音色生成的控制，我们提出了一个正则化的基于VAE的潜在空间，采用音色描述符。此外，我们建议一个更简洁的声音表示，利用其谐波含量，以尽量减少潜在空间的维数。摘要：Controllable timbre synthesis has been a subject of research for several decades, and deep neural networks have been the most successful in this area. Deep generative models such as Variational Autoencoders (VAEs) have the ability to generate a high-level representation of audio while providing a structured latent space. Despite their advantages, the interpretability of these latent spaces in terms of human perception is often limited. To address this limitation and enhance the control over timbre generation, we propose a regularized VAE-based latent space that incorporates timbre descriptors. Moreover, we suggest a more concise representation of sound by utilizing its harmonic content, in order to minimize the dimensionality of the latent space.
机器翻译，仅供参考