今日论文合集：cs.SD语音5篇，eess.AS音频处理4篇。
cs.SD语音
【1】 Generating music with sentiment using Transformer-GANs
标题：用《Transformer-甘斯》创作充满感情的音乐
链接：https://arxiv.org/abs/2212.11134
作者：Pedro Neves,Jose Fornari,João Florindo机构：State University of Campinas, João B. Florindo摘要：由于深度学习的出现，自动音乐生成领域取得了重大进展。但是，这些结果大多是由无条件的模型产生的，这些模型缺乏与其用户互动的能力，不允许它们以有意义和实用的方式指导生成过程。此外，合成音乐在较长的时间尺度上保持连贯性，同时仍然捕捉到使其听起来"现实“或”类似人类“的局部特征，这仍然是一个挑战。这是由于处理长数据序列所需的大量计算要求，并且还由于经常采用的训练方案所施加的限制。本文提出了一个以情感数据为条件的符号音乐生成模型。该模型是用标签训练的Transformer-GAN，所述标签对应于定量表示人类情感状态的效价和唤醒维度的不同配置。我们试图通过采用有效的线性版本的注意力和使用鉴别器来解决上述两个问题，这两个问题都作为改善所生成的音乐的整体质量及其跟随调节信号的能力的工具。摘要：The field of Automatic Music Generation has seen significant progress thanks to the advent of Deep Learning. However, most of these results have been produced by unconditional models, which lack the ability to interact with their users, not allowing them to guide the generative process in meaningful and practical ways. Moreover, synthesizing music that remains coherent across longer timescales while still capturing the local aspects that make it sound ``realistic'' or ``human-like'' is still challenging. This is due to the large computational requirements needed to work with long sequences of data, and also to limitations imposed by the training schemes that are often employed. In this paper, we propose a generative model of symbolic music conditioned by data retrieved from human sentiment. The model is a Transformer-GAN trained with labels that correspond to different configurations of the valence and arousal dimensions that quantitatively represent human affective states. We try to tackle both of the problems above by employing an efficient linear version of Attention and using a Discriminator both as a tool to improve the overall quality of the generated music and its ability to follow the conditioning signals.
【2】 Polytopic Analysis of Music
标题：音乐的多义性分析
链接：https://arxiv.org/abs/2212.11054
作者：Axel Marmoret,Jérémy E. Cohen,Frédéric Bibmot摘要：音乐的结构分段是指找到歌曲组织的符号表示的任务，将音乐流减少到不重叠片段的划分。在这个定义下，音乐结构可能不是唯一的，甚至可能是模糊的。解决这种不确定性的一种方法是将该任务视为压缩过程，并且将音乐结构视为给定压缩标准的优化。在这种观点下，C. Guichaoua开发了一个压缩驱动的模型，用于检索音乐结构，该模型基于“系统和对比”模型和多面体，多面体是n超立方体的扩展。我们提出了这个模型，我们称之为“音乐的多面体分析”，以及一个新的开源专用工具箱，称为MusicOnPolytopes（Python）。这个模型也被扩展到使用Tonnetz作为一个关系系统。在RWC Pop数据集上进行了结构分割实验。结果表明，与C.吉查瓦。摘要：Structural segmentation of music refers to the task of finding a symbolic representation of the organisation of a song, reducing the musical flow to a partition of non-overlapping segments. Under this definition, the musical structure may not be unique, and may even be ambiguous. One way to resolve that ambiguity is to see this task as a compression process, and to consider the musical structure as the optimization of a given compression criteria. In that viewpoint, C. Guichaoua developed a compression-driven model for retrieving the musical structure, based on the "System and Contrast" model, and on polytopes, which are extension of nhypercubes. We present this model, which we call "polytopic analysis of music", along with a new opensource dedicated toolbox called MusicOnPolytopes (in Python). This model is also extended to the use of the Tonnetz as a relation system. Structural segmentation experiments are conducted on the RWC Pop dataset. Results show improvements compared to the previous ones, presented by C. Guichaoua.
【3】 RECAP: Retrieval Augmented Music Captioner
标题：概述：检索增强音乐捕获器
链接：https://arxiv.org/abs/2212.10901
作者：Zihao He,Weituo Hao,Xuchen Song机构：USC Information Sciences Institute, ByteDance摘要：随着流媒体音乐搜索和推荐平台的普及，通过交互式理解音频和歌词来解读音乐已成为一项重要而富有挑战性的任务。然而，许多以前的工作集中在细化编码器-解码器架构的各个组件，将音乐映射到字幕令牌，忽略了音频和歌词对应的潜在用途。在这篇文章中，我们提出通过对比学习来明确地学习带有检索增强的多模态对齐。通过学习音频-歌词对应关系，引导模型学习更好的跨模态注意力权重，从而生成高质量的字幕词。我们提供了理论和实证结果，证明了所提方法的优越性。摘要：With the prevalence of stream media platforms serving music search and recommendation, interpreting music by understanding audio and lyrics interactively has become an important and challenging task. However, many previous works focus on refining individual components of encoder-decoder architecture mapping music to caption tokens, ignoring the potential usage of audio and lyrics correspondence. In this paper, we propose to explicitly learn the multi-modal alignment with retrieval augmentation by contrastive learning. By learning audio-lyrics correspondence, the model is guided to learn better cross-modal attention weights, thus generating high-quality caption words. We provide both theoretical and empirical results that demonstrate the advantage of the proposed method.
【4】 4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders
标题：4D ASR：CTC、注意、传感器和掩码预测解码器的联合建模
链接：https://arxiv.org/abs/2212.10818
作者：Yui Sudo,Muhammad Shakeel,Brian Yan,Jiatong Shi,Shinji Watanabe机构：Honda Research Institute Japan Co., Ltd., Saitama, Japan, Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA备注：Submitted to ICASSP 2023摘要：端到端（E2 E）自动语音识别（ASR）的网络结构可以分为几种模型，包括连接主义时间分类（CTC）模型、递归神经网络转换器（RNN-T）模型、注意机制模型和非自回归掩蔽预测模型。由于这些网络体系结构各有优缺点，因此典型的使用情形是根据应用程序要求切换这些单独的模型，从而导致维护所有模型的开销增加。已经提出了用于集成这些互补模型中的两个以减轻开销问题的若干方法;但是，如果我们集成更多的模型，我们将进一步受益于这些互补的模型，并在单一系统中实现更广泛的应用。本文提出CTC、注意、RNN-T和掩蔽预测的四解码器联合建模（4D），其具有以下三个优点：1）四个解码器被联合训练，使得它们可以根据应用场景容易地切换。2)由于联合训练具有互补性，可以使模型正则化，提高模型的鲁棒性。3)使用CTC、注意力和RNN-T的新的一遍联合解码方法进一步提高了性能。实验结果表明，所提出的模型能够持续降低WER。摘要：The network architecture of end-to-end (E2E) automatic speech recognition (ASR) can be classified into several models, including connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), attention mechanism, and non-autoregressive mask-predict models. Since each of these network architectures has pros and cons, a typical use case is to switch these separate models depending on the application requirement, resulting in the increased overhead of maintaining all models. Several methods for integrating two of these complementary models to mitigate the overhead issue have been proposed; however, if we integrate more models, we will further benefit from these complementary models and realize broader applications with a single system. This paper proposes four-decoder joint modeling (4D) of CTC, attention, RNN-T, and mask-predict, which has the following three advantages: 1) The four decoders are jointly trained so that they can be easily switched depending on the application scenarios. 2) Joint training may bring model regularization and improve the model robustness thanks to their complementary properties. 3) Novel one-pass joint decoding methods using CTC, attention, and RNN-T further improves the performance. The experimental results showed that the proposed model consistently reduced the WER.
【5】 An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits
标题：一种受大脑皮质-丘脑-大脑皮质回路启发的视听语音分离模型
链接：https://arxiv.org/abs/2212.10744
作者：Kai Li,Fenghua Xie,Hang Chen,Kexin Yuan,Xiaolin Hu摘要：涉及视觉输入的视听方法为语音分离的最新进展奠定了基础。然而，听觉和视觉输入同时使用的优化仍然是一个活跃的研究领域。受皮质-丘脑-皮质回路的启发，不同通道的感觉处理机制通过非丘系感觉丘脑相互调节，提出了一种新的用于视听语音分离（AVSS）的皮质-丘脑-皮质神经网络（CTCNet）。首先，CTCNet在独立的听觉和视觉子网络中以自下而上的方式学习分级听觉和视觉表征，模仿听觉和视觉皮层区域的功能。然后，受大脑皮层区域和丘脑之间大量连接的启发，该模型通过自上而下的连接将听觉和视觉信息融合在丘脑的一个子网络中。最后，模型将融合后的信息传回听觉和视觉子网络，并重复上述过程若干次。在3个语音分离基准数据集上的实验结果表明，CTCNet算法以较少的参数显著优于现有的AVSS算法。这些结果表明，模仿哺乳动物大脑的解剖学连接体对于推进深层神经网络的发展具有巨大的潜力。项目报告库是https://github.com/JusperLee/CTCNet。摘要：Audio-visual approaches involving visual inputs have laid the foundation for recent progress in speech separation. However, the optimization of the concurrent usage of auditory and visual inputs is still an active research area. Inspired by the cortico-thalamo-cortical circuit, in which the sensory processing mechanisms of different modalities modulate one another via the non-lemniscal sensory thalamus, we propose a novel cortico-thalamo-cortical neural network (CTCNet) for audio-visual speech separation (AVSS). First, the CTCNet learns hierarchical auditory and visual representations in a bottom-up manner in separate auditory and visual subnetworks, mimicking the functions of the auditory and visual cortical areas. Then, inspired by the large number of connections between cortical regions and the thalamus, the model fuses the auditory and visual information in a thalamic subnetwork through top-down connections. Finally, the model transmits this fused information back to the auditory and visual subnetworks, and the above process is repeated several times. The results of experiments on three speech separation benchmark datasets show that CTCNet remarkably outperforms existing AVSS methods with considerablely fewer parameters. These results suggest that mimicking the anatomical connectome of the mammalian brain has great potential for advancing the development of deep neural networks. Project repo is https://github.com/JusperLee/CTCNet.
【1】 Generating music with sentiment using Transformer-GANs
标题：用《Transformer-甘斯》创作充满感情的音乐
链接：https://arxiv.org/abs/2212.11134
*与cs.SD语音【1】为同一篇
作者：Pedro Neves,Jose Fornari,João Florindo机构：State University of Campinas, João B. Florindo摘要：由于深度学习的出现，自动音乐生成领域取得了重大进展。但是，这些结果大多是由无条件的模型产生的，这些模型缺乏与其用户互动的能力，不允许它们以有意义和实用的方式指导生成过程。此外，合成音乐在较长的时间尺度上保持连贯性，同时仍然捕捉到使其听起来"现实“或”类似人类“的局部特征，这仍然是一个挑战。这是由于处理长数据序列所需的大量计算要求，并且还由于经常采用的训练方案所施加的限制。本文提出了一个以情感数据为条件的符号音乐生成模型。该模型是用标签训练的Transformer-GAN，所述标签对应于定量表示人类情感状态的效价和唤醒维度的不同配置。我们试图通过采用有效的线性版本的注意力和使用鉴别器来解决上述两个问题，这两个问题都作为改善所生成的音乐的整体质量及其跟随调节信号的能力的工具。摘要：The field of Automatic Music Generation has seen significant progress thanks to the advent of Deep Learning. However, most of these results have been produced by unconditional models, which lack the ability to interact with their users, not allowing them to guide the generative process in meaningful and practical ways. Moreover, synthesizing music that remains coherent across longer timescales while still capturing the local aspects that make it sound ``realistic'' or ``human-like'' is still challenging. This is due to the large computational requirements needed to work with long sequences of data, and also to limitations imposed by the training schemes that are often employed. In this paper, we propose a generative model of symbolic music conditioned by data retrieved from human sentiment. The model is a Transformer-GAN trained with labels that correspond to different configurations of the valence and arousal dimensions that quantitatively represent human affective states. We try to tackle both of the problems above by employing an efficient linear version of Attention and using a Discriminator both as a tool to improve the overall quality of the generated music and its ability to follow the conditioning signals.
【2】 Polytopic Analysis of Music
标题：音乐的多义性分析
链接：https://arxiv.org/abs/2212.11054
*与cs.SD语音【2】为同一篇
作者：Axel Marmoret,Jérémy E. Cohen,Frédéric Bibmot摘要：音乐的结构分段是指找到歌曲组织的符号表示的任务，将音乐流减少到不重叠片段的划分。在这个定义下，音乐结构可能不是唯一的，甚至可能是模糊的。解决这种不确定性的一种方法是将该任务视为压缩过程，并且将音乐结构视为给定压缩标准的优化。在这种观点下，C. Guichaoua开发了一个压缩驱动的模型，用于检索音乐结构，该模型基于“系统和对比”模型和多面体，多面体是n超立方体的扩展。我们提出了这个模型，我们称之为“音乐的多面体分析”，以及一个新的开源专用工具箱，称为MusicOnPolytopes（Python）。这个模型也被扩展到使用Tonnetz作为一个关系系统。在RWC Pop数据集上进行了结构分割实验。结果表明，与C.吉查瓦。摘要：Structural segmentation of music refers to the task of finding a symbolic representation of the organisation of a song, reducing the musical flow to a partition of non-overlapping segments. Under this definition, the musical structure may not be unique, and may even be ambiguous. One way to resolve that ambiguity is to see this task as a compression process, and to consider the musical structure as the optimization of a given compression criteria. In that viewpoint, C. Guichaoua developed a compression-driven model for retrieving the musical structure, based on the "System and Contrast" model, and on polytopes, which are extension of nhypercubes. We present this model, which we call "polytopic analysis of music", along with a new opensource dedicated toolbox called MusicOnPolytopes (in Python). This model is also extended to the use of the Tonnetz as a relation system. Structural segmentation experiments are conducted on the RWC Pop dataset. Results show improvements compared to the previous ones, presented by C. Guichaoua.
【3】 RECAP: Retrieval Augmented Music Captioner
标题：概述：检索增强音乐捕获器
链接：https://arxiv.org/abs/2212.10901
*与cs.SD语音【3】为同一篇
作者：Zihao He,Weituo Hao,Xuchen Song机构：USC Information Sciences Institute, ByteDance摘要：随着流媒体音乐搜索和推荐平台的普及，通过交互式理解音频和歌词来解读音乐已成为一项重要而富有挑战性的任务。然而，许多以前的工作集中在细化编码器-解码器架构的各个组件，将音乐映射到字幕令牌，忽略了音频和歌词对应的潜在用途。在这篇文章中，我们提出通过对比学习来明确地学习带有检索增强的多模态对齐。通过学习音频-歌词对应关系，引导模型学习更好的跨模态注意力权重，从而生成高质量的字幕词。我们提供了理论和实证结果，证明了所提方法的优越性。摘要：With the prevalence of stream media platforms serving music search and recommendation, interpreting music by understanding audio and lyrics interactively has become an important and challenging task. However, many previous works focus on refining individual components of encoder-decoder architecture mapping music to caption tokens, ignoring the potential usage of audio and lyrics correspondence. In this paper, we propose to explicitly learn the multi-modal alignment with retrieval augmentation by contrastive learning. By learning audio-lyrics correspondence, the model is guided to learn better cross-modal attention weights, thus generating high-quality caption words. We provide both theoretical and empirical results that demonstrate the advantage of the proposed method.
【4】 4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders
标题：4D ASR：CTC、注意、传感器和掩码预测解码器的联合建模
链接：https://arxiv.org/abs/2212.10818
*与cs.SD语音【4】为同一篇
作者：Yui Sudo,Muhammad Shakeel,Brian Yan,Jiatong Shi,Shinji Watanabe机构：Honda Research Institute Japan Co., Ltd., Saitama, Japan, Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA备注：Submitted to ICASSP 2023摘要：端到端（E2 E）自动语音识别（ASR）的网络结构可以分为几种模型，包括连接主义时间分类（CTC）模型、递归神经网络转换器（RNN-T）模型、注意机制模型和非自回归掩蔽预测模型。由于这些网络体系结构各有优缺点，因此典型的使用情形是根据应用程序要求切换这些单独的模型，从而导致维护所有模型的开销增加。已经提出了用于集成这些互补模型中的两个以减轻开销问题的若干方法;但是，如果我们集成更多的模型，我们将进一步受益于这些互补的模型，并在单一系统中实现更广泛的应用。本文提出CTC、注意、RNN-T和掩蔽预测的四解码器联合建模（4D），其具有以下三个优点：1）四个解码器被联合训练，使得它们可以根据应用场景容易地切换。2)由于联合训练具有互补性，可以使模型正则化，提高模型的鲁棒性。3)使用CTC、注意力和RNN-T的新的一遍联合解码方法进一步提高了性能。实验结果表明，所提出的模型能够持续降低WER。摘要：The network architecture of end-to-end (E2E) automatic speech recognition (ASR) can be classified into several models, including connectionist temporal classification (CTC), recurrent neural network transducer (RNN-T), attention mechanism, and non-autoregressive mask-predict models. Since each of these network architectures has pros and cons, a typical use case is to switch these separate models depending on the application requirement, resulting in the increased overhead of maintaining all models. Several methods for integrating two of these complementary models to mitigate the overhead issue have been proposed; however, if we integrate more models, we will further benefit from these complementary models and realize broader applications with a single system. This paper proposes four-decoder joint modeling (4D) of CTC, attention, RNN-T, and mask-predict, which has the following three advantages: 1) The four decoders are jointly trained so that they can be easily switched depending on the application scenarios. 2) Joint training may bring model regularization and improve the model robustness thanks to their complementary properties. 3) Novel one-pass joint decoding methods using CTC, attention, and RNN-T further improves the performance. The experimental results showed that the proposed model consistently reduced the WER.
机器翻译，仅供参考
每日语音速递