今日论文合集：cs.SD语音6篇，eess.AS音频处理6篇。【1】 InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
标题：InstructTTS：用自然语言风格提示在离散潜在空间中建模富有表现力的TTS
链接：https://arxiv.org/abs/2301.13662
作者：Dongchao Yang,Songxiang Liu,Rongjie Huang,Guangzhi Lei,Chao Weng,Helen Meng,Dong Yu机构：there still exists a big gapbetween TTS-synthetic speech and human speech in terms ofDongchao Yang and Helen Meng are with the Chinese University of HongKong, Rongjie Huang is with the Zhejiang University摘要：可表达的文语转换（TTS）旨在根据人的需要合成不同说话风格的语音。如今，控制说话风格有两种常见的方法：（1）预先定义一组说话风格，用范畴索引来表示不同的说话风格。然而，由于这些模型只能生成预定义的样式，因此在表达的多样性方面存在局限性。(2)使用参考语音作为风格输入，这导致提取的风格信息不直观或不可解释的问题。在本研究中，我们尝试使用自然语言作为风格提示来控制合成语音中的风格，“叹息”的语气中充满了悲伤的情绪，还带着一些无奈的感觉。考虑到现有的TTS语料库尚不适合于测试这一新任务，我们首先构建了一个语音语料库，其语音样本不仅使用内容转录而且使用自然语言的风格描述进行标注。在此基础上，提出了一种具有表达能力的TTS模型InstructTTS，该模型具有以下几个方面的新颖性：（1）充分利用自监督学习和跨模态度量学习的优点，提出了一种新的三阶段训练方法，得到了一个鲁棒的句子嵌入模型，该模型能够有效地从风格提示中获取语义信息，并控制生成语音中的说话风格。(2)我们提出在离散潜在空间中对声学特征建模，并训练一种新的离散扩散概率模型来生成矢量量化（VQ）声学令牌，而不是通常使用的梅尔谱图。(3)在声学模型训练过程中联合应用互信息（MI）估计和最小化，以最小化风格-说话人和风格-内容MI，避免风格提示可能的内容和说话人信息泄漏。摘要：Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, \textit{e.g.}, ``Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named as InstructTTS, which is novel in the sense of following aspects: (1) We fully take the advantage of self-supervised learning and cross-modal metric learning, and propose a novel three-stage training procedure to obtain a robust sentence embedding model, which can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt.
【2】 An Analysis of Classification Approaches for Hit Song Prediction using Engineered Metadata Features with Lyrics and Audio Features
标题：结合歌词和音频特征的工程元数据特征用于热门歌曲预测的分类方法分析
链接：https://arxiv.org/abs/2301.13507
作者：Mengyisong Zhao,Morgan Harvey,David Cameron,Frank Hopfgartner,Valerie J. Gillet机构：Gillet 1 1 The University of Sheffield, uk 2 Universit¨at Koblenz摘要：热门歌曲预测是音乐信息检索领域的一个新兴领域，目前仍然面临着巨大的挑战。能够理解是什么让一首给定的歌曲成为热门歌曲显然对整个音乐产业都有好处。先前的热门歌曲预测方法集中于使用唱片的音频特征。本研究旨在利用Spotify提供的歌曲音频特征、歌词以及新颖的基于元数据的特征（标题主题、流行度连续性和流派类别）等替代性元数据，提高Billboard Hot 100歌曲前10名的预测效果。应用了五种机器学习方法，包括：k近邻、朴素贝叶斯、随机森林、逻辑回归和多层感知器。实验结果表明，随机森林（RF）和Logistic回归（LR）模型的预测准确率分别为89.1%和87.2%，AUC分别为0.91和0.93，优于其他模型。我们的发现也证明了我们新的音乐元数据特征的效用，这对模型的区分性能贡献最大。摘要：Hit song prediction, one of the emerging fields in music information retrieval (MIR), remains a considerable challenge. Being able to understand what makes a given song a hit is clearly beneficial to the whole music industry. Previous approaches to hit song prediction have focused on using audio features of a record. This study aims to improve the prediction result of the top 10 hits among Billboard Hot 100 songs using more alternative metadata, including song audio features provided by Spotify, song lyrics, and novel metadata-based features (title topic, popularity continuity and genre class). Five machine learning approaches are applied, including: k-nearest neighbours, Naive Bayes, Random Forest, Logistic Regression and Multilayer Perceptron. Our results show that Random Forest (RF) and Logistic Regression (LR) with all features (including novel features, song audio features and lyrics features) outperforms other models, achieving 89.1% and 87.2% accuracy, and 0.91 and 0.93 AUC, respectively. Our findings also demonstrate the utility of our novel music metadata features, which contributed most to the models' discriminative performance.
【3】 An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generation
标题：序列音乐生成任务中不同基音和节拍网格编码方法的比较分析
链接：https://arxiv.org/abs/2301.13383
作者：Yuqiang Li,Shengchen Li,George Fazekas机构：Xi’an Jiaotong-Liverpool Universiy†Xi’an Jiaotong-Liverpool Universiy‡Queen Mary University of London 1arXiv备注：This is a draft before submitted to TISMIR as a journal paper摘要：音高和节拍是符号音乐生成任务中的两个基本音乐特征，研究人员通常根据特定的目标选择不同的编码方法。然而，不同编码方法的优点和缺点还没有被频繁地讨论。该文综合分析了音高和节拍两个低层特征对基于符号的序列音乐生成模型性能的影响。首先，比较了常用的MIDI数字编码和一种较少使用的八度编码。第二，将密集的条内度量网格作为辅助特征强加给编码序列。比较了度量网格的不同复杂度和分辨率。在复杂度方面，比较了单标记方法和多标记方法;对于网格分辨率，比较0（消融）、1（小节级）、4（下拍级）、12、（第8三连音级）直到64（第64音符网格级）;对于持续时间分辨率，比较了每拍4、8、12和16个细分。所有不同的编码都在单独训练的Transformer-XL模型上进行了测试，用于旋律生成任务。关于几个客观评价指标与测试数据集的分布相似性，结果表明，在音高相关指标上，类八度音编码显著优于公认的MIDI编码;更细的网格和多标记网格改善了节奏质量，但是在早期训练阶段也遭受过拟合。实验结果从基音嵌入空间和单令牌网格编码的测试损失两个方面揭示了过拟合的普遍现象。从实际应用的角度出发，我们既论证了该方法的可行性，又提出了在生成任务中使用较小网络和较低嵌入维数容易出现过拟合问题的担忧。研究结果也可以为未来的特征工程模型提供参考。摘要：Pitch and meter are two fundamental music features for symbolic music generation tasks, where researchers usually choose different encoding methods depending on specific goals. However, the advantages and drawbacks of different encoding methods have not been frequently discussed. This paper presents a integrated analysis of the influence of two low-level feature, pitch and meter, on the performance of a token-based sequential music generation model. First, the commonly used MIDI number encoding and a less used class-octave encoding are compared. Second, an dense intra-bar metric grid is imposed to the encoded sequence as auxiliary features. Different complexity and resolutions of the metric grid are compared. For complexity, the single token approach and the multiple token approach are compared; for grid resolution, 0 (ablation), 1 (bar-level), 4 (downbeat-level) 12, (8th-triplet-level) up to 64 (64th-note-grid-level) are compared; for duration resolution, 4, 8, 12 and 16 subdivisions per beat are compared. All different encodings are tested on separately trained Transformer-XL models for a melody generation task. Regarding distribution similarity of several objective evaluation metrics to the test dataset, results suggest that the class-octave encoding significantly outperforms the taken-for-granted MIDI encoding on pitch-related metrics; finer grids and multiple-token grids improve the rhythmic quality, but also suffer from over-fitting at early training stage. Results display a general phenomenon of over-fitting from two aspects, the pitch embedding space and the test loss of the single-token grid encoding. From a practical perspective, we both demonstrate the feasibility and raise the concern of easy over-fitting problem of using smaller networks and lower embedding dimensions on the generation task. The findings can also contribute to futural models in terms of feature engineering.
【4】 Automated Time-frequency Domain Audio Crossfades using Graph Cuts
标题：基于图割的自动时频域音频交错衰落
链接：https://arxiv.org/abs/2301.13380
作者：Kyle Robinson,Dan Brown机构：University of Waterloo, Time, Hz摘要：从一个音频剪辑平滑地过渡到另一个音频剪辑的问题出现在许多音乐消费场景中，尤其是当音乐消费已经从专业策划和实况流传输的收音机移动到个人回放设备和服务时。我们给出了朝向通过将频谱离散化为仓（bin）并随后找到每个仓的过渡时间来从一个音频剪辑自动过渡到另一个音频剪辑的新方法的第一步。我们把这个问题描述为一个图流优化问题;特别是最小切割/最大流量。摘要：The problem of transitioning smoothly from one audio clip to another arises in many music consumption scenarios, especially as music consumption has moved from professionally curated and live-streamed radios to personal playback devices and services. we present the first steps toward a new method of automatically transitioning from one audio clip to another by discretizing the frequency spectrum into bins and then finding transition times for each bin. We phrase the problem as one of graph flow optimization; specifically min-cut/max-flow.
【5】 ArchiSound: Audio Generation with Diffusion
标题：档案声音：具有扩散功能的音频生成
链接：https://arxiv.org/abs/2301.13267
机构：Master’s Thesis, Supervised by Zhijing Jin, Prof. Bernhard Schölkopf, ETH Zurich备注：Master Thesis at ETH Zurich摘要：近年来，图像生成的扩散模型的流行使人们对这些模型在媒体生成的其他领域中的潜力产生了新的关注。一个尚未充分探索的领域是扩散模型在音频生成中的应用。音频生成需要理解多个方面，例如时间维度、长期结构、多层重叠声音以及只有受过训练的听众才能察觉的细微差别。在这项工作中，我们研究了扩散模型在音频生成方面的潜力。我们提出了一组模型来解决多个方面的问题，包括一种新的使用堆叠的一维U网进行文本条件潜在音频扩散的方法，该方法可以从文本描述中生成多分钟的音乐。对于每个模型，我们都努力保持合理的推理速度，目标是在单个消费级GPU上实现实时性。除了经过训练的模型之外，我们还提供了一个开源库集合，希望简化该领域未来的工作。样本可在www.example.com上找到。https://bit.ly/audio-diffusion.代码见www.example.com。https://github.com/archinetai/audio-diffusion-pytorch.摘要：The recent surge in popularity of diffusion models for image generation has brought new attention to the potential of these models in other areas of media generation. One area that has yet to be fully explored is the application of diffusion models to audio generation. Audio generation requires an understanding of multiple aspects, such as the temporal dimension, long term structure, multiple layers of overlapping sounds, and the nuances that only trained listeners can detect. In this work, we investigate the potential of diffusion models for audio generation. We propose a set of models to tackle multiple aspects, including a new method for text-conditional latent audio diffusion with stacked 1D U-Nets, that can generate multiple minutes of music from a textual description. For each model, we make an effort to maintain reasonable inference speed, targeting real-time on a single consumer GPU. In addition to trained models, we provide a collection of open source libraries with the hope of simplifying future work in the field. Samples can be found at https://bit.ly/audio-diffusion. Codes are at https://github.com/archinetai/audio-diffusion-pytorch.
【6】 Neural Target Speech Extraction: An Overview
标题：神经目标语音提取：综述
链接：https://arxiv.org/abs/2301.13341
作者：Katerina Zmolikova,Marc Delcroix,Tsubasa Ochiai,Keisuke Kinoshita,Jan Černocký,Dong Yu机构：whichestimates the speech signal of a target speaker in a mixtureKaterina Zmolikova and Jan ˇCernock´y are with Brno University of Tech-nology备注：Submitted to IEEE Signal Processing Magazine on Apr. 25, 2022, and accepted on Jan. 12, 2023摘要：即使在具有噪声、混响和干扰说话者的挑战性声学条件下，人类也可以收听目标说话者。这种现象被称为鸡尾酒会效应。几十年来，研究人员一直致力于研究人类的听力。一个关键问题是处理干扰说话者，因为目标和非目标语音信号共享相似的特性，使得它们的辨别复杂化。目标语音/说话人提取（TSE）使用识别混合物中的说话人的线索从具有或不具有噪声和混响的若干说话人的混合物中分离目标说话人的语音信号。这样的线索可以是指示目标说话者的方向的空间线索、说话者的嘴唇的视频、或者从中可以导出他们的话音特征的预先记录的登记话语。TSE是一个新兴的研究领域，近年来受到越来越多的关注，因为它为鸡尾酒会问题提供了一种实用的方法，涉及音频、视频、阵列处理和深度学习等信号处理方面。本文重点介绍了最新的基于神经元的方法，并对TSE进行了深入的概述。我们引导读者通过不同的主要方法，强调框架之间的相似性，并讨论潜在的未来方向。摘要：Humans can listen to a target speaker even in challenging acoustic conditions that have noise, reverberation, and interfering speakers. This phenomenon is known as the cocktail-party effect. For decades, researchers have focused on approaching the listening ability of humans. One critical issue is handling interfering speakers because the target and non-target speech signals share similar characteristics, complicating their discrimination. Target speech/speaker extraction (TSE) isolates the speech signal of a target speaker from a mixture of several speakers with or without noises and reverberations using clues that identify the speaker in the mixture. Such clues might be a spatial clue indicating the direction of the target speaker, a video of the speaker's lips, or a pre-recorded enrollment utterance from which their voice characteristics can be derived. TSE is an emerging field of research that has received increased attention in recent years because it offers a practical approach to the cocktail-party problem and involves such aspects of signal processing as audio, visual, array processing, and deep learning. This paper focuses on recent neural-based approaches and presents an in-depth overview of TSE. We guide readers through the different major approaches, emphasizing the similarities among frameworks and discussing potential future directions.
【1】 Neural Target Speech Extraction: An Overview
标题：神经目标语音提取：综述
链接：https://arxiv.org/abs/2301.13341
*与cs.SD语音【6】为同一篇
作者：Katerina Zmolikova,Marc Delcroix,Tsubasa Ochiai,Keisuke Kinoshita,Jan Černocký,Dong Yu机构：whichestimates the speech signal of a target speaker in a mixtureKaterina Zmolikova and Jan ˇCernock´y are with Brno University of Tech-nology备注：Submitted to IEEE Signal Processing Magazine on Apr. 25, 2022, and accepted on Jan. 12, 2023摘要：即使在具有噪声、混响和干扰说话者的挑战性声学条件下，人类也可以收听目标说话者。这种现象被称为鸡尾酒会效应。几十年来，研究人员一直致力于研究人类的听力。一个关键问题是处理干扰说话者，因为目标和非目标语音信号共享相似的特性，使得它们的辨别复杂化。目标语音/说话人提取（TSE）使用识别混合物中的说话人的线索从具有或不具有噪声和混响的若干说话人的混合物中分离目标说话人的语音信号。这样的线索可以是指示目标说话者的方向的空间线索、说话者的嘴唇的视频、或者从中可以导出他们的话音特征的预先记录的登记话语。TSE是一个新兴的研究领域，近年来受到越来越多的关注，因为它为鸡尾酒会问题提供了一种实用的方法，涉及音频、视频、阵列处理和深度学习等信号处理方面。本文重点介绍了最新的基于神经元的方法，并对TSE进行了深入的概述。我们引导读者通过不同的主要方法，强调框架之间的相似性，并讨论潜在的未来方向。摘要：Humans can listen to a target speaker even in challenging acoustic conditions that have noise, reverberation, and interfering speakers. This phenomenon is known as the cocktail-party effect. For decades, researchers have focused on approaching the listening ability of humans. One critical issue is handling interfering speakers because the target and non-target speech signals share similar characteristics, complicating their discrimination. Target speech/speaker extraction (TSE) isolates the speech signal of a target speaker from a mixture of several speakers with or without noises and reverberations using clues that identify the speaker in the mixture. Such clues might be a spatial clue indicating the direction of the target speaker, a video of the speaker's lips, or a pre-recorded enrollment utterance from which their voice characteristics can be derived. TSE is an emerging field of research that has received increased attention in recent years because it offers a practical approach to the cocktail-party problem and involves such aspects of signal processing as audio, visual, array processing, and deep learning. This paper focuses on recent neural-based approaches and presents an in-depth overview of TSE. We guide readers through the different major approaches, emphasizing the similarities among frameworks and discussing potential future directions.
【2】 InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
标题：InstructTTS：用自然语言风格提示在离散潜在空间中建模富有表现力的TTS
链接：https://arxiv.org/abs/2301.13662
*与cs.SD语音【1】为同一篇
作者：Dongchao Yang,Songxiang Liu,Rongjie Huang,Guangzhi Lei,Chao Weng,Helen Meng,Dong Yu机构：there still exists a big gapbetween TTS-synthetic speech and human speech in terms ofDongchao Yang and Helen Meng are with the Chinese University of HongKong, Rongjie Huang is with the Zhejiang University摘要：可表达的文语转换（TTS）旨在根据人的需要合成不同说话风格的语音。如今，控制说话风格有两种常见的方法：（1）预先定义一组说话风格，用范畴索引来表示不同的说话风格。然而，由于这些模型只能生成预定义的样式，因此在表达的多样性方面存在局限性。(2)使用参考语音作为风格输入，这导致提取的风格信息不直观或不可解释的问题。在本研究中，我们尝试使用自然语言作为风格提示来控制合成语音中的风格，“叹息”的语气中充满了悲伤的情绪，还带着一些无奈的感觉。考虑到现有的TTS语料库尚不适合于测试这一新任务，我们首先构建了一个语音语料库，其语音样本不仅使用内容转录而且使用自然语言的风格描述进行标注。在此基础上，提出了一种具有表达能力的TTS模型InstructTTS，该模型具有以下几个方面的新颖性：（1）充分利用自监督学习和跨模态度量学习的优点，提出了一种新的三阶段训练方法，得到了一个鲁棒的句子嵌入模型，该模型能够有效地从风格提示中获取语义信息，并控制生成语音中的说话风格。(2)我们提出在离散潜在空间中对声学特征建模，并训练一种新的离散扩散概率模型来生成矢量量化（VQ）声学令牌，而不是通常使用的梅尔谱图。(3)在声学模型训练过程中联合应用互信息（MI）估计和最小化，以最小化风格-说话人和风格-内容MI，避免风格提示可能的内容和说话人信息泄漏。摘要：Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, \textit{e.g.}, ``Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named as InstructTTS, which is novel in the sense of following aspects: (1) We fully take the advantage of self-supervised learning and cross-modal metric learning, and propose a novel three-stage training procedure to obtain a robust sentence embedding model, which can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt.
【3】 An Analysis of Classification Approaches for Hit Song Prediction using Engineered Metadata Features with Lyrics and Audio Features
标题：结合歌词和音频特征的工程元数据特征用于热门歌曲预测的分类方法分析
链接：https://arxiv.org/abs/2301.13507
*与cs.SD语音【2】为同一篇
作者：Mengyisong Zhao,Morgan Harvey,David Cameron,Frank Hopfgartner,Valerie J. Gillet机构：Gillet 1 1 The University of Sheffield, uk 2 Universit¨at Koblenz摘要：热门歌曲预测是音乐信息检索领域的一个新兴领域，目前仍然面临着巨大的挑战。能够理解是什么让一首给定的歌曲成为热门歌曲显然对整个音乐产业都有好处。先前的热门歌曲预测方法集中于使用唱片的音频特征。本研究旨在利用Spotify提供的歌曲音频特征、歌词以及新颖的基于元数据的特征（标题主题、流行度连续性和流派类别）等替代性元数据，提高Billboard Hot 100歌曲前10名的预测效果。应用了五种机器学习方法，包括：k近邻、朴素贝叶斯、随机森林、逻辑回归和多层感知器。实验结果表明，随机森林（RF）和Logistic回归（LR）模型的预测准确率分别为89.1%和87.2%，AUC分别为0.91和0.93，优于其他模型。我们的发现也证明了我们新的音乐元数据特征的效用，这对模型的区分性能贡献最大。摘要：Hit song prediction, one of the emerging fields in music information retrieval (MIR), remains a considerable challenge. Being able to understand what makes a given song a hit is clearly beneficial to the whole music industry. Previous approaches to hit song prediction have focused on using audio features of a record. This study aims to improve the prediction result of the top 10 hits among Billboard Hot 100 songs using more alternative metadata, including song audio features provided by Spotify, song lyrics, and novel metadata-based features (title topic, popularity continuity and genre class). Five machine learning approaches are applied, including: k-nearest neighbours, Naive Bayes, Random Forest, Logistic Regression and Multilayer Perceptron. Our results show that Random Forest (RF) and Logistic Regression (LR) with all features (including novel features, song audio features and lyrics features) outperforms other models, achieving 89.1% and 87.2% accuracy, and 0.91 and 0.93 AUC, respectively. Our findings also demonstrate the utility of our novel music metadata features, which contributed most to the models' discriminative performance.
【4】 An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generation
标题：序列音乐生成任务中不同基音和节拍网格编码方法的比较分析
链接：https://arxiv.org/abs/2301.13383
*与cs.SD语音【3】为同一篇
作者：Yuqiang Li,Shengchen Li,George Fazekas机构：Xi’an Jiaotong-Liverpool Universiy†Xi’an Jiaotong-Liverpool Universiy‡Queen Mary University of London 1arXiv备注：This is a draft before submitted to TISMIR as a journal paper摘要：音高和节拍是符号音乐生成任务中的两个基本音乐特征，研究人员通常根据特定的目标选择不同的编码方法。然而，不同编码方法的优点和缺点还没有被频繁地讨论。该文综合分析了音高和节拍两个低层特征对基于符号的序列音乐生成模型性能的影响。首先，比较了常用的MIDI数字编码和一种较少使用的八度编码。第二，将密集的条内度量网格作为辅助特征强加给编码序列。比较了度量网格的不同复杂度和分辨率。在复杂度方面，比较了单标记方法和多标记方法;对于网格分辨率，比较0（消融）、1（小节级）、4（下拍级）、12、（第8三连音级）直到64（第64音符网格级）;对于持续时间分辨率，比较了每拍4、8、12和16个细分。所有不同的编码都在单独训练的Transformer-XL模型上进行了测试，用于旋律生成任务。关于几个客观评价指标与测试数据集的分布相似性，结果表明，在音高相关指标上，类八度音编码显著优于公认的MIDI编码;更细的网格和多标记网格改善了节奏质量，但是在早期训练阶段也遭受过拟合。实验结果从基音嵌入空间和单令牌网格编码的测试损失两个方面揭示了过拟合的普遍现象。从实际应用的角度出发，我们既论证了该方法的可行性，又提出了在生成任务中使用较小网络和较低嵌入维数容易出现过拟合问题的担忧。研究结果也可以为未来的特征工程模型提供参考。摘要：Pitch and meter are two fundamental music features for symbolic music generation tasks, where researchers usually choose different encoding methods depending on specific goals. However, the advantages and drawbacks of different encoding methods have not been frequently discussed. This paper presents a integrated analysis of the influence of two low-level feature, pitch and meter, on the performance of a token-based sequential music generation model. First, the commonly used MIDI number encoding and a less used class-octave encoding are compared. Second, an dense intra-bar metric grid is imposed to the encoded sequence as auxiliary features. Different complexity and resolutions of the metric grid are compared. For complexity, the single token approach and the multiple token approach are compared; for grid resolution, 0 (ablation), 1 (bar-level), 4 (downbeat-level) 12, (8th-triplet-level) up to 64 (64th-note-grid-level) are compared; for duration resolution, 4, 8, 12 and 16 subdivisions per beat are compared. All different encodings are tested on separately trained Transformer-XL models for a melody generation task. Regarding distribution similarity of several objective evaluation metrics to the test dataset, results suggest that the class-octave encoding significantly outperforms the taken-for-granted MIDI encoding on pitch-related metrics; finer grids and multiple-token grids improve the rhythmic quality, but also suffer from over-fitting at early training stage. Results display a general phenomenon of over-fitting from two aspects, the pitch embedding space and the test loss of the single-token grid encoding. From a practical perspective, we both demonstrate the feasibility and raise the concern of easy over-fitting problem of using smaller networks and lower embedding dimensions on the generation task. The findings can also contribute to futural models in terms of feature engineering.
【5】 Automated Time-frequency Domain Audio Crossfades using Graph Cuts
标题：基于图割的自动时频域音频交错衰落
链接：https://arxiv.org/abs/2301.13380
*与cs.SD语音【4】为同一篇
作者：Kyle Robinson,Dan Brown机构：University of Waterloo, Time, Hz摘要：从一个音频剪辑平滑地过渡到另一个音频剪辑的问题出现在许多音乐消费场景中，尤其是当音乐消费已经从专业策划和实况流传输的收音机移动到个人回放设备和服务时。我们给出了朝向通过将频谱离散化为仓（bin）并随后找到每个仓的过渡时间来从一个音频剪辑自动过渡到另一个音频剪辑的新方法的第一步。我们把这个问题描述为一个图流优化问题;特别是最小切割/最大流量。摘要：The problem of transitioning smoothly from one audio clip to another arises in many music consumption scenarios, especially as music consumption has moved from professionally curated and live-streamed radios to personal playback devices and services. we present the first steps toward a new method of automatically transitioning from one audio clip to another by discretizing the frequency spectrum into bins and then finding transition times for each bin. We phrase the problem as one of graph flow optimization; specifically min-cut/max-flow.
【6】 ArchiSound: Audio Generation with Diffusion
标题：档案声音：具有扩散功能的音频生成
链接：https://arxiv.org/abs/2301.13267
*与cs.SD语音【5】为同一篇
机构：Master’s Thesis, Supervised by Zhijing Jin, Prof. Bernhard Schölkopf, ETH Zurich备注：Master Thesis at ETH Zurich摘要：近年来，图像生成的扩散模型的流行使人们对这些模型在媒体生成的其他领域中的潜力产生了新的关注。一个尚未充分探索的领域是扩散模型在音频生成中的应用。音频生成需要理解多个方面，例如时间维度、长期结构、多层重叠声音以及只有受过训练的听众才能察觉的细微差别。在这项工作中，我们研究了扩散模型在音频生成方面的潜力。我们提出了一组模型来解决多个方面的问题，包括一种新的使用堆叠的一维U网进行文本条件潜在音频扩散的方法，该方法可以从文本描述中生成多分钟的音乐。对于每个模型，我们都努力保持合理的推理速度，目标是在单个消费级GPU上实现实时性。除了经过训练的模型之外，我们还提供了一个开源库集合，希望简化该领域未来的工作。样本可在www.example.com上找到。https://bit.ly/audio-diffusion.代码见www.example.com。https://github.com/archinetai/audio-diffusion-pytorch.摘要：The recent surge in popularity of diffusion models for image generation has brought new attention to the potential of these models in other areas of media generation. One area that has yet to be fully explored is the application of diffusion models to audio generation. Audio generation requires an understanding of multiple aspects, such as the temporal dimension, long term structure, multiple layers of overlapping sounds, and the nuances that only trained listeners can detect. In this work, we investigate the potential of diffusion models for audio generation. We propose a set of models to tackle multiple aspects, including a new method for text-conditional latent audio diffusion with stacked 1D U-Nets, that can generate multiple minutes of music from a textual description. For each model, we make an effort to maintain reasonable inference speed, targeting real-time on a single consumer GPU. In addition to trained models, we provide a collection of open source libraries with the hope of simplifying future work in the field. Samples can be found at https://bit.ly/audio-diffusion. Codes are at https://github.com/archinetai/audio-diffusion-pytorch.
机器翻译，仅供参考
每日语音速递