今天跟大家分享一篇语音相关的论文合集：eess.AS音频处理7篇。

【1】 Automatic music mixing with deep learning and out-of-domain data

标题：具有深度学习和域外数据的自动音乐混合

链接：https://arxiv.org/abs/2208.11428

作者：Marco A. Martínez-Ramírez,Wei-Hsiang Liao,Giorgio Fabbro,Stefan Uhlich,Chihiro Nagashima,Yuki Mitsufuji
机构：ZSony Group Corporation, Tokyo, Japan, Sony Europe B.V., Stuttgart, Germany
备注：23rd International Society for Music Information Retrieval Conference (ISMIR), December, 2022. Source code, demo and audio examples: this https URL
摘要：音乐混合传统上涉及以干净的、单独的音轨的形式记录乐器，并使用音频效果和专业知识（例如，混合工程师）。近年来，音乐制作任务的自动化已经成为一个新兴的领域，其中已经探索了基于规则的方法和机器学习方法。尽管如此，缺乏干燥或干净的乐器录音限制了这类模型的性能，与专业的人工混音仍有很大差距。我们探索是否可以使用域外数据（如湿音乐或经过处理的多轨音乐录音），并将其重新用于训练监督式深度学习模型，以弥补当前自动混音质量方面的差距。为了实现这一点，我们提出了一种新颖的数据预处理方法，允许模型执行自动音乐混合。我们还重新设计了一种用于评估音乐混音系统的听力测试方法。我们使用经验丰富的混合工程师作为参与者，通过这种主观测试来验证我们的结果。
摘要：Music mixing traditionally involves recording instruments in the form of clean, individual tracks and blending them into a final mixture using audio effects and expert knowledge (e.g., a mixing engineer). The automation of music production tasks has become an emerging field in recent years, where rule-based methods and machine learning approaches have been explored. Nevertheless, the lack of dry or clean instrument recordings limits the performance of such models, which is still far from professional human-made mixes. We explore whether we can use out-of-domain data such as wet or processed multitrack music recordings and repurpose it to train supervised deep learning models that can bridge the current gap in automatic mixing quality. To achieve this we propose a novel data preprocessing method that allows the models to perform automatic music mixing. We also redesigned a listening test method for evaluating music mixing systems. We validate our results through such subjective tests using highly experienced mixing engineers as participants.


【2】 A convolutional plane wave model for sound field reconstruction

标题：一种用于声场重建的卷积平面波模型

链接：https://arxiv.org/abs/2208.11324

作者：Manuel Hahmann,Efren Fernandez-Grande
机构：Acoustic Technology Group, Department of Electrical Engineering, Technical University of Denmark, Building , Ørsteds Plads, Kgs. Lyngby, Denmark
摘要：空间声场内插依赖于合适的模型以既符合可用的测量又预测感兴趣域中的声场。当感兴趣的空间域与波长相比较大时，或者当存在球形和平面波阵面时，或者声场复杂时，如在近场中，可能难以确定合适的模型。为了跨越这样的复杂声场，全局重建任务可以被划分为局部子域问题。先前的研究已经表明，由于模型系数的数量较多，划分方法依赖于每个域内的足够测量。该研究提出了一种所有局部子域的联合分析，同时增强了相邻分区之间的自相似性。更具体地，寻求局部平面波表示的系数以具有空间平滑幅度。建立了平面波滤波器的声场卷积模型，并利用乘子交替方向法求解了声场的逆重构问题。仿真和实测声场的实验表明，该方法既保留了局部模型适应复杂声场的灵活性，又保留了全局结构，可以通过较少的测量值进行重构。
摘要：Spatial sound field interpolation relies on suitable models to both conform to available measurements and predict the sound field in the domain of interest. A suitable model can be difficult to determine when the spatial domain of interest is large compared to the wavelength or when spherical and planar wavefronts are present or the sound field is complex, as in the near-field. To span such complex sound fields, the global reconstruction task can be partitioned into local subdomain problems. Previous studies have shown that partitioning approaches rely on sufficient measurements within each domain, due to the higher number of model coefficients. This study proposes a joint analysis of all local subdomains, while enforcing self-similarity between neighbouring partitions. More specifically, the coefficients of local plane wave representations are sought to have spatially smooth magnitudes. A convolutional model of the sound field in terms of plane wave filters is formulated and the inverse reconstruction problem is solved via the alternating direction method of multipliers. The experiments on simulated and measured sound fields suggest, that the proposed method both retains the flexibility of local models to conform to complex sound fields and also preserves the global structure to reconstruct from fewer measurements.


【3】 Interpreting Song Lyrics with an Audio-Informed Pre-trained Language  Model

标题：基于音频预训练语言模型的歌词理解

链接：https://arxiv.org/abs/2208.11671

作者：Yixiao Zhang,Junyan Jiang,Gus Xia,Simon Dixon
机构：Centre for Digital Music, Queen Mary University of London,  Music X Lab, NYU Shanghai,  MBZUAI
备注：Accepted to ISMIR 2022
摘要：歌词解释可以帮助人们快速理解歌曲及其歌词，也可以更容易地管理、检索和从不断增长的音乐档案库中高效地发现歌曲。本文提出了一种基于大规模预训练语言模型和音频编码器相结合的新模型—-BART-fusion，用于从歌词和音乐音频生成歌词解释。我们使用跨模态注意模块将音频表示合并到歌词表示中，以帮助预先训练的语言模型从音频角度理解歌曲，同时保持语言模型的原始生成性能。我们还发布了歌曲解释数据集，这是一个新的大规模数据集，用于训练和评估我们的模型。实验结果表明，附加的音频信息有助于我们的模型更好地理解单词和音乐，并产生准确流畅的解释。另一个跨通道音乐检索实验表明，与原始的BART相比，BART融合生成的解释也可以帮助人们更准确地检索音乐。
摘要：Lyric interpretations can help people understand songs and their lyrics quickly, and can also make it easier to manage, retrieve and discover songs efficiently from the growing mass of music archives. In this paper we propose BART-fusion, a novel model for generating lyric interpretations from lyrics and music audio that combines a large-scale pre-trained language model with an audio encoder. We employ a cross-modal attention module to incorporate the audio representation into the lyrics representation to help the pre-trained language model understand the song from an audio perspective, while preserving the language model's original generative performance. We also release the Song Interpretation Dataset, a new large-scale dataset for training and evaluating our model. Experimental results show that the additional audio information helps our model to understand words and music better, and to generate precise and fluent interpretations. An additional experiment on cross-modal music retrieval shows that interpretations generated by BART-fusion can also help people retrieve music more accurately than with the original BART.


【4】 MEG-MASC: a high-quality magneto-encephalography dataset for evaluating  natural speech processing

标题：MEG-MASC：一种用于评估自然语音处理的高质量脑磁图数据集

链接：https://arxiv.org/abs/2208.11488

作者：Laura Gwilliams,Graham Flick,Alec Marantz,Liina Pylkkanen,David Poeppel,Jean-Remi King
机构：Poeppel, and Jean-Re´ mi King, Department of Neurological Surgery, University of California, San Francisco, Department of Psychology, New York University, New York, NYU Abu Dhabi Institute, Abu Dhabi, Department of Linguistics, New York University, New York
备注：11 pages, 4 figures
摘要：“MEG-MASC”数据集提供了一组精选的原始脑磁图（MEG）记录，记录了27名讲英语的人听了两个小时的自然主义故事。每个参与者进行两个相同的会话，包括听四个虚构的故事，从手动注释子语料库（MASC）混合随机单词表和理解问题。我们在记录的元数据中为每个单词和音素的起始和偏移打上时间戳，并根据“脑成像数据结构”（BIDS）组织数据集。该数据收集为对语音的时间分辨脑反应的大规模编码和解码分析提供了合适的基准。我们提供了Python代码来复制MEG诱发相关字段的几个验证分析，例如语音特征和词频的时间解码。所有代码和MEG、音频和文本数据都是公开的，以保持透明和可复制研究的最佳实践。
摘要：The "MEG-MASC" dataset provides a curated set of raw magnetoencephalography (MEG) recordings of 27 English speakers who listened to two hours of naturalistic stories. Each participant performed two identical sessions, involving listening to four fictional stories from the Manually Annotated Sub-Corpus (MASC) intermixed with random word lists and comprehension questions. We time-stamp the onset and offset of each word and phoneme in the metadata of the recording, and organize the dataset according to the 'Brain Imaging Data Structure' (BIDS). This data collection provides a suitable benchmark to large-scale encoding and decoding analyses of temporally-resolved brain responses to speech. We provide the Python code to replicate several validations analyses of the MEG evoked related fields such as the temporal decoding of phonetic features and word frequency. All code and MEG, audio and text data are publicly available to keep with best practices in transparent and reproducible research.


【5】 Improving Natural-Language-based Audio Retrieval with Transfer Learning  and Audio & Text Augmentations

标题：利用迁移学习和音文增强改进基于自然语言的音频检索

链接：https://arxiv.org/abs/2208.11460

作者：Paul Primus,Gerhard Widmer
机构：Institute of Computational Perception (CP-JKU), LIT Artificial Intelligence Lab, Johannes Kepler University, Austria
备注：submitted to DCASE Workshop 2022
摘要：在深度学习的许多应用领域中，缺乏大的标记数据集仍然是一个重大挑战。研究人员和实践者通常求助于迁移学习和数据扩充来缓解这个问题。我们在使用自然语言查询进行音频检索的背景下研究这些策略（DCASE 2022挑战的任务6b）。我们提出的系统使用预先训练的嵌入模型来将记录和文本描述投影到共享的音频字幕空间中，在该空间中来自不同模态的相关示例是接近的。我们在音频和文本输入上采用各种数据扩增技术，并通过基于模型的顺序优化系统地调整其相应的超参数。实验结果表明，所采用的增强策略减少了过拟合现象，提高了检索性能。我们进一步表明，在AudioCaps数据集上对系统进行预训练会带来额外的改进。
摘要：The absence of large labeled datasets remains a significant challenge in many application areas of deep learning. Researchers and practitioners typically resort to transfer learning and data augmentation to alleviate this issue. We study these strategies in the context of audio retrieval with natural language queries (Task 6b of the DCASE 2022 Challenge). Our proposed system uses pre-trained embedding models to project recordings and textual descriptions into a shared audio-caption space in which related examples from different modalities are close. We employ various data augmentation techniques on audio and text inputs and systematically tune their corresponding hyperparameters with sequential model-based optimization. Our results show that the used augmentations strategies reduce overfitting and improve retrieval performance. We further show that pre-training the system on the AudioCaps dataset leads to additional improvements.


【6】 Improved Zero-Shot Audio Tagging & Classification with Patchout  Spectrogram Transformers

标题：基于Patchout频谱变换的改进零射音频标注与分类

链接：https://arxiv.org/abs/2208.11402

作者：Paul Primus,Gerhard Widmer
机构：Institute of Computational Perception (CP-JKU), LIT Artificial Intelligence Lab, Johannes Kepler University, Austria
备注：published in EUSIPCO 2022
摘要：用于标记和分类声音信号的标准机器学习模型不能处理在训练期间未看到的类别。Zero-Shot（ZS）学习通过基于可适应的类描述来预测类，从而克服了这一限制。本研究旨在探讨基于自我注意的音频嵌入结构对ZS学习的有效性。为此，我们将最近的patchout谱图Transformer与两种经典的卷积结构进行了比较。我们在三个任务和三个不同的基准测试数据集上对这三种架构进行了评估：AudioSet上的通用标签、ESC-50上的环境声音分类以及OpenMIC上的乐器标签。我们的结果表明，在所有这些设置中，基于自注意的嵌入方法都优于两种比较的卷积结构。通过相应地设计训练和测试数据，我们观察到当训练和新测试类之间的'语义距离'很大时，预测性能会显著受损，这种影响值得更详细的研究。
摘要：Standard machine learning models for tagging and classifying acoustic signals cannot handle classes that were not seen during training. Zero-Shot (ZS) learning overcomes this restriction by predicting classes based on adaptable class descriptions. This study sets out to investigate the effectiveness of self-attention-based audio embedding architectures for ZS learning. To this end, we compare the very recent patchout spectrogram transformer with two classic convolutional architectures. We evaluate these three architectures on three tasks and on three different benchmark datasets: general-purpose tagging on AudioSet, environmental sound classification on ESC-50, and instrument tagging on OpenMIC. Our results show that the self-attention-based embedding methods outperform both compared convolutional architectures in all of these settings. By designing training and test data accordingly, we observe that prediction performance suffers significantly when the `semantic distance' between training and new test classes is large, an effect that will deserve more detailed investigations.


【7】 Deep model with built-in self-attention alignment for acoustic echo  cancellation

标题：内置自注意对准的声回声消除深度模型

链接：https://arxiv.org/abs/2208.11308

作者：Evgenii Indenbom,Nicolae-Cătălin Ristea,Ando Saabas,Tanel Pärnamaa,Jegor Gužvin
机构：Microsoft Corporation
摘要：随着最近的研究进展，深度学习模型已成为实时电话会议应用中声学回声消除（AEC）的一个有吸引力的选择。由于声学回声是差的音频质量的主要来源之一，因此已经提出了各种各样的深度模型。然而，对于良好的回声消除质量，一个重要但经常被忽略的要求是麦克风和远端信号的同步。通常使用基于互相关的经典算法来实现，对准模块是具有已知设计限制的单独功能块。在我们的工作中，我们提出了一种深度学习架构，内置基于自我注意的对齐功能，能够处理未对齐的输入，在简化通信管道的同时提高回声消除性能。此外，我们还在AEC Challenge数据集的真实记录上证明了我们的方法在困难的延迟估计情况下取得了显著的改进。
摘要：With recent research advances, deep learning models have become an attractive choice for acoustic echo cancellation (AEC) in real-time teleconferencing applications. Since acoustic echo is one of the major sources of poor audio quality, a wide variety of deep models have been proposed. However, an important but often omitted requirement for good echo cancellation quality is the synchronization of the microphone and far end signals. Typically implemented using classical algorithms based on cross-correlation, the alignment module is a separate functional block with known design limitations. In our work we propose a deep learning architecture with built-in self-attention based alignment, which is able to handle unaligned inputs, improving echo cancellation performance while simplifying the communication pipeline. Moreover, we show that our approach achieves significant improvements for difficult delay estimation cases on real recordings from AEC Challenge data set.


机器翻译，仅供参考