语音/音频处理学术速递[7.12]

今日论文合集：cs.SD语音8篇，eess.AS音频处理10篇。

cs.SD语音

【1】 ProgGP: From GuitarPro Tablature Neural Generation To Progressive Metal Production标题：ProgGP：从吉他专业神经生成到渐进式金属生产链接：https://arxiv.org/abs/2307.05328

作者：Jackson Loth,Pedro Sarmento,CJ Carr,Zack Zukowski,Mathieu Barthet机构：Queen Mary University of London, United Kingdom备注：Pre-print accepted for publication at CMMR2023摘要：最近在符号音乐生成领域的工作已经显示出使用基于GuitarPro格式的标记化作为输入和输出表示的价值，GuitarPro格式是支持吉他表达属性的符号表示。我们通过在ProgGP上微调预训练的Transformer模型来扩展这项工作，ProgGP是一个包含173首渐进金属歌曲的自定义数据集，目的是通过人类-AI合作伙伴关系从该类型中创建作品。我们的模型能够生成多个吉他，低音吉他，鼓，钢琴和管弦乐部分。我们研究的有效性，生成的音乐使用混合方法的方法相结合的定量分析，计算音乐学的范式和定性分析，以实践为基础的研究范式。最后，我们通过使用该模型作为创建渐进金属歌曲的工具来展示该模型的价值，该歌曲由人类金属制作人基于AI生成的音乐完全制作和混合。摘要：Recent work in the field of symbolic music generation has shown value in using a tokenization based on the GuitarPro format, a symbolic representation supporting guitar expressive attributes, as an input and output representation. We extend this work by fine-tuning a pre-trained Transformer model on ProgGP, a custom dataset of 173 progressive metal songs, for the purposes of creating compositions from that genre through a human-AI partnership. Our model is able to generate multiple guitar, bass guitar, drums, piano and orchestral parts. We examine the validity of the generated music using a mixed methods approach by combining quantitative analyses following a computational musicology paradigm and qualitative analyses following a practice-based research paradigm. Finally, we demonstrate the value of the model by using it as a tool to create a progressive metal song, fully produced and mixed by a human metal producer based on AI-generated music.

【2】 ShredGP: Guitarist Style-Conditioned Tablature Generation标题：ShredGP：吉他手风格受限的表演者一代链接：https://arxiv.org/abs/2307.05324

作者：Pedro Sarmento,Adarsh Kumar,Dekun Xie,CJ Carr,Zack Zukowski,Mathieu Barthet机构：Queen Mary University of London, Indian Institute of Technology Kharaghpur, Dadabots备注：Accepted for publication at CMMR 2023摘要：GuitarPro格式表格是一种数字音乐符号，它封装了有关吉他演奏技术和指法的信息。我们介绍ShredGP，一个基于GuitarPro指法生成Transformer的模型，用于模仿四个不同的标志性电吉他手的风格。为了评估每个吉他手的特质，我们采用了计算音乐学方法，通过分析从DadaGP编码方案产生的令牌计算的功能。统计分析的特点证据之间的显着差异的四个吉他手。我们训练了ShredGP模型的两个变体，一个使用多乐器语料库，另一个使用独奏吉他数据。我们提出了一个基于BERT的吉他手分类模型，并用它来评估生成的例子。总体而言，来自分类器的结果表明，ShredGP能够生成与目标吉他手的风格一致的内容。最后，我们反思了ShredGP在人类-AI音乐交互中的应用前景。摘要：GuitarPro format tablatures are a type of digital music notation that encapsulates information about guitar playing techniques and fingerings. We introduce ShredGP, a GuitarPro tablature generative Transformer-based model conditioned to imitate the style of four distinct iconic electric guitarists. In order to assess the idiosyncrasies of each guitar player, we adopt a computational musicology methodology by analysing features computed from the tokens yielded by the DadaGP encoding scheme. Statistical analyses of the features evidence significant differences between the four guitarists. We trained two variants of the ShredGP model, one using a multi-instrument corpus, the other using solo guitar data. We present a BERT-based model for guitar player classification and use it to evaluate the generated examples. Overall, results from the classifier show that ShredGP is able to generate content congruent with the style of the targeted guitar player. Finally, we reflect on prospective applications for ShredGP for human-AI music interaction.

【3】 On the Effectiveness of Speech Self-supervised Learning for Music标题：浅谈音乐语音自主学习的有效性链接：https://arxiv.org/abs/2307.05161

作者：Yinghao Ma,Ruibin Yuan,Yizhi Li,Ge Zhang,Xingran Chen,Hanzhi Yin,Chenghua Lin,Emmanouil Benetos,Anton Ragni,Norbert Gyenge,Ruibo Liu,Gus Xia,Roger Dannenberg,Yike Guo,Jie Fu机构：, Multimodal Art Projection Research Community, Queen Mary University of London, Beijing Academy of Artificial Intelligence, Carnegie Mellon University, University of Sheffield, University of Waterloo, University of Michigan Ann Arbor, Dartmouth College摘要：自监督学习（SSL）在各种语音和自然语言处理应用中显示出有希望的结果。然而，它在音乐信息检索（MIR）的功效仍然在很大程度上未被探索。虽然以前在音乐录音上预训练的SSL模型可能大多是闭源的，但最近的语音模型（如wav2vec2.0）在音乐建模方面表现出了希望。然而，探索将语音SSL模型应用于音乐录音的有效性的研究有限。我们探索SSL的音乐适应与两个独特的语音相关的模型，data2vec1.0和Hubert，并将它们分别称为music 2 vec和musicHuBERT。我们在各种预训练配置下用95 M参数训练了12 $ SSL模型，并用13个不同的MIR任务系统地评估了MIR任务的性能。我们的研究结果表明，使用音乐数据进行训练通常可以提高MIR任务的性能，即使模型是使用为语音设计的范例进行训练的。然而，我们确定的局限性，这种现有的面向语音的设计，特别是在建模复调信息。基于实验结果，实证建议，设计未来的音乐SSL的策略和范式。摘要：Self-supervised learning (SSL) has shown promising results in various speech and natural language processing applications. However, its efficacy in music information retrieval (MIR) still remains largely unexplored. While previous SSL models pre-trained on music recordings may have been mostly closed-sourced, recent speech models such as wav2vec2.0 have shown promise in music modelling. Nevertheless, research exploring the effectiveness of applying speech SSL models to music recordings has been limited. We explore the music adaption of SSL with two distinctive speech-related models, data2vec1.0 and Hubert, and refer to them as music2vec and musicHuBERT, respectively. We train $12$ SSL models with 95M parameters under various pre-training configurations and systematically evaluate the MIR task performances with 13 different MIR tasks. Our findings suggest that training with music data can generally improve performance on MIR tasks, even when models are trained using paradigms designed for speech. However, we identify the limitations of such existing speech-oriented designs, especially in modelling polyphonic information. Based on the experimental results, empirical suggestions are also given for designing future musical SSL strategies and paradigms.

【4】 Optimizing Feature Extraction for Symbolic Music标题：符号音乐特征提取的优化方法链接：https://arxiv.org/abs/2307.05107

作者：Federico Simonetta,Ana Llorens,Martín Serrano,Eduardo García-Portugués,Álvaro Torrente机构：ICCMU - Instituto Complutense de Ciencias Musicales, Madrid, Universidad Complutense de Madrid, Madrid, Universidad Carlos III, Madrid备注：Published at ISMIR 2023摘要：本文提出了一个全面的调查现有的特征提取工具的符号音乐和对比他们的性能，以确定一个给定的乐谱的音乐风格的特征集。在这方面，我们提出了一种新的特征提取工具，名为musif，并评估其功效的各种曲目和文件格式，包括MIDI，MusicXML，和 **kern。Musif在计算效率方面接近现有的工具，如jSymbolic和music21，同时试图提高自定义功能开发的可用性。所提出的工具还提高了分类精度时，与其他集的功能。我们展示了每一组功能和它们所需的计算资源的贡献。我们的研究结果表明，特征提取的最佳工具是从每个工具，而不是一个单一的最佳功能的组合。为了方便今后的研究在音乐信息检索，我们发布的工具和基准测试的源代码。摘要：This paper presents a comprehensive investigation of existing feature extraction tools for symbolic music and contrasts their performance to determine the set of features that best characterizes the musical style of a given music score. In this regard, we propose a novel feature extraction tool, named musif, and evaluate its efficacy on various repertoires and file formats, including MIDI, MusicXML, and **kern. Musif approximates existing tools such as jSymbolic and music21 in terms of computational efficiency while attempting to enhance the usability for custom feature development. The proposed tool also enhances classification accuracy when combined with other sets of features. We demonstrate the contribution of each set of features and the computational resources they require. Our findings indicate that the optimal tool for feature extraction is a combination of the best features from each tool rather than those of a single one. To facilitate future research in music information retrieval, we release the source code of the tool and benchmarks.

【5】 The smarty4covid dataset and knowledge base: a framework enabling interpretable analysis of audio signals标题：Smarty4covid数据集和知识库：实现音频信号可解释分析的框架链接：https://arxiv.org/abs/2307.05096

作者：Konstantia Zarkogianni,Edmund Dervakos,George Filandrianos,Theofanis Ganitidis,Vasiliki Gkatzou,Aikaterini Sakagianni,Raghu Raghavendra,C. L. Max Nikias,Giorgos Stamou,Konstantina S. Nikita机构：National Technical University of Athens, School of Electrical and Computer Engineering, Athens, Greece, Sismanoglion General Hospital, Department of Intensive Care Unit, Athens, Greece备注：Submitted for publication in Nature Scientific Data摘要：利用人工智能（AI）和移动健康的力量来检测指示呼吸异常/病症的发作和进展的新生物标志物，极大地吸引了科学和研究兴趣，尤其是在COVID-19大流行期间。smarty 4covid数据集包含咳嗽（4，676），规律呼吸（4，665），深呼吸（4，695）和语音（4，291）的音频信号，这些信号是通过移动设备按照众包方法记录的。还包括其他自我报告的信息（例如：COVID-19病毒检测），从而为开发COVID-19风险检测模型提供全面的数据集。smarty 4covid数据集以网络本体语言（OWL）知识库的形式发布，能够从其他相关数据集、复杂查询和推理中整合数据。它已被用于开发能够：（i）从常规呼吸记录中提取临床信息性呼吸指标，以及（ii）识别众包音频记录中的咳嗽、呼吸和语音片段。提出并验证了一个新的框架，该框架利用smarty 4covid OWL知识库，在不透明的基于AI的COVID-19风险检测模型中生成反事实解释。摘要：Harnessing the power of Artificial Intelligence (AI) and m-health towards detecting new bio-markers indicative of the onset and progress of respiratory abnormalities/conditions has greatly attracted the scientific and research interest especially during COVID-19 pandemic. The smarty4covid dataset contains audio signals of cough (4,676), regular breathing (4,665), deep breathing (4,695) and voice (4,291) as recorded by means of mobile devices following a crowd-sourcing approach. Other self reported information is also included (e.g. COVID-19 virus tests), thus providing a comprehensive dataset for the development of COVID-19 risk detection models. The smarty4covid dataset is released in the form of a web-ontology language (OWL) knowledge base enabling data consolidation from other relevant datasets, complex queries and reasoning. It has been utilized towards the development of models able to: (i) extract clinically informative respiratory indicators from regular breathing records, and (ii) identify cough, breath and voice segments in crowd-sourced audio recordings. A new framework utilizing the smarty4covid OWL knowledge base towards generating counterfactual explanations in opaque AI-based COVID-19 risk detection models is proposed and validated.

【6】 LaunchpadGPT: Language Model as Music Visualization Designer on Launchpad标题：LaunchpadGPT：LaunchPad上音乐可视化设计者的语言模型链接：https://arxiv.org/abs/2307.04827

作者：Siting Xu,Yunlong Tang,Feng Zheng机构：Southern University of Science and Technology备注：Accepted by International Computer Music Conference (ICMC) 2023摘要：Launchpad是一种乐器，允许用户通过按下发光按钮来创建和演奏音乐。为了帮助和启发Launchpad灯光效果的设计，并为初学者提供一个更容易使用该仪器创建音乐可视化的方法，我们提出了LaunchpadGPT模型自动生成Launchpad上的音乐可视化设计。基于具有良好的生成能力的语言模型，我们提出的LaunchpadGPT以音频音乐作为输入，并以视频（Launchpad-playing video）的形式输出Launchpad-playing的灯光效果。我们收集Launchpad播放视频并对其进行处理，以获得音乐和Launchpad播放的相应视频帧作为提示-完成对，以训练语言模型。实验结果表明，该方法可以创建更好的音乐可视化比随机生成方法，并持有更广泛的音乐可视化应用的潜力。我们的代码可在www.example.com上获得https://github.com/yunlong10/LaunchpadGPT/。摘要：Launchpad is a musical instrument that allows users to create and perform music by pressing illuminated buttons. To assist and inspire the design of the Launchpad light effect, and provide a more accessible approach for beginners to create music visualization with this instrument, we proposed the LaunchpadGPT model to generate music visualization designs on Launchpad automatically. Based on the language model with excellent generation ability, our proposed LaunchpadGPT takes an audio piece of music as input and outputs the lighting effects of Launchpad-playing in the form of a video (Launchpad-playing video). We collect Launchpad-playing videos and process them to obtain music and corresponding video frame of Launchpad-playing as prompt-completion pairs, to train the language model. The experiment result shows the proposed method can create better music visualization than random generation methods and hold the potential for a broader range of music visualization applications. Our code is available at https://github.com/yunlong10/LaunchpadGPT/.

【7】 Aeroacoustic testing on a full aircraft model at high Reynolds numbers in the European Transonic Windtunnel标题：欧洲跨音速风洞中高雷诺数全飞机模型的气动声学试验链接：https://arxiv.org/abs/2307.05140

作者：Thomas Ahlefeldt,Daniel Ernst,Armin Goudarzi,Hans-Georg-Raumer,Carsten Spehr机构：German Aerospace Center (DLR), G¨ottingen, Germany备注：None摘要：本文提出了一种端到端的方法，用于评估巴航工业公司的一个缩比完整模型的加压和低温风洞测量接近真实世界的雷诺数。讨论了传声器的选择、测量参数、阵列的设计和流量参数的选择。提出了不同的风洞条件，允许分离的雷诺数的影响，从马赫数，以及开槽和封闭的测试部分的影响。本文提供了三维波束形成的结果与CLEAN-SC反卷积，感兴趣的区域的选择，以及相应的源谱。结果表明，开槽的测试部分的波束形成的结果相比，封闭的测试部分和雷诺数有一个深刻的，非线性的影响，减少与雷诺数的航空声发射的影响很小。此外，源显示出在恒定雷诺数下的非线性马赫数依赖性，但在观察到的马赫数范围内是自相似的。研究结果表明，有可能在真实世界的雷诺数，这使得在未来的进一步调查，如源的方向性的小规模的完整模型上研究现实世界的现象。摘要：This paper presents an end-to-end approach for the assessment of pressurized and cryogenic wind tunnel measurements of an EMBRAER scaled full model close to real-world Reynolds numbers. The choice of microphones, measurement parameters, the design of the array, and the selection of flow parameters are discussed. Different wind tunnel conditions are proposed which allow separating the influence of the Reynolds number from the Mach number, as well as the influence of slotted and closed test sections. The paper provides three-dimensional beamforming results with CLEAN-SC deconvolution, the selection of regions of interest, and the corresponding source spectra. The results suggest that slotted test sections have little influence on the beamforming results compared to closed test sections and that the Reynolds number has a profound, non-linear impact on the aeroacoustic emission that lessens with increasing Reynolds number. Further, sources show a non-linear Mach number dependency at constant Reynolds number but are self-similar in the observed Mach number range. The findings suggest that it is possible to study real-world phenomena on small-scale full models at real-world Reynolds numbers, which enable further investigations in the future such as the directivity of sources.

【8】 On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis标题：自监督语音表征在自发语音合成中的应用链接：https://arxiv.org/abs/2307.05132

作者：Siyang Wang,Gustav Eje Henter,Joakim Gustafson,Éva Székely机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：7 pages, 2 figures. 12th ISCA Speech Synthesis Workshop (SSW) 2023摘要：自监督学习（SSL）语音表示从大量不同的、混合质量的语音数据中学习，而不需要转录，在许多语音技术应用中越来越普及。先前的工作已经表明，SSL是一个有效的中间表示在两个阶段的文本到语音（TTS）的阅读和自发语音。然而，它仍然不清楚哪个SSL和哪个层从每个SSL模型是最适合自发TTS。我们通过将自发TTS中SSL的比较范围扩展到6个不同的SSL和每个SSL内的3层来解决这个缺点。此外，SSL也显示出潜在的预测合成语音的平均意见分数（MOS），但这只在阅读语音MOS预测。我们扩展了一个基于SSL的MOS预测框架以前开发的评分阅读语音合成和评估其性能的合成自发的语音。所有实验在两个不同的自发语料库上进行了两次，以找到可推广的趋势。总体而言，我们提出了全面的实验结果，使用SSL在自发TTS和MOS预测，以进一步量化和了解SSL如何可以用于自发TTS。音频样本：https://www.speech.kth.se/tts-demos/sp_ssl_tts摘要：Self-supervised learning (SSL) speech representations learned from large amounts of diverse, mixed-quality speech data without transcriptions are gaining ground in many speech technology applications. Prior work has shown that SSL is an effective intermediate representation in two-stage text-to-speech (TTS) for both read and spontaneous speech. However, it is still not clear which SSL and which layer from each SSL model is most suited for spontaneous TTS. We address this shortcoming by extending the scope of comparison for SSL in spontaneous TTS to 6 different SSLs and 3 layers within each SSL. Furthermore, SSL has also shown potential in predicting the mean opinion scores (MOS) of synthesized speech, but this has only been done in read-speech MOS prediction. We extend an SSL-based MOS prediction framework previously developed for scoring read speech synthesis and evaluate its performance on synthesized spontaneous speech. All experiments are conducted twice on two different spontaneous corpora in order to find generalizable trends. Overall, we present comprehensive experimental results on the use of SSL in spontaneous TTS and MOS prediction to further quantify and understand how SSL can be used in spontaneous TTS. Audios samples: https://www.speech.kth.se/tts-demos/sp_ssl_tts

eess.AS音频处理

【1】 On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis标题：自监督语音表征在自发语音合成中的应用链接：https://arxiv.org/abs/2307.05132

【2】 Predicting Tuberculosis from Real-World Cough Audio Recordings and Metadata标题：从真实世界的咳嗽录音和元数据预测结核病链接：https://arxiv.org/abs/2307.04842

作者：George P. Kafentzis,Stephane Tetsing,Joe Brew,Lola Jover,Mindaugas Galvosas,Carlos Chaccour,Peter M. Small机构：Hyfe Inc., U.S.A, ISGlobal, Barcelona Institute for Global Health, Spain, University of Washington, Department of Global Health摘要：结核病（TB）是由结核分枝杆菌（Mycobacterium tuberculosis）细菌引起的感染性疾病，并且主要影响肺以及其他身体部位。当感染者咳嗽、打喷嚏或说话时，结核病通过空气传播。医生通过临床检查和专门测试来诊断患者的结核病。然而，咳嗽是肺结核等呼吸道疾病的常见症状。文献表明，来自不同呼吸系统疾病的咳嗽声可以由医生和计算机算法两者区分。因此，与患有和不患有TB的患者相关的咳嗽记录似乎是合理的调查途径。在这项工作中，我们利用了一个非常大的数据集的结核病和非结核病咳嗽录音从非洲东南部，印度和东南亚的亚洲使用一个完全自动化的基于电话的应用程序（Hyfe），没有手动注释。我们适合统计分类器的基础上，频谱和时域特征，有和没有临床元数据。分层分组交叉验证的方法表明，平均曲线下面积（AUC）约0.70 $\pm $0.05的咳嗽级别和参与者级别的分类，可以实现单独使用咳嗽声。人口统计学和临床因素的加入增加了性能，导致平均AUC约为0.81 $\pm $0.05。我们的研究结果表明，基于手机的应用程序，整合临床症状和咳嗽声分析，可以帮助社区卫生工作者，最重要的是，卫生服务计划，以改善结核病病例发现工作，同时降低成本，这可能会大大改善公共卫生。摘要：Tuberculosis (TB) is an infectious disease caused by the bacterium Mycobacterium tuberculosis and primarily affects the lungs, as well as other body parts. TB is spread through the air when an infected person coughs, sneezes, or talks. Medical doctors diagnose TB in patients via clinical examinations and specialized tests. However, coughing is a common symptom of respiratory diseases such as TB. Literature suggests that cough sounds coming from different respiratory diseases can be distinguished by both medical doctors and computer algorithms. Therefore, cough recordings associated with patients with and without TB seems to be a reasonable avenue of investigation. In this work, we utilize a very large dataset of TB and non-TB cough audio recordings obtained from the south-east of Africa, India, and the south-east of Asia using a fully automated phone-based application (Hyfe), without manual annotation. We fit statistical classifiers based on spectral and time domain features with and without clinical metadata. A stratified grouped cross-validation approach shows that an average Area Under Curve (AUC) of approximately 0.70 $\pm$ 0.05 both for a cough-level and a participant-level classification can be achieved using cough sounds alone. The addition of demographic and clinical factors increases performance, resulting in an average AUC of approximately 0.81 $\pm$ 0.05. Our results suggest mobile phone-based applications that integrate clinical symptoms and cough sound analysis could help community health workers and, most importantly, health service programs to improve TB case-finding efforts while reducing costs, which could substantially improve public health.

【3】 ProgGP: From GuitarPro Tablature Neural Generation To Progressive Metal Production标题：ProgGP：从吉他专业神经生成到渐进式金属生产链接：https://arxiv.org/abs/2307.05328

【4】 ShredGP: Guitarist Style-Conditioned Tablature Generation标题：ShredGP：吉他手风格受限的表演者一代链接：https://arxiv.org/abs/2307.05324

【5】 On the Effectiveness of Speech Self-supervised Learning for Music标题：浅谈音乐语音自主学习的有效性链接：https://arxiv.org/abs/2307.05161

【6】 Aeroacoustic testing on a full aircraft model at high Reynolds numbers in the European Transonic Windtunnel标题：欧洲跨音速风洞中高雷诺数全飞机模型的气动声学试验链接：https://arxiv.org/abs/2307.05140

【7】 Optimizing Feature Extraction for Symbolic Music标题：符号音乐特征提取的优化方法链接：https://arxiv.org/abs/2307.05107

【8】 The smarty4covid dataset and knowledge base: a framework enabling interpretable analysis of audio signals标题：Smarty4covid数据集和知识库：实现音频信号可解释分析的框架链接：https://arxiv.org/abs/2307.05096

【9】 Improving RNN-Transducers with Acoustic LookAhead标题：用声学前视技术改进RNN换能器链接：https://arxiv.org/abs/2307.05006

作者：Vinit S. Unni,Ashish Mittal,Preethi Jyothi,Sunita Sarawagi机构：Indian Institute of Technology Bombay, India, IBM Research, India备注：5 pages, 1 fig, 7 tables, Proceedings of Interspeech 2023摘要：RNN-Transducers（RNN-T）由于其高准确性和流式传输功能，已被广泛接受为语音到文本转换的端到端模型。典型的RNN-T独立地对输入音频和文本上下文进行编码，并通过瘦联合网络将这两种编码组合起来。虽然这种架构提供了SOTA流式传输的准确性，但它也使模型容易受到强烈的LM偏置的影响，这表现为没有声学证据的文本的多步幻觉。在本文中，我们提出了LookAhead，使文本表示更声学接地，展望未来的音频输入。这种技术产生了显着的5%-20%的相对减少，在域内和域外的评估集的字错误率。摘要：RNN-Transducers (RNN-Ts) have gained widespread acceptance as an end-to-end model for speech to text conversion because of their high accuracy and streaming capabilities. A typical RNN-T independently encodes the input audio and the text context, and combines the two encodings by a thin joint network. While this architecture provides SOTA streaming accuracy, it also makes the model vulnerable to strong LM biasing which manifests as multi-step hallucination of text without acoustic evidence. In this paper we propose LookAhead that makes text representations more acoustically grounded by looking ahead into the future within the audio input. This technique yields a significant 5%-20% relative reduction in word error rate on both in-domain and out-of-domain evaluation sets.

【10】 LaunchpadGPT: Language Model as Music Visualization Designer on Launchpad标题：LaunchpadGPT：LaunchPad上音乐可视化设计者的语言模型链接：https://arxiv.org/abs/2307.04827

机器翻译，仅供参考

语音/音频处理学术速递[7.12]

评论 0