今日论文合集：cs.SD语音5篇，eess.AS音频处理6篇。
【1】 Validity in Music Information Research Experiments
标题：音乐信息研究实验中的效度
链接：https://arxiv.org/abs/2301.01578
作者：Bob L. T. Sturm,Arthur Flexer机构：Division of Speech, Music and Hearing, KTH Stockholm, Sweden, Institute of Computational Perception, Johannes Kepler摘要：有效性是根据证据（如实验中收集的数据）做出的推论的真实性，是科学工作的核心。考虑到音乐信息研究领域的成熟，我们认为有效性应该比迄今为止得到更多的讨论和考虑。在工作中考虑有效性可以提高其科学价值和工程价值。令人困惑的MIR现象，如对抗性攻击和性能玻璃天花板，通过有效性的镜头变得不那么神秘。在这篇文章中，我们回顾了有效性的主题一般，考虑四个主要类型的有效性从一个关键的参考：Shadish等人，2002年。我们以一个典型的MIR实验为基础来讨论这些类型：使用机器学习的音乐分类。通过这种方法，可以指导MIR实验者从他们的实验中收集的数据中做出有效的推断。摘要：Validity is the truth of an inference made from evidence, such as data collected in an experiment, and is central to working scientifically. Given the maturity of the domain of music information research (MIR), validity in our opinion should be discussed and considered much more than it has been so far. Considering validity in one's work can improve its scientific and engineering value. Puzzling MIR phenomena like adversarial attacks and performance glass ceilings become less mysterious through the lens of validity. In this article, we review the subject of validity in general, considering the four major types of validity from a key reference: Shadish et al. 2002. We ground our discussion of these types with a prototypical MIR experiment: music classification using machine learning. Through this MIR experimentalists can be guided to make valid inferences from data collected from their experiments.
【2】 Audio-Visual Efficient Conformer for Robust Speech Recognition
标题：一种用于稳健语音识别的高效音视同步器
链接：https://arxiv.org/abs/2301.01456
作者：Maxime Burchi,Radu Timofte机构：Computer Vision Lab, CAIDAS, IFI, University of W¨urzburg, Germany摘要：近年来，基于神经网络的端到端自动语音识别（ASR）系统得到了很大的发展。大规模人工标注数据集的可用性和充足的计算资源使得训练强大的深度神经网络成为可能，在学术基准上达到非常低的字错误率（WER）。然而，尽管在干净的音频样本上有令人印象深刻的性能，但在嘈杂的语音上经常观察到性能下降。在这项工作中，我们建议通过处理音频和视频模态来提高最近提出的基于高效一致性连接主义时间分类（CTC）的架构的噪声鲁棒性。我们在ResNet-18视觉前端的基础上使用高效Conformer后端，并在区块之间添加中间CTC损失，从而改进了之前的唇读方法。我们使用CTC间残差模块将中间块特征条件化在早期预测上，以放松基于CTC的模型的条件独立性假设。我们还用一种更有效、更简单的注意机制（我们称之为补丁注意）来取代高效构象器分组注意。我们使用公开的唇读句子2（LRS2）和唇读句子3（LRS3）数据集进行实验。我们的实验表明，使用音频和视觉模态允许在存在环境噪声的情况下更好地识别语音，并显著加速训练，以少4倍的训练步骤达到较低的WER。我们的视听高效一致性（AVEC）模型实现了最先进的性能，在LRS2和LRS3测试集上的WER分别为2.3%和1.8%。Code and pretrained models are available at https://github.com/burchim/AVEC.摘要：End-to-end Automatic Speech Recognition (ASR) systems based on neural networks have seen large improvements in recent years. The availability of large scale hand-labeled datasets and sufficient computing resources made it possible to train powerful deep neural networks, reaching very low Word Error Rate (WER) on academic benchmarks. However, despite impressive performance on clean audio samples, a drop of performance is often observed on noisy speech. In this work, we propose to improve the noise robustness of the recently proposed Efficient Conformer Connectionist Temporal Classification (CTC)-based architecture by processing both audio and visual modalities. We improve previous lip reading methods using an Efficient Conformer back-end on top of a ResNet-18 visual front-end and by adding intermediate CTC losses between blocks. We condition intermediate block features on early predictions using Inter CTC residual modules to relax the conditional independence assumption of CTC-based models. We also replace the Efficient Conformer grouped attention by a more efficient and simpler attention mechanism that we call patch attention. We experiment with publicly available Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3) datasets. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps. Our Audio-Visual Efficient Conformer (AVEC) model achieves state-of-the-art performance, reaching WER of 2.3% and 1.8% on LRS2 and LRS3 test sets. Code and pretrained models are available at https://github.com/burchim/AVEC.
【3】 An ensemble-based framework for mispronunciation detection of Arabic phonemes
标题：一种基于集成的阿拉伯音素发音错误检测框架
链接：https://arxiv.org/abs/2301.01378
作者：Sukru Selim Calik,Ayhan Kucukmanisa,Zeynep Hilal Kilimci机构：Maviay Consultancy Company, Kocaeli University Technopark, Kocaeli, Turkey, Department of Electronics and Communication Engineering, Kocaeli University, Kocaeli, Turkey, Department of Information Systems Engineering, Kocaeli University, Kocaeli, Turkey摘要：通过计算机辅助语言学习（CALL）系统来确定错误发音并确保向用户提供反馈。在这项工作中，我们介绍了一个集合模型，定义阿拉伯语音素的错误发音，并帮助学习阿拉伯语，有效。据我们所知，这是第一次尝试确定阿拉伯语音素的错误发音采用集成学习技术和传统的机器学习模型，全面。为了观察特征提取技术的效果，将梅尔倒谱系数（MFCC）和梅尔谱图与每种学习算法混合。为了显示所提出的模型的成功，阿拉伯语音素中的29个字母，其中8个是hafiz，由总共11个不同的人发声。采用加噪、时移、时间拉伸、基音偏移等方法提高了数据集的容量。实验结果表明，投票分类器与Mel谱图特征提取技术的集成算法具有显著的分类效果，分类准确率达95.9%。摘要：Determination of mispronunciations and ensuring feedback to users are maintained by computer-assisted language learning (CALL) systems. In this work, we introduce an ensemble model that defines the mispronunciation of Arabic phonemes and assists learning of Arabic, effectively. To the best of our knowledge, this is the very first attempt to determine the mispronunciations of Arabic phonemes employing ensemble learning techniques and conventional machine learning models, comprehensively. In order to observe the effect of feature extraction techniques, mel-frequency cepstrum coefficients (MFCC), and Mel spectrogram are blended with each learning algorithm. To show the success of proposed model, 29 letters in the Arabic phonemes, 8 of which are hafiz, are voiced by a total of 11 different person. The amount of data set has been enhanced employing the methods of adding noise, time shifting, time stretching, pitch shifting. Extensive experiment results demonstrate that the utilization of voting classifier as an ensemble algorithm with Mel spectrogram feature extraction technique exhibits remarkable classification result with 95.9% of accuracy.
【4】 Quantum Representations of Sound: from mechanical waves to quantum circuits
标题：声音的量子表示：从机械波到量子电路
链接：https://arxiv.org/abs/2301.01595
作者：Paulo V. Itaborai,Eduardo R. Miranda机构：Interdisciplinary Centre for Computer Music Research (ICCMR), University of Plymouth, Plymouth, UK备注：29 pages,26 figures. Accompanying Python package is available: this https URL摘要：在写这篇文章的时候，量子音频仍然是一个非常年轻的研究领域，即使在量子信号处理社区也是如此。本章介绍了量子音频的发展现状，并讨论了音频信号的量子表示方法。目前，还没有量子表示策略声称是音频应用的最佳策略。每一种方法都有其优点和缺点。可以认为，未来的量子音频表示方案将利用针对特定应用的多种策略。注：这是一个未经编辑的节略版本的预提交草案的一章，具有相同的标题，出版在书量子计算机音乐：基础、方法和高级概念》，作者E.红米兰达（第223 - 274页）。请参阅本书中的版本，了解应用示例和基于量子音频表示的声音合成方法及其开发新型乐器的潜力的讨论。https://link.springer.com/book/10.1007/978-3-031-13909-3摘要：By the time of writing, quantum audio still is a very young area of study, even within the quantum signal processing community. This chapter introduces the state of the art in quantum audio and discusses methods for the quantum representation of audio signals. Currently, no quantum representation strategy claims to be the best one for audio applications. Each one presents advantages and disadvantages. It can be argued that future quantum audio representation schemes will make use of multiple strategies aimed at specific applications. NOTE: This is an unedited abridged version of the pre-submission draft of a chapter, with the same title, published in the book Quantum Computer Music: Foundations, Methods and Advanced Concepts, by E. R. Miranda (pp. 223 - 274). Please refer to the version in this book for application examples and a discussion on sound synthesis methods based on quantum audio representation and their potential for developing new types of musical instruments. https://link.springer.com/book/10.1007/978-3-031-13909-3
【5】 Modeling the Rhythm from Lyrics for Melody Generation of Pop Song
标题：流行歌曲旋律生成中的歌词节奏建模
链接：https://arxiv.org/abs/2301.01361
作者：Daiyu Zhang,Ju-Chiang Wang,Katerina Kosta,Jordan B. L. Smith,Shicen Zhou备注：Published in ISMIR 2022摘要：根据预先写好的歌词创作流行歌曲旋律是作曲家的典型实践。歌词如何被设置为旋律的计算模型对于自动作曲系统是重要的，但是端到端歌词到旋律模型将需要大量的成对训练数据。为了减少数据的限制，我们采用了两阶段的方法，将任务分为歌词到节奏和节奏到旋律模块。然而，歌词到节奏的任务仍然是具有挑战性的，由于其多模态。在这篇文章中，我们提出了一个新的歌词到节奏框架，其中包括词性标签以实现更好的文本设置，以及一个Transformer架构，旨在建模长期的音节到音符的关联。对于节奏到旋律的任务，我们采用了一个经过验证的和弦条件旋律Transformer，它已经取得了最先进的结果。中文歌词旋律生成实验表明，该框架能够对数据集中的节奏和音高分布等关键特征进行建模，在主观评价中，该系统生成的旋律与现有技术生成的旋律相似或更好.摘要：Creating a pop song melody according to pre-written lyrics is a typical practice for composers. A computational model of how lyrics are set as melodies is important for automatic composition systems, but an end-to-end lyric-to-melody model would require enormous amounts of paired training data. To mitigate the data constraints, we adopt a two-stage approach, dividing the task into lyric-to-rhythm and rhythm-to-melody modules. However, the lyric-to-rhythm task is still challenging due to its multimodality. In this paper, we propose a novel lyric-to-rhythm framework that includes part-of-speech tags to achieve better text setting, and a Transformer architecture designed to model long-term syllable-to-note associations. For the rhythm-to-melody task, we adapt a proven chord-conditioned melody Transformer, which has achieved state-of-the-art results. Experiments for Chinese lyric-to-melody generation show that the proposed framework is able to model key characteristics of rhythm and pitch distributions in the dataset, and in a subjective evaluation, the melodies generated by our system were rated as similar to or better than those of a state-of-the-art alternative.【1】 Grid-Based Decimation for Wavelet Transforms with Stably Invertible Implementation
标题：基于网格的小波抽取及其稳定可逆实现
链接：https://arxiv.org/abs/2301.01640
作者：Nicki Holighaus,Günther Koliander,Clara Hollomey,Friedrich Pillichshammer机构：Pillichshammer is with the Institute of Financial Mathematics andApplied Number Theory, Johannes Kepler University Linz摘要：小波变换的恒定中心频率与带宽比（Q因子）为音频数据提供了非常自然的表示。然而，可逆小波变换要么需要非均匀抽取--导致难以处理的不规则数据结构--要么需要具有不可接受的计算开销的过高过采样。在这里，我们提出了一种新的抽取策略，小波变换，导致稳定的表示与过采样率接近1和均匀抽取。特别地，我们证明了所得表示的有限实现在框架论意义下是能量守恒的。所获得的小波系数可以存储在时频矩阵中，其中列作为时间帧并且行作为频率信道的自然解释。这种矩阵结构立即允许访问大量的算法，这些算法成功地用于时间-频率音频处理，但是先前不能与小波变换结合使用。我们展示了我们的方法在基于非负矩阵分解的处理中、在开始检测中和在无相位重建中的应用。摘要：The constant center frequency to bandwidth ratio (Q-factor) of wavelet transforms provides a very natural representation for audio data. However, invertible wavelet transforms have either required non-uniform decimation -- leading to irregular data structures that are cumbersome to work with -- or require excessively high oversampling with unacceptable computational overhead. Here, we present a novel decimation strategy for wavelet transforms that leads to stable representations with oversampling rates close to one and uniform decimation. Specifically, we show that finite implementations of the resulting representation are energy-preserving in the sense of frame theory. The obtained wavelet coefficients can be stored in a timefrequency matrix with a natural interpretation of columns as time frames and rows as frequency channels. This matrix structure immediately grants access to a large number of algorithms that are successfully used in time-frequency audio processing, but could not previously be used jointly with wavelet transforms. We demonstrate the application of our method in processing based on nonnegative matrix factorization, in onset detection, and in phaseless reconstruction.
【2】 Modeling the Rhythm from Lyrics for Melody Generation of Pop Song
标题：流行歌曲旋律生成中的歌词节奏建模
链接：https://arxiv.org/abs/2301.01361
*与cs.SD语音【5】为同一篇
作者：Daiyu Zhang,Ju-Chiang Wang,Katerina Kosta,Jordan B. L. Smith,Shicen Zhou备注：Published in ISMIR 2022摘要：根据预先写好的歌词创作流行歌曲旋律是作曲家的典型实践。歌词如何被设置为旋律的计算模型对于自动作曲系统是重要的，但是端到端歌词到旋律模型将需要大量的成对训练数据。为了减少数据的限制，我们采用了两阶段的方法，将任务分为歌词到节奏和节奏到旋律模块。然而，歌词到节奏的任务仍然是具有挑战性的，由于其多模态。在这篇文章中，我们提出了一个新的歌词到节奏框架，其中包括词性标签以实现更好的文本设置，以及一个Transformer架构，旨在建模长期的音节到音符的关联。对于节奏到旋律的任务，我们采用了一个经过验证的和弦条件旋律Transformer，它已经取得了最先进的结果。中文歌词旋律生成实验表明，该框架能够对数据集中的节奏和音高分布等关键特征进行建模，在主观评价中，该系统生成的旋律与现有技术生成的旋律相似或更好.摘要：Creating a pop song melody according to pre-written lyrics is a typical practice for composers. A computational model of how lyrics are set as melodies is important for automatic composition systems, but an end-to-end lyric-to-melody model would require enormous amounts of paired training data. To mitigate the data constraints, we adopt a two-stage approach, dividing the task into lyric-to-rhythm and rhythm-to-melody modules. However, the lyric-to-rhythm task is still challenging due to its multimodality. In this paper, we propose a novel lyric-to-rhythm framework that includes part-of-speech tags to achieve better text setting, and a Transformer architecture designed to model long-term syllable-to-note associations. For the rhythm-to-melody task, we adapt a proven chord-conditioned melody Transformer, which has achieved state-of-the-art results. Experiments for Chinese lyric-to-melody generation show that the proposed framework is able to model key characteristics of rhythm and pitch distributions in the dataset, and in a subjective evaluation, the melodies generated by our system were rated as similar to or better than those of a state-of-the-art alternative.
【3】 Quantum Representations of Sound: from mechanical waves to quantum circuits
标题：声音的量子表示：从机械波到量子电路
链接：https://arxiv.org/abs/2301.01595
*与cs.SD语音【4】为同一篇
作者：Paulo V. Itaborai,Eduardo R. Miranda机构：Interdisciplinary Centre for Computer Music Research (ICCMR), University of Plymouth, Plymouth, UK备注：29 pages,26 figures. Accompanying Python package is available: this https URL摘要：在写这篇文章的时候，量子音频仍然是一个非常年轻的研究领域，即使在量子信号处理社区也是如此。本章介绍了量子音频的发展现状，并讨论了音频信号的量子表示方法。目前，还没有量子表示策略声称是音频应用的最佳策略。每一种方法都有其优点和缺点。可以认为，未来的量子音频表示方案将利用针对特定应用的多种策略。注：这是一个未经编辑的节略版本的预提交草案的一章，具有相同的标题，出版在书量子计算机音乐：基础、方法和高级概念》，作者E.红米兰达（第223 - 274页）。请参阅本书中的版本，了解应用示例和基于量子音频表示的声音合成方法及其开发新型乐器的潜力的讨论。https://link.springer.com/book/10.1007/978-3-031-13909-3摘要：By the time of writing, quantum audio still is a very young area of study, even within the quantum signal processing community. This chapter introduces the state of the art in quantum audio and discusses methods for the quantum representation of audio signals. Currently, no quantum representation strategy claims to be the best one for audio applications. Each one presents advantages and disadvantages. It can be argued that future quantum audio representation schemes will make use of multiple strategies aimed at specific applications. NOTE: This is an unedited abridged version of the pre-submission draft of a chapter, with the same title, published in the book Quantum Computer Music: Foundations, Methods and Advanced Concepts, by E. R. Miranda (pp. 223 - 274). Please refer to the version in this book for application examples and a discussion on sound synthesis methods based on quantum audio representation and their potential for developing new types of musical instruments. https://link.springer.com/book/10.1007/978-3-031-13909-3
【4】 Validity in Music Information Research Experiments
标题：音乐信息研究实验中的效度
链接：https://arxiv.org/abs/2301.01578
*与cs.SD语音【1】为同一篇
作者：Bob L. T. Sturm,Arthur Flexer机构：Division of Speech, Music and Hearing, KTH Stockholm, Sweden, Institute of Computational Perception, Johannes Kepler摘要：有效性是根据证据（如实验中收集的数据）做出的推论的真实性，是科学工作的核心。考虑到音乐信息研究领域的成熟，我们认为有效性应该比迄今为止得到更多的讨论和考虑。在工作中考虑有效性可以提高其科学价值和工程价值。令人困惑的MIR现象，如对抗性攻击和性能玻璃天花板，通过有效性的镜头变得不那么神秘。在这篇文章中，我们回顾了有效性的主题一般，考虑四个主要类型的有效性从一个关键的参考：Shadish等人，2002年。我们以一个典型的MIR实验为基础来讨论这些类型：使用机器学习的音乐分类。通过这种方法，可以指导MIR实验者从他们的实验中收集的数据中做出有效的推断。摘要：Validity is the truth of an inference made from evidence, such as data collected in an experiment, and is central to working scientifically. Given the maturity of the domain of music information research (MIR), validity in our opinion should be discussed and considered much more than it has been so far. Considering validity in one's work can improve its scientific and engineering value. Puzzling MIR phenomena like adversarial attacks and performance glass ceilings become less mysterious through the lens of validity. In this article, we review the subject of validity in general, considering the four major types of validity from a key reference: Shadish et al. 2002. We ground our discussion of these types with a prototypical MIR experiment: music classification using machine learning. Through this MIR experimentalists can be guided to make valid inferences from data collected from their experiments.
【5】 Audio-Visual Efficient Conformer for Robust Speech Recognition
标题：一种用于稳健语音识别的高效音视同步器
链接：https://arxiv.org/abs/2301.01456
*与cs.SD语音【2】为同一篇
作者：Maxime Burchi,Radu Timofte机构：Computer Vision Lab, CAIDAS, IFI, University of W¨urzburg, Germany摘要：近年来，基于神经网络的端到端自动语音识别（ASR）系统得到了很大的发展。大规模人工标注数据集的可用性和充足的计算资源使得训练强大的深度神经网络成为可能，在学术基准上达到非常低的字错误率（WER）。然而，尽管在干净的音频样本上有令人印象深刻的性能，但在嘈杂的语音上经常观察到性能下降。在这项工作中，我们建议通过处理音频和视频模态来提高最近提出的基于高效一致性连接主义时间分类（CTC）的架构的噪声鲁棒性。我们在ResNet-18视觉前端的基础上使用高效Conformer后端，并在区块之间添加中间CTC损失，从而改进了之前的唇读方法。我们使用CTC间残差模块将中间块特征条件化在早期预测上，以放松基于CTC的模型的条件独立性假设。我们还用一种更有效、更简单的注意机制（我们称之为补丁注意）来取代高效构象器分组注意。我们使用公开的唇读句子2（LRS2）和唇读句子3（LRS3）数据集进行实验。我们的实验表明，使用音频和视觉模态允许在存在环境噪声的情况下更好地识别语音，并显著加速训练，以少4倍的训练步骤达到较低的WER。我们的视听高效一致性（AVEC）模型实现了最先进的性能，在LRS2和LRS3测试集上的WER分别为2.3%和1.8%。Code and pretrained models are available at https://github.com/burchim/AVEC.摘要：End-to-end Automatic Speech Recognition (ASR) systems based on neural networks have seen large improvements in recent years. The availability of large scale hand-labeled datasets and sufficient computing resources made it possible to train powerful deep neural networks, reaching very low Word Error Rate (WER) on academic benchmarks. However, despite impressive performance on clean audio samples, a drop of performance is often observed on noisy speech. In this work, we propose to improve the noise robustness of the recently proposed Efficient Conformer Connectionist Temporal Classification (CTC)-based architecture by processing both audio and visual modalities. We improve previous lip reading methods using an Efficient Conformer back-end on top of a ResNet-18 visual front-end and by adding intermediate CTC losses between blocks. We condition intermediate block features on early predictions using Inter CTC residual modules to relax the conditional independence assumption of CTC-based models. We also replace the Efficient Conformer grouped attention by a more efficient and simpler attention mechanism that we call patch attention. We experiment with publicly available Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3) datasets. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps. Our Audio-Visual Efficient Conformer (AVEC) model achieves state-of-the-art performance, reaching WER of 2.3% and 1.8% on LRS2 and LRS3 test sets. Code and pretrained models are available at https://github.com/burchim/AVEC.
【6】 An ensemble-based framework for mispronunciation detection of Arabic phonemes
标题：一种基于集成的阿拉伯音素发音错误检测框架
链接：https://arxiv.org/abs/2301.01378
*与cs.SD语音【3】为同一篇
作者：Sukru Selim Calik,Ayhan Kucukmanisa,Zeynep Hilal Kilimci机构：Maviay Consultancy Company, Kocaeli University Technopark, Kocaeli, Turkey, Department of Electronics and Communication Engineering, Kocaeli University, Kocaeli, Turkey, Department of Information Systems Engineering, Kocaeli University, Kocaeli, Turkey摘要：通过计算机辅助语言学习（CALL）系统来确定错误发音并确保向用户提供反馈。在这项工作中，我们介绍了一个集合模型，定义阿拉伯语音素的错误发音，并帮助学习阿拉伯语，有效。据我们所知，这是第一次尝试确定阿拉伯语音素的错误发音采用集成学习技术和传统的机器学习模型，全面。为了观察特征提取技术的效果，将梅尔倒谱系数（MFCC）和梅尔谱图与每种学习算法混合。为了显示所提出的模型的成功，阿拉伯语音素中的29个字母，其中8个是hafiz，由总共11个不同的人发声。采用加噪、时移、时间拉伸、基音偏移等方法提高了数据集的容量。实验结果表明，投票分类器与Mel谱图特征提取技术的集成算法具有显著的分类效果，分类准确率达95.9%。摘要：Determination of mispronunciations and ensuring feedback to users are maintained by computer-assisted language learning (CALL) systems. In this work, we introduce an ensemble model that defines the mispronunciation of Arabic phonemes and assists learning of Arabic, effectively. To the best of our knowledge, this is the very first attempt to determine the mispronunciations of Arabic phonemes employing ensemble learning techniques and conventional machine learning models, comprehensively. In order to observe the effect of feature extraction techniques, mel-frequency cepstrum coefficients (MFCC), and Mel spectrogram are blended with each learning algorithm. To show the success of proposed model, 29 letters in the Arabic phonemes, 8 of which are hafiz, are voiced by a total of 11 different person. The amount of data set has been enhanced employing the methods of adding noise, time shifting, time stretching, pitch shifting. Extensive experiment results demonstrate that the utilization of voting classifier as an ensemble algorithm with Mel spectrogram feature extraction technique exhibits remarkable classification result with 95.9% of accuracy.机器翻译，仅供参考
每日语音速递