【1】CCOM-HuQin: an Annotated Multimodal Chinese Fiddle Performance Dataset
标题：CCOM-胡琴：一个带注释的多模式中文小提琴演奏数据集
链接：http://arxiv.org/pdf/2209.06496v1
作者：Yu Zhang,Ziya Zhou,Xiaobing Li,Feng Yu,Maosong Sun摘要：胡琴是中国传统弓弦乐器的一个家族。不同的演奏风格所体现的演奏技巧，为胡琴演奏增添了丰富的情感色彩和美感。复杂的应用技术使胡琴音乐成为一个具有挑战性的来源，为基本的MIR任务，如音高分析，转录和乐谱音频对齐。本文介绍了一个胡琴音乐多模态演奏数据集，该数据集包含11，992个单PT片段和57个经典摘录注释音乐片段的视听记录。本文以音乐学理论为基础，结合实际应用案例，对胡琴PT分类进行了系统的描述。然后介绍数据集创建方法，并重点介绍以PT为特征的标注原则。本文从不同角度分析了胡琴子类别中演奏的PT的多样性，并进行了初步的实验，以展示该数据集在各种MIR任务和跨文化音乐研究中的潜在应用。最后，提出了该数据集有待进一步扩展的工作.摘要：HuQin is a family of traditional Chinese bowed string instruments. Playing techniques(PTs) embodied in various playing styles add abundant emotional coloring and aesthetic feelings to HuQin performance. The complex applied techniques make HuQin music a challenging source for fundamental MIR tasks such as pitch analysis, transcription and score-audio alignment. In this paper, we present a multimodal performance dataset of HuQin music that contains audio-visual recordings of 11,992 single PT clips and 57 annotated musical pieces of classical excerpts. We systematically describe the HuQin PT taxonomy based on musicological theory and practical use cases. Then we introduce the dataset creation methodology and highlight the annotation principles featuring PTs. We analyze the statistics in different aspects to demonstrate the variety of PTs played in HuQin subcategories and perform preliminary experiments to show the potential applications of the dataset in various MIR tasks and cross-cultural music studies. Finally, we propose future work to be extended on the dataset.
【2】ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTS
标题：ParaTTS：在基于段落的TTS中学习语言和韵律跨句信息
链接：http://arxiv.org/pdf/2209.06484v1
作者：Liumeng Xue,Frank K. Soong,Shaofei Zhang,Lei Xie机构：School of Computer Science, NorthwesternPolytechnical University备注：Published in IEEEACM Transactions on Audio, Speech, and Language Processing摘要：神经端到端TTS模型的最新进展已经显示了在传统的基于句子的TTS中的高质量、自然的合成语音。然而，在建立基于段落的TTS模型时，需要考虑大量的上下文信息，因此在考虑整个段落的情况下，要再现同样高的质量仍然是一个挑战。为了减轻训练的难度，我们提出在训练中考虑跨句、嵌入式结构等语言和韵律信息的建模。利用改进的Tacotron2. 0训练了语言感知、韵律感知和句位网络3个子模块。具体来说，我们利用语言感知和韵律感知网络来学习段落中的信息以及相应的组成句子之间的关系。通过编码器捕获段落中的信息，并利用多头部注意机制学习段落中的句间信息。句子在段落中的相对位置被句子位置网络明确地利用。在一个4.08小时的讲故事的有声读物语料库上，由一个女性普通话说话人录制的语音，证明了所提出的TTS模型可以产生相当自然和高质量的分段语音。与基于句子的模型相比，该模型可以更好地预测和呈现跨句子的上下文信息，如连续句子之间的中断和韵律变化。在长度与训练数据的典型段落长度相似、长于或远长于训练数据的段落文本上进行测试，新模型产生的TTS语音在主观测试中始终优于基于句子的模型，并在客观测量中得到证实。摘要：Recent advancements in neural end-to-end TTS models have shown high-quality, natural synthesized speech in a conventional sentence-based TTS. However, it is still challenging to reproduce similar high quality when a whole paragraph is considered in TTS, where a large amount of contextual information needs to be considered in building a paragraph-based TTS model. To alleviate the difficulty in training, we propose to model linguistic and prosodic information by considering cross-sentence, embedded structure in training. Three sub-modules, including linguistics-aware, prosody-aware and sentence-position networks, are trained together with a modified Tacotron2. Specifically, to learn the information embedded in a paragraph and the relations among the corresponding component sentences, we utilize linguistics-aware and prosody-aware networks. The information in a paragraph is captured by encoders and the inter-sentence information in a paragraph is learned with multi-head attention mechanisms. The relative sentence position in a paragraph is explicitly exploited by a sentence-position network. Trained on a storytelling audio-book corpus (4.08 hours), recorded by a female Mandarin Chinese speaker, the proposed TTS model demonstrates that it can produce rather natural and good-quality speech paragraph-wise. The cross-sentence contextual information, such as break and prosodic variations between consecutive sentences, can be better predicted and rendered than the sentence-based model. Tested on paragraph texts, of which the lengths are similar to, longer than, or much longer than the typical paragraph length of the training data, the TTS speech produced by the new model is consistently preferred over the sentence-based model in subjective tests and confirmed in objective measures.
【3】ConvNext Based Neural Network for Anti-Spoofing
标题：基于ConvNext的神经网络反欺骗
链接：http://arxiv.org/pdf/2209.06434v1
作者：Qiaowei Ma,Jinghui Zhong,Yitao Yang,Weiheng Liu,Ying Gao,Wing W. Y. Ng机构：Y. Ng, School of Computer Science and Engineering, South China University of Technology, Guangzhou, China摘要：自动说话人确认（ASV）在现实生活中广泛应用于身份认证。然而，随着语音转换、语音合成算法的快速发展和录音设备质量的提高，ASV系统越来越容易受到欺骗攻击。近年来，在合成与重放语音检测方面有很多研究工作，研究人员提出了一些基于人工特征的反欺骗方法，以提高合成与重放语音检测系统的准确性和鲁棒性。然而，使用手工制作的特征而不是原始波形会丢失用于反欺骗的某些信息，这将降低系统的检测性能。受ConvNext在图像分类任务中良好性能的启发，对ConvNext网络结构进行了相应的扩展，并将其应用于欺骗攻击检测任务，提出了一种端到端的反欺骗模型。通过将扩展结构与信道注意块相结合，该模型可以集中于语音表示中信息量最大的子带，从而提高了抗欺骗性能。实验结果表明，在ASVSpoof 2019 LA和PA评测数据集上，本文提出的最佳单系统分别获得了1.88%和2.79%的相同错误率，证明了该模型的抗欺骗能力。摘要：Automatic speaker verification (ASV) has been widely used in the real life for identity authentication. However, with the rapid development of speech conversion, speech synthesis algorithms and the improvement of the quality of recording devices, ASV systems are vulnerable for spoof attacks. In recent years, there have many works about synthetic and replay speech detection, researchers had proposed a number of anti-spoofing methods based on hand-crafted features to improve the accuracy and robustness of synthetic and replay speech detection system. However, using hand-crafted features rather than raw waveform would lose certain information for anti-spoofing, which will reduce the detection performance of the system. Inspired by the promising performance of ConvNext in image classification tasks, we extend the ConvNext network architecture accordingly for spoof attacks detection task and propose an end-to-end anti-spoofing model. By integrating the extended architecture with the channel attention block, the proposed model can focus on the most informative sub-bands of speech representations to improve the anti-spoofing performance. Experiments show that our proposed best single system could achieve an equal error rate of 1.88% and 2.79% for the ASVSpoof 2019 LA evaluation dataset and PA evaluation dataset respectively, which demonstrate the model's capacity for anti-spoofing.
【4】I2CR: Improving Noise Robustness on Keyword Spotting Using Inter-Intra Contrastive Regularization
标题：I2CR：利用帧内对比正则化提高关键词检测的噪声稳健性
链接：http://arxiv.org/pdf/2209.06360v1
作者：Dianwen Ng,Jia Qi Yip,Tanmay Surana,Zhao Yang,Chong Zhang,Yukun Ma,Chongjia Ni,Eng Siong Chng,Bin Ma机构：Alibaba Group, Singapore, School of Computer Science and Engineering, Nanyang Technological University, Singapore, Xi’an Jiaotong University, China摘要：关键词识别中的噪声鲁棒性问题一直是一个难题，因为许多模型无法克服噪声的严重影响，导致特征嵌入质量下降。提出了一种称为I2CR（Inter-Intra Contrasive Regularization）的对比正则化方法，通过引导模型学习特定于聚类的基本语音信息来改进特征表示。这涉及最大化相同类的帧内和帧间样本之间的相似性。结果，它将实例拉得更接近于形成更突出的聚类的更一般化的表示，并且减少了噪声的不利影响。我们证明了我们的方法在不同的噪声环境下对不同的骨干模型架构提供了一致的准确性改进。我们还证明了我们提出的框架提高了未知域外噪声和未知变异噪声信噪比的准确性。这表明了我们在噪声鲁棒性方面的整体改进工作的重要性。摘要：Noise robustness in keyword spotting remains a challenge as many models fail to overcome the heavy influence of noises, causing the deterioration of the quality of feature embeddings. We proposed a contrastive regularization method called Inter-Intra Contrastive Regularization (I2CR) to improve the feature representations by guiding the model to learn the fundamental speech information specific to the cluster. This involves maximizing the similarity across Intra and Inter samples of the same class. As a result, it pulls the instances closer to more generalized representations that form more prominent clusters and reduces the adverse impact of noises. We show that our method provides consistent improvements in accuracy over different backbone model architectures under different noise environments. We also demonstrate that our proposed framework has improved the accuracy of unseen out-of-domain noises and unseen variant noise SNRs. This indicates the significance of our work with the overall refinement in noise robustness.
【5】Using Rater and System Metadata to Explain Variance in the VoiceMOS Challenge 2022 Dataset
标题：使用评分员和系统元数据解释VoiceMOS挑战2022数据集中的差异
链接：http://arxiv.org/pdf/2209.06358v1
作者：Michael Chinen,Jan Skoglund,Chandan K A Reddy,Alessandro Ragano,Andrew Hines机构：Google, University College Dublin, School of Computer Science, Dublin, Ireland备注：Preprint; accepted for Interspeech 2022摘要：非参考语音质量模型对于越来越多的应用是重要的。VoiceMOS 2022挑战赛提供了一个带有主观标签的合成语音转换和文本到语音样本的数据集。本研究着眼于从元数据和数据集的分布不平衡的语音质量的主观评级中可以解释的方差量。使用wav2vec 2.0构建语音质量模型，该模型具有额外的元数据特征，包括评分者组和系统标识符，并获得竞争性度量，包括在系统级的0.934的Spearman秩相关系数（SRCC）和0.088的MSE，以及在话语级的0.877和0.198。使用测试限制或设盲的数据和元数据进一步改进了指标。元数据分析表明，由于验证和测试数据集上每个系统使用的话语数量差异很大，系统级度量并不代表模型的系统级预测。我们的结论是，一般情况下，测试集中应该有足够的话语来限制样本平均误差，并且系统之间的话语计数应该相对平衡，否则话语级度量可能更可靠和可解释。摘要：Non-reference speech quality models are important for a growing number of applications. The VoiceMOS 2022 challenge provided a dataset of synthetic voice conversion and text-to-speech samples with subjective labels. This study looks at the amount of variance that can be explained in subjective ratings of speech quality from metadata and the distribution imbalances of the dataset. Speech quality models were constructed using wav2vec 2.0 with additional metadata features that included rater groups and system identifiers and obtained competitive metrics including a Spearman rank correlation coefficient (SRCC) of 0.934 and MSE of 0.088 at the system-level, and 0.877 and 0.198 at the utterance-level. Using data and metadata that the test restricted or blinded further improved the metrics. A metadata analysis showed that the system-level metrics do not represent the model's system-level prediction as a result of the wide variation in the number of utterances used for each system on the validation and test datasets. We conclude that, in general, conditions should have enough utterances in the test set to bound the sample mean error, and be relatively balanced in utterance count between systems, otherwise the utterance-level metrics may be more reliable and interpretable.
【6】A Universally-Deployable ASR Frontend for Joint Acoustic Echo Cancellation, Speech Enhancement, and Voice Separation
标题：一种通用的声学回波抵消、语音增强和语音分离联合ASR前端
链接：http://arxiv.org/pdf/2209.06410v1
作者：Tom O'Malley,Arun Narayanan,Quan Wang摘要：最近的工作已经表明，可以训练单个模型来执行联合声学回声消除（AEC）、语音增强和话音分离，从而用作鲁棒自动语音识别（ASR）的统一前端。联合模型使用上下文信息，诸如回放音频的参考、噪声上下文和说话者嵌入。在这项工作中，我们提出了一些新的改进这样一个模型。首先，我们改进了用于将噪声上下文摄取到模型中的交叉注意力一致器的体系结构。第二，我们将该模型推广为能够处理不同长度的噪声上下文。第三，我们提出了一种新的策略--Signal Dropout，它可以对缺失的上下文信息进行建模。在缺少一个或多个信号的情况下，所提出的模型与没有这些信号训练的任务特定模型表现得几乎一样好;并且当这样的信号存在时，我们的系统与需要所有上下文信号的系统相比很好。在基线上，当不存在说话者嵌入时，最终模型在背景语音上保持25.0%的相对单词错误率减少，并且当不存在设备回放时，在AEC上保持61.2%的相对单词错误率减少。摘要：Recent work has shown that it is possible to train a single model to perform joint acoustic echo cancellation (AEC), speech enhancement, and voice separation, thereby serving as a unified frontend for robust automatic speech recognition (ASR). The joint model uses contextual information, such as a reference of the playback audio, noise context, and speaker embedding. In this work, we propose a number of novel improvements to such a model. First, we improve the architecture of the Cross-Attention Conformer that is used to ingest noise context into the model. Second, we generalize the model to be able to handle varying lengths of noise context. Third, we propose Signal Dropout, a novel strategy that models missing contextual information. In the absence of one or more signals, the proposed model performs nearly as well as task-specific models trained without these signals; and when such signals are present, our system compares well against systems that require all context signals. Over the baseline, the final model retains a relative word error rate reduction of 25.0% on background speech when speaker embedding is absent, and 61.2% on AEC when device playback is absent.
【7】Deep Speech Synthesis from Articulatory Representations
标题：基于节律表示的深度语音合成
链接：http://arxiv.org/pdf/2209.06337v1
作者：Peter Wu,Shinji Watanabe,Louis Goldstein,Alan W Black,Gopala K. Anumanchipalli机构：University of California, Berkeley, United States, Carnegie Mellon University, United States, University of Southern California, United States摘要：在发音合成任务中，从包含关于人类声道的物理行为的信息的输入特征合成语音。由于发音空间紧凑、平滑、可解释，该任务为语音合成研究提供了一个很有前景的方向。当前的研究强调了深度学习模型执行发音合成的潜力。但是，这些模型是否能达到人类语音产生系统的效率和保真度还不清楚。为了帮助弥补这一差距，我们提出了一种时域发音合成方法，并通过电磁发音描记术（EMA）和合成发音特征输入证明了其有效性。我们的模型计算效率高，对于EMA-to-speech任务实现了18.5%的转录单词错误率（WER），与现有工作相比产生了11.6%的改进。通过插值实验，我们也突出了我们方法的可推广性和可解释性。摘要：In the articulatory synthesis task, speech is synthesized from input features containing information about the physical behavior of the human vocal tract. This task provides a promising direction for speech synthesis research, as the articulatory space is compact, smooth, and interpretable. Current works have highlighted the potential for deep learning models to perform articulatory synthesis. However, it remains unclear whether these models can achieve the efficiency and fidelity of the human speech production system. To help bridge this gap, we propose a time-domain articulatory synthesis methodology and demonstrate its efficacy with both electromagnetic articulography (EMA) and synthetic articulatory feature inputs. Our model is computationally efficient and achieves a transcription word error rate (WER) of 18.5% for the EMA-to-speech task, yielding an improvement of 11.6% compared to prior work. Through interpolation experiments, we also highlight the generalizability and interpretability of our approach.
【8】Automated detection of pronunciation errors in non-native English speech employing deep learning
标题：基于深度学习的非母语英语语音发音错误自动检测
链接：http://arxiv.org/pdf/2209.06265v1
机构：Scientific discipline: Technical Informatics and Telecommunications, DOCTORAL DISSERTATION, Title of doctoral dissertation: Automated detection of pronunciation errors in non-native English, speech employing deep learning摘要：尽管近年来取得了显著进步，但现有的计算机辅助发音训练（CAPT）方法以相对较低的准确度（在40%-80%的召回率下，准确度为60%）检测发音错误。这个博士工作提出了用于检测非母语（L2）英语语音中的发音错误的新颖的深度学习方法，在AUC度量（曲线下面积）方面比现有技术的方法好41%，即，从0.528变化到0.749。现有CAPT方法的问题之一是发音错误检测模型的可靠训练所需的注释的发错音的语音的低可用性。因此，发音错误的检测被重新表述为生成合成的发错音的语音的任务。直观地说，如果我们能够模仿发音错误的语音并产生任意数量的训练数据，检测发音错误将更加有效。此外，为了消除对齐标准音素和识别音素的需要，提出了一种新颖的端到端多任务技术来直接检测发音错误。发音错误检测模型已经在Amazon中用于自动检测合成语音中的发音错误，以加速对新的语音合成方法的研究。实验结果表明，提出的深度学习方法适用于构音障碍语音的检测和重构。摘要：Despite significant advances in recent years, the existing Computer-Assisted Pronunciation Training (CAPT) methods detect pronunciation errors with a relatively low accuracy (precision of 60% at 40%-80% recall). This Ph.D. work proposes novel deep learning methods for detecting pronunciation errors in non-native (L2) English speech, outperforming the state-of-the-art method in AUC metric (Area under the Curve) by 41%, i.e., from 0.528 to 0.749. One of the problems with existing CAPT methods is the low availability of annotated mispronounced speech needed for reliable training of pronunciation error detection models. Therefore, the detection of pronunciation errors is reformulated to the task of generating synthetic mispronounced speech. Intuitively, if we could mimic mispronounced speech and produce any amount of training data, detecting pronunciation errors would be more effective. Furthermore, to eliminate the need to align canonical and recognized phonemes, a novel end-to-end multi-task technique to directly detect pronunciation errors was proposed. The pronunciation error detection models have been used at Amazon to automatically detect pronunciation errors in synthetic speech to accelerate the research into new speech synthesis methods. It was demonstrated that the proposed deep learning methods are applicable in the tasks of detecting and reconstructing dysarthric speech.
【1】 Decoupled Pronunciation and Prosody Modeling in Meta-Learning-Based Multilingual Speech Synthesis
标题：基于元学习的多语种语音合成中的语音韵律解耦建模
链接：http://arxiv.org/pdf/2209.06789v1
作者：Yukun Peng,Zhenhua Ling机构：National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China备注：Submitted to Interspeech 2022摘要：为了提高基于元学习的多语言语音合成的性能，提出了一种发音和韵律解耦建模方法。基线元学习合成方法采用单个文本编码器和单个解码器来预测所有语言的Mel谱图，该文本编码器具有以语言嵌入为条件的参数生成器。相比之下，考虑到语音知识和韵律知识在不同语言之间应该以不同的方式共享，我们提出的方法设计了一个双流模型结构，包含两个编码器和两个解码器，分别用于语音和韵律建模。实验结果表明，与基本的元学习合成方法相比，该方法有效地提高了多语种语音合成的可懂度和自然度。摘要：This paper presents a method of decoupled pronunciation and prosody modeling to improve the performance of meta-learning-based multilingual speech synthesis. The baseline meta-learning synthesis method adopts a single text encoder with a parameter generator conditioned on language embeddings and a single decoder to predict mel-spectrograms for all languages. In contrast, our proposed method designs a two-stream model structure that contains two encoders and two decoders for pronunciation and prosody modeling, respectively, considering that the pronunciation knowledge and the prosody knowledge should be shared in different ways among languages. In our experiments, our proposed method effectively improved the intelligibility and naturalness of multilingual speech synthesis comparing with the baseline meta-learning synthesis method.
【2】 Applying wav2vec2 for Speech Recognition on Bengali Common Voices Dataset
标题：Wave2ve2在孟加拉语音库上的语音识别应用
链接：http://arxiv.org/pdf/2209.06581v1
作者：H. A. Z. Sameen Shahgir,Khondker Salman Sayeed,Tanjeem Azwad Zaman机构：Undergraduate, Computer Science and Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh摘要：语音本身是连续的，其中离散的词、音素和其他单元没有被清楚地分割，因此语音识别几十年来一直是一个活跃的研究问题。在这项工作中，我们对wav 2 vec 2.0进行了微调，以识别和转录孟加拉语语音--在孟加拉语普通语音语音数据集上训练它。在训练了71个时期之后，在由36919个mp3文件组成的训练集上，我们在大小为7，747的验证集上实现了0.3172的训练损失和0.2524的WER。使用5元语法语言模型，在大小为7，747的测试集上，Levenshtein距离为2.6446。然后将训练集和验证集合并、混洗并拆分成85-15的比例。在该组合数据集上再训练7个历元，在测试集上产生2.60753的改进Levenshtein距离。我们的模型是性能最好的模型，在隐藏数据集上实现了6.234的Levenshtein距离，比其他竞争提交的模型低1.1049个单位。摘要：Speech is inherently continuous, where discrete words, phonemes and other units are not clearly segmented, and so speech recognition has been an active research problem for decades. In this work we have fine-tuned wav2vec 2.0 to recognize and transcribe Bengali speech -- training it on the Bengali Common Voice Speech Dataset. After training for 71 epochs, on a training set consisting of 36919 mp3 files, we achieved a training loss of 0.3172 and WER of 0.2524 on a validation set of size 7,747. Using a 5-gram language model, the Levenshtein Distance was 2.6446 on a test set of size 7,747. Then the training set and validation set were combined, shuffled and split into 85-15 ratio. Training for 7 more epochs on this combined dataset yielded an improved Levenshtein Distance of 2.60753 on the test set. Our model was the best performing one, achieving a Levenshtein Distance of 6.234 on a hidden dataset, which was 1.1049 units lower than other competing submissions.
【3】A Universally-Deployable ASR Frontend for Joint Acoustic Echo Cancellation, Speech Enhancement, and Voice Separation
标题：一种通用的声学回波抵消、语音增强和语音分离联合ASR前端
链接：http://arxiv.org/pdf/2209.06410v1* 与cs.SD语音【6】为同一篇
作者：Tom O'Malley,Arun Narayanan,Quan Wang摘要：最近的工作已经表明，可以训练单个模型来执行联合声学回声消除（AEC）、语音增强和话音分离，从而用作鲁棒自动语音识别（ASR）的统一前端。联合模型使用上下文信息，诸如回放音频的参考、噪声上下文和说话者嵌入。在这项工作中，我们提出了一些新的改进这样一个模型。首先，我们改进了用于将噪声上下文摄取到模型中的交叉注意力一致器的体系结构。第二，我们将该模型推广为能够处理不同长度的噪声上下文。第三，我们提出了一种新的策略--Signal Dropout，它可以对缺失的上下文信息进行建模。在缺少一个或多个信号的情况下，所提出的模型与没有这些信号训练的任务特定模型表现得几乎一样好;并且当这样的信号存在时，我们的系统与需要所有上下文信号的系统相比很好。在基线上，当不存在说话者嵌入时，最终模型在背景语音上保持25.0%的相对单词错误率减少，并且当不存在设备回放时，在AEC上保持61.2%的相对单词错误率减少。摘要：Recent work has shown that it is possible to train a single model to perform joint acoustic echo cancellation (AEC), speech enhancement, and voice separation, thereby serving as a unified frontend for robust automatic speech recognition (ASR). The joint model uses contextual information, such as a reference of the playback audio, noise context, and speaker embedding. In this work, we propose a number of novel improvements to such a model. First, we improve the architecture of the Cross-Attention Conformer that is used to ingest noise context into the model. Second, we generalize the model to be able to handle varying lengths of noise context. Third, we propose Signal Dropout, a novel strategy that models missing contextual information. In the absence of one or more signals, the proposed model performs nearly as well as task-specific models trained without these signals; and when such signals are present, our system compares well against systems that require all context signals. Over the baseline, the final model retains a relative word error rate reduction of 25.0% on background speech when speaker embedding is absent, and 61.2% on AEC when device playback is absent.
【4】Deep Speech Synthesis from Articulatory Representations
标题：基于节律表示的深度语音合成
链接：http://arxiv.org/pdf/2209.06337v1* 与cs.SD语音【7】为同一篇
作者：Peter Wu,Shinji Watanabe,Louis Goldstein,Alan W Black,Gopala K. Anumanchipalli机构：University of California, Berkeley, United States, Carnegie Mellon University, United States, University of Southern California, United States摘要：在发音合成任务中，从包含关于人类声道的物理行为的信息的输入特征合成语音。由于发音空间紧凑、平滑、可解释，该任务为语音合成研究提供了一个很有前景的方向。当前的研究强调了深度学习模型执行发音合成的潜力。但是，这些模型是否能达到人类语音产生系统的效率和保真度还不清楚。为了帮助弥补这一差距，我们提出了一种时域发音合成方法，并通过电磁发音描记术（EMA）和合成发音特征输入证明了其有效性。我们的模型计算效率高，对于EMA-to-speech任务实现了18.5%的转录单词错误率（WER），与现有工作相比产生了11.6%的改进。通过插值实验，我们也突出了我们方法的可推广性和可解释性。摘要：In the articulatory synthesis task, speech is synthesized from input features containing information about the physical behavior of the human vocal tract. This task provides a promising direction for speech synthesis research, as the articulatory space is compact, smooth, and interpretable. Current works have highlighted the potential for deep learning models to perform articulatory synthesis. However, it remains unclear whether these models can achieve the efficiency and fidelity of the human speech production system. To help bridge this gap, we propose a time-domain articulatory synthesis methodology and demonstrate its efficacy with both electromagnetic articulography (EMA) and synthetic articulatory feature inputs. Our model is computationally efficient and achieves a transcription word error rate (WER) of 18.5% for the EMA-to-speech task, yielding an improvement of 11.6% compared to prior work. Through interpolation experiments, we also highlight the generalizability and interpretability of our approach.
【5】Automated detection of pronunciation errors in non-native English speech employing deep learning
标题：基于深度学习的非母语英语语音发音错误自动检测
链接：http://arxiv.org/pdf/2209.06265v1
* 与cs.SD语音【8】为同一篇
机构：Scientific discipline: Technical Informatics and Telecommunications, DOCTORAL DISSERTATION, Title of doctoral dissertation: Automated detection of pronunciation errors in non-native English, speech employing deep learning摘要：尽管近年来取得了显著进步，但现有的计算机辅助发音训练（CAPT）方法以相对较低的准确度（在40%-80%的召回率下，准确度为60%）检测发音错误。这个博士工作提出了用于检测非母语（L2）英语语音中的发音错误的新颖的深度学习方法，在AUC度量（曲线下面积）方面比现有技术的方法好41%，即，从0.528变化到0.749。现有CAPT方法的问题之一是发音错误检测模型的可靠训练所需的注释的发错音的语音的低可用性。因此，发音错误的检测被重新表述为生成合成的发错音的语音的任务。直观地说，如果我们能够模仿发音错误的语音并产生任意数量的训练数据，检测发音错误将更加有效。此外，为了消除对齐标准音素和识别音素的需要，提出了一种新颖的端到端多任务技术来直接检测发音错误。发音错误检测模型已经在Amazon中用于自动检测合成语音中的发音错误，以加速对新的语音合成方法的研究。实验结果表明，提出的深度学习方法适用于构音障碍语音的检测和重构。摘要：Despite significant advances in recent years, the existing Computer-Assisted Pronunciation Training (CAPT) methods detect pronunciation errors with a relatively low accuracy (precision of 60% at 40%-80% recall). This Ph.D. work proposes novel deep learning methods for detecting pronunciation errors in non-native (L2) English speech, outperforming the state-of-the-art method in AUC metric (Area under the Curve) by 41%, i.e., from 0.528 to 0.749. One of the problems with existing CAPT methods is the low availability of annotated mispronounced speech needed for reliable training of pronunciation error detection models. Therefore, the detection of pronunciation errors is reformulated to the task of generating synthetic mispronounced speech. Intuitively, if we could mimic mispronounced speech and produce any amount of training data, detecting pronunciation errors would be more effective. Furthermore, to eliminate the need to align canonical and recognized phonemes, a novel end-to-end multi-task technique to directly detect pronunciation errors was proposed. The pronunciation error detection models have been used at Amazon to automatically detect pronunciation errors in synthetic speech to accelerate the research into new speech synthesis methods. It was demonstrated that the proposed deep learning methods are applicable in the tasks of detecting and reconstructing dysarthric speech.
【6】 Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings
标题：形义结合：一种声词嵌入的多任务学习模型
链接：http://arxiv.org/pdf/2209.06633v1
作者：Badr M. Abdullah,Bernd Möbius,Dietrich Klakow机构：Language Science and Technology (LST), Saarland University, Germany, Saarland Informatics Campus, Germany备注：Accepted in INTERSPEECH 2022摘要：声学单词嵌入模型（AWE）学习将可变长度的口语单词片段映射到固定维度的向量表示上，使得同一单词的不同声学样本在嵌入空间中被投影到附近。除了语音技术应用外，AWE模型还可以预测人类在各种听觉词汇处理任务中的表现。当前的AWE模型基于神经网络，并且以自底向上的方法进行训练，该方法集成声音线索以在给定声音或符号监督信号的情况下建立单词表示。因此，这些模型在学习过程中不利用或捕获高级词汇知识。%并捕获有关单词形式的低级信息。本文提出了一种多任务学习模型，将自顶向下的词汇知识融入到AWE的训练过程中。我们的模型学习了声学输入和词汇表示之间的映射，词汇表示除了自底向上的基于形式的监督之外，还编码了诸如单词语义的高级信息。我们在三种语言上进行了实验，结果表明，词汇知识的加入提高了嵌入空间的可区分性，并促使模型更好地分离词汇类别。摘要：Models of acoustic word embeddings (AWEs) learn to map variable-length spoken word segments onto fixed-dimensionality vector representations such that different acoustic exemplars of the same word are projected nearby in the embedding space. In addition to their speech technology applications, AWE models have been shown to predict human performance on a variety of auditory lexical processing tasks. Current AWE models are based on neural networks and trained in a bottom-up approach that integrates acoustic cues to build up a word representation given an acoustic or symbolic supervision signal. Therefore, these models do not leverage or capture high-level lexical knowledge during the learning process. % and capture low-level information about word forms. In this paper, we propose a multi-task learning model that incorporates top-down lexical knowledge into the training procedure of AWEs. Our model learns a mapping between the acoustic input and a lexical representation that encodes high-level information such as word semantics in addition to bottom-up form-based supervision. We experiment with three languages and demonstrate that incorporating lexical knowledge improves the embedding space discriminability and encourages the model to better separate lexical categories.
【7】CCOM-HuQin: an Annotated Multimodal Chinese Fiddle Performance Dataset
标题：CCOM-胡琴：一个带注释的多模式中文小提琴演奏数据集
链接：http://arxiv.org/pdf/2209.06496v1* 与cs.SD语音【1】为同一篇
作者：Yu Zhang,Ziya Zhou,Xiaobing Li,Feng Yu,Maosong Sun摘要：胡琴是中国传统弓弦乐器的一个家族。不同的演奏风格所体现的演奏技巧，为胡琴演奏增添了丰富的情感色彩和美感。复杂的应用技术使胡琴音乐成为一个具有挑战性的来源，为基本的MIR任务，如音高分析，转录和乐谱音频对齐。本文介绍了一个胡琴音乐多模态演奏数据集，该数据集包含11，992个单PT片段和57个经典摘录注释音乐片段的视听记录。本文以音乐学理论为基础，结合实际应用案例，对胡琴PT分类进行了系统的描述。然后介绍数据集创建方法，并重点介绍以PT为特征的标注原则。本文从不同角度分析了胡琴子类别中演奏的PT的多样性，并进行了初步的实验，以展示该数据集在各种MIR任务和跨文化音乐研究中的潜在应用。最后，提出了该数据集有待进一步扩展的工作.摘要：HuQin is a family of traditional Chinese bowed string instruments. Playing techniques(PTs) embodied in various playing styles add abundant emotional coloring and aesthetic feelings to HuQin performance. The complex applied techniques make HuQin music a challenging source for fundamental MIR tasks such as pitch analysis, transcription and score-audio alignment. In this paper, we present a multimodal performance dataset of HuQin music that contains audio-visual recordings of 11,992 single PT clips and 57 annotated musical pieces of classical excerpts. We systematically describe the HuQin PT taxonomy based on musicological theory and practical use cases. Then we introduce the dataset creation methodology and highlight the annotation principles featuring PTs. We analyze the statistics in different aspects to demonstrate the variety of PTs played in HuQin subcategories and perform preliminary experiments to show the potential applications of the dataset in various MIR tasks and cross-cultural music studies. Finally, we propose future work to be extended on the dataset.
【8】ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTS
标题：ParaTTS：在基于段落的TTS中学习语言和韵律跨句信息
链接：http://arxiv.org/pdf/2209.06484v1
* 与cs.SD语音【2】为同一篇
作者：Liumeng Xue,Frank K. Soong,Shaofei Zhang,Lei Xie机构：School of Computer Science, NorthwesternPolytechnical University备注：Published in IEEEACM Transactions on Audio, Speech, and Language Processing摘要：神经端到端TTS模型的最新进展已经显示了在传统的基于句子的TTS中的高质量、自然的合成语音。然而，在建立基于段落的TTS模型时，需要考虑大量的上下文信息，因此在考虑整个段落的情况下，要再现同样高的质量仍然是一个挑战。为了减轻训练的难度，我们提出在训练中考虑跨句、嵌入式结构等语言和韵律信息的建模。利用改进的Tacotron2. 0训练了语言感知、韵律感知和句位网络3个子模块。具体来说，我们利用语言感知和韵律感知网络来学习段落中的信息以及相应的组成句子之间的关系。通过编码器捕获段落中的信息，并利用多头部注意机制学习段落中的句间信息。句子在段落中的相对位置被句子位置网络明确地利用。在一个4.08小时的讲故事的有声读物语料库上，由一个女性普通话说话人录制的语音，证明了所提出的TTS模型可以产生相当自然和高质量的分段语音。与基于句子的模型相比，该模型可以更好地预测和呈现跨句子的上下文信息，如连续句子之间的中断和韵律变化。在长度与训练数据的典型段落长度相似、长于或远长于训练数据的段落文本上进行测试，新模型产生的TTS语音在主观测试中始终优于基于句子的模型，并在客观测量中得到证实。摘要：Recent advancements in neural end-to-end TTS models have shown high-quality, natural synthesized speech in a conventional sentence-based TTS. However, it is still challenging to reproduce similar high quality when a whole paragraph is considered in TTS, where a large amount of contextual information needs to be considered in building a paragraph-based TTS model. To alleviate the difficulty in training, we propose to model linguistic and prosodic information by considering cross-sentence, embedded structure in training. Three sub-modules, including linguistics-aware, prosody-aware and sentence-position networks, are trained together with a modified Tacotron2. Specifically, to learn the information embedded in a paragraph and the relations among the corresponding component sentences, we utilize linguistics-aware and prosody-aware networks. The information in a paragraph is captured by encoders and the inter-sentence information in a paragraph is learned with multi-head attention mechanisms. The relative sentence position in a paragraph is explicitly exploited by a sentence-position network. Trained on a storytelling audio-book corpus (4.08 hours), recorded by a female Mandarin Chinese speaker, the proposed TTS model demonstrates that it can produce rather natural and good-quality speech paragraph-wise. The cross-sentence contextual information, such as break and prosodic variations between consecutive sentences, can be better predicted and rendered than the sentence-based model. Tested on paragraph texts, of which the lengths are similar to, longer than, or much longer than the typical paragraph length of the training data, the TTS speech produced by the new model is consistently preferred over the sentence-based model in subjective tests and confirmed in objective measures.
【9】ConvNext Based Neural Network for Anti-Spoofing
标题：基于ConvNext的神经网络反欺骗
链接：http://arxiv.org/pdf/2209.06434v1* 与cs.SD语音【3】为同一篇
作者：Qiaowei Ma,Jinghui Zhong,Yitao Yang,Weiheng Liu,Ying Gao,Wing W. Y. Ng机构：Y. Ng, School of Computer Science and Engineering, South China University of Technology, Guangzhou, China摘要：自动说话人确认（ASV）在现实生活中广泛应用于身份认证。然而，随着语音转换、语音合成算法的快速发展和录音设备质量的提高，ASV系统越来越容易受到欺骗攻击。近年来，在合成与重放语音检测方面有很多研究工作，研究人员提出了一些基于人工特征的反欺骗方法，以提高合成与重放语音检测系统的准确性和鲁棒性。然而，使用手工制作的特征而不是原始波形会丢失用于反欺骗的某些信息，这将降低系统的检测性能。受ConvNext在图像分类任务中良好性能的启发，对ConvNext网络结构进行了相应的扩展，并将其应用于欺骗攻击检测任务，提出了一种端到端的反欺骗模型。通过将扩展结构与信道注意块相结合，该模型可以集中于语音表示中信息量最大的子带，从而提高了抗欺骗性能。实验结果表明，在ASVSpoof 2019 LA和PA评测数据集上，本文提出的最佳单系统分别获得了1.88%和2.79%的相同错误率，证明了该模型的抗欺骗能力。摘要：Automatic speaker verification (ASV) has been widely used in the real life for identity authentication. However, with the rapid development of speech conversion, speech synthesis algorithms and the improvement of the quality of recording devices, ASV systems are vulnerable for spoof attacks. In recent years, there have many works about synthetic and replay speech detection, researchers had proposed a number of anti-spoofing methods based on hand-crafted features to improve the accuracy and robustness of synthetic and replay speech detection system. However, using hand-crafted features rather than raw waveform would lose certain information for anti-spoofing, which will reduce the detection performance of the system. Inspired by the promising performance of ConvNext in image classification tasks, we extend the ConvNext network architecture accordingly for spoof attacks detection task and propose an end-to-end anti-spoofing model. By integrating the extended architecture with the channel attention block, the proposed model can focus on the most informative sub-bands of speech representations to improve the anti-spoofing performance. Experiments show that our proposed best single system could achieve an equal error rate of 1.88% and 2.79% for the ASVSpoof 2019 LA evaluation dataset and PA evaluation dataset respectively, which demonstrate the model's capacity for anti-spoofing.
【10】I2CR: Improving Noise Robustness on Keyword Spotting Using Inter-Intra Contrastive Regularization
标题：I2CR：利用帧内对比正则化提高关键词检测的噪声稳健性
链接：http://arxiv.org/pdf/2209.06360v1
* 与cs.SD语音【4】为同一篇
作者：Dianwen Ng,Jia Qi Yip,Tanmay Surana,Zhao Yang,Chong Zhang,Yukun Ma,Chongjia Ni,Eng Siong Chng,Bin Ma机构：Alibaba Group, Singapore, School of Computer Science and Engineering, Nanyang Technological University, Singapore, Xi’an Jiaotong University, China摘要：关键词识别中的噪声鲁棒性问题一直是一个难题，因为许多模型无法克服噪声的严重影响，导致特征嵌入质量下降。提出了一种称为I2CR（Inter-Intra Contrasive Regularization）的对比正则化方法，通过引导模型学习特定于聚类的基本语音信息来改进特征表示。这涉及最大化相同类的帧内和帧间样本之间的相似性。结果，它将实例拉得更接近于形成更突出的聚类的更一般化的表示，并且减少了噪声的不利影响。我们证明了我们的方法在不同的噪声环境下对不同的骨干模型架构提供了一致的准确性改进。我们还证明了我们提出的框架提高了未知域外噪声和未知变异噪声信噪比的准确性。这表明了我们在噪声鲁棒性方面的整体改进工作的重要性。摘要：Noise robustness in keyword spotting remains a challenge as many models fail to overcome the heavy influence of noises, causing the deterioration of the quality of feature embeddings. We proposed a contrastive regularization method called Inter-Intra Contrastive Regularization (I2CR) to improve the feature representations by guiding the model to learn the fundamental speech information specific to the cluster. This involves maximizing the similarity across Intra and Inter samples of the same class. As a result, it pulls the instances closer to more generalized representations that form more prominent clusters and reduces the adverse impact of noises. We show that our method provides consistent improvements in accuracy over different backbone model architectures under different noise environments. We also demonstrate that our proposed framework has improved the accuracy of unseen out-of-domain noises and unseen variant noise SNRs. This indicates the significance of our work with the overall refinement in noise robustness.
【11】Using Rater and System Metadata to Explain Variance in the VoiceMOS Challenge 2022 Dataset
标题：使用评分员和系统元数据解释VoiceMOS挑战2022数据集中的差异
链接：http://arxiv.org/pdf/2209.06358v1
* 与cs.SD语音【5】为同一篇
作者：Michael Chinen,Jan Skoglund,Chandan K A Reddy,Alessandro Ragano,Andrew Hines机构：Google, University College Dublin, School of Computer Science, Dublin, Ireland备注：Preprint; accepted for Interspeech 2022摘要：非参考语音质量模型对于越来越多的应用是重要的。VoiceMOS 2022挑战赛提供了一个带有主观标签的合成语音转换和文本到语音样本的数据集。本研究着眼于从元数据和数据集的分布不平衡的语音质量的主观评级中可以解释的方差量。使用wav2vec 2.0构建语音质量模型，该模型具有额外的元数据特征，包括评分者组和系统标识符，并获得竞争性度量，包括在系统级的0.934的Spearman秩相关系数（SRCC）和0.088的MSE，以及在话语级的0.877和0.198。使用测试限制或设盲的数据和元数据进一步改进了指标。元数据分析表明，由于验证和测试数据集上每个系统使用的话语数量差异很大，系统级度量并不代表模型的系统级预测。我们的结论是，一般情况下，测试集中应该有足够的话语来限制样本平均误差，并且系统之间的话语计数应该相对平衡，否则话语级度量可能更可靠和可解释。摘要：Non-reference speech quality models are important for a growing number of applications. The VoiceMOS 2022 challenge provided a dataset of synthetic voice conversion and text-to-speech samples with subjective labels. This study looks at the amount of variance that can be explained in subjective ratings of speech quality from metadata and the distribution imbalances of the dataset. Speech quality models were constructed using wav2vec 2.0 with additional metadata features that included rater groups and system identifiers and obtained competitive metrics including a Spearman rank correlation coefficient (SRCC) of 0.934 and MSE of 0.088 at the system-level, and 0.877 and 0.198 at the utterance-level. Using data and metadata that the test restricted or blinded further improved the metrics. A metadata analysis showed that the system-level metrics do not represent the model's system-level prediction as a result of the wide variation in the number of utterances used for each system on the validation and test datasets. We conclude that, in general, conditions should have enough utterances in the test set to bound the sample mean error, and be relatively balanced in utterance count between systems, otherwise the utterance-level metrics may be more reliable and interpretable.
机器翻译，仅供参考
每日语音速递