今日论文合集：cs.SD语音34篇，eess.AS音频处理34篇。
【1】 Text-to-speech synthesis from dark data with evaluation-in-the-loop data selection
标题：利用循环求值数据选择从暗数据合成文本到语音的方法
链接：https://arxiv.org/abs/2210.14850
作者：Kentaro Seki,Shinnosuke Takamichi,Takaaki Saeki,Hiroshi Saruwatari机构：The University of Tokyo, Japan.备注：Submitted to ICASSP 2023摘要：本文提出了一种从暗数据中选择用于文本语音合成（TTS）的训练数据的方法。TTS模型通常是在高质量的语音语料库上训练的，这需要花费大量的时间和金钱来收集数据，这使得增加说话人变化非常具有挑战性。相比之下，有大量的数据，其可用性是未知的（又名，“暗数据”），如YouTube视频。为了利用TTS语料库之外的数据，先前的研究基于声学质量从语料库中选择语音数据。然而，考虑到已经提出了对数据噪声鲁棒的TTS模型，我们应该基于数据作为给定TTS模型的训练数据的重要性来选择数据，而不是语音本身的质量。我们的方法具有训练和评估循环，基于给定TTS模型的合成语音的自动预测质量来选择训练数据。YouTube数据的评测结果表明，该方法优于传统的基于音质的方法。摘要：This paper proposes a method for selecting training data for text-to-speech (TTS) synthesis from dark data. TTS models are typically trained on high-quality speech corpora that cost much time and money for data collection, which makes it very challenging to increase speaker variation. In contrast, there is a large amount of data whose availability is unknown (a.k.a, "dark data"), such as YouTube videos. To utilize data other than TTS corpora, previous studies have selected speech data from the corpora on the basis of acoustic quality. However, considering that TTS models robust to data noise have been proposed, we should select data on the basis of its importance as training data to the given TTS model, not the quality of speech itself. Our method with a loop of training and evaluation selects training data on the basis of the automatically predicted quality of synthetic speech of a given TTS model. Results of evaluations using YouTube data reveal that our method outperforms the conventional acoustic-quality-based method.
【2】 Monotonic segmental attention for automatic speech recognition标题：用于自动语音识别的单调分段注意链接：https://arxiv.org/abs/2210.14742
作者：Albert Zeyer,Robin Schmitt,Wei Zhou,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：accepted at SLT: this https URL摘要：本文提出了一种新的分段注意模型用于语音识别。我们将解码器的注意力限制在片段上，以避免全局注意力的二次运行时间，更好地推广到长序列，并最终实现流。我们直接比较了全局注意力和不同的分段注意力模型变量。我们开发并比较了两个独立的时间同步解码器，其中一个特别考虑了分段性质，从而产生了进一步的改进。对分段模型使用时间同步解码是新颖的，并且是朝向流应用的一个步骤。我们的实验显示了长度模型对预测片段边界的重要性。与文献中的其他单调注意方法相比，使用分段解码的最终最佳分段注意模型的性能优于全局注意。此外，我们观察到分段模型更好地推广到长达几分钟的长序列。摘要：We introduce a novel segmental-attention model for automatic speech recognition. We restrict the decoder attention to segments to avoid quadratic runtime of global attention, better generalize to long sequences, and eventually enable streaming. We directly compare global-attention and different segmental-attention modeling variants. We develop and compare two separate time-synchronous decoders, one specifically taking the segmental nature into account, yielding further improvements. Using time-synchronous decoding for segmental models is novel and a step towards streaming applications. Our experiments show the importance of a length model to predict the segment boundaries. The final best segmental-attention model using segmental decoding performs better than global-attention, in contrast to other monotonic attention approaches in the literature. Further, we observe that the segmental model generalizes much better to long sequences of up to several minutes.
【3】 Linguistic-Enhanced Transformer with CTC Embedding for Speech Recognition标题：用于语音识别的嵌入CTC的语言增强型转换器链接：https://arxiv.org/abs/2210.14725
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Mengyuan Zhao,Zhiyong Zhang,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by ECAISS2022, The Fourth International Workshop on Edge Computing and Artificial Intelligence based Sensor-Cloud System摘要：近年来，CTC-Attention联合模型的出现对自动语音识别（ASR）有了很大的改进。改进之处主要在于解码器对语言信息的建模。与声学编码器联合优化的解码器在训练期间以自回归方式从地面实况序列再现语言模型。然而，解码器的训练语料库限于语音转录，其远小于训练可接受的语言模型所需的语料库。这导致解码器的鲁棒性差。针对这一问题，本文提出了一种语言增强的变换器，在训练过程中将细化的CTC信息引入到译码器中，使译码器具有更强的鲁棒性。在AISHELL-1语音语料库上的实验表明，该算法的字符错误率（CER）相对降低了7%。在CTC-Attention ASR联合模型中，解码器对语言信息的敏感度高于对声音信息的敏感度。摘要：The recent emergence of joint CTC-Attention model shows significant improvement in automatic speech recognition (ASR). The improvement largely lies in the modeling of linguistic information by decoder. The decoder joint-optimized with an acoustic encoder renders the language model from ground-truth sequences in an auto-regressive manner during training. However, the training corpus of the decoder is limited to the speech transcriptions, which is far less than the corpus needed to train an acceptable language model. This leads to poor robustness of decoder. To alleviate this problem, we propose linguistic-enhanced transformer, which introduces refined CTC information to decoder during training process, so that the decoder can be more robust. Our experiments on AISHELL-1 speech corpus show that the character error rate (CER) is relatively reduced by up to 7%. We also find that in joint CTC-Attention ASR model, decoder is more sensitive to linguistic information than acoustic information.
【4】 Semi-Supervised Learning Based on Reference Model for Low-resource TTS标题：基于参考模型的半监督学习在低资源TTS中的应用链接：https://arxiv.org/abs/2210.14723
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by NMIC2022, The Fourth International Workshop on Network Meets Intelligent Computations摘要：以往的神经文本语音转换（TTS）方法主要基于监督学习方法，这意味着它们依赖于大规模的训练数据集，在低资源条件下难以达到与之相当的性能。针对这一问题，提出了一种基于神经TTS的半监督学习方法，该方法在标记目标数据有限的情况下，还可以解决以往自回归模型中存在的曝光偏差问题.具体地说，我们使用大量的源数据对基于Fastspeech2的参考模型进行预训练，并在有限的目标数据集上进行微调。同时，利用原始参考模型生成的伪标签进一步指导微调模型的训练，达到正则化的效果，减少微调模型在有限目标数据上训练时的过拟合。实验结果表明，本文提出的半监督学习方法在目标数据有限的情况下，显著提高了测试数据的语音质量，实现了语音合成的自然性和鲁棒性。摘要：Most previous neural text-to-speech (TTS) methods are mainly based on supervised learning methods, which means they depend on a large training dataset and hard to achieve comparable performance under low-resource conditions. To address this issue, we propose a semi-supervised learning method for neural TTS in which labeled target data is limited, which can also resolve the problem of exposure bias in the previous auto-regressive models. Specifically, we pre-train the reference model based on Fastspeech2 with much source data, fine-tuned on a limited target dataset. Meanwhile, pseudo labels generated by the original reference model are used to guide the fine-tuned model's training further, achieve a regularization effect, and reduce the overfitting of the fine-tuned model during training on the limited target data. Experimental results show that our proposed semi-supervised learning scheme with limited target data significantly improves the voice quality for test data to achieve naturalness and robustness in speech synthesis.
【5】 Pretrained audio neural networks for Speech emotion recognition in Portuguese标题：用于葡萄牙语语音情感识别的预训练音频神经网络链接：https://arxiv.org/abs/2210.14716
作者：Marcelo Matheus Gauy,Marcelo Finger机构：Universidade de São Paulo, Rua do Matão , São Paulo, Brazil备注：None摘要：语音情感识别（SER）的目标是识别语音的情感方面。针对巴西葡萄牙语语音的SER挑战是利用葡萄牙语的短片段提出的，这些短片段根据副语言元素（笑、哭等）被分类为中性、非中性女性和非中性男性。该数据集包含大约$50$分钟的巴西葡萄牙语语音。由于数据集偏小，我们研究迁移学习和数据扩充技术的组合是否可以产生积极的结果。因此，通过将称为SpecAugment的数据扩增技术与用于迁移学习的预训练音频神经网络（PANN）的使用相结合，我们能够获得有趣的结果。PANN（CNN 6、CNN 10和CNN 14）在包含超过$5000$小时的音频的称为AudioSet的大数据集上被预训练。在SER数据集上对它们进行微调，并将验证集上性能最佳的模型（CNN 10）提交给挑战，从挑战提供的基线的$0.54$提高到$0.73$的$F1$分数。此外，我们还测试了Transformer神经架构的使用，该架构在大约600 $小时的巴西葡萄牙语音频数据上进行了预训练。变压器以及更复杂的PANN模型（CNN 14）未能推广到SER数据集中的测试集，也没有超过基线。考虑到数据集大小的限制，目前用于SER的最佳方法是使用PANN（具体地，CNN 6和CNN 10）。摘要：The goal of speech emotion recognition (SER) is to identify the emotional aspects of speech. The SER challenge for Brazilian Portuguese speech was proposed with short snippets of Portuguese which are classified as neutral, non-neutral female and non-neutral male according to paralinguistic elements (laughing, crying, etc). This dataset contains about $50$ minutes of Brazilian Portuguese speech. As the dataset leans on the small side, we investigate whether a combination of transfer learning and data augmentation techniques can produce positive results. Thus, by combining a data augmentation technique called SpecAugment, with the use of Pretrained Audio Neural Networks (PANNs) for transfer learning we are able to obtain interesting results. The PANNs (CNN6, CNN10 and CNN14) are pretrained on a large dataset called AudioSet containing more than $5000$ hours of audio. They were finetuned on the SER dataset and the best performing model (CNN10) on the validation set was submitted to the challenge, achieving an $F1$ score of $0.73$ up from $0.54$ from the baselines provided by the challenge. Moreover, we also tested the use of Transformer neural architecture, pretrained on about $600$ hours of Brazilian Portuguese audio data. Transformers, as well as more complex models of PANNs (CNN14), fail to generalize to the test set in the SER dataset and do not beat the baseline. Considering the limitation of the dataset sizes, currently the best approach for SER is using PANNs (specifically, CNN6 and CNN10).
【6】 Pronunciation Generation for Foreign Language Words in Intra-Sentential Code-Switching Speech Recognition标题：句内码转换语音识别中外语单词的发音生成链接：https://arxiv.org/abs/2210.14691
作者：Wei Wang,Chao Zhang,Xiaopei Wu机构：Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei , China, A R T I C L E I N F O摘要：语码转换是指在一个句子或语篇中转换语言的现象。然而，有限的语码转换、不同语言的音素集以及高昂的重建成本给语码转换语音识别的声学模型的建立带来了挑战。本文以有限的语码转换数据为驱动材料，探索了一条在委托的母语声学模型上快速培养句内语码转换识别技能的捷径，提出了一种数据驱动的方法来制作种子词典，用于训练字形-音素模型来预测语码转换句子中外语词的映射发音。本文的数据驱动技术的核心工作包括语音解码方法和不同的选择方法。而对于词汇层面驱动材料不平衡的问题，我们有一个内在的辅助启示，那就是利用字形-音素模型，在拥有充足材料的词汇中学习好的发音规则，以帮助稀缺的词汇。实验结果表明，在纯汉语识别器上获得的汉英语码转换识别混合错误率为29.15%，通过数据驱动的方法加入外语单词的发音后，混合错误率为12.13%，最后通过不同的选择方法和内部辅助策略的组合，混合错误率达到了最好的11.14%。摘要：Code-Switching refers to the phenomenon of switching languages within a sentence or discourse. However, limited code-switching , different language phoneme-sets and high rebuilding costs throw a challenge to make the specialized acoustic model for code-switching speech recognition. In this paper, we make use of limited code-switching data as driving materials and explore a shortcut to quickly develop intra-sentential code-switching recognition skill on the commissioned native language acoustic model, where we propose a data-driven method to make the seed lexicon which is used to train grapheme-to-phoneme model to predict mapping pronunciations for foreign language word in code-switching sentences. The core work of the data-driven technology in this paper consists of a phonetic decoding method and different selection methods. And for imbalanced word-level driving materials problem, we have an internal assistance inspiration that learning the good pronunciation rules in the words that possess sufficient materials using the grapheme-to-phoneme model to help the scarce. Our experiments show that the Mixed Error Rate in intra-sentential Chinese-English code-switching recognition reduced from 29.15\%, acquired on the pure Chinese recognizer, to 12.13\% by adding foreign language words' pronunciation through our data-driven approach, and finally get the best result 11.14\% with the combination of different selection methods and internal assistance tactic.
【7】 In search of strong embedding extractors for speaker diarisation标题：寻找用于说话人腹泻的强嵌入抽取器链接：https://arxiv.org/abs/2210.14682
作者：Jee-weon Jung,Hee-Soo Heo,Bong-Jin Lee,Jaesung Huh,Andrew Brown,Youngki Kwon,Shinji Watanabe,Joon Son Chung机构：Naver Corporation, South Korea, Visual Geometry Group, Department of Engineering Science, University of Oxford, UK, Carnegie Mellon University, Pittsburgh, PA, USA, Korea Advanced Institute of Science and Technology, South Korea备注：5pages, 1 figure, 2 tables, submitted to ICASSP摘要：说话人嵌入提取器（EE）将输入音频映射到说话人判别潜空间，在说话人识别中具有重要作用。然而，在采用EE进行日记化时存在几个挑战，我们从这两个方面解决了两个关键问题。首先，评估不是直接的，因为更好的性能所需的特征在说话者确认和日记化之间不同。我们表明，在广泛采用的说话人确认评估协议上的更好的性能并不导致更好的对话性能。第二，嵌入提取器没有看到其中存在多个说话者的话语。这些输入不可避免地出现在说话人日记中，因为重叠的语音和说话人的变化;它们降低了性能。为了减轻第一个问题，我们生成更好地模拟日记化场景的说话者验证评估协议。我们提出了两种数据增强技术来缓解第二个问题，使嵌入提取器意识到重叠语音或说话人改变输入。一种技术生成重叠的语音段，而另一种技术生成两个说话者顺序说出的段。实验结果表明，两种方法都是有效的。摘要：Speaker embedding extractors (EEs), which map input audio to a speaker discriminant latent space, are of paramount importance in speaker diarisation. However, there are several challenges when adopting EEs for diarisation, from which we tackle two key problems. First, the evaluation is not straightforward because the features required for better performance differ between speaker verification and diarisation. We show that better performance on widely adopted speaker verification evaluation protocols does not lead to better diarisation performance. Second, embedding extractors have not seen utterances in which multiple speakers exist. These inputs are inevitably present in speaker diarisation because of overlapped speech and speaker changes; they degrade the performance. To mitigate the first problem, we generate speaker verification evaluation protocols that mimic the diarisation scenario better. We propose two data augmentation techniques to alleviate the second problem, making embedding extractors aware of overlapped speech or speaker change input. One technique generates overlapped speech segments, and the other generates segments where two speakers utter sequentially. Extensive experimental results using three state-of-the-art speaker embedding extractors demonstrate that both proposed approaches are effective.
【8】 Full-band General Audio Synthesis with Score-based Diffusion标题：基于分数扩散的全频段通用音频合成链接：https://arxiv.org/abs/2210.14661
作者：Santiago Pascual,Gautam Bhattacharya,Chunghsin Yeh,Jordi Pons,Joan Serrà机构：Dolby Laboratories摘要：最近的研究表明，深度生成模型能够处理来自单个标签的一般音频合成，产生各种脉冲、音调和环境声音。这样的模型对带限信号进行操作，并且作为自回归方法的结果，它们通常由预先训练的潜在编码器和/或若干级联模块来符合。本文提出了一种基于扩散的通用音频合成生成模型DAG，该模型在波形域端到端处理全频带信号。实验结果表明，与现有的标签条件生成器相比，DAG在质量和多样性方面都具有优势。更具体地，当与现有技术相比时，DAG的带限和全带版本实现了分别高达40%和65%的相对改进。我们相信DAG是足够灵活的，以适应不同的条件模式，同时提供高质量的合成。摘要：Recent works have shown the capability of deep generative models to tackle general audio synthesis from a single label, producing a variety of impulsive, tonal, and environmental sounds. Such models operate on band-limited signals and, as a result of an autoregressive approach, they are typically conformed by pre-trained latent encoders and/or several cascaded modules. In this work, we propose a diffusion-based generative model for general audio synthesis, named DAG, which deals with full-band signals end-to-end in the waveform domain. Results show the superiority of DAG over existing label-conditioned generators in terms of both quality and diversity. More specifically, when compared to the state of the art, the band-limited and full-band versions of DAG achieve relative improvements that go up to 40 and 65%, respectively. We believe DAG is flexible enough to accommodate different conditioning schemas while providing good quality synthesis.
【9】 TSUP Speaker Diarization System for Conversational Short-phrase Speaker Diarization Challenge标题：用于会话短短语说话人二值化挑战的TUP说话人二值化系统链接：https://arxiv.org/abs/2210.14653
作者：Bowen Pang,Huan Zhao,Gaosheng Zhang,Xiaoyue Yang,Yang Sun,Li Zhang,Qing Wang,Lei Xie机构：Northwestern Polytechnical University (NPU), Xi’an, China, Shenzhen Transsion Holding Limited摘要：本文介绍了TSUP团队对ISCSLP 2022会话短语说话人对话化（CSSD）挑战的提交，该挑战特别关注短语会话，并采用了一种新的评估指标--会话对话化错误率（CDER）。在这一挑战中，我们研究了三种典型的说话人识别系统，分别是基于谱聚类的识别系统、目标说话人语音活动检测系统和端到端神经网络识别系统。我们的主要发现总结如下。首先，在新的CDER度量下，SC方法比其他两种方法更受欢迎。第二，对于所有三种类型的扬声器诊断系统，超参数的调谐对于CDER是必不可少的。具体地，当子段的长度设置得更长时，CDER变得更小。最后，通过DOVER-LAP的多系统融合将恶化挑战数据上的CDER度量。我们提交的SC系统最终在挑战中排名第三。摘要：This paper describes the TSUP team's submission to the ISCSLP 2022 conversational short-phrase speaker diarization (CSSD) challenge which particularly focuses on short-phrase conversations with a new evaluation metric called conversational diarization error rate (CDER). In this challenge, we explore three kinds of typical speaker diarization systems, which are spectral clustering(SC) based diarization, target-speaker voice activity detection(TS-VAD) and end-to-end neural diarization(EEND) respectively. Our major findings are summarized as follows. First, the SC approach is more favored over the other two approaches under the new CDER metric. Second, tuning on hyperparameters is essential to CDER for all three types of speaker diarization systems. Specifically, CDER becomes smaller when the length of sub-segments setting longer. Finally, multi-system fusion through DOVER-LAP will worsen the CDER metric on the challenge data. Our submitted SC system eventually ranks the third place in the challenge.
【10】 Speaker Diarization Based on Multi-channel Microphone Array in Small-scale Meeting标题：小型会议中基于多通道麦克风阵列的说话人二值化链接：https://arxiv.org/abs/2210.14644
作者：Yuxuan Du,Ruohua Zhou摘要：在发言人日记化的任务中，小规模会议的数量占了很大的比重。当麦克风阵列被用作录音设备时，其空间信息通常被大多数研究者所忽略。本文借鉴d-向量和麦克风阵列空间向量相结合的聚类方法，针对4人以内的会议，提出了一种利用多通道麦克风阵列的二值化方法。我们利用语音增强技术对麦克风阵列输出的音频信号进行预处理。采用受控响应功率相位变换（SRP-PHAT）算法对语音片段进行分类，得到更准确的说话人，并根据说话人数目对语音片段进行聚类，以获得更好的性能。最后，利用DOVER-LAP算法对系统进行融合，得到最佳融合结果。我们在AMI语料库上对我们的系统进行了评估。与目前最好的实验结果相比，本系统在图像处理的误码率（DER）方面有了很大的改善。摘要：In the task of speaker diarization, the number of small-scale meetings accounts for a large proportion. When microphone arrays are employed as a recording device, its spatial information is usually ignored by most researchers. In this paper, inspired by the clustering method combining d-vector and microphone array spatial vector, we proposed a diarization method which using multi-channel microphone arrays for a meeting with no more than 4 speakers. We utilize speech enhancement to preprocess the audio from the microphone array. The Steered-Response Power Phase Transform (SRP-PHAT) algorithm are employed to get more accurate speakers, and apply the number of speakers to recluster the speech segments to achieve better performance. Finally, we fuse our system by DOVER-LAP to get the best result. We evaluated our system on the AMI corpus. Compared with the best experimental results so far, our system has achieved largely improvement in the diarization error rate (DER).
【11】 Fast Yet Effective Speech Emotion Recognition with Self-distillation标题：一种快速有效的自提炼语音情感识别链接：https://arxiv.org/abs/2210.14636
作者：Zhao Ren,Thanh Tam Nguyen,Yi Chang,Björn W. Schuller机构：L,S Research Center, Leibniz University Hannover, Germany, Griffith University, Australia, GLAM – Group on Language, Audio, & Music, Imperial College London, United Kingdom备注：Submitted to ICASSP 2023摘要：语音情感识别是从语音中识别人的情感状态的任务。SER在帮助对话系统真正理解我们的情绪并成为值得信赖的人类对话伙伴方面非常普遍。由于语音的冗长特性，SER还受到缺乏用于诸如深度神经网络之类的强大模型的丰富标记数据的困扰。基于大规模语音数据集的预训练复杂模型已经通过迁移学习成功地应用于SER。然而，对复杂模型进行微调仍然需要较大的存储空间，并且导致推理效率较低。在这篇文章中，我们认为通过自蒸馏可以实现快速而有效的SER，自蒸馏是一种同时微调预训练模型和训练其较浅版本的方法。我们的自我提炼框架有三方面的好处：（1）对声学模态采用自提取方法，突破了语音数据的有限真实性，在SER数据集上的性能优于现有模型;（2）在不同深度执行强大的模型可以在资源有限的边缘设备上实现自适应的准确性-效率折衷;（3）新的微调过程而不是从头开始训练以进行自蒸馏，导致了更快的学习时间和对具有少量标签信息的数据的现有技术水平的准确性。摘要：Speech emotion recognition (SER) is the task of recognising human's emotional states from speech. SER is extremely prevalent in helping dialogue systems to truly understand our emotions and become a trustworthy human conversational partner. Due to the lengthy nature of speech, SER also suffers from the lack of abundant labelled data for powerful models like deep neural networks. Pre-trained complex models on large-scale speech datasets have been successfully applied to SER via transfer learning. However, fine-tuning complex models still requires large memory space and results in low inference efficiency. In this paper, we argue achieving a fast yet effective SER is possible with self-distillation, a method of simultaneously fine-tuning a pretrained model and training shallower versions of itself. The benefits of our self-distillation framework are threefold: (1) the adoption of self-distillation method upon the acoustic modality breaks through the limited ground-truth of speech data, and outperforms the existing models' performance on an SER dataset; (2) executing powerful models at different depth can achieve adaptive accuracy-efficiency trade-offs on resource-limited edge devices; (3) a new fine-tuning process rather than training from scratch for self-distillation leads to faster learning time and the state-of-the-art accuracy on data with small quantities of label information.
【12】 Audio Mosaicing with Simulation-based Inference标题：基于仿真推理的音频拼接链接：https://arxiv.org/abs/2210.14602
作者：Andrew Gambardella,Youngjun Choi,Doyo Choi,Jinjoon Lee机构：Graduate School of Culture Technology, KAIST摘要：几十年来，马赛克和拼贴画一直是艺术不可或缺的一部分。在当代媒体艺术中特别重要的是音频马赛克，其中艺术家手动组合几个音频源以便构建一个单一的连贯声音，组合来自不同源的元素。在这里，我们提出了一个算法，以自动创建音频马赛克使用模拟为基础的推理范例。我们的算法将希望近似的音频文件和可以用于近似的音频文件列表作为输入，以可解释和解开的方式使用源，找到可以从中采样原始音频文件的重建的后验分布。我们通过创建一个音频马赛克来验证我们的方法，该马赛克使用100首韩国流行歌曲重新排列和重叠来重建传统韩国葬礼的声音。摘要：Mosaics and collages have been an integral part of art for decades. Particularly important in contemporary media art is the audio mosaic, in which an artist manually combines several audio sources in order to construct one single coherent sound, combining elements from disparate sources. Here we propose an algorithm to automatically create audio mosaics using the simulation-based inference paradigm. Our algorithm takes as input an audio file that one wishes to approximate, and a list of audio files one can use for approximation, finding a posterior distribution from which one can sample reconstructions of the original audio file, using the sources in an interpretable and disentangled manner. We validate our approach by creating an audio mosaic which reconstructs the sound of a traditional Korean funeral using 100 K-pop songs rearranged and overlapped.
【13】 Improving Speech-to-Speech Translation Through Unlabeled Text标题：通过无标记文本改进语音到语音的翻译链接：https://arxiv.org/abs/2210.14514
作者：Xuan-Phi Nguyen,Sravya Popuri,Changhan Wang,Yun Tang,Ilia Kulikov,Hongyu Gong机构：⋆ Meta AI, USA, † Nanyang Technological University, Singapore摘要：由于直接语音到语音翻译（S2 ST）数据的严重匮乏，S2 ST是翻译范式中最具挑战性的问题之一。虽然已经努力通过级联预训练语音识别（ASR）、机器翻译（MT）和文本到语音（TTS）模型来增加来自未标记语音的数据量;未标记的文本在改进S2 ST方面仍然相对未被充分利用。提出了一种有效的方法，利用现有的大量不同语言的未标记文本来创建大量的S2 ST数据，通过对生成的合成数据应用各种声学效果来提高S2 ST性能。根据经验，我们的方法在西班牙语-英语翻译方面比现有技术的水平高出2 BLEU。实验结果表明，该方法在资源非常少的情况下，对西班牙语-英语和俄语-英语的翻译都能取得显著的效果。摘要：Direct speech-to-speech translation (S2ST) is among the most challenging problems in the translation paradigm due to the significant scarcity of S2ST data. While effort has been made to increase the data size from unlabeled speech by cascading pretrained speech recognition (ASR), machine translation (MT) and text-to-speech (TTS) models; unlabeled text has remained relatively under-utilized to improve S2ST. We propose an effective way to utilize the massive existing unlabeled text from different languages to create a large amount of S2ST data to improve S2ST performance by applying various acoustic effects to the generated synthetic data. Empirically our method outperforms the state of the art in Spanish-English translation by up to 2 BLEU. Significant gains by the proposed method are demonstrated in extremely low-resource settings for both Spanish-English and Russian-English translations.
【14】 Parallel Gated Neural Network With Attention Mechanisim For Speech Enhancement标题：具有注意机制的并行门控神经网络语音增强链接：https://arxiv.org/abs/2210.14509
作者：Jianqiao Cui,Stefan Bleeck机构：Institute of Sound and Vibration Research, University of Southampton, UK备注：5 pages, 6 figures, references added摘要：深度学习算法越来越多地用于语音增强（SE）。在有监督的方法中，需要全局和局部信息来进行精确的谱映射。关键限制通常是对关键上下文信息的捕获不足。为了充分利用目标说话人的长时变特性，补偿干净语音的失真，采用序列到序列（S2S）映射结构，提出了一种新的单耳语音增强系统，该系统由特征提取模块（FEB）、补偿增强模块（ComEB）和掩蔽模块（MB）组成。在FEB中，U网模块利用复值谱提取抽象特征，单路利用掩蔽方法抑制幅值域的背景噪声，MB从FEB中提取幅值特征，并补偿ComEB中丢失的复值域特征，从而恢复出最终的干净语音。在Librispeech数据集上的实验结果表明，该模型在ESTOI和PESQ评分方面均优于现有模型.摘要：Deep learning algorithm are increasingly used for speech enhancement (SE). In supervised methods, global and local information is required for accurate spectral mapping. A key restriction is often poor capture of key contextual information. To leverage long-term for target speakers and compensate distortions of cleaned speech, this paper adopts a sequence-to-sequence (S2S) mapping structure and proposes a novel monaural speech enhancement system, consisting of a Feature Extraction Block (FEB), a Compensation Enhancement Block (ComEB) and a Mask Block (MB). In the FEB a U-net block is used to extract abstract features using complex-valued spectra with one path to suppress the background noise in the magnitude domain using masking methods and the MB takes magnitude features from the FEBand compensates the lost complex-domain features produced from ComEB to restore the final cleaned speech. Experiments are conducted on the Librispeech dataset and results show that the proposed model obtains better performance than recent models in terms of ESTOI and PESQ scores.
【15】 Two-stage dimensional emotion recognition by fusing predictions of acoustic and text networks using SVM标题：基于支持向量机的声学和文本网络预测融合的两阶段情感识别链接：https://arxiv.org/abs/2210.14495
作者：Bagus Tris Atmaja,Masato Akagi机构：Japan Advanced Institute of Science and Technology,-, Asahidai, Nomi, Ishikawa, -, Japan, Sepuluh Nopember Insitute of Technology, Sukolilo, Surabaya , Indonesia备注：None摘要：计算机自动语音情感识别（SER）是实现更自然的人机交互的关键。在人与人的互动中，正确感知情绪的能力对于在特定情况下采取进一步的措施至关重要。SER中的一个问题是是否有必要将声学特征与诸如面部表情、文本和运动捕捉等其他数据相结合。本研究提出一种结合声音与文字信息的后融合方法，包含两个步骤。首先，在深度学习系统中分别训练声学和文本特征。其次，将深度学习系统的预测结果输入支持向量机（SVM），以预测最终的回归分数。此外，本研究的任务是维度情感建模，因为它可以更深入地分析情感状态。实验结果表明，该两阶段后融合方法的性能优于任何一阶段处理方法，且从一阶段处理到两阶段处理具有线性相关性。这种后期融合方法改善了先前在一致性相关系数得分中测量的早期融合结果。摘要：Automatic speech emotion recognition (SER) by a computer is a critical component for more natural human-machine interaction. As in human-human interaction, the capability to perceive emotion correctly is essential to take further steps in a particular situation. One issue in SER is whether it is necessary to combine acoustic features with other data such as facial expressions, text, and motion capture. This research proposes to combine acoustic and text information by applying a late-fusion approach consisting of two steps. First, acoustic and text features are trained separately in deep learning systems. Second, the prediction results from the deep learning systems are fed into a support vector machine (SVM) to predict the final regression score. Furthermore, the task in this research is dimensional emotion modeling because it can enable a deeper analysis of affective states. Experimental results show that this two-stage, late-fusion approach, obtains higher performance than that of any one-stage processing, with a linear correlation from one-stage to two-stage processing. This late-fusion approach improves previous early fusion results measured in concordance correlation coefficients score.
【16】 AVES: Animal Vocalization Encoder based on Self-Supervision标题：AVES：基于自我监控的动物发声编码器链接：https://arxiv.org/abs/2210.14493
作者：Masato Hagiwara机构：Earth Species Project摘要：在生物声学中缺乏注释的训练数据阻碍了以监督方式训练的大规模神经网络模型的使用。为了充分利用大量未标注的音频数据，提出了一种基于自监督变换器的动物发声编码模型AVES（Animal Vocalization Encoder based on Self-Supervisory）.我们在一组不同的未注释音频数据集上对AVES进行预训练，并针对下游的生物声学任务对其进行微调。一系列分类和检测任务的综合实验表明，AVES优于所有强基线，甚至优于在注释音频分类数据集上训练的监督“顶线”模型。研究结果也表明，挑选一个与下游任务相关的小训练子集是训练高质量音频表示模型的有效方法。我们在以下网址开放了模型的源代码：摘要：The lack of annotated training data in bioacoustics hinders the use of large-scale neural network models trained in a supervised way. In order to leverage a large amount of unannotated audio data, we propose AVES (Animal Vocalization Encoder based on Self-Supervision), a self-supervised, transformer-based audio representation model for encoding animal vocalizations. We pretrain AVES on a diverse set of unannotated audio datasets and fine-tune them for downstream bioacoustics tasks. Comprehensive experiments with a suite of classification and detection tasks have shown that AVES outperforms all the strong baselines and even the supervised "topline" models trained on annotated audio classification datasets. The results also suggest that curating a small training subset related to downstream tasks is an efficient way to train high-quality audio representation models. We open-source our models at \url{https://github.com/earthspecies/aves}.
【17】 SCP-GAN: Self-Correcting Discriminator Optimization for Training Consistency Preserving Metric GAN on Speech Enhancement Tasks标题：SCP-GAN：语音增强任务中训练一致性保持度量GAN的自校正判别器优化链接：https://arxiv.org/abs/2210.14474
作者：Vasily Zadorozhnyy,Qiang Ye,Kazuhito Koishida机构：Department of Mathematics, University of Kentucky, Lexington, USA, Applied Sciences Group, Microsoft Corporation, Redmond, USA备注：5 pages (4 - manuscript and 1 - references), 2 figures, 2 tables摘要：近年来，生成式对抗网络（GANs）在语音增强（SE）任务中取得了显著的改进效果。然而，他们很难训练。在本文中，我们对GAN的训练方案进行了一些改进，这些改进可以应用于大多数基于GAN的SE模型。我们提出使用一致性损失函数，其针对由傅立叶变换和傅立叶逆变换引起的时间域和时频域中的不一致性。我们还提出了在SE任务上训练GAN鉴别器的自校正优化，这有助于避免鉴别器损失函数部分的“有害”训练方向。我们已经在几个最先进的基于GAN的SE模型上测试了我们提出的方法，并获得了一致的改进，包括对Voice Bank+DEMAND数据集的新的最先进的结果。摘要：In recent years, Generative Adversarial Networks (GANs) have produced significantly improved results in speech enhancement (SE) tasks. They are difficult to train, however. In this work, we introduce several improvements to the GAN training schemes, which can be applied to most GAN-based SE models. We propose using consistency loss functions, which target the inconsistency in time and time-frequency domains caused by Fourier and Inverse Fourier Transforms. We also present self-correcting optimization for training a GAN discriminator on SE tasks, which helps avoid "harmful" training directions for parts of the discriminator loss function. We have tested our proposed methods on several state-of-the-art GAN-based SE models and obtained consistent improvements, including new state-of-the-art results for the Voice Bank+DEMAND dataset.
【18】 The NPU-ASLP System for The ISCSLP 2022 Magichub Code-Swiching ASR Challenge标题：用于ISCSLP 2022 MagicHub代码交换ASR挑战赛的NPU-ASLP系统链接：https://arxiv.org/abs/2210.14448
作者：Yuhao Liang,Peikun Chen,Fan Yu,Xinfa Zhu,Tianyi Xu,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China备注：Submitted to ISCSLP 2022摘要：本文介绍了我们提交给ISCSLP 2022 Magichub代码切换ASR挑战赛的NPU-ASLP系统。在这一挑战中，我们首先探讨了几种流行的端到端ASR架构和训练策略，包括双编码器、语言感知编码器（LAE）和专家混合（MoE）。为了提高系统的语言建模能力，我们进一步尝试了内部语言模型和长上下文语言模型。鉴于挑战中有限的训练数据，我们进一步研究了数据增强的影响，包括速度扰动、基音偏移、语音编解码、SpecAugment和从文本到语音（TTS）的合成数据。最后，探讨了基于ROVER的分数融合，以充分利用不同模型的互补假设。我们提交的系统在测试集上的混合错误率（MER）达到了16.87%，在挑战排名中排名第二。摘要：This paper describes our NPU-ASLP system submitted to the ISCSLP 2022 Magichub Code-Switching ASR Challenge. In this challenge, we first explore several popular end-to-end ASR architectures and training strategies, including bi-encoder, language-aware encoder (LAE) and mixture of experts (MoE). To improve our system's language modeling ability, we further attempt the internal language model as well as the long context language model. Given the limited training data in the challenge, we further investigate the effects of data augmentation, including speed perturbation, pitch shifting, speech codec, SpecAugment and synthetic data from text-to-speech (TTS). Finally, we explore ROVER-based score fusion to make full use of complementary hypotheses from different models. Our submitted system achieves 16.87% on mix error rate (MER) on the test set and comes to the 2nd place in the challenge ranking.
【19】 Smart Speech Segmentation using Acousto-Linguistic Features with look-ahead标题：基于声学-语言特征的智能语音分割链接：https://arxiv.org/abs/2210.14446
作者：Piyush Behre,Naveen Parihar,Sharman Tan,Amy Shah,Eva Sharma,Geoffrey Liu,Shuangyu Chang,Hosam Khalil,Chris Basoglu,Sayan Pathak机构：Microsoft Corporation摘要：用于连续自动语音识别（ASR）的分段传统上使用静默超时或话音活动检测器（VAD），这两者都限于声学特征。考虑到人们说话时自然会停下来思考，这种细分往往过于激进。因此，分割发生在句子中间，阻碍了标点符号和下游任务，如机器翻译，高质量的分割对这些任务至关重要。利用声学特征的基于模型的分割方法是强大的，但是如果不理解语言本身，这些方法是有限的。我们提出了一种混合方法，利用声学和语言信息来改善分割。此外，我们还表明，包括一个单词作为前瞻提高了分割质量。平均而言，我们的模型将分割-F0.5评分提高了9.8%。我们证明了这种方法适用于多种语言。对于机器翻译的下游任务，它将翻译BLEU评分平均提高了1. 05分。摘要：Segmentation for continuous Automatic Speech Recognition (ASR) has traditionally used silence timeouts or voice activity detectors (VADs), which are both limited to acoustic features. This segmentation is often overly aggressive, given that people naturally pause to think as they speak. Consequently, segmentation happens mid-sentence, hindering both punctuation and downstream tasks like machine translation for which high-quality segmentation is critical. Model-based segmentation methods that leverage acoustic features are powerful, but without an understanding of the language itself, these approaches are limited. We present a hybrid approach that leverages both acoustic and language information to improve segmentation. Furthermore, we show that including one word as a look-ahead boosts segmentation quality. On average, our models improve segmentation-F0.5 score by 9.8% over baseline. We show that this approach works for multiple languages. For the downstream task of machine translation, it improves the translation BLEU score by an average of 1.05 points.
【20】 Dynamic Speech Endpoint Detection with Regression Targets标题：基于回归目标的动态语音端点检测链接：https://arxiv.org/abs/2210.14252
作者：Dawei Liang,Hang Su,Tarun Singh,Jay Mahadeokar,Shanil Puri,Jiedan Zhu,Edison Thomaz,Mike Seltzer机构：⋆ University of Texas at Austin, † Meta AI.备注：Manuscript submitted to ICASSP 2023摘要：交互式语音助手已被广泛用作各种场景中的输入接口，例如在智能家居设备、可穿戴设备和AR设备上。检测语音查询的结束，即语音端点指示，是语音助手与用户交互的重要任务。传统上，语音端点定位是基于纯分类方法和任意二进制目标。本文提出了一种新的基于回归的语音端点检测模型，该模型使得端点检测器能够根据用户查询的上下文调整其检测行为。具体地说，我们提出了一种暂停建模方法，并证明了其对动态端点定位的有效性。基于我们对供应商收集的智能手机和可穿戴设备语音查询的实验，与传统的基于分类的方法相比，我们的策略在端点延迟和准确性之间表现出更好的平衡。本文进一步讨论了该模型的优点和该框架的推广。摘要：Interactive voice assistants have been widely used as input interfaces in various scenarios, e.g. on smart homes devices, wearables and on AR devices. Detecting the end of a speech query, i.e. speech end-pointing, is an important task for voice assistants to interact with users. Traditionally, speech end-pointing is based on pure classification methods along with arbitrary binary targets. In this paper, we propose a novel regression-based speech end-pointing model, which enables an end-pointer to adjust its detection behavior based on context of user queries. Specifically, we present a pause modeling method and show its effectiveness for dynamic end-pointing. Based on our experiments with vendor-collected smartphone and wearables speech queries, our strategy shows a better trade-off between endpointing latency and accuracy, compared to the traditional classification-based method. We further discuss the benefits of this model and generalization of the framework in the paper.
【21】 Naturalistic Head Motion Generation from Speech标题：从语音生成自然主义的头部运动链接：https://arxiv.org/abs/2210.14800
作者：Trisha Mittal,Zakaria Aldeneh,Masha Fedzechkina,Anurag Ranjan,Barry-John Theobald机构：Department of Computer Science, University of Maryland, College Park, Apple备注：Submitted to ICASSP 2023摘要：为了提供丰富的交互体验，需要合成自然的头部运动以伴随具体化的会话代理的语音。大多数现有工作通过使用客观度量将所产生的头部运动与单个地面实况进行比较来评估所产生的头部运动的质量。然而，有许多看似合理的头部运动序列伴随着语音发声。在这项工作中，我们研究了从生成模型中采样的头部运动的感知质量的变化。我们表明，尽管提供了更多样的头部运动，但生成模型产生的运动具有不同程度的感知质量。我们最后表明，以往研究中常用的客观指标并不能准确地反映生成的头部运动的感知质量。这些结果为未来研究与人类质量感知相关的更好的客观度量开辟了一条有趣的途径。摘要：Synthesizing natural head motion to accompany speech for an embodied conversational agent is necessary for providing a rich interactive experience. Most prior works assess the quality of generated head motion by comparing them against a single ground-truth using an objective metric. Yet there are many plausible head motion sequences to accompany a speech utterance. In this work, we study the variation in the perceptual quality of head motions sampled from a generative model. We show that, despite providing more diverse head motions, the generative model produces motions with varying degrees of perceptual quality. We finally show that objective metrics commonly used in previous research do not accurately reflect the perceptual quality of generated head motions. These results open an interesting avenue for future work to investigate better objective metrics that correlate with human perception of quality.
【22】 Multitask Detection of Speaker Changes, Overlapping Speech and Voice Activity Using wav2vec 2.0标题：基于Wav2vec 2.0的说话人变化、重叠语音和语音活动的多任务检测链接：https://arxiv.org/abs/2210.14755
作者：Marie Kunešová,Zbyněk Zajíc机构：0Marie Kuneˇsov´a and Zbynˇek Zaj´ıcNew Technologies for the Information Society and Department of Cybernetics, University of West Bohemia备注：Submitted to ICASSP 2023摘要：自监督学习方法最近在广泛的机器学习问题上取得了巨大的成功。在语音处理领域，最近最成功的自监督模型之一是wav 2 vec 2.0。在本文中，我们探讨了该模型在三个基本语音分类任务上的有效性：说话者变化检测、重叠语音检测和话音活动检测。首先，我们只专注于一个任务--说话人变化检测--在这个任务中，我们提出的系统超过了以前在四个不同语料库上的报告结果，并且即使在来自人工设计的数据集的域外数据上训练时也达到了相当的性能。然后，我们扩展了我们的方法，在一个多任务系统中处理所有三个任务，并在AMI语料库上实现了最先进的性能。本文中的算法的实现可在www.example.com上公开获得https://github.com/mkunes/w2v2_audioFrameClassification。摘要：Self-supervised learning approaches have lately achieved great success on a broad spectrum of machine learning problems. In the field of speech processing, one of the most successful recent self-supervised models is wav2vec 2.0. In this paper, we explore the effectiveness of this model on three basic speech classification tasks: speaker change detection, overlapped speech detection, and voice activity detection. First, we concentrate on only one task -- speaker change detection -- where our proposed system surpasses the previously reported results on four different corpora, and achieves comparable performance even when trained on out-of-domain data from an artificially designed dataset. Then we expand our approach to tackle all three tasks in a single multitask system with state-of-the-art performance on the AMI corpus. The implementation of the algorithms in this paper is publicly available at https://github.com/mkunes/w2v2_audioFrameClassification.
【23】 Weighted Pressure Matching Based on Kernel Interpolation For Sound Field Reproduction标题：基于核插值法的声场再现加权声压匹配链接：https://arxiv.org/abs/2210.14711
作者：Shoichi Koyama,Kazuyuki Arikawa机构：Weighted pressure matching based on kernel interpolationfor sound field reproductionShoichi KOYAMA and Kazuyuki ARIKAWA( 1)( 1)Graduate School of Information Science and Technology, The University of Tokyo备注：Presented at 24th International Congress on Acoustics (ICA) 2022摘要：提出了一种称为加权声压匹配的声场再现方法。声场再现的目的在于使用目标区域内的多个扬声器来合成期望的声场。基于优化的方法是从合成声场和期望声场之间的误差的最小化中导出的，与基于积分方程的方法相比，这使得能够使用任意阵列几何形状。压力匹配由于其实现简单而被广泛应用于基于优化的声场再现方法中。其代价函数定义为目标区域内多个控制点的综合误差;然后，通过求解最小二乘问题得到扬声器的驱动信号。然而，在压力匹配中，不考虑控制点之间的区域。我们将成本函数定义为目标区域上的合成误差的区域积分。基于声场的核插值，该代价函数被表示为在控制点处的合成压力的加权平方误差。实验结果表明，加权压力匹配方法优于传统的压力匹配方法。摘要：A sound field reproduction method called weighted pressure matching is proposed. Sound field reproduction is aimed at synthesizing the desired sound field using multiple loudspeakers inside a target region. Optimization-based methods are derived from the minimization of errors between synthesized and desired sound fields, which enable the use of an arbitrary array geometry in contrast with integral-equation-based methods. Pressure matching is widely used in the optimization-based sound field reproduction methods because of its simplicity of implementation. Its cost function is defined as the synthesis errors at multiple control points inside the target region; then, the driving signals of the loudspeakers are obtained by solving a least-squares problem. However, in pressure matching, the region between the control points is not taken into consideration. We define the cost function as the regional integration of the synthesis error over the target region. On the basis of the kernel interpolation of the sound field, this cost function is represented as the weighted square error of the synthesized pressures at the control points. Experimental results indicate that the proposed weighted pressure matching outperforms conventional pressure matching.
【24】 Xiaoicesing 2: A High-Fidelity Singing Voice Synthesizer Based on Generative Adversarial Network标题：基于产生式对抗性网络的高保真歌声合成器--晓策2号链接：https://arxiv.org/abs/2210.14666
作者：Chunhui Wang,Chang Zeng,Xing He机构：Beijing Bombax XiaoIce Technology Co., Ltd, China, National Institute of Informatics, Japan ,SOKENDAI, Japan备注：submitted to icassp2023摘要：小冰唱是一个以产生48kHz歌唱声为目标的歌唱声合成（SVS）系统。然而，由于没有对这些部分的细节进行建模的特殊设计，由其生成的梅尔谱图在中高频区域过度平滑。本文提出了一种基于小波变换的Mel谱图生成算法XiaoiceSing2，该算法能够生成Mel谱图的中高频细节，从而更好地构建出全频带Mel谱图。具体来说，为了缓解这一问题，小冰唱2采用了一种生成式对抗网络（GAN），它由一个基于FastSpeech的生成器和一个多频段鉴别器组成。通过在自注意模块的基础上增加多个残差卷积模块，对前馈变换模块进行改进，以平衡局部和全局特征。多波段鉴别器包含三个子鉴别器，分别负责Mel谱图的低频、中频和高频部分。每个子鉴别器由多个片段鉴别器（SD）和细节鉴别器（DD）组成，以从不同方面区分音频。在我们内部的48kHz演唱数据集上的实验表明，XiaoiceSing2比XiaoiceSing的演唱质量有了显著的提高。摘要：XiaoiceSing is a singing voice synthesis (SVS) system that aims at generating 48kHz singing voices. However, the mel-spectrogram generated by it is over-smoothing in middle- and high-frequency areas due to no special design for modeling the details of these parts. In this paper, we propose XiaoiceSing2, which can generate the details of middle- and high-frequency parts to better construct the full-band mel-spectrogram. Specifically, in order to alleviate this problem, XiaoiceSing2 adopts a generative adversarial network (GAN), which consists of a FastSpeech-based generator and a multi-band discriminator. We improve the feed-forward Transformer (FFT) block by adding multiple residual convolutional blocks in parallel with the self-attention block to balance the local and global features. The multi-band discriminator contains three sub-discriminators responsible for low-, middle-, and high-frequency parts of the mel-spectrogram, respectively. Each sub-discriminator is composed of several segment discriminators (SD) and detail discriminators (DD) to distinguish the audio from different aspects. The experiment on our internal 48kHz singing voice dataset shows XiaoiceSing2 significantly improves the quality of the singing voice over XiaoiceSing.
【25】 Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input标题：屏蔽建模二人组：通过鼓励两个网络对输入建模来学习表示链接：https://arxiv.org/abs/2210.14648
作者：Daisuke Niizumi,Daiki Takeuchi,Yasunori Ohishi,Noboru Harada,Kunio Kashino机构：NTT Corporation, Japan备注：5 pages, 3 figures, and 5 tables. Under review摘要：掩码自动编码器是一种简单而强大的自监督学习方法。然而，它通过重建被屏蔽的输入面片来间接地学习表示。几种方法通过预测掩蔽的片的表示来直接学习表示;然而，我们认为使用所有面片来编码训练信号表示是次优的。提出了一种新的方法--M2 D（Masked Modeling Duo），该方法在获得训练信号的同时，直接使用掩模面片来学习表示。在M2 D中，在线网络编码可见的块并预测掩蔽的块表示，而目标网络，动量编码器，编码掩蔽的块。为了更好地预测目标表示，在线网络应该很好地对输入进行建模，而目标网络也应该很好地对输入进行建模，以与在线预测一致。然后，学习的表示应该更好地对输入建模。我们通过学习通用音频表示法验证了M2 D，M2 D在UrbanSound 8 K、VoxCeleb 1、AudioSet 20 K、GTZAN和SpeechCommandsV 2等任务上设置了新的一流性能。摘要：Masked Autoencoders is a simple yet powerful self-supervised learning method. However, it learns representations indirectly by reconstructing masked input patches. Several methods learn representations directly by predicting representations of masked patches; however, we think using all patches to encode training signal representations is suboptimal. We propose a new method, Masked Modeling Duo (M2D), that learns representations directly while obtaining training signals using only masked patches. In the M2D, the online network encodes visible patches and predicts masked patch representations, and the target network, a momentum encoder, encodes masked patches. To better predict target representations, the online network should model the input well, while the target network should also model it well to agree with online predictions. Then the learned representations should better model the input. We validated the M2D by learning general-purpose audio representations, and M2D set new state-of-the-art performance on tasks such as UrbanSound8K, VoxCeleb1, AudioSet20K, GTZAN, and SpeechCommandsV2.
【26】 Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function标题：基于深度学习的无置换损失函数多说话人DOA估计链接：https://arxiv.org/abs/2210.14581
作者：Qing Wang,Hang Chen,Ya Jiang,Zhe Wang,Yuyang Wang,Jun Du,Chin-Hui Lee机构：University of Science and Technology of China, Hefei, China, Georgia Institute of Technology, Atlanta, GA. USA备注：5 pages, 3 figures, accepted by ISCSLP 2022摘要：本文提出了一种基于深度学习的多说话人到达方向（DOA）估计算法，该算法利用无排列损失函数对音视频信号进行DOA估计。我们首先收集一个数据集用于多模态声源定位（SSL），其中音频和视频信号都记录在现实生活中的家庭电视场景中。在此基础上，提出了一种基于针孔摄像机模型的空间标注方法，通过摄像机坐标与像素坐标之间的转换，利用视频数据生成每个说话人的DOA基准。将空间位置信息与声学特征一起作为另一个输入，多说话人DOA估计可以作为活动说话人检测的分类任务来解决。由于每个说话人的位置被用作输入，因此将解决多说话人相关任务中的标签排列问题。仿真数据和真实数据的实验结果表明，该模型的估计性能明显优于纯音频DOA估计模型。摘要：In this paper, we propose a deep learning based multi-speaker direction of arrival (DOA) estimation with audio and visual signals by using permutation-free loss function. We first collect a data set for multi-modal sound source localization (SSL) where both audio and visual signals are recorded in real-life home TV scenarios. Then we propose a novel spatial annotation method to produce the ground truth of DOA for each speaker with the video data by transformation between camera coordinate and pixel coordinate according to the pin-hole camera model. With spatial location information served as another input along with acoustic feature, multi-speaker DOA estimation could be solved as a classification task of active speaker detection. Label permutation problem in multi-speaker related tasks will be addressed since the locations of each speaker are used as input. Experiments conducted on both simulated data and real data show that the proposed audio-visual DOA estimation model outperforms audio-only DOA estimation model by a large margin.
【27】 Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarization标题：使用令牌级语言二值化减少代码转换语音识别中的语言混淆链接：https://arxiv.org/abs/2210.14567
作者：Hexin Liu,Haihua Xu,Leibny Paola Garcia,Andy W. H. Khong,Yi He,Sanjeev Khudanpur机构：Bytedance AI Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, CLSP and HLT-COE, Johns Hopkins University, USA备注：Submitted to ICASSP 2023摘要：语码转换（CS）是指在语音信号中语言发生转换，导致语言混淆的现象，用于自动语音识别（ASR）。本文旨在从两个角度解决语言混淆问题，以改进CS-ASR：整合和分离语言信息。我们通过动态偏置CS-ASR模型中的标记级语言后验值来将语言信息合并到CS-ASR模型中，标记级语言后验值是序列到序列辅助语言日记化模块的输出。相反，解纠缠过程通过对抗性训练来减少语言之间的差异，以便规范化两种语言。我们在SEAME数据集上进行了实验。与基线模型相比，LD联合优化和语言后验偏差联合优化均获得了性能提升。通过对两种方法的比较表明，在减少CS语音中的语言混淆方面，融合语言信息比去纠缠更有效。摘要：Code-switching (CS) refers to the phenomenon that languages switch within a speech signal and leads to language confusion for automatic speech recognition (ASR). This paper aims to address language confusion for improving CS-ASR from two perspectives: incorporating and disentangling language information. We incorporate language information in the CS-ASR model by dynamically biasing the model with token-level language posteriors which are outputs of a sequence-to-sequence auxiliary language diarization module. In contrast, the disentangling process reduces the difference between languages via adversarial training so as to normalize two languages. We conduct the experiments on the SEAME dataset. Compared to the baseline model, both the joint optimization with LD and the language posterior bias achieve performance improvement. The comparison of the proposed methods indicates that incorporating language information is more effective than disentangling for reducing language confusion in CS speech.
【28】 Deep Metric Learning with Adaptive Margin and Adaptive Scale for Acoustic Word Discrimination标题：具有自适应边距和自适应尺度的声学单词识别深度度量学习链接：https://arxiv.org/abs/2210.14564
作者：Myunghun Jung,Hoirin Kim机构：School of Electrical Engineering, KAIST, Daejeon, Republic of Korea备注：Submitted to ICASSP2023摘要：目前深度度量学习中的损失函数多以对数和指数形式表示，且包含边界和尺度等重要的超参数。由于每个数据类都有一个固有的特性，一些先前的工作试图通过引入自适应的裕度来学习接近真实分布的嵌入空间。然而，根本没有关于自适应尺度的工作。我们认为在训练过程中，边界和尺度都应该是自适应可调的。本文提出了一种自适应间隔和尺度（AdaMS）方法，该方法将每个类的间隔和尺度的超参数替换为可学习的自适应间隔和自适应尺度参数.在《华尔街日报》数据集上对该方法进行了测试，取得了较好的单词识别效果.摘要：Many recent loss functions in deep metric learning are expressed with logarithmic and exponential forms, and they involve margin and scale as essential hyper-parameters. Since each data class has an intrinsic characteristic, several previous works have tried to learn embedding space close to the real distribution by introducing adaptive margins. However, there was no work on adaptive scales at all. We argue that both margin and scale should be adaptively adjustable during the training. In this paper, we propose a method called Adaptive Margin and Scale (AdaMS), where hyper-parameters of margin and scale are replaced with learnable parameters of adaptive margins and adaptive scales for each class. Our method is evaluated on Wall Street Journal dataset, and we achieve outperforming results for word discrimination tasks.
【29】 Position tracking of a varying number of sound sources with sliding permutation invariant training标题：利用滑动置换不变训练对不同数量声源的位置跟踪链接：https://arxiv.org/abs/2210.14536
作者：David Diaz-Guerra,Archontis Politis,Tuomas Virtanen机构：⋆ Dept. of Electronic Engineering and Communications, University of Zaragoza, Spain, †Audio Research Group, Tampere University, Finland备注：Submitted to the ICASSP 2023摘要：最近基于数据和学习的声源定位（SSL）方法在具有挑战性的声学场景中表现出了很强的性能。然而，在使这种方法适应于如现实中将发生的那样一致地跟踪出现和消失的多个源方面，几乎没有做什么工作。在本文中，我们提出了一种新的深度学习SSL模型的训练策略，该策略基于先前时间帧中估计位置和参考位置之间的最佳关联的均方误差，实现简单。它优化了跟踪系统的所需特性：处理时变数量的源，并根据它们的轨迹对定位估计排序，最小化身份切换（IDS）。通过对多混响运动源的仿真数据和两种模型结构的评估，证明了该算法在不影响帧间定位精度的前提下，有效地减少了身份切换。摘要：Recent data- and learning-based sound source localization (SSL) methods have shown strong performance in challenging acoustic scenarios. However, little work has been done on adapting such methods to track consistently multiple sources appearing and disappearing, as would occur in reality. In this paper, we present a new training strategy for deep learning SSL models with a straightforward implementation based on the mean squared error of the optimal association between estimated and reference positions in the preceding time frames. It optimizes the desired properties of a tracking system: handling a time-varying number of sources and ordering localization estimates according to their trajectories, minimizing identity switches (IDSs). Evaluation on simulated data of multiple reverberant moving sources and on two model architectures proves its effectiveness on reducing identity switches without compromising frame-wise localization accuracy.
【30】 UFO2: A unified pre-training framework for online and offline speech recognition标题：UFO2：一种统一的线上线下语音识别预训练框架链接：https://arxiv.org/abs/2210.14515
作者：Li Fu,Siqi Li,Qingtao Li,Liping Deng,Fangzhu Li,Lu Fan,Meng Chen,Xiaodong He机构：JD AI Research, Beijing, China备注：Submitted to ICASSP 2023摘要：本文提出了一种在线和离线自动语音识别的统一预训练框架（UFO 2），该框架1）将在线和离线两种独立的训练流程简化为一个过程，2）在有限的话语标注下提高了误词率（WER）性能.具体地说，我们将传统的基于离线模式的自监督学习（SSL）的ASR方法扩展到统一的方式，其中模型训练同时基于全上下文和动态组块输入。为了增强预训练的表示模型，使用停止梯度操作来将在线模式目标解耦到量化器。在预训练阶段和后级微调阶段，引入联合损失，训练出两种模式全权值分担的统一模型。在LibriSpeech数据集上的实验结果表明，UFO 2在离线和在线模式下的相对WER减少量分别为29.7%和18.2%，优于基于SSL的基线方法.摘要：In this paper, we propose a Unified pre-training Framework for Online and Offline (UFO2) Automatic Speech Recognition (ASR), which 1) simplifies the two separate training workflows for online and offline modes into one process, and 2) improves the Word Error Rate (WER) performance with limited utterance annotating. Specifically, we extend the conventional offline-mode Self-Supervised Learning (SSL)-based ASR approach to a unified manner, where the model training is conditioned on both the full-context and dynamic-chunked inputs. To enhance the pre-trained representation model, stop-gradient operation is applied to decouple the online-mode objectives to the quantizer. Moreover, in both the pre-training and the downstream fine-tuning stages, joint losses are proposed to train the unified model with full-weight sharing for the two modes. Experimental results on the LibriSpeech dataset show that UFO2 outperforms the SSL-based baseline method by 29.7% and 18.2% relative WER reduction in offline and online modes, respectively.
【31】 Effect of different splitting criteria on the performance of speech emotion recognition标题：不同分割标准对语音情感识别性能的影响链接：https://arxiv.org/abs/2210.14501
作者：Bagus Tris Atmaja,Akira Sasou机构：National Institute of Advanced Industrial Science and Technology, Japan备注：None摘要：传统的语音情感识别（SER）评估仅在说话者无关的条件下执行;他们中一些人甚至没有在这个条件下评价他们的结果。本文强调了通过脚本分割训练和测试数据的重要性，即句子开放或文本无关准则。实验结果表明，采用句子开放准则会降低SER的识别性能，这一发现表明，在不同的语言信息嵌入在声学信息中时，从语音中识别情感是困难的。令人惊讶的是，文本无关标准的表现始终比说话人+文本无关标准差。在SER表现上，划分标准的难度从最难到最容易的顺序是：文本无关、说话者+文本无关、说话者无关和说话者+文本相关。说话人+文本无关和文本无关的差异小于其他标准，增加了从不同句子中的语音识别情感的难度。摘要：Traditional speech emotion recognition (SER) evaluations have been performed merely on a speaker-independent condition; some of them even did not evaluate their result on this condition. This paper highlights the importance of splitting training and test data for SER by script, known as sentence-open or text-independent criteria. The results show that employing sentence-open criteria degraded the performance of SER. This finding implies the difficulties of recognizing emotion from speech in different linguistic information embedded in acoustic information. Surprisingly, text-independent criteria consistently performed worse than speaker+text-independent criteria. The full order of difficulties for splitting criteria on SER performances from the most difficult to the easiest is text-independent, speaker+text-independent, speaker-independent, and speaker+text-dependent. The gap between speaker+text-independent and text-independent was smaller than other criteria, strengthening the difficulties of recognizing emotion from speech in different sentences.
【32】 Sinusoidal Frequency Estimation by Gradient Descent标题：基于梯度下降法的正弦频率估计链接：https://arxiv.org/abs/2210.14476
作者：Ben Hayes,Charalampos Saitis,György Fazekas机构：Centre for Digital Music, Queen Mary University of London备注：Submitted to ICASSP 2023摘要：正弦参数估计是从谱分析到时间序列预测等应用中的一项基本任务。然而，通过梯度下降来估计正弦频率参数通常是不可能的，因为误差函数是非凸的并且密集地填充有局部最小值。因此，不断增长的可微分信号处理方法家族已经不能调谐振荡分量的频率，从而阻碍了它们在广泛应用中的使用。本文提出了一种利用复指数代理函数的Wirtinger导数和任意一阶梯度优化器进行正弦频率和幅值联合估计的技术，使得无约束正弦模型的神经网络控制器的端到端训练成为可能。摘要：Sinusoidal parameter estimation is a fundamental task in applications from spectral analysis to time-series forecasting. Estimating the sinusoidal frequency parameter by gradient descent is, however, often impossible as the error function is non-convex and densely populated with local minima. The growing family of differentiable signal processing methods has therefore been unable to tune the frequency of oscillatory components, preventing their use in a broad range of applications. This work presents a technique for joint sinusoidal frequency and amplitude estimation using the Wirtinger derivatives of a complex exponential surrogate and any first order gradient-based optimizer, enabling end to-end training of neural network controllers for unconstrained sinusoidal models.
【33】 RedPen: Region- and Reason-Annotated Dataset of Unnatural Speech标题：红笔：非自然语言的区域和原因注释数据集链接：https://arxiv.org/abs/2210.14406
作者：Kyumin Park,Keon Lee,Daeyoung Kim,Dongyeop Kang机构：School of Computing, KAIST, Republic of Korea, KRAFTON Inc., Republic of Korea, University of Minnesota, US备注：Submitted to ICASSP 2023摘要：即使在语音合成模型方面有了最新的进展，对这种模型的评估也纯粹基于作为单个自然度分数的人类判断，例如平均意见分数（MOS）。基于分数的度量没有给出关于哪些词类是不自然的或者人类判断者为什么认为它们是不自然的任何进一步信息。提出了一个新的语音数据集RedPen，它对不自然的语音区域进行了人工标注，并分析了其原因. RedPen由180个合成语音组成，其中有由群组工作者注释的非自然区域;然后，这些区域被推理并按错误类型分类，例如语音抖动和背景噪声。我们发现我们的数据集显示了比模型驱动的不自然度预测更好的解释不自然的语音区域。我们的分析还表明，每个模型包括不同类型的错误类型。总之，我们的数据集成功地显示了不同的错误区域和类型存在于单一自然度得分下的可能性。我们相信，我们的数据集将有助于在未来评估和开发更可解释的语音模型。我们的数据集将在接受后公开。摘要：Even with recent advances in speech synthesis models, the evaluation of such models is based purely on human judgement as a single naturalness score, such as the Mean Opinion Score (MOS). The score-based metric does not give any further information about which parts of speech are unnatural or why human judges believe they are unnatural. We present a novel speech dataset, RedPen, with human annotations on unnatural speech regions and their corresponding reasons. RedPen consists of 180 synthesized speeches with unnatural regions annotated by crowd workers; These regions are then reasoned and categorized by error types, such as voice trembling and background noise. We find that our dataset shows a better explanation for unnatural speech regions than the model-driven unnaturalness prediction. Our analysis also shows that each model includes different types of error types. Summing up, our dataset successfully shows the possibility that various error regions and types lie under the single naturalness score. We believe that our dataset will shed light on the evaluation and development of more interpretable speech models in the future. Our dataset will be publicly available upon acceptance.
【34】 Artificial ASMR: A Cyber-Psychological Study标题：人工ASMR：一种网络心理学研究链接：https://arxiv.org/abs/2210.14321
作者：Zexiu Wu,Bin Han,C. Clark Cao,Hans. D. Schotten机构：∗Technische Universit¨at Kaiserslautern, †Lingnan University, ‡German Research Center of Artificial Intelligence (DFKI)备注：Submitted to ICASSP 2023摘要：自主感觉经络反应（ASMR）在过去的十年里得到了广泛的应用，但对其的研究还很少，而且还不成熟。针对ASMR音频中常见的声学模式，研究了音频信号的时频特征和周期特征之间的相关性及其在触发ASMR效应中的有效性。采用了一种结合信号处理、人工智能和实验心理学的网络心理学方法，利用该方法，我们能够识别与ASMR相关的声学特征，并由此合成随机人工ASMR音频。摘要：The popularity of Autonomous Sensory Meridian Response (ASMR) has skyrockteted over the past decade, but scientific studies on it are still few and immature. With our attention caught by the common acoustic patterns in ASMR audios, we investigate the correlation between the time-frequency and cyclic features of audio signals and their effectiveness in triggering ASMR effects. A cyber-psychological approach that combines signal processing, artificial intelligence, and experimental psychology is taken, with which we are able to identify ASMR-related acoustic features, and therewith synthesize random artificial ASMR audios.
【1】 Naturalistic Head Motion Generation from Speech标题：从语音生成自然主义的头部运动链接：https://arxiv.org/abs/2210.14800
* 与cs.SD语音【21】为同一篇
作者：Trisha Mittal,Zakaria Aldeneh,Masha Fedzechkina,Anurag Ranjan,Barry-John Theobald机构：Department of Computer Science, University of Maryland, College Park, Apple备注：Submitted to ICASSP 2023摘要：为了提供丰富的交互体验，需要合成自然的头部运动以伴随具体化的会话代理的语音。大多数现有工作通过使用客观度量将所产生的头部运动与单个地面实况进行比较来评估所产生的头部运动的质量。然而，有许多看似合理的头部运动序列伴随着语音发声。在这项工作中，我们研究了从生成模型中采样的头部运动的感知质量的变化。我们表明，尽管提供了更多样的头部运动，但生成模型产生的运动具有不同程度的感知质量。我们最后表明，以往研究中常用的客观指标并不能准确地反映生成的头部运动的感知质量。这些结果为未来研究与人类质量感知相关的更好的客观度量开辟了一条有趣的途径。摘要：Synthesizing natural head motion to accompany speech for an embodied conversational agent is necessary for providing a rich interactive experience. Most prior works assess the quality of generated head motion by comparing them against a single ground-truth using an objective metric. Yet there are many plausible head motion sequences to accompany a speech utterance. In this work, we study the variation in the perceptual quality of head motions sampled from a generative model. We show that, despite providing more diverse head motions, the generative model produces motions with varying degrees of perceptual quality. We finally show that objective metrics commonly used in previous research do not accurately reflect the perceptual quality of generated head motions. These results open an interesting avenue for future work to investigate better objective metrics that correlate with human perception of quality.
【2】 Multitask Detection of Speaker Changes, Overlapping Speech and Voice Activity Using wav2vec 2.0标题：基于Wav2vec 2.0的说话人变化、重叠语音和语音活动的多任务检测链接：https://arxiv.org/abs/2210.14755
* 与cs.SD语音【22】为同一篇
作者：Marie Kunešová,Zbyněk Zajíc机构：0Marie Kuneˇsov´a and Zbynˇek Zaj´ıcNew Technologies for the Information Society and Department of Cybernetics, University of West Bohemia备注：Submitted to ICASSP 2023摘要：自监督学习方法最近在广泛的机器学习问题上取得了巨大的成功。在语音处理领域，最近最成功的自监督模型之一是wav 2 vec 2.0。在本文中，我们探讨了该模型在三个基本语音分类任务上的有效性：说话者变化检测、重叠语音检测和话音活动检测。首先，我们只专注于一个任务--说话人变化检测--在这个任务中，我们提出的系统超过了以前在四个不同语料库上的报告结果，并且即使在来自人工设计的数据集的域外数据上训练时也达到了相当的性能。然后，我们扩展了我们的方法，在一个多任务系统中处理所有三个任务，并在AMI语料库上实现了最先进的性能。本文中的算法的实现可在www.example.com上公开获得https://github.com/mkunes/w2v2_audioFrameClassification。摘要：Self-supervised learning approaches have lately achieved great success on a broad spectrum of machine learning problems. In the field of speech processing, one of the most successful recent self-supervised models is wav2vec 2.0. In this paper, we explore the effectiveness of this model on three basic speech classification tasks: speaker change detection, overlapped speech detection, and voice activity detection. First, we concentrate on only one task -- speaker change detection -- where our proposed system surpasses the previously reported results on four different corpora, and achieves comparable performance even when trained on out-of-domain data from an artificially designed dataset. Then we expand our approach to tackle all three tasks in a single multitask system with state-of-the-art performance on the AMI corpus. The implementation of the algorithms in this paper is publicly available at https://github.com/mkunes/w2v2_audioFrameClassification.
【3】 Weighted Pressure Matching Based on Kernel Interpolation For Sound Field Reproduction标题：基于核插值法的声场再现加权声压匹配链接：https://arxiv.org/abs/2210.14711
* 与cs.SD语音【23】为同一篇
作者：Shoichi Koyama,Kazuyuki Arikawa机构：Weighted pressure matching based on kernel interpolationfor sound field reproductionShoichi KOYAMA and Kazuyuki ARIKAWA( 1)( 1)Graduate School of Information Science and Technology, The University of Tokyo备注：Presented at 24th International Congress on Acoustics (ICA) 2022摘要：提出了一种称为加权声压匹配的声场再现方法。声场再现的目的在于使用目标区域内的多个扬声器来合成期望的声场。基于优化的方法是从合成声场和期望声场之间的误差的最小化中导出的，与基于积分方程的方法相比，这使得能够使用任意阵列几何形状。压力匹配由于其实现简单而被广泛应用于基于优化的声场再现方法中。其代价函数定义为目标区域内多个控制点的综合误差;然后，通过求解最小二乘问题得到扬声器的驱动信号。然而，在压力匹配中，不考虑控制点之间的区域。我们将成本函数定义为目标区域上的合成误差的区域积分。基于声场的核插值，该代价函数被表示为在控制点处的合成压力的加权平方误差。实验结果表明，加权压力匹配方法优于传统的压力匹配方法。摘要：A sound field reproduction method called weighted pressure matching is proposed. Sound field reproduction is aimed at synthesizing the desired sound field using multiple loudspeakers inside a target region. Optimization-based methods are derived from the minimization of errors between synthesized and desired sound fields, which enable the use of an arbitrary array geometry in contrast with integral-equation-based methods. Pressure matching is widely used in the optimization-based sound field reproduction methods because of its simplicity of implementation. Its cost function is defined as the synthesis errors at multiple control points inside the target region; then, the driving signals of the loudspeakers are obtained by solving a least-squares problem. However, in pressure matching, the region between the control points is not taken into consideration. We define the cost function as the regional integration of the synthesis error over the target region. On the basis of the kernel interpolation of the sound field, this cost function is represented as the weighted square error of the synthesized pressures at the control points. Experimental results indicate that the proposed weighted pressure matching outperforms conventional pressure matching.
【4】 Xiaoicesing 2: A High-Fidelity Singing Voice Synthesizer Based on Generative Adversarial Network标题：基于产生式对抗性网络的高保真歌声合成器--晓策2号链接：https://arxiv.org/abs/2210.14666
* 与cs.SD语音【24】为同一篇
作者：Chunhui Wang,Chang Zeng,Xing He机构：Beijing Bombax XiaoIce Technology Co., Ltd, China, National Institute of Informatics, Japan ,SOKENDAI, Japan备注：submitted to icassp2023摘要：小冰唱是一个以产生48kHz歌唱声为目标的歌唱声合成（SVS）系统。然而，由于没有对这些部分的细节进行建模的特殊设计，由其生成的梅尔谱图在中高频区域过度平滑。本文提出了一种基于小波变换的Mel谱图生成算法XiaoiceSing2，该算法能够生成Mel谱图的中高频细节，从而更好地构建出全频带Mel谱图。具体来说，为了缓解这一问题，小冰唱2采用了一种生成式对抗网络（GAN），它由一个基于FastSpeech的生成器和一个多频段鉴别器组成。通过在自注意模块的基础上增加多个残差卷积模块，对前馈变换模块进行改进，以平衡局部和全局特征。多波段鉴别器包含三个子鉴别器，分别负责Mel谱图的低频、中频和高频部分。每个子鉴别器由多个片段鉴别器（SD）和细节鉴别器（DD）组成，以从不同方面区分音频。在我们内部的48kHz演唱数据集上的实验表明，XiaoiceSing2比XiaoiceSing的演唱质量有了显著的提高。摘要：XiaoiceSing is a singing voice synthesis (SVS) system that aims at generating 48kHz singing voices. However, the mel-spectrogram generated by it is over-smoothing in middle- and high-frequency areas due to no special design for modeling the details of these parts. In this paper, we propose XiaoiceSing2, which can generate the details of middle- and high-frequency parts to better construct the full-band mel-spectrogram. Specifically, in order to alleviate this problem, XiaoiceSing2 adopts a generative adversarial network (GAN), which consists of a FastSpeech-based generator and a multi-band discriminator. We improve the feed-forward Transformer (FFT) block by adding multiple residual convolutional blocks in parallel with the self-attention block to balance the local and global features. The multi-band discriminator contains three sub-discriminators responsible for low-, middle-, and high-frequency parts of the mel-spectrogram, respectively. Each sub-discriminator is composed of several segment discriminators (SD) and detail discriminators (DD) to distinguish the audio from different aspects. The experiment on our internal 48kHz singing voice dataset shows XiaoiceSing2 significantly improves the quality of the singing voice over XiaoiceSing.
【5】 Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input标题：屏蔽建模二人组：通过鼓励两个网络对输入建模来学习表示链接：https://arxiv.org/abs/2210.14648
* 与cs.SD语音【25】为同一篇
作者：Daisuke Niizumi,Daiki Takeuchi,Yasunori Ohishi,Noboru Harada,Kunio Kashino机构：NTT Corporation, Japan备注：5 pages, 3 figures, and 5 tables. Under review摘要：掩码自动编码器是一种简单而强大的自监督学习方法。然而，它通过重建被屏蔽的输入面片来间接地学习表示。几种方法通过预测掩蔽的片的表示来直接学习表示;然而，我们认为使用所有面片来编码训练信号表示是次优的。提出了一种新的方法--M2 D（Masked Modeling Duo），该方法在获得训练信号的同时，直接使用掩模面片来学习表示。在M2 D中，在线网络编码可见的块并预测掩蔽的块表示，而目标网络，动量编码器，编码掩蔽的块。为了更好地预测目标表示，在线网络应该很好地对输入进行建模，而目标网络也应该很好地对输入进行建模，以与在线预测一致。然后，学习的表示应该更好地对输入建模。我们通过学习通用音频表示法验证了M2 D，M2 D在UrbanSound 8 K、VoxCeleb 1、AudioSet 20 K、GTZAN和SpeechCommandsV 2等任务上设置了新的一流性能。摘要：Masked Autoencoders is a simple yet powerful self-supervised learning method. However, it learns representations indirectly by reconstructing masked input patches. Several methods learn representations directly by predicting representations of masked patches; however, we think using all patches to encode training signal representations is suboptimal. We propose a new method, Masked Modeling Duo (M2D), that learns representations directly while obtaining training signals using only masked patches. In the M2D, the online network encodes visible patches and predicts masked patch representations, and the target network, a momentum encoder, encodes masked patches. To better predict target representations, the online network should model the input well, while the target network should also model it well to agree with online predictions. Then the learned representations should better model the input. We validated the M2D by learning general-purpose audio representations, and M2D set new state-of-the-art performance on tasks such as UrbanSound8K, VoxCeleb1, AudioSet20K, GTZAN, and SpeechCommandsV2.
【6】 Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function标题：基于深度学习的无置换损失函数多说话人DOA估计链接：https://arxiv.org/abs/2210.14581
* 与cs.SD语音【26】为同一篇
作者：Qing Wang,Hang Chen,Ya Jiang,Zhe Wang,Yuyang Wang,Jun Du,Chin-Hui Lee机构：University of Science and Technology of China, Hefei, China, Georgia Institute of Technology, Atlanta, GA. USA备注：5 pages, 3 figures, accepted by ISCSLP 2022摘要：本文提出了一种基于深度学习的多说话人到达方向（DOA）估计算法，该算法利用无排列损失函数对音视频信号进行DOA估计。我们首先收集一个数据集用于多模态声源定位（SSL），其中音频和视频信号都记录在现实生活中的家庭电视场景中。在此基础上，提出了一种基于针孔摄像机模型的空间标注方法，通过摄像机坐标与像素坐标之间的转换，利用视频数据生成每个说话人的DOA基准。将空间位置信息与声学特征一起作为另一个输入，多说话人DOA估计可以作为活动说话人检测的分类任务来解决。由于每个说话人的位置被用作输入，因此将解决多说话人相关任务中的标签排列问题。仿真数据和真实数据的实验结果表明，该模型的估计性能明显优于纯音频DOA估计模型。摘要：In this paper, we propose a deep learning based multi-speaker direction of arrival (DOA) estimation with audio and visual signals by using permutation-free loss function. We first collect a data set for multi-modal sound source localization (SSL) where both audio and visual signals are recorded in real-life home TV scenarios. Then we propose a novel spatial annotation method to produce the ground truth of DOA for each speaker with the video data by transformation between camera coordinate and pixel coordinate according to the pin-hole camera model. With spatial location information served as another input along with acoustic feature, multi-speaker DOA estimation could be solved as a classification task of active speaker detection. Label permutation problem in multi-speaker related tasks will be addressed since the locations of each speaker are used as input. Experiments conducted on both simulated data and real data show that the proposed audio-visual DOA estimation model outperforms audio-only DOA estimation model by a large margin.
【7】 Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarization标题：使用令牌级语言二值化减少代码转换语音识别中的语言混淆链接：https://arxiv.org/abs/2210.14567
* 与cs.SD语音【27】为同一篇
作者：Hexin Liu,Haihua Xu,Leibny Paola Garcia,Andy W. H. Khong,Yi He,Sanjeev Khudanpur机构：Bytedance AI Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, CLSP and HLT-COE, Johns Hopkins University, USA备注：Submitted to ICASSP 2023摘要：语码转换（CS）是指在语音信号中语言发生转换，导致语言混淆的现象，用于自动语音识别（ASR）。本文旨在从两个角度解决语言混淆问题，以改进CS-ASR：整合和分离语言信息。我们通过动态偏置CS-ASR模型中的标记级语言后验值来将语言信息合并到CS-ASR模型中，标记级语言后验值是序列到序列辅助语言日记化模块的输出。相反，解纠缠过程通过对抗性训练来减少语言之间的差异，以便规范化两种语言。我们在SEAME数据集上进行了实验。与基线模型相比，LD联合优化和语言后验偏差联合优化均获得了性能提升。通过对两种方法的比较表明，在减少CS语音中的语言混淆方面，融合语言信息比去纠缠更有效。摘要：Code-switching (CS) refers to the phenomenon that languages switch within a speech signal and leads to language confusion for automatic speech recognition (ASR). This paper aims to address language confusion for improving CS-ASR from two perspectives: incorporating and disentangling language information. We incorporate language information in the CS-ASR model by dynamically biasing the model with token-level language posteriors which are outputs of a sequence-to-sequence auxiliary language diarization module. In contrast, the disentangling process reduces the difference between languages via adversarial training so as to normalize two languages. We conduct the experiments on the SEAME dataset. Compared to the baseline model, both the joint optimization with LD and the language posterior bias achieve performance improvement. The comparison of the proposed methods indicates that incorporating language information is more effective than disentangling for reducing language confusion in CS speech.
【8】 Deep Metric Learning with Adaptive Margin and Adaptive Scale for Acoustic Word Discrimination标题：具有自适应边距和自适应尺度的声学单词识别深度度量学习链接：https://arxiv.org/abs/2210.14564
* 与cs.SD语音【28】为同一篇
作者：Myunghun Jung,Hoirin Kim机构：School of Electrical Engineering, KAIST, Daejeon, Republic of Korea备注：Submitted to ICASSP2023摘要：目前深度度量学习中的损失函数多以对数和指数形式表示，且包含边界和尺度等重要的超参数。由于每个数据类都有一个固有的特性，一些先前的工作试图通过引入自适应的裕度来学习接近真实分布的嵌入空间。然而，根本没有关于自适应尺度的工作。我们认为在训练过程中，边界和尺度都应该是自适应可调的。本文提出了一种自适应间隔和尺度（AdaMS）方法，该方法将每个类的间隔和尺度的超参数替换为可学习的自适应间隔和自适应尺度参数.在《华尔街日报》数据集上对该方法进行了测试，取得了较好的单词识别效果.摘要：Many recent loss functions in deep metric learning are expressed with logarithmic and exponential forms, and they involve margin and scale as essential hyper-parameters. Since each data class has an intrinsic characteristic, several previous works have tried to learn embedding space close to the real distribution by introducing adaptive margins. However, there was no work on adaptive scales at all. We argue that both margin and scale should be adaptively adjustable during the training. In this paper, we propose a method called Adaptive Margin and Scale (AdaMS), where hyper-parameters of margin and scale are replaced with learnable parameters of adaptive margins and adaptive scales for each class. Our method is evaluated on Wall Street Journal dataset, and we achieve outperforming results for word discrimination tasks.
【9】 Position tracking of a varying number of sound sources with sliding permutation invariant training标题：利用滑动置换不变训练对不同数量声源的位置跟踪链接：https://arxiv.org/abs/2210.14536
* 与cs.SD语音【29】为同一篇
作者：David Diaz-Guerra,Archontis Politis,Tuomas Virtanen机构：⋆ Dept. of Electronic Engineering and Communications, University of Zaragoza, Spain, †Audio Research Group, Tampere University, Finland备注：Submitted to the ICASSP 2023摘要：最近基于数据和学习的声源定位（SSL）方法在具有挑战性的声学场景中表现出了很强的性能。然而，在使这种方法适应于如现实中将发生的那样一致地跟踪出现和消失的多个源方面，几乎没有做什么工作。在本文中，我们提出了一种新的深度学习SSL模型的训练策略，该策略基于先前时间帧中估计位置和参考位置之间的最佳关联的均方误差，实现简单。它优化了跟踪系统的所需特性：处理时变数量的源，并根据它们的轨迹对定位估计排序，最小化身份切换（IDS）。通过对多混响运动源的仿真数据和两种模型结构的评估，证明了该算法在不影响帧间定位精度的前提下，有效地减少了身份切换。摘要：Recent data- and learning-based sound source localization (SSL) methods have shown strong performance in challenging acoustic scenarios. However, little work has been done on adapting such methods to track consistently multiple sources appearing and disappearing, as would occur in reality. In this paper, we present a new training strategy for deep learning SSL models with a straightforward implementation based on the mean squared error of the optimal association between estimated and reference positions in the preceding time frames. It optimizes the desired properties of a tracking system: handling a time-varying number of sources and ordering localization estimates according to their trajectories, minimizing identity switches (IDSs). Evaluation on simulated data of multiple reverberant moving sources and on two model architectures proves its effectiveness on reducing identity switches without compromising frame-wise localization accuracy.
【10】 UFO2: A unified pre-training framework for online and offline speech recognition标题：UFO2：一种统一的线上线下语音识别预训练框架链接：https://arxiv.org/abs/2210.14515
* 与cs.SD语音【30】为同一篇
作者：Li Fu,Siqi Li,Qingtao Li,Liping Deng,Fangzhu Li,Lu Fan,Meng Chen,Xiaodong He机构：JD AI Research, Beijing, China备注：Submitted to ICASSP 2023摘要：本文提出了一种在线和离线自动语音识别的统一预训练框架（UFO 2），该框架1）将在线和离线两种独立的训练流程简化为一个过程，2）在有限的话语标注下提高了误词率（WER）性能.具体地说，我们将传统的基于离线模式的自监督学习（SSL）的ASR方法扩展到统一的方式，其中模型训练同时基于全上下文和动态组块输入。为了增强预训练的表示模型，使用停止梯度操作来将在线模式目标解耦到量化器。在预训练阶段和后级微调阶段，引入联合损失，训练出两种模式全权值分担的统一模型。在LibriSpeech数据集上的实验结果表明，UFO 2在离线和在线模式下的相对WER减少量分别为29.7%和18.2%，优于基于SSL的基线方法.摘要：In this paper, we propose a Unified pre-training Framework for Online and Offline (UFO2) Automatic Speech Recognition (ASR), which 1) simplifies the two separate training workflows for online and offline modes into one process, and 2) improves the Word Error Rate (WER) performance with limited utterance annotating. Specifically, we extend the conventional offline-mode Self-Supervised Learning (SSL)-based ASR approach to a unified manner, where the model training is conditioned on both the full-context and dynamic-chunked inputs. To enhance the pre-trained representation model, stop-gradient operation is applied to decouple the online-mode objectives to the quantizer. Moreover, in both the pre-training and the downstream fine-tuning stages, joint losses are proposed to train the unified model with full-weight sharing for the two modes. Experimental results on the LibriSpeech dataset show that UFO2 outperforms the SSL-based baseline method by 29.7% and 18.2% relative WER reduction in offline and online modes, respectively.
【11】 Effect of different splitting criteria on the performance of speech emotion recognition标题：不同分割标准对语音情感识别性能的影响链接：https://arxiv.org/abs/2210.14501
* 与cs.SD语音【31】为同一篇
作者：Bagus Tris Atmaja,Akira Sasou机构：National Institute of Advanced Industrial Science and Technology, Japan备注：None摘要：传统的语音情感识别（SER）评估仅在说话者无关的条件下执行;他们中一些人甚至没有在这个条件下评价他们的结果。本文强调了通过脚本分割训练和测试数据的重要性，即句子开放或文本无关准则。实验结果表明，采用句子开放准则会降低SER的识别性能，这一发现表明，在不同的语言信息嵌入在声学信息中时，从语音中识别情感是困难的。令人惊讶的是，文本无关标准的表现始终比说话人+文本无关标准差。在SER表现上，划分标准的难度从最难到最容易的顺序是：文本无关、说话者+文本无关、说话者无关和说话者+文本相关。说话人+文本无关和文本无关的差异小于其他标准，增加了从不同句子中的语音识别情感的难度。摘要：Traditional speech emotion recognition (SER) evaluations have been performed merely on a speaker-independent condition; some of them even did not evaluate their result on this condition. This paper highlights the importance of splitting training and test data for SER by script, known as sentence-open or text-independent criteria. The results show that employing sentence-open criteria degraded the performance of SER. This finding implies the difficulties of recognizing emotion from speech in different linguistic information embedded in acoustic information. Surprisingly, text-independent criteria consistently performed worse than speaker+text-independent criteria. The full order of difficulties for splitting criteria on SER performances from the most difficult to the easiest is text-independent, speaker+text-independent, speaker-independent, and speaker+text-dependent. The gap between speaker+text-independent and text-independent was smaller than other criteria, strengthening the difficulties of recognizing emotion from speech in different sentences.
【12】 Sinusoidal Frequency Estimation by Gradient Descent标题：基于梯度下降法的正弦频率估计链接：https://arxiv.org/abs/2210.14476
* 与cs.SD语音【32】为同一篇
作者：Ben Hayes,Charalampos Saitis,György Fazekas机构：Centre for Digital Music, Queen Mary University of London备注：Submitted to ICASSP 2023摘要：正弦参数估计是从谱分析到时间序列预测等应用中的一项基本任务。然而，通过梯度下降来估计正弦频率参数通常是不可能的，因为误差函数是非凸的并且密集地填充有局部最小值。因此，不断增长的可微分信号处理方法家族已经不能调谐振荡分量的频率，从而阻碍了它们在广泛应用中的使用。本文提出了一种利用复指数代理函数的Wirtinger导数和任意一阶梯度优化器进行正弦频率和幅值联合估计的技术，使得无约束正弦模型的神经网络控制器的端到端训练成为可能。摘要：Sinusoidal parameter estimation is a fundamental task in applications from spectral analysis to time-series forecasting. Estimating the sinusoidal frequency parameter by gradient descent is, however, often impossible as the error function is non-convex and densely populated with local minima. The growing family of differentiable signal processing methods has therefore been unable to tune the frequency of oscillatory components, preventing their use in a broad range of applications. This work presents a technique for joint sinusoidal frequency and amplitude estimation using the Wirtinger derivatives of a complex exponential surrogate and any first order gradient-based optimizer, enabling end to-end training of neural network controllers for unconstrained sinusoidal models.
【13】 RedPen: Region- and Reason-Annotated Dataset of Unnatural Speech标题：红笔：非自然语言的区域和原因注释数据集链接：https://arxiv.org/abs/2210.14406
* 与cs.SD语音【33】为同一篇
作者：Kyumin Park,Keon Lee,Daeyoung Kim,Dongyeop Kang机构：School of Computing, KAIST, Republic of Korea, KRAFTON Inc., Republic of Korea, University of Minnesota, US备注：Submitted to ICASSP 2023摘要：即使在语音合成模型方面有了最新的进展，对这种模型的评估也纯粹基于作为单个自然度分数的人类判断，例如平均意见分数（MOS）。基于分数的度量没有给出关于哪些词类是不自然的或者人类判断者为什么认为它们是不自然的任何进一步信息。提出了一个新的语音数据集RedPen，它对不自然的语音区域进行了人工标注，并分析了其原因. RedPen由180个合成语音组成，其中有由群组工作者注释的非自然区域;然后，这些区域被推理并按错误类型分类，例如语音抖动和背景噪声。我们发现我们的数据集显示了比模型驱动的不自然度预测更好的解释不自然的语音区域。我们的分析还表明，每个模型包括不同类型的错误类型。总之，我们的数据集成功地显示了不同的错误区域和类型存在于单一自然度得分下的可能性。我们相信，我们的数据集将有助于在未来评估和开发更可解释的语音模型。我们的数据集将在接受后公开。摘要：Even with recent advances in speech synthesis models, the evaluation of such models is based purely on human judgement as a single naturalness score, such as the Mean Opinion Score (MOS). The score-based metric does not give any further information about which parts of speech are unnatural or why human judges believe they are unnatural. We present a novel speech dataset, RedPen, with human annotations on unnatural speech regions and their corresponding reasons. RedPen consists of 180 synthesized speeches with unnatural regions annotated by crowd workers; These regions are then reasoned and categorized by error types, such as voice trembling and background noise. We find that our dataset shows a better explanation for unnatural speech regions than the model-driven unnaturalness prediction. Our analysis also shows that each model includes different types of error types. Summing up, our dataset successfully shows the possibility that various error regions and types lie under the single naturalness score. We believe that our dataset will shed light on the evaluation and development of more interpretable speech models in the future. Our dataset will be publicly available upon acceptance.
【14】 Artificial ASMR: A Cyber-Psychological Study标题：人工ASMR：一种网络心理学研究链接：https://arxiv.org/abs/2210.14321
* 与cs.SD语音【24】为同一篇
作者：Zexiu Wu,Bin Han,C. Clark Cao,Hans. D. Schotten机构：∗Technische Universit¨at Kaiserslautern, †Lingnan University, ‡German Research Center of Artificial Intelligence (DFKI)备注：Submitted to ICASSP 2023摘要：自主感觉经络反应（ASMR）在过去的十年里得到了广泛的应用，但对其的研究还很少，而且还不成熟。针对ASMR音频中常见的声学模式，研究了音频信号的时频特征和周期特征之间的相关性及其在触发ASMR效应中的有效性。采用了一种结合信号处理、人工智能和实验心理学的网络心理学方法，利用该方法，我们能够识别与ASMR相关的声学特征，并由此合成随机人工ASMR音频。摘要：The popularity of Autonomous Sensory Meridian Response (ASMR) has skyrockteted over the past decade, but scientific studies on it are still few and immature. With our attention caught by the common acoustic patterns in ASMR audios, we investigate the correlation between the time-frequency and cyclic features of audio signals and their effectiveness in triggering ASMR effects. A cyber-psychological approach that combines signal processing, artificial intelligence, and experimental psychology is taken, with which we are able to identify ASMR-related acoustic features, and therewith synthesize random artificial ASMR audios.
【15】 Text-to-speech synthesis from dark data with evaluation-in-the-loop data selection标题：利用循环求值数据选择从暗数据合成文本到语音的方法链接：https://arxiv.org/abs/2210.14850
* 与cs.SD语音【1】为同一篇
作者：Kentaro Seki,Shinnosuke Takamichi,Takaaki Saeki,Hiroshi Saruwatari机构：The University of Tokyo, Japan.备注：Submitted to ICASSP 2023摘要：本文提出了一种从暗数据中选择用于文本语音合成（TTS）的训练数据的方法。TTS模型通常是在高质量的语音语料库上训练的，这需要花费大量的时间和金钱来收集数据，这使得增加说话人变化非常具有挑战性。相比之下，有大量的数据，其可用性是未知的（又名，“暗数据”），如YouTube视频。为了利用TTS语料库之外的数据，先前的研究基于声学质量从语料库中选择语音数据。然而，考虑到已经提出了对数据噪声鲁棒的TTS模型，我们应该基于数据作为给定TTS模型的训练数据的重要性来选择数据，而不是语音本身的质量。我们的方法具有训练和评估循环，基于给定TTS模型的合成语音的自动预测质量来选择训练数据。YouTube数据的评测结果表明，该方法优于传统的基于音质的方法。摘要：This paper proposes a method for selecting training data for text-to-speech (TTS) synthesis from dark data. TTS models are typically trained on high-quality speech corpora that cost much time and money for data collection, which makes it very challenging to increase speaker variation. In contrast, there is a large amount of data whose availability is unknown (a.k.a, "dark data"), such as YouTube videos. To utilize data other than TTS corpora, previous studies have selected speech data from the corpora on the basis of acoustic quality. However, considering that TTS models robust to data noise have been proposed, we should select data on the basis of its importance as training data to the given TTS model, not the quality of speech itself. Our method with a loop of training and evaluation selects training data on the basis of the automatically predicted quality of synthetic speech of a given TTS model. Results of evaluations using YouTube data reveal that our method outperforms the conventional acoustic-quality-based method.
【16】 Monotonic segmental attention for automatic speech recognition标题：用于自动语音识别的单调分段注意链接：https://arxiv.org/abs/2210.14742
* 与cs.SD语音【2】为同一篇
作者：Albert Zeyer,Robin Schmitt,Wei Zhou,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：accepted at SLT: this https URL摘要：本文提出了一种新的分段注意模型用于语音识别。我们将解码器的注意力限制在片段上，以避免全局注意力的二次运行时间，更好地推广到长序列，并最终实现流。我们直接比较了全局注意力和不同的分段注意力模型变量。我们开发并比较了两个独立的时间同步解码器，其中一个特别考虑了分段性质，从而产生了进一步的改进。对分段模型使用时间同步解码是新颖的，并且是朝向流应用的一个步骤。我们的实验显示了长度模型对预测片段边界的重要性。与文献中的其他单调注意方法相比，使用分段解码的最终最佳分段注意模型的性能优于全局注意。此外，我们观察到分段模型更好地推广到长达几分钟的长序列。摘要：We introduce a novel segmental-attention model for automatic speech recognition. We restrict the decoder attention to segments to avoid quadratic runtime of global attention, better generalize to long sequences, and eventually enable streaming. We directly compare global-attention and different segmental-attention modeling variants. We develop and compare two separate time-synchronous decoders, one specifically taking the segmental nature into account, yielding further improvements. Using time-synchronous decoding for segmental models is novel and a step towards streaming applications. Our experiments show the importance of a length model to predict the segment boundaries. The final best segmental-attention model using segmental decoding performs better than global-attention, in contrast to other monotonic attention approaches in the literature. Further, we observe that the segmental model generalizes much better to long sequences of up to several minutes.
【17】 Linguistic-Enhanced Transformer with CTC Embedding for Speech Recognition标题：用于语音识别的嵌入CTC的语言增强型转换器链接：https://arxiv.org/abs/2210.14725
* 与cs.SD语音【3】为同一篇
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Mengyuan Zhao,Zhiyong Zhang,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by ECAISS2022, The Fourth International Workshop on Edge Computing and Artificial Intelligence based Sensor-Cloud System摘要：近年来，CTC-Attention联合模型的出现对自动语音识别（ASR）有了很大的改进。改进之处主要在于解码器对语言信息的建模。与声学编码器联合优化的解码器在训练期间以自回归方式从地面实况序列再现语言模型。然而，解码器的训练语料库限于语音转录，其远小于训练可接受的语言模型所需的语料库。这导致解码器的鲁棒性差。针对这一问题，本文提出了一种语言增强的变换器，在训练过程中将细化的CTC信息引入到译码器中，使译码器具有更强的鲁棒性。在AISHELL-1语音语料库上的实验表明，该算法的字符错误率（CER）相对降低了7%。在CTC-Attention ASR联合模型中，解码器对语言信息的敏感度高于对声音信息的敏感度。摘要：The recent emergence of joint CTC-Attention model shows significant improvement in automatic speech recognition (ASR). The improvement largely lies in the modeling of linguistic information by decoder. The decoder joint-optimized with an acoustic encoder renders the language model from ground-truth sequences in an auto-regressive manner during training. However, the training corpus of the decoder is limited to the speech transcriptions, which is far less than the corpus needed to train an acceptable language model. This leads to poor robustness of decoder. To alleviate this problem, we propose linguistic-enhanced transformer, which introduces refined CTC information to decoder during training process, so that the decoder can be more robust. Our experiments on AISHELL-1 speech corpus show that the character error rate (CER) is relatively reduced by up to 7%. We also find that in joint CTC-Attention ASR model, decoder is more sensitive to linguistic information than acoustic information.
【18】 Semi-Supervised Learning Based on Reference Model for Low-resource TTS标题：基于参考模型的半监督学习在低资源TTS中的应用链接：https://arxiv.org/abs/2210.14723
* 与cs.SD语音【4】为同一篇
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by NMIC2022, The Fourth International Workshop on Network Meets Intelligent Computations摘要：以往的神经文本语音转换（TTS）方法主要基于监督学习方法，这意味着它们依赖于大规模的训练数据集，在低资源条件下难以达到与之相当的性能。针对这一问题，提出了一种基于神经TTS的半监督学习方法，该方法在标记目标数据有限的情况下，还可以解决以往自回归模型中存在的曝光偏差问题.具体地说，我们使用大量的源数据对基于Fastspeech2的参考模型进行预训练，并在有限的目标数据集上进行微调。同时，利用原始参考模型生成的伪标签进一步指导微调模型的训练，达到正则化的效果，减少微调模型在有限目标数据上训练时的过拟合。实验结果表明，本文提出的半监督学习方法在目标数据有限的情况下，显著提高了测试数据的语音质量，实现了语音合成的自然性和鲁棒性。摘要：Most previous neural text-to-speech (TTS) methods are mainly based on supervised learning methods, which means they depend on a large training dataset and hard to achieve comparable performance under low-resource conditions. To address this issue, we propose a semi-supervised learning method for neural TTS in which labeled target data is limited, which can also resolve the problem of exposure bias in the previous auto-regressive models. Specifically, we pre-train the reference model based on Fastspeech2 with much source data, fine-tuned on a limited target dataset. Meanwhile, pseudo labels generated by the original reference model are used to guide the fine-tuned model's training further, achieve a regularization effect, and reduce the overfitting of the fine-tuned model during training on the limited target data. Experimental results show that our proposed semi-supervised learning scheme with limited target data significantly improves the voice quality for test data to achieve naturalness and robustness in speech synthesis.
【19】 Pretrained audio neural networks for Speech emotion recognition in Portuguese标题：用于葡萄牙语语音情感识别的预训练音频神经网络链接：https://arxiv.org/abs/2210.14716
* 与cs.SD语音【5】为同一篇
作者：Marcelo Matheus Gauy,Marcelo Finger机构：Universidade de São Paulo, Rua do Matão , São Paulo, Brazil备注：None摘要：语音情感识别（SER）的目标是识别语音的情感方面。针对巴西葡萄牙语语音的SER挑战是利用葡萄牙语的短片段提出的，这些短片段根据副语言元素（笑、哭等）被分类为中性、非中性女性和非中性男性。该数据集包含大约$50$分钟的巴西葡萄牙语语音。由于数据集偏小，我们研究迁移学习和数据扩充技术的组合是否可以产生积极的结果。因此，通过将称为SpecAugment的数据扩增技术与用于迁移学习的预训练音频神经网络（PANN）的使用相结合，我们能够获得有趣的结果。PANN（CNN 6、CNN 10和CNN 14）在包含超过$5000$小时的音频的称为AudioSet的大数据集上被预训练。在SER数据集上对它们进行微调，并将验证集上性能最佳的模型（CNN 10）提交给挑战，从挑战提供的基线的$0.54$提高到$0.73$的$F1$分数。此外，我们还测试了Transformer神经架构的使用，该架构在大约600 $小时的巴西葡萄牙语音频数据上进行了预训练。变压器以及更复杂的PANN模型（CNN 14）未能推广到SER数据集中的测试集，也没有超过基线。考虑到数据集大小的限制，目前用于SER的最佳方法是使用PANN（具体地，CNN 6和CNN 10）。摘要：The goal of speech emotion recognition (SER) is to identify the emotional aspects of speech. The SER challenge for Brazilian Portuguese speech was proposed with short snippets of Portuguese which are classified as neutral, non-neutral female and non-neutral male according to paralinguistic elements (laughing, crying, etc). This dataset contains about $50$ minutes of Brazilian Portuguese speech. As the dataset leans on the small side, we investigate whether a combination of transfer learning and data augmentation techniques can produce positive results. Thus, by combining a data augmentation technique called SpecAugment, with the use of Pretrained Audio Neural Networks (PANNs) for transfer learning we are able to obtain interesting results. The PANNs (CNN6, CNN10 and CNN14) are pretrained on a large dataset called AudioSet containing more than $5000$ hours of audio. They were finetuned on the SER dataset and the best performing model (CNN10) on the validation set was submitted to the challenge, achieving an $F1$ score of $0.73$ up from $0.54$ from the baselines provided by the challenge. Moreover, we also tested the use of Transformer neural architecture, pretrained on about $600$ hours of Brazilian Portuguese audio data. Transformers, as well as more complex models of PANNs (CNN14), fail to generalize to the test set in the SER dataset and do not beat the baseline. Considering the limitation of the dataset sizes, currently the best approach for SER is using PANNs (specifically, CNN6 and CNN10).
【20】 Pronunciation Generation for Foreign Language Words in Intra-Sentential Code-Switching Speech Recognition标题：句内码转换语音识别中外语单词的发音生成链接：https://arxiv.org/abs/2210.14691
* 与cs.SD语音【6】为同一篇
作者：Wei Wang,Chao Zhang,Xiaopei Wu机构：Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei , China, A R T I C L E I N F O摘要：语码转换是指在一个句子或语篇中转换语言的现象。然而，有限的语码转换、不同语言的音素集以及高昂的重建成本给语码转换语音识别的声学模型的建立带来了挑战。本文以有限的语码转换数据为驱动材料，探索了一条在委托的母语声学模型上快速培养句内语码转换识别技能的捷径，提出了一种数据驱动的方法来制作种子词典，用于训练字形-音素模型来预测语码转换句子中外语词的映射发音。本文的数据驱动技术的核心工作包括语音解码方法和不同的选择方法。而对于词汇层面驱动材料不平衡的问题，我们有一个内在的辅助启示，那就是利用字形-音素模型，在拥有充足材料的词汇中学习好的发音规则，以帮助稀缺的词汇。实验结果表明，在纯汉语识别器上获得的汉英语码转换识别混合错误率为29.15%，通过数据驱动的方法加入外语单词的发音后，混合错误率为12.13%，最后通过不同的选择方法和内部辅助策略的组合，混合错误率达到了最好的11.14%。摘要：Code-Switching refers to the phenomenon of switching languages within a sentence or discourse. However, limited code-switching , different language phoneme-sets and high rebuilding costs throw a challenge to make the specialized acoustic model for code-switching speech recognition. In this paper, we make use of limited code-switching data as driving materials and explore a shortcut to quickly develop intra-sentential code-switching recognition skill on the commissioned native language acoustic model, where we propose a data-driven method to make the seed lexicon which is used to train grapheme-to-phoneme model to predict mapping pronunciations for foreign language word in code-switching sentences. The core work of the data-driven technology in this paper consists of a phonetic decoding method and different selection methods. And for imbalanced word-level driving materials problem, we have an internal assistance inspiration that learning the good pronunciation rules in the words that possess sufficient materials using the grapheme-to-phoneme model to help the scarce. Our experiments show that the Mixed Error Rate in intra-sentential Chinese-English code-switching recognition reduced from 29.15\%, acquired on the pure Chinese recognizer, to 12.13\% by adding foreign language words' pronunciation through our data-driven approach, and finally get the best result 11.14\% with the combination of different selection methods and internal assistance tactic.
【21】 In search of strong embedding extractors for speaker diarisation标题：寻找用于说话人腹泻的强嵌入抽取器链接：https://arxiv.org/abs/2210.14682
* 与cs.SD语音【7】为同一篇
作者：Jee-weon Jung,Hee-Soo Heo,Bong-Jin Lee,Jaesung Huh,Andrew Brown,Youngki Kwon,Shinji Watanabe,Joon Son Chung机构：Naver Corporation, South Korea, Visual Geometry Group, Department of Engineering Science, University of Oxford, UK, Carnegie Mellon University, Pittsburgh, PA, USA, Korea Advanced Institute of Science and Technology, South Korea备注：5pages, 1 figure, 2 tables, submitted to ICASSP摘要：说话人嵌入提取器（EE）将输入音频映射到说话人判别潜空间，在说话人识别中具有重要作用。然而，在采用EE进行日记化时存在几个挑战，我们从这两个方面解决了两个关键问题。首先，评估不是直接的，因为更好的性能所需的特征在说话者确认和日记化之间不同。我们表明，在广泛采用的说话人确认评估协议上的更好的性能并不导致更好的对话性能。第二，嵌入提取器没有看到其中存在多个说话者的话语。这些输入不可避免地出现在说话人日记中，因为重叠的语音和说话人的变化;它们降低了性能。为了减轻第一个问题，我们生成更好地模拟日记化场景的说话者验证评估协议。我们提出了两种数据增强技术来缓解第二个问题，使嵌入提取器意识到重叠语音或说话人改变输入。一种技术生成重叠的语音段，而另一种技术生成两个说话者顺序说出的段。实验结果表明，两种方法都是有效的。摘要：Speaker embedding extractors (EEs), which map input audio to a speaker discriminant latent space, are of paramount importance in speaker diarisation. However, there are several challenges when adopting EEs for diarisation, from which we tackle two key problems. First, the evaluation is not straightforward because the features required for better performance differ between speaker verification and diarisation. We show that better performance on widely adopted speaker verification evaluation protocols does not lead to better diarisation performance. Second, embedding extractors have not seen utterances in which multiple speakers exist. These inputs are inevitably present in speaker diarisation because of overlapped speech and speaker changes; they degrade the performance. To mitigate the first problem, we generate speaker verification evaluation protocols that mimic the diarisation scenario better. We propose two data augmentation techniques to alleviate the second problem, making embedding extractors aware of overlapped speech or speaker change input. One technique generates overlapped speech segments, and the other generates segments where two speakers utter sequentially. Extensive experimental results using three state-of-the-art speaker embedding extractors demonstrate that both proposed approaches are effective.
【22】 Full-band General Audio Synthesis with Score-based Diffusion标题：基于分数扩散的全频段通用音频合成链接：https://arxiv.org/abs/2210.14661
* 与cs.SD语音【8】为同一篇
作者：Santiago Pascual,Gautam Bhattacharya,Chunghsin Yeh,Jordi Pons,Joan Serrà机构：Dolby Laboratories摘要：最近的研究表明，深度生成模型能够处理来自单个标签的一般音频合成，产生各种脉冲、音调和环境声音。这样的模型对带限信号进行操作，并且作为自回归方法的结果，它们通常由预先训练的潜在编码器和/或若干级联模块来符合。本文提出了一种基于扩散的通用音频合成生成模型DAG，该模型在波形域端到端处理全频带信号。实验结果表明，与现有的标签条件生成器相比，DAG在质量和多样性方面都具有优势。更具体地，当与现有技术相比时，DAG的带限和全带版本实现了分别高达40%和65%的相对改进。我们相信DAG是足够灵活的，以适应不同的条件模式，同时提供高质量的合成。摘要：Recent works have shown the capability of deep generative models to tackle general audio synthesis from a single label, producing a variety of impulsive, tonal, and environmental sounds. Such models operate on band-limited signals and, as a result of an autoregressive approach, they are typically conformed by pre-trained latent encoders and/or several cascaded modules. In this work, we propose a diffusion-based generative model for general audio synthesis, named DAG, which deals with full-band signals end-to-end in the waveform domain. Results show the superiority of DAG over existing label-conditioned generators in terms of both quality and diversity. More specifically, when compared to the state of the art, the band-limited and full-band versions of DAG achieve relative improvements that go up to 40 and 65%, respectively. We believe DAG is flexible enough to accommodate different conditioning schemas while providing good quality synthesis.
【23】 TSUP Speaker Diarization System for Conversational Short-phrase Speaker Diarization Challenge标题：用于会话短短语说话人二值化挑战的TUP说话人二值化系统链接：https://arxiv.org/abs/2210.14653
* 与cs.SD语音【9】为同一篇
作者：Bowen Pang,Huan Zhao,Gaosheng Zhang,Xiaoyue Yang,Yang Sun,Li Zhang,Qing Wang,Lei Xie机构：Northwestern Polytechnical University (NPU), Xi’an, China, Shenzhen Transsion Holding Limited摘要：本文介绍了TSUP团队对ISCSLP 2022会话短语说话人对话化（CSSD）挑战的提交，该挑战特别关注短语会话，并采用了一种新的评估指标--会话对话化错误率（CDER）。在这一挑战中，我们研究了三种典型的说话人识别系统，分别是基于谱聚类的识别系统、目标说话人语音活动检测系统和端到端神经网络识别系统。我们的主要发现总结如下。首先，在新的CDER度量下，SC方法比其他两种方法更受欢迎。第二，对于所有三种类型的扬声器诊断系统，超参数的调谐对于CDER是必不可少的。具体地，当子段的长度设置得更长时，CDER变得更小。最后，通过DOVER-LAP的多系统融合将恶化挑战数据上的CDER度量。我们提交的SC系统最终在挑战中排名第三。摘要：This paper describes the TSUP team's submission to the ISCSLP 2022 conversational short-phrase speaker diarization (CSSD) challenge which particularly focuses on short-phrase conversations with a new evaluation metric called conversational diarization error rate (CDER). In this challenge, we explore three kinds of typical speaker diarization systems, which are spectral clustering(SC) based diarization, target-speaker voice activity detection(TS-VAD) and end-to-end neural diarization(EEND) respectively. Our major findings are summarized as follows. First, the SC approach is more favored over the other two approaches under the new CDER metric. Second, tuning on hyperparameters is essential to CDER for all three types of speaker diarization systems. Specifically, CDER becomes smaller when the length of sub-segments setting longer. Finally, multi-system fusion through DOVER-LAP will worsen the CDER metric on the challenge data. Our submitted SC system eventually ranks the third place in the challenge.
【24】 Speaker Diarization Based on Multi-channel Microphone Array in Small-scale Meeting标题：小型会议中基于多通道麦克风阵列的说话人二值化链接：https://arxiv.org/abs/2210.14644
* 与cs.SD语音【10】为同一篇
作者：Yuxuan Du,Ruohua Zhou摘要：在发言人日记化的任务中，小规模会议的数量占了很大的比重。当麦克风阵列被用作录音设备时，其空间信息通常被大多数研究者所忽略。本文借鉴d-向量和麦克风阵列空间向量相结合的聚类方法，针对4人以内的会议，提出了一种利用多通道麦克风阵列的二值化方法。我们利用语音增强技术对麦克风阵列输出的音频信号进行预处理。采用受控响应功率相位变换（SRP-PHAT）算法对语音片段进行分类，得到更准确的说话人，并根据说话人数目对语音片段进行聚类，以获得更好的性能。最后，利用DOVER-LAP算法对系统进行融合，得到最佳融合结果。我们在AMI语料库上对我们的系统进行了评估。与目前最好的实验结果相比，本系统在图像处理的误码率（DER）方面有了很大的改善。摘要：In the task of speaker diarization, the number of small-scale meetings accounts for a large proportion. When microphone arrays are employed as a recording device, its spatial information is usually ignored by most researchers. In this paper, inspired by the clustering method combining d-vector and microphone array spatial vector, we proposed a diarization method which using multi-channel microphone arrays for a meeting with no more than 4 speakers. We utilize speech enhancement to preprocess the audio from the microphone array. The Steered-Response Power Phase Transform (SRP-PHAT) algorithm are employed to get more accurate speakers, and apply the number of speakers to recluster the speech segments to achieve better performance. Finally, we fuse our system by DOVER-LAP to get the best result. We evaluated our system on the AMI corpus. Compared with the best experimental results so far, our system has achieved largely improvement in the diarization error rate (DER).
【25】 Fast Yet Effective Speech Emotion Recognition with Self-distillation标题：一种快速有效的自提炼语音情感识别链接：https://arxiv.org/abs/2210.14636
* 与cs.SD语音【11】为同一篇
作者：Zhao Ren,Thanh Tam Nguyen,Yi Chang,Björn W. Schuller机构：L,S Research Center, Leibniz University Hannover, Germany, Griffith University, Australia, GLAM – Group on Language, Audio, & Music, Imperial College London, United Kingdom备注：Submitted to ICASSP 2023摘要：语音情感识别是从语音中识别人的情感状态的任务。SER在帮助对话系统真正理解我们的情绪并成为值得信赖的人类对话伙伴方面非常普遍。由于语音的冗长特性，SER还受到缺乏用于诸如深度神经网络之类的强大模型的丰富标记数据的困扰。基于大规模语音数据集的预训练复杂模型已经通过迁移学习成功地应用于SER。然而，对复杂模型进行微调仍然需要较大的存储空间，并且导致推理效率较低。在这篇文章中，我们认为通过自蒸馏可以实现快速而有效的SER，自蒸馏是一种同时微调预训练模型和训练其较浅版本的方法。我们的自我提炼框架有三方面的好处：（1）对声学模态采用自提取方法，突破了语音数据的有限真实性，在SER数据集上的性能优于现有模型;（2）在不同深度执行强大的模型可以在资源有限的边缘设备上实现自适应的准确性-效率折衷;（3）新的微调过程而不是从头开始训练以进行自蒸馏，导致了更快的学习时间和对具有少量标签信息的数据的现有技术水平的准确性。摘要：Speech emotion recognition (SER) is the task of recognising human's emotional states from speech. SER is extremely prevalent in helping dialogue systems to truly understand our emotions and become a trustworthy human conversational partner. Due to the lengthy nature of speech, SER also suffers from the lack of abundant labelled data for powerful models like deep neural networks. Pre-trained complex models on large-scale speech datasets have been successfully applied to SER via transfer learning. However, fine-tuning complex models still requires large memory space and results in low inference efficiency. In this paper, we argue achieving a fast yet effective SER is possible with self-distillation, a method of simultaneously fine-tuning a pretrained model and training shallower versions of itself. The benefits of our self-distillation framework are threefold: (1) the adoption of self-distillation method upon the acoustic modality breaks through the limited ground-truth of speech data, and outperforms the existing models' performance on an SER dataset; (2) executing powerful models at different depth can achieve adaptive accuracy-efficiency trade-offs on resource-limited edge devices; (3) a new fine-tuning process rather than training from scratch for self-distillation leads to faster learning time and the state-of-the-art accuracy on data with small quantities of label information.
【26】 Audio Mosaicing with Simulation-based Inference标题：基于仿真推理的音频拼接链接：https://arxiv.org/abs/2210.14602
* 与cs.SD语音【12】为同一篇
作者：Andrew Gambardella,Youngjun Choi,Doyo Choi,Jinjoon Lee机构：Graduate School of Culture Technology, KAIST摘要：几十年来，马赛克和拼贴画一直是艺术不可或缺的一部分。在当代媒体艺术中特别重要的是音频马赛克，其中艺术家手动组合几个音频源以便构建一个单一的连贯声音，组合来自不同源的元素。在这里，我们提出了一个算法，以自动创建音频马赛克使用模拟为基础的推理范例。我们的算法将希望近似的音频文件和可以用于近似的音频文件列表作为输入，以可解释和解开的方式使用源，找到可以从中采样原始音频文件的重建的后验分布。我们通过创建一个音频马赛克来验证我们的方法，该马赛克使用100首韩国流行歌曲重新排列和重叠来重建传统韩国葬礼的声音。摘要：Mosaics and collages have been an integral part of art for decades. Particularly important in contemporary media art is the audio mosaic, in which an artist manually combines several audio sources in order to construct one single coherent sound, combining elements from disparate sources. Here we propose an algorithm to automatically create audio mosaics using the simulation-based inference paradigm. Our algorithm takes as input an audio file that one wishes to approximate, and a list of audio files one can use for approximation, finding a posterior distribution from which one can sample reconstructions of the original audio file, using the sources in an interpretable and disentangled manner. We validate our approach by creating an audio mosaic which reconstructs the sound of a traditional Korean funeral using 100 K-pop songs rearranged and overlapped.
【27】 Improving Speech-to-Speech Translation Through Unlabeled Text标题：通过无标记文本改进语音到语音的翻译链接：https://arxiv.org/abs/2210.14514
* 与cs.SD语音【13】为同一篇
作者：Xuan-Phi Nguyen,Sravya Popuri,Changhan Wang,Yun Tang,Ilia Kulikov,Hongyu Gong机构：⋆ Meta AI, USA, † Nanyang Technological University, Singapore摘要：由于直接语音到语音翻译（S2 ST）数据的严重匮乏，S2 ST是翻译范式中最具挑战性的问题之一。虽然已经努力通过级联预训练语音识别（ASR）、机器翻译（MT）和文本到语音（TTS）模型来增加来自未标记语音的数据量;未标记的文本在改进S2 ST方面仍然相对未被充分利用。提出了一种有效的方法，利用现有的大量不同语言的未标记文本来创建大量的S2 ST数据，通过对生成的合成数据应用各种声学效果来提高S2 ST性能。根据经验，我们的方法在西班牙语-英语翻译方面比现有技术的水平高出2 BLEU。实验结果表明，该方法在资源非常少的情况下，对西班牙语-英语和俄语-英语的翻译都能取得显著的效果。摘要：Direct speech-to-speech translation (S2ST) is among the most challenging problems in the translation paradigm due to the significant scarcity of S2ST data. While effort has been made to increase the data size from unlabeled speech by cascading pretrained speech recognition (ASR), machine translation (MT) and text-to-speech (TTS) models; unlabeled text has remained relatively under-utilized to improve S2ST. We propose an effective way to utilize the massive existing unlabeled text from different languages to create a large amount of S2ST data to improve S2ST performance by applying various acoustic effects to the generated synthetic data. Empirically our method outperforms the state of the art in Spanish-English translation by up to 2 BLEU. Significant gains by the proposed method are demonstrated in extremely low-resource settings for both Spanish-English and Russian-English translations.
【28】 Parallel Gated Neural Network With Attention Mechanisim For Speech Enhancement标题：具有注意机制的并行门控神经网络语音增强链接：https://arxiv.org/abs/2210.14509
* 与cs.SD语音【14】为同一篇
作者：Jianqiao Cui,Stefan Bleeck机构：Institute of Sound and Vibration Research, University of Southampton, UK备注：5 pages, 6 figures, references added摘要：深度学习算法越来越多地用于语音增强（SE）。在有监督的方法中，需要全局和局部信息来进行精确的谱映射。关键限制通常是对关键上下文信息的捕获不足。为了充分利用目标说话人的长时变特性，补偿干净语音的失真，采用序列到序列（S2S）映射结构，提出了一种新的单耳语音增强系统，该系统由特征提取模块（FEB）、补偿增强模块（ComEB）和掩蔽模块（MB）组成。在FEB中，U网模块利用复值谱提取抽象特征，单路利用掩蔽方法抑制幅值域的背景噪声，MB从FEB中提取幅值特征，并补偿ComEB中丢失的复值域特征，从而恢复出最终的干净语音。在Librispeech数据集上的实验结果表明，该模型在ESTOI和PESQ评分方面均优于现有模型.摘要：Deep learning algorithm are increasingly used for speech enhancement (SE). In supervised methods, global and local information is required for accurate spectral mapping. A key restriction is often poor capture of key contextual information. To leverage long-term for target speakers and compensate distortions of cleaned speech, this paper adopts a sequence-to-sequence (S2S) mapping structure and proposes a novel monaural speech enhancement system, consisting of a Feature Extraction Block (FEB), a Compensation Enhancement Block (ComEB) and a Mask Block (MB). In the FEB a U-net block is used to extract abstract features using complex-valued spectra with one path to suppress the background noise in the magnitude domain using masking methods and the MB takes magnitude features from the FEBand compensates the lost complex-domain features produced from ComEB to restore the final cleaned speech. Experiments are conducted on the Librispeech dataset and results show that the proposed model obtains better performance than recent models in terms of ESTOI and PESQ scores.
【29】 Two-stage dimensional emotion recognition by fusing predictions of acoustic and text networks using SVM标题：基于支持向量机的声学和文本网络预测融合的两阶段情感识别链接：https://arxiv.org/abs/2210.14495
* 与cs.SD语音【15】为同一篇
作者：Bagus Tris Atmaja,Masato Akagi机构：Japan Advanced Institute of Science and Technology,-, Asahidai, Nomi, Ishikawa, -, Japan, Sepuluh Nopember Insitute of Technology, Sukolilo, Surabaya , Indonesia备注：None摘要：计算机自动语音情感识别（SER）是实现更自然的人机交互的关键。在人与人的互动中，正确感知情绪的能力对于在特定情况下采取进一步的措施至关重要。SER中的一个问题是是否有必要将声学特征与诸如面部表情、文本和运动捕捉等其他数据相结合。本研究提出一种结合声音与文字信息的后融合方法，包含两个步骤。首先，在深度学习系统中分别训练声学和文本特征。其次，将深度学习系统的预测结果输入支持向量机（SVM），以预测最终的回归分数。此外，本研究的任务是维度情感建模，因为它可以更深入地分析情感状态。实验结果表明，该两阶段后融合方法的性能优于任何一阶段处理方法，且从一阶段处理到两阶段处理具有线性相关性。这种后期融合方法改善了先前在一致性相关系数得分中测量的早期融合结果。摘要：Automatic speech emotion recognition (SER) by a computer is a critical component for more natural human-machine interaction. As in human-human interaction, the capability to perceive emotion correctly is essential to take further steps in a particular situation. One issue in SER is whether it is necessary to combine acoustic features with other data such as facial expressions, text, and motion capture. This research proposes to combine acoustic and text information by applying a late-fusion approach consisting of two steps. First, acoustic and text features are trained separately in deep learning systems. Second, the prediction results from the deep learning systems are fed into a support vector machine (SVM) to predict the final regression score. Furthermore, the task in this research is dimensional emotion modeling because it can enable a deeper analysis of affective states. Experimental results show that this two-stage, late-fusion approach, obtains higher performance than that of any one-stage processing, with a linear correlation from one-stage to two-stage processing. This late-fusion approach improves previous early fusion results measured in concordance correlation coefficients score.
【30】 AVES: Animal Vocalization Encoder based on Self-Supervision标题：AVES：基于自我监控的动物发声编码器链接：https://arxiv.org/abs/2210.14493
* 与cs.SD语音【16】为同一篇
作者：Masato Hagiwara机构：Earth Species Project摘要：在生物声学中缺乏注释的训练数据阻碍了以监督方式训练的大规模神经网络模型的使用。为了充分利用大量未标注的音频数据，提出了一种基于自监督变换器的动物发声编码模型AVES（Animal Vocalization Encoder based on Self-Supervisory）.我们在一组不同的未注释音频数据集上对AVES进行预训练，并针对下游的生物声学任务对其进行微调。一系列分类和检测任务的综合实验表明，AVES优于所有强基线，甚至优于在注释音频分类数据集上训练的监督“顶线”模型。研究结果也表明，挑选一个与下游任务相关的小训练子集是训练高质量音频表示模型的有效方法。我们在以下网址开放了模型的源代码：摘要：The lack of annotated training data in bioacoustics hinders the use of large-scale neural network models trained in a supervised way. In order to leverage a large amount of unannotated audio data, we propose AVES (Animal Vocalization Encoder based on Self-Supervision), a self-supervised, transformer-based audio representation model for encoding animal vocalizations. We pretrain AVES on a diverse set of unannotated audio datasets and fine-tune them for downstream bioacoustics tasks. Comprehensive experiments with a suite of classification and detection tasks have shown that AVES outperforms all the strong baselines and even the supervised "topline" models trained on annotated audio classification datasets. The results also suggest that curating a small training subset related to downstream tasks is an efficient way to train high-quality audio representation models. We open-source our models at \url{https://github.com/earthspecies/aves}.
【31】 SCP-GAN: Self-Correcting Discriminator Optimization for Training Consistency Preserving Metric GAN on Speech Enhancement Tasks标题：SCP-GAN：语音增强任务中训练一致性保持度量GAN的自校正判别器优化链接：https://arxiv.org/abs/2210.14474
* 与cs.SD语音【17】为同一篇
作者：Vasily Zadorozhnyy,Qiang Ye,Kazuhito Koishida机构：Department of Mathematics, University of Kentucky, Lexington, USA, Applied Sciences Group, Microsoft Corporation, Redmond, USA备注：5 pages (4 - manuscript and 1 - references), 2 figures, 2 tables摘要：近年来，生成式对抗网络（GANs）在语音增强（SE）任务中取得了显著的改进效果。然而，他们很难训练。在本文中，我们对GAN的训练方案进行了一些改进，这些改进可以应用于大多数基于GAN的SE模型。我们提出使用一致性损失函数，其针对由傅立叶变换和傅立叶逆变换引起的时间域和时频域中的不一致性。我们还提出了在SE任务上训练GAN鉴别器的自校正优化，这有助于避免鉴别器损失函数部分的“有害”训练方向。我们已经在几个最先进的基于GAN的SE模型上测试了我们提出的方法，并获得了一致的改进，包括对Voice Bank+DEMAND数据集的新的最先进的结果。摘要：In recent years, Generative Adversarial Networks (GANs) have produced significantly improved results in speech enhancement (SE) tasks. They are difficult to train, however. In this work, we introduce several improvements to the GAN training schemes, which can be applied to most GAN-based SE models. We propose using consistency loss functions, which target the inconsistency in time and time-frequency domains caused by Fourier and Inverse Fourier Transforms. We also present self-correcting optimization for training a GAN discriminator on SE tasks, which helps avoid "harmful" training directions for parts of the discriminator loss function. We have tested our proposed methods on several state-of-the-art GAN-based SE models and obtained consistent improvements, including new state-of-the-art results for the Voice Bank+DEMAND dataset.
【32】 The NPU-ASLP System for The ISCSLP 2022 Magichub Code-Swiching ASR Challenge标题：用于ISCSLP 2022 MagicHub代码交换ASR挑战赛的NPU-ASLP系统链接：https://arxiv.org/abs/2210.14448
* 与cs.SD语音【18】为同一篇
作者：Yuhao Liang,Peikun Chen,Fan Yu,Xinfa Zhu,Tianyi Xu,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China备注：Submitted to ISCSLP 2022摘要：None摘要：This paper describes our NPU-ASLP system submitted to the ISCSLP 2022 Magichub Code-Switching ASR Challenge. In this challenge, we first explore several popular end-to-end ASR architectures and training strategies, including bi-encoder, language-aware encoder (LAE) and mixture of experts (MoE). To improve our system's language modeling ability, we further attempt the internal language model as well as the long context language model. Given the limited training data in the challenge, we further investigate the effects of data augmentation, including speed perturbation, pitch shifting, speech codec, SpecAugment and synthetic data from text-to-speech (TTS). Finally, we explore ROVER-based score fusion to make full use of complementary hypotheses from different models. Our submitted system achieves 16.87% on mix error rate (MER) on the test set and comes to the 2nd place in the challenge ranking.
【33】 Smart Speech Segmentation using Acousto-Linguistic Features with look-ahead标题：基于声学-语言特征的智能语音分割链接：https://arxiv.org/abs/2210.14446
* 与cs.SD语音【19】为同一篇
作者：Piyush Behre,Naveen Parihar,Sharman Tan,Amy Shah,Eva Sharma,Geoffrey Liu,Shuangyu Chang,Hosam Khalil,Chris Basoglu,Sayan Pathak机构：Microsoft Corporation摘要：None摘要：Segmentation for continuous Automatic Speech Recognition (ASR) has traditionally used silence timeouts or voice activity detectors (VADs), which are both limited to acoustic features. This segmentation is often overly aggressive, given that people naturally pause to think as they speak. Consequently, segmentation happens mid-sentence, hindering both punctuation and downstream tasks like machine translation for which high-quality segmentation is critical. Model-based segmentation methods that leverage acoustic features are powerful, but without an understanding of the language itself, these approaches are limited. We present a hybrid approach that leverages both acoustic and language information to improve segmentation. Furthermore, we show that including one word as a look-ahead boosts segmentation quality. On average, our models improve segmentation-F0.5 score by 9.8% over baseline. We show that this approach works for multiple languages. For the downstream task of machine translation, it improves the translation BLEU score by an average of 1.05 points.
【34】 Dynamic Speech Endpoint Detection with Regression Targets标题：基于回归目标的动态语音端点检测链接：https://arxiv.org/abs/2210.14252
* 与cs.SD语音【20】为同一篇
作者：Dawei Liang,Hang Su,Tarun Singh,Jay Mahadeokar,Shanil Puri,Jiedan Zhu,Edison Thomaz,Mike Seltzer机构：⋆ University of Texas at Austin, † Meta AI.备注：Manuscript submitted to ICASSP 2023摘要：交互式语音助手已被广泛用作各种场景中的输入接口，例如在智能家居设备、可穿戴设备和AR设备上。检测语音查询的结束，即语音端点指示，是语音助手与用户交互的重要任务。传统上，语音端点定位是基于纯分类方法和任意二进制目标。本文提出了一种新的基于回归的语音端点检测模型，该模型使得端点检测器能够根据用户查询的上下文调整其检测行为。具体地说，我们提出了一种暂停建模方法，并证明了其对动态端点定位的有效性。基于我们对供应商收集的智能手机和可穿戴设备语音查询的实验，与传统的基于分类的方法相比，我们的策略在端点延迟和准确性之间表现出更好的平衡。本文进一步讨论了该模型的优点和该框架的推广。摘要：Interactive voice assistants have been widely used as input interfaces in various scenarios, e.g. on smart homes devices, wearables and on AR devices. Detecting the end of a speech query, i.e. speech end-pointing, is an important task for voice assistants to interact with users. Traditionally, speech end-pointing is based on pure classification methods along with arbitrary binary targets. In this paper, we propose a novel regression-based speech end-pointing model, which enables an end-pointer to adjust its detection behavior based on context of user queries. Specifically, we present a pause modeling method and show its effectiveness for dynamic end-pointing. Based on our experiments with vendor-collected smartphone and wearables speech queries, our strategy shows a better trade-off between endpointing latency and accuracy, compared to the traditional classification-based method. We further discuss the benefits of this model and generalization of the framework in the paper.
机器翻译，仅供参考
每日语音速递