今日论文合集：cs.SD语音10篇，eess.AS音频处理14篇。【1】 ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representations
标题：ParrotTTS：基于自监督表示的文语合成
链接：http://arxiv.org/pdf/2303.01261v1
作者：Saiteja Kosgi,Neil Kumar Shah,Vishal Tambrahalli,Neha Sherin,Vineet Gandhi机构：Kohli Centre on Intelligent Systems, IIIT Hyderabad, TCS Research, Pune摘要：自统计TTS时代以来，TTS系统被建模为梅尔合成器，随后是语音声码器，并被推进到神经设计中。我们提出了一种替代TTS建模的方法，称为ParrotTTS，借用自监督学习（SSL）方法。ParrotTTS采用两步法，首先在大量可用的未标记数据上训练语音到语音模型，然后是文本到嵌入模型，利用对齐转录的语音将其扩展到TTS。与传统TTS模型相比，ParrotTTS在自然度上获得了有竞争力的平均意见得分，但显著提高了后者转录对的数据效率和无转录的说话人自适应。这进一步为在一般训练的SSL语音模型上训练TTS模型铺平了道路。摘要：Text-to-speech (TTS) systems are modelled as mel-synthesizers followed by speech-vocoders since the era of statistical TTS that is carried forward into neural designs. We propose an alternative approach to TTS modelling referred to as ParrotTTS borrowing from self-supervised learning (SSL) methods. ParrotTTS takes a two-step approach by initially training a speech-to-speech model on unlabelled data that is abundantly available, followed by a text-to-embedding model that leverages speech with aligned transcriptions to extend it to TTS. ParrotTTS achieves competitive mean opinion scores on naturalness compared to traditional TTS models but significantly improves over the latter's data efficiency of transcribed pairs and speaker adaptation without transcriptions. This further paves the path to training TTS models on generically trained SSL speech models.【2】 Language-Universal Adapter Learning with Knowledge Distillation for End-to-End Multilingual Speech Recognition
标题：用于端到端多语言语音识别的具有知识提取的语言通用适配器学习
链接：http://arxiv.org/pdf/2303.01249v1
作者：Zhijie Shen,Wu Guo,Bin Gu机构：The Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Hefei, China摘要：本文提出了一种基于预训练模型的端到端多语言自动语音识别（ASR）的语言通用适配器学习框架。对于声学建模，wav2vec 2.0预训练模型通过插入特定语言和通用语言适配器进行了微调。然后使用在线知识提炼来使语言通用适配器能够学习语言专用特征和通用特征。通过利用语言标识符（LID）也减少了语言信息混淆。使用LID，我们对多个头部注意力输出执行位置式修改。在推理过程中，语言专用适配器被移除，而语言通用适配器保持激活。该方法提高了识别精度，解决了常见多语种ASR系统中适配器参数数量随语种数量线性增加的问题。在BABEL数据集上的实验验证了该框架的有效性。与传统的多语言模型相比，绝对错误率降低了3.3%。该代码可从以下网址获得：https: github.com shen9712 UniversalAdapterLearning。摘要：In this paper, we propose a language-universal adapter learning framework based on a pre-trained model for end-to-end multilingual automatic speech recognition (ASR). For acoustic modeling, the wav2vec 2.0 pre-trained model is fine-tuned by inserting language-specific and language-universal adapters. An online knowledge distillation is then used to enable the language-universal adapters to learn both language-specific and universal features. The linguistic information confusion is also reduced by leveraging language identifiers (LIDs). With LIDs we perform a position-wise modification on the multi-head attention outputs. In the inference procedure, the language-specific adapters are removed while the language-universal adapters are kept activated. The proposed method improves the recognition accuracy and addresses the linear increase of the number of adapters' parameters with the number of languages in common multilingual ASR systems. Experiments on the BABEL dataset confirm the effectiveness of the proposed framework. Compared to the conventional multilingual model, a 3.3% absolute error rate reduction is achieved. The code is available at: https: github.com shen9712 UniversalAdapterLearning.【3】 Learning From Yourself: A Self-Distillation Method for Fake Speech Detection
标题：向自己学习：一种用于虚假语音检测的自蒸馏方法
链接：http://arxiv.org/pdf/2303.01211v1
作者：Jun Xue,Cunhang Fan,Jiangyan Yi,Chenglong Wang,Zhengqi Wen,Dan Zhang,Zhao Lv机构：Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei , China, NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing , China, Qiyuan Laboratory, Beijing , China备注：Accepted by ICASSP 2023摘要：本文提出了一种新的自提取伪语音检测方法，该方法能够在不增加模型复杂度的前提下显著提高伪语音检测的性能。对于FSD来说，一些细粒度的信息是非常重要的，如谱图缺陷、哑段等，这些信息往往被浅网络感知。然而，浅网络有很多噪声，不能很好地捕捉这一点。针对这一问题，提出用最深网络指导浅网络来增强浅网络。具体地，FSD的网络被分成若干段，最深的网络被用作教师模型，并且所有浅的网络通过添加分类器而成为多个学生模型。同时，利用最深网络特征和浅网络特征之间的蒸馏路径来减小特征差异。在ASVspoof 2019 LA和PA数据集上的一系列实验结果表明了所提方法的有效性，与基线相比有显著的改进.摘要：In this paper, we propose a novel self-distillation method for fake speech detection (FSD), which can significantly improve the performance of FSD without increasing the model complexity. For FSD, some fine-grained information is very important, such as spectrogram defects, mute segments, and so on, which are often perceived by shallow networks. However, shallow networks have much noise, which can not capture this very well. To address this problem, we propose using the deepest network instruct shallow network for enhancing shallow networks. Specifically, the networks of FSD are divided into several segments, the deepest network being used as the teacher model, and all shallow networks become multiple student models by adding classifiers. Meanwhile, the distillation path between the deepest network feature and shallow network features is used to reduce the feature difference. A series of experimental results on the ASVspoof 2019 LA and PA datasets show the effectiveness of the proposed method, with significant improvements compared to the baseline.【4】 Speaker-Aware Anti-Spoofing
标题：扬声器感知反欺骗
链接：http://arxiv.org/pdf/2303.01126v1
作者：Xuechen Liu,Md Sahidullah,Kong Aik Lee,Tomi Kinnunen机构：School of Computing, University of Eastern Finland, Joensuu, Finland, Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France, Institute for Infocomm Research, A⋆STAR, Singapore摘要：我们提出了说话人感知的反欺骗方法，其中目标说话人的先验知识被结合到语音欺骗对策中。与常用的非特定人训练方法不同，我们采用了非特定人条件化的方法来训练CM。作为概念的证明，我们考虑了说话人感知对最先进的AASIST（使用集成的谱时图注意网络的音频反欺骗）模型的扩展。为此，我们考虑了两种不同的策略，分别在帧和话语两个层次上整合目标说话人信息。基于ASVspoof 2019数据集的自定义协议的实验结果表明了通过注册获取说话人信息的效率：我们在与说话者无关的基线上分别获得了相等错误率（EER）和最小串联检测成本函数（t-DCF）的25.1%和11.6%的最大相对改进。摘要：We address speaker-aware anti-spoofing, where prior knowledge of the target speaker is incorporated into a voice spoofing countermeasure (CM). In contrast to the frequently used speaker-independent solutions, we train the CM in a speaker-conditioned way. As a proof of concept, we consider speaker-aware extension to the state-of-the-art AASIST (audio anti-spoofing using integrated spectro-temporal graph attention networks) model. To this end, we consider two alternative strategies to incorporate target speaker information at the frame and utterance levels, respectively. The experimental results on a custom protocol based on ASVspoof 2019 dataset indicates the efficiency of the speaker information via enrollment: we obtain maximum relative improvements of 25.1% and 11.6% in equal error rate (EER) and minimum tandem detection cost function (t-DCF) over a speaker-independent baseline, respectively.【5】 Distilling Multi-Level X-vector Knowledge for Small-footprint Speaker Verification
标题：用于小占地说话人确认的多层X向量知识提取
链接：http://arxiv.org/pdf/2303.01125v1
作者：Xuechen Liu,Md Sahidullah,Tomi Kinnunen机构：School of Computing, University of Eastern Finland, Joensuu, Finland, Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：深层说话人模型在说话人确认中具有较低的错误率。尽管如此，高性能往往是模型大小和计算时间的交换，使这些模型在有限的条件下运行具有挑战性。我们专注于小尺寸深说话人嵌入提取，利用知识提炼。虽然先前关于这个主题的工作已经解决了话语级别的说话人嵌入提取，但是我们建议结合来自x向量模型（教师网络）的各个级别的嵌入来训练小足迹的学生网络。结果表明了帧级信息的有用性，根据教师嵌入的大小，学生模型比他们的教师小85%-91%。教师嵌入的级联导致学生网络与教师一起达到可比较的性能，同时利用教师的75%的相对大小减少。这些发现和类比进一步推广到其他x载体变体。摘要：Deep speaker models yield low error rates in speaker verification. Nonetheless, the high performance tends to be exchanged for model size and computation time, making these models challenging to run under limited conditions. We focus on small-footprint deep speaker embedding extraction, leveraging knowledge distillation. While prior work on this topic has addressed speaker embedding extraction at the utterance level, we propose to combine embeddings from various levels of the x-vector model (teacher network) to train small-footprint student networks. Results indicate the usefulness of frame-level information, with the student models being 85%-91% smaller than their teacher, depending on the size of the teacher embeddings. Concatenation of teacher embeddings results in student networks that reach comparable performance along with the teacher while utilizing a 75% relative size reduction from the teacher. The findings and analogies are furthered to other x-vector variants.【6】 LiteG2P: A fast, light and high accuracy model for grapheme-to-phoneme conversion
标题：LiteG2P：一种快速、轻便、高精度的字音转换模型
链接：http://arxiv.org/pdf/2303.01086v1
作者：Chunfeng Wang,Peisong Huang,Yuxiang Zou,Haoyu Zhang,Shichao Liu,Xiang Yin,Zejun Ma机构：Bytedance AI-Lab, China备注：Accepted by ICASSP2023摘要：作为自动语音识别（ASR）的关键组成部分和文语转换（TTS）的前端，字素转音素（G2P）技术的作用是将字母转换为相应的发音。现有方法要么速度慢，要么性能差，应用场景受限，尤其是在设备上推理的过程中。本文综合了专家知识和基于连接主义时态分类（CTC）的神经网络的优点，提出了一种快速、轻量级、理论上并行的新方法LiteG2P。LiteG2P采用精心领先的设计，可在云和设备上应用。在CMU数据集上的实验结果表明，该方法的性能优于基于CTC的方法，参数减少了10倍，甚至与基于Transformer的序列到序列模型相当，参数减少了33倍，计算量减少了33倍。摘要：As a key component of automated speech recognition (ASR) and the front-end in text-to-speech (TTS), grapheme-to-phoneme (G2P) plays the role of converting letters to their corresponding pronunciations. Existing methods are either slow or poor in performance, and are limited in application scenarios, particularly in the process of on-device inference. In this paper, we integrate the advantages of both expert knowledge and connectionist temporal classification (CTC) based neural network and propose a novel method named LiteG2P which is fast, light and theoretically parallel. With the carefully leading design, LiteG2P can be applied both on cloud and on device. Experimental results on the CMU dataset show that the performance of the proposed method is superior to the state-of-the-art CTC based method with 10 times fewer parameters, and even comparable to the state-of-the-art Transformer-based sequence-to-sequence model with less parameters and 33 times less computation.【7】 Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
标题：谷歌USM：将自动语音识别扩展到100多种语言
链接：http://arxiv.org/pdf/2303.01037v1
作者：Yu Zhang,Wei Han,James Qin,Yongqiang Wang,Ankur Bapna,Zhehuai Chen,Nanxin Chen,Bo Li,Vera Axelrod,Gary Wang,Zhong Meng,Ke Hu,Andrew Rosenberg,Rohit Prabhavalkar,Daniel S. Park,Parisa Haghani,Jason Riesa,Ginger Perng,Hagen Soltau,Trevor Strohman,Bhuvana Ramabhadran,Tara Sainath,Pedro Moreno,Chung-Cheng Chiu,Johan Schalkwyk,Françoise Beaufays,Yonghui Wu备注：20 pages, 7 figures, 8 tables摘要：我们介绍了通用语音模型（USM），这是一个可以跨100多种语言执行自动语音识别（ASR）的大型模型。这是通过在跨越300种语言的1200万（M）小时的大型未标记多语言数据集上预训练模型的编码器，并在较小的标记数据集上进行微调来实现的。我们使用带有随机投影量化和语音-文本模态匹配的多语言预训练来实现下游多语言ASR和语音到文本翻译任务的最先进性能。我们还证明，尽管使用的标记训练集的大小是Whisper模型的1 7，但我们的模型在跨许多语言的域内和域外语音识别任务上表现出相当或更好的性能。摘要：We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1 7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.【8】Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition
标题：用于自动语音识别的合成交叉口音数据增强
链接：http://arxiv.org/pdf/2303.00802v1
作者：Philipp Klumpp,Pooja Chitkara,Leda Sarı,Prashant Serai,Jilong Wu,Irina-Elena Veliche,Rongqing Huang,Qing He机构：Meta AI, Pattern Recognition Lab, FAU Erlangen-N¨urnberg摘要：近年来，对有偏倚的ASR数据集或模型的认识显著增加。即使对于英语，尽管有大量可用的训练数据，但系统对于非母语使用者表现更差。在本文中，我们改进了一个重音转换模型（ACM），它可以将母语为美国英语的语音转换为重音发音。我们在ACM训练中包括语音知识，以提供关于在合成波形中恢复某些发音模式的准确反馈。此外，我们研究了学习重音表征代替静态嵌入的可行性。生成的数据然后用于训练两个最先进的ASR系统。我们在母语和非母语英语数据集上评估了我们的方法，发现综合重音数据有助于ASR更好地从可见重音理解语音。这种观察结果没有转化为看不见的口音，并且对于专门用母语语音预先训练的模型没有观察到这种现象。摘要：The awareness for biased ASR datasets or models has increased notably in recent years. Even for English, despite a vast amount of available training data, systems perform worse for non-native speakers. In this work, we improve an accent-conversion model (ACM) which transforms native US-English speech into accented pronunciation. We include phonetic knowledge in the ACM training to provide accurate feedback about how well certain pronunciation patterns were recovered in the synthesized waveform. Furthermore, we investigate the feasibility of learned accent representations instead of static embeddings. Generated data was then used to train two state-of-the-art ASR systems. We evaluated our approach on native and non-native English datasets and found that synthetically accented data helped the ASR to better understand speech from seen accents. This observation did not translate to unseen accents, and it was not observed for a model that had been pre-trained exclusively with native speech.【9】 Improving Transformer-based End-to-End Speaker Diarization by Assigning Auxiliary Losses to Attention Heads
标题：通过将辅助损耗分配给注意头来改善基于Transformer的端到端说话人二值化
链接：http://arxiv.org/pdf/2303.01192v1
作者：Ye-Rin Jeoung,Joon-Young Yang,Jeong-Hwan Choi,Joon-Hyuk Chang机构：Department of Electronic Engineering, Hanyang University, Seoul, Republic of Korea备注：5 pages, ICASSP2023 (Accepted)摘要：基于变换器的端到端神经说话人识别（EEND）模型利用多脑自注意（SA）机制实现重叠语音区域的准确说话人标记预测。在本研究中，为了提高SA-EEND模型的训练效果，我们建议使用Transformer层的SA头的辅助损耗。具体地，我们假设SA层的注意力权重矩阵是冗余的，如果它们的模式类似于单位矩阵的模式。然后，我们显式地约束这样的矩阵以展示与话音活动检测或重叠语音检测任务相关的特定说话者活动模式。因此，我们期望所提出的辅助损耗引导Transformer层在注意力权重中表现出更多样的模式，从而减少SA头部中的假设冗余。仿真和CALLHOME数据集上的双说话人二值化实验结果表明了该方法的有效性，与传统的SA-EEND模型相比，二值化错误率分别降低了32.58%和17.11%。摘要：Transformer-based end-to-end neural speaker diarization (EEND) models utilize the multi-head self-attention (SA) mechanism to enable accurate speaker label prediction in overlapped speech regions. In this study, to enhance the training effectiveness of SA-EEND models, we propose the use of auxiliary losses for the SA heads of the transformer layers. Specifically, we assume that the attention weight matrices of an SA layer are redundant if their patterns are similar to those of the identity matrix. We then explicitly constrain such matrices to exhibit specific speaker activity patterns relevant to voice activity detection or overlapped speech detection tasks. Consequently, we expect the proposed auxiliary losses to guide the transformer layers to exhibit more diverse patterns in the attention weights, thereby reducing the assumed redundancies in the SA heads. The effectiveness of the proposed method is demonstrated using the simulated and CALLHOME datasets for two-speaker diarization tasks, reducing the diarization error rate of the conventional SA-EEND model by 32.58% and 17.11%, respectively.【10】DISPLACE Challenge: DIarization of SPeaker and LAnguage in Conversational Environments
标题：取代挑战：对话环境中说话人和语言的二元化
链接：http://arxiv.org/pdf/2303.00830v1
作者：Shikha Baghel,Shreyas Ramoji,Sidharth,Ranjana H,Prachi Singh,Somil Jain,Pratik Roy Chowdhuri,Kaustubh Kulkarni,Swapnil Padhi,Deepu Vijayasenan,Sriram Ganapathy机构：LEAP Lab, Department of Electrical Engineering, Indian Institute of Science, Bengaluru, India, Department of Electronics and Communication, National Institute of Technology Karnataka, Surathkal, India摘要：DISPLACE挑战需要对同一数据执行说话人和语言日记化的第一个任务，因为数据包含多语言代码混合语音中的多说话人社交对话。该挑战试图在多语言环境中对说话人对话（SD）和多说话人环境中的语言对话（LD）进行基准测试和改进。为了应对这一挑战，我们分发了一个自然的多语种、多发言者对话数据集，以用于开发和评估目的。自动系统是在单通道远场录音上进行评估的，其中包括自然码混合、码切换、重叠、混响、短转弯、短停顿和同一语言的多种方言。共有60支来自工业界和学术界的队伍报名参加这一挑战。摘要：The DISPLACE challenge entails a first-of-kind task to perform speaker and language diarization on the same data, as the data contains multi-speaker social conversations in multilingual code-mixed speech. The challenge attempts to benchmark and improve Speaker Diarization (SD) in multilingual settings and Language Diarization (LD) in multi-speaker settings. For this challenge, a natural multilingual, multi-speaker conversational dataset is distributed for development and evaluation purposes. Automatic systems are evaluated on single-channel far-field recordings containing natural code-mix, code-switch, overlap, reverberation, short turns, short pauses, and multiple dialects of the same language. A total of 60 teams from industry and academia have registered for this challenge.【1】 Improving Transformer-based End-to-End Speaker Diarization by Assigning Auxiliary Losses to Attention Heads
标题：通过将辅助损耗分配给注意头来改善基于Transformer的端到端说话人二值化
链接：http://arxiv.org/pdf/2303.01192v1
*与cs.SD语音【9】为同一篇
作者：Ye-Rin Jeoung,Joon-Young Yang,Jeong-Hwan Choi,Joon-Hyuk Chang机构：Department of Electronic Engineering, Hanyang University, Seoul, Republic of Korea备注：5 pages, ICASSP2023 (Accepted)摘要：基于变换器的端到端神经说话人识别（EEND）模型利用多脑自注意（SA）机制实现重叠语音区域的准确说话人标记预测。在本研究中，为了提高SA-EEND模型的训练效果，我们建议使用Transformer层的SA头的辅助损耗。具体地，我们假设SA层的注意力权重矩阵是冗余的，如果它们的模式类似于单位矩阵的模式。然后，我们显式地约束这样的矩阵以展示与话音活动检测或重叠语音检测任务相关的特定说话者活动模式。因此，我们期望所提出的辅助损耗引导Transformer层在注意力权重中表现出更多样的模式，从而减少SA头部中的假设冗余。仿真和CALLHOME数据集上的双说话人二值化实验结果表明了该方法的有效性，与传统的SA-EEND模型相比，二值化错误率分别降低了32.58%和17.11%。摘要：Transformer-based end-to-end neural speaker diarization (EEND) models utilize the multi-head self-attention (SA) mechanism to enable accurate speaker label prediction in overlapped speech regions. In this study, to enhance the training effectiveness of SA-EEND models, we propose the use of auxiliary losses for the SA heads of the transformer layers. Specifically, we assume that the attention weight matrices of an SA layer are redundant if their patterns are similar to those of the identity matrix. We then explicitly constrain such matrices to exhibit specific speaker activity patterns relevant to voice activity detection or overlapped speech detection tasks. Consequently, we expect the proposed auxiliary losses to guide the transformer layers to exhibit more diverse patterns in the attention weights, thereby reducing the assumed redundancies in the SA heads. The effectiveness of the proposed method is demonstrated using the simulated and CALLHOME datasets for two-speaker diarization tasks, reducing the diarization error rate of the conventional SA-EEND model by 32.58% and 17.11%, respectively.【2】Real-time Audio Video Enhancement with a Microphone Array and Headphones
标题：使用麦克风阵列和耳机进行实时音视频增强
链接：http://arxiv.org/pdf/2303.00949v1
作者：Jacob Kealey,Anthony Gosselin,Étienne Deshaies-Samson,Francis Cardinal,Félix Ducharme-Turcotte,Olivier Bergeron,Amélie Rioux-Joyal,Jérémy Bélec,François Grondin机构：Authors are with the Department of Electrical Engineeringand Computer Engineering, Universit´e de Sherbrooke备注：Submitted to IROS 2023摘要：本文提出了一个完整的硬件和软件流水线，用于在噪声和混响条件下的实时语音增强。该设备由安装在眼镜上的麦克风阵列和摄像头组成，连接到增强语音并在耳机中播放音频的嵌入式系统，最大延迟为120毫秒。提出的方法依赖于人脸检测、跟踪和验证，使用波束形成器和后滤波神经网络来增强目标说话人的语音。实验结果验证了该方法的可行性，为探索和验证各种波束形成和语音增强方法用于实时语音增强奠定了基础。摘要：This paper presents a complete hardware and software pipeline for real-time speech enhancement in noisy and reverberant conditions. The device consists of a microphone array and a camera mounted on eyeglasses, connected to an embedded system that enhances speech and plays back the audio in headphones, with a latency of maximum 120 msec. The proposed approach relies on face detection, tracking and verification to enhance the speech of a target speaker using a beamformer and a postfiltering neural network. Results demonstrate the feasibility of the approach, and opens the door to the exploration and validation of a wide range of beamformer and speech enhancement methods for real-time speech enhancement.【3】DISPLACE Challenge: DIarization of SPeaker and LAnguage in Conversational Environments
标题：取代挑战：对话环境中说话人和语言的二元化
链接：http://arxiv.org/pdf/2303.00830v1
*与cs.SD语音【10】为同一篇
作者：Shikha Baghel,Shreyas Ramoji,Sidharth,Ranjana H,Prachi Singh,Somil Jain,Pratik Roy Chowdhuri,Kaustubh Kulkarni,Swapnil Padhi,Deepu Vijayasenan,Sriram Ganapathy机构：LEAP Lab, Department of Electrical Engineering, Indian Institute of Science, Bengaluru, India, Department of Electronics and Communication, National Institute of Technology Karnataka, Surathkal, India摘要：DISPLACE挑战需要对同一数据执行说话人和语言日记化的第一个任务，因为数据包含多语言代码混合语音中的多说话人社交对话。该挑战试图在多语言环境中对说话人对话（SD）和多说话人环境中的语言对话（LD）进行基准测试和改进。为了应对这一挑战，我们分发了一个自然的多语种、多发言者对话数据集，以用于开发和评估目的。自动系统是在单通道远场录音上进行评估的，其中包括自然码混合、码切换、重叠、混响、短转弯、短停顿和同一语言的多种方言。共有60支来自工业界和学术界的队伍报名参加这一挑战。摘要：The DISPLACE challenge entails a first-of-kind task to perform speaker and language diarization on the same data, as the data contains multi-speaker social conversations in multilingual code-mixed speech. The challenge attempts to benchmark and improve Speaker Diarization (SD) in multilingual settings and Language Diarization (LD) in multi-speaker settings. For this challenge, a natural multilingual, multi-speaker conversational dataset is distributed for development and evaluation purposes. Automatic systems are evaluated on single-channel far-field recordings containing natural code-mix, code-switch, overlap, reverberation, short turns, short pauses, and multiple dialects of the same language. A total of 60 teams from industry and academia have registered for this challenge.【4】 Ego-noise reduction of a mobile robot using noise spatial covariance matrix learning and minimum variance distortionless response
标题：基于噪声空间协方差矩阵学习和最小方差无失真响应的移动机器人自降噪
链接：http://arxiv.org/pdf/2303.00829v1
作者：Philippe-Olivier Lagacé,François Ferland,François Grondin机构：Grondin are with theDepartment of Electrical Engineering and Computer Engineering, Universit´ede Sherbrooke备注：Submitted to IROS 2023摘要：语音和事件识别系统的性能最近由于深度学习方法而显著提高。然而，当算法被部署在机器人上时，由于在训练神经网络时由其致动器产生的不可见的机械噪声和电气干扰，这些任务中的一些仍然具有挑战性。因此，当在机器人上使用预先训练的语音和事件识别算法时，作为预处理步骤的自我噪声减少可以帮助解决这个问题。在这篇文章中，我们提出了一种新的方法来减少自我噪声只使用麦克风阵列和不到两分钟的噪音记录。使用主成分分析（PCA），从校准期间在线创建的字典中选择最佳协方差矩阵候选，并与最小方差无失真响应（MVDR）波束形成器一起使用。结果表明，该方法实时性好，信号失真比（SDR）提高了10 dB，误字率（WER）在某些情况下降低了55%，事件检测的平均精度（AP）提高了0.2。摘要：The performance of speech and events recognition systems significantly improved recently thanks to deep learning methods. However, some of these tasks remain challenging when algorithms are deployed on robots due to the unseen mechanical noise and electrical interference generated by their actuators while training the neural networks. Ego-noise reduction as a preprocessing step therefore can help solve this issue when using pre-trained speech and event recognition algorithms on robots. In this paper, we propose a new method to reduce ego-noise using only a microphone array and less than two minute of noise recordings. Using Principal Component Analysis (PCA), the best covariance matrix candidate is selected from a dictionary created online during calibration and used with the Minimum Variance Distortionless Response (MVDR) beamformer. Results show that the proposed method runs in real-time, improves the signal-to-distortion ratio (SDR) by up to 10 dB, decreases the word error rate (WER) by 55 % in some cases and increases the Average Precision (AP) of event detection by up to 0.2.【5】 ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representations
标题：ParrotTTS：基于自监督表示的文语合成
链接：http://arxiv.org/pdf/2303.01261v1
*与cs.SD语音【1】为同一篇
作者：Saiteja Kosgi,Neil Kumar Shah,Vishal Tambrahalli,Neha Sherin,Vineet Gandhi机构：Kohli Centre on Intelligent Systems, IIIT Hyderabad, TCS Research, Pune摘要：自统计TTS时代以来，TTS系统被建模为梅尔合成器，随后是语音声码器，并被推进到神经设计中。我们提出了一种替代TTS建模的方法，称为ParrotTTS，借用自监督学习（SSL）方法。ParrotTTS采用两步法，首先在大量可用的未标记数据上训练语音到语音模型，然后是文本到嵌入模型，利用对齐转录的语音将其扩展到TTS。与传统TTS模型相比，ParrotTTS在自然度上获得了有竞争力的平均意见得分，但显著提高了后者转录对的数据效率和无转录的说话人自适应。这进一步为在一般训练的SSL语音模型上训练TTS模型铺平了道路。摘要：Text-to-speech (TTS) systems are modelled as mel-synthesizers followed by speech-vocoders since the era of statistical TTS that is carried forward into neural designs. We propose an alternative approach to TTS modelling referred to as ParrotTTS borrowing from self-supervised learning (SSL) methods. ParrotTTS takes a two-step approach by initially training a speech-to-speech model on unlabelled data that is abundantly available, followed by a text-to-embedding model that leverages speech with aligned transcriptions to extend it to TTS. ParrotTTS achieves competitive mean opinion scores on naturalness compared to traditional TTS models but significantly improves over the latter's data efficiency of transcribed pairs and speaker adaptation without transcriptions. This further paves the path to training TTS models on generically trained SSL speech models.【6】 Language-Universal Adapter Learning with Knowledge Distillation for End-to-End Multilingual Speech Recognition
标题：用于端到端多语言语音识别的具有知识提取的语言通用适配器学习
链接：http://arxiv.org/pdf/2303.01249v1
*与cs.SD语音【2】为同一篇
作者：Zhijie Shen,Wu Guo,Bin Gu机构：The Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Hefei, China摘要：本文提出了一种基于预训练模型的端到端多语言自动语音识别（ASR）的语言通用适配器学习框架。对于声学建模，wav2vec 2.0预训练模型通过插入特定语言和通用语言适配器进行了微调。然后使用在线知识提炼来使语言通用适配器能够学习语言专用特征和通用特征。通过利用语言标识符（LID）也减少了语言信息混淆。使用LID，我们对多个头部注意力输出执行位置式修改。在推理过程中，语言专用适配器被移除，而语言通用适配器保持激活。该方法提高了识别精度，解决了常见多语种ASR系统中适配器参数数量随语种数量线性增加的问题。在BABEL数据集上的实验验证了该框架的有效性。与传统的多语言模型相比，绝对错误率降低了3.3%。该代码可从以下网址获得：https: github.com shen9712 UniversalAdapterLearning。摘要：In this paper, we propose a language-universal adapter learning framework based on a pre-trained model for end-to-end multilingual automatic speech recognition (ASR). For acoustic modeling, the wav2vec 2.0 pre-trained model is fine-tuned by inserting language-specific and language-universal adapters. An online knowledge distillation is then used to enable the language-universal adapters to learn both language-specific and universal features. The linguistic information confusion is also reduced by leveraging language identifiers (LIDs). With LIDs we perform a position-wise modification on the multi-head attention outputs. In the inference procedure, the language-specific adapters are removed while the language-universal adapters are kept activated. The proposed method improves the recognition accuracy and addresses the linear increase of the number of adapters' parameters with the number of languages in common multilingual ASR systems. Experiments on the BABEL dataset confirm the effectiveness of the proposed framework. Compared to the conventional multilingual model, a 3.3% absolute error rate reduction is achieved. The code is available at: https: github.com shen9712 UniversalAdapterLearning.【7】 Learning From Yourself: A Self-Distillation Method for Fake Speech Detection
标题：向自己学习：一种用于虚假语音检测的自蒸馏方法
链接：http://arxiv.org/pdf/2303.01211v1
*与cs.SD语音【3】为同一篇
作者：Jun Xue,Cunhang Fan,Jiangyan Yi,Chenglong Wang,Zhengqi Wen,Dan Zhang,Zhao Lv机构：Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei , China, NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing , China, Qiyuan Laboratory, Beijing , China备注：Accepted by ICASSP 2023摘要：本文提出了一种新的自提取伪语音检测方法，该方法能够在不增加模型复杂度的前提下显著提高伪语音检测的性能。对于FSD来说，一些细粒度的信息是非常重要的，如谱图缺陷、哑段等，这些信息往往被浅网络感知。然而，浅网络有很多噪声，不能很好地捕捉这一点。针对这一问题，提出用最深网络指导浅网络来增强浅网络。具体地，FSD的网络被分成若干段，最深的网络被用作教师模型，并且所有浅的网络通过添加分类器而成为多个学生模型。同时，利用最深网络特征和浅网络特征之间的蒸馏路径来减小特征差异。在ASVspoof 2019 LA和PA数据集上的一系列实验结果表明了所提方法的有效性，与基线相比有显著的改进.摘要：In this paper, we propose a novel self-distillation method for fake speech detection (FSD), which can significantly improve the performance of FSD without increasing the model complexity. For FSD, some fine-grained information is very important, such as spectrogram defects, mute segments, and so on, which are often perceived by shallow networks. However, shallow networks have much noise, which can not capture this very well. To address this problem, we propose using the deepest network instruct shallow network for enhancing shallow networks. Specifically, the networks of FSD are divided into several segments, the deepest network being used as the teacher model, and all shallow networks become multiple student models by adding classifiers. Meanwhile, the distillation path between the deepest network feature and shallow network features is used to reduce the feature difference. A series of experimental results on the ASVspoof 2019 LA and PA datasets show the effectiveness of the proposed method, with significant improvements compared to the baseline.【8】 Speaker-Aware Anti-Spoofing
标题：扬声器感知反欺骗
链接：http://arxiv.org/pdf/2303.01126v1
*与cs.SD语音【4】为同一篇
作者：Xuechen Liu,Md Sahidullah,Kong Aik Lee,Tomi Kinnunen机构：School of Computing, University of Eastern Finland, Joensuu, Finland, Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France, Institute for Infocomm Research, A⋆STAR, Singapore摘要：我们提出了说话人感知的反欺骗方法，其中目标说话人的先验知识被结合到语音欺骗对策中。与常用的非特定人训练方法不同，我们采用了非特定人条件化的方法来训练CM。作为概念的证明，我们考虑了说话人感知对最先进的AASIST（使用集成的谱时图注意网络的音频反欺骗）模型的扩展。为此，我们考虑了两种不同的策略，分别在帧和话语两个层次上整合目标说话人信息。基于ASVspoof 2019数据集的自定义协议的实验结果表明了通过注册获取说话人信息的效率：我们在与说话者无关的基线上分别获得了相等错误率（EER）和最小串联检测成本函数（t-DCF）的25.1%和11.6%的最大相对改进。摘要：We address speaker-aware anti-spoofing, where prior knowledge of the target speaker is incorporated into a voice spoofing countermeasure (CM). In contrast to the frequently used speaker-independent solutions, we train the CM in a speaker-conditioned way. As a proof of concept, we consider speaker-aware extension to the state-of-the-art AASIST (audio anti-spoofing using integrated spectro-temporal graph attention networks) model. To this end, we consider two alternative strategies to incorporate target speaker information at the frame and utterance levels, respectively. The experimental results on a custom protocol based on ASVspoof 2019 dataset indicates the efficiency of the speaker information via enrollment: we obtain maximum relative improvements of 25.1% and 11.6% in equal error rate (EER) and minimum tandem detection cost function (t-DCF) over a speaker-independent baseline, respectively.【9】 Distilling Multi-Level X-vector Knowledge for Small-footprint Speaker Verification
标题：用于小占地说话人确认的多层X向量知识提取
链接：http://arxiv.org/pdf/2303.01125v1
*与cs.SD语音【5】为同一篇
作者：Xuechen Liu,Md Sahidullah,Tomi Kinnunen机构：School of Computing, University of Eastern Finland, Joensuu, Finland, Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：深层说话人模型在说话人确认中具有较低的错误率。尽管如此，高性能往往是模型大小和计算时间的交换，使这些模型在有限的条件下运行具有挑战性。我们专注于小尺寸深说话人嵌入提取，利用知识提炼。虽然先前关于这个主题的工作已经解决了话语级别的说话人嵌入提取，但是我们建议结合来自x向量模型（教师网络）的各个级别的嵌入来训练小足迹的学生网络。结果表明了帧级信息的有用性，根据教师嵌入的大小，学生模型比他们的教师小85%-91%。教师嵌入的级联导致学生网络与教师一起达到可比较的性能，同时利用教师的75%的相对大小减少。这些发现和类比进一步推广到其他x载体变体。摘要：Deep speaker models yield low error rates in speaker verification. Nonetheless, the high performance tends to be exchanged for model size and computation time, making these models challenging to run under limited conditions. We focus on small-footprint deep speaker embedding extraction, leveraging knowledge distillation. While prior work on this topic has addressed speaker embedding extraction at the utterance level, we propose to combine embeddings from various levels of the x-vector model (teacher network) to train small-footprint student networks. Results indicate the usefulness of frame-level information, with the student models being 85%-91% smaller than their teacher, depending on the size of the teacher embeddings. Concatenation of teacher embeddings results in student networks that reach comparable performance along with the teacher while utilizing a 75% relative size reduction from the teacher. The findings and analogies are furthered to other x-vector variants.【10】 LiteG2P: A fast, light and high accuracy model for grapheme-to-phoneme conversion
标题：LiteG2P：一种快速、轻便、高精度的字音转换模型
链接：http://arxiv.org/pdf/2303.01086v1
*与cs.SD语音【6】为同一篇
作者：Chunfeng Wang,Peisong Huang,Yuxiang Zou,Haoyu Zhang,Shichao Liu,Xiang Yin,Zejun Ma机构：Bytedance AI-Lab, China备注：Accepted by ICASSP2023摘要：作为自动语音识别（ASR）的关键组成部分和文语转换（TTS）的前端，字素转音素（G2P）技术的作用是将字母转换为相应的发音。现有方法要么速度慢，要么性能差，应用场景受限，尤其是在设备上推理的过程中。本文综合了专家知识和基于连接主义时态分类（CTC）的神经网络的优点，提出了一种快速、轻量级、理论上并行的新方法LiteG2P。LiteG2P采用精心领先的设计，可在云和设备上应用。在CMU数据集上的实验结果表明，该方法的性能优于基于CTC的方法，参数减少了10倍，甚至与基于Transformer的序列到序列模型相当，参数减少了33倍，计算量减少了33倍。摘要：As a key component of automated speech recognition (ASR) and the front-end in text-to-speech (TTS), grapheme-to-phoneme (G2P) plays the role of converting letters to their corresponding pronunciations. Existing methods are either slow or poor in performance, and are limited in application scenarios, particularly in the process of on-device inference. In this paper, we integrate the advantages of both expert knowledge and connectionist temporal classification (CTC) based neural network and propose a novel method named LiteG2P which is fast, light and theoretically parallel. With the carefully leading design, LiteG2P can be applied both on cloud and on device. Experimental results on the CMU dataset show that the performance of the proposed method is superior to the state-of-the-art CTC based method with 10 times fewer parameters, and even comparable to the state-of-the-art Transformer-based sequence-to-sequence model with less parameters and 33 times less computation.【11】 Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
标题：谷歌USM：将自动语音识别扩展到100多种语言
链接：http://arxiv.org/pdf/2303.01037v1
*与cs.SD语音【7】为同一篇
作者：Yu Zhang,Wei Han,James Qin,Yongqiang Wang,Ankur Bapna,Zhehuai Chen,Nanxin Chen,Bo Li,Vera Axelrod,Gary Wang,Zhong Meng,Ke Hu,Andrew Rosenberg,Rohit Prabhavalkar,Daniel S. Park,Parisa Haghani,Jason Riesa,Ginger Perng,Hagen Soltau,Trevor Strohman,Bhuvana Ramabhadran,Tara Sainath,Pedro Moreno,Chung-Cheng Chiu,Johan Schalkwyk,Françoise Beaufays,Yonghui Wu备注：20 pages, 7 figures, 8 tables摘要：我们介绍了通用语音模型（USM），这是一个可以跨100多种语言执行自动语音识别（ASR）的大型模型。这是通过在跨越300种语言的1200万（M）小时的大型未标记多语言数据集上预训练模型的编码器，并在较小的标记数据集上进行微调来实现的。我们使用带有随机投影量化和语音-文本模态匹配的多语言预训练来实现下游多语言ASR和语音到文本翻译任务的最先进性能。我们还证明，尽管使用的标记训练集的大小是Whisper模型的1 7，但我们的模型在跨许多语言的域内和域外语音识别任务上表现出相当或更好的性能。摘要：We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1 7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.【12】 Leveraging Large Text Corpora for End-to-End Speech Summarization
标题：利用大型文本语料库进行端到端语音摘要
链接：http://arxiv.org/pdf/2303.00978v1
作者：Kohei Matsuura,Takanori Ashihara,Takafumi Moriya,Tomohiro Tanaka,Atsunori Ogawa,Marc Delcroix,Ryo Masumura机构：NTT Corporation, Japan备注：Accepted to ICASSP 2023摘要：端到端语音摘要（E2E SSum）是一种从语音直接生成摘要句的技术。与结合了自动语音识别（ASR）和文本摘要模型的级联方法相比，E2E方法更有前途，因为它减少了ASR错误，合并了非语言信息，并简化了整个系统。然而，由于收集大量的配对数据（即，语音和概要）是困难的，所以训练数据通常不足以训练鲁棒的E2E SSum系统。本文提出了两种利用大量外部文本摘要数据进行E2E SSum训练的新方法。第一种技术是利用文本到语音（TTS）系统来生成合成语音，该合成语音用于具有文本摘要的E2E SSum训练。第二种是无TTS的方法，直接将音素序列而不是合成语音输入到E2E SSum模型。实验表明，本文提出的基于TTS和音素的方法在How2数据集上提高了多个指标。特别地，我们的最佳系统以很大的幅度优于先前的现有技术（即，METEOR评分改善超过6分）。据我们所知，这是第一个使用外部语言资源进行E2E SSum的工作。此外，我们报告了How2数据集的详细分析，以确认我们提出的E2E SSum系统的有效性。摘要：End-to-end speech summarization (E2E SSum) is a technique to directly generate summary sentences from speech. Compared with the cascade approach, which combines automatic speech recognition (ASR) and text summarization models, the E2E approach is more promising because it mitigates ASR errors, incorporates nonverbal information, and simplifies the overall system. However, since collecting a large amount of paired data (i.e., speech and summary) is difficult, the training data is usually insufficient to train a robust E2E SSum system. In this paper, we present two novel methods that leverage a large amount of external text summarization data for E2E SSum training. The first technique is to utilize a text-to-speech (TTS) system to generate synthesized speech, which is used for E2E SSum training with the text summary. The second is a TTS-free method that directly inputs phoneme sequence instead of synthesized speech to the E2E SSum model. Experiments show that our proposed TTS- and phoneme-based methods improve several metrics on the How2 dataset. In particular, our best system outperforms a previous state-of-the-art one by a large margin (i.e., METEOR score improvements of more than 6 points). To the best of our knowledge, this is the first work to use external language resources for E2E SSum. Moreover, we report a detailed analysis of the How2 dataset to confirm the validity of our proposed E2E SSum system.【13】Synthetic Cross-accent Data Augmentation for Automatic Speech Recognition
标题：用于自动语音识别的合成交叉口音数据增强
链接：http://arxiv.org/pdf/2303.00802v1
*与cs.SD语音【8】为同一篇
作者：Philipp Klumpp,Pooja Chitkara,Leda Sarı,Prashant Serai,Jilong Wu,Irina-Elena Veliche,Rongqing Huang,Qing He机构：Meta AI, Pattern Recognition Lab, FAU Erlangen-N¨urnberg摘要：近年来，对有偏倚的ASR数据集或模型的认识显著增加。即使对于英语，尽管有大量可用的训练数据，但系统对于非母语使用者表现更差。在本文中，我们改进了一个重音转换模型（ACM），它可以将母语为美国英语的语音转换为重音发音。我们在ACM训练中包括语音知识，以提供关于在合成波形中恢复某些发音模式的准确反馈。此外，我们研究了学习重音表征代替静态嵌入的可行性。生成的数据然后用于训练两个最先进的ASR系统。我们在母语和非母语英语数据集上评估了我们的方法，发现综合重音数据有助于ASR更好地从可见重音理解语音。这种观察结果没有转化为看不见的口音，并且对于专门用母语语音预先训练的模型没有观察到这种现象。摘要：The awareness for biased ASR datasets or models has increased notably in recent years. Even for English, despite a vast amount of available training data, systems perform worse for non-native speakers. In this work, we improve an accent-conversion model (ACM) which transforms native US-English speech into accented pronunciation. We include phonetic knowledge in the ACM training to provide accurate feedback about how well certain pronunciation patterns were recovered in the synthesized waveform. Furthermore, we investigate the feasibility of learned accent representations instead of static embeddings. Generated data was then used to train two state-of-the-art ASR systems. We evaluated our approach on native and non-native English datasets and found that synthetically accented data helped the ASR to better understand speech from seen accents. This observation did not translate to unseen accents, and it was not observed for a model that had been pre-trained exclusively with native speech.【14】Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training
标题：用门控语言专家和课程训练建设高精度的多语种ASR
链接：http://arxiv.org/pdf/2303.00786v1
作者：Eric Sun,Jinyu Li,Yuxuan Hu,Yimeng Zhu,Long Zhou,Jian Xue,Peidong Wang,Linquan Liu,Shujie Liu,Edward Lin,Yifan Gong摘要：我们提出门控语言专家来改进多语言Transformer换能器模型，而无需在推理期间从用户输入任何语言标识（LID）。我们定义了门控机制和LID损耗以使Transformer编码器学习语言相关信息，使用门控转换器专家和共享转换器层构建多语言转换器块以实现紧凑模型，并在联合网络输出上应用线性专家以更好地正则化语音声学和令牌标签联合信息。此外，还提出了一个课程培训方案，让LID指导被选语言专家更好地服务于相应的语言。在英语和西班牙语双语任务上进行评估，我们的方法分别比基线双语模型和单语模型平均减少了12.5%和7.3%的相对单词错误，获得了与用oracle LID训练和推断的上限模型相似的结果。我们在三语、四语和五语模型上进一步探索了我们的方法，并观察到了类似于双语模型的优点，这表明我们的方法易于扩展到更多的语言。摘要：We propose gated language experts to improve multilingual transformer transducer models without any language identification (LID) input from users during inference. We define gating mechanism and LID loss to let transformer encoders learn language-dependent information, construct the multilingual transformer block with gated transformer experts and shared transformer layers for compact models, and apply linear experts on joint network output to better regularize speech acoustic and token label joint information. Furthermore, a curriculum training scheme is proposed to let LID guide the gated language experts for better serving their corresponding languages. Evaluated on the English and Spanish bilingual task, our methods achieve average 12.5% and 7.3% relative word error reductions over the baseline bilingual model and monolingual models, respectively, obtaining similar results to the upper bound model trained and inferred with oracle LID. We further explore our method on trilingual, quadrilingual, and pentalingual models, and observe similar advantages as in the bilingual models, which demonstrates the easy extension to more languages.机器翻译，仅供参考
每日语音速递