今日论文合集：cs.SD语音11篇，eess.AS音频处理13篇。
cs.SD语音
【1】 Towards spoken dialect identification of Irish标题：爱尔兰人的口语方言识别研究链接：https://arxiv.org/abs/2307.07436
作者：Liam Lonergan,Mengjie Qian,Neasa Ní Chiaráin,Christer Gobl,Ailbhe Ní Chasaide机构：Phonetics and Speech Laboratory, School of Linguistics, Speech and Communication Sciences, Trinity College Dublin, Ireland, Engineering Department, Cambridge University, UK备注：Accepted to Interspeech 2023 Workshop of the 2nd Annual Meeting of the Special Interest Group of Under-resourced Languages Workshop, Dublin (SiGUL)摘要：爱尔兰语有着丰富的方言和口音。这增加了为低资源语言创建语音识别系统的难度，因为这样的系统必须用有限的语料库应对高度的可变性。最近的一项研究调查方言偏见在爱尔兰ASR发现，均衡的培训语料库引起了不平等的方言性能，与阿尔斯特方言的性能始终比康诺特或明斯特方言差。出于这一动机，本实验研究口语方言识别爱尔兰语，以期将这样一个系统的语音识别管道。两个声学分类模型进行了测试，XLS-R和ECAPA-TDNN，结合基于文本的分类器使用预训练的爱尔兰语BERT模型。ECAPA-TDNN，特别是在VoxLingua 107数据集上进行语言识别预训练的模型，总体表现最好，准确率为73%。通过将模型的输出与基于文本的模型融合，这一比例进一步提高到76%。阿尔斯特方言是最准确的识别，准确率为94%，但该模型难以消除康诺特和明斯特方言之间的歧义，这表明可能需要更细致的方法来区分爱尔兰方言。摘要：The Irish language is rich in its diversity of dialects and accents. This compounds the difficulty of creating a speech recognition system for the low-resource language, as such a system must contend with a high degree of variability with limited corpora. A recent study investigating dialect bias in Irish ASR found that balanced training corpora gave rise to unequal dialect performance, with performance for the Ulster dialect being consistently worse than for the Connacht or Munster dialects. Motivated by this, the present experiments investigate spoken dialect identification of Irish, with a view to incorporating such a system into the speech recognition pipeline. Two acoustic classification models are tested, XLS-R and ECAPA-TDNN, in conjunction with a text-based classifier using a pretrained Irish-language BERT model. The ECAPA-TDNN, particularly a model pretrained for language identification on the VoxLingua107 dataset, performed best overall, with an accuracy of 73%. This was further improved to 76% by fusing the model's outputs with the text-based model. The Ulster dialect was most accurately identified, with an accuracy of 94%, however the model struggled to disambiguate between the Connacht and Munster dialects, suggesting a more nuanced approach may be necessary to robustly distinguish between the dialects of Irish.
【2】 Real-time Percussive Technique Recognition and Embedding Learning for  the Acoustic Guitar标题：吉他敲击技术的实时识别与嵌入学习链接：https://arxiv.org/abs/2307.07426
作者：Andrea Martelloni,Andrew P McPherson,Mathieu Barthet机构：Queen Mary University, of London, Imperial College备注：Accepted at the 24th Int. Society for Music Information Retrieval Conf., Milan, Italy, 2023摘要：实时音乐信息检索（RT-MIR）在增强传统乐器的性能方面具有很大的潜力。我们开发的RT-MIR技术旨在增强打击指法，它融合了原声吉他演奏与吉他身体打击。我们为增强仪器性能的RT-MIR系统制定了几个设计目标：（i）因果约束，（ii）感知上可忽略的动作-声音延迟，（iii）控制亲密度支持，（iv）合成控制支持。我们提出并评估实时吉他身体打击乐识别和嵌入学习技术的基础上卷积神经网络（CNN）和CNN联合训练变分自动编码器（VAE）。我们介绍了一个分类的吉他身体打击的手部分和位置的基础上。我们遵循一个跨数据集的评估方法，收集三个数据集标记根据分类。模型的嵌入质量使用KL-散度跨对应于不同分类类的分布进行评估。结果表明，网络是强大的分类器，特别是在简化的2类识别任务中，与CNN相比，VAE产生了改进的类分离，这一点可以通过增加分布中的KL散度来证明。我们认为，VAE嵌入质量可以支持控制亲密度和丰富的互动时，潜在空间的参数被用来控制外部合成引擎。已经确定了围绕不同数据集的推广的进一步设计挑战。摘要：Real-time music information retrieval (RT-MIR) has much potential to augment the capabilities of traditional acoustic instruments. We develop RT-MIR techniques aimed at augmenting percussive fingerstyle, which blends acoustic guitar playing with guitar body percussion. We formulate several design objectives for RT-MIR systems for augmented instrument performance: (i) causal constraint, (ii) perceptually negligible action-to-sound latency, (iii) control intimacy support, (iv) synthesis control support. We present and evaluate real-time guitar body percussion recognition and embedding learning techniques based on convolutional neural networks (CNNs) and CNNs jointly trained with variational autoencoders (VAEs). We introduce a taxonomy of guitar body percussion based on hand part and location. We follow a cross-dataset evaluation approach by collecting three datasets labelled according to the taxonomy. The embedding quality of the models is assessed using KL-Divergence across distributions corresponding to different taxonomic classes. Results indicate that the networks are strong classifiers especially in a simplified 2-class recognition task, and the VAEs yield improved class separation compared to CNNs as evidenced by increased KL-Divergence across distributions. We argue that the VAE embedding quality could support control intimacy and rich interaction when the latent space's parameters are used to control an external synthesis engine. Further design challenges around generalisation to different datasets have been identified.
【3】 Sumformer: A Linear-Complexity Alternative to Self-Attention for Speech  Recognition标题：Sumform：语音识别中自我注意的一种线性复杂性替代方法链接：https://arxiv.org/abs/2307.07421
作者：Titouan Parcollet,Rogier van Dalen,Shucong Zhang,Sourav Bhattacharya机构：Samsung AI Center, Cambridge, United Kingdom备注：Submitted to NeurIPS 2023摘要：现代语音识别系统依赖于自我注意。不幸的是，令牌与自我注意力的混合在语音话语的长度上需要二次时间，从而减慢推理以及训练并增加存储器消耗。已经开发了用于ASR的自我注意力的更便宜的替代方案，但未能始终达到相同的准确性水平。然而，在实践中，经训练的语音识别器的自我注意力权重采取随时间的全局平均值的形式。因此，本文提出了一种线性时间的替代自我注意的语音识别。它总结了一个完整的话语与平均向量的所有时间步长。然后将该单个摘要与特定时间的信息组合。我们称这种方法为“汇总混合”。在最先进的ASR模型中引入摘要混合，可以保持或超过以前的语音识别性能，同时将训练和推理时间降低高达27%，并将内存预算减少两倍。摘要：Modern speech recognition systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference as well as training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but fail to consistently reach the same level of accuracy. In practice, however, the self-attention weights of trained speech recognizers take the form of a global average over time. This paper, therefore, proposes a linear-time alternative to self-attention for speech recognition. It summarises a whole utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method ``Summary Mixing''. Introducing Summary Mixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while lowering the training and inference times by up to 27% and reducing the memory budget by a factor of two.
【4】 Phoneme-retrieval; voice recognition; vowels recognition标题：音素检索；语音识别；元音识别链接：https://arxiv.org/abs/2307.07407
作者：Brunello Tirozzi,Orchidea Maria Lecian机构：Sapienza University of Rome, Department of Civil and Industrial Engineering, Via Eudossiana,- , Rome, Italy;, Department of Information Engineering, Sapienza Unviersity of Rome, Physics Department, Piazzale Aldo Mororo,- , Rome, Italy;, Key-words备注：10 pages摘要：由于网络的特殊构造方式，提出了一种音素检索技术。给出了神经元的初始集合。这些神经元的数量近似等于数据的典型结构的数量。例如，如果网络是为语音检索而构建的，则神经元的数量必须等于特定人所属的社会群体所讲语言的字母表的特征音素的数量。通常，这个任务非常复杂，并且网络可能严重依赖于用于学习的样本。如果网络是为图像检索而构建的，那么只有当要检索的数据属于特定的图像集时，它才能工作。如果网络是为语音识别而构建的，它只适用于某些特定的单词集。一个典型的例子是用于飞机飞行的单词。例如，如果使用合适的学习过程，则像“飞机应该向东转弯120度”这样的命令可以容易地被网络识别。摘要：A phoneme-retrieval technique is proposed, which is due to the particular way of the construction of the network. An initial set of neurons is given. The number of these neurons is approximately equal to the number of typical structures of the data. For example if the network is built for voice retrieval then the number of neurons must be equal to the number of characteristic phonemes of the alphabet of the language spoken by the social group to which the particular person belongs. Usually this task is very complicated and the network can depend critically on the samples used for the learning. If the network is built for image retrieval then it works only if the data to be retrieved belong to a particular set of images. If the network is built for voice recognition it works only for some particular set of words. A typical example is the words used for the flight of airplanes. For example a command like the "airplane should make a turn of 120 degrees towards the east" can be easily recognized by the network if a suitable learning procedure is used.
【5】 Towards dialect-inclusive recognition in a low-resource language: are  balanced corpora the answer?标题：在低资源语言中实现包括方言的识别：平衡语料库是答案吗？链接：https://arxiv.org/abs/2307.07295
作者：Liam Lonergan,Mengjie Qian,Neasa Ní Chiaráin,Christer Gobl,Ailbhe Ní Chasaide机构：Phonetics and Speech Laboratory, School of Linguistics, Speech and Communication Sciences, Trinity College Dublin, Ireland, Engineering Department, Cambridge University, UK备注：Accepted to Interspeech 2023, Dublin摘要：ASR系统通常是为口语“标准”而构建的，并且它们的性能对于非标准方言/变体而下降。这对于像爱尔兰语这样的语言来说是一个问题，那里没有单一的口语标准，而是三种主要的方言：阿尔斯特（Ul）、康诺特（Co）和明斯特（Mu）。作为一个诊断，以量化的效果，说话者的方言识别性能，12 ASR系统进行了训练，首先使用基线方言平衡的训练语料库，然后使用修改后的版本的基线语料库，其中方言特定的材料被减去或添加。结果表明，方言平衡语料库不产生类似的表现跨方言：UI方言始终表现不佳，而Mu产生最低的WER。Co话与Mu话有着密切的关系，但这种关系并不对称。这些结果将指导未来的语料库收集和系统建设策略，以优化跨方言的性能公平。摘要：ASR systems are generally built for the spoken 'standard', and their performance declines for non-standard dialects/varieties. This is a problem for a language like Irish, where there is no single spoken standard, but rather three major dialects: Ulster (Ul), Connacht (Co) and Munster (Mu). As a diagnostic to quantify the effect of the speaker's dialect on recognition performance, 12 ASR systems were trained, firstly using baseline dialect-balanced training corpora, and then using modified versions of the baseline corpora, where dialect-specific materials were either subtracted or added. Results indicate that dialect-balanced corpora do not yield a similar performance across the dialects: the Ul dialect consistently underperforms, whereas Mu yields lowest WERs. There is a close relationship between Co and Mu dialects, but one that is not symmetrical. These results will guide future corpus collection and system building strategies to optimise for cross-dialect performance equity.
【6】 Replay to Remember: Continual Layer-Specific Fine-tuning for German  Speech Recognition标题：回放到记忆：针对德语语音识别的连续层特定微调链接：https://arxiv.org/abs/2307.07280
作者：Theresa Pekarek Rosin,Stefan Wermter机构：Knowledge Technology, Department of Informatics, University of Hamburg, Vogt-Koelln-Str. , Hamburg, Germany, www.knowledge-technology.info备注：13 pages, 7 figures, submitted to ICANN 2023摘要：虽然自动语音识别（ASR）模型随着无监督或自监督训练技术的引入已经显示出显著的进步，但这些改进仍然仅限于语言和说话者的一部分。迁移学习使大规模多语言模型不仅能够适应低资源语言，而且能够适应更具体的说话者群体。然而，对来自新域的数据进行微调通常伴随着原始域的性能下降。因此，在我们的实验中，我们研究了大规模ASR模型的性能可以在较小的域中近似，使用我们自己的德国高级语音命令（SVC-de）数据集，以及通过在训练期间选择性地冻结模型的部分可以保留多少一般语音识别性能。为了进一步提高ASR模型对微调域之外的词汇和说话者的鲁棒性，我们应用经验重放进行持续学习。通过仅添加来自原始域的一小部分数据，我们能够在新域上达到低于5%的字错误率（WERs），同时在可接受的WERs下稳定一般语音识别的性能。摘要：While Automatic Speech Recognition (ASR) models have shown significant advances with the introduction of unsupervised or self-supervised training techniques, these improvements are still only limited to a subsection of languages and speakers. Transfer learning enables the adaptation of large-scale multilingual models to not only low-resource languages but also to more specific speaker groups. However, fine-tuning on data from new domains is usually accompanied by a decrease in performance on the original domain. Therefore, in our experiments, we examine how well the performance of large-scale ASR models can be approximated for smaller domains, with our own dataset of German Senior Voice Commands (SVC-de), and how much of the general speech recognition performance can be preserved by selectively freezing parts of the model during training. To further increase the robustness of the ASR model to vocabulary and speakers outside of the fine-tuned domain, we apply Experience Replay for continual learning. By adding only a fraction of data from the original domain, we are able to reach Word-Error-Rates (WERs) below 5\% on the new domain, while stabilizing performance for general speech recognition at acceptable WERs.
【7】 AudioInceptionNeXt: TCL AI LAB Submission to EPIC-SOUND  Audio-Based-Interaction-Recognition Challenge 2023标题：AudioInceptionNeXt：TCL AI实验室提交给EPIC-声音-基于音频的交互-识别挑战赛2023链接：https://arxiv.org/abs/2307.07265
作者：Kin Wai Lau,Yasar Abbas Ur Rehman,Yuyang Xie,Lan Ma机构：TCL AI Lab摘要：本报告介绍了我们提交给2023年Epic-Kitchen EPIC-SOUNDS基于音频的交互识别挑战赛的技术细节。任务是学习从音频样本到其对应的动作标签的映射。为了实现这一目标，我们提出了一个简单而有效的单流基于CNN的架构，称为AudioInceptionNeXt，它对音频样本的时频对数梅尔频谱图进行操作。受InceptionNeXt设计的启发，我们在AudioInceptionNeXt块中提出了并行多尺度深度可分离卷积核，这使得模型能够更有效地学习时间和频率信息。大规模可分离核函数能够捕获活动的长持续时间和全局频率语义信息，而小规模可分离核函数能够捕获活动的短持续时间和频率信息的局部细节。我们的方法在挑战测试集上达到了55.43%的前1名准确率，在公共排行榜上排名第一。代码可在https://github.com/StevenLauHKHK/AudioInceptionNeXt.git上匿名获取。摘要：This report presents the technical details of our submission to the 2023 Epic-Kitchen EPIC-SOUNDS Audio-Based Interaction Recognition Challenge. The task is to learn the mapping from audio samples to their corresponding action labels. To achieve this goal, we propose a simple yet effective single-stream CNN-based architecture called AudioInceptionNeXt that operates on the time-frequency log-mel-spectrogram of the audio samples. Motivated by the design of the InceptionNeXt, we propose parallel multi-scale depthwise separable convolutional kernels in the AudioInceptionNeXt block, which enable the model to learn the time and frequency information more effectively. The large-scale separable kernels capture the long duration of activities and the global frequency semantic information, while the small-scale separable kernels capture the short duration of activities and local details of frequency information. Our approach achieved 55.43% of top-1 accuracy on the challenge test set, ranked as 1st on the public leaderboard. Codes are available anonymously at https://github.com/StevenLauHKHK/AudioInceptionNeXt.git.
【8】 Leveraging Pretrained ASR Encoders for Effective and Efficient  End-to-End Speech Intent Classification and Slot Filling标题：利用预先训练的ASR编码器实现有效且高效的端到端语音意图分类和空位填充链接：https://arxiv.org/abs/2307.07057
作者：He Huang,Jagadeesh Balam,Boris Ginsburg机构：NVIDIA, USA备注：INTERSPEECH 2023摘要：我们研究了语音意图分类和时隙填充（SICSF），提出使用在语音识别（ASR）上预训练的编码器来初始化端到端（E2 E）Conformer-Transformer模型，该模型在SLURP数据集上实现了新的最先进的结果，具有90.14%的意图准确度和82.27%的SLURP-F1。我们将我们的模型与在自监督学习（SSL）上预训练的编码器进行比较，并表明ASR预训练比SSL对SICSF更有效。为了探索参数效率，我们冻结了编码器并添加了适配器模块，并表明参数效率只能通过ASR预训练的编码器实现，而SSL编码器需要完全微调才能实现可比的结果。此外，我们提供了一个深入的比较端到端模型与级联模型（ASR+NLU），并表明，E2 E模型优于级联模型，除非提供一个Oracle ASR模型。最后但并非最不重要的是，我们的模型是第一个E2 E模型，实现了与Oracle ASR级联模型相同的性能。代码、检查点和配置可用。摘要：We study speech intent classification and slot filling (SICSF) by proposing to use an encoder pretrained on speech recognition (ASR) to initialize an end-to-end (E2E) Conformer-Transformer model, which achieves the new state-of-the-art results on the SLURP dataset, with 90.14% intent accuracy and 82.27% SLURP-F1. We compare our model with encoders pretrained on self-supervised learning (SSL), and show that ASR pretraining is much more effective than SSL for SICSF. To explore parameter efficiency, we freeze the encoder and add Adapter modules, and show that parameter efficiency is only achievable with an ASR-pretrained encoder, while the SSL encoder needs full finetuning to achieve comparable results. In addition, we provide an in-depth comparison on end-to-end models versus cascading models (ASR+NLU), and show that E2E models are better than cascaded models unless an oracle ASR model is provided. Last but not least, our model is the first E2E model that achieves the same performance as cascading models with oracle ASR. Code, checkpoints and configs are available.
【9】 Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech  Prompts标题：Mega-TTS 2：具有任意长度语音提示的零命中率文本到语音转换链接：https://arxiv.org/abs/2307.07218
作者：Ziyue Jiang,Jinglin Liu,Yi Ren,Jinzheng He,Chen Zhang,Zhenhui Ye,Pengfei Wei,Chunfeng Wang,Xiang Yin,Zejun Ma,Zhou Zhao机构：♠Zhejiang University & ♡ByteDance摘要：Zero-shot文本到语音的目的是合成语音与看不见的语音提示。以前的大规模多扬声器TTS模型已经成功地实现了这一目标，在10秒内注册录音。然而，它们中的大多数被设计成仅利用短语音提示。短语音提示中的有限信息显著阻碍了细粒度身份模仿的表现。在本文中，我们介绍了兆TTS 2，通用zero-shot多说话者TTS模型，能够合成语音看不见的扬声器与任意长度的提示。具体来说，我们1）设计一个多参考音色编码器，从多个参考语音中提取音色信息; 2）训练具有任意长度语音提示的韵律语言模型;通过这些设计，我们的模型适用于不同长度的提示，这扩展了zero-shot文本到语音的语音质量的上限。除了任意长度的提示，我们引入了任意源提示，它利用来自多个P-LLM输出的概率来产生表达和控制韵律。此外，我们提出了一个音素级的自回归持续时间模型，引入上下文学习能力的持续时间建模。实验表明，我们的方法不仅可以合成身份保持语音与一个看不见的发言人的短提示，但也实现了较长的语音提示，提高了性能。音频样本可以在https://mega-tts.github.io/mega2_demo/上找到。摘要：Zero-shot text-to-speech aims at synthesizing voices with unseen speech prompts. Previous large-scale multispeaker TTS models have successfully achieved this goal with an enrolled recording within 10 seconds. However, most of them are designed to utilize only short speech prompts. The limited information in short speech prompts significantly hinders the performance of fine-grained identity imitation. In this paper, we introduce Mega-TTS 2, a generic zero-shot multispeaker TTS model that is capable of synthesizing speech for unseen speakers with arbitrary-length prompts. Specifically, we 1) design a multi-reference timbre encoder to extract timbre information from multiple reference speeches; 2) and train a prosody language model with arbitrary-length speech prompts; With these designs, our model is suitable for prompts of different lengths, which extends the upper bound of speech quality for zero-shot text-to-speech. Besides arbitrary-length prompts, we introduce arbitrary-source prompts, which leverages the probabilities derived from multiple P-LLM outputs to produce expressive and controlled prosody. Furthermore, we propose a phoneme-level auto-regressive duration model to introduce in-context learning capabilities to duration modeling. Experiments demonstrate that our method could not only synthesize identity-preserving speech with a short prompt of an unseen speaker but also achieve improved performance with longer speech prompts. Audio samples can be found in https://mega-tts.github.io/mega2_demo/.
【10】 Low Rank Properties for Estimating Microphones Start Time and Sources  Emission Time标题：估计麦克风起始时间和源发射时间的低秩性链接：https://arxiv.org/abs/2307.07096
作者：Faxian Cao,Yongqiang Cheng,Adil Mehmood Khan,Zhijing Yang,Yingxiu Chang备注：13 pages for main content; 9 pages for proof of proposed low rank properties; 13 figures摘要：关于麦克风记录开始时间和源发射时间的未知定时信息的缺乏在包括联合麦克风和源定位的若干应用中提出了挑战。与传统的直接估计未知定时信息的优化方法相比，低秩性质（LRP）包含一个额外的低秩结构，便于对未知定时信息进行线性约束，以形成相应的低秩结构信息，从而在适当初始化的情况下实现未知定时信息的全局最优解。然而，未知定时信息的初始化是随机的，导致用于估计未知定时信息的局部最小值。在本文中，我们提出了一种组合的低秩近似方法，以减轻随机初始化的影响，对未知的定时信息的估计。我们定义了三个新的变种LRP支持的证明，允许未知的定时信息，以受益于更低的秩结构信息。然后，通过利用低秩结构信息从LRP和LRP的建议的变体，四个线性约束的未知定时信息。最后，我们使用所提出的组合低秩近似算法，通过四个可用的线性约束，获得全局最优解的未知定时信息。实验结果表明，我们的方法相比，国家的最先进的方法在恢复率（任何配置的成功初始化的数量），收敛率（成功恢复的配置的数量），和未知的定时信息的估计误差的优越性能。摘要：The absence of unknown timing information about the microphones recording start time and the sources emission time presents a challenge in several applications, including joint microphones and sources localization. Compared with traditional optimization methods that try to estimate unknown timing information directly, low rank property (LRP) contains an additional low rank structure that facilitates a linear constraint of unknown timing information for formulating corresponding low rank structure information, enabling the achievement of global optimal solutions of unknown timing information with suitable initialization. However, the initialization of unknown timing information is random, resulting in local minimal values for estimation of the unknown timing information. In this paper, we propose a combined low rank approximation method to alleviate the effect of random initialization on the estimation of unknown timing information. We define three new variants of LRP supported by proof that allows unknown timing information to benefit from more low rank structure information. Then, by utilizing the low rank structure information from both LRP and proposed variants of LRP, four linear constraints of unknown timing information are presented. Finally, we use the proposed combined low rank approximation algorithm to obtain global optimal solutions of unknown timing information through the four available linear constraints. Experimental results demonstrate superior performance of our method compared to state-of-the-art approaches in terms of recovery rate (the number of successful initialization for any configuration), convergency rate (the number of successfully recovered configurations), and estimation errors of unknown timing information.
【11】 Controllable Emphasis with zero data for text-to-speech标题：文本到语音转换的零数据可控强调链接：https://arxiv.org/abs/2307.07062
作者：Arnaud Joly,Marco Nicolis,Ekaterina Peterova,Alessandro Lombardi,Ammar Abbas,Arent van Korlaar,Aman Hussain,Parul Sharma,Alexis Moinet,Mateusz Lajszczak,Penny Karanasou,Antonio Bonafonte,Thomas Drugman,Elena Sokolova机构：Amazon, United Kingdom备注：In proceeding of 12th Speech Synthesis Workshop (SSW) 2023摘要：我们提出了一种可扩展的方法来产生高质量的强调文本到语音（TTS），不需要录音或注释。许多TTS模型包括音素持续时间模型。实现强调语音的简单但有效的方法在于增加强调词的预测持续时间。我们表明，这是显着优于频谱图修改技术，提高自然度由$7.3\%$和正确的测试人员的识别强调的词在一个句子中的$40\%$的参考女性EN-美国的声音。我们表明，这种技术显着关闭的差距，需要明确的录音方法。该方法被证明是可扩展的，并在所有四种语言测试（英语，西班牙语，意大利语，德语），不同的声音和多种说话风格的首选。摘要：We present a scalable method to produce high quality emphasis for text-to-speech (TTS) that does not require recordings or annotations. Many TTS models include a phoneme duration model. A simple but effective method to achieve emphasized speech consists in increasing the predicted duration of the emphasised word. We show that this is significantly better than spectrogram modification techniques improving naturalness by $7.3\%$ and correct testers' identification of the emphasized word in a sentence by $40\%$ on a reference female en-US voice. We show that this technique significantly closes the gap to methods that require explicit recordings. The method proved to be scalable and preferred in all four languages tested (English, Spanish, Italian, German), for different voices and multiple speaking styles.
eess.AS音频处理
【1】 Representation Learning With Hidden Unit Clustering For Low Resource  Speech Applications标题：用于低资源语音应用的隐元聚类表示学习链接：https://arxiv.org/abs/2307.07325
作者：Varun Krishna,Tarun Sai,Sriram Ganapathy机构：Department of Electrical Engineering,  IndianInstitute of Science摘要：语音的表示学习，没有文本资源，是许多低资源语音应用的显着感兴趣的领域。在本文中，我们描述了一种方法，从原始音频使用隐藏单元聚类（HUC）框架的自我监督表示学习。模型的输入由加窗并用1-D卷积层处理的音频样本组成。从卷积神经网络（CNN）模块学习的“时间-频率”表示进一步用长短期记忆（LSTM）层处理，LSTM层为每个窗口分段生成上下文向量表示。的HUC框架，允许分类成少量的音素样的单位的表示，用于训练学习语义丰富的语音表示的模型。目标包括音素样的伪标签为每个音频段和这些生成与迭代k均值算法。我们探索的技术，提高的说话人不变性的学习表示，并说明所提出的方法的有效性两个设置，i）完全无监督语音应用程序的子任务描述为零语音2021挑战的一部分和ii）半监督自动语音识别（ASR）应用程序的TIMIT数据集和GramVaani挑战印地语数据集。在这些实验中，我们实现了各种ZeroSpeech任务的最新结果。此外，在ASR实验上，HUC表示被示出显着改善了其他建立的基准的基础上Wav 2 vec，HuBERT和Best-RQ。摘要：The representation learning of speech, without textual resources, is an area of significant interest for many low resource speech applications. In this paper, we describe an approach to self-supervised representation learning from raw audio using a hidden unit clustering (HUC) framework. The input to the model consists of audio samples that are windowed and processed with 1-D convolutional layers. The learned "time-frequency" representations from the convolutional neural network (CNN) module are further processed with long short term memory (LSTM) layers which generate a contextual vector representation for every windowed segment. The HUC framework, allowing the categorization of the representations into a small number of phoneme-like units, is used to train the model for learning semantically rich speech representations. The targets consist of phoneme-like pseudo labels for each audio segment and these are generated with an iterative k-means algorithm. We explore techniques that improve the speaker invariance of the learned representations and illustrate the effectiveness of the proposed approach on two settings, i) completely unsupervised speech applications on the sub-tasks described as part of the ZeroSpeech 2021 challenge and ii) semi-supervised automatic speech recognition (ASR) applications on the TIMIT dataset and on the GramVaani challenge Hindi dataset. In these experiments, we achieve state-of-art results for various ZeroSpeech tasks. Further, on the ASR experiments, the HUC representations are shown to improve significantly over other established benchmarks based on Wav2vec, HuBERT and Best-RQ.
【2】 Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech  Prompts标题：Mega-TTS 2：具有任意长度语音提示的零命中率文本到语音转换链接：https://arxiv.org/abs/2307.07218
作者：Ziyue Jiang,Jinglin Liu,Yi Ren,Jinzheng He,Chen Zhang,Zhenhui Ye,Pengfei Wei,Chunfeng Wang,Xiang Yin,Zejun Ma,Zhou Zhao机构：♠Zhejiang University & ♡ByteDance摘要：Zero-shot文本到语音的目的是合成语音与看不见的语音提示。以前的大规模多扬声器TTS模型已经成功地实现了这一目标，在10秒内注册录音。然而，它们中的大多数被设计成仅利用短语音提示。短语音提示中的有限信息显著阻碍了细粒度身份模仿的表现。在本文中，我们介绍了兆TTS 2，通用zero-shot多说话者TTS模型，能够合成语音看不见的扬声器与任意长度的提示。具体来说，我们1）设计一个多参考音色编码器，从多个参考语音中提取音色信息; 2）训练具有任意长度语音提示的韵律语言模型;通过这些设计，我们的模型适用于不同长度的提示，这扩展了zero-shot文本到语音的语音质量的上限。除了任意长度的提示，我们引入了任意源提示，它利用来自多个P-LLM输出的概率来产生表达和控制韵律。此外，我们提出了一个音素级的自回归持续时间模型，引入上下文学习能力的持续时间建模。实验表明，我们的方法不仅可以合成身份保持语音与一个看不见的发言人的短提示，但也实现了较长的语音提示，提高了性能。音频样本可以在https://mega-tts.github.io/mega2_demo/上找到。摘要：Zero-shot text-to-speech aims at synthesizing voices with unseen speech prompts. Previous large-scale multispeaker TTS models have successfully achieved this goal with an enrolled recording within 10 seconds. However, most of them are designed to utilize only short speech prompts. The limited information in short speech prompts significantly hinders the performance of fine-grained identity imitation. In this paper, we introduce Mega-TTS 2, a generic zero-shot multispeaker TTS model that is capable of synthesizing speech for unseen speakers with arbitrary-length prompts. Specifically, we 1) design a multi-reference timbre encoder to extract timbre information from multiple reference speeches; 2) and train a prosody language model with arbitrary-length speech prompts; With these designs, our model is suitable for prompts of different lengths, which extends the upper bound of speech quality for zero-shot text-to-speech. Besides arbitrary-length prompts, we introduce arbitrary-source prompts, which leverages the probabilities derived from multiple P-LLM outputs to produce expressive and controlled prosody. Furthermore, we propose a phoneme-level auto-regressive duration model to introduce in-context learning capabilities to duration modeling. Experiments demonstrate that our method could not only synthesize identity-preserving speech with a short prompt of an unseen speaker but also achieve improved performance with longer speech prompts. Audio samples can be found in https://mega-tts.github.io/mega2_demo/.
【3】 Reproducing the Velocity Vectors in the Listening Region标题：在监听区域中再现速度矢量链接：https://arxiv.org/abs/2307.07200
作者：Jiarui Wang,Thushara Abhayapala,Jihui Aimee Zhang,Prasanga Samarasinghe机构：∗The Australian National University, †University of Southampton备注：T摘要：提出了一种基于球面听音区域速度矢量匹配的声场再现算法。使用声场平移的概念，在一个球形区域中的速度矢量的球谐系数推导出所需的压力分布。期望的压力分布可以对应于诸如平面波和点源之类的源，或者从使用球形麦克风阵列的测量获得。不同于先前的工作，其中速度矢量仅在收听区域的边界上或在离散的甜蜜点处被控制，该工作直接操纵整个收听区域中的速度矢量，这有望改善在低频下对期望声场的感知。摘要：This paper proposes a sound field reproduction algorithm based on matching the velocity vectors in a spherical listening region. Using the concept of sound field translation, the spherical harmonic coefficients of the velocity vectors in a spherical region are derived from the desired pressure distribution. The desired pressure distribution can either correspond to sources such as plane waves and point sources, or be obtained from measurements using a spherical microphone array. Unlike previous work in which the velocity vectors are only controlled on the boundary of the listening region or at discrete sweet spots, this work directly manipulates the velocity vectors in the whole listening region, which is expected to improve the perception of the desired sound field at low frequencies.
【4】 Low Rank Properties for Estimating Microphones Start Time and Sources  Emission Time标题：估计麦克风起始时间和源发射时间的低秩性链接：https://arxiv.org/abs/2307.07096
作者：Faxian Cao,Yongqiang Cheng,Adil Mehmood Khan,Zhijing Yang,Yingxiu Chang备注：13 pages for main content; 9 pages for proof of proposed low rank properties; 13 figures摘要：关于麦克风记录开始时间和源发射时间的未知定时信息的缺乏在包括联合麦克风和源定位的若干应用中提出了挑战。与传统的直接估计未知定时信息的优化方法相比，低秩性质（LRP）包含一个额外的低秩结构，便于对未知定时信息进行线性约束，以形成相应的低秩结构信息，从而在适当初始化的情况下实现未知定时信息的全局最优解。然而，未知定时信息的初始化是随机的，导致用于估计未知定时信息的局部最小值。在本文中，我们提出了一种组合的低秩近似方法，以减轻随机初始化的影响，对未知的定时信息的估计。我们定义了三个新的变种LRP支持的证明，允许未知的定时信息，以受益于更低的秩结构信息。然后，通过利用低秩结构信息从LRP和LRP的建议的变体，四个线性约束的未知定时信息。最后，我们使用所提出的组合低秩近似算法，通过四个可用的线性约束，获得全局最优解的未知定时信息。实验结果表明，我们的方法相比，国家的最先进的方法在恢复率（任何配置的成功初始化的数量），收敛率（成功恢复的配置的数量），和未知的定时信息的估计误差的优越性能。摘要：The absence of unknown timing information about the microphones recording start time and the sources emission time presents a challenge in several applications, including joint microphones and sources localization. Compared with traditional optimization methods that try to estimate unknown timing information directly, low rank property (LRP) contains an additional low rank structure that facilitates a linear constraint of unknown timing information for formulating corresponding low rank structure information, enabling the achievement of global optimal solutions of unknown timing information with suitable initialization. However, the initialization of unknown timing information is random, resulting in local minimal values for estimation of the unknown timing information. In this paper, we propose a combined low rank approximation method to alleviate the effect of random initialization on the estimation of unknown timing information. We define three new variants of LRP supported by proof that allows unknown timing information to benefit from more low rank structure information. Then, by utilizing the low rank structure information from both LRP and proposed variants of LRP, four linear constraints of unknown timing information are presented. Finally, we use the proposed combined low rank approximation algorithm to obtain global optimal solutions of unknown timing information through the four available linear constraints. Experimental results demonstrate superior performance of our method compared to state-of-the-art approaches in terms of recovery rate (the number of successful initialization for any configuration), convergency rate (the number of successfully recovered configurations), and estimation errors of unknown timing information.
【5】 Controllable Emphasis with zero data for text-to-speech标题：文本到语音转换的零数据可控强调链接：https://arxiv.org/abs/2307.07062
作者：Arnaud Joly,Marco Nicolis,Ekaterina Peterova,Alessandro Lombardi,Ammar Abbas,Arent van Korlaar,Aman Hussain,Parul Sharma,Alexis Moinet,Mateusz Lajszczak,Penny Karanasou,Antonio Bonafonte,Thomas Drugman,Elena Sokolova机构：Amazon, United Kingdom备注：In proceeding of 12th Speech Synthesis Workshop (SSW) 2023摘要：我们提出了一种可扩展的方法来产生高质量的强调文本到语音（TTS），不需要录音或注释。许多TTS模型包括音素持续时间模型。实现强调语音的简单但有效的方法在于增加强调词的预测持续时间。我们表明，这是显着优于频谱图修改技术，提高自然度由$7.3\%$和正确的测试人员的识别强调的词在一个句子中的$40\%$的参考女性EN-美国的声音。我们表明，这种技术显着关闭的差距，需要明确的录音方法。该方法被证明是可扩展的，并在所有四种语言测试（英语，西班牙语，意大利语，德语），不同的声音和多种说话风格的首选。摘要：We present a scalable method to produce high quality emphasis for text-to-speech (TTS) that does not require recordings or annotations. Many TTS models include a phoneme duration model. A simple but effective method to achieve emphasized speech consists in increasing the predicted duration of the emphasised word. We show that this is significantly better than spectrogram modification techniques improving naturalness by $7.3\%$ and correct testers' identification of the emphasized word in a sentence by $40\%$ on a reference female en-US voice. We show that this technique significantly closes the gap to methods that require explicit recordings. The method proved to be scalable and preferred in all four languages tested (English, Spanish, Italian, German), for different voices and multiple speaking styles.
【6】 Towards spoken dialect identification of Irish标题：爱尔兰人的口语方言识别研究链接：https://arxiv.org/abs/2307.07436
作者：Liam Lonergan,Mengjie Qian,Neasa Ní Chiaráin,Christer Gobl,Ailbhe Ní Chasaide机构：Phonetics and Speech Laboratory, School of Linguistics, Speech and Communication Sciences, Trinity College Dublin, Ireland, Engineering Department, Cambridge University, UK备注：Accepted to Interspeech 2023 Workshop of the 2nd Annual Meeting of the Special Interest Group of Under-resourced Languages Workshop, Dublin (SiGUL)摘要：爱尔兰语有着丰富的方言和口音。这增加了为低资源语言创建语音识别系统的难度，因为这样的系统必须用有限的语料库应对高度的可变性。最近的一项研究调查方言偏见在爱尔兰ASR发现，均衡的培训语料库引起了不平等的方言性能，与阿尔斯特方言的性能始终比康诺特或明斯特方言差。出于这一动机，本实验研究口语方言识别爱尔兰语，以期将这样一个系统的语音识别管道。两个声学分类模型进行了测试，XLS-R和ECAPA-TDNN，结合基于文本的分类器使用预训练的爱尔兰语BERT模型。ECAPA-TDNN，特别是在VoxLingua 107数据集上进行语言识别预训练的模型，总体表现最好，准确率为73%。通过将模型的输出与基于文本的模型融合，这一比例进一步提高到76%。阿尔斯特方言是最准确的识别，准确率为94%，但该模型难以消除康诺特和明斯特方言之间的歧义，这表明可能需要更细致的方法来区分爱尔兰方言。摘要：The Irish language is rich in its diversity of dialects and accents. This compounds the difficulty of creating a speech recognition system for the low-resource language, as such a system must contend with a high degree of variability with limited corpora. A recent study investigating dialect bias in Irish ASR found that balanced training corpora gave rise to unequal dialect performance, with performance for the Ulster dialect being consistently worse than for the Connacht or Munster dialects. Motivated by this, the present experiments investigate spoken dialect identification of Irish, with a view to incorporating such a system into the speech recognition pipeline. Two acoustic classification models are tested, XLS-R and ECAPA-TDNN, in conjunction with a text-based classifier using a pretrained Irish-language BERT model. The ECAPA-TDNN, particularly a model pretrained for language identification on the VoxLingua107 dataset, performed best overall, with an accuracy of 73%. This was further improved to 76% by fusing the model's outputs with the text-based model. The Ulster dialect was most accurately identified, with an accuracy of 94%, however the model struggled to disambiguate between the Connacht and Munster dialects, suggesting a more nuanced approach may be necessary to robustly distinguish between the dialects of Irish.
【7】 Real-time Percussive Technique Recognition and Embedding Learning for  the Acoustic Guitar标题：吉他敲击技术的实时识别与嵌入学习链接：https://arxiv.org/abs/2307.07426
作者：Andrea Martelloni,Andrew P McPherson,Mathieu Barthet机构：Queen Mary University, of London, Imperial College备注：Accepted at the 24th Int. Society for Music Information Retrieval Conf., Milan, Italy, 2023摘要：实时音乐信息检索（RT-MIR）在增强传统乐器的性能方面具有很大的潜力。我们开发的RT-MIR技术旨在增强打击指法，它融合了原声吉他演奏与吉他身体打击。我们为增强仪器性能的RT-MIR系统制定了几个设计目标：（i）因果约束，（ii）感知上可忽略的动作-声音延迟，（iii）控制亲密度支持，（iv）合成控制支持。我们提出并评估实时吉他身体打击乐识别和嵌入学习技术的基础上卷积神经网络（CNN）和CNN联合训练变分自动编码器（VAE）。我们介绍了一个分类的吉他身体打击的手部分和位置的基础上。我们遵循一个跨数据集的评估方法，收集三个数据集标记根据分类。模型的嵌入质量使用KL-散度跨对应于不同分类类的分布进行评估。结果表明，网络是强大的分类器，特别是在简化的2类识别任务中，与CNN相比，VAE产生了改进的类分离，这一点可以通过增加分布中的KL散度来证明。我们认为，VAE嵌入质量可以支持控制亲密度和丰富的互动时，潜在空间的参数被用来控制外部合成引擎。已经确定了围绕不同数据集的推广的进一步设计挑战。摘要：Real-time music information retrieval (RT-MIR) has much potential to augment the capabilities of traditional acoustic instruments. We develop RT-MIR techniques aimed at augmenting percussive fingerstyle, which blends acoustic guitar playing with guitar body percussion. We formulate several design objectives for RT-MIR systems for augmented instrument performance: (i) causal constraint, (ii) perceptually negligible action-to-sound latency, (iii) control intimacy support, (iv) synthesis control support. We present and evaluate real-time guitar body percussion recognition and embedding learning techniques based on convolutional neural networks (CNNs) and CNNs jointly trained with variational autoencoders (VAEs). We introduce a taxonomy of guitar body percussion based on hand part and location. We follow a cross-dataset evaluation approach by collecting three datasets labelled according to the taxonomy. The embedding quality of the models is assessed using KL-Divergence across distributions corresponding to different taxonomic classes. Results indicate that the networks are strong classifiers especially in a simplified 2-class recognition task, and the VAEs yield improved class separation compared to CNNs as evidenced by increased KL-Divergence across distributions. We argue that the VAE embedding quality could support control intimacy and rich interaction when the latent space's parameters are used to control an external synthesis engine. Further design challenges around generalisation to different datasets have been identified.
【8】 Sumformer: A Linear-Complexity Alternative to Self-Attention for Speech  Recognition标题：Sumform：语音识别中自我注意的一种线性复杂性替代方法链接：https://arxiv.org/abs/2307.07421
作者：Titouan Parcollet,Rogier van Dalen,Shucong Zhang,Sourav Bhattacharya机构：Samsung AI Center, Cambridge, United Kingdom备注：Submitted to NeurIPS 2023摘要：现代语音识别系统依赖于自我注意。不幸的是，令牌与自我注意力的混合在语音话语的长度上需要二次时间，从而减慢推理以及训练并增加存储器消耗。已经开发了用于ASR的自我注意力的更便宜的替代方案，但未能始终达到相同的准确性水平。然而，在实践中，经训练的语音识别器的自我注意力权重采取随时间的全局平均值的形式。因此，本文提出了一种线性时间的替代自我注意的语音识别。它总结了一个完整的话语与平均向量的所有时间步长。然后将该单个摘要与特定时间的信息组合。我们称这种方法为“汇总混合”。在最先进的ASR模型中引入摘要混合，可以保持或超过以前的语音识别性能，同时将训练和推理时间降低高达27%，并将内存预算减少两倍。摘要：Modern speech recognition systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference as well as training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but fail to consistently reach the same level of accuracy. In practice, however, the self-attention weights of trained speech recognizers take the form of a global average over time. This paper, therefore, proposes a linear-time alternative to self-attention for speech recognition. It summarises a whole utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method ``Summary Mixing''. Introducing Summary Mixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while lowering the training and inference times by up to 27% and reducing the memory budget by a factor of two.
【9】 Phoneme-retrieval; voice recognition; vowels recognition标题：音素检索；语音识别；元音识别链接：https://arxiv.org/abs/2307.07407
作者：Brunello Tirozzi,Orchidea Maria Lecian机构：Sapienza University of Rome, Department of Civil and Industrial Engineering, Via Eudossiana,- , Rome, Italy;, Department of Information Engineering, Sapienza Unviersity of Rome, Physics Department, Piazzale Aldo Mororo,- , Rome, Italy;, Key-words备注：10 pages摘要：由于网络的特殊构造方式，提出了一种音素检索技术。给出了神经元的初始集合。这些神经元的数量近似等于数据的典型结构的数量。例如，如果网络是为语音检索而构建的，则神经元的数量必须等于特定人所属的社会群体所讲语言的字母表的特征音素的数量。通常，这个任务非常复杂，并且网络可能严重依赖于用于学习的样本。如果网络是为图像检索而构建的，那么只有当要检索的数据属于特定的图像集时，它才能工作。如果网络是为语音识别而构建的，它只适用于某些特定的单词集。一个典型的例子是用于飞机飞行的单词。例如，如果使用合适的学习过程，则像“飞机应该向东转弯120度”这样的命令可以容易地被网络识别。摘要：A phoneme-retrieval technique is proposed, which is due to the particular way of the construction of the network. An initial set of neurons is given. The number of these neurons is approximately equal to the number of typical structures of the data. For example if the network is built for voice retrieval then the number of neurons must be equal to the number of characteristic phonemes of the alphabet of the language spoken by the social group to which the particular person belongs. Usually this task is very complicated and the network can depend critically on the samples used for the learning. If the network is built for image retrieval then it works only if the data to be retrieved belong to a particular set of images. If the network is built for voice recognition it works only for some particular set of words. A typical example is the words used for the flight of airplanes. For example a command like the "airplane should make a turn of 120 degrees towards the east" can be easily recognized by the network if a suitable learning procedure is used.
【10】 Towards dialect-inclusive recognition in a low-resource language: are  balanced corpora the answer?标题：在低资源语言中实现包括方言的识别：平衡语料库是答案吗？链接：https://arxiv.org/abs/2307.07295
作者：Liam Lonergan,Mengjie Qian,Neasa Ní Chiaráin,Christer Gobl,Ailbhe Ní Chasaide机构：Phonetics and Speech Laboratory, School of Linguistics, Speech and Communication Sciences, Trinity College Dublin, Ireland, Engineering Department, Cambridge University, UK备注：Accepted to Interspeech 2023, Dublin摘要：ASR系统通常是为口语“标准”而构建的，并且它们的性能对于非标准方言/变体而下降。这对于像爱尔兰语这样的语言来说是一个问题，那里没有单一的口语标准，而是三种主要的方言：阿尔斯特（Ul）、康诺特（Co）和明斯特（Mu）。作为一个诊断，以量化的效果，说话者的方言识别性能，12 ASR系统进行了训练，首先使用基线方言平衡的训练语料库，然后使用修改后的版本的基线语料库，其中方言特定的材料被减去或添加。结果表明，方言平衡语料库不产生类似的表现跨方言：UI方言始终表现不佳，而Mu产生最低的WER。Co话与Mu话有着密切的关系，但这种关系并不对称。这些结果将指导未来的语料库收集和系统建设策略，以优化跨方言的性能公平。摘要：ASR systems are generally built for the spoken 'standard', and their performance declines for non-standard dialects/varieties. This is a problem for a language like Irish, where there is no single spoken standard, but rather three major dialects: Ulster (Ul), Connacht (Co) and Munster (Mu). As a diagnostic to quantify the effect of the speaker's dialect on recognition performance, 12 ASR systems were trained, firstly using baseline dialect-balanced training corpora, and then using modified versions of the baseline corpora, where dialect-specific materials were either subtracted or added. Results indicate that dialect-balanced corpora do not yield a similar performance across the dialects: the Ul dialect consistently underperforms, whereas Mu yields lowest WERs. There is a close relationship between Co and Mu dialects, but one that is not symmetrical. These results will guide future corpus collection and system building strategies to optimise for cross-dialect performance equity.
【11】 Replay to Remember: Continual Layer-Specific Fine-tuning for German  Speech Recognition标题：回放到记忆：针对德语语音识别的连续层特定微调链接：https://arxiv.org/abs/2307.07280
作者：Theresa Pekarek Rosin,Stefan Wermter机构：Knowledge Technology, Department of Informatics, University of Hamburg, Vogt-Koelln-Str. , Hamburg, Germany, www.knowledge-technology.info备注：13 pages, 7 figures, submitted to ICANN 2023摘要：虽然自动语音识别（ASR）模型随着无监督或自监督训练技术的引入已经显示出显著的进步，但这些改进仍然仅限于语言和说话者的一部分。迁移学习使大规模多语言模型不仅能够适应低资源语言，而且能够适应更具体的说话者群体。然而，对来自新域的数据进行微调通常伴随着原始域的性能下降。因此，在我们的实验中，我们研究了大规模ASR模型的性能可以在较小的域中近似，使用我们自己的德国高级语音命令（SVC-de）数据集，以及通过在训练期间选择性地冻结模型的部分可以保留多少一般语音识别性能。为了进一步提高ASR模型对微调域之外的词汇和说话者的鲁棒性，我们应用经验重放进行持续学习。通过仅添加来自原始域的一小部分数据，我们能够在新域上达到低于5%的字错误率（WERs），同时在可接受的WERs下稳定一般语音识别的性能。摘要：While Automatic Speech Recognition (ASR) models have shown significant advances with the introduction of unsupervised or self-supervised training techniques, these improvements are still only limited to a subsection of languages and speakers. Transfer learning enables the adaptation of large-scale multilingual models to not only low-resource languages but also to more specific speaker groups. However, fine-tuning on data from new domains is usually accompanied by a decrease in performance on the original domain. Therefore, in our experiments, we examine how well the performance of large-scale ASR models can be approximated for smaller domains, with our own dataset of German Senior Voice Commands (SVC-de), and how much of the general speech recognition performance can be preserved by selectively freezing parts of the model during training. To further increase the robustness of the ASR model to vocabulary and speakers outside of the fine-tuned domain, we apply Experience Replay for continual learning. By adding only a fraction of data from the original domain, we are able to reach Word-Error-Rates (WERs) below 5\% on the new domain, while stabilizing performance for general speech recognition at acceptable WERs.
【12】 AudioInceptionNeXt: TCL AI LAB Submission to EPIC-SOUND  Audio-Based-Interaction-Recognition Challenge 2023标题：AudioInceptionNeXt：TCL AI实验室提交给EPIC-声音-基于音频的交互-识别挑战赛2023链接：https://arxiv.org/abs/2307.07265
作者：Kin Wai Lau,Yasar Abbas Ur Rehman,Yuyang Xie,Lan Ma机构：TCL AI Lab摘要：本报告介绍了我们提交给2023年Epic-Kitchen EPIC-SOUNDS基于音频的交互识别挑战赛的技术细节。任务是学习从音频样本到其对应的动作标签的映射。为了实现这一目标，我们提出了一个简单而有效的单流基于CNN的架构，称为AudioInceptionNeXt，它对音频样本的时频对数梅尔频谱图进行操作。受InceptionNeXt设计的启发，我们在AudioInceptionNeXt块中提出了并行多尺度深度可分离卷积核，这使得模型能够更有效地学习时间和频率信息。大规模可分离核函数能够捕获活动的长持续时间和全局频率语义信息，而小规模可分离核函数能够捕获活动的短持续时间和频率信息的局部细节。我们的方法在挑战测试集上达到了55.43%的前1名准确率，在公共排行榜上排名第一。代码可在https://github.com/StevenLauHKHK/AudioInceptionNeXt.git上匿名获取。摘要：This report presents the technical details of our submission to the 2023 Epic-Kitchen EPIC-SOUNDS Audio-Based Interaction Recognition Challenge. The task is to learn the mapping from audio samples to their corresponding action labels. To achieve this goal, we propose a simple yet effective single-stream CNN-based architecture called AudioInceptionNeXt that operates on the time-frequency log-mel-spectrogram of the audio samples. Motivated by the design of the InceptionNeXt, we propose parallel multi-scale depthwise separable convolutional kernels in the AudioInceptionNeXt block, which enable the model to learn the time and frequency information more effectively. The large-scale separable kernels capture the long duration of activities and the global frequency semantic information, while the small-scale separable kernels capture the short duration of activities and local details of frequency information. Our approach achieved 55.43% of top-1 accuracy on the challenge test set, ranked as 1st on the public leaderboard. Codes are available anonymously at https://github.com/StevenLauHKHK/AudioInceptionNeXt.git.
【13】 Leveraging Pretrained ASR Encoders for Effective and Efficient  End-to-End Speech Intent Classification and Slot Filling标题：利用预先训练的ASR编码器实现有效且高效的端到端语音意图分类和空位填充链接：https://arxiv.org/abs/2307.07057
作者：He Huang,Jagadeesh Balam,Boris Ginsburg机构：NVIDIA, USA备注：INTERSPEECH 2023摘要：我们研究了语音意图分类和时隙填充（SICSF），提出使用在语音识别（ASR）上预训练的编码器来初始化端到端（E2 E）Conformer-Transformer模型，该模型在SLURP数据集上实现了新的最先进的结果，具有90.14%的意图准确度和82.27%的SLURP-F1。我们将我们的模型与在自监督学习（SSL）上预训练的编码器进行比较，并表明ASR预训练比SSL对SICSF更有效。为了探索参数效率，我们冻结了编码器并添加了适配器模块，并表明参数效率只能通过ASR预训练的编码器实现，而SSL编码器需要完全微调才能实现可比的结果。此外，我们提供了一个深入的比较端到端模型与级联模型（ASR+NLU），并表明，E2 E模型优于级联模型，除非提供一个Oracle ASR模型。最后但并非最不重要的是，我们的模型是第一个E2 E模型，实现了与Oracle ASR级联模型相同的性能。代码、检查点和配置可用。摘要：We study speech intent classification and slot filling (SICSF) by proposing to use an encoder pretrained on speech recognition (ASR) to initialize an end-to-end (E2E) Conformer-Transformer model, which achieves the new state-of-the-art results on the SLURP dataset, with 90.14% intent accuracy and 82.27% SLURP-F1. We compare our model with encoders pretrained on self-supervised learning (SSL), and show that ASR pretraining is much more effective than SSL for SICSF. To explore parameter efficiency, we freeze the encoder and add Adapter modules, and show that parameter efficiency is only achievable with an ASR-pretrained encoder, while the SSL encoder needs full finetuning to achieve comparable results. In addition, we provide an in-depth comparison on end-to-end models versus cascading models (ASR+NLU), and show that E2E models are better than cascaded models unless an oracle ASR model is provided. Last but not least, our model is the first E2E model that achieves the same performance as cascading models with oracle ASR. Code, checkpoints and configs are available.

机器翻译，仅供参考