今日论文合集：cs.SD语音12篇，eess.AS音频处理17篇。
cs.SD语音
【1】 Robust Wake-Up Word Detection by Two-stage Multi-resolution Ensembles标题：基于两级多分辨率集成的稳健唤醒词检测链接：https://arxiv.org/abs/2310.11379
作者：Fernando López,Jordi Luque,Carlos Segura,Pablo Gómez备注：5 pages, 3 figures摘要：基于语音的界面依赖于唤醒词机制来发起与设备的通信。然而，实现稳健、节能和快速的检测仍然是一个挑战。本文解决了这些实际的生产需求，通过增强数据的时间对齐和使用检测的基础上，两个阶段的多分辨率。它采用两种模式：用于音频流实时处理的轻量级设备上模型和服务器端的验证模型，该验证模型是改进检测的异构架构的集合。该方案允许两个操作点的优化。为了保护隐私，音频功能被发送到云端，而不是原始音频。该研究调查了用于特征提取的不同参数配置，以选择一个用于设备上检测，另一个用于验证模型。此外，十三个不同的音频分类器进行了比较，在性能和推理时间。建议的集成优于我们更强的分类器在每一个噪声条件。摘要：Voice-based interfaces rely on a wake-up word mechanism to initiate communication with devices. However, achieving a robust, energy-efficient, and fast detection remains a challenge. This paper addresses these real production needs by enhancing data with temporal alignments and using detection based on two phases with multi-resolution. It employs two models: a lightweight on-device model for real-time processing of the audio stream and a verification model on the server-side, which is an ensemble of heterogeneous architectures that refine detection. This scheme allows the optimization of two operating points. To protect privacy, audio features are sent to the cloud instead of raw audio. The study investigated different parametric configurations for feature extraction to select one for on-device detection and another for the verification model. Furthermore, thirteen different audio classifiers were compared in terms of performance and inference time. The proposed ensemble outperforms our stronger classifier in every noise condition.
【2】 High-Fidelity Noise Reduction with Differentiable Signal Processing标题：利用可差分信号处理实现高保真降噪链接：https://arxiv.org/abs/2310.11364
作者：Christian J. Steinmetz,Thomas Walther,Joshua D. Reiss备注：Accepted for publication at the 155th Convention of the Audio Engineering Society摘要：基于深度学习的降噪技术在提高录制语音的整体质量方面表现出令人印象深刻的性能。虽然这些方法是高性能的，但由于许多因素，它们在音频工程中的应用可能受到限制。这些缺点包括仅对语音进行操作而不支持音乐、缺乏实时能力、缺乏可解释的控制参数、以较低的采样率进行操作以及引入伪像的趋势。另一方面，基于信号处理的降噪算法提供了对广泛内容的细粒度控制和操作，然而，它们通常需要手动操作才能达到最佳效果。为了解决这两种方法的局限性，在这项工作中，我们介绍了一种方法，利用基于信号处理的去噪器，当与神经网络控制器相结合时，可以对语音和音乐信号进行全自动和高保真降噪。我们评估我们提出的方法与客观指标和感知听力测试。我们的评估表明，语音增强模型可以扩展到音乐，但是训练模型只去除静态噪声是至关重要的。此外，我们提出的方法实现了与深度学习模型相当的性能，同时在某些情况下效率更高，引入的工件更少。听力示例可在https://tape.it/research/denoiser网站上找到。摘要：Noise reduction techniques based on deep learning have demonstrated impressive performance in enhancing the overall quality of recorded speech. While these approaches are highly performant, their application in audio engineering can be limited due to a number of factors. These include operation only on speech without support for music, lack of real-time capability, lack of interpretable control parameters, operation at lower sample rates, and a tendency to introduce artifacts. On the other hand, signal processing-based noise reduction algorithms offer fine-grained control and operation on a broad range of content, however, they often require manual operation to achieve the best results. To address the limitations of both approaches, in this work we introduce a method that leverages a signal processing-based denoiser that when combined with a neural network controller, enables fully automatic and high-fidelity noise reduction on both speech and music signals. We evaluate our proposed method with objective metrics and a perceptual listening test. Our evaluation reveals that speech enhancement models can be extended to music, however training the model to remove only stationary noise is critical. Furthermore, our proposed approach achieves performance on par with the deep learning models, while being significantly more efficient and introducing fewer artifacts in some cases. Listening examples are available online at https://tape.it/research/denoiser .
【3】 Serenade: A Model for Human-in-the-loop Automatic Chord Estimation标题：小夜曲：一种人在环中的和弦自动估计模型链接：https://arxiv.org/abs/2310.11165
作者：Hendrik Vincent Koops,Gianluca Micchi,Ilaria Manco,Elio Quinton备注：Accepted at MMRP23. 7 pages, 5 figures, 2 tables摘要：计算和声分析对于自动切分、语料库分析和自动和弦标签估计等MIR任务非常重要。然而，最近对音乐和谐的模糊性的研究，导致有限的评分员之间的协议，已经明显地表明，有一个玻璃天花板的共同指标，如准确性。通常，这些问题要么在训练数据本身中通过创建多数规则注释来解决，要么在训练阶段通过学习软目标来解决。我们提出了一种新的替代方法，其中人类和自回归模型一起共同创建音轨的谐波注释。在自动生成和声预测后，人类稀疏地注释具有低模型置信度的部分，然后模型在人类指导下调整其预测。我们在流行音乐的数据集上评估了我们的模型，我们表明，通过这种人在回路中的方法，谐波分析性能比仅模型的方法有所提高。人类的贡献被模型的第二个约束预测放大。摘要：Computational harmony analysis is important for MIR tasks such as automatic segmentation, corpus analysis and automatic chord label estimation. However, recent research into the ambiguous nature of musical harmony, causing limited inter-rater agreement, has made apparent that there is a glass ceiling for common metrics such as accuracy. Commonly, these issues are addressed either in the training data itself by creating majority-rule annotations or during the training phase by learning soft targets. We propose a novel alternative approach in which a human and an autoregressive model together co-create a harmonic annotation for an audio track. After automatically generating harmony predictions, a human sparsely annotates parts with low model confidence and the model then adjusts its predictions following human guidance. We evaluate our model on a dataset of popular music and we show that, with this human-in-the-loop approach, harmonic analysis performance improves over a model-only approach. The human contribution is amplified by the second, constrained prediction of the model.
【4】 Leveraging Content-based Features from Multiple Acoustic Models for  Singing Voice Conversion标题：利用来自多个声学模型的基于内容的特征进行歌唱语音转换链接：https://arxiv.org/abs/2310.11160
作者：Xueyao Zhang,Yicheng Gu,Haopeng Chen,Zihao Fang,Lexiao Zou,Liumeng Xue,Zhizheng Wu摘要：歌声转换（SVC）是一种使任意歌手演唱任意歌曲的技术。为了实现这一点，重要的是从源音频中获得说话者不可知的表示，这是一项具有挑战性的任务。常见的解决方案是提取基于内容的特征（例如，PPG）。然而，声学模型的选择是广泛而多样的。来自不同声学模型的内容特征的特征是什么，以及集成多个内容特征是否可以互相帮助，还有待探索。受此启发，本研究调查了三个不同的内容特征，分别来自WeNet，Whisper和ContentVec。我们探讨他们的互补作用，可理解性，韵律，转换相似性SVC。通过将多个内容特征与基于扩散的SVC模型相结合，我们的SVC系统在客观和主观评价上都实现了优于单一内容特征源的转换性能。我们的演示页面和代码可以在https://www.zhangxueyao.com/data/MultipleContentsSVC/index.html上找到。摘要：Singing voice conversion (SVC) is a technique to enable an arbitrary singer to sing an arbitrary song. To achieve that, it is important to obtain speaker-agnostic representations from source audio, which is a challenging task. A common solution is to extract content-based features (e.g., PPGs) from a pretrained acoustic model. However, the choices for acoustic models are vast and varied. It is yet to be explored what characteristics of content features from different acoustic models are, and whether integrating multiple content features can help each other. Motivated by that, this study investigates three distinct content features, sourcing from WeNet, Whisper, and ContentVec, respectively. We explore their complementary roles in intelligibility, prosody, and conversion similarity for SVC. By integrating the multiple content features with a diffusion-based SVC model, our SVC system achieves superior conversion performance on both objective and subjective evaluation in comparison to a single source of content features. Our demo page and code can be available https://www.zhangxueyao.com/data/MultipleContentsSVC/index.html.
【5】 Long-form Simultaneous Speech Translation: Thesis Proposal标题：长篇同声传译：论文建议链接：https://arxiv.org/abs/2310.11141
作者：Peter Polák备注：IJCNLP-AACL SRW 2023 - camera-ready version摘要：同步语音翻译（SST）旨在提供实时的口语翻译，甚至在说话者完成句子之前。传统上，SST主要由级联系统解决，这些系统将任务分解为子任务，包括语音识别，分割和机器翻译。然而，深度学习的出现引发了人们对端到端（E2E）系统的极大兴趣。然而，在目前的文献中报道的大多数E2E SST方法的一个主要限制是，它们假设源语音被预先分割成句子，这是实际的，现实世界的应用的一个重大障碍。本论文提出了端到端的同步语音翻译，特别是在长格式的设置，即，没有预先分割。我们提出了E2E SST的最新进展的调查，评估SST的主要障碍及其与长期情景的相关性，并提出了应对这些挑战的方法。摘要：Simultaneous speech translation (SST) aims to provide real-time translation of spoken language, even before the speaker finishes their sentence. Traditionally, SST has been addressed primarily by cascaded systems that decompose the task into subtasks, including speech recognition, segmentation, and machine translation. However, the advent of deep learning has sparked significant interest in end-to-end (E2E) systems. Nevertheless, a major limitation of most approaches to E2E SST reported in the current literature is that they assume that the source speech is pre-segmented into sentences, which is a significant obstacle for practical, real-world applications. This thesis proposal addresses end-to-end simultaneous speech translation, particularly in the long-form setting, i.e., without pre-segmentation. We present a survey of the latest advancements in E2E SST, assess the primary obstacles in SST and its relevance to long-form scenarios, and suggest approaches to tackle these challenges.
【6】 VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System标题：VoxArabica：一种稳健的方言感知阿拉伯语语音识别系统链接：https://arxiv.org/abs/2310.11069
作者：Abdul Waheed,Bashar Talafha,Peter Suvellin,Abdelrahman Elmadney,Muhammad Abdul-Mageed备注：Accepted at ArabicNLP conference co-located with EMNLP'23摘要：阿拉伯语是一种复杂的语言，有许多变体和方言，全世界有超过4.5亿人使用。由于语言的多样性和变化，这是具有挑战性的，建立一个强大的和通用的阿拉伯语自动语音识别系统。在这项工作中，我们通过开发和演示一个名为VoxArabica的系统来解决这一差距，该系统用于方言识别（DID）以及阿拉伯语的自动语音识别（ASR）。我们训练了各种各样的模型，如HuBERT（DID），Whisper和XLS-R（ASR），在有监督的环境中进行阿拉伯语DID和ASR任务。我们的DID模型经过训练，除了MSA之外，还可以识别17种不同的方言。我们在MSA、埃及、摩洛哥和混合数据上微调我们的ASR模型。此外，对于ASR中的其余方言，我们提供了在zero-shot设置中选择各种型号（如Whisper和MMS）的选项。我们将这些模型集成到一个具有多种功能的Web界面中，例如音频录制，文件上传，模型选择以及为不正确的输出提高标志的选项。总的来说，我们相信VoxArabica将对关注阿拉伯语研究的广泛受众有用。我们的系统目前在https://cdce-206-12-100-168.ngrok.io/上运行。摘要：Arabic is a complex language with many varieties and dialects spoken by over 450 millions all around the world. Due to the linguistic diversity and variations, it is challenging to build a robust and generalized ASR system for Arabic. In this work, we address this gap by developing and demoing a system, dubbed VoxArabica, for dialect identification (DID) as well as automatic speech recognition (ASR) of Arabic. We train a wide range of models such as HuBERT (DID), Whisper, and XLS-R (ASR) in a supervised setting for Arabic DID and ASR tasks. Our DID models are trained to identify 17 different dialects in addition to MSA. We finetune our ASR models on MSA, Egyptian, Moroccan, and mixed data. Additionally, for the remaining dialects in ASR, we provide the option to choose various models such as Whisper and MMS in a zero-shot setting. We integrate these models into a single web interface with diverse features such as audio recording, file upload, model selection, and the option to raise flags for incorrect outputs. Overall, we believe VoxArabica will be useful for a wide range of audiences concerned with Arabic research. Our system is currently running at https://cdce-206-12-100-168.ngrok.io/.
【7】 Lyricist-Singer Entropy Affects Lyric-Lyricist Classification  Performance标题：词人-歌手信息熵影响词人分类性能链接：https://arxiv.org/abs/2310.11035
作者：Mitsuki Morita,Masato Kikuchi,Tadachika Ozono备注：The 10th International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA 2023)摘要：歌词是音乐的重要组成部分，但关于歌词作者特征的音乐信息处理研究却很少。因为这些特征对于音乐应用（例如推荐）可能是有价值的，所以它们值得进一步研究。我们考虑了一种潜在的方法，提取代表歌词的歌词作者的特点。因为这些特征必须在提取之前被识别，所以我们专注于具有容易识别特征的抒情诗人。我们认为，这是理想的歌手表演独特的歌曲，共享特定的歌手的某些特征。因此，我们假设作词人解释了他们为之写歌词的歌手的独特特征。换句话说，歌词-作词人分类性能或从歌词捕捉作词人的特征的容易程度可以取决于歌手的种类。在这项研究中，我们观察到一个歌词歌手熵或各种歌手与一个单一的歌词和歌词歌手分类性能之间的关系。作为一个例子，当作词人只为一个歌手写歌词时，作词人-歌手熵是最小的。在我们的实验中，我们将抒情诗人分为五组的抒情诗人歌手熵，并评估各组内的抒情诗人分类性能。因此，最好的F1得分是获得最低作词歌手熵的组。我们的研究结果表明，进一步分析的功能，有助于抒情歌手分类性能的最低抒情歌手歌手熵组可能会提高抒情歌手的特征提取任务。摘要：Although lyrics represent an essential component of music, few music information processing studies have been conducted on the characteristics of lyricists. Because these characteristics may be valuable for musical applications, such as recommendations, they warrant further study. We considered a potential method that extracts features representing the characteristics of lyricists from lyrics. Because these features must be identified prior to extraction, we focused on lyricists with easily identifiable features. We believe that it is desirable for singers to perform unique songs that share certain characteristics specific to the singer. Accordingly, we hypothesized that lyricists account for the unique characteristics of the singers they write lyrics for. In other words, lyric-lyricist classification performance or the ease of capturing the features of a lyricist from the lyrics may depend on the variety of singers. In this study, we observed a relationship between lyricist-singer entropy or the variety of singers associated with a single lyricist and lyric-lyricist classification performance. As an example, the lyricist-singer entropy is minimal when the lyricist writes lyrics for only one singer. In our experiments, we grouped lyricists among five groups in terms of lyricist-singer entropy and assessed the lyric-lyricist classification performance within each group. Consequently, the best F1 score was obtained for the group with the lowest lyricist-singer entropy. Our results suggest that further analyses of the features contributing to lyric-lyricist classification performance on the lowest lyricist-singer entropy group may improve the feature extraction task for lyricists.
【8】 Correction Focused Language Model Training for Speech Recognition标题：面向语音识别的纠错语言模型训练链接：https://arxiv.org/abs/2310.11003
作者：Yingyi Ma,Zhe Liu,Ozlem Kalinli摘要：语言模型（LM）已被普遍采用，以提高自动语音识别（ASR）的性能，特别是在领域适应任务。传统的LM训练方法对语料库中的所有单词都一视同仁，导致ASR性能的改善不理想。在这项工作中，我们介绍了一种新的校正重点LM训练方法，其目的是优先考虑ASR易错的话。词级ASR易错性分数表示ASR误识别的可能性，被定义和成形为先验词分布以指导LM训练。为了实现仅文本语料库的校正集中训练，大型语言模型（LLM）通过多任务微调被用作易错性分数预测器和文本生成器。领域自适应任务的实验结果证明了我们提出的方法的有效性。与传统的LM相比，校正集中的训练在足够的文本场景中实现了高达5.5%的字错误率（WER）降低。在文本不足的情况下，使用LLM生成的文本进行LM训练可以实现高达相对13%的WER降低，而校正集中的训练可以进一步获得高达相对6%的WER降低。摘要：Language models (LMs) have been commonly adopted to boost the performance of automatic speech recognition (ASR) particularly in domain adaptation tasks. Conventional way of LM training treats all the words in corpora equally, resulting in suboptimal improvements in ASR performance. In this work, we introduce a novel correction focused LM training approach which aims to prioritize ASR fallible words. The word-level ASR fallibility score, representing the likelihood of ASR mis-recognition, is defined and shaped as a prior word distribution to guide the LM training. To enable correction focused training with text-only corpora, large language models (LLMs) are employed as fallibility score predictors and text generators through multi-task fine-tuning. Experimental results for domain adaptation tasks demonstrate the effectiveness of our proposed method. Compared with conventional LMs, correction focused training achieves up to relatively 5.5% word error rate (WER) reduction in sufficient text scenarios. In insufficient text scenarios, LM training with LLM-generated text achieves up to relatively 13% WER reduction, while correction focused training further obtains up to relatively 6% WER reduction.
【9】 A High Fidelity and Low Complexity Neural Audio Coding标题：一种高保真低复杂度的神经音频编码链接：https://arxiv.org/abs/2310.10992
作者：Wenzhe Liu,Wei Xiao,Meng Wang,Shan Yang,Yupeng Shi,Yuyong Kang,Dan Su,Shidong Shang,Dong Yu摘要：音频编码是实时通信系统中必不可少的模块。由于深度神经网络的强大建模和生成能力，神经音频编解码器可以以低比特率压缩音频样本。为了解决高频表达差和高计算成本和存储消耗，我们提出了一个集成的框架，利用神经网络模型的宽带分量，并采用传统的信号处理压缩高频分量根据心理听觉知识。受听觉感知理论的启发，设计了一种基于感知的损失函数来改进谐波建模。此外，生成对抗网络（GAN）压缩首次提出了神经音频编解码器。我们的方法在主观和客观指标上优于先前的高级神经编解码器，并允许在桌面和移动设备上进行实时推理。摘要：Audio coding is an essential module in the real-time communication system. Neural audio codecs can compress audio samples with a low bitrate due to the strong modeling and generative capabilities of deep neural networks. To address the poor high-frequency expression and high computational cost and storage consumption, we proposed an integrated framework that utilizes a neural network to model wide-band components and adopts traditional signal processing to compress high-band components according to psychological hearing knowledge. Inspired by auditory perception theory, a perception-based loss function is designed to improve harmonic modeling. Besides, generative adversarial network (GAN) compression is proposed for the first time for neural audio codecs. Our method is superior to prior advanced neural codecs across subjective and objective metrics and allows real-time inference on desktop and mobile.
【10】 Spatial HuBERT: Self-supervised Spatial Speech Representation Learning  for a Single Talker from Multi-channel Audio标题：空间休BERT：多声道音频中单个说话人的自监督空间语音表示学习链接：https://arxiv.org/abs/2310.10922
作者：Antoni Dimitriadis,Siqi Pan,Vidhyasaharan Sethu,Beena Ahmed摘要：自监督学习已被用于利用未标记的数据，通过训练表示模型来提高语音系统的准确性和泛化能力。虽然最近的许多研究试图在各种声学领域、语言、模态甚至同时说话者之间产生有效的表示，但这些研究都仅限于单通道音频记录。本文介绍了空间HuBERT，一个自我监督的语音表示模型，学习声学和空间信息有关的一个潜在的嘈杂环境中的单个扬声器，通过使用多通道音频输入。Spatial HuBERT在各种空间下游任务中学习表现优于最先进的单通道语音表示的表示，特别是在混响和嘈杂的环境中。我们还展示了空间休伯特学习的语音定位下游任务的表示的效用。随着这篇论文，我们公开发布了一个新的数据集100 000模拟一阶高保真度立体声房间脉冲响应。摘要：Self-supervised learning has been used to leverage unlabelled data, improving accuracy and generalisation of speech systems through the training of representation models. While many recent works have sought to produce effective representations across a variety of acoustic domains, languages, modalities and even simultaneous speakers, these studies have all been limited to single-channel audio recordings. This paper presents Spatial HuBERT, a self-supervised speech representation model that learns both acoustic and spatial information pertaining to a single speaker in a potentially noisy environment by using multi-channel audio inputs. Spatial HuBERT learns representations that outperform state-of-the-art single-channel speech representations on a variety of spatial downstream tasks, particularly in reverberant and noisy environments. We also demonstrate the utility of the representations learned by Spatial HuBERT on a speech localisation downstream task. Along with this paper, we publicly release a new dataset of 100 000 simulated first-order ambisonics room impulse responses.
【11】 Unsupervised Lead Sheet Generation via Semantic Compression标题：基于语义压缩的无监督引导单生成链接：https://arxiv.org/abs/2310.10772
作者：Zachary Novack,Nikita Srivatsan,Taylor Berg-Kirkpatrick,Julian McAuley摘要：在生成音乐的研究中，铅板已经变得司空见惯，被用作多轨音乐生成和自动编曲等下游任务的初始压缩表示。尽管如此，研究人员在寻找配对的铅表和满分时，经常依靠确定性还原方法（如天际线算法）来生成铅表，很少关注铅表本身的质量以及它们如何准确地反映它们的协调对应物。为了解决这些问题，我们提出了有条件的铅片生成的问题（即。在给定其完整乐谱版本的情况下生成引导表），并且示出了该任务可以被公式化为无监督音乐压缩任务，其中引导表表示乐谱的压缩的潜在版本。我们介绍了一种新的模型，称为铅AE，模型的铅片作为一个离散的子选择的原始序列，使用可微的top-k算子，允许可控的局部稀疏约束。在自动代理任务和直接的人类评价，我们发现，我们的方法改进了确定性的基线，并产生一致的减少大多轨分数。摘要：Lead sheets have become commonplace in generative music research, being used as an initial compressed representation for downstream tasks like multitrack music generation and automatic arrangement. Despite this, researchers have often fallen back on deterministic reduction methods (such as the skyline algorithm) to generate lead sheets when seeking paired lead sheets and full scores, with little attention being paid toward the quality of the lead sheets themselves and how they accurately reflect their orchestrated counterparts. To address these issues, we propose the problem of conditional lead sheet generation (i.e. generating a lead sheet given its full score version), and show that this task can be formulated as an unsupervised music compression task, where the lead sheet represents a compressed latent version of the score. We introduce a novel model, called Lead-AE, that models the lead sheets as a discrete subselection of the original sequence, using a differentiable top-k operator to allow for controllable local sparsity constraints. Across both automatic proxy tasks and direct human evaluations, we find that our method improves upon the established deterministic baseline and produces coherent reductions of large multitrack scores.
【12】 Optimized Tokenization for Transcribed Error Correction标题：用于转录纠错的优化标记化链接：https://arxiv.org/abs/2310.10704
作者：Tomer Wullach,Shlomo E. Chazan摘要：语音识别系统面临的挑战，如发音的变化，不利的音频条件和标记数据的稀缺性，强调了纠正重复错误的后处理步骤的必要性。以前的研究已经表明了采用专用纠错模型的优势，但训练这样的模型需要大量的标记数据，这是不容易获得的。为了克服这一限制，通常使用合成的类似转录的数据，然而，桥接转录错误和合成噪声之间的分布差距并不是微不足道的。在本文中，我们证明了校正模型的性能可以显着提高训练单独使用合成数据。具体而言，我们经验表明：（1）使用从一组转录数据导出的误差分布生成的合成数据优于应用随机扰动的常见方法;（2）对BPE分词器的词汇表应用语言特定的调整，在适应看不见的分布和保留转录错误的知识之间取得平衡。我们展示了这些关键观察的好处，并使用多种语言，语音识别系统和突出的语音识别数据集来评估我们的方法。摘要：The challenges facing speech recognition systems, such as variations in pronunciations, adverse audio conditions, and the scarcity of labeled data, emphasize the necessity for a post-processing step that corrects recurring errors. Previous research has shown the advantages of employing dedicated error correction models, yet training such models requires large amounts of labeled data which is not easily obtained. To overcome this limitation, synthetic transcribed-like data is often utilized, however, bridging the distribution gap between transcribed errors and synthetic noise is not trivial. In this paper, we demonstrate that the performance of correction models can be significantly increased by training solely using synthetic data. Specifically, we empirically show that: (1) synthetic data generated using the error distribution derived from a set of transcribed data outperforms the common approach of applying random perturbations; (2) applying language-specific adjustments to the vocabulary of a BPE tokenizer strike a balance between adapting to unseen distributions and retaining knowledge of transcribed errors. We showcase the benefits of these key observations, and evaluate our approach using multiple languages, speech recognition systems and prominent speech recognition datasets.
eess.AS音频处理
【1】 Zipformer: A faster and better encoder for automatic speech recognition标题：Zipform：一种更快更好的自动语音识别编码器链接：https://arxiv.org/abs/2310.11230
作者：Zengwei Yao,Liyong Guo,Xiaoyu Yang,Wei Kang,Fangjun Kuang,Yifan Yang,Zengrui Jin,Long Lin,Daniel Povey摘要：Conformer已成为自动语音识别（ASR）中最流行的编码器模型。它将卷积模块添加到Transformer中，以学习局部和全局依赖关系。在这项工作中，我们描述了一个更快，更高效的内存，性能更好的Transformer，称为Zipformer。建模更改包括：1）类似于U-Net的编码器结构，其中中间栈以较低的帧速率操作; 2）重组块结构，增加模块，在其中我们重新使用注意力权重以提高效率; 3）LayerNorm的一种修改形式BiasNorm允许我们保留一些长度信息; 4）新的激活功能SwooshR和SwooshL比Swish工作得更好。我们还提出了一个新的优化器，称为ScaledAdam，它通过每个张量的当前尺度来缩放更新，以保持相对变化大致相同，并且还显式地学习参数尺度。它实现了比Adam更快的收敛和更好的性能。在LibriSpeech、Aishell-1和WenetSpeech数据集上进行的大量实验证明了我们提出的Zipformer比其他最先进的ASR模型更有效。我们的代码可在https://github.com/k2-fsa/icefall上公开获取。摘要：The Conformer has become the most popular encoder model for automatic speech recognition (ASR). It adds convolution modules to a transformer to learn both local and global dependencies. In this work we describe a faster, more memory-efficient, and better-performing transformer, called Zipformer. Modeling changes include: 1) a U-Net-like encoder structure where middle stacks operate at lower frame rates; 2) reorganized block structure with more modules, within which we re-use attention weights for efficiency; 3) a modified form of LayerNorm called BiasNorm allows us to retain some length information; 4) new activation functions SwooshR and SwooshL work better than Swish. We also propose a new optimizer, called ScaledAdam, which scales the update by each tensor's current scale to keep the relative change about the same, and also explictly learns the parameter scale. It achieves faster convergence and better performance than Adam. Extensive experiments on LibriSpeech, Aishell-1, and WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer over other state-of-the-art ASR models. Our code is publicly available at https://github.com/k2-fsa/icefall.
【2】 Iterative Shallow Fusion of Backward Language Model for End-to-End  Speech Recognition标题：端到端语音识别的后向语言模型迭代浅融合链接：https://arxiv.org/abs/2310.11010
作者：Atsunori Ogawa,Takafumi Moriya,Naoyuki Kamo,Naohiro Tawara,Marc Delcroix备注：Accepted to ICASSP 2023摘要：我们提出了一种新的浅融合（SF）方法，利用外部后向语言模型（BLM）的端到端的自动语音识别（ASR）。BLM与前向语言模型（FLM）具有互补的特征，并且它们组合的有效性已经通过将ASR假设重新评分作为后处理来证实。在所提出的SF中，我们迭代地将BLM应用于后向方向上的部分ASR假设（即，从可能的下一个令牌到开始符号），用新计算的BLM分数代替在最后一次迭代中计算的分数。为了提高这种迭代SF（ISF）的有效性，我们训练的一个部分识别BLM（PBLM）使用反向文本数据，包括部分句子，考虑ISF的框架。在使用基于注意力的编码器-解码器ASR系统的实验中，我们证实了使用PBLM的ISF显示出与使用FLM的SF相当的性能。通过执行ISF，在解码期间可以防止预期假设的早期修剪，并且与将PBLM应用于后处理相比，我们可以获得性能改进。最后，我们证实，通过结合SF和ISF，由于FLM和PBLM的互补性，可以获得进一步的性能改善。摘要：We propose a new shallow fusion (SF) method to exploit an external backward language model (BLM) for end-to-end automatic speech recognition (ASR). The BLM has complementary characteristics with a forward language model (FLM), and the effectiveness of their combination has been confirmed by rescoring ASR hypotheses as post-processing. In the proposed SF, we iteratively apply the BLM to partial ASR hypotheses in the backward direction (i.e., from the possible next token to the start symbol) during decoding, substituting the newly calculated BLM scores for the scores calculated at the last iteration. To enhance the effectiveness of this iterative SF (ISF), we train a partial sentence-aware BLM (PBLM) using reversed text data including partial sentences, considering the framework of ISF. In experiments using an attention-based encoder-decoder ASR system, we confirmed that ISF using the PBLM shows comparable performance with SF using the FLM. By performing ISF, early pruning of prospective hypotheses can be prevented during decoding, and we can obtain a performance improvement compared to applying the PBLM as post-processing. Finally, we confirmed that, by combining SF and ISF, further performance improvement can be obtained thanks to the complementarity of the FLM and PBLM.
【3】 Advanced accent/dialect identification and accentedness assessment with  multi-embedding models and automatic speech recognition标题：具有多嵌入模型和自动语音识别的高级口音/方言识别和重音评估链接：https://arxiv.org/abs/2310.11004
作者：Shahram Ghorbani,John H. L. Hansen备注：Submitted to The Journal of the Acoustical Society of America摘要：由于口音和方言变异的复杂性和多样性，准确地对口音进行分类和评估重音都是具有挑战性的任务。在这项研究中，利用先进的预训练语言识别（LID）和说话人识别（SID）模型的嵌入来提高口音分类和非母语重音评估的准确性。研究结果表明，采用预先训练的LID和SID模型有效地编码语音中的口音/方言信息。此外，LID和SID编码的口音信息补充了从头开始训练的端到端口音识别（AID）模型。通过将所有三个嵌入，所提出的多嵌入AID系统实现了优越的准确性，在口音识别。接下来，我们研究利用自动语音识别（ASR）和口音识别模型来探索重音估计。ASR模型是一个端到端的连接主义时间分类（CTC）模型，专门使用en-US话语进行训练。利用AID模型的ASR错误率和en-US输出作为客观加重评分。评价结果表明，这两个模型估计的分数之间有很强的相关性。此外，一个强大的相关性之间的客观重音分数和主观分数的基础上，人类的感知证明，提供证据的可靠性和有效性，利用艾滋病为基础的系统和ASR为基础的重音评估在非母语的讲话。摘要：Accurately classifying accents and assessing accentedness in non-native speakers are both challenging tasks due to the complexity and diversity of accent and dialect variations. In this study, embeddings from advanced pre-trained language identification (LID) and speaker identification (SID) models are leveraged to improve the accuracy of accent classification and non-native accentedness assessment. Findings demonstrate that employing pre-trained LID and SID models effectively encodes accent/dialect information in speech. Furthermore, the LID and SID encoded accent information complement an end-to-end accent identification (AID) model trained from scratch. By incorporating all three embeddings, the proposed multi-embedding AID system achieves superior accuracy in accent identification. Next, we investigate leveraging automatic speech recognition (ASR) and accent identification models to explore accentedness estimation. The ASR model is an end-to-end connectionist temporal classification (CTC) model trained exclusively with en-US utterances. The ASR error rate and en-US output of the AID model are leveraged as objective accentedness scores. Evaluation results demonstrate a strong correlation between the scores estimated by the two models. Additionally, a robust correlation between the objective accentedness scores and subjective scores based on human perception is demonstrated, providing evidence for the reliability and validity of utilizing AID-based and ASR-based systems for accentedness assessment in non-native speech.
【4】 Self-Supervised Models of Speech Infer Universal Articulatory Kinematics标题：语音自监督模型推断通用关节运动学链接：https://arxiv.org/abs/2310.10788
作者：Cheol Jun Cho,Abdelrahman Mohamed,Alan W Black,Gopala K. Anumanchipalli摘要：基于自监督学习（SSL）的语音模型在一系列下游任务上表现出卓越的性能。这些最先进的模型仍然是黑箱，但最近的许多研究已经开始“探索”像休伯特这样的模型，将它们的内部表征与言语的不同方面联系起来。在本文中，我们表明“关节运动学的推断”作为SSL模型的基本属性，即，这些模型将声学转换为语音信号下的因果发音动力学的能力。我们还表明，这种抽象在很大程度上是重叠的语言的数据用于训练模型，优先于语言具有相似的语音系统。此外，我们表明，与简单的仿射变换，声学发音反转（AAI）是可转移的扬声器，甚至跨性别，语言和方言，显示这种属性的推广。总之，这些结果揭示了SSL模型的内部结构，这些内部结构对SSL模型的卓越性能至关重要，并为语音工程的语言不可知通用模型开辟了新的途径，这些模型是可解释的，并以语音科学为基础。摘要：Self-Supervised Learning (SSL) based models of speech have shown remarkable performance on a range of downstream tasks. These state-of-the-art models have remained blackboxes, but many recent studies have begun "probing" models like HuBERT, to correlate their internal representations to different aspects of speech. In this paper, we show "inference of articulatory kinematics" as fundamental property of SSL models, i.e., the ability of these models to transform acoustics into the causal articulatory dynamics underlying the speech signal. We also show that this abstraction is largely overlapping across the language of the data used to train the model, with preference to the language with similar phonological system. Furthermore, we show that with simple affine transformations, Acoustic-to-Articulatory inversion (AAI) is transferrable across speakers, even across genders, languages, and dialects, showing the generalizability of this property. Together, these results shed new light on the internals of SSL models that are critical to their superior performance, and open up new avenues into language-agnostic universal models for speech engineering, that are interpretable and grounded in speech science.
【5】 Robust Wake-Up Word Detection by Two-stage Multi-resolution Ensembles标题：基于两级多分辨率集成的稳健唤醒词检测链接：https://arxiv.org/abs/2310.11379
作者：Fernando López,Jordi Luque,Carlos Segura,Pablo Gómez备注：5 pages, 3 figures摘要：基于语音的界面依赖于唤醒词机制来发起与设备的通信。然而，实现稳健、节能和快速的检测仍然是一个挑战。本文解决了这些实际的生产需求，通过增强数据的时间对齐和使用检测的基础上，两个阶段的多分辨率。它采用两种模式：用于音频流实时处理的轻量级设备上模型和服务器端的验证模型，该验证模型是改进检测的异构架构的集合。该方案允许两个操作点的优化。为了保护隐私，音频功能被发送到云端，而不是原始音频。该研究调查了用于特征提取的不同参数配置，以选择一个用于设备上检测，另一个用于验证模型。此外，十三个不同的音频分类器进行了比较，在性能和推理时间。建议的集成优于我们更强的分类器在每一个噪声条件。摘要：Voice-based interfaces rely on a wake-up word mechanism to initiate communication with devices. However, achieving a robust, energy-efficient, and fast detection remains a challenge. This paper addresses these real production needs by enhancing data with temporal alignments and using detection based on two phases with multi-resolution. It employs two models: a lightweight on-device model for real-time processing of the audio stream and a verification model on the server-side, which is an ensemble of heterogeneous architectures that refine detection. This scheme allows the optimization of two operating points. To protect privacy, audio features are sent to the cloud instead of raw audio. The study investigated different parametric configurations for feature extraction to select one for on-device detection and another for the verification model. Furthermore, thirteen different audio classifiers were compared in terms of performance and inference time. The proposed ensemble outperforms our stronger classifier in every noise condition.
【6】 High-Fidelity Noise Reduction with Differentiable Signal Processing标题：利用可差分信号处理实现高保真降噪链接：https://arxiv.org/abs/2310.11364
作者：Christian J. Steinmetz,Thomas Walther,Joshua D. Reiss备注：Accepted for publication at the 155th Convention of the Audio Engineering Society摘要：None摘要：Noise reduction techniques based on deep learning have demonstrated impressive performance in enhancing the overall quality of recorded speech. While these approaches are highly performant, their application in audio engineering can be limited due to a number of factors. These include operation only on speech without support for music, lack of real-time capability, lack of interpretable control parameters, operation at lower sample rates, and a tendency to introduce artifacts. On the other hand, signal processing-based noise reduction algorithms offer fine-grained control and operation on a broad range of content, however, they often require manual operation to achieve the best results. To address the limitations of both approaches, in this work we introduce a method that leverages a signal processing-based denoiser that when combined with a neural network controller, enables fully automatic and high-fidelity noise reduction on both speech and music signals. We evaluate our proposed method with objective metrics and a perceptual listening test. Our evaluation reveals that speech enhancement models can be extended to music, however training the model to remove only stationary noise is critical. Furthermore, our proposed approach achieves performance on par with the deep learning models, while being significantly more efficient and introducing fewer artifacts in some cases. Listening examples are available online at https://tape.it/research/denoiser .
【7】 Serenade: A Model for Human-in-the-loop Automatic Chord Estimation标题：小夜曲：一种人在环中的和弦自动估计模型链接：https://arxiv.org/abs/2310.11165
作者：Hendrik Vincent Koops,Gianluca Micchi,Ilaria Manco,Elio Quinton备注：Accepted at MMRP23. 7 pages, 5 figures, 2 tables摘要：计算和声分析对于自动切分、语料库分析和自动和弦标签估计等MIR任务非常重要。然而，最近对音乐和谐的模糊性的研究，导致有限的评分员之间的协议，已经明显地表明，有一个玻璃天花板的共同指标，如准确性。通常，这些问题要么在训练数据本身中通过创建多数规则注释来解决，要么在训练阶段通过学习软目标来解决。我们提出了一种新的替代方法，其中人类和自回归模型一起共同创建音轨的谐波注释。在自动生成和声预测后，人类稀疏地注释具有低模型置信度的部分，然后模型在人类指导下调整其预测。我们在流行音乐的数据集上评估了我们的模型，我们表明，通过这种人在回路中的方法，谐波分析性能比仅模型的方法有所提高。人类的贡献被模型的第二个约束预测放大。摘要：Computational harmony analysis is important for MIR tasks such as automatic segmentation, corpus analysis and automatic chord label estimation. However, recent research into the ambiguous nature of musical harmony, causing limited inter-rater agreement, has made apparent that there is a glass ceiling for common metrics such as accuracy. Commonly, these issues are addressed either in the training data itself by creating majority-rule annotations or during the training phase by learning soft targets. We propose a novel alternative approach in which a human and an autoregressive model together co-create a harmonic annotation for an audio track. After automatically generating harmony predictions, a human sparsely annotates parts with low model confidence and the model then adjusts its predictions following human guidance. We evaluate our model on a dataset of popular music and we show that, with this human-in-the-loop approach, harmonic analysis performance improves over a model-only approach. The human contribution is amplified by the second, constrained prediction of the model.
【8】 Leveraging Content-based Features from Multiple Acoustic Models for  Singing Voice Conversion标题：利用来自多个声学模型的基于内容的特征进行歌唱语音转换链接：https://arxiv.org/abs/2310.11160
作者：Xueyao Zhang,Yicheng Gu,Haopeng Chen,Zihao Fang,Lexiao Zou,Liumeng Xue,Zhizheng Wu摘要：歌声转换（SVC）是一种使任意歌手演唱任意歌曲的技术。为了实现这一点，重要的是从源音频中获得说话者不可知的表示，这是一项具有挑战性的任务。常见的解决方案是提取基于内容的特征（例如，PPG）。然而，声学模型的选择是广泛而多样的。来自不同声学模型的内容特征的特征是什么，以及集成多个内容特征是否可以互相帮助，还有待探索。受此启发，本研究调查了三个不同的内容特征，分别来自WeNet，Whisper和ContentVec。我们探讨他们的互补作用，可理解性，韵律，转换相似性SVC。通过将多个内容特征与基于扩散的SVC模型相结合，我们的SVC系统在客观和主观评价上都实现了优于单一内容特征源的转换性能。我们的演示页面和代码可以在https://www.zhangxueyao.com/data/MultipleContentsSVC/index.html上找到。摘要：Singing voice conversion (SVC) is a technique to enable an arbitrary singer to sing an arbitrary song. To achieve that, it is important to obtain speaker-agnostic representations from source audio, which is a challenging task. A common solution is to extract content-based features (e.g., PPGs) from a pretrained acoustic model. However, the choices for acoustic models are vast and varied. It is yet to be explored what characteristics of content features from different acoustic models are, and whether integrating multiple content features can help each other. Motivated by that, this study investigates three distinct content features, sourcing from WeNet, Whisper, and ContentVec, respectively. We explore their complementary roles in intelligibility, prosody, and conversion similarity for SVC. By integrating the multiple content features with a diffusion-based SVC model, our SVC system achieves superior conversion performance on both objective and subjective evaluation in comparison to a single source of content features. Our demo page and code can be available https://www.zhangxueyao.com/data/MultipleContentsSVC/index.html.
【9】 Long-form Simultaneous Speech Translation: Thesis Proposal标题：长篇同声传译：论文建议链接：https://arxiv.org/abs/2310.11141
作者：Peter Polák备注：IJCNLP-AACL SRW 2023 - camera-ready version摘要：同步语音翻译（SST）旨在提供实时的口语翻译，甚至在说话者完成句子之前。传统上，SST主要由级联系统解决，这些系统将任务分解为子任务，包括语音识别，分割和机器翻译。然而，深度学习的出现引发了人们对端到端（E2E）系统的极大兴趣。然而，在目前的文献中报道的大多数E2E SST方法的一个主要限制是，它们假设源语音被预先分割成句子，这是实际的，现实世界的应用的一个重大障碍。本论文提出了端到端的同步语音翻译，特别是在长格式的设置，即，没有预先分割。我们提出了E2E SST的最新进展的调查，评估SST的主要障碍及其与长期情景的相关性，并提出了应对这些挑战的方法。摘要：Simultaneous speech translation (SST) aims to provide real-time translation of spoken language, even before the speaker finishes their sentence. Traditionally, SST has been addressed primarily by cascaded systems that decompose the task into subtasks, including speech recognition, segmentation, and machine translation. However, the advent of deep learning has sparked significant interest in end-to-end (E2E) systems. Nevertheless, a major limitation of most approaches to E2E SST reported in the current literature is that they assume that the source speech is pre-segmented into sentences, which is a significant obstacle for practical, real-world applications. This thesis proposal addresses end-to-end simultaneous speech translation, particularly in the long-form setting, i.e., without pre-segmentation. We present a survey of the latest advancements in E2E SST, assess the primary obstacles in SST and its relevance to long-form scenarios, and suggest approaches to tackle these challenges.
【10】 VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System标题：VoxArabica：一种稳健的方言感知阿拉伯语语音识别系统链接：https://arxiv.org/abs/2310.11069
作者：Abdul Waheed,Bashar Talafha,Peter Suvellin,Abdelrahman Elmadney,Muhammad Abdul-Mageed备注：Accepted at ArabicNLP conference co-located with EMNLP'23摘要：阿拉伯语是一种复杂的语言，有许多变体和方言，全世界有超过4.5亿人使用。由于语言的多样性和变化，这是具有挑战性的，建立一个强大的和通用的阿拉伯语自动语音识别系统。在这项工作中，我们通过开发和演示一个名为VoxArabica的系统来解决这一差距，该系统用于方言识别（DID）以及阿拉伯语的自动语音识别（ASR）。我们训练了各种各样的模型，如HuBERT（DID），Whisper和XLS-R（ASR），在有监督的环境中进行阿拉伯语DID和ASR任务。我们的DID模型经过训练，除了MSA之外，还可以识别17种不同的方言。我们在MSA、埃及、摩洛哥和混合数据上微调我们的ASR模型。此外，对于ASR中的其余方言，我们提供了在zero-shot设置中选择各种型号（如Whisper和MMS）的选项。我们将这些模型集成到一个具有多种功能的Web界面中，例如音频录制，文件上传，模型选择以及为不正确的输出提高标志的选项。总的来说，我们相信VoxArabica将对关注阿拉伯语研究的广泛受众有用。我们的系统目前在https://cdce-206-12-100-168.ngrok.io/上运行。摘要：Arabic is a complex language with many varieties and dialects spoken by over 450 millions all around the world. Due to the linguistic diversity and variations, it is challenging to build a robust and generalized ASR system for Arabic. In this work, we address this gap by developing and demoing a system, dubbed VoxArabica, for dialect identification (DID) as well as automatic speech recognition (ASR) of Arabic. We train a wide range of models such as HuBERT (DID), Whisper, and XLS-R (ASR) in a supervised setting for Arabic DID and ASR tasks. Our DID models are trained to identify 17 different dialects in addition to MSA. We finetune our ASR models on MSA, Egyptian, Moroccan, and mixed data. Additionally, for the remaining dialects in ASR, we provide the option to choose various models such as Whisper and MMS in a zero-shot setting. We integrate these models into a single web interface with diverse features such as audio recording, file upload, model selection, and the option to raise flags for incorrect outputs. Overall, we believe VoxArabica will be useful for a wide range of audiences concerned with Arabic research. Our system is currently running at https://cdce-206-12-100-168.ngrok.io/.
【11】 Lyricist-Singer Entropy Affects Lyric-Lyricist Classification  Performance标题：词人-歌手信息熵影响词人分类性能链接：https://arxiv.org/abs/2310.11035
作者：Mitsuki Morita,Masato Kikuchi,Tadachika Ozono备注：The 10th International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA 2023)摘要：歌词是音乐的重要组成部分，但关于歌词作者特征的音乐信息处理研究却很少。因为这些特征对于音乐应用（例如推荐）可能是有价值的，所以它们值得进一步研究。我们考虑了一种潜在的方法，提取代表歌词的歌词作者的特点。因为这些特征必须在提取之前被识别，所以我们专注于具有容易识别特征的抒情诗人。我们认为，这是理想的歌手表演独特的歌曲，共享特定的歌手的某些特征。因此，我们假设作词人解释了他们为之写歌词的歌手的独特特征。换句话说，歌词-作词人分类性能或从歌词捕捉作词人的特征的容易程度可以取决于歌手的种类。在这项研究中，我们观察到一个歌词歌手熵或各种歌手与一个单一的歌词和歌词歌手分类性能之间的关系。作为一个例子，当作词人只为一个歌手写歌词时，作词人-歌手熵是最小的。在我们的实验中，我们将抒情诗人分为五组的抒情诗人歌手熵，并评估各组内的抒情诗人分类性能。因此，最好的F1得分是获得最低作词歌手熵的组。我们的研究结果表明，进一步分析的功能，有助于抒情歌手分类性能的最低抒情歌手歌手熵组可能会提高抒情歌手的特征提取任务。摘要：Although lyrics represent an essential component of music, few music information processing studies have been conducted on the characteristics of lyricists. Because these characteristics may be valuable for musical applications, such as recommendations, they warrant further study. We considered a potential method that extracts features representing the characteristics of lyricists from lyrics. Because these features must be identified prior to extraction, we focused on lyricists with easily identifiable features. We believe that it is desirable for singers to perform unique songs that share certain characteristics specific to the singer. Accordingly, we hypothesized that lyricists account for the unique characteristics of the singers they write lyrics for. In other words, lyric-lyricist classification performance or the ease of capturing the features of a lyricist from the lyrics may depend on the variety of singers. In this study, we observed a relationship between lyricist-singer entropy or the variety of singers associated with a single lyricist and lyric-lyricist classification performance. As an example, the lyricist-singer entropy is minimal when the lyricist writes lyrics for only one singer. In our experiments, we grouped lyricists among five groups in terms of lyricist-singer entropy and assessed the lyric-lyricist classification performance within each group. Consequently, the best F1 score was obtained for the group with the lowest lyricist-singer entropy. Our results suggest that further analyses of the features contributing to lyric-lyricist classification performance on the lowest lyricist-singer entropy group may improve the feature extraction task for lyricists.
【12】 Correction Focused Language Model Training for Speech Recognition标题：面向语音识别的纠错语言模型训练链接：https://arxiv.org/abs/2310.11003
作者：Yingyi Ma,Zhe Liu,Ozlem Kalinli摘要：语言模型（LM）已被普遍采用，以提高自动语音识别（ASR）的性能，特别是在领域适应任务。传统的LM训练方法对语料库中的所有单词都一视同仁，导致ASR性能的改善不理想。在这项工作中，我们介绍了一种新的校正重点LM训练方法，其目的是优先考虑ASR易错的话。词级ASR易错性分数表示ASR误识别的可能性，被定义和成形为先验词分布以指导LM训练。为了实现仅文本语料库的校正集中训练，大型语言模型（LLM）通过多任务微调被用作易错性分数预测器和文本生成器。领域自适应任务的实验结果证明了我们提出的方法的有效性。与传统的LM相比，校正集中的训练在足够的文本场景中实现了高达5.5%的字错误率（WER）降低。在文本不足的情况下，使用LLM生成的文本进行LM训练可以实现高达相对13%的WER降低，而校正集中的训练可以进一步获得高达相对6%的WER降低。摘要：Language models (LMs) have been commonly adopted to boost the performance of automatic speech recognition (ASR) particularly in domain adaptation tasks. Conventional way of LM training treats all the words in corpora equally, resulting in suboptimal improvements in ASR performance. In this work, we introduce a novel correction focused LM training approach which aims to prioritize ASR fallible words. The word-level ASR fallibility score, representing the likelihood of ASR mis-recognition, is defined and shaped as a prior word distribution to guide the LM training. To enable correction focused training with text-only corpora, large language models (LLMs) are employed as fallibility score predictors and text generators through multi-task fine-tuning. Experimental results for domain adaptation tasks demonstrate the effectiveness of our proposed method. Compared with conventional LMs, correction focused training achieves up to relatively 5.5% word error rate (WER) reduction in sufficient text scenarios. In insufficient text scenarios, LM training with LLM-generated text achieves up to relatively 13% WER reduction, while correction focused training further obtains up to relatively 6% WER reduction.
【13】 A High Fidelity and Low Complexity Neural Audio Coding标题：一种高保真低复杂度的神经音频编码链接：https://arxiv.org/abs/2310.10992
作者：Wenzhe Liu,Wei Xiao,Meng Wang,Shan Yang,Yupeng Shi,Yuyong Kang,Dan Su,Shidong Shang,Dong Yu摘要：音频编码是实时通信系统中必不可少的模块。由于深度神经网络的强大建模和生成能力，神经音频编解码器可以以低比特率压缩音频样本。为了解决高频表达差和高计算成本和存储消耗，我们提出了一个集成的框架，利用神经网络模型的宽带分量，并采用传统的信号处理压缩高频分量根据心理听觉知识。受听觉感知理论的启发，设计了一种基于感知的损失函数来改进谐波建模。此外，生成对抗网络（GAN）压缩首次提出了神经音频编解码器。我们的方法在主观和客观指标上优于先前的高级神经编解码器，并允许在桌面和移动设备上进行实时推理。摘要：Audio coding is an essential module in the real-time communication system. Neural audio codecs can compress audio samples with a low bitrate due to the strong modeling and generative capabilities of deep neural networks. To address the poor high-frequency expression and high computational cost and storage consumption, we proposed an integrated framework that utilizes a neural network to model wide-band components and adopts traditional signal processing to compress high-band components according to psychological hearing knowledge. Inspired by auditory perception theory, a perception-based loss function is designed to improve harmonic modeling. Besides, generative adversarial network (GAN) compression is proposed for the first time for neural audio codecs. Our method is superior to prior advanced neural codecs across subjective and objective metrics and allows real-time inference on desktop and mobile.
【14】 Spatial HuBERT: Self-supervised Spatial Speech Representation Learning  for a Single Talker from Multi-channel Audio标题：空间休BERT：多声道音频中单个说话人的自监督空间语音表示学习链接：https://arxiv.org/abs/2310.10922
作者：Antoni Dimitriadis,Siqi Pan,Vidhyasaharan Sethu,Beena Ahmed摘要：自监督学习已被用于利用未标记的数据，通过训练表示模型来提高语音系统的准确性和泛化能力。虽然最近的许多研究试图在各种声学领域、语言、模态甚至同时说话者之间产生有效的表示，但这些研究都仅限于单通道音频记录。本文介绍了空间HuBERT，一个自我监督的语音表示模型，学习声学和空间信息有关的一个潜在的嘈杂环境中的单个扬声器，通过使用多通道音频输入。Spatial HuBERT在各种空间下游任务中学习表现优于最先进的单通道语音表示的表示，特别是在混响和嘈杂的环境中。我们还展示了空间休伯特学习的语音定位下游任务的表示的效用。随着这篇论文，我们公开发布了一个新的数据集100 000模拟一阶高保真度立体声房间脉冲响应。摘要：Self-supervised learning has been used to leverage unlabelled data, improving accuracy and generalisation of speech systems through the training of representation models. While many recent works have sought to produce effective representations across a variety of acoustic domains, languages, modalities and even simultaneous speakers, these studies have all been limited to single-channel audio recordings. This paper presents Spatial HuBERT, a self-supervised speech representation model that learns both acoustic and spatial information pertaining to a single speaker in a potentially noisy environment by using multi-channel audio inputs. Spatial HuBERT learns representations that outperform state-of-the-art single-channel speech representations on a variety of spatial downstream tasks, particularly in reverberant and noisy environments. We also demonstrate the utility of the representations learned by Spatial HuBERT on a speech localisation downstream task. Along with this paper, we publicly release a new dataset of 100 000 simulated first-order ambisonics room impulse responses.
【15】 SD-HuBERT: Self-Distillation Induces Syllabic Organization in HuBERT标题：SD-Hubert：自我蒸馏导致休BERT的音节组织链接：https://arxiv.org/abs/2310.10803
作者：Cheol Jun Cho,Abdelrahman Mohamed,Shang-Wen Li,Alan W Black,Gopala K. Anumanchipalli摘要：语音自监督学习（SSL）中的数据驱动单元发现开启了口语处理的新时代。然而，所发现的单元通常保留在语音空间中，限制了SSL表示的效用。在这里，我们证明了一个音节的组织出现在学习语音的双音级表示。特别是，我们采用“自我蒸馏”目标来微调预训练的HuBERT，并使用汇总整个句子的聚合器令牌。在没有任何监督的情况下，所产生的模型在语音中划出了明确的边界，并且跨框架的表示显示出突出的音节结构。我们证明，这种新兴的结构在很大程度上对应于地面真理音节。此外，我们提出了一个新的基准任务，口语ABX，用于评估语音的高级表示。与以前的模型相比，我们的模型在无监督音节发现和学习音节级表示方面都表现出色。总之，我们证明了HuBERT的自我升华在不依赖外部标签或模态的情况下产生了音节组织，并可能为口语建模提供新的数据驱动单元。摘要：Data-driven unit discovery in self-supervised learning (SSL) of speech has embarked on a new era of spoken language processing. Yet, the discovered units often remain in phonetic space, limiting the utility of SSL representations. Here, we demonstrate that a syllabic organization emerges in learning sentence-level representation of speech. In particular, we adopt "self-distillation" objective to fine-tune the pretrained HuBERT with an aggregator token that summarizes the entire sentence. Without any supervision, the resulting model draws definite boundaries in speech, and the representations across frames show salient syllabic structures. We demonstrate that this emergent structure largely corresponds to the ground truth syllables. Furthermore, we propose a new benchmark task, Spoken Speech ABX, for evaluating sentence-level representation of speech. When compared to previous models, our model outperforms in both unsupervised syllable discovery and learning sentence-level representation. Together, we demonstrate that the self-distillation of HuBERT gives rise to syllabic organization without relying on external labels or modalities, and potentially provides novel data-driven units for spoken language modeling.
【16】 Unsupervised Lead Sheet Generation via Semantic Compression标题：基于语义压缩的无监督引导单生成链接：https://arxiv.org/abs/2310.10772
作者：Zachary Novack,Nikita Srivatsan,Taylor Berg-Kirkpatrick,Julian McAuley摘要：在生成音乐的研究中，铅板已经变得司空见惯，被用作多轨音乐生成和自动编曲等下游任务的初始压缩表示。尽管如此，研究人员在寻找配对的铅表和满分时，经常依靠确定性还原方法（如天际线算法）来生成铅表，很少关注铅表本身的质量以及它们如何准确地反映它们的协调对应物。为了解决这些问题，我们提出了有条件的铅片生成的问题（即。在给定其完整乐谱版本的情况下生成引导表），并且示出了该任务可以被公式化为无监督音乐压缩任务，其中引导表表示乐谱的压缩的潜在版本。我们介绍了一种新的模型，称为铅AE，模型的铅片作为一个离散的子选择的原始序列，使用可微的top-k算子，允许可控的局部稀疏约束。在自动代理任务和直接的人类评价，我们发现，我们的方法改进了确定性的基线，并产生一致的减少大多轨分数。摘要：Lead sheets have become commonplace in generative music research, being used as an initial compressed representation for downstream tasks like multitrack music generation and automatic arrangement. Despite this, researchers have often fallen back on deterministic reduction methods (such as the skyline algorithm) to generate lead sheets when seeking paired lead sheets and full scores, with little attention being paid toward the quality of the lead sheets themselves and how they accurately reflect their orchestrated counterparts. To address these issues, we propose the problem of conditional lead sheet generation (i.e. generating a lead sheet given its full score version), and show that this task can be formulated as an unsupervised music compression task, where the lead sheet represents a compressed latent version of the score. We introduce a novel model, called Lead-AE, that models the lead sheets as a discrete subselection of the original sequence, using a differentiable top-k operator to allow for controllable local sparsity constraints. Across both automatic proxy tasks and direct human evaluations, we find that our method improves upon the established deterministic baseline and produces coherent reductions of large multitrack scores.
【17】 Optimized Tokenization for Transcribed Error Correction标题：用于转录纠错的优化标记化链接：https://arxiv.org/abs/2310.10704
作者：Tomer Wullach,Shlomo E. Chazan摘要：语音识别系统面临的挑战，如发音的变化，不利的音频条件和标记数据的稀缺性，强调了纠正重复错误的后处理步骤的必要性。以前的研究已经表明了采用专用纠错模型的优势，但训练这样的模型需要大量的标记数据，这是不容易获得的。为了克服这一限制，通常使用合成的类似转录的数据，然而，桥接转录错误和合成噪声之间的分布差距并不是微不足道的。在本文中，我们证明了校正模型的性能可以显着提高训练单独使用合成数据。具体而言，我们经验表明：（1）使用从一组转录数据导出的误差分布生成的合成数据优于应用随机扰动的常见方法;（2）对BPE分词器的词汇表应用语言特定的调整，在适应看不见的分布和保留转录错误的知识之间取得平衡。我们展示了这些关键观察的好处，并使用多种语言，语音识别系统和突出的语音识别数据集来评估我们的方法。摘要：The challenges facing speech recognition systems, such as variations in pronunciations, adverse audio conditions, and the scarcity of labeled data, emphasize the necessity for a post-processing step that corrects recurring errors. Previous research has shown the advantages of employing dedicated error correction models, yet training such models requires large amounts of labeled data which is not easily obtained. To overcome this limitation, synthetic transcribed-like data is often utilized, however, bridging the distribution gap between transcribed errors and synthetic noise is not trivial. In this paper, we demonstrate that the performance of correction models can be significantly increased by training solely using synthetic data. Specifically, we empirically show that: (1) synthetic data generated using the error distribution derived from a set of transcribed data outperforms the common approach of applying random perturbations; (2) applying language-specific adjustments to the vocabulary of a BPE tokenizer strike a balance between adapting to unseen distributions and retaining knowledge of transcribed errors. We showcase the benefits of these key observations, and evaluate our approach using multiple languages, speech recognition systems and prominent speech recognition datasets.
机器翻译，仅供参考