今日论文合集：cs.SD语音17篇，eess.AS音频处理17篇。

cs.SD语音

【1】 Audio Retrieval for Multimodal Design Documents: A New Dataset and Algorithms标题：多通道设计文档音频检索：一种新的数据集和算法链接：http://arxiv.org/pdf/2302.14757v1

作者：Prachi Singh,Srikrishna Karanam,Sumit Shekhar机构：Adobe Research, Bangalore, India, Indian Institute of Science, Bangalore, India备注：5 pages including references摘要：我们考虑并提出了一种检索与包括文本元素和视觉图像两者的多模态设计文档输入相关的音频文件的新问题，生日 贺卡。除了增强用户体验之外，集成与这些输入的主题 风格相匹配的音频还有助于改善这些文档的可访问性（例如，视觉受损的人可以改为收听音频）。虽然最近在音频检索方面的工作存在，但是这些方法和数据集明确地针对自然图像。然而，我们的问题认为多模态设计文档（由用户使用创造性软件创建）与自然点击的照片有本质上的不同。为此，我们的第一个贡献是收集和管理一个新的大规模数据集，称为旋律设计（或MELON），包括代表各种风格、主题、模板、插图等的设计文档。与音乐音频配对。给定我们的配对图像-文本-音频数据集，我们的下一个贡献是一个新颖的多模态交叉注意音频检索（MMCAR）算法，该算法使训练神经网络能够学习跨图像、文本和音频维度的公共共享特征空间。我们使用这些学习到的特征来证明我们的方法优于现有的最先进的方法，并为研究团体在我们的新数据集上产生一个新的参考基准。摘要：We consider and propose a new problem of retrieving audio files relevant to multimodal design document inputs comprising both textual elements and visual imagery, e.g., birthday greeting cards. In addition to enhancing user experience, integrating audio that matches the theme style of these inputs also helps improve the accessibility of these documents (e.g., visually impaired people can listen to the audio instead). While recent work in audio retrieval exists, these methods and datasets are targeted explicitly towards natural images. However, our problem considers multimodal design documents (created by users using creative software) substantially different from a naturally clicked photograph. To this end, our first contribution is collecting and curating a new large-scale dataset called Melodic-Design (or MELON), comprising design documents representing various styles, themes, templates, illustrations, etc., paired with music audio. Given our paired image-text-audio dataset, our next contribution is a novel multimodal cross-attention audio retrieval (MMCAR) algorithm that enables training neural networks to learn a common shared feature space across image, text, and audio dimensions. We use these learned features to demonstrate that our method outperforms existing state-of-the-art methods and produce a new reference benchmark for the research community on our new dataset.

【2】 deHuBERT: Disentangling Noise in a Self-supervised Model for Robust Speech Recognition标题：DeHuBERT：用于稳健语音识别的自监督模型中的噪声分离链接：http://arxiv.org/pdf/2302.14597v1

作者：Dianwen Ng,Ruixi Zhang,Jia Qi Yip,Zhao Yang,Jinjie Ni,Chong Zhang,Yukun Ma,Chongjia Ni,Eng Siong Chng,Bin Ma机构：Alibaba Group, School of Computer Science and Engineering, Nanyang Technological University, Singapore, Xi’an Jiaotong University, China备注：Accepted by ICASSP 2023摘要：现有的自监督预训练语音模型为利用海量未标注语料库建立良好的自动语音识别（ASR）提供了一种有效的方法。然而，许多当前的模型是在来自单个源的干净语料库上训练的，这在测试期间存在噪声时往往表现不佳。然而，克服噪声对实际应用的不利影响是至关重要的。在这项工作中，我们提出了一个新的训练框架，称为deHuBERT，用于降噪编码的启发H。巴洛的冗余减少原理。新框架通过引入辅助损失来改进HuBERT训练算法，辅助损失将成对噪声失真嵌入之间的自相关矩阵和互相关矩阵推向单位矩阵。这鼓励模型产生噪声不可知的语音表示。通过这种方法，我们报告了在噪声环境中（包括看不见的噪声）改进的鲁棒性，而不损害在清洁集上的性能。摘要：Existing self-supervised pre-trained speech models have offered an effective way to leverage massive unannotated corpora to build good automatic speech recognition (ASR). However, many current models are trained on a clean corpus from a single source, which tends to do poorly when noise is present during testing. Nonetheless, it is crucial to overcome the adverse influence of noise for real-world applications. In this work, we propose a novel training framework, called deHuBERT, for noise reduction encoding inspired by H. Barlow's redundancy-reduction principle. The new framework improves the HuBERT training algorithm by introducing auxiliary losses that drive the self- and cross-correlation matrix between pairwise noise-distorted embeddings towards identity matrix. This encourages the model to produce noise-agnostic speech representations. With this method, we report improved robustness in noisy environments, including unseen noises, without impairing the performance on the clean set.

【3】 Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition标题：用于节律障碍和老年人语音识别的自监督预训练ASR模型探索链接：http://arxiv.org/pdf/2302.14564v1

作者：Shujie Hu,Xurong Xie,Zengrui Jin,Mengzhe Geng,Yi Wang,Mingyu Cui,Jiajun Deng,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Institute of Software, Chinese Academy of Sciences, China摘要：由于难以大量收集这类数据，迄今为止，无序和老年语音的自动识别仍然是一项极具挑战性的任务。本文探索了一系列方法，将域自适应SSL预训练模型集成到TDNN和Conformer ASR系统中，用于构音障碍和老年人语音识别：a）标准声学前端和域自适应wav2vec2.0语音表示之间的输入特征融合;b）单独使用标准声学特征和使用附加wav2vec2.0特征单独训练的TDNN系统的帧级联合解码;以及c）涉及使用域自适应wav2vec2.0模型重新评分的TDNN Conformer系统输出的多通路解码。此外，将域自适应wav 2 vec 2. 0表示用于声学到发音（A2 A）反演，以构建多模态构音障碍和老年人语音识别系统。在UASpeech构音障碍和DementiaBank Pitt老年人语音语料库上进行的实验表明，TDNN和Conformer ASR系统集成的域自适应wav2vec2.0模型在统计学上显著优于独立的wav2vec2.0模型，在两个任务上分别降低了8.22%和3.43%绝对值（26.71%和15.88%相对值）。在16名构音障碍者的UASpeech测试集和DementiaBank Pitt测试集上分别获得了22.56%的最低公布WER（极低可懂度52.53%，看不见的单词39.09%）和18.17%。摘要：Automatic recognition of disordered and elderly speech remains a highly challenging task to date due to the difficulty in collecting such data in large quantities. This paper explores a series of approaches to integrate domain adapted SSL pre-trained models into TDNN and Conformer ASR systems for dysarthric and elderly speech recognition: a) input feature fusion between standard acoustic frontends and domain adapted wav2vec2.0 speech representations; b) frame-level joint decoding of TDNN systems separately trained using standard acoustic features alone and with additional wav2vec2.0 features; and c) multi-pass decoding involving the TDNN Conformer system outputs to be rescored using domain adapted wav2vec2.0 models. In addition, domain adapted wav2vec2.0 representations are utilized in acoustic-to-articulatory (A2A) inversion to construct multi-modal dysarthric and elderly speech recognition systems. Experiments conducted on the UASpeech dysarthric and DementiaBank Pitt elderly speech corpora suggest TDNN and Conformer ASR systems integrated domain adapted wav2vec2.0 models consistently outperform the standalone wav2vec2.0 models by statistically significant WER reductions of 8.22% and 3.43% absolute (26.71% and 15.88% relative) on the two tasks respectively. The lowest published WERs of 22.56% (52.53% on very low intelligibility, 39.09% on unseen words) and 18.17% are obtained on the UASpeech test set of 16 dysarthric speakers, and the DementiaBank Pitt test set respectively.

【4】 CrossSpeech: Speaker-independent Acoustic Representation for Cross-lingual Speech Synthesis标题：CrossSpeech：跨语言语音合成的非特定人声学表示链接：http://arxiv.org/pdf/2302.14370v1

作者：Ji-Hoon Kim,Hong-Sun Yang,Yoon-Cheol Ju,Il-Hwan Kim,Byeong-Yeol Kim机构：dot Inc., Seoul, Republic of Korea备注：Accepted to ICASSP 2023摘要：虽然最近的文本到语音（TTS）系统在人类水平的质量方面取得了显著的进步，但是跨语言TTS的性能落后于语言内TTS的性能。这一差距主要源于跨语言TTS中的说话人-语言纠缠问题。本文提出了CrossSpeech算法，该算法通过在声学特征空间层次上有效地分离说话人和语言信息，提高了跨语言语音的质量。具体来说，CrossSpeech将语音生成流水线分解为说话人无关生成器（SIG）和说话人相关生成器（SDG）。SIG产生不偏向于特定说话者分布的说话者无关的声学表示。另一方面，SDG对表征说话人属性的说话人相关语音变化进行建模。通过分别处理每一种信息，CrossSpeech可以获得分离的说话人和语言表示。通过实验，我们验证了CrossSpeech在跨语言TTS中取得了显著的改进，特别是在说话人与目标说话人的相似度方面。摘要：While recent text-to-speech (TTS) systems have made remarkable strides toward human-level quality, the performance of cross-lingual TTS lags behind that of intra-lingual TTS. This gap is mainly rooted from the speaker-language entanglement problem in cross-lingual TTS. In this paper, we propose CrossSpeech which improves the quality of cross-lingual speech by effectively disentangling speaker and language information in the level of acoustic feature space. Specifically, CrossSpeech decomposes the speech generation pipeline into the speaker-independent generator (SIG) and speaker-dependent generator (SDG). The SIG produces the speaker-independent acoustic representation which is not biased to specific speaker distributions. On the other hand, the SDG models speaker-dependent speech variation that characterizes speaker attributes. By handling each information separately, CrossSpeech can obtain disentangled speaker and language representations. From the experiments, we verify that CrossSpeech achieves significant improvements in cross-lingual TTS, especially in terms of speaker similarity to the target speaker.

【5】 A Token-Wise Beam Search Algorithm for RNN-T标题：一种适用于RNN-T的令牌式波束搜索算法链接：http://arxiv.org/pdf/2302.14357v1

作者：Gil Keren机构：Meta AI, USA摘要：用于语音识别的标准递归神经网络转换器（RNN-T）解码算法在时间轴上迭代，使得在移动到下一个时间步长之前解码一个时间步长。这些算法导致对联合网络的大量调用，这在先前的工作中被示出为降低解码速度的重要因素。我们提出了一种解码波束搜索算法，该算法在一段时间步长上分批处理联合网络调用，这导致了40%-70%的解码加速比，在所有模型和实验设置中一致。此外，在一个片段上聚集发射概率可以被看作是找到最可能的模型输出的更好近似，使得我们的算法随着片段大小的增加相对提高甲骨文单词错误率高达10%，并且稍微提高一般单词错误率。摘要：Standard Recurrent Neural Network Transducers (RNN-T) decoding algorithms for speech recognition are iterating over the time axis, such that one time step is decoded before moving on to the next time step. Those algorithms result in a large number of calls to the joint network, that were shown in previous work to be an important factor that reduces decoding speed. We present a decoding beam search algorithm that batches the joint network calls across a segment of time steps, which results in 40%-70% decoding speedups, consistently across all models and settings experimented with. In addition, aggregating emission probabilities over a segment may be seen as a better approximation to finding the most likely model output, causing our algorithm to improve oracle word error rate by up to 10% relative as the segment size increases, and to slightly improve general word error rate.

【6】 UniFLG: Unified Facial Landmark Generator from Text or Speech标题：UniFLG：文本或语音的统一人脸标志点生成器链接：http://arxiv.org/pdf/2302.14337v1

作者：Kentaro Mitsui,Yukiya Hono,Kei Sawada机构：rinna Co., Ltd., Tokyo, Japan备注：5 pages, 2 figures, 3 tables摘要：人脸生成由于其广泛的应用性而得到了广泛的研究。用于说话面部生成的两个主要框架包括文本驱动框架和语音驱动框架，文本驱动框架从文本生成同步语音和说话面部，语音驱动框架从语音生成说话面部。为了整合这些框架，本文提出了一种统一的面部标志生成器（unified facial landmark generator，UniFLG）。所提出的系统利用端到端文本到语音不仅用于合成语音，而且用于提取文本和语音共有的一系列潜在表示，并将其馈送到界标解码器以生成面部界标。我们证明，我们的系统取得了更高的自然度在语音合成和面部标志生成相比，国家的最先进的文本驱动的方法。我们进一步证明了我们的系统可以从说话人的语音中生成面部标志，而不需要面部视频数据甚至语音数据。摘要：Talking face generation has been extensively investigated owing to its wide applicability. The two primary frameworks used for talking face generation comprise a text-driven framework, which generates synchronized speech and talking faces from text, and a speech-driven framework, which generates talking faces from speech. To integrate these frameworks, this paper proposes a unified facial landmark generator (UniFLG). The proposed system exploits end-to-end text-to-speech not only for synthesizing speech but also for extracting a series of latent representations that are common to text and speech, and feeds it to a landmark decoder to generate facial landmarks. We demonstrate that our system achieves higher naturalness in both speech synthesis and facial landmark generation compared to the state-of-the-art text-driven method. We further demonstrate that our system can generate facial landmarks from speech of speakers without facial video data or even speech data.

【7】 Adapter Incremental Continual Learning of Efficient Audio Spectrogram Transformers标题：高效音谱图转换器的自适应增量式连续学习链接：http://arxiv.org/pdf/2302.14314v1

作者：Nithish Muthuchamy Selvaraj,Xiaobao Guo,Adams Kong,Bingquan Shen,Alex Kot机构：Nanyang Technological University, Singapore, DSO National Laboratories, Singapore摘要：连续学习涉及在保留先前任务的知识的同时为新任务递增地训练神经网络。然而，用最少的计算资源有效地微调用于顺序任务的模型仍然是一个挑战。在这篇文章中，我们提出了任务增量连续学习（TI-CL）的音频分类器与参数有效和计算有效的音频谱图Transformers（AST）。为了减少TI-CL的可训练参数而不降低性能，我们比较了几种参数有效传递（PET）方法，并提出了用于TI-CL的具有卷积适配器的AST，其可训练参数少于完全微调对应物的5%。为了降低计算复杂度，提出了一种新的频率-时间因子化注意力（FTA）方法来代替传统的自注意力变换方法。FTA仅需要全局自我注意（GSA）所需计算的一个因子就可以获得有竞争力的性能。最后，我们制定了TI-CL的方法，称为适配器增量连续学习（AI-CL），作为“参数高效”卷积适配器和“计算高效”FTA的组合。在ESC-50、SpeechCommandsV 2（SCv 2）和AVE（Audio-Visual Event）基准测试上的实验表明，该方法在保持较低计算量的同时，有效地防止了TI-CL中的灾难性遗忘。摘要：Continual learning involves training neural networks incrementally for new tasks while retaining the knowledge of previous tasks. However, efficiently fine-tuning the model for sequential tasks with minimal computational resources remains a challenge. In this paper, we propose Task Incremental Continual Learning (TI-CL) of audio classifiers with both parameter-efficient and compute-efficient Audio Spectrogram Transformers (AST). To reduce the trainable parameters without performance degradation for TI-CL, we compare several Parameter Efficient Transfer (PET) methods and propose AST with Convolutional Adapters for TI-CL, which has less than 5% of trainable parameters of the fully fine-tuned counterparts. To reduce the computational complexity, we introduce a novel Frequency-Time factorized Attention (FTA) method that replaces the traditional self-attention in transformers for audio spectrograms. FTA achieves competitive performance with only a factor of the computations required by Global Self-Attention (GSA). Finally, we formulate our method for TI-CL, called Adapter Incremental Continual Learning (AI-CL), as a combination of the "parameter-efficient" Convolutional Adapter and the "compute-efficient" FTA. Experiments on ESC-50, SpeechCommandsV2 (SCv2), and Audio-Visual Event (AVE) benchmarks show that our proposed method prevents catastrophic forgetting in TI-CL while maintaining a lower computational budget.

【8】 HalluAudio: Hallucinating Frequency as Concepts for Few-Shot Audio Classification标题：HalluAudio：幻觉频率作为Few-Shot音频分类的概念链接：http://arxiv.org/pdf/2302.14204v1

作者：Zhongjie Yu,Shuyang Wang,Lin Chen,Zhongwei Cheng机构：Wyze Labs, Inc.备注：Accepted at ICASSP 2023摘要：Few-Shot音频分类是一个新兴的研究课题，受到了研究界越来越多的关注。大多数现有的研究忽略了音频谱图形式的特殊性，而主要关注于从图像任务中借用的嵌入空间，而在本研究中，我们旨在利用这种特殊的音频格式，通过将高频和低频部分幻觉为结构化概念，提出一种新的方法。在ESC-50数据集和平衡Kaggle 18数据集上的大量实验表明，该方法的性能明显优于基线方法.我们的方法幻觉高频和低频部分的方式也使其具有可解释性，并为Few-Shot音频分类开辟了新的潜力。摘要：Few-shot audio classification is an emerging topic that attracts more and more attention from the research community. Most existing work ignores the specificity of the form of the audio spectrogram and focuses largely on the embedding space borrowed from image tasks, while in this work, we aim to take advantage of this special audio format and propose a new method by hallucinating high-frequency and low-frequency parts as structured concepts. Extensive experiments on ESC-50 and our curated balanced Kaggle18 dataset show the proposed method outperforms the baseline by a notable margin. The way that our method hallucinates high-frequency and low-frequency parts also enables its interpretability and opens up new potentials for the few-shot audio classification.

【9】 Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding标题：用于语音识别和理解的自监督预训练模型的结构化剪枝链接：http://arxiv.org/pdf/2302.14132v1

作者：Yifan Peng,Kwangyoun Kim,Felix Wu,Prashant Sridhar,Shinji Watanabe机构：ASAPP Inc., Mountain View, CA, USA, Carnegie Mellon University, Pittsburgh, PA, USA备注：Accepted at ICASSP 2023摘要：自监督语音表示学习（SSL）在各种下游任务中已经显示出有效性，但是SSL模型通常较大且较慢。模型压缩技术（如修剪）旨在减少模型大小和计算量，而不降低精度。先前的研究集中在Transformers的修剪;然而，语音模型不仅利用一堆Transformer块，而且还结合基于多个卷积层的前端网络用于低级特征表示学习。该前端具有较小的尺寸，但是计算成本很高。在这项工作中，我们提出了三个任务特定的结构化修剪方法来处理这种异构网络。在LibriSpeech和SLURP上的实验结果表明，该方法比原始的wav 2 vec 2-base算法准确率高，计算量减少10% ~ 30%，在不降低性能的前提下，计算量减少40% ~ 50%.摘要：Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques such as pruning aim to reduce the model size and computation without degradation in accuracy. Prior studies focus on the pruning of Transformers; however, speech models not only utilize a stack of Transformer blocks, but also combine a frontend network based on multiple convolutional layers for low-level feature representation learning. This frontend has a small size but a heavy computational cost. In this work, we propose three task-specific structured pruning methods to deal with such heterogeneous networks. Experiments on LibriSpeech and SLURP show that the proposed method is more accurate than the original wav2vec2-base with 10% to 30% less computation, and is able to reduce the computation by 40% to 50% without any degradation.

【10】 Explanations for Automatic Speech Recognition标题：自动语音识别说明链接：http://arxiv.org/pdf/2302.14062v1

作者：Xiaoliang Wu,Peter Bell,Ajitha Rajan机构：School of Informatics, University of Edinburgh备注：Accepted by Speech Track, ICASSP 2023摘要：我们通过提供有助于增加我们对系统的理解并最终有助于建立对系统的信任的解释来解决基于神经网络的ASR的质量评估。与简单的分类标签相比，解释转录更具挑战性，因为判断它们的正确性并不简单，并且现有的可解释机器学习模型无法处理作为可变长度序列的转录。我们将ASR转录解释为音频帧的子集，它既是转录的最小原因也是充分原因。为此，我们从图像分类-统计故障定位（SFL）和因果关系调整现有的可解释AI（XAI）技术。此外，我们使用了ASR的局部可解释模型不可知解释（LIME）的修改版本作为我们实验中的基线。我们在三种不同的ASR、Google API、Sphinx的基线模型、Deepspeech和来自Commonvoice数据集的100个音频样本上评估了由所提出的技术生成的解释的质量。摘要：We address quality assessment for neural network based ASR by providing explanations that help increase our understanding of the system and ultimately help build trust in the system. Compared to simple classification labels, explaining transcriptions is more challenging as judging their correctness is not straightforward and transcriptions as a variable-length sequence is not handled by existing interpretable machine learning models. We provide an explanation for an ASR transcription as a subset of audio frames that is both a minimal and sufficient cause of the transcription. To do this, we adapt existing explainable AI (XAI) techniques from image classification-Statistical Fault Localisation(SFL) and Causal. Additionally, we use an adapted version of Local Interpretable Model-Agnostic Explanations (LIME) for ASR as a baseline in our experiments. We evaluate the quality of the explanations generated by the proposed techniques over three different ASR ,Google API, the baseline model of Sphinx, Deepspeech and 100 audio samples from the Commonvoice dataset.

【11】Phone and speaker spatial organization in self-supervised speech representations标题：自监督语音表征中的音素和说话人空间组织链接：http://arxiv.org/pdf/2302.14055v1

作者：Pablo Riera,Manuela Cerdeiro,Leonardo Pepino,Luciana Ferrer机构：⋆Instituto de Investigaci´on en Ciencias de la Computaci´on (ICC), CONICET-UBA, Argentina, †Departamento de Computaci´on, FCEyN, Universidad de Buenos Aires (UBA), Argentina摘要：语音的自监督表示目前正广泛用于大量应用。最近，已经做出了一些努力来尝试分析这些表示中的每一个中存在的信息类型。大多数这样的工作使用下游模型来测试表示是否可以成功地用于特定任务。然而，下游模型通常对表示执行非线性操作，提取在原始表示中可能不容易获得的信息。在这项工作中，我们使用不需要下游模型的方法分析了几种最先进的语音表示中音素和说话人信息的空间组织。我们使用表示相似性分析来测量不同层如何编码基本声学参数，例如共振峰和音调。此外，我们使用非参数统计测试来研究每种表示法按音素或说话人类别对语音样本进行聚类的程度。我们的结果表明，根据预训练中使用的目标任务，模型不同地表示这些语音属性。摘要：Self-supervised representations of speech are currently being widely used for a large number of applications. Recently, some efforts have been made in trying to analyze the type of information present in each of these representations. Most such work uses downstream models to test whether the representations can be successfully used for a specific task. The downstream models, though, typically perform nonlinear operations on the representation extracting information that may not have been readily available in the original representation. In this work, we analyze the spatial organization of phone and speaker information in several state-of-the-art speech representations using methods that do not require a downstream model. We measure how different layers encode basic acoustic parameters such as formants and pitch using representation similarity analysis. Further, we study the extent to which each representation clusters the speech samples by phone or speaker classes using non-parametric statistical testing. Our results indicate that models represent these speech attributes differently depending on the target task used during pretraining.

【12】  Incremental Learning of Acoustic Scenes and Sound Events标题：声音场景和声音事件的增量学习

链接：http://arxiv.org/pdf/2302.14815v1

作者：Manjunath Mulimani,Annamaria Mesaros机构：Computing Sciences, Tampere University, Tampere, Finland摘要：在本文中，我们提出了一种用于两个不同任务随时间的增量学习的方法：声学场景分类（ASC）和音频标记（AT）。我们使用一个简单的卷积神经网络（CNN）模型作为增量学习器来解决这些问题。通常，增量学习方法在对新任务顺序地训练时灾难性地忘记先前任务。为了缓解这个问题，我们在学习的时间步之间使用了独立学习和知识提炼（KD）。实验在TUT 2016 2017数据集上进行，该数据集包含4个声学场景类和25个声音事件类。增量学习器在增量时间步内完成AT任务的F1得分为54.4%，完成ASC任务的准确率为88.9%，优于同时完成ASC和AT的多任务系统。ASC任务性能仅比初始时间ASC准确率94.0%下降5.1%。摘要：In this paper, we propose a method for incremental learning of two distinct tasks over time: acoustic scene classification (ASC) and audio tagging (AT). We use a simple convolutional neural network (CNN) model as an incremental learner to solve the tasks. Generally, incremental learning methods catastrophically forget the previous task when sequentially trained on a new task. To alleviate this problem, we use independent learning and knowledge distillation (KD) between the timesteps in learning. Experiments are performed on TUT 2016 2017 dataset, containing 4 acoustic scene classes and 25 sound event classes. The proposed incremental learner solves the AT task with an F1 score of 54.4% and the ASC task with an accuracy of 88.9% in an incremental time step, outperforming a multi-task system which solves ASC and AT at the same time. The ASC task performance degrades only by 5.1% from the initial time ASC accuracy of 94.0%.

【13】 Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement标题：基于扩散的语音增强降低随机微分方程先验失配链接：http://arxiv.org/pdf/2302.14748v1

作者：Bunlong Lay,Simon Welker,Julius Richter,Timo Gerkmann机构：Signal Processing (SP), University of Hamburg, Germany, Center for Free-Electron Laser Science, DESY, Hamburg, Germany备注：5 pages, 3 figures摘要：近年来，基于分数的生成模型已经成功地应用于语音增强任务。使用随机微分方程来模拟迭代前向过程，其中在每一步将环境噪声和白高斯噪声添加到干净语音信号。虽然在极限情况下，前向过程的平均值在噪声混合处结束，但在实践中，它停止得更早，因此仅在噪声混合的近似处停止。这导致正向过程的终止分布与用于在推理时求解反向过程的先验分布之间的差异。在本文中，我们解决了这一矛盾。为此，我们提出了一个基于布朗桥的正向过程，并证明了与以前的扩散过程相比，这种过程导致失配的减少。更重要的是，我们证明了我们的方法在客观度量上比基线过程有所改进，迭代步骤只有一半，并且需要调优的超参数也少了一个。摘要：Recently, score-based generative models have been successfully employed for the task of speech enhancement. A stochastic differential equation is used to model the iterative forward process, where at each step environmental noise and white Gaussian noise are added to the clean speech signal. While in limit the mean of the forward process ends at the noisy mixture, in practice it stops earlier and thus only at an approximation of the noisy mixture. This results in a discrepancy between the terminating distribution of the forward process and the prior used for solving the reverse process at inference. In this paper, we address this discrepancy. To this end, we propose a forward process based on a Brownian bridge and show that such a process leads to a reduction of the mismatch compared to previous diffusion processes. More importantly, we show that our approach improves in objective metrics over the baseline process with only half of the iteration steps and having one hyperparameter less to tune.

【14】 SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing标题：SpeechFormer++：一种分层高效的副语言语音处理框架链接：http://arxiv.org/pdf/2302.14638v1

作者：Weidong Chen,Xiaofen Xing,Xiangmin Xu,Jianxin Pang,Lan Du备注：14 pages, 7 figures, 14 tables, TASLP 2023 paper摘要：副语言语音处理在情感和神经认知障碍分析等许多领域都有重要的应用。近年来，Transformer在自然语言处理领域取得了显著的成功，并展示了其对语音的适应性。然而，在语音领域中，先前关于Transformer的工作没有结合语音的属性，使得Transformer的全部潜力未被开发。本文结合语音的特点，提出了一个基于结构的副语言语音处理框架SpeechFormer++。更具体地，遵循语音信号中的分量关系，我们设计单元编码器来对单元内和单元间信息（即，帧、音素和单词）。根据层次关系，利用分块合并生成不同粒度的特征，这与语音信号的结构模式是一致的。此外，引入了一个字编码器，将字粒度特征集成到每个单元编码器中，有效地平衡了细粒度和粗粒度信息。SpeechFormer++在语音情感识别（IEMOCAP & MELD）、抑郁症分类（DAIC-WOZ）和阿尔茨海默病检测（Pitt）任务上进行了评估。实验结果表明，SpeechFormer++的性能优于标准Transformer，同时大大降低了计算成本。此外，与最先进的方法相比，它提供了更好的结果。摘要：Paralinguistic speech processing is important in addressing many issues, such as sentiment and neurocognitive disorder analyses. Recently, Transformer has achieved remarkable success in the natural language processing field and has demonstrated its adaptation to speech. However, previous works on Transformer in the speech field have not incorporated the properties of speech, leaving the full potential of Transformer unexplored. In this paper, we consider the characteristics of speech and propose a general structure-based framework, called SpeechFormer++, for paralinguistic speech processing. More concretely, following the component relationship in the speech signal, we design a unit encoder to model the intra- and inter-unit information (i.e., frames, phones, and words) efficiently. According to the hierarchical relationship, we utilize merging blocks to generate features at different granularities, which is consistent with the structural pattern in the speech signal. Moreover, a word encoder is introduced to integrate word-grained features into each unit encoder, which effectively balances fine-grained and coarse-grained information. SpeechFormer++ is evaluated on the speech emotion recognition (IEMOCAP & MELD), depression classification (DAIC-WOZ) and Alzheimer's disease detection (Pitt) tasks. The results show that SpeechFormer++ outperforms the standard Transformer while greatly reducing the computational cost. Furthermore, it delivers superior results compared to the state-of-the-art approaches.

【15】 Training sound event detection with soft labels from crowdsourced annotations标题：利用众包标注中的软标签训练声音事件检测链接：http://arxiv.org/pdf/2302.14572v1

作者：Irene Martín-Morató,Manu Harju,Paul Ahokas,Annamaria Mesaros机构：Computing Sciences, Tampere University, Tampere, Finland备注：ICASSP 2023摘要：在这篇文章中，我们研究了使用软标签来训练一个声音事件检测（SED）系统。软标签可以由考虑了人类对类别的不确定性的注释产生，或者作为注释中多个观点的自然表示出现。将注释转换为硬标签会产生用于训练的明确类别，代价是丢失有关标签分布的详细信息。这项工作调查了如何软标签可以使用，以及他们带来了什么好处，在培训一个SED系统。结果表明，该系统能够学习关于反映在软标签中的声音的活动的信息，并且能够检测在典型的二元目标训练设置中遗漏的声音。我们还发布了一个通过众包产生的新数据集，其中包含现实生活录音中声音事件的时间强标签，包括软标签和硬标签。摘要：In this paper, we study the use of soft labels to train a system for sound event detection (SED). Soft labels can result from annotations which account for human uncertainty about categories, or emerge as a natural representation of multiple opinions in annotation. Converting annotations to hard labels results in unambiguous categories for training, at the cost of losing the details about the labels distribution. This work investigates how soft labels can be used, and what benefits they bring in training a SED system. The results show that the system is capable of learning information about the activity of the sounds which is reflected in the soft labels and is able to detect sounds that are missed in the typical binary target training setup. We also release a new dataset produced through crowdsourcing, containing temporally strong labels for sound events in real-life recordings, with both soft and hard labels.

【16】 Renaissance canons with asymmetric schemes标题：文艺复兴时期的非对称方案链接：http://arxiv.org/pdf/2302.14160v1

作者：Evan M. O'Dorney备注：17 pages, including 13 figures, 3 tables, and a 2-page illustrative musical composition摘要：所谓音乐经典的“方案”，我们指的是声音进入的顺序，以及每个进入声音的时间和音高位移。当时间位移不相等时，要达到协和的响度就特别困难。使用第一物种理论模型，我们量化了文艺复兴时期作曲家使用或可能使用的方案的灵活性。我们设计了一个算法来精确计算柔度值（在这个过程中发现它是皮索数的$h$次根）。我们发现，帕莱斯特里纳一贯选择一些最灵活的方案，比他的前任，但他绝没有用尽可行的方案。我们结束了提出一个新的组成范围内的风格利用一个未经探索的佳能计划。摘要：By a "scheme" of a musical canon, we mean the order of voice entry with the time and pitch displacement of each entering voice. When the time displacements are unequal, achieving consonant sonorities is especially challenging. Using a first-species theoretical model, we quantify the flexibility of schemes that Renaissance composers used or could have used. We craft an algorithm to compute the flexibility value precisely (finding in the process that it is an $h$-th root of a Pisot number). We find that Palestrina consistently selected some of the most flexible schemes, more so than his predecessors, but that he by no means exhausted the feasible schemes. We close by presenting a new composition within the limits of the style utilizing an unexplored canonic scheme.

【17】 Diagonal State Space Augmented Transformers for Speech Recognition标题：对角状态空间增广变换在语音识别中的应用链接：http://arxiv.org/pdf/2302.14120v1

作者：George Saon,Ankit Gupta,Xiaodong Cui机构：IBM Research AI, Yorktown Heights, USA备注：to be presented at ICASSP 2023摘要：我们通过用对角状态空间（DSS）模型代替深度时间卷积来改进流行的构象器结构。DSS是最近引入的线性RNN的变体，其通过使用对角状态转移矩阵离散化线性动态系统而获得。DSS层将输入序列投影到正交多项式空间上，其中基函数、度量和支持的选择由转移矩阵的特征值控制。我们在三个公共语料库上比较神经传感器与构象器或我们提出的DSS增强Transformer（DSSformer）编码器：总机英语会话电话演讲300小时，总机+费舍尔2000小时，和一个大屠杀幸存者的口头档案称为马拉赫176小时。在Switchboard 300 2000小时上，我们在Hub 5 2000评估的组合测试集上分别达到了8.9% 6.7%的WER的单个模型性能，并且在MALACH上，我们相对于先前最佳公布结果将WER提高了7%。此外，我们提出的经验证据表明，DSS层学习阻尼傅里叶基函数，其中衰减系数是层特定的，而频率系数收敛到几乎相同的线性间隔值跨越所有层。摘要：We improve on the popular conformer architecture by replacing the depthwise temporal convolutions with diagonal state space (DSS) models. DSS is a recently introduced variant of linear RNNs obtained by discretizing a linear dynamical system with a diagonal state transition matrix. DSS layers project the input sequence onto a space of orthogonal polynomials where the choice of basis functions, metric and support is controlled by the eigenvalues of the transition matrix. We compare neural transducers with either conformer or our proposed DSS-augmented transformer (DSSformer) encoders on three public corpora: Switchboard English conversational telephone speech 300 hours, Switchboard+Fisher 2000 hours, and a spoken archive of holocaust survivor testimonials called MALACH 176 hours. On Switchboard 300 2000 hours, we reach a single model performance of 8.9% 6.7% WER on the combined test set of the Hub5 2000 evaluation, respectively, and on MALACH we improve the WER by 7% relative over the previous best published result. In addition, we present empirical evidence suggesting that DSS layers learn damped Fourier basis functions where the attenuation coefficients are layer specific whereas the frequency coefficients converge to almost identical linearly-spaced values across all layers.
eess.AS音频处理

【1】  Incremental Learning of Acoustic Scenes and Sound Events标题：声音场景和声音事件的增量学习

链接：http://arxiv.org/pdf/2302.14815v1

*与cs.SD语音【12】为同一篇

作者：Manjunath Mulimani,Annamaria Mesaros机构：Computing Sciences, Tampere University, Tampere, Finland摘要：在本文中，我们提出了一种用于两个不同任务随时间的增量学习的方法：声学场景分类（ASC）和音频标记（AT）。我们使用一个简单的卷积神经网络（CNN）模型作为增量学习器来解决这些问题。通常，增量学习方法在对新任务顺序地训练时灾难性地忘记先前任务。为了缓解这个问题，我们在学习的时间步之间使用了独立学习和知识提炼（KD）。实验在TUT 2016 2017数据集上进行，该数据集包含4个声学场景类和25个声音事件类。增量学习器在增量时间步内完成AT任务的F1得分为54.4%，完成ASC任务的准确率为88.9%，优于同时完成ASC和AT的多任务系统。ASC任务性能仅比初始时间ASC准确率94.0%下降5.1%。摘要：In this paper, we propose a method for incremental learning of two distinct tasks over time: acoustic scene classification (ASC) and audio tagging (AT). We use a simple convolutional neural network (CNN) model as an incremental learner to solve the tasks. Generally, incremental learning methods catastrophically forget the previous task when sequentially trained on a new task. To alleviate this problem, we use independent learning and knowledge distillation (KD) between the timesteps in learning. Experiments are performed on TUT 2016 2017 dataset, containing 4 acoustic scene classes and 25 sound event classes. The proposed incremental learner solves the AT task with an F1 score of 54.4% and the ASC task with an accuracy of 88.9% in an incremental time step, outperforming a multi-task system which solves ASC and AT at the same time. The ASC task performance degrades only by 5.1% from the initial time ASC accuracy of 94.0%.

【2】 Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement标题：基于扩散的语音增强降低随机微分方程先验失配链接：http://arxiv.org/pdf/2302.14748v1

*与cs.SD语音【13】为同一篇

作者：Bunlong Lay,Simon Welker,Julius Richter,Timo Gerkmann机构：Signal Processing (SP), University of Hamburg, Germany, Center for Free-Electron Laser Science, DESY, Hamburg, Germany备注：5 pages, 3 figures摘要：近年来，基于分数的生成模型已经成功地应用于语音增强任务。使用随机微分方程来模拟迭代前向过程，其中在每一步将环境噪声和白高斯噪声添加到干净语音信号。虽然在极限情况下，前向过程的平均值在噪声混合处结束，但在实践中，它停止得更早，因此仅在噪声混合的近似处停止。这导致正向过程的终止分布与用于在推理时求解反向过程的先验分布之间的差异。在本文中，我们解决了这一矛盾。为此，我们提出了一个基于布朗桥的正向过程，并证明了与以前的扩散过程相比，这种过程导致失配的减少。更重要的是，我们证明了我们的方法在客观度量上比基线过程有所改进，迭代步骤只有一半，并且需要调优的超参数也少了一个。摘要：Recently, score-based generative models have been successfully employed for the task of speech enhancement. A stochastic differential equation is used to model the iterative forward process, where at each step environmental noise and white Gaussian noise are added to the clean speech signal. While in limit the mean of the forward process ends at the noisy mixture, in practice it stops earlier and thus only at an approximation of the noisy mixture. This results in a discrepancy between the terminating distribution of the forward process and the prior used for solving the reverse process at inference. In this paper, we address this discrepancy. To this end, we propose a forward process based on a Brownian bridge and show that such a process leads to a reduction of the mismatch compared to previous diffusion processes. More importantly, we show that our approach improves in objective metrics over the baseline process with only half of the iteration steps and having one hyperparameter less to tune.

【3】 SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing标题：SpeechFormer++：一种分层高效的副语言语音处理框架链接：http://arxiv.org/pdf/2302.14638v1

*与cs.SD语音【14】为同一篇

作者：Weidong Chen,Xiaofen Xing,Xiangmin Xu,Jianxin Pang,Lan Du备注：14 pages, 7 figures, 14 tables, TASLP 2023 paper摘要：副语言语音处理在情感和神经认知障碍分析等许多领域都有重要的应用。近年来，Transformer在自然语言处理领域取得了显著的成功，并展示了其对语音的适应性。然而，在语音领域中，先前关于Transformer的工作没有结合语音的属性，使得Transformer的全部潜力未被开发。本文结合语音的特点，提出了一个基于结构的副语言语音处理框架SpeechFormer++。更具体地，遵循语音信号中的分量关系，我们设计单元编码器来对单元内和单元间信息（即，帧、音素和单词）。根据层次关系，利用分块合并生成不同粒度的特征，这与语音信号的结构模式是一致的。此外，引入了一个字编码器，将字粒度特征集成到每个单元编码器中，有效地平衡了细粒度和粗粒度信息。SpeechFormer++在语音情感识别（IEMOCAP & MELD）、抑郁症分类（DAIC-WOZ）和阿尔茨海默病检测（Pitt）任务上进行了评估。实验结果表明，SpeechFormer++的性能优于标准Transformer，同时大大降低了计算成本。此外，与最先进的方法相比，它提供了更好的结果。摘要：Paralinguistic speech processing is important in addressing many issues, such as sentiment and neurocognitive disorder analyses. Recently, Transformer has achieved remarkable success in the natural language processing field and has demonstrated its adaptation to speech. However, previous works on Transformer in the speech field have not incorporated the properties of speech, leaving the full potential of Transformer unexplored. In this paper, we consider the characteristics of speech and propose a general structure-based framework, called SpeechFormer++, for paralinguistic speech processing. More concretely, following the component relationship in the speech signal, we design a unit encoder to model the intra- and inter-unit information (i.e., frames, phones, and words) efficiently. According to the hierarchical relationship, we utilize merging blocks to generate features at different granularities, which is consistent with the structural pattern in the speech signal. Moreover, a word encoder is introduced to integrate word-grained features into each unit encoder, which effectively balances fine-grained and coarse-grained information. SpeechFormer++ is evaluated on the speech emotion recognition (IEMOCAP & MELD), depression classification (DAIC-WOZ) and Alzheimer's disease detection (Pitt) tasks. The results show that SpeechFormer++ outperforms the standard Transformer while greatly reducing the computational cost. Furthermore, it delivers superior results compared to the state-of-the-art approaches.

【4】 Training sound event detection with soft labels from crowdsourced annotations标题：利用众包标注中的软标签训练声音事件检测链接：http://arxiv.org/pdf/2302.14572v1

*与cs.SD语音【15】为同一篇

作者：Irene Martín-Morató,Manu Harju,Paul Ahokas,Annamaria Mesaros机构：Computing Sciences, Tampere University, Tampere, Finland备注：ICASSP 2023摘要：在这篇文章中，我们研究了使用软标签来训练一个声音事件检测（SED）系统。软标签可以由考虑了人类对类别的不确定性的注释产生，或者作为注释中多个观点的自然表示出现。将注释转换为硬标签会产生用于训练的明确类别，代价是丢失有关标签分布的详细信息。这项工作调查了如何软标签可以使用，以及他们带来了什么好处，在培训一个SED系统。结果表明，该系统能够学习关于反映在软标签中的声音的活动的信息，并且能够检测在典型的二元目标训练设置中遗漏的声音。我们还发布了一个通过众包产生的新数据集，其中包含现实生活录音中声音事件的时间强标签，包括软标签和硬标签。摘要：In this paper, we study the use of soft labels to train a system for sound event detection (SED). Soft labels can result from annotations which account for human uncertainty about categories, or emerge as a natural representation of multiple opinions in annotation. Converting annotations to hard labels results in unambiguous categories for training, at the cost of losing the details about the labels distribution. This work investigates how soft labels can be used, and what benefits they bring in training a SED system. The results show that the system is capable of learning information about the activity of the sounds which is reflected in the soft labels and is able to detect sounds that are missed in the typical binary target training setup. We also release a new dataset produced through crowdsourcing, containing temporally strong labels for sound events in real-life recordings, with both soft and hard labels.

【5】 Diagonal State Space Augmented Transformers for Speech Recognition标题：对角状态空间增广变换在语音识别中的应用链接：http://arxiv.org/pdf/2302.14120v1

*与cs.SD语音【17】为同一篇

作者：George Saon,Ankit Gupta,Xiaodong Cui机构：IBM Research AI, Yorktown Heights, USA备注：to be presented at ICASSP 2023摘要：我们通过用对角状态空间（DSS）模型代替深度时间卷积来改进流行的构象器结构。DSS是最近引入的线性RNN的变体，其通过使用对角状态转移矩阵离散化线性动态系统而获得。DSS层将输入序列投影到正交多项式空间上，其中基函数、度量和支持的选择由转移矩阵的特征值控制。我们在三个公共语料库上比较神经传感器与构象器或我们提出的DSS增强Transformer（DSSformer）编码器：总机英语会话电话演讲300小时，总机+费舍尔2000小时，和一个大屠杀幸存者的口头档案称为马拉赫176小时。在Switchboard 300 2000小时上，我们在Hub 5 2000评估的组合测试集上分别达到了8.9% 6.7%的WER的单个模型性能，并且在MALACH上，我们相对于先前最佳公布结果将WER提高了7%。此外，我们提出的经验证据表明，DSS层学习阻尼傅里叶基函数，其中衰减系数是层特定的，而频率系数收敛到几乎相同的线性间隔值跨越所有层。摘要：We improve on the popular conformer architecture by replacing the depthwise temporal convolutions with diagonal state space (DSS) models. DSS is a recently introduced variant of linear RNNs obtained by discretizing a linear dynamical system with a diagonal state transition matrix. DSS layers project the input sequence onto a space of orthogonal polynomials where the choice of basis functions, metric and support is controlled by the eigenvalues of the transition matrix. We compare neural transducers with either conformer or our proposed DSS-augmented transformer (DSSformer) encoders on three public corpora: Switchboard English conversational telephone speech 300 hours, Switchboard+Fisher 2000 hours, and a spoken archive of holocaust survivor testimonials called MALACH 176 hours. On Switchboard 300 2000 hours, we reach a single model performance of 8.9% 6.7% WER on the combined test set of the Hub5 2000 evaluation, respectively, and on MALACH we improve the WER by 7% relative over the previous best published result. In addition, we present empirical evidence suggesting that DSS layers learn damped Fourier basis functions where the attenuation coefficients are layer specific whereas the frequency coefficients converge to almost identical linearly-spaced values across all layers.

【6】 Audio Retrieval for Multimodal Design Documents: A New Dataset and Algorithms标题：多通道设计文档音频检索：一种新的数据集和算法链接：http://arxiv.org/pdf/2302.14757v1

*与cs.SD语音【1】为同一篇

作者：Prachi Singh,Srikrishna Karanam,Sumit Shekhar机构：Adobe Research, Bangalore, India, Indian Institute of Science, Bangalore, India备注：5 pages including references摘要：我们考虑并提出了一种检索与包括文本元素和视觉图像两者的多模态设计文档输入相关的音频文件的新问题，生日 贺卡。除了增强用户体验之外，集成与这些输入的主题 风格相匹配的音频还有助于改善这些文档的可访问性（例如，视觉受损的人可以改为收听音频）。虽然最近在音频检索方面的工作存在，但是这些方法和数据集明确地针对自然图像。然而，我们的问题认为多模态设计文档（由用户使用创造性软件创建）与自然点击的照片有本质上的不同。为此，我们的第一个贡献是收集和管理一个新的大规模数据集，称为旋律设计（或MELON），包括代表各种风格、主题、模板、插图等的设计文档。与音乐音频配对。给定我们的配对图像-文本-音频数据集，我们的下一个贡献是一个新颖的多模态交叉注意音频检索（MMCAR）算法，该算法使训练神经网络能够学习跨图像、文本和音频维度的公共共享特征空间。我们使用这些学习到的特征来证明我们的方法优于现有的最先进的方法，并为研究团体在我们的新数据集上产生一个新的参考基准。摘要：We consider and propose a new problem of retrieving audio files relevant to multimodal design document inputs comprising both textual elements and visual imagery, e.g., birthday greeting cards. In addition to enhancing user experience, integrating audio that matches the theme style of these inputs also helps improve the accessibility of these documents (e.g., visually impaired people can listen to the audio instead). While recent work in audio retrieval exists, these methods and datasets are targeted explicitly towards natural images. However, our problem considers multimodal design documents (created by users using creative software) substantially different from a naturally clicked photograph. To this end, our first contribution is collecting and curating a new large-scale dataset called Melodic-Design (or MELON), comprising design documents representing various styles, themes, templates, illustrations, etc., paired with music audio. Given our paired image-text-audio dataset, our next contribution is a novel multimodal cross-attention audio retrieval (MMCAR) algorithm that enables training neural networks to learn a common shared feature space across image, text, and audio dimensions. We use these learned features to demonstrate that our method outperforms existing state-of-the-art methods and produce a new reference benchmark for the research community on our new dataset.

【7】 deHuBERT: Disentangling Noise in a Self-supervised Model for Robust Speech Recognition标题：DeHuBERT：用于稳健语音识别的自监督模型中的噪声分离链接：http://arxiv.org/pdf/2302.14597v1

*与cs.SD语音【2】为同一篇

作者：Dianwen Ng,Ruixi Zhang,Jia Qi Yip,Zhao Yang,Jinjie Ni,Chong Zhang,Yukun Ma,Chongjia Ni,Eng Siong Chng,Bin Ma机构：Alibaba Group, School of Computer Science and Engineering, Nanyang Technological University, Singapore, Xi’an Jiaotong University, China备注：Accepted by ICASSP 2023摘要：现有的自监督预训练语音模型为利用海量未标注语料库建立良好的自动语音识别（ASR）提供了一种有效的方法。然而，许多当前的模型是在来自单个源的干净语料库上训练的，这在测试期间存在噪声时往往表现不佳。然而，克服噪声对实际应用的不利影响是至关重要的。在这项工作中，我们提出了一个新的训练框架，称为deHuBERT，用于降噪编码的启发H。巴洛的冗余减少原理。新框架通过引入辅助损失来改进HuBERT训练算法，辅助损失将成对噪声失真嵌入之间的自相关矩阵和互相关矩阵推向单位矩阵。这鼓励模型产生噪声不可知的语音表示。通过这种方法，我们报告了在噪声环境中（包括看不见的噪声）改进的鲁棒性，而不损害在清洁集上的性能。摘要：Existing self-supervised pre-trained speech models have offered an effective way to leverage massive unannotated corpora to build good automatic speech recognition (ASR). However, many current models are trained on a clean corpus from a single source, which tends to do poorly when noise is present during testing. Nonetheless, it is crucial to overcome the adverse influence of noise for real-world applications. In this work, we propose a novel training framework, called deHuBERT, for noise reduction encoding inspired by H. Barlow's redundancy-reduction principle. The new framework improves the HuBERT training algorithm by introducing auxiliary losses that drive the self- and cross-correlation matrix between pairwise noise-distorted embeddings towards identity matrix. This encourages the model to produce noise-agnostic speech representations. With this method, we report improved robustness in noisy environments, including unseen noises, without impairing the performance on the clean set.

【8】 Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition标题：用于节律障碍和老年人语音识别的自监督预训练ASR模型探索链接：http://arxiv.org/pdf/2302.14564v1

*与cs.SD语音【3】为同一篇

作者：Shujie Hu,Xurong Xie,Zengrui Jin,Mengzhe Geng,Yi Wang,Mingyu Cui,Jiajun Deng,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Institute of Software, Chinese Academy of Sciences, China摘要：由于难以大量收集这类数据，迄今为止，无序和老年语音的自动识别仍然是一项极具挑战性的任务。本文探索了一系列方法，将域自适应SSL预训练模型集成到TDNN和Conformer ASR系统中，用于构音障碍和老年人语音识别：a）标准声学前端和域自适应wav2vec2.0语音表示之间的输入特征融合;b）单独使用标准声学特征和使用附加wav2vec2.0特征单独训练的TDNN系统的帧级联合解码;以及c）涉及使用域自适应wav2vec2.0模型重新评分的TDNN Conformer系统输出的多通路解码。此外，将域自适应wav 2 vec 2. 0表示用于声学到发音（A2 A）反演，以构建多模态构音障碍和老年人语音识别系统。在UASpeech构音障碍和DementiaBank Pitt老年人语音语料库上进行的实验表明，TDNN和Conformer ASR系统集成的域自适应wav2vec2.0模型在统计学上显著优于独立的wav2vec2.0模型，在两个任务上分别降低了8.22%和3.43%绝对值（26.71%和15.88%相对值）。在16名构音障碍者的UASpeech测试集和DementiaBank Pitt测试集上分别获得了22.56%的最低公布WER（极低可懂度52.53%，看不见的单词39.09%）和18.17%。摘要：Automatic recognition of disordered and elderly speech remains a highly challenging task to date due to the difficulty in collecting such data in large quantities. This paper explores a series of approaches to integrate domain adapted SSL pre-trained models into TDNN and Conformer ASR systems for dysarthric and elderly speech recognition: a) input feature fusion between standard acoustic frontends and domain adapted wav2vec2.0 speech representations; b) frame-level joint decoding of TDNN systems separately trained using standard acoustic features alone and with additional wav2vec2.0 features; and c) multi-pass decoding involving the TDNN Conformer system outputs to be rescored using domain adapted wav2vec2.0 models. In addition, domain adapted wav2vec2.0 representations are utilized in acoustic-to-articulatory (A2A) inversion to construct multi-modal dysarthric and elderly speech recognition systems. Experiments conducted on the UASpeech dysarthric and DementiaBank Pitt elderly speech corpora suggest TDNN and Conformer ASR systems integrated domain adapted wav2vec2.0 models consistently outperform the standalone wav2vec2.0 models by statistically significant WER reductions of 8.22% and 3.43% absolute (26.71% and 15.88% relative) on the two tasks respectively. The lowest published WERs of 22.56% (52.53% on very low intelligibility, 39.09% on unseen words) and 18.17% are obtained on the UASpeech test set of 16 dysarthric speakers, and the DementiaBank Pitt test set respectively.

【9】 CrossSpeech: Speaker-independent Acoustic Representation for Cross-lingual Speech Synthesis标题：CrossSpeech：跨语言语音合成的非特定人声学表示链接：http://arxiv.org/pdf/2302.14370v1

*与cs.SD语音【4】为同一篇

作者：Ji-Hoon Kim,Hong-Sun Yang,Yoon-Cheol Ju,Il-Hwan Kim,Byeong-Yeol Kim机构：dot Inc., Seoul, Republic of Korea备注：Accepted to ICASSP 2023摘要：虽然最近的文本到语音（TTS）系统在人类水平的质量方面取得了显著的进步，但是跨语言TTS的性能落后于语言内TTS的性能。这一差距主要源于跨语言TTS中的说话人-语言纠缠问题。本文提出了CrossSpeech算法，该算法通过在声学特征空间层次上有效地分离说话人和语言信息，提高了跨语言语音的质量。具体来说，CrossSpeech将语音生成流水线分解为说话人无关生成器（SIG）和说话人相关生成器（SDG）。SIG产生不偏向于特定说话者分布的说话者无关的声学表示。另一方面，SDG对表征说话人属性的说话人相关语音变化进行建模。通过分别处理每一种信息，CrossSpeech可以获得分离的说话人和语言表示。通过实验，我们验证了CrossSpeech在跨语言TTS中取得了显著的改进，特别是在说话人与目标说话人的相似度方面。摘要：While recent text-to-speech (TTS) systems have made remarkable strides toward human-level quality, the performance of cross-lingual TTS lags behind that of intra-lingual TTS. This gap is mainly rooted from the speaker-language entanglement problem in cross-lingual TTS. In this paper, we propose CrossSpeech which improves the quality of cross-lingual speech by effectively disentangling speaker and language information in the level of acoustic feature space. Specifically, CrossSpeech decomposes the speech generation pipeline into the speaker-independent generator (SIG) and speaker-dependent generator (SDG). The SIG produces the speaker-independent acoustic representation which is not biased to specific speaker distributions. On the other hand, the SDG models speaker-dependent speech variation that characterizes speaker attributes. By handling each information separately, CrossSpeech can obtain disentangled speaker and language representations. From the experiments, we verify that CrossSpeech achieves significant improvements in cross-lingual TTS, especially in terms of speaker similarity to the target speaker.

【10】 A Token-Wise Beam Search Algorithm for RNN-T标题：一种适用于RNN-T的令牌式波束搜索算法链接：http://arxiv.org/pdf/2302.14357v1

*与cs.SD语音【5】为同一篇

作者：Gil Keren机构：Meta AI, USA摘要：用于语音识别的标准递归神经网络转换器（RNN-T）解码算法在时间轴上迭代，使得在移动到下一个时间步长之前解码一个时间步长。这些算法导致对联合网络的大量调用，这在先前的工作中被示出为降低解码速度的重要因素。我们提出了一种解码波束搜索算法，该算法在一段时间步长上分批处理联合网络调用，这导致了40%-70%的解码加速比，在所有模型和实验设置中一致。此外，在一个片段上聚集发射概率可以被看作是找到最可能的模型输出的更好近似，使得我们的算法随着片段大小的增加相对提高甲骨文单词错误率高达10%，并且稍微提高一般单词错误率。摘要：Standard Recurrent Neural Network Transducers (RNN-T) decoding algorithms for speech recognition are iterating over the time axis, such that one time step is decoded before moving on to the next time step. Those algorithms result in a large number of calls to the joint network, that were shown in previous work to be an important factor that reduces decoding speed. We present a decoding beam search algorithm that batches the joint network calls across a segment of time steps, which results in 40%-70% decoding speedups, consistently across all models and settings experimented with. In addition, aggregating emission probabilities over a segment may be seen as a better approximation to finding the most likely model output, causing our algorithm to improve oracle word error rate by up to 10% relative as the segment size increases, and to slightly improve general word error rate.

【11】 UniFLG: Unified Facial Landmark Generator from Text or Speech标题：UniFLG：文本或语音的统一人脸标志点生成器链接：http://arxiv.org/pdf/2302.14337v1

*与cs.SD语音【6】为同一篇

作者：Kentaro Mitsui,Yukiya Hono,Kei Sawada机构：rinna Co., Ltd., Tokyo, Japan备注：5 pages, 2 figures, 3 tables摘要：人脸生成由于其广泛的应用性而得到了广泛的研究。用于说话面部生成的两个主要框架包括文本驱动框架和语音驱动框架，文本驱动框架从文本生成同步语音和说话面部，语音驱动框架从语音生成说话面部。为了整合这些框架，本文提出了一种统一的面部标志生成器（unified facial landmark generator，UniFLG）。所提出的系统利用端到端文本到语音不仅用于合成语音，而且用于提取文本和语音共有的一系列潜在表示，并将其馈送到界标解码器以生成面部界标。我们证明，我们的系统取得了更高的自然度在语音合成和面部标志生成相比，国家的最先进的文本驱动的方法。我们进一步证明了我们的系统可以从说话人的语音中生成面部标志，而不需要面部视频数据甚至语音数据。摘要：Talking face generation has been extensively investigated owing to its wide applicability. The two primary frameworks used for talking face generation comprise a text-driven framework, which generates synchronized speech and talking faces from text, and a speech-driven framework, which generates talking faces from speech. To integrate these frameworks, this paper proposes a unified facial landmark generator (UniFLG). The proposed system exploits end-to-end text-to-speech not only for synthesizing speech but also for extracting a series of latent representations that are common to text and speech, and feeds it to a landmark decoder to generate facial landmarks. We demonstrate that our system achieves higher naturalness in both speech synthesis and facial landmark generation compared to the state-of-the-art text-driven method. We further demonstrate that our system can generate facial landmarks from speech of speakers without facial video data or even speech data.

【12】 Adapter Incremental Continual Learning of Efficient Audio Spectrogram Transformers标题：高效音谱图转换器的自适应增量式连续学习链接：http://arxiv.org/pdf/2302.14314v1

*与cs.SD语音【7】为同一篇

作者：Nithish Muthuchamy Selvaraj,Xiaobao Guo,Adams Kong,Bingquan Shen,Alex Kot机构：Nanyang Technological University, Singapore, DSO National Laboratories, Singapore摘要：连续学习涉及在保留先前任务的知识的同时为新任务递增地训练神经网络。然而，用最少的计算资源有效地微调用于顺序任务的模型仍然是一个挑战。在这篇文章中，我们提出了任务增量连续学习（TI-CL）的音频分类器与参数有效和计算有效的音频谱图Transformers（AST）。为了减少TI-CL的可训练参数而不降低性能，我们比较了几种参数有效传递（PET）方法，并提出了用于TI-CL的具有卷积适配器的AST，其可训练参数少于完全微调对应物的5%。为了降低计算复杂度，提出了一种新的频率-时间因子化注意力（FTA）方法来代替传统的自注意力变换方法。FTA仅需要全局自我注意（GSA）所需计算的一个因子就可以获得有竞争力的性能。最后，我们制定了TI-CL的方法，称为适配器增量连续学习（AI-CL），作为“参数高效”卷积适配器和“计算高效”FTA的组合。在ESC-50、SpeechCommandsV 2（SCv 2）和AVE（Audio-Visual Event）基准测试上的实验表明，该方法在保持较低计算量的同时，有效地防止了TI-CL中的灾难性遗忘。摘要：Continual learning involves training neural networks incrementally for new tasks while retaining the knowledge of previous tasks. However, efficiently fine-tuning the model for sequential tasks with minimal computational resources remains a challenge. In this paper, we propose Task Incremental Continual Learning (TI-CL) of audio classifiers with both parameter-efficient and compute-efficient Audio Spectrogram Transformers (AST). To reduce the trainable parameters without performance degradation for TI-CL, we compare several Parameter Efficient Transfer (PET) methods and propose AST with Convolutional Adapters for TI-CL, which has less than 5% of trainable parameters of the fully fine-tuned counterparts. To reduce the computational complexity, we introduce a novel Frequency-Time factorized Attention (FTA) method that replaces the traditional self-attention in transformers for audio spectrograms. FTA achieves competitive performance with only a factor of the computations required by Global Self-Attention (GSA). Finally, we formulate our method for TI-CL, called Adapter Incremental Continual Learning (AI-CL), as a combination of the "parameter-efficient" Convolutional Adapter and the "compute-efficient" FTA. Experiments on ESC-50, SpeechCommandsV2 (SCv2), and Audio-Visual Event (AVE) benchmarks show that our proposed method prevents catastrophic forgetting in TI-CL while maintaining a lower computational budget.

【13】 HalluAudio: Hallucinating Frequency as Concepts for Few-Shot Audio Classification标题：HalluAudio：幻觉频率作为Few-Shot音频分类的概念链接：http://arxiv.org/pdf/2302.14204v1

*与cs.SD语音【8】为同一篇

作者：Zhongjie Yu,Shuyang Wang,Lin Chen,Zhongwei Cheng机构：Wyze Labs, Inc.备注：Accepted at ICASSP 2023摘要：Few-Shot音频分类是一个新兴的研究课题，受到了研究界越来越多的关注。大多数现有的研究忽略了音频谱图形式的特殊性，而主要关注于从图像任务中借用的嵌入空间，而在本研究中，我们旨在利用这种特殊的音频格式，通过将高频和低频部分幻觉为结构化概念，提出一种新的方法。在ESC-50数据集和平衡Kaggle 18数据集上的大量实验表明，该方法的性能明显优于基线方法.我们的方法幻觉高频和低频部分的方式也使其具有可解释性，并为Few-Shot音频分类开辟了新的潜力。摘要：Few-shot audio classification is an emerging topic that attracts more and more attention from the research community. Most existing work ignores the specificity of the form of the audio spectrogram and focuses largely on the embedding space borrowed from image tasks, while in this work, we aim to take advantage of this special audio format and propose a new method by hallucinating high-frequency and low-frequency parts as structured concepts. Extensive experiments on ESC-50 and our curated balanced Kaggle18 dataset show the proposed method outperforms the baseline by a notable margin. The way that our method hallucinates high-frequency and low-frequency parts also enables its interpretability and opens up new potentials for the few-shot audio classification.

【14】 Renaissance canons with asymmetric schemes标题：文艺复兴时期的非对称方案链接：http://arxiv.org/pdf/2302.14160v1

*与cs.SD语音【16】为同一篇

作者：Evan M. O'Dorney备注：17 pages, including 13 figures, 3 tables, and a 2-page illustrative musical composition摘要：所谓音乐经典的“方案”，我们指的是声音进入的顺序，以及每个进入声音的时间和音高位移。当时间位移不相等时，要达到协和的响度就特别困难。使用第一物种理论模型，我们量化了文艺复兴时期作曲家使用或可能使用的方案的灵活性。我们设计了一个算法来精确计算柔度值（在这个过程中发现它是皮索数的$h$次根）。我们发现，帕莱斯特里纳一贯选择一些最灵活的方案，比他的前任，但他绝没有用尽可行的方案。我们结束了提出一个新的组成范围内的风格利用一个未经探索的佳能计划。摘要：By a "scheme" of a musical canon, we mean the order of voice entry with the time and pitch displacement of each entering voice. When the time displacements are unequal, achieving consonant sonorities is especially challenging. Using a first-species theoretical model, we quantify the flexibility of schemes that Renaissance composers used or could have used. We craft an algorithm to compute the flexibility value precisely (finding in the process that it is an $h$-th root of a Pisot number). We find that Palestrina consistently selected some of the most flexible schemes, more so than his predecessors, but that he by no means exhausted the feasible schemes. We close by presenting a new composition within the limits of the style utilizing an unexplored canonic scheme.

【15】 Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding标题：用于语音识别和理解的自监督预训练模型的结构化剪枝链接：http://arxiv.org/pdf/2302.14132v1

*与cs.SD语音【9】为同一篇

作者：Yifan Peng,Kwangyoun Kim,Felix Wu,Prashant Sridhar,Shinji Watanabe机构：ASAPP Inc., Mountain View, CA, USA, Carnegie Mellon University, Pittsburgh, PA, USA备注：Accepted at ICASSP 2023摘要：自监督语音表示学习（SSL）在各种下游任务中已经显示出有效性，但是SSL模型通常较大且较慢。模型压缩技术（如修剪）旨在减少模型大小和计算量，而不降低精度。先前的研究集中在Transformers的修剪;然而，语音模型不仅利用一堆Transformer块，而且还结合基于多个卷积层的前端网络用于低级特征表示学习。该前端具有较小的尺寸，但是计算成本很高。在这项工作中，我们提出了三个任务特定的结构化修剪方法来处理这种异构网络。在LibriSpeech和SLURP上的实验结果表明，该方法比原始的wav 2 vec 2-base算法准确率高，计算量减少10% ~ 30%，在不降低性能的前提下，计算量减少40% ~ 50%.摘要：Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques such as pruning aim to reduce the model size and computation without degradation in accuracy. Prior studies focus on the pruning of Transformers; however, speech models not only utilize a stack of Transformer blocks, but also combine a frontend network based on multiple convolutional layers for low-level feature representation learning. This frontend has a small size but a heavy computational cost. In this work, we propose three task-specific structured pruning methods to deal with such heterogeneous networks. Experiments on LibriSpeech and SLURP show that the proposed method is more accurate than the original wav2vec2-base with 10% to 30% less computation, and is able to reduce the computation by 40% to 50% without any degradation.

【16】 Explanations for Automatic Speech Recognition标题：自动语音识别说明链接：http://arxiv.org/pdf/2302.14062v1

*与cs.SD语音【10】为同一篇

作者：Xiaoliang Wu,Peter Bell,Ajitha Rajan机构：School of Informatics, University of Edinburgh备注：Accepted by Speech Track, ICASSP 2023摘要：我们通过提供有助于增加我们对系统的理解并最终有助于建立对系统的信任的解释来解决基于神经网络的ASR的质量评估。与简单的分类标签相比，解释转录更具挑战性，因为判断它们的正确性并不简单，并且现有的可解释机器学习模型无法处理作为可变长度序列的转录。我们将ASR转录解释为音频帧的子集，它既是转录的最小原因也是充分原因。为此，我们从图像分类-统计故障定位（SFL）和因果关系调整现有的可解释AI（XAI）技术。此外，我们使用了ASR的局部可解释模型不可知解释（LIME）的修改版本作为我们实验中的基线。我们在三种不同的ASR、Google API、Sphinx的基线模型、Deepspeech和来自Commonvoice数据集的100个音频样本上评估了由所提出的技术生成的解释的质量。摘要：We address quality assessment for neural network based ASR by providing explanations that help increase our understanding of the system and ultimately help build trust in the system. Compared to simple classification labels, explaining transcriptions is more challenging as judging their correctness is not straightforward and transcriptions as a variable-length sequence is not handled by existing interpretable machine learning models. We provide an explanation for an ASR transcription as a subset of audio frames that is both a minimal and sufficient cause of the transcription. To do this, we adapt existing explainable AI (XAI) techniques from image classification-Statistical Fault Localisation(SFL) and Causal. Additionally, we use an adapted version of Local Interpretable Model-Agnostic Explanations (LIME) for ASR as a baseline in our experiments. We evaluate the quality of the explanations generated by the proposed techniques over three different ASR ,Google API, the baseline model of Sphinx, Deepspeech and 100 audio samples from the Commonvoice dataset.

【17】Phone and speaker spatial organization in self-supervised speech representations标题：自监督语音表征中的音素和说话人空间组织链接：http://arxiv.org/pdf/2302.14055v1

*与cs.SD语音【11】为同一篇

作者：Pablo Riera,Manuela Cerdeiro,Leonardo Pepino,Luciana Ferrer机构：⋆Instituto de Investigaci´on en Ciencias de la Computaci´on (ICC), CONICET-UBA, Argentina, †Departamento de Computaci´on, FCEyN, Universidad de Buenos Aires (UBA), Argentina摘要：语音的自监督表示目前正广泛用于大量应用。最近，已经做出了一些努力来尝试分析这些表示中的每一个中存在的信息类型。大多数这样的工作使用下游模型来测试表示是否可以成功地用于特定任务。然而，下游模型通常对表示执行非线性操作，提取在原始表示中可能不容易获得的信息。在这项工作中，我们使用不需要下游模型的方法分析了几种最先进的语音表示中音素和说话人信息的空间组织。我们使用表示相似性分析来测量不同层如何编码基本声学参数，例如共振峰和音调。此外，我们使用非参数统计测试来研究每种表示法按音素或说话人类别对语音样本进行聚类的程度。我们的结果表明，根据预训练中使用的目标任务，模型不同地表示这些语音属性。摘要：Self-supervised representations of speech are currently being widely used for a large number of applications. Recently, some efforts have been made in trying to analyze the type of information present in each of these representations. Most such work uses downstream models to test whether the representations can be successfully used for a specific task. The downstream models, though, typically perform nonlinear operations on the representation extracting information that may not have been readily available in the original representation. In this work, we analyze the spatial organization of phone and speaker information in several state-of-the-art speech representations using methods that do not require a downstream model. We measure how different layers encode basic acoustic parameters such as formants and pitch using representation similarity analysis. Further, we study the extent to which each representation clusters the speech samples by phone or speaker classes using non-parametric statistical testing. Our results indicate that models represent these speech attributes differently depending on the target task used during pretraining.

机器翻译，仅供参考