【1】 A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer
标题：一种基于张量融合的多模式自发语音痴呆检测方法
链接：https://arxiv.org/abs/2211.04368
作者：Loukas Ilias,Dimitris Askounis,John Psarras机构：Decision Support Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, Athens, Greece备注：2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI) - Oral Presentation摘要：阿尔茨海默病（AD）是一种进行性神经系统疾病，意味着症状在多年中逐渐发展。它也是痴呆症的主要原因，痴呆症会影响记忆力、思维能力和智力。现在，研究人员已经将他们的兴趣转移到从自发语音中检测AD，因为它构成了一个时间有效的过程。然而，现有的提出多模态方法的最新工作没有考虑模态间和模态内的相互作用，并且提出了早期和晚期融合方法。为了解决这些局限性，我们提出了深度神经网络，它可以以端到端可训练的方式训练，并捕捉模态间和模态内的交互作用。首先，将每一个音频文件转换成由三个声道组成的图像，即，log-Mel谱图、δ谱图和δ-δ谱图。接下来，每个副本通过BERT模型，然后通过门控自注意层。类似地，每个图像通过Swin Transformer，然后通过独立的门控自关注层。还从每个音频文件中提取声学特征。最后，来自不同模态的表示向量被馈送到张量融合层，用于捕获模态间交互。在ADReSS Challenge数据集上进行的大量实验表明，与现有的研究方法相比，本文提出的方法取得了有价值的优势，准确率和F1-score分别达到86.25%和85.48%。摘要：Alzheimer's disease (AD) is a progressive neurological disorder, meaning that the symptoms develop gradually throughout the years. It is also the main cause of dementia, which affects memory, thinking skills, and mental abilities. Nowadays, researchers have moved their interest towards AD detection from spontaneous speech, since it constitutes a time-effective procedure. However, existing state-of-the-art works proposing multimodal approaches do not take into consideration the inter- and intra-modal interactions and propose early and late fusion approaches. To tackle these limitations, we propose deep neural networks, which can be trained in an end-to-end trainable way and capture the inter- and intra-modal interactions. Firstly, each audio file is converted to an image consisting of three channels, i.e., log-Mel spectrogram, delta, and delta-delta. Next, each transcript is passed through a BERT model followed by a gated self-attention layer. Similarly, each image is passed through a Swin Transformer followed by an independent gated self-attention layer. Acoustic features are extracted also from each audio file. Finally, the representation vectors from the different modalities are fed to a tensor fusion layer for capturing the inter-modal interactions. Extensive experiments conducted on the ADReSS Challenge dataset indicate that our introduced approaches obtain valuable advantages over existing research initiatives reaching Accuracy and F1-score up to 86.25% and 85.48% respectively.
【2】 BER: Balanced Error Rate For Speaker Diarization
标题：BER：说话人二值化的均衡误码率
链接：https://arxiv.org/abs/2211.04304
机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Shanghai Jiao Tong University备注：5 pages, 2 figures摘要：DER是评估透析性能的主要指标，同时面临着一个困境：短话语或片段中的错误往往被较长的错误淹没。短片段，例如：'yes'或'no'仍然有语义信息。此外，DER忽略了较少说话者的错误。尽管JER平衡了说话人的错误，但它仍然面临着同样的困境。考虑到时长误差、分段误差和说话人加权误差构成了完整的发音评估，我们提出了一种平衡误差率（BER）来评估说话人发音。首先，提出了一种基于连通子图的分段级错误率（SER）和自适应IoU阈值来实现精确的分段匹配。第二，为了统一地评价二重性，我们采用了一种特定于说话人的时长和片段之间的调和平均值，然后是说话人加权平均值。第三，我们通过模块化系统、EEND和多模态方法在真实数据集上分析了我们的度量。SER和BER可在www.example.com上公开获得https://github.com/X-LANCE/BER。摘要：DER is the primary metric to evaluate diarization performance while facing a dilemma: the errors in short utterances or segments tend to be overwhelmed by longer ones. Short segments, e.g., `yes' or `no,' still have semantic information. Besides, DER overlooks errors in less-talked speakers. Although JER balances speaker errors, it still suffers from the same dilemma. Considering all those aspects, duration error, segment error, and speaker-weighted error constituting a complete diarization evaluation, we propose a Balanced Error Rate (BER) to evaluate speaker diarization. First, we propose a segment-level error rate (SER) via connected sub-graphs and adaptive IoU threshold to get accurate segment matching. Second, to evaluate diarization in a unified way, we adopt a speaker-specific harmonic mean between duration and segment, followed by a speaker-weighted average. Third, we analyze our metric via the modularized system, EEND, and the multi-modal method on real datasets. SER and BER are publicly available at https://github.com/X-LANCE/BER.
【3】 Improving performance of real-time full-band blind packet-loss concealment with predictive network
标题：利用预测网络提高实时全频带盲丢包隐藏的性能
链接：https://arxiv.org/abs/2211.04071
作者：Viet-Anh Nguyen,Anh H. T. Nguyen,Andy W. H. Khong机构：Crystalsound Team, NamiTech JSC, Ho Chi Minh City, Vietnam, Nanyang Technological University, Singapore备注：Submitted to ICASSP 2023, 5 pages, 1 figure, 4 tables摘要：分组丢失隐藏（PLC）是一种用于增强由不良网络条件或音频处理流水线中的下溢/上溢引起的语音降级的工具。我们提出一种实时递归方法，利用先前的输出来减轻丢失分组的假象，而无需丢失掩码的先验知识。提出的全频带递归网络（FRN）模型工作在48 kHz，适合于高质量的通信应用。实验结果突出了FRN相对于离线非因果基线的优越性，并且在最近的PLC挑战中表现最好。摘要：Packet loss concealment (PLC) is a tool for enhancing speech degradation caused by poor network conditions or underflow/overflow in audio processing pipelines. We propose a real-time recurrent method that leverages previous outputs to mitigate artefact of lost packets without the prior knowledge of loss mask. The proposed full-band recurrent network (FRN) model operates at 48~kHz, which is suitable for high-quality telecommunication applications. Experiment results highlight the superiority of FRN over an offline non-causal baseline and a top performer in a recent PLC challenge.
【4】 High-resolution embedding extractor for speaker diarisation
标题：一种用于说话人腹泻的高分辨率嵌入式抽取器
链接：https://arxiv.org/abs/2211.04060
作者：Hee-Soo Heo,Youngki Kwon,Bong-Jin Lee,You Jin Kim,Jee-weon Jung机构：Naver Corporation, South Korea备注：5pages, 2 figure, 3 tables, submitted to ICASSP摘要：说话人嵌入提取器对基于聚类的说话人语音识别系统的性能有重要影响。传统上，从每个语音段中仅提取一个嵌入。然而，由于滑动窗口方法，由于说话者改变点，片段容易包括两个或更多个说话者。本研究提出一种新颖的嵌入式撷取器架构，称为高解析度嵌入式撷取器（HEE），其可从每一语音片段中撷取多个高解析度嵌入。Hee由特征图提取器和增强器组成，其中具有自注意机制的增强器是成功的关键。HEE的增强剂取代了聚集过程;代替全局池层，增强器通过注意力杠杆作用于全局上下文来将相对信息组合到每个帧。提取的密集帧级嵌入可以各自表示一个说话者。因此，多个说话者可以由每个片段中的不同帧级特征来表示。我们还提出了一个人工生成的混合数据训练框架来训练所提出的HEE。通过在5个评价集（包括4个公共数据集）上的实验，HEE在每个评价集上都有至少10%的改进，除了一个数据集，我们分析该数据集不存在快速说话人变化。摘要：Speaker embedding extractors significantly influence the performance of clustering-based speaker diarisation systems. Conventionally, only one embedding is extracted from each speech segment. However, because of the sliding window approach, a segment easily includes two or more speakers owing to speaker change points. This study proposes a novel embedding extractor architecture, referred to as a high-resolution embedding extractor (HEE), which extracts multiple high-resolution embeddings from each speech segment. Hee consists of a feature-map extractor and an enhancer, where the enhancer with the self-attention mechanism is the key to success. The enhancer of HEE replaces the aggregation process; instead of a global pooling layer, the enhancer combines relative information to each frame via attention leveraging the global context. Extracted dense frame-level embeddings can each represent a speaker. Thus, multiple speakers can be represented by different frame-level features in each segment. We also propose an artificially generating mixture data training framework to train the proposed HEE. Through experiments on five evaluation sets, including four public datasets, the proposed HEE demonstrates at least 10% improvement on each evaluation set, except for one dataset, which we analyse that rapid speaker changes less exist.
【5】 ATCO2 corpus: A Large-Scale Dataset for Research on Automatic Speech Recognition and Natural Language Understanding of Air Traffic Control Communications
标题：ATCO2语料库：用于空中交通管制通信自动语音识别和自然语言理解研究的大规模数据集
链接：https://arxiv.org/abs/2211.04054
作者：Juan Zuluaga-Gomez,Karel Veselý,Igor Szöke,Petr Motlicek,Martin Kocour,Mickael Rigault,Khalid Choukri,Amrutha Prasad,Seyyed Saeed Sarfjoo,Iuliia Nigmatulina,Claudia Cevenini,Pavel Kolčárek,Allan Tart,Jan Černocký机构：Saeed Sarfjoo, Pavel Kolˇcárek, Jan ˇCernocký, Idiap Research Institute, Martigny, Switzerland, Ecole Polytechnique Federale de Lausanne (EPFL), Lausanne, Switzerland, ReplayWell, Brno, Czech Republic备注：Manuscript under review; The code will be available at this https URL摘要：个人助理、自动语音识别器和对话理解系统在我们互联的数字世界中正变得越来越重要。一个明显的例子是空中交通管制（ATC）通信。空中交通管制（ATC）的目的是引导飞行器，以安全和最佳的方式控制空域。这些基于语音的对话通过甚高频无线电信道在空中交通管制员（ATCO）和飞行员之间传送。为了将这些新技术应用于低资源领域，需要大规模的标注数据集来开发数据驱动的人工智能系统。两个例子是自动语音识别（ASR）和自然语言理解（NLU）。本文介绍了ATCO 2语料库，该数据集旨在促进具有挑战性的ATC领域的研究，该领域由于缺乏注释数据而滞后。ATCO 2语料库包括1）数据收集和预处理，2）语音数据的伪注释，3）与ATC相关的命名实体的提取。ATCO 2语料库被分成三个子集。1)ATCO 2-测试集语料库包含4小时的ATC语音与手动转录和一个子集与黄金注释的命名实体识别（呼号，命令，价值）。2)ATCO 2-PL-集语料库由5281小时的未标记ATC数据组成，其中包含来自域内语音识别器的自动转录、上下文信息、说话人转向信息、信噪比估计和每个样本的英语语言检测分数。这两款产品均可通过ELDA购买，网址为http://catalog.elra.info/en-us/repository/browse/ELRA-S0484。3)ATCO 2-test-set-1h语料库是我们在www.example.com上免费提供的原始测试集语料库的一个小时子集https://www.atco2.org/data。我们希望ATCO 2语料库不仅能促进ATC通信领域的鲁棒ASR和NLU研究，而且能促进一般研究社区的研究。摘要：Personal assistants, automatic speech recognizers and dialogue understanding systems are becoming more critical in our interconnected digital world. A clear example is air traffic control (ATC) communications. ATC aims at guiding aircraft and controlling the airspace in a safe and optimal manner. These voice-based dialogues are carried between an air traffic controller (ATCO) and pilots via very-high frequency radio channels. In order to incorporate these novel technologies into ATC (low-resource domain), large-scale annotated datasets are required to develop the data-driven AI systems. Two examples are automatic speech recognition (ASR) and natural language understanding (NLU). In this paper, we introduce the ATCO2 corpus, a dataset that aims at fostering research on the challenging ATC field, which has lagged behind due to lack of annotated data. The ATCO2 corpus covers 1) data collection and pre-processing, 2) pseudo-annotations of speech data, and 3) extraction of ATC-related named entities. The ATCO2 corpus is split into three subsets. 1) ATCO2-test-set corpus contains 4 hours of ATC speech with manual transcripts and a subset with gold annotations for named-entity recognition (callsign, command, value). 2) The ATCO2-PL-set corpus consists of 5281 hours of unlabeled ATC data enriched with automatic transcripts from an in-domain speech recognizer, contextual information, speaker turn information, signal-to-noise ratio estimate and English language detection score per sample. Both available for purchase through ELDA at http://catalog.elra.info/en-us/repository/browse/ELRA-S0484. 3) The ATCO2-test-set-1h corpus is a one-hour subset from the original test set corpus, that we are offering for free at https://www.atco2.org/data. We expect the ATCO2 corpus will foster research on robust ASR and NLU not only in the field of ATC communications but also in the general research community.
【6】 Comparative layer-wise analysis of self-supervised speech models
标题：自监督语音模型的分层比较分析
链接：https://arxiv.org/abs/2211.03929
作者：Ankita Pasad,Bowen Shi,Karen Livescu机构：Toyota Technological Institute at Chicago备注：submitted to ICASSP 2022摘要：在过去的几年中，已经提出了许多在预训练目标、输入模态和预训练数据方面不同的自监督语音模型。尽管在下游任务上取得了令人印象深刻的经验性成功，但我们对模型编码的性质和模型之间的差异仍然了解有限。在这项工作中，我们检查了各种最新模型的中间表示。具体来说，我们使用基于典型相关分析（CCA）的轻量级分析工具，测量各个层中编码的声学、语音和单词级属性。我们发现，这些属性在不同的层之间根据模型的不同而不同地演化，并且这些变化与预训练目标的选择有关。我们进一步研究了我们的分析对下游任务的效用，通过比较性能趋势与语音识别和口语理解任务的表现。我们发现CCA趋势为下游任务选择感兴趣的层提供了可靠的指导，并且单层性能通常在使用所有层时匹配或改善，这暗示了更有效地使用预训练模型。摘要：Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive empirical successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models.
【7】 Cross-Attention is all you need: Real-Time Streaming Transformers for Personalised Speech Enhancement
标题：交叉关注是您所需要的：用于个性化语音增强的实时流转换器
链接：https://arxiv.org/abs/2211.04346
作者：Shucong Zhang,Malcolm Chadwick,Alberto Gil C. P. Ramos,Sourav Bhattacharya机构：Samsung AI Centre, Cambridge, UK摘要：个性化语音增强（PSE）只提取目标用户的语音，并从录制的音频片段中删除其他所有内容，可以潜在地改善用户在野外部署的音频AI模块的体验。为了支持诸如实时ASR和音频呼叫增强之类的大量下游音频任务，PSE解决方案应当以流模式操作，即，输入音频清理应当以小的等待时间和实时因素实时发生。个性化典型地通过以静态嵌入向量的形式从登记音频中提取目标说话人的声音简档，然后使用它来调节PSE模型的输出来实现。然而，固定的目标说话人嵌入可能不是在所有条件下都是最优的。在本文中，我们提出了一个基于流式变压器的PSE模型，并提出了一种新颖的交叉注意方法，给出了自适应的目标说话人表示。我们展示了大量的实验，并表明我们提出的交叉注意方法始终优于竞争基线，即使我们的模型只有大约一半的大小。摘要：Personalised speech enhancement (PSE), which extracts only the speech of a target user and removes everything else from a recorded audio clip, can potentially improve users' experiences of audio AI modules deployed in the wild. To support a large variety of downstream audio tasks, such as real-time ASR and audio-call enhancement, a PSE solution should operate in a streaming mode, i.e., input audio cleaning should happen in real-time with a small latency and real-time factor. Personalisation is typically achieved by extracting a target speaker's voice profile from an enrolment audio, in the form of a static embedding vector, and then using it to condition the output of a PSE model. However, a fixed target speaker embedding may not be optimal under all conditions. In this work, we present a streaming Transformer-based PSE model and propose a novel cross-attention approach that gives adaptive target speaker representations. We present extensive experiments and show that our proposed cross-attention approach outperforms competitive baselines consistently, even when our model is only approximately half the size.
【8】 DiffPhase: Generative Diffusion-based STFT Phase Retrieval
标题：基于产生式扩散的短时傅里叶变换相位检索
链接：https://arxiv.org/abs/2211.04332
作者：Tal Peer,Simon Welker,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany, Center for Free-Electron Laser Science, DESY, Hamburg, Germany备注：Submitted to ICASSP 2023摘要：扩散概率模型最近已经被用于各种任务，包括语音增强和合成。作为一种生成方法，扩散模型已被证明特别适合于填补问题，在这种情况下，缺失数据是根据现有数据生成的。相位恢复本质上是一个插补问题，其中相位信息必须基于给定的幅度生成。在本工作中，我们建立在语音领域的先前工作的基础上，采用语音增强扩散模型专门用于STFT相位检索。使用语音质量和可懂度度量进行的评估表明扩散方法非常适合相位恢复任务，其性能超过了经典和现代方法。摘要：Diffusion probabilistic models have been recently used in a variety of tasks, including speech enhancement and synthesis. As a generative approach, diffusion models have been shown to be especially suitable for imputation problems, where missing data is generated based on existing data. Phase retrieval is inherently an imputation problem, where phase information has to be generated based on the given magnitude. In this work we build upon previous work in the speech domain, adapting a speech enhancement diffusion model specifically for STFT phase retrieval. Evaluation using speech quality and intelligibility metrics shows the diffusion approach is well-suited to the phase retrieval task, with performance surpassing both classical and modern methods.
【9】 Pushing the limits of self-supervised speaker verification using regularized distillation framework
标题：利用正则化蒸馏框架突破自监督说话人确认的极限
链接：https://arxiv.org/abs/2211.04168
作者：Yafeng Chen,Siqi Zheng,Hui Wang,Luyao Cheng,Qian Chen机构：Speech Lab, Alibaba Group摘要：训练鲁棒的没有说话人标签的说话人确认系统长期以来一直是一项具有挑战性的任务。以往的研究观察到自监督和全监督方法之间存在很大的性能差距。本文应用一种非对比自监督学习框架--无标签蒸馏（DINO），提出了两个正则化项用于DINO中的嵌入.其中一个正则项保证了嵌入的多样性，而另一个正则项则保证了每个嵌入变量的去相关性。在时域和频域上，探讨了各种数据增强技术的有效性。在VoxCelib数据集上进行的一系列实验证明了正则化DINO框架在说话人确认中的优越性。该方法在VoxCeleb上实现了单级自监督的说话人确认性能。这些代码将公开发布。摘要：Training robust speaker verification systems without speaker labels has long been a challenging task. Previous studies observed a large performance gap between self-supervised and fully supervised methods. In this paper, we apply a non-contrastive self-supervised learning framework called DIstillation with NO labels (DINO) and propose two regularization terms applied to embeddings in DINO. One regularization term guarantees the diversity of the embeddings, while the other regularization term decorrelates the variables of each embedding. The effectiveness of various data augmentation techniques are explored, on both time and frequency domain. A range of experiments conducted on the VoxCeleb datasets demonstrate the superiority of the regularized DINO framework in speaker verification. Our method achieves the state-of-the-art speaker verification performance under a single-stage self-supervised setting on VoxCeleb. The codes will be made publicly-available.
【10】 Unsupervised vocal dereverberation with diffusion-based generative models
标题：基于扩散的产生式模型的无监督语音去混响
链接：https://arxiv.org/abs/2211.04124
作者：Koichi Saito,Naoki Murata,Toshimitsu Uesaka,Chieh-Hsin Lai,Yuhta Takida,Takao Fukui,Yuki Mitsufuji机构：Sony Group Corporation, Tokyo, Japan备注：6 pages, 2 figures, submitted to ICASSP 2023摘要：从混响音乐中去除混响是为下游音乐处理清理音频的必要技术。音乐的混响分为自然混响和人工混响两类。人工混响由于其各种参数设置和混响类型而具有比自然混响更广泛的多样性。然而，最近的监督去混响方法可能失败，因为它们依赖于足够多样和大量的混响观测和检索数据对来训练，以便在推断期间可推广到未看过的观测。为了解决这些问题，我们提出了一种无监督的方法，该方法可以去除音乐的一般类型的人工混响，而不需要用于训练的数据对。所提出的方法基于扩散模型，其中它利用传统的信号处理技术初始化未知混响算子，并且同时借助于扩散模型来细化估计。通过客观和感知的评价，我们的方法优于目前主要的人声去混响基准。摘要：Removing reverb from reverberant music is a necessary technique to clean up audio for downstream music manipulations. Reverberation of music contains two categories, natural reverb, and artificial reverb. Artificial reverb has a wider diversity than natural reverb due to its various parameter setups and reverberation types. However, recent supervised dereverberation methods may fail because they rely on sufficiently diverse and numerous pairs of reverberant observations and retrieved data for training in order to be generalizable to unseen observations during inference. To resolve these problems, we propose an unsupervised method that can remove a general kind of artificial reverb for music without requiring pairs of data for training. The proposed method is based on diffusion models, where it initializes the unknown reverberation operator with a conventional signal processing technique and simultaneously refines the estimate with the help of diffusion models. We show through objective and perceptual evaluations that our method outperforms the current leading vocal dereverberation benchmarks.
【11】 On Negative Sampling for Contrastive Audio-Text Retrieval
标题：对比音文检索中的负抽样问题研究
链接：https://arxiv.org/abs/2211.04070
作者：Huang Xie,Okko Räsänen,Tuomas Virtanen机构：Unit of Computing Sciences, Tampere University, Finland备注：Submitted to ICASSP2023摘要：本文研究了负抽样在音频文本检索中的对比学习。用于负采样的策略指的是从用于正音频-文本对的候选池中选择负（音频剪辑或文本描述）。我们通过模型估计的音频和文本样本的通道内和跨通道相关性得分来探索采样策略。在文献[1]的检索系统上，我们研究了8种抽样策略，包括硬负抽样和半硬负抽样。实验结果表明，不同策略的检索性能差异很大。特别地，通过选择具有跨模态得分的半硬否定，检索系统在文本到音频和音频到文本检索中获得改进的性能。此外，我们还证明了在对具有跨模态得分的硬否定进行采样时，特征会发生崩溃。摘要：This paper investigates negative sampling for contrastive learning in the context of audio-text retrieval. The strategy for negative sampling refers to selecting negatives (either audio clips or textual descriptions) from a pool of candidates for a positive audio-text pair. We explore sampling strategies via model-estimated within-modality and cross-modality relevance scores for audio and text samples. With a constant training setting on the retrieval system from [1], we study eight sampling strategies, including hard and semi-hard negative sampling. Experimental results show that retrieval performance varies dramatically among different strategies. Particularly, by selecting semi-hard negatives with cross-modality scores, the retrieval system gains improved performance in both text-to-audio and audio-to-text retrieval. Besides, we show that feature collapse occurs while sampling hard negatives with cross-modality scores.
【1】 Cross-Attention is all you need: Real-Time Streaming Transformers for Personalised Speech Enhancement
标题：交叉关注是您所需要的：用于个性化语音增强的实时流转换器
链接：https://arxiv.org/abs/2211.04346
作者：Shucong Zhang,Malcolm Chadwick,Alberto Gil C. P. Ramos,Sourav Bhattacharya机构：Samsung AI Centre, Cambridge, UK摘要：个性化语音增强（PSE）只提取目标用户的语音，并从录制的音频片段中删除其他所有内容，可以潜在地改善用户在野外部署的音频AI模块的体验。为了支持诸如实时ASR和音频呼叫增强之类的大量下游音频任务，PSE解决方案应当以流模式操作，即，输入音频清理应当以小的等待时间和实时因素实时发生。个性化典型地通过以静态嵌入向量的形式从登记音频中提取目标说话人的声音简档，然后使用它来调节PSE模型的输出来实现。然而，固定的目标说话人嵌入可能不是在所有条件下都是最优的。在本文中，我们提出了一个基于流式变压器的PSE模型，并提出了一种新颖的交叉注意方法，给出了自适应的目标说话人表示。我们展示了大量的实验，并表明我们提出的交叉注意方法始终优于竞争基线，即使我们的模型只有大约一半的大小。摘要：Personalised speech enhancement (PSE), which extracts only the speech of a target user and removes everything else from a recorded audio clip, can potentially improve users' experiences of audio AI modules deployed in the wild. To support a large variety of downstream audio tasks, such as real-time ASR and audio-call enhancement, a PSE solution should operate in a streaming mode, i.e., input audio cleaning should happen in real-time with a small latency and real-time factor. Personalisation is typically achieved by extracting a target speaker's voice profile from an enrolment audio, in the form of a static embedding vector, and then using it to condition the output of a PSE model. However, a fixed target speaker embedding may not be optimal under all conditions. In this work, we present a streaming Transformer-based PSE model and propose a novel cross-attention approach that gives adaptive target speaker representations. We present extensive experiments and show that our proposed cross-attention approach outperforms competitive baselines consistently, even when our model is only approximately half the size.
【2】 DiffPhase: Generative Diffusion-based STFT Phase Retrieval
标题：基于产生式扩散的短时傅里叶变换相位检索
链接：https://arxiv.org/abs/2211.04332
作者：Tal Peer,Simon Welker,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany, Center for Free-Electron Laser Science, DESY, Hamburg, Germany备注：Submitted to ICASSP 2023摘要：扩散概率模型最近已经被用于各种任务，包括语音增强和合成。作为一种生成方法，扩散模型已被证明特别适合于填补问题，在这种情况下，缺失数据是根据现有数据生成的。相位恢复本质上是一个插补问题，其中相位信息必须基于给定的幅度生成。在本工作中，我们建立在语音领域的先前工作的基础上，采用语音增强扩散模型专门用于STFT相位检索。使用语音质量和可懂度度量进行的评估表明扩散方法非常适合相位恢复任务，其性能超过了经典和现代方法。摘要：Diffusion probabilistic models have been recently used in a variety of tasks, including speech enhancement and synthesis. As a generative approach, diffusion models have been shown to be especially suitable for imputation problems, where missing data is generated based on existing data. Phase retrieval is inherently an imputation problem, where phase information has to be generated based on the given magnitude. In this work we build upon previous work in the speech domain, adapting a speech enhancement diffusion model specifically for STFT phase retrieval. Evaluation using speech quality and intelligibility metrics shows the diffusion approach is well-suited to the phase retrieval task, with performance surpassing both classical and modern methods.
【3】 Pushing the limits of self-supervised speaker verification using regularized distillation framework
标题：利用正则化蒸馏框架突破自监督说话人确认的极限
链接：https://arxiv.org/abs/2211.04168
作者：Yafeng Chen,Siqi Zheng,Hui Wang,Luyao Cheng,Qian Chen机构：Speech Lab, Alibaba Group摘要：训练鲁棒的没有说话人标签的说话人确认系统长期以来一直是一项具有挑战性的任务。以往的研究观察到自监督和全监督方法之间存在很大的性能差距。本文应用一种非对比自监督学习框架--无标签蒸馏（DINO），提出了两个正则化项用于DINO中的嵌入.其中一个正则项保证了嵌入的多样性，而另一个正则项则保证了每个嵌入变量的去相关性。在时域和频域上，探讨了各种数据增强技术的有效性。在VoxCelib数据集上进行的一系列实验证明了正则化DINO框架在说话人确认中的优越性。该方法在VoxCeleb上实现了单级自监督的说话人确认性能。这些代码将公开发布。摘要：Training robust speaker verification systems without speaker labels has long been a challenging task. Previous studies observed a large performance gap between self-supervised and fully supervised methods. In this paper, we apply a non-contrastive self-supervised learning framework called DIstillation with NO labels (DINO) and propose two regularization terms applied to embeddings in DINO. One regularization term guarantees the diversity of the embeddings, while the other regularization term decorrelates the variables of each embedding. The effectiveness of various data augmentation techniques are explored, on both time and frequency domain. A range of experiments conducted on the VoxCeleb datasets demonstrate the superiority of the regularized DINO framework in speaker verification. Our method achieves the state-of-the-art speaker verification performance under a single-stage self-supervised setting on VoxCeleb. The codes will be made publicly-available.
【4】 Unsupervised vocal dereverberation with diffusion-based generative models
标题：基于扩散的产生式模型的无监督语音去混响
链接：https://arxiv.org/abs/2211.04124
作者：Koichi Saito,Naoki Murata,Toshimitsu Uesaka,Chieh-Hsin Lai,Yuhta Takida,Takao Fukui,Yuki Mitsufuji机构：Sony Group Corporation, Tokyo, Japan备注：6 pages, 2 figures, submitted to ICASSP 2023摘要：从混响音乐中去除混响是为下游音乐处理清理音频的必要技术。音乐的混响分为自然混响和人工混响两类。人工混响由于其各种参数设置和混响类型而具有比自然混响更广泛的多样性。然而，最近的监督去混响方法可能失败，因为它们依赖于足够多样和大量的混响观测和检索数据对来训练，以便在推断期间可推广到未看过的观测。为了解决这些问题，我们提出了一种无监督的方法，该方法可以去除音乐的一般类型的人工混响，而不需要用于训练的数据对。所提出的方法基于扩散模型，其中它利用传统的信号处理技术初始化未知混响算子，并且同时借助于扩散模型来细化估计。通过客观和感知的评价，我们的方法优于目前主要的人声去混响基准。摘要：Removing reverb from reverberant music is a necessary technique to clean up audio for downstream music manipulations. Reverberation of music contains two categories, natural reverb, and artificial reverb. Artificial reverb has a wider diversity than natural reverb due to its various parameter setups and reverberation types. However, recent supervised dereverberation methods may fail because they rely on sufficiently diverse and numerous pairs of reverberant observations and retrieved data for training in order to be generalizable to unseen observations during inference. To resolve these problems, we propose an unsupervised method that can remove a general kind of artificial reverb for music without requiring pairs of data for training. The proposed method is based on diffusion models, where it initializes the unknown reverberation operator with a conventional signal processing technique and simultaneously refines the estimate with the help of diffusion models. We show through objective and perceptual evaluations that our method outperforms the current leading vocal dereverberation benchmarks.
【5】 On Negative Sampling for Contrastive Audio-Text Retrieval
标题：对比音文检索中的负抽样问题研究
链接：https://arxiv.org/abs/2211.04070
作者：Huang Xie,Okko Räsänen,Tuomas Virtanen机构：Unit of Computing Sciences, Tampere University, Finland备注：Submitted to ICASSP2023摘要：本文研究了负抽样在音频文本检索中的对比学习。用于负采样的策略指的是从用于正音频-文本对的候选池中选择负（音频剪辑或文本描述）。我们通过模型估计的音频和文本样本的通道内和跨通道相关性得分来探索采样策略。在文献[1]的检索系统上，我们研究了8种抽样策略，包括硬负抽样和半硬负抽样。实验结果表明，不同策略的检索性能差异很大。特别地，通过选择具有跨模态得分的半硬否定，检索系统在文本到音频和音频到文本检索中获得改进的性能。此外，我们还证明了在对具有跨模态得分的硬否定进行采样时，特征会发生崩溃。摘要：This paper investigates negative sampling for contrastive learning in the context of audio-text retrieval. The strategy for negative sampling refers to selecting negatives (either audio clips or textual descriptions) from a pool of candidates for a positive audio-text pair. We explore sampling strategies via model-estimated within-modality and cross-modality relevance scores for audio and text samples. With a constant training setting on the retrieval system from [1], we study eight sampling strategies, including hard and semi-hard negative sampling. Experimental results show that retrieval performance varies dramatically among different strategies. Particularly, by selecting semi-hard negatives with cross-modality scores, the retrieval system gains improved performance in both text-to-audio and audio-to-text retrieval. Besides, we show that feature collapse occurs while sampling hard negatives with cross-modality scores.
【6】 A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer
标题：一种基于张量融合的多模式自发语音痴呆检测方法
链接：https://arxiv.org/abs/2211.04368
作者：Loukas Ilias,Dimitris Askounis,John Psarras机构：Decision Support Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, Athens, Greece备注：2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI) - Oral Presentation摘要：阿尔茨海默病（AD）是一种进行性神经系统疾病，意味着症状在多年中逐渐发展。它也是痴呆症的主要原因，痴呆症会影响记忆力、思维能力和智力。现在，研究人员已经将他们的兴趣转移到从自发语音中检测AD，因为它构成了一个时间有效的过程。然而，现有的提出多模态方法的最新工作没有考虑模态间和模态内的相互作用，并且提出了早期和晚期融合方法。为了解决这些局限性，我们提出了深度神经网络，它可以以端到端可训练的方式训练，并捕捉模态间和模态内的交互作用。首先，将每一个音频文件转换成由三个声道组成的图像，即，log-Mel谱图、δ谱图和δ-δ谱图。接下来，每个副本通过BERT模型，然后通过门控自注意层。类似地，每个图像通过Swin Transformer，然后通过独立的门控自关注层。还从每个音频文件中提取声学特征。最后，来自不同模态的表示向量被馈送到张量融合层，用于捕获模态间交互。在ADReSS Challenge数据集上进行的大量实验表明，与现有的研究方法相比，本文提出的方法取得了有价值的优势，准确率和F1-score分别达到86.25%和85.48%。摘要：Alzheimer's disease (AD) is a progressive neurological disorder, meaning that the symptoms develop gradually throughout the years. It is also the main cause of dementia, which affects memory, thinking skills, and mental abilities. Nowadays, researchers have moved their interest towards AD detection from spontaneous speech, since it constitutes a time-effective procedure. However, existing state-of-the-art works proposing multimodal approaches do not take into consideration the inter- and intra-modal interactions and propose early and late fusion approaches. To tackle these limitations, we propose deep neural networks, which can be trained in an end-to-end trainable way and capture the inter- and intra-modal interactions. Firstly, each audio file is converted to an image consisting of three channels, i.e., log-Mel spectrogram, delta, and delta-delta. Next, each transcript is passed through a BERT model followed by a gated self-attention layer. Similarly, each image is passed through a Swin Transformer followed by an independent gated self-attention layer. Acoustic features are extracted also from each audio file. Finally, the representation vectors from the different modalities are fed to a tensor fusion layer for capturing the inter-modal interactions. Extensive experiments conducted on the ADReSS Challenge dataset indicate that our introduced approaches obtain valuable advantages over existing research initiatives reaching Accuracy and F1-score up to 86.25% and 85.48% respectively.
【7】 BER: Balanced Error Rate For Speaker Diarization
标题：BER：说话人二值化的均衡误码率
链接：https://arxiv.org/abs/2211.04304
机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Shanghai Jiao Tong University备注：5 pages, 2 figures摘要：DER是评估透析性能的主要指标，同时面临着一个困境：短话语或片段中的错误往往被较长的错误淹没。短片段，例如：'yes'或'no'仍然有语义信息。此外，DER忽略了较少说话者的错误。尽管JER平衡了说话人的错误，但它仍然面临着同样的困境。考虑到时长误差、分段误差和说话人加权误差构成了完整的发音评估，我们提出了一种平衡误差率（BER）来评估说话人发音。首先，提出了一种基于连通子图的分段级错误率（SER）和自适应IoU阈值来实现精确的分段匹配。第二，为了统一地评价二重性，我们采用了一种特定于说话人的时长和片段之间的调和平均值，然后是说话人加权平均值。第三，我们通过模块化系统、EEND和多模态方法在真实数据集上分析了我们的度量。SER和BER可在www.example.com上公开获得https://github.com/X-LANCE/BER。摘要：DER is the primary metric to evaluate diarization performance while facing a dilemma: the errors in short utterances or segments tend to be overwhelmed by longer ones. Short segments, e.g., `yes' or `no,' still have semantic information. Besides, DER overlooks errors in less-talked speakers. Although JER balances speaker errors, it still suffers from the same dilemma. Considering all those aspects, duration error, segment error, and speaker-weighted error constituting a complete diarization evaluation, we propose a Balanced Error Rate (BER) to evaluate speaker diarization. First, we propose a segment-level error rate (SER) via connected sub-graphs and adaptive IoU threshold to get accurate segment matching. Second, to evaluate diarization in a unified way, we adopt a speaker-specific harmonic mean between duration and segment, followed by a speaker-weighted average. Third, we analyze our metric via the modularized system, EEND, and the multi-modal method on real datasets. SER and BER are publicly available at https://github.com/X-LANCE/BER.
【8】 Improving performance of real-time full-band blind packet-loss concealment with predictive network
标题：利用预测网络提高实时全频带盲丢包隐藏的性能
链接：https://arxiv.org/abs/2211.04071
作者：Viet-Anh Nguyen,Anh H. T. Nguyen,Andy W. H. Khong机构：Crystalsound Team, NamiTech JSC, Ho Chi Minh City, Vietnam, Nanyang Technological University, Singapore备注：Submitted to ICASSP 2023, 5 pages, 1 figure, 4 tables摘要：Packet loss concealment (PLC) is a tool for enhancing speech degradation caused by poor network conditions or underflow/overflow in audio processing pipelines. We propose a real-time recurrent method that leverages previous outputs to mitigate artefact of lost packets without the prior knowledge of loss mask. The proposed full-band recurrent network (FRN) model operates at 48~kHz, which is suitable for high-quality telecommunication applications. Experiment results highlight the superiority of FRN over an offline non-causal baseline and a top performer in a recent PLC challenge.
【9】 High-resolution embedding extractor for speaker diarisation
标题：一种用于说话人腹泻的高分辨率嵌入式抽取器
链接：https://arxiv.org/abs/2211.04060
作者：Hee-Soo Heo,Youngki Kwon,Bong-Jin Lee,You Jin Kim,Jee-weon Jung机构：Naver Corporation, South Korea备注：5pages, 2 figure, 3 tables, submitted to ICASSP摘要：说话人嵌入提取器对基于聚类的说话人语音识别系统的性能有重要影响。传统上，从每个语音段中仅提取一个嵌入。然而，由于滑动窗口方法，由于说话者改变点，片段容易包括两个或更多个说话者。本研究提出一种新颖的嵌入式撷取器架构，称为高解析度嵌入式撷取器（HEE），其可从每一语音片段中撷取多个高解析度嵌入。Hee由特征图提取器和增强器组成，其中具有自注意机制的增强器是成功的关键。HEE的增强剂取代了聚集过程;代替全局池层，增强器通过注意力杠杆作用于全局上下文来将相对信息组合到每个帧。提取的密集帧级嵌入可以各自表示一个说话者。因此，多个说话者可以由每个片段中的不同帧级特征来表示。我们还提出了一个人工生成的混合数据训练框架来训练所提出的HEE。通过在5个评价集（包括4个公共数据集）上的实验，HEE在每个评价集上都有至少10%的改进，除了一个数据集，我们分析该数据集不存在快速说话人变化。摘要：Speaker embedding extractors significantly influence the performance of clustering-based speaker diarisation systems. Conventionally, only one embedding is extracted from each speech segment. However, because of the sliding window approach, a segment easily includes two or more speakers owing to speaker change points. This study proposes a novel embedding extractor architecture, referred to as a high-resolution embedding extractor (HEE), which extracts multiple high-resolution embeddings from each speech segment. Hee consists of a feature-map extractor and an enhancer, where the enhancer with the self-attention mechanism is the key to success. The enhancer of HEE replaces the aggregation process; instead of a global pooling layer, the enhancer combines relative information to each frame via attention leveraging the global context. Extracted dense frame-level embeddings can each represent a speaker. Thus, multiple speakers can be represented by different frame-level features in each segment. We also propose an artificially generating mixture data training framework to train the proposed HEE. Through experiments on five evaluation sets, including four public datasets, the proposed HEE demonstrates at least 10% improvement on each evaluation set, except for one dataset, which we analyse that rapid speaker changes less exist.
【10】 ATCO2 corpus: A Large-Scale Dataset for Research on Automatic Speech Recognition and Natural Language Understanding of Air Traffic Control Communications
标题：ATCO2语料库：用于空中交通管制通信自动语音识别和自然语言理解研究的大规模数据集
链接：https://arxiv.org/abs/2211.04054
作者：Juan Zuluaga-Gomez,Karel Veselý,Igor Szöke,Petr Motlicek,Martin Kocour,Mickael Rigault,Khalid Choukri,Amrutha Prasad,Seyyed Saeed Sarfjoo,Iuliia Nigmatulina,Claudia Cevenini,Pavel Kolčárek,Allan Tart,Jan Černocký机构：Saeed Sarfjoo, Pavel Kolˇcárek, Jan ˇCernocký, Idiap Research Institute, Martigny, Switzerland, Ecole Polytechnique Federale de Lausanne (EPFL), Lausanne, Switzerland, ReplayWell, Brno, Czech Republic备注：Manuscript under review; The code will be available at this https URL摘要：个人助理、自动语音识别器和对话理解系统在我们互联的数字世界中正变得越来越重要。一个明显的例子是空中交通管制（ATC）通信。空中交通管制（ATC）的目的是引导飞行器，以安全和最佳的方式控制空域。这些基于语音的对话通过甚高频无线电信道在空中交通管制员（ATCO）和飞行员之间传送。为了将这些新技术应用于低资源领域，需要大规模的标注数据集来开发数据驱动的人工智能系统。两个例子是自动语音识别（ASR）和自然语言理解（NLU）。本文介绍了ATCO 2语料库，该数据集旨在促进具有挑战性的ATC领域的研究，该领域由于缺乏注释数据而滞后。ATCO 2语料库包括1）数据收集和预处理，2）语音数据的伪注释，3）与ATC相关的命名实体的提取。ATCO 2语料库被分成三个子集。1)ATCO 2-测试集语料库包含4小时的ATC语音与手动转录和一个子集与黄金注释的命名实体识别（呼号，命令，价值）。2)ATCO 2-PL-集语料库由5281小时的未标记ATC数据组成，其中包含来自域内语音识别器的自动转录、上下文信息、说话人转向信息、信噪比估计和每个样本的英语语言检测分数。这两款产品均可通过ELDA购买，网址为http://catalog.elra.info/en-us/repository/browse/ELRA-S0484。3)ATCO 2-test-set-1h语料库是我们在www.example.com上免费提供的原始测试集语料库的一个小时子集https://www.atco2.org/data。我们希望ATCO 2语料库不仅能促进ATC通信领域的鲁棒ASR和NLU研究，而且能促进一般研究社区的研究。摘要：Personal assistants, automatic speech recognizers and dialogue understanding systems are becoming more critical in our interconnected digital world. A clear example is air traffic control (ATC) communications. ATC aims at guiding aircraft and controlling the airspace in a safe and optimal manner. These voice-based dialogues are carried between an air traffic controller (ATCO) and pilots via very-high frequency radio channels. In order to incorporate these novel technologies into ATC (low-resource domain), large-scale annotated datasets are required to develop the data-driven AI systems. Two examples are automatic speech recognition (ASR) and natural language understanding (NLU). In this paper, we introduce the ATCO2 corpus, a dataset that aims at fostering research on the challenging ATC field, which has lagged behind due to lack of annotated data. The ATCO2 corpus covers 1) data collection and pre-processing, 2) pseudo-annotations of speech data, and 3) extraction of ATC-related named entities. The ATCO2 corpus is split into three subsets. 1) ATCO2-test-set corpus contains 4 hours of ATC speech with manual transcripts and a subset with gold annotations for named-entity recognition (callsign, command, value). 2) The ATCO2-PL-set corpus consists of 5281 hours of unlabeled ATC data enriched with automatic transcripts from an in-domain speech recognizer, contextual information, speaker turn information, signal-to-noise ratio estimate and English language detection score per sample. Both available for purchase through ELDA at http://catalog.elra.info/en-us/repository/browse/ELRA-S0484. 3) The ATCO2-test-set-1h corpus is a one-hour subset from the original test set corpus, that we are offering for free at https://www.atco2.org/data. We expect the ATCO2 corpus will foster research on robust ASR and NLU not only in the field of ATC communications but also in the general research community.
【11】 Comparative layer-wise analysis of self-supervised speech models
标题：自监督语音模型的分层比较分析
链接：https://arxiv.org/abs/2211.03929
作者：Ankita Pasad,Bowen Shi,Karen Livescu机构：Toyota Technological Institute at Chicago备注：submitted to ICASSP 2022摘要：在过去的几年中，已经提出了许多在预训练目标、输入模态和预训练数据方面不同的自监督语音模型。尽管在下游任务上取得了令人印象深刻的经验性成功，但我们对模型编码的性质和模型之间的差异仍然了解有限。在这项工作中，我们检查了各种最新模型的中间表示。具体来说，我们使用基于典型相关分析（CCA）的轻量级分析工具，测量各个层中编码的声学、语音和单词级属性。我们发现，这些属性在不同的层之间根据模型的不同而不同地演化，并且这些变化与预训练目标的选择有关。我们进一步研究了我们的分析对下游任务的效用，通过比较性能趋势与语音识别和口语理解任务的表现。我们发现CCA趋势为下游任务选择感兴趣的层提供了可靠的指导，并且单层性能通常在使用所有层时匹配或改善，这暗示了更有效地使用预训练模型。摘要：Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive empirical successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models.
机器翻译，仅供参考
每日语音速递