今日论文合集：cs.SD语音22篇，eess.AS音频处理22篇。
cs.SD语音
【1】 Biased Self-supervised learning for ASR
标题：ASR中的有偏自监督学习
链接：https://arxiv.org/abs/2211.02536
作者：Florian L. Kreyssig,Yangyang Shi,Jinxi Guo,Leda Sari,Abdelrahman Mohamed,Philip C. Woodland机构：University of Cambridge, UK,Meta AI, USA备注：Submitted to ICASSP 2023摘要：通过掩蔽预测预训练（MPPT）的自监督学习在一系列语音处理任务中表现出了令人印象深刻的性能。本文提出了一种将自监督学习偏向于特定任务的方法。其核心思想是对用于获得目标序列的模型进行微调。这会带来更好的性能和训练速度的大幅提高。此外，本文还提出了一种MPPT的变体，通过计算屏蔽帧和非屏蔽帧的MPPT损失，可以有效地训练低占用量的流模型。在Librispeech语料库上对这些方法进行了自动语音识别评估，其中100小时的数据用作已标记数据，860小时的数据用作未标记数据。在测试-其他上，有偏训练在250 k更新之后优于无偏训练15.5%，在100 k更新之后优于无偏训练23.8%。对于流模型，预训练方法产生了44.1%的字错误率的减少。摘要：Self-supervised learning via masked prediction pre-training (MPPT) has shown impressive performance on a range of speech-processing tasks. This paper proposes a method to bias self-supervised learning towards a specific task. The core idea is to slightly finetune the model that is used to obtain the target sequence. This leads to better performance and a substantial increase in training speed. Furthermore, this paper proposes a variant of MPPT that allows low-footprint streaming models to be trained effectively by computing the MPPT loss on masked and unmasked frames. These approaches are evaluated for automatic speech recognition on the Librispeech corpus, where 100 hours of data served as the labelled data and 860 hours as the unlabelled data. The biased training outperforms the unbiased training by 15.5% after 250k updates and 23.8% after 100k updates on test-other. For the streaming models, the pre-training approach yields a reduction in word error rate of 44.1%.
【2】 A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability
标题：一种具有真正零射能力的弱监督流媒体多语言语音模型
链接：https://arxiv.org/abs/2211.02499
作者：Jian Xue,Peidong Wang,Jinyu Li,Eric Sun机构：Microsoft Speech Group, Redmond, WA, USA备注：submitted to ICASSP 2023摘要：本文介绍了我们建立一个流式多语言语音模型（SM 2）的工作，该模型可以将多种口语转换成目标语言的文本。SM 2的主干是Transformer传感器，它具有很高的流传输能力。代替人工标记的语音翻译（ST）数据，SM 2模型使用通过用机器翻译服务转换语音识别语料库中的转录而生成的弱监督数据来训练。基于25种语言的35.1万小时的匿名语音训练数据，SM 2模型的ST质量与一些最近流行的大规模非流语音模型相当，甚至更好。更重要的是，我们表明SM 2在扩展到新的目标语言时具有真正的zero-shot能力，为训练期间未看到的{源-语音，目标-文本}对产生高质量的ST结果。摘要：In this paper, we introduce our work of building a Streaming Multilingual Speech Model (SM2), which can transcribe or translate multiple spoken languages into texts of the target language. The backbone of SM2 is Transformer Transducer, which has high streaming capability. Instead of human labeled speech translation (ST) data, SM2 models are trained using weakly supervised data generated by converting the transcriptions in speech recognition corpora with a machine translation service. With 351 thousand hours of anonymized speech training data from 25 languages, SM2 models achieve comparable or even better ST quality than some recent popular large-scale non-streaming speech models. More importantly, we show that SM2 has the truly zero-shot capability when expanding to new target languages, yielding high quality ST results for {source-speech, target-text} pairs that are not seen during training.
【3】 NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS
标题：NoreSpeech：基于知识提取的抗噪表达TTS条件扩散模型
链接：https://arxiv.org/abs/2211.02448
作者：Dongchao Yang,Songxiang Liu,Jianwei Yu,Helin Wang,Chao Weng,Yuexian Zou机构：ADSPLAB, School of ECE, Peking University, China, Tencent AI Lab, Shenzhen, China, Johns Hopkins University, USA备注：Submitted to ICASSP2023摘要：表现型文本-语音转换（TTS）技术可以通过模仿参考音频的韵律和音色来合成新的说话风格，但它面临着以下挑战：（1）难以提取参考音频中的高动态韵律信息，特别是当参考音频包含背景噪声时。(2)TTS系统应该对看不见的说话风格有很好的概括。本文提出了一种鲁棒的可表达的TTS模型（NoreSpeech），该模型能够鲁棒地将带噪参考话语中的说话风格转换为合成语音.具体来说，我们的NoreSpeech包括几个组件：（1）新颖的DiffStyle模块，其利用强大的概率去噪扩散模型来通过知识提炼从教师模型学习与噪声无关的说话风格特征;（2）VQ-VAE块，将风格特征映射到一个可控的量化潜在空间，以提高风格迁移的泛化性;以及（3）直接但有效的无参数文本风格对齐模块，其使得NoreSpeech能够将风格从长度不匹配的参考话语转移到文本输入。实验结果表明，在噪声环境下，NoreSpeech比已有的表现型TTS模型具有更好的效果。音频样本和代码可从以下网址获得：第二章：关于汉语语音教学的几点思考摘要：Expressive text-to-speech (TTS) can synthesize a new speaking style by imiating prosody and timbre from a reference audio, which faces the following challenges: (1) The highly dynamic prosody information in the reference audio is difficult to extract, especially, when the reference audio contains background noise. (2) The TTS systems should have good generalization for unseen speaking styles. In this paper, we present a \textbf{no}ise-\textbf{r}obust \textbf{e}xpressive TTS model (NoreSpeech), which can robustly transfer speaking style in a noisy reference utterance to synthesized speech. Specifically, our NoreSpeech includes several components: (1) a novel DiffStyle module, which leverages powerful probabilistic denoising diffusion models to learn noise-agnostic speaking style features from a teacher model by knowledge distillation; (2) a VQ-VAE block, which maps the style features into a controllable quantized latent space for improving the generalization of style transfer; and (3) a straight-forward but effective parameter-free text-style alignment module, which enables NoreSpeech to transfer style to a textual input from a length-mismatched reference utterance. Experiments demonstrate that NoreSpeech is more effective than previous expressive TTS models in noise environments. Audio samples and code are available at: \href{http://dongchaoyang.top/NoreSpeech\_demo/}{http://dongchaoyang.top/NoreSpeech\_demo/}
【4】 SPEAKER VGG CCT: Cross-corpus Speech Emotion Recognition with Speaker Embedding and Vision Transformers
标题：说话人VGG CCT：基于说话人嵌入和视觉变换的跨语料语音情感识别
链接：https://arxiv.org/abs/2211.02366
机构：Cross-corpus Speech Emotion Recognitionwith Speaker Embedding and Vision TransformersAlessandro ArezzoUniversity of FlorenceFlorence, comStefano BerrettiUniversity of FlorenceFlorence摘要：近年来，语音情感识别（Speech Emotion Recognition，SER）的研究主要是将语音信号转换为语谱图，然后利用在普通图像上预先训练并利用语谱图进行微调的卷积神经网络对语谱图进行分类。本文从上述基本思想出发，提出了一种基于紧凑卷积Transformers（CCT）和说话人嵌入相结合的SER学习方法。有了CCT，Vision Transformers（ViT）的学习能力与卷积所带来的对大量数据的需求减少相结合。这在SER中很重要，因为在SER中通常无法获得大量的数据。说话人嵌入允许网络提取说话人的身份表示，然后通过自注意机制将其与CCT从声谱图提取的特征相结合。总的来说，该解决方案能够在跨语料库场景中实时操作，显示有希望的结果，其中训练和测试数据集保持分离。在文献中很少使用的跨语料库环境中对几个基准进行了实验，结果与最先进的网络体系结构获得的结果相当或更好。您可以在www.example.com上找到我们的代码https://github.com/JabuMlDev/Speaker-VGG-CCT。摘要：In recent years, Speech Emotion Recognition (SER) has been investigated mainly transforming the speech signal into spectrograms that are then classified using Convolutional Neural Networks pretrained on generic images and fine tuned with spectrograms. In this paper, we start from the general idea above and develop a new learning solution for SER, which is based on Compact Convolutional Transformers (CCTs) combined with a speaker embedding. With CCTs, the learning power of Vision Transformers (ViT) is combined with a diminished need for large volume of data as made possible by the convolution. This is important in SER, where large corpora of data are usually not available. The speaker embedding allows the network to extract an identity representation of the speaker, which is then integrated by means of a self-attention mechanism with the features that the CCT extracts from the spectrogram. Overall, the solution is capable of operating in real-time showing promising results in a cross-corpus scenario, where training and test datasets are kept separate. Experiments have been performed on several benchmarks in a cross-corpus setting as rarely used in the literature, with results that are comparable or superior to those obtained with state-of-the-art network architectures. Our code is available at https://github.com/JabuMlDev/Speaker-VGG-CCT.
【5】 Improving Speech Prosody of Audiobook Text-to-Speech Synthesis with Acoustic and Textual Contexts
标题：利用声学和文本语境改进有声读物文语转换合成的语音韵律
链接：https://arxiv.org/abs/2211.02336
作者：Detai Xin,Sharath Adavanne,Federico Ang,Ashish Kulkarni,Shinnosuke Takamichi,Hiroshi Saruwatari机构：Rakuten Group, Inc., Japan, Graduate School of Information Science and Technology, The University of Tokyo, Japan摘要：本文提出了一个多说话人日语有声读物文本到语音（TTS）系统，该系统充分利用先前声学上下文和双边文本上下文的多模态上下文信息来改善合成语音的韵律。以往的研究要么使用单模态的上下文信息，要么使用单模态的上下文信息，不能充分反映上下文信息。该方法利用声学上下文编码器和文本上下文编码器来聚合上下文信息，并将其反馈给TTS模型，使其能够预测上下文相关的韵律。我们对一个多人日语有声读物数据集进行了客观和主观的综合评价。实验结果表明，该方法的性能明显优于已有的两种方法。此外，我们提出的见解，不同的选择上下文-模态，横向信息和长度-有声读物TTS，从来没有讨论过的文献之前。摘要：We present a multi-speaker Japanese audiobook text-to-speech (TTS) system that leverages multimodal context information of preceding acoustic context and bilateral textual context to improve the prosody of synthetic speech. Previous work either uses unilateral or single-modality context, which does not fully represent the context information. The proposed method uses an acoustic context encoder and a textual context encoder to aggregate context information and feeds it to the TTS model, which enables the model to predict context-dependent prosody. We conducted comprehensive objective and subjective evaluations on a multi-speaker Japanese audiobook dataset. Experimental results demonstrate that the proposed method significantly outperforms two previous works. Additionally, we present insights about the different choices of context - modalities, lateral information and length - for audiobook TTS that have never been discussed in the literature before.
【6】 Once-for-All Sequence Compression for Self-Supervised Speech Models
标题：用于自监督语音模型的一次性序列压缩
链接：https://arxiv.org/abs/2211.02332
作者：Hsuan-Jui Chen,Yen Meng,Hung-yi Lee机构：Nation Taiwan University备注：Submitted to ICASSP 2023摘要：沿时间轴的序列长度通常是自监督语音模型的计算成本的主要因素。已经提出了减少序列长度以降低计算成本的工作。然而，不同的下游任务具有不同的序列压缩容忍度，因此产生固定压缩率的模型可能不适合所有的任务。在本研究中，我们提出一种适用于自我监督语音模型的一次所有（once-for-all，OFA）序列压缩架构，其支持连续范围的压缩率。该框架在各种任务上进行了评估，与固定压缩率变体相比，显示出边际退化，具有平滑的性能-效率权衡。我们进一步探索了自适应压缩率学习，展示了在不需要网格搜索的情况下选择特定任务的优选帧周期的能力。摘要：The sequence length along the time axis is often the dominant factor of the computational cost of self-supervised speech models. Works have been proposed to reduce the sequence length for lowering the computational cost. However, different downstream tasks have different tolerance of sequence compressing, so a model that produces a fixed compressing rate may not fit all tasks. In this work, we introduce a once-for-all (OFA) sequence compression framework for self-supervised speech models that supports a continuous range of compressing rates. The framework is evaluated on various tasks, showing marginal degradation compared to the fixed compressing rate variants with a smooth performance-efficiency trade-off. We further explore adaptive compressing rate learning, demonstrating the ability to select task-specific preferred frame periods without needing a grid search.
【7】 Binaural Rendering of Ambisonic Signals by Neural Networks
标题：基于神经网络的两性声信号双耳再现
链接：https://arxiv.org/abs/2211.02301
作者：Yin Zhu,Qiuqiang Kong,Junjie Shi,Shilei Liu,Xuzhou Ye,Ju-chiang Wang,Junping Zhang机构：Department of Computer Science, Fudan University, Audio & Music Intelligence (SAMI), ByteDance Inc.摘要：环绕声信号的双耳再现在虚拟现实和沉浸式媒体中具有广泛的兴趣。传统方法通常需要手动测量头部相关传递函数（HRTF）。为了解决这一问题，我们收集了一个成对的双声道立体声数据集，并提出了一个端到端的深度学习框架。实验结果表明，神经网络在客观评价指标上优于传统方法，在主观评价指标上与传统方法具有可比性。为了验证所提出的框架，我们实验性地探索了输入特征、模型结构、输出特征和损失函数的不同设置。我们提出的系统在质量、音色、定位和沉浸维度上实现了7.32的SDR和3.83、3.58、3.87、3.58的MOS。摘要：Binaural rendering of ambisonic signals is of broad interest to virtual reality and immersive media. Conventional methods often require manually measured Head-Related Transfer Functions (HRTFs). To address this issue, we collect a paired ambisonic-binaural dataset and propose a deep learning framework in an end-to-end manner. Experimental results show that neural networks outperform the conventional method in objective metrics and achieve comparable subjective metrics. To validate the proposed framework, we experimentally explore different settings of the input features, model structures, output features, and loss functions. Our proposed system achieves an SDR of 7.32 and MOSs of 3.83, 3.58, 3.87, 3.58 in quality, timbre, localization, and immersion dimensions.
【8】 Wireless Deep Speech Semantic Transmission
标题：无线深度语音语义传输
链接：https://arxiv.org/abs/2211.02283
作者：Zixuan Xiao,Shengshi Yao,Jincheng Dai,Sixian Wang,Kai Niu,Ping Zhang机构：⋆ Beijing University of Posts and Telecommunications, Beijing, China, † Peng Cheng Laboratory, Shenzhen, China摘要：本文提出了一种新的高效的语义编码传输方法，用于无线信道上的端到端语音传输。我们称整个系统为深层语音语义传输（DSST）。具体地说，我们引入了一个非线性变换将语音源映射到语义潜在空间，并将语义特征反馈到源-通道编码器以生成通道-输入序列。在变分建模思想的指导下，在隐空间上建立熵模型来估计语义特征嵌入之间的重要性差异。相应地，这些不同重要性的语义特征可以被合理地分配不同的编码速率，从而使系统编码增益最大化。此外，我们引入了信道信噪比（SNR）自适应机制，使得单个模型可以应用于各种信道状态。通过对模型进行端到端的优化，实现了灵活的率失真（RD）折衷，支持多种无线语音语义传输。实验结果表明，DSST系统在主观和客观两个方面都明显优于现有的工程语音传输系统。与现有的神经语音语义传输方法相比，在达到相同质量的情况下，该模型最多可节省75%的信道带宽开销。您可以在www.example.com上找到音频演示的直观比较https://ximoo123.github.io/DSST。摘要：In this paper, we propose a new class of high-efficiency semantic coded transmission methods for end-to-end speech transmission over wireless channels. We name the whole system as deep speech semantic transmission (DSST). Specifically, we introduce a nonlinear transform to map the speech source to semantic latent space and feed semantic features into source-channel encoder to generate the channel-input sequence. Guided by the variational modeling idea, we build an entropy model on the latent space to estimate the importance diversity among semantic feature embeddings. Accordingly, these semantic features of different importance can be allocated with different coding rates reasonably, which maximizes the system coding gain. Furthermore, we introduce a channel signal-to-noise ratio (SNR) adaptation mechanism such that a single model can be applied over various channel states. The end-to-end optimization of our model leads to a flexible rate-distortion (RD) trade-off, supporting versatile wireless speech semantic transmission. Experimental results verify that our DSST system clearly outperforms current engineered speech transmission systems on both objective and subjective metrics. Compared with existing neural speech semantic transmission methods, our model saves up to 75% of channel bandwidth costs when achieving the same quality. An intuitive comparison of audio demos can be found at https://ximoo123.github.io/DSST.
【9】 Real-Time Target Sound Extraction
标题：目标声音的实时提取
链接：https://arxiv.org/abs/2211.02250
作者：Bandhav Veluri,Justin Chan,Malek Itani,Tuochao Chen,Takuya Yoshioka,Shyamnath Gollakota机构：⋄Paul G. Allen School of Computer Science & Engineering, University of Washington, USA, •Microsoft, One Microsoft Way, Redmond, WA, USA摘要：我们提出了第一个神经网络模型，以实现实时和流式目标声音提取。为了实现这一点，我们提出了Waveformer，一种编码器-解码器架构，其具有作为编码器的膨胀因果卷积层的堆叠，以及作为解码器的Transformer解码器层。该混合架构使用扩张因果卷积，用于以计算上有效的方式处理大的感受野，同时还受益于基于变换器的架构所提供的性能。我们的评估显示，与之前的模型相比，针对此任务，SI-SNRi提高了2.2-3.3 dB，同时模型大小缩小了1.2- 4倍，运行时间缩短了1.5- 2倍。开放源代码和数据集：https://github.com/vb000/Waveformer摘要：We present the first neural network model to achieve real-time and streaming target sound extraction. To accomplish this, we propose Waveformer, an encoder-decoder architecture with a stack of dilated causal convolution layers as the encoder, and a transformer decoder layer as the decoder. This hybrid architecture uses dilated causal convolutions for processing large receptive fields in a computationally efficient manner, while also benefiting from the performance transformer-based architectures provide. Our evaluations show as much as 2.2-3.3 dB improvement in SI-SNRi compared to the prior models for this task while having a 1.2-4x smaller model size and a 1.5-2x lower runtime. Open-source code and datasets: https://github.com/vb000/Waveformer
【10】 CCATMos: Convolutional Context-aware Transformer Network for Non-intrusive Speech Quality Assessment
标题：CCATMos：用于非干扰语音质量评估的卷积上下文感知转换网络
链接：https://arxiv.org/abs/2211.02577
作者：Yuchen Liu,Li-Chia Yang,Alex Pawlicki,Marko Stamenovic机构：Bose Corporation, USA, Department of Computer Science, Indiana University, USA摘要：语音质量评估已经成为诸如电话和在线会议的许多语音通信相关应用中的关键组成部分。传统的侵入式语音质量评估需要清晰地参考劣化的话语以提供准确的质量测量。这一要求限制了这些方法在现实场景中的可用性。另一方面，非侵入性的主观测量是评价语音质量的"黄金标准”，因为人类听众可以很容易地从本质上评价任何劣化语音的质量。本文提出了一种新的端到端模型结构--卷积上下文感知转换器（ConvolutionalContext-Aware Transformer，CCAT）网络，用于预测评分者的平均意见得分（MeanOpinionScore，MOS）。我们在跨多种语言和失真类型的三个MOS标注数据集上评估我们的模型，并将结果提交给ConferencingSpeech 2022挑战赛。实验结果表明，与现有的非侵入式语音评估模型相比，CCAT能够提供较好的MOS预测，在挑战性评估测试集上，与基线模型相比，平均Pearson相关系数（PCC）从0.530提高到0.697，平均RMSE从0.768降低到0.570。摘要：Speech quality assessment has been a critical component in many voice communication related applications such as telephony and online conferencing. Traditional intrusive speech quality assessment requires the clean reference of the degraded utterance to provide an accurate quality measurement. This requirement limits the usability of these methods in real-world scenarios. On the other hand, non-intrusive subjective measurement is the ``golden standard" in evaluating speech quality as human listeners can intrinsically evaluate the quality of any degraded speech with ease. In this paper, we propose a novel end-to-end model structure called Convolutional Context-Aware Transformer (CCAT) network to predict the mean opinion score (MOS) of human raters. We evaluate our model on three MOS-annotated datasets spanning multiple languages and distortion types and submit our results to the ConferencingSpeech 2022 Challenge. Our experiments show that CCAT provides promising MOS predictions compared to current state-of-art non-intrusive speech assessment models with average Pearson correlation coefficient (PCC) increasing from 0.530 to 0.697 and average RMSE decreasing from 0.768 to 0.570 compared to the baseline model on the challenge evaluation test set.
【11】 Self-Supervised Learning for Speech Enhancement through Synthesis
标题：基于自监督学习的语音合成增强算法
链接：https://arxiv.org/abs/2211.02542
作者：Bryce Irvin,Marko Stamenovic,Mikolaj Kegler,Li-Chia Yang机构：Bose Corporation, USA, Georgia Institute of Technology, USA摘要：现代语音增强（SE）网络通常通过时频掩蔽、潜在表示掩蔽或区别性信号预测来实现噪声抑制。与此相反，一些最近的工作通过生成语音合成来探索SE，其中系统的输出在固有的有损特征去噪步骤之后由神经声码器合成。本文提出了一种去噪声码器（DeVo）方法，其中声码器接受噪声表示并学习直接合成干净的语音。我们利用来自自监督学习（SSL）语音模型的丰富表示来发现相关特征。我们在15个潜在的SSL前端中进行候选搜索，然后使用最佳SSL配置对我们的声码器进行反向训练。此外，我们还演示了一个因果版本，该版本能够在流音频上运行，延迟为10毫秒，性能下降最小。最后，我们进行了客观评估和主观听力研究，以表明我们的系统改善了客观指标，并在主观上优于现有的最先进的SE模型。摘要：Modern speech enhancement (SE) networks typically implement noise suppression through time-frequency masking, latent representation masking, or discriminative signal prediction. In contrast, some recent works explore SE via generative speech synthesis, where the system's output is synthesized by a neural vocoder after an inherently lossy feature-denoising step. In this paper, we propose a denoising vocoder (DeVo) approach, where a vocoder accepts noisy representations and learns to directly synthesize clean speech. We leverage rich representations from self-supervised learning (SSL) speech models to discover relevant features. We conduct a candidate search across 15 potential SSL front-ends and subsequently train our vocoder adversarially with the best SSL configuration. Additionally, we demonstrate a causal version capable of running on streaming audio with 10ms latency and minimal performance degradation. Finally, we conduct both objective evaluations and subjective listening studies to show our system improves objective metrics and outperforms an existing state-of-the-art SE model subjectively.
【12】 Cold Diffusion for Speech Enhancement
标题：用于语音增强的冷扩散
链接：https://arxiv.org/abs/2211.02527
作者：Hao Yen,François G. Germain,Gordon Wichern,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, School of Electrical and Computer Engineering, Georgia Institute of Technology, GA, USA备注：5 pages, 1 figure, 1 table, 3 algorithms. Submitted to ICASSP 2023摘要：扩散模型最近已经显示出对于困难的增强任务（例如自然图像和音频信号的有条件和无条件恢复）的有希望的结果。在本研究中，我们探讨了利用最近提出的一种先进的迭代扩散模型，即冷扩散，从噪声信号中恢复干净语音信号的可能性。冷扩散采样过程的独特数学性质可用于恢复任意退化的高质量样品。基于这些特性，我们提出了一种改进的训练算法和目标，以帮助模型在采样过程中更好地泛化。我们通过研究两个模型体系结构来验证我们提出的框架。在基准语音增强数据集VoiceBank-DEMAND上的实验结果表明，与典型的判别式模型和基于扩散的增强模型相比，该方法具有更好的性能。摘要：Diffusion models have recently shown promising results for difficult enhancement tasks such as the conditional and unconditional restoration of natural images and audio signals. In this work, we explore the possibility of leveraging a recently proposed advanced iterative diffusion model, namely cold diffusion, to recover clean speech signals from noisy signals. The unique mathematical properties of the sampling process from cold diffusion could be utilized to restore high-quality samples from arbitrary degradations. Based on these properties, we propose an improved training algorithm and objective to help the model generalize better during the sampling process. We verify our proposed framework by investigating two model architectures. Experimental results on benchmark speech enhancement dataset VoiceBank-DEMAND demonstrate the strong performance of the proposed approach compared to representative discriminative models and diffusion-based enhancement models.
【13】 Neural Feature Predictor and Discriminative Residual Coding for Low-Bitrate Speech Coding
标题：用于低比特率语音编码的神经特征预测器和判别残差编码
链接：https://arxiv.org/abs/2211.02506
作者：Haici Yang,Wootaek Lim,Minje Kim机构：Indiana University, Luddy School of Informatics, Computing, and Engineering, Bloomington, IN, USA, Electronics and Telecommunications Research Institute, Daejeon , South Korea摘要：低比特率和超低比特率神经语音编码通过从紧凑的语音特征生成语音信号来实现前所未有的编码增益。通过递归神经预测器减少帧级特征序列中存在的时间冗余，提高了神经语音编码的编码效率。预测可以实现低熵残差表示，我们基于它们对信号重构的贡献来区别性地编码该低熵残差表示。特征预测和区别编码的协调产生了动态比特分配算法，该算法在不可预测但罕见的事件上花费更多比特。因此，我们开发了一个可伸缩的、轻量级的、低延迟的、低比特率的神经语音编码系统。我们使用LPCNet作为神经声码器来证明所提出的方法的优点。主观测试和特征空间分析结果表明，该模型在保证预测结果的因果性的同时，在极低的码率下，与LPCNet和Lyra V2相比，具有更高的编码效率。摘要：Low and ultra-low-bitrate neural speech coding achieves unprecedented coding gain by generating speech signals from compact speech features. This paper introduces additional coding efficiency in neural speech coding by reducing the temporal redundancy existing in the frame-level feature sequence via a recurrent neural predictor. The prediction can achieve a low-entropy residual representation, which we discriminatively code based on their contribution to the signal reconstruction. The harmonization of feature prediction and discriminative coding results in a dynamic bit allocation algorithm that spends more bits on unpredictable but rare events. As a result, we develop a scalable, lightweight, low-latency, and low-bitrate neural speech coding system. We demonstrate the advantage of the proposed methods using the LPCNet as a neural vocoder. While the proposed method guarantees causality in its prediction, the subjective tests and feature space analysis show that our model achieves superior coding efficiency compared to LPCNet and Lyra V2 in the very low bitrates.
【14】 Sampling Rate Offset Estimation and Compensation for Distributed Adaptive Node-Specific Signal Estimation in Wireless Acoustic Sensor Networks
标题：无线水声传感器网络中分布式自适应特定节点信号估计的采样率偏差估计与补偿
链接：https://arxiv.org/abs/2211.02489
作者：Paul Didier,Toon van Waterschoot,Simon Doclo,Marc Moonen机构：STADIUS Center for Dynamical Systems, Department of Electrical Engineering (ESAT), KU Leuven, Leuven, Belgium, Signal Processing Group, Department of Medical Physics and Acoustics and Cluster of Excellence Hearing,all摘要：异构无线声学传感器网络（WASN）中设备之间的采样率偏移（SRO）会阻碍分布式自适应算法在依赖于相干信号处理时按预期执行的能力。本文提出了一种SRO估计和补偿方法，以允许分布式自适应节点特定信号估计（DANSE）算法在由异步设备组成的WASN中的部署。首先在基于相干漂移的方法中利用在每个节点处可用的信号来盲估计SRO，然后经由频域中的相移来补偿SRO。引入了DANSE的加权叠加（WOLA）实现的修改，以解决SRO引起的全采样漂移，允许经由作为时域卷积的WOLA过程的近似的每采样信号传输。在异步WASN中对目标语音信号估计的分布式降噪环境中评估了所提出算法的性能。摘要：Sampling rate offsets (SROs) between devices in a heterogeneous wireless acoustic sensor network (WASN) can hinder the ability of distributed adaptive algorithms to perform as intended when they rely on coherent signal processing. In this paper, we present an SRO estimation and compensation method to allow the deployment of the distributed adaptive node-specific signal estimation (DANSE) algorithm in WASNs composed of asynchronous devices. The signals available at each node are first utilised in a coherence-drift-based method to blindly estimate SROs which are then compensated for via phase shifts in the frequency domain. A modification of the weighted overlap-add (WOLA) implementation of DANSE is introduced to account for SRO-induced full-sample drifts, permitting per-sample signal transmission via an approximation of the WOLA process as a time-domain convolution. The performance of the proposed algorithm is evaluated in the context of distributed noise reduction for the estimation of a target speech signal in an asynchronous WASN.
【15】 Spatially Selective Deep Non-linear Filters for Speaker Extraction
标题：用于说话人提取的空间选择性深度非线性滤波器
链接：https://arxiv.org/abs/2211.02420
作者：Kristina Tesch,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany备注：Submitted to ICASSP 2023摘要：在多人同时讲话的场景中，信号的空间特性是用于提取目标信号的最明显特征。本文提出了一种深度联合空谱非线性滤波器，它可以在任意目标方向上进行控制。为此，提出了一种简单有效的条件化机制，根据目标方向设置滤波器递归层的初始状态.我们表明，该方案比基线方法更有效，并且在不损失性能的情况下增加了滤波器的灵活性。所得到的空间选择性非线性滤波器还可以用于任意数量的说话人的语音分离，并且能够实现非常精确的多说话人定位，如我们在本文中所演示的。摘要：In a scenario with multiple persons talking simultaneously, the spatial characteristics of the signals are the most distinct feature for extracting the target signal. In this work, we develop a deep joint spatial-spectral non-linear filter that can be steered in an arbitrary target direction. For this we propose a simple and effective conditioning mechanism, which sets the initial state of the filter's recurrent layers based on the target direction. We show that this scheme is more effective than the baseline approach and increases the flexibility of the filter at no performance cost. The resulting spatially selective non-linear filters can also be used for speech separation of an arbitrary number of speakers and enable very accurate multi-speaker localization as we demonstrate in this paper.
【16】 Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration
标题：语音恢复中基于扩散的生成方法与区分方法的比较
链接：https://arxiv.org/abs/2211.02397
作者：Jean-Marie Lemercier,Julius Richter,Simon Welker,Timo Gerkmann机构：⋆Signal Processing (SP), Universit¨at Hamburg, Germany, × Center for Free-Electron Laser Science, DESY, Germany备注：Submitted to ICASSP 2023摘要：近年来，基于扩散的生成模型对计算机视觉和语音处理领域产生了很大的影响。除了数据生成任务之外，它们还被用于数据恢复任务，如语音增强和去混响。虽然传统上认为判别模型例如对于语音增强更强大，但是最近已经显示出生成扩散方法显著地缩小了该性能差距。本文系统地比较了生成扩散模型和判别方法在不同语音恢复任务中的性能。为此，我们将我们先前在复时频域中的基于扩散的语音增强的贡献扩展到带宽扩展的任务。然后将其与具有相同网络结构的区别训练神经网络在语音去噪、去混响和带宽扩展三个恢复任务上进行比较。我们观察到，在所有任务上，生成式方法的整体性能优于其鉴别式对应物，对于非加性失真模型，如在去混响和带宽扩展中，具有最大的益处。代码和音频示例可在www.example.com上在线找到https://uhh.de/inf-sp-sgmsemultitask摘要：Diffusion-based generative models have had a high impact on the computer vision and speech processing communities these past years. Besides data generation tasks, they have also been employed for data restoration tasks like speech enhancement and dereverberation. While discriminative models have traditionally been argued to be more powerful e.g. for speech enhancement, generative diffusion approaches have recently been shown to narrow this performance gap considerably. In this paper, we systematically compare the performance of generative diffusion models and discriminative approaches on different speech restoration tasks. For this, we extend our prior contributions on diffusion-based speech enhancement in the complex time-frequency domain to the task of bandwith extension. We then compare it to a discriminatively trained neural network with the same network architecture on three restoration tasks, namely speech denoising, dereverberation and bandwidth extension. We observe that the generative approach performs globally better than its discriminative counterpart on all tasks, with the strongest benefit for non-additive distortion models, like in dereverberation and bandwidth extension. Code and audio examples can be found online at https://uhh.de/inf-sp-sgmsemultitask
【17】 Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition
标题：用于流端到端语音识别的序列换能器的最小延迟训练
链接：https://arxiv.org/abs/2211.02333
作者：Yusuke Shinohara,Shinji Watanabe机构：Yahoo Japan Corporation, Japan,Carnegie Mellon University, USA备注：Presented at INTERSPEECH 2022摘要：序列转换器，如RNN-T和Conformer-T，是最有前途的端到端语音识别模型之一，特别是在延迟和准确性都很重要的流场景中。尽管已经研究了各种方法来减少延迟，例如对准限制训练和FastEmit，但是延迟减少通常伴随着准确性的显著降低。我们认为，这种次优性能可能是由于先前的方法没有显式地建模和减少延迟而导致的。本文提出了一种新的训练方法来显式地建模和减少序列传感器模型的延迟。首先，我们定义了格上每条对角线上的期望延迟，并证明了它的梯度可以在前向-后向算法中有效地计算。然后，我们用该预期延迟来增加换能器损耗，从而实现延迟和准确性之间的最佳折衷。在WSJ数据集上的实验结果表明，在WER下降0.7%的情况下，所提出的最小时延训练方法将因果Conformer-T的时延从220 ms降低到27 ms，优于传统的对齐限制训练方法（110 ms）和FastEmit方法（67 ms）.摘要：Sequence transducers, such as the RNN-T and the Conformer-T, are one of the most promising models of end-to-end speech recognition, especially in streaming scenarios where both latency and accuracy are important. Although various methods, such as alignment-restricted training and FastEmit, have been studied to reduce the latency, latency reduction is often accompanied with a significant degradation in accuracy. We argue that this suboptimal performance might be caused because none of the prior methods explicitly model and reduce the latency. In this paper, we propose a new training method to explicitly model and reduce the latency of sequence transducer models. First, we define the expected latency at each diagonal line on the lattice, and show that its gradient can be computed efficiently within the forward-backward algorithm. Then we augment the transducer loss with this expected latency, so that an optimal trade-off between latency and accuracy is achieved. Experimental results on the WSJ dataset show that the proposed minimum latency training reduces the latency of causal Conformer-T from 220 ms to 27 ms within a WER degradation of 0.7%, and outperforms conventional alignment-restricted training (110 ms) and FastEmit (67 ms) methods.
【18】 CochlScene: Acquisition of acoustic scene data using crowdsourcing
标题：CochlScene：使用众包方式获取声音场景数据
链接：https://arxiv.org/abs/2211.02289
作者：Il-Young Jeong,Jeongsoo Park机构：Cochl, Inc., Seoul, Republic of Korea备注：Accept by APSIPA ASC 2022, 5 pages, 2 figures摘要：本文描述了一种利用众包技术收集声学场景数据的管道。解释了众包的详细过程，包括规划、验证标准和实际的用户界面。作为数据收集的结果，我们提出了一种新的用于声学场景分类的数据集CochlScene。我们的数据集包括从13个声学场景中的831个参与者收集的76k个样本。我们还提出了训练集、验证集和测试集的手动数据分割，以增加评估结果的可靠性。最后，本研究为后续研究提供了一个基本的系统。摘要：This paper describes a pipeline for collecting acoustic scene data by using crowdsourcing. The detailed process of crowdsourcing is explained, including planning, validation criteria, and actual user interfaces. As a result of data collection, we present CochlScene, a novel dataset for acoustic scene classification. Our dataset consists of 76k samples collected from 831 participants in 13 acoustic scenes. We also propose a manual data split of training, validation, and test sets to increase the reliability of the evaluation results. Finally, we provide a baseline system for future research.
【19】 Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects
标题：音乐混合风格迁移：一种解开音效的对比学习方法
链接：https://arxiv.org/abs/2211.02247
作者：Junghyun Koo,Marco A. Martinez-Ramirez,Wei-Hsiang Liao,Stefan Uhlich,Kyogu Lee,Yuki Mitsufuji机构：Marco A. Mart´ınez-Ram´ırez, Sony Group Corporation, Tokyo, Japan, Sony Europe B.V., Stuttgart, Germany, Music and Audio Research Group, Department of Intelligence and Information, Seoul National University摘要：我们提出了一种端到端的音乐混音风格转换系统，该系统将输入多音轨的混音风格转换为参考歌曲的混音风格。这是利用以对比目标预先训练的编码器来实现的，以从参考音乐录音中仅提取音频效果相关信息。所有模型均采用自监督的方式，利用一种有效的数据预处理方法，从一个已经处理过的湿多轨数据集上训练而成，从而缓解了未处理干数据的数据稀缺性。我们分析了所提出的编码器对音频效果的解纠缠能力，并通过客观和主观评价验证了其混合风格转换的性能。从结果中，我们表明所提出的系统不仅将多声道音频的混合风格转换成接近参考的混合风格，而且在使用音乐源分离模型时对于混合风格转换也是鲁棒的。摘要：We propose an end-to-end music mixing style transfer system that converts the mixing style of an input multitrack to that of a reference song. This is achieved with an encoder pre-trained with a contrastive objective to extract only audio effects related information from a reference music recording. All our models are trained in a self-supervised manner from an already-processed wet multitrack dataset with an effective data preprocessing method that alleviates the data scarcity of obtaining unprocessed dry data. We analyze the proposed encoder for the disentanglement capability of audio effects and also validate its performance for mixing style transfer through both objective and subjective evaluations. From the results, we show the proposed system not only converts the mixing style of multitrack audio close to a reference but is also robust with mixture-wise style transfer upon using a music source separation model.
【20】 Integrated Parameter-Efficient Tuning for General-Purpose Audio Models
标题：通用音频模型的集成参数高效调谐
链接：https://arxiv.org/abs/2211.02227
作者：Ju-ho Kim,Jungwoo Heo,Hyun-seo Shin,Chan-yeong Lim,Ha-Jin Yu机构：School of Computer Science, University of Seoul备注：5 pages, 3 figures, submit to ICASSP2023摘要：超大规模和通用预训练模型的出现正在改变为目标任务构建特定任务模型的范式。在音频研究领域，任务无关的预训练模型具有很高的可移植性和适应性，通过对下游任务的微调，已经取得了最好的性能。然而，重新训练这些庞大模型的所有参数需要大量的时间和成本，以及巨大的碳足迹。为了克服这些局限性，本研究在音频领域探索并应用了有效的迁移学习方法。我们还提出了一个集成的参数有效调整（IPET）框架，通过聚合嵌入提示（一种基于提示的学习方法）和适配器（一种有效的迁移学习方法）。我们使用具有不同特征的两个骨干预训练音频模型来证明所提出的框架的功效：音频频谱图Transformer和WAV2VEC2.0。与可训练参数较少的微调方法相比，所提出的IPET框架在以下四个下游任务中表现出了显著的性能：声音事件分类、音乐流派分类、关键词识别和说话人验证。此外，作者还指出并分析了IPET框架的不足，为音频领域参数的有效调整提供了借鉴和研究方向。摘要：The advent of hyper-scale and general-purpose pre-trained models is shifting the paradigm of building task-specific models for target tasks. In the field of audio research, task-agnostic pre-trained models with high transferability and adaptability have achieved state-of-the-art performances through fine-tuning for downstream tasks. Nevertheless, re-training all the parameters of these massive models entails an enormous amount of time and cost, along with a huge carbon footprint. To overcome these limitations, the present study explores and applies efficient transfer learning methods in the audio domain. We also propose an integrated parameter-efficient tuning (IPET) framework by aggregating the embedding prompt (a prompt-based learning approach), and the adapter (an effective transfer learning method). We demonstrate the efficacy of the proposed framework using two backbone pre-trained audio models with different characteristics: the audio spectrogram transformer and wav2vec 2.0. The proposed IPET framework exhibits remarkable performance compared to fine-tuning method with fewer trainable parameters in four downstream tasks: sound event classification, music genre classification, keyword spotting, and speaker verification. Furthermore, the authors identify and analyze the shortcomings of the IPET framework, providing lessons and research directions for parameter efficient tuning in the audio domain.
【21】 Streaming Audio-Visual Speech Recognition with Alignment Regularization
标题：基于对齐正则化的流媒体视听语音识别
链接：https://arxiv.org/abs/2211.02133
作者：Pingchuan Ma,Niko Moritz,Stavros Petridis,Christian Fuegen,Maja Pantic备注：Submitted to ICASSP2023摘要：在说出一个单词后立即对其进行识别是自动语音识别（ASR）系统在真实世界场景中的一个重要要求。因此，在文献中已经提出了关于仅流音频ASR模型的大量工作。然而，流媒体音频视频自动语音识别（AV-ASR）在早期的工作中很少受到关注。本文提出了一种基于CTC/注意力神经网络混合结构的流媒体AV-ASR系统。音频和视频编码器神经网络都基于构象器结构，使用块式自注意（CSA）和因果卷积使构象器结构可流化。采用触发注意技术实现了解码器神经网络的流识别，该技术采用CTC/注意力联合评分进行时间同步解码。对于帧级ASR标准，例如CTC，来自音频和视频编码器的同步响应对于联合AV决策过程是关键的。在这项工作中，我们提出了一种新的对齐正则化技术，该技术促进了音频和视频编码器的同步，这进而导致在所有SNR水平下的流和离线AV-ASR模型的更好的误字率（WER）。该模型在LRS 3（Lip Reading Sentences 3）数据集上的离线和在线测试分别获得了2.0%和2.6%的WER，在不使用外部训练数据的情况下，这两个测试结果都达到了最佳水平.摘要：Recognizing a word shortly after it is spoken is an important requirement for automatic speech recognition (ASR) systems in real-world scenarios. As a result, a large body of work on streaming audio-only ASR models has been presented in the literature. However, streaming audio-visual automatic speech recognition (AV-ASR) has received little attention in earlier works. In this work, we propose a streaming AV-ASR system based on a hybrid connectionist temporal classification (CTC)/attention neural network architecture. The audio and the visual encoder neural networks are both based on the conformer architecture, which is made streamable using chunk-wise self-attention (CSA) and causal convolution. Streaming recognition with a decoder neural network is realized by using the triggered attention technique, which performs time-synchronous decoding with joint CTC/attention scoring. For frame-level ASR criteria, such as CTC, a synchronized response from the audio and visual encoders is critical for a joint AV decision making process. In this work, we propose a novel alignment regularization technique that promotes synchronization of the audio and visual encoder, which in turn results in better word error rates (WERs) at all SNR levels for streaming and offline AV-ASR models. The proposed AV-ASR model achieves WERs of 2.0% and 2.6% on the Lip Reading Sentences 3 (LRS3) dataset in an offline and online setup, respectively, which both present state-of-the-art results when no external training data are used.
【22】 Fearless Steps Challenge Phase-1 Evaluation Plan
标题：无畏步骤挑战阶段-第一阶段评估计划
链接：https://arxiv.org/abs/2211.02051
作者：Aditya Joglekar,John H. L. Hansen机构：arXiv:,.,v, [eess.AS] , Nov备注：Document Generated in February 2019 for conducting the Fearless Steps Challenge Phase-1 and its associated ISCA Interspeech-2019 Special Session摘要：无畏步伐挑战赛2019第1阶段（FSC-P1）是由达拉斯德克萨斯大学鲁棒语音系统中心（CRSS）主办的无畏步伐倡议的首届挑战赛。本挑战赛的目标是评估最先进的语音和语言系统在具有挑战性的环境中的表现，这些系统适用于具有自然主义音频的大型任务导向型团队。研究人员可以选择参与这些挑战性任务中的任何一项或多项。研究人员也可以选择将Fearless STEPS语料库用于其他相关的语音应用。我们鼓励所有与会者提交他们的解决方案和成果，供ISCA INTERSPEECH-2019特别会议审议。摘要：The Fearless Steps Challenge 2019 Phase-1 (FSC-P1) is the inaugural Challenge of the Fearless Steps Initiative hosted by the Center for Robust Speech Systems (CRSS) at the University of Texas at Dallas. The goal of this Challenge is to evaluate the performance of state-of-the-art speech and language systems for large task-oriented teams with naturalistic audio in challenging environments. Researchers may select to participate in any single or multiple of these challenge tasks. Researchers may also choose to employ the FEARLESS STEPS corpus for other related speech applications. All participants are encouraged to submit their solutions and results for consideration in the ISCA INTERSPEECH-2019 special session.
【1】 CCATMos: Convolutional Context-aware Transformer Network for Non-intrusive Speech Quality Assessment
标题：CCATMos：用于非干扰语音质量评估的卷积上下文感知转换网络
链接：https://arxiv.org/abs/2211.02577
* 与cs.SD语音【10】为同一篇
作者：Yuchen Liu,Li-Chia Yang,Alex Pawlicki,Marko Stamenovic机构：Bose Corporation, USA, Department of Computer Science, Indiana University, USA摘要：语音质量评估已经成为诸如电话和在线会议的许多语音通信相关应用中的关键组成部分。传统的侵入式语音质量评估需要清晰地参考劣化的话语以提供准确的质量测量。这一要求限制了这些方法在现实场景中的可用性。另一方面，非侵入性的主观测量是评价语音质量的"黄金标准”，因为人类听众可以很容易地从本质上评价任何劣化语音的质量。本文提出了一种新的端到端模型结构--卷积上下文感知转换器（ConvolutionalContext-Aware Transformer，CCAT）网络，用于预测评分者的平均意见得分（MeanOpinionScore，MOS）。我们在跨多种语言和失真类型的三个MOS标注数据集上评估我们的模型，并将结果提交给ConferencingSpeech 2022挑战赛。实验结果表明，与现有的非侵入式语音评估模型相比，CCAT能够提供较好的MOS预测，在挑战性评估测试集上，与基线模型相比，平均Pearson相关系数（PCC）从0.530提高到0.697，平均RMSE从0.768降低到0.570。摘要：Speech quality assessment has been a critical component in many voice communication related applications such as telephony and online conferencing. Traditional intrusive speech quality assessment requires the clean reference of the degraded utterance to provide an accurate quality measurement. This requirement limits the usability of these methods in real-world scenarios. On the other hand, non-intrusive subjective measurement is the ``golden standard" in evaluating speech quality as human listeners can intrinsically evaluate the quality of any degraded speech with ease. In this paper, we propose a novel end-to-end model structure called Convolutional Context-Aware Transformer (CCAT) network to predict the mean opinion score (MOS) of human raters. We evaluate our model on three MOS-annotated datasets spanning multiple languages and distortion types and submit our results to the ConferencingSpeech 2022 Challenge. Our experiments show that CCAT provides promising MOS predictions compared to current state-of-art non-intrusive speech assessment models with average Pearson correlation coefficient (PCC) increasing from 0.530 to 0.697 and average RMSE decreasing from 0.768 to 0.570 compared to the baseline model on the challenge evaluation test set.
【2】 Self-Supervised Learning for Speech Enhancement through Synthesis
标题：基于自监督学习的语音合成增强算法
链接：https://arxiv.org/abs/2211.02542
* 与cs.SD语音【11】为同一篇
作者：Bryce Irvin,Marko Stamenovic,Mikolaj Kegler,Li-Chia Yang机构：Bose Corporation, USA, Georgia Institute of Technology, USA摘要：现代语音增强（SE）网络通常通过时频掩蔽、潜在表示掩蔽或区别性信号预测来实现噪声抑制。与此相反，一些最近的工作通过生成语音合成来探索SE，其中系统的输出在固有的有损特征去噪步骤之后由神经声码器合成。本文提出了一种去噪声码器（DeVo）方法，其中声码器接受噪声表示并学习直接合成干净的语音。我们利用来自自监督学习（SSL）语音模型的丰富表示来发现相关特征。我们在15个潜在的SSL前端中进行候选搜索，然后使用最佳SSL配置对我们的声码器进行反向训练。此外，我们还演示了一个因果版本，该版本能够在流音频上运行，延迟为10毫秒，性能下降最小。最后，我们进行了客观评估和主观听力研究，以表明我们的系统改善了客观指标，并在主观上优于现有的最先进的SE模型。摘要：Modern speech enhancement (SE) networks typically implement noise suppression through time-frequency masking, latent representation masking, or discriminative signal prediction. In contrast, some recent works explore SE via generative speech synthesis, where the system's output is synthesized by a neural vocoder after an inherently lossy feature-denoising step. In this paper, we propose a denoising vocoder (DeVo) approach, where a vocoder accepts noisy representations and learns to directly synthesize clean speech. We leverage rich representations from self-supervised learning (SSL) speech models to discover relevant features. We conduct a candidate search across 15 potential SSL front-ends and subsequently train our vocoder adversarially with the best SSL configuration. Additionally, we demonstrate a causal version capable of running on streaming audio with 10ms latency and minimal performance degradation. Finally, we conduct both objective evaluations and subjective listening studies to show our system improves objective metrics and outperforms an existing state-of-the-art SE model subjectively.
【3】 Cold Diffusion for Speech Enhancement
标题：用于语音增强的冷扩散
链接：https://arxiv.org/abs/2211.02527
* 与cs.SD语音【12】为同一篇
作者：Hao Yen,François G. Germain,Gordon Wichern,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, School of Electrical and Computer Engineering, Georgia Institute of Technology, GA, USA备注：5 pages, 1 figure, 1 table, 3 algorithms. Submitted to ICASSP 2023摘要：扩散模型最近已经显示出对于困难的增强任务（例如自然图像和音频信号的有条件和无条件恢复）的有希望的结果。在本研究中，我们探讨了利用最近提出的一种先进的迭代扩散模型，即冷扩散，从噪声信号中恢复干净语音信号的可能性。冷扩散采样过程的独特数学性质可用于恢复任意退化的高质量样品。基于这些特性，我们提出了一种改进的训练算法和目标，以帮助模型在采样过程中更好地泛化。我们通过研究两个模型体系结构来验证我们提出的框架。在基准语音增强数据集VoiceBank-DEMAND上的实验结果表明，与典型的判别式模型和基于扩散的增强模型相比，该方法具有更好的性能。摘要：Diffusion models have recently shown promising results for difficult enhancement tasks such as the conditional and unconditional restoration of natural images and audio signals. In this work, we explore the possibility of leveraging a recently proposed advanced iterative diffusion model, namely cold diffusion, to recover clean speech signals from noisy signals. The unique mathematical properties of the sampling process from cold diffusion could be utilized to restore high-quality samples from arbitrary degradations. Based on these properties, we propose an improved training algorithm and objective to help the model generalize better during the sampling process. We verify our proposed framework by investigating two model architectures. Experimental results on benchmark speech enhancement dataset VoiceBank-DEMAND demonstrate the strong performance of the proposed approach compared to representative discriminative models and diffusion-based enhancement models.
【4】 Neural Feature Predictor and Discriminative Residual Coding for Low-Bitrate Speech Coding
标题：用于低比特率语音编码的神经特征预测器和判别残差编码
链接：https://arxiv.org/abs/2211.02506
* 与cs.SD语音【13】为同一篇
作者：Haici Yang,Wootaek Lim,Minje Kim机构：Indiana University, Luddy School of Informatics, Computing, and Engineering, Bloomington, IN, USA, Electronics and Telecommunications Research Institute, Daejeon , South Korea摘要：低比特率和超低比特率神经语音编码通过从紧凑的语音特征生成语音信号来实现前所未有的编码增益。通过递归神经预测器减少帧级特征序列中存在的时间冗余，提高了神经语音编码的编码效率。预测可以实现低熵残差表示，我们基于它们对信号重构的贡献来区别性地编码该低熵残差表示。特征预测和区别编码的协调产生了动态比特分配算法，该算法在不可预测但罕见的事件上花费更多比特。因此，我们开发了一个可伸缩的、轻量级的、低延迟的、低比特率的神经语音编码系统。我们使用LPCNet作为神经声码器来证明所提出的方法的优点。主观测试和特征空间分析结果表明，该模型在保证预测结果的因果性的同时，在极低的码率下，与LPCNet和Lyra V2相比，具有更高的编码效率。摘要：Low and ultra-low-bitrate neural speech coding achieves unprecedented coding gain by generating speech signals from compact speech features. This paper introduces additional coding efficiency in neural speech coding by reducing the temporal redundancy existing in the frame-level feature sequence via a recurrent neural predictor. The prediction can achieve a low-entropy residual representation, which we discriminatively code based on their contribution to the signal reconstruction. The harmonization of feature prediction and discriminative coding results in a dynamic bit allocation algorithm that spends more bits on unpredictable but rare events. As a result, we develop a scalable, lightweight, low-latency, and low-bitrate neural speech coding system. We demonstrate the advantage of the proposed methods using the LPCNet as a neural vocoder. While the proposed method guarantees causality in its prediction, the subjective tests and feature space analysis show that our model achieves superior coding efficiency compared to LPCNet and Lyra V2 in the very low bitrates.
【5】 Sampling Rate Offset Estimation and Compensation for Distributed Adaptive Node-Specific Signal Estimation in Wireless Acoustic Sensor Networks
标题：无线水声传感器网络中分布式自适应特定节点信号估计的采样率偏差估计与补偿
链接：https://arxiv.org/abs/2211.02489
* 与cs.SD语音【14】为同一篇
作者：Paul Didier,Toon van Waterschoot,Simon Doclo,Marc Moonen机构：STADIUS Center for Dynamical Systems, Department of Electrical Engineering (ESAT), KU Leuven, Leuven, Belgium, Signal Processing Group, Department of Medical Physics and Acoustics and Cluster of Excellence Hearing,all摘要：异构无线声学传感器网络（WASN）中设备之间的采样率偏移（SRO）会阻碍分布式自适应算法在依赖于相干信号处理时按预期执行的能力。本文提出了一种SRO估计和补偿方法，以允许分布式自适应节点特定信号估计（DANSE）算法在由异步设备组成的WASN中的部署。首先在基于相干漂移的方法中利用在每个节点处可用的信号来盲估计SRO，然后经由频域中的相移来补偿SRO。引入了DANSE的加权叠加（WOLA）实现的修改，以解决SRO引起的全采样漂移，允许经由作为时域卷积的WOLA过程的近似的每采样信号传输。在异步WASN中对目标语音信号估计的分布式降噪环境中评估了所提出算法的性能。摘要：Sampling rate offsets (SROs) between devices in a heterogeneous wireless acoustic sensor network (WASN) can hinder the ability of distributed adaptive algorithms to perform as intended when they rely on coherent signal processing. In this paper, we present an SRO estimation and compensation method to allow the deployment of the distributed adaptive node-specific signal estimation (DANSE) algorithm in WASNs composed of asynchronous devices. The signals available at each node are first utilised in a coherence-drift-based method to blindly estimate SROs which are then compensated for via phase shifts in the frequency domain. A modification of the weighted overlap-add (WOLA) implementation of DANSE is introduced to account for SRO-induced full-sample drifts, permitting per-sample signal transmission via an approximation of the WOLA process as a time-domain convolution. The performance of the proposed algorithm is evaluated in the context of distributed noise reduction for the estimation of a target speech signal in an asynchronous WASN.
【6】 Spatially Selective Deep Non-linear Filters for Speaker Extraction
标题：用于说话人提取的空间选择性深度非线性滤波器
链接：https://arxiv.org/abs/2211.02420
* 与cs.SD语音【15】为同一篇
作者：Kristina Tesch,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany备注：Submitted to ICASSP 2023摘要：在多人同时讲话的场景中，信号的空间特性是用于提取目标信号的最明显特征。本文提出了一种深度联合空谱非线性滤波器，它可以在任意目标方向上进行控制。为此，提出了一种简单有效的条件化机制，根据目标方向设置滤波器递归层的初始状态.我们表明，该方案比基线方法更有效，并且在不损失性能的情况下增加了滤波器的灵活性。所得到的空间选择性非线性滤波器还可以用于任意数量的说话人的语音分离，并且能够实现非常精确的多说话人定位，如我们在本文中所演示的。摘要：In a scenario with multiple persons talking simultaneously, the spatial characteristics of the signals are the most distinct feature for extracting the target signal. In this work, we develop a deep joint spatial-spectral non-linear filter that can be steered in an arbitrary target direction. For this we propose a simple and effective conditioning mechanism, which sets the initial state of the filter's recurrent layers based on the target direction. We show that this scheme is more effective than the baseline approach and increases the flexibility of the filter at no performance cost. The resulting spatially selective non-linear filters can also be used for speech separation of an arbitrary number of speakers and enable very accurate multi-speaker localization as we demonstrate in this paper.
【7】 Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration
标题：语音恢复中基于扩散的生成方法与区分方法的比较
链接：https://arxiv.org/abs/2211.02397
* 与cs.SD语音【16】为同一篇
作者：Jean-Marie Lemercier,Julius Richter,Simon Welker,Timo Gerkmann机构：⋆Signal Processing (SP), Universit¨at Hamburg, Germany, × Center for Free-Electron Laser Science, DESY, Germany备注：Submitted to ICASSP 2023摘要：近年来，基于扩散的生成模型对计算机视觉和语音处理领域产生了很大的影响。除了数据生成任务之外，它们还被用于数据恢复任务，如语音增强和去混响。虽然传统上认为判别模型例如对于语音增强更强大，但是最近已经显示出生成扩散方法显著地缩小了该性能差距。本文系统地比较了生成扩散模型和判别方法在不同语音恢复任务中的性能。为此，我们将我们先前在复时频域中的基于扩散的语音增强的贡献扩展到带宽扩展的任务。然后将其与具有相同网络结构的区别训练神经网络在语音去噪、去混响和带宽扩展三个恢复任务上进行比较。我们观察到，在所有任务上，生成式方法的整体性能优于其鉴别式对应物，对于非加性失真模型，如在去混响和带宽扩展中，具有最大的益处。代码和音频示例可在www.example.com上在线找到https://uhh.de/inf-sp-sgmsemultitask摘要：Diffusion-based generative models have had a high impact on the computer vision and speech processing communities these past years. Besides data generation tasks, they have also been employed for data restoration tasks like speech enhancement and dereverberation. While discriminative models have traditionally been argued to be more powerful e.g. for speech enhancement, generative diffusion approaches have recently been shown to narrow this performance gap considerably. In this paper, we systematically compare the performance of generative diffusion models and discriminative approaches on different speech restoration tasks. For this, we extend our prior contributions on diffusion-based speech enhancement in the complex time-frequency domain to the task of bandwith extension. We then compare it to a discriminatively trained neural network with the same network architecture on three restoration tasks, namely speech denoising, dereverberation and bandwidth extension. We observe that the generative approach performs globally better than its discriminative counterpart on all tasks, with the strongest benefit for non-additive distortion models, like in dereverberation and bandwidth extension. Code and audio examples can be found online at https://uhh.de/inf-sp-sgmsemultitask
【8】 Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition
标题：用于流端到端语音识别的序列换能器的最小延迟训练
链接：https://arxiv.org/abs/2211.02333
* 与cs.SD语音【17】为同一篇
作者：Yusuke Shinohara,Shinji Watanabe机构：Yahoo Japan Corporation, Japan,Carnegie Mellon University, USA备注：Presented at INTERSPEECH 2022摘要：序列转换器，如RNN-T和Conformer-T，是最有前途的端到端语音识别模型之一，特别是在延迟和准确性都很重要的流场景中。尽管已经研究了各种方法来减少延迟，例如对准限制训练和FastEmit，但是延迟减少通常伴随着准确性的显著降低。我们认为，这种次优性能可能是由于先前的方法没有显式地建模和减少延迟而导致的。本文提出了一种新的训练方法来显式地建模和减少序列传感器模型的延迟。首先，我们定义了格上每条对角线上的期望延迟，并证明了它的梯度可以在前向-后向算法中有效地计算。然后，我们用该预期延迟来增加换能器损耗，从而实现延迟和准确性之间的最佳折衷。在WSJ数据集上的实验结果表明，在WER下降0.7%的情况下，所提出的最小时延训练方法将因果Conformer-T的时延从220 ms降低到27 ms，优于传统的对齐限制训练方法（110 ms）和FastEmit方法（67 ms）.摘要：Sequence transducers, such as the RNN-T and the Conformer-T, are one of the most promising models of end-to-end speech recognition, especially in streaming scenarios where both latency and accuracy are important. Although various methods, such as alignment-restricted training and FastEmit, have been studied to reduce the latency, latency reduction is often accompanied with a significant degradation in accuracy. We argue that this suboptimal performance might be caused because none of the prior methods explicitly model and reduce the latency. In this paper, we propose a new training method to explicitly model and reduce the latency of sequence transducer models. First, we define the expected latency at each diagonal line on the lattice, and show that its gradient can be computed efficiently within the forward-backward algorithm. Then we augment the transducer loss with this expected latency, so that an optimal trade-off between latency and accuracy is achieved. Experimental results on the WSJ dataset show that the proposed minimum latency training reduces the latency of causal Conformer-T from 220 ms to 27 ms within a WER degradation of 0.7%, and outperforms conventional alignment-restricted training (110 ms) and FastEmit (67 ms) methods.
【9】 CochlScene: Acquisition of acoustic scene data using crowdsourcing
标题：CochlScene：使用众包方式获取声音场景数据
链接：https://arxiv.org/abs/2211.02289
* 与cs.SD语音【18】为同一篇
作者：Il-Young Jeong,Jeongsoo Park机构：Cochl, Inc., Seoul, Republic of Korea备注：Accept by APSIPA ASC 2022, 5 pages, 2 figures摘要：本文描述了一种利用众包技术收集声学场景数据的管道。解释了众包的详细过程，包括规划、验证标准和实际的用户界面。作为数据收集的结果，我们提出了一种新的用于声学场景分类的数据集CochlScene。我们的数据集包括从13个声学场景中的831个参与者收集的76k个样本。我们还提出了训练集、验证集和测试集的手动数据分割，以增加评估结果的可靠性。最后，本研究为后续研究提供了一个基本的系统。摘要：This paper describes a pipeline for collecting acoustic scene data by using crowdsourcing. The detailed process of crowdsourcing is explained, including planning, validation criteria, and actual user interfaces. As a result of data collection, we present CochlScene, a novel dataset for acoustic scene classification. Our dataset consists of 76k samples collected from 831 participants in 13 acoustic scenes. We also propose a manual data split of training, validation, and test sets to increase the reliability of the evaluation results. Finally, we provide a baseline system for future research.
【10】 Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects
标题：音乐混合风格迁移：一种解开音效的对比学习方法
链接：https://arxiv.org/abs/2211.02247
* 与cs.SD语音【19】为同一篇
作者：Junghyun Koo,Marco A. Martinez-Ramirez,Wei-Hsiang Liao,Stefan Uhlich,Kyogu Lee,Yuki Mitsufuji机构：Marco A. Mart´ınez-Ram´ırez, Sony Group Corporation, Tokyo, Japan, Sony Europe B.V., Stuttgart, Germany, Music and Audio Research Group, Department of Intelligence and Information, Seoul National University摘要：我们提出了一种端到端的音乐混音风格转换系统，该系统将输入多音轨的混音风格转换为参考歌曲的混音风格。这是利用以对比目标预先训练的编码器来实现的，以从参考音乐录音中仅提取音频效果相关信息。所有模型均采用自监督的方式，利用一种有效的数据预处理方法，从一个已经处理过的湿多轨数据集上训练而成，从而缓解了未处理干数据的数据稀缺性。我们分析了所提出的编码器对音频效果的解纠缠能力，并通过客观和主观评价验证了其混合风格转换的性能。从结果中，我们表明所提出的系统不仅将多声道音频的混合风格转换成接近参考的混合风格，而且在使用音乐源分离模型时对于混合风格转换也是鲁棒的。摘要：We propose an end-to-end music mixing style transfer system that converts the mixing style of an input multitrack to that of a reference song. This is achieved with an encoder pre-trained with a contrastive objective to extract only audio effects related information from a reference music recording. All our models are trained in a self-supervised manner from an already-processed wet multitrack dataset with an effective data preprocessing method that alleviates the data scarcity of obtaining unprocessed dry data. We analyze the proposed encoder for the disentanglement capability of audio effects and also validate its performance for mixing style transfer through both objective and subjective evaluations. From the results, we show the proposed system not only converts the mixing style of multitrack audio close to a reference but is also robust with mixture-wise style transfer upon using a music source separation model.
【11】 Integrated Parameter-Efficient Tuning for General-Purpose Audio Models
标题：通用音频模型的集成参数高效调谐
链接：https://arxiv.org/abs/2211.02227
* 与cs.SD语音【20】为同一篇
作者：Ju-ho Kim,Jungwoo Heo,Hyun-seo Shin,Chan-yeong Lim,Ha-Jin Yu机构：School of Computer Science, University of Seoul备注：5 pages, 3 figures, submit to ICASSP2023摘要：超大规模和通用预训练模型的出现正在改变为目标任务构建特定任务模型的范式。在音频研究领域，任务无关的预训练模型具有很高的可移植性和适应性，通过对下游任务的微调，已经取得了最好的性能。然而，重新训练这些庞大模型的所有参数需要大量的时间和成本，以及巨大的碳足迹。为了克服这些局限性，本研究在音频领域探索并应用了有效的迁移学习方法。我们还提出了一个集成的参数有效调整（IPET）框架，通过聚合嵌入提示（一种基于提示的学习方法）和适配器（一种有效的迁移学习方法）。我们使用具有不同特征的两个骨干预训练音频模型来证明所提出的框架的功效：音频频谱图Transformer和WAV2VEC2.0。与可训练参数较少的微调方法相比，所提出的IPET框架在以下四个下游任务中表现出了显著的性能：声音事件分类、音乐流派分类、关键词识别和说话人验证。此外，作者还指出并分析了IPET框架的不足，为音频领域参数的有效调整提供了借鉴和研究方向。摘要：The advent of hyper-scale and general-purpose pre-trained models is shifting the paradigm of building task-specific models for target tasks. In the field of audio research, task-agnostic pre-trained models with high transferability and adaptability have achieved state-of-the-art performances through fine-tuning for downstream tasks. Nevertheless, re-training all the parameters of these massive models entails an enormous amount of time and cost, along with a huge carbon footprint. To overcome these limitations, the present study explores and applies efficient transfer learning methods in the audio domain. We also propose an integrated parameter-efficient tuning (IPET) framework by aggregating the embedding prompt (a prompt-based learning approach), and the adapter (an effective transfer learning method). We demonstrate the efficacy of the proposed framework using two backbone pre-trained audio models with different characteristics: the audio spectrogram transformer and wav2vec 2.0. The proposed IPET framework exhibits remarkable performance compared to fine-tuning method with fewer trainable parameters in four downstream tasks: sound event classification, music genre classification, keyword spotting, and speaker verification. Furthermore, the authors identify and analyze the shortcomings of the IPET framework, providing lessons and research directions for parameter efficient tuning in the audio domain.
【12】 Streaming Audio-Visual Speech Recognition with Alignment Regularization
标题：基于对齐正则化的流媒体视听语音识别
链接：https://arxiv.org/abs/2211.02133
* 与cs.SD语音【21】为同一篇
作者：Pingchuan Ma,Niko Moritz,Stavros Petridis,Christian Fuegen,Maja Pantic备注：Submitted to ICASSP2023摘要：在说出一个单词后立即对其进行识别是自动语音识别（ASR）系统在真实世界场景中的一个重要要求。因此，在文献中已经提出了关于仅流音频ASR模型的大量工作。然而，流媒体音频视频自动语音识别（AV-ASR）在早期的工作中很少受到关注。本文提出了一种基于CTC/注意力神经网络混合结构的流媒体AV-ASR系统。音频和视频编码器神经网络都基于构象器结构，使用块式自注意（CSA）和因果卷积使构象器结构可流化。采用触发注意技术实现了解码器神经网络的流识别，该技术采用CTC/注意力联合评分进行时间同步解码。对于帧级ASR标准，例如CTC，来自音频和视频编码器的同步响应对于联合AV决策过程是关键的。在这项工作中，我们提出了一种新的对齐正则化技术，该技术促进了音频和视频编码器的同步，这进而导致在所有SNR水平下的流和离线AV-ASR模型的更好的误字率（WER）。该模型在LRS 3（Lip Reading Sentences 3）数据集上的离线和在线测试分别获得了2.0%和2.6%的WER，在不使用外部训练数据的情况下，这两个测试结果都达到了最佳水平.摘要：Recognizing a word shortly after it is spoken is an important requirement for automatic speech recognition (ASR) systems in real-world scenarios. As a result, a large body of work on streaming audio-only ASR models has been presented in the literature. However, streaming audio-visual automatic speech recognition (AV-ASR) has received little attention in earlier works. In this work, we propose a streaming AV-ASR system based on a hybrid connectionist temporal classification (CTC)/attention neural network architecture. The audio and the visual encoder neural networks are both based on the conformer architecture, which is made streamable using chunk-wise self-attention (CSA) and causal convolution. Streaming recognition with a decoder neural network is realized by using the triggered attention technique, which performs time-synchronous decoding with joint CTC/attention scoring. For frame-level ASR criteria, such as CTC, a synchronized response from the audio and visual encoders is critical for a joint AV decision making process. In this work, we propose a novel alignment regularization technique that promotes synchronization of the audio and visual encoder, which in turn results in better word error rates (WERs) at all SNR levels for streaming and offline AV-ASR models. The proposed AV-ASR model achieves WERs of 2.0% and 2.6% on the Lip Reading Sentences 3 (LRS3) dataset in an offline and online setup, respectively, which both present state-of-the-art results when no external training data are used.
【13】 Fearless Steps Challenge Phase-1 Evaluation Plan
标题：无畏步骤挑战阶段-第一阶段评估计划
链接：https://arxiv.org/abs/2211.02051
* 与cs.SD语音【22】为同一篇
作者：Aditya Joglekar,John H. L. Hansen机构：arXiv:,.,v, [eess.AS] , Nov备注：Document Generated in February 2019 for conducting the Fearless Steps Challenge Phase-1 and its associated ISCA Interspeech-2019 Special Session摘要：无畏步伐挑战赛2019第1阶段（FSC-P1）是由达拉斯德克萨斯大学鲁棒语音系统中心（CRSS）主办的无畏步伐倡议的首届挑战赛。本挑战赛的目标是评估最先进的语音和语言系统在具有挑战性的环境中的表现，这些系统适用于具有自然主义音频的大型任务导向型团队。研究人员可以选择参与这些挑战性任务中的任何一项或多项。研究人员也可以选择将Fearless STEPS语料库用于其他相关的语音应用。我们鼓励所有与会者提交他们的解决方案和成果，供ISCA INTERSPEECH-2019特别会议审议。摘要：The Fearless Steps Challenge 2019 Phase-1 (FSC-P1) is the inaugural Challenge of the Fearless Steps Initiative hosted by the Center for Robust Speech Systems (CRSS) at the University of Texas at Dallas. The goal of this Challenge is to evaluate the performance of state-of-the-art speech and language systems for large task-oriented teams with naturalistic audio in challenging environments. Researchers may select to participate in any single or multiple of these challenge tasks. Researchers may also choose to employ the FEARLESS STEPS corpus for other related speech applications. All participants are encouraged to submit their solutions and results for consideration in the ISCA INTERSPEECH-2019 special session.
【14】 Biased Self-supervised learning for ASR
标题：ASR中的有偏自监督学习
链接：https://arxiv.org/abs/2211.02536
* 与cs.SD语音【1】为同一篇
作者：Florian L. Kreyssig,Yangyang Shi,Jinxi Guo,Leda Sari,Abdelrahman Mohamed,Philip C. Woodland机构：University of Cambridge, UK,Meta AI, USA备注：Submitted to ICASSP 2023摘要：通过掩蔽预测预训练（MPPT）的自监督学习在一系列语音处理任务中表现出了令人印象深刻的性能。本文提出了一种将自监督学习偏向于特定任务的方法。其核心思想是对用于获得目标序列的模型进行微调。这会带来更好的性能和训练速度的大幅提高。此外，本文还提出了一种MPPT的变体，通过计算屏蔽帧和非屏蔽帧的MPPT损失，可以有效地训练低占用量的流模型。在Librispeech语料库上对这些方法进行了自动语音识别评估，其中100小时的数据用作已标记数据，860小时的数据用作未标记数据。在测试-其他上，有偏训练在250 k更新之后优于无偏训练15.5%，在100 k更新之后优于无偏训练23.8%。对于流模型，预训练方法产生了44.1%的字错误率的减少。摘要：Self-supervised learning via masked prediction pre-training (MPPT) has shown impressive performance on a range of speech-processing tasks. This paper proposes a method to bias self-supervised learning towards a specific task. The core idea is to slightly finetune the model that is used to obtain the target sequence. This leads to better performance and a substantial increase in training speed. Furthermore, this paper proposes a variant of MPPT that allows low-footprint streaming models to be trained effectively by computing the MPPT loss on masked and unmasked frames. These approaches are evaluated for automatic speech recognition on the Librispeech corpus, where 100 hours of data served as the labelled data and 860 hours as the unlabelled data. The biased training outperforms the unbiased training by 15.5% after 250k updates and 23.8% after 100k updates on test-other. For the streaming models, the pre-training approach yields a reduction in word error rate of 44.1%.
【15】 A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability
标题：一种具有真正零射能力的弱监督流媒体多语言语音模型
链接：https://arxiv.org/abs/2211.02499
* 与cs.SD语音【2】为同一篇
作者：Jian Xue,Peidong Wang,Jinyu Li,Eric Sun机构：Microsoft Speech Group, Redmond, WA, USA备注：submitted to ICASSP 2023摘要：本文介绍了我们建立一个流式多语言语音模型（SM 2）的工作，该模型可以将多种口语转换成目标语言的文本。SM 2的主干是Transformer传感器，它具有很高的流传输能力。代替人工标记的语音翻译（ST）数据，SM 2模型使用通过用机器翻译服务转换语音识别语料库中的转录而生成的弱监督数据来训练。基于25种语言的35.1万小时的匿名语音训练数据，SM 2模型的ST质量与一些最近流行的大规模非流语音模型相当，甚至更好。更重要的是，我们表明SM 2在扩展到新的目标语言时具有真正的zero-shot能力，为训练期间未看到的{源-语音，目标-文本}对产生高质量的ST结果。摘要：In this paper, we introduce our work of building a Streaming Multilingual Speech Model (SM2), which can transcribe or translate multiple spoken languages into texts of the target language. The backbone of SM2 is Transformer Transducer, which has high streaming capability. Instead of human labeled speech translation (ST) data, SM2 models are trained using weakly supervised data generated by converting the transcriptions in speech recognition corpora with a machine translation service. With 351 thousand hours of anonymized speech training data from 25 languages, SM2 models achieve comparable or even better ST quality than some recent popular large-scale non-streaming speech models. More importantly, we show that SM2 has the truly zero-shot capability when expanding to new target languages, yielding high quality ST results for {source-speech, target-text} pairs that are not seen during training.
【16】 NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS
标题：NoreSpeech：基于知识提取的抗噪表达TTS条件扩散模型
链接：https://arxiv.org/abs/2211.02448
* 与cs.SD语音【3】为同一篇
作者：Dongchao Yang,Songxiang Liu,Jianwei Yu,Helin Wang,Chao Weng,Yuexian Zou机构：ADSPLAB, School of ECE, Peking University, China, Tencent AI Lab, Shenzhen, China, Johns Hopkins University, USA备注：Submitted to ICASSP2023摘要：表现型文本-语音转换（TTS）技术可以通过模仿参考音频的韵律和音色来合成新的说话风格，但它面临着以下挑战：（1）难以提取参考音频中的高动态韵律信息，特别是当参考音频包含背景噪声时。(2)TTS系统应该对看不见的说话风格有很好的概括。本文提出了一种鲁棒的可表达的TTS模型（NoreSpeech），该模型能够鲁棒地将带噪参考话语中的说话风格转换为合成语音.具体来说，我们的NoreSpeech包括几个组件：（1）新颖的DiffStyle模块，其利用强大的概率去噪扩散模型来通过知识提炼从教师模型学习与噪声无关的说话风格特征;（2）VQ-VAE块，将风格特征映射到一个可控的量化潜在空间，以提高风格迁移的泛化性;以及（3）直接但有效的无参数文本风格对齐模块，其使得NoreSpeech能够将风格从长度不匹配的参考话语转移到文本输入。实验结果表明，在噪声环境下，NoreSpeech比已有的表现型TTS模型具有更好的效果。音频样本和代码可从以下网址获得：第二章：关于汉语语音教学的几点思考摘要：Expressive text-to-speech (TTS) can synthesize a new speaking style by imiating prosody and timbre from a reference audio, which faces the following challenges: (1) The highly dynamic prosody information in the reference audio is difficult to extract, especially, when the reference audio contains background noise. (2) The TTS systems should have good generalization for unseen speaking styles. In this paper, we present a \textbf{no}ise-\textbf{r}obust \textbf{e}xpressive TTS model (NoreSpeech), which can robustly transfer speaking style in a noisy reference utterance to synthesized speech. Specifically, our NoreSpeech includes several components: (1) a novel DiffStyle module, which leverages powerful probabilistic denoising diffusion models to learn noise-agnostic speaking style features from a teacher model by knowledge distillation; (2) a VQ-VAE block, which maps the style features into a controllable quantized latent space for improving the generalization of style transfer; and (3) a straight-forward but effective parameter-free text-style alignment module, which enables NoreSpeech to transfer style to a textual input from a length-mismatched reference utterance. Experiments demonstrate that NoreSpeech is more effective than previous expressive TTS models in noise environments. Audio samples and code are available at: \href{http://dongchaoyang.top/NoreSpeech\_demo/}{http://dongchaoyang.top/NoreSpeech\_demo/}
【17】 SPEAKER VGG CCT: Cross-corpus Speech Emotion Recognition with Speaker Embedding and Vision Transformers
标题：说话人VGG CCT：基于说话人嵌入和视觉变换的跨语料语音情感识别
链接：https://arxiv.org/abs/2211.02366
* 与cs.SD语音【4】为同一篇
机构：Cross-corpus Speech Emotion Recognitionwith Speaker Embedding and Vision TransformersAlessandro ArezzoUniversity of FlorenceFlorence, comStefano BerrettiUniversity of FlorenceFlorence摘要：近年来，语音情感识别（Speech Emotion Recognition，SER）的研究主要是将语音信号转换为语谱图，然后利用在普通图像上预先训练并利用语谱图进行微调的卷积神经网络对语谱图进行分类。本文从上述基本思想出发，提出了一种基于紧凑卷积Transformers（CCT）和说话人嵌入相结合的SER学习方法。有了CCT，Vision Transformers（ViT）的学习能力与卷积所带来的对大量数据的需求减少相结合。这在SER中很重要，因为在SER中通常无法获得大量的数据。说话人嵌入允许网络提取说话人的身份表示，然后通过自注意机制将其与CCT从声谱图提取的特征相结合。总的来说，该解决方案能够在跨语料库场景中实时操作，显示有希望的结果，其中训练和测试数据集保持分离。在文献中很少使用的跨语料库环境中对几个基准进行了实验，结果与最先进的网络体系结构获得的结果相当或更好。您可以在www.example.com上找到我们的代码https://github.com/JabuMlDev/Speaker-VGG-CCT。摘要：In recent years, Speech Emotion Recognition (SER) has been investigated mainly transforming the speech signal into spectrograms that are then classified using Convolutional Neural Networks pretrained on generic images and fine tuned with spectrograms. In this paper, we start from the general idea above and develop a new learning solution for SER, which is based on Compact Convolutional Transformers (CCTs) combined with a speaker embedding. With CCTs, the learning power of Vision Transformers (ViT) is combined with a diminished need for large volume of data as made possible by the convolution. This is important in SER, where large corpora of data are usually not available. The speaker embedding allows the network to extract an identity representation of the speaker, which is then integrated by means of a self-attention mechanism with the features that the CCT extracts from the spectrogram. Overall, the solution is capable of operating in real-time showing promising results in a cross-corpus scenario, where training and test datasets are kept separate. Experiments have been performed on several benchmarks in a cross-corpus setting as rarely used in the literature, with results that are comparable or superior to those obtained with state-of-the-art network architectures. Our code is available at https://github.com/JabuMlDev/Speaker-VGG-CCT.
【18】 Improving Speech Prosody of Audiobook Text-to-Speech Synthesis with Acoustic and Textual Contexts
标题：利用声学和文本语境改进有声读物文语转换合成的语音韵律
链接：https://arxiv.org/abs/2211.02336
* 与cs.SD语音【5】为同一篇
作者：Detai Xin,Sharath Adavanne,Federico Ang,Ashish Kulkarni,Shinnosuke Takamichi,Hiroshi Saruwatari机构：Rakuten Group, Inc., Japan, Graduate School of Information Science and Technology, The University of Tokyo, Japan摘要：本文提出了一个多说话人日语有声读物文本到语音（TTS）系统，该系统充分利用先前声学上下文和双边文本上下文的多模态上下文信息来改善合成语音的韵律。以往的研究要么使用单模态的上下文信息，要么使用单模态的上下文信息，不能充分反映上下文信息。该方法利用声学上下文编码器和文本上下文编码器来聚合上下文信息，并将其反馈给TTS模型，使其能够预测上下文相关的韵律。我们对一个多人日语有声读物数据集进行了客观和主观的综合评价。实验结果表明，该方法的性能明显优于已有的两种方法。此外，我们提出的见解，不同的选择上下文-模态，横向信息和长度-有声读物TTS，从来没有讨论过的文献之前。摘要：We present a multi-speaker Japanese audiobook text-to-speech (TTS) system that leverages multimodal context information of preceding acoustic context and bilateral textual context to improve the prosody of synthetic speech. Previous work either uses unilateral or single-modality context, which does not fully represent the context information. The proposed method uses an acoustic context encoder and a textual context encoder to aggregate context information and feeds it to the TTS model, which enables the model to predict context-dependent prosody. We conducted comprehensive objective and subjective evaluations on a multi-speaker Japanese audiobook dataset. Experimental results demonstrate that the proposed method significantly outperforms two previous works. Additionally, we present insights about the different choices of context - modalities, lateral information and length - for audiobook TTS that have never been discussed in the literature before.
【19】 Once-for-All Sequence Compression for Self-Supervised Speech Models
标题：用于自监督语音模型的一次性序列压缩
链接：https://arxiv.org/abs/2211.02332
* 与cs.SD语音【6】为同一篇
作者：Hsuan-Jui Chen,Yen Meng,Hung-yi Lee机构：Nation Taiwan University备注：Submitted to ICASSP 2023摘要：沿时间轴的序列长度通常是自监督语音模型的计算成本的主要因素。已经提出了减少序列长度以降低计算成本的工作。然而，不同的下游任务具有不同的序列压缩容忍度，因此产生固定压缩率的模型可能不适合所有的任务。在本研究中，我们提出一种适用于自我监督语音模型的一次所有（once-for-all，OFA）序列压缩架构，其支持连续范围的压缩率。该框架在各种任务上进行了评估，与固定压缩率变体相比，显示出边际退化，具有平滑的性能-效率权衡。我们进一步探索了自适应压缩率学习，展示了在不需要网格搜索的情况下选择特定任务的优选帧周期的能力。摘要：The sequence length along the time axis is often the dominant factor of the computational cost of self-supervised speech models. Works have been proposed to reduce the sequence length for lowering the computational cost. However, different downstream tasks have different tolerance of sequence compressing, so a model that produces a fixed compressing rate may not fit all tasks. In this work, we introduce a once-for-all (OFA) sequence compression framework for self-supervised speech models that supports a continuous range of compressing rates. The framework is evaluated on various tasks, showing marginal degradation compared to the fixed compressing rate variants with a smooth performance-efficiency trade-off. We further explore adaptive compressing rate learning, demonstrating the ability to select task-specific preferred frame periods without needing a grid search.
【20】 Binaural Rendering of Ambisonic Signals by Neural Networks
标题：基于神经网络的两性声信号双耳再现
链接：https://arxiv.org/abs/2211.02301
* 与cs.SD语音【7】为同一篇
作者：Yin Zhu,Qiuqiang Kong,Junjie Shi,Shilei Liu,Xuzhou Ye,Ju-chiang Wang,Junping Zhang机构：Department of Computer Science, Fudan University, Audio & Music Intelligence (SAMI), ByteDance Inc.摘要：环绕声信号的双耳再现在虚拟现实和沉浸式媒体中具有广泛的兴趣。传统方法通常需要手动测量头部相关传递函数（HRTF）。为了解决这一问题，我们收集了一个成对的双声道立体声数据集，并提出了一个端到端的深度学习框架。实验结果表明，神经网络在客观评价指标上优于传统方法，在主观评价指标上与传统方法具有可比性。为了验证所提出的框架，我们实验性地探索了输入特征、模型结构、输出特征和损失函数的不同设置。我们提出的系统在质量、音色、定位和沉浸维度上实现了7.32的SDR和3.83、3.58、3.87、3.58的MOS。摘要：Binaural rendering of ambisonic signals is of broad interest to virtual reality and immersive media. Conventional methods often require manually measured Head-Related Transfer Functions (HRTFs). To address this issue, we collect a paired ambisonic-binaural dataset and propose a deep learning framework in an end-to-end manner. Experimental results show that neural networks outperform the conventional method in objective metrics and achieve comparable subjective metrics. To validate the proposed framework, we experimentally explore different settings of the input features, model structures, output features, and loss functions. Our proposed system achieves an SDR of 7.32 and MOSs of 3.83, 3.58, 3.87, 3.58 in quality, timbre, localization, and immersion dimensions.
【21】 Wireless Deep Speech Semantic Transmission
标题：无线深度语音语义传输
链接：https://arxiv.org/abs/2211.02283
* 与cs.SD语音【8】为同一篇
作者：Zixuan Xiao,Shengshi Yao,Jincheng Dai,Sixian Wang,Kai Niu,Ping Zhang机构：⋆ Beijing University of Posts and Telecommunications, Beijing, China, † Peng Cheng Laboratory, Shenzhen, China摘要：本文提出了一种新的高效的语义编码传输方法，用于无线信道上的端到端语音传输。我们称整个系统为深层语音语义传输（DSST）。具体地说，我们引入了一个非线性变换将语音源映射到语义潜在空间，并将语义特征反馈到源-通道编码器以生成通道-输入序列。在变分建模思想的指导下，在隐空间上建立熵模型来估计语义特征嵌入之间的重要性差异。相应地，这些不同重要性的语义特征可以被合理地分配不同的编码速率，从而使系统编码增益最大化。此外，我们引入了信道信噪比（SNR）自适应机制，使得单个模型可以应用于各种信道状态。通过对模型进行端到端的优化，实现了灵活的率失真（RD）折衷，支持多种无线语音语义传输。实验结果表明，DSST系统在主观和客观两个方面都明显优于现有的工程语音传输系统。与现有的神经语音语义传输方法相比，在达到相同质量的情况下，该模型最多可节省75%的信道带宽开销。您可以在www.example.com上找到音频演示的直观比较https://ximoo123.github.io/DSST。摘要：In this paper, we propose a new class of high-efficiency semantic coded transmission methods for end-to-end speech transmission over wireless channels. We name the whole system as deep speech semantic transmission (DSST). Specifically, we introduce a nonlinear transform to map the speech source to semantic latent space and feed semantic features into source-channel encoder to generate the channel-input sequence. Guided by the variational modeling idea, we build an entropy model on the latent space to estimate the importance diversity among semantic feature embeddings. Accordingly, these semantic features of different importance can be allocated with different coding rates reasonably, which maximizes the system coding gain. Furthermore, we introduce a channel signal-to-noise ratio (SNR) adaptation mechanism such that a single model can be applied over various channel states. The end-to-end optimization of our model leads to a flexible rate-distortion (RD) trade-off, supporting versatile wireless speech semantic transmission. Experimental results verify that our DSST system clearly outperforms current engineered speech transmission systems on both objective and subjective metrics. Compared with existing neural speech semantic transmission methods, our model saves up to 75% of channel bandwidth costs when achieving the same quality. An intuitive comparison of audio demos can be found at https://ximoo123.github.io/DSST.
【22】 Real-Time Target Sound Extraction
标题：目标声音的实时提取
链接：https://arxiv.org/abs/2211.02250
* 与cs.SD语音【9】为同一篇
作者：Bandhav Veluri,Justin Chan,Malek Itani,Tuochao Chen,Takuya Yoshioka,Shyamnath Gollakota机构：⋄Paul G. Allen School of Computer Science & Engineering, University of Washington, USA, •Microsoft, One Microsoft Way, Redmond, WA, USA摘要：我们提出了第一个神经网络模型，以实现实时和流式目标声音提取。为了实现这一点，我们提出了Waveformer，一种编码器-解码器架构，其具有作为编码器的膨胀因果卷积层的堆叠，以及作为解码器的Transformer解码器层。该混合架构使用扩张因果卷积，用于以计算上有效的方式处理大的感受野，同时还受益于基于变换器的架构所提供的性能。我们的评估显示，与之前的模型相比，针对此任务，SI-SNRi提高了2.2-3.3 dB，同时模型大小缩小了1.2- 4倍，运行时间缩短了1.5- 2倍。开放源代码和数据集：https://github.com/vb000/Waveformer摘要：We present the first neural network model to achieve real-time and streaming target sound extraction. To accomplish this, we propose Waveformer, an encoder-decoder architecture with a stack of dilated causal convolution layers as the encoder, and a transformer decoder layer as the decoder. This hybrid architecture uses dilated causal convolutions for processing large receptive fields in a computationally efficient manner, while also benefiting from the performance transformer-based architectures provide. Our evaluations show as much as 2.2-3.3 dB improvement in SI-SNRi compared to the prior models for this task while having a 1.2-4x smaller model size and a 1.5-2x lower runtime. Open-source code and datasets: https://github.com/vb000/Waveformer
机器翻译，仅供参考
每日语音速递