今日论文合集：cs.SD语音9篇，eess.AS音频处理9篇。
cs.SD语音

【1】 ProsAudit, a prosodic benchmark for self-supervised speech models

标题：ProsAudit，一个用于自我监督语音模型的韵律基准

链接：https://arxiv.org/abs/2302.12057

作者：Maureen de Seyssel,Marvin Lavechin,Hadrien Titeux,Arthur Thomas,Gwendal Virlet,Andrea Santos Revilla,Guillaume Wisniewski,Bogdan Ludusan,Emmanuel Dupoux
机构：Cognitive Machine Learning (ENS–CNRS–EHESS–INRIA–PSL Research University), France,  Universit´e de Paris Cit´e, CNRS, Laboratoire de linguistique formelle, F-, Paris, France, Bielefeld University, Germany,  PEGASE, INRAE, Institut Agro, Saint-Gilles, France
备注：4 pages + references, 1 figure
摘要：本文介绍了一个用于评估自监督学习（SSL）语音模型中结构韵律知识的英语基准ProsAudit。它由两个子任务、相应的指标和一个评估数据集组成。在原语法任务中，模型必须正确识别强韵律边界和弱韵律边界。在词汇任务中，模型需要正确区分插入在单词之间和单词内的停顿。我们还提供了该基准的人体评估分数。我们评估了一系列SSL模型，发现它们都能够在两个任务上执行高于概率的任务，即使是在一种看不见的语言上进行训练。然而，非母语模型在词汇任务上的表现显著低于母语模型，这凸显了词汇知识在词汇任务中的重要性。我们还发现了规模的明显影响，在两个子任务中，使用更多数据训练的模型表现更好。
摘要：We present ProsAudit, a benchmark in English to assess structural prosodic knowledge in self-supervised learning (SSL) speech models. It consists of two subtasks, their corresponding metrics, an evaluation dataset. In the protosyntax task, the model must correctly identify strong versus weak prosodic boundaries. In the lexical task, the model needs to correctly distinguish between pauses inserted between words and within words. We also provide human evaluation scores on this benchmark. We evaluated a series of SSL models and found that they were all able to perform above chance on both tasks, even when trained on an unseen language. However, non-native models performed significantly worse than native ones on the lexical task, highlighting the importance of lexical knowledge in this task. We also found a clear effect of size with models trained on more data performing better in the two subtasks.

【2】 Evaluating Automatic Speech Recognition in an Incremental Setting

标题：在增量设置中评估自动语音识别

链接：https://arxiv.org/abs/2302.12049

作者：Ryan Whetten,Mir Tahsin Imtiaz,Casey Kennington
机构：Boise State University, Department of Computer Science,  W University Dr., Boise, ID
备注：5 pages
摘要：自动语音识别的可靠性不断提高，使其日常使用激增。然而，出于研究目的，通常不清楚应该为任务选择哪种模型，特别是如果既需要速度又需要精度。在这篇文章中，我们使用包括单词错误率、延迟和对英语测试数据中已识别单词的更新次数在内的指标系统地评估了六个语音识别器，并提出和比较了两种将音频流传输到识别器进行增量识别的方法。我们进一步提出每秒撤销数作为评估增量识别的新指标，并证明它提供了对整体模型性能的见解。我们发现，一般来说，本地识别器比基于云的识别器更快，需要更少的更新。最后，我们发现Meta的Wav2Vec模型是最快的，而Mozilla的DeepSpeech模型在预测方面是最稳定的。
摘要：The increasing reliability of automatic speech recognition has proliferated its everyday use. However, for research purposes, it is often unclear which model one should choose for a task, particularly if there is a requirement for speed as well as accuracy. In this paper, we systematically evaluate six speech recognizers using metrics including word error rate, latency, and the number of updates to already recognized words on English test data, as well as propose and compare two methods for streaming audio into recognizers for incremental recognition. We further propose Revokes per Second as a new metric for evaluating incremental recognition and demonstrate that it provides insights into overall model performance. We find that, generally, local recognizers are faster and require fewer updates than cloud-based recognizers. Finally, we find Meta's Wav2Vec model to be the fastest, and find Mozilla's DeepSpeech model to be the most stable in its predictions.

【3】 Metric-oriented Speech Enhancement using Diffusion Probabilistic Model

标题：基于扩散概率模型的面向度量的语音增强

链接：https://arxiv.org/abs/2302.11989

作者：Chen Chen,Yuchen Hu,Weiwei Weng,Eng Siong Chng
机构：School of Computer Science and Engineering, Nanyang Technological University, Singapore
备注：Accepted by ICASSP2023
摘要：基于深度神经网络的语音增强技术的核心是在成对训练数据的监督下学习噪声到干净的转换。然而，任务专用评估度量（例如，PESQ）通常是不可微的，不能直接构造在训练准则中。训练目标和评估度量之间的这种不匹配可能导致次优性能。为了解决这个问题，我们提出了一种面向度量的语音增强方法（MOSE），该方法利用了扩散概率模型的最新进展，并将面向度量的训练策略集成到其逆过程中。具体而言，我们设计了一个基于行动者-批评者的框架，将评价指标视为一种后验回报，从而引导逆向过程向指标递增的方向发展。实验结果表明，MOSE明显受益于指标导向训练，在所有评价指标上均优于生成性基线。
摘要：Deep neural network based speech enhancement technique focuses on learning a noisy-to-clean transformation supervised by paired training data. However, the task-specific evaluation metric (e.g., PESQ) is usually non-differentiable and can not be directly constructed in the training criteria. This mismatch between the training objective and evaluation metric likely results in sub-optimal performance. To alleviate it, we propose a metric-oriented speech enhancement method (MOSE), which leverages the recent advances in the diffusion probabilistic model and integrates a metric-oriented training strategy into its reverse process. Specifically, we design an actor-critic based framework that considers the evaluation metric as a posterior reward, thus guiding the reverse process to the metric-increasing direction. The experimental results demonstrate that MOSE obviously benefits from metric-oriented training and surpasses the generative baselines in terms of all evaluation metrics.

【4】 Unsupervised Noise adaptation using Data Simulation

标题：基于数据模拟的无监督噪声自适应

链接：https://arxiv.org/abs/2302.11981

作者：Chen Chen,Yuchen Hu,Heqing Zou,Linhui Sun,Eng Siong Chng
机构：School of Computer Science and Engineering, Nanyang Technological University, Singapore,  Nanjing University of Posts and Telecommunications, China
备注：Accepted by ICASSP2023
摘要：基于深度神经网络的语音增强方法旨在使用监督学习范例来学习噪声到干净的变换。然而，这种训练井变换易受训练集中未包含的看不见的噪声的影响。在这项工作中，我们专注于语音增强中的无监督噪声自适应问题，其中目标域数据的真实数据是完全不可用的。具体地，我们提出了一种基于生成对抗网络的方法，以使用几分钟的不成对目标域数据来有效地学习逆清洁到噪声变换。然后，利用该变换来生成用于增强模型的域自适应的足够的模拟数据。实验结果表明，该方法有效地抑制了训练集与测试集之间的领域失配，并大幅优于最佳基线。
摘要：Deep neural network based speech enhancement approaches aim to learn a noisy-to-clean transformation using a supervised learning paradigm. However, such a trained-well transformation is vulnerable to unseen noises that are not included in training set. In this work, we focus on the unsupervised noise adaptation problem in speech enhancement, where the ground truth of target domain data is completely unavailable. Specifically, we propose a generative adversarial network based method to efficiently learn a converse clean-to-noisy transformation using a few minutes of unpaired target domain data. Then this transformation is utilized to generate sufficient simulated data for domain adaptation of the enhancement model. Experimental results show that our method effectively mitigates the domain mismatch between training and test sets, and surpasses the best baseline by a large margin.

【5】 D2Former: A Fully Complex Dual-Path Dual-Decoder Conformer Network using  Joint Complex Masking and Complex Spectral Mapping for Monaural Speech  Enhancement

标题：D2Former：一种用于单声道语音增强的联合复数掩蔽和复谱映射的全复数双路双解码器同构网络

链接：https://arxiv.org/abs/2302.11832

作者：Shengkui Zhao,Bin Ma
机构：Alibaba Group
备注：5 pages, 3 figures, accepted by ICASSP 2023
摘要：单耳语音增强已经在时频（TF）域中使用真实网络进行了广泛的研究。然而，输入和目标在TF域中自然是复值的，完全复杂的网络对于有效地学习特征表示和在复杂域中对序列建模是非常期望的。此外，相位，语音感知质量的重要因素，已经被证明可以使用复掩蔽或复谱映射从含噪语音中与幅度一起学习。许多最近的研究集中于复掩蔽或复频谱映射，忽略了它们的性能边界。针对上述问题，提出了一种联合复掩蔽和复谱映射的全复双路双解码器构象器网络（D2Former），用于单耳语音增强。在D2Former中，我们将构象网络扩展到复数域，形成了一个双路径复数TF自注意结构，以有效地对复值TF序列进行建模。通过利用时间依赖性上的复杂扩张卷积和用于频率递归的复杂前馈顺序记忆网络（CFSMN），我们使用双路径学习结构进一步增强编码器和解码器中的TF特征表示。此外，我们通过将复掩蔽和复谱映射两种训练目标的优势结合到一个联合学习框架中，改善了这两种训练目标的性能边界。因此，D2Former充分利用了复值运算、双路径处理和联合训练目标。与之前的模型相比，D2Former在VoiceBank+Demand基准测试中以最小的0.87M参数模型规模实现了最先进的结果。
摘要：Monaural speech enhancement has been widely studied using real networks in the time-frequency (TF) domain. However, the input and the target are naturally complex-valued in the TF domain, a fully complex network is highly desirable for effectively learning the feature representation and modelling the sequence in the complex domain. Moreover, phase, an important factor for perceptual quality of speech, has been proved learnable together with magnitude from noisy speech using complex masking or complex spectral mapping. Many recent studies focus on either complex masking or complex spectral mapping, ignoring their performance boundaries. To address above issues, we propose a fully complex dual-path dual-decoder conformer network (D2Former) using joint complex masking and complex spectral mapping for monaural speech enhancement. In D2Former, we extend the conformer network into the complex domain and form a dual-path complex TF self-attention architecture for effectively modelling the complex-valued TF sequence. We further boost the TF feature representation in the encoder and the decoders using a dual-path learning structure by exploiting complex dilated convolutions on time dependency and complex feedforward sequential memory networks (CFSMN) for frequency recurrence. In addition, we improve the performance boundaries of complex masking and complex spectral mapping by combining the strengths of the two training targets into a joint-learning framework. As a consequence, D2Former takes fully advantages of the complex-valued operations, the dual-path processing, and the joint-training targets. Compared to the previous models, D2Former achieves state-of-the-art results on the VoiceBank+Demand benchmark with the smallest model size of 0.87M parameters.

【6】 MossFormer: Pushing the Performance Limit of Monaural Speech Separation  using Gated Single-Head Transformer with Convolution-Augmented Joint  Self-Attentions

标题：MossFormer：利用带卷积增强联合注意的门控单头Transformer突破单声道语音分离的性能极限

链接：https://arxiv.org/abs/2302.11824

作者：Shengkui Zhao,Bin Ma
机构：Alibaba Group
备注：5 pages, 3 figures, accepted by ICASSP 2023
摘要：基于变换器的模型在单声道语音分离中提供了显著的性能改进。然而，与最近提出的上限相比，仍然存在性能差距。目前双路径变压器模型的主要局限性是对长程元素相互作用和局部特征模式的建模效率低。在本文中，我们提出了一种具有卷积增强联合自注意的门控单头变换器结构，命名为\textit {MossFormer}（\textit {Mo} naural\textit {s} peech\textit {s} eparation Transs\textit {Former}），从而得到了这个上界.为了有效解决双路径结构中组块间的间接元素交互，MossFormer采用了联合局部和全局自注意结构，该结构同时对局部组块执行全计算自注意和对全序列执行线性化低成本自注意。联合注意力直接实现了MossFormer模型全序列元素交互作用。此外，我们使用了一个强大的注意门控机制与简化的单头自我注意。除了关注的远程建模外，我们还使用卷积来增强MossFormer，以进行位置方式的局部模式建模。因此，MossFormer的性能明显优于之前的型号，并在WSJ0 - 2/3mix和WHAM上实现了最先进的结果!/哇!基准。我们的模型在WSJ0 - 3mix上实现了21.2 dB的SI-SDRi上限，在WSJ0 - 2mix上仅比上限23.1 dB低0.3 dB。
摘要：Transformer based models have provided significant performance improvements in monaural speech separation. However, there is still a performance gap compared to a recent proposed upper bound. The major limitation of the current dual-path Transformer models is the inefficient modelling of long-range elemental interactions and local feature patterns. In this work, we achieve the upper bound by proposing a gated single-head transformer architecture with convolution-augmented joint self-attentions, named \textit{MossFormer} (\textit{Mo}naural \textit{s}peech \textit{s}eparation Trans\textit{Former}). To effectively solve the indirect elemental interactions across chunks in the dual-path architecture, MossFormer employs a joint local and global self-attention architecture that simultaneously performs a full-computation self-attention on local chunks and a linearised low-cost self-attention over the full sequence. The joint attention enables MossFormer model full-sequence elemental interaction directly. In addition, we employ a powerful attentive gating mechanism with simplified single-head self-attentions. Besides the attentive long-range modelling, we also augment MossFormer with convolutions for the position-wise local pattern modelling. As a consequence, MossFormer significantly outperforms the previous models and achieves the state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR! benchmarks. Our model achieves the SI-SDRi upper bound of 21.2 dB on WSJ0-3mix and only 0.3 dB below the upper bound of 23.1 dB on WSJ0-2mix.

【7】 Frequency bin-wise single channel speech presence probability estimation  using multiple DNNs

标题：基于多DNN的频域单通道语音存在概率估计

链接：https://arxiv.org/abs/2302.12048

作者：Shuai Tao,Himavanth Reddy,Jesper Rindom Jensen,Mads Græsbøll Christensen
机构：Audio Analysis Lab, CREATE, Aalborg University, Aalborg, Denmark
备注：Accepted for ICASSP 2023
摘要：在这项工作中，我们提出了一个频率区间的方法来估计单通道语音存在概率（SPP）与多个深度神经网络（DNNs）在短时傅里叶变换域。由于所有频率仓通常被同时视为传统的基于DNN的SPP估计器的输入特征，因此高模型复杂度是不可避免的。为了降低模型复杂度和对训练数据的要求，我们考虑了单个频率单元和它的一些相邻频率单元来训练单独的门递归单元。此外，我们还使用含噪语音和后验概率SPP表示来训练我们的模型。在深度噪声抑制挑战数据集上执行实验。实验结果表明，采用频率区间模型可以提高语音检测的准确率。最后，我们也证明了我们提出的方法在语音检测精度和模型复杂度方面优于大多数最先进的SPP估计方法。
摘要：In this work, we propose a frequency bin-wise method to estimate the single-channel speech presence probability (SPP) with multiple deep neural networks (DNNs) in the short-time Fourier transform domain. Since all frequency bins are typically considered simultaneously as input features for conventional DNN-based SPP estimators, high model complexity is inevitable. To reduce the model complexity and the requirements on the training data, we take a single frequency bin and some of its neighboring frequency bins into account to train separate gate recurrent units. In addition, the noisy speech and the a posteriori probability SPP representation are used to train our model. The experiments were performed on the Deep Noise Suppression challenge dataset. The experimental results show that the speech detection accuracy can be improved when we employ the frequency bin-wise model. Finally, we also demonstrate that our proposed method outperforms most of the state-of-the-art SPP estimation methods in terms of speech detection accuracy and model complexity.

【8】 A Framework for Unified Real-time Personalized and Non-Personalized  Speech Enhancement

标题：一种统一的实时个性化和非个性化语音增强框架

链接：https://arxiv.org/abs/2302.11768

作者：Zhepei Wang,Ritwik Giri,Devansh Shah,Jean-Marc Valin,Michael M. Goodwin,Paris Smaragdis
机构：†Amazon Web Services, ♯University of Illinois at Urbana-Champaign
备注：Accepted by ICASSP 2023
摘要：在本研究中，我们提出一种训练单一语音增强网路的方法，此网路可以执行个人化与非个人化语音增强。这是通过合并指定增强输出类型的逐帧调节输入来实现的。为了提高增强输出的质量并减轻过抑制，我们实验了通过语音活动的存在或不存在来重新加权帧，并将增强应用于说话人嵌入。通过在多任务学习设置下训练，我们经验性地表明，所提出的统一模型在个性化和非个性化语音增强基准上都获得了有希望的结果，并且达到了与专门针对任一任务训练的模型相似的性能。所提方法的强大性能表明，与在推理过程中保持单独的任务特定模型相比，统一模型是一种更经济的选择。
摘要：In this study, we present an approach to train a single speech enhancement network that can perform both personalized and non-personalized speech enhancement. This is achieved by incorporating a frame-wise conditioning input that specifies the type of enhancement output. To improve the quality of the enhanced output and mitigate oversuppression, we experiment with re-weighting frames by the presence or absence of speech activity and applying augmentations to speaker embeddings. By training under a multi-task learning setting, we empirically show that the proposed unified model obtains promising results on both personalized and non-personalized speech enhancement benchmarks and reaches similar performance to models that are trained specialized for either task. The strong performance of the proposed method demonstrates that the unified model is a more economical alternative compared to keeping separate task-specific models during inference.

【9】 Incorporating Uncertainty from Speaker Embedding Estimation to Speaker  Verification

标题：将说话人嵌入估计的不确定性引入说话人确认

链接：https://arxiv.org/abs/2302.11763

作者：Qiongqiong Wang,Kong Aik Lee,Tianchi Liu
机构：Institute for Infocomm Research (I,R), A⋆STAR, Singapore
备注：Accepted in ICASSP 2023 conference
摘要：在不同条件下记录的语音话语在它们的嵌入估计中表现出不同程度的置信度，即，不确定性，即使使用相同的神经网络提取它们。本文旨在将xi向量网络前端产生的不确定性估计与概率线性判别分析（PLDA）后端评分相结合用于说话人确认。为了实现这一点，我们推导出一个后验协方差矩阵，它衡量不确定性，从帧精度到嵌入空间。我们提出了一个对数似然比函数的PLDA评分与不确定性传播。我们还建议用长度缩放技术代替长度归一化预处理技术，以用于后端的不确定性传播应用。在VoxCeleb-1、SITW测试集以及域不匹配的CNCeleb 1-E测试集上的实验结果表明，所提方法的有效性达到了14.5% ~ 41.3%的EER缩减率和4.6% ~ 25.3%的minDCF缩减率.
摘要：Speech utterances recorded under differing conditions exhibit varying degrees of confidence in their embedding estimates, i.e., uncertainty, even if they are extracted using the same neural network. This paper aims to incorporate the uncertainty estimate produced in the xi-vector network front-end with a probabilistic linear discriminant analysis (PLDA) back-end scoring for speaker verification. To achieve this we derive a posterior covariance matrix, which measures the uncertainty, from the frame-wise precisions to the embedding space. We propose a log-likelihood ratio function for the PLDA scoring with the uncertainty propagation. We also propose to replace the length normalization pre-processing technique with a length scaling technique for the application of uncertainty propagation in the back-end. Experimental results on the VoxCeleb-1, SITW test sets as well as a domain-mismatched CNCeleb1-E set show the effectiveness of the proposed techniques with 14.5%-41.3% EER reductions and 4.6%-25.3% minDCF reductions.
eess.AS音频处理

【1】 Frequency bin-wise single channel speech presence probability estimation  using multiple DNNs

标题：基于多DNN的频域单通道语音存在概率估计

链接：https://arxiv.org/abs/2302.12048

*与cs.SD语音【7】为同一篇

作者：Shuai Tao,Himavanth Reddy,Jesper Rindom Jensen,Mads Græsbøll Christensen
机构：Audio Analysis Lab, CREATE, Aalborg University, Aalborg, Denmark
备注：Accepted for ICASSP 2023
摘要：在这项工作中，我们提出了一个频率区间的方法来估计单通道语音存在概率（SPP）与多个深度神经网络（DNNs）在短时傅里叶变换域。由于所有频率仓通常被同时视为传统的基于DNN的SPP估计器的输入特征，因此高模型复杂度是不可避免的。为了降低模型复杂度和对训练数据的要求，我们考虑了单个频率单元和它的一些相邻频率单元来训练单独的门递归单元。此外，我们还使用含噪语音和后验概率SPP表示来训练我们的模型。在深度噪声抑制挑战数据集上执行实验。实验结果表明，采用频率区间模型可以提高语音检测的准确率。最后，我们也证明了我们提出的方法在语音检测精度和模型复杂度方面优于大多数最先进的SPP估计方法。
摘要：In this work, we propose a frequency bin-wise method to estimate the single-channel speech presence probability (SPP) with multiple deep neural networks (DNNs) in the short-time Fourier transform domain. Since all frequency bins are typically considered simultaneously as input features for conventional DNN-based SPP estimators, high model complexity is inevitable. To reduce the model complexity and the requirements on the training data, we take a single frequency bin and some of its neighboring frequency bins into account to train separate gate recurrent units. In addition, the noisy speech and the a posteriori probability SPP representation are used to train our model. The experiments were performed on the Deep Noise Suppression challenge dataset. The experimental results show that the speech detection accuracy can be improved when we employ the frequency bin-wise model. Finally, we also demonstrate that our proposed method outperforms most of the state-of-the-art SPP estimation methods in terms of speech detection accuracy and model complexity.

【2】 A Framework for Unified Real-time Personalized and Non-Personalized  Speech Enhancement

标题：一种统一的实时个性化和非个性化语音增强框架

链接：https://arxiv.org/abs/2302.11768

*与cs.SD语音【8】为同一篇

作者：Zhepei Wang,Ritwik Giri,Devansh Shah,Jean-Marc Valin,Michael M. Goodwin,Paris Smaragdis
机构：†Amazon Web Services, ♯University of Illinois at Urbana-Champaign
备注：Accepted by ICASSP 2023
摘要：在本研究中，我们提出一种训练单一语音增强网路的方法，此网路可以执行个人化与非个人化语音增强。这是通过合并指定增强输出类型的逐帧调节输入来实现的。为了提高增强输出的质量并减轻过抑制，我们实验了通过语音活动的存在或不存在来重新加权帧，并将增强应用于说话人嵌入。通过在多任务学习设置下训练，我们经验性地表明，所提出的统一模型在个性化和非个性化语音增强基准上都获得了有希望的结果，并且达到了与专门针对任一任务训练的模型相似的性能。所提方法的强大性能表明，与在推理过程中保持单独的任务特定模型相比，统一模型是一种更经济的选择。
摘要：In this study, we present an approach to train a single speech enhancement network that can perform both personalized and non-personalized speech enhancement. This is achieved by incorporating a frame-wise conditioning input that specifies the type of enhancement output. To improve the quality of the enhanced output and mitigate oversuppression, we experiment with re-weighting frames by the presence or absence of speech activity and applying augmentations to speaker embeddings. By training under a multi-task learning setting, we empirically show that the proposed unified model obtains promising results on both personalized and non-personalized speech enhancement benchmarks and reaches similar performance to models that are trained specialized for either task. The strong performance of the proposed method demonstrates that the unified model is a more economical alternative compared to keeping separate task-specific models during inference.

【3】 Incorporating Uncertainty from Speaker Embedding Estimation to Speaker  Verification

标题：将说话人嵌入估计的不确定性引入说话人确认

链接：https://arxiv.org/abs/2302.11763

*与cs.SD语音【9】为同一篇

作者：Qiongqiong Wang,Kong Aik Lee,Tianchi Liu
机构：Institute for Infocomm Research (I,R), A⋆STAR, Singapore
备注：Accepted in ICASSP 2023 conference
摘要：在不同条件下记录的语音话语在它们的嵌入估计中表现出不同程度的置信度，即，不确定性，即使使用相同的神经网络提取它们。本文旨在将xi向量网络前端产生的不确定性估计与概率线性判别分析（PLDA）后端评分相结合用于说话人确认。为了实现这一点，我们推导出一个后验协方差矩阵，它衡量不确定性，从帧精度到嵌入空间。我们提出了一个对数似然比函数的PLDA评分与不确定性传播。我们还建议用长度缩放技术代替长度归一化预处理技术，以用于后端的不确定性传播应用。在VoxCeleb-1、SITW测试集以及域不匹配的CNCeleb 1-E测试集上的实验结果表明，所提方法的有效性达到了14.5% ~ 41.3%的EER缩减率和4.6% ~ 25.3%的minDCF缩减率.
摘要：Speech utterances recorded under differing conditions exhibit varying degrees of confidence in their embedding estimates, i.e., uncertainty, even if they are extracted using the same neural network. This paper aims to incorporate the uncertainty estimate produced in the xi-vector network front-end with a probabilistic linear discriminant analysis (PLDA) back-end scoring for speaker verification. To achieve this we derive a posterior covariance matrix, which measures the uncertainty, from the frame-wise precisions to the embedding space. We propose a log-likelihood ratio function for the PLDA scoring with the uncertainty propagation. We also propose to replace the length normalization pre-processing technique with a length scaling technique for the application of uncertainty propagation in the back-end. Experimental results on the VoxCeleb-1, SITW test sets as well as a domain-mismatched CNCeleb1-E set show the effectiveness of the proposed techniques with 14.5%-41.3% EER reductions and 4.6%-25.3% minDCF reductions.

【4】 ProsAudit, a prosodic benchmark for self-supervised speech models

标题：ProsAudit，一个用于自我监督语音模型的韵律基准

链接：https://arxiv.org/abs/2302.12057

*与cs.SD语音【1】为同一篇

作者：Maureen de Seyssel,Marvin Lavechin,Hadrien Titeux,Arthur Thomas,Gwendal Virlet,Andrea Santos Revilla,Guillaume Wisniewski,Bogdan Ludusan,Emmanuel Dupoux
机构：Cognitive Machine Learning (ENS–CNRS–EHESS–INRIA–PSL Research University), France,  Universit´e de Paris Cit´e, CNRS, Laboratoire de linguistique formelle, F-, Paris, France, Bielefeld University, Germany,  PEGASE, INRAE, Institut Agro, Saint-Gilles, France
备注：4 pages + references, 1 figure
摘要：本文介绍了一个用于评估自监督学习（SSL）语音模型中结构韵律知识的英语基准ProsAudit。它由两个子任务、相应的指标和一个评估数据集组成。在原语法任务中，模型必须正确识别强韵律边界和弱韵律边界。在词汇任务中，模型需要正确区分插入在单词之间和单词内的停顿。我们还提供了该基准的人体评估分数。我们评估了一系列SSL模型，发现它们都能够在两个任务上执行高于概率的任务，即使是在一种看不见的语言上进行训练。然而，非母语模型在词汇任务上的表现显著低于母语模型，这凸显了词汇知识在词汇任务中的重要性。我们还发现了规模的明显影响，在两个子任务中，使用更多数据训练的模型表现更好。
摘要：We present ProsAudit, a benchmark in English to assess structural prosodic knowledge in self-supervised learning (SSL) speech models. It consists of two subtasks, their corresponding metrics, an evaluation dataset. In the protosyntax task, the model must correctly identify strong versus weak prosodic boundaries. In the lexical task, the model needs to correctly distinguish between pauses inserted between words and within words. We also provide human evaluation scores on this benchmark. We evaluated a series of SSL models and found that they were all able to perform above chance on both tasks, even when trained on an unseen language. However, non-native models performed significantly worse than native ones on the lexical task, highlighting the importance of lexical knowledge in this task. We also found a clear effect of size with models trained on more data performing better in the two subtasks.

【5】 Evaluating Automatic Speech Recognition in an Incremental Setting

标题：在增量设置中评估自动语音识别

链接：https://arxiv.org/abs/2302.12049

*与cs.SD语音【2】为同一篇

作者：Ryan Whetten,Mir Tahsin Imtiaz,Casey Kennington
机构：Boise State University, Department of Computer Science,  W University Dr., Boise, ID
备注：5 pages
摘要：自动语音识别的可靠性不断提高，使其日常使用激增。然而，出于研究目的，通常不清楚应该为任务选择哪种模型，特别是如果既需要速度又需要精度。在这篇文章中，我们使用包括单词错误率、延迟和对英语测试数据中已识别单词的更新次数在内的指标系统地评估了六个语音识别器，并提出和比较了两种将音频流传输到识别器进行增量识别的方法。我们进一步提出每秒撤销数作为评估增量识别的新指标，并证明它提供了对整体模型性能的见解。我们发现，一般来说，本地识别器比基于云的识别器更快，需要更少的更新。最后，我们发现Meta的Wav2Vec模型是最快的，而Mozilla的DeepSpeech模型在预测方面是最稳定的。
摘要：The increasing reliability of automatic speech recognition has proliferated its everyday use. However, for research purposes, it is often unclear which model one should choose for a task, particularly if there is a requirement for speed as well as accuracy. In this paper, we systematically evaluate six speech recognizers using metrics including word error rate, latency, and the number of updates to already recognized words on English test data, as well as propose and compare two methods for streaming audio into recognizers for incremental recognition. We further propose Revokes per Second as a new metric for evaluating incremental recognition and demonstrate that it provides insights into overall model performance. We find that, generally, local recognizers are faster and require fewer updates than cloud-based recognizers. Finally, we find Meta's Wav2Vec model to be the fastest, and find Mozilla's DeepSpeech model to be the most stable in its predictions.

【6】 Metric-oriented Speech Enhancement using Diffusion Probabilistic Model

标题：基于扩散概率模型的面向度量的语音增强

链接：https://arxiv.org/abs/2302.11989

*与cs.SD语音【3】为同一篇

作者：Chen Chen,Yuchen Hu,Weiwei Weng,Eng Siong Chng
机构：School of Computer Science and Engineering, Nanyang Technological University, Singapore
备注：Accepted by ICASSP2023
摘要：基于深度神经网络的语音增强技术的核心是在成对训练数据的监督下学习噪声到干净的转换。然而，任务专用评估度量（例如，PESQ）通常是不可微的，不能直接构造在训练准则中。训练目标和评估度量之间的这种不匹配可能导致次优性能。为了解决这个问题，我们提出了一种面向度量的语音增强方法（MOSE），该方法利用了扩散概率模型的最新进展，并将面向度量的训练策略集成到其逆过程中。具体而言，我们设计了一个基于行动者-批评者的框架，将评价指标视为一种后验回报，从而引导逆向过程向指标递增的方向发展。实验结果表明，MOSE明显受益于指标导向训练，在所有评价指标上均优于生成性基线。
摘要：Deep neural network based speech enhancement technique focuses on learning a noisy-to-clean transformation supervised by paired training data. However, the task-specific evaluation metric (e.g., PESQ) is usually non-differentiable and can not be directly constructed in the training criteria. This mismatch between the training objective and evaluation metric likely results in sub-optimal performance. To alleviate it, we propose a metric-oriented speech enhancement method (MOSE), which leverages the recent advances in the diffusion probabilistic model and integrates a metric-oriented training strategy into its reverse process. Specifically, we design an actor-critic based framework that considers the evaluation metric as a posterior reward, thus guiding the reverse process to the metric-increasing direction. The experimental results demonstrate that MOSE obviously benefits from metric-oriented training and surpasses the generative baselines in terms of all evaluation metrics.

【7】 Unsupervised Noise adaptation using Data Simulation

标题：基于数据模拟的无监督噪声自适应

链接：https://arxiv.org/abs/2302.11981

*与cs.SD语音【4】为同一篇

作者：Chen Chen,Yuchen Hu,Heqing Zou,Linhui Sun,Eng Siong Chng
机构：School of Computer Science and Engineering, Nanyang Technological University, Singapore,  Nanjing University of Posts and Telecommunications, China
备注：Accepted by ICASSP2023
摘要：基于深度神经网络的语音增强方法旨在使用监督学习范例来学习噪声到干净的变换。然而，这种训练井变换易受训练集中未包含的看不见的噪声的影响。在这项工作中，我们专注于语音增强中的无监督噪声自适应问题，其中目标域数据的真实数据是完全不可用的。具体地，我们提出了一种基于生成对抗网络的方法，以使用几分钟的不成对目标域数据来有效地学习逆清洁到噪声变换。然后，利用该变换来生成用于增强模型的域自适应的足够的模拟数据。实验结果表明，该方法有效地抑制了训练集与测试集之间的领域失配，并大幅优于最佳基线。
摘要：Deep neural network based speech enhancement approaches aim to learn a noisy-to-clean transformation using a supervised learning paradigm. However, such a trained-well transformation is vulnerable to unseen noises that are not included in training set. In this work, we focus on the unsupervised noise adaptation problem in speech enhancement, where the ground truth of target domain data is completely unavailable. Specifically, we propose a generative adversarial network based method to efficiently learn a converse clean-to-noisy transformation using a few minutes of unpaired target domain data. Then this transformation is utilized to generate sufficient simulated data for domain adaptation of the enhancement model. Experimental results show that our method effectively mitigates the domain mismatch between training and test sets, and surpasses the best baseline by a large margin.

【8】 D2Former: A Fully Complex Dual-Path Dual-Decoder Conformer Network using  Joint Complex Masking and Complex Spectral Mapping for Monaural Speech  Enhancement

标题：D2Former：一种用于单声道语音增强的联合复数掩蔽和复谱映射的全复数双路双解码器同构网络

链接：https://arxiv.org/abs/2302.11832

*与cs.SD语音【5】为同一篇

作者：Shengkui Zhao,Bin Ma
机构：Alibaba Group
备注：5 pages, 3 figures, accepted by ICASSP 2023
摘要：None
摘要：Monaural speech enhancement has been widely studied using real networks in the time-frequency (TF) domain. However, the input and the target are naturally complex-valued in the TF domain, a fully complex network is highly desirable for effectively learning the feature representation and modelling the sequence in the complex domain. Moreover, phase, an important factor for perceptual quality of speech, has been proved learnable together with magnitude from noisy speech using complex masking or complex spectral mapping. Many recent studies focus on either complex masking or complex spectral mapping, ignoring their performance boundaries. To address above issues, we propose a fully complex dual-path dual-decoder conformer network (D2Former) using joint complex masking and complex spectral mapping for monaural speech enhancement. In D2Former, we extend the conformer network into the complex domain and form a dual-path complex TF self-attention architecture for effectively modelling the complex-valued TF sequence. We further boost the TF feature representation in the encoder and the decoders using a dual-path learning structure by exploiting complex dilated convolutions on time dependency and complex feedforward sequential memory networks (CFSMN) for frequency recurrence. In addition, we improve the performance boundaries of complex masking and complex spectral mapping by combining the strengths of the two training targets into a joint-learning framework. As a consequence, D2Former takes fully advantages of the complex-valued operations, the dual-path processing, and the joint-training targets. Compared to the previous models, D2Former achieves state-of-the-art results on the VoiceBank+Demand benchmark with the smallest model size of 0.87M parameters.

【9】 MossFormer: Pushing the Performance Limit of Monaural Speech Separation  using Gated Single-Head Transformer with Convolution-Augmented Joint  Self-Attentions

标题：MossFormer：利用带卷积增强联合注意的门控单头Transformer突破单声道语音分离的性能极限

链接：https://arxiv.org/abs/2302.11824

*与cs.SD语音【6】为同一篇

作者：Shengkui Zhao,Bin Ma
机构：Alibaba Group
备注：5 pages, 3 figures, accepted by ICASSP 2023
摘要：基于变换器的模型在单声道语音分离中提供了显著的性能改进。然而，与最近提出的上限相比，仍然存在性能差距。目前双路径变压器模型的主要局限性是对长程元素相互作用和局部特征模式的建模效率低。在本文中，我们提出了一种具有卷积增强联合自注意的门控单头变换器结构，命名为\textit {MossFormer}（\textit {Mo} naural\textit {s} peech\textit {s} eparation Transs\textit {Former}），从而得到了这个上界.为了有效解决双路径结构中组块间的间接元素交互，MossFormer采用了联合局部和全局自注意结构，该结构同时对局部组块执行全计算自注意和对全序列执行线性化低成本自注意。联合注意力直接实现了MossFormer模型全序列元素交互作用。此外，我们使用了一个强大的注意门控机制与简化的单头自我注意。除了关注的远程建模外，我们还使用卷积来增强MossFormer，以进行位置方式的局部模式建模。因此，MossFormer的性能明显优于之前的型号，并在WSJ0 - 2/3mix和WHAM上实现了最先进的结果!/哇!基准。我们的模型在WSJ0 - 3mix上实现了21.2 dB的SI-SDRi上限，在WSJ0 - 2mix上仅比上限23.1 dB低0.3 dB。
摘要：Transformer based models have provided significant performance improvements in monaural speech separation. However, there is still a performance gap compared to a recent proposed upper bound. The major limitation of the current dual-path Transformer models is the inefficient modelling of long-range elemental interactions and local feature patterns. In this work, we achieve the upper bound by proposing a gated single-head transformer architecture with convolution-augmented joint self-attentions, named \textit{MossFormer} (\textit{Mo}naural \textit{s}peech \textit{s}eparation Trans\textit{Former}). To effectively solve the indirect elemental interactions across chunks in the dual-path architecture, MossFormer employs a joint local and global self-attention architecture that simultaneously performs a full-computation self-attention on local chunks and a linearised low-cost self-attention over the full sequence. The joint attention enables MossFormer model full-sequence elemental interaction directly. In addition, we employ a powerful attentive gating mechanism with simplified single-head self-attentions. Besides the attentive long-range modelling, we also augment MossFormer with convolutions for the position-wise local pattern modelling. As a consequence, MossFormer significantly outperforms the previous models and achieves the state-of-the-art results on WSJ0-2/3mix and WHAM!/WHAMR! benchmarks. Our model achieves the SI-SDRi upper bound of 21.2 dB on WSJ0-3mix and only 0.3 dB below the upper bound of 23.1 dB on WSJ0-2mix.

机器翻译，仅供参考