今日论文合集：cs.SD语音11篇，eess.AS音频处理11篇。
cs.SD语音

【1】 Efficient CTC Regularization via Coarse Labels for End-to-End Speech  Translation

标题：端到端语音翻译中基于粗标签的高效CTC正则化

链接：https://arxiv.org/abs/2302.10871

作者：Biao Zhang,Barry Haddow,Rico Sennrich
机构：School of Informatics, University of Edinburgh, Department of Computational Linguistics, University of Zurich
备注：EACL 2023
摘要：对于端到端的语音翻译，使用源转录本或目标翻译作为标签，使用连接主义时间分类（CTC）目标来正则化编码器可以极大地提高质量度量。然而，CTC需要词汇空间上的额外预测层，这带来了不可忽略的模型参数和计算开销，尽管该层通常不用于推理。在本文中，我们重新检查了CTC正则化对真正词汇标签的需求，并探索了减少CTC标签空间的策略，目标是提高效率而不降低质量。我们提出了CTC的粗标注（CoLaCTC），它通过简单的启发式规则合并词汇标注，例如使用截断、除法或模（MOD）运算。尽管简单，我们在4种源语言和8种目标语言上的实验表明，CoLaCTC与MOD尤其可以将标签空间大幅压缩到256，甚至更多，获得训练效率（1.18倍~ 1.77倍加速比，取决于原始词汇量大小），但仍然提供与CTC基线相当或更好的性能。我们还表明，CoLaCTC成功地推广到CTC规则化，无论使用转录本或翻译标记。
摘要：For end-to-end speech translation, regularizing the encoder with the Connectionist Temporal Classification (CTC) objective using the source transcript or target translation as labels can greatly improve quality metrics. However, CTC demands an extra prediction layer over the vocabulary space, bringing in nonnegligible model parameters and computational overheads, although this layer is typically not used for inference. In this paper, we re-examine the need for genuine vocabulary labels for CTC for regularization and explore strategies to reduce the CTC label space, targeting improved efficiency without quality degradation. We propose coarse labeling for CTC (CoLaCTC), which merges vocabulary labels via simple heuristic rules, such as using truncation, division or modulo (MOD) operations. Despite its simplicity, our experiments on 4 source and 8 target languages show that CoLaCTC with MOD particularly can compress the label space aggressively to 256 and even further, gaining training efficiency (1.18x ~ 1.77x speedup depending on the original vocabulary size) yet still delivering comparable or better performance than the CTC baseline. We also show that CoLaCTC successfully generalizes to CTC regularization regardless of using transcript or translation for labeling.

【2】 Interpretable Spectrum Transformation Attacks to Speaker Recognition

标题：对说话人识别的可解释谱变换攻击

链接：https://arxiv.org/abs/2302.10686

作者：Jiadi Yao,Hong Luo,Xiao-Lei Zhang
机构：Xiao-Lei Zhang are with the School of Marine Science andTechnology,  Northwestern Polytechnical University
摘要：对抗性攻击对说话人识别的成功主要是在白盒场景中。将攻击白盒代理模型所产生的对抗性声音应用于黑盒受害者模型，即基于传递的黑盒攻击时，对抗性声音的传递性不仅远不能令人满意，而且缺乏可解释的基础.针对这些问题，本文提出了一种基于改进离散余弦变换的谱变换攻击（STA-MDCT），以提高敌对语音向黑箱受害者模型的可转移性。具体地，我们首先将MDCT应用于输入语音。然后，我们稍微修改不同频带的能量以捕获对抗性噪声在时频域中的显著区域，这些区域对于攻击的成功至关重要。与现有的在时域操作语音的方法不同，所提出的框架在时频域操作语音，这提高了攻击的可解释性、可转移性和不可察觉性。而且，它可以被任何基于梯度的攻击者实现。为了利用模型集成的优势，我们不仅使用单个白盒代理模型来实现STA-MDCT，而且使用代理模型的集成来实现STA-MDCT。最后，我们利用类激活图（CAM）可视化了敌对语音的显著图，首次为说话人识别中基于迁移的攻击提供了可解释的基础。与五种典型攻击者的广泛比较结果表明，CAM可视化清楚地解释了STA-MDCT的有效性，以及比较方法的缺点;所提出的方法大大优于比较方法。
摘要：The success of adversarial attacks to speaker recognition is mainly in white-box scenarios. When applying the adversarial voices that are generated by attacking white-box surrogate models to black-box victim models, i.e. \textit{transfer-based} black-box attacks, the transferability of the adversarial voices is not only far from satisfactory, but also lacks interpretable basis. To address these issues, in this paper, we propose a general framework, named spectral transformation attack based on modified discrete cosine transform (STA-MDCT), to improve the transferability of the adversarial voices to a black-box victim model. Specifically, we first apply MDCT to the input voice. Then, we slightly modify the energy of different frequency bands for capturing the salient regions of the adversarial noise in the time-frequency domain that are critical to a successful attack. Unlike existing approaches that operate voices in the time domain, the proposed framework operates voices in the time-frequency domain, which improves the interpretability, transferability, and imperceptibility of the attack. Moreover, it can be implemented with any gradient-based attackers. To utilize the advantage of model ensembling, we not only implement STA-MDCT with a single white-box surrogate model, but also with an ensemble of surrogate models. Finally, we visualize the saliency maps of adversarial voices by the class activation maps (CAM), which offers an interpretable basis to transfer-based attacks in speaker recognition for the first time. Extensive comparison results with five representative attackers show that the CAM visualization clearly explains the effectiveness of STA-MDCT, and the weaknesses of the comparison methods; the proposed method outperforms the comparison methods by a large margin.

【3】 DasFormer: Deep Alternating Spectrogram Transformer for  Multi/Single-Channel Speech Separation

标题：DasFormer：用于多声道/单声道语音分离的深度交替语谱图转换器

链接：https://arxiv.org/abs/2302.10657

作者：Shuo Wang,Xiangyu Kong,Xiulian Peng,Hesam Movassagh,Vinod Prakash,Yan Lu
机构：Institute of Acoustics, Chinese Academy of Sciences, Beijing, China, Microsoft Research Asia, Beijing, China, Microsoft Corporation, Redmond WA , USA
备注：5 pages, accepted by ICASSP2023
摘要：对于语音分离的任务，以往的研究通常将多声道和单声道场景作为两条研究轨道，分别开发专门的解决方案。为此，我们提出了一种简单而统一的结构- DasFormer（Deep alternating spectrogram Transformer）来处理这两种复杂混响环境。与逐帧序列建模不同，频谱图中的每个TF bin被分配有嵌入编码频谱和空间信息。有了这样的输入，DasFormer然后由简单块的多次重复形成，每个简单块集成1）两个多头自注意（MHSA）模块，在谱图的每个频率仓和时间帧内交替处理2）每个MHSA之前的MBConv，用于对谱图上的局部特征建模。实验结果表明，DasFormer具有强大的时频建模能力，在多通道语音分离中的性能远远优于现有的SOTA模型，在混响场景中的性能也远优于单通道SOTA模型。
摘要：For the task of speech separation, previous study usually treats multi-channel and single-channel scenarios as two research tracks with specialized solutions developed respectively. Instead, we propose a simple and unified architecture - DasFormer (Deep alternating spectrogram transFormer) to handle both of them in the challenging reverberant environments. Unlike frame-wise sequence modeling, each TF-bin in the spectrogram is assigned with an embedding encoding spectral and spatial information. With such input, DasFormer is then formed by multiple repetition of simple blocks each of which integrates 1) two multi-head self-attention (MHSA) modules alternately processing within each frequency bin & temporal frame of the spectrogram 2) MBConv before each MHSA for modeling local features on the spectrogram. Experiments show that DasFormer has a powerful ability to model the time-frequency representation, whose performance far exceeds the current SOTA models in multi-channel speech separation, and also single-channel SOTA in the more challenging yet realistic reverberation scenario.

【4】 Nonparallel Emotional Voice Conversion For Unseen Speaker-Emotion Pairs  Using Dual Domain Adversarial Network & Virtual Domain Pairing

标题：基于双域对抗网络和虚拟域配对的非并行情感语音转换

链接：https://arxiv.org/abs/2302.10536

作者：Nirmesh Shah,Mayank Kumar Singh,Naoya Takahashi,Naoyuki Onoe
机构：Sony Research India,Sony Group Corporation, Japan
备注：Demo Samples at this https URL
摘要：情感语音转换（EVC）系统的主要目标是将给定语音信号的情感从一种风格转换成另一种风格而不修改信号的语言内容。大多数最先进的方法仅针对可见的说话者-情感组合来转换情感。在这篇论文中，我们解决了在训练和测试期间只存在中性数据的说话者的情感转换问题（即，看不见的说话者-情感组合）。为此，我们扩展了最近提出的StartGANv 2-VC架构，利用双编码器分别学习说话人和情感风格嵌入以及双域源分类器。为了实现向看不见的说话人-情感组合的转换，提出了一种虚拟域配对（VDP）训练策略，该策略虚拟地将真实数据中不存在的说话人-情感对合并在一起，而不损害对抗训练中鉴别器和生成器的最小最大博弈。我们使用印地语情感数据库对所提出的方法进行了评估。
摘要：Primary goal of an emotional voice conversion (EVC) system is to convert the emotion of a given speech signal from one style to another style without modifying the linguistic content of the signal. Most of the state-of-the-art approaches convert emotions for seen speaker-emotion combinations only. In this paper, we tackle the problem of converting the emotion of speakers whose only neutral data are present during the time of training and testing (i.e., unseen speaker-emotion combinations). To this end, we extend a recently proposed StartGANv2-VC architecture by utilizing dual encoders for learning the speaker and emotion style embeddings separately along with dual domain source classifiers. For achieving the conversion to unseen speaker-emotion combinations, we propose a Virtual Domain Pairing (VDP) training strategy, which virtually incorporates the speaker-emotion pairs that are not present in the real data without compromising the min-max game of a discriminator and generator in adversarial training. We evaluate the proposed method using a Hindi emotional database.

【5】 pykanto: a python library to accelerate research on wild bird song

标题：Pykanto：一个加速野生鸟鸣研究的巨蟒文库

链接：https://arxiv.org/abs/2302.10340

作者：Nilo Merino Recalde
机构：∗,  Department of Biology, University of Oxford, Oxford, UK
备注：9 pages, 3 figures
摘要：由于传统计算方法的局限性，研究野生动物的发声是一个挑战，这些方法通常耗时且缺乏可重复性。在这里，我介绍一个新的软件包pikanto，它提供了一套工具来构建、管理和探索大型声音数据库。它可以自动发现动物发声中的离散单元，使用新的交互式网络应用程序对个体曲目进行半监督标记，并将数据馈送到深度学习模型，以研究个体签名以及个体与种群之间的声学相似性等问题。为了证明它的能力，我把图书馆放在英国牛津附近威瑟姆森林的雄性大山雀的发声上进行测试。结果表明，通过鸣唱可以准确地确定个体鸟类的身份，并且使用Pykanto提高了该过程的效率和可重复性。
摘要：Studying the vocalisations of wild animals can be a challenge due to the limitations of traditional computational methods, which often are time-consuming and lack reproducibility. Here, I present pykanto, a new software package that provides a set of tools to build, manage, and explore large sound databases. It can automatically find discrete units in animal vocalisations, perform semi-supervised labelling of individual repertoires with a new interactive web app, and feed data to deep learning models to study things like individual signatures and acoustic similarity between individuals and populations. To demonstrate its capabilities, I put the library to the test on the vocalisations of male great tits in Wytham Woods, near Oxford, UK. The results show that the identities of individual birds can be accurately determined from their songs and that the use of pykanto improves the efficiency and reproducibility of the process.

【6】 VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge

标题：VoxSRC 2022：第四届VoxCeleb说话人识别挑战赛

链接：https://arxiv.org/abs/2302.10248

作者：Jaesung Huh,Andrew Brown,Jee-weon Jung,Joon Son Chung,Arsha Nagrani,Daniel Garcia-Romero,Andrew Zisserman
机构：Visual Geometry Group, Department of Engineering Science, University of Oxford, UK, Naver Corporation, South Korea, Korea Advanced Institute of Science and Technology, South Korea, AWS AI Labs, USA
摘要：本文总结了与INTERSPEECH 2022联合举办的2022年VoxCeleb说话人识别挑战赛（VoxSRC-22）的结果。这个挑战的目标是评估最先进的说话人识别系统能在多大程度上从“野外”获得的语音中识别出说话人。挑战包括：（i）提供YouTube视频中公开提供的说话人识别和日记数据，以及地面实况注释和标准化评价软件;以及（ii）在INTERSPEECH 2022举办的公众挑战和混合研讨会。我们描述了挑战的四个方面，以及基线、方法和结果。我们最后讨论了VoxSRC-22的新领域转移焦点，以及前三个版本挑战的进展。
摘要：This paper summarises the findings from the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22), which was held in conjunction with INTERSPEECH 2022. The goal of this challenge was to evaluate how well state-of-the-art speaker recognition systems can diarise and recognise speakers from speech obtained "in the wild". The challenge consisted of: (i) the provision of publicly available speaker recognition and diarisation data from YouTube videos together with ground truth annotation and standardised evaluation software; and (ii) a public challenge and hybrid workshop held at INTERSPEECH 2022. We describe the four tracks of our challenge along with the baselines, methods, and results. We conclude with a discussion on the new domain-transfer focus of VoxSRC-22, and on the progression of the challenge from the previous three editions.

【7】 Federated Learning for ASR based on Wav2vec 2.0

标题：基于Wav2vec 2.0的ASR联合学习

链接：https://arxiv.org/abs/2302.10790

作者：Tuan Nguyen,Salima Mdhaffar,Natalia Tomashenko,Jean-François Bonastre,Yannick Estève
机构：LIA - Avignon University, France
备注：5 pages, accepted in ICASSP 2023
摘要：本文提出了一种使用联邦学习来训练ASR模型的研究，该模型基于通过自监督预先训练的wav 2 vec 2.0模型。在著名的TED-LIUM 3数据集上进行的实验表明，该模型在不使用语言模型的情况下，在官方TED-LIUM 3测试集上获得了10.92%的单词错误率，并且不需要共享来自不同用户的任何数据。我们还分析了说话人参与联邦学习的程度对ASR的影响。由于联邦学习最初是出于隐私保护的目的而引入的，因此我们也测量了它保护说话人身份的能力。为此，我们采用了一种方法，根据指标数据集上的神经网络足迹分析交换模型中包含的信息。该分析是逐层进行的，并且示出了基于交换的wav 2 vec 2.0的模型中的哪些层带来说话者身份信息。
摘要：This paper presents a study on the use of federated learning to train an ASR model based on a wav2vec 2.0 model pre-trained by self supervision. Carried out on the well-known TED-LIUM 3 dataset, our experiments show that such a model can obtain, with no use of a language model, a word error rate of 10.92% on the official TED-LIUM 3 test set, without sharing any data from the different users. We also analyse the ASR performance for speakers depending to their participation to the federated learning. Since federated learning was first introduced for privacy purposes, we also measure its ability to protect speaker identity. To do that, we exploit an approach to analyze information contained in exchanged models based on a neural network footprint on an indicator dataset. This analysis is made layer-wise and shows which layers in an exchanged wav2vec 2.0 based model bring the speaker identity information.

【8】 Leveraging phone-level linguistic-acoustic similarity for  utterance-level pronunciation scoring

标题：利用语音级语言-声学相似性进行发音级发音评分

链接：https://arxiv.org/abs/2302.10444

作者：Wei Liu,Kaiqi Fu,Xiaohai Tian,Shuju Shi,Wei Li,Zejun Ma,Tan Lee
机构：Department of Electronic Engineering, The Chinese University of Hong Kong, ByteDance AI Lab Speech & Audio Team
备注：Accepted by ICASSP 2023
摘要：最近对发音评分的研究已经探索了引入音素嵌入作为参考发音的效果，但大多数是以隐含的方式，即，添加或连接参考音素嵌入和目标音素的实际发音作为音素级发音质量表示。在这篇文章中，我们建议使用语言声学相似性来明确地测量非母语产出与其母语参考的偏差，以进行发音评估。具体地，首先通过参考音素嵌入和对应的声学嵌入之间的余弦相似性来估计偏差。接下来，引入音素级发音良好度（GOP）预训练阶段来指导这种基于相似性的学习，以便更好地初始化上述两个嵌入。最后，使用基于变换器的分级发音评分器来映射音素嵌入、声学嵌入序列以及它们的相似性度量，以预测最终的话语级别评分。在非本地数据库上的实验结果表明，所提出的系统明显优于基线，其中声学和音素嵌入是简单地添加或级联。进一步的实验结果表明，该方法学习的音素嵌入能够捕捉母语语音的语言声学属性作为参考。
摘要：Recent studies on pronunciation scoring have explored the effect of introducing phone embeddings as reference pronunciation, but mostly in an implicit manner, i.e., addition or concatenation of reference phone embedding and actual pronunciation of the target phone as the phone-level pronunciation quality representation. In this paper, we propose to use linguistic-acoustic similarity to explicitly measure the deviation of non-native production from its native reference for pronunciation assessment. Specifically, the deviation is first estimated by the cosine similarity between reference phone embedding and corresponding acoustic embedding. Next, a phone-level Goodness of pronunciation (GOP) pre-training stage is introduced to guide this similarity-based learning for better initialization of the aforementioned two embeddings. Finally, a transformer-based hierarchical pronunciation scorer is used to map a sequence of phone embeddings, acoustic embeddings along with their similarity measures to predict the final utterance-level score. Experimental results on the non-native databases suggest that the proposed system significantly outperforms the baselines, where the acoustic and phone embeddings are simply added or concatenated. A further examination shows that the phone embeddings learned in the proposed approach are able to capture linguistic-acoustic attributes of native pronunciation as reference.

【9】 Real-time speech enhancement with dynamic attention span

标题：具有动态注意广度的实时语音增强

链接：https://arxiv.org/abs/2302.10377

作者：Chengyu Zheng,Yuan Zhou,Xiulian Peng,Yuan Zhang,Yan Lu
机构：†Communication University of China, Beijing, China, ‡Microsoft Research Asia, Beijing, China
备注：ICASSP 2023 (Accepted)
摘要：对于包括噪声抑制、去混响和声学回声消除的实时语音增强（SE），音频信号的时变成为一个严峻的挑战。因果关系和内存占用限制了系统只能利用历史信息来捕捉时变特征。提出了在基于深度神经网络的SE模型中，根据输入信号自适应地改变感受野。具体地，在编解码器架构中，在所有关注模块中引入动态关注跨度机制，用于控制处理当前帧所使用的历史内容的大小。实验结果表明，该动态机制能较好地跟踪时变因素，捕捉语音相关特征，有利于干扰去除和语音质量保持。
摘要：For real-time speech enhancement (SE) including noise suppression, dereverberation and acoustic echo cancellation, the time-variance of the audio signals becomes a severe challenge. The causality and memory usage limit that only the historical information can be used for the system to capture the time-variant characteristics. We propose to adaptively change the receptive field according to the input signal in deep neural network based SE model. Specifically, in an encoder-decoder framework, a dynamic attention span mechanism is introduced to all the attention modules for controlling the size of historical content used for processing the current frame. Experimental results verify that this dynamic mechanism can better track time-variant factors and capture speech-related characteristics, benefiting to both interference removing and speech quality retaining.

【10】 Real-Time Speech Enhancement Using Spectral Subtraction with Minimum  Statistics and Spectral Floor

标题：基于最小统计量和谱底的谱减法实时语音增强

链接：https://arxiv.org/abs/2302.10313

作者：Georgios Ioannides,Vasilios Rallis
摘要：提出了一种初步的实时语音增强方法，以减小加性噪声的影响。该方法在频域中操作，并且是谱减法的一种形式。最初，使用最小统计量来生成频域中噪声信号的估计。最小统计量的使用避免了对语音活动检测器（VAD）的需要，语音活动检测器（VAD）已被证明是难以创建的。由于使用了最小统计量，在从噪声破坏的语音信号中减去之前，噪声信号估计必须乘以一个比例因子。频谱基底被应用于该差值以抑制“音乐噪声”的影响。最后，考虑一系列进一步的增强以进一步减少残留噪声的影响。使用时间-频率图比较这些方法以创建最终的语音增强设计
摘要：An initial real-time speech enhancement method is presented to reduce the effects of additive noise. The method operates in the frequency domain and is a form of spectral subtraction. Initially, minimum statistics are used to generate an estimate of the noise signal in the frequency domain. The use of minimum statistics avoids the need for a voice activity detector (VAD) which has proven to be challenging to create. As minimum statistics are used, the noise signal estimate must be multiplied by a scaling factor before subtraction from the noise corrupted speech signal can take place. A spectral floor is applied to the difference to suppress the effects of "musical noise". Finally, a series of further enhancements are considered to reduce the effects of residual noise even further. These methods are compared using time-frequency plots to create the final speech enhancement design

【11】 Spoken Entity Extraction for Virtual Agents

标题：面向虚拟主体的语音实体提取

链接：https://arxiv.org/abs/2302.10186

作者：Karan Singla,Yeon-Jun Kim,Ryan Price,Shahab Jalalvand,Srinivas Bangalore
机构：Interactions AI
摘要：本文重新考虑了使用语音编码器进行语音处理的一些方面，特别是直接从语音中提取实体，而不需要中间文本表示。在人机对话中，从语音中提取姓名、邮政地址和电子邮件地址等实体是一项具有挑战性的任务。在本文中，我们研究了微调预训练语音编码器对直接从语音中提取人类可读形式的口语实体而不需要文本转录的影响。我们说明了这种直接方法优化了编码器以仅转录语音的实体相关部分，而忽略诸如载体短语和实体拼写之类的多余部分。在来自企业虚拟代理的对话的背景下，我们证明了1步方法优于典型的2步级联，即首先生成词汇转录，然后进行基于文本的实体提取以识别口语实体。
摘要：This paper reimagines some aspects of speech processing using speech encoders, specifically about extracting entities directly from speech, with no intermediate textual representation. In human-computer conversations, extracting entities such as names, postal addresses and email addresses from speech is a challenging task. In this paper, we study the impact of fine-tuning pre-trained speech encoders on extracting spoken entities in human-readable form directly from speech without the need for text transcription. We illustrate that such a direct approach optimizes the encoder to transcribe only the entity relevant portions of speech, ignoring the superfluous portions such as carrier phrases and spellings of entities. In the context of dialogs from an enterprise virtual agent, we demonstrate that the 1-step approach outperforms the typical 2-step cascade of first generating lexical transcriptions followed by text-based entity extraction for identifying spoken entities.
eess.AS音频处理

【1】 Federated Learning for ASR based on Wav2vec 2.0

标题：基于Wav2vec 2.0的ASR联合学习

链接：https://arxiv.org/abs/2302.10790

*与cs.SD语音【7】为同一篇

作者：Tuan Nguyen,Salima Mdhaffar,Natalia Tomashenko,Jean-François Bonastre,Yannick Estève
机构：LIA - Avignon University, France
备注：5 pages, accepted in ICASSP 2023
摘要：本文提出了一种使用联邦学习来训练ASR模型的研究，该模型基于通过自监督预先训练的wav 2 vec 2.0模型。在著名的TED-LIUM 3数据集上进行的实验表明，该模型在不使用语言模型的情况下，在官方TED-LIUM 3测试集上获得了10.92%的单词错误率，并且不需要共享来自不同用户的任何数据。我们还分析了说话人参与联邦学习的程度对ASR的影响。由于联邦学习最初是出于隐私保护的目的而引入的，因此我们也测量了它保护说话人身份的能力。为此，我们采用了一种方法，根据指标数据集上的神经网络足迹分析交换模型中包含的信息。该分析是逐层进行的，并且示出了基于交换的wav 2 vec 2.0的模型中的哪些层带来说话者身份信息。
摘要：This paper presents a study on the use of federated learning to train an ASR model based on a wav2vec 2.0 model pre-trained by self supervision. Carried out on the well-known TED-LIUM 3 dataset, our experiments show that such a model can obtain, with no use of a language model, a word error rate of 10.92% on the official TED-LIUM 3 test set, without sharing any data from the different users. We also analyse the ASR performance for speakers depending to their participation to the federated learning. Since federated learning was first introduced for privacy purposes, we also measure its ability to protect speaker identity. To do that, we exploit an approach to analyze information contained in exchanged models based on a neural network footprint on an indicator dataset. This analysis is made layer-wise and shows which layers in an exchanged wav2vec 2.0 based model bring the speaker identity information.

【2】 Leveraging phone-level linguistic-acoustic similarity for  utterance-level pronunciation scoring

标题：利用语音级语言-声学相似性进行发音级发音评分

链接：https://arxiv.org/abs/2302.10444

*与cs.SD语音【8】为同一篇

作者：Wei Liu,Kaiqi Fu,Xiaohai Tian,Shuju Shi,Wei Li,Zejun Ma,Tan Lee
机构：Department of Electronic Engineering, The Chinese University of Hong Kong, ByteDance AI Lab Speech & Audio Team
备注：Accepted by ICASSP 2023
摘要：最近对发音评分的研究已经探索了引入音素嵌入作为参考发音的效果，但大多数是以隐含的方式，即，添加或连接参考音素嵌入和目标音素的实际发音作为音素级发音质量表示。在这篇文章中，我们建议使用语言声学相似性来明确地测量非母语产出与其母语参考的偏差，以进行发音评估。具体地，首先通过参考音素嵌入和对应的声学嵌入之间的余弦相似性来估计偏差。接下来，引入音素级发音良好度（GOP）预训练阶段来指导这种基于相似性的学习，以便更好地初始化上述两个嵌入。最后，使用基于变换器的分级发音评分器来映射音素嵌入、声学嵌入序列以及它们的相似性度量，以预测最终的话语级别评分。在非本地数据库上的实验结果表明，所提出的系统明显优于基线，其中声学和音素嵌入是简单地添加或级联。进一步的实验结果表明，该方法学习的音素嵌入能够捕捉母语语音的语言声学属性作为参考。
摘要：Recent studies on pronunciation scoring have explored the effect of introducing phone embeddings as reference pronunciation, but mostly in an implicit manner, i.e., addition or concatenation of reference phone embedding and actual pronunciation of the target phone as the phone-level pronunciation quality representation. In this paper, we propose to use linguistic-acoustic similarity to explicitly measure the deviation of non-native production from its native reference for pronunciation assessment. Specifically, the deviation is first estimated by the cosine similarity between reference phone embedding and corresponding acoustic embedding. Next, a phone-level Goodness of pronunciation (GOP) pre-training stage is introduced to guide this similarity-based learning for better initialization of the aforementioned two embeddings. Finally, a transformer-based hierarchical pronunciation scorer is used to map a sequence of phone embeddings, acoustic embeddings along with their similarity measures to predict the final utterance-level score. Experimental results on the non-native databases suggest that the proposed system significantly outperforms the baselines, where the acoustic and phone embeddings are simply added or concatenated. A further examination shows that the phone embeddings learned in the proposed approach are able to capture linguistic-acoustic attributes of native pronunciation as reference.

【3】 Real-time speech enhancement with dynamic attention span

标题：具有动态注意广度的实时语音增强

链接：https://arxiv.org/abs/2302.10377

*与cs.SD语音【9】为同一篇

作者：Chengyu Zheng,Yuan Zhou,Xiulian Peng,Yuan Zhang,Yan Lu
机构：†Communication University of China, Beijing, China, ‡Microsoft Research Asia, Beijing, China
备注：ICASSP 2023 (Accepted)
摘要：对于包括噪声抑制、去混响和声学回声消除的实时语音增强（SE），音频信号的时变成为一个严峻的挑战。因果关系和内存占用限制了系统只能利用历史信息来捕捉时变特征。提出了在基于深度神经网络的SE模型中，根据输入信号自适应地改变感受野。具体地，在编解码器架构中，在所有关注模块中引入动态关注跨度机制，用于控制处理当前帧所使用的历史内容的大小。实验结果表明，该动态机制能较好地跟踪时变因素，捕捉语音相关特征，有利于干扰去除和语音质量保持。
摘要：For real-time speech enhancement (SE) including noise suppression, dereverberation and acoustic echo cancellation, the time-variance of the audio signals becomes a severe challenge. The causality and memory usage limit that only the historical information can be used for the system to capture the time-variant characteristics. We propose to adaptively change the receptive field according to the input signal in deep neural network based SE model. Specifically, in an encoder-decoder framework, a dynamic attention span mechanism is introduced to all the attention modules for controlling the size of historical content used for processing the current frame. Experimental results verify that this dynamic mechanism can better track time-variant factors and capture speech-related characteristics, benefiting to both interference removing and speech quality retaining.

【4】 Real-Time Speech Enhancement Using Spectral Subtraction with Minimum  Statistics and Spectral Floor

标题：基于最小统计量和谱底的谱减法实时语音增强

链接：https://arxiv.org/abs/2302.10313

*与cs.SD语音【10】为同一篇

作者：Georgios Ioannides,Vasilios Rallis
摘要：提出了一种初步的实时语音增强方法，以减小加性噪声的影响。该方法在频域中操作，并且是谱减法的一种形式。最初，使用最小统计量来生成频域中噪声信号的估计。最小统计量的使用避免了对语音活动检测器（VAD）的需要，语音活动检测器（VAD）已被证明是难以创建的。由于使用了最小统计量，在从噪声破坏的语音信号中减去之前，噪声信号估计必须乘以一个比例因子。频谱基底被应用于该差值以抑制“音乐噪声”的影响。最后，考虑一系列进一步的增强以进一步减少残留噪声的影响。使用时间-频率图比较这些方法以创建最终的语音增强设计
摘要：An initial real-time speech enhancement method is presented to reduce the effects of additive noise. The method operates in the frequency domain and is a form of spectral subtraction. Initially, minimum statistics are used to generate an estimate of the noise signal in the frequency domain. The use of minimum statistics avoids the need for a voice activity detector (VAD) which has proven to be challenging to create. As minimum statistics are used, the noise signal estimate must be multiplied by a scaling factor before subtraction from the noise corrupted speech signal can take place. A spectral floor is applied to the difference to suppress the effects of "musical noise". Finally, a series of further enhancements are considered to reduce the effects of residual noise even further. These methods are compared using time-frequency plots to create the final speech enhancement design

【5】 Spoken Entity Extraction for Virtual Agents

标题：面向虚拟主体的语音实体提取

链接：https://arxiv.org/abs/2302.10186

*与cs.SD语音【11】为同一篇

作者：Karan Singla,Yeon-Jun Kim,Ryan Price,Shahab Jalalvand,Srinivas Bangalore
机构：Interactions AI
摘要：本文重新考虑了使用语音编码器进行语音处理的一些方面，特别是直接从语音中提取实体，而不需要中间文本表示。在人机对话中，从语音中提取姓名、邮政地址和电子邮件地址等实体是一项具有挑战性的任务。在本文中，我们研究了微调预训练语音编码器对直接从语音中提取人类可读形式的口语实体而不需要文本转录的影响。我们说明了这种直接方法优化了编码器以仅转录语音的实体相关部分，而忽略诸如载体短语和实体拼写之类的多余部分。在来自企业虚拟代理的对话的背景下，我们证明了1步方法优于典型的2步级联，即首先生成词汇转录，然后进行基于文本的实体提取以识别口语实体。
摘要：This paper reimagines some aspects of speech processing using speech encoders, specifically about extracting entities directly from speech, with no intermediate textual representation. In human-computer conversations, extracting entities such as names, postal addresses and email addresses from speech is a challenging task. In this paper, we study the impact of fine-tuning pre-trained speech encoders on extracting spoken entities in human-readable form directly from speech without the need for text transcription. We illustrate that such a direct approach optimizes the encoder to transcribe only the entity relevant portions of speech, ignoring the superfluous portions such as carrier phrases and spellings of entities. In the context of dialogs from an enterprise virtual agent, we demonstrate that the 1-step approach outperforms the typical 2-step cascade of first generating lexical transcriptions followed by text-based entity extraction for identifying spoken entities.

【6】 Efficient CTC Regularization via Coarse Labels for End-to-End Speech  Translation

标题：端到端语音翻译中基于粗标签的高效CTC正则化

链接：https://arxiv.org/abs/2302.10871

*与cs.SD语音【1】为同一篇

作者：Biao Zhang,Barry Haddow,Rico Sennrich
机构：School of Informatics, University of Edinburgh, Department of Computational Linguistics, University of Zurich
备注：EACL 2023
摘要：对于端到端的语音翻译，使用源转录本或目标翻译作为标签，使用连接主义时间分类（CTC）目标来正则化编码器可以极大地提高质量度量。然而，CTC需要词汇空间上的额外预测层，这带来了不可忽略的模型参数和计算开销，尽管该层通常不用于推理。在本文中，我们重新检查了CTC正则化对真正词汇标签的需求，并探索了减少CTC标签空间的策略，目标是提高效率而不降低质量。我们提出了CTC的粗标注（CoLaCTC），它通过简单的启发式规则合并词汇标注，例如使用截断、除法或模（MOD）运算。尽管简单，我们在4种源语言和8种目标语言上的实验表明，CoLaCTC与MOD尤其可以将标签空间大幅压缩到256，甚至更多，获得训练效率（1.18倍~ 1.77倍加速比，取决于原始词汇量大小），但仍然提供与CTC基线相当或更好的性能。我们还表明，CoLaCTC成功地推广到CTC规则化，无论使用转录本或翻译标记。
摘要：For end-to-end speech translation, regularizing the encoder with the Connectionist Temporal Classification (CTC) objective using the source transcript or target translation as labels can greatly improve quality metrics. However, CTC demands an extra prediction layer over the vocabulary space, bringing in nonnegligible model parameters and computational overheads, although this layer is typically not used for inference. In this paper, we re-examine the need for genuine vocabulary labels for CTC for regularization and explore strategies to reduce the CTC label space, targeting improved efficiency without quality degradation. We propose coarse labeling for CTC (CoLaCTC), which merges vocabulary labels via simple heuristic rules, such as using truncation, division or modulo (MOD) operations. Despite its simplicity, our experiments on 4 source and 8 target languages show that CoLaCTC with MOD particularly can compress the label space aggressively to 256 and even further, gaining training efficiency (1.18x ~ 1.77x speedup depending on the original vocabulary size) yet still delivering comparable or better performance than the CTC baseline. We also show that CoLaCTC successfully generalizes to CTC regularization regardless of using transcript or translation for labeling.

【7】 Interpretable Spectrum Transformation Attacks to Speaker Recognition

标题：对说话人识别的可解释谱变换攻击

链接：https://arxiv.org/abs/2302.10686

*与cs.SD语音【2】为同一篇

作者：Jiadi Yao,Hong Luo,Xiao-Lei Zhang
机构：Xiao-Lei Zhang are with the School of Marine Science andTechnology,  Northwestern Polytechnical University
摘要：None
摘要：The success of adversarial attacks to speaker recognition is mainly in white-box scenarios. When applying the adversarial voices that are generated by attacking white-box surrogate models to black-box victim models, i.e. \textit{transfer-based} black-box attacks, the transferability of the adversarial voices is not only far from satisfactory, but also lacks interpretable basis. To address these issues, in this paper, we propose a general framework, named spectral transformation attack based on modified discrete cosine transform (STA-MDCT), to improve the transferability of the adversarial voices to a black-box victim model. Specifically, we first apply MDCT to the input voice. Then, we slightly modify the energy of different frequency bands for capturing the salient regions of the adversarial noise in the time-frequency domain that are critical to a successful attack. Unlike existing approaches that operate voices in the time domain, the proposed framework operates voices in the time-frequency domain, which improves the interpretability, transferability, and imperceptibility of the attack. Moreover, it can be implemented with any gradient-based attackers. To utilize the advantage of model ensembling, we not only implement STA-MDCT with a single white-box surrogate model, but also with an ensemble of surrogate models. Finally, we visualize the saliency maps of adversarial voices by the class activation maps (CAM), which offers an interpretable basis to transfer-based attacks in speaker recognition for the first time. Extensive comparison results with five representative attackers show that the CAM visualization clearly explains the effectiveness of STA-MDCT, and the weaknesses of the comparison methods; the proposed method outperforms the comparison methods by a large margin.

【8】 DasFormer: Deep Alternating Spectrogram Transformer for  Multi/Single-Channel Speech Separation

标题：DasFormer：用于多声道/单声道语音分离的深度交替语谱图转换器

链接：https://arxiv.org/abs/2302.10657

*与cs.SD语音【3】为同一篇

作者：Shuo Wang,Xiangyu Kong,Xiulian Peng,Hesam Movassagh,Vinod Prakash,Yan Lu
机构：Institute of Acoustics, Chinese Academy of Sciences, Beijing, China, Microsoft Research Asia, Beijing, China, Microsoft Corporation, Redmond WA , USA
备注：5 pages, accepted by ICASSP2023
摘要：对于语音分离的任务，以往的研究通常将多声道和单声道场景作为两条研究轨道，分别开发专门的解决方案。为此，我们提出了一种简单而统一的结构- DasFormer（Deep alternating spectrogram Transformer）来处理这两种复杂混响环境。与逐帧序列建模不同，频谱图中的每个TF bin被分配有嵌入编码频谱和空间信息。有了这样的输入，DasFormer然后由简单块的多次重复形成，每个简单块集成1）两个多头自注意（MHSA）模块，在谱图的每个频率仓和时间帧内交替处理2）每个MHSA之前的MBConv，用于对谱图上的局部特征建模。实验结果表明，DasFormer具有强大的时频建模能力，在多通道语音分离中的性能远远优于现有的SOTA模型，在混响场景中的性能也远优于单通道SOTA模型。
摘要：For the task of speech separation, previous study usually treats multi-channel and single-channel scenarios as two research tracks with specialized solutions developed respectively. Instead, we propose a simple and unified architecture - DasFormer (Deep alternating spectrogram transFormer) to handle both of them in the challenging reverberant environments. Unlike frame-wise sequence modeling, each TF-bin in the spectrogram is assigned with an embedding encoding spectral and spatial information. With such input, DasFormer is then formed by multiple repetition of simple blocks each of which integrates 1) two multi-head self-attention (MHSA) modules alternately processing within each frequency bin & temporal frame of the spectrogram 2) MBConv before each MHSA for modeling local features on the spectrogram. Experiments show that DasFormer has a powerful ability to model the time-frequency representation, whose performance far exceeds the current SOTA models in multi-channel speech separation, and also single-channel SOTA in the more challenging yet realistic reverberation scenario.

【9】 Nonparallel Emotional Voice Conversion For Unseen Speaker-Emotion Pairs  Using Dual Domain Adversarial Network & Virtual Domain Pairing

标题：基于双域对抗网络和虚拟域配对的非并行情感语音转换

链接：https://arxiv.org/abs/2302.10536

*与cs.SD语音【4】为同一篇

作者：Nirmesh Shah,Mayank Kumar Singh,Naoya Takahashi,Naoyuki Onoe
机构：Sony Research India,Sony Group Corporation, Japan
备注：Demo Samples at this https URL
摘要：情感语音转换（EVC）系统的主要目标是将给定语音信号的情感从一种风格转换成另一种风格而不修改信号的语言内容。大多数最先进的方法仅针对可见的说话者-情感组合来转换情感。在这篇论文中，我们解决了在训练和测试期间只存在中性数据的说话者的情感转换问题（即，看不见的说话者-情感组合）。为此，我们扩展了最近提出的StartGANv 2-VC架构，利用双编码器分别学习说话人和情感风格嵌入以及双域源分类器。为了实现向看不见的说话人-情感组合的转换，提出了一种虚拟域配对（VDP）训练策略，该策略虚拟地将真实数据中不存在的说话人-情感对合并在一起，而不损害对抗训练中鉴别器和生成器的最小最大博弈。我们使用印地语情感数据库对所提出的方法进行了评估。
摘要：Primary goal of an emotional voice conversion (EVC) system is to convert the emotion of a given speech signal from one style to another style without modifying the linguistic content of the signal. Most of the state-of-the-art approaches convert emotions for seen speaker-emotion combinations only. In this paper, we tackle the problem of converting the emotion of speakers whose only neutral data are present during the time of training and testing (i.e., unseen speaker-emotion combinations). To this end, we extend a recently proposed StartGANv2-VC architecture by utilizing dual encoders for learning the speaker and emotion style embeddings separately along with dual domain source classifiers. For achieving the conversion to unseen speaker-emotion combinations, we propose a Virtual Domain Pairing (VDP) training strategy, which virtually incorporates the speaker-emotion pairs that are not present in the real data without compromising the min-max game of a discriminator and generator in adversarial training. We evaluate the proposed method using a Hindi emotional database.

【10】 pykanto: a python library to accelerate research on wild bird song

标题：Pykanto：一个加速野生鸟鸣研究的巨蟒文库

链接：https://arxiv.org/abs/2302.10340

*与cs.SD语音【5】为同一篇

作者：Nilo Merino Recalde
机构：∗,  Department of Biology, University of Oxford, Oxford, UK
备注：9 pages, 3 figures
摘要：由于传统计算方法的局限性，研究野生动物的发声是一个挑战，这些方法通常耗时且缺乏可重复性。在这里，我介绍一个新的软件包pikanto，它提供了一套工具来构建、管理和探索大型声音数据库。它可以自动发现动物发声中的离散单元，使用新的交互式网络应用程序对个体曲目进行半监督标记，并将数据馈送到深度学习模型，以研究个体签名以及个体与种群之间的声学相似性等问题。为了证明它的能力，我把图书馆放在英国牛津附近威瑟姆森林的雄性大山雀的发声上进行测试。结果表明，通过鸣唱可以准确地确定个体鸟类的身份，并且使用Pykanto提高了该过程的效率和可重复性。
摘要：Studying the vocalisations of wild animals can be a challenge due to the limitations of traditional computational methods, which often are time-consuming and lack reproducibility. Here, I present pykanto, a new software package that provides a set of tools to build, manage, and explore large sound databases. It can automatically find discrete units in animal vocalisations, perform semi-supervised labelling of individual repertoires with a new interactive web app, and feed data to deep learning models to study things like individual signatures and acoustic similarity between individuals and populations. To demonstrate its capabilities, I put the library to the test on the vocalisations of male great tits in Wytham Woods, near Oxford, UK. The results show that the identities of individual birds can be accurately determined from their songs and that the use of pykanto improves the efficiency and reproducibility of the process.

【11】 VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge

标题：VoxSRC 2022：第四届VoxCeleb说话人识别挑战赛

链接：https://arxiv.org/abs/2302.10248

*与cs.SD语音【6】为同一篇

作者：Jaesung Huh,Andrew Brown,Jee-weon Jung,Joon Son Chung,Arsha Nagrani,Daniel Garcia-Romero,Andrew Zisserman
机构：Visual Geometry Group, Department of Engineering Science, University of Oxford, UK, Naver Corporation, South Korea, Korea Advanced Institute of Science and Technology, South Korea, AWS AI Labs, USA
摘要：None
摘要：This paper summarises the findings from the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22), which was held in conjunction with INTERSPEECH 2022. The goal of this challenge was to evaluate how well state-of-the-art speaker recognition systems can diarise and recognise speakers from speech obtained "in the wild". The challenge consisted of: (i) the provision of publicly available speaker recognition and diarisation data from YouTube videos together with ground truth annotation and standardised evaluation software; and (ii) a public challenge and hybrid workshop held at INTERSPEECH 2022. We describe the four tracks of our challenge along with the baselines, methods, and results. We conclude with a discussion on the new domain-transfer focus of VoxSRC-22, and on the progression of the challenge from the previous three editions.

机器翻译，仅供参考