今日论文合集：cs.SD语音10篇，eess.AS音频处理15篇。cs.SD语音【1】 Multilingual Speech-to-Speech Translation into Multiple Target Languages标题：多语言语音到多目标语言的翻译链接：https://arxiv.org/abs/2307.08655作者：Hongyu Gong,Ning Dong,Sravya Popuri,Vedanuj Goswami,Ann Lee,Juan Pino机构：Meta AI Research, USA摘要：语音到语音翻译（S2 ST）使人们能够在用不同语言交谈的人之间进行口头交流。尽管有一些关于多语言S2 ST的研究，但他们的重点是源端的多语言性，即，从多种源语言到一种目标语言的翻译。我们提出了支持多种目标语言的多语言S2 ST的第一个工作。利用语音到单元和声码器的直接S2 ST的最新进展，我们为这些关键组件配备了多语言功能。语音掩蔽单元（Speech-to-Masked-unit，S2 MU）是S2 U的多语言扩展，它对不属于目标语言的单元进行掩蔽，以减少语言干扰。我们还提出了多语言声码器，这是训练语言嵌入和辅助语言识别损失。在基准翻译测试集，我们提出的多语言模型显示出优越的性能比双语模型从英语翻译成16美元的目标语言。摘要：Speech-to-speech translation (S2ST) enables spoken communication between people talking in different languages. Despite a few studies on multilingual S2ST, their focus is the multilinguality on the source side, i.e., the translation from multiple source languages to one target language. We present the first work on multilingual S2ST supporting multiple target languages. Leveraging recent advance in direct S2ST with speech-to-unit and vocoder, we equip these key components with multilingual capability. Speech-to-masked-unit (S2MU) is the multilingual extension of S2U, which applies masking to units which don't belong to the given target language to reduce the language interference. We also propose multilingual vocoder which is trained with language embedding and the auxiliary loss of language identification. On benchmark translation testsets, our proposed multilingual model shows superior performance than bilingual models in the translation from English into $16$ target languages.【2】 TST: Time-Sparse Transducer for Automatic Speech Recognition标题：TST：用于自动语音识别的时间稀疏换能器链接：https://arxiv.org/abs/2307.08323作者：Xiaohui Zhang,Mangui Liang,Zhengkun Tian,Jiangyan Yi,Jianhua Tao机构：Tao, School of Computer and Information Technology, Beijing Jiaotong University, Institute of Automation, Chinese Academy of Science, Beijing, China, Department of Automation, Tsinghua University, Beijing, China备注：None摘要：端到端模型，特别是递归神经网络传感器（RNN-T）在语音识别中取得了巨大的成功。然而，换能器在处理长解码序列时需要大量的存储器占用空间和计算时间。为了解决这个问题，我们提出了一种称为时间稀疏换能器的模型，该模型将时间稀疏机制引入到换能器中。在这种机制中，我们通过降低隐态的时间分辨率来获得中间表示。然后加权平均算法被用来组合这些表示到稀疏隐藏状态，然后由解码器。所有的实验都在汉语数据集AISHELL-1上进行。与RNN-T相比，时间稀疏传感器的特征误差率接近RNN-T，实时性为原传感器的50.00%。通过调整时间分辨率，时间稀疏传感器还可以将实时因子降低到原来的16.54%，代价是4.94%的精度损失。摘要：End-to-end model, especially Recurrent Neural Network Transducer (RNN-T), has achieved great success in speech recognition. However, transducer requires a great memory footprint and computing time when processing a long decoding sequence. To solve this problem, we propose a model named time-sparse transducer, which introduces a time-sparse mechanism into transducer. In this mechanism, we obtain the intermediate representations by reducing the time resolution of the hidden states. Then the weighted average algorithm is used to combine these representations into sparse hidden states followed by the decoder. All the experiments are conducted on a Mandarin dataset AISHELL-1. Compared with RNN-T, the character error rate of the time-sparse transducer is close to RNN-T and the real-time factor is 50.00% of the original. By adjusting the time resolution, the time-sparse transducer can also reduce the real-time factor to 16.54% of the original at the expense of a 4.94% loss of precision.【3】 BASS: Block-wise Adaptation for Speech Summarization标题：BASS：语音摘要的分块自适应算法链接：https://arxiv.org/abs/2307.08217作者：Roshan Sharma,Kenneth Zheng,Siddhant Arora,Shinji Watanabe,Rita Singh,Bhiksha Raj机构：Electrical and Computer Engineering, Language Technologies Institute , Carnegie Mellon University, USA备注：Accepted at Interspeech 2023摘要：端到端语音摘要已被证明可以在级联基线上提高性能。然而，由于计算限制，这样的模型很难在非常大的输入（几十分钟或小时）上训练，因此用截断的模型输入来训练。截断导致较差的模型，并且该问题的解决方案在于逐块建模，即，每次处理输入帧的一部分。在本文中，我们开发了一种方法，允许一个很长的序列上以增量的方式训练摘要模型。语音摘要被实现为流处理，其中假设摘要基于新的声学信息在每个块更新。我们设计和测试的策略，通过语义上下文跨块。How2数据集上的实验表明，所提出的分块训练方法提高了3个点的绝对ROUGE-L上的截断输入基线。摘要：End-to-end speech summarization has been shown to improve performance over cascade baselines. However, such models are difficult to train on very large inputs (dozens of minutes or hours) owing to compute restrictions and are hence trained with truncated model inputs. Truncation leads to poorer models, and a solution to this problem rests in block-wise modeling, i.e., processing a portion of the input frames at a time. In this paper, we develop a method that allows one to train summarization models on very long sequences in an incremental manner. Speech summarization is realized as a streaming process, where hypothesis summaries are updated every block based on new acoustic information. We devise and test strategies to pass semantic context across the blocks. Experiments on the How2 dataset demonstrate that the proposed block-wise training method improves by 3 points absolute on ROUGE-L over a truncated input baseline.【4】 Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Sound标题：利用声音元素对语音识别进行隐秘的后门攻击链接：https://arxiv.org/abs/2307.08208作者：Hanbo Cai,Pengcheng Zhang,Hai Dong,Yan Xiao,Stefanos Koffas,Yiming Li机构：Sun Yat-senUniversity备注：13 pages摘要：深度神经网络（DNN）已被广泛且成功地采用并部署在语音识别的各种应用中。最近，一些研究表明，这些模型容易受到后门攻击，攻击者可以通过毒化受害者模型的训练过程来将恶意预测行为植入受害者模型中。在本文中，我们重新审视毒药只有后门攻击语音识别。我们发现，现有的方法是不隐秘的，因为它们的触发模式是可察觉的人类或机器检测。这种限制主要是因为它们的触发模式是简单的噪声或可分离且独特的剪辑。受这些发现的启发，我们建议利用声音的元素（例如，音高和音色）来设计更隐蔽但有效的仅毒后门攻击。具体来说，我们插入一个短时间的高音调信号作为触发器，并增加剩余的音频剪辑的间距，以“掩盖”它的设计隐身间距为基础的触发器。我们操纵受害者音频的音色特征来设计基于音色的隐形攻击，并设计声纹选择模块来促进多后门攻击。我们的攻击可以产生更多的“天然”有毒样本，因此更隐蔽。在基准数据集上进行了大量的实验，验证了我们的攻击在不同设置下的有效性（$e.g.$，所有对一、所有对所有、干净标签、物理和多后门设置）及其隐蔽性。复制主要实验的代码可在\url{https：//github.com/HanboCai/BadSpeech_SoE}获得。摘要：Deep neural networks (DNNs) have been widely and successfully adopted and deployed in various applications of speech recognition. Recently, a few works revealed that these models are vulnerable to backdoor attacks, where the adversaries can implant malicious prediction behaviors into victim models by poisoning their training process. In this paper, we revisit poison-only backdoor attacks against speech recognition. We reveal that existing methods are not stealthy since their trigger patterns are perceptible to humans or machine detection. This limitation is mostly because their trigger patterns are simple noises or separable and distinctive clips. Motivated by these findings, we propose to exploit elements of sound ($e.g.$, pitch and timbre) to design more stealthy yet effective poison-only backdoor attacks. Specifically, we insert a short-duration high-pitched signal as the trigger and increase the pitch of remaining audio clips to `mask' it for designing stealthy pitch-based triggers. We manipulate timbre features of victim audios to design the stealthy timbre-based attack and design a voiceprint selection module to facilitate the multi-backdoor attack. Our attacks can generate more `natural' poisoned samples and therefore are more stealthy. Extensive experiments are conducted on benchmark datasets, which verify the effectiveness of our attacks under different settings ($e.g.$, all-to-one, all-to-all, clean-label, physical, and multi-backdoor settings) and their stealthiness. The code for reproducing main experiments are available at \url{https://github.com/HanboCai/BadSpeech_SoE}.【5】 NoiseBandNet: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks标题：NoiseBandNet：基于滤波器组的可控时变神经网络音效合成链接：https://arxiv.org/abs/2307.08007作者：Adrián Barahona-Ríos,Tom Collins机构：Frost School of Music, University of Miami备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：由于数据的潜在稀缺性和频谱-时间变化，声音效果的可控神经音频合成是一项具有挑战性的任务。差分数字信号处理（DDSP）合成器已经成功地用于使用相对有限的数据和计算资源来建模和控制音乐和谐波信号。在这里，我们提出了NoiseBandNet，一种能够通过滤波器组过滤白噪声来合成和控制声音效果的架构，从而比以前的系统更进一步，对声音的谐波性质作出假设。我们通过一系列的实验来评估我们的方法，建模的脚步声，雷雨，陶器，敲门声和金属声音效果。将NoiseBandNet音频重建能力与DDSP滤波噪声合成器的四种变体进行比较，NoiseBandNet在10个评估类别中有9个得分较高，从而建立了一种灵活的DDSP方法，用于生成任意长度的时变、非谐波声音效果，同时具有良好的时间和频率分辨率。最后，我们介绍了NoiseBandNet的一些潜在的创造性用途，通过生成变化，执行响度转移，并通过在用户定义的控制曲线上对其进行训练。摘要：Controllable neural audio synthesis of sound effects is a challenging task due to the potential scarcity and spectro-temporal variance of the data. Differentiable digital signal processing (DDSP) synthesisers have been successfully employed to model and control musical and harmonic signals using relatively limited data and computational resources. Here we propose NoiseBandNet, an architecture capable of synthesising and controlling sound effects by filtering white noise through a filterbank, thus going further than previous systems that make assumptions about the harmonic nature of sounds. We evaluate our approach via a series of experiments, modelling footsteps, thunderstorm, pottery, knocking, and metal sound effects. Comparing NoiseBandNet audio reconstruction capabilities to four variants of the DDSP-filtered noise synthesiser, NoiseBandNet scores higher in nine out of ten evaluation categories, establishing a flexible DDSP method for generating time-varying, inharmonic sound effects of arbitrary length with both good time and frequency resolution. Finally, we introduce some potential creative uses of NoiseBandNet, by generating variations, performing loudness transfer, and by training it on user-defined control curves.【6】 Single and Multi-Speaker Cloned Voice Detection: From Perceptual to Learned Features标题：单说话人和多说话人克隆语音检测：从感知特征到学习特征链接：https://arxiv.org/abs/2307.07683作者：Sarah Barrington,Romit Barua,Gautham Koorma,Hany Farid机构：School of Information, Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA USA摘要：近年来，合成语音克隆技术取得了重大进展，带来了一系列潜在危害。从小规模和大规模的金融欺诈到虚假信息活动，需要可靠的方法来区分真实和合成的声音。我们描述了三种技术，用于区分一个真正的克隆的声音，旨在模仿一个特定的人。这三种方法在其特征提取阶段不同，其中低维感知特征提供高可解释性但较低准确性，通用光谱特征，以及端到端学习特征提供较少可解释性但较高准确性。我们展示了这些方法的有效性时，一个单一的扬声器的声音和多个声音训练时。学习的功能始终产生相等的错误率在$0\%$和$4\%$之间，并且对对抗性洗钱相当稳健。摘要：Synthetic-voice cloning technologies have seen significant advances in recent years, giving rise to a range of potential harms. From small- and large-scale financial fraud to disinformation campaigns, the need for reliable methods to differentiate real and synthesized voices is imperative. We describe three techniques for differentiating a real from a cloned voice designed to impersonate a specific person. These three approaches differ in their feature extraction stage with low-dimensional perceptual features offering high interpretability but lower accuracy, to generic spectral features, and end-to-end learned features offering less interpretability but higher accuracy. We show the efficacy of these approaches when trained on a single speaker's voice and when trained on multiple voices. The learned features consistently yield an equal error rate between $0\%$ and $4\%$, and are reasonably robust to adversarial laundering.【7】 Corticomorphic Hybrid CNN-SNN Architecture for EEG-based Low-footprint Low-latency Auditory Attention Detection标题：基于皮质形态混合CNN-SNN结构的EEG低足迹低潜伏期听觉注意检测链接：https://arxiv.org/abs/2307.08501作者：Richard Gall,Deniz Kocanaogullari,Murat Akcakaya,Deniz Erdogmus,Rajkumar Kubendran机构：Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, Northeastern University, Boston, MA, USA摘要：在多说话者“鸡尾酒会”场景中，听众可以选择性地关注感兴趣的说话者。对人类听觉注意力网络的研究表明，皮层夹带到语音包络导致高度相关的脑电图（EEG）测量。基于EEG的听觉注意力检测（AAD）使用人工神经网络（ANN）的当前趋势对于边缘计算平台是不实用的，这是由于使用多个EEG通道的较长的决策窗口，具有更高的功耗和更大的内存占用要求。人工神经网络也不能准确地模拟大脑的自上而下的注意力网络，因为皮层组织是复杂的和层次的。在本文中，我们提出了一种混合卷积神经网络尖峰神经网络（CNN-SNN）皮质结构，灵感来自听觉皮层，它使用EEG数据与多扬声器语音包络，成功解码听觉注意低延迟至1秒，仅使用8个EEG电极战略性地放置在听觉皮层附近，在一个显着更高的准确率91.03%，与最先进的相比。同时，与传统的CNN参考模型相比，我们的模型在较低的位精度下使用的参数少了约15%，从而减少了约57%的内存占用。研究结果显示，大脑嵌入式设备（如智能助听器）中的边缘计算有很大的前景。摘要：In a multi-speaker "cocktail party" scenario, a listener can selectively attend to a speaker of interest. Studies into the human auditory attention network demonstrate cortical entrainment to speech envelopes resulting in highly correlated Electroencephalography (EEG) measurements. Current trends in EEG-based auditory attention detection (AAD) using artificial neural networks (ANN) are not practical for edge-computing platforms due to longer decision windows using several EEG channels, with higher power consumption and larger memory footprint requirements. Nor are ANNs capable of accurately modeling the brain's top-down attention network since the cortical organization is complex and layer. In this paper, we propose a hybrid convolutional neural network-spiking neural network (CNN-SNN) corticomorphic architecture, inspired by the auditory cortex, which uses EEG data along with multi-speaker speech envelopes to successfully decode auditory attention with low latency down to 1 second, using only 8 EEG electrodes strategically placed close to the auditory cortex, at a significantly higher accuracy of 91.03%, compared to the state-of-the-art. Simultaneously, when compared to a traditional CNN reference model, our model uses ~15% fewer parameters at a lower bit precision resulting in ~57% memory footprint reduction. The results show great promise for edge-computing in brain-embedded devices, like smart hearing aids.【8】 Vocoder drift compensation by x-vector alignment in speaker anonymisation标题：说话人匿名化中基于x向量对齐的声码器漂移补偿链接：https://arxiv.org/abs/2307.08403作者：Michele Panariello,Massimiliano Todisco,Nicholas Evans机构：EURECOM, Sophia Antipolis, France备注：Accepted at the ISCA SPSC Symposium 2023摘要：对于最流行的基于X向量的说话者匿名化方法，大部分的匿名化可以源于声码，而不是来自用于用虚拟伪说话者的X向量替换原始说话者X向量的核心匿名化函数。这种现象可能阻碍更好的匿名化系统的设计，因为缺乏对x向量空间的细粒度控制。本文报道的工作探讨了所谓的声码器漂移的起源，并表明，这是由于取代的x-向量和原始的语言内容，语调和韵律表示之间的不匹配。还报道了声码器漂移补偿的原始方法。虽然匿名化性能下降，如预期，补偿减少了声码器漂移，提供了改进的控制x向量空间，并奠定了基础，更好的匿名化功能在未来的设计。摘要：For the most popular x-vector-based approaches to speaker anonymisation, the bulk of the anonymisation can stem from vocoding rather than from the core anonymisation function which is used to substitute an original speaker x-vector with that of a fictitious pseudo-speaker. This phenomenon can impede the design of better anonymisation systems since there is a lack of fine-grained control over the x-vector space. The work reported in this paper explores the origin of so-called vocoder drift and shows that it is due to the mismatch between the substituted x-vector and the original representations of the linguistic content, intonation and prosody. Also reported is an original approach to vocoder drift compensation. While anonymisation performance degrades as expected, compensation reduces vocoder drift substantially, offers improved control over the x-vector space and lays a foundation for the design of better anonymisation functions in the future.【9】 Exploring Binary Classification Loss For Speaker Verification标题：用于说话人确认的二值分类丢失问题研究链接：https://arxiv.org/abs/2307.08205作者：Bing Han,Zhengyang Chen,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted by ICASSP 2023摘要：闭集训练和开集测试的不匹配导致说话人验证任务的性能显著下降。对于现有的损失函数，度量学习为基础的目标强烈依赖于搜索有效的配对，这可能会阻碍进一步的改进。而流行的多分类方法在对看不见的说话人进行评估时通常会出现退化。在这项工作中，我们介绍了SphereFace2框架，它使用几个二进制分类器来训练说话人模型在成对的方式，而不是执行多分类。受益于这种学习范式，它可以有效地缓解培训和评估之间的差距。在Voxceleb上进行的实验表明，SphereFace2优于其他现有的损失函数，特别是在硬试验中。此外，大裕度微调策略被证明是兼容的，它的进一步改进。最后，SphereFace2还显示了其对类噪声标签的强大鲁棒性，这有可能应用于具有不准确估计伪标签的半监督训练场景。代码可在https://github.com/Hunterhuan/sphereface2_speaker_verification摘要：The mismatch between close-set training and open-set testing usually leads to significant performance degradation for speaker verification task. For existing loss functions, metric learning-based objectives depend strongly on searching effective pairs which might hinder further improvements. And popular multi-classification methods are usually observed with degradation when evaluated on unseen speakers. In this work, we introduce SphereFace2 framework which uses several binary classifiers to train the speaker model in a pair-wise manner instead of performing multi-classification. Benefiting from this learning paradigm, it can efficiently alleviate the gap between training and evaluation. Experiments conducted on Voxceleb show that the SphereFace2 outperforms other existing loss functions, especially on hard trials. Besides, large margin fine-tuning strategy is proven to be compatible with it for further improvements. Finally, SphereFace2 also shows its strong robustness to class-wise noisy labels which has the potential to be applied in the semi-supervised training scenario with inaccurate estimated pseudo labels. Codes are available in https://github.com/Hunterhuan/sphereface2_speaker_verification【10】 Noise-aware Speech Enhancement using Diffusion Probabilistic Model标题：基于扩散概率模型的噪声感知语音增强链接：https://arxiv.org/abs/2307.08029作者：Yuchen Hu,Chen Chen,Ruizhe Li,Qiushi Zhu,Eng Siong Chng机构：Nanyang Technological University, Singapore, University of Aberdeen, UK, University of Science and Technology of China, China备注：5 pages, 2 figures摘要：随着扩散模型的发展，生成语音增强技术（generative speech enhancement，SE）因其对未知测试噪声的巨大潜力而受到了广泛的关注。然而，现有的努力主要集中在固有属性的干净语音推理，未充分开发的变化的噪声信息在现实世界的条件。在本文中，我们提出了一种噪声感知语音增强（NASE）方法，提取噪声特定的信息，以指导扩散模型的反向过程。具体而言，我们设计了一个噪声分类（NC）模型，产生声学嵌入作为噪声调节器，用于指导反向去噪过程。同时，设计了一种多任务学习方案，联合优化SE和NC任务，以提高提取的噪声条件的噪声特异性。我们提出的NASE是一个即插即用的模块，可以推广到任何扩散SE模型。在VoiceBank-DEMAND数据集上的实验结果表明，NASE算法比多种主流的扩散SE模型有了显著的改进，尤其是在未知测试噪声方面。摘要：With recent advances of diffusion model, generative speech enhancement (SE) has attracted a surge of research interest due to its great potential for unseen testing noises. However, existing efforts mainly focus on inherent properties of clean speech for inference, underexploiting the varying noise information in real-world conditions. In this paper, we propose a noise-aware speech enhancement (NASE) approach that extracts noise-specific information to guide the reverse process in diffusion model. Specifically, we design a noise classification (NC) model to produce acoustic embedding as a noise conditioner for guiding the reverse denoising process. Meanwhile, a multi-task learning scheme is devised to jointly optimize SE and NC tasks, in order to enhance the noise specificity of extracted noise conditioner. Our proposed NASE is shown to be a plug-and-play module that can be generalized to any diffusion SE models. Experiment evidence on VoiceBank-DEMAND dataset shows that NASE achieves significant improvement over multiple mainstream diffusion SE models, especially on unseen testing noises.eess.AS音频处理【1】 Semi-supervised multi-channel speaker diarization with cross-channel attention标题：具有跨声道注意的半监督多声道说话人二值化链接：https://arxiv.org/abs/2307.08688作者：Shilong Wu,Jun Du,Maokui He,Shutong Niu,Hang Chen,Haitao Tang,Chin-Hui Lee机构：University of Science and Technology of China, China, iFlytek, China , Georgia Institute of Technology, USA备注：8 pages,3 figures摘要：大多数神经说话人日记系统依赖于足够的人工训练数据标签，这是很难收集在现实世界的场景。本文提出了一种半监督的说话人日志化系统，利用大规模的多通道的训练数据生成伪标签的未标记的数据。此外，我们将跨通道注意力引入到使用记忆感知多说话人嵌入的神经说话人日记（NSD-MA-MSE）中，以更好地学习说话人嵌入的通道上下文信息。在仅包含训练集的部分说话人标签的CHiME-7 Mixer 6数据集上的实验结果表明，与开发集上的基于聚类的模型相比，我们的系统实现了57.01%的相对DER降低。我们进一步在CHiME-6数据集上进行实验，以模拟缺失部分训练集标签的场景。当使用80%和50%标记的训练数据时，我们的系统与使用100%标记的数据进行训练所获得的结果相当。摘要：Most neural speaker diarization systems rely on sufficient manual training data labels, which are hard to collect under real-world scenarios. This paper proposes a semi-supervised speaker diarization system to utilize large-scale multi-channel training data by generating pseudo-labels for unlabeled data. Furthermore, we introduce cross-channel attention into the Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding (NSD-MA-MSE) to learn channel contextual information of speaker embeddings better. Experimental results on the CHiME-7 Mixer6 dataset which only contains partial speakers' labels of the training set, show that our system achieved 57.01% relative DER reduction compared to the clustering-based model on the development set. We further conducted experiments on the CHiME-6 dataset to simulate the scenario of missing partial training set labels. When using 80% and 50% labeled training data, our system performs comparably to the results obtained using 100% labeled data for training.【2】 Corticomorphic Hybrid CNN-SNN Architecture for EEG-based Low-footprint Low-latency Auditory Attention Detection标题：基于皮质形态混合CNN-SNN结构的EEG低足迹低潜伏期听觉注意检测链接：https://arxiv.org/abs/2307.08501作者：Richard Gall,Deniz Kocanaogullari,Murat Akcakaya,Deniz Erdogmus,Rajkumar Kubendran机构：Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, Northeastern University, Boston, MA, USA摘要：在多说话者“鸡尾酒会”场景中，听众可以选择性地关注感兴趣的说话者。对人类听觉注意力网络的研究表明，皮层夹带到语音包络导致高度相关的脑电图（EEG）测量。基于EEG的听觉注意力检测（AAD）使用人工神经网络（ANN）的当前趋势对于边缘计算平台是不实用的，这是由于使用多个EEG通道的较长的决策窗口，具有更高的功耗和更大的内存占用要求。人工神经网络也不能准确地模拟大脑的自上而下的注意力网络，因为皮层组织是复杂的和层次的。在本文中，我们提出了一种混合卷积神经网络尖峰神经网络（CNN-SNN）皮质结构，灵感来自听觉皮层，它使用EEG数据与多扬声器语音包络，成功解码听觉注意低延迟至1秒，仅使用8个EEG电极战略性地放置在听觉皮层附近，在一个显着更高的准确率91.03%，与最先进的相比。同时，与传统的CNN参考模型相比，我们的模型在较低的位精度下使用的参数少了约15%，从而减少了约57%的内存占用。研究结果显示，大脑嵌入式设备（如智能助听器）中的边缘计算有很大的前景。摘要：In a multi-speaker "cocktail party" scenario, a listener can selectively attend to a speaker of interest. Studies into the human auditory attention network demonstrate cortical entrainment to speech envelopes resulting in highly correlated Electroencephalography (EEG) measurements. Current trends in EEG-based auditory attention detection (AAD) using artificial neural networks (ANN) are not practical for edge-computing platforms due to longer decision windows using several EEG channels, with higher power consumption and larger memory footprint requirements. Nor are ANNs capable of accurately modeling the brain's top-down attention network since the cortical organization is complex and layer. In this paper, we propose a hybrid convolutional neural network-spiking neural network (CNN-SNN) corticomorphic architecture, inspired by the auditory cortex, which uses EEG data along with multi-speaker speech envelopes to successfully decode auditory attention with low latency down to 1 second, using only 8 EEG electrodes strategically placed close to the auditory cortex, at a significantly higher accuracy of 91.03%, compared to the state-of-the-art. Simultaneously, when compared to a traditional CNN reference model, our model uses ~15% fewer parameters at a lower bit precision resulting in ~57% memory footprint reduction. The results show great promise for edge-computing in brain-embedded devices, like smart hearing aids.【3】 Vocoder drift compensation by x-vector alignment in speaker anonymisation标题：说话人匿名化中基于x向量对齐的声码器漂移补偿链接：https://arxiv.org/abs/2307.08403作者：Michele Panariello,Massimiliano Todisco,Nicholas Evans机构：EURECOM, Sophia Antipolis, France备注：Accepted at the ISCA SPSC Symposium 2023摘要：对于最流行的基于X向量的说话者匿名化方法，大部分的匿名化可以源于声码，而不是来自用于用虚拟伪说话者的X向量替换原始说话者X向量的核心匿名化函数。这种现象可能阻碍更好的匿名化系统的设计，因为缺乏对x向量空间的细粒度控制。本文报道的工作探讨了所谓的声码器漂移的起源，并表明，这是由于取代的x-向量和原始的语言内容，语调和韵律表示之间的不匹配。还报道了声码器漂移补偿的原始方法。虽然匿名化性能下降，如预期，补偿减少了声码器漂移，提供了改进的控制x向量空间，并奠定了基础，更好的匿名化功能在未来的设计。摘要：For the most popular x-vector-based approaches to speaker anonymisation, the bulk of the anonymisation can stem from vocoding rather than from the core anonymisation function which is used to substitute an original speaker x-vector with that of a fictitious pseudo-speaker. This phenomenon can impede the design of better anonymisation systems since there is a lack of fine-grained control over the x-vector space. The work reported in this paper explores the origin of so-called vocoder drift and shows that it is due to the mismatch between the substituted x-vector and the original representations of the linguistic content, intonation and prosody. Also reported is an original approach to vocoder drift compensation. While anonymisation performance degrades as expected, compensation reduces vocoder drift substantially, offers improved control over the x-vector space and lays a foundation for the design of better anonymisation functions in the future.【4】 Dynamic Kernel Convolution Network with Scene-dedicate Training for Sound Event Localization and Detection标题：具有场景专用训练的动态核卷积网络用于声事件定位和检测链接：https://arxiv.org/abs/2307.08239作者：Siwei Huang,Jianfeng Chen,Jisheng Bai,Yafei Jia,Dongzhe Zhang机构：Zhang备注：11 pages, 6 figures摘要：基于DNN的方法在声音事件定位和检测（SELD）中显示出高性能。而在真实的空间声音场景中，混响和各种声音事件的不平衡存在增加了SELD任务的复杂性。在本文中，我们提出了一个有效的空间场景SELD系统，在我们的方法中，在卷积块之后引入动态核卷积模块，以适应性地建模不同的通道特征。其次，我们结合SELDnet和EINv 2框架到建议的SELD系统与多轨道ACCDOA。此外，两个场景专用的策略被引入到训练阶段，以提高系统在现实的空间声音场景的泛化。最后，我们应用数据增强的方法来扩展数据集，使用通道旋转，空间数据合成。在Sony-TAu Realistic Spatial Soundscapes 2022数据集上，采用四种联合度量对SELD系统的性能进行了评估，实验结果表明，该系统优于固定核卷积SELD系统。此外，所提出的系统实现了0.348在DCASE SELD任务的SELD分数，并超过了SOTA方法。摘要：DNN-based methods have shown high performance in sound event localization and detection(SELD). While in real spatial sound scenes, reverberation and the imbalanced presence of various sound events increase the complexity of the SELD task. In this paper, we propose an effective SELD system in real spatial scenes.In our approach, a dynamic kernel convolution module is introduced after the convolution blocks to adaptively model the channel-wise features with different receptive fields. Secondly, we incorporate the SELDnet and EINv2 framework into the proposed SELD system with multi-track ACCDOA. Moreover, two scene-dedicated strategies are introduced into the training stage to improve the generalization of the system in realistic spatial sound scenes. Finally, we apply data augmentation methods to extend the dataset using channel rotation, spatial data synthesis. Four joint metrics are used to evaluate the performance of the SELD system on the Sony-TAu Realistic Spatial Soundscapes 2022 dataset.Experimental results show that the proposed systems outperform the fixed-kernel convolution SELD systems. In addition, the proposed system achieved an SELD score of 0.348 in the DCASE SELD task and surpassed the SOTA methods.【5】 Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition标题：用于完全格式化的端到端语音识别的语音适应大语言模型链接：https://arxiv.org/abs/2307.08234作者：Shaoshi Ling,Yuxuan Hu,Shuangbei Qian,Guoli Ye,Yao Qian,Yifan Gong,Ed Lin,Michael Zeng机构：Microsoft Could and AI摘要：大多数端到端（E2 E）语音识别模型由执行声学和语言建模功能的编码器和解码器块组成。预训练的大型语言模型（LLM）有潜力提高E2 E ASR的性能。然而，将预训练的语言模型集成到E2 E语音识别模型中已经示出了有限的益处，这是由于基于文本的LLM和在E2 E ASR中使用的LLM之间的不匹配。在本文中，我们探索了一种替代的方法，通过适应预训练的LLM语音。我们在不同领域的全格式E2 E ASR转录任务的实验表明，我们的方法可以有效地利用预先训练的LLM的优势，以产生更可读的ASR转录。我们的模型，这是基于预训练的大型语言模型的编码器-解码器或解码器唯一的结构，超过强大的ASR模型，如Whisper，在识别错误率方面，考虑到格式，标点符号和大写以及。摘要：Most end-to-end (E2E) speech recognition models are composed of encoder and decoder blocks that perform acoustic and language modeling functions. Pretrained large language models (LLMs) have the potential to improve the performance of E2E ASR. However, integrating a pretrained language model into an E2E speech recognition model has shown limited benefits due to the mismatches between text-based LLMs and those used in E2E ASR. In this paper, we explore an alternative approach by adapting a pretrained LLMs to speech. Our experiments on fully-formatted E2E ASR transcription tasks across various domains demonstrate that our approach can effectively leverage the strengths of pretrained LLMs to produce more readable ASR transcriptions. Our model, which is based on the pretrained large language models with either an encoder-decoder or decoder-only structure, surpasses strong ASR models such as Whisper, in terms of recognition error rate, considering formats like punctuation and capitalization as well.【6】 Exploring Binary Classification Loss For Speaker Verification标题：用于说话人确认的二值分类丢失问题研究链接：https://arxiv.org/abs/2307.08205作者：Bing Han,Zhengyang Chen,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted by ICASSP 2023摘要：闭集训练和开集测试的不匹配导致说话人验证任务的性能显著下降。对于现有的损失函数，度量学习为基础的目标强烈依赖于搜索有效的配对，这可能会阻碍进一步的改进。而流行的多分类方法在对看不见的说话人进行评估时通常会出现退化。在这项工作中，我们介绍了SphereFace2框架，它使用几个二进制分类器来训练说话人模型在成对的方式，而不是执行多分类。受益于这种学习范式，它可以有效地缓解培训和评估之间的差距。在Voxceleb上进行的实验表明，SphereFace2优于其他现有的损失函数，特别是在硬试验中。此外，大裕度微调策略被证明是兼容的，它的进一步改进。最后，SphereFace2还显示了其对类噪声标签的强大鲁棒性，这有可能应用于具有不准确估计伪标签的半监督训练场景。代码可在https://github.com/Hunterhuan/sphereface2_speaker_verification摘要：The mismatch between close-set training and open-set testing usually leads to significant performance degradation for speaker verification task. For existing loss functions, metric learning-based objectives depend strongly on searching effective pairs which might hinder further improvements. And popular multi-classification methods are usually observed with degradation when evaluated on unseen speakers. In this work, we introduce SphereFace2 framework which uses several binary classifiers to train the speaker model in a pair-wise manner instead of performing multi-classification. Benefiting from this learning paradigm, it can efficiently alleviate the gap between training and evaluation. Experiments conducted on Voxceleb show that the SphereFace2 outperforms other existing loss functions, especially on hard trials. Besides, large margin fine-tuning strategy is proven to be compatible with it for further improvements. Finally, SphereFace2 also shows its strong robustness to class-wise noisy labels which has the potential to be applied in the semi-supervised training scenario with inaccurate estimated pseudo labels. Codes are available in https://github.com/Hunterhuan/sphereface2_speaker_verification【7】 Noise-aware Speech Enhancement using Diffusion Probabilistic Model标题：基于扩散概率模型的噪声感知语音增强链接：https://arxiv.org/abs/2307.08029作者：Yuchen Hu,Chen Chen,Ruizhe Li,Qiushi Zhu,Eng Siong Chng机构：Nanyang Technological University, Singapore, University of Aberdeen, UK, University of Science and Technology of China, China备注：5 pages, 2 figures摘要：随着扩散模型的发展，生成语音增强技术（generative speech enhancement，SE）因其对未知测试噪声的巨大潜力而受到了广泛的关注。然而，现有的努力主要集中在固有属性的干净语音推理，未充分开发的变化的噪声信息在现实世界的条件。在本文中，我们提出了一种噪声感知语音增强（NASE）方法，提取噪声特定的信息，以指导扩散模型的反向过程。具体而言，我们设计了一个噪声分类（NC）模型，产生声学嵌入作为噪声调节器，用于指导反向去噪过程。同时，设计了一种多任务学习方案，联合优化SE和NC任务，以提高提取的噪声条件的噪声特异性。我们提出的NASE是一个即插即用的模块，可以推广到任何扩散SE模型。在VoiceBank-DEMAND数据集上的实验结果表明，NASE算法比多种主流的扩散SE模型有了显著的改进，尤其是在未知测试噪声方面。摘要：With recent advances of diffusion model, generative speech enhancement (SE) has attracted a surge of research interest due to its great potential for unseen testing noises. However, existing efforts mainly focus on inherent properties of clean speech for inference, underexploiting the varying noise information in real-world conditions. In this paper, we propose a noise-aware speech enhancement (NASE) approach that extracts noise-specific information to guide the reverse process in diffusion model. Specifically, we design a noise classification (NC) model to produce acoustic embedding as a noise conditioner for guiding the reverse denoising process. Meanwhile, a multi-task learning scheme is devised to jointly optimize SE and NC tasks, in order to enhance the noise specificity of extracted noise conditioner. Our proposed NASE is shown to be a plug-and-play module that can be generalized to any diffusion SE models. Experiment evidence on VoiceBank-DEMAND dataset shows that NASE achieves significant improvement over multiple mainstream diffusion SE models, especially on unseen testing noises.【8】 Model Adaptation for ASR in low-resource Indian Languages标题：ASR在低资源的印度语言中的模型适配链接：https://arxiv.org/abs/2307.07948作者：Abhayjeet Singh,Arjun Singh Mehta,Ashish Khuraishi K S,Deekshitha G,Gauri Date,Jai Nanavati,Jesuraja Bandekar,Karnalius Basumatary,Karthika P,Sandhya Badiger,Sathvik Udupa,Saurabh Kumar,Savitha,Prasanta Kumar Ghosh,Prashanthi V,Priyanka Pai,Raoul Nanavati,Rohan Saxena,Sai Praneeth Reddy Mora,Srinivasa Raghavan机构：Department of Electrical Engineering, Indian Institute of Science (IISc), Bangalore-, India, Navana Tech (NT), Nanavati Mahalaya , Homi Mody St. Fort, Mumbai-, India备注：ASRU Special session overview paper摘要：近年来，自动语音识别（ASR）性能得到了大幅提高，主要是通过基于自监督学习（SSL）的声学模型（如wav2vec2）和大规模多语言训练（如Whisper）实现的。对于低资源语言来说，音频和文本的可用性都有限，仍然存在巨大的挑战。这种情况因印度语中的多种方言而变得更加复杂。然而，许多印度语言可以归入相同的家族，共享相同的脚本和语法结构。这就是可以应用大量的适应和微调技术来克服数据的低资源性质，通过利用资源丰富的类似语言。 在这样的场景中，重要的是要理解每种模态（如声学和文本）在构建可靠的ASR中的重要程度。这可能是这样的情况，在一种语言中丰富的声学数据减少了对大型纯文本语料库的需求。或者，由于各种预先训练的声学模型的可用性，反之亦然也可能是真的。在这次拟议的特别会议上，我们鼓励社区利用孟加拉语和Bhojpuri两种低资源印度语言的数据来探索这些想法。这些方法不限于印度语言，解决方案可能适用于世界各地使用的各种语言。摘要：Automatic speech recognition (ASR) performance has improved drastically in recent years, mainly enabled by self-supervised learning (SSL) based acoustic models such as wav2vec2 and large-scale multi-lingual training like Whisper. A huge challenge still exists for low-resource languages where the availability of both audio and text is limited. This is further complicated by the presence of multiple dialects like in Indian languages. However, many Indian languages can be grouped into the same families and share the same script and grammatical structure. This is where a lot of adaptation and fine-tuning techniques can be applied to overcome the low-resource nature of the data by utilising well-resourced similar languages. In such scenarios, it is important to understand the extent to which each modality, like acoustics and text, is important in building a reliable ASR. It could be the case that an abundance of acoustic data in a language reduces the need for large text-only corpora. Or, due to the availability of various pretrained acoustic models, the vice-versa could also be true. In this proposed special session, we encourage the community to explore these ideas with the data in two low-resource Indian languages of Bengali and Bhojpuri. These approaches are not limited to Indian languages, the solutions are potentially applicable to various languages spoken around the world.【9】 Audio-Visual Speech Enhancement Using Self-supervised Learning to Improve Speech Intelligibility in Cochlear Implant Simulations标题：基于自监督学习的人工耳蜗声清晰度语音增强链接：https://arxiv.org/abs/2307.07748作者：Richard Lee Lai,Jen-Cheng Hou,Mandar Gogate,Kia Dashtipour,Amir Hussain,Yu Tsao机构：Member, IEEE摘要：患有听力障碍的个体在理解语音的能力方面面临挑战，特别是在嘈杂的环境中。本研究的目的是探讨视听语音增强（AVSE）在提高声码语音在人工耳蜗（CI）模拟中的可懂度的有效性。值得注意的是，该研究侧重于具有挑战性的场景，其中AVSE任务的训练数据的可用性有限。为了解决这个问题，我们提出了一种新的深度神经网络框架，称为基于自我监督学习的AVSE（SSL-AVSE）。所提出的SSL-AVSE相结合的视觉线索，如嘴唇和嘴部的运动，从目标扬声器与相应的音频信号。然后将上下文组合的音频和视频数据馈送到基于变换器的SSL AV-HuBERT模型中以提取特征，使用基于BLSTM的SE模型进一步处理这些特征。结果显示了几个关键的发现。首先，SSL-AVSE通过利用AV-HuBERT模型成功地克服了数据有限的问题。其次，通过微调AV-HuBERT模型参数的目标SE任务，实现了显着的性能改进。具体而言，PESQ（语音质量的感知评估）从1.43提高到1.67，STOI（短时客观清晰度）从0.70提高到0.74。此外，SSL-AVSE的性能进行了评估，使用CI声码语音来评估CI用户的可懂度。比较实验结果表明，在人类对话过程中遇到的动态噪声的存在下，SSL-AVSE表现出了很大的改善。NCM（正态相关矩阵）值表明与噪声基线相比增加了26.5%至87.2%。摘要：Individuals with hearing impairments face challenges in their ability to comprehend speech, particularly in noisy environments. The aim of this study is to explore the effectiveness of audio-visual speech enhancement (AVSE) in enhancing the intelligibility of vocoded speech in cochlear implant (CI) simulations. Notably, the study focuses on a challenged scenario where there is limited availability of training data for the AVSE task. To address this problem, we propose a novel deep neural network framework termed Self-Supervised Learning-based AVSE (SSL-AVSE). The proposed SSL-AVSE combines visual cues, such as lip and mouth movements, from the target speakers with corresponding audio signals. The contextually combined audio and visual data are then fed into a Transformer-based SSL AV-HuBERT model to extract features, which are further processed using a BLSTM-based SE model. The results demonstrate several key findings. Firstly, SSL-AVSE successfully overcomes the issue of limited data by leveraging the AV-HuBERT model. Secondly, by fine-tuning the AV-HuBERT model parameters for the target SE task, significant performance improvements are achieved. Specifically, there is a notable enhancement in PESQ (Perceptual Evaluation of Speech Quality) from 1.43 to 1.67 and in STOI (Short-Time Objective Intelligibility) from 0.70 to 0.74. Furthermore, the performance of the SSL-AVSE was evaluated using CI vocoded speech to assess the intelligibility for CI users. Comparative experimental outcomes reveal that in the presence of dynamic noises encountered during human conversations, SSL-AVSE exhibits a substantial improvement. The NCM (Normal Correlation Matrix) values indicate an increase of 26.5% to 87.2% compared to the noisy baseline.【10】 Multilingual Speech-to-Speech Translation into Multiple Target Languages标题：多语言语音到多目标语言的翻译链接：https://arxiv.org/abs/2307.08655作者：Hongyu Gong,Ning Dong,Sravya Popuri,Vedanuj Goswami,Ann Lee,Juan Pino机构：Meta AI Research, USA摘要：语音到语音翻译（S2 ST）使人们能够在用不同语言交谈的人之间进行口头交流。尽管有一些关于多语言S2 ST的研究，但他们的重点是源端的多语言性，即，从多种源语言到一种目标语言的翻译。我们提出了支持多种目标语言的多语言S2 ST的第一个工作。利用语音到单元和声码器的直接S2 ST的最新进展，我们为这些关键组件配备了多语言功能。语音掩蔽单元（Speech-to-Masked-unit，S2 MU）是S2 U的多语言扩展，它对不属于目标语言的单元进行掩蔽，以减少语言干扰。我们还提出了多语言声码器，这是训练语言嵌入和辅助语言识别损失。在基准翻译测试集，我们提出的多语言模型显示出优越的性能比双语模型从英语翻译成16美元的目标语言。摘要：Speech-to-speech translation (S2ST) enables spoken communication between people talking in different languages. Despite a few studies on multilingual S2ST, their focus is the multilinguality on the source side, i.e., the translation from multiple source languages to one target language. We present the first work on multilingual S2ST supporting multiple target languages. Leveraging recent advance in direct S2ST with speech-to-unit and vocoder, we equip these key components with multilingual capability. Speech-to-masked-unit (S2MU) is the multilingual extension of S2U, which applies masking to units which don't belong to the given target language to reduce the language interference. We also propose multilingual vocoder which is trained with language embedding and the auxiliary loss of language identification. On benchmark translation testsets, our proposed multilingual model shows superior performance than bilingual models in the translation from English into $16$ target languages.【11】 TST: Time-Sparse Transducer for Automatic Speech Recognition标题：TST：用于自动语音识别的时间稀疏换能器链接：https://arxiv.org/abs/2307.08323作者：Xiaohui Zhang,Mangui Liang,Zhengkun Tian,Jiangyan Yi,Jianhua Tao机构：Tao, School of Computer and Information Technology, Beijing Jiaotong University, Institute of Automation, Chinese Academy of Science, Beijing, China, Department of Automation, Tsinghua University, Beijing, China备注：None摘要：端到端模型，特别是递归神经网络传感器（RNN-T）在语音识别中取得了巨大的成功。然而，换能器在处理长解码序列时需要大量的存储器占用空间和计算时间。为了解决这个问题，我们提出了一种称为时间稀疏换能器的模型，该模型将时间稀疏机制引入到换能器中。在这种机制中，我们通过降低隐态的时间分辨率来获得中间表示。然后加权平均算法被用来组合这些表示到稀疏隐藏状态，然后由解码器。所有的实验都在汉语数据集AISHELL-1上进行。与RNN-T相比，时间稀疏传感器的特征误差率接近RNN-T，实时性为原传感器的50.00%。通过调整时间分辨率，时间稀疏传感器还可以将实时因子降低到原来的16.54%，代价是4.94%的精度损失。摘要：End-to-end model, especially Recurrent Neural Network Transducer (RNN-T), has achieved great success in speech recognition. However, transducer requires a great memory footprint and computing time when processing a long decoding sequence. To solve this problem, we propose a model named time-sparse transducer, which introduces a time-sparse mechanism into transducer. In this mechanism, we obtain the intermediate representations by reducing the time resolution of the hidden states. Then the weighted average algorithm is used to combine these representations into sparse hidden states followed by the decoder. All the experiments are conducted on a Mandarin dataset AISHELL-1. Compared with RNN-T, the character error rate of the time-sparse transducer is close to RNN-T and the real-time factor is 50.00% of the original. By adjusting the time resolution, the time-sparse transducer can also reduce the real-time factor to 16.54% of the original at the expense of a 4.94% loss of precision.【12】 BASS: Block-wise Adaptation for Speech Summarization标题：BASS：语音摘要的分块自适应算法链接：https://arxiv.org/abs/2307.08217作者：Roshan Sharma,Kenneth Zheng,Siddhant Arora,Shinji Watanabe,Rita Singh,Bhiksha Raj机构：Electrical and Computer Engineering, Language Technologies Institute , Carnegie Mellon University, USA备注：Accepted at Interspeech 2023摘要：端到端语音摘要已被证明可以在级联基线上提高性能。然而，由于计算限制，这样的模型很难在非常大的输入（几十分钟或小时）上训练，因此用截断的模型输入来训练。截断导致较差的模型，并且该问题的解决方案在于逐块建模，即，每次处理输入帧的一部分。在本文中，我们开发了一种方法，允许一个很长的序列上以增量的方式训练摘要模型。语音摘要被实现为流处理，其中假设摘要基于新的声学信息在每个块更新。我们设计和测试的策略，通过语义上下文跨块。How2数据集上的实验表明，所提出的分块训练方法提高了3个点的绝对ROUGE-L上的截断输入基线。摘要：End-to-end speech summarization has been shown to improve performance over cascade baselines. However, such models are difficult to train on very large inputs (dozens of minutes or hours) owing to compute restrictions and are hence trained with truncated model inputs. Truncation leads to poorer models, and a solution to this problem rests in block-wise modeling, i.e., processing a portion of the input frames at a time. In this paper, we develop a method that allows one to train summarization models on very long sequences in an incremental manner. Speech summarization is realized as a streaming process, where hypothesis summaries are updated every block based on new acoustic information. We devise and test strategies to pass semantic context across the blocks. Experiments on the How2 dataset demonstrate that the proposed block-wise training method improves by 3 points absolute on ROUGE-L over a truncated input baseline.【13】 Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Sound标题：利用声音元素对语音识别进行隐秘的后门攻击链接：https://arxiv.org/abs/2307.08208作者：Hanbo Cai,Pengcheng Zhang,Hai Dong,Yan Xiao,Stefanos Koffas,Yiming Li机构：Sun Yat-senUniversity备注：13 pages摘要：深度神经网络（DNN）已被广泛且成功地采用并部署在语音识别的各种应用中。最近，一些研究表明，这些模型容易受到后门攻击，攻击者可以通过毒化受害者模型的训练过程来将恶意预测行为植入受害者模型中。在本文中，我们重新审视毒药只有后门攻击语音识别。我们发现，现有的方法是不隐秘的，因为它们的触发模式是可察觉的人类或机器检测。这种限制主要是因为它们的触发模式是简单的噪声或可分离且独特的剪辑。受这些发现的启发，我们建议利用声音的元素（例如，音高和音色）来设计更隐蔽但有效的仅毒后门攻击。具体来说，我们插入一个短时间的高音调信号作为触发器，并增加剩余的音频剪辑的间距，以“掩盖”它的设计隐身间距为基础的触发器。我们操纵受害者音频的音色特征来设计基于音色的隐形攻击，并设计声纹选择模块来促进多后门攻击。我们的攻击可以产生更多的“天然”有毒样本，因此更隐蔽。在基准数据集上进行了大量的实验，验证了我们的攻击在不同设置下的有效性（$e.g.$，所有对一、所有对所有、干净标签、物理和多后门设置）及其隐蔽性。复制主要实验的代码可在\url{https：//github.com/HanboCai/BadSpeech_SoE}获得。摘要：Deep neural networks (DNNs) have been widely and successfully adopted and deployed in various applications of speech recognition. Recently, a few works revealed that these models are vulnerable to backdoor attacks, where the adversaries can implant malicious prediction behaviors into victim models by poisoning their training process. In this paper, we revisit poison-only backdoor attacks against speech recognition. We reveal that existing methods are not stealthy since their trigger patterns are perceptible to humans or machine detection. This limitation is mostly because their trigger patterns are simple noises or separable and distinctive clips. Motivated by these findings, we propose to exploit elements of sound ($e.g.$, pitch and timbre) to design more stealthy yet effective poison-only backdoor attacks. Specifically, we insert a short-duration high-pitched signal as the trigger and increase the pitch of remaining audio clips to `mask' it for designing stealthy pitch-based triggers. We manipulate timbre features of victim audios to design the stealthy timbre-based attack and design a voiceprint selection module to facilitate the multi-backdoor attack. Our attacks can generate more `natural' poisoned samples and therefore are more stealthy. Extensive experiments are conducted on benchmark datasets, which verify the effectiveness of our attacks under different settings ($e.g.$, all-to-one, all-to-all, clean-label, physical, and multi-backdoor settings) and their stealthiness. The code for reproducing main experiments are available at \url{https://github.com/HanboCai/BadSpeech_SoE}.【14】 NoiseBandNet: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks标题：NoiseBandNet：基于滤波器组的可控时变神经网络音效合成链接：https://arxiv.org/abs/2307.08007作者：Adrián Barahona-Ríos,Tom Collins机构：Frost School of Music, University of Miami备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：由于数据的潜在稀缺性和频谱-时间变化，声音效果的可控神经音频合成是一项具有挑战性的任务。差分数字信号处理（DDSP）合成器已经成功地用于使用相对有限的数据和计算资源来建模和控制音乐和谐波信号。在这里，我们提出了NoiseBandNet，一种能够通过滤波器组过滤白噪声来合成和控制声音效果的架构，从而比以前的系统更进一步，对声音的谐波性质作出假设。我们通过一系列的实验来评估我们的方法，建模的脚步声，雷雨，陶器，敲门声和金属声音效果。将NoiseBandNet音频重建能力与DDSP滤波噪声合成器的四种变体进行比较，NoiseBandNet在10个评估类别中有9个得分较高，从而建立了一种灵活的DDSP方法，用于生成任意长度的时变、非谐波声音效果，同时具有良好的时间和频率分辨率。最后，我们介绍了NoiseBandNet的一些潜在的创造性用途，通过生成变化，执行响度转移，并通过在用户定义的控制曲线上对其进行训练。摘要：Controllable neural audio synthesis of sound effects is a challenging task due to the potential scarcity and spectro-temporal variance of the data. Differentiable digital signal processing (DDSP) synthesisers have been successfully employed to model and control musical and harmonic signals using relatively limited data and computational resources. Here we propose NoiseBandNet, an architecture capable of synthesising and controlling sound effects by filtering white noise through a filterbank, thus going further than previous systems that make assumptions about the harmonic nature of sounds. We evaluate our approach via a series of experiments, modelling footsteps, thunderstorm, pottery, knocking, and metal sound effects. Comparing NoiseBandNet audio reconstruction capabilities to four variants of the DDSP-filtered noise synthesiser, NoiseBandNet scores higher in nine out of ten evaluation categories, establishing a flexible DDSP method for generating time-varying, inharmonic sound effects of arbitrary length with both good time and frequency resolution. Finally, we introduce some potential creative uses of NoiseBandNet, by generating variations, performing loudness transfer, and by training it on user-defined control curves.【15】 Single and Multi-Speaker Cloned Voice Detection: From Perceptual to Learned Features标题：单说话人和多说话人克隆语音检测：从感知特征到学习特征链接：https://arxiv.org/abs/2307.07683作者：Sarah Barrington,Romit Barua,Gautham Koorma,Hany Farid机构：School of Information, Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA USA摘要：近年来，合成语音克隆技术取得了重大进展，带来了一系列潜在危害。从小规模和大规模的金融欺诈到虚假信息活动，需要可靠的方法来区分真实和合成的声音。我们描述了三种技术，用于区分一个真正的克隆的声音，旨在模仿一个特定的人。这三种方法在其特征提取阶段不同，其中低维感知特征提供高可解释性但较低准确性，通用光谱特征，以及端到端学习特征提供较少可解释性但较高准确性。我们展示了这些方法的有效性时，一个单一的扬声器的声音和多个声音训练时。学习的功能始终产生相等的错误率在$0\%$和$4\%$之间，并且对对抗性洗钱相当稳健。摘要：Synthetic-voice cloning technologies have seen significant advances in recent years, giving rise to a range of potential harms. From small- and large-scale financial fraud to disinformation campaigns, the need for reliable methods to differentiate real and synthesized voices is imperative. We describe three techniques for differentiating a real from a cloned voice designed to impersonate a specific person. These three approaches differ in their feature extraction stage with low-dimensional perceptual features offering high interpretability but lower accuracy, to generic spectral features, and end-to-end learned features offering less interpretability but higher accuracy. We show the efficacy of these approaches when trained on a single speaker's voice and when trained on multiple voices. The learned features consistently yield an equal error rate between $0\%$ and $4\%$, and are reasonably robust to adversarial laundering.
每日语音速递
