今日论文合集：cs.SD语音13篇，eess.AS音频处理13篇。【1】 AutoMatch: A Large-scale Audio Beat Matching Benchmark for Boosting Deep Learning Assistant Video Editing
标题：AutoMatch：一种支持深度学习助手视频编辑的大规模音频节拍匹配基准
链接：http://arxiv.org/pdf/2303.01884v1
作者：Sen Pei,Jingya Yu,Qi Chen,Wozhou He机构：ByteDance Inc, NLPR, Institute of Automation, Chinese Academy of Sciences摘要：短视频的爆炸式增长极大地重塑了人们的社交方式，产生了日常分享和获取最新信息的新趋势。这些丰富的视频资源，一方面得益于带摄像头的便携式设备的普及，但另一方面也离不开众多视频创作者贡献的宝贵编辑工作。本文研究了一个新颖而实用的问题，即音频节拍匹配（ABM），旨在根据背景音乐推荐合适的过渡时间戳。这项技术有助于减轻视频编辑过程中的劳动密集型工作，为创作者节省能源，使他们能够更专注于视频内容的创意。我们正式定义了反弹道导弹问题及其评估协议。同时，大规模音频数据集，即，为了促进这一新的研究方向，提出了具有超过87k精细注释背景音乐的AutoMatch。为了进一步为后续的研究打下坚实的基础，我们还提出了一个新的模型BeatX来解决这个具有挑战性的问题。同时，我们创造性地提出了标签范围的概念，它消除了数据不平衡的问题，并在训练过程中为地面真值分配自适应权重。虽然短视频平台数量众多，但针对该场景的相关研究并不充分，据我们所知，AutoMatch是第一个解决音频节拍匹配问题的大规模数据集。我们希望发布的数据集和我们的竞争基线可以鼓励更多的关注这一系列的研究。数据集和编码将公开提供。摘要：The explosion of short videos has dramatically reshaped the manners people socialize, yielding a new trend for daily sharing and access to the latest information. These rich video resources, on the one hand, benefited from the popularization of portable devices with cameras, but on the other, they can not be independent of the valuable editing work contributed by numerous video creators. In this paper, we investigate a novel and practical problem, namely audio beat matching (ABM), which aims to recommend the proper transition time stamps based on the background music. This technique helps to ease the labor-intensive work during video editing, saving energy for creators so that they can focus more on the creativity of video content. We formally define the ABM problem and its evaluation protocol. Meanwhile, a large-scale audio dataset, i.e., the AutoMatch with over 87k finely annotated background music, is presented to facilitate this newly opened research direction. To further lay solid foundations for the following study, we also propose a novel model termed BeatX to tackle this challenging task. Alongside, we creatively present the concept of label scope, which eliminates the data imbalance issues and assigns adaptive weights for the ground truth during the training procedure in one stop. Though plentiful short video platforms have flourished for a long time, the relevant research concerning this scenario is not sufficient, and to the best of our knowledge, AutoMatch is the first large-scale dataset to tackle the audio beat ma【2】 Low-Complexity Audio Embedding Extractors
标题：低复杂度的音频嵌入抽取器
链接：http://arxiv.org/pdf/2303.01879v1
作者：Florian Schmid,Khaled Koutini,Gerhard Widmer机构：Institute of Computational Perception,LIT Artificial Intelligence Lab, Johannes Kepler University, Linz, Austria备注：Submitted to the 31st European Signal Processing Conference, EUSIPCO 2023. Source Code available at: this https URL摘要：使用深度学习模型解决说话人识别、音乐分类或语义音频事件标记等任务通常需要计算要求很高的网络。通用音频嵌入（GPAE）是音频信号的密集表示，允许轻量级、浅分类器处理各种音频任务。其思想是单个复杂特征提取器将提取密集的GPAE，而浅MLP可以产生特定于任务的预测。如果所提取的密集表示足够通用以允许简单的下游分类器推广到音频域中的各种任务，则单个代价高昂的前向传递足以并行地解决多个任务。在这项工作中，我们试图降低成本的GPAE提取器，使他们适合于资源有限的设备。我们使用在AudioSet上训练的高效移动网络，使用来自Transformer集合的知识蒸馏作为高效GPAE提取器。我们探索了如何从模型中获得高质量的GPAE，研究了模型复杂度与提取的GPAE质量之间的关系，并得出结论：低复杂度模型可以生成有竞争力的GPAE，为在边缘设备上分析音频流铺平了道路。多个音频分类和识别任务。摘要：Solving tasks such as speaker recognition, music classification, or semantic audio event tagging with deep learning models typically requires computationally demanding networks. General-purpose audio embeddings (GPAEs) are dense representations of audio signals that allow lightweight, shallow classifiers to tackle various audio tasks. The idea is that a single complex feature extractor would extract dense GPAEs, while shallow MLPs can produce task-specific predictions. If the extracted dense representations are general enough to allow the simple downstream classifiers to generalize to a variety of tasks in the audio domain, a single costly forward pass suffices to solve multiple tasks in parallel. In this work, we try to reduce the cost of GPAE extractors to make them suitable for resource-constrained devices. We use efficient MobileNets trained on AudioSet using Knowledge Distillation from a Transformer ensemble as efficient GPAE extractors. We explore how to obtain high-quality GPAEs from the model, study how model complexity relates to the quality of extracted GPAEs, and conclude that low-complexity models can generate competitive GPAEs, paving the way for analyzing audio streams on edge devices w.r.t. multiple audio classification and recognition tasks.【3】 Decoding and Visualising Intended Emotion in an Expressive Piano Performance
标题：在富有表现力的钢琴演奏中解码和形象化意向情感
链接：http://arxiv.org/pdf/2303.01875v1
作者：Institute of Computational Perception, Johannes Kepler University Linz, Austria, LIT AI Lab, Linz Institute of Technology, Austria备注：Extended version of Late-Breaking Demo Session paper accepted at ISMIR 2022 (23rd Int. Society for Music Information Retrieval Conf., Bengaluru, India, 2022)摘要：专业音乐家可以塑造一个音乐作品来传达他们打算传达的特定情感。本文将一个基于中层特征的音乐情感模型应用于演奏者与听者的交流场景中，并通过一个小型可视化的音乐情感解码器进行实时演示。我们还使用感知速度和感知动态的类似物来扩展现有的中级特征集。摘要：Expert musicians can mould a musical piece to convey specific emotions that they intend to communicate. In this paper, we place a mid-level features based music emotion model in this performer-to-listener communication scenario, and demonstrate via a small visualisation music emotion decoding in real time. We also extend the existing set of mid-level features using analogues of perceptual speed and perceived dynamics.【4】 Spectrogram Inversion for Audio Source Separation via Consistency, Mixing, and Magnitude Constraints
标题：基于一致性、混合和幅度约束的音频源分离谱图反演法
链接：http://arxiv.org/pdf/2303.01864v1
作者：Paul Magron,Tuomas Virtanen机构：Universit´e de Lorraine, CNRS, Inria, LORIA, Nancy, France, Audio Research Group, Tampere University, Tampere, Finland摘要：音频源分离通常通过估计每个源的幅度谱图，然后应用相位恢复（或谱图反演）算法来恢复时域信号来实现。典型地，谱图反演被视为涉及一个或多个项的优化问题，以便促进符合一致性属性、混合约束和 或目标幅度目标的估计。尽管如此，哪一套限制和问题表述在实践中最合适，仍然不清楚。本文设计了一个推导谱图反演算法的通用框架，该框架基于通过将这些目标组合为软惩罚或硬约束来公式化优化问题。我们通过在对应于每个目标 约束的子集上执行交替投影的算法来解决这些问题。我们的框架包括现有的技术，从文献以及新的算法。我们研究了这些方法在语音增强任务中的潜力。特别地，我们的一个新颖算法在使用神经网络预先估计幅值的现实设置中优于其他方法。摘要：Audio source separation is often achieved by estimating the magnitude spectrogram of each source, and then applying a phase recovery (or spectrogram inversion) algorithm to retrieve time-domain signals. Typically, spectrogram inversion is treated as an optimization problem involving one or several terms in order to promote estimates that comply with a consistency property, a mixing constraint, and or a target magnitude objective. Nonetheless, it is still unclear which set of constraints and problem formulation is the most appropriate in practice. In this paper, we design a general framework for deriving spectrogram inversion algorithm, which is based on formulating optimization problems by combining these objectives either as soft penalties or hard constraints. We solve these by means of algorithms that perform alternating projections on the subsets corresponding to each objective constraint. Our framework encompasses existing techniques from the literature as well as novel algorithms. We investigate the potential of these approaches for a speech enhancement task. In particular, one of our novel algorithms outperforms other approaches in a realistic setting where the magnitudes are estimated beforehand using a neural network.【5】 Unified Keyword Spotting and Audio Tagging on Mobile Devices with Transformers
标题：带Transformer的移动设备上统一的关键字识别和音频标记
链接：http://arxiv.org/pdf/2303.01812v1
作者：Heinrich Dinkel,Yongqing Wang,Zhiyong Yan,Junbo Zhang,Yujun Wang机构：Xiaomi Corperation, Beijing, China摘要：关键词识别是现代智能助手人机交互前端的核心任务。最近，已经提出了一种统一（UniKW-AT）框架，其以音频标记（AT）的形式向KWS模型添加附加能力。然而，先前的工作没有考虑UniKW-AT模型的真实世界部署，其中模型大小和推理速度等因素比性能本身更重要。本文介绍了三种移动设备可部署模型，即统一Transformers（UiT）。我们的最佳模型在Audioset上达到了34.09的mAP，在公共Google Speech Commands V1数据集上达到了97.76的准确率。此外，我们在四个移动平台上对我们提出的方法进行了基准测试，结果表明，与竞争对手MobileNetV 2相比，我们提出的UiT模型可以实现2 - 6倍的加速比。摘要：Keyword spotting (KWS) is a core human-machine-interaction front-end task for most modern intelligent assistants. Recently, a unified (UniKW-AT) framework has been proposed that adds additional capabilities in the form of audio tagging (AT) to a KWS model. However, previous work did not consider the real-world deployment of a UniKW-AT model, where factors such as model size and inference speed are more important than performance alone. This work introduces three mobile-device deployable models named Unified Transformers (UiT). Our best model achieves an mAP of 34.09 on Audioset, and an accuracy of 97.76 on the public Google Speech Commands V1 dataset. Further, we benchmark our proposed approaches on four mobile platforms, revealing that the proposed UiT models can achieve a speedup of 2 - 6 times against a competitive MobileNetV2.【6】 SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks
标题：SottoVoce：一种基于超声成像的深度神经网络无声语音交互
链接：http://arxiv.org/pdf/2303.01758v1
作者：Naoki Kimura,Michinari Kono,Jun Rekimoto机构：The University of Tokyo, -,-, Hongo, Bunkyo-ku, Tokyo, Sony Computer Science Laboratories, -,-, Higashigotanda, Shinagawa-ku, Tokyo, JapanJournal-ref：CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019)摘要：通过语音操作的数字设备的可用性正在迅速扩展。然而，语音接口的应用仍然受到限制。比如，在公共场所讲话成为周围人的烦恼，机密信息更不宜说出口。环境噪声可能降低语音识别的准确性。为了解决这些限制，提出了一种检测用户的清音话语的系统。根据由附着于颌下侧的超声成像传感器观察到的内部信息，我们提出的系统识别说话内容而无需用户的说话语音。我们提出的深度神经网络模型用于从超声图像序列中获取声学特征。我们确认我们的系统生成的音频信号可以控制现有的智能音箱。我们还观察到用户可以调整他们的口腔运动来学习和提高他们的语音识别的准确性。摘要：The availability of digital devices operated by voice is expanding rapidly. However, the applications of voice interfaces are still restricted. For example, speaking in public places becomes an annoyance to the surrounding people, and secret information should not be uttered. Environmental noise may reduce the accuracy of speech recognition. To address these limitations, a system to detect a user's unvoiced utterance is proposed. From internal information observed by an ultrasonic imaging sensor attached to the underside of the jaw, our proposed system recognizes the utterance contents without the user's uttering voice. Our proposed deep neural network model is used to obtain acoustic features from a sequence of ultrasound images. We confirmed that audio signals generated by our system can control the existing smart speakers. We also observed that a user can adjust their oral movement to learn and improve the accuracy of their voice recognition.【7】 DWFormer: Dynamic Window transFormer for Speech Emotion Recognition
标题：DWFormer：用于语音情感识别的动态窗口转换器
链接：http://arxiv.org/pdf/2303.01694v1
作者：Shuaiqi Chen,Xiaofen Xing,Weibin Zhang,Weidong Chen,Xiangmin Xu机构：School of Electronic and Information Engineering, South China University of Technology, China, VoiceAI Technologies, Shenzhen, China , Pazhou Laboratory, China, School of Future Technology, South China University of Technology, China备注：4 pages, 5 figures, 3 tables, accepted by 2023 International Conference on Acoustics, Speech, and Signal Processing (ICASSP2023)摘要：语音情感识别是人机交互的关键。表示不同情感的时域局部分散在语音的不同部分。此外，重要信息的时间尺度可在语音段内和跨语音段在大范围上变化。虽然基于变换的模型在这一领域取得了一定的进展，但现有的模型不能精确定位不同时间尺度上的重要区域。为了解决这一问题，我们提出了动态窗口转换器（DWFormer），这是一种通过动态地将样本分割成窗口来利用时间重要性的新体系结构。在窗口内应用自注意机制，以细粒度方式局部捕获时间重要信息。全局通信中还考虑了跨窗口信息交互。在IEMOCAP和MELD数据集上评价DWFormer。实验结果表明，该模型比现有方法具有更好的性能。摘要：Speech emotion recognition is crucial to human-computer interaction. The temporal regions that represent different emotions scatter in different parts of the speech locally. Moreover, the temporal scales of important information may vary over a large range within and across speech segments. Although transformer-based models have made progress in this field, the existing models could not precisely locate important regions at different temporal scales. To address the issue, we propose Dynamic Window transFormer (DWFormer), a new architecture that leverages temporal importance by dynamically splitting samples into windows. Self-attention mechanism is applied within windows for capturing temporal important information locally in a fine-grained way. Cross-window information interaction is also taken into account for global communication. DWFormer is evaluated on both the IEMOCAP and the MELD datasets. Experimental results show that the proposed model achieves better performance than the previous state-of-the-art methods.【8】LooperGP: A Loopable Sequence Model for Live Coding Performance using GuitarPro Tablature
标题：Loop GP：基于GuitarPro平台的实时编码性能可循环序列模型
链接：http://arxiv.org/pdf/2303.01665v1
作者：Sara Adkins,Pedro Sarmento,Mathieu Barthet机构：Barthet,[,−,−,−,], Centre for Digital Music, Queen Mary University of London, United KingdomJournal-ref：EvoMUSART: International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar) 2023摘要：尽管离线效果令人印象深刻，但用于符号音乐生成的深度学习模型并没有广泛用于现场表演，原因是缺乏具有音乐意义的控制参数，并且其输出中缺乏结构化的音乐形式。为了解决这些问题，我们引入了LooperGP，这是一种引导Transformer-XL模型生成具有指定小节数和拍号的可循环乐句的方法，支持实时编码性能的工具。我们展示了通过在从DadaGP数据集提取的93，681个音乐循环的数据集上训练LooperGP，我们能够引导其生成输出生成3倍于基线的可循环短语。在由31名参与者进行的主观听力测试中，LooperGP循环在原创性、音乐连贯性和循环流畅性方面均获得了积极的中值评级，显示了其作为演奏工具的潜力。摘要：Despite their impressive offline results, deep learning models for symbolic music generation are not widely used in live performances due to a deficit of musically meaningful control parameters and a lack of structured musical form in their outputs. To address these issues we introduce LooperGP, a method for steering a Transformer-XL model towards generating loopable musical phrases of a specified number of bars and time signature, enabling a tool for live coding performances. We show that by training LooperGP on a dataset of 93,681 musical loops extracted from the DadaGP dataset, we are able to steer its generative output towards generating 3x as many loopable phrases as our baseline. In a subjective listening test conducted by 31 participants, LooperGP loops achieved positive median ratings in originality, musical coherence and loop smoothness, demonstrating its potential as a performance tool.【9】 Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations
标题：Miipher：一种融合自监督语音和文本表示的稳健语音恢复模型
链接：http://arxiv.org/pdf/2303.01664v1
作者：Yuma Koizumi,Heiga Zen,Shigeki Karita,Yifan Ding,Kohei Yatabe,Nobuyuki Morioka,Yu Zhang,Wei Han,Ankur Bapna,Michiel Bacchiani机构：Google, Japan, Tokyo University of Agriculture & Technology, Japan, Google, USA摘要：语音恢复（SR）是将退化的语音信号转换成高质量语音信号的一项任务。在本研究中，我们提出了一个鲁棒的SR模型Miipher，并将Miipher应用到一个新的SR应用中：通过将从Web收集的语音样本转换为录音室质量来增加用于语音生成的高质量训练数据的量。为了使我们的SR模型对各种退化具有鲁棒性，我们使用（i）从w2 v-BERT提取的语音表示作为输入特征，以及（ii）经由PnG-BERT从转录本提取的文本表示作为语言条件作用特征。实验表明Miipher（i）对各种音频退化都是鲁棒的，（ii）使我们能够从从从Web收集的恢复语音样本训练高质量的文本到语音（TTS）模型。音频样本可在我们的演示页面：google.github.io df-conformer miipher摘要：Speech restoration (SR) is a task of converting degraded speech signals into high-quality ones. In this study, we propose a robust SR model called Miipher, and apply Miipher to a new SR application: increasing the amount of high-quality training data for speech generation by converting speech samples collected from the Web to studio-quality. To make our SR model robust against various degradation, we use (i) a speech representation extracted from w2v-BERT for the input feature, and (ii) a text representation extracted from transcripts via PnG-BERT as a linguistic conditioning feature. Experiments show that Miipher (i) is robust against various audio degradation and (ii) enable us to train a high-quality text-to-speech (TTS) model from restored speech samples collected from the Web. Audio samples are available at our demo page: google.github.io df-conformer miipher【10】 WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions
标题：WESPER：基于密语的语音交互的Zero-Shot和实时密语到正常语音的转换
链接：http://arxiv.org/pdf/2303.01639v1
机构：The University of Tokyo, -,-, Hongo, Bunkyo-ku, Tokyo, Japan, Sony Computer Science Laboratories, Kyoto, -, Hontoro-cho, Shimogyo-ku, Kyoto-shi, Kyoto, Japan, Energy Predictor, Pitch Predictor, Whisper Hoarse Normal Voices, Transformer Layers, ReconstructedJournal-ref：Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), April 23--28, 2023摘要：识别耳语音并将其转换为正常语音为语音交互创造了许多可能性。由于耳语语音的声压明显低于正常语音，因此可以在公共场所作为半无声的语音互动，而不会被他人听到。将耳语转换为正常语音还可以提高有语言或听力障碍的人的语音质量。然而，传统的语音转换技术不能提供足够的转换质量，或者需要由耳语和正常语音发声对组成的说话者相关数据集。为了解决这些问题，我们提出了WESPER，一种基于自监督学习的zero-shot、实时的耳语到正常语音转换机制。WESPER由语音到单元（STU）编码器和单元到语音（UTS）解码器组成，STU编码器生成耳语音和正常语音共同的隐藏语音单元，UTS解码器从编码的语音单元重构语音。与现有方法不同，这种转换是用户无关的，并且不需要用于耳语音和正常语音的配对数据集。UTS解码器可以从语音单元中重构任何目标说话人的语音中的语音，并且它仅需要未标记的目标说话人的语音数据。我们证实，从耳语转换的语音质量得到了改善，同时保留了其自然韵律。此外，我们证实了所提出的方法对有语言或听力障碍的人进行语音重建的有效性。（项目页面：http: lab.rekimoto.org projects wesper）摘要：Recognizing whispered speech and converting it to normal speech creates many possibilities for speech interaction. Because the sound pressure of whispered speech is significantly lower than that of normal speech, it can be used as a semi-silent speech interaction in public places without being audible to others. Converting whispers to normal speech also improves the speech quality for people with speech or hearing impairments. However, conventional speech conversion techniques do not provide sufficient conversion quality or require speaker-dependent datasets consisting of pairs of whispered and normal speech utterances. To address these problems, we propose WESPER, a zero-shot, real-time whisper-to-normal speech conversion mechanism based on self-supervised learning. WESPER consists of a speech-to-unit (STU) encoder, which generates hidden speech units common to both whispered and normal speech, and a unit-to-speech (UTS) decoder, which reconstructs speech from the encoded speech units. Unlike the existing methods, this conversion is user-independent and does not require a paired dataset for whispered and normal speech. The UTS decoder can reconstruct speech in any target speaker's voice from speech units, and it requires only an unlabeled target speaker's speech data. We confirmed that the quality of the speech converted from a whisper was improved while preserving its natural prosody. Additionally, we confirmed the effectiveness of the proposed approach to perform speech reconstruction for people with speech or h【11】 Fine-grained Emotional Control of Text-To-Speech: Learning To Rank Inter- And Intra-Class Emotion Intensities
标题：文本到语音转换的细粒度情绪控制：学习对班级间和班内情绪强度进行排序
链接：http://arxiv.org/pdf/2303.01508v1
作者：Shijun Wang,Jón Guðnason,Damian Borth机构：University of St.Gallen, Switzerland, Reykjavik University, Iceland备注：Accepted by ICASSP2023摘要：最先进的文本到语音（TTS）模型能够产生高质量的语音。然而，所生成的语音在情感表达上通常是中性的，而人们常常希望对单词或音素进行细粒度的情感控制。尽管仍然具有挑战性，但最近已经提出了能够通过手动分配情感强度来控制语音的第一个TTS模型。不幸的是，由于忽略了类内距离，强度差异往往是不可识别的。本文提出了一种细粒度可控的情感TTS，它同时考虑了类间和类内距离，能够合成具有可识别强度差异的语音。主观和客观实验表明，该模型在可控性、情感表达性和自然性方面均优于现有的两种可控TTS模型。摘要：State-of-the-art Text-To-Speech (TTS) models are capable of producing high-quality speech. The generated speech, however, is usually neutral in emotional expression, whereas very often one would want fine-grained emotional control of words or phonemes. Although still challenging, the first TTS models have been recently proposed that are able to control voice by manually assigning emotion intensity. Unfortunately, due to the neglect of intra-class distance, the intensity differences are often unrecognizable. In this paper, we propose a fine-grained controllable emotional TTS, that considers both inter- and intra-class distances and be able to synthesize speech with recognizable intensity difference. Our subjective and objective experiments demonstrate that our model exceeds two state-of-the-art controllable TTS models for controllability, emotion expressiveness and naturalness.【12】 Defending against Adversarial Audio via Diffusion Model
标题：利用扩散模型防御恶意音频
链接：http://arxiv.org/pdf/2303.01507v1
作者：Shutong Wu,Jiongxiao Wang,Wei Ping,Weili Nie,Chaowei Xiao机构：Arizona State University, Shanghai Jiao Tong University, NVIDIA摘要：近年来，深度学习模型已经广泛应用于商业声学系统中。然而，对抗性音频示例可能导致这些声学系统的异常行为，同时人类难以感知。已经提出了诸如基于变换的防御和对抗性训练的各种方法来保护声学系统免受对抗性攻击，但是它们对于自适应攻击不太有效。此外，由于音频数据的独特属性，直接应用来自图像域的方法可能导致次优结果。在本文中，我们提出了一个对抗性的净化为基础的防御管道，音频纯，为声学系统通过现成的扩散模型。利用扩散模型强大的生成能力，AudioPure首先在对抗性音频中加入少量噪声，然后运行反向采样步骤，净化嘈杂的音频，恢复干净的音频。AudioPure是一种即插即用的方法，可以直接应用于任何预先训练的分类器，而无需任何微调或重新训练。我们在语音命令识别任务上进行了广泛的实验来评估AudioPure的鲁棒性。该方法可以有效地抵抗各种对抗性攻击（如$ mathcal{L}_2$或$ mathcal{L}_ infty$-norm）.在$ mathcal{L}_2$或$ mathcal{L}_ infty$-norm约束下的强自适应白盒和黑盒攻击下，该算法的鲁棒性优于现有算法（鲁棒准确率高达+20%）.此外，我们还利用随机光滑方法对$ mathcal{L}_2$-范数有界扰动进行了鲁棒性验证.我们的管道实现了比基准更高的认证精度。摘要：Deep learning models have been widely used in commercial acoustic systems in recent years. However, adversarial audio examples can cause abnormal behaviors for those acoustic systems, while being hard for humans to perceive. Various methods, such as transformation-based defenses and adversarial training, have been proposed to protect acoustic systems from adversarial attacks, but they are less effective against adaptive attacks. Furthermore, directly applying the methods from the image domain can lead to suboptimal results because of the unique properties of audio data. In this paper, we propose an adversarial purification-based defense pipeline, AudioPure, for acoustic systems via off-the-shelf diffusion models. Taking advantage of the strong generation ability of diffusion models, AudioPure first adds a small amount of noise to the adversarial audio and then runs the reverse sampling step to purify the noisy audio and recover clean audio. AudioPure is a plug-and-play method that can be directly applied to any pretrained classifier without any fine-tuning or re-training. We conduct extensive experiments on speech command recognition task to evaluate the robustness of AudioPure. Our method is effective against diverse adversarial attacks (e.g. $ mathcal{L}_2$ or $ mathcal{L}_ infty$-norm). It outperforms the existing methods under both strong adaptive white-box and black-box attacks bounded by $ mathcal{L}_2$ or $ mathcal{L}_ infty$-norm (up to +20 % in【13】An investigation into the adaptability of a diffusion-based TTS model
标题：基于扩散的TTS模型的适应性研究
链接：http://arxiv.org/pdf/2303.01849v1
作者：Haolin Chen,Philip N. Garner机构：Idiap Research Institute, Martigny, Switzerland摘要：考虑到扩散技术最近在产生自然发音的合成语音方面取得的成功，我们研究如何将扩散技术应用于说话人自适应TTS。从更传统的自适应方法中获得线索，我们证明了自适应可以包括在使用条件层归一化和步骤嵌入的扩散管道中。然而，我们通过实验表明，虽然该方法有优点，但单独的这种自适应无法达到基于Transformer的技术的性能。在第二个实验中，我们展示了扩散可以与Transformer进行最佳组合，后者承担了大部分自适应负载，而前者有助于提高自然度。摘要：Given the recent success of diffusion in producing natural-sounding synthetic speech, we investigate how diffusion can be used in speaker adaptive TTS. Taking cues from more traditional adaptation approaches, we show that adaptation can be included in a diffusion pipeline using conditional layer normalization with a step embedding. However, we show experimentally that, whilst the approach has merit, such adaptation alone cannot approach the performance of Transformer-based techniques. In a second experiment, we show that diffusion can be optimally combined with Transformer, with the latter taking the bulk of the adaptation load and the former contributing to improved naturalness.【1】An investigation into the adaptability of a diffusion-based TTS model
标题：基于扩散的TTS模型的适应性研究
链接：http://arxiv.org/pdf/2303.01849v1
*与cs.SD语音【13】为同一篇
作者：Haolin Chen,Philip N. Garner机构：Idiap Research Institute, Martigny, Switzerland摘要：考虑到扩散技术最近在产生自然发音的合成语音方面取得的成功，我们研究如何将扩散技术应用于说话人自适应TTS。从更传统的自适应方法中获得线索，我们证明了自适应可以包括在使用条件层归一化和步骤嵌入的扩散管道中。然而，我们通过实验表明，虽然该方法有优点，但单独的这种自适应无法达到基于Transformer的技术的性能。在第二个实验中，我们展示了扩散可以与Transformer进行最佳组合，后者承担了大部分自适应负载，而前者有助于提高自然度。摘要：Given the recent success of diffusion in producing natural-sounding synthetic speech, we investigate how diffusion can be used in speaker adaptive TTS. Taking cues from more traditional adaptation approaches, we show that adaptation can be included in a diffusion pipeline using conditional layer normalization with a step embedding. However, we show experimentally that, whilst the approach has merit, such adaptation alone cannot approach the performance of Transformer-based techniques. In a second experiment, we show that diffusion can be optimally combined with Transformer, with the latter taking the bulk of the adaptation load and the former contributing to improved naturalness.【2】 AutoMatch: A Large-scale Audio Beat Matching Benchmark for Boosting Deep Learning Assistant Video Editing
标题：AutoMatch：一种支持深度学习助手视频编辑的大规模音频节拍匹配基准
链接：http://arxiv.org/pdf/2303.01884v1
*与cs.SD语音【1】为同一篇
作者：Sen Pei,Jingya Yu,Qi Chen,Wozhou He机构：ByteDance Inc, NLPR, Institute of Automation, Chinese Academy of Sciences摘要：短视频的爆炸式增长极大地重塑了人们的社交方式，产生了日常分享和获取最新信息的新趋势。这些丰富的视频资源，一方面得益于带摄像头的便携式设备的普及，但另一方面也离不开众多视频创作者贡献的宝贵编辑工作。本文研究了一个新颖而实用的问题，即音频节拍匹配（ABM），旨在根据背景音乐推荐合适的过渡时间戳。这项技术有助于减轻视频编辑过程中的劳动密集型工作，为创作者节省能源，使他们能够更专注于视频内容的创意。我们正式定义了反弹道导弹问题及其评估协议。同时，大规模音频数据集，即，为了促进这一新的研究方向，提出了具有超过87k精细注释背景音乐的AutoMatch。为了进一步为后续的研究打下坚实的基础，我们还提出了一个新的模型BeatX来解决这个具有挑战性的问题。同时，我们创造性地提出了标签范围的概念，它消除了数据不平衡的问题，并在训练过程中为地面真值分配自适应权重。虽然短视频平台数量众多，但针对该场景的相关研究并不充分，据我们所知，AutoMatch是第一个解决音频节拍匹配问题的大规模数据集。我们希望发布的数据集和我们的竞争基线可以鼓励更多的关注这一系列的研究。数据集和编码将公开提供。摘要：The explosion of short videos has dramatically reshaped the manners people socialize, yielding a new trend for daily sharing and access to the latest information. These rich video resources, on the one hand, benefited from the popularization of portable devices with cameras, but on the other, they can not be independent of the valuable editing work contributed by numerous video creators. In this paper, we investigate a novel and practical problem, namely audio beat matching (ABM), which aims to recommend the proper transition time stamps based on the background music. This technique helps to ease the labor-intensive work during video editing, saving energy for creators so that they can focus more on the creativity of video content. We formally define the ABM problem and its evaluation protocol. Meanwhile, a large-scale audio dataset, i.e., the AutoMatch with over 87k finely annotated background music, is presented to facilitate this newly opened research direction. To further lay solid foundations for the following study, we also propose a novel model termed BeatX to tackle this challenging task. Alongside, we creatively present the concept of label scope, which eliminates the data imbalance issues and assigns adaptive weights for the ground truth during the training procedure in one stop. Though plentiful short video platforms have flourished for a long time, the relevant research concerning this scenario is not sufficient, and to the best of our knowledge, AutoMatch is the first large-scale dataset to tackle the audio beat ma【3】 Low-Complexity Audio Embedding Extractors
标题：低复杂度的音频嵌入抽取器
链接：http://arxiv.org/pdf/2303.01879v1
*与cs.SD语音【2】为同一篇
作者：Florian Schmid,Khaled Koutini,Gerhard Widmer机构：Institute of Computational Perception,LIT Artificial Intelligence Lab, Johannes Kepler University, Linz, Austria备注：Submitted to the 31st European Signal Processing Conference, EUSIPCO 2023. Source Code available at: this https URL摘要：使用深度学习模型解决说话人识别、音乐分类或语义音频事件标记等任务通常需要计算要求很高的网络。通用音频嵌入（GPAE）是音频信号的密集表示，允许轻量级、浅分类器处理各种音频任务。其思想是单个复杂特征提取器将提取密集的GPAE，而浅MLP可以产生特定于任务的预测。如果所提取的密集表示足够通用以允许简单的下游分类器推广到音频域中的各种任务，则单个代价高昂的前向传递足以并行地解决多个任务。在这项工作中，我们试图降低成本的GPAE提取器，使他们适合于资源有限的设备。我们使用在AudioSet上训练的高效移动网络，使用来自Transformer集合的知识蒸馏作为高效GPAE提取器。我们探索了如何从模型中获得高质量的GPAE，研究了模型复杂度与提取的GPAE质量之间的关系，并得出结论：低复杂度模型可以生成有竞争力的GPAE，为在边缘设备上分析音频流铺平了道路。多个音频分类和识别任务。摘要：Solving tasks such as speaker recognition, music classification, or semantic audio event tagging with deep learning models typically requires computationally demanding networks. General-purpose audio embeddings (GPAEs) are dense representations of audio signals that allow lightweight, shallow classifiers to tackle various audio tasks. The idea is that a single complex feature extractor would extract dense GPAEs, while shallow MLPs can produce task-specific predictions. If the extracted dense representations are general enough to allow the simple downstream classifiers to generalize to a variety of tasks in the audio domain, a single costly forward pass suffices to solve multiple tasks in parallel. In this work, we try to reduce the cost of GPAE extractors to make them suitable for resource-constrained devices. We use efficient MobileNets trained on AudioSet using Knowledge Distillation from a Transformer ensemble as efficient GPAE extractors. We explore how to obtain high-quality GPAEs from the model, study how model complexity relates to the quality of extracted GPAEs, and conclude that low-complexity models can generate competitive GPAEs, paving the way for analyzing audio streams on edge devices w.r.t. multiple audio classification and recognition tasks.【4】 Decoding and Visualising Intended Emotion in an Expressive Piano Performance
标题：在富有表现力的钢琴演奏中解码和形象化意向情感
链接：http://arxiv.org/pdf/2303.01875v1
*与cs.SD语音【3】为同一篇
作者：Institute of Computational Perception, Johannes Kepler University Linz, Austria, LIT AI Lab, Linz Institute of Technology, Austria备注：Extended version of Late-Breaking Demo Session paper accepted at ISMIR 2022 (23rd Int. Society for Music Information Retrieval Conf., Bengaluru, India, 2022)摘要：专业音乐家可以塑造一个音乐作品来传达他们打算传达的特定情感。本文将一个基于中层特征的音乐情感模型应用于演奏者与听者的交流场景中，并通过一个小型可视化的音乐情感解码器进行实时演示。我们还使用感知速度和感知动态的类似物来扩展现有的中级特征集。摘要：Expert musicians can mould a musical piece to convey specific emotions that they intend to communicate. In this paper, we place a mid-level features based music emotion model in this performer-to-listener communication scenario, and demonstrate via a small visualisation music emotion decoding in real time. We also extend the existing set of mid-level features using analogues of perceptual speed and perceived dynamics.【5】 Spectrogram Inversion for Audio Source Separation via Consistency, Mixing, and Magnitude Constraints
标题：基于一致性、混合和幅度约束的音频源分离谱图反演法
链接：http://arxiv.org/pdf/2303.01864v1
*与cs.SD语音【4】为同一篇
作者：Paul Magron,Tuomas Virtanen机构：Universit´e de Lorraine, CNRS, Inria, LORIA, Nancy, France, Audio Research Group, Tampere University, Tampere, Finland摘要：音频源分离通常通过估计每个源的幅度谱图，然后应用相位恢复（或谱图反演）算法来恢复时域信号来实现。典型地，谱图反演被视为涉及一个或多个项的优化问题，以便促进符合一致性属性、混合约束和 或目标幅度目标的估计。尽管如此，哪一套限制和问题表述在实践中最合适，仍然不清楚。本文设计了一个推导谱图反演算法的通用框架，该框架基于通过将这些目标组合为软惩罚或硬约束来公式化优化问题。我们通过在对应于每个目标 约束的子集上执行交替投影的算法来解决这些问题。我们的框架包括现有的技术，从文献以及新的算法。我们研究了这些方法在语音增强任务中的潜力。特别地，我们的一个新颖算法在使用神经网络预先估计幅值的现实设置中优于其他方法。摘要：Audio source separation is often achieved by estimating the magnitude spectrogram of each source, and then applying a phase recovery (or spectrogram inversion) algorithm to retrieve time-domain signals. Typically, spectrogram inversion is treated as an optimization problem involving one or several terms in order to promote estimates that comply with a consistency property, a mixing constraint, and or a target magnitude objective. Nonetheless, it is still unclear which set of constraints and problem formulation is the most appropriate in practice. In this paper, we design a general framework for deriving spectrogram inversion algorithm, which is based on formulating optimization problems by combining these objectives either as soft penalties or hard constraints. We solve these by means of algorithms that perform alternating projections on the subsets corresponding to each objective constraint. Our framework encompasses existing techniques from the literature as well as novel algorithms. We investigate the potential of these approaches for a speech enhancement task. In particular, one of our novel algorithms outperforms other approaches in a realistic setting where the magnitudes are estimated beforehand using a neural network.【6】 Unified Keyword Spotting and Audio Tagging on Mobile Devices with Transformers
标题：带Transformer的移动设备上统一的关键字识别和音频标记
链接：http://arxiv.org/pdf/2303.01812v1
*与cs.SD语音【5】为同一篇
作者：Heinrich Dinkel,Yongqing Wang,Zhiyong Yan,Junbo Zhang,Yujun Wang机构：Xiaomi Corperation, Beijing, China摘要：关键词识别是现代智能助手人机交互前端的核心任务。最近，已经提出了一种统一（UniKW-AT）框架，其以音频标记（AT）的形式向KWS模型添加附加能力。然而，先前的工作没有考虑UniKW-AT模型的真实世界部署，其中模型大小和推理速度等因素比性能本身更重要。本文介绍了三种移动设备可部署模型，即统一Transformers（UiT）。我们的最佳模型在Audioset上达到了34.09的mAP，在公共Google Speech Commands V1数据集上达到了97.76的准确率。此外，我们在四个移动平台上对我们提出的方法进行了基准测试，结果表明，与竞争对手MobileNetV 2相比，我们提出的UiT模型可以实现2 - 6倍的加速比。摘要：Keyword spotting (KWS) is a core human-machine-interaction front-end task for most modern intelligent assistants. Recently, a unified (UniKW-AT) framework has been proposed that adds additional capabilities in the form of audio tagging (AT) to a KWS model. However, previous work did not consider the real-world deployment of a UniKW-AT model, where factors such as model size and inference speed are more important than performance alone. This work introduces three mobile-device deployable models named Unified Transformers (UiT). Our best model achieves an mAP of 34.09 on Audioset, and an accuracy of 97.76 on the public Google Speech Commands V1 dataset. Further, we benchmark our proposed approaches on four mobile platforms, revealing that the proposed UiT models can achieve a speedup of 2 - 6 times against a competitive MobileNetV2.【7】 SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks
标题：SottoVoce：一种基于超声成像的深度神经网络无声语音交互
链接：http://arxiv.org/pdf/2303.01758v1
*与cs.SD语音【6】为同一篇
作者：Naoki Kimura,Michinari Kono,Jun Rekimoto机构：The University of Tokyo, -,-, Hongo, Bunkyo-ku, Tokyo, Sony Computer Science Laboratories, -,-, Higashigotanda, Shinagawa-ku, Tokyo, JapanJournal-ref：CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019)摘要：通过语音操作的数字设备的可用性正在迅速扩展。然而，语音接口的应用仍然受到限制。比如，在公共场所讲话成为周围人的烦恼，机密信息更不宜说出口。环境噪声可能降低语音识别的准确性。为了解决这些限制，提出了一种检测用户的清音话语的系统。根据由附着于颌下侧的超声成像传感器观察到的内部信息，我们提出的系统识别说话内容而无需用户的说话语音。我们提出的深度神经网络模型用于从超声图像序列中获取声学特征。我们确认我们的系统生成的音频信号可以控制现有的智能音箱。我们还观察到用户可以调整他们的口腔运动来学习和提高他们的语音识别的准确性。摘要：The availability of digital devices operated by voice is expanding rapidly. However, the applications of voice interfaces are still restricted. For example, speaking in public places becomes an annoyance to the surrounding people, and secret information should not be uttered. Environmental noise may reduce the accuracy of speech recognition. To address these limitations, a system to detect a user's unvoiced utterance is proposed. From internal information observed by an ultrasonic imaging sensor attached to the underside of the jaw, our proposed system recognizes the utterance contents without the user's uttering voice. Our proposed deep neural network model is used to obtain acoustic features from a sequence of ultrasound images. We confirmed that audio signals generated by our system can control the existing smart speakers. We also observed that a user can adjust their oral movement to learn and improve the accuracy of their voice recognition.【8】 DWFormer: Dynamic Window transFormer for Speech Emotion Recognition
标题：DWFormer：用于语音情感识别的动态窗口转换器
链接：http://arxiv.org/pdf/2303.01694v1
*与cs.SD语音【7】为同一篇
作者：Shuaiqi Chen,Xiaofen Xing,Weibin Zhang,Weidong Chen,Xiangmin Xu机构：School of Electronic and Information Engineering, South China University of Technology, China, VoiceAI Technologies, Shenzhen, China , Pazhou Laboratory, China, School of Future Technology, South China University of Technology, China备注：4 pages, 5 figures, 3 tables, accepted by 2023 International Conference on Acoustics, Speech, and Signal Processing (ICASSP2023)摘要：语音情感识别是人机交互的关键。表示不同情感的时域局部分散在语音的不同部分。此外，重要信息的时间尺度可在语音段内和跨语音段在大范围上变化。虽然基于变换的模型在这一领域取得了一定的进展，但现有的模型不能精确定位不同时间尺度上的重要区域。为了解决这一问题，我们提出了动态窗口转换器（DWFormer），这是一种通过动态地将样本分割成窗口来利用时间重要性的新体系结构。在窗口内应用自注意机制，以细粒度方式局部捕获时间重要信息。全局通信中还考虑了跨窗口信息交互。在IEMOCAP和MELD数据集上评价DWFormer。实验结果表明，该模型比现有方法具有更好的性能。摘要：Speech emotion recognition is crucial to human-computer interaction. The temporal regions that represent different emotions scatter in different parts of the speech locally. Moreover, the temporal scales of important information may vary over a large range within and across speech segments. Although transformer-based models have made progress in this field, the existing models could not precisely locate important regions at different temporal scales. To address the issue, we propose Dynamic Window transFormer (DWFormer), a new architecture that leverages temporal importance by dynamically splitting samples into windows. Self-attention mechanism is applied within windows for capturing temporal important information locally in a fine-grained way. Cross-window information interaction is also taken into account for global communication. DWFormer is evaluated on both the IEMOCAP and the MELD datasets. Experimental results show that the proposed model achieves better performance than the previous state-of-the-art methods.【9】LooperGP: A Loopable Sequence Model for Live Coding Performance using GuitarPro Tablature
标题：Loop GP：基于GuitarPro平台的实时编码性能可循环序列模型
链接：http://arxiv.org/pdf/2303.01665v1
*与cs.SD语音【8】为同一篇
作者：Sara Adkins,Pedro Sarmento,Mathieu Barthet机构：Barthet,[,−,−,−,], Centre for Digital Music, Queen Mary University of London, United KingdomJournal-ref：EvoMUSART: International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar) 2023摘要：尽管离线效果令人印象深刻，但用于符号音乐生成的深度学习模型并没有广泛用于现场表演，原因是缺乏具有音乐意义的控制参数，并且其输出中缺乏结构化的音乐形式。为了解决这些问题，我们引入了LooperGP，这是一种引导Transformer-XL模型生成具有指定小节数和拍号的可循环乐句的方法，支持实时编码性能的工具。我们展示了通过在从DadaGP数据集提取的93，681个音乐循环的数据集上训练LooperGP，我们能够引导其生成输出生成3倍于基线的可循环短语。在由31名参与者进行的主观听力测试中，LooperGP循环在原创性、音乐连贯性和循环流畅性方面均获得了积极的中值评级，显示了其作为演奏工具的潜力。摘要：Despite their impressive offline results, deep learning models for symbolic music generation are not widely used in live performances due to a deficit of musically meaningful control parameters and a lack of structured musical form in their outputs. To address these issues we introduce LooperGP, a method for steering a Transformer-XL model towards generating loopable musical phrases of a specified number of bars and time signature, enabling a tool for live coding performances. We show that by training LooperGP on a dataset of 93,681 musical loops extracted from the DadaGP dataset, we are able to steer its generative output towards generating 3x as many loopable phrases as our baseline. In a subjective listening test conducted by 31 participants, LooperGP loops achieved positive median ratings in originality, musical coherence and loop smoothness, demonstrating its potential as a performance tool.【10】 Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations
标题：Miipher：一种融合自监督语音和文本表示的稳健语音恢复模型
链接：http://arxiv.org/pdf/2303.01664v1
*与cs.SD语音【9】为同一篇
作者：Yuma Koizumi,Heiga Zen,Shigeki Karita,Yifan Ding,Kohei Yatabe,Nobuyuki Morioka,Yu Zhang,Wei Han,Ankur Bapna,Michiel Bacchiani机构：Google, Japan, Tokyo University of Agriculture & Technology, Japan, Google, USA摘要：语音恢复（SR）是将退化的语音信号转换成高质量语音信号的一项任务。在本研究中，我们提出了一个鲁棒的SR模型Miipher，并将Miipher应用到一个新的SR应用中：通过将从Web收集的语音样本转换为录音室质量来增加用于语音生成的高质量训练数据的量。为了使我们的SR模型对各种退化具有鲁棒性，我们使用（i）从w2 v-BERT提取的语音表示作为输入特征，以及（ii）经由PnG-BERT从转录本提取的文本表示作为语言条件作用特征。实验表明Miipher（i）对各种音频退化都是鲁棒的，（ii）使我们能够从从从Web收集的恢复语音样本训练高质量的文本到语音（TTS）模型。音频样本可在我们的演示页面：google.github.io df-conformer miipher摘要：Speech restoration (SR) is a task of converting degraded speech signals into high-quality ones. In this study, we propose a robust SR model called Miipher, and apply Miipher to a new SR application: increasing the amount of high-quality training data for speech generation by converting speech samples collected from the Web to studio-quality. To make our SR model robust against various degradation, we use (i) a speech representation extracted from w2v-BERT for the input feature, and (ii) a text representation extracted from transcripts via PnG-BERT as a linguistic conditioning feature. Experiments show that Miipher (i) is robust against various audio degradation and (ii) enable us to train a high-quality text-to-speech (TTS) model from restored speech samples collected from the Web. Audio samples are available at our demo page: google.github.io df-conformer miipher【11】 WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions
标题：WESPER：基于密语的语音交互的Zero-Shot和实时密语到正常语音的转换
链接：http://arxiv.org/pdf/2303.01639v1
*与cs.SD语音【10】为同一篇
机构：The University of Tokyo, -,-, Hongo, Bunkyo-ku, Tokyo, Japan, Sony Computer Science Laboratories, Kyoto, -, Hontoro-cho, Shimogyo-ku, Kyoto-shi, Kyoto, Japan, Energy Predictor, Pitch Predictor, Whisper Hoarse Normal Voices, Transformer Layers, ReconstructedJournal-ref：Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), April 23--28, 2023摘要：识别耳语音并将其转换为正常语音为语音交互创造了许多可能性。由于耳语语音的声压明显低于正常语音，因此可以在公共场所作为半无声的语音互动，而不会被他人听到。将耳语转换为正常语音还可以提高有语言或听力障碍的人的语音质量。然而，传统的语音转换技术不能提供足够的转换质量，或者需要由耳语和正常语音发声对组成的说话者相关数据集。为了解决这些问题，我们提出了WESPER，一种基于自监督学习的zero-shot、实时的耳语到正常语音转换机制。WESPER由语音到单元（STU）编码器和单元到语音（UTS）解码器组成，STU编码器生成耳语音和正常语音共同的隐藏语音单元，UTS解码器从编码的语音单元重构语音。与现有方法不同，这种转换是用户无关的，并且不需要用于耳语音和正常语音的配对数据集。UTS解码器可以从语音单元中重构任何目标说话人的语音中的语音，并且它仅需要未标记的目标说话人的语音数据。我们证实，从耳语转换的语音质量得到了改善，同时保留了其自然韵律。此外，我们证实了所提出的方法对有语言或听力障碍的人进行语音重建的有效性。（项目页面：http: lab.rekimoto.org projects wesper）摘要：Recognizing whispered speech and converting it to normal speech creates many possibilities for speech interaction. Because the sound pressure of whispered speech is significantly lower than that of normal speech, it can be used as a semi-silent speech interaction in public places without being audible to others. Converting whispers to normal speech also improves the speech quality for people with speech or hearing impairments. However, conventional speech conversion techniques do not provide sufficient conversion quality or require speaker-dependent datasets consisting of pairs of whispered and normal speech utterances. To address these problems, we propose WESPER, a zero-shot, real-time whisper-to-normal speech conversion mechanism based on self-supervised learning. WESPER consists of a speech-to-unit (STU) encoder, which generates hidden speech units common to both whispered and normal speech, and a unit-to-speech (UTS) decoder, which reconstructs speech from the encoded speech units. Unlike the existing methods, this conversion is user-independent and does not require a paired dataset for whispered and normal speech. The UTS decoder can reconstruct speech in any target speaker's voice from speech units, and it requires only an unlabeled target speaker's speech data. We confirmed that the quality of the speech converted from a whisper was improved while preserving its natural prosody. Additionally, we confirmed the effectiveness of the proposed approach to perform speech reconstruction for people with speech or h【12】 Fine-grained Emotional Control of Text-To-Speech: Learning To Rank Inter- And Intra-Class Emotion Intensities
标题：文本到语音转换的细粒度情绪控制：学习对班级间和班内情绪强度进行排序
链接：http://arxiv.org/pdf/2303.01508v1
*与cs.SD语音【11】为同一篇
作者：Shijun Wang,Jón Guðnason,Damian Borth机构：University of St.Gallen, Switzerland, Reykjavik University, Iceland备注：Accepted by ICASSP2023摘要：最先进的文本到语音（TTS）模型能够产生高质量的语音。然而，所生成的语音在情感表达上通常是中性的，而人们常常希望对单词或音素进行细粒度的情感控制。尽管仍然具有挑战性，但最近已经提出了能够通过手动分配情感强度来控制语音的第一个TTS模型。不幸的是，由于忽略了类内距离，强度差异往往是不可识别的。本文提出了一种细粒度可控的情感TTS，它同时考虑了类间和类内距离，能够合成具有可识别强度差异的语音。主观和客观实验表明，该模型在可控性、情感表达性和自然性方面均优于现有的两种可控TTS模型。摘要：State-of-the-art Text-To-Speech (TTS) models are capable of producing high-quality speech. The generated speech, however, is usually neutral in emotional expression, whereas very often one would want fine-grained emotional control of words or phonemes. Although still challenging, the first TTS models have been recently proposed that are able to control voice by manually assigning emotion intensity. Unfortunately, due to the neglect of intra-class distance, the intensity differences are often unrecognizable. In this paper, we propose a fine-grained controllable emotional TTS, that considers both inter- and intra-class distances and be able to synthesize speech with recognizable intensity difference. Our subjective and objective experiments demonstrate that our model exceeds two state-of-the-art controllable TTS models for controllability, emotion expressiveness and naturalness.【13】 Defending against Adversarial Audio via Diffusion Model
标题：利用扩散模型防御恶意音频
链接：http://arxiv.org/pdf/2303.01507v1
*与cs.SD语音【12】为同一篇
作者：Shutong Wu,Jiongxiao Wang,Wei Ping,Weili Nie,Chaowei Xiao机构：Arizona State University, Shanghai Jiao Tong University, NVIDIA摘要：近年来，深度学习模型已经广泛应用于商业声学系统中。然而，对抗性音频示例可能导致这些声学系统的异常行为，同时人类难以感知。已经提出了诸如基于变换的防御和对抗性训练的各种方法来保护声学系统免受对抗性攻击，但是它们对于自适应攻击不太有效。此外，由于音频数据的独特属性，直接应用来自图像域的方法可能导致次优结果。在本文中，我们提出了一个对抗性的净化为基础的防御管道，音频纯，为声学系统通过现成的扩散模型。利用扩散模型强大的生成能力，AudioPure首先在对抗性音频中加入少量噪声，然后运行反向采样步骤，净化嘈杂的音频，恢复干净的音频。AudioPure是一种即插即用的方法，可以直接应用于任何预先训练的分类器，而无需任何微调或重新训练。我们在语音命令识别任务上进行了广泛的实验来评估AudioPure的鲁棒性。该方法可以有效地抵抗各种对抗性攻击（如$ mathcal{L}_2$或$ mathcal{L}_ infty$-norm）.在$ mathcal{L}_2$或$ mathcal{L}_ infty$-norm约束下的强自适应白盒和黑盒攻击下，该算法的鲁棒性优于现有算法（鲁棒准确率高达+20%）.此外，我们还利用随机光滑方法对$ mathcal{L}_2$-范数有界扰动进行了鲁棒性验证.我们的管道实现了比基准更高的认证精度。摘要：Deep learning models have been widely used in commercial acoustic systems in recent years. However, adversarial audio examples can cause abnormal behaviors for those acoustic systems, while being hard for humans to perceive. Various methods, such as transformation-based defenses and adversarial training, have been proposed to protect acoustic systems from adversarial attacks, but they are less effective against adaptive attacks. Furthermore, directly applying the methods from the image domain can lead to suboptimal results because of the unique properties of audio data. In this paper, we propose an adversarial purification-based defense pipeline, AudioPure, for acoustic systems via off-the-shelf diffusion models. Taking advantage of the strong generation ability of diffusion models, AudioPure first adds a small amount of noise to the adversarial audio and then runs the reverse sampling step to purify the noisy audio and recover clean audio. AudioPure is a plug-and-play method that can be directly applied to any pretrained classifier without any fine-tuning or re-training. We conduct extensive experiments on speech command recognition task to evaluate the robustness of AudioPure. Our method is effective against diverse adversarial attacks (e.g. $ mathcal{L}_2$ or $ mathcal{L}_ infty$-norm). It outperforms the existing methods under both strong adaptive white-box and black-box attacks bounded by $ mathcal{L}_2$ or $ mathcal{L}_ infty$-norm (up to +20 % in机器翻译，仅供参考
每日语音速递