今日论文合集：cs.SD语音10篇，eess.AS音频处理13篇。
cs.SD语音
【1】AVSegFormer: Audio-Visual Segmentation with Transformer
标题：AVSegFormer：基于变换的视听分割
链接：http://arxiv.org/pdf/2307.01146v1
作者：Shengyi Gao,Zhe Chen,Guo Chen,Wenhai Wang,Tong Lu机构：Nanjing University, Shanghai AI Laboratory备注：9 pages, 7 figures摘要：音频和视觉的结合长期以来一直是多模态社区感兴趣的话题。最近，已经引入了一种新的视听分割（AVS）任务，旨在定位和分割给定视频中的声音对象。这项任务首次需要音频驱动的像素级场景理解，这构成了重大挑战。在本文中，我们提出了AVSegFormer，一个新的框架AVS任务，利用Transformer架构。具体来说，我们引入音频查询和学习查询到Transformer解码器，使网络有选择地关注感兴趣的视觉功能。此外，我们提出了一个视听混合器，它可以动态调整视觉特征的放大和抑制不相关的空间通道。此外，我们设计了一个中间掩码损失，以加强监督的解码器，鼓励网络产生更准确的中间预测。大量的实验表明，AVSegFormer实现了最先进的AVS基准测试结果。该代码可在www.example.com获得https: github.com vvvb-github AVSegFormer。摘要：The combination of audio and vision has long been a topic of interest in the multi-modal community. Recently, a new audio-visual segmentation (AVS) task has been introduced, aiming to locate and segment the sounding objects in a given video. This task demands audio-driven pixel-level scene understanding for the first time, posing significant challenges. In this paper, we propose AVSegFormer, a novel framework for AVS tasks that leverages the transformer architecture. Specifically, we introduce audio queries and learnable queries into the transformer decoder, enabling the network to selectively attend to interested visual features. Besides, we present an audio-visual mixer, which can dynamically adjust visual features by amplifying relevant and suppressing irrelevant spatial channels. Additionally, we devise an intermediate mask loss to enhance the supervision of the decoder, encouraging the network to produce more accurate intermediate predictions. Extensive experiments demonstrate that AVSegFormer achieves state-of-the-art results on the AVS benchmark. The code is available at https: github.com vvvb-github AVSegFormer.
【2】musif: a Python package for symbolic music feature extraction
标题：Musif：一个用于符号音乐特征提取的Python包
链接：http://arxiv.org/pdf/2307.01120v1
作者：Ana Llorens,Federico Simonetta,Martín Serrano,Álvaro Torrente机构：Department of Musicology, Universidad Complutense de Madrid, Madrid, Spain, Instituto Complutense de Ciencias Musicales, Madrid, Spain备注：Published at the Sound and Music Computing Conference 2023摘要：在这项工作中，我们介绍musif，一个Python包，便于自动提取的功能，从符号乐谱。该软件包包括大量功能的实现，这些功能是由音乐学，音乐理论，统计学和计算机科学的专家团队开发的。此外，该包允许使用常用的Python库轻松创建自定义功能。musif主要用于处理以MusicXML格式编码的高质量音乐学数据，但也支持音乐信息检索任务中常用的其他格式，包括MIDI，MEI，Kern等。我们提供了全面的文档和教程，以帮助扩展框架，并促进新的和没有经验的用户介绍其使用。摘要：In this work, we introduce musif, a Python package that facilitates the automatic extraction of features from symbolic music scores. The package includes the implementation of a large number of features, which have been developed by a team of experts in musicology, music theory, statistics, and computer science. Additionally, the package allows for the easy creation of custom features using commonly available Python libraries. musif is primarily geared towards processing high-quality musicological data encoded in MusicXML format, but also supports other formats commonly used in music information retrieval tasks, including MIDI, MEI, Kern, and others. We provide comprehensive documentation and tutorials to aid in the extension of the framework and to facilitate the introduction of new and inexperienced users to its usage.
【3】Multilingual Contextual Adapters To Improve Custom Word Recognition In Low-resource Languages标题：改进低资源语言中自定义单词识别的多语言上下文适配器链接：http://arxiv.org/pdf/2307.00759v1
作者：Devang Kulshreshtha,Saket Dingliwal,Brady Houston,Sravan Bodapati机构：AWS AI Labs备注：Published at INTERSPEECH 2023摘要：连接时态分类（CTC）模型在自动语音识别（ASR）中的速度和性能之间取得了平衡。然而，这些CTC模型仍然在其他领域挣扎，例如个性化定制词。最近的方法探索了上下文适配器，其中CTC的基于注意力的偏置模型被用来改善自定义实体的识别。虽然这种方法在足够的数据下工作得很好，但我们展示了它对于低资源语言不是一个有效的策略。在这项工作中，我们提出了一个监督损失的上下文适配器的顺利培训。此外，我们探索了一种多语言策略，以提高有限的训练数据的性能。我们的方法实现了48%的F1改进，检索看不见的自定义实体的低资源语言。有趣的是，作为训练上下文适配器的副产品，我们看到基本CTC模型的性能也降低了5-11%的字错误率（WER）。摘要：Connectionist Temporal Classification (CTC) models are popular for their balance between speed and performance for Automatic Speech Recognition (ASR). However, these CTC models still struggle in other areas, such as personalization towards custom words. A recent approach explores Contextual Adapters, wherein an attention-based biasing model for CTC is used to improve the recognition of custom entities. While this approach works well with enough data, we showcase that it isn't an effective strategy for low-resource languages. In this work, we propose a supervision loss for smoother training of the Contextual Adapters. Further, we explore a multilingual strategy to improve performance with limited training data. Our method achieves 48% F1 improvement in retrieving unseen custom entities for a low-resource language. Interestingly, as a by-product of training the Contextual Adapters, we see a 5-11% Word Error Rate (WER) reduction in the performance of the base CTC model as well.
【4】An End-to-End Multi-Module Audio Deepfake Generation System for ADD Challenge 2023
标题：面向ADD挑战2023的端到端多模块音频深伪生成系统
链接：http://arxiv.org/pdf/2307.00729v1
作者：Sheng Zhao,Qilong Yuan,Yibo Duan,Zhuoyue Chen摘要：合成语音生成的任务是从给定的文本中生成语言内容，然后模拟出虚假的人声，决定合成语音生成效果的关键因素主要包括生成速度、分词精度、合成语音的自然度等本文建立了一个端到端多模块合成语音生成模型，包括说话人编码器、基于Tacotron2的合成器和基于WaveRNN的声码器。此外，我们在不同的数据集和各种模型结构上进行了大量的比较实验。最终，我们在ADD 2023挑战赛道1.1中以44.97%的加权欺骗成功率（WDSR）获得第一名。摘要：The task of synthetic speech generation is to generate language content from a given text, then simulating fake human voice.The key factors that determine the effect of synthetic speech generation mainly include speed of generation, accuracy of word segmentation, naturalness of synthesized speech, etc. This paper builds an end-to-end multi-module synthetic speech generation model, including speaker encoder, synthesizer based on Tacotron2, and vocoder based on WaveRNN. In addition, we perform a lot of comparative experiments on different datasets and various model structures. Finally, we won the first place in the ADD 2023 challenge Track 1.1 with the weighted deception success rate (WDSR) of 44.97%.
【5】Conformer LLMs -- Convolution Augmented Large Language Models标题：Conform LLMS--卷积增强的大型语言模型链接：http://arxiv.org/pdf/2307.00461v1
作者：Prateek Verma机构：Stanford University, Jane Stanford Way, Stanford, California备注：6 pages, 1 figure摘要：这项工作为大型语言模型（LLM）构建了两个流行的神经架构块，即卷积层和Transformers。非因果构象在自动语音识别中被广泛使用。这项工作的目的是适应这些架构在因果设置训练LLM。Transformers解码器有效地捕获了多种模式的远程依赖关系，并形成了机器学习现代进步的核心支柱。卷积架构在提取诸如原始1-D信号、语音和图像等领域中的特征方面已经很流行。在本文中，通过将局部和全局依赖性结合在潜在表示上使用因果卷积滤波器和Transformer，我们实现了显着的性能增益。这项工作展示了一个强大的语音架构，可以集成和适应的因果设置超出语音应用程序的大规模语言建模。摘要：This work builds together two popular blocks of neural architecture, namely convolutional layers and Transformers, for large language models (LLMs). Non-causal conformers are used ubiquitously in automatic speech recognition. This work aims to adapt these architectures in a causal setup for training LLMs. Transformers decoders effectively capture long-range dependencies over several modalities and form a core backbone of modern advancements in machine learning. Convolutional architectures have been popular in extracting features in domains such as raw 1-D signals, speech, and images, to name a few. In this paper, by combining local and global dependencies over latent representations using causal convolutional filters and Transformer, we achieve significant gains in performance. This work showcases a robust speech architecture that can be integrated and adapted in a causal setup beyond speech applications for large-scale language modeling.
【6】Don't Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapters
标题：不要停止自我监督：通过残差适配器实现语音表征的重音适应链接：http://arxiv.org/pdf/2307.00453v1
作者：Anshu Bhatia,Sanchit Sinha,Saket Dingliwal,Karthik Gopalakrishnan,Sravan Bodapati,Katrin Kirchhoff机构：AWS AI Labs, University of Virginia摘要：语音表示学习的自我监督的方式从大量的未标记语音语料库已经成功地适应几个下游任务。然而，这样的表示可能偏向于这样的语料库的规范数据特性，并且在非典型的非母语口音说话者群体上表现不佳。与国家的最先进的HuBERT模型作为基线，我们提出和调查的自我监督适应的语音表示，以参数有效的方式，通过训练口音特定的残差适配器。我们的实验与4口音和选择自动语音识别（ASR）的下游任务感兴趣。我们获得强大的字错误率降低（WERR）超过HuBERT大所有4个口音，平均WERR为22.7%与口音特定适配器和平均WERR为25.1%，如果整个编码器是口音适应。虽然我们的实验利用HuBERT和ASR作为下游任务，我们提出的方法是模型和任务不可知的。摘要：Speech representations learned in a self-supervised fashion from massive unlabeled speech corpora have been adapted successfully toward several downstream tasks. However, such representations may be skewed toward canonical data characteristics of such corpora and perform poorly on atypical, non-native accented speaker populations. With the state-of-the-art HuBERT model as a baseline, we propose and investigate self-supervised adaptation of speech representations to such populations in a parameter-efficient way via training accent-specific residual adapters. We experiment with 4 accents and choose automatic speech recognition (ASR) as the downstream task of interest. We obtain strong word error rate reductions (WERR) over HuBERT-large for all 4 accents, with a mean WERR of 22.7% with accent-specific adapters and a mean WERR of 25.1% if the entire encoder is accent-adapted. While our experiments utilize HuBERT and ASR as the downstream task, our proposed approach is both model and task-agnostic.
【7】Dataset balancing can hurt model performance标题：数据集平衡可能会损害模型性能
链接：http://arxiv.org/pdf/2307.00079v1
作者：R. Channing Moore,Daniel P. W. Ellis,Eduardo Fonseca,Shawn Hershey,Aren Jansen,Manoj Plakal机构：Google Research, Mountain View, CA, and New York, NY, USAJournal-ref：ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5摘要：从每个类样本的倾斜分布的训练数据中进行机器学习，可以导致模型在常见类上以牺牲稀有类的性能为代价来支持性能。AudioSet在其527个声音事件类中有非常广泛的先验。AudioSet上的分类性能通常通过对每个类度量的简单平均来评估，这意味着稀有类上的性能在重要性上与常见类上的性能相同。最近的几篇论文使用了数据集平衡技术来提高AudioSet的性能。然而，我们发现，虽然平衡提高了公共AudioSet评估数据的性能，它同时伤害了在相同条件下收集的未发布评估集的性能。通过改变平衡的程度，我们表明，它的好处是脆弱的，取决于评估集。我们也没有发现证据表明，平衡提高罕见类的性能相对于普通类。因此，我们告诫不要盲目地应用平衡，也不要过分关注公共评价集的微小改进。摘要：Machine learning from training data with a skewed distribution of examples per class can lead to models that favor performance on common classes at the expense of performance on rare ones. AudioSet has a very wide range of priors over its 527 sound event classes. Classification performance on AudioSet is usually evaluated by a simple average over per-class metrics, meaning that performance on rare classes is equal in importance to the performance on common ones. Several recent papers have used dataset balancing techniques to improve performance on AudioSet. We find, however, that while balancing improves performance on the public AudioSet evaluation data it simultaneously hurts performance on an unpublished evaluation set collected under the same conditions. By varying the degree of balancing, we show that its benefits are fragile and depend on the evaluation set. We also do not find evidence indicating that balancing improves rare class performance relative to common classes. We therefore caution against blind application of balancing, as well as against paying too much attention to small improvements on a public evaluation set.
【8】CASEIN: Cascading Explicit and Implicit Control for Fine-grained Emotion Intensity Regulation标题：酪蛋白：细粒度情绪强度调控的显性和隐性级联控制链接：http://arxiv.org/pdf/2307.00020v1
作者：Yuhao Cui,Xiongwei Wang,Zhongzhou Zhao,Wei Zhou,Haiqing Chen机构：Alibaba Group, China备注：Accepted at Interspeech 2023摘要：现有的细粒度强度调节方法依赖于通过预测情绪概率的显式控制。然而，这些高层次的语义概率往往是不准确的和不顺利的音素水平，导致学习的偏见。特别是当我们试图混合特定音素的多种情感强度时，导致合成的可控性和自然性明显降低。为了解决这个问题，我们提出了CAScaded显式和隐式控制框架（CASEIN），它利用准确的解纠缠的情绪流形从参考语音学习隐式表示在较低的语义水平。这种表示弥合了显式概率和合成模型之间的语义差距，减少了学习中的偏差。在实验中，我们的CASEIN超越现有的方法在可控性和自然性。值得注意的是，我们是第一个实现对多种情绪的混合强度进行细粒度控制的人。摘要：Existing fine-grained intensity regulation methods rely on explicit control through predicted emotion probabilities. However, these high-level semantic probabilities are often inaccurate and unsmooth at the phoneme level, leading to bias in learning. Especially when we attempt to mix multiple emotion intensities for specific phonemes, resulting in markedly reduced controllability and naturalness of the synthesis. To address this issue, we propose the CAScaded Explicit and Implicit coNtrol framework (CASEIN), which leverages accurate disentanglement of emotion manifolds from the reference speech to learn the implicit representation at a lower semantic level. This representation bridges the semantical gap between explicit probabilities and the synthesis model, reducing bias in learning. In experiments, our CASEIN surpasses existing methods in both controllability and naturalness. Notably, we are the first to achieve fine-grained control over the mixed intensity of multiple emotions.
【9】The Human Auditory System and Audio标题：人类的听觉系统和音频链接：http://arxiv.org/pdf/2307.00084v1
作者：Milind N. Kunchur备注：32 pages + contents, 22 figures, 19 equations, 221 cited references摘要：这项工作回顾了人类的听觉系统，阐明了一些专门的机制和非线性路径沿物理声音和感知之间的事件链。频率、时间和相位之间的习惯关系--例如不确定性原理--适用于线性系统，并不直接适用于听觉过程。某些过程的听觉时间分辨率可以是信号周期的百分之一，并且可以向下延伸到微秒时间尺度。来自3500个内毛细胞的30000个听觉神经纤维的神经兴奋模式所对应的惊人的大量变化，说明了听觉系统对声音细节的分辨率的巨大能力。耳朵的灵敏度足够高，可以检测到一个皮米级别的基底膜振幅，或者说比原子小100倍。本文调查和提供了一些令人印象深刻的人类听觉系统的能力的新见解，并探讨了它们的关系，在复制的声音保真度。摘要：This work reviews the human auditory system, elucidating some of the specialized mechanisms and non-linear pathways along the chain of events between physical sound and its perception. Customary relationships between frequency, time, and phase--such as the uncertainty principle--that hold for linear systems, do not apply straightforwardly to the hearing process. Auditory temporal resolution for certain processes can be a hundredth of the period of the signal, and can extend down to the microseconds time scale. The astonishingly large number of variations that correspond to the neural excitation pattern of 30000 auditory nerve fibers, originating from 3500 inner hair cells, explicates the vast capacity of the auditory system for the resolution of sonic detail. And the ear is sensitive enough to detect a basilar-membrane amplitude at the level of a picometer, or about a hundred times smaller than an atom. This article surveys and provides new insights into some of the impressive capabilities of the human auditory system and explores their relationship to fidelity in reproduced sound.
【10】EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech标题：EmoSpeech：引导FastSpeech2走向情感文本到语音链接：http://arxiv.org/pdf/2307.00024v1
作者：Daria Diatlova,Vitaly Shutov机构：VK, deepvk, Saint Petersburg, Russia摘要：最先进的语音合成模型试图尽可能接近人类的声音。因此，情感建模是文本到语音（TTS）研究的重要组成部分。在我们的工作中，我们选择了FastSpeech2作为起点，并提出了一系列的修改来合成情感语音。根据自动和人类的评价，我们的模型，EmoSpeech，超过现有的模型在生成的语音MOS得分和情感识别的准确性。我们为FastSpeech2架构的每个扩展提供了详细的消融研究，以形成EmoSpeech。文本中情感的不均匀分布对于更好的合成语音和语调感知至关重要。我们的模型包括一个调节机制，通过允许情绪以不同的强度水平对每个电话做出贡献来有效地处理这个问题。人类评估表明，所提出的修改生成具有更高MOS和情感表现力的音频。摘要：State-of-the-art speech synthesis models try to get as close as possible to the human voice. Hence, modelling emotions is an essential part of Text-To-Speech (TTS) research. In our work, we selected FastSpeech2 as the starting point and proposed a series of modifications for synthesizing emotional speech. According to automatic and human evaluation, our model, EmoSpeech, surpasses existing models regarding both MOS score and emotion recognition accuracy in generated speech. We provided a detailed ablation study for every extension to FastSpeech2 architecture that forms EmoSpeech. The uneven distribution of emotions in the text is crucial for better, synthesized speech and intonation perception. Our model includes a conditioning mechanism that effectively handles this issue by allowing emotions to contribute to each phone with varying intensity levels. The human assessment indicates that proposed modifications generate audio with higher MOS and emotional expressiveness.
eess.AS音频处理
【1】Using joint training speaker encoder with consistency loss to achieve cross-lingual voice conversion and expressive voice conversion
标题：利用具有一致性损失的联合训练说话人编码器实现跨语言语音转换和表现力语音转换
链接：http://arxiv.org/pdf/2307.00393v1
作者：Houjian Guo,Chaoran Liu,Carlos Toshinori Ishi,Hiroshi Ishiguro机构：Interactive Robot Research Team, Guardian Robot Project, RIKEN, Japan, Graduate School of Engineering Science, Osaka University, Japan, Advanced Telecommunications Research Institute International, Japan摘要：语音转换系统在常见语音转换任务中的自然性和相似性方面已经取得了显著的进步。然而，他们的表现在更复杂的任务，如跨语言语音转换和表达语音转换仍然不完美。在这项研究中，我们提出了一种新的方法，结合了一个联合训练的说话人编码器和内容特征的跨语言语音识别模型Whisper提取，以实现高质量的跨语言语音转换。此外，我们引入了一个说话者一致性损失的联合编码器，这提高了转换后的语音和参考语音之间的相似性。为了进一步探索的联合说话人编码器的能力，我们使用的语音posteriorgram的内容特征，这使得模型能够有效地再现说话人的特征和参考语音的情感方面。摘要：Voice conversion systems have made significant advancements in terms of naturalness and similarity in common voice conversion tasks. However, their performance in more complex tasks such as cross-lingual voice conversion and expressive voice conversion remains imperfect. In this study, we propose a novel approach that combines a jointly trained speaker encoder and content features extracted from the cross-lingual speech recognition model Whisper to achieve high-quality cross-lingual voice conversion. Additionally, we introduce a speaker consistency loss to the joint encoder, which improves the similarity between the converted speech and the reference speech. To further explore the capabilities of the joint speaker encoder, we use the phonetic posteriorgram as the content feature, which enables the model to effectively reproduce both the speaker characteristics and the emotional aspects of the reference speech.
【2】Enhancing the EEG Speech Match Mismatch Tasks With Word Boundaries标题：利用词边界增强EEG语音匹配失配任务链接：http://arxiv.org/pdf/2307.00366v1
作者：Akshara Soman,Vidhi Sinha,Sriram Ganapathy机构：LEAP Lab, Indian Institute of Science Bangalore, India备注：5 pages, 4 figures, 4 tables, accepted to Interspeech2023 conference摘要：最近的研究表明，人类语音理解的潜在神经机制，可以分析使用的语音刺激和神经反应的匹配-不匹配分类。然而，这样的研究已经进行了固定的持续时间段没有考虑到离散的处理语音在大脑中。在这项工作中，我们建立了词的边界信息在句子处理中起着重要的作用，脑电的语音输入。我们使用卷积层网络处理语音和EEG信号。然后，对表示执行基于词边界的平均池化，并且使用递归层并入词间上下文。实验表明，建模精度可以显着提高（匹配不匹配的分类精度）93%的公开可用的语音EEG数据集，而以前的努力实现了65-75%的准确性，这项任务。摘要：Recent studies have shown that the underlying neural mechanisms of human speech comprehension can be analyzed using a match-mismatch classification of the speech stimulus and the neural response. However, such studies have been conducted for fixed-duration segments without accounting for the discrete processing of speech in the brain. In this work, we establish that word boundary information plays a significant role in sentence processing by relating EEG to its speech input. We process the speech and the EEG signals using a network of convolution layers. Then, a word boundary-based average pooling is performed on the representations, and the inter-word context is incorporated using a recurrent layer. The experiments show that the modeling accuracy can be significantly improved (match-mismatch classification accuracy) to 93% on a publicly available speech-EEG data set, while previous efforts achieved an accuracy of 65-75% for this task.
【3】VoxWatch: An open-set speaker recognition benchmark on VoxCeleb标题：VoxWatch：一种基于VoxCeleb的开放式说话人识别基准链接：http://arxiv.org/pdf/2307.00169v1
作者：Raghuveer Peri,Seyed Omid Sadjadi,Daniel Garcia-Romero机构：AWS AI Labs备注：8 pages摘要：尽管开放集说话人识别（OSI）具有广泛的实际应用，如在欺诈预防，但在说话人识别社区相比，说话人验证（SV）受到较少的关注。OSI处理确定测试语音样本是否属于来自一组预先登记的个体（集合内）的说话者，或者它是否来自集合外的说话者。除了与语音可变性相关的典型挑战外，OSI还容易出现“假警报问题”;随着设置内扬声器群体（也称为监视列表）的大小的增长，设置外得分变得更大，导致增加的虚警率。这对于金融机构和边境安全中的应用尤其具有挑战性，其中监视列表大小通常为数千个扬声器的数量级。因此，重要的是系统地量化误报问题，并开发技术，减轻监视列表大小对检测性能的影响。以前对这个问题的研究很少，缺乏一个共同的基准进行系统评估。在本文中，我们提出了第一个公开的基准OSI，开发使用VoxCeleb数据集。我们量化的监视列表大小和语音持续时间的影响，基于监视列表的说话人检测任务使用三个强大的神经网络为基础的系统。与以往的研究结果相比，我们表明，普遍采用的自适应分数归一化并不能保证提高这项任务的性能。另一方面，我们表明，得分校准和得分融合，两个其他常用的技术SV，结果显着改善OSI性能。摘要：Despite its broad practical applications such as in fraud prevention, open-set speaker identification (OSI) has received less attention in the speaker recognition community compared to speaker verification (SV). OSI deals with determining if a test speech sample belongs to a speaker from a set of pre-enrolled individuals (in-set) or if it is from an out-of-set speaker. In addition to the typical challenges associated with speech variability, OSI is prone to the "false-alarm problem"; as the size of the in-set speaker population (a.k.a watchlist) grows, the out-of-set scores become larger, leading to increased false alarm rates. This is in particular challenging for applications in financial institutions and border security where the watchlist size is typically of the order of several thousand speakers. Therefore, it is important to systematically quantify the false-alarm problem, and develop techniques that alleviate the impact of watchlist size on detection performance. Prior studies on this problem are sparse, and lack a common benchmark for systematic evaluations. In this paper, we present the first public benchmark for OSI, developed using the VoxCeleb dataset. We quantify the effect of the watchlist size and speech duration on the watchlist-based speaker detection task using three strong neural network based systems. In contrast to the findings from prior research, we show that the commonly adopted adaptive score normalization is not guaranteed to improve the performance for this task. On the other hand, we show that score calibration and score fusion, two other commonly used techniques in SV, result in significant improvements in OSI performance.
【4】EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech标题：EmoSpeech：引导FastSpeech2走向情感文本到语音链接：http://arxiv.org/pdf/2307.00024v1
*与cs.SD语音【10】为同一篇
作者：Daria Diatlova,Vitaly Shutov机构：VK, deepvk, Saint Petersburg, Russia摘要：最先进的语音合成模型试图尽可能接近人类的声音。因此，情感建模是文本到语音（TTS）研究的重要组成部分。在我们的工作中，我们选择了FastSpeech2作为起点，并提出了一系列的修改来合成情感语音。根据自动和人类的评价，我们的模型，EmoSpeech，超过现有的模型在生成的语音MOS得分和情感识别的准确性。我们为FastSpeech2架构的每个扩展提供了详细的消融研究，以形成EmoSpeech。文本中情感的不均匀分布对于更好的合成语音和语调感知至关重要。我们的模型包括一个调节机制，通过允许情绪以不同的强度水平对每个电话做出贡献来有效地处理这个问题。人类评估表明，所提出的修改生成具有更高MOS和情感表现力的音频。摘要：State-of-the-art speech synthesis models try to get as close as possible to the human voice. Hence, modelling emotions is an essential part of Text-To-Speech (TTS) research. In our work, we selected FastSpeech2 as the starting point and proposed a series of modifications for synthesizing emotional speech. According to automatic and human evaluation, our model, EmoSpeech, surpasses existing models regarding both MOS score and emotion recognition accuracy in generated speech. We provided a detailed ablation study for every extension to FastSpeech2 architecture that forms EmoSpeech. The uneven distribution of emotions in the text is crucial for better, synthesized speech and intonation perception. Our model includes a conditioning mechanism that effectively handles this issue by allowing emotions to contribute to each phone with varying intensity levels. The human assessment indicates that proposed modifications generate audio with higher MOS and emotional expressiveness.
【5】AVSegFormer: Audio-Visual Segmentation with Transformer
标题：AVSegFormer：基于变换的视听分割
链接：http://arxiv.org/pdf/2307.01146v1
*与cs.SD语音【1】为同一篇
作者：Shengyi Gao,Zhe Chen,Guo Chen,Wenhai Wang,Tong Lu机构：Nanjing University, Shanghai AI Laboratory备注：9 pages, 7 figures摘要：音频和视觉的结合长期以来一直是多模态社区感兴趣的话题。最近，已经引入了一种新的视听分割（AVS）任务，旨在定位和分割给定视频中的声音对象。这项任务首次需要音频驱动的像素级场景理解，这构成了重大挑战。在本文中，我们提出了AVSegFormer，一个新的框架AVS任务，利用Transformer架构。具体来说，我们引入音频查询和学习查询到Transformer解码器，使网络有选择地关注感兴趣的视觉功能。此外，我们提出了一个视听混合器，它可以动态调整视觉特征的放大和抑制不相关的空间通道。此外，我们设计了一个中间掩码损失，以加强监督的解码器，鼓励网络产生更准确的中间预测。大量的实验表明，AVSegFormer实现了最先进的AVS基准测试结果。该代码可在www.example.com获得https: github.com vvvb-github AVSegFormer。摘要：The combination of audio and vision has long been a topic of interest in the multi-modal community. Recently, a new audio-visual segmentation (AVS) task has been introduced, aiming to locate and segment the sounding objects in a given video. This task demands audio-driven pixel-level scene understanding for the first time, posing significant challenges. In this paper, we propose AVSegFormer, a novel framework for AVS tasks that leverages the transformer architecture. Specifically, we introduce audio queries and learnable queries into the transformer decoder, enabling the network to selectively attend to interested visual features. Besides, we present an audio-visual mixer, which can dynamically adjust visual features by amplifying relevant and suppressing irrelevant spatial channels. Additionally, we devise an intermediate mask loss to enhance the supervision of the decoder, encouraging the network to produce more accurate intermediate predictions. Extensive experiments demonstrate that AVSegFormer achieves state-of-the-art results on the AVS benchmark. The code is available at https: github.com vvvb-github AVSegFormer.
【6】musif: a Python package for symbolic music feature extraction
标题：Musif：一个用于符号音乐特征提取的Python包
链接：http://arxiv.org/pdf/2307.01120v1
*与cs.SD语音【2】为同一篇
作者：Ana Llorens,Federico Simonetta,Martín Serrano,Álvaro Torrente机构：Department of Musicology, Universidad Complutense de Madrid, Madrid, Spain, Instituto Complutense de Ciencias Musicales, Madrid, Spain备注：Published at the Sound and Music Computing Conference 2023摘要：在这项工作中，我们介绍musif，一个Python包，便于自动提取的功能，从符号乐谱。该软件包包括大量功能的实现，这些功能是由音乐学，音乐理论，统计学和计算机科学的专家团队开发的。此外，该包允许使用常用的Python库轻松创建自定义功能。musif主要用于处理以MusicXML格式编码的高质量音乐学数据，但也支持音乐信息检索任务中常用的其他格式，包括MIDI，MEI，Kern等。我们提供了全面的文档和教程，以帮助扩展框架，并促进新的和没有经验的用户介绍其使用。摘要：In this work, we introduce musif, a Python package that facilitates the automatic extraction of features from symbolic music scores. The package includes the implementation of a large number of features, which have been developed by a team of experts in musicology, music theory, statistics, and computer science. Additionally, the package allows for the easy creation of custom features using commonly available Python libraries. musif is primarily geared towards processing high-quality musicological data encoded in MusicXML format, but also supports other formats commonly used in music information retrieval tasks, including MIDI, MEI, Kern, and others. We provide comprehensive documentation and tutorials to aid in the extension of the framework and to facilitate the introduction of new and inexperienced users to its usage.
【7】ContextSpeech: Expressive and Efficient Text-to-Speech for Paragraph Reading
标题：ConextSpeech：用于段落阅读的富有表现力和高效的文语转换
链接：http://arxiv.org/pdf/2307.00782v1
作者：Yujia Xiao,Shaofei Zhang,Xi Wang,Xu Tan,Lei He,Sheng Zhao,Frank K. Soong,Tan Lee机构：Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, Microsoft, China备注：5 pages, 4 figures, accepted by INTERSPEECH 2023摘要：虽然最先进的文本到语音系统可以在句子级别生成非常高质量的自然语音，但它们在段落 长形式阅读的语音生成方面仍然面临巨大挑战。这种缺陷是由于i）忽略跨句子上下文信息，以及ii）长形式合成的高计算和存储成本。为了解决这些问题，这项工作开发了一个轻量级而有效的TTS系统，ContextSpeech。具体来说，我们首先设计了一个内存缓存的递归机制，将全局文本和语音上下文纳入句子编码。然后，我们构建层次结构的文本语义，以扩大范围的全球上下文增强。此外，我们集成线性化的自我注意，以提高模型的效率。实验结果表明，ContextSpeech显著提高了语音质量和韵律表达能力，并具有竞争力的模型效率。音频样本可在以下网址获得：https: contextspeech.github.io demo摘要：While state-of-the-art Text-to-Speech systems can generate natural speech of very high quality at sentence level, they still meet great challenges in speech generation for paragraph long-form reading. Such deficiencies are due to i) ignorance of cross-sentence contextual information, and ii) high computation and memory cost for long-form synthesis. To address these issues, this work develops a lightweight yet effective TTS system, ContextSpeech. Specifically, we first design a memory-cached recurrence mechanism to incorporate global text and speech context into sentence encoding. Then we construct hierarchically-structured textual semantics to broaden the scope for global context enhancement. Additionally, we integrate linearized self-attention to improve model efficiency. Experiments show that ContextSpeech significantly improves the voice quality and prosody expressiveness in paragraph reading with competitive model efficiency. Audio samples are available at: https: contextspeech.github.io demo
【8】Multilingual Contextual Adapters To Improve Custom Word Recognition In Low-resource Languages标题：改进低资源语言中自定义单词识别的多语言上下文适配器链接：http://arxiv.org/pdf/2307.00759v1
*与cs.SD语音【3】为同一篇
作者：Devang Kulshreshtha,Saket Dingliwal,Brady Houston,Sravan Bodapati机构：AWS AI Labs备注：Published at INTERSPEECH 2023摘要：连接时态分类（CTC）模型在自动语音识别（ASR）中的速度和性能之间取得了平衡。然而，这些CTC模型仍然在其他领域挣扎，例如个性化定制词。最近的方法探索了上下文适配器，其中CTC的基于注意力的偏置模型被用来改善自定义实体的识别。虽然这种方法在足够的数据下工作得很好，但我们展示了它对于低资源语言不是一个有效的策略。在这项工作中，我们提出了一个监督损失的上下文适配器的顺利培训。此外，我们探索了一种多语言策略，以提高有限的训练数据的性能。我们的方法实现了48%的F1改进，检索看不见的自定义实体的低资源语言。有趣的是，作为训练上下文适配器的副产品，我们看到基本CTC模型的性能也降低了5-11%的字错误率（WER）。摘要：Connectionist Temporal Classification (CTC) models are popular for their balance between speed and performance for Automatic Speech Recognition (ASR). However, these CTC models still struggle in other areas, such as personalization towards custom words. A recent approach explores Contextual Adapters, wherein an attention-based biasing model for CTC is used to improve the recognition of custom entities. While this approach works well with enough data, we showcase that it isn't an effective strategy for low-resource languages. In this work, we propose a supervision loss for smoother training of the Contextual Adapters. Further, we explore a multilingual strategy to improve performance with limited training data. Our method achieves 48% F1 improvement in retrieving unseen custom entities for a low-resource language. Interestingly, as a by-product of training the Contextual Adapters, we see a 5-11% Word Error Rate (WER) reduction in the performance of the base CTC model as well.
【9】Don't Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapters
标题：不要停止自我监督：通过残差适配器实现语音表征的重音适应链接：http://arxiv.org/pdf/2307.00453v1
*与cs.SD语音【6】为同一篇
作者：Anshu Bhatia,Sanchit Sinha,Saket Dingliwal,Karthik Gopalakrishnan,Sravan Bodapati,Katrin Kirchhoff机构：AWS AI Labs, University of Virginia摘要：语音表示学习的自我监督的方式从大量的未标记语音语料库已经成功地适应几个下游任务。然而，这样的表示可能偏向于这样的语料库的规范数据特性，并且在非典型的非母语口音说话者群体上表现不佳。与国家的最先进的HuBERT模型作为基线，我们提出和调查的自我监督适应的语音表示，以参数有效的方式，通过训练口音特定的残差适配器。我们的实验与4口音和选择自动语音识别（ASR）的下游任务感兴趣。我们获得强大的字错误率降低（WERR）超过HuBERT大所有4个口音，平均WERR为22.7%与口音特定适配器和平均WERR为25.1%，如果整个编码器是口音适应。虽然我们的实验利用HuBERT和ASR作为下游任务，我们提出的方法是模型和任务不可知的。摘要：Speech representations learned in a self-supervised fashion from massive unlabeled speech corpora have been adapted successfully toward several downstream tasks. However, such representations may be skewed toward canonical data characteristics of such corpora and perform poorly on atypical, non-native accented speaker populations. With the state-of-the-art HuBERT model as a baseline, we propose and investigate self-supervised adaptation of speech representations to such populations in a parameter-efficient way via training accent-specific residual adapters. We experiment with 4 accents and choose automatic speech recognition (ASR) as the downstream task of interest. We obtain strong word error rate reductions (WERR) over HuBERT-large for all 4 accents, with a mean WERR of 22.7% with accent-specific adapters and a mean WERR of 25.1% if the entire encoder is accent-adapted. While our experiments utilize HuBERT and ASR as the downstream task, our proposed approach is both model and task-agnostic.
【10】What do self-supervised speech models know about words?
标题：自我监督语音模型对单词了解多少？
链接：http://arxiv.org/pdf/2307.00162v1
作者：Ankita Pasad,Chung-Ming Chien,Shane Settle,Karen Livescu机构：Toyota Technological Institute at Chicago摘要：在过去的几年中，已经引入了许多自监督语音模型（S3M），产生了各种语音任务的性能和数据效率的改进。越来越多的证据表明，不同的S3M在不同的层编码语言信息，并且一些S3M似乎学习类似于语音的子词单位。然而，这些模型在多大程度上捕获了更大的语言单位，如单词，以及单词相关信息在哪里编码，仍然不清楚。在这项研究中，我们进行了几个分析的词段表示提取的不同层的三个S3M：wav2vec2、HuBERT和WavLM。我们采用典型相关分析（CCA），轻量级的分析工具，来衡量这些表示和词级语言属性之间的相似性。我们发现，最大的词级语言内容往往被发现在中间模型层，而一些较低层次的信息，如发音也保留在高层HuBERT和WavLM。句法和语义词属性具有相似的分层行为。我们还发现，对于所有测试的模型，词的身份信息集中在每个词段的中心附近。然后，我们测试了相同模型的分层性能，当直接使用时，没有额外的学习参数，在几个任务：声学单词辨别、分词和语义句子相似性。我们发现类似的分层趋势的性能，而且，发现，当使用性能最好的HuBERT或WavLM层时，有可能实现的分词和句子相似性的性能，竞争对手更复杂的现有方法。摘要：Many self-supervised speech models (S3Ms) have been introduced over the last few years, producing performance and data efficiency improvements for a variety of speech tasks. Evidence is emerging that different S3Ms encode linguistic information in different layers, and also that some S3Ms appear to learn phone-like sub-word units. However, the extent to which these models capture larger linguistic units, such as words, and where word-related information is encoded, remains unclear. In this study, we conduct several analyses of word segment representations extracted from different layers of three S3Ms: wav2vec2, HuBERT, and WavLM. We employ canonical correlation analysis (CCA), a lightweight analysis tool, to measure the similarity between these representations and word-level linguistic properties. We find that the maximal word-level linguistic content tends to be found in intermediate model layers, while some lower-level information like pronunciation is also retained in higher layers of HuBERT and WavLM. Syntactic and semantic word attributes have similar layer-wise behavior. We also find that, for all of the models tested, word identity information is concentrated near the center of each word segment. We then test the layer-wise performance of the same models, when used directly with no additional learned parameters, on several tasks: acoustic word discrimination, word segmentation, and semantic sentence similarity. We find similar layer-wise trends in performance, and furthermore, find that when using the best-performing layer of HuBERT or WavLM, it is possible to achieve performance on word segmentation and sentence similarity that rivals more complex existing approaches.
【11】The Human Auditory System and Audio标题：人类的听觉系统和音频链接：http://arxiv.org/pdf/2307.00084v1
*与cs.SD语音【9】为同一篇
作者：Milind N. Kunchur备注：32 pages + contents, 22 figures, 19 equations, 221 cited references摘要：这项工作回顾了人类的听觉系统，阐明了一些专门的机制和非线性路径沿物理声音和感知之间的事件链。频率、时间和相位之间的习惯关系--例如不确定性原理--适用于线性系统，并不直接适用于听觉过程。某些过程的听觉时间分辨率可以是信号周期的百分之一，并且可以向下延伸到微秒时间尺度。来自3500个内毛细胞的30000个听觉神经纤维的神经兴奋模式所对应的惊人的大量变化，说明了听觉系统对声音细节的分辨率的巨大能力。耳朵的灵敏度足够高，可以检测到一个皮米级别的基底膜振幅，或者说比原子小100倍。本文调查和提供了一些令人印象深刻的人类听觉系统的能力的新见解，并探讨了它们的关系，在复制的声音保真度。摘要：This work reviews the human auditory system, elucidating some of the specialized mechanisms and non-linear pathways along the chain of events between physical sound and its perception. Customary relationships between frequency, time, and phase--such as the uncertainty principle--that hold for linear systems, do not apply straightforwardly to the hearing process. Auditory temporal resolution for certain processes can be a hundredth of the period of the signal, and can extend down to the microseconds time scale. The astonishingly large number of variations that correspond to the neural excitation pattern of 30000 auditory nerve fibers, originating from 3500 inner hair cells, explicates the vast capacity of the auditory system for the resolution of sonic detail. And the ear is sensitive enough to detect a basilar-membrane amplitude at the level of a picometer, or about a hundred times smaller than an atom. This article surveys and provides new insights into some of the impressive capabilities of the human auditory system and explores their relationship to fidelity in reproduced sound.
【12】Dataset balancing can hurt model performance标题：数据集平衡可能会损害模型性能
链接：http://arxiv.org/pdf/2307.00079v1
*与cs.SD语音【7】为同一篇
作者：R. Channing Moore,Daniel P. W. Ellis,Eduardo Fonseca,Shawn Hershey,Aren Jansen,Manoj Plakal机构：Google Research, Mountain View, CA, and New York, NY, USAJournal-ref：ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5摘要：从每个类样本的倾斜分布的训练数据中进行机器学习，可以导致模型在常见类上以牺牲稀有类的性能为代价来支持性能。AudioSet在其527个声音事件类中有非常广泛的先验。AudioSet上的分类性能通常通过对每个类度量的简单平均来评估，这意味着稀有类上的性能在重要性上与常见类上的性能相同。最近的几篇论文使用了数据集平衡技术来提高AudioSet的性能。然而，我们发现，虽然平衡提高了公共AudioSet评估数据的性能，它同时伤害了在相同条件下收集的未发布评估集的性能。通过改变平衡的程度，我们表明，它的好处是脆弱的，取决于评估集。我们也没有发现证据表明，平衡提高罕见类的性能相对于普通类。因此，我们告诫不要盲目地应用平衡，也不要过分关注公共评价集的微小改进。摘要：Machine learning from training data with a skewed distribution of examples per class can lead to models that favor performance on common classes at the expense of performance on rare ones. AudioSet has a very wide range of priors over its 527 sound event classes. Classification performance on AudioSet is usually evaluated by a simple average over per-class metrics, meaning that performance on rare classes is equal in importance to the performance on common ones. Several recent papers have used dataset balancing techniques to improve performance on AudioSet. We find, however, that while balancing improves performance on the public AudioSet evaluation data it simultaneously hurts performance on an unpublished evaluation set collected under the same conditions. By varying the degree of balancing, we show that its benefits are fragile and depend on the evaluation set. We also do not find evidence indicating that balancing improves rare class performance relative to common classes. We therefore caution against blind application of balancing, as well as against paying too much attention to small improvements on a public evaluation set.
【13】CASEIN: Cascading Explicit and Implicit Control for Fine-grained Emotion Intensity Regulation标题：酪蛋白：细粒度情绪强度调控的显性和隐性级联控制链接：http://arxiv.org/pdf/2307.00020v1
*与cs.SD语音【8】为同一篇
作者：Yuhao Cui,Xiongwei Wang,Zhongzhou Zhao,Wei Zhou,Haiqing Chen机构：Alibaba Group, China备注：Accepted at Interspeech 2023摘要：现有的细粒度强度调节方法依赖于通过预测情绪概率的显式控制。然而，这些高层次的语义概率往往是不准确的和不顺利的音素水平，导致学习的偏见。特别是当我们试图混合特定音素的多种情感强度时，导致合成的可控性和自然性明显降低。为了解决这个问题，我们提出了CAScaded显式和隐式控制框架（CASEIN），它利用准确的解纠缠的情绪流形从参考语音学习隐式表示在较低的语义水平。这种表示弥合了显式概率和合成模型之间的语义差距，减少了学习中的偏差。在实验中，我们的CASEIN超越现有的方法在可控性和自然性。值得注意的是，我们是第一个实现对多种情绪的混合强度进行细粒度控制的人。摘要：Existing fine-grained intensity regulation methods rely on explicit control through predicted emotion probabilities. However, these high-level semantic probabilities are often inaccurate and unsmooth at the phoneme level, leading to bias in learning. Especially when we attempt to mix multiple emotion intensities for specific phonemes, resulting in markedly reduced controllability and naturalness of the synthesis. To address this issue, we propose the CAScaded Explicit and Implicit coNtrol framework (CASEIN), which leverages accurate disentanglement of emotion manifolds from the reference speech to learn the implicit representation at a lower semantic level. This representation bridges the semantical gap between explicit probabilities and the synthesis model, reducing bias in learning. In experiments, our CASEIN surpasses existing methods in both controllability and naturalness. Notably, we are the first to achieve fine-grained control over the mixed intensity of multiple emotions.

机器翻译，仅供参考