今日论文合集：cs.SD语音15篇，eess.AS音频处理13篇。本文经arXiv每日学术速递授权转载
【1】Analysis and Visualization of Musical Structure using Networks
作者：Alberto Alcalá-Alvarez,Pablo Padilla-Longoria摘要：在这篇文章中，定义和分析一个家庭的图形或网络的符号音乐信息的框架进行了讨论。这些图表涉及不同类型的元素，例如音高、和弦和节奏，以及它们之间的关系，并且是从数字乐谱中包含的定量或分类数据构建的。它们有助于立即可视化音乐特征，从而为理解音乐片段的一般结构元素提供了计算工具。从数字分数中获得的数据经历了图和网络理论的不同分析过程，例如计算其中心性度量和熵，以及检测其社区。我们分析了来自不同风格的音乐作品，并将我们的一些结果与传统音乐分析技术的结论进行了比较。摘要：In this article, a framework for defining and analysing a family of graphs or networks from symbolic music information is discussed. Such graphs concern different types of elements, such as pitches, chords and rhythms, and the relations among them, and are built from quantitative or categorical data contained in digital music scores. They are helpful in visualizing musical features at once, thus leading to a computational tool for understanding the general structural elements of a music fragment. Data obtained from a digital score undergoes different analytical procedures from graph and network theory, such as computing their centrality measures and entropy, and detecting their communities. We analyze pieces of music coming from different styles, and compare some of our results with conclusions from traditional music analysis techniques.
【2】Tailors: New Music Timbre Visualizer to Entertain Music Through Imagery作者：47 pages, 9 figures, 5 tables摘要：在本文中，我已经实现了一个音色可视化系统称为裁缝。通过与27名MIR用户的实验，发现与唯一的音乐条件和基本可视化相比，Tailors在传达音乐的音色温暖，亮度，深度，浅度，硬度，粗糙度和锐度特征方面是有效的。在音乐意象和音乐娱乐调查中，裁缝的所有分数在三种情况中都是最高的。音色意象与意象娱乐性之间的多元线性回归分析显示两者之间存在显著的正相关。通过与Fisher变换的相关系数比较，发现Tailors通过改善音乐视觉意象，使用户的音乐娱乐性更好。调查结果显示，Tailors在最佳音色表达、音乐体验和再次使用意愿方面排名第一。虽然有些用户觉得眼睛有负担，但Tailors留下了音色可视化映射规则的数据驱动方法的未来工作，以获得许多用户的同意。此外，还讨论了减少音色特征以专注于裁缝可以表达的特征，并以更艺术的方式使用空间感的裁缝未来的工作。摘要：In this paper, I have implemented a timbre visualization system called Tailors. Through the experiment with 27 MIR users, Tailors was found to be effective in conveying timbral warmth, brightness, depth, shallowness, hardness, roughness, and sharpness features of music compared to the only music condition and basic visualization. All scores of Tailors in the music imagery and music entertainment surveys were valued highest among the three conditions. Multiple linear regression analysis between timbre-imagery and imagery-entertainment shows significant and positive correlations. Coefficients comparing results from Fisher Transformation show that Tailors made user's music entertainment better through improved music visual imagery. The post-survey result represents that Tailors ranked first for the best timbre expression, music experience, and willingness to use it again. While some users felt a burden in the eye, Tailors left the future work of the data-driven approach of the mapping rule of timbre visualization to gain consent from many users. Furthermore, reducing timbre features to focus on features that Tailors can express well was also discussed, with future work of Tailors in a more artistic way using the sense of space.
【3】Vector Signal Reconstruction Sparse and Parametric Approach of direction of arrival Using Single Vector Hydrophone标题：使用单分量声纳的分量信号重建稀疏和到达方向参数化方法备注：22 pages. arXiv admin note: substantial text overlap with arXiv:2404.13568摘要：本文讨论了单矢量水听器在水声信号处理中的波达方向（DOA）估计问题。针对传统DOA估计方法在多源环境和噪声干扰下的局限性，提出了一种矢量信号重构稀疏参数化方法（VSRSPA）。该方法通过重构单个矢量水听器的信号模型，将其协方差矩阵转换为适合于稀疏参数化算法的Toeplitz结构。然后，该过程使用SPA算法对其进行优化，以实现更准确的DOA估计。通过详细的仿真分析，验证了该算法在单目标和双目标情况下，特别是在各种信噪比条件下的DOA估计性能。仿真结果表明，与传统的DOA估计方法相比，该算法在估计精度和分辨力方面具有明显的优势，特别是在多源信号和低信噪比环境下。本研究的贡献在于为复杂环境下的单矢量水听器波达方向估计提供了一种有效的新方法，为矢量水听器信号处理领域引入了新的研究方向和解决方案。摘要：This article discusses the application of single vector hydrophones in the field of underwater acoustic signal processing for Direction Of Arrival (DOA) estimation. Addressing the limitations of traditional DOA estimation methods in multi-source environments and under noise interference, this study introduces a Vector Signal Reconstruction Sparse and Parametric Approach (VSRSPA). This method involves reconstructing the signal model of a single vector hydrophone, converting its covariance matrix into a Toeplitz structure suitable for the Sparse and Parametric Approach (SPA) algorithm. The process then optimizes it using the SPA algorithm to achieve more accurate DOA estimation. Through detailed simulation analysis, this research has confirmed the performance of the proposed algorithm in single and dual-target DOA estimation scenarios, especially under various signal-to-noise ratio(SNR) conditions. The simulation results show that, compared to traditional DOA estimation methods, this algorithm has significant advantages in estimation accuracy and resolution, particularly in multi-source signals and low SNR environments. The contribution of this study lies in providing an effective new method for DOA estimation with single vector hydrophones in complex environments, introducing new research directions and solutions in the field of vector hydrophone signal processing.【4】Every Breath You Don't Take: Deepfake Speech Detection Using Breath标题：你没有呼吸的每一次呼吸：使用呼吸进行Deepfake语音检测作者：Seth Layton,Thiago De Andrade,Daniel Olszewski,Kevin Warren,Carrie Gates,Kevin Butler,Patrick Traynor备注：Submitted to ACM journal -- Digital Threats: Research and Practice摘要：Deepfake言论对系统和社会构成了真正且不断增长的威胁。已经创建了许多检测器来帮助防御语音deepfake。虽然这些检测器实现了无数的方法，但许多检测器依赖于语音生成过程的低级片段。我们假设，呼吸，一种更高级别的语音部分，是自然语音的关键组成部分，因此在deepfake语音中不正确的生成是一种性能错误。为了评估这一点，我们创建了一个呼吸检测器，并将其与在线新闻文章音频的自定义数据集相结合，以区分真实 deepfake语音。此外，我们将此自定义数据集公开提供，以便于将来的工作进行比较。将我们的简单呼吸检测器作为deepfake语音识别器应用于野外样本，可以在33.6小时的音频中实现准确的分类（测试数据上的完美1.0 AUPRC和0.0 EER）。我们将我们的模型与最先进的SSL-wav 2 vec模型进行了比较，结果表明，这种复杂的深度学习模型完全无法对相同的野生样本进行分类（0.72 AUPRC和0.99 EER）。摘要：Deepfake speech represents a real and growing threat to systems and society. Many detectors have been created to aid in defense against speech deepfakes. While these detectors implement myriad methodologies, many rely on low-level fragments of the speech generation process. We hypothesize that breath, a higher-level part of speech, is a key component of natural speech and thus improper generation in deepfake speech is a performant discriminator. To evaluate this, we create a breath detector and leverage this against a custom dataset of online news article audio to discriminate between real deepfake speech. Additionally, we make this custom dataset publicly available to facilitate comparison for future work. Applying our simple breath detector as a deepfake speech discriminator on in-the-wild samples allows for accurate classification (perfect 1.0 AUPRC and 0.0 EER on test data) across 33.6 hours of audio. We compare our model with the state-of-the-art SSL-wav2vec model and show that this complex deep learning model completely fails to classify the same in-the-wild samples (0.72 AUPRC and 0.99 EER).
【5】StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations标题：StoryTTC：具有丰富文本表达性注释的高度表达文本到语音数据集作者：Sen Liu,Yiwei Guo,Xie Chen,Kai YuJournal-ref：IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 11521-11525摘要：表达性文语转换（ETTS）中的声学表达性研究由来已久，但对文本的内在表达性缺乏足够的关注，尤其是对艺术作品的ETTS。在本文中，我们介绍了StoryTTS，一个高度ETTS数据集，包含丰富的表现力，无论是在声学和文本的角度，从记录的普通话讲故事的节目。本文提出了一个系统而全面的文本表达性标注框架。我们分析和定义语音相关的文本表达在StoryTTS包括五个不同的维度，通过语言学，修辞学等，然后我们采用大型语言模型，并提示他们与一些手动注释的例子批量注释。由此产生的语料库包含61个小时的连续和高度韵律的语音配备了准确的文本transmandency和丰富的文本表现力的注释。因此，StoryTTS可以帮助未来的ETTS研究，充分挖掘丰富的内在文本和声学特征。通过实验验证了TTS模型在与StoryTTS中的标注文本标签相结合时，可以生成具有更好表现力的语音。摘要：While acoustic expressiveness has long been studied in expressive text-to-speech (ETTS), the inherent expressiveness in text lacks sufficient attention, especially for ETTS of artistic works. In this paper, we introduce StoryTTS, a highly ETTS dataset that contains rich expressiveness both in acoustic and textual perspective, from the recording of a Mandarin storytelling show. A systematic and comprehensive labeling framework is proposed for textual expressiveness. We analyze and define speech-related textual expressiveness in StoryTTS to include five distinct dimensions through linguistics, rhetoric, etc. Then we employ large language models and prompt them with a few manual annotation examples for batch annotation. The resulting corpus contains 61 hours of consecutive and highly prosodic speech equipped with accurate text transcriptions and rich textual expressiveness annotations. Therefore, StoryTTS can aid future ETTS research to fully mine the abundant intrinsic textual and acoustic features. Experiments are conducted to validate that TTS models can generate speech with improved expressiveness when integrating with the annotated textual labels in StoryTTS.【6】Music Style Transfer With Diffusion Model作者：Hong Huang,Yuyi Wang,Luyao Li,Jun LinJournal-ref：International Computer Music Conference (ICMC 2023) pp. 40-47, October 2023摘要：以往对音乐风格迁移的研究主要集中在一对一的风格转换上，相对有限。当考虑多种风格之间的转换时，以前的方法需要设计多个模式来理清音乐的复杂风格，导致计算成本高且音频生成缓慢。现有的音乐风格转移方法生成具有伪像的频谱图，导致所生成的音频中的显著噪声。为了解决这些问题，本研究提出了一个基于扩散模型（DM）的音乐风格迁移框架，并使用基于谱图的方法来实现多对多的音乐风格迁移。GuideDiff方法用于将声谱图恢复为高保真音频，加快音频生成速度并降低生成的音频中的噪声。实验结果表明，该模型在多模式音乐风格传输方面具有良好的性能，能够在消费级GPU上实时生成高质量的音频。摘要：Previous studies on music style transfer have mainly focused on one-to-one style conversion, which is relatively limited. When considering the conversion between multiple styles, previous methods required designing multiple modes to disentangle the complex style of the music, resulting in large computational costs and slow audio generation. The existing music style transfer methods generate spectrograms with artifacts, leading to significant noise in the generated audio. To address these issues, this study proposes a music style transfer framework based on diffusion models (DM) and uses spectrogram-based methods to achieve multi-to-multi music style transfer. The GuideDiff method is used to restore spectrograms to high-fidelity audio, accelerating audio generation speed and reducing noise in the generated audio. Experimental results show that our model has good performance in multi-mode music style transfer compared to the baseline and can generate high-quality audio in real-time on consumer-grade GPUs.
【7】Qualitative Approaches to Voice UX作者：Katie Seaborn,Jacqueline Urakami,Peter Pennefather,Norihisa P. MiyakeJournal-ref：ACM Computing Surveys (2024)摘要：语音是现代计算机系统提供的一种自然表达方式。基于语音的用户体验（语音UX）的定性视角提供了对复杂交互的丰富描述，这些交互仅凭数字无法完全表示。我们对有关语音用户体验定性方法的文献进行了系统的回顾，在系统的地图中捕捉了这一工作的性质，并提供了定性的综合结果。我们强调了定性方法对语音用户体验研究的好处，确定了提高方法和结果严谨性的机会，并在各种设备和定性实践模式中提取了体验模式。摘要：Voice is a natural mode of expression offered by modern computer-based systems. Qualitative perspectives on voice-based user experiences (voice UX) offer rich descriptions of complex interactions that numbers alone cannot fully represent. We conducted a systematic review of the literature on qualitative approaches to voice UX, capturing the nature of this body of work in a systematic map and offering a qualitative synthesis of findings. We highlight the benefits of qualitative methods for voice UX research, identify opportunities for increasing rigour in methods and outcomes, and distill patterns of experience across a diversity of devices and modes of qualitative praxis.
【8】Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities标题：Bayesian示例选择改善语音、文本和视觉模式的上下文学习作者：Siyin Wang,Chao-Han Huck Yang,Ji Wu,Chao Zhang摘要：大型语言模型（LLM）可以通过基于对话历史中呈现的一些示例的上下文学习（ICL）来适应新任务，而无需任何模型参数更新。尽管如此方便，ICL的性能在很大程度上取决于上下文的例子，这使得上下文的例子选择方法的关键选择的质量。提出了一种新的贝叶斯情境样本选择方法（ByCS）。ByCS基于贝叶斯定理，扩展了上下文内样本条件下的推理概率，重点研究了测试输入条件下的逆推理。在假设准确的逆推理概率（似然）将导致准确的推理概率（后验）之后，基于其逆推理结果来选择上下文中的示例。多样化和广泛的跨任务和跨模态实验进行语音，文本和图像的例子。实验结果表明，我们的ByCS方法的有效性和鲁棒性的各种模型，任务和方式。摘要：Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of ICL heavily depends on the quality of the in-context examples presented, which makes the in-context example selection approach a critical choice. This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL. Extending the inference probability conditioned on in-context examples based on Bayes' theorem, ByCS focuses on the inverse inference conditioned on test input. Following the assumption that accurate inverse inference probability (likelihood) will result in accurate inference probability (posterior), in-context examples are selected based on their inverse inference results. Diverse and extensive cross-tasking and cross-modality experiments are performed with speech, text, and image examples. Experimental results show the efficacy and robustness of our ByCS method on various models, tasks and modalities.【9】Voice Passing : a Non-Binary Voice Gender Prediction System for evaluating Transgender voice transition标题：语音传递：用于评估跨性别语音转变的非二进制语音性别预测系统作者：David Doukhan,Simon Devauchelle,Lucile Girard-Monneron,Mía Chávez Ruz,V. Chaddouk,Isabelle Wagner,Albert RilliardJournal-ref：Proc. INTERSPEECH 2023, 5207-5211摘要：本文介绍了一个软件，允许使用连续的声音女性化百分比（VFP）描述的声音。该系统旨在为跨性别扬声器在他们的声音过渡和语音治疗师支持他们在这个过程中。一个语料库的41个法语顺-和跨性别的发言者被记录。感知评估允许57名参与者估计每个声音的VFP。二元性别分类模型在外部性别平衡数据上进行训练，并在重叠窗口上使用，以获得平均性别预测估计值，该估计值经过校准以预测VFP，并获得比$F_0$或基于声道长度的模型更高的准确性。训练数据说话风格和DNN架构被证明会影响VFP估计。模型的准确性受到说话人年龄的影响。这突出了风格，年龄和性别的概念作为二元或非二元的重要性，以建立适当的文化概念的统计表示。摘要：This paper presents a software allowing to describe voices using a continuous Voice Femininity Percentage (VFP). This system is intended for transgender speakers during their voice transition and for voice therapists supporting them in this process. A corpus of 41 French cis- and transgender speakers was recorded. A perceptual evaluation allowed 57 participants to estimate the VFP for each voice. Binary gender classification models were trained on external gender-balanced data and used on overlapping windows to obtain average gender prediction estimates, which were calibrated to predict VFP and obtained higher accuracy than $F_0$ or vocal track length-based models. Training data speaking style and DNN architecture were shown to impact VFP estimation. Accuracy of the models was affected by speakers' age. This highlights the importance of style, age, and the conception of gender as binary or not, to build adequate statistical representations of cultural concepts.【10】Artificial Neural Networks to Recognize Speakers Division from Continuous Bengali Speech作者：Hasmot Ali,Md. Fahad Hossain,Md. Mehedi Hasan,Sheikh Abujar,Sheak Rashed Haider Noori摘要：基于语音的应用程序正在统治自动化时代，因为语音有很多因素决定了说话者的信息以及语音。现代自动语音识别（ASR）是人机交互（HCI）领域的一个福音，用于使用人工智能技术在人与设备之间进行有效的通信。语音是最简单的沟通媒介之一，因为它有很多相同的功能，不同的发言者。如今，在说话人识别方面，可以使用他们的语音来确定说话人及其身份。在本文中，我们提出了一种方法，将提供一个扬声器的地理身份在一定区域使用连续孟加拉语语音。我们认为孟加拉国的八个不同的分区作为地理区域。我们将Mel频率倒谱系数（MFCC）和Delta特征应用于人工神经网络来分类说话人。我们在特征提取之前执行了一些预处理任务，如降噪和8-10秒的原始音频分割。我们使用了来自633名男性和女性说话者超过45小时的音频数据。我们记录了85.44%的最高准确率。摘要：Voice based applications are ruling over the era of automation because speech has a lot of factors that determine a speakers information as well as speech. Modern Automatic Speech Recognition (ASR) is a blessing in the field of Human-Computer Interaction (HCI) for efficient communication among humans and devices using Artificial Intelligence technology. Speech is one of the easiest mediums of communication because it has a lot of identical features for different speakers. Nowadays it is possible to determine speakers and their identity using their speech in terms of speaker recognition. In this paper, we presented a method that will provide a speakers geographical identity in a certain region using continuous Bengali speech. We consider eight different divisions of Bangladesh as the geographical region. We applied the Mel Frequency Cepstral Coefficient (MFCC) and Delta features on an Artificial Neural Network to classify speakers division. We performed some preprocessing tasks like noise reduction and 8-10 second segmentation of raw audio before feature extraction. We used our dataset of more than 45 hours of audio data from 633 individual male and female speakers. We recorded the highest accuracy of 85.44%.【11】Additive Margin in Contrastive Self-Supervised Frameworks to Learn Discriminative Speaker Representations标题：对比自我监督框架中的添加裕度来学习区分性说话者表示作者：Theo Lepage,Reda Dehak备注：accepted at Odyssey 2024: The Speaker and Language Recognition Workshop. arXiv admin note: text overlap with arXiv:2306.03664摘要：自监督学习（SSL）框架通过受益于大型未标记数据集而成为学习鲁棒类表示的标准。对于说话人验证（SV），大多数SSL系统依赖于基于对比度的损失函数。我们探索不同的方法来提高这些技术的性能，通过重新审视NT-Xent对比损失。我们的主要贡献是定义的NT-Xent-AM损失和研究的重要性，添加剂的利润（AM）在Simplified和MoCo SSL方法，以进一步分离积极的负对。尽管类冲突，我们表明，AM增强了同一扬声器嵌入的紧凑性，并减少了SV上的假阴性和假阳性的数量。此外，我们还证明了对称对比损失的有效性，它为SSL任务提供了更多的监督。对Simplified实现这两个修改可以提高性能，并在VoxCeleb 1-O上获得7.85%的EER，优于其他等效方法。摘要：Self-Supervised Learning (SSL) frameworks became the standard for learning robust class representations by benefiting from large unlabeled datasets. For Speaker Verification (SV), most SSL systems rely on contrastive-based loss functions. We explore different ways to improve the performance of these techniques by revisiting the NT-Xent contrastive loss. Our main contribution is the definition of the NT-Xent-AM loss and the study of the importance of Additive Margin (AM) in SimCLR and MoCo SSL methods to further separate positive from negative pairs. Despite class collisions, we show that AM enhances the compactness of same-speaker embeddings and reduces the number of false negatives and false positives on SV. Additionally, we demonstrate the effectiveness of the symmetric contrastive loss, which provides more supervision for the SSL task. Implementing these two modifications to SimCLR improves performance and results in 7.85% EER on VoxCeleb1-O, outperforming other equivalent methods.
【12】Multi-Sample Dynamic Time Warping for Few-Shot Keyword Spotting标题：用于Few-Shot关键词发现的多样本动态时间扭曲作者：Kevin Wilkinghoff,Alessia Cornaggia-Urrigshardt摘要：在多样本关键字识别中，每个关键字类由多个口语实例（称为样本）表示。一种在目标序列中检测关键词的简单方法是使用子序列动态时间规整查询所有类的所有样本。然而，所得到的处理时间相对于属于每个类的样本的数量线性增加。或者，每个类别只能查询单个Fr 'echet平均值，从而减少了处理时间，但通常也会导致检测性能变差，因为查询样本的变异性没有充分捕获。在这项工作中，多样本动态时间规整提出了计算类特定的成本张量，包括所有查询样本的变化。为了显著降低推理过程中的计算复杂度，在应用动态时间规整之前将这些成本张量转换为成本矩阵。在Few-Shot关键字定位的实验评估中，结果表明，该方法产生与使用所有单独的查询样本作为模板非常相似的性能，同时具有仅比使用Fr 'echet方法时稍慢的运行时间。摘要：In multi-sample keyword spotting, each keyword class is represented by multiple spoken instances, called samples. A na "ive approach to detect keywords in a target sequence consists of querying all samples of all classes using sub-sequence dynamic time warping. However, the resulting processing time increases linearly with respect to the number of samples belonging to each class. Alternatively, only a single Fr 'echet mean can be queried for each class, resulting in reduced processing time but usually also in worse detection performance as the variability of the query samples is not captured sufficiently well. In this work, multi-sample dynamic time warping is proposed to compute class-specific cost-tensors that include the variability of all query samples. To significantly reduce the computational complexity during inference, these cost tensors are converted to cost matrices before applying dynamic time warping. In experimental evaluations for few-shot keyword spotting, it is shown that this method yields a very similar performance as using all individual query samples as templates while having a runtime that is only slightly slower than when using Fr 'echet means.【13】Rethinking Processing Distortions: Disentangling the Impact of Speech Enhancement Errors on Speech Recognition Performance标题：重新思考处理失真：理清语音增强错误对语音识别性能的影响作者：Tsubasa Ochiai,Kazuma Iwamoto,Marc Delcroix,Rintaro Ikeshita,Hiroshi Sato,Shoko Araki,Shigeru Katagiri备注：13 pages, 6 figures, Submitted to IEEEACM Trans. Audio, Speech, and Language Processing摘要：单通道语音增强（SE）前端在噪声条件下提高自动语音识别（ASR）性能具有挑战性。这通常归因于由单通道SE前端的非线性处理引起的处理失真。然而，这种降低的ASR性能的原因还没有得到充分的调查。如何设计单通道SE前端的方式，显着提高ASR性能仍然是一个开放的研究问题。在这项研究中，我们研究了一个信号级的数值度量，可以解释ASR性能下降的原因。为此，我们提出了一种新的分析方案的基础上正交投影为基础的分解SE错误。该方案手动修改分解的干扰，噪声和伪影错误的比例，它使我们能够直接评估每种错误类型对ASR性能的影响。我们的分析揭示了与其他类型的错误相比，伪影错误对ASR性能的特别不利的影响。这为我们提供了导致ASR性能下降的处理失真的更原则性的定义。然后，我们研究了两种实用的方法来减少伪影误差的影响。首先，我们证明了简单观察添加（OA）后处理（即，内插增强的和观察到的信号）可以单调地提高信号与伪像的比率。其次，我们提出了一种新的训练目标，称为伪影增强的信号失真比（AB-SDR），它迫使模型以更少的伪影误差来估计增强的信号。通过实验，我们证实了OA和AB-SDR方法在减少由单通道SE前端引起的伪影误差方面是有效的，从而使它们能够显着提高ASR性能。摘要：It is challenging to improve automatic speech recognition (ASR) performance in noisy conditions with a single-channel speech enhancement (SE) front-end. This is generally attributed to the processing distortions caused by the nonlinear processing of single-channel SE front-ends. However, the causes of such degraded ASR performance have not been fully investigated. How to design single-channel SE front-ends in a way that significantly improves ASR performance remains an open research question. In this study, we investigate a signal-level numerical metric that can explain the cause of degradation in ASR performance. To this end, we propose a novel analysis scheme based on the orthogonal projection-based decomposition of SE errors. This scheme manually modifies the ratio of the decomposed interference, noise, and artifact errors, and it enables us to directly evaluate the impact of each error type on ASR performance. Our analysis reveals the particularly detrimental effect of artifact errors on ASR performance compared to the other types of errors. This provides us with a more principled definition of processing distortions that cause the ASR performance degradation. Then, we study two practical approaches for reducing the impact of artifact errors. First, we prove that the simple observation adding (OA) post-processing (i.e., interpolating the enhanced and observed signals) can monotonically improve the signal-to-artifact ratio. Second, we propose a novel training objective, called artifact-boosted signal-to-distortion ratio (AB-SDR), which forces the model to estimate the enhanced signals with fewer artifact errors. Through experiments, we confirm that both the OA and AB-SDR approaches are effective in decreasing artifact errors caused by single-channel SE front-ends, allowing them to significantly improve ASR performance.
【14】FlashSpeech: Efficient Zero-Shot Speech Synthesis标题：Flash Speech：高效的Zero-Shot语音合成作者：Zhen Ye,Zeqian Ju,Haohe Liu,Xu Tan,Jianyi Chen,Yiwen Lu,Peiwen Sun,Jiahao Pan,Weizhen Bian,Shulin He,Qifeng Liu,Yike Guo,Wei Xue备注：Efficient zero-shot speech synthesis摘要：语言模型和扩散模型极大地促进了大规模zero-shot语音合成的最新进展。然而，这两种方法的生成过程是缓慢的和计算密集型的。使用较低的计算预算来实现与以前的工作相当的质量的有效语音合成仍然是一个重大的挑战。在本文中，我们提出了FlashSpeech，一个大规模的zero-shot语音合成系统，与以前的工作相比，约5 %的推理时间。FlashSpeech建立在潜在一致性模型的基础上，并应用了一种新的对抗性一致性训练方法，该方法可以从头开始训练，而不需要预先训练的扩散模型作为教师。此外，一个新的韵律发生器模块，增强了韵律的多样性，使语音的节奏更自然。FlashSpeech的生成过程可以用一个或两个采样步骤有效地实现，同时保持高的音频质量和与用于zero-shot语音生成的音频提示的高相似性。我们的实验结果证明了FlashSpeech的优越性能。值得注意的是，FlashSpeech可以比其他zero-shot语音合成系统快20倍，同时在语音质量和相似性方面保持相当的性能。此外，FlashSpeech通过有效地执行语音转换、语音编辑和多样化语音采样等任务来展示其多功能性。音频样本可以在https: flashspeech.github.io 上找到。摘要：Recent progress in large-scale zero-shot speech synthesis has been significantly advanced by language models and diffusion models. However, the generation process of both methods is slow and computationally intensive. Efficient speech synthesis using a lower computing budget to achieve quality on par with previous work remains a significant challenge. In this paper, we present FlashSpeech, a large-scale zero-shot speech synthesis system with approximately 5 % of the inference time compared with previous work. FlashSpeech is built on the latent consistency model and applies a novel adversarial consistency training approach that can train from scratch without the need for a pre-trained diffusion model as the teacher. Furthermore, a new prosody generator module enhances the diversity of prosody, making the rhythm of the speech sound more natural. The generation processes of FlashSpeech can be achieved efficiently with one or two sampling steps while maintaining high audio quality and high similarity to the audio prompt for zero-shot speech generation. Our experimental results demonstrate the superior performance of FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other zero-shot speech synthesis systems while maintaining comparable performance in terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates its versatility by efficiently performing tasks like voice conversion, speech editing, and diverse speech sampling. Audio samples can be found in https: flashspeech.github.io .【15】Exploring the Potential of Data-Driven Spatial Audio Enhancement Using a Single-Channel Model标题：使用单通道模型探索数据驱动空间音频增强的潜力作者：Arthur N. dos Santos,Bruno S. Masiero,Túlio C. L. Mateus摘要：区分数据驱动的单通道和多通道语音增强和去混响方法的一个关键方面是，在后一种情况下，问题的制定和解决方案的复杂性都更具挑战性。此外，由于计算资源有限，训练需要管理较大数据集或具有更复杂设计的模型是繁琐的。在这种情况下，一个未经验证的假设，即单通道方法可以通过简单地独立处理每个通道来适应多通道场景，这具有重要的意义，提高了声音场景捕获和系统输入输出格式之间的兼容性，同时也使现代研究能够专注于其他具有挑战性的方面，例如全带宽音频增强，竞争性噪声抑制和无监督学习。本研究通过比较基本的单通道语音增强和去混响模型与其他两个多通道模型的增强效果来验证这一假设，这些模型用于将干净的语音与嘈杂的3D混合信号分开。利用波达方向估计模型，通过将输出信号与地面真实坐标值进行比较，客观地评估其保留空间信息的能力。因此，在以获得较低的可懂度分数增益为代价的更直接的单通道解决方案与保留空间信息之间出现权衡。摘要：One key aspect differentiating data-driven single- and multi-channel speech enhancement and dereverberation methods is that both the problem formulation and complexity of the solutions are considerably more challenging in the latter case. Additionally, with limited computational resources, it is cumbersome to train models that require the management of larger datasets or those with more complex designs. In this scenario, an unverified hypothesis that single-channel methods can be adapted to multi-channel scenarios simply by processing each channel independently holds significant implications, boosting compatibility between sound scene capture and system input-output formats, while also allowing modern research to focus on other challenging aspects, such as full-bandwidth audio enhancement, competitive noise suppression, and unsupervised learning. This study verifies this hypothesis by comparing the enhancement promoted by a basic single-channel speech enhancement and dereverberation model with two other multi-channel models tailored to separate clean speech from noisy 3D mixes. A direction of arrival estimation model was used to objectively evaluate its capacity to preserve spatial information by comparing the output signals with ground-truth coordinate values. Consequently, a trade-off arises between preserving spatial information with a more straightforward single-channel solution at the cost of obtaining lower gains in intelligibility scores.【1】Voice Passing : a Non-Binary Voice Gender Prediction System for evaluating Transgender voice transition标题：语音传递：用于评估跨性别语音转变的非二进制语音性别预测系统作者：David Doukhan,Simon Devauchelle,Lucile Girard-Monneron,Mía Chávez Ruz,V. Chaddouk,Isabelle Wagner,Albert RilliardJournal-ref：Proc. INTERSPEECH 2023, 5207-5211摘要：本文介绍了一个软件，允许使用连续的声音女性化百分比（VFP）描述的声音。该系统旨在为跨性别扬声器在他们的声音过渡和语音治疗师支持他们在这个过程中。一个语料库的41个法语顺-和跨性别的发言者被记录。感知评估允许57名参与者估计每个声音的VFP。二元性别分类模型在外部性别平衡数据上进行训练，并在重叠窗口上使用，以获得平均性别预测估计值，该估计值经过校准以预测VFP，并获得比$F_0$或基于声道长度的模型更高的准确性。训练数据说话风格和DNN架构被证明会影响VFP估计。模型的准确性受到说话人年龄的影响。这突出了风格，年龄和性别的概念作为二元或非二元的重要性，以建立适当的文化概念的统计表示。摘要：This paper presents a software allowing to describe voices using a continuous Voice Femininity Percentage (VFP). This system is intended for transgender speakers during their voice transition and for voice therapists supporting them in this process. A corpus of 41 French cis- and transgender speakers was recorded. A perceptual evaluation allowed 57 participants to estimate the VFP for each voice. Binary gender classification models were trained on external gender-balanced data and used on overlapping windows to obtain average gender prediction estimates, which were calibrated to predict VFP and obtained higher accuracy than $F_0$ or vocal track length-based models. Training data speaking style and DNN architecture were shown to impact VFP estimation. Accuracy of the models was affected by speakers' age. This highlights the importance of style, age, and the conception of gender as binary or not, to build adequate statistical representations of cultural concepts.【2】Artificial Neural Networks to Recognize Speakers Division from Continuous Bengali Speech作者：Hasmot Ali,Md. Fahad Hossain,Md. Mehedi Hasan,Sheikh Abujar,Sheak Rashed Haider Noori摘要：基于语音的应用程序正在统治自动化时代，因为语音有很多因素决定了说话者的信息以及语音。现代自动语音识别（ASR）是人机交互（HCI）领域的一个福音，用于使用人工智能技术在人与设备之间进行有效的通信。语音是最简单的沟通媒介之一，因为它有很多相同的功能，不同的发言者。如今，在说话人识别方面，可以使用他们的语音来确定说话人及其身份。在本文中，我们提出了一种方法，将提供一个扬声器的地理身份在一定区域使用连续孟加拉语语音。我们认为孟加拉国的八个不同的分区作为地理区域。我们将Mel频率倒谱系数（MFCC）和Delta特征应用于人工神经网络来分类说话人。我们在特征提取之前执行了一些预处理任务，如降噪和8-10秒的原始音频分割。我们使用了来自633名男性和女性说话者超过45小时的音频数据。我们记录了85.44%的最高准确率。摘要：Voice based applications are ruling over the era of automation because speech has a lot of factors that determine a speakers information as well as speech. Modern Automatic Speech Recognition (ASR) is a blessing in the field of Human-Computer Interaction (HCI) for efficient communication among humans and devices using Artificial Intelligence technology. Speech is one of the easiest mediums of communication because it has a lot of identical features for different speakers. Nowadays it is possible to determine speakers and their identity using their speech in terms of speaker recognition. In this paper, we presented a method that will provide a speakers geographical identity in a certain region using continuous Bengali speech. We consider eight different divisions of Bangladesh as the geographical region. We applied the Mel Frequency Cepstral Coefficient (MFCC) and Delta features on an Artificial Neural Network to classify speakers division. We performed some preprocessing tasks like noise reduction and 8-10 second segmentation of raw audio before feature extraction. We used our dataset of more than 45 hours of audio data from 633 individual male and female speakers. We recorded the highest accuracy of 85.44%.
【3】Additive Margin in Contrastive Self-Supervised Frameworks to Learn Discriminative Speaker Representations标题：对比自我监督框架中的添加裕度来学习区分性说话者表示作者：Theo Lepage,Reda Dehak备注：accepted at Odyssey 2024: The Speaker and Language Recognition Workshop. arXiv admin note: text overlap with arXiv:2306.03664摘要：自监督学习（SSL）框架通过受益于大型未标记数据集而成为学习鲁棒类表示的标准。对于说话人验证（SV），大多数SSL系统依赖于基于对比度的损失函数。我们探索不同的方法来提高这些技术的性能，通过重新审视NT-Xent对比损失。我们的主要贡献是定义的NT-Xent-AM损失和研究的重要性，添加剂的利润（AM）在Simplified和MoCo SSL方法，以进一步分离积极的负对。尽管类冲突，我们表明，AM增强了同一扬声器嵌入的紧凑性，并减少了SV上的假阴性和假阳性的数量。此外，我们还证明了对称对比损失的有效性，它为SSL任务提供了更多的监督。对Simplified实现这两个修改可以提高性能，并在VoxCeleb 1-O上获得7.85%的EER，优于其他等效方法。摘要：Self-Supervised Learning (SSL) frameworks became the standard for learning robust class representations by benefiting from large unlabeled datasets. For Speaker Verification (SV), most SSL systems rely on contrastive-based loss functions. We explore different ways to improve the performance of these techniques by revisiting the NT-Xent contrastive loss. Our main contribution is the definition of the NT-Xent-AM loss and the study of the importance of Additive Margin (AM) in SimCLR and MoCo SSL methods to further separate positive from negative pairs. Despite class collisions, we show that AM enhances the compactness of same-speaker embeddings and reduces the number of false negatives and false positives on SV. Additionally, we demonstrate the effectiveness of the symmetric contrastive loss, which provides more supervision for the SSL task. Implementing these two modifications to SimCLR improves performance and results in 7.85% EER on VoxCeleb1-O, outperforming other equivalent methods.
【4】Multi-Sample Dynamic Time Warping for Few-Shot Keyword Spotting标题：用于Few-Shot关键词发现的多样本动态时间扭曲作者：Kevin Wilkinghoff,Alessia Cornaggia-Urrigshardt摘要：在多样本关键字识别中，每个关键字类由多个口语实例（称为样本）表示。一种在目标序列中检测关键词的简单方法是使用子序列动态时间规整查询所有类的所有样本。然而，所得到的处理时间相对于属于每个类的样本的数量线性增加。或者，每个类别只能查询单个Fr 'echet平均值，从而减少了处理时间，但通常也会导致检测性能变差，因为查询样本的变异性没有充分捕获。在这项工作中，多样本动态时间规整提出了计算类特定的成本张量，包括所有查询样本的变化。为了显著降低推理过程中的计算复杂度，在应用动态时间规整之前将这些成本张量转换为成本矩阵。在Few-Shot关键字定位的实验评估中，结果表明，该方法产生与使用所有单独的查询样本作为模板非常相似的性能，同时具有仅比使用Fr 'echet方法时稍慢的运行时间。摘要：In multi-sample keyword spotting, each keyword class is represented by multiple spoken instances, called samples. A na "ive approach to detect keywords in a target sequence consists of querying all samples of all classes using sub-sequence dynamic time warping. However, the resulting processing time increases linearly with respect to the number of samples belonging to each class. Alternatively, only a single Fr 'echet mean can be queried for each class, resulting in reduced processing time but usually also in worse detection performance as the variability of the query samples is not captured sufficiently well. In this work, multi-sample dynamic time warping is proposed to compute class-specific cost-tensors that include the variability of all query samples. To significantly reduce the computational complexity during inference, these cost tensors are converted to cost matrices before applying dynamic time warping. In experimental evaluations for few-shot keyword spotting, it is shown that this method yields a very similar performance as using all individual query samples as templates while having a runtime that is only slightly slower than when using Fr 'echet means.【5】Rethinking Processing Distortions: Disentangling the Impact of Speech Enhancement Errors on Speech Recognition Performance标题：重新思考处理失真：理清语音增强错误对语音识别性能的影响作者：Tsubasa Ochiai,Kazuma Iwamoto,Marc Delcroix,Rintaro Ikeshita,Hiroshi Sato,Shoko Araki,Shigeru Katagiri备注：13 pages, 6 figures, Submitted to IEEEACM Trans. Audio, Speech, and Language Processing摘要：单通道语音增强（SE）前端在噪声条件下提高自动语音识别（ASR）性能具有挑战性。这通常归因于由单通道SE前端的非线性处理引起的处理失真。然而，这种降低的ASR性能的原因还没有得到充分的调查。如何设计单通道SE前端的方式，显着提高ASR性能仍然是一个开放的研究问题。在这项研究中，我们研究了一个信号级的数值度量，可以解释ASR性能下降的原因。为此，我们提出了一种新的分析方案的基础上正交投影为基础的分解SE错误。该方案手动修改分解的干扰，噪声和伪影错误的比例，它使我们能够直接评估每种错误类型对ASR性能的影响。我们的分析揭示了与其他类型的错误相比，伪影错误对ASR性能的特别不利的影响。这为我们提供了导致ASR性能下降的处理失真的更原则性的定义。然后，我们研究了两种实用的方法来减少伪影误差的影响。首先，我们证明了简单观察添加（OA）后处理（即，内插增强的和观察到的信号）可以单调地提高信号与伪像的比率。其次，我们提出了一种新的训练目标，称为伪影增强的信号失真比（AB-SDR），它迫使模型以更少的伪影误差来估计增强的信号。通过实验，我们证实了OA和AB-SDR方法在减少由单通道SE前端引起的伪影误差方面是有效的，从而使它们能够显着提高ASR性能。摘要：It is challenging to improve automatic speech recognition (ASR) performance in noisy conditions with a single-channel speech enhancement (SE) front-end. This is generally attributed to the processing distortions caused by the nonlinear processing of single-channel SE front-ends. However, the causes of such degraded ASR performance have not been fully investigated. How to design single-channel SE front-ends in a way that significantly improves ASR performance remains an open research question. In this study, we investigate a signal-level numerical metric that can explain the cause of degradation in ASR performance. To this end, we propose a novel analysis scheme based on the orthogonal projection-based decomposition of SE errors. This scheme manually modifies the ratio of the decomposed interference, noise, and artifact errors, and it enables us to directly evaluate the impact of each error type on ASR performance. Our analysis reveals the particularly detrimental effect of artifact errors on ASR performance compared to the other types of errors. This provides us with a more principled definition of processing distortions that cause the ASR performance degradation. Then, we study two practical approaches for reducing the impact of artifact errors. First, we prove that the simple observation adding (OA) post-processing (i.e., interpolating the enhanced and observed signals) can monotonically improve the signal-to-artifact ratio. Second, we propose a novel training objective, called artifact-boosted signal-to-distortion ratio (AB-SDR), which forces the model to estimate the enhanced signals with fewer artifact errors. Through experiments, we confirm that both the OA and AB-SDR approaches are effective in decreasing artifact errors caused by single-channel SE front-ends, allowing them to significantly improve ASR performance.
【6】FlashSpeech: Efficient Zero-Shot Speech Synthesis标题：Flash Speech：高效的Zero-Shot语音合成作者：Zhen Ye,Zeqian Ju,Haohe Liu,Xu Tan,Jianyi Chen,Yiwen Lu,Peiwen Sun,Jiahao Pan,Weizhen Bian,Shulin He,Qifeng Liu,Yike Guo,Wei Xue备注：Efficient zero-shot speech synthesis摘要：语言模型和扩散模型极大地促进了大规模zero-shot语音合成的最新进展。然而，这两种方法的生成过程是缓慢的和计算密集型的。使用较低的计算预算来实现与以前的工作相当的质量的有效语音合成仍然是一个重大的挑战。在本文中，我们提出了FlashSpeech，一个大规模的zero-shot语音合成系统，与以前的工作相比，约5 %的推理时间。FlashSpeech建立在潜在一致性模型的基础上，并应用了一种新的对抗性一致性训练方法，该方法可以从头开始训练，而不需要预先训练的扩散模型作为教师。此外，一个新的韵律发生器模块，增强了韵律的多样性，使语音的节奏更自然。FlashSpeech的生成过程可以用一个或两个采样步骤有效地实现，同时保持高的音频质量和与用于zero-shot语音生成的音频提示的高相似性。我们的实验结果证明了FlashSpeech的优越性能。值得注意的是，FlashSpeech可以比其他zero-shot语音合成系统快20倍，同时在语音质量和相似性方面保持相当的性能。此外，FlashSpeech通过有效地执行语音转换、语音编辑和多样化语音采样等任务来展示其多功能性。音频样本可以在https: flashspeech.github.io 上找到。摘要：Recent progress in large-scale zero-shot speech synthesis has been significantly advanced by language models and diffusion models. However, the generation process of both methods is slow and computationally intensive. Efficient speech synthesis using a lower computing budget to achieve quality on par with previous work remains a significant challenge. In this paper, we present FlashSpeech, a large-scale zero-shot speech synthesis system with approximately 5 % of the inference time compared with previous work. FlashSpeech is built on the latent consistency model and applies a novel adversarial consistency training approach that can train from scratch without the need for a pre-trained diffusion model as the teacher. Furthermore, a new prosody generator module enhances the diversity of prosody, making the rhythm of the speech sound more natural. The generation processes of FlashSpeech can be achieved efficiently with one or two sampling steps while maintaining high audio quality and high similarity to the audio prompt for zero-shot speech generation. Our experimental results demonstrate the superior performance of FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other zero-shot speech synthesis systems while maintaining comparable performance in terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates its versatility by efficiently performing tasks like voice conversion, speech editing, and diverse speech sampling. Audio samples can be found in https: flashspeech.github.io .
【7】Exploring the Potential of Data-Driven Spatial Audio Enhancement Using a Single-Channel Model标题：使用单通道模型探索数据驱动空间音频增强的潜力作者：Arthur N. dos Santos,Bruno S. Masiero,Túlio C. L. Mateus摘要：区分数据驱动的单通道和多通道语音增强和去混响方法的一个关键方面是，在后一种情况下，问题的制定和解决方案的复杂性都更具挑战性。此外，由于计算资源有限，训练需要管理较大数据集或具有更复杂设计的模型是繁琐的。在这种情况下，一个未经验证的假设，即单通道方法可以通过简单地独立处理每个通道来适应多通道场景，这具有重要的意义，提高了声音场景捕获和系统输入输出格式之间的兼容性，同时也使现代研究能够专注于其他具有挑战性的方面，例如全带宽音频增强，竞争性噪声抑制和无监督学习。本研究通过比较基本的单通道语音增强和去混响模型与其他两个多通道模型的增强效果来验证这一假设，这些模型用于将干净的语音与嘈杂的3D混合信号分开。利用波达方向估计模型，通过将输出信号与地面真实坐标值进行比较，客观地评估其保留空间信息的能力。因此，在以获得较低的可懂度分数增益为代价的更直接的单通道解决方案与保留空间信息之间出现权衡。摘要：One key aspect differentiating data-driven single- and multi-channel speech enhancement and dereverberation methods is that both the problem formulation and complexity of the solutions are considerably more challenging in the latter case. Additionally, with limited computational resources, it is cumbersome to train models that require the management of larger datasets or those with more complex designs. In this scenario, an unverified hypothesis that single-channel methods can be adapted to multi-channel scenarios simply by processing each channel independently holds significant implications, boosting compatibility between sound scene capture and system input-output formats, while also allowing modern research to focus on other challenging aspects, such as full-bandwidth audio enhancement, competitive noise suppression, and unsupervised learning. This study verifies this hypothesis by comparing the enhancement promoted by a basic single-channel speech enhancement and dereverberation model with two other multi-channel models tailored to separate clean speech from noisy 3D mixes. A direction of arrival estimation model was used to objectively evaluate its capacity to preserve spatial information by comparing the output signals with ground-truth coordinate values. Consequently, a trade-off arises between preserving spatial information with a more straightforward single-channel solution at the cost of obtaining lower gains in intelligibility scores.
【8】Analysis and Visualization of Musical Structure using Networks
作者：Alberto Alcalá-Alvarez,Pablo Padilla-Longoria摘要：在这篇文章中，定义和分析一个家庭的图形或网络的符号音乐信息的框架进行了讨论。这些图表涉及不同类型的元素，例如音高、和弦和节奏，以及它们之间的关系，并且是从数字乐谱中包含的定量或分类数据构建的。它们有助于立即可视化音乐特征，从而为理解音乐片段的一般结构元素提供了计算工具。从数字分数中获得的数据经历了图和网络理论的不同分析过程，例如计算其中心性度量和熵，以及检测其社区。我们分析了来自不同风格的音乐作品，并将我们的一些结果与传统音乐分析技术的结论进行了比较。摘要：In this article, a framework for defining and analysing a family of graphs or networks from symbolic music information is discussed. Such graphs concern different types of elements, such as pitches, chords and rhythms, and the relations among them, and are built from quantitative or categorical data contained in digital music scores. They are helpful in visualizing musical features at once, thus leading to a computational tool for understanding the general structural elements of a music fragment. Data obtained from a digital score undergoes different analytical procedures from graph and network theory, such as computing their centrality measures and entropy, and detecting their communities. We analyze pieces of music coming from different styles, and compare some of our results with conclusions from traditional music analysis techniques.
【9】Tailors: New Music Timbre Visualizer to Entertain Music Through Imagery作者：47 pages, 9 figures, 5 tables摘要：在本文中，我已经实现了一个音色可视化系统称为裁缝。通过与27名MIR用户的实验，发现与唯一的音乐条件和基本可视化相比，Tailors在传达音乐的音色温暖，亮度，深度，浅度，硬度，粗糙度和锐度特征方面是有效的。在音乐意象和音乐娱乐调查中，裁缝的所有分数在三种情况中都是最高的。音色意象与意象娱乐性之间的多元线性回归分析显示两者之间存在显著的正相关。通过与Fisher变换的相关系数比较，发现Tailors通过改善音乐视觉意象，使用户的音乐娱乐性更好。调查结果显示，Tailors在最佳音色表达、音乐体验和再次使用意愿方面排名第一。虽然有些用户觉得眼睛有负担，但Tailors留下了音色可视化映射规则的数据驱动方法的未来工作，以获得许多用户的同意。此外，还讨论了减少音色特征以专注于裁缝可以表达的特征，并以更艺术的方式使用空间感的裁缝未来的工作。摘要：In this paper, I have implemented a timbre visualization system called Tailors. Through the experiment with 27 MIR users, Tailors was found to be effective in conveying timbral warmth, brightness, depth, shallowness, hardness, roughness, and sharpness features of music compared to the only music condition and basic visualization. All scores of Tailors in the music imagery and music entertainment surveys were valued highest among the three conditions. Multiple linear regression analysis between timbre-imagery and imagery-entertainment shows significant and positive correlations. Coefficients comparing results from Fisher Transformation show that Tailors made user's music entertainment better through improved music visual imagery. The post-survey result represents that Tailors ranked first for the best timbre expression, music experience, and willingness to use it again. While some users felt a burden in the eye, Tailors left the future work of the data-driven approach of the mapping rule of timbre visualization to gain consent from many users. Furthermore, reducing timbre features to focus on features that Tailors can express well was also discussed, with future work of Tailors in a more artistic way using the sense of space.【10】Every Breath You Don't Take: Deepfake Speech Detection Using Breath标题：你没有呼吸的每一次呼吸：使用呼吸进行Deepfake语音检测作者：Seth Layton,Thiago De Andrade,Daniel Olszewski,Kevin Warren,Carrie Gates,Kevin Butler,Patrick Traynor备注：Submitted to ACM journal -- Digital Threats: Research and Practice摘要：Deepfake言论对系统和社会构成了真正且不断增长的威胁。已经创建了许多检测器来帮助防御语音deepfake。虽然这些检测器实现了无数的方法，但许多检测器依赖于语音生成过程的低级片段。我们假设，呼吸，一种更高级别的语音部分，是自然语音的关键组成部分，因此在deepfake语音中不正确的生成是一种性能错误。为了评估这一点，我们创建了一个呼吸检测器，并将其与在线新闻文章音频的自定义数据集相结合，以区分真实 deepfake语音。此外，我们将此自定义数据集公开提供，以便于将来的工作进行比较。将我们的简单呼吸检测器作为deepfake语音识别器应用于野外样本，可以在33.6小时的音频中实现准确的分类（测试数据上的完美1.0 AUPRC和0.0 EER）。我们将我们的模型与最先进的SSL-wav 2 vec模型进行了比较，结果表明，这种复杂的深度学习模型完全无法对相同的野生样本进行分类（0.72 AUPRC和0.99 EER）。摘要：Deepfake speech represents a real and growing threat to systems and society. Many detectors have been created to aid in defense against speech deepfakes. While these detectors implement myriad methodologies, many rely on low-level fragments of the speech generation process. We hypothesize that breath, a higher-level part of speech, is a key component of natural speech and thus improper generation in deepfake speech is a performant discriminator. To evaluate this, we create a breath detector and leverage this against a custom dataset of online news article audio to discriminate between real deepfake speech. Additionally, we make this custom dataset publicly available to facilitate comparison for future work. Applying our simple breath detector as a deepfake speech discriminator on in-the-wild samples allows for accurate classification (perfect 1.0 AUPRC and 0.0 EER on test data) across 33.6 hours of audio. We compare our model with the state-of-the-art SSL-wav2vec model and show that this complex deep learning model completely fails to classify the same in-the-wild samples (0.72 AUPRC and 0.99 EER).
【11】StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations标题：StoryTTC：具有丰富文本表达性注释的高度表达文本到语音数据集作者：Sen Liu,Yiwei Guo,Xie Chen,Kai YuJournal-ref：IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 11521-11525摘要：表达性文语转换（ETTS）中的声学表达性研究由来已久，但对文本的内在表达性缺乏足够的关注，尤其是对艺术作品的ETTS。在本文中，我们介绍了StoryTTS，一个高度ETTS数据集，包含丰富的表现力，无论是在声学和文本的角度，从记录的普通话讲故事的节目。本文提出了一个系统而全面的文本表达性标注框架。我们分析和定义语音相关的文本表达在StoryTTS包括五个不同的维度，通过语言学，修辞学等，然后我们采用大型语言模型，并提示他们与一些手动注释的例子批量注释。由此产生的语料库包含61个小时的连续和高度韵律的语音配备了准确的文本transmandency和丰富的文本表现力的注释。因此，StoryTTS可以帮助未来的ETTS研究，充分挖掘丰富的内在文本和声学特征。通过实验验证了TTS模型在与StoryTTS中的标注文本标签相结合时，可以生成具有更好表现力的语音。摘要：While acoustic expressiveness has long been studied in expressive text-to-speech (ETTS), the inherent expressiveness in text lacks sufficient attention, especially for ETTS of artistic works. In this paper, we introduce StoryTTS, a highly ETTS dataset that contains rich expressiveness both in acoustic and textual perspective, from the recording of a Mandarin storytelling show. A systematic and comprehensive labeling framework is proposed for textual expressiveness. We analyze and define speech-related textual expressiveness in StoryTTS to include five distinct dimensions through linguistics, rhetoric, etc. Then we employ large language models and prompt them with a few manual annotation examples for batch annotation. The resulting corpus contains 61 hours of consecutive and highly prosodic speech equipped with accurate text transcriptions and rich textual expressiveness annotations. Therefore, StoryTTS can aid future ETTS research to fully mine the abundant intrinsic textual and acoustic features. Experiments are conducted to validate that TTS models can generate speech with improved expressiveness when integrating with the annotated textual labels in StoryTTS.【12】Qualitative Approaches to Voice UX作者：Katie Seaborn,Jacqueline Urakami,Peter Pennefather,Norihisa P. MiyakeJournal-ref：ACM Computing Surveys (2024)摘要：语音是现代计算机系统提供的一种自然表达方式。基于语音的用户体验（语音UX）的定性视角提供了对复杂交互的丰富描述，这些交互仅凭数字无法完全表示。我们对有关语音用户体验定性方法的文献进行了系统的回顾，在系统的地图中捕捉了这一工作的性质，并提供了定性的综合结果。我们强调了定性方法对语音用户体验研究的好处，确定了提高方法和结果严谨性的机会，并在各种设备和定性实践模式中提取了体验模式。摘要：Voice is a natural mode of expression offered by modern computer-based systems. Qualitative perspectives on voice-based user experiences (voice UX) offer rich descriptions of complex interactions that numbers alone cannot fully represent. We conducted a systematic review of the literature on qualitative approaches to voice UX, capturing the nature of this body of work in a systematic map and offering a qualitative synthesis of findings. We highlight the benefits of qualitative methods for voice UX research, identify opportunities for increasing rigour in methods and outcomes, and distill patterns of experience across a diversity of devices and modes of qualitative praxis.
【13】Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities标题：Bayesian示例选择改善语音、文本和视觉模式的上下文学习作者：Siyin Wang,Chao-Han Huck Yang,Ji Wu,Chao Zhang摘要：大型语言模型（LLM）可以通过基于对话历史中呈现的一些示例的上下文学习（ICL）来适应新任务，而无需任何模型参数更新。尽管如此方便，ICL的性能在很大程度上取决于上下文的例子，这使得上下文的例子选择方法的关键选择的质量。提出了一种新的贝叶斯情境样本选择方法（ByCS）。ByCS基于贝叶斯定理，扩展了上下文内样本条件下的推理概率，重点研究了测试输入条件下的逆推理。在假设准确的逆推理概率（似然）将导致准确的推理概率（后验）之后，基于其逆推理结果来选择上下文中的示例。多样化和广泛的跨任务和跨模态实验进行语音，文本和图像的例子。实验结果表明，我们的ByCS方法的有效性和鲁棒性的各种模型，任务和方式。摘要：Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of ICL heavily depends on the quality of the in-context examples presented, which makes the in-context example selection approach a critical choice. This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL. Extending the inference probability conditioned on in-context examples based on Bayes' theorem, ByCS focuses on the inverse inference conditioned on test input. Following the assumption that accurate inverse inference probability (likelihood) will result in accurate inference probability (posterior), in-context examples are selected based on their inverse inference results. Diverse and extensive cross-tasking and cross-modality experiments are performed with speech, text, and image examples. Experimental results show the efficacy and robustness of our ByCS method on various models, tasks and modalities. 
每日语音速递
