今日论文合集：cs.SD语音13篇，eess.AS音频处理17篇。
cs.SD语音
【1】 UniverSLU: Universal Spoken Language Understanding for Diverse  Classification and Sequence Generation Tasks with a Single Network标题：通过单一网络实现对不同分类和序列生成任务的通用口语理解链接：https://arxiv.org/abs/2310.02973
作者：Siddhant Arora,Hayato Futami,Jee-weon Jung,Yifan Peng,Roshan Sharma,Yosuke Kashiwagi,Emiru Tsunoo,Shinji Watanabe机构：Carnegie Mellon University, Sony Group Corporation摘要：最近的研究表明，通过采用具有多任务处理能力的大型语言模型，可以获得很好的结果。它们利用提示来指导模型的行为，并超越特定任务模型的性能。基于此，我们问：我们是否可以建立一个单一的模型，共同执行各种口语理解（SLU）的任务？为了解决这个问题，我们利用预训练的自动语音识别（ASR）模型，并采用各种任务和数据集说明符作为离散提示。我们展示了我们的单一多任务学习（MTL）模型“UniversSLU”在17个数据集和9种语言中的12个不同的语音分类和序列生成任务的有效性。结果表明，UniverSLU实现了有竞争力的性能，甚至超过了特定任务的模型。我们还进行了初步调查，使人类可解释的自然短语，而不是任务说明符作为离散提示和测试模型的泛化能力，以新的释义。摘要：Recent studies have demonstrated promising outcomes by employing large language models with multi-tasking capabilities. They utilize prompts to guide the model's behavior and surpass performance of task-specific models. Motivated by this, we ask: can we build a single model that jointly perform various spoken language understanding (SLU) tasks? To address this, we utilize pre-trained automatic speech recognition (ASR) models and employ various task and dataset specifiers as discrete prompts. We demonstrate efficacy of our single multi-task learning (MTL) model "UniverSLU" for 12 different speech classification and sequence generation tasks across 17 datasets and 9 languages. Results show that UniverSLU achieves competitive performance and even surpasses task-specific models. We also conduct preliminary investigations into enabling human-interpretable natural phrases instead of task specifiers as discrete prompts and test the model's generalization capabilities to new paraphrases.
【2】 End-to-End Training of a Neural HMM with Label and Transition  Probabilities标题：具有标签和转移概率的神经HMM的端到端训练链接：https://arxiv.org/abs/2310.02724
作者：Daniel Mann,Tina Raissi,Wilfried Michel,Ralf Schlüter,Hermann Ney机构：AppTek GmbH, Aachen, Germany, Machine Learning and Human Language Technology, Computer Science Department, RWTH Aachen University, Aachen, Germany备注：Accepted for Presentation at ASRU2023摘要：我们研究了一种使用隐马尔可夫模型（HMM）进行端到端神经网络训练的新建模方法，其中隐状态之间的转移概率被明确建模和学习。大多数当代序列到序列模型允许通过对给定拓扑中的所有可能的标签分割求和来从头开始训练。在我们的方法中，有明确的，可学习的概率段之间的过渡，而不是一个空白的标签，隐式编码的持续时间统计。我们实现了一个基于GPU的向前向后算法，使标签和转移概率的同时训练。我们调查识别结果，并另外维特比对齐我们的模型。我们发现，虽然过渡模型训练并没有提高识别性能，但它对对齐质量有积极的影响。所产生的对齐被示出为在最先进的维特比训练中是可行的目标。摘要：We investigate a novel modeling approach for end-to-end neural network training using hidden Markov models (HMM) where the transition probabilities between hidden states are modeled and learned explicitly. Most contemporary sequence-to-sequence models allow for from-scratch training by summing over all possible label segmentations in a given topology. In our approach there are explicit, learnable probabilities for transitions between segments as opposed to a blank label that implicitly encodes duration statistics. We implement a GPU-based forward-backward algorithm that enables the simultaneous training of label and transition probabilities. We investigate recognition results and additionally Viterbi alignments of our models. We find that while the transition model training does not improve recognition performance, it has a positive impact on the alignment quality. The generated alignments are shown to be viable targets in state-of-the-art Viterbi trainings.
【3】 Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised  Learning with Masked Unit Prediction标题：多分辨率Hubert：带掩蔽单元预测的多分辨率语音自监督学习链接：https://arxiv.org/abs/2310.02720
作者：Jiatong Shi,Hirofumi Inaguma,Xutai Ma,Ilia Kulikov,Anna Sun机构： Language Technologies Institute, Carnegie Mellon University, Meta AI备注：35 pages, 4 figures, 21 tables摘要：现有的语音自监督学习（SSL）模型通常以20毫秒的固定分辨率处理语音信号。这种方法忽略了语音信号中以不同分辨率存在的变化的信息内容。相比之下，本文旨在将多分辨率信息纳入语音自监督表示学习。我们引入了一个SSL模型，利用层次化的Transformer架构，辅以HUBERT风格的掩蔽预测目标，在多个分辨率处理语音。实验结果表明，该模型不仅实现了更有效的推理，但也表现出优越的或可比的性能，在各种任务的原始HuBERT模型。具体而言，在LibriSpeech语音识别基准的微调实验以及使用语音通用性能基准（SUPERB）和多语言SUPERB（ML-SUPERB）的评估中，已经观察到了原始HuBERT的显着性能改进。摘要：Existing Self-Supervised Learning (SSL) models for speech typically process speech signals at a fixed resolution of 20 milliseconds. This approach overlooks the varying informational content present at different resolutions in speech signals. In contrast, this paper aims to incorporate multi-resolution information into speech self-supervised representation learning. We introduce a SSL model that leverages a hierarchical Transformer architecture, complemented by HuBERT-style masked prediction objectives, to process speech at multiple resolutions. Experimental results indicate that the proposed model not only achieves more efficient inference but also exhibits superior or comparable performance to the original HuBERT model over various tasks. Specifically, significant performance improvements over the original HuBERT have been observed in fine-tuning experiments on the LibriSpeech speech recognition benchmark as well as in evaluations using the Speech Universal PERformance Benchmark (SUPERB) and Multilingual SUPERB (ML-SUPERB).
【4】 BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching  Speech Recognition标题：BA-MOE：用于码型转换语音识别的边界感知混合专家适配器链接：https://arxiv.org/abs/2310.02629
作者：Peikun Chen,Fan Yu,Yuhao Lian,Hongfei Xue,Xucheng Wan,Naijun Zheng,Huan Zhou,Lei Xie机构：Northwestern Polytechnical University, Xian, China, Huawei备注：5 pages,3figures,Accept by ASRU2023摘要：基于专家混合的模型利用语言专家有效地提取特定语言的表征，在语码转换的自动语音识别中得到了很好的应用。然而，仍然有很大的改进空间，因为跨语言的相似发音可能导致无效的多语言建模和不准确的语言边界估计。为了消除这些缺点，我们提出了一个跨层的语言适配器和边界感知的训练方法，即边界感知混合专家（BA-MoE）。具体来说，我们引入特定于语言的适配器来分离特定于语言的表示和统一的门控层来融合每个编码器层内的表示。其次，我们计算每个特定语言适配器的平均输出的语言适应损失，以改善适配器模块的特定语言表示学习。此外，我们利用一个边界感知预测器来学习边界表示，以处理语言边界混淆。我们的方法实现了显着的性能改进，与ASRU 2019年普通话-英语代码转换挑战数据集的基线相比，混合错误率降低了16.55%。摘要：Mixture-of-experts based models, which use language experts to extract language-specific representations effectively, have been well applied in code-switching automatic speech recognition. However, there is still substantial space to improve as similar pronunciation across languages may result in ineffective multi-language modeling and inaccurate language boundary estimation. To eliminate these drawbacks, we propose a cross-layer language adapter and a boundary-aware training method, namely Boundary-Aware Mixture-of-Experts (BA-MoE). Specifically, we introduce language-specific adapters to separate language-specific representations and a unified gating layer to fuse representations within each encoder layer. Second, we compute language adaptation loss of the mean output of each language-specific adapter to improve the adapter module's language-specific representation learning. Besides, we utilize a boundary-aware predictor to learn boundary representations for dealing with language boundary confusion. Our approach achieves significant performance improvement, reducing the mixture error rate by 16.55\% compared to the baseline on the ASRU 2019 Mandarin-English code-switching challenge dataset.
【5】 Improving severity preservation of healthy-to-pathological voice  conversion with global style tokens标题：使用全局样式令牌改进健康到病态语音转换的严重性保存链接：https://arxiv.org/abs/2310.02570
作者：Bence Mark Halpern,Wen-Chin Huang,Lester Phillip Violeta,R. J. J. H. van Son,Tomoki Toda机构：Nagoya University, Japan, Netherlands Cancer Institute, The Netherlands备注：7 pages, 3 figures, 5 tables. Accepted to IEEE Automatic Speech Recognition and Understanding Workshop 2023摘要：在健康到病态语音转换（H2 P-VC）中，健康语音被转换为病态语音，同时保留身份。本文改进了先前的H2 P-VC的两阶段方法，其中（1）首先以适当的严重性创建语音，（2）然后在保持语音的严重性的同时转换语音的说话者身份。具体而言，我们提出了改进（2）通过使用语音后验图（PPG）和全球风格令牌（GST）。此外，我们提出了一个新的数据集，其中包含具有相同身份的病理和健康扬声器的并行记录，从而可以进行更精确的评估。专家听众的听力测试表明，该框架保留了源样本的严重性，同时建模目标扬声器的声音。我们还表明，（a）病理影响x向量，但不是所有的扬声器信息丢失，（b）选择源扬声器的严重性标签的基础上单独是不够的。摘要：In healthy-to-pathological voice conversion (H2P-VC), healthy speech is converted into pathological while preserving the identity. The paper improves on previous two-stage approach to H2P-VC where (1) speech is created first with the appropriate severity, (2) then the speaker identity of the voice is converted while preserving the severity of the voice. Specifically, we propose improvements to (2) by using phonetic posteriorgrams (PPG) and global style tokens (GST). Furthermore, we present a new dataset that contains parallel recordings of pathological and healthy speakers with the same identity which allows more precise evaluation. Listening tests by expert listeners show that the framework preserves severity of the source sample, while modelling target speaker's voice. We also show that (a) pathology impacts x-vectors but not all speaker information is lost, (b) choosing source speakers based on severity labels alone is insufficient.
【6】 Shaping the Epochal Individuality and Generality: The Temporal Dynamics  of Uncertainty and Prediction Error in Musical Improvisation标题：塑造时代的个性和共性：音乐即兴创作中不确定性和预测误差的时间动力学链接：https://arxiv.org/abs/2310.02518
作者：Tatsuya Daikoku机构： 3 1 Graduate School of Information Science and Technology, The University of Tokyo 2 Centre for Neuroscience in Education, University of Cambridge 3 Center for Brain摘要：音乐即兴创作，就像即兴演讲一样，揭示了即兴创作者的精神状态和情感特征的复杂方面。然而，揭示这种个性的特定音乐成分在很大程度上仍未被探索。在大脑的统计学习和预测处理的框架内，本研究探讨了音乐即兴创作中的不确定性和惊喜（预测错误）的时间动态。本研究采用HBSL模型分析了从1905年到2009年的456首爵士乐即兴作品，来自78位不同的爵士音乐家。结果表明，独特的时间模式的惊喜和不确定性，特别是在音高和音高节奏序列，揭示了特定时代的功能，从20世纪初到21世纪。相反，节奏序列表现出跨时代的一致程度的不确定性。此外，声学特性在不同时期保持不变。这些发现强调了即兴音乐中惊喜和不确定性的时间动态如何随着时间的推移而变化的重要性，深刻影响了每个时代艺术家为即兴创作所采用的独特方法。此外，有人建议，即兴音乐的发展可以归因于大脑的自适应统计学习机制，不断完善内部模型，以反映各自时代的文化和情感的细微差别。这项研究揭示了即兴音乐的演变轨迹，并强调了艺术家们为与他们所处时代的文化和情感景观产生共鸣而进行的微妙转变。摘要：Musical improvisation, much like spontaneous speech, reveals intricate facets of the improviser's state of mind and emotional character. However, the specific musical components that reveal such individuality remain largely unexplored. Within the framework of brain's statistical learning and predictive processing, this study examined the temporal dynamics of uncertainty and surprise (prediction error) in a piece of musical improvisation. This study employed the HBSL model to analyze a corpus of 456 Jazz improvisations, spanning 1905 to 2009, from 78 distinct Jazz musicians. The results indicated distinctive temporal patterns of surprise and uncertainty, especially in pitch and pitch-rhythm sequences, revealing era-specific features from the early 20th to the 21st centuries. Conversely, rhythm sequences exhibited a consistent degree of uncertainty across eras. Further, the acoustic properties remain unchanged across different periods. These findings highlight the importance of how temporal dynamics of surprise and uncertainty in improvisational music change over periods, profoundly influencing the distinctive methodologies artists adopt for improvisation in each era. Further, it is suggested that the development of improvisational music can be attributed to the brain's adaptive statistical learning mechanisms, which constantly refine internal models to mirror the cultural and emotional nuances of their respective epochs. This study unravels the evolutionary trajectory of improvisational music and highlights the nuanced shifts artists employ to resonate with the cultural and emotional landscapes of their times.
【7】 Towards an Interpretable Representation of Speaker Identity via  Perceptual Voice Qualities标题：通过感知语音质量实现说话人身份的可解释表征链接：https://arxiv.org/abs/2310.02497
作者：Robin Netzorg,Bohan Yu,Andrea Guzman,Peter Wu,Luna McNulty,Gopala Anumanchipalli机构：University of California, Berkeley,Brown University摘要：与文本和视觉等其他数据形式不同，语音并不容易解释。虽然外行人可以理解如何通过感知来描述图像或句子，但非专家对语音的描述通常以高级人口统计信息（如性别或年龄）结束。在本文中，我们提出了一个可能的可解释的表示的扬声器身份的感知语音质量（PQ）的基础上。通过将性别PQ添加到以病理学为中心的共识听觉感知评价的语音（CAPE-V）协议，我们的基于PQ的方法提供了一个感性的潜在空间的字符的成人的声音，是一个中介的抽象之间的高层次的人口统计和低层次的声学，物理，或学习表示。与之前的信念相反，我们证明了这些PQ是由非专家的合奏可听，并进一步证明了在基于PQ的表示编码的信息是可预测的各种语音表示。摘要：Unlike other data modalities such as text and vision, speech does not lend itself to easy interpretation. While lay people can understand how to describe an image or sentence via perception, non-expert descriptions of speech often end at high-level demographic information, such as gender or age. In this paper, we propose a possible interpretable representation of speaker identity based on perceptual voice qualities (PQs). By adding gendered PQs to the pathology-focused Consensus Auditory-Perceptual Evaluation of Voice (CAPE-V) protocol, our PQ-based approach provides a perceptual latent space of the character of adult voices that is an intermediary of abstraction between high-level demographics and low-level acoustic, physical, or learned representations. Contrary to prior belief, we demonstrate that these PQs are hearable by ensembles of non-experts, and further demonstrate that the information encoded in a PQ-based representation is predictable by various speech representations.
【8】 ResidualTransformer: Residual Low-rank Learning with Weight-sharing for  Transformer Layers标题：残差Transformer层的加权残差低阶学习链接：https://arxiv.org/abs/2310.02489
作者：Yiming Wang,Jinyu Li机构：Microsoft Corporation, Redmond, WA, USA备注：Submitted to IEEE ICASSP 2024. 5 pages, 1 figure摘要：在这些设备上部署语音处理模型时，始终在线设备的存储器约束是主要关注点之一。虽然用足够大量的数据训练的较大模型通常表现更好，但使它们适合设备存储器是一项艰巨的挑战。在本文中，我们的目标是通过重新参数化模型权重跨Transformer编码器层，并假设一个特殊的权重组成和结构，以减少模型的大小。更具体地说，受ResNet和最近LoRA工作的启发，我们提出了一种名为ResidualTransformer的方法，其中Transformer层中的每个权重矩阵包括1）与其相邻层共享的满秩分量，以及2）其自身的唯一低秩分量。低秩矩阵仅占少量的模型大小增加。此外，我们加入对角权重矩阵，以提高建模能力的低秩矩阵。我们的10k小时的语音识别和语音翻译任务的实验表明，Transformer编码器的大小可以减少约3倍，性能下降非常轻微。摘要：Memory constraint of always-on devices is one of the major concerns when deploying speech processing models on these devices. While larger models trained with sufficiently large amount of data generally perform better, making them fit in the device memory is a demanding challenge. In this paper, we aim to reduce model size by reparameterizing model weights across Transformer encoder layers and assuming a special weight composition and structure. More specifically, inspired by ResNet and the more recent LoRA work, we propose an approach named ResidualTransformer, where each weight matrix in a Transformer layer comprises 1) a shared full-rank component with its adjacent layers, and 2) a unique low-rank component to itself. The low-rank matrices only account for a small amount of model size increase. In addition, we add diagonal weight matrices to improve modeling capacity of the low-rank matrices. Experiments of our 10k-hour speech recognition and speech translation tasks show that the Transformer encoder size can be reduced by ~3X with very slight performance degradation.
【9】 Unsupervised Speech Recognition with N-Skipgram and Positional Unigram  Matching标题：基于N-Skipgram和位置单字匹配的无监督语音识别链接：https://arxiv.org/abs/2310.02382
作者：Liming Wang,Mark Hasegawa-Johnson,Chang D. Yoo机构：⋆Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, †Artificial Intelligence and Machine Learning Lab, KAIST摘要：由于GAN相关的不稳定性，语音和文本之间的不对齐以及大量的内存需求，训练无监督语音识别系统面临着挑战。为了应对这些挑战，我们介绍了一种新的ASR系统，ESPUM。该系统利用低阶N-跳图（最多N=3）的能力，并结合从一小批样本中收集的位置一元统计数据。评估TIMIT基准，我们的模型展示了竞争力的表现，在ASR和音素分割任务。访问我们的公开代码https://github.com/lwang114/GraphUnsupASR。摘要：Training unsupervised speech recognition systems presents challenges due to GAN-associated instability, misalignment between speech and text, and significant memory demands. To tackle these challenges, we introduce a novel ASR system, ESPUM. This system harnesses the power of lower-order N-skipgrams (up to N=3) combined with positional unigram statistics gathered from a small batch of samples. Evaluated on the TIMIT benchmark, our model showcases competitive performance in ASR and phoneme segmentation tasks. Access our publicly available code at https://github.com/lwang114/GraphUnsupASR.
【10】 Prompting Audios Using Acoustic Properties For Emotion Representation标题：利用声学特性进行情感表征的提示音频链接：https://arxiv.org/abs/2310.02298
作者：Hira Dhamyal,Benjamin Elizalde,Soham Deshmukh,Huaming Wang,Bhiksha Raj,Rita Singh机构： Carnegie Mellon University,Microsoft,Mohammed bin Zayed University of AI备注：arXiv admin note: substantial text overlap with arXiv:2211.07737摘要：情绪是一个连续体，但目前的模型将情绪视为一个有限值的离散变量。这种表达方式没有捕捉到情感表达的多样性。为了更好地表示情感，我们建议使用自然语言描述（或提示）。在这项工作中，我们解决了自动生成这些提示并训练模型以更好地从音频和提示对中学习情感表示的挑战。我们使用与情绪相关的声学属性，如音高，强度，语速和清晰度，自动生成提示，即“声音提示”。我们使用一个对比的学习目标，以映射语音到各自的声学提示。我们评估我们的模型情感音频检索和语音情感识别。我们的研究结果表明，声学提示显着提高模型的性能在EAR中，在各种精度@K指标。在SER中，我们观察到Ravdess数据集的相对准确度提高了3.8%。摘要：Emotions lie on a continuum, but current models treat emotions as a finite valued discrete variable. This representation does not capture the diversity in the expression of emotion. To better represent emotions we propose the use of natural language descriptions (or prompts). In this work, we address the challenge of automatically generating these prompts and training a model to better learn emotion representations from audio and prompt pairs. We use acoustic properties that are correlated to emotion like pitch, intensity, speech rate, and articulation rate to automatically generate prompts i.e. 'acoustic prompts'. We use a contrastive learning objective to map speech to their respective acoustic prompts. We evaluate our model on Emotion Audio Retrieval and Speech Emotion Recognition. Our results show that the acoustic prompts significantly improve the model's performance in EAR, in various Precision@K metrics. In SER, we observe a 3.8% relative accuracy improvement on the Ravdess dataset.
【11】 Zero Resource Code-switched Speech Benchmark Using Speech Utterance  Pairs For Multiple Spoken Languages标题：使用多语言语音话语对的零资源码切换语音基准链接：https://arxiv.org/abs/2310.03018
作者：Kuan-Po Huang,Chih-Kai Yang,Yu-Kuan Fu,Ewan Dunbar,Hung-yi Lee机构：National Taiwan University, ASUS Intelligent Cloud Services, University of Toronto备注：Submitted to ICASSP 2024摘要：我们引入了一个新的零资源代码切换语音基准，旨在直接评估自监督语音编码器的代码切换能力。我们展示了一个基线系统的离散单元语言建模，以演示如何代码切换能力的语音编码器可以在零资源的方式进行评估。我们的实验涵盖了各种知名的语音编码器，包括Wav2vec 2.0，HuBERT，XLSR等。我们研究了预训练语言和模型大小对基准测试性能的影响。值得注意的是，虽然我们的研究结果表明，以XLSR为例的多语言预训练语音编码器在代码切换场景中的表现优于单语变体（Wav2vec 2.0，HuBERT），但其代码切换语言能力仍有很大的改进空间。摘要：We introduce a new zero resource code-switched speech benchmark designed to directly assess the code-switching capabilities of self-supervised speech encoders. We showcase a baseline system of language modeling on discrete units to demonstrate how the code-switching abilities of speech encoders can be assessed in a zero-resource manner. Our experiments encompass a variety of well-known speech encoders, including Wav2vec 2.0, HuBERT, XLSR, etc. We examine the impact of pre-training languages and model size on benchmark performance. Notably, though our results demonstrate that speech encoders with multilingual pre-training, exemplified by XLSR, outperform monolingual variants (Wav2vec 2.0, HuBERT) in code-switching scenarios, there is still substantial room for improvement in their code-switching linguistic abilities.
【12】 Discriminative Training of VBx Diarization标题：VBx二值化的判别训练链接：https://arxiv.org/abs/2310.02732
作者：Dominik Klement,Mireia Diez,Federico Landini,Lukáš Burget,Anna Silnova,Marc Delcroix,Naohiro Tawara机构： Naohiro Tawara 2 1Brno University of Technology备注：Submitted to ICASSP 2024摘要：x向量序列的贝叶斯HMM聚类（VBx）已成为出版物和挑战中广泛采用的日志化基线模型。它使用HMM来建模扬声器回合，生成训练的概率线性判别分析（PLDA）扬声器分布建模，和贝叶斯推理来估计分配的x向量的扬声器。本文提出了一种新的框架更新VBx参数使用判别式训练，直接优化预定义的损失。我们还提出了一个新的损失，更好地与日记化错误率相比，二进制交叉熵$\unicode{x2013}$日记化端到端系统的默认选择。跨三个数据集（AMI，CALLHOME和DIHARD II）的概念验证结果证明了该方法自动查找超参数的能力，实现了与广泛网格搜索（通常需要额外的超参数行为知识）所找到的性能相当的性能。此外，我们表明，歧视微调PLDA可以进一步提高模型的性能。我们发布的源代码与此出版物。摘要：Bayesian HMM clustering of x-vector sequences (VBx) has become a widely adopted diarization baseline model in publications and challenges. It uses an HMM to model speaker turns, a generatively trained probabilistic linear discriminant analysis (PLDA) for speaker distribution modeling, and Bayesian inference to estimate the assignment of x-vectors to speakers. This paper presents a new framework for updating the VBx parameters using discriminative training, which directly optimizes a predefined loss. We also propose a new loss that better correlates with the diarization error rate compared to binary cross-entropy $\unicode{x2013}$ the default choice for diarization end-to-end systems. Proof-of-concept results across three datasets (AMI, CALLHOME, and DIHARD II) demonstrate the method's capability of automatically finding hyperparameters, achieving comparable performance to those found by extensive grid search, which typically requires additional hyperparameter behavior knowledge. Moreover, we show that discriminative fine-tuning of PLDA can further improve the model's performance. We release the source code with this publication.
【13】 End-to-End Continuous Speech Emotion Recognition in Real-life Customer  Service Call Center Conversations标题：真实客户服务呼叫中心对话中的端到端连续语音情感识别链接：https://arxiv.org/abs/2310.02281
作者：Yajing Feng,Laurence Devillers机构：LISN, and AI Lab, CNRS, Paris-Saclay University, and Axys Consultants, Orsay, France备注：None摘要：语音情感识别（SER）在呼叫中心的对话已经成为一个有价值的工具，用于评估客户端和代理之间的互动质量。与受控的实验室环境相反，现实生活中的对话发生在不受控制的条件下，并且受到影响情感表达的上下文因素的影响。在本文中，我们提出了我们的方法来构建一个大规模的现实生活中的数据集（CusEmo）连续SER在客户服务呼叫中心的对话。我们采用维度情感标注方法来捕捉真实呼叫中心对话中情感的微妙性、复杂性和连续性，同时标注上下文信息。该研究还解决了端到端（E2E）SER系统的数据集的应用过程中遇到的挑战，包括确定适当的标签采样率和输入段长度，以及整合上下文信息（对话者的性别和同理心水平）与不同的权重使用多任务学习。结果表明，引入共情水平信息后，模型的性能得到了提高。摘要：Speech Emotion recognition (SER) in call center conversations has emerged as a valuable tool for assessing the quality of interactions between clients and agents. In contrast to controlled laboratory environments, real-life conversations take place under uncontrolled conditions and are subject to contextual factors that influence the expression of emotions. In this paper, we present our approach to constructing a large-scale reallife dataset (CusEmo) for continuous SER in customer service call center conversations. We adopted the dimensional emotion annotation approach to capture the subtlety, complexity, and continuity of emotions in real-life call center conversations, while annotating contextual information. The study also addresses the challenges encountered during the application of the End-to-End (E2E) SER system to the dataset, including determining the appropriate label sampling rate and input segment length, as well as integrating contextual information (interlocutor's gender and empathy level) with different weights using multitask learning. The result shows that incorporating the empathy level information improved the model's performance.
eess.AS音频处理
【1】 Zero Resource Code-switched Speech Benchmark Using Speech Utterance  Pairs For Multiple Spoken Languages标题：使用多语言语音话语对的零资源码切换语音基准链接：https://arxiv.org/abs/2310.03018
作者：Kuan-Po Huang,Chih-Kai Yang,Yu-Kuan Fu,Ewan Dunbar,Hung-yi Lee机构：National Taiwan University, ASUS Intelligent Cloud Services, University of Toronto备注：Submitted to ICASSP 2024摘要：我们引入了一个新的零资源代码切换语音基准，旨在直接评估自监督语音编码器的代码切换能力。我们展示了一个基线系统的离散单元语言建模，以演示如何代码切换能力的语音编码器可以在零资源的方式进行评估。我们的实验涵盖了各种知名的语音编码器，包括Wav2vec 2.0，HuBERT，XLSR等。我们研究了预训练语言和模型大小对基准测试性能的影响。值得注意的是，虽然我们的研究结果表明，以XLSR为例的多语言预训练语音编码器在代码切换场景中的表现优于单语变体（Wav2vec 2.0，HuBERT），但其代码切换语言能力仍有很大的改进空间。摘要：We introduce a new zero resource code-switched speech benchmark designed to directly assess the code-switching capabilities of self-supervised speech encoders. We showcase a baseline system of language modeling on discrete units to demonstrate how the code-switching abilities of speech encoders can be assessed in a zero-resource manner. Our experiments encompass a variety of well-known speech encoders, including Wav2vec 2.0, HuBERT, XLSR, etc. We examine the impact of pre-training languages and model size on benchmark performance. Notably, though our results demonstrate that speech encoders with multilingual pre-training, exemplified by XLSR, outperform monolingual variants (Wav2vec 2.0, HuBERT) in code-switching scenarios, there is still substantial room for improvement in their code-switching linguistic abilities.
【2】 Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech  Model标题：自监督编解码器语音模型的提示和适配调整链接：https://arxiv.org/abs/2310.02971
作者：Kai-Wei Chang,Ming-Hsin Chen,Yun-Ping Lin,Jing Neng Hsu,Paul Kuo-Ming Huang,Chien-yu Huang,Shang-Wen Li,Hung-yi Lee机构：Graduate Institute of Communication Engineering, National Taiwan University, Taiwan,  Department of Computer Science and Information Engineering, National Taiwan University,Meta AI备注：Accepted to IEEE ASRU 2023摘要：提示和适配器调整已经成为微调（FT）方法的有效替代方案。然而，现有的语音提示研究主要集中在分类任务上，而在更复杂的序列生成任务上则失败了。此外，适配器调优主要应用于仅编码器的自监督模型。我们的实验表明，提示Wav 2Seq，一个自我监督的编码器-解码器模型，超过以前的作品序列生成任务。它实现了一个显着的53%的相对改善，在字错误率为ASR和27%的F1分数槽填充。此外，在低资源场景中，提示与FT方法竞争。此外，我们展示了跨语言ASR中Wav 2Seq上的提示和适配器调整的可移植性。当涉及有限的可训练参数时，提示和适配器调整在7种语言中始终优于传统的FT。值得注意的是，在低资源场景中，提示始终优于适配器调优。摘要：Prompting and adapter tuning have emerged as efficient alternatives to fine-tuning (FT) methods. However, existing studies on speech prompting focused on classification tasks and failed on more complex sequence generation tasks. Besides, adapter tuning is primarily applied with a focus on encoder-only self-supervised models. Our experiments show that prompting on Wav2Seq, a self-supervised encoder-decoder model, surpasses previous works in sequence generation tasks. It achieves a remarkable 53% relative improvement in word error rate for ASR and a 27% in F1 score for slot filling. Additionally, prompting competes with the FT method in the low-resource scenario. Moreover, we show the transferability of prompting and adapter tuning on Wav2Seq in cross-lingual ASR. When limited trainable parameters are involved, prompting and adapter tuning consistently outperform conventional FT across 7 languages. Notably, in the low-resource scenario, prompting consistently outperforms adapter tuning.
【3】 VITS-Based Singing Voice Conversion Leveraging Whisper and multi-scale  F0 Modeling标题：利用Whisper和多尺度F0建模的VITS歌唱语音转换链接：https://arxiv.org/abs/2310.02802
作者：Ziqian Ning,Yuepeng Jiang,Zhichao Wang,Bin Zhang,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, Lyra Lab, Tencent Music Entertainment, Shenzhen, China摘要：本文介绍了T23团队提交给2023年歌唱声音转换挑战赛的系统。遵循发音-合成框架，我们的歌唱转换模型基于VITS，包含四个关键模块：先验编码器、后验编码器、解码器和并行转置卷积组（PBTC）模块。我们特别利用Whisper，一个强大的预训练ASR模型，提取瓶颈特征（BNF）作为先前编码器的输入。在BNF提取之前，我们对源信号进行基音扰动以去除说话人音色，有效地避免了源说话人音色泄漏到目标中。此外，PBTC模块提取多尺度F0作为到现有编码器的辅助输入，从而捕获歌唱的更好的音高变化。我们设计了一个三阶段的训练策略，以更好地适应基础模型的目标说话人有限的目标说话人数据。官方挑战结果显示，我们的系统在自然度方面表现优异，在任务1和任务2中分别排名第1和第2。进一步消融证明了我们的系统设计的有效性。摘要：This paper introduces the T23 team's system submitted to the Singing Voice Conversion Challenge 2023. Following the recognition-synthesis framework, our singing conversion model is based on VITS, incorporating four key modules: a prior encoder, a posterior encoder, a decoder, and a parallel bank of transposed convolutions (PBTC) module. We particularly leverage Whisper, a powerful pre-trained ASR model, to extract bottleneck features (BNF) as the input of the prior encoder. Before BNF extraction, we perform pitch perturbation to the source signal to remove speaker timbre, which effectively avoids the leakage of the source speaker timbre to the target. Moreover, the PBTC module extracts multi-scale F0 as the auxiliary input to the prior encoder, thereby capturing better pitch variations of singing. We design a three-stage training strategy to better adapt the base model to the target speaker with limited target speaker data. Official challenge results show that our system has superior performance in naturalness, ranking 1st and 2nd respectively in Task 1 and 2. Further ablation justifies the effectiveness of our system design.
【4】 Discriminative Training of VBx Diarization标题：VBx二值化的判别训练链接：https://arxiv.org/abs/2310.02732
作者：Dominik Klement,Mireia Diez,Federico Landini,Lukáš Burget,Anna Silnova,Marc Delcroix,Naohiro Tawara机构： Naohiro Tawara 2 1Brno University of Technology备注：Submitted to ICASSP 2024摘要：x向量序列的贝叶斯HMM聚类（VBx）已成为出版物和挑战中广泛采用的日志化基线模型。它使用HMM来建模扬声器回合，生成训练的概率线性判别分析（PLDA）扬声器分布建模，和贝叶斯推理来估计分配的x向量的扬声器。本文提出了一种新的框架更新VBx参数使用判别式训练，直接优化预定义的损失。我们还提出了一个新的损失，更好地与日记化错误率相比，二进制交叉熵$\unicode{x2013}$日记化端到端系统的默认选择。跨三个数据集（AMI，CALLHOME和DIHARD II）的概念验证结果证明了该方法自动查找超参数的能力，实现了与广泛网格搜索（通常需要额外的超参数行为知识）所找到的性能相当的性能。此外，我们表明，歧视微调PLDA可以进一步提高模型的性能。我们发布的源代码与此出版物。摘要：Bayesian HMM clustering of x-vector sequences (VBx) has become a widely adopted diarization baseline model in publications and challenges. It uses an HMM to model speaker turns, a generatively trained probabilistic linear discriminant analysis (PLDA) for speaker distribution modeling, and Bayesian inference to estimate the assignment of x-vectors to speakers. This paper presents a new framework for updating the VBx parameters using discriminative training, which directly optimizes a predefined loss. We also propose a new loss that better correlates with the diarization error rate compared to binary cross-entropy $\unicode{x2013}$ the default choice for diarization end-to-end systems. Proof-of-concept results across three datasets (AMI, CALLHOME, and DIHARD II) demonstrate the method's capability of automatically finding hyperparameters, achieving comparable performance to those found by extensive grid search, which typically requires additional hyperparameter behavior knowledge. Moreover, we show that discriminative fine-tuning of PLDA can further improve the model's performance. We release the source code with this publication.
【5】 Continual Contrastive Spoken Language Understanding标题：连续对比口语理解链接：https://arxiv.org/abs/2310.02699
作者：Umberto Cappellazzo,Enrico Fini,Muqiao Yang,Daniele Falavigna,Alessio Brutti,Bhiksha Raj机构：University of Trento,Independent Researcher,Carnegie Mellon University,Fondazione Bruno, Kessler摘要：最近，神经网络在各个领域都取得了令人印象深刻的进展，语音处理也不例外。然而，该领域的最新突破需要使用大型数据集和巨大的计算资源进行广泛的离线训练。不幸的是，这些模型在不断学习新任务时很难保留先前获得的知识，从头开始重新训练几乎总是不切实际的。在本文中，我们研究的问题，学习序列到序列模型的口语理解类增量学习（CIL）设置，我们提出了COCONUT，CIL方法，依赖于经验重放和对比学习的结合。通过仅应用于排练样本的标准监督对比损失的修改版本，COCONUT通过从同一类中提取更接近的样本并推开其他样本来保留学习的表示。此外，我们利用多模态对比损失，通过对齐音频和文本特征，帮助模型学习新数据的更多区分性表示。我们还研究了不同的对比设计，结合对比损失的优势与师生架构用于蒸馏。两个已建立的SLU数据集上的实验表明，我们提出的方法的有效性和显着的改进基线。我们还表明，COCONUT可以与操作在解码器侧的模型，从而导致进一步的度量改进的方法相结合。摘要：Recently, neural networks have shown impressive progress across diverse fields, with speech processing being no exception. However, recent breakthroughs in this area require extensive offline training using large datasets and tremendous computing resources. Unfortunately, these models struggle to retain their previously acquired knowledge when learning new tasks continually, and retraining from scratch is almost always impractical. In this paper, we investigate the problem of learning sequence-to-sequence models for spoken language understanding in a class-incremental learning (CIL) setting and we propose COCONUT, a CIL method that relies on the combination of experience replay and contrastive learning. Through a modified version of the standard supervised contrastive loss applied only to the rehearsal samples, COCONUT preserves the learned representations by pulling closer samples from the same class and pushing away the others. Moreover, we leverage a multimodal contrastive loss that helps the model learn more discriminative representations of the new data by aligning audio and text features. We also investigate different contrastive designs to combine the strengths of the contrastive loss with teacher-student architectures used for distillation. Experiments on two established SLU datasets reveal the effectiveness of our proposed approach and significant improvements over the baselines. We also show that COCONUT can be combined with methods that operate on the decoder side of the model, resulting in further metrics improvements.
【6】 The VoiceMOS Challenge 2023: Zero-shot Subjective Speech Quality  Prediction for Multiple Domains标题：VoiceMOS挑战赛2023：多域零发声主观语音质量预测链接：https://arxiv.org/abs/2310.02640
作者：Erica Cooper,Wen-Chin Huang,Yu Tsao,Hsin-Min Wang,Tomoki Toda,Junichi Yamagishi机构：National Institute of Informatics, Japan, Nagoya University, Japan, Academia Sinica, Taiwan备注：Accepted to ASRU 2023摘要：我们提出了第二版的VoiceMOS挑战，一个科学的事件，旨在促进自动预测的平均意见得分（MOS）的合成和处理的语音的研究。今年，我们强调现实世界和具有挑战性的zero-shot域外MOS预测与三个不同的语音评估方案的三个轨道。来自7个不同国家的工业界和学术界的10个小组参加了会议。令人惊讶的是，我们发现法语文本到语音合成的两个子轨道在其可预测性方面存在很大差异，并且唱歌语音转换的样本并不像我们预期的那样难以预测。在培训期间使用各种数据集和听众信息似乎是成功的办法。摘要：We present the second edition of the VoiceMOS Challenge, a scientific event that aims to promote the study of automatic prediction of the mean opinion score (MOS) of synthesized and processed speech. This year, we emphasize real-world and challenging zero-shot out-of-domain MOS prediction with three tracks for three different voice evaluation scenarios. Ten teams from industry and academia in seven different countries participated. Surprisingly, we found that the two sub-tracks of French text-to-speech synthesis had large differences in their predictability, and that singing voice-converted samples were not as difficult to predict as we had expected. Use of diverse datasets and listener information during training appeared to be successful approaches.
【7】 End-to-End Continuous Speech Emotion Recognition in Real-life Customer  Service Call Center Conversations标题：真实客户服务呼叫中心对话中的端到端连续语音情感识别链接：https://arxiv.org/abs/2310.02281
作者：Yajing Feng,Laurence Devillers机构：LISN, and AI Lab, CNRS, Paris-Saclay University, and Axys Consultants, Orsay, France备注：None摘要：语音情感识别（SER）在呼叫中心的对话已经成为一个有价值的工具，用于评估客户端和代理之间的互动质量。与受控的实验室环境相反，现实生活中的对话发生在不受控制的条件下，并且受到影响情感表达的上下文因素的影响。在本文中，我们提出了我们的方法来构建一个大规模的现实生活中的数据集（CusEmo）连续SER在客户服务呼叫中心的对话。我们采用维度情感标注方法来捕捉真实呼叫中心对话中情感的微妙性、复杂性和连续性，同时标注上下文信息。该研究还解决了端到端（E2E）SER系统的数据集的应用过程中遇到的挑战，包括确定适当的标签采样率和输入段长度，以及整合上下文信息（对话者的性别和同理心水平）与不同的权重使用多任务学习。结果表明，引入共情水平信息后，模型的性能得到了提高。摘要：Speech Emotion recognition (SER) in call center conversations has emerged as a valuable tool for assessing the quality of interactions between clients and agents. In contrast to controlled laboratory environments, real-life conversations take place under uncontrolled conditions and are subject to contextual factors that influence the expression of emotions. In this paper, we present our approach to constructing a large-scale reallife dataset (CusEmo) for continuous SER in customer service call center conversations. We adopted the dimensional emotion annotation approach to capture the subtlety, complexity, and continuity of emotions in real-life call center conversations, while annotating contextual information. The study also addresses the challenges encountered during the application of the End-to-End (E2E) SER system to the dataset, including determining the appropriate label sampling rate and input segment length, as well as integrating contextual information (interlocutor's gender and empathy level) with different weights using multitask learning. The result shows that incorporating the empathy level information improved the model's performance.
【8】 UniverSLU: Universal Spoken Language Understanding for Diverse  Classification and Sequence Generation Tasks with a Single Network标题：通过单一网络实现对不同分类和序列生成任务的通用口语理解链接：https://arxiv.org/abs/2310.02973
作者：Siddhant Arora,Hayato Futami,Jee-weon Jung,Yifan Peng,Roshan Sharma,Yosuke Kashiwagi,Emiru Tsunoo,Shinji Watanabe机构：Carnegie Mellon University, Sony Group Corporation摘要：最近的研究表明，通过采用具有多任务处理能力的大型语言模型，可以获得很好的结果。它们利用提示来指导模型的行为，并超越特定任务模型的性能。基于此，我们问：我们是否可以建立一个单一的模型，共同执行各种口语理解（SLU）的任务？为了解决这个问题，我们利用预训练的自动语音识别（ASR）模型，并采用各种任务和数据集说明符作为离散提示。我们展示了我们的单一多任务学习（MTL）模型“UniversSLU”在17个数据集和9种语言中的12个不同的语音分类和序列生成任务的有效性。结果表明，UniverSLU实现了有竞争力的性能，甚至超过了特定任务的模型。我们还进行了初步调查，使人类可解释的自然短语，而不是任务说明符作为离散提示和测试模型的泛化能力，以新的释义。摘要：Recent studies have demonstrated promising outcomes by employing large language models with multi-tasking capabilities. They utilize prompts to guide the model's behavior and surpass performance of task-specific models. Motivated by this, we ask: can we build a single model that jointly perform various spoken language understanding (SLU) tasks? To address this, we utilize pre-trained automatic speech recognition (ASR) models and employ various task and dataset specifiers as discrete prompts. We demonstrate efficacy of our single multi-task learning (MTL) model "UniverSLU" for 12 different speech classification and sequence generation tasks across 17 datasets and 9 languages. Results show that UniverSLU achieves competitive performance and even surpasses task-specific models. We also conduct preliminary investigations into enabling human-interpretable natural phrases instead of task specifiers as discrete prompts and test the model's generalization capabilities to new paraphrases.
【9】 End-to-End Training of a Neural HMM with Label and Transition  Probabilities标题：具有标签和转移概率的神经HMM的端到端训练链接：https://arxiv.org/abs/2310.02724
作者：Daniel Mann,Tina Raissi,Wilfried Michel,Ralf Schlüter,Hermann Ney机构：AppTek GmbH, Aachen, Germany, Machine Learning and Human Language Technology, Computer Science Department, RWTH Aachen University, Aachen, Germany备注：Accepted for Presentation at ASRU2023摘要：我们研究了一种使用隐马尔可夫模型（HMM）进行端到端神经网络训练的新建模方法，其中隐状态之间的转移概率被明确建模和学习。大多数当代序列到序列模型允许通过对给定拓扑中的所有可能的标签分割求和来从头开始训练。在我们的方法中，有明确的，可学习的概率段之间的过渡，而不是一个空白的标签，隐式编码的持续时间统计。我们实现了一个基于GPU的向前向后算法，使标签和转移概率的同时训练。我们调查识别结果，并另外维特比对齐我们的模型。我们发现，虽然过渡模型训练并没有提高识别性能，但它对对齐质量有积极的影响。所产生的对齐被示出为在最先进的维特比训练中是可行的目标。摘要：We investigate a novel modeling approach for end-to-end neural network training using hidden Markov models (HMM) where the transition probabilities between hidden states are modeled and learned explicitly. Most contemporary sequence-to-sequence models allow for from-scratch training by summing over all possible label segmentations in a given topology. In our approach there are explicit, learnable probabilities for transitions between segments as opposed to a blank label that implicitly encodes duration statistics. We implement a GPU-based forward-backward algorithm that enables the simultaneous training of label and transition probabilities. We investigate recognition results and additionally Viterbi alignments of our models. We find that while the transition model training does not improve recognition performance, it has a positive impact on the alignment quality. The generated alignments are shown to be viable targets in state-of-the-art Viterbi trainings.
【10】 Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised  Learning with Masked Unit Prediction标题：多分辨率Hubert：带掩蔽单元预测的多分辨率语音自监督学习链接：https://arxiv.org/abs/2310.02720
作者：Jiatong Shi,Hirofumi Inaguma,Xutai Ma,Ilia Kulikov,Anna Sun机构： Language Technologies Institute, Carnegie Mellon University, Meta AI备注：35 pages, 4 figures, 21 tables摘要：现有的语音自监督学习（SSL）模型通常以20毫秒的固定分辨率处理语音信号。这种方法忽略了语音信号中以不同分辨率存在的变化的信息内容。相比之下，本文旨在将多分辨率信息纳入语音自监督表示学习。我们引入了一个SSL模型，利用层次化的Transformer架构，辅以HUBERT风格的掩蔽预测目标，在多个分辨率处理语音。实验结果表明，该模型不仅实现了更有效的推理，但也表现出优越的或可比的性能，在各种任务的原始HuBERT模型。具体而言，在LibriSpeech语音识别基准的微调实验以及使用语音通用性能基准（SUPERB）和多语言SUPERB（ML-SUPERB）的评估中，已经观察到了原始HuBERT的显着性能改进。摘要：Existing Self-Supervised Learning (SSL) models for speech typically process speech signals at a fixed resolution of 20 milliseconds. This approach overlooks the varying informational content present at different resolutions in speech signals. In contrast, this paper aims to incorporate multi-resolution information into speech self-supervised representation learning. We introduce a SSL model that leverages a hierarchical Transformer architecture, complemented by HuBERT-style masked prediction objectives, to process speech at multiple resolutions. Experimental results indicate that the proposed model not only achieves more efficient inference but also exhibits superior or comparable performance to the original HuBERT model over various tasks. Specifically, significant performance improvements over the original HuBERT have been observed in fine-tuning experiments on the LibriSpeech speech recognition benchmark as well as in evaluations using the Speech Universal PERformance Benchmark (SUPERB) and Multilingual SUPERB (ML-SUPERB).
【11】 BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching  Speech Recognition标题：BA-MOE：用于码型转换语音识别的边界感知混合专家适配器链接：https://arxiv.org/abs/2310.02629
作者：Peikun Chen,Fan Yu,Yuhao Lian,Hongfei Xue,Xucheng Wan,Naijun Zheng,Huan Zhou,Lei Xie机构：Northwestern Polytechnical University, Xian, China, Huawei备注：5 pages,3figures,Accept by ASRU2023摘要：基于专家混合的模型利用语言专家有效地提取特定语言的表征，在语码转换的自动语音识别中得到了很好的应用。然而，仍然有很大的改进空间，因为跨语言的相似发音可能导致无效的多语言建模和不准确的语言边界估计。为了消除这些缺点，我们提出了一个跨层的语言适配器和边界感知的训练方法，即边界感知混合专家（BA-MoE）。具体来说，我们引入特定于语言的适配器来分离特定于语言的表示和统一的门控层来融合每个编码器层内的表示。其次，我们计算每个特定语言适配器的平均输出的语言适应损失，以改善适配器模块的特定语言表示学习。此外，我们利用一个边界感知预测器来学习边界表示，以处理语言边界混淆。我们的方法实现了显着的性能改进，与ASRU 2019年普通话-英语代码转换挑战数据集的基线相比，混合错误率降低了16.55%。摘要：Mixture-of-experts based models, which use language experts to extract language-specific representations effectively, have been well applied in code-switching automatic speech recognition. However, there is still substantial space to improve as similar pronunciation across languages may result in ineffective multi-language modeling and inaccurate language boundary estimation. To eliminate these drawbacks, we propose a cross-layer language adapter and a boundary-aware training method, namely Boundary-Aware Mixture-of-Experts (BA-MoE). Specifically, we introduce language-specific adapters to separate language-specific representations and a unified gating layer to fuse representations within each encoder layer. Second, we compute language adaptation loss of the mean output of each language-specific adapter to improve the adapter module's language-specific representation learning. Besides, we utilize a boundary-aware predictor to learn boundary representations for dealing with language boundary confusion. Our approach achieves significant performance improvement, reducing the mixture error rate by 16.55\% compared to the baseline on the ASRU 2019 Mandarin-English code-switching challenge dataset.
【12】 Improving severity preservation of healthy-to-pathological voice  conversion with global style tokens标题：使用全局样式令牌改进健康到病态语音转换的严重性保存链接：https://arxiv.org/abs/2310.02570
作者：Bence Mark Halpern,Wen-Chin Huang,Lester Phillip Violeta,R. J. J. H. van Son,Tomoki Toda机构：Nagoya University, Japan, Netherlands Cancer Institute, The Netherlands备注：7 pages, 3 figures, 5 tables. Accepted to IEEE Automatic Speech Recognition and Understanding Workshop 2023摘要：在健康到病态语音转换（H2 P-VC）中，健康语音被转换为病态语音，同时保留身份。本文改进了先前的H2 P-VC的两阶段方法，其中（1）首先以适当的严重性创建语音，（2）然后在保持语音的严重性的同时转换语音的说话者身份。具体而言，我们提出了改进（2）通过使用语音后验图（PPG）和全球风格令牌（GST）。此外，我们提出了一个新的数据集，其中包含具有相同身份的病理和健康扬声器的并行记录，从而可以进行更精确的评估。专家听众的听力测试表明，该框架保留了源样本的严重性，同时建模目标扬声器的声音。我们还表明，（a）病理影响x向量，但不是所有的扬声器信息丢失，（b）选择源扬声器的严重性标签的基础上单独是不够的。摘要：In healthy-to-pathological voice conversion (H2P-VC), healthy speech is converted into pathological while preserving the identity. The paper improves on previous two-stage approach to H2P-VC where (1) speech is created first with the appropriate severity, (2) then the speaker identity of the voice is converted while preserving the severity of the voice. Specifically, we propose improvements to (2) by using phonetic posteriorgrams (PPG) and global style tokens (GST). Furthermore, we present a new dataset that contains parallel recordings of pathological and healthy speakers with the same identity which allows more precise evaluation. Listening tests by expert listeners show that the framework preserves severity of the source sample, while modelling target speaker's voice. We also show that (a) pathology impacts x-vectors but not all speaker information is lost, (b) choosing source speakers based on severity labels alone is insufficient.
【13】 Shaping the Epochal Individuality and Generality: The Temporal Dynamics  of Uncertainty and Prediction Error in Musical Improvisation标题：塑造时代的个性和共性：音乐即兴创作中不确定性和预测误差的时间动力学链接：https://arxiv.org/abs/2310.02518
作者：Tatsuya Daikoku机构： 3 1 Graduate School of Information Science and Technology, The University of Tokyo 2 Centre for Neuroscience in Education, University of Cambridge 3 Center for Brain摘要：音乐即兴创作，就像即兴演讲一样，揭示了即兴创作者的精神状态和情感特征的复杂方面。然而，揭示这种个性的特定音乐成分在很大程度上仍未被探索。在大脑的统计学习和预测处理的框架内，本研究探讨了音乐即兴创作中的不确定性和惊喜（预测错误）的时间动态。本研究采用HBSL模型分析了从1905年到2009年的456首爵士乐即兴作品，来自78位不同的爵士音乐家。结果表明，独特的时间模式的惊喜和不确定性，特别是在音高和音高节奏序列，揭示了特定时代的功能，从20世纪初到21世纪。相反，节奏序列表现出跨时代的一致程度的不确定性。此外，声学特性在不同时期保持不变。这些发现强调了即兴音乐中惊喜和不确定性的时间动态如何随着时间的推移而变化的重要性，深刻影响了每个时代艺术家为即兴创作所采用的独特方法。此外，有人建议，即兴音乐的发展可以归因于大脑的自适应统计学习机制，不断完善内部模型，以反映各自时代的文化和情感的细微差别。这项研究揭示了即兴音乐的演变轨迹，并强调了艺术家们为与他们所处时代的文化和情感景观产生共鸣而进行的微妙转变。摘要：Musical improvisation, much like spontaneous speech, reveals intricate facets of the improviser's state of mind and emotional character. However, the specific musical components that reveal such individuality remain largely unexplored. Within the framework of brain's statistical learning and predictive processing, this study examined the temporal dynamics of uncertainty and surprise (prediction error) in a piece of musical improvisation. This study employed the HBSL model to analyze a corpus of 456 Jazz improvisations, spanning 1905 to 2009, from 78 distinct Jazz musicians. The results indicated distinctive temporal patterns of surprise and uncertainty, especially in pitch and pitch-rhythm sequences, revealing era-specific features from the early 20th to the 21st centuries. Conversely, rhythm sequences exhibited a consistent degree of uncertainty across eras. Further, the acoustic properties remain unchanged across different periods. These findings highlight the importance of how temporal dynamics of surprise and uncertainty in improvisational music change over periods, profoundly influencing the distinctive methodologies artists adopt for improvisation in each era. Further, it is suggested that the development of improvisational music can be attributed to the brain's adaptive statistical learning mechanisms, which constantly refine internal models to mirror the cultural and emotional nuances of their respective epochs. This study unravels the evolutionary trajectory of improvisational music and highlights the nuanced shifts artists employ to resonate with the cultural and emotional landscapes of their times.
【14】 Towards an Interpretable Representation of Speaker Identity via  Perceptual Voice Qualities标题：通过感知语音质量实现说话人身份的可解释表征链接：https://arxiv.org/abs/2310.02497
作者：Robin Netzorg,Bohan Yu,Andrea Guzman,Peter Wu,Luna McNulty,Gopala Anumanchipalli机构：University of California, Berkeley,Brown University摘要：与文本和视觉等其他数据形式不同，语音并不容易解释。虽然外行人可以理解如何通过感知来描述图像或句子，但非专家对语音的描述通常以高级人口统计信息（如性别或年龄）结束。在本文中，我们提出了一个可能的可解释的表示的扬声器身份的感知语音质量（PQ）的基础上。通过将性别PQ添加到以病理学为中心的共识听觉感知评价的语音（CAPE-V）协议，我们的基于PQ的方法提供了一个感性的潜在空间的字符的成人的声音，是一个中介的抽象之间的高层次的人口统计和低层次的声学，物理，或学习表示。与之前的信念相反，我们证明了这些PQ是由非专家的合奏可听，并进一步证明了在基于PQ的表示编码的信息是可预测的各种语音表示。摘要：Unlike other data modalities such as text and vision, speech does not lend itself to easy interpretation. While lay people can understand how to describe an image or sentence via perception, non-expert descriptions of speech often end at high-level demographic information, such as gender or age. In this paper, we propose a possible interpretable representation of speaker identity based on perceptual voice qualities (PQs). By adding gendered PQs to the pathology-focused Consensus Auditory-Perceptual Evaluation of Voice (CAPE-V) protocol, our PQ-based approach provides a perceptual latent space of the character of adult voices that is an intermediary of abstraction between high-level demographics and low-level acoustic, physical, or learned representations. Contrary to prior belief, we demonstrate that these PQs are hearable by ensembles of non-experts, and further demonstrate that the information encoded in a PQ-based representation is predictable by various speech representations.
【15】 ResidualTransformer: Residual Low-rank Learning with Weight-sharing for  Transformer Layers标题：残差Transformer层的加权残差低阶学习链接：https://arxiv.org/abs/2310.02489
作者：Yiming Wang,Jinyu Li机构：Microsoft Corporation, Redmond, WA, USA备注：Submitted to IEEE ICASSP 2024. 5 pages, 1 figure摘要：在这些设备上部署语音处理模型时，始终在线设备的存储器约束是主要关注点之一。虽然用足够大量的数据训练的较大模型通常表现更好，但使它们适合设备存储器是一项艰巨的挑战。在本文中，我们的目标是通过重新参数化模型权重跨Transformer编码器层，并假设一个特殊的权重组成和结构，以减少模型的大小。更具体地说，受ResNet和最近LoRA工作的启发，我们提出了一种名为ResidualTransformer的方法，其中Transformer层中的每个权重矩阵包括1）与其相邻层共享的满秩分量，以及2）其自身的唯一低秩分量。低秩矩阵仅占少量的模型大小增加。此外，我们加入对角权重矩阵，以提高建模能力的低秩矩阵。我们的10k小时的语音识别和语音翻译任务的实验表明，Transformer编码器的大小可以减少约3倍，性能下降非常轻微。摘要：Memory constraint of always-on devices is one of the major concerns when deploying speech processing models on these devices. While larger models trained with sufficiently large amount of data generally perform better, making them fit in the device memory is a demanding challenge. In this paper, we aim to reduce model size by reparameterizing model weights across Transformer encoder layers and assuming a special weight composition and structure. More specifically, inspired by ResNet and the more recent LoRA work, we propose an approach named ResidualTransformer, where each weight matrix in a Transformer layer comprises 1) a shared full-rank component with its adjacent layers, and 2) a unique low-rank component to itself. The low-rank matrices only account for a small amount of model size increase. In addition, we add diagonal weight matrices to improve modeling capacity of the low-rank matrices. Experiments of our 10k-hour speech recognition and speech translation tasks show that the Transformer encoder size can be reduced by ~3X with very slight performance degradation.
【16】 Unsupervised Speech Recognition with N-Skipgram and Positional Unigram  Matching标题：基于N-Skipgram和位置单字匹配的无监督语音识别链接：https://arxiv.org/abs/2310.02382
作者：Liming Wang,Mark Hasegawa-Johnson,Chang D. Yoo机构：⋆Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, †Artificial Intelligence and Machine Learning Lab, KAIST摘要：由于GAN相关的不稳定性，语音和文本之间的不对齐以及大量的内存需求，训练无监督语音识别系统面临着挑战。为了应对这些挑战，我们介绍了一种新的ASR系统，ESPUM。该系统利用低阶N-跳图（最多N=3）的能力，并结合从一小批样本中收集的位置一元统计数据。评估TIMIT基准，我们的模型展示了竞争力的表现，在ASR和音素分割任务。访问我们的公开代码https://github.com/lwang114/GraphUnsupASR。摘要：Training unsupervised speech recognition systems presents challenges due to GAN-associated instability, misalignment between speech and text, and significant memory demands. To tackle these challenges, we introduce a novel ASR system, ESPUM. This system harnesses the power of lower-order N-skipgrams (up to N=3) combined with positional unigram statistics gathered from a small batch of samples. Evaluated on the TIMIT benchmark, our model showcases competitive performance in ASR and phoneme segmentation tasks. Access our publicly available code at https://github.com/lwang114/GraphUnsupASR.
【17】 Prompting Audios Using Acoustic Properties For Emotion Representation标题：利用声学特性进行情感表征的提示音频链接：https://arxiv.org/abs/2310.02298
作者：Hira Dhamyal,Benjamin Elizalde,Soham Deshmukh,Huaming Wang,Bhiksha Raj,Rita Singh机构： Carnegie Mellon University,Microsoft,Mohammed bin Zayed University of AI备注：arXiv admin note: substantial text overlap with arXiv:2211.07737摘要：情绪是一个连续体，但目前的模型将情绪视为一个有限值的离散变量。这种表达方式没有捕捉到情感表达的多样性。为了更好地表示情感，我们建议使用自然语言描述（或提示）。在这项工作中，我们解决了自动生成这些提示并训练模型以更好地从音频和提示对中学习情感表示的挑战。我们使用与情绪相关的声学属性，如音高，强度，语速和清晰度，自动生成提示，即“声音提示”。我们使用一个对比的学习目标，以映射语音到各自的声学提示。我们评估我们的模型情感音频检索和语音情感识别。我们的研究结果表明，声学提示显着提高模型的性能在EAR中，在各种精度@K指标。在SER中，我们观察到Ravdess数据集的相对准确度提高了3.8%。摘要：Emotions lie on a continuum, but current models treat emotions as a finite valued discrete variable. This representation does not capture the diversity in the expression of emotion. To better represent emotions we propose the use of natural language descriptions (or prompts). In this work, we address the challenge of automatically generating these prompts and training a model to better learn emotion representations from audio and prompt pairs. We use acoustic properties that are correlated to emotion like pitch, intensity, speech rate, and articulation rate to automatically generate prompts i.e. 'acoustic prompts'. We use a contrastive learning objective to map speech to their respective acoustic prompts. We evaluate our model on Emotion Audio Retrieval and Speech Emotion Recognition. Our results show that the acoustic prompts significantly improve the model's performance in EAR, in various Precision@K metrics. In SER, we observe a 3.8% relative accuracy improvement on the Ravdess dataset.
机器翻译，仅供参考