今日论文合集：cs.SD语音6篇，eess.AS音频处理9篇。
cs.SD语音
【1】 Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis标题：用于语音合成的特定说话人潜在语音特征编码链接：https://arxiv.org/abs/2311.11745
作者：Jungil Kong,Junmo Lee,Jeongmin Kim,Beomjeong Kim,Jihoon Park,Dohee Kong,Changheon Lee,Sangjin Kim摘要：在这项工作中，我们提出了一种新的方法来建模众多的扬声器，它能够表达扬声器的整体特征的细节，就像一个训练有素的多扬声器模型，而无需对目标扬声器的数据集进行额外的训练。虽然具有类似目的的各种作品已被积极研究，但由于其基本局限性，其性能尚未达到训练的多说话人模型的性能。为了克服以前的局限性，我们提出了有效的方法进行特征学习和表示目标说话人的语音特征，通过离散化的功能和条件，他们的语音合成模型。我们的方法获得了显着更高的相似性平均意见得分（SMOS）在主观相似性评价比看到的扬声器的性能最好的多扬声器模型，即使与看不见的扬声器。所提出的方法也优于一个zero-shot方法显着的利润率。此外，我们的方法在生成新的人工扬声器方面表现出显着的性能。此外，我们证明了编码的潜在功能是足够的信息，以重建原始扬声器的语音完全。这意味着我们的方法可以作为一个通用的方法来编码和重建说话人的特点在各种任务。摘要：In this work, we propose a novel method for modeling numerous speakers, which enables expressing the overall characteristics of speakers in detail like a trained multi-speaker model without additional training on the target speaker's dataset. Although various works with similar purposes have been actively studied, their performance has not yet reached that of trained multi-speaker models due to their fundamental limitations. To overcome previous limitations, we propose effective methods for feature learning and representing target speakers' speech characteristics by discretizing the features and conditioning them to a speech synthesis model. Our method obtained a significantly higher similarity mean opinion score (SMOS) in subjective similarity evaluation than seen speakers of a best-performing multi-speaker model, even with unseen speakers. The proposed method also outperforms a zero-shot method by significant margins. Furthermore, our method shows remarkable performance in generating new artificial speakers. In addition, we demonstrate that the encoded latent features are sufficiently informative to reconstruct an original speaker's speech completely. It implies that our method can be used as a general methodology to encode and reconstruct speakers' characteristics in various tasks.
【2】 Encoding Performance Data in MEI with the Automatic Music Performance  Analysis and Comparison Toolkit (AMPACT)标题：用自动音乐演奏分析比较工具包(AMPACT)对MEI中的演奏数据进行编码链接：https://arxiv.org/abs/2311.11363
作者：Johanna Devaney,Cecilia Beauchamp备注：None摘要：本文提出了一种新的方法，编码的性能数据在MEI使用最近添加的\texttt{}元素。使用自动音乐性能分析和比较工具包（AMPACT）提取性能数据，并将其编码为链接到特定音符的\texttt{ }元素中的JSON对象。一组流行音乐的人声已被编码，以展示可以被编码的描述符的范围，以及在没有完全指定的乐谱的情况下如何使用AMPACT来提取性能数据。摘要：This paper presents a new method of encoding performance data in MEI using the recently added \texttt{} element. Performance data was extracted using the Automatic Music Performance Analysis and Comparison Toolkit (AMPACT) and encoded as a JSON object within an \texttt{} element linked to a specific musical note. A set of pop music vocals has was encoded to demonstrate both the range of descriptors that can be encoded inand how AMPACT can be used for extracting performance data in the absence of a fully specified musical score.
【3】 M$^{2}$UGen: Multi-modal Music Understanding and Generation with the  Power of Large Language Models标题：M$^{2}$Ugen：大型语言模型的力量下的多模式音乐理解和生成链接：https://arxiv.org/abs/2311.11255
作者：Atin Sakkeer Hussain,Shansong Liu,Chenshuo Sun,Ying Shan摘要：目前利用大型语言模型（LLM）的研究正在经历一场浪潮。许多作品利用这些模型强大的推理能力来理解各种形式，如文本，语音，图像，视频等，它们还利用LLM来理解人类的意图并生成所需的输出，如图像，视频和音乐。然而，使用LLM将理解和生成相结合的研究仍然有限，并且处于起步阶段。为了解决这个差距，我们引入了多模态音乐理解和生成（M$^{2}$UGen）框架，集成了LLM的能力，理解和生成不同形式的音乐。M$^{2}$UGen框架是专门构建的，旨在通过分别使用预训练的MERT、ViT和ViViT模型，从各种灵感来源（包括音乐、图像和视频）中释放创造潜力。为了实现音乐生成，我们探索了AudioLDM 2和MusicGen的使用。通过LLaMA 2模型的集成，实现了多模态理解和音乐生成的桥梁。此外，我们利用MU-LLaMA模型来生成支持文本/图像/视频到音乐生成的广泛数据集，从而促进了我们的M$^{2}$UGen框架的训练。我们对我们提出的框架进行了全面评估。实验结果表明，我们的模型达到或超过目前最先进的模型的性能。摘要：The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.
【4】 The Persian Piano Corpus: A Collection Of Instrument-Based Feature  Extracted Data Considering Dastgah标题：波斯钢琴语料库：基于乐器的Dastgah特征提取数据集合链接：https://arxiv.org/abs/2311.11074
作者：Parsa Rasouli,Azam Bastanfard备注：including 11 pages and 6 figures. we want to inform related data PPC is submitted to Harvard Dataverse: this https URL摘要：音乐领域的研究正在迅速增长，这一趋势强调了对全面数据的需求。尽管研究人员努力贡献自己的数据集，但许多数据集缺乏全面研究所需的包容性，因为它们经常集中在音乐的特定组成部分或其他特定主题上。我们有endeavor，以解决数据稀缺，采用基于仪器的方法，提供一个完整的语料库相关的波斯钢琴。我们的钢琴语料库包括波斯音乐模式（Dastgah）的相关标签和全面的元数据，允许在各种流行的研究领域中使用。从波斯钢琴语料库（PPC）中的2022首波斯钢琴作品中提取的特征已被收集并提供给研究人员，旨在更深入地了解波斯音乐以及钢琴在其中的作用。摘要：The research in the field of music is rapidly growing, and this trend emphasizes the need for comprehensive data. Though researchers have made an effort to contribute their own datasets, many data collections lack the requisite inclusivity for comprehensive study because they are frequently focused on particular components of music or other specific topics. We have endeavored to address data scarcity by employing an instrument-based approach to provide a complete corpus related to the Persian piano. Our piano corpus includes relevant labels for Persian music mode (Dastgah) and comprehensive metadata, allowing for utilization in various popular research areas. The features extracted from 2022 Persian piano pieces in The Persian Piano Corpus (PPC) have been collected and made available to researchers, aiming for a more thorough understanding of Persian music and the role of the piano in it in subsequent steps.
【5】 Automatic Restoration of Diacritics for Speech Data Sets标题：语音数据集的变音符号自动恢复链接：https://arxiv.org/abs/2311.10771
作者：Sara Shatnawi,Sawsan Alqahtani,Hanan Aldarmaki摘要：自动基于文本的变音符号恢复模型通常具有高变音符号错误率时，应用到语音转录的领域和风格的变化，在口语。在这项工作中，我们探讨的可能性，提高性能的自动变音符号恢复时，应用到语音数据，利用并行的口语。特别是，我们使用预先训练的Whisper ASR模型，对相对少量的变音阿拉伯语语音数据进行微调，以生成语音话语的粗略变音转录，然后将其用作基于transformer的变音恢复模型的额外输入。该模型一致地提高变音恢复性能相比，一个等效的纯文本模型，至少有5\%的绝对减少变音错误率在同一个域和两个域外测试集。我们的研究结果强调了当前基于文本的语音数据集的变音符号恢复模型的不足，并为基于语音的变音符号恢复提供了一个新的基线。摘要：Automatic text-based diacritic restoration models generally have high diacritic error rates when applied to speech transcripts as a result of domain and style shifts in spoken language. In this work, we explore the possibility of improving the performance of automatic diacritic restoration when applied to speech data by utilizing the parallel spoken utterances. In particular, we use the pre-trained Whisper ASR model fine-tuned on relatively small amounts of diacritized Arabic speech data to produce rough diacritized transcripts for the speech utterances, which we then use as an additional input for a transformer-based diacritic restoration model. The proposed model consistently improve diacritic restoration performance compared to an equivalent text-only model, with at least 5\% absolute reduction in diacritic error rate within the same domain and on two out-of-domain test sets. Our results underscore the inadequacy of current text-based diacritic restoration models for speech data sets and provide a new baseline for speech-based diacritic restoration.
【6】 MSPB: a longitudinal multi-sensor dataset with phenotypic trait  measurements from honey bees标题：MSPB：一个带有蜜蜂表型特征测量的纵向多传感器数据集链接：https://arxiv.org/abs/2311.10876
作者：Yi Zhu,Mahsa Abdollahi,Ségolène Maucourt,Nico Coallier,Heitor R. Guimarães,Pierre Giovenazzo,Tiago H. Falk备注：Under review; project webpage: this https URL摘要：我们提出了一个纵向的多传感器数据集收集蜜蜂的殖民地（意大利蜜蜂）与丰富的表型测量。在2020年5月至2021年4月期间，从位于加拿大魁北克省两个养蜂场的53个蜂箱中连续收集数据。传感器数据包括音频特征、温度和相对湿度。表型测量包括蜂箱数量、育雏细胞（卵、幼虫和蛹）数量、狄斯瓦螨侵染水平、防御和卫生行为、蜂蜜产量和冬季死亡率。我们的研究是第一个提供由养蜂科学专家注释的各种各样的表型性状测量的研究之一，这有助于更广泛的分析。我们首先总结了数据采集过程，传感器数据预处理步骤，和数据组成。然后，我们提供了表型数据分布的概述以及传感器数据模式的可视化。最后，我们展示了几个基于传感器数据分析和机器学习的蜂巢监测应用程序，例如冬季死亡率预测，蜂巢人口估计，以及活跃和产卵女王的存在。摘要：We present a longitudinal multi-sensor dataset collected from honey bee colonies (Apis mellifera) with rich phenotypic measurements. Data were continuously collected between May-2020 and April-2021 from 53 hives located at two apiaries in Qu\'ebec, Canada. The sensor data included audio features, temperature, and relative humidity. The phenotypic measurements contained beehive population, number of brood cells (eggs, larva and pupa), Varroa destructor infestation levels, defensive and hygienic behaviors, honey yield, and winter mortality. Our study is amongst the first to provide a wide variety of phenotypic trait measurements annotated by apicultural science experts, which facilitate a broader scope of analysis. We first summarize the data collection procedure, sensor data pre-processing steps, and data composition. We then provide an overview of the phenotypic data distribution as well as a visualization of the sensor data patterns. Lastly, we showcase several hive monitoring applications based on sensor data analysis and machine learning, such as winter mortality prediction, hive population estimation, and the presence of an active and laying queen.
eess.AS音频处理
【1】 How does end-to-end speech recognition training impact speech  enhancement artifacts?标题：端到端语音识别训练如何影响语音增强伪像？链接：https://arxiv.org/abs/2311.11599
作者：Kazuma Iwamoto,Tsubasa Ochiai,Marc Delcroix,Rintaro Ikeshita,Hiroshi Sato,Shoko Araki,Shigeru Katagiri备注：5 pages, 1 figure, 1 table摘要：联合训练语音增强（SE）前端和自动语音识别（ASR）后端已被研究作为一种方式，以减轻由单通道SE产生的处理失真对ASR的影响。在本文中，我们调查的效果，这样的联合训练增强信号的信号电平特性的分解噪声和伪影错误的观点。实验分析提供了两个新的发现：1）SE前端的ASR级训练减少了伪影误差，同时增加了噪声误差，以及2）简单地内插增强和观察到的信号，这实现了减少伪影和增加噪声的类似效果，提高了ASR性能，而无需联合修改SE和ASR模块，即使对于使用WavLM特征提取器的强ASR后端也是如此。我们的研究结果提供了一个更好的理解联合训练的效果和设计一个ASR不可知SE前端的新见解。摘要：Jointly training a speech enhancement (SE) front-end and an automatic speech recognition (ASR) back-end has been investigated as a way to mitigate the influence of \emph{processing distortion} generated by single-channel SE on ASR. In this paper, we investigate the effect of such joint training on the signal-level characteristics of the enhanced signals from the viewpoint of the decomposed noise and artifact errors. The experimental analyses provide two novel findings: 1) ASR-level training of the SE front-end reduces the artifact errors while increasing the noise errors, and 2) simply interpolating the enhanced and observed signals, which achieves a similar effect of reducing artifacts and increasing noise, improves ASR performance without jointly modifying the SE and ASR modules, even for a strong ASR back-end using a WavLM feature extractor. Our findings provide a better understanding of the effect of joint training and a novel insight for designing an ASR agnostic SE front-end.
【2】 Neural network-based virtual microphone estimation with virtual  microphone and beamformer-level multi-task loss标题：具有虚拟麦克风和波束形成器级多任务丢失的基于神经网络的虚拟麦克风估计链接：https://arxiv.org/abs/2311.11595
作者：Hanako Segawa,Tsubasa Ochiai,Marc Delcroix,Tomohiro Nakatani,Rintaro Ikeshita,Shoko Araki,Takeshi Yamada,Shoji Makino备注：5 pages, 2 figures, 1 table摘要：阵列处理性能取决于可用麦克风的数量。虚拟麦克风估计（VME）已经被提出来人为地增加麦克风信号的数量。基于神经网络的VME（NN-VME）训练具有VM级损失的NN，以预测在训练期间可用但在推断时不可用的麦克风位置处的信号。然而，该训练目标对于特定阵列处理后端（诸如波束成形）可能不是最佳的。另一种方法是使用考虑阵列处理后端的训练目标，例如波束形成器输出的损失。该方法可以生成对于波束成形最佳的信号，但不是物理接地的。为了结合这两种方法的优点，本文提出了一种多任务损失的NN-VME，结合VM级和波束形成器级的损失。我们在多说话者欠定条件下评估了所提出的多任务NN-VME，并表明与仅使用真实麦克风相比，它实现了33.1%的相对WER改进，与使用先前的NN-VME方法相比，它实现了10.8%的相对WER改进。摘要：Array processing performance depends on the number of microphones available. Virtual microphone estimation (VME) has been proposed to increase the number of microphone signals artificially. Neural network-based VME (NN-VME) trains an NN with a VM-level loss to predict a signal at a microphone location that is available during training but not at inference. However, this training objective may not be optimal for a specific array processing back-end, such as beamforming. An alternative approach is to use a training objective considering the array-processing back-end, such as a loss on the beamformer output. This approach may generate signals optimal for beamforming but not physically grounded. To combine the advantages of both approaches, this paper proposes a multi-task loss for NN-VME that combines both VM-level and beamformer-level losses. We evaluate the proposed multi-task NN-VME on multi-talker underdetermined conditions and show that it achieves a 33.1 % relative WER improvement compared to using only real microphones and 10.8 % compared to using a prior NN-VME approach.
【3】 APNet2: High-quality and High-efficiency Neural Vocoder with Direct  Prediction of Amplitude and Phase Spectra标题：APNet2：高质量、高效率的直接预测幅相谱的神经声码器链接：https://arxiv.org/abs/2311.11545
作者：Hui-Peng Du,Ye-Xin Lu,Yang Ai,Zhen-Hua Ling摘要：在我们以前的工作中，我们提出了一种称为APNet的神经声码器，它直接预测语音幅度和相位谱，从输入的声学特征并行5 ms的帧移，然后使用逆短时傅立叶变换（ISTFT）重建16 kHz的语音波形。APNet演示了生成与HiFi-GAN声码器质量相当的合成语音的能力，但推理速度大大提高。然而，APNet声码器的性能受到波形采样率和频谱帧移位的限制，限制了其用于高质量语音合成的实用性。因此，本文提出了一种改进的迭代APNet，命名为APNet 2。所提出的APNet 2声码器采用ConvNeXt v2作为幅度和相位预测的骨干网络，期望增强建模能力。此外，我们将多分辨率滤波器（MRD）引入到基于GAN的损失中，并优化某些损失的形式。在具有22.05 kHz的波形采样率和256个点的频谱帧移位（即，大约11.6ms），我们提出的APNet 2声码器在合成语音质量方面优于原始的APNet和Vocos声码器。APNet 2的合成语音质量也与HiFi-GAN和iSTFTNet相当，同时提供更快的推理速度。摘要：In our previous work, we proposed a neural vocoder called APNet, which directly predicts speech amplitude and phase spectra with a 5 ms frame shift in parallel from the input acoustic features, and then reconstructs the 16 kHz speech waveform using inverse short-time Fourier transform (ISTFT). APNet demonstrates the capability to generate synthesized speech of comparable quality to the HiFi-GAN vocoder but with a considerably improved inference speed. However, the performance of the APNet vocoder is constrained by the waveform sampling rate and spectral frame shift, limiting its practicality for high-quality speech synthesis. Therefore, this paper proposes an improved iteration of APNet, named APNet2. The proposed APNet2 vocoder adopts ConvNeXt v2 as the backbone network for amplitude and phase predictions, expecting to enhance the modeling capability. Additionally, we introduce a multi-resolution discriminator (MRD) into the GAN-based losses and optimize the form of certain losses. At a common configuration with a waveform sampling rate of 22.05 kHz and spectral frame shift of 256 points (i.e., approximately 11.6ms), our proposed APNet2 vocoder outperformed the original APNet and Vocos vocoders in terms of synthesized speech quality. The synthesized speech quality of APNet2 is also comparable to that of HiFi-GAN and iSTFTNet, while offering a significantly faster inference speed.
【4】 Label-Synchronous Neural Transducer for Adaptable Online E2E Speech  Recognition标题：用于自适应在线E2E语音识别的标签同步神经传感器链接：https://arxiv.org/abs/2311.11353
作者：Keqi Deng,Philip C. Woodland备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：虽然端到端（E2 E）自动语音识别（ASR）已经显示出最先进的识别精度，但它往往会隐含地偏向于训练数据分布，这可能会降低泛化能力。本文提出了一种标签同步神经传感器（LS-传感器），它提供了一个自然的方法，域适应的基础上，纯文本数据。LS传感器在将标签级编码器表示与预测网络输出组合之前提取标签级编码器表示。由于不再需要空白标记，预测网络可以作为标准语言模型，可以使用纯文本数据轻松进行调整。提出了一种自回归积分触发（AIF）机制来生成标签级编码器表示，同时保留可用于流式传输的低延迟操作。此外，流联合解码方法的设计，以提高ASR的准确性，同时保持与AIF同步。实验表明，与标准神经传感器相比，所提出的LS-传感器对于域内LibriSpeech数据给出了12.9%的相对WER降低（WRR），对于跨域TED-LIUM 2和AESRC 2020数据，具有自适应预测网络的相对WRR分别为21.4%和24.6%。摘要：Although end-to-end (E2E) automatic speech recognition (ASR) has shown state-of-the-art recognition accuracy, it tends to be implicitly biased towards the training data distribution which can degrade generalisation. This paper proposes a label-synchronous neural transducer (LS-Transducer), which provides a natural approach to domain adaptation based on text-only data. The LS-Transducer extracts a label-level encoder representation before combining it with the prediction network output. Since blank tokens are no longer needed, the prediction network performs as a standard language model, which can be easily adapted using text-only data. An Auto-regressive Integrate-and-Fire (AIF) mechanism is proposed to generate the label-level encoder representation while retaining low latency operation that can be used for streaming. In addition, a streaming joint decoding method is designed to improve ASR accuracy while retaining synchronisation with AIF. Experiments show that compared to standard neural transducers, the proposed LS-Transducer gave a 12.9% relative WER reduction (WERR) for intra-domain LibriSpeech data, as well as 21.4% and 24.6% relative WERRs on cross-domain TED-LIUM 2 and AESRC2020 data with an adapted prediction network.
【5】 MSPB: a longitudinal multi-sensor dataset with phenotypic trait  measurements from honey bees标题：MSPB：一个带有蜜蜂表型特征测量的纵向多传感器数据集链接：https://arxiv.org/abs/2311.10876
作者：Yi Zhu,Mahsa Abdollahi,Ségolène Maucourt,Nico Coallier,Heitor R. Guimarães,Pierre Giovenazzo,Tiago H. Falk备注：Under review; project webpage: this https URL摘要：我们提出了一个纵向的多传感器数据集收集蜜蜂的殖民地（意大利蜜蜂）与丰富的表型测量。在2020年5月至2021年4月期间，从位于加拿大魁北克省两个养蜂场的53个蜂箱中连续收集数据。传感器数据包括音频特征、温度和相对湿度。表型测量包括蜂箱数量、育雏细胞（卵、幼虫和蛹）数量、狄斯瓦螨侵染水平、防御和卫生行为、蜂蜜产量和冬季死亡率。我们的研究是第一个提供由养蜂科学专家注释的各种各样的表型性状测量的研究之一，这有助于更广泛的分析。我们首先总结了数据采集过程，传感器数据预处理步骤，和数据组成。然后，我们提供了表型数据分布的概述以及传感器数据模式的可视化。最后，我们展示了几个基于传感器数据分析和机器学习的蜂巢监测应用程序，例如冬季死亡率预测，蜂巢人口估计，以及活跃和产卵女王的存在。摘要：We present a longitudinal multi-sensor dataset collected from honey bee colonies (Apis mellifera) with rich phenotypic measurements. Data were continuously collected between May-2020 and April-2021 from 53 hives located at two apiaries in Qu\'ebec, Canada. The sensor data included audio features, temperature, and relative humidity. The phenotypic measurements contained beehive population, number of brood cells (eggs, larva and pupa), Varroa destructor infestation levels, defensive and hygienic behaviors, honey yield, and winter mortality. Our study is amongst the first to provide a wide variety of phenotypic trait measurements annotated by apicultural science experts, which facilitate a broader scope of analysis. We first summarize the data collection procedure, sensor data pre-processing steps, and data composition. We then provide an overview of the phenotypic data distribution as well as a visualization of the sensor data patterns. Lastly, we showcase several hive monitoring applications based on sensor data analysis and machine learning, such as winter mortality prediction, hive population estimation, and the presence of an active and laying queen.
【6】 Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis标题：用于语音合成的特定说话人潜在语音特征编码链接：https://arxiv.org/abs/2311.11745
作者：Jungil Kong,Junmo Lee,Jeongmin Kim,Beomjeong Kim,Jihoon Park,Dohee Kong,Changheon Lee,Sangjin Kim摘要：在这项工作中，我们提出了一种新的方法来建模众多的扬声器，它能够表达扬声器的整体特征的细节，就像一个训练有素的多扬声器模型，而无需对目标扬声器的数据集进行额外的训练。虽然具有类似目的的各种作品已被积极研究，但由于其基本局限性，其性能尚未达到训练的多说话人模型的性能。为了克服以前的局限性，我们提出了有效的方法进行特征学习和表示目标说话人的语音特征，通过离散化的功能和条件，他们的语音合成模型。我们的方法获得了显着更高的相似性平均意见得分（SMOS）在主观相似性评价比看到的扬声器的性能最好的多扬声器模型，即使与看不见的扬声器。所提出的方法也优于一个zero-shot方法显着的利润率。此外，我们的方法在生成新的人工扬声器方面表现出显着的性能。此外，我们证明了编码的潜在功能是足够的信息，以重建原始扬声器的语音完全。这意味着我们的方法可以作为一个通用的方法来编码和重建说话人的特点在各种任务。摘要：In this work, we propose a novel method for modeling numerous speakers, which enables expressing the overall characteristics of speakers in detail like a trained multi-speaker model without additional training on the target speaker's dataset. Although various works with similar purposes have been actively studied, their performance has not yet reached that of trained multi-speaker models due to their fundamental limitations. To overcome previous limitations, we propose effective methods for feature learning and representing target speakers' speech characteristics by discretizing the features and conditioning them to a speech synthesis model. Our method obtained a significantly higher similarity mean opinion score (SMOS) in subjective similarity evaluation than seen speakers of a best-performing multi-speaker model, even with unseen speakers. The proposed method also outperforms a zero-shot method by significant margins. Furthermore, our method shows remarkable performance in generating new artificial speakers. In addition, we demonstrate that the encoded latent features are sufficiently informative to reconstruct an original speaker's speech completely. It implies that our method can be used as a general methodology to encode and reconstruct speakers' characteristics in various tasks.
【7】 M$^{2}$UGen: Multi-modal Music Understanding and Generation with the  Power of Large Language Models标题：M$^{2}$Ugen：大型语言模型的力量下的多模式音乐理解和生成链接：https://arxiv.org/abs/2311.11255
作者：Atin Sakkeer Hussain,Shansong Liu,Chenshuo Sun,Ying Shan摘要：目前利用大型语言模型（LLM）的研究正在经历一场浪潮。许多作品利用这些模型强大的推理能力来理解各种形式，如文本，语音，图像，视频等，它们还利用LLM来理解人类的意图并生成所需的输出，如图像，视频和音乐。然而，使用LLM将理解和生成相结合的研究仍然有限，并且处于起步阶段。为了解决这个差距，我们引入了多模态音乐理解和生成（M$^{2}$UGen）框架，集成了LLM的能力，理解和生成不同形式的音乐。M$^{2}$UGen框架是专门构建的，旨在通过分别使用预训练的MERT、ViT和ViViT模型，从各种灵感来源（包括音乐、图像和视频）中释放创造潜力。为了实现音乐生成，我们探索了AudioLDM 2和MusicGen的使用。通过LLaMA 2模型的集成，实现了多模态理解和音乐生成的桥梁。此外，我们利用MU-LLaMA模型来生成支持文本/图像/视频到音乐生成的广泛数据集，从而促进了我们的M$^{2}$UGen框架的训练。我们对我们提出的框架进行了全面评估。实验结果表明，我们的模型达到或超过目前最先进的模型的性能。摘要：The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.
【8】 Is one brick enough to break the wall of spoken dialogue state tracking?标题：一块砖就足以打破口头对话状态跟踪的障碍吗？链接：https://arxiv.org/abs/2311.04923
作者：Lucas Druart,Valentin Vielzeuf,Yannick Estève备注：Submitted to IEEE ICASSP 2024{\copyright} 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：在面向任务的对话（TOD）系统中，正确地更新系统对用户需求的理解（即对话状态跟踪）是顺利交互的关键。传统上，TOD系统在三个步骤中执行此更新：用户话语的转录、关键概念的语义提取以及与先前识别的概念的上下文化。这种级联方法遭受级联误差和单独优化。端到端方法已被证明有助于语义提取步骤。本文通过比较三种方法进一步铺平了实现完全神经口语对话状态跟踪的道路：（1）最先进的级联方法，（2）基于规则的上下文化的局部E2E方法和（3）完全神经方法。我们的研究强调，尽管它们都优于最近的DSTC 11最佳模型，特别是使用滤波后处理步骤，但（1）仍然是最准确的方法。事实上，（2）和（3）在对话展开时都难以传播上下文，这表明完全神经方法中的上下文传播是一个开放的挑战。摘要：In Task-Oriented Dialogue (TOD) systems, correctly updating the system's understanding of the user's needs (a.k.a dialogue state tracking) is key to a smooth interaction. Traditionally, TOD systems perform this update in three steps: transcription of the user's utterance, semantic extraction of the key concepts, and contextualization with the previously identified concepts. Such cascade approaches suffer from cascading errors and separate optimization. End-to-End approaches have been proved helpful up to the semantic extraction step. This paper goes one step further paving the path towards completely neural spoken dialogue state tracking by comparing three approaches: (1) a state of the art cascade approach, (2) a locally E2E approach with rule-based contextualization and (3) a completely neural approach. Our study highlights that although they all outperform the recent DSTC11 best model, especially with a filtering post-processing step, (1) remains the most accurate approach. Indeed, both (2) and (3) have trouble propagating context as dialogues unfold showing that context propagation in completely neural approaches is an open challenge.
【9】 Are cascade dialogue state tracking models speaking out of turn in  spoken dialogues?标题：级联对话状态跟踪模型是否在口头对话中不按顺序发言？链接：https://arxiv.org/abs/2311.04922
作者：Lucas Druart,Léo Jacqmin,Benoît Favre,Lina Maria Rojas-Barahona,Valentin Vielzeuf备注：Submitted to IEEE ICASSP 2024{\copyright} 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：在面向任务的对话（TOD）系统中，正确地更新系统对用户需求的理解是顺利交互的关键。传统上，TOD系统由几个相互交互的模块组成。虽然这些组件中的每一个都是活跃的研究社区的焦点，但它们在交互中的行为可能会被忽视。本文提出了一个全面的分析，如对话状态跟踪高度依赖于对话上下文的复杂设置的最先进的系统的错误。基于口语MultiWoz，我们确定，非分类槽的值上的错误是必不可少的，以解决口语和基于聊天的对话系统之间的差距。我们探索潜在的解决方案，以提高transmittance和帮助对话状态跟踪生成模型纠正这些错误。摘要：In Task-Oriented Dialogue (TOD) systems, correctly updating the system's understanding of the user's needs is key to a smooth interaction. Traditionally TOD systems are composed of several modules that interact with one another. While each of these components is the focus of active research communities, their behavior in interaction can be overlooked. This paper proposes a comprehensive analysis of the errors of state of the art systems in complex settings such as Dialogue State Tracking which highly depends on the dialogue context. Based on spoken MultiWoz, we identify that errors on non-categorical slots' values are essential to address in order to bridge the gap between spoken and chat-based dialogue systems. We explore potential solutions to improve transcriptions and help dialogue state tracking generative models correct such errors.
机器翻译，仅供参考