今日论文合集：cs.SD语音5篇，eess.AS音频处理4篇。
cs.SD语音
【1】 Improving Joint Speech-Text Representations Without Alignment标题：改进无对齐的语音-文本联合表示链接：https://arxiv.org/abs/2308.06125
作者：Cal Peyser,Zhong Meng,Ke Hu,Rohit Prabhavalkar,Andrew Rosenberg,Tara N. Sainath,Michael Picheny,Kyunghyun Cho备注：None摘要：在过去的一年里，在文本提示的图像生成的前提下，一个跨模态的表示空间，其中的文本和图像域表示联合的想法取得了惊人的进展。在ASR中，这个想法已经被应用为联合语音-文本编码器，其可以通过在未配对的语音和文本上进行训练来扩展到非常大的参数模型的容量。虽然这些方法显示出希望，但它们需要对语音和文本中固有的序列长度不匹配进行特殊处理，无论是通过上采样启发法还是显式对齐模型。在这项工作中，我们提供的证据表明，联合语音文本编码器自然地实现一致的表示方式，无视序列长度，并认为一致性损失可以原谅长度差异，并简单地假设最佳对齐。我们发现，这样的损失提高了下游WER在一个大参数单语和多语言系统。摘要：The last year has seen astonishing progress in text-prompted image generation premised on the idea of a cross-modal representation space in which the text and image domains are represented jointly. In ASR, this idea has found application as joint speech-text encoders that can scale to the capacities of very large parameter models by being trained on both unpaired speech and text. While these methods show promise, they have required special treatment of the sequence-length mismatch inherent in speech and text, either by up-sampling heuristics or an explicit alignment model. In this work, we offer evidence that joint speech-text encoders naturally achieve consistent representations across modalities by disregarding sequence length, and argue that consistency losses could forgive length differences and simply assume the best alignment. We show that such a loss improves downstream WER in both a large-parameter monolingual and multilingual system.
【2】 Lip2Vec: Efficient and Robust Visual Speech Recognition via  Latent-to-Latent Visual to Audio Representation Mapping标题：Lip2Vec：基于潜在视听表征映射的高效稳健视觉语音识别链接：https://arxiv.org/abs/2308.06112
作者：Yasser Abdelaziz Dahou Djilali,Sanath Narayan,Haithem Boussaid,Ebtessam Almazrouei,Merouane Debbah摘要：视觉语音识别（VSR）不同于常见的感知任务，因为它需要对视频序列进行更深入的推理，即使是人类专家。尽管最近在VSR方面取得了进展，但当前的方法依赖于标记数据来完全训练或微调其预测目标语音的模型。这阻碍了它们在训练集之外进行泛化的能力，并导致在分布外的挑战性场景下性能退化。与以前的作品，涉及辅助损失或复杂的训练程序和架构，我们提出了一个简单的方法，名为Lip2Vec，是基于学习先验模型。给定一个鲁棒的视觉语音编码器，该网络将唇序列的编码的潜在表示映射到来自音频对的其对应的潜在表示，其对于有效的文本解码是足够不变的。然后使用现成的音频语音识别（ASR）模型将生成的音频表示解码为文本。该模型与LRS3数据集上实现26 WER的全监督学习方法相比毫不逊色。与SoTA方法不同，我们的模型在VoxCeleb测试集上保持了合理的性能。我们认为，重新编程的VSR作为ASR任务缩小了两者之间的性能差距，并铺平了道路，更灵活的配方唇读。摘要：Visual Speech Recognition (VSR) differs from the common perception tasks as it requires deeper reasoning over the video sequence, even by human experts. Despite the recent advances in VSR, current approaches rely on labeled data to fully train or finetune their models predicting the target speech. This hinders their ability to generalize well beyond the training set and leads to performance degeneration under out-of-distribution challenging scenarios. Unlike previous works that involve auxiliary losses or complex training procedures and architectures, we propose a simple approach, named Lip2Vec that is based on learning a prior model. Given a robust visual speech encoder, this network maps the encoded latent representations of the lip sequence to their corresponding latents from the audio pair, which are sufficiently invariant for effective text decoding. The generated audio representation is then decoded to text using an off-the-shelf Audio Speech Recognition (ASR) model. The proposed model compares favorably with fully-supervised learning methods on the LRS3 dataset achieving 26 WER. Unlike SoTA approaches, our model keeps a reasonable performance on the VoxCeleb test set. We believe that reprogramming the VSR as an ASR task narrows the performance gap between the two and paves the way for more flexible formulations of lip reading.
【3】 An Autoethnographic Exploration of XAI in Algorithmic Composition标题：《算术作文》中的夏民族志探微链接：https://arxiv.org/abs/2308.06089
作者：Ashley Noel-Hirst,Nick Bryan-Kinns摘要：机器学习模型能够生成从民间音乐到古典音乐的各种类型的复杂音乐。然而，当前的生成音乐AI模型通常难以理解和以有意义的方式控制。虽然研究已经开始探索如何为音乐创建可解释的AI（XAI）生成模型，但在音乐制作实践中还没有研究过生成的XAI模型。本文介绍了一个autoethnographic研究的MeasureVAE生成音乐XAI模型与解释的潜在维度训练爱尔兰民间音乐的使用。研究结果表明，音乐制作工作流程的探索性突出了训练数据集的音乐特征，而不是生成模型本身的特征。在迭代工作流中占用XAI模型突出了XAI模型形成比它们最初设计的更丰富和更复杂的工作流的一部分的潜力。摘要：Machine Learning models are capable of generating complex music across a range of genres from folk to classical music. However, current generative music AI models are typically difficult to understand and control in meaningful ways. Whilst research has started to explore how explainable AI (XAI) generative models might be created for music, no generative XAI models have been studied in music making practice. This paper introduces an autoethnographic study of the use of the MeasureVAE generative music XAI model with interpretable latent dimensions trained on Irish folk music. Findings suggest that the exploratory nature of the music-making workflow foregrounds musical features of the training dataset rather than features of the generative model itself. The appropriation of an XAI model within an iterative workflow highlights the potential of XAI models to form part of a richer and more complex workflow than they were initially designed for.
【4】 Audio is all in one: speech-driven gesture synthetics using WavLM  pre-trained model标题：音频一体化：使用WavLM预训练模型的语音驱动手势合成链接：https://arxiv.org/abs/2308.05995
作者：Fan Zhang,Naye Ji,Fuxing Gao,Siyuan Zhao,Zhaohan Wang,Shunman Li备注：10 pages, 5 figures, 1 table摘要：用于数字人的协同语音手势的生成是虚拟人创建领域中的新兴领域。先前的研究已经取得了进展，通过使用声学和语义信息作为输入，并采用分类方法来识别人的ID和情感，以驱动协同语音手势生成。然而，这一努力仍然面临重大挑战。这些挑战超越了协同语音手势、语音声学和语义之间错综复杂的相互作用;它们还包括与个性、情感和其他模糊但重要的因素有关的复杂性。本文介绍了“diffmotion-v2”，这是一种基于语音条件扩散和基于非自回归变换的生成模型，具有WavLM预训练模型。它可以仅使用原始语音音频来产生个人和风格化的全身协同语音手势，从而消除了对复杂的多模态处理和手动注释的需要。首先，考虑到语音音频不仅包含声学和语义特征，而且还传达了个性特征，情感以及与伴随手势相关的更微妙的信息，我们开创性地采用WavLM（一种大规模预训练模型）来提取低级和高级音频信息。其次，我们在基于变换器的层中引入自适应层规范架构来学习语音信息和伴随的手势之间的关系。在Trinity、ZEGGS和BEAT数据集上进行了广泛的主观评价实验，以确认WavLM和模型合成具有各种风格的自然协同语音手势的能力。摘要：The generation of co-speech gestures for digital humans is an emerging area in the field of virtual human creation. Prior research has made progress by using acoustic and semantic information as input and adopting classify method to identify the person's ID and emotion for driving co-speech gesture generation. However, this endeavour still faces significant challenges. These challenges go beyond the intricate interplay between co-speech gestures, speech acoustic, and semantics; they also encompass the complexities associated with personality, emotion, and other obscure but important factors. This paper introduces "diffmotion-v2," a speech-conditional diffusion-based and non-autoregressive transformer-based generative model with WavLM pre-trained model. It can produce individual and stylized full-body co-speech gestures only using raw speech audio, eliminating the need for complex multimodal processing and manually annotated. Firstly, considering that speech audio not only contains acoustic and semantic features but also conveys personality traits, emotions, and more subtle information related to accompanying gestures, we pioneer the adaptation of WavLM, a large-scale pre-trained model, to extract low-level and high-level audio information. Secondly, we introduce an adaptive layer norm architecture in the transformer-based layer to learn the relationship between speech information and accompanying gestures. Extensive subjective evaluation experiments are conducted on the Trinity, ZEGGS, and BEAT datasets to confirm the WavLM and the model's ability to synthesize natural co-speech gestures with various styles.
【5】 Advancing the study of Large-Scale Learning in Overlapped Speech  Detection标题：推进大规模学习在重叠语音检测中的研究链接：https://arxiv.org/abs/2308.05987
作者：Zhaohui Yin,Jingguang Tian,Xinhui Hu,Xinkang Xu摘要：重叠语音检测（OSD）是涉及多方会话分析的语音应用的重要组成部分。然而，现有的OSD模型大多是在特定的数据集上训练和评估的，这限制了这些模型的应用场景。为了解决这个问题，我们进行了大规模学习（LSL）的OSD研究，并提出了一个更通用的16 K单通道OSD模型。在我们的研究中，522小时的不同语言和风格的标记音频被收集和使用作为大规模数据集。设计了严格的对比实验，并用于评估LSL在OSD任务中的有效性和基于不同深度神经网络的OSD模型的性能。结果表明，LSL能显著提高OSD模型的性能和鲁棒性，基于LSL的Conformer OSD模型（CF-OSD）是目前性能最好的16 K单通道OSD模型。此外，具有LSL的CF-OSD建立了最先进的性能，Aliemeeting测试集和DIHARD II评估集的F1评分分别为80.8%和52.0%。摘要：Overlapped Speech Detection (OSD) is an important part of speech applications involving analysis of multi-party conversations. However, Most of the existing OSD models are trained and evaluated on specific dataset, which limits the application scenarios of these models. In order to solve this problem, we conduct a study of large-scale learning (LSL) in OSD and propose a more general 16K single-channel OSD model. In our study, 522 hours of labeled audio in different languages and styles are collected and used as the large-scale dataset. Rigorous comparative experiments are designed and used to evaluate the effectiveness of LSL in OSD task and the performance of OSD models based on different deep neural networks. The results show that LSL can significantly improve the performance and robustness of OSD models, and the OSD model based on Conformer (CF-OSD) with LSL is currently the best 16K single-channel OSD model. Moreover, the CF-OSD with LSL establishes a state-of-the-art performance with a F1-score of 80.8% and 52.0% on the Alimeeting test set and DIHARD II evaluation set, respectively.
eess.AS音频处理
【1】 Improving Joint Speech-Text Representations Without Alignment标题：改进无对齐的语音-文本联合表示链接：https://arxiv.org/abs/2308.06125
作者：Cal Peyser,Zhong Meng,Ke Hu,Rohit Prabhavalkar,Andrew Rosenberg,Tara N. Sainath,Michael Picheny,Kyunghyun Cho备注：None摘要：在过去的一年里，在文本提示的图像生成的前提下，一个跨模态的表示空间，其中的文本和图像域表示联合的想法取得了惊人的进展。在ASR中，这个想法已经被应用为联合语音-文本编码器，其可以通过在未配对的语音和文本上进行训练来扩展到非常大的参数模型的容量。虽然这些方法显示出希望，但它们需要对语音和文本中固有的序列长度不匹配进行特殊处理，无论是通过上采样启发法还是显式对齐模型。在这项工作中，我们提供的证据表明，联合语音文本编码器自然地实现一致的表示方式，无视序列长度，并认为一致性损失可以原谅长度差异，并简单地假设最佳对齐。我们发现，这样的损失提高了下游WER在一个大参数单语和多语言系统。摘要：The last year has seen astonishing progress in text-prompted image generation premised on the idea of a cross-modal representation space in which the text and image domains are represented jointly. In ASR, this idea has found application as joint speech-text encoders that can scale to the capacities of very large parameter models by being trained on both unpaired speech and text. While these methods show promise, they have required special treatment of the sequence-length mismatch inherent in speech and text, either by up-sampling heuristics or an explicit alignment model. In this work, we offer evidence that joint speech-text encoders naturally achieve consistent representations across modalities by disregarding sequence length, and argue that consistency losses could forgive length differences and simply assume the best alignment. We show that such a loss improves downstream WER in both a large-parameter monolingual and multilingual system.
【2】 Lip2Vec: Efficient and Robust Visual Speech Recognition via  Latent-to-Latent Visual to Audio Representation Mapping标题：Lip2Vec：基于潜在视听表征映射的高效稳健视觉语音识别链接：https://arxiv.org/abs/2308.06112
作者：Yasser Abdelaziz Dahou Djilali,Sanath Narayan,Haithem Boussaid,Ebtessam Almazrouei,Merouane Debbah摘要：视觉语音识别（VSR）不同于常见的感知任务，因为它需要对视频序列进行更深入的推理，即使是人类专家。尽管最近在VSR方面取得了进展，但当前的方法依赖于标记数据来完全训练或微调其预测目标语音的模型。这阻碍了它们在训练集之外进行泛化的能力，并导致在分布外的挑战性场景下性能退化。与以前的作品，涉及辅助损失或复杂的训练程序和架构，我们提出了一个简单的方法，名为Lip2Vec，是基于学习先验模型。给定一个鲁棒的视觉语音编码器，该网络将唇序列的编码的潜在表示映射到来自音频对的其对应的潜在表示，其对于有效的文本解码是足够不变的。然后使用现成的音频语音识别（ASR）模型将生成的音频表示解码为文本。该模型与LRS3数据集上实现26 WER的全监督学习方法相比毫不逊色。与SoTA方法不同，我们的模型在VoxCeleb测试集上保持了合理的性能。我们认为，重新编程的VSR作为ASR任务缩小了两者之间的性能差距，并铺平了道路，更灵活的配方唇读。摘要：Visual Speech Recognition (VSR) differs from the common perception tasks as it requires deeper reasoning over the video sequence, even by human experts. Despite the recent advances in VSR, current approaches rely on labeled data to fully train or finetune their models predicting the target speech. This hinders their ability to generalize well beyond the training set and leads to performance degeneration under out-of-distribution challenging scenarios. Unlike previous works that involve auxiliary losses or complex training procedures and architectures, we propose a simple approach, named Lip2Vec that is based on learning a prior model. Given a robust visual speech encoder, this network maps the encoded latent representations of the lip sequence to their corresponding latents from the audio pair, which are sufficiently invariant for effective text decoding. The generated audio representation is then decoded to text using an off-the-shelf Audio Speech Recognition (ASR) model. The proposed model compares favorably with fully-supervised learning methods on the LRS3 dataset achieving 26 WER. Unlike SoTA approaches, our model keeps a reasonable performance on the VoxCeleb test set. We believe that reprogramming the VSR as an ASR task narrows the performance gap between the two and paves the way for more flexible formulations of lip reading.
【3】 Audio is all in one: speech-driven gesture synthetics using WavLM  pre-trained model标题：音频一体化：使用WavLM预训练模型的语音驱动手势合成链接：https://arxiv.org/abs/2308.05995
作者：Fan Zhang,Naye Ji,Fuxing Gao,Siyuan Zhao,Zhaohan Wang,Shunman Li备注：10 pages, 5 figures, 1 table摘要：用于数字人的协同语音手势的生成是虚拟人创建领域中的新兴领域。先前的研究已经取得了进展，通过使用声学和语义信息作为输入，并采用分类方法来识别人的ID和情感，以驱动协同语音手势生成。然而，这一努力仍然面临重大挑战。这些挑战超越了协同语音手势、语音声学和语义之间错综复杂的相互作用;它们还包括与个性、情感和其他模糊但重要的因素有关的复杂性。本文介绍了“diffmotion-v2”，这是一种基于语音条件扩散和基于非自回归变换的生成模型，具有WavLM预训练模型。它可以仅使用原始语音音频来产生个人和风格化的全身协同语音手势，从而消除了对复杂的多模态处理和手动注释的需要。首先，考虑到语音音频不仅包含声学和语义特征，而且还传达了个性特征，情感以及与伴随手势相关的更微妙的信息，我们开创性地采用WavLM（一种大规模预训练模型）来提取低级和高级音频信息。其次，我们在基于变换器的层中引入自适应层规范架构来学习语音信息和伴随的手势之间的关系。在Trinity、ZEGGS和BEAT数据集上进行了广泛的主观评价实验，以确认WavLM和模型合成具有各种风格的自然协同语音手势的能力。摘要：The generation of co-speech gestures for digital humans is an emerging area in the field of virtual human creation. Prior research has made progress by using acoustic and semantic information as input and adopting classify method to identify the person's ID and emotion for driving co-speech gesture generation. However, this endeavour still faces significant challenges. These challenges go beyond the intricate interplay between co-speech gestures, speech acoustic, and semantics; they also encompass the complexities associated with personality, emotion, and other obscure but important factors. This paper introduces "diffmotion-v2," a speech-conditional diffusion-based and non-autoregressive transformer-based generative model with WavLM pre-trained model. It can produce individual and stylized full-body co-speech gestures only using raw speech audio, eliminating the need for complex multimodal processing and manually annotated. Firstly, considering that speech audio not only contains acoustic and semantic features but also conveys personality traits, emotions, and more subtle information related to accompanying gestures, we pioneer the adaptation of WavLM, a large-scale pre-trained model, to extract low-level and high-level audio information. Secondly, we introduce an adaptive layer norm architecture in the transformer-based layer to learn the relationship between speech information and accompanying gestures. Extensive subjective evaluation experiments are conducted on the Trinity, ZEGGS, and BEAT datasets to confirm the WavLM and the model's ability to synthesize natural co-speech gestures with various styles.
【4】 Advancing the study of Large-Scale Learning in Overlapped Speech  Detection标题：推进大规模学习在重叠语音检测中的研究链接：https://arxiv.org/abs/2308.05987
作者：Zhaohui Yin,Jingguang Tian,Xinhui Hu,Xinkang Xu摘要：重叠语音检测（OSD）是涉及多方会话分析的语音应用的重要组成部分。然而，现有的OSD模型大多是在特定的数据集上训练和评估的，这限制了这些模型的应用场景。为了解决这个问题，我们进行了大规模学习（LSL）的OSD研究，并提出了一个更通用的16 K单通道OSD模型。在我们的研究中，522小时的不同语言和风格的标记音频被收集和使用作为大规模数据集。设计了严格的对比实验，并用于评估LSL在OSD任务中的有效性和基于不同深度神经网络的OSD模型的性能。结果表明，LSL能显著提高OSD模型的性能和鲁棒性，基于LSL的Conformer OSD模型（CF-OSD）是目前性能最好的16 K单通道OSD模型。此外，具有LSL的CF-OSD建立了最先进的性能，Aliemeeting测试集和DIHARD II评估集的F1评分分别为80.8%和52.0%。摘要：Overlapped Speech Detection (OSD) is an important part of speech applications involving analysis of multi-party conversations. However, Most of the existing OSD models are trained and evaluated on specific dataset, which limits the application scenarios of these models. In order to solve this problem, we conduct a study of large-scale learning (LSL) in OSD and propose a more general 16K single-channel OSD model. In our study, 522 hours of labeled audio in different languages and styles are collected and used as the large-scale dataset. Rigorous comparative experiments are designed and used to evaluate the effectiveness of LSL in OSD task and the performance of OSD models based on different deep neural networks. The results show that LSL can significantly improve the performance and robustness of OSD models, and the OSD model based on Conformer (CF-OSD) with LSL is currently the best 16K single-channel OSD model. Moreover, the CF-OSD with LSL establishes a state-of-the-art performance with a F1-score of 80.8% and 52.0% on the Alimeeting test set and DIHARD II evaluation set, respectively.
机器翻译，仅供参考