今日论文合集：cs.SD语音2篇，eess.AS音频处理2篇。
cs.SD语音

【1】XAI-based Comparison of Input Representations for Audio Event Classification标题：基于XAI的音频事件分类输入表示法比较链接：http://arxiv.org/pdf/2304.14019v1

作者：Annika Frommholz,Fabian Seipel,Sebastian Lapuschkin,Wojciech Samek,Johanna Vielhaben机构：Fraunhofer Heinrich-Hertz Institute, Berlin, Germany, Technische Universität Berlin &, BIFOLD - Berlin Institute for the, Foundations of Learning and Data备注：7 pages, 4 figures摘要：深度神经网络是音频事件分类的一种很有前途的工具。与自然图像等其他数据相比，音频数据有许多合理和不明显的表示，可以作为这些模型的输入。由于它们的黑盒性质，不同的输入表示的效果到目前为止主要是通过测量分类性能来研究的。在这项工作中，我们利用可解释AI（XAI）来理解在不同输入表示上训练的模型的底层分类策略。具体来说，我们比较了两种模型架构，它们与用于音频事件检测的相关输入特征有关：一个直接处理信号作为原始波形，而另一个接收其时间-频率频谱图表示。我们展示了如何通过“警笛”{分层相关传播}获得的相关热图揭示代表依赖的决策策略。有了这些见解，我们可以在鲁棒性和代表性方面做出关于最佳输入表示的明智决定，并确认模型的分类策略符合人类需求。摘要：Deep neural networks are a promising tool for Audio Event Classification. In contrast to other data like natural images, there are many sensible and non-obvious representations for audio data, which could serve as input to these models. Due to their black-box nature, the effect of different input representations has so far mostly been investigated by measuring classification performance. In this work, we leverage eXplainable AI (XAI), to understand the underlying classification strategies of models trained on different input representations. Specifically, we compare two model architectures with regard to relevant input features used for Audio Event Detection: one directly processes the signal as the raw waveform, and the other takes in its time-frequency spectrogram representation. We show how relevance heatmaps obtained via "Siren"{Layer-wise Relevance Propagation} uncover representation-dependent decision strategies. With these insights, we can make a well-informed decision about the best input representation in terms of robustness and representativity and confirm that the model's classification strategies align with human requirements.

【2】Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model标题：基于指令调谐LLM和潜在扩散模型的文本-音频转换链接：http://arxiv.org/pdf/2304.13731v1

作者：Deepanway Ghosal,Navonil Majumder,Ambuj Mehrish,Soujanya Poria机构：‡ DeCLaRe Lab, Singapore University of Technology and Design, Singapore备注：this https URL摘要：最近的大型语言模型（LLM）的巨大规模允许许多有趣的属性，例如基于指令和思想链的微调，这显着提高了许多自然语言处理（NLP）任务中的零次和Few-Shot性能。受这些成功的启发，我们采用了这样一个指令调谐的LLM Flan-T5作为文本到音频（TTA）生成的文本编码器-目标是从其文本描述生成音频的任务。现有技术在TTA上的工作要么预先训练联合文本音频编码器，要么使用非指令调谐模型，诸如T5。因此，我们的基于潜在扩散模型（LDM）的方法TANGO在大多数指标上都优于最先进的AudioLDM，并且在AudioCaps测试集的其余部分上保持可比性，尽管在63倍小的数据集上训练LDM并保持文本编码器冻结。这种改进也可能归因于采用基于音频压力水平的声音混合用于训练集增强，而现有方法采用随机混合。摘要：The immense scale of the recent large language models (LLM) allows many interesting properties, such as, instruction- and chain-of-thought-based fine-tuning, that has significantly improved zero- and few-shot performance in many natural language processing (NLP) tasks. Inspired by such successes, we adopt such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio (TTA) generation -- a task where the goal is to generate an audio from its textual description. The prior works on TTA either pre-trained a joint text-audio encoder or used a non-instruction-tuned model, such as, T5. Consequently, our latent diffusion model (LDM)-based approach TANGO outperforms the state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set, despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. This improvement might also be attributed to the adoption of audio pressure level-based sound mixing for training set augmentation, whereas the prior methods take a random mix.
eess.AS音频处理

【1】XAI-based Comparison of Input Representations for Audio Event Classification标题：基于XAI的音频事件分类输入表示法比较链接：http://arxiv.org/pdf/2304.14019v1

*与cs.SD语音【1】为同一篇

作者：Annika Frommholz,Fabian Seipel,Sebastian Lapuschkin,Wojciech Samek,Johanna Vielhaben机构：Fraunhofer Heinrich-Hertz Institute, Berlin, Germany, Technische Universität Berlin &, BIFOLD - Berlin Institute for the, Foundations of Learning and Data备注：7 pages, 4 figures摘要：深度神经网络是音频事件分类的一种很有前途的工具。与自然图像等其他数据相比，音频数据有许多合理和不明显的表示，可以作为这些模型的输入。由于它们的黑盒性质，不同的输入表示的效果到目前为止主要是通过测量分类性能来研究的。在这项工作中，我们利用可解释AI（XAI）来理解在不同输入表示上训练的模型的底层分类策略。具体来说，我们比较了两种模型架构，它们与用于音频事件检测的相关输入特征有关：一个直接处理信号作为原始波形，而另一个接收其时间-频率频谱图表示。我们展示了如何通过“警笛”{分层相关传播}获得的相关热图揭示代表依赖的决策策略。有了这些见解，我们可以在鲁棒性和代表性方面做出关于最佳输入表示的明智决定，并确认模型的分类策略符合人类需求。摘要：Deep neural networks are a promising tool for Audio Event Classification. In contrast to other data like natural images, there are many sensible and non-obvious representations for audio data, which could serve as input to these models. Due to their black-box nature, the effect of different input representations has so far mostly been investigated by measuring classification performance. In this work, we leverage eXplainable AI (XAI), to understand the underlying classification strategies of models trained on different input representations. Specifically, we compare two model architectures with regard to relevant input features used for Audio Event Detection: one directly processes the signal as the raw waveform, and the other takes in its time-frequency spectrogram representation. We show how relevance heatmaps obtained via "Siren"{Layer-wise Relevance Propagation} uncover representation-dependent decision strategies. With these insights, we can make a well-informed decision about the best input representation in terms of robustness and representativity and confirm that the model's classification strategies align with human requirements.

【2】Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model标题：基于指令调谐LLM和潜在扩散模型的文本-音频转换链接：http://arxiv.org/pdf/2304.13731v1

*与cs.SD语音【2】为同一篇

作者：Deepanway Ghosal,Navonil Majumder,Ambuj Mehrish,Soujanya Poria机构：‡ DeCLaRe Lab, Singapore University of Technology and Design, Singapore备注：this https URL摘要：最近的大型语言模型（LLM）的巨大规模允许许多有趣的属性，例如基于指令和思想链的微调，这显着提高了许多自然语言处理（NLP）任务中的零次和Few-Shot性能。受这些成功的启发，我们采用了这样一个指令调谐的LLM Flan-T5作为文本到音频（TTA）生成的文本编码器-目标是从其文本描述生成音频的任务。现有技术在TTA上的工作要么预先训练联合文本音频编码器，要么使用非指令调谐模型，诸如T5。因此，我们的基于潜在扩散模型（LDM）的方法TANGO在大多数指标上都优于最先进的AudioLDM，并且在AudioCaps测试集的其余部分上保持可比性，尽管在63倍小的数据集上训练LDM并保持文本编码器冻结。这种改进也可能归因于采用基于音频压力水平的声音混合用于训练集增强，而现有方法采用随机混合。摘要：The immense scale of the recent large language models (LLM) allows many interesting properties, such as, instruction- and chain-of-thought-based fine-tuning, that has significantly improved zero- and few-shot performance in many natural language processing (NLP) tasks. Inspired by such successes, we adopt such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio (TTA) generation -- a task where the goal is to generate an audio from its textual description. The prior works on TTA either pre-trained a joint text-audio encoder or used a non-instruction-tuned model, such as, T5. Consequently, our latent diffusion model (LDM)-based approach TANGO outperforms the state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set, despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. This improvement might also be attributed to the adoption of audio pressure level-based sound mixing for training set augmentation, whereas the prior methods take a random mix.

机器翻译，仅供参考