今日论文合集：cs.SD语音8篇，eess.AS音频处理9篇。
cs.SD语音
【1】 A Multimodal Prototypical Approach for Unsupervised Sound Classification标题：一种无监督声音分类的多模式原型方法链接：https://arxiv.org/abs/2306.12300
作者：Saksham Singh Kushwaha,Magdalena Fuentes机构：Courant Institute of Mathematical Sciences, New York University, NY, USA, MARL, New York University, NY, USA, IDM, New York University, NY, USA备注：Accepted to INTERSPEECH 2023摘要：在环境声音分类方面，系统的适应性是关键：哪些声音类是感兴趣的取决于上下文和用户的需要。文本到音频检索的最新进展允许zero-shot音频分类，但与监督模型相比，性能仍然有限。这项工作提出了一种多模态的原型方法，利用本地音频文本嵌入音频查询提供更相关的答案，增强了适应性的声音检测在野外。我们首先使用文本查询附近的音频嵌入社区，最好地表征每个查询声音，并选择该组的质心作为我们的原型。其次，我们将看不见的音频与这些原型进行比较以进行分类。我们进行了多次消融研究，以了解嵌入模型和提示的影响。我们的无监督方法在三个声音识别基准测试中的zero-shot最先进水平平均提高了12%。摘要：In the context of environmental sound classification, the adaptability of systems is key: which sound classes are interesting depends on the context and the user's needs. Recent advances in text-to-audio retrieval allow for zero-shot audio classification, but performance compared to supervised models remains limited. This work proposes a multimodal prototypical approach that exploits local audio-text embeddings to provide more relevant answers to audio queries, augmenting the adaptability of sound detection in the wild. We do this by first using text to query a nearby community of audio embeddings that best characterize each query sound, and select the group's centroids as our prototypes. Second, we compare unseen audio to these prototypes for classification. We perform multiple ablation studies to understand the impact of the embedding models and prompts. Our unsupervised approach improves upon the zero-shot state-of-the-art in three sound recognition benchmarks by an average of 12%.
【2】 Automatic Speech Disentanglement for Voice Conversion using Rank Module  and Speech Augmentation标题：利用秩模和语音增强实现语音转换的语音解缠链接：https://arxiv.org/abs/2306.12259
作者：Zhonghua Liu,Shijun Wang,Ning Chen机构：East China University of Science and Technology, Shanghai, China, University of St.Gallen, St.Gallen, Switzerland备注：Accepted by INTERSPEECH2023摘要：语音转换（VC）将源语音的语音转换为目标语音，同时保持源语音的内容。语音主要可以分解为四个部分：内容、音色、节奏和音高。不幸的是，大多数相关作品只考虑内容和音色，这导致语音不太自然。最近的一些作品能够解开语音成几个组件，但他们需要费力的瓶颈调谐或各种手工制作的功能，每个假设包含解开的语音信息。在本文中，我们提出了一个VC模型，可以自动解开语音成四个组件，只使用两个增强功能，而无需多个手工制作的功能或费力的瓶颈调整。该模型是简单而有效的，实证结果表明，我们的模型可以实现更好的性能比基线，关于解纠缠的有效性和语音自然度。摘要：Voice Conversion (VC) converts the voice of a source speech to that of a target while maintaining the source's content. Speech can be mainly decomposed into four components: content, timbre, rhythm and pitch. Unfortunately, most related works only take into account content and timbre, which results in less natural speech. Some recent works are able to disentangle speech into several components, but they require laborious bottleneck tuning or various hand-crafted features, each assumed to contain disentangled speech information. In this paper, we propose a VC model that can automatically disentangle speech into four components using only two augmentation functions, without the requirement of multiple hand-crafted features or laborious bottleneck tuning. The proposed model is straightforward yet efficient, and the empirical results demonstrate that our model can achieve a better performance than the baseline, regarding disentanglement effectiveness and speech naturalness.
【3】 Knowledge-based Multimodal Music Similarity标题：基于知识的多通道音乐相似度研究链接：https://arxiv.org/abs/2306.12249
作者：Andrea Poltronieri机构：Department of Computer Science and Engineering, University of Bologna, Italy备注：11 pages, 1 figure摘要：音乐相似性是音乐检索、推荐系统和音乐分析的一个重要方面。此外，相似性对音乐专家来说是至关重要的，因为它允许研究作曲家和历史时期之间的类比和影响。当前的音乐相似性的方法主要依赖于符号内容，其生产成本可能是昂贵的，并且不总是现成的。相反，使用音频信号的方法通常不能提供关于观察到的相似性背后的原因的任何洞察。本研究解决了目前的方法的局限性，重点研究音乐的相似性，使用符号和音频内容。本研究的目的是开发一个完全可解释和解释的系统，可以提供终端用户更多的控制和理解音乐相似性和分类系统。摘要：Music similarity is an essential aspect of music retrieval, recommendation systems, and music analysis. Moreover, similarity is of vital interest for music experts, as it allows studying analogies and influences among composers and historical periods. Current approaches to musical similarity rely mainly on symbolic content, which can be expensive to produce and is not always readily available. Conversely, approaches using audio signals typically fail to provide any insight about the reasons behind the observed similarity. This research addresses the limitations of current approaches by focusing on the study of musical similarity using both symbolic and audio content. The aim of this research is to develop a fully explainable and interpretable system that can provide end-users with more control and understanding of music similarity and classification systems.
【4】 Diffusion Posterior Sampling for Informed Single-Channel Dereverberation标题：基于扩散后验抽样的单通道有理解混响链接：https://arxiv.org/abs/2306.12286
作者：Jean-Marie Lemercier,Simon Welker,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany,  Center for Free-Electron Laser Science, DESY, Hamburg, Germany摘要：在本文中，我们提出了一个明智的单通道去混响方法的基础上，扩散模型的条件生成。利用房间脉冲响应的知识，使用与表示干净语音先验的神经网络耦合的测量一致性标准经由反向扩散生成无回声话语。所提出的方法在很大程度上是更强大的测量噪声相比，一个国家的最先进的通知单通道去混响方法，特别是对于非平稳噪声。此外，我们比较其他盲去混响方法使用扩散模型，并显示所提出的方法的优越性大混响时间。我们激励所提出的算法，通过引入一个扩展盲去混响，允许联合估计的房间脉冲响应和消声语音。音频样本和代码可以在网上找到（https：//uhh.de/inf-sp-derev-dps）。摘要：We present in this paper an informed single-channel dereverberation method based on conditional generation with diffusion models. With knowledge of the room impulse response, the anechoic utterance is generated via reverse diffusion using a measurement consistency criterion coupled with a neural network that represents the clean speech prior. The proposed approach is largely more robust to measurement noise compared to a state-of-the-art informed single-channel dereverberation method, especially for non-stationary noise. Furthermore, we compare to other blind dereverberation methods using diffusion models and show superiority of the proposed approach for large reverberation times. We motivate the presented algorithm by introducing an extension for blind dereverberation allowing joint estimation of the room impulse response and anechoic speech. Audio samples and code can be found online (https://uhh.de/inf-sp-derev-dps).
【5】 Visual-Aware Text-to-Speech标题：视觉感知的文本到语音转换链接：https://arxiv.org/abs/2306.12020
作者：Mohan Zhou,Yalong Bai,Wei Zhang,Ting Yao,Tiejun Zhao,Tao Mei机构：Harbin Institute of Technology, Harbin, China, JD Explore Academy, Beijing, China备注：None摘要：在面对面交互期间，动态地合成主动响应于收听头部的说话语音是至关重要的。例如，说话者可以利用听者的面部表情来调整音调、重读音节或停顿。在这项工作中，我们提出了一个新的视觉感知文本到语音（VA-TTS）的任务来合成语音的文本输入和顺序的视觉反馈（例如，点头、微笑）。与传统的文语转换不同，VA-TTS突出了视觉模态的影响。在这个新的任务中，我们设计了一个基线模型来融合音素语言信息和听者视觉信号的语音合成。在多模态会话数据集ViCo-X上进行的大量实验验证了我们的建议，即用场景适当的节奏和韵律生成更自然的音频。摘要：Dynamically synthesizing talking speech that actively responds to a listening head is critical during the face-to-face interaction. For example, the speaker could take advantage of the listener's facial expression to adjust the tones, stressed syllables, or pauses. In this work, we present a new visual-aware text-to-speech (VA-TTS) task to synthesize speech conditioned on both textual inputs and sequential visual feedback (e.g., nod, smile) of the listener in face-to-face communication. Different from traditional text-to-speech, VA-TTS highlights the impact of visual modality. On this newly-minted task, we devise a baseline model to fuse phoneme linguistic information and listener visual signals for speech synthesis. Extensive experiments on multimodal conversation dataset ViCo-X verify our proposal for generating more natural audio with scenario-appropriate rhythm and prosody.
【6】 Federated Self-Learning with Weak Supervision for Speech Recognition标题：弱监督下的联合自学习语音识别链接：https://arxiv.org/abs/2306.12015
作者：Milind Rao,Gopinath Chennupati,Gautam Tiwari,Anit Kumar Sahu,Anirudh Raju,Ariya Rastrow,Jasha Droppo机构：Amazon Alexa AI, U.S.A.备注：Proceedings of ICASSP 2023摘要：具有低占用空间的自动语音识别（ASR）模型越来越多地部署在用于会话代理的边缘设备上，这增强了隐私。我们研究的问题，联邦持续增量学习的递归神经网络传感器（RNN-T）ASR模型的隐私增强计划上学习的设备上，没有访问地面真相人类转录或机器转录从一个更强的ASR模型。特别是，我们研究了基于自学习的方案的性能，通过ASR模型的指数移动平均更新配对的教师模型。此外，我们建议使用可能嘈杂的弱监督信号，如反馈分数和自然语言理解语义确定的用户行为在会话中的交互与会话代理的多个回合。在多任务策略梯度训练方法中利用这些信号来提高ASR的自学习性能。最后，我们展示了如何通过将设备上学习与使用选定历史数据集的记忆重放方法相结合来减轻灾难性遗忘。这些创新允许在新用例上WER相对提高10%，在没有强监督信号（如地面实况转录）的情况下，其他测试集的退化最小。摘要：Automatic speech recognition (ASR) models with low-footprint are increasingly being deployed on edge devices for conversational agents, which enhances privacy. We study the problem of federated continual incremental learning for recurrent neural network-transducer (RNN-T) ASR models in the privacy-enhancing scheme of learning on-device, without access to ground truth human transcripts or machine transcriptions from a stronger ASR model. In particular, we study the performance of a self-learning based scheme, with a paired teacher model updated through an exponential moving average of ASR models. Further, we propose using possibly noisy weak-supervision signals such as feedback scores and natural language understanding semantics determined from user behavior across multiple turns in a session of interactions with the conversational agent. These signals are leveraged in a multi-task policy-gradient training approach to improve the performance of self-learning for ASR. Finally, we show how catastrophic forgetting can be mitigated by combining on-device learning with a memory-replay approach using selected historical datasets. These innovations allow for 10% relative improvement in WER on new use cases with minimal degradation on other test sets in the absence of strong-supervision signals such as ground-truth transcriptions.
【7】 Learning When to Trust Which Teacher for Weakly Supervised ASR标题：学习什么时候应该信任哪位教师监督不力的ASR链接：https://arxiv.org/abs/2306.12012
作者：Aakriti Agrawal,Milind Rao,Anit Kumar Sahu,Gopinath Chennupati,Andreas Stolcke机构：∗University of Maryland, U.S.A., †Amazon Alexa AI, U.S.A.备注：Proceedings of INTERSPEECH 2023摘要：自动语音识别（ASR）训练可以利用多个专家作为教师模型，每个专家在特定的领域或口音上进行训练。教师模型本质上可能是不透明的，因为它们的架构可能是未知的，或者它们的训练节奏与学生ASR模型的训练节奏不同。尽管如此，学生模型使用由专家教师独立生成的伪标签进行增量更新。在本文中，我们利用多个领域的专家在训练学生ASR模型的监督。这种训练策略在很少或没有人工转录可用的情况下特别有用。为此，我们提出了一种智能加权机制，该机制根据输入音频选择适当的专家，然后在无监督设置中训练学生模型。我们使用LibriSpeech和LibriLight基准测试显示了我们的方法的有效性，并发现比基线提高了4到25%，这些基线统一加权所有专家，使用单个专家模型，或使用ROVER结合专家。摘要：Automatic speech recognition (ASR) training can utilize multiple experts as teacher models, each trained on a specific domain or accent. Teacher models may be opaque in nature since their architecture may be not be known or their training cadence is different from that of the student ASR model. Still, the student models are updated incrementally using the pseudo-labels generated independently by the expert teachers. In this paper, we exploit supervision from multiple domain experts in training student ASR models. This training strategy is especially useful in scenarios where few or no human transcriptions are available. To that end, we propose a Smart-Weighter mechanism that selects an appropriate expert based on the input audio, and then trains the student model in an unsupervised setting. We show the efficacy of our approach using LibriSpeech and LibriLight benchmarks and find an improvement of 4 to 25\% over baselines that uniformly weight all the experts, use a single expert model, or combine experts using ROVER.
【8】 On Frequency-Wise Normalizations for Better Recording Device  Generalization in Audio Spectrogram Transformers标题：音频频谱变换中提高录音设备通用性的频域归一化链接：https://arxiv.org/abs/2306.11764
作者：Paul Primus and,Gerhard Widmer机构：Institute of Computational Perception (CP-JKU), LIT Artificial Intelligence Lab, Johannes Kepler University, Austria备注：EUSIPCO 2023摘要：在训练和应用时看到的数据之间的变化仍然是机器学习的主要挑战。我们研究这个问题的背景下，声学场景分类（ASC）与不匹配的记录设备。以前的作品成功地采用卷积神经网络中的输入和隐藏层激活的频率归一化，以减少记录设备的差异。这项工作的主要目标是采用频率方面的标准化音频频谱图Transformers（ASTs），最近已成为占主导地位的模型架构在ASC。为此，我们首先研究如何记录设备的特性编码在隐藏层激活的AST。我们发现记录设备信息最初是在频率维度上编码的;然而，在第一个自我注意块之后，它在很大程度上被转换为令牌维度。基于这种观察，我们推测，抑制输入频谱图中的记录设备特性是最有效的。我们提出了一个频率为中心的操作频谱图，提高了ASC的性能上看不见的记录设备平均高达18.2个百分点。摘要：Varying conditions between the data seen at training and at application time remain a major challenge for machine learning. We study this problem in the context of Acoustic Scene Classification (ASC) with mismatching recording devices. Previous works successfully employed frequency-wise normalization of inputs and hidden layer activations in convolutional neural networks to reduce the recording device discrepancy. The main objective of this work was to adopt frequency-wise normalization for Audio Spectrogram Transformers (ASTs), which have recently become the dominant model architecture in ASC. To this end, we first investigate how recording device characteristics are encoded in the hidden layer activations of ASTs. We find that recording device information is initially encoded in the frequency dimension; however, after the first self-attention block, it is largely transformed into the token dimension. Based on this observation, we conjecture that suppressing recording device characteristics in the input spectrogram is the most effective. We propose a frequency-centering operation for spectrograms that improves the ASC performance on unseen recording devices on average by up to 18.2 percentage points.
eess.AS音频处理
【1】 Diffusion Posterior Sampling for Informed Single-Channel Dereverberation标题：基于扩散后验抽样的单通道有理解混响链接：https://arxiv.org/abs/2306.12286
作者：Jean-Marie Lemercier,Simon Welker,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany,  Center for Free-Electron Laser Science, DESY, Hamburg, Germany摘要：在本文中，我们提出了一个明智的单通道去混响方法的基础上，扩散模型的条件生成。利用房间脉冲响应的知识，使用与表示干净语音先验的神经网络耦合的测量一致性标准经由反向扩散生成无回声话语。所提出的方法在很大程度上是更强大的测量噪声相比，一个国家的最先进的通知单通道去混响方法，特别是对于非平稳噪声。此外，我们比较其他盲去混响方法使用扩散模型，并显示所提出的方法的优越性大混响时间。我们激励所提出的算法，通过引入一个扩展盲去混响，允许联合估计的房间脉冲响应和消声语音。音频样本和代码可以在网上找到（https：//uhh.de/inf-sp-derev-dps）。摘要：We present in this paper an informed single-channel dereverberation method based on conditional generation with diffusion models. With knowledge of the room impulse response, the anechoic utterance is generated via reverse diffusion using a measurement consistency criterion coupled with a neural network that represents the clean speech prior. The proposed approach is largely more robust to measurement noise compared to a state-of-the-art informed single-channel dereverberation method, especially for non-stationary noise. Furthermore, we compare to other blind dereverberation methods using diffusion models and show superiority of the proposed approach for large reverberation times. We motivate the presented algorithm by introducing an extension for blind dereverberation allowing joint estimation of the room impulse response and anechoic speech. Audio samples and code can be found online (https://uhh.de/inf-sp-derev-dps).
【2】 Visual-Aware Text-to-Speech标题：视觉感知的文本到语音转换链接：https://arxiv.org/abs/2306.12020
作者：Mohan Zhou,Yalong Bai,Wei Zhang,Ting Yao,Tiejun Zhao,Tao Mei机构：Harbin Institute of Technology, Harbin, China, JD Explore Academy, Beijing, China备注：None摘要：在面对面交互期间，动态地合成主动响应于收听头部的说话语音是至关重要的。例如，说话者可以利用听者的面部表情来调整音调、重读音节或停顿。在这项工作中，我们提出了一个新的视觉感知文本到语音（VA-TTS）的任务来合成语音的文本输入和顺序的视觉反馈（例如，点头、微笑）。与传统的文语转换不同，VA-TTS突出了视觉模态的影响。在这个新的任务中，我们设计了一个基线模型来融合音素语言信息和听者视觉信号的语音合成。在多模态会话数据集ViCo-X上进行的大量实验验证了我们的建议，即用场景适当的节奏和韵律生成更自然的音频。摘要：Dynamically synthesizing talking speech that actively responds to a listening head is critical during the face-to-face interaction. For example, the speaker could take advantage of the listener's facial expression to adjust the tones, stressed syllables, or pauses. In this work, we present a new visual-aware text-to-speech (VA-TTS) task to synthesize speech conditioned on both textual inputs and sequential visual feedback (e.g., nod, smile) of the listener in face-to-face communication. Different from traditional text-to-speech, VA-TTS highlights the impact of visual modality. On this newly-minted task, we devise a baseline model to fuse phoneme linguistic information and listener visual signals for speech synthesis. Extensive experiments on multimodal conversation dataset ViCo-X verify our proposal for generating more natural audio with scenario-appropriate rhythm and prosody.
【3】 Federated Self-Learning with Weak Supervision for Speech Recognition标题：弱监督下的联合自学习语音识别链接：https://arxiv.org/abs/2306.12015
作者：Milind Rao,Gopinath Chennupati,Gautam Tiwari,Anit Kumar Sahu,Anirudh Raju,Ariya Rastrow,Jasha Droppo机构：Amazon Alexa AI, U.S.A.备注：Proceedings of ICASSP 2023摘要：具有低占用空间的自动语音识别（ASR）模型越来越多地部署在用于会话代理的边缘设备上，这增强了隐私。我们研究的问题，联邦持续增量学习的递归神经网络传感器（RNN-T）ASR模型的隐私增强计划上学习的设备上，没有访问地面真相人类转录或机器转录从一个更强的ASR模型。特别是，我们研究了基于自学习的方案的性能，通过ASR模型的指数移动平均更新配对的教师模型。此外，我们建议使用可能嘈杂的弱监督信号，如反馈分数和自然语言理解语义确定的用户行为在会话中的交互与会话代理的多个回合。在多任务策略梯度训练方法中利用这些信号来提高ASR的自学习性能。最后，我们展示了如何通过将设备上学习与使用选定历史数据集的记忆重放方法相结合来减轻灾难性遗忘。这些创新允许在新用例上WER相对提高10%，在没有强监督信号（如地面实况转录）的情况下，其他测试集的退化最小。摘要：Automatic speech recognition (ASR) models with low-footprint are increasingly being deployed on edge devices for conversational agents, which enhances privacy. We study the problem of federated continual incremental learning for recurrent neural network-transducer (RNN-T) ASR models in the privacy-enhancing scheme of learning on-device, without access to ground truth human transcripts or machine transcriptions from a stronger ASR model. In particular, we study the performance of a self-learning based scheme, with a paired teacher model updated through an exponential moving average of ASR models. Further, we propose using possibly noisy weak-supervision signals such as feedback scores and natural language understanding semantics determined from user behavior across multiple turns in a session of interactions with the conversational agent. These signals are leveraged in a multi-task policy-gradient training approach to improve the performance of self-learning for ASR. Finally, we show how catastrophic forgetting can be mitigated by combining on-device learning with a memory-replay approach using selected historical datasets. These innovations allow for 10% relative improvement in WER on new use cases with minimal degradation on other test sets in the absence of strong-supervision signals such as ground-truth transcriptions.
【4】 Learning When to Trust Which Teacher for Weakly Supervised ASR标题：学习什么时候应该信任哪位教师监督不力的ASR链接：https://arxiv.org/abs/2306.12012
作者：Aakriti Agrawal,Milind Rao,Anit Kumar Sahu,Gopinath Chennupati,Andreas Stolcke机构：∗University of Maryland, U.S.A., †Amazon Alexa AI, U.S.A.备注：Proceedings of INTERSPEECH 2023摘要：自动语音识别（ASR）训练可以利用多个专家作为教师模型，每个专家在特定的领域或口音上进行训练。教师模型本质上可能是不透明的，因为它们的架构可能是未知的，或者它们的训练节奏与学生ASR模型的训练节奏不同。尽管如此，学生模型使用由专家教师独立生成的伪标签进行增量更新。在本文中，我们利用多个领域的专家在训练学生ASR模型的监督。这种训练策略在很少或没有人工转录可用的情况下特别有用。为此，我们提出了一种智能加权机制，该机制根据输入音频选择适当的专家，然后在无监督设置中训练学生模型。我们使用LibriSpeech和LibriLight基准测试显示了我们的方法的有效性，并发现比基线提高了4到25%，这些基线统一加权所有专家，使用单个专家模型，或使用ROVER结合专家。摘要：Automatic speech recognition (ASR) training can utilize multiple experts as teacher models, each trained on a specific domain or accent. Teacher models may be opaque in nature since their architecture may be not be known or their training cadence is different from that of the student ASR model. Still, the student models are updated incrementally using the pseudo-labels generated independently by the expert teachers. In this paper, we exploit supervision from multiple domain experts in training student ASR models. This training strategy is especially useful in scenarios where few or no human transcriptions are available. To that end, we propose a Smart-Weighter mechanism that selects an appropriate expert based on the input audio, and then trains the student model in an unsupervised setting. We show the efficacy of our approach using LibriSpeech and LibriLight benchmarks and find an improvement of 4 to 25\% over baselines that uniformly weight all the experts, use a single expert model, or combine experts using ROVER.
【5】 On Frequency-Wise Normalizations for Better Recording Device  Generalization in Audio Spectrogram Transformers标题：音频频谱变换中提高录音设备通用性的频域归一化链接：https://arxiv.org/abs/2306.11764
作者：Paul Primus and,Gerhard Widmer机构：Institute of Computational Perception (CP-JKU), LIT Artificial Intelligence Lab, Johannes Kepler University, Austria备注：EUSIPCO 2023摘要：在训练和应用时看到的数据之间的变化仍然是机器学习的主要挑战。我们研究这个问题的背景下，声学场景分类（ASC）与不匹配的记录设备。以前的作品成功地采用卷积神经网络中的输入和隐藏层激活的频率归一化，以减少记录设备的差异。这项工作的主要目标是采用频率方面的标准化音频频谱图Transformers（ASTs），最近已成为占主导地位的模型架构在ASC。为此，我们首先研究如何记录设备的特性编码在隐藏层激活的AST。我们发现记录设备信息最初是在频率维度上编码的;然而，在第一个自我注意块之后，它在很大程度上被转换为令牌维度。基于这种观察，我们推测，抑制输入频谱图中的记录设备特性是最有效的。我们提出了一个频率为中心的操作频谱图，提高了ASC的性能上看不见的记录设备平均高达18.2个百分点。摘要：Varying conditions between the data seen at training and at application time remain a major challenge for machine learning. We study this problem in the context of Acoustic Scene Classification (ASC) with mismatching recording devices. Previous works successfully employed frequency-wise normalization of inputs and hidden layer activations in convolutional neural networks to reduce the recording device discrepancy. The main objective of this work was to adopt frequency-wise normalization for Audio Spectrogram Transformers (ASTs), which have recently become the dominant model architecture in ASC. To this end, we first investigate how recording device characteristics are encoded in the hidden layer activations of ASTs. We find that recording device information is initially encoded in the frequency dimension; however, after the first self-attention block, it is largely transformed into the token dimension. Based on this observation, we conjecture that suppressing recording device characteristics in the input spectrogram is the most effective. We propose a frequency-centering operation for spectrograms that improves the ASC performance on unseen recording devices on average by up to 18.2 percentage points.
【6】 A Multimodal Prototypical Approach for Unsupervised Sound Classification标题：一种无监督声音分类的多模式原型方法链接：https://arxiv.org/abs/2306.12300
作者：Saksham Singh Kushwaha,Magdalena Fuentes机构：Courant Institute of Mathematical Sciences, New York University, NY, USA, MARL, New York University, NY, USA, IDM, New York University, NY, USA备注：Accepted to INTERSPEECH 2023摘要：在环境声音分类方面，系统的适应性是关键：哪些声音类是感兴趣的取决于上下文和用户的需要。文本到音频检索的最新进展允许零拍摄音频分类，但与监督模型相比，性能仍然有限。这项工作提出了一种多模态的原型方法，利用本地音频文本嵌入音频查询提供更相关的答案，增强了适应性的声音检测在野外。我们首先使用文本查询附近的音频嵌入社区，最好地表征每个查询声音，并选择该组的质心作为我们的原型。其次，我们将看不见的音频与这些原型进行比较以进行分类。我们进行了多次消融研究，以了解嵌入模型和提示的影响。我们的无监督方法在三个声音识别基准测试中的零拍最先进水平平均提高了12%。摘要：In the context of environmental sound classification, the adaptability of systems is key: which sound classes are interesting depends on the context and the user's needs. Recent advances in text-to-audio retrieval allow for zero-shot audio classification, but performance compared to supervised models remains limited. This work proposes a multimodal prototypical approach that exploits local audio-text embeddings to provide more relevant answers to audio queries, augmenting the adaptability of sound detection in the wild. We do this by first using text to query a nearby community of audio embeddings that best characterize each query sound, and select the group's centroids as our prototypes. Second, we compare unseen audio to these prototypes for classification. We perform multiple ablation studies to understand the impact of the embedding models and prompts. Our unsupervised approach improves upon the zero-shot state-of-the-art in three sound recognition benchmarks by an average of 12%.
【7】 Automatic Speech Disentanglement for Voice Conversion using Rank Module  and Speech Augmentation标题：利用秩模和语音增强实现语音转换的语音解缠链接：https://arxiv.org/abs/2306.12259
作者：Zhonghua Liu,Shijun Wang,Ning Chen机构：East China University of Science and Technology, Shanghai, China, University of St.Gallen, St.Gallen, Switzerland备注：Accepted by INTERSPEECH2023摘要：语音转换（VC）将源语音的语音转换为目标语音，同时保持源语音的内容。语音主要可以分解为四个部分：内容、音色、节奏和音高。不幸的是，大多数相关作品只考虑内容和音色，这导致语音不太自然。最近的一些作品能够解开语音成几个组件，但他们需要费力的瓶颈调谐或各种手工制作的功能，每个假设包含解开的语音信息。在本文中，我们提出了一个VC模型，可以自动解开语音成四个组件，只使用两个增强功能，而无需多个手工制作的功能或费力的瓶颈调整。该模型是简单而有效的，实证结果表明，我们的模型可以实现更好的性能比基线，关于解纠缠的有效性和语音自然度。摘要：Voice Conversion (VC) converts the voice of a source speech to that of a target while maintaining the source's content. Speech can be mainly decomposed into four components: content, timbre, rhythm and pitch. Unfortunately, most related works only take into account content and timbre, which results in less natural speech. Some recent works are able to disentangle speech into several components, but they require laborious bottleneck tuning or various hand-crafted features, each assumed to contain disentangled speech information. In this paper, we propose a VC model that can automatically disentangle speech into four components using only two augmentation functions, without the requirement of multiple hand-crafted features or laborious bottleneck tuning. The proposed model is straightforward yet efficient, and the empirical results demonstrate that our model can achieve a better performance than the baseline, regarding disentanglement effectiveness and speech naturalness.
【8】 Knowledge-based Multimodal Music Similarity标题：基于知识的多通道音乐相似度研究链接：https://arxiv.org/abs/2306.12249
作者：Andrea Poltronieri机构：Department of Computer Science and Engineering, University of Bologna, Italy备注：11 pages, 1 figure摘要：音乐相似性是音乐检索、推荐系统和音乐分析的一个重要方面。此外，相似性对音乐专家来说是至关重要的，因为它允许研究作曲家和历史时期之间的类比和影响。当前的音乐相似性的方法主要依赖于符号内容，其生产成本可能是昂贵的，并且不总是现成的。相反，使用音频信号的方法通常不能提供关于观察到的相似性背后的原因的任何洞察。本研究解决了目前的方法的局限性，重点研究音乐的相似性，使用符号和音频内容。本研究的目的是开发一个完全可解释和解释的系统，可以提供终端用户更多的控制和理解音乐相似性和分类系统。摘要：Music similarity is an essential aspect of music retrieval, recommendation systems, and music analysis. Moreover, similarity is of vital interest for music experts, as it allows studying analogies and influences among composers and historical periods. Current approaches to musical similarity rely mainly on symbolic content, which can be expensive to produce and is not always readily available. Conversely, approaches using audio signals typically fail to provide any insight about the reasons behind the observed similarity. This research addresses the limitations of current approaches by focusing on the study of musical similarity using both symbolic and audio content. The aim of this research is to develop a fully explainable and interpretable system that can provide end-users with more control and understanding of music similarity and classification systems.
【9】 Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone  Mapping, Features Input, and Source Language Selection标题：低资源语音合成中的迁移学习策略：音素映射、特征输入和源语言选择链接：https://arxiv.org/abs/2306.12040
作者：Phat Do,Matt Coler,Jelske Dijkstra,Esther Klabbers机构：Campus Fryslˆan, University of Groningen, the Netherlands, Fryske AkademyMercator Research Centre, the Netherlands, ReadSpeaker, the Netherlands备注：Accepted at the Speech Synthesis Workshop 2023摘要：我们比较使用PHOIBLE为基础的电话映射方法，并使用语音特征输入的迁移学习TTS在低资源的语言。我们使用不同的源语言（英语，芬兰语，印地语，日语和俄语）和目标语言（保加利亚语，格鲁吉亚语，哈萨克语，斯瓦希里语，乌尔都语和乌兹别克语）来测试语言的独立性的方法，并提高结果的适用性。我们使用自动语音识别的字符错误率和预测的平均意见得分进行评估。结果表明，音素映射和特征输入都能提高输出质量，且特征输入的效果更好，但这些效果也取决于具体的语言组合。我们还比较了最近提出的电话频率的角度相似性（ASPF）与基于家谱的距离度量作为迁移学习中选择源语言的标准。ASPF被证明是有效的，如果使用基于标签的电话输入，而语言距离没有预期的效果。摘要：We compare using a PHOIBLE-based phone mapping method and using phonological features input in transfer learning for TTS in low-resource languages. We use diverse source languages (English, Finnish, Hindi, Japanese, and Russian) and target languages (Bulgarian, Georgian, Kazakh, Swahili, Urdu, and Uzbek) to test the language-independence of the methods and enhance the findings' applicability. We use Character Error Rates from automatic speech recognition and predicted Mean Opinion Scores for evaluation. Results show that both phone mapping and features input improve the output quality and the latter performs better, but these effects also depend on the specific language combination. We also compare the recently-proposed Angular Similarity of Phone Frequencies (ASPF) with a family tree-based distance measure as a criterion to select source languages in transfer learning. ASPF proves effective if label-based phone input is used, while the language distance does not have expected effects.
机器翻译，仅供参考