今日论文合集：cs.SD语音10篇，eess.AS音频处理10篇。
cs.SD语音
【1】 A Low-rank Matching Attention based Cross-modal Feature Fusion Method  for Conversational Emotion Recognition标题：一种基于低阶匹配注意力的跨通道特征融合会话情感识别方法链接：https://arxiv.org/abs/2306.17799
作者：Yuntao Shou,Xiangyong Cao,Deyu Meng,Bo Dong,Qinghua Zheng机构：Xi’an Jiaotong University, Xi’an, China备注：10 pages, 4 figures摘要：会话情感识别是人机交互中的一个重要研究课题。虽然基于深度学习（DL）的CER方法已经取得了优异的性能，但在这些基于DL的方法中使用的现有跨模态特征融合方法要么忽略了模态内和模态间的情感交互，要么具有较高的计算复杂度。为了解决这些问题，本文开发了一种新的跨模态特征融合方法的CER任务，即，低秩匹配注意法（LMAM）。通过设置匹配权重并逐行计算模态特征之间的注意力得分，LMAM包含的参数比自注意法少。我们进一步利用权值的低秩分解方法，使LMAM的参数数小于自注意的三分之一。因此，LMAM可以潜在地缓解由大量参数引起的过拟合问题。另外，通过计算和融合模态内和模态间特征的相似性，LMAM还可以充分利用每个模态内的模态内上下文信息和跨模态的互补语义信息（即，文本、视频和音频）同时进行。在一些基准数据集上的实验结果表明，LMAM可以嵌入到任何现有的国家的最先进的基于DL的CER方法，并有助于提高其性能，即插即用的方式。实验结果验证了LMAM算法与其他常用的跨模态融合方法相比的优越性。此外，LMAM是通用的跨模态融合方法，并且因此可以应用于其他多模态识别任务，例如，会话推荐和幽默检测。摘要：Conversational emotion recognition (CER) is an important research topic in human-computer interactions. Although deep learning (DL) based CER approaches have achieved excellent performance, existing cross-modal feature fusion methods used in these DL-based approaches either ignore the intra-modal and inter-modal emotional interaction or have high computational complexity. To address these issues, this paper develops a novel cross-modal feature fusion method for the CER task, i.e., the low-rank matching attention method (LMAM). By setting a matching weight and calculating attention scores between modal features row by row, LMAM contains fewer parameters than the self-attention method. We further utilize the low-rank decomposition method on the weight to make the parameter number of LMAM less than one-third of the self-attention. Therefore, LMAM can potentially alleviate the over-fitting issue caused by a large number of parameters. Additionally, by computing and fusing the similarity of intra-modal and inter-modal features, LMAM can also fully exploit the intra-modal contextual information within each modality and the complementary semantic information across modalities (i.e., text, video and audio) simultaneously. Experimental results on some benchmark datasets show that LMAM can be embedded into any existing state-of-the-art DL-based CER methods and help boost their performance in a plug-and-play manner. Also, experimental results verify the superiority of LMAM compared with other popular cross-modal fusion methods. Moreover, LMAM is a general cross-modal fusion method and can thus be applied to other multi-modal recognition tasks, e.g., session recommendation and humour detection.
【2】 Towards Improving the Performance of Pre-Trained Speech Models for  Low-Resource Languages Through Lateral Inhibition标题：通过侧向抑制提高低资源语言语音模型的训练性能链接：https://arxiv.org/abs/2306.17792
作者：Andrei-Marius Avram,Răzvan-Alexandru Smădu,Vasile Păiş,Dumitru-Clementin Cercel,Radu Ion,Dan Tufiş机构：University Politehnica of Bucharest, Romania, †Research Institute for Artificial Intelligence “Mihai Dr˘ag˘anescu”, Romanian Academy备注：Accepted at TSP2023摘要：随着自然语言处理中Transformer模型中双向编码器表示的兴起，语音社区已经采用了他们的一些开发方法。因此，引入Wav2Vec模型以减少获得最先进结果所需的数据。这项工作利用了这些知识，并通过简单地替换微调密集层的生物过程启发的横向抑制层来提高预训练语音模型的性能。我们的实验罗马尼亚语，低资源的语言，显示出平均改善12.5%的字错误率（WER）使用横向抑制层。此外，我们获得了最先进的罗马尼亚语音语料库和罗宾技术采集语料库1.78%和29.64%，分别为最先进的结果。摘要：With the rise of bidirectional encoder representations from Transformer models in natural language processing, the speech community has adopted some of their development methodologies. Therefore, the Wav2Vec models were introduced to reduce the data required to obtain state-of-the-art results. This work leverages this knowledge and improves the performance of the pre-trained speech models by simply replacing the fine-tuning dense layer with a lateral inhibition layer inspired by the biological process. Our experiments on Romanian, a low-resource language, show an average improvement of 12.5% word error rate (WER) using the lateral inhibition layer. In addition, we obtain state-of-the-art results on both the Romanian Speech Corpus and the Robin Technical Acquisition Corpus with 1.78% WER and 29.64% WER, respectively.
【3】 Empirical Interpretation of the Relationship Between Speech Acoustic  Context and Emotion Recognition标题：语音声学语境与情绪识别关系的实证研究链接：https://arxiv.org/abs/2306.17500
作者：Anna Ollerenshaw,Md Asif Jalal,Rosanna Milner,Thomas Hain机构：Speech and Hearing Research Group, Department of Computer Science, University of Sheffield, UK摘要：语音情感识别（SER）对于获取情感智力和理解语音语境意义至关重要。辅音-元音音位边界的变化可以丰富声学语境中的语言线索，从而影响SER。在实践中，语音情绪被视为在给定持续时间内声学片段上的单个标签。然而，语音内的电话边界不是离散事件，因此感知的情绪状态也应该分布在潜在的连续时间窗口。  本研究探讨了声学背景和电话边界对SER本地标记使用注意为基础的方法的影响。跨语料库分析实验的结果支持使用分布式方法来理解语音情感的好处。实验中的音素和单词被映射到注意向量连同基频，以观察重叠分布，从而观察声学上下文和情感之间的关系。这项工作的目的是桥梁心理语言学理论研究与计算建模的SER。摘要：Speech emotion recognition (SER) is vital for obtaining emotional intelligence and understanding the contextual meaning of speech. Variations of consonant-vowel (CV) phonemic boundaries can enrich acoustic context with linguistic cues, which impacts SER. In practice, speech emotions are treated as single labels over an acoustic segment for a given time duration. However, phone boundaries within speech are not discrete events, therefore the perceived emotion state should also be distributed over potentially continuous time-windows.  This research explores the implication of acoustic context and phone boundaries on local markers for SER using an attention-based approach. The benefits of using a distributed approach to speech emotion understanding are supported by the results of cross-corpora analysis experiments. Experiments where phones and words are mapped to the attention vectors along with the fundamental frequency to observe the overlapping distributions and thereby the relationship between acoustic context and emotion. This work aims to bridge psycholinguistic theory research with computational modelling for SER.
【4】 Beyond-Voice: Towards Continuous 3D Hand Pose Tracking on Commercial  Home Assistant Devices标题：Beyond-Voice：商用家用辅助设备上的连续3D手势跟踪链接：https://arxiv.org/abs/2306.17477
作者：Yin Li,Rohan Reddy,Cheng Zhang,Rajalakshmi Nandakumar机构：Cornell University, United States摘要：越来越流行的家庭助理被广泛用作智能家居设备的中央控制器。然而，当前的设计严重依赖语音接口，具有可访问性和可用性问题;一些最新的设备配备了额外的摄像机和显示器，这很昂贵，并引起了隐私问题。这些担忧共同推动了Beyond-Voice，这是一种新型的深度学习驱动的声学传感系统，允许家用助理设备连续跟踪和重建手部姿势。它利用现有的机载麦克风和扬声器将家庭助理转变为主动声纳系统。我们向深度学习模型提供了高分辨率的距离轮廓，该模型可以分析多个身体部位的运动并预测21个手指关节的3D位置，从而将声学手部跟踪的粒度提升到一个新的水平。它可以跨不同的环境和用户运行，而不需要个性化的训练数据。在3种不同环境中对11名参与者进行的用户研究表明，Beyond-Voice可以跟踪关节，平均绝对误差为16.47mm，而无需测试对象提供任何训练数据。摘要：Increasingly popular home assistants are widely utilized as the central controller for smart home devices. However, current designs heavily rely on voice interfaces with accessibility and usability issues; some latest ones are equipped with additional cameras and displays, which are costly and raise privacy concerns. These concerns jointly motivate Beyond-Voice, a novel deep-learning-driven acoustic sensing system that allows commodity home assistant devices to track and reconstruct hand poses continuously. It transforms the home assistant into an active sonar system using its existing onboard microphones and speakers. We feed a high-resolution range profile to the deep learning model that can analyze the motions of multiple body parts and predict the 3D positions of 21 finger joints, bringing the granularity for acoustic hand tracking to the next level. It operates across different environments and users without the need for personalized training data. A user study with 11 participants in 3 different environments shows that Beyond-Voice can track joints with an average mean absolute error of 16.47mm without any training data provided by the testing subject.
【5】 Audio Embeddings as Teachers for Music Classification标题：音频嵌入作为音乐分类的教师链接：https://arxiv.org/abs/2306.17424
作者：Yiwei Ding,Alexander Lerch机构：Music Informatics Group, Georgia Institute of Technology备注：Accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 9 pages, 2 figures摘要：音乐分类一直是音乐信息检索领域的热门课题之一。随着深度学习模型的发展，在过去的十年里，各种分类任务都取得了令人印象深刻的进步。然而，不断增加的模型复杂性使得训练和推理在计算上都是昂贵的。在本文中，我们整合迁移学习和基于特征的知识提炼的思想，系统地研究使用预先训练的音频嵌入作为教师来指导低复杂度学生网络的训练。通过对学生网络的特征空间进行正则化，教师嵌入中的知识可以传递给学生。我们使用各种预训练的音频嵌入和测试的有效性的乐器分类和音乐自动标记的任务的方法。结果表明，我们的方法显着提高了结果相比，相同的模型训练没有教师的知识。该技术也可以结合经典的知识提炼方法，以进一步提高模型的性能。摘要：Music classification has been one of the most popular tasks in the field of music information retrieval. With the development of deep learning models, the last decade has seen impressive improvements in a wide range of classification tasks. However, the increasing model complexity makes both training and inference computationally expensive. In this paper, we integrate the ideas of transfer learning and feature-based knowledge distillation and systematically investigate using pre-trained audio embeddings as teachers to guide the training of low-complexity student networks. By regularizing the feature space of the student networks with the pre-trained embeddings, the knowledge in the teacher embeddings can be transferred to the students. We use various pre-trained audio embeddings and test the effectiveness of the method on the tasks of musical instrument classification and music auto-tagging. Results show that our method significantly improves the results in comparison to the identical model trained without the teacher's knowledge. This technique can also be combined with classical knowledge distillation approaches to further improve the model's performance.
【6】 Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion  Models标题：Diff-Foley：基于潜在扩散模型的同步视音频合成链接：https://arxiv.org/abs/2306.17203
作者：Simian Luo,Chuanhao Yan,Chenxu Hu,Hang Zhao机构：IIIS, Tsinghua University, Shanghai Qi Zhi Institute摘要：视频到音频（V2 A）模型最近由于其在直接从无声视频生成音频中的实际应用而受到关注，特别是在视频/电影制作中。然而，V2 A中的先前方法在时间同步和视听相关性方面具有有限的生成质量。我们提出了Diff-Foley，一个同步的视频到音频合成方法与潜在的扩散模型（LDM），产生高质量的音频与改进的同步和视听相关性。我们采用对比视听预训练（CAVP）来学习更多的时间和语义对齐的特征，然后在声谱图潜在空间上训练LDM与CAVP对齐的视觉特征。CAVP对齐的特征使LDM能够通过交叉注意模块捕获更微妙的视听相关性。我们通过“双重指导”进一步显著提高样品质量。Diff-Foley在当前大规模V2 A数据集上实现了最先进的V2 A性能。此外，我们证明Diff-Foley的实际适用性和泛化能力，通过下游微调。项目页面：参见https://diff-foley.github.io/摘要：The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present Diff-Foley, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross-attention module. We further significantly improve sample quality with `double guidance'. Diff-Foley achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate Diff-Foley practical applicability and generalization capabilities via downstream finetuning. Project Page: see https://diff-foley.github.io/
【7】 Beyond Neural-on-Neural Approaches to Speaker Gender Protection标题：超越神经对神经的说话人性别保护方法链接：https://arxiv.org/abs/2306.17700
作者：Loes van Bemmel,Zhuoran Liu,Nik Vaessen,Martha Larson机构：Institute for Computing and Information Sciences,Department of Artificial Intelligence, Center for Language Studies, Radboud University Nijmegen, the Netherlands备注：None摘要：最近的研究提出了修改语音以抵御性别推理攻击的方法。这些保护算法的目标是控制关于说话者性别的信息的可用性，这是隐私敏感的属性。目前，开发和测试性别保护算法的常用实践是“神经对神经”，即，利用神经网络产生扰动并对其进行测试。在本文中，我们建议超越这种做法，加强性别保护的研究。首先，我们证明了测试性别推理攻击的重要性，这是基于语音科学家历史上开发的语音特征，以及传统使用的神经分类器。接下来，我们认为，研究人员应该使用语音特征来深入了解保护性修改如何改变语音信号。最后，我们指出，性别保护算法应该比较新的“声音对手”，人类执行的声音适应，以提高可解释性，使前麦克风保护。摘要：Recent research has proposed approaches that modify speech to defend against gender inference attacks. The goal of these protection algorithms is to control the availability of information about a speaker's gender, a privacy-sensitive attribute. Currently, the common practice for developing and testing gender protection algorithms is "neural-on-neural", i.e., perturbations are generated and tested with a neural network. In this paper, we propose to go beyond this practice to strengthen the study of gender protection. First, we demonstrate the importance of testing gender inference attacks that are based on speech features historically developed by speech scientists, alongside the conventionally used neural classifiers. Next, we argue that researchers should use speech features to gain insight into how protective modifications change the speech signal. Finally, we point out that gender-protection algorithms should be compared with novel "vocal adversaries", human-executed voice adaptations, in order to improve interpretability and enable before-the-mic protection.
【8】 Modified Parametric Multichannel Wiener Filter \\for Low-latency  Enhancement of Speech Mixtures with Unknown Number of Speakers标题：改进的参数多通道维纳滤波用于低延时增强说话人数目未知的混合语音链接：https://arxiv.org/abs/2306.17317
作者：Ning Guo,Tomohiro Nakatani,Shoko Araki,Takehiro Moriya机构：∗ NTT Corporation, Japan, † International Audio Laboratories Erlangen, Germany摘要：本文介绍了一种新的低延迟在线波束形成算法，称为修改参数多通道维纳滤波器（Mod-PMWF），增强未知和变化的说话人的混合语音。尽管诸如线性约束最小方差BF（LCMV BF）之类的常规BF可以增强语音混合，但是它们通常需要语音混合的诸如扬声器的数量和从扬声器到麦克风的声学传递函数（ATF）之类的属性。当混合属性不可用时，通过低延迟处理估计它们是具有挑战性的，阻碍了BF的应用到该问题。在本文中，我们克服了这个问题，通过修改一个传统的参数多通道维纳滤波器（PMWF）。提出的Mod-PMWF可以自适应地形成一个方向性图案，增强所有的扬声器的混合，而不显式估计这些属性。我们的实验将显示建议BF的有效性，干扰减少率和主观听力测试。摘要：This paper introduces a novel low-latency online beamforming (BF) algorithm, named Modified Parametric Multichannel Wiener Filter (Mod-PMWF), for enhancing speech mixtures with unknown and varying number of speakers. Although conventional BFs such as linearly constrained minimum variance BF (LCMV BF) can enhance a speech mixture, they typically require such attributes of the speech mixture as the number of speakers and the acoustic transfer functions (ATFs) from the speakers to the microphones. When the mixture attributes are unavailable, estimating them by low-latency processing is challenging, hindering the application of the BFs to the problem. In this paper, we overcome this problem by modifying a conventional Parametric Multichannel Wiener Filter (PMWF). The proposed Mod-PMWF can adaptively form a directivity pattern that enhances all the speakers in the mixture without explicitly estimating these attributes. Our experiments will show the proposed BF's effectiveness in interference reduction ratios and subjective listening tests.
【9】 Singing Voice Synthesis Using Differentiable LPC and  Glottal-Flow-Inspired Wavetables标题：基于可差分LPC和声门流激励波表的歌声合成链接：https://arxiv.org/abs/2306.17252
作者：Chin-Yun Yu,György Fazekas机构：Centre for Digital Music, Queen Mary University of London, UK备注：9 pages, 4 figures. Accepted at ISMIR 2023摘要：本文介绍了GlOttal-flow LPC滤波器（GOLF），一种新颖的方法唱歌的声音合成（SVS），利用人的声音的物理特性，使用可微分数字信号处理。GOLF采用声门模型作为谐波源和IIR滤波器来模拟声道，产生了一个可解释和有效的方法。我们表明，它是有竞争力的国家的最先进的歌唱声码器，需要更少的合成参数和更少的内存来训练，并运行一个数量级更快的推理。此外，我们证明，GOLF可以建模的人类的声音，这具有巨大的潜力，渲染和分析唱歌的声音在一个可区分的方式的相位分量。我们的研究结果突出了将人类语音机制的物理属性纳入SVS的有效性，并强调了基于信号处理的方法的优势，这提供了更大的解释性和效率的合成。音频样本可在www.example.com获得https://yoyololicon.github.io/golf-demo/。摘要：This paper introduces GlOttal-flow LPC Filter (GOLF), a novel method for singing voice synthesis (SVS) that exploits the physical characteristics of the human voice using differentiable digital signal processing. GOLF employs a glottal model as the harmonic source and IIR filters to simulate the vocal tract, resulting in an interpretable and efficient approach. We show it is competitive with state-of-the-art singing voice vocoders, requiring fewer synthesis parameters and less memory to train, and runs an order of magnitude faster for inference. Additionally, we demonstrate that GOLF can model the phase components of the human voice, which has immense potential for rendering and analysing singing voices in a differentiable manner. Our results highlight the effectiveness of incorporating the physical properties of the human voice mechanism into SVS and underscore the advantages of signal-processing-based approaches, which offer greater interpretability and efficiency in synthesis. Audio samples are available at https://yoyololicon.github.io/golf-demo/.
【10】 Learning Multilingual Expressive Speech Representation for Prosody  Prediction without Parallel Data标题：无需并行数据的韵律预测多语言表达语音表示学习链接：https://arxiv.org/abs/2306.17199
作者：Jarod Duret,Titouan Parcollet,Yannick Estève机构：LIA - Avignon Universite, France, University of Cambridge, United-Kingdom备注：None摘要：我们提出了一种语音到语音情感保留翻译的方法，操作在离散语音单元的水平。我们的方法依赖于使用多语言情感嵌入，可以捕捉情感信息的语言独立的方式。我们表明，这种嵌入可以用来预测目标语言中的语音单元的音高和持续时间，使我们能够重新合成具有相同情感内容的源语音信号。我们评估了我们的方法，英语和法语语音信号，并表明它优于不使用情感信息的基线方法，包括从不同的语言提取情感嵌入。即使这项初步的研究没有直接解决机器翻译的问题，我们的研究结果表明，我们的方法在语音再合成的背景下跨语言情感保存的有效性。摘要：We propose a method for speech-to-speech emotionpreserving translation that operates at the level of discrete speech units. Our approach relies on the use of multilingual emotion embedding that can capture affective information in a language-independent manner. We show that this embedding can be used to predict the pitch and duration of speech units in a target language, allowing us to resynthesize the source speech signal with the same emotional content. We evaluate our approach to English and French speech signals and show that it outperforms a baseline method that does not use emotional information, including when the emotion embedding is extracted from a different language. Even if this preliminary study does not address directly the machine translation issue, our results demonstrate the effectiveness of our approach for cross-lingual emotion preservation in the context of speech resynthesis.
eess.AS音频处理
【1】 Beyond Neural-on-Neural Approaches to Speaker Gender Protection标题：超越神经对神经的说话人性别保护方法链接：https://arxiv.org/abs/2306.17700
作者：Loes van Bemmel,Zhuoran Liu,Nik Vaessen,Martha Larson机构：Institute for Computing and Information Sciences,Department of Artificial Intelligence, Center for Language Studies, Radboud University Nijmegen, the Netherlands备注：None摘要：最近的研究提出了修改语音以抵御性别推断攻击的方法。这些保护算法的目标是控制关于说话者性别的信息的可用性，这是一种隐私敏感属性。目前，开发和测试性别保护算法的常见做法是“神经对神经”，即，产生扰动并利用神经网络进行测试。在本文中，我们建议超越这种做法，加强性别保护的研究。首先，我们展示了测试性别推断攻击的重要性，这些攻击是基于语音科学家历史上开发的语音特征，以及传统使用的神经分类器。接下来，我们认为，研究人员应该使用语音功能来深入了解保护性修改如何改变语音信号。最后，我们指出，性别保护算法应与新的“声音的对手”，人类执行的声音适应，以提高可解释性，使麦克风前的保护。摘要：Recent research has proposed approaches that modify speech to defend against gender inference attacks. The goal of these protection algorithms is to control the availability of information about a speaker's gender, a privacy-sensitive attribute. Currently, the common practice for developing and testing gender protection algorithms is "neural-on-neural", i.e., perturbations are generated and tested with a neural network. In this paper, we propose to go beyond this practice to strengthen the study of gender protection. First, we demonstrate the importance of testing gender inference attacks that are based on speech features historically developed by speech scientists, alongside the conventionally used neural classifiers. Next, we argue that researchers should use speech features to gain insight into how protective modifications change the speech signal. Finally, we point out that gender-protection algorithms should be compared with novel "vocal adversaries", human-executed voice adaptations, in order to improve interpretability and enable before-the-mic protection.
【2】 Modified Parametric Multichannel Wiener Filter \\for Low-latency  Enhancement of Speech Mixtures with Unknown Number of Speakers标题：改进的参数多通道维纳滤波用于低延时增强说话人数目未知的混合语音链接：https://arxiv.org/abs/2306.17317
作者：Ning Guo,Tomohiro Nakatani,Shoko Araki,Takehiro Moriya机构：∗ NTT Corporation, Japan, † International Audio Laboratories Erlangen, Germany摘要：本文介绍了一种新的低延迟在线波束形成（BF）算法，命名为修改参数多通道维纳滤波器（Mod-PMWF），用于增强语音混合未知和不同数量的发言人。尽管诸如线性约束最小方差BF（LCMV BF）的常规BF可以增强语音混合，但是它们通常需要语音混合的诸如扬声器的数量和从扬声器到麦克风的声学传递函数（ATF）的属性。当混合物属性不可用时，通过低延迟处理来估计它们是具有挑战性的，阻碍了BF对问题的应用。在本文中，我们克服了这个问题，通过修改传统的参数多通道维纳滤波器（PMWF）。所提出的Mod-PMWF可以自适应地形成一个方向性模式，增强了所有的扬声器在混合物中，而不显式地估计这些属性。我们的实验将显示所提出的BF的有效性，在干扰减少率和主观听力测试。摘要：This paper introduces a novel low-latency online beamforming (BF) algorithm, named Modified Parametric Multichannel Wiener Filter (Mod-PMWF), for enhancing speech mixtures with unknown and varying number of speakers. Although conventional BFs such as linearly constrained minimum variance BF (LCMV BF) can enhance a speech mixture, they typically require such attributes of the speech mixture as the number of speakers and the acoustic transfer functions (ATFs) from the speakers to the microphones. When the mixture attributes are unavailable, estimating them by low-latency processing is challenging, hindering the application of the BFs to the problem. In this paper, we overcome this problem by modifying a conventional Parametric Multichannel Wiener Filter (PMWF). The proposed Mod-PMWF can adaptively form a directivity pattern that enhances all the speakers in the mixture without explicitly estimating these attributes. Our experiments will show the proposed BF's effectiveness in interference reduction ratios and subjective listening tests.
【3】 Singing Voice Synthesis Using Differentiable LPC and  Glottal-Flow-Inspired Wavetables标题：基于可差分LPC和声门流激励波表的歌声合成链接：https://arxiv.org/abs/2306.17252
作者：Chin-Yun Yu,György Fazekas机构：Centre for Digital Music, Queen Mary University of London, UK备注：9 pages, 4 figures. Accepted at ISMIR 2023摘要：本文介绍了一种新的歌唱声音合成方法----GLOTAL流LPC滤波器（GOLF），该方法利用了人声的物理特性，采用可微分数字信号处理。GOLF采用声门模型作为谐波源和IIR滤波器来模拟声道，从而产生可解释和有效的方法。我们表明，它是有竞争力的国家的最先进的歌声声码器，需要更少的合成参数和更少的内存来训练，并运行一个数量级更快的推理。此外，我们表明，高尔夫可以模拟人类的声音，这具有巨大的潜力，渲染和分析歌唱的声音在一个可区分的方式的相位分量。我们的研究结果强调了将人类语音机制的物理特性纳入SVS的有效性，并强调了基于信号处理的方法的优势，该方法提供了更高的可解释性和合成效率。音频样本可在www.example.com上获得https://yoyololicon.github.io/golf-demo/。摘要：This paper introduces GlOttal-flow LPC Filter (GOLF), a novel method for singing voice synthesis (SVS) that exploits the physical characteristics of the human voice using differentiable digital signal processing. GOLF employs a glottal model as the harmonic source and IIR filters to simulate the vocal tract, resulting in an interpretable and efficient approach. We show it is competitive with state-of-the-art singing voice vocoders, requiring fewer synthesis parameters and less memory to train, and runs an order of magnitude faster for inference. Additionally, we demonstrate that GOLF can model the phase components of the human voice, which has immense potential for rendering and analysing singing voices in a differentiable manner. Our results highlight the effectiveness of incorporating the physical properties of the human voice mechanism into SVS and underscore the advantages of signal-processing-based approaches, which offer greater interpretability and efficiency in synthesis. Audio samples are available at https://yoyololicon.github.io/golf-demo/.
【4】 Learning Multilingual Expressive Speech Representation for Prosody  Prediction without Parallel Data标题：无需并行数据的韵律预测多语言表达语音表示学习链接：https://arxiv.org/abs/2306.17199
作者：Jarod Duret,Titouan Parcollet,Yannick Estève机构：LIA - Avignon Universite, France, University of Cambridge, United-Kingdom备注：None摘要：我们提出了一种方法，语音到语音emotionpreserving的翻译，在离散语音单元的水平。我们的方法依赖于使用多语言的情感嵌入，可以捕获情感信息的语言无关的方式。我们表明，这种嵌入可以用来预测在目标语言中的语音单元的音高和持续时间，使我们能够重新合成源语音信号具有相同的情感内容。我们评估我们的方法，英语和法语的语音信号，并表明它优于基线方法，不使用情感信息，包括当情感嵌入是从不同的语言中提取。即使这个初步的研究没有直接解决机器翻译的问题，我们的研究结果表明，我们的方法的有效性，在语音合成的背景下，跨语言的情感保存。摘要：We propose a method for speech-to-speech emotionpreserving translation that operates at the level of discrete speech units. Our approach relies on the use of multilingual emotion embedding that can capture affective information in a language-independent manner. We show that this embedding can be used to predict the pitch and duration of speech units in a target language, allowing us to resynthesize the source speech signal with the same emotional content. We evaluate our approach to English and French speech signals and show that it outperforms a baseline method that does not use emotional information, including when the emotion embedding is extracted from a different language. Even if this preliminary study does not address directly the machine translation issue, our results demonstrate the effectiveness of our approach for cross-lingual emotion preservation in the context of speech resynthesis.
【5】 A Low-rank Matching Attention based Cross-modal Feature Fusion Method  for Conversational Emotion Recognition标题：一种基于低阶匹配注意力的跨通道特征融合会话情感识别方法链接：https://arxiv.org/abs/2306.17799
作者：Yuntao Shou,Xiangyong Cao,Deyu Meng,Bo Dong,Qinghua Zheng机构：Xi’an Jiaotong University, Xi’an, China备注：10 pages, 4 figures摘要：会话情感识别是人机交互中的一个重要研究课题。虽然基于深度学习（DL）的CER方法已经取得了优异的性能，但是在这些基于DL的方法中使用的现有跨模态特征融合方法要么忽略模态内和模态间的情感交互，要么具有高计算复杂度。为了解决这些问题，本文开发了一种新的跨模态特征融合方法的CER任务，即，低秩匹配注意方法（LMAM）。通过设置匹配权重并逐行计算模态特征之间的注意力得分，LMAM包含的参数比自注意法少。我们进一步利用低秩分解方法的权重，使LMAM的参数数量小于自注意的三分之一。因此，LMAM可以潜在地缓解由大量参数引起的过拟合问题。另外，通过计算和融合模态内和模态间特征的相似性，LMAM还可以充分利用每个模态内的模态内上下文信息和跨模态的互补语义信息（即，文本、视频和音频）。一些基准数据集上的实验结果表明，LMAM可以嵌入到任何现有的国家的最先进的基于DL的CER方法，并有助于提高他们的性能在一个即插即用的方式。实验结果也验证了LMAM与其他流行的跨模态融合方法相比的优越性。此外，LMAM是通用的跨模态融合方法，并且因此可以应用于其他多模态识别任务，例如，会话推荐和幽默检测。摘要：Conversational emotion recognition (CER) is an important research topic in human-computer interactions. Although deep learning (DL) based CER approaches have achieved excellent performance, existing cross-modal feature fusion methods used in these DL-based approaches either ignore the intra-modal and inter-modal emotional interaction or have high computational complexity. To address these issues, this paper develops a novel cross-modal feature fusion method for the CER task, i.e., the low-rank matching attention method (LMAM). By setting a matching weight and calculating attention scores between modal features row by row, LMAM contains fewer parameters than the self-attention method. We further utilize the low-rank decomposition method on the weight to make the parameter number of LMAM less than one-third of the self-attention. Therefore, LMAM can potentially alleviate the over-fitting issue caused by a large number of parameters. Additionally, by computing and fusing the similarity of intra-modal and inter-modal features, LMAM can also fully exploit the intra-modal contextual information within each modality and the complementary semantic information across modalities (i.e., text, video and audio) simultaneously. Experimental results on some benchmark datasets show that LMAM can be embedded into any existing state-of-the-art DL-based CER methods and help boost their performance in a plug-and-play manner. Also, experimental results verify the superiority of LMAM compared with other popular cross-modal fusion methods. Moreover, LMAM is a general cross-modal fusion method and can thus be applied to other multi-modal recognition tasks, e.g., session recommendation and humour detection.
【6】 Towards Improving the Performance of Pre-Trained Speech Models for  Low-Resource Languages Through Lateral Inhibition标题：通过侧向抑制提高低资源语言语音模型的训练性能链接：https://arxiv.org/abs/2306.17792
作者：Andrei-Marius Avram,Răzvan-Alexandru Smădu,Vasile Păiş,Dumitru-Clementin Cercel,Radu Ion,Dan Tufiş机构：University Politehnica of Bucharest, Romania, †Research Institute for Artificial Intelligence “Mihai Dr˘ag˘anescu”, Romanian Academy备注：Accepted at TSP2023摘要：随着自然语言处理中Transformer模型的双向编码器表示的兴起，语音社区已经采用了他们的一些开发方法。因此，引入Wav2Vec模型以减少获得最新结果所需的数据。这项工作利用了这一知识，并通过简单地将微调密集层替换为受生物过程启发的侧抑制层来提高预训练语音模型的性能。我们的实验罗马尼亚语，一种低资源的语言，显示了平均改善12.5%的字错误率（WER）使用侧抑制层。此外，我们获得国家的最先进的结果，罗马尼亚语语音语料库和罗宾技术收购语料库与1.78%的WER和29.64%的WER，分别。摘要：With the rise of bidirectional encoder representations from Transformer models in natural language processing, the speech community has adopted some of their development methodologies. Therefore, the Wav2Vec models were introduced to reduce the data required to obtain state-of-the-art results. This work leverages this knowledge and improves the performance of the pre-trained speech models by simply replacing the fine-tuning dense layer with a lateral inhibition layer inspired by the biological process. Our experiments on Romanian, a low-resource language, show an average improvement of 12.5% word error rate (WER) using the lateral inhibition layer. In addition, we obtain state-of-the-art results on both the Romanian Speech Corpus and the Robin Technical Acquisition Corpus with 1.78% WER and 29.64% WER, respectively.
【7】 Empirical Interpretation of the Relationship Between Speech Acoustic  Context and Emotion Recognition标题：语音声学语境与情绪识别关系的实证研究链接：https://arxiv.org/abs/2306.17500
作者：Anna Ollerenshaw,Md Asif Jalal,Rosanna Milner,Thomas Hain机构：Speech and Hearing Research Group, Department of Computer Science, University of Sheffield, UK摘要：语音情感识别对于获得情感智力和理解语音的上下文意义至关重要。辅音-元音（CV）音位边界的变化可以丰富声学语境与语言线索，这影响SER。在实践中，语音情感被视为给定持续时间的声学段上的单个标签。然而，语音内的电话边界不是离散事件，因此感知的情感状态也应该分布在潜在的连续时间窗口上。  本研究采用基于注意力的方法，探讨了声学语境和音素边界对SER局部标记的影响。跨语料库分析实验的结果支持使用分布式方法的语音情感理解的好处。实验中，电话和单词被映射到注意力向量以及基本频率，以观察重叠的分布，从而声学上下文和情感之间的关系。这项工作的目的是桥梁心理语言学理论研究与计算建模SER。摘要：Speech emotion recognition (SER) is vital for obtaining emotional intelligence and understanding the contextual meaning of speech. Variations of consonant-vowel (CV) phonemic boundaries can enrich acoustic context with linguistic cues, which impacts SER. In practice, speech emotions are treated as single labels over an acoustic segment for a given time duration. However, phone boundaries within speech are not discrete events, therefore the perceived emotion state should also be distributed over potentially continuous time-windows.  This research explores the implication of acoustic context and phone boundaries on local markers for SER using an attention-based approach. The benefits of using a distributed approach to speech emotion understanding are supported by the results of cross-corpora analysis experiments. Experiments where phones and words are mapped to the attention vectors along with the fundamental frequency to observe the overlapping distributions and thereby the relationship between acoustic context and emotion. This work aims to bridge psycholinguistic theory research with computational modelling for SER.
【8】 Beyond-Voice: Towards Continuous 3D Hand Pose Tracking on Commercial  Home Assistant Devices标题：Beyond-Voice：商用家用辅助设备上的连续3D手势跟踪链接：https://arxiv.org/abs/2306.17477
作者：Yin Li,Rohan Reddy,Cheng Zhang,Rajalakshmi Nandakumar机构：Cornell University, United States摘要：越来越流行的家庭助理被广泛用作智能家庭设备的中央控制器。然而，当前的设计严重依赖语音接口，具有可访问性和可用性问题;一些最新的设备配备了额外的摄像机和显示器，这很昂贵，并引起了隐私问题。这些担忧共同推动了Beyond-Voice，这是一种新型的深度学习驱动的声学传感系统，允许家用助理设备连续跟踪和重建手部姿势。它使用现有的机载麦克风和扬声器将家庭助理转换为主动声纳系统。我们向深度学习模型提供了高分辨率的距离轮廓，该模型可以分析多个身体部位的运动并预测21个手指关节的3D位置，从而将声学手部跟踪的粒度提升到一个新的水平。它可以跨不同的环境和用户运行，而无需个性化的训练数据。在3种不同环境中对11名参与者进行的用户研究表明，Beyond-Voice可以在没有测试对象提供任何训练数据的情况下以16.47mm的平均绝对误差跟踪关节。摘要：Increasingly popular home assistants are widely utilized as the central controller for smart home devices. However, current designs heavily rely on voice interfaces with accessibility and usability issues; some latest ones are equipped with additional cameras and displays, which are costly and raise privacy concerns. These concerns jointly motivate Beyond-Voice, a novel deep-learning-driven acoustic sensing system that allows commodity home assistant devices to track and reconstruct hand poses continuously. It transforms the home assistant into an active sonar system using its existing onboard microphones and speakers. We feed a high-resolution range profile to the deep learning model that can analyze the motions of multiple body parts and predict the 3D positions of 21 finger joints, bringing the granularity for acoustic hand tracking to the next level. It operates across different environments and users without the need for personalized training data. A user study with 11 participants in 3 different environments shows that Beyond-Voice can track joints with an average mean absolute error of 16.47mm without any training data provided by the testing subject.
【9】 Audio Embeddings as Teachers for Music Classification标题：音频嵌入作为音乐分类的教师链接：https://arxiv.org/abs/2306.17424
作者：Yiwei Ding,Alexander Lerch机构：Music Informatics Group, Georgia Institute of Technology备注：Accepted at the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), 9 pages, 2 figures摘要：音乐分类一直是音乐信息检索领域的研究热点之一。随着深度学习模型的发展，过去十年在广泛的分类任务中取得了令人印象深刻的进步。然而，不断增加的模型复杂度使得训练和推理计算都很昂贵。在本文中，我们整合了迁移学习和基于特征的知识提取的思想，并系统地研究了使用预训练的音频嵌入作为教师来指导低复杂度学生网络的训练。通过用预训练的嵌入正则化学生网络的特征空间，可以将教师嵌入中的知识传递给学生。我们使用各种预训练的音频嵌入，并测试该方法在乐器分类和音乐自动标记任务上的有效性。结果表明，我们的方法显着改善的结果相比，没有教师的知识训练的相同的模型。该技术还可以与经典的知识蒸馏方法相结合，以进一步提高模型的性能。摘要：Music classification has been one of the most popular tasks in the field of music information retrieval. With the development of deep learning models, the last decade has seen impressive improvements in a wide range of classification tasks. However, the increasing model complexity makes both training and inference computationally expensive. In this paper, we integrate the ideas of transfer learning and feature-based knowledge distillation and systematically investigate using pre-trained audio embeddings as teachers to guide the training of low-complexity student networks. By regularizing the feature space of the student networks with the pre-trained embeddings, the knowledge in the teacher embeddings can be transferred to the students. We use various pre-trained audio embeddings and test the effectiveness of the method on the tasks of musical instrument classification and music auto-tagging. Results show that our method significantly improves the results in comparison to the identical model trained without the teacher's knowledge. This technique can also be combined with classical knowledge distillation approaches to further improve the model's performance.
【10】 Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion  Models标题：Diff-Foley：基于潜在扩散模型的同步视音频合成链接：https://arxiv.org/abs/2306.17203
作者：Simian Luo,Chuanhao Yan,Chenxu Hu,Hang Zhao机构：IIIS, Tsinghua University, Shanghai Qi Zhi Institute摘要：视频到音频（V2 A）模型最近因其在直接从无声视频生成音频（特别是在视频/电影制作中）中的实际应用而受到关注。然而，V2 A中的先前方法在时间同步和视听相关性方面具有有限的生成质量。我们提出了Diff-Foley，一个同步的视频到音频的合成方法与潜在的扩散模型（LDM），产生高品质的音频与改进的同步和视听相关性。我们采用对比视听预训练（CAVP），学习更多的时间和语义对齐的功能，然后训练一个LDM与CAVP对齐的视觉特征的频谱图的潜在空间。CAVP对齐的功能使LDM能够通过交叉注意模块捕获更微妙的视听相关性。我们通过“双重指导”进一步显著提高样品质量。Diff-Foley在当前大规模V2 A数据集上实现了最先进的V2 A性能。此外，我们证明了Diff-Foley的实用性和泛化能力，通过下游微调。项目页面：参见https://diff-foley.github.io/摘要：The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present Diff-Foley, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross-attention module. We further significantly improve sample quality with `double guidance'. Diff-Foley achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate Diff-Foley practical applicability and generalization capabilities via downstream finetuning. Project Page: see https://diff-foley.github.io/
机器翻译，仅供参考