今日论文合集：cs.SD语音24篇，eess.AS音频处理31篇。【1】 Multimodal Speech Recognition for Language-Guided Embodied Agents标题：面向语言制导的具身智能体的多模式语音识别链接：https://arxiv.org/abs/2302.14030
作者：Allen Chang,Xiaoyuan Zhu,Aarav Monga,Seoho Ahn,Tejas Srinivasan,Jesse Thomason机构：Department of Computer Science, University of Southern California, USA, Center for AI In Society’s Student Branch, University of Southern California, USA备注：5 pages, 5 figures摘要：语言引导的具体化代理的基准通常假定基于文本的指令，但是部署的代理将遇到口头指令。虽然自动语音识别（ASR）模型可以弥补输入差距，但错误的ASR转录会损害代理完成任务的能力。在这项工作中，我们提出训练一个多模态ASR模型，通过考虑伴随的视觉上下文来减少转录口头指令时的错误。我们在由ALFRED任务完成数据集合成的口语指令数据集上训练我们的模型，其中我们通过系统地屏蔽口语单词来模拟声学噪声。我们发现，利用视觉观察有利于掩蔽词的恢复，与单峰基线相比，多模态ASR模型恢复高达30%的掩蔽词。我们还发现，文本训练的具身智能体通过遵循多模态ASR模型的转录指令更频繁地成功完成任务。摘要：Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions. While Automatic Speech Recognition (ASR) models can bridge the input gap, erroneous ASR transcripts can hurt the agents' ability to complete tasks. In this work, we propose training a multimodal ASR model to reduce errors in transcribing spoken instructions by considering the accompanying visual context. We train our model on a dataset of spoken instructions, synthesized from the ALFRED task completion dataset, where we simulate acoustic noise by systematically masking spoken words. We find that utilizing visual observations facilitates masked word recovery, with multimodal ASR models recovering up to 30% more masked words than unimodal baselines. We also find that a text-trained embodied agent successfully completes tasks more often by following transcribed instructions from multimodal ASR models.【2】 Cross-modal Face- and Voice-style Transfer标题：跨模式脸型和语音型转换链接：https://arxiv.org/abs/2302.13838
作者：Naoya Takahashi,Mayank K. Singh,Yuki Mitsufuji机构：Sony Group Corporation, Japan备注：arXiv admin note: text overlap with arXiv:1912.01865 by other authors摘要：图像到图像的翻译和语音转换使得能够生成新的面部图像和语音，同时分别保持诸如图像中的姿势和音频中的语言内容之类的一些语义。它们可以帮助许多应用程序中的内容创建过程。然而，由于它们局限于每种模态内的转换，匹配所生成的面部和语音的印象仍然是一个未决问题。我们提出了一种称为XFaVoT的跨模态风格传输框架，该框架联合学习四个任务：具有音频或图像引导的图像翻译和语音转换任务，其使得能够生成”匹配给定语音的面部”和“匹配给定面部的语音”，以及具有单个框架的模态内翻译任务。在多个数据集上的实验结果表明，XFaVoT实现了跨模态风格的图像和语音翻译，在质量、多样性和人脸-语音对应性方面优于基线。摘要：Image-to-image translation and voice conversion enable the generation of a new facial image and voice while maintaining some of the semantics such as a pose in an image and linguistic content in audio, respectively. They can aid in the content-creation process in many applications. However, as they are limited to the conversion within each modality, matching the impression of the generated face and voice remains an open question. We propose a cross-modal style transfer framework called XFaVoT that jointly learns four tasks: image translation and voice conversion tasks with audio or image guidance, which enables the generation of ``face that matches given voice" and ``voice that matches given face", and intra-modality translation tasks with a single framework. Experimental results on multiple datasets show that XFaVoT achieves cross-modal style translation of image and voice, outperforming baselines in terms of quality, diversity, and face-voice correspondence.【3】 DST: Deformable Speech Transformer for Emotion Recognition标题：DST：用于情感识别的可变形语音转换器链接：https://arxiv.org/abs/2302.13729
作者：Weidong Chen,Xiaofen Xing,Xiangmin Xu,Jianxin Pang,Lan Du机构： School of Electronic and Information Engineering, South China University of Technology, China, School of Future Technology, South China University of Technology, China, Pazhou Laboratory, China, UBTECH Robotics Corp, China, iFLYTEK Research, China备注：5 pages, 4 figures, 2tables, accepted by ICASSP 2023摘要：Transformer在多人自注意的支持下，在语音情感识别（SER）中取得了显著的效果。与原有的全注意机制相比，基于窗口的注意机制在学习细粒度特征时更有效，同时大大减少了模型冗余。然而，情感线索以多粒度方式呈现，使得预定义固定窗口可严重降低模型灵活性。此外，很难手动获得最佳窗口设置。本文提出了一种用于SER任务的可变形语音Transformer DST。DST通过一个轻量级的判决网络确定以输入语音为条件的窗口大小的使用。同时，利用基于声学特征的数据相关偏移量来调整注意窗口的位置，使DST能够自适应地发现和注意语音中的有价值信息。在IEMOCAP和MELD上的大量实验证明了DST的优越性。摘要：Enabled by multi-head self-attention, Transformer has exhibited remarkable results in speech emotion recognition (SER). Compared to the original full attention mechanism, window-based attention is more effective in learning fine-grained features while greatly reducing model redundancy. However, emotional cues are present in a multi-granularity manner such that the pre-defined fixed window can severely degrade the model flexibility. In addition, it is difficult to obtain the optimal window settings manually. In this paper, we propose a Deformable Speech Transformer, named DST, for SER task. DST determines the usage of window sizes conditioned on input speech via a light-weight decision network. Meanwhile, data-dependent offsets derived from acoustic features are utilized to adjust the positions of the attention windows, allowing DST to adaptively discover and attend to the valuable information embedded in the speech. Extensive experiments on IEMOCAP and MELD demonstrate the superiority of DST.【4】 Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech标题：虚声：面向文语转换的人脸扩散模型链接：https://arxiv.org/abs/2302.13700
作者：Jiyoung Lee,Joon Son Chung,Soo-Whan Chung机构：NAVER AI Lab, South Korea, Korea Advanced Institute of Science and Technology, South Korea, NAVER Cloud, South Korea备注：ICASSP 2023. Project page: this https URL摘要：这项工作的目标是zero-shot的文本到语音合成，说话风格和声音学习的面部特征。受人们在看到某人的脸时可以想象其声音这一自然事实的启发，我们在一个从可见属性学习的统一框架内引入了一个人脸风格的扩散文本到语音（TTS）模型，称为Face-TTS。这是第一次使用人脸图像作为训练TTS模型的条件。 我们联合训练跨模型生物特征和TTS模型，以保持人脸图像和生成的语音片段之间的说话人身份。我们还提出了一个说话人特征绑定损失来增强生成语音片段和真实语音片段在说话人嵌入空间中的相似性。由于生物特征信息是直接从人脸图像中提取的，因此我们的方法不需要额外的微调步骤来从看不见和听不见的说话者生成语音。我们在LRS 3数据集上训练和评估该模型，该数据集是一个包含背景噪声和多种说话风格的野生视听语料库。该项目的网页是https://facetts.github.io。摘要：The goal of this work is zero-shot text-to-speech synthesis, with speaking styles and voices learnt from facial characteristics. Inspired by the natural fact that people can imagine the voice of someone when they look at his or her face, we introduce a face-styled diffusion text-to-speech (TTS) model within a unified framework learnt from visible attributes, called Face-TTS. This is the first time that face images are used as a condition to train a TTS model. We jointly train cross-model biometrics and TTS models to preserve speaker identity between face images and generated speech segments. We also propose a speaker feature binding loss to enforce the similarity of the generated and the ground truth speech segments in speaker embedding space. Since the biometric information is extracted directly from the face image, our method does not require extra fine-tuning steps to generate speech from unseen and unheard speakers. We train and evaluate the model on the LRS3 dataset, an in-the-wild audio-visual corpus containing background noise and diverse speaking styles. The project page is https://facetts.github.io.【5】 A Comparative Analysis Of Latent Regressor Losses For Singing Voice Conversion标题：歌唱嗓音转换中潜在阻滞器损耗的比较分析链接：https://arxiv.org/abs/2302.13678
作者：Brendan O'Connor,Simon Dixon机构：Centre For Digital Music, Queen Mary University of London备注：Submitted to the Sound and Music Computing Conference 2023摘要：先前的研究已经表明，当应用于歌声转换（SVC）时，用于口语转换（VC）的已建立的技术不能很好地执行。我们提出了一个替代损失的组成部分，在损失函数中，否则是建立良好的VC任务，这已被证明，以改善我们的模型的SVC性能。我们首先在歌手录音的梅尔谱图上训练歌手身份嵌入（SIE）网络，以使用对比学习产生歌手特定的方差编码。随后，我们在这些SIE的基础上训练了一个众所周知的自动编码器框架（AutoVC），并测量了使用不同潜在回归因子损失分量时SVC性能的差异。我们发现使用这个损失加权平均。SIE比w.r.t.具有更好的性能。瓶颈嵌入，其中转换后的音频对于目标歌手更自然且更特定。包含这个损失分量的优点是明确地迫使网络用音色相似性来重构，并且还否定了AutoVC的瓶颈嵌入中的不良解纠缠的影响。我们证明了计算和人的评价歌手转换音频剪辑之间的特殊多样性，这突出了两者的必要性。我们还提出了一种源歌手和目标歌手之间的音高匹配机制，以确保这些评估不受音高寄存器差异的影响。摘要：Previous research has shown that established techniques for spoken voice conversion (VC) do not perform as well when applied to singing voice conversion (SVC). We propose an alternative loss component in a loss function that is otherwise well-established among VC tasks, which has been shown to improve our model's SVC performance. We first trained a singer identity embedding (SIE) network on mel-spectrograms of singer recordings to produce singer-specific variance encodings using contrastive learning. We subsequently trained a well-known autoencoder framework (AutoVC) conditioned on these SIEs, and measured differences in SVC performance when using different latent regressor loss components. We found that using this loss w.r.t. SIEs leads to better performance than w.r.t. bottleneck embeddings, where converted audio is more natural and specific towards target singers. The inclusion of this loss component has the advantage of explicitly forcing the network to reconstruct with timbral similarity, and also negates the effect of poor disentanglement in AutoVC's bottleneck embeddings. We demonstrate peculiar diversity between computational and human evaluations on singer-converted audio clips, which highlights the necessity of both. We also propose a pitch-matching mechanism between source and target singers to ensure these evaluations are not influenced by differences in pitch register.【6】 Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognition标题：辅助任务在Wav2vec 2.0和ERT多模式融合中的多模式情感识别链接：https://arxiv.org/abs/2302.13661
作者：Dekai Sun,Yancheng He,Jiqing Han机构：Harbin Institute of Technology, Harbin, China摘要：数据的缺乏和多模态融合的困难一直是多模态情感识别所面临的挑战。本文提出使用预训练的模型作为上游网络，wav2vec 2.0用于音频模态，BERT用于文本模态，并在MER的下游任务中对它们进行微调以应对数据不足的问题。针对多模式融合的困难性，采用K层多头注意机制作为下游融合模块。从MER任务本身出发，设计了两个辅助任务，以缓解模态间融合不足的问题，引导网络捕捉和对齐情绪相关特征。在IEMOCAP数据集上，与已有的模型相比，加权准确率（WA）和未加权准确率（UA）分别提高了78.42%和79.71%。摘要：The lack of data and the difficulty of multimodal fusion have always been challenges for multimodal emotion recognition (MER). In this paper, we propose to use pretrained models as upstream network, wav2vec 2.0 for audio modality and BERT for text modality, and finetune them in downstream task of MER to cope with the lack of data. For the difficulty of multimodal fusion, we use a K-layer multi-head attention mechanism as a downstream fusion module. Starting from the MER task itself, we design two auxiliary tasks to alleviate the insufficient fusion between modalities and guide the network to capture and align emotion-related features. Compared to the previous state-of-the-art models, we achieve a better performance by 78.42% Weighted Accuracy (WA) and 79.71% Unweighted Accuracy (UA) on the IEMOCAP dataset.【7】 Predicting EEG Responses to Attended Speech via Deep Neural Networks for Speech标题：基于语音深度神经网络的听音脑电反应预测链接：https://arxiv.org/abs/2302.13553
作者：Emina Alickovic,Tobias Dorszewski,Thomas U. Christiansen,Kasper Eskelund,Leonardo Gizzi,Martin A. Skoglund,Dorothea Wendt机构： are with the Institute for Modellingand Simulation of Biomechanical Systems, University of Stuttgart摘要：在多说话者环境中关注感兴趣的语音流可能是一项具有挑战性的任务，特别是对于听力受损的收听者。研究表明，用脑电图（EEG）评估的神经反应受听者听觉注意的调节，揭示了对所听语音的选择性神经跟踪（NT）。NT方法大多依赖于人工设计的声学和语言语音特征来预测神经响应。直到最近，没有特定语言信息的深度神经网络（DNN）模型才被用于提取NT的语音特征，证明了分级DNN层中的语音特征可以预测整个听觉通路的神经响应。在本研究中，我们更进一步地研究了相似的DNN语音模型对预测EEG中观察到的竞争语音的神经反应的适用性。我们使用64通道采集系统记录了17名听力正常的听者的脑电图数据，这些听者被指示关注两个竞争说话者中的一个。我们的数据显示，与人工设计的声学特征相比，DNN提取的语音特征能显著更好地预测EEG反应。此外，对分级DNN层的分析表明，早期层产生最高的预测。此外，我们发现使用DNN提取的语音特征比使用手工设计的声学特征在听觉注意力分类准确性方面有显著提高。这些发现为制定新的NT措施以评估和进一步推进听力技术开辟了新途径。摘要：Attending to the speech stream of interest in multi-talker environments can be a challenging task, particularly for listeners with hearing impairment. Research suggests that neural responses assessed with electroencephalography (EEG) are modulated by listener`s auditory attention, revealing selective neural tracking (NT) of the attended speech. NT methods mostly rely on hand-engineered acoustic and linguistic speech features to predict the neural response. Only recently, deep neural network (DNN) models without specific linguistic information have been used to extract speech features for NT, demonstrating that speech features in hierarchical DNN layers can predict neural responses throughout the auditory pathway. In this study, we go one step further to investigate the suitability of similar DNN models for speech to predict neural responses to competing speech observed in EEG. We recorded EEG data using a 64-channel acquisition system from 17 listeners with normal hearing instructed to attend to one of two competing talkers. Our data revealed that EEG responses are significantly better predicted by DNN-extracted speech features than by hand-engineered acoustic features. Furthermore, analysis of hierarchical DNN layers showed that early layers yielded the highest predictions. Moreover, we found a significant increase in auditory attention classification accuracies with the use of DNN-extracted speech features over the use of hand-engineered acoustic features. These findings open a new avenue for development of new NT measures to evaluate and further advance hearing technology.【8】 Continuous descriptor-based control for deep audio synthesis标题：基于连续描述符的深度音频合成控制链接：https://arxiv.org/abs/2302.13542
作者：Ninon Devis,Nils Demerlé,Sarah Nabi,David Genova,Philippe Esling机构：IRCAM - Sorbonne Universit´e, CNRS UMR , place Igor Stravinsky, Paris, France备注：ICASSP 2023摘要：尽管在用于音乐生成的深度模型方面取得了重大进展，但这些技术的使用仍然限于专家用户。在音乐家中普及之前，生成模型必须首先提供对一代人的表达控制，因为这是深度生成模型在创作工作流中整合的条件。在本文中，我们通过引入一种深度生成音频模型来解决这一问题，该模型提供表达性和连续的基于描述符的控制，同时保持足够的轻量级以嵌入硬件合成器。我们使用对抗性混淆准则，通过显式地去除潜在空间中的显著音乐特征来增强实时生成的可控性。然后，用户指定的特征作为附加的调节信息被重新引入，允许对生成进行连续控制，类似于合成器旋钮。我们评估了我们的方法在各种各样的声音上的表现，包括乐器，打击乐和语音录音，同时提供音色和属性传输，允许新的声音生成方法。摘要：Despite significant advances in deep models for music generation, the use of these techniques remains restricted to expert users. Before being democratized among musicians, generative models must first provide expressive control over the generation, as this conditions the integration of deep generative models in creative workflows. In this paper, we tackle this issue by introducing a deep generative audio model providing expressive and continuous descriptor-based control, while remaining lightweight enough to be embedded in a hardware synthesizer. We enforce the controllability of real-time generation by explicitly removing salient musical features in the latent space using an adversarial confusion criterion. User-specified features are then reintroduced as additional conditioning information, allowing for continuous control of the generation, akin to a synthesizer knob. We assess the performance of our method on a wide variety of sounds including instrumental, percussive and speech recordings while providing both timbre and attributes transfer, allowing new ways of generating sounds.【9】 VE-KWS: Visual Modality Enhanced End-to-End Keyword Spotting标题：VE-KWS：视觉通道增强型端到端关键词定位链接：https://arxiv.org/abs/2302.13523
作者：Ao Zhang,He Wang,Pengcheng Guo,Yihui Fu,Lei Xie,Yingying Gao,Shilei Zhang,Junlan Feng机构：Northwestern Polytechnical University, Xi’an, China, China Mobile Research Institute, Beijing, China备注：5 pages. Accepted at ICASSP2023摘要：基于音频模态的关键词识别（KWS）系统在远场和噪声环境下性能显著下降，通常以虚警和拒识来衡量。因此，利用多种模态之间的互补关系的视听关键词识别最近获得了很多关注。然而，目前的研究主要集中在整合不同模态的专门学习表征，而不是在每个单独的建模过程中探索模态关系。本文提出了一种新的视觉通道增强的端到端知识共享系统框架（VE-KWS），从两个方面融合音频和视觉通道。第一种是利用从视频中的嘴唇区域获取的说话人位置信息来辅助多通道音频波束形成器的训练。通过将波束形成器用作音频增强模块，可以显著地抑制由远场或噪声环境引起的声学失真。另一种方法是在不同的通道之间进行交叉注意，以捕捉通道间的关系，并帮助每一个通道的表征学习。在MSIP挑战语料库上的实验表明，该模型在Eval集上获得了2. 79%的误拒率和2. 95%的误警率，与ICASSP 2022 MISP挑战中排名靠前的系统相比，具有更好的SOTA性能.摘要：The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting cross-attention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves 2.79% false rejection rate and 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge.【10】 3D Neural Beamforming for Multi-channel Speech Separation Against Location Uncertainty标题：位置不确定条件下的三维神经波束形成多通道语音分离链接：https://arxiv.org/abs/2302.13462
作者：Rongzhi Gu,Shi-Xiong Zhang,Dong Yu机构： Tencent AI Lab, Shenzhen, China, Tencent AI Lab, Seattle, WA, USA摘要：利用说话人方向信息进行多通道语音分离比盲语音分离有明显的优势。但是，它有两个局限性。第一，当两个声音的到来方向接近时，观察到显著的性能劣化。第二，结果高度依赖于对说话者方向的精确估计。为了克服这些问题，本文提出了一种用于多通道语音分离的三维特征和相关的三维神经波束形成器。在这一领域的先前工作在两个重要方向上得到了扩展。首先，将传统的1D方向性波束图案推广到3D。这使得模型能够从3D空间中的任何目标区域提取语音。因此，具有相似方向但不同高度或距离的扬声器变得可分离。其次，为了处理说话人位置的不确定性，将先前提出的空间特征扩展为一种新的3D区域特征。提出的三维区域特征和三维神经波束形成器在车内场景下进行了评估。实验结果表明，三维特征和三维波束形成器相结合可以获得与以真实说话人位置作为输入的分离模型相当的性能。摘要：Multi-channel speech separation using speaker's directional information has demonstrated significant gains over blind speech separation. However, it has two limitations. First, substantial performance degradation is observed when the coming directions of two sounds are close. Second, the result highly relies on the precise estimation of the speaker's direction. To overcome these issues, this paper proposes 3D features and an associated 3D neural beamformer for multi-channel speech separation. Previous works in this area are extended in two important directions. First, the traditional 1D directional beam patterns are generalized to 3D. This enables the model to extract speech from any target region in the 3D space. Thus, speakers with similar directions but different elevations or distances become separable. Second, to handle the speaker location uncertainty, previously proposed spatial feature is extended to a new 3D region feature. The proposed 3D region feature and 3D neural beamformer are evaluated under an in-car scenario. Experimental results demonstrated that the combination of 3D feature and 3D beamformer can achieve comparable performance to the separation model with ground truth speaker location as input.【11】 Low latency transformers for speech processing标题：用于语音处理的低延迟转换器链接：https://arxiv.org/abs/2302.13451
作者：Jianbo Ma,Siqi Pan,Deepak Chandran,Andrea Fanelli,Richard Cartwright机构：Dolby Laboratories备注：6 pages, 3 figures摘要：Transformer是现代神经网络中广泛使用的构建模块。然而，当应用于音频数据时，Transformer的因果行为，我们称之为无因注意（AA），通常将其应用限于离线任务。本文介绍了流注意（SA）算法，该算法具有固定的延迟，并且训练所需的计算和内存资源比AA算法要少。接下来，我们介绍低延迟流注意（LLSA），这是一种组合多个SA层而不产生与层数成比例的延迟累积的方法。对自动语音识别（ASR）和语音情感识别（SER）任务中的AA、SA和LLSA进行了比较分析。结果表明，具有几秒（例如1.8秒）的固定延迟的基于因果SA的网络和具有短至300ms的延迟的LLSA网络可以与因果（AA）网络相当地执行。我们得出结论，SA和LLSA方法保留了传统因果Transformers的许多优点，但具有延迟特性，使它们能够在实时流应用中运行。摘要：The transformer is a widely-used building block in modern neural networks. However, when applied to audio data, the transformer's acausal behaviour, which we term Acausal Attention (AA), has generally limited its application to offline tasks. In this paper we introduce Streaming Attention (SA), which operates causally with fixed latency, and requires lower compute and memory resources than AA to train. Next, we introduce Low Latency Streaming Attention (LLSA), a method which combines multiple SA layers without latency build-up proportional to the layer count. Comparative analysis between AA, SA and LLSA on Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER) tasks are presented. The results show that causal SA-based networks with fixed latencies of a few seconds (e.g. 1.8 seconds) and LLSA networks with latencies as short as 300 ms can perform comparably with acausal (AA) networks. We conclude that SA and LLSA methods retain many of the benefits of conventional acausal transformers, but with latency characteristics that make them practical to run in real-time streaming applications.【12】 From Audio to Symbolic Encoding标题：从音频编码到符号编码链接：https://arxiv.org/abs/2302.13401
作者：Shenli Yuan,Lingjie Kong,Jiushuang Guo机构：Department of Mechanical Engineering, Center for Computer Research in Music and Acoustics (CCRMA), Stanford University, Department of Computer Science, Department of Statistics摘要：自动音乐转录（AMT）旨在将原始音频转换为符号化的音乐表示。作为音乐信息检索的一个基本问题，由于声学信号中的多个谐波的重叠，即使对于受过训练的人类专家来说，AMT也被认为是一个困难的任务。另一方面，语音识别作为自然语言处理中最热门的任务之一，旨在将人类的口语翻译成文本。基于AMT和语音识别的相似性质（因为它们都处理将音频信号转换为符号编码的任务），本文研究了通用神经网络架构是否可能同时处理这两个任务。本文介绍了我们在现有Onsets和Frames基础上构建的新的神经网络结构，并比较了其多种变体在AMT任务上的性能。我们还用语音识别任务测试了我们的体系结构。对于AMT，与使用最先进架构训练的模型相比，我们的模型能够产生更好的结果;然而，尽管类似的体系结构能够在语音识别任务上被训练，但是与其它任务特定的模型相比，它没有产生非常理想的结果。摘要：Automatic music transcription (AMT) aims to convert raw audio to symbolic music representation. As a fundamental problem of music information retrieval (MIR), AMT is considered a difficult task even for trained human experts due to overlap of multiple harmonics in the acoustic signal. On the other hand, speech recognition, as one of the most popular tasks in natural language processing, aims to translate human spoken language to texts. Based on the similar nature of AMT and speech recognition (as they both deal with tasks of translating audio signal to symbolic encoding), this paper investigated whether a generic neural network architecture could possibly work on both tasks. In this paper, we introduced our new neural network architecture built on top of the current state-of-the-art Onsets and Frames, and compared the performances of its multiple variations on AMT task. We also tested our architecture with the task of speech recognition. For AMT, our models were able to produce better results compared to the model trained using the state-of-art architecture; however, although similar architecture was able to be trained on the speech recognition task, it did not generate very ideal result compared to other task-specific models.【13】 Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics标题：音乐中的多模式：从高水平的音频特征和歌词预测音乐中的情感链接：https://arxiv.org/abs/2302.13321
作者：Tibor Krols,Yana Nikolova,Ninell Oldenburg机构：University of Copenhagen, Department of Nordic Studies and Linguistics, and Department of Computer Science备注：12 pages, incl. 2 pages appendix摘要：本文旨在测试一种多模态的音乐情感识别方法是否比单模态的方法在高层次的歌曲特征和歌词上有更好的表现。我们使用从Spotify API检索到的11首歌曲特征，结合歌词特征（包括情感、TF-IDF和Anew）来预测Deezer情绪检测数据集（DMDD）上的效价和唤醒（Russell，1980）评分（Delbouys等人，2018年），采用4种不同的回归模型。我们发现，在11个高层次的歌曲特征中，主要有5个对表现有贡献，多模态特征在预测效价时比单独的音频特征做得更好。我们公开了我们的代码。摘要：This paper aims to test whether a multi-modal approach for music emotion recognition (MER) performs better than a uni-modal one on high-level song features and lyrics. We use 11 song features retrieved from the Spotify API, combined lyrics features including sentiment, TF-IDF, and Anew to predict valence and arousal (Russell, 1980) scores on the Deezer Mood Detection Dataset (DMDD) (Delbouys et al., 2018) with 4 different regression models. We find that out of the 11 high-level song features, mainly 5 contribute to the performance, multi-modal features do better than audio alone when predicting valence. We made our code publically available.【14】 Implementation of an aeroacoustic simulation pipeline using openCFS-Acoustics and openCFS-Data applied to human phonation标题：利用OpenCFS声学和OpenCFS数据实现用于人体发声的气动声学模拟管道链接：https://arxiv.org/abs/2302.13290
作者：Stefan Schoder机构：Group of Aeroacoustics and Vibroacoustics, IGTE, TU Graz, Inffeldgasse , Graz备注：9 pages, 4 figures摘要：人类发声过程可以使用有限元方法（FEM）来建模，该有限元方法提供了语音产生过程的详细表示。用C++语言编写的有限元软件（openCFS）模拟了发声过程。FEM模型由人体上呼吸道的3D网格组成。simVoice模型提供了发声过程的准确表示，并在一些出版物中有效。在本文中，我们将展示如何使用openCFS和openCFS-Data建立模型。摘要：The human phonation process be modeled using the Finite Element Method (FEM) which provides a detailed representation of the voice production process. A software implementation in C++ using FEM (openCFS) has been used to simulate the phonation process. The FEM model consists of a 3D mesh of the upper human airways. The simVoice model provides an accurate representation of the phonation process and was valid in several publications. In this article, we show how to set up the model using openCFS and openCFS-Data.【15】 Contrast-PLC: Contrastive Learning for Packet Loss Concealment标题：对比度-PLC：分组丢失隐藏的对比学习链接：https://arxiv.org/abs/2302.13284
作者：Huaying Xue,Xiulian Peng,Yan Lu机构：Microsoft Research Asia, Beijing, China备注：5 pages, ICASSP 2023(Accepted)摘要：当只有有限的可用上下文可供使用时，分组丢失隐藏（PLC）在真实地和自然地隐藏丢失内容方面具有挑战性。最近，基于深度学习的PLC算法已经证明了它们优于传统对应物;但是它们的隐藏能力仍然主要限于最大120 ms的丢失。即使使用强大的基于GAN的生成模型，预测可能发生在音素内/之间的长突发丢失仍然是非常具有挑战性的。在这篇文章中，我们提出使用对比学习来学习一个损失鲁棒的PLC语义表示。设计了一种结合语义预测和基于遗传网络的生成模型的混合神经元PLC体系结构，验证了该方法的有效性。在Interspeech 2022 PLC Challenge的盲测试集上的测试结果表明，该方法优于常用的UNet式框架和无对比学习的框架，尤其是在（120，220] ms的较长突发丢失方面。摘要：Packet loss concealment (PLC) is challenging in concealing missing contents both plausibly and naturally when there are only limited available context to use. Recently deep-learning based PLC algorithms have demonstrated their superiority over traditional counterparts; but their concealment ability is still mostly limited to a maximum of 120ms loss. Even with strong GAN-based generative models, it is still very challenging to predict long burst losses that could happen within/in-between phonemes. In this paper, we propose to use contrastive learning to learn a loss-robust semantic representation for PLC. A hybrid neural PLC architecture combining the semantic prediction and GAN-based generative model is designed to verify its effectiveness. Results on the blind test set of Interspeech2022 PLC Challenge show its superiority over commonly used UNet-style framework and the one without contrastive learning, especially for the longer burst loss at (120, 220] ms.【16】 Mingling or Misalignment? Temporal Shift for Speech Emotion Recognition with Pre-trained Representations标题：混搭还是错位？基于预训练表征的语音情感识别时移方法链接：https://arxiv.org/abs/2302.13277
作者：Siyuan Shen,Feng Liu,Aimin Zhou机构：East China Normal University, Shanghai, China, Shanghai Institute of AI for Education, Shanghai, China备注：Accepted by 48th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)摘要：随着自监督模型的发展，预训练的语音表示被证明对下游的语音情感识别（SER）任务是有效的。大多数现有的工作主要集中在利用预先训练的表示，并且仅仅在预先训练的模型之上采用线性头，而忽略了下游网络的设计。在这篇论文中，我们提出了一个时间移位模块来混合通道信息，而不需要引入任何参数或FLOP。通过时间移位模块，三个设计的基线构建块演变成相应的移位变体，即ShiftCNN、ShiftLSTM和Shiftformer。此外，为了平衡混合和错位之间的权衡，我们提出了两种技术策略，移位的位置和移位的比例。在微调和特征提取设置下，时间移位模型家族在基准IEMOCAP数据集上的性能都优于最先进的方法。我们的代码可在www.example.com上获得https://github.com/ECNU-Cross-Innovation-Lab/ShiftSER。摘要：Fueled by recent advances of self-supervised models, pre-trained speech representations proved effective for the downstream speech emotion recognition (SER) task. Most prior works mainly focus on exploiting pre-trained representations and just adopt a linear head on top of the pre-trained model, neglecting the design of the downstream network. In this paper, we propose a temporal shift module to mingle channel-wise information without introducing any parameter or FLOP. With the temporal shift module, three designed baseline building blocks evolve into corresponding shift variants, i.e. ShiftCNN, ShiftLSTM, and Shiftformer. Moreover, to balance the trade-off between mingling and misalignment, we propose two technical strategies, placement of shift and proportion of shift. The family of temporal shift models all outperforms the state-of-the-art methods on the benchmark IEMOCAP dataset under both finetuning and feature extraction settings. Our code is available at https://github.com/ECNU-Cross-Innovation-Lab/ShiftSER.【17】 Two-Stream Joint-Training for Speaker Independent Acoustic-to-Articulatory Inversion标题：非说话人声律倒置的双流联合训练链接：https://arxiv.org/abs/2302.13273
作者：Jianrong Wang,Jinyu Liu,Li Liu,Xuewei Li,Mei Yu,Jie Gao,Qiang Fang机构： College of Intelligence and Computing, Tianjin University, Tianjin, China, Tianjin International Engineering Institute, Tianjin University, Tianjin, China, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China摘要：声学到发音反演（AAI）旨在从语音音频中估计发音器官的参数。AAI面临着两个共同的挑战，即有限的数据和在非特定人场景下的性能不佳。目前的研究大多集中于直接从语音中提取特征，而忽略了音素信息的重要性，这可能会限制AAI的性能。为此，我们提出了一种新的网络称为SPN，使用两个不同的流来执行AAI任务。首先，为了提高非特定人语音识别实验的性能，我们提出了一种新的音素流网络来估计发音参数作为音素特征。据我们所知，这是第一个从音素中提取非特定人特征来提高AAI性能的工作。其次，为了更好地表达语音信息，我们训练了一个语音流网络，将局部特征和全局特征结合起来。在非特定人实验中，与现有的SOTA方法相比，该方法的RMSE降低了0.18mm，Pearson相关系数提高了6.0%。该代码已在www.example.com上发布。https://github.com/liujinyu123/AAINetwork-SPN.摘要：Acoustic-to-articulatory inversion (AAI) aims to estimate the parameters of articulators from speech audio. There are two common challenges in AAI, which are the limited data and the unsatisfactory performance in speaker independent scenario. Most current works focus on extracting features directly from speech and ignoring the importance of phoneme information which may limit the performance of AAI. To this end, we propose a novel network called SPN that uses two different streams to carry out the AAI task. Firstly, to improve the performance of speaker-independent experiment, we propose a new phoneme stream network to estimate the articulatory parameters as the phoneme features. To the best of our knowledge, this is the first work that extracts the speaker-independent features from phonemes to improve the performance of AAI. Secondly, in order to better represent the speech information, we train a speech stream network to combine the local features and the global features. Compared with state-of-the-art (SOTA), the proposed method reduces 0.18mm on RMSE and increases 6.0% on Pearson correlation coefficient in the speaker-independent experiment. The code has been released at https://github.com/liujinyu123/AAINetwork-SPN.【18】 Speech Corpora Divergence Based Unsupervised Data Selection for ASR标题：基于语料库离散度的ASR非监督数据选择链接：https://arxiv.org/abs/2302.13222
作者：Changfeng Gao,Gaofeng Cheng,Pengyuan Zhang,Yonghong Yan机构： [ 1 1] usesa baseline ASR model to decode the N-Best hypothesis andthen calculate the term frequency-inverse document frequencyThe authors are with the Key Laboratory of Speech Acoustics andContent Understanding, Institute of Acoustics摘要：应用场景匹配数据的选取是自动语音识别（ASR）训练的重要内容，但训练语料库的匹配程度难以度量。提出了一种基于语料库散度的无监督目标感知数据选择方法，该方法可以度量两个语料库之间的相似性。首先利用自监督Hubert模型将语音语料离散化为标签序列，计算N-gram概率分布。然后计算N元文法之间的Kullback-Leibler散度作为SCD。最后，选择与目标语料库具有最小SCD的子集进行标注和训练。与以往的数据选择方法相比，SCD数据选择方法能够关注更多的声学细节，保证选择集的多样性。我们在来自共同之声的不同口音上评估我们的方法。实验结果表明，提出的SCD数据选择方法相对于随机选择方法的性能提高了14.8%，与监督选择方法的效果相当甚至更好。摘要：Selecting application scenarios matching data is important for the automatic speech recognition (ASR) training, but it is difficult to measure the matching degree of the training corpus. This study proposes a unsupervised target-aware data selection method based on speech corpora divergence (SCD), which can measure the similarity between two speech corpora. We first use the self-supervised Hubert model to discretize the speech corpora into label sequence and calculate the N-gram probability distribution. Then we calculate the Kullback-Leibler divergence between the N-grams as the SCD. Finally, we can choose the subset which has minimum SCD to the target corpus for annotation and training. Compared to previous data selection method, the SCD data selection method can focus on more acoustic details and guarantee the diversity of the selected set. We evaluate our method on different accents from Common Voice. Experiments show that the proposed SCD data selection can realize 14.8% relative improvements to the random selection, comparable or even superior to the result of supervised selection.【19】 Speaker Recognition in Realistic Scenario Using Multimodal Data标题：基于多模式数据的真实场景说话人识别链接：https://arxiv.org/abs/2302.13033
作者：Saqlain Hussain Shah,Muhammad Saad Saeed,Shah Nawaz,Muhammad Haroon Yousaf机构：University of Engineering and Technology Taxila,Swarm Robotics Lab NCRA, Deutsches Elektronen-Synchrotron DESY备注：Accepted at the International Conference on Artificial Intelligence (ICAI'2023)摘要：近年来，利用来自YouTube的大规模视听信息，在名人的面部和声音之间建立了关联。大规模视听数据集的可用性有助于开发基于标准卷积神经网络的说话人识别方法。因此，本文的目的是利用大规模的音视频信息来提高说话人识别的性能。为了达到这个目的，我们提出了一个双分支网络来学习多模态系统中人脸和语音的联合表征。然后，从该两分支网络中提取特征来训练用于说话人识别的分类器。我们在一个名为VoxCeleb$1$的大规模视听数据集上评估了我们提出的框架。实验结果表明，人脸信息的加入提高了说话人识别的性能。此外，我们的结果表明，有一个重叠的面孔和声音。摘要：In recent years, an association is established between faces and voices of celebrities leveraging large scale audio-visual information from YouTube. The availability of large scale audio-visual datasets is instrumental in developing speaker recognition methods based on standard Convolutional Neural Networks. Thus, the aim of this paper is to leverage large scale audio-visual information to improve speaker recognition task. To achieve this task, we proposed a two-branch network to learn joint representations of faces and voices in a multimodal system. Afterwards, features are extracted from the two-branch network to train a classifier for speaker recognition. We evaluated our proposed framework on a large scale audio-visual dataset named VoxCeleb$1$. Our results show that addition of facial information improved the performance of speaker recognition. Moreover, our results indicate that there is an overlap between face and voice.【20】 Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing标题：联合优化翻译和语音计时提高自动配音的等时性链接：https://arxiv.org/abs/2302.12979
作者：Alexandra Chronopoulou,Brian Thompson,Prashant Mathur,Yogesh Virkar,Surafel M. Lakew,Marcello Federico机构：AWS AI Labs, Center for Information and Language Processing, LMU Munich, Germany备注：5 pages摘要：自动配音（AD）是将视频中的原始语音翻译成目标语言语音的任务。新的目标语语音应满足等时性;也就是说，新的语音应该与原始视频在时间上对齐，包括嘴部运动、停顿、手势等。在本文中，我们建议训练一个模型，该模型可以直接优化翻译以及生成的翻译的语音持续时间。我们表明，与以前的工作相比，该系统生成的语音更好地匹配原始语音的定时，同时简化了系统架构。摘要：Automatic dubbing (AD) is the task of translating the original speech in a video into target language speech. The new target language speech should satisfy isochrony; that is, the new speech should be time aligned with the original video, including mouth movements, pauses, hand gestures, etc. In this paper, we propose training a model that directly optimizes both the translation as well as the speech duration of the generated translations. We show that this system generates speech that better matches the timing of the original speech, compared to prior work, while simplifying the system architecture.【21】 Pre-Finetuning for Few-Shot Emotional Speech Recognition标题：Few-Shot情感语音识别的预调优链接：https://arxiv.org/abs/2302.12921
作者：Maximillian Chen,Zhou Yu机构：Columbia University备注：5 pages, 4 figures. Code available at this https URL摘要：语音模型长期以来已知对于许多分类任务来说过拟合个体说话者。这导致在扬声器位于域外或分布外的设置中泛化效果较差，这在生产环境中很常见。我们把说话人适应看作是一个Few-Shot的学习问题，并建议调查迁移学习方法的灵感来自最近成功的预训练模型在自然语言任务。我们提出在困难的任务上预先微调语音模型，以将知识提取到Few-Shot的下游分类目标。我们在四个多类情感语音识别语料库的每个排列上预微调Wav2Vec2.0，并通过在情感语音数据集上进行33，600次Few-Shot微调试验来评估我们的预微调模型。摘要：Speech models have long been known to overfit individual speakers for many classification tasks. This leads to poor generalization in settings where the speakers are out-of-domain or out-of-distribution, as is common in production environments. We view speaker adaptation as a few-shot learning problem and propose investigating transfer learning approaches inspired by recent success with pre-trained models in natural language tasks. We propose pre-finetuning speech models on difficult tasks to distill knowledge into few-shot downstream classification objectives. We pre-finetune Wav2Vec2.0 on every permutation of four multiclass emotional speech recognition corpora and evaluate our pre-finetuned models through 33,600 few-shot fine-tuning trials on the Emotional Speech Dataset.【22】 Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator标题：使用集成文本到MEL语谱图生成器的端到端ASR的纯文本域适配链接：https://arxiv.org/abs/2302.14036
作者：Vladimir Bataev,Roman Korostik,Evgeny Shabalin,Vitaly Lavrukhin,Boris Ginsburg机构：NVIDIA, University of London, UK;, ITMO University, Russia;, Higher School of Economics, Russia摘要：我们提出了一种端到端的ASR系统，它可以在转录的语音数据、文本数据或两者的混合上进行训练。对于纯文本训练，我们的扩展ASR模型使用了一个集成的辅助TTS模块，该模块从文本创建mel谱图。该模块包含传统的非自回归文本到梅尔谱图生成器，该生成器用GAN增强器增强以提高谱图质量。所提出的系统可以通过使用纯文本数据来提高ASR模型在新领域上的准确性，并且允许通过使用大文本语料库来显著优于传统的音频文本训练。摘要：We propose an end-to-end ASR system that can be trained on transcribed speech data, text data, or a mixture of both. For text-only training, our extended ASR model uses an integrated auxiliary TTS block that creates mel spectrograms from the text. This block contains a conventional non-autoregressive text-to-mel-spectrogram generator augmented with a GAN enhancer to improve the spectrogram quality. The proposed system can improve the accuracy of the ASR model on a new domain by using text-only data, and allows to significantly surpass conventional audio-text training by using large text corpora.【23】 A Deep Neural Network Based Reverse Radio Spectrogram Search Algorithm标题：一种基于深度神经网络的反向射电频谱搜索算法链接：https://arxiv.org/abs/2302.13854
作者：Peter Xiangyuan Ma,Steve Croft,Andrew P. V. Siemion机构：Department of Mathematics, University of Toronto, St. George Street, Toronto, ON M,S ,E, Canada, Radio Astronomy Laboratory, Campbell Hall, University of California, Berkeley, CA , USA, SETI Institute, Mountain View, CA , USA备注：8 pages, 8 figures摘要：我们开发了一种快速的模块化深度学习算法，用于在无线电频谱图数据中搜索感兴趣的相似信号。首先，我们根据能量检测算法返回的滤波数据训练自动编码器。然后，我们将位置嵌入层从经典的Transformer架构调整为基于频率的嵌入。接下来，我们使用自动编码器的编码器组件从无线电频谱图的小窗口（~ 715，Hz，每个频率窗口的分辨率为2.79 Hz）中提取特征。我们使用我们的算法在一组信号（搜索项目的编码特征）上对给定查询（感兴趣的编码信号）进行搜索，以产生具有相似特征的最佳候选。我们成功地证明了该算法只在给定原始无线电频谱图数据的情况下检索到具有相似外观的信号。摘要：We developed a fast and modular deep learning algorithm to search for lookalike signals of interest in radio spectrogram data. First, we trained an autoencoder on filtered data returned by an energy detection algorithm. We then adapted a positional embedding layer from classical Transformer architecture to a frequency-based embedding. Next we used the encoder component of the autoencoder to extract features from small (~ 715,Hz with a resolution of 2.79Hz per frequency bin) windows in the radio spectrogram. We used our algorithm to conduct a search for a given query (encoded signal of interest) on a set of signals (encoded features of searched items) to produce the top candidates with similar features. We successfully demonstrate that the algorithm retrieves signals with similar appearance, given only the original radio spectrogram data.【24】 Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow标题：VariancFlow：通过归一化流程利用方差信息实现高质量、可控的文语转换链接：https://arxiv.org/abs/2302.13458
作者：Yoonhyung Lee,Jinhyeok Yang,Kyomin Jung机构：Seoul National University, Dept. of Electrical and Computer Engineering, Republic of Korea, Speech AI Lab, NCSOFT, Republic of Korea备注：Accepted for ICASSP 2022摘要：非自回归文语转换模型可以有效地学习文语之间的一对多关系，其方法有两种。第一种方法是使用一个先进的生成式框架，如规范化流（NF）。第二种是在生成语音时一起使用诸如音调或能量的方差信息。对于第二种类型，还可以通过调整提供给模型的方差值来控制方差因子。本文结合这两种模型的优点，提出了一种新的模型--方差流模型。通过使用NF对方差建模，VarianceFlow可以更精确地预测方差信息，从而提高语音质量。同时，NF的目标函数使得模型能够以一种非纠缠的方式使用方差信息和文本，从而实现更精确的方差控制。实验结果表明，VarianceFlow在语音质量和可控性方面均优于现有的TTS模型。摘要：There are two types of methods for non-autoregressive text-to-speech models to learn the one-to-many relationship between text and speech effectively. The first one is to use an advanced generative framework such as normalizing flow (NF). The second one is to use variance information such as pitch or energy together when generating speech. For the second type, it is also possible to control the variance factors by adjusting the variance values provided to a model. In this paper, we propose a novel model called VarianceFlow combining the advantages of the two types. By modeling the variance with NF, VarianceFlow predicts the variance information more precisely with improved speech quality. Also, the objective function of NF makes the model use the variance information and the text in a disentangled manner resulting in more precise variance control. In experiments, VarianceFlow shows superior performance over other state-of-the-art TTS models both in terms of speech quality and controllability.【1】 Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator标题：使用集成文本到MEL语谱图生成器的端到端ASR的纯文本域适配链接：https://arxiv.org/abs/2302.14036
*与cs.SD语音【22】为同一篇
作者：Vladimir Bataev,Roman Korostik,Evgeny Shabalin,Vitaly Lavrukhin,Boris Ginsburg机构：NVIDIA, University of London, UK;, ITMO University, Russia;, Higher School of Economics, Russia摘要：我们提出了一种端到端的ASR系统，它可以在转录的语音数据、文本数据或两者的混合上进行训练。对于纯文本训练，我们的扩展ASR模型使用了一个集成的辅助TTS模块，该模块从文本创建mel谱图。该模块包含传统的非自回归文本到梅尔谱图生成器，该生成器用GAN增强器增强以提高谱图质量。所提出的系统可以通过使用纯文本数据来提高ASR模型在新领域上的准确性，并且允许通过使用大文本语料库来显著优于传统的音频文本训练。摘要：We propose an end-to-end ASR system that can be trained on transcribed speech data, text data, or a mixture of both. For text-only training, our extended ASR model uses an integrated auxiliary TTS block that creates mel spectrograms from the text. This block contains a conventional non-autoregressive text-to-mel-spectrogram generator augmented with a GAN enhancer to improve the spectrogram quality. The proposed system can improve the accuracy of the ASR model on a new domain by using text-only data, and allows to significantly surpass conventional audio-text training by using large text corpora.【2】 A Deep Neural Network Based Reverse Radio Spectrogram Search Algorithm标题：一种基于深度神经网络的反向射电频谱搜索算法链接：https://arxiv.org/abs/2302.13854
*与cs.SD语音【23】为同一篇
作者：Peter Xiangyuan Ma,Steve Croft,Andrew P. V. Siemion机构：Department of Mathematics, University of Toronto, St. George Street, Toronto, ON M,S ,E, Canada, Radio Astronomy Laboratory, Campbell Hall, University of California, Berkeley, CA , USA, SETI Institute, Mountain View, CA , USA备注：8 pages, 8 figures摘要：我们开发了一种快速的模块化深度学习算法，用于在无线电频谱图数据中搜索感兴趣的相似信号。首先，我们根据能量检测算法返回的滤波数据训练自动编码器。然后，我们将位置嵌入层从经典的Transformer架构调整为基于频率的嵌入。接下来，我们使用自动编码器的编码器组件从无线电频谱图的小窗口（~ 715，Hz，每个频率窗口的分辨率为2.79 Hz）中提取特征。我们使用我们的算法在一组信号（搜索项目的编码特征）上对给定查询（感兴趣的编码信号）进行搜索，以产生具有相似特征的最佳候选。我们成功地证明了该算法只在给定原始无线电频谱图数据的情况下检索到具有相似外观的信号。摘要：We developed a fast and modular deep learning algorithm to search for lookalike signals of interest in radio spectrogram data. First, we trained an autoencoder on filtered data returned by an energy detection algorithm. We then adapted a positional embedding layer from classical Transformer architecture to a frequency-based embedding. Next we used the encoder component of the autoencoder to extract features from small (~ 715,Hz with a resolution of 2.79Hz per frequency bin) windows in the radio spectrogram. We used our algorithm to conduct a search for a given query (encoded signal of interest) on a set of signals (encoded features of searched items) to produce the top candidates with similar features. We successfully demonstrate that the algorithm retrieves signals with similar appearance, given only the original radio spectrogram data.【3】 MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognition标题：Mole：用于多语言自动语音识别的混合语言专家链接：https://arxiv.org/abs/2302.13750
作者：Yoohwan Kwon,Soo-Whan Chung机构：NAVER Cloud, South Korea备注：Accepted by ICASSP 2023摘要：多语言语音识别的目的是区分不同语言的语言表达，同时集成声学处理。相比之下，目前的多语言语音识别研究遵循语言感知的范式，主要目标是提高识别性能，而不是区分语言特征。在这篇文章中，我们提出了一个名为混合语言专家（MoLE）的多语言语音识别网络，它可以对各种语言的语音进行摘要。具体地说，MoLE从任意语言的输入语音中分析语言表达，用轻量级语言标记器激活特定于语言的专家。符号化器不仅激活专家，而且估计激活的可靠性。基于可信度，将激活专家和语言不可知专家聚合以表示语言条件嵌入，从而实现高效的语音识别。在5种语言的场景下对该模型进行了测试，实验结果表明，该模型在多语言识别中具有较好的性能，尤其是对低资源语言的语音识别。摘要：Multi-lingual speech recognition aims to distinguish linguistic expressions in different languages and integrate acoustic processing simultaneously. In contrast, current multi-lingual speech recognition research follows a language-aware paradigm, mainly targeted to improve recognition performance rather than discriminate language characteristics. In this paper, we present a multi-lingual speech recognition network named Mixture-of-Language-Expert(MoLE), which digests speech in a variety of languages. Specifically, MoLE analyzes linguistic expression from input speech in arbitrary languages, activating a language-specific expert with a lightweight language tokenizer. The tokenizer not only activates experts, but also estimates the reliability of the activation. Based on the reliability, the activated expert and the language-agnostic expert are aggregated to represent language-conditioned embedding for efficient speech recognition. Our proposed model is evaluated in 5 languages scenario, and the experimental results show that our structure is advantageous on multi-lingual recognition, especially for speech in low-resource language.【4】 Duration-aware pause insertion using pre-trained language model for multi-speaker text-to-speech标题：使用用于多说话人文本到语音转换的预训练语言模型的时长感知暂停插入链接：https://arxiv.org/abs/2302.13652
作者：Dong Yang,Tomoki Koriyama,Yuki Saito,Takaaki Saeki,Detai Xin,Hiroshi Saruwatari备注：Accepted by ICASSP2023摘要：None摘要：Pause insertion, also known as phrase break prediction and phrasing, is an essential part of TTS systems because proper pauses with natural duration significantly enhance the rhythm and intelligibility of synthetic speech. However, conventional phrasing models ignore various speakers' different styles of inserting silent pauses, which can degrade the performance of the model trained on a multi-speaker speech corpus. To this end, we propose more powerful pause insertion frameworks based on a pre-trained language model. Our approach uses bidirectional encoder representations from transformers (BERT) pre-trained on a large-scale text corpus, injecting speaker embedding to capture various speaker characteristics. We also leverage duration-aware pause insertion for more natural multi-speaker TTS. We develop and evaluate two types of models. The first improves conventional phrasing models on the position prediction of respiratory pauses (RPs), i.e., silent pauses at word transitions without punctuation. It performs speaker-conditioned RP prediction considering contextual information and is used to demonstrate the effect of speaker information on the prediction. The second model is further designed for phoneme-based TTS models and performs duration-aware pause insertion, predicting both RPs and punctuation-indicated pauses (PIPs) that are categorized by duration. The evaluation results show that our models improve the precision and recall of pause insertion and the rhythm of synthetic speech.【5】 Complex Clipping for Improved Generalization in Machine Learning标题：机器学习中改进泛化的复杂剪裁链接：https://arxiv.org/abs/2302.13527
作者：Les Atlas,Nicholas Rasmussen,Felix Schwock,Mert Pilanci机构：Los Altos, CA and The University of South Dakota, Department of Computer, Science, East, Clark, Street, Vermillion, SD, Engineering, University of Washington, Seattle, WA, USA (e-mail:, training, these critical points are singularities in neural networks备注：Submitted to IEEE Signal Processing Letters摘要：对于许多机器学习应用，常见的输入表示是谱图。频谱图的基本表示是给出复值的短时傅里叶变换（STFT）。谱图使用这些复值的幅度，这是一种常用的检测器。现代机器学习系统通常是过参数化的，其中可能的病态问题通过正则化来改善。深度网络层之间整流线性单元（ReLU）激活函数的共同使用已被证明有助于这种正则化，从而改善系统性能。我们将ReLU激活的这一思想扩展到复杂STFT的检测，提供了一个易于计算的修改和正则化谱图，这可能导致更好的行为训练。然后，我们确认了该方法在用于真实世界应用的噪声声学数据集上的益处。泛化性能显著提高。该方法可以有益于使用时间-频率映射的其他应用，用于声学、音频和其他应用。摘要：For many machine learning applications, a common input representation is a spectrogram. The underlying representation for a spectrogram is a short time Fourier transform (STFT) which gives complex values. The spectrogram uses the magnitude of these complex values, a commonly used detector. Modern machine learning systems are commonly overparameterized, where possible ill-conditioning problems are ameliorated by regularization. The common use of rectified linear unit (ReLU) activation functions between layers of a deep net has been shown to help this regularization, improving system performance. We extend this idea of ReLU activation to detection for the complex STFT, providing a simple-to-compute modified and regularized spectrogram, which potentially results in better behaved training. We then confirmed the benefit of this approach on a noisy acoustic data set used for a real-world application. Generalization performance improved substantially. This approach might benefit other applications which use time-frequency mappings, for acoustic, audio, and other applications.【6】 Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow标题：VariancFlow：通过归一化流程利用方差信息实现高质量、可控的文语转换链接：https://arxiv.org/abs/2302.13458
*与cs.SD语音【24】为同一篇
作者：Yoonhyung Lee,Jinhyeok Yang,Kyomin Jung机构：Seoul National University, Dept. of Electrical and Computer Engineering, Republic of Korea, Speech AI Lab, NCSOFT, Republic of Korea备注：Accepted for ICASSP 2022摘要：非自回归文语转换模型可以有效地学习文语之间的一对多关系，其方法有两种。第一种方法是使用一个先进的生成式框架，如规范化流（NF）。第二种是在生成语音时一起使用诸如音调或能量的方差信息。对于第二种类型，还可以通过调整提供给模型的方差值来控制方差因子。本文结合这两种模型的优点，提出了一种新的模型--方差流模型。通过使用NF对方差建模，VarianceFlow可以更精确地预测方差信息，从而提高语音质量。同时，NF的目标函数使得模型能够以一种非纠缠的方式使用方差信息和文本，从而实现更精确的方差控制。实验结果表明，VarianceFlow在语音质量和可控性方面均优于现有的TTS模型。摘要：There are two types of methods for non-autoregressive text-to-speech models to learn the one-to-many relationship between text and speech effectively. The first one is to use an advanced generative framework such as normalizing flow (NF). The second one is to use variance information such as pitch or energy together when generating speech. For the second type, it is also possible to control the variance factors by adjusting the variance values provided to a model. In this paper, we propose a novel model called VarianceFlow combining the advantages of the two types. By modeling the variance with NF, VarianceFlow predicts the variance information more precisely with improved speech quality. Also, the objective function of NF makes the model use the variance information and the text in a disentangled manner resulting in more precise variance control. In experiments, VarianceFlow shows superior performance over other state-of-the-art TTS models both in terms of speech quality and controllability.【7】 DFSNet: A Steerable Neural Beamformer Invariant to Microphone Array Configuration for Real-Time, Low-Latency Speech Enhancement标题：DFSNet：一种用于实时、低延迟语音增强的麦克风阵列配置的可控神经波束形成器链接：https://arxiv.org/abs/2302.13407
作者：Anton Kovalyov,Kashyap Patel,Issa Panahi备注：5 pages, 1 figure, 2 tables摘要：对麦克风阵列配置的不变性是神经波束形成器中的罕见属性。此类中的滤波求和（FS）方法定义相对于参考通道的目标信号。然而，这不仅使混响条件下的公式化复杂化，而且使网络复杂化，网络必须具有推断参考信道是什么的机制。为了解决这些问题，本研究提出延迟滤波与求和网路（DFSNet），一种不受麦克风数目和阵列几何形状影响的可操控神经波束形成器，用于因果语音增强。在DFSNet中，在FS操作之前，首先将采集的信号导向语音源方向，这将任务简化为估计延迟和求和的混响干净语音。该模型具有低延迟、低失真、低存储和计算负担的特点，在助听器应用中具有很大的潜力。模拟结果显示，其性能与非因果最新技术水平相当。摘要：Invariance to microphone array configuration is a rare attribute in neural beamformers. Filter-and-sum (FS) methods in this class define the target signal with respect to a reference channel. However, this not only complicates formulation in reverberant conditions but also the network, which must have a mechanism to infer what the reference channel is. To address these issues, this study presents Delay Filter-and-Sum Network (DFSNet), a steerable neural beamformer invariant to microphone number and array geometry for causal speech enhancement. In DFSNet, acquired signals are first steered toward the speech source direction prior to the FS operation, which simplifies the task into the estimation of delay-and-summed reverberant clean speech. The proposed model is designed to incur low latency, distortion, and memory and computational burden, giving rise to high potential in hearing aid applications. Simulation results reveal comparable performance to noncausal state-of-the-art.【8】 I-MSV 2022: Indic-Multilingual and Multi-sensor Speaker Verification Challenge标题：I-MSV 2022：印度-多语言和多传感器说话人确认挑战链接：https://arxiv.org/abs/2302.13209
作者：Jagabandhu Mishra,Mrinmoy Bhattacharjee,S. R. Mahadeva Prasanna摘要：说话人验证（SV）是使用他/她的语音样本来验证索赔人所声称的身份的任务。尽管SV技术已经有了大量的研究，但是关于多语言会话的发展是有限的。在印度这样的国家，几乎所有的说话者都是通晓多种语言的。因此，根据在印度情况下收集的数据开发一个多语种统计卫星（MSV）系统更具挑战性。基于这一动机，2022年印度语-多语言说话人确认（I-MSV）挑战赛旨在了解和比较最先进的说话人确认技术。对于该挑战，已经使用5个不同传感器以13种印度语言收集了100个说话者所讲的大约100个小时的数据。数据被分为开发、训练和测试集，并已公开供进一步研究。本挑战的目标是使SV系统对入组和测试之间的语言和传感器变化具有鲁棒性。在挑战中，参与者被要求在两种情况下开发SV系统，即约束和非约束。在有约束和无约束的情况下，最佳系统的等误码率（EER）分别为2 .12 $和0 .2 6 $.摘要：Speaker Verification (SV) is a task to verify the claimed identity of the claimant using his/her voice sample. Though there exists an ample amount of research in SV technologies, the development concerning a multilingual conversation is limited. In a country like India, almost all the speakers are polyglot in nature. Consequently, the development of a Multilingual SV (MSV) system on the data collected in the Indian scenario is more challenging. With this motivation, the Indic- Multilingual Speaker Verification (I-MSV) Challenge 2022 has been designed for understanding and comparing the state-of-the-art SV techniques. For the challenge, approximately $100$ hours of data spoken by $100$ speakers has been collected using $5$ different sensors in $13$ Indian languages. The data is divided into development, training, and testing sets and has been made publicly available for further research. The goal of this challenge is to make the SV system robust to language and sensor variations between enrollment and testing. In the challenge, participants were asked to develop the SV system in two scenarios, viz. constrained and unconstrained. The best system in the constrained and unconstrained scenario achieved a performance of $2.12\%$ and $0.26\%$ in terms of Equal Error Rate (EER), respectively.【9】 Time-Variance Aware Real-Time Speech Enhancement标题：时变感知的实时语音增强链接：https://arxiv.org/abs/2302.13063
作者：Chengyu Zheng,Yuan Zhou,Xiulian Peng,Yuan Zhang,Yan Lu机构：Communication University of China, Beijing, China, Microsoft Research Asia, Beijing, China摘要：在实际全双工通信应用中经常会出现时变因素。其中有些是由复杂的环境引起的，如非平稳的环境噪声和变化的声路径，而有些是由通信系统引起的，如远端和近端信号之间的动态延迟。目前基于端到端深度神经网络（DNN）的语音增强方法通常对时变分量进行隐式建模，难以处理语音增强中不可预测的时变。为了显式地捕获时变分量，我们提出了一个动态核生成（DKG）模块，它可以作为一个可学习的插件引入到基于DNN的端到端流水线中。具体地，DKG模块生成关于每个输入音频帧的卷积核，使得DNN模型能够在推理期间根据输入信号动态地调整其权重。实验结果表明，DKG模块提高了模型在时变场景下的联合声学回声消除（AEC）和深度噪声抑制（DNS）任务中的性能。摘要：Time-variant factors often occur in real-world full-duplex communication applications. Some of them are caused by the complex environment such as non-stationary environmental noises and varying acoustic path while some are caused by the communication system such as the dynamic delay between the far-end and near-end signals. Current end-to-end deep neural network (DNN) based methods usually model the time-variant components implicitly and can hardly handle the unpredictable time-variance in real-time speech enhancement. To explicitly capture the time-variant components, we propose a dynamic kernel generation (DKG) module that can be introduced as a learnable plug-in to a DNN-based end-to-end pipeline. Specifically, the DKG module generates a convolutional kernel regarding to each input audio frame, so that the DNN model is able to dynamically adjust its weights according to the input signal during inference. Experimental results verify that DKG module improves the performance of the model under time-variant scenarios, in the joint acoustic echo cancellation (AEC) and deep noise suppression (DNS) tasks.【10】 Multimodal Speech Recognition for Language-Guided Embodied Agents标题：面向语言制导的具身智能体的多模式语音识别链接：https://arxiv.org/abs/2302.14030
*与cs.SD语音【1】为同一篇
作者：Allen Chang,Xiaoyuan Zhu,Aarav Monga,Seoho Ahn,Tejas Srinivasan,Jesse Thomason机构：Department of Computer Science, University of Southern California, USA, Center for AI In Society’s Student Branch, University of Southern California, USA备注：5 pages, 5 figures摘要：语言引导的具体化代理的基准通常假定基于文本的指令，但是部署的代理将遇到口头指令。虽然自动语音识别（ASR）模型可以弥补输入差距，但错误的ASR转录会损害代理完成任务的能力。在这项工作中，我们提出训练一个多模态ASR模型，通过考虑伴随的视觉上下文来减少转录口头指令时的错误。我们在由ALFRED任务完成数据集合成的口语指令数据集上训练我们的模型，其中我们通过系统地屏蔽口语单词来模拟声学噪声。我们发现，利用视觉观察有利于掩蔽词的恢复，与单峰基线相比，多模态ASR模型恢复高达30%的掩蔽词。我们还发现，文本训练的具身智能体通过遵循多模态ASR模型的转录指令更频繁地成功完成任务。摘要：Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions. While Automatic Speech Recognition (ASR) models can bridge the input gap, erroneous ASR transcripts can hurt the agents' ability to complete tasks. In this work, we propose training a multimodal ASR model to reduce errors in transcribing spoken instructions by considering the accompanying visual context. We train our model on a dataset of spoken instructions, synthesized from the ALFRED task completion dataset, where we simulate acoustic noise by systematically masking spoken words. We find that utilizing visual observations facilitates masked word recovery, with multimodal ASR models recovering up to 30% more masked words than unimodal baselines. We also find that a text-trained embodied agent successfully completes tasks more often by following transcribed instructions from multimodal ASR models.【11】 Cross-modal Face- and Voice-style Transfer标题：跨模式脸型和语音型转换链接：https://arxiv.org/abs/2302.13838
*与cs.SD语音【2】为同一篇
作者：Naoya Takahashi,Mayank K. Singh,Yuki Mitsufuji机构：Sony Group Corporation, Japan备注：arXiv admin note: text overlap with arXiv:1912.01865 by other authors摘要：None摘要：Image-to-image translation and voice conversion enable the generation of a new facial image and voice while maintaining some of the semantics such as a pose in an image and linguistic content in audio, respectively. They can aid in the content-creation process in many applications. However, as they are limited to the conversion within each modality, matching the impression of the generated face and voice remains an open question. We propose a cross-modal style transfer framework called XFaVoT that jointly learns four tasks: image translation and voice conversion tasks with audio or image guidance, which enables the generation of ``face that matches given voice" and ``voice that matches given face", and intra-modality translation tasks with a single framework. Experimental results on multiple datasets show that XFaVoT achieves cross-modal style translation of image and voice, outperforming baselines in terms of quality, diversity, and face-voice correspondence.【12】 DST: Deformable Speech Transformer for Emotion Recognition标题：DST：用于情感识别的可变形语音转换器链接：https://arxiv.org/abs/2302.13729
*与cs.SD语音【3】为同一篇
作者：Weidong Chen,Xiaofen Xing,Xiangmin Xu,Jianxin Pang,Lan Du机构： School of Electronic and Information Engineering, South China University of Technology, China, School of Future Technology, South China University of Technology, China, Pazhou Laboratory, China, UBTECH Robotics Corp, China, iFLYTEK Research, China备注：5 pages, 4 figures, 2tables, accepted by ICASSP 2023摘要：Transformer在多人自注意的支持下，在语音情感识别（SER）中取得了显著的效果。与原有的全注意机制相比，基于窗口的注意机制在学习细粒度特征时更有效，同时大大减少了模型冗余。然而，情感线索以多粒度方式呈现，使得预定义固定窗口可严重降低模型灵活性。此外，很难手动获得最佳窗口设置。本文提出了一种用于SER任务的可变形语音Transformer DST。DST通过一个轻量级的判决网络确定以输入语音为条件的窗口大小的使用。同时，利用基于声学特征的数据相关偏移量来调整注意窗口的位置，使DST能够自适应地发现和注意语音中的有价值信息。在IEMOCAP和MELD上的大量实验证明了DST的优越性。摘要：Enabled by multi-head self-attention, Transformer has exhibited remarkable results in speech emotion recognition (SER). Compared to the original full attention mechanism, window-based attention is more effective in learning fine-grained features while greatly reducing model redundancy. However, emotional cues are present in a multi-granularity manner such that the pre-defined fixed window can severely degrade the model flexibility. In addition, it is difficult to obtain the optimal window settings manually. In this paper, we propose a Deformable Speech Transformer, named DST, for SER task. DST determines the usage of window sizes conditioned on input speech via a light-weight decision network. Meanwhile, data-dependent offsets derived from acoustic features are utilized to adjust the positions of the attention windows, allowing DST to adaptively discover and attend to the valuable information embedded in the speech. Extensive experiments on IEMOCAP and MELD demonstrate the superiority of DST.【13】 Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech标题：虚声：面向文语转换的人脸扩散模型链接：https://arxiv.org/abs/2302.13700
*与cs.SD语音【4】为同一篇
作者：Jiyoung Lee,Joon Son Chung,Soo-Whan Chung机构：NAVER AI Lab, South Korea, Korea Advanced Institute of Science and Technology, South Korea, NAVER Cloud, South Korea备注：ICASSP 2023. Project page: this https URL摘要：这项工作的目标是zero-shot的文本到语音合成，说话风格和声音学习的面部特征。受人们在看到某人的脸时可以想象其声音这一自然事实的启发，我们在一个从可见属性学习的统一框架内引入了一个人脸风格的扩散文本到语音（TTS）模型，称为Face-TTS。这是第一次使用人脸图像作为训练TTS模型的条件。 我们联合训练跨模型生物特征和TTS模型，以保持人脸图像和生成的语音片段之间的说话人身份。我们还提出了一个说话人特征绑定损失来增强生成语音片段和真实语音片段在说话人嵌入空间中的相似性。由于生物特征信息是直接从人脸图像中提取的，因此我们的方法不需要额外的微调步骤来从看不见和听不见的说话者生成语音。我们在LRS 3数据集上训练和评估该模型，该数据集是一个包含背景噪声和多种说话风格的野生视听语料库。该项目的网页是https://facetts.github.io。摘要：The goal of this work is zero-shot text-to-speech synthesis, with speaking styles and voices learnt from facial characteristics. Inspired by the natural fact that people can imagine the voice of someone when they look at his or her face, we introduce a face-styled diffusion text-to-speech (TTS) model within a unified framework learnt from visible attributes, called Face-TTS. This is the first time that face images are used as a condition to train a TTS model. We jointly train cross-model biometrics and TTS models to preserve speaker identity between face images and generated speech segments. We also propose a speaker feature binding loss to enforce the similarity of the generated and the ground truth speech segments in speaker embedding space. Since the biometric information is extracted directly from the face image, our method does not require extra fine-tuning steps to generate speech from unseen and unheard speakers. We train and evaluate the model on the LRS3 dataset, an in-the-wild audio-visual corpus containing background noise and diverse speaking styles. The project page is https://facetts.github.io.【14】 A Comparative Analysis Of Latent Regressor Losses For Singing Voice Conversion标题：歌唱嗓音转换中潜在阻滞器损耗的比较分析链接：https://arxiv.org/abs/2302.13678
*与cs.SD语音【5】为同一篇
作者：Brendan O'Connor,Simon Dixon机构：Centre For Digital Music, Queen Mary University of London备注：Submitted to the Sound and Music Computing Conference 2023摘要：先前的研究已经表明，当应用于歌声转换（SVC）时，用于口语转换（VC）的已建立的技术不能很好地执行。我们提出了一个替代损失的组成部分，在损失函数中，否则是建立良好的VC任务，这已被证明，以改善我们的模型的SVC性能。我们首先在歌手录音的梅尔谱图上训练歌手身份嵌入（SIE）网络，以使用对比学习产生歌手特定的方差编码。随后，我们在这些SIE的基础上训练了一个众所周知的自动编码器框架（AutoVC），并测量了使用不同潜在回归因子损失分量时SVC性能的差异。我们发现使用这个损失加权平均。SIE比w.r.t.具有更好的性能。瓶颈嵌入，其中转换后的音频对于目标歌手更自然且更特定。包含这个损失分量的优点是明确地迫使网络用音色相似性来重构，并且还否定了AutoVC的瓶颈嵌入中的不良解纠缠的影响。我们证明了计算和人的评价歌手转换音频剪辑之间的特殊多样性，这突出了两者的必要性。我们还提出了一种源歌手和目标歌手之间的音高匹配机制，以确保这些评估不受音高寄存器差异的影响。摘要：Previous research has shown that established techniques for spoken voice conversion (VC) do not perform as well when applied to singing voice conversion (SVC). We propose an alternative loss component in a loss function that is otherwise well-established among VC tasks, which has been shown to improve our model's SVC performance. We first trained a singer identity embedding (SIE) network on mel-spectrograms of singer recordings to produce singer-specific variance encodings using contrastive learning. We subsequently trained a well-known autoencoder framework (AutoVC) conditioned on these SIEs, and measured differences in SVC performance when using different latent regressor loss components. We found that using this loss w.r.t. SIEs leads to better performance than w.r.t. bottleneck embeddings, where converted audio is more natural and specific towards target singers. The inclusion of this loss component has the advantage of explicitly forcing the network to reconstruct with timbral similarity, and also negates the effect of poor disentanglement in AutoVC's bottleneck embeddings. We demonstrate peculiar diversity between computational and human evaluations on singer-converted audio clips, which highlights the necessity of both. We also propose a pitch-matching mechanism between source and target singers to ensure these evaluations are not influenced by differences in pitch register.【15】 Using Auxiliary Tasks In Multimodal Fusion Of Wav2vec 2.0 And BERT For Multimodal Emotion Recognition标题：辅助任务在Wav2vec 2.0和ERT多模式融合中的多模式情感识别链接：https://arxiv.org/abs/2302.13661
*与cs.SD语音【6】为同一篇
作者：Dekai Sun,Yancheng He,Jiqing Han机构：Harbin Institute of Technology, Harbin, China摘要：数据的缺乏和多模态融合的困难一直是多模态情感识别所面临的挑战。本文提出使用预训练的模型作为上游网络，wav2vec 2.0用于音频模态，BERT用于文本模态，并在MER的下游任务中对它们进行微调以应对数据不足的问题。针对多模式融合的困难性，采用K层多头注意机制作为下游融合模块。从MER任务本身出发，设计了两个辅助任务，以缓解模态间融合不足的问题，引导网络捕捉和对齐情绪相关特征。在IEMOCAP数据集上，与已有的模型相比，加权准确率（WA）和未加权准确率（UA）分别提高了78.42%和79.71%。摘要：The lack of data and the difficulty of multimodal fusion have always been challenges for multimodal emotion recognition (MER). In this paper, we propose to use pretrained models as upstream network, wav2vec 2.0 for audio modality and BERT for text modality, and finetune them in downstream task of MER to cope with the lack of data. For the difficulty of multimodal fusion, we use a K-layer multi-head attention mechanism as a downstream fusion module. Starting from the MER task itself, we design two auxiliary tasks to alleviate the insufficient fusion between modalities and guide the network to capture and align emotion-related features. Compared to the previous state-of-the-art models, we achieve a better performance by 78.42% Weighted Accuracy (WA) and 79.71% Unweighted Accuracy (UA) on the IEMOCAP dataset.【16】 Predicting EEG Responses to Attended Speech via Deep Neural Networks for Speech标题：基于语音深度神经网络的听音脑电反应预测链接：https://arxiv.org/abs/2302.13553
*与cs.SD语音【7】为同一篇
作者：Emina Alickovic,Tobias Dorszewski,Thomas U. Christiansen,Kasper Eskelund,Leonardo Gizzi,Martin A. Skoglund,Dorothea Wendt机构： are with the Institute for Modellingand Simulation of Biomechanical Systems, University of Stuttgart摘要：在多说话者环境中关注感兴趣的语音流可能是一项具有挑战性的任务，特别是对于听力受损的收听者。研究表明，用脑电图（EEG）评估的神经反应受听者听觉注意的调节，揭示了对所听语音的选择性神经跟踪（NT）。NT方法大多依赖于人工设计的声学和语言语音特征来预测神经响应。直到最近，没有特定语言信息的深度神经网络（DNN）模型才被用于提取NT的语音特征，证明了分级DNN层中的语音特征可以预测整个听觉通路的神经响应。在本研究中，我们更进一步地研究了相似的DNN语音模型对预测EEG中观察到的竞争语音的神经反应的适用性。我们使用64通道采集系统记录了17名听力正常的听者的脑电图数据，这些听者被指示关注两个竞争说话者中的一个。我们的数据显示，与人工设计的声学特征相比，DNN提取的语音特征能显著更好地预测EEG反应。此外，对分级DNN层的分析表明，早期层产生最高的预测。此外，我们发现使用DNN提取的语音特征比使用手工设计的声学特征在听觉注意力分类准确性方面有显著提高。这些发现为制定新的NT措施以评估和进一步推进听力技术开辟了新途径。摘要：Attending to the speech stream of interest in multi-talker environments can be a challenging task, particularly for listeners with hearing impairment. Research suggests that neural responses assessed with electroencephalography (EEG) are modulated by listener`s auditory attention, revealing selective neural tracking (NT) of the attended speech. NT methods mostly rely on hand-engineered acoustic and linguistic speech features to predict the neural response. Only recently, deep neural network (DNN) models without specific linguistic information have been used to extract speech features for NT, demonstrating that speech features in hierarchical DNN layers can predict neural responses throughout the auditory pathway. In this study, we go one step further to investigate the suitability of similar DNN models for speech to predict neural responses to competing speech observed in EEG. We recorded EEG data using a 64-channel acquisition system from 17 listeners with normal hearing instructed to attend to one of two competing talkers. Our data revealed that EEG responses are significantly better predicted by DNN-extracted speech features than by hand-engineered acoustic features. Furthermore, analysis of hierarchical DNN layers showed that early layers yielded the highest predictions. Moreover, we found a significant increase in auditory attention classification accuracies with the use of DNN-extracted speech features over the use of hand-engineered acoustic features. These findings open a new avenue for development of new NT measures to evaluate and further advance hearing technology.【17】 Continuous descriptor-based control for deep audio synthesis标题：基于连续描述符的深度音频合成控制链接：https://arxiv.org/abs/2302.13542
*与cs.SD语音【8】为同一篇
作者：Ninon Devis,Nils Demerlé,Sarah Nabi,David Genova,Philippe Esling机构：IRCAM - Sorbonne Universit´e, CNRS UMR , place Igor Stravinsky, Paris, France备注：ICASSP 2023摘要：尽管在用于音乐生成的深度模型方面取得了重大进展，但这些技术的使用仍然限于专家用户。在音乐家中普及之前，生成模型必须首先提供对一代人的表达控制，因为这是深度生成模型在创作工作流中整合的条件。在本文中，我们通过引入一种深度生成音频模型来解决这一问题，该模型提供表达性和连续的基于描述符的控制，同时保持足够的轻量级以嵌入硬件合成器。我们使用对抗性混淆准则，通过显式地去除潜在空间中的显著音乐特征来增强实时生成的可控性。然后，用户指定的特征作为附加的调节信息被重新引入，允许对生成进行连续控制，类似于合成器旋钮。我们评估了我们的方法在各种各样的声音上的表现，包括乐器，打击乐和语音录音，同时提供音色和属性传输，允许新的声音生成方法。摘要：Despite significant advances in deep models for music generation, the use of these techniques remains restricted to expert users. Before being democratized among musicians, generative models must first provide expressive control over the generation, as this conditions the integration of deep generative models in creative workflows. In this paper, we tackle this issue by introducing a deep generative audio model providing expressive and continuous descriptor-based control, while remaining lightweight enough to be embedded in a hardware synthesizer. We enforce the controllability of real-time generation by explicitly removing salient musical features in the latent space using an adversarial confusion criterion. User-specified features are then reintroduced as additional conditioning information, allowing for continuous control of the generation, akin to a synthesizer knob. We assess the performance of our method on a wide variety of sounds including instrumental, percussive and speech recordings while providing both timbre and attributes transfer, allowing new ways of generating sounds.【18】 VE-KWS: Visual Modality Enhanced End-to-End Keyword Spotting标题：VE-KWS：视觉通道增强型端到端关键词定位链接：https://arxiv.org/abs/2302.13523
*与cs.SD语音【9】为同一篇
作者：Ao Zhang,He Wang,Pengcheng Guo,Yihui Fu,Lei Xie,Yingying Gao,Shilei Zhang,Junlan Feng机构：Northwestern Polytechnical University, Xi’an, China, China Mobile Research Institute, Beijing, China备注：5 pages. Accepted at ICASSP2023摘要：基于音频模态的关键词识别（KWS）系统在远场和噪声环境下性能显著下降，通常以虚警和拒识来衡量。因此，利用多种模态之间的互补关系的视听关键词识别最近获得了很多关注。然而，目前的研究主要集中在整合不同模态的专门学习表征，而不是在每个单独的建模过程中探索模态关系。本文提出了一种新的视觉通道增强的端到端知识共享系统框架（VE-KWS），从两个方面融合音频和视觉通道。第一种是利用从视频中的嘴唇区域获取的说话人位置信息来辅助多通道音频波束形成器的训练。通过将波束形成器用作音频增强模块，可以显著地抑制由远场或噪声环境引起的声学失真。另一种方法是在不同的通道之间进行交叉注意，以捕捉通道间的关系，并帮助每一个通道的表征学习。在MSIP挑战语料库上的实验表明，该模型在Eval集上获得了2. 79%的误拒率和2. 95%的误警率，与ICASSP 2022 MISP挑战中排名靠前的系统相比，具有更好的SOTA性能.摘要：The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting cross-attention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves 2.79% false rejection rate and 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge.【19】 3D Neural Beamforming for Multi-channel Speech Separation Against Location Uncertainty标题：位置不确定条件下的三维神经波束形成多通道语音分离链接：https://arxiv.org/abs/2302.13462
*与cs.SD语音【10】为同一篇
作者：Rongzhi Gu,Shi-Xiong Zhang,Dong Yu机构： Tencent AI Lab, Shenzhen, China, Tencent AI Lab, Seattle, WA, USA摘要：利用说话人方向信息进行多通道语音分离比盲语音分离有明显的优势。但是，它有两个局限性。第一，当两个声音的到来方向接近时，观察到显著的性能劣化。第二，结果高度依赖于对说话者方向的精确估计。为了克服这些问题，本文提出了一种用于多通道语音分离的三维特征和相关的三维神经波束形成器。在这一领域的先前工作在两个重要方向上得到了扩展。首先，将传统的1D方向性波束图案推广到3D。这使得模型能够从3D空间中的任何目标区域提取语音。因此，具有相似方向但不同高度或距离的扬声器变得可分离。其次，为了处理说话人位置的不确定性，将先前提出的空间特征扩展为一种新的3D区域特征。提出的三维区域特征和三维神经波束形成器在车内场景下进行了评估。实验结果表明，三维特征和三维波束形成器相结合可以获得与以真实说话人位置作为输入的分离模型相当的性能。摘要：Multi-channel speech separation using speaker's directional information has demonstrated significant gains over blind speech separation. However, it has two limitations. First, substantial performance degradation is observed when the coming directions of two sounds are close. Second, the result highly relies on the precise estimation of the speaker's direction. To overcome these issues, this paper proposes 3D features and an associated 3D neural beamformer for multi-channel speech separation. Previous works in this area are extended in two important directions. First, the traditional 1D directional beam patterns are generalized to 3D. This enables the model to extract speech from any target region in the 3D space. Thus, speakers with similar directions but different elevations or distances become separable. Second, to handle the speaker location uncertainty, previously proposed spatial feature is extended to a new 3D region feature. The proposed 3D region feature and 3D neural beamformer are evaluated under an in-car scenario. Experimental results demonstrated that the combination of 3D feature and 3D beamformer can achieve comparable performance to the separation model with ground truth speaker location as input.【20】 Low latency transformers for speech processing标题：用于语音处理的低延迟转换器链接：https://arxiv.org/abs/2302.13451
*与cs.SD语音【11】为同一篇
作者：Jianbo Ma,Siqi Pan,Deepak Chandran,Andrea Fanelli,Richard Cartwright机构：Dolby Laboratories备注：6 pages, 3 figures摘要：None摘要：The transformer is a widely-used building block in modern neural networks. However, when applied to audio data, the transformer's acausal behaviour, which we term Acausal Attention (AA), has generally limited its application to offline tasks. In this paper we introduce Streaming Attention (SA), which operates causally with fixed latency, and requires lower compute and memory resources than AA to train. Next, we introduce Low Latency Streaming Attention (LLSA), a method which combines multiple SA layers without latency build-up proportional to the layer count. Comparative analysis between AA, SA and LLSA on Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER) tasks are presented. The results show that causal SA-based networks with fixed latencies of a few seconds (e.g. 1.8 seconds) and LLSA networks with latencies as short as 300 ms can perform comparably with acausal (AA) networks. We conclude that SA and LLSA methods retain many of the benefits of conventional acausal transformers, but with latency characteristics that make them practical to run in real-time streaming applications.【21】 From Audio to Symbolic Encoding标题：从音频编码到符号编码链接：https://arxiv.org/abs/2302.13401
*与cs.SD语音【12】为同一篇
作者：Shenli Yuan,Lingjie Kong,Jiushuang Guo机构：Department of Mechanical Engineering, Center for Computer Research in Music and Acoustics (CCRMA), Stanford University, Department of Computer Science, Department of Statistics摘要：自动音乐转录（AMT）旨在将原始音频转换为符号化的音乐表示。作为音乐信息检索的一个基本问题，由于声学信号中的多个谐波的重叠，即使对于受过训练的人类专家来说，AMT也被认为是一个困难的任务。另一方面，语音识别作为自然语言处理中最热门的任务之一，旨在将人类的口语翻译成文本。基于AMT和语音识别的相似性质（因为它们都处理将音频信号转换为符号编码的任务），本文研究了通用神经网络架构是否可能同时处理这两个任务。本文介绍了我们在现有Onsets和Frames基础上构建的新的神经网络结构，并比较了其多种变体在AMT任务上的性能。我们还用语音识别任务测试了我们的体系结构。对于AMT，与使用最先进架构训练的模型相比，我们的模型能够产生更好的结果;然而，尽管类似的体系结构能够在语音识别任务上被训练，但是与其它任务特定的模型相比，它没有产生非常理想的结果。摘要：Automatic music transcription (AMT) aims to convert raw audio to symbolic music representation. As a fundamental problem of music information retrieval (MIR), AMT is considered a difficult task even for trained human experts due to overlap of multiple harmonics in the acoustic signal. On the other hand, speech recognition, as one of the most popular tasks in natural language processing, aims to translate human spoken language to texts. Based on the similar nature of AMT and speech recognition (as they both deal with tasks of translating audio signal to symbolic encoding), this paper investigated whether a generic neural network architecture could possibly work on both tasks. In this paper, we introduced our new neural network architecture built on top of the current state-of-the-art Onsets and Frames, and compared the performances of its multiple variations on AMT task. We also tested our architecture with the task of speech recognition. For AMT, our models were able to produce better results compared to the model trained using the state-of-art architecture; however, although similar architecture was able to be trained on the speech recognition task, it did not generate very ideal result compared to other task-specific models.【22】 Efficient Ensemble Architecture for Multimodal Acoustic and Textual Embeddings in Punctuation Restoration using Time-Delay Neural Networks标题：时延神经网络标点符号恢复中多模式声学和文本嵌入的高效集成结构链接：https://arxiv.org/abs/2302.13376
作者：Xing Yi Liu,Homayoon Beigi备注：6 pages, 1 figure, 5 tables, technical report at Recognition Technologies, Inc摘要：标点符号恢复在自动语音识别的后处理过程中起着至关重要的作用，而模型效率是完成这一任务的关键。为此，我们提出了EfficientPunct，这是一种使用多模式时延神经网络的集成方法，它比当前最好的模型高出1.0个F1点，使用不到十分之一的参数来处理嵌入。我们简化了语音识别器，以有效地输出隐藏层潜在向量作为标点符号恢复的音频嵌入，以及BERT提取有意义的文本嵌入。通过使用强制对齐和时间卷积，我们消除了对基于注意力的多头融合的需要，大大提高了计算效率，同时也提高了性能。EfficientPunct在性能和效率两方面都开创了新的技术水平，它的集合体对BERT纯粹基于语言的预测的权重略高于多模网络的预测。摘要：Punctuation restoration plays an essential role in the post-processing procedure of automatic speech recognition, but model efficiency is a key requirement for this task. To that end, we present EfficientPunct, an ensemble method with a multimodal time-delay neural network that outperforms the current best model by 1.0 F1 points, using less than a tenth of its parameters to process embeddings. We streamline a speech recognizer to efficiently output hidden layer latent vectors as audio embeddings for punctuation restoration, as well as BERT to extract meaningful text embeddings. By using forced alignment and temporal convolutions, we eliminate the need for multi-head attention-based fusion, greatly increasing computational efficiency but also raising performance. EfficientPunct sets a new state of the art, in terms of both performance and efficiency, with an ensemble that weights BERT's purely language-based predictions slightly more than the multimodal network's predictions.【23】 Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics标题：音乐中的多模式：从高水平的音频特征和歌词预测音乐中的情感链接：https://arxiv.org/abs/2302.13321
*与cs.SD语音【13】为同一篇
作者：Tibor Krols,Yana Nikolova,Ninell Oldenburg机构：University of Copenhagen, Department of Nordic Studies and Linguistics, and Department of Computer Science备注：12 pages, incl. 2 pages appendix摘要：本文旨在测试一种多模态的音乐情感识别方法是否比单模态的方法在高层次的歌曲特征和歌词上有更好的表现。我们使用从Spotify API检索到的11首歌曲特征，结合歌词特征（包括情感、TF-IDF和Anew）来预测Deezer情绪检测数据集（DMDD）上的效价和唤醒（Russell，1980）评分（Delbouys等人，2018年），采用4种不同的回归模型。我们发现，在11个高层次的歌曲特征中，主要有5个对表现有贡献，多模态特征在预测效价时比单独的音频特征做得更好。我们公开了我们的代码。摘要：This paper aims to test whether a multi-modal approach for music emotion recognition (MER) performs better than a uni-modal one on high-level song features and lyrics. We use 11 song features retrieved from the Spotify API, combined lyrics features including sentiment, TF-IDF, and Anew to predict valence and arousal (Russell, 1980) scores on the Deezer Mood Detection Dataset (DMDD) (Delbouys et al., 2018) with 4 different regression models. We find that out of the 11 high-level song features, mainly 5 contribute to the performance, multi-modal features do better than audio alone when predicting valence. We made our code publically available.【24】 Implementation of an aeroacoustic simulation pipeline using openCFS-Acoustics and openCFS-Data applied to human phonation标题：利用OpenCFS声学和OpenCFS数据实现用于人体发声的气动声学模拟管道链接：https://arxiv.org/abs/2302.13290
*与cs.SD语音【14】为同一篇
作者：Stefan Schoder机构：Group of Aeroacoustics and Vibroacoustics, IGTE, TU Graz, Inffeldgasse , Graz备注：9 pages, 4 figures摘要：人类发声过程可以使用有限元方法（FEM）来建模，该有限元方法提供了语音产生过程的详细表示。用C++语言编写的有限元软件（openCFS）模拟了发声过程。FEM模型由人体上呼吸道的3D网格组成。simVoice模型提供了发声过程的准确表示，并在一些出版物中有效。在本文中，我们将展示如何使用openCFS和openCFS-Data建立模型。摘要：The human phonation process be modeled using the Finite Element Method (FEM) which provides a detailed representation of the voice production process. A software implementation in C++ using FEM (openCFS) has been used to simulate the phonation process. The FEM model consists of a 3D mesh of the upper human airways. The simVoice model provides an accurate representation of the phonation process and was valid in several publications. In this article, we show how to set up the model using openCFS and openCFS-Data.【25】 Contrast-PLC: Contrastive Learning for Packet Loss Concealment标题：对比度-PLC：分组丢失隐藏的对比学习链接：https://arxiv.org/abs/2302.13284
*与cs.SD语音【15】为同一篇
作者：Huaying Xue,Xiulian Peng,Yan Lu机构：Microsoft Research Asia, Beijing, China备注：5 pages, ICASSP 2023(Accepted)摘要：当只有有限的可用上下文可供使用时，分组丢失隐藏（PLC）在真实地和自然地隐藏丢失内容方面具有挑战性。最近，基于深度学习的PLC算法已经证明了它们优于传统对应物;但是它们的隐藏能力仍然主要限于最大120 ms的丢失。即使使用强大的基于GAN的生成模型，预测可能发生在音素内/之间的长突发丢失仍然是非常具有挑战性的。在这篇文章中，我们提出使用对比学习来学习一个损失鲁棒的PLC语义表示。设计了一种结合语义预测和基于遗传网络的生成模型的混合神经元PLC体系结构，验证了该方法的有效性。在Interspeech 2022 PLC Challenge的盲测试集上的测试结果表明，该方法优于常用的UNet式框架和无对比学习的框架，尤其是在（120，220] ms的较长突发丢失方面。摘要：Packet loss concealment (PLC) is challenging in concealing missing contents both plausibly and naturally when there are only limited available context to use. Recently deep-learning based PLC algorithms have demonstrated their superiority over traditional counterparts; but their concealment ability is still mostly limited to a maximum of 120ms loss. Even with strong GAN-based generative models, it is still very challenging to predict long burst losses that could happen within/in-between phonemes. In this paper, we propose to use contrastive learning to learn a loss-robust semantic representation for PLC. A hybrid neural PLC architecture combining the semantic prediction and GAN-based generative model is designed to verify its effectiveness. Results on the blind test set of Interspeech2022 PLC Challenge show its superiority over commonly used UNet-style framework and the one without contrastive learning, especially for the longer burst loss at (120, 220] ms.【26】 Mingling or Misalignment? Temporal Shift for Speech Emotion Recognition with Pre-trained Representations标题：混搭还是错位？基于预训练表征的语音情感识别时移方法链接：https://arxiv.org/abs/2302.13277
*与cs.SD语音【16】为同一篇
作者：Siyuan Shen,Feng Liu,Aimin Zhou机构：East China Normal University, Shanghai, China, Shanghai Institute of AI for Education, Shanghai, China备注：Accepted by 48th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)摘要：随着自监督模型的发展，预训练的语音表示被证明对下游的语音情感识别（SER）任务是有效的。大多数现有的工作主要集中在利用预先训练的表示，并且仅仅在预先训练的模型之上采用线性头，而忽略了下游网络的设计。在这篇论文中，我们提出了一个时间移位模块来混合通道信息，而不需要引入任何参数或FLOP。通过时间移位模块，三个设计的基线构建块演变成相应的移位变体，即ShiftCNN、ShiftLSTM和Shiftformer。此外，为了平衡混合和错位之间的权衡，我们提出了两种技术策略，移位的位置和移位的比例。在微调和特征提取设置下，时间移位模型家族在基准IEMOCAP数据集上的性能都优于最先进的方法。我们的代码可在www.example.com上获得https://github.com/ECNU-Cross-Innovation-Lab/ShiftSER。摘要：Fueled by recent advances of self-supervised models, pre-trained speech representations proved effective for the downstream speech emotion recognition (SER) task. Most prior works mainly focus on exploiting pre-trained representations and just adopt a linear head on top of the pre-trained model, neglecting the design of the downstream network. In this paper, we propose a temporal shift module to mingle channel-wise information without introducing any parameter or FLOP. With the temporal shift module, three designed baseline building blocks evolve into corresponding shift variants, i.e. ShiftCNN, ShiftLSTM, and Shiftformer. Moreover, to balance the trade-off between mingling and misalignment, we propose two technical strategies, placement of shift and proportion of shift. The family of temporal shift models all outperforms the state-of-the-art methods on the benchmark IEMOCAP dataset under both finetuning and feature extraction settings. Our code is available at https://github.com/ECNU-Cross-Innovation-Lab/ShiftSER.【27】 Two-Stream Joint-Training for Speaker Independent Acoustic-to-Articulatory Inversion标题：非说话人声律倒置的双流联合训练链接：https://arxiv.org/abs/2302.13273
*与cs.SD语音【17】为同一篇
作者：Jianrong Wang,Jinyu Liu,Li Liu,Xuewei Li,Mei Yu,Jie Gao,Qiang Fang机构： College of Intelligence and Computing, Tianjin University, Tianjin, China, Tianjin International Engineering Institute, Tianjin University, Tianjin, China, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China摘要：None摘要：Acoustic-to-articulatory inversion (AAI) aims to estimate the parameters of articulators from speech audio. There are two common challenges in AAI, which are the limited data and the unsatisfactory performance in speaker independent scenario. Most current works focus on extracting features directly from speech and ignoring the importance of phoneme information which may limit the performance of AAI. To this end, we propose a novel network called SPN that uses two different streams to carry out the AAI task. Firstly, to improve the performance of speaker-independent experiment, we propose a new phoneme stream network to estimate the articulatory parameters as the phoneme features. To the best of our knowledge, this is the first work that extracts the speaker-independent features from phonemes to improve the performance of AAI. Secondly, in order to better represent the speech information, we train a speech stream network to combine the local features and the global features. Compared with state-of-the-art (SOTA), the proposed method reduces 0.18mm on RMSE and increases 6.0% on Pearson correlation coefficient in the speaker-independent experiment. The code has been released at https://github.com/liujinyu123/AAINetwork-SPN.【28】 Speech Corpora Divergence Based Unsupervised Data Selection for ASR标题：基于语料库离散度的ASR非监督数据选择链接：https://arxiv.org/abs/2302.13222
*与cs.SD语音【18】为同一篇
作者：Changfeng Gao,Gaofeng Cheng,Pengyuan Zhang,Yonghong Yan机构： [ 1 1] usesa baseline ASR model to decode the N-Best hypothesis andthen calculate the term frequency-inverse document frequencyThe authors are with the Key Laboratory of Speech Acoustics andContent Understanding, Institute of Acoustics摘要：应用场景匹配数据的选取是自动语音识别（ASR）训练的重要内容，但训练语料库的匹配程度难以度量。提出了一种基于语料库散度的无监督目标感知数据选择方法，该方法可以度量两个语料库之间的相似性。首先利用自监督Hubert模型将语音语料离散化为标签序列，计算N-gram概率分布。然后计算N元文法之间的Kullback-Leibler散度作为SCD。最后，选择与目标语料库具有最小SCD的子集进行标注和训练。与以往的数据选择方法相比，SCD数据选择方法能够关注更多的声学细节，保证选择集的多样性。我们在来自共同之声的不同口音上评估我们的方法。实验结果表明，提出的SCD数据选择方法相对于随机选择方法的性能提高了14.8%，与监督选择方法的效果相当甚至更好。摘要：Selecting application scenarios matching data is important for the automatic speech recognition (ASR) training, but it is difficult to measure the matching degree of the training corpus. This study proposes a unsupervised target-aware data selection method based on speech corpora divergence (SCD), which can measure the similarity between two speech corpora. We first use the self-supervised Hubert model to discretize the speech corpora into label sequence and calculate the N-gram probability distribution. Then we calculate the Kullback-Leibler divergence between the N-grams as the SCD. Finally, we can choose the subset which has minimum SCD to the target corpus for annotation and training. Compared to previous data selection method, the SCD data selection method can focus on more acoustic details and guarantee the diversity of the selected set. We evaluate our method on different accents from Common Voice. Experiments show that the proposed SCD data selection can realize 14.8% relative improvements to the random selection, comparable or even superior to the result of supervised selection.【29】 Speaker Recognition in Realistic Scenario Using Multimodal Data标题：基于多模式数据的真实场景说话人识别链接：https://arxiv.org/abs/2302.13033
*与cs.SD语音【19】为同一篇
作者：Saqlain Hussain Shah,Muhammad Saad Saeed,Shah Nawaz,Muhammad Haroon Yousaf机构：University of Engineering and Technology Taxila,Swarm Robotics Lab NCRA, Deutsches Elektronen-Synchrotron DESY备注：Accepted at the International Conference on Artificial Intelligence (ICAI'2023)摘要：近年来，利用来自YouTube的大规模视听信息，在名人的面部和声音之间建立了关联。大规模视听数据集的可用性有助于开发基于标准卷积神经网络的说话人识别方法。因此，本文的目的是利用大规模的音视频信息来提高说话人识别的性能。为了达到这个目的，我们提出了一个双分支网络来学习多模态系统中人脸和语音的联合表征。然后，从该两分支网络中提取特征来训练用于说话人识别的分类器。我们在一个名为VoxCeleb$1$的大规模视听数据集上评估了我们提出的框架。实验结果表明，人脸信息的加入提高了说话人识别的性能。此外，我们的结果表明，有一个重叠的面孔和声音。摘要：In recent years, an association is established between faces and voices of celebrities leveraging large scale audio-visual information from YouTube. The availability of large scale audio-visual datasets is instrumental in developing speaker recognition methods based on standard Convolutional Neural Networks. Thus, the aim of this paper is to leverage large scale audio-visual information to improve speaker recognition task. To achieve this task, we proposed a two-branch network to learn joint representations of faces and voices in a multimodal system. Afterwards, features are extracted from the two-branch network to train a classifier for speaker recognition. We evaluated our proposed framework on a large scale audio-visual dataset named VoxCeleb$1$. Our results show that addition of facial information improved the performance of speaker recognition. Moreover, our results indicate that there is an overlap between face and voice.【30】 Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing标题：联合优化翻译和语音计时提高自动配音的等时性链接：https://arxiv.org/abs/2302.12979
*与cs.SD语音【20】为同一篇
作者：Alexandra Chronopoulou,Brian Thompson,Prashant Mathur,Yogesh Virkar,Surafel M. Lakew,Marcello Federico机构：AWS AI Labs, Center for Information and Language Processing, LMU Munich, Germany备注：5 pages摘要：自动配音（AD）是将视频中的原始语音翻译成目标语言语音的任务。新的目标语语音应满足等时性;也就是说，新的语音应该与原始视频在时间上对齐，包括嘴部运动、停顿、手势等。在本文中，我们建议训练一个模型，该模型可以直接优化翻译以及生成的翻译的语音持续时间。我们表明，与以前的工作相比，该系统生成的语音更好地匹配原始语音的定时，同时简化了系统架构。摘要：Automatic dubbing (AD) is the task of translating the original speech in a video into target language speech. The new target language speech should satisfy isochrony; that is, the new speech should be time aligned with the original video, including mouth movements, pauses, hand gestures, etc. In this paper, we propose training a model that directly optimizes both the translation as well as the speech duration of the generated translations. We show that this system generates speech that better matches the timing of the original speech, compared to prior work, while simplifying the system architecture.【31】 Pre-Finetuning for Few-Shot Emotional Speech Recognition标题：Few-Shot情感语音识别的预调优链接：https://arxiv.org/abs/2302.12921
*与cs.SD语音【21】为同一篇
作者：Maximillian Chen,Zhou Yu机构：Columbia University备注：5 pages, 4 figures. Code available at this https URL摘要：语音模型长期以来已知对于许多分类任务来说过拟合个体说话者。这导致在扬声器位于域外或分布外的设置中泛化效果较差，这在生产环境中很常见。我们把说话人适应看作是一个Few-Shot的学习问题，并建议调查迁移学习方法的灵感来自最近成功的预训练模型在自然语言任务。我们提出在困难的任务上预先微调语音模型，以将知识提取到Few-Shot的下游分类目标。我们在四个多类情感语音识别语料库的每个排列上预微调Wav2Vec2.0，并通过在情感语音数据集上进行33，600次Few-Shot微调试验来评估我们的预微调模型。摘要：Speech models have long been known to overfit individual speakers for many classification tasks. This leads to poor generalization in settings where the speakers are out-of-domain or out-of-distribution, as is common in production environments. We view speaker adaptation as a few-shot learning problem and propose investigating transfer learning approaches inspired by recent success with pre-trained models in natural language tasks. We propose pre-finetuning speech models on difficult tasks to distill knowledge into few-shot downstream classification objectives. We pre-finetune Wav2Vec2.0 on every permutation of four multiclass emotional speech recognition corpora and evaluate our pre-finetuned models through 33,600 few-shot fine-tuning trials on the Emotional Speech Dataset.机器翻译，仅供参考
每日语音速递