今日论文合集：cs.SD语音7篇，eess.AS音频处理7篇。
cs.SD语音

【1】 TOLD: A Novel Two-Stage Overlap-Aware Framework for Speaker Diarization

标题：Tell：一种新的两阶段重叠感知说话人二值化框架

链接：http://arxiv.org/pdf/2303.05397v1

作者：Jiaming Wang,Zhihao Du,Shiliang Zhang
机构：Speech Lab, Alibaba Group, China
备注：Accepted by ICASSP2023
摘要：近年来，端到端神经元日记化（EEND）被引入并在说话人重叠的场景中取得了良好的效果。在EEND中，说话人的日记化被描述为一个多标记预测问题，说话人活动是独立估计的，没有很好地考虑它们之间的相关性。为了克服这些缺点，本文采用幂集编码将说话人二分法转化为单标记分类问题，提出了重叠感知的EEND（EEND-OLA）模型，该模型可以显式地描述说话人的重叠和相关性。受两级混合系统成功的启发，我们进一步提出了一种新的两级重叠感知Diarization框架（TOLD），通过引入说话人重叠感知后处理（SOAP）模型来迭代地改进EEND-OLA的Diarization结果。实验结果表明，与原始的EEND相比，EEND-OLA在二值化错误率（DER）方面获得了14.39%的相对改善，而SOAP的使用又提供了19.33%的相对改善。因此，我们的方法TOLD在CALLHOME数据集上实现了10.14%的DER，据我们所知，这是该基准测试的最新结果。
摘要：Recently, end-to-end neural diarization (EEND) is introduced and achieves promising results in speaker-overlapped scenarios. In EEND, speaker diarization is formulated as a multi-label prediction problem, where speaker activities are estimated independently and their dependency are not well considered. To overcome these disadvantages, we employ the power set encoding to reformulate speaker diarization as a single-label classification problem and propose the overlap-aware EEND (EEND-OLA) model, in which speaker overlaps and dependency can be modeled explicitly. Inspired by the success of two-stage hybrid systems, we further propose a novel Two-stage OverLap-aware Diarization framework (TOLD) by involving a speaker overlap-aware post-processing (SOAP) model to iteratively refine the diarization results of EEND-OLA. Experimental results show that, compared with the original EEND, the proposed EEND-OLA achieves a 14.39% relative improvement in terms of diarization error rates (DER), and utilizing SOAP provides another 19.33% relative improvement. As a result, our method TOLD achieves a DER of 10.14% on the CALLHOME dataset, which is a new state-of-the-art result on this benchmark to the best of our knowledge.

【2】 MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning

标题：MMCosine：面向均衡视听细粒度学习的多模式余弦损失

链接：http://arxiv.org/pdf/2303.05338v1

作者：Ruize Xu,Ruoxuan Feng,Shi-Xiong Zhang,Di Hu
机构：Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China, Tencent AI Lab, Bellevue, WA, USA
摘要：视听学习通过融合多种形式的实用信息，有助于全面理解世界。然而，最近的研究表明，在联合学习模型中，单模态编码器的不平衡优化是提高模型性能的瓶颈。我们进一步发现，现有的不平衡抑制方法在一些对可区分特征分布要求较高的细粒度视听任务上表现不佳。基于余弦损失法构建超球面特征空间和降低类内角度变化的成功经验，提出了多模态余弦损失法（Multi-Modal Cosine Loss，MMCosine）。它对特征和权重进行模态方面的$L_2$归一化，以实现平衡和更好的多模态细粒度学习。从权范数的角度证明了该方法能够有效地缓解非平衡优化问题，并充分利用余弦度量的可区分性。大量的实验证明了该方法的有效性以及与先进的多模态融合策略和最新的不平衡抑制方法的通用性。
摘要：Audio-visual learning helps to comprehensively understand the world by fusing practical information from multiple modalities. However, recent studies show that the imbalanced optimization of uni-modal encoders in a joint-learning model is a bottleneck to enhancing the model's performance. We further find that the up-to-date imbalance-mitigating methods fail on some audio-visual fine-grained tasks, which have a higher demand for distinguishable feature distribution. Fueled by the success of cosine loss that builds hyperspherical feature spaces and achieves lower intra-class angular variability, this paper proposes Multi-Modal Cosine loss, MMCosine. It performs a modality-wise $L_2$ normalization to features and weights towards balanced and better multi-modal fine-grained learning. We demonstrate that our method can alleviate the imbalanced optimization from the perspective of weight norm and fully exploit the discriminability of the cosine metric. Extensive experiments prove the effectiveness of our method and the versatility with advanced multi-modal fusion strategies and up-to-date imbalance-mitigating methods.

【3】 Improving Few-Shot Learning for Talking Face System with TTS Data Augmentation

标题：利用TTS数据增强改进说话人脸系统的Few-Shot学习

链接：http://arxiv.org/pdf/2303.05322v1

作者：Qi Chen,Ziyang Ma,Tao Liu,Xu Tan,Qu Lu,Xie Chen,Kai Yu
机构：X-LANCE Lab, Department of Computer Science and Engineering, MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China, Microsoft Research Asia,Shanghai Media Tech
备注：4 pages. Accepted by ICASSP 2023
摘要：近年来，音频驱动的人脸识别技术引起了学术界和工业界的广泛关注。然而，音频驱动的说话脸中的数据采集和标记是劳动密集型和成本高昂的。数据资源的缺乏导致合成效果不佳。为了解决这个问题，我们提出使用TTS（Text-To-Speech）进行数据扩充，以提高说话人脸系统的Few-Shot能力。软DTW的引入解决了TTS音频带来的错位问题，并首次应用于人脸任务。此外，HuBERT提取的特征被探索以利用音频的潜在信息，并且发现其优于其他特征。与基线模型相比，该方法在MSE、DTW和用户学习偏好上分别取得了17%、14%和38%的优势，证明了TTS增强技术对人脸系统Few-Shot学习的有效性。
摘要：Audio-driven talking face has attracted broad interest from academia and industry recently. However, data acquisition and labeling in audio-driven talking face are labor-intensive and costly. The lack of data resource results in poor synthesis effect. To alleviate this issue, we propose to use TTS (Text-To-Speech) for data augmentation to improve few-shot ability of the talking face system. The misalignment problem brought by the TTS audio is solved with the introduction of soft-DTW, which is first adopted in the talking face task. Moreover, features extracted by HuBERT are explored to utilize underlying information of audio, and found to be superior over other features. The proposed method achieves 17%, 14%, 38% dominance on MSE score, DTW score and user study preference repectively over the baseline model, which shows the effectiveness of improving few-shot learning for talking face system with TTS augmentation.

【4】 WASD: A Wilder Active Speaker Detection Dataset

标题：WASD：一种Wilder主动说话人检测数据集

链接：http://arxiv.org/pdf/2303.05321v1

作者：Tiago Roxo,Joana C. Costa,Pedro R. M. Inácio,Hugo Proença
机构：IT - Instituto de Telecomunicac¸˜oes, University of Beira Interior, Portugal
摘要：现有的主动说话人检测（ASD）模型仅利用声音和人脸特征，在AVA-ActiveSpeaker（AVA）上取得了很好的效果。虽然这种方法适用于电影设置（AVA），但它不适合约束较少的条件。为了证明这一局限性，我们提出了Wilder主动说话人检测（WASD）数据集，通过针对当前ASD的两个关键组件来增加难度：音频和面部。WASD分为5个类别，从最佳条件到监控设置，包含音频和面部数据战术损伤对ASD的增量挑战。我们选择最先进的型号，并在两组WASD中评估其性能：“简单”（协作设置）和“困难”（音频和 或面部被专门降级）。结果表明：1）AVA训练的模型在WASD Easy组中保持了最先进的性能，而在Hard组中表现不佳，显示了2）AVA和Easy数据之间的相似性;以及3）WASD中的训练没有将模型性能提高到AVA水平，特别是对于音频损伤和监视设置。这表明AVA没有为野生型ASD准备模型，目前的方法在处理此类条件方面不合格。申报的数据集还包含身体数据注释，为ASD提供了新的来源，可在www.example.com上获得https: github.com Tiago-Roxo WASD。
摘要：Current Active Speaker Detection (ASD) models achieve great results on AVA-ActiveSpeaker (AVA), using only sound and facial features. Although this approach is applicable in movie setups (AVA), it is not suited for less constrained conditions. To demonstrate this limitation, we propose a Wilder Active Speaker Detection (WASD) dataset, with increased difficulty by targeting the two key components of current ASD: audio and face. Grouped into 5 categories, ranging from optimal conditions to surveillance settings, WASD contains incremental challenges for ASD with tactical impairment of audio and face data. We select state-of-the-art models and assess their performance in two groups of WASD: Easy (cooperative settings) and Hard (audio and or face are specifically degraded). The results show that: 1) AVA trained models maintain a state-of-the-art performance in WASD Easy group, while underperforming in the Hard one, showing the 2) similarity between AVA and Easy data; and 3) training in WASD does not improve models performance to AVA levels, particularly for audio impairment and surveillance settings. This shows that AVA does not prepare models for wild ASD and current approaches are subpar to deal with such conditions. The proposed dataset also contains body data annotations to provide a new source for ASD, and is available at https: github.com Tiago-Roxo WASD.

【5】 hierarchical network with decoupled knowledge distillation for speech emotion recognition

标题：用于语音情感识别的解耦知识提取的层次网络

链接：http://arxiv.org/pdf/2303.05134v1

作者：Ziping Zhao,Huan Wang,Haishuai Wang,Bjorn Schuller
机构：College of Computer and Information Engineering, Tianjin Normal University, Tianjin, China, College of Computer Science, Zhejiang University, China, Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Germany
备注：5 pages,4 figures,icassp 2023
摘要：语音情感识别（SER）的目标是使计算机能够以与人类相同的方式识别给定话语的情感类别。SER的准确性很大程度上取决于由该模型获得的话语级别表示的有效性。然而，非目标阶层所携带的“暗知识”却一直被先前的研究所忽略。本文提出了一种层次化网络DKDFMH，它采用了深度卷积神经网络中的解耦知识提取和融合多头注意机制。该方法利用logit蒸馏从不同尺度的注意集获取更高层次的语义特征，深入挖掘非目标类所携带的知识，从而引导模型更加关注情感特征之间的差异。为了验证该模型的有效性，在交互式情感二进运动捕获（IEMOCAP）数据集上进行了实验。我们实现了具有竞争力的性能，加权准确率（WA）为79.1%，未加权准确率（UA）为77.1%。据我们所知，这是自2015年以来，Logit蒸馏首次回归最先进状态。
摘要：The goal of Speech Emotion Recognition (SER) is to enable computers to recognize the emotion category of a given utterance in the same way that humans do. The accuracy of SER is strongly dependent on the validity of the utterance-level representation obtained by the model. Nevertheless, the dark knowledge" carried by non-target classes is always ignored by previous studies. In this paper, we propose a hierarchical network, called DKDFMH, which employs decoupled knowledge distillation in a deep convolutional neural network with a fused multi-head attention mechanism. Our approach applies logit distillation to obtain higher-level semantic features from different scales of attention sets and delve into the knowledge carried by non-target classes, thus guiding the model to focus more on the differences between sentiment features. To validate the effectiveness of our model, we conducted experiments on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. We achieved competitive performance, with 79.1% weighted accuracy (WA) and 77.1% unweighted accuracy (UA). To the best of our knowledge, this is the first time since 2015 that logit distillation has been returned to state-of-the-art status.

【6】Towards Robust Image-in-Audio Deep Steganography

标题：一种稳健的音频图像深度隐写算法

链接：http://arxiv.org/pdf/2303.05007v1

作者：Jaume Ros Alonso,Margarita Geleta,Jordi Pons,Xavier Giro-i-Nieto
机构：Universitat Politecnica de Catalunya, UC Berkeley, Dolby Laboratories
备注：8 pages, 5 figures, 2 tables
摘要：由于人工智能技术的最新进展，特别是在能够在不同性质的信号内隐藏信号的多模式设置的背景下，隐写术领域经历了兴趣的激增。所有隐写方法的主要目标都是实现感知透明性、鲁棒性和大的嵌入容量--这往往会出现经典方法难以调和的冲突目标。该文对现有的一种音频图像深度隐写方法进行了扩展和改进，重点提高了其鲁棒性。所提出的增强包括对损失函数的修改、利用短时傅立叶变换（STFT）、在用于纠错的编码过程中引入冗余、以及在像素子卷积操作中缓冲附加信息。实验结果表明，该方法在鲁棒性和感知透明性方面优于现有方法。
摘要：The field of steganography has experienced a surge of interest due to the recent advancements in AI-powered techniques, particularly in the context of multimodal setups that enable the concealment of signals within signals of a different nature. The primary objectives of all steganographic methods are to achieve perceptual transparency, robustness, and large embedding capacity - which often present conflicting goals that classical methods have struggled to reconcile. This paper extends and enhances an existing image-in-audio deep steganography method by focusing on improving its robustness. The proposed enhancements include modifications to the loss function, utilization of the Short-Time Fourier Transform (STFT), introduction of redundancy in the encoding process for error correction, and buffering of additional information in the pixel subconvolution operation. The results demonstrate that our approach outperforms the existing method in terms of robustness and perceptual transparency.

【7】 X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion

标题：X-SepFormer：对说话人混淆进行显式优化的端到端说话人提取网络

链接：http://arxiv.org/pdf/2303.05023v1

作者：Kai Liu,Ziqing Du,Xucheng Wan,Huan Zhou
机构：AI Application Research Center, Huawei Technologies Co., Ltd., Shenzhen, China
备注：Accepted by ICASSP 2023
摘要：目标语音提取（TSE）系统被设计成从多说话者混合中提取目标语音。对于大多数现有的TSE网络，普遍的训练目标是提高提取的语音波形的重构性能。然而，据报道，提供高重建性能的TSE系统在实践中可能仍然遭受低质量体验问题。其中一个经验问题是错误的说话人提取（称为说话人混淆，SC），它导致强烈的负面经验，并阻碍有效的对话。为了缓解强制性SC问题，我们重新制定了训练目标，并提出了两种新的损失方案，探索了在小块级别定义的重建改进性能的度量，并利用了度量相关的分布信息。两种丢失方案旨在鼓励TSE网络基于所述分布信息来关注那些SC组块。在此基础上，我们提出了一个端到端的TSE模型X-SepFormer，它是SepFormer的主干。在基准WSJ 0 - 2 mix数据集上的实验结果验证了本文方法的有效性，SC误差得到了一致的改善（相对改善14.8%）.此外，SI-SDRi为19.4 dB，PESQ为3.81，我们的最佳系统显著优于当前的SOTA系统，并且提供了迄今为止在WSJ 0 - 2 mix上报告的最佳TSE结果。
摘要：Target speech extraction (TSE) systems are designed to extract target speech from a multi-talker mixture. The popular training objective for most prior TSE networks is to enhance reconstruction performance of extracted speech waveform. However, it has been reported that a TSE system delivers high reconstruction performance may still suffer low-quality experience problems in practice. One such experience problem is wrong speaker extraction (called speaker confusion, SC), which leads to strong negative experience and hampers effective conversations. To mitigate the imperative SC issue, we reformulate the training objective and propose two novel loss schemes that explore the metric of reconstruction improvement performance defined at small chunk-level and leverage the metric associated distribution information. Both loss schemes aim to encourage a TSE network to pay attention to those SC chunks based on the said distribution information. On this basis, we present X-SepFormer, an end-to-end TSE model with proposed loss schemes and a backbone of SepFormer. Experimental results on the benchmark WSJ0-2mix dataset validate the effectiveness of our proposals, showing consistent improvements on SC errors (by 14.8% relative). Moreover, with SI-SDRi of 19.4 dB and PESQ of 3.81, our best system significantly outperforms the current SOTA systems and offers the top TSE results reported till date on the WSJ0-2mix.
eess.AS音频处理

【1】 X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion

标题：X-SepFormer：对说话人混淆进行显式优化的端到端说话人提取网络

链接：http://arxiv.org/pdf/2303.05023v1

*与cs.SD语音【7】为同一篇

作者：Kai Liu,Ziqing Du,Xucheng Wan,Huan Zhou
机构：AI Application Research Center, Huawei Technologies Co., Ltd., Shenzhen, China
备注：Accepted by ICASSP 2023
摘要：目标语音提取（TSE）系统被设计成从多说话者混合中提取目标语音。对于大多数现有的TSE网络，普遍的训练目标是提高提取的语音波形的重构性能。然而，据报道，提供高重建性能的TSE系统在实践中可能仍然遭受低质量体验问题。其中一个经验问题是错误的说话人提取（称为说话人混淆，SC），它导致强烈的负面经验，并阻碍有效的对话。为了缓解强制性SC问题，我们重新制定了训练目标，并提出了两种新的损失方案，探索了在小块级别定义的重建改进性能的度量，并利用了度量相关的分布信息。两种丢失方案旨在鼓励TSE网络基于所述分布信息来关注那些SC组块。在此基础上，我们提出了一个端到端的TSE模型X-SepFormer，它是SepFormer的主干。在基准WSJ 0 - 2 mix数据集上的实验结果验证了本文方法的有效性，SC误差得到了一致的改善（相对改善14.8%）.此外，SI-SDRi为19.4 dB，PESQ为3.81，我们的最佳系统显著优于当前的SOTA系统，并且提供了迄今为止在WSJ 0 - 2 mix上报告的最佳TSE结果。
摘要：Target speech extraction (TSE) systems are designed to extract target speech from a multi-talker mixture. The popular training objective for most prior TSE networks is to enhance reconstruction performance of extracted speech waveform. However, it has been reported that a TSE system delivers high reconstruction performance may still suffer low-quality experience problems in practice. One such experience problem is wrong speaker extraction (called speaker confusion, SC), which leads to strong negative experience and hampers effective conversations. To mitigate the imperative SC issue, we reformulate the training objective and propose two novel loss schemes that explore the metric of reconstruction improvement performance defined at small chunk-level and leverage the metric associated distribution information. Both loss schemes aim to encourage a TSE network to pay attention to those SC chunks based on the said distribution information. On this basis, we present X-SepFormer, an end-to-end TSE model with proposed loss schemes and a backbone of SepFormer. Experimental results on the benchmark WSJ0-2mix dataset validate the effectiveness of our proposals, showing consistent improvements on SC errors (by 14.8% relative). Moreover, with SI-SDRi of 19.4 dB and PESQ of 3.81, our best system significantly outperforms the current SOTA systems and offers the top TSE results reported till date on the WSJ0-2mix.

【2】 TOLD: A Novel Two-Stage Overlap-Aware Framework for Speaker Diarization

标题：Tell：一种新的两阶段重叠感知说话人二值化框架

链接：http://arxiv.org/pdf/2303.05397v1

*与cs.SD语音【1】为同一篇

作者：Jiaming Wang,Zhihao Du,Shiliang Zhang
机构：Speech Lab, Alibaba Group, China
备注：Accepted by ICASSP2023
摘要：近年来，端到端神经元日记化（EEND）被引入并在说话人重叠的场景中取得了良好的效果。在EEND中，说话人的日记化被描述为一个多标记预测问题，说话人活动是独立估计的，没有很好地考虑它们之间的相关性。为了克服这些缺点，本文采用幂集编码将说话人二分法转化为单标记分类问题，提出了重叠感知的EEND（EEND-OLA）模型，该模型可以显式地描述说话人的重叠和相关性。受两级混合系统成功的启发，我们进一步提出了一种新的两级重叠感知Diarization框架（TOLD），通过引入说话人重叠感知后处理（SOAP）模型来迭代地改进EEND-OLA的Diarization结果。实验结果表明，与原始的EEND相比，EEND-OLA在二值化错误率（DER）方面获得了14.39%的相对改善，而SOAP的使用又提供了19.33%的相对改善。因此，我们的方法TOLD在CALLHOME数据集上实现了10.14%的DER，据我们所知，这是该基准测试的最新结果。
摘要：Recently, end-to-end neural diarization (EEND) is introduced and achieves promising results in speaker-overlapped scenarios. In EEND, speaker diarization is formulated as a multi-label prediction problem, where speaker activities are estimated independently and their dependency are not well considered. To overcome these disadvantages, we employ the power set encoding to reformulate speaker diarization as a single-label classification problem and propose the overlap-aware EEND (EEND-OLA) model, in which speaker overlaps and dependency can be modeled explicitly. Inspired by the success of two-stage hybrid systems, we further propose a novel Two-stage OverLap-aware Diarization framework (TOLD) by involving a speaker overlap-aware post-processing (SOAP) model to iteratively refine the diarization results of EEND-OLA. Experimental results show that, compared with the original EEND, the proposed EEND-OLA achieves a 14.39% relative improvement in terms of diarization error rates (DER), and utilizing SOAP provides another 19.33% relative improvement. As a result, our method TOLD achieves a DER of 10.14% on the CALLHOME dataset, which is a new state-of-the-art result on this benchmark to the best of our knowledge.

【3】 MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning

标题：MMCosine：面向均衡视听细粒度学习的多模式余弦损失

 链接：http://arxiv.org/pdf/2303.05338v1

*与cs.SD语音【2】为同一篇

作者：Ruize Xu,Ruoxuan Feng,Shi-Xiong Zhang,Di Hu
机构：Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China, Tencent AI Lab, Bellevue, WA, USA
摘要：视听学习通过融合多种形式的实用信息，有助于全面理解世界。然而，最近的研究表明，在联合学习模型中，单模态编码器的不平衡优化是提高模型性能的瓶颈。我们进一步发现，现有的不平衡抑制方法在一些对可区分特征分布要求较高的细粒度视听任务上表现不佳。基于余弦损失法构建超球面特征空间和降低类内角度变化的成功经验，提出了多模态余弦损失法（Multi-Modal Cosine Loss，MMCosine）。它对特征和权重进行模态方面的$L_2$归一化，以实现平衡和更好的多模态细粒度学习。从权范数的角度证明了该方法能够有效地缓解非平衡优化问题，并充分利用余弦度量的可区分性。大量的实验证明了该方法的有效性以及与先进的多模态融合策略和最新的不平衡抑制方法的通用性。
摘要：Audio-visual learning helps to comprehensively understand the world by fusing practical information from multiple modalities. However, recent studies show that the imbalanced optimization of uni-modal encoders in a joint-learning model is a bottleneck to enhancing the model's performance. We further find that the up-to-date imbalance-mitigating methods fail on some audio-visual fine-grained tasks, which have a higher demand for distinguishable feature distribution. Fueled by the success of cosine loss that builds hyperspherical feature spaces and achieves lower intra-class angular variability, this paper proposes Multi-Modal Cosine loss, MMCosine. It performs a modality-wise $L_2$ normalization to features and weights towards balanced and better multi-modal fine-grained learning. We demonstrate that our method can alleviate the imbalanced optimization from the perspective of weight norm and fully exploit the discriminability of the cosine metric. Extensive experiments prove the effectiveness of our method and the versatility with advanced multi-modal fusion strategies and up-to-date imbalance-mitigating methods.

【4】 Improving Few-Shot Learning for Talking Face System with TTS Data Augmentation

标题：利用TTS数据增强改进说话人脸系统的Few-Shot学习

链接：http://arxiv.org/pdf/2303.05322v1

*与cs.SD语音【3】为同一篇

作者：Qi Chen,Ziyang Ma,Tao Liu,Xu Tan,Qu Lu,Xie Chen,Kai Yu
机构：X-LANCE Lab, Department of Computer Science and Engineering, MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China, Microsoft Research Asia,Shanghai Media Tech
备注：4 pages. Accepted by ICASSP 2023
摘要：近年来，音频驱动的人脸识别技术引起了学术界和工业界的广泛关注。然而，音频驱动的说话脸中的数据采集和标记是劳动密集型和成本高昂的。数据资源的缺乏导致合成效果不佳。为了解决这个问题，我们提出使用TTS（Text-To-Speech）进行数据扩充，以提高说话人脸系统的Few-Shot能力。软DTW的引入解决了TTS音频带来的错位问题，并首次应用于人脸任务。此外，HuBERT提取的特征被探索以利用音频的潜在信息，并且发现其优于其他特征。与基线模型相比，该方法在MSE、DTW和用户学习偏好上分别取得了17%、14%和38%的优势，证明了TTS增强技术对人脸系统Few-Shot学习的有效性。
摘要：Audio-driven talking face has attracted broad interest from academia and industry recently. However, data acquisition and labeling in audio-driven talking face are labor-intensive and costly. The lack of data resource results in poor synthesis effect. To alleviate this issue, we propose to use TTS (Text-To-Speech) for data augmentation to improve few-shot ability of the talking face system. The misalignment problem brought by the TTS audio is solved with the introduction of soft-DTW, which is first adopted in the talking face task. Moreover, features extracted by HuBERT are explored to utilize underlying information of audio, and found to be superior over other features. The proposed method achieves 17%, 14%, 38% dominance on MSE score, DTW score and user study preference repectively over the baseline model, which shows the effectiveness of improving few-shot learning for talking face system with TTS augmentation.

【5】 WASD: A Wilder Active Speaker Detection Dataset

标题：WASD：一种Wilder主动说话人检测数据集

链接：http://arxiv.org/pdf/2303.05321v1

*与cs.SD语音【4】为同一篇

作者：Tiago Roxo,Joana C. Costa,Pedro R. M. Inácio,Hugo Proença
机构：IT - Instituto de Telecomunicac¸˜oes, University of Beira Interior, Portugal
摘要：现有的主动说话人检测（ASD）模型仅利用声音和人脸特征，在AVA-ActiveSpeaker（AVA）上取得了很好的效果。虽然这种方法适用于电影设置（AVA），但它不适合约束较少的条件。为了证明这一局限性，我们提出了Wilder主动说话人检测（WASD）数据集，通过针对当前ASD的两个关键组件来增加难度：音频和面部。WASD分为5个类别，从最佳条件到监控设置，包含音频和面部数据战术损伤对ASD的增量挑战。我们选择最先进的型号，并在两组WASD中评估其性能：“简单”（协作设置）和“困难”（音频和 或面部被专门降级）。结果表明：1）AVA训练的模型在WASD Easy组中保持了最先进的性能，而在Hard组中表现不佳，显示了2）AVA和Easy数据之间的相似性;以及3）WASD中的训练没有将模型性能提高到AVA水平，特别是对于音频损伤和监视设置。这表明AVA没有为野生型ASD准备模型，目前的方法在处理此类条件方面不合格。申报的数据集还包含身体数据注释，为ASD提供了新的来源，可在www.example.com上获得https: github.com Tiago-Roxo WASD。
摘要：Current Active Speaker Detection (ASD) models achieve great results on AVA-ActiveSpeaker (AVA), using only sound and facial features. Although this approach is applicable in movie setups (AVA), it is not suited for less constrained conditions. To demonstrate this limitation, we propose a Wilder Active Speaker Detection (WASD) dataset, with increased difficulty by targeting the two key components of current ASD: audio and face. Grouped into 5 categories, ranging from optimal conditions to surveillance settings, WASD contains incremental challenges for ASD with tactical impairment of audio and face data. We select state-of-the-art models and assess their performance in two groups of WASD: Easy (cooperative settings) and Hard (audio and or face are specifically degraded). The results show that: 1) AVA trained models maintain a state-of-the-art performance in WASD Easy group, while underperforming in the Hard one, showing the 2) similarity between AVA and Easy data; and 3) training in WASD does not improve models performance to AVA levels, particularly for audio impairment and surveillance settings. This shows that AVA does not prepare models for wild ASD and current approaches are subpar to deal with such conditions. The proposed dataset also contains body data annotations to provide a new source for ASD, and is available at https: github.com Tiago-Roxo WASD.

【6】 hierarchical network with decoupled knowledge distillation for speech emotion recognition

标题：用于语音情感识别的解耦知识提取的层次网络

链接：http://arxiv.org/pdf/2303.05134v1

*与cs.SD语音【5】为同一篇

作者：Ziping Zhao,Huan Wang,Haishuai Wang,Bjorn Schuller
机构：College of Computer and Information Engineering, Tianjin Normal University, Tianjin, China, College of Computer Science, Zhejiang University, China, Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Germany
备注：5 pages,4 figures,icassp 2023
摘要：语音情感识别（SER）的目标是使计算机能够以与人类相同的方式识别给定话语的情感类别。SER的准确性很大程度上取决于由该模型获得的话语级别表示的有效性。然而，非目标阶层所携带的“暗知识”却一直被先前的研究所忽略。本文提出了一种层次化网络DKDFMH，它采用了深度卷积神经网络中的解耦知识提取和融合多头注意机制。该方法利用logit蒸馏从不同尺度的注意集获取更高层次的语义特征，深入挖掘非目标类所携带的知识，从而引导模型更加关注情感特征之间的差异。为了验证该模型的有效性，在交互式情感二进运动捕获（IEMOCAP）数据集上进行了实验。我们实现了具有竞争力的性能，加权准确率（WA）为79.1%，未加权准确率（UA）为77.1%。据我们所知，这是自2015年以来，Logit蒸馏首次回归最先进状态。
摘要：The goal of Speech Emotion Recognition (SER) is to enable computers to recognize the emotion category of a given utterance in the same way that humans do. The accuracy of SER is strongly dependent on the validity of the utterance-level representation obtained by the model. Nevertheless, the dark knowledge" carried by non-target classes is always ignored by previous studies. In this paper, we propose a hierarchical network, called DKDFMH, which employs decoupled knowledge distillation in a deep convolutional neural network with a fused multi-head attention mechanism. Our approach applies logit distillation to obtain higher-level semantic features from different scales of attention sets and delve into the knowledge carried by non-target classes, thus guiding the model to focus more on the differences between sentiment features. To validate the effectiveness of our model, we conducted experiments on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. We achieved competitive performance, with 79.1% weighted accuracy (WA) and 77.1% unweighted accuracy (UA). To the best of our knowledge, this is the first time since 2015 that logit distillation has been returned to state-of-the-art status.

【7】 Towards Robust Image-in-Audio Deep Steganography

标题：一种稳健的音频图像深度隐写算法

链接：http://arxiv.org/pdf/2303.05007v1

*与cs.SD语音【6】为同一篇

作者：Jaume Ros Alonso,Margarita Geleta,Jordi Pons,Xavier Giro-i-Nieto
机构：Universitat Politecnica de Catalunya, UC Berkeley, Dolby Laboratories
备注：8 pages, 5 figures, 2 tables
摘要：由于人工智能技术的最新进展，特别是在能够在不同性质的信号内隐藏信号的多模式设置的背景下，隐写术领域经历了兴趣的激增。所有隐写方法的主要目标都是实现感知透明性、鲁棒性和大的嵌入容量--这往往会出现经典方法难以调和的冲突目标。该文对现有的一种音频图像深度隐写方法进行了扩展和改进，重点提高了其鲁棒性。所提出的增强包括对损失函数的修改、利用短时傅立叶变换（STFT）、在用于纠错的编码过程中引入冗余、以及在像素子卷积操作中缓冲附加信息。实验结果表明，该方法在鲁棒性和感知透明性方面优于现有方法。
摘要：The field of steganography has experienced a surge of interest due to the recent advancements in AI-powered techniques, particularly in the context of multimodal setups that enable the concealment of signals within signals of a different nature. The primary objectives of all steganographic methods are to achieve perceptual transparency, robustness, and large embedding capacity - which often present conflicting goals that classical methods have struggled to reconcile. This paper extends and enhances an existing image-in-audio deep steganography method by focusing on improving its robustness. The proposed enhancements include modifications to the loss function, utilization of the Short-Time Fourier Transform (STFT), introduction of redundancy in the encoding process for error correction, and buffering of additional information in the pixel subconvolution operation. The results demonstrate that our approach outperforms the existing method in terms of robustness and perceptual transparency.

机器翻译，仅供参考