今日论文合集：cs.SD语音6篇，eess.AS音频处理7篇。
cs.SD语音
【1】 Efficient Monaural Speech Enhancement using Spectrum Attention Fusion标题：基于频谱注意力融合的高效单声道语音增强链接：https://arxiv.org/abs/2308.02263
作者：Jinyu Long,Jetic Gū,Binhao Bai,Zhibo Yang,Ping Wei,Junli Li机构：School of Computing Science, Sichuan Normal University, Chengdu , China,  School of Computing Science, Simon Fraser University, Burnaby, BC V,A ,S, Canada,  Visual Computing and Virtual Reality Key Laboratory of Sichuan, Sichuan Normal摘要：语音增强是自动语音处理流水线中的一项要求很高的任务，其重点是将干净的语音与噪声通道分离。基于Transformer的模型最近在语音增强方面击败了RNN和CNN模型，但同时它们的计算成本要高得多，并且需要更多高质量的训练数据，这总是很难获得。在本文中，我们提出了一种改进的语音增强模型，保持自我关注的表现力，同时显着降低模型的复杂性，我们称之为频谱注意力融合。我们仔细构建了一个卷积模块来替换语音Transformer中的几个自注意层，使模型能够更有效地融合频谱特征。我们提出的模型能够实现与SOTA模型相当或更好的结果，但在Voice Bank + DEMAND数据集上具有显着较小的参数（0.58M）。摘要：Speech enhancement is a demanding task in automated speech processing pipelines, focusing on separating clean speech from noisy channels. Transformer based models have recently bested RNN and CNN models in speech enhancement, however at the same time they are much more computationally expensive and require much more high quality training data, which is always hard to come by. In this paper, we present an improvement for speech enhancement models that maintains the expressiveness of self-attention while significantly reducing model complexity, which we have termed Spectrum Attention Fusion. We carefully construct a convolutional module to replace several self-attention layers in a speech Transformer, allowing the model to more efficiently fuse spectral features. Our proposed model is able to achieve comparable or better results against SOTA models but with significantly smaller parameters (0.58M) on the Voice Bank + DEMAND dataset.
【2】 Finding Tori: Self-supervised Learning for Analyzing Korean Folk Song标题：寻找Tori：分析朝鲜族民歌的自我监督学习链接：https://arxiv.org/abs/2308.02249
作者：Danbinaerin Han,Rafael Caro Repetto,Dasaem Jeong机构：Department of Art & Technology, Sogang University, Kunstuniversität Graz备注：Accepted at 24th International Society for Music Information Retrieval Conference (ISMIR 2023)摘要：在本文中，我们介绍了大约700小时的韩国民歌，这是在20世纪80 - 90年代录制的现场录音数据集的计算分析。由于大多数歌曲都是由非专业音乐家在没有伴奏的情况下演唱的，因此数据集提供了一些挑战。为了解决这一挑战，我们利用基于音高轮廓的卷积神经网络进行自监督学习，然后分析了模型如何捕获tori的音乐概念，tori是一个由特定音阶，装饰音符和惯用旋律轮廓定义的分类系统。实验结果表明，我们的方法可以更好地捕捉环面的特点相比，传统的基音直方图。使用我们的方法，我们已经研究了如何在现有的学术界提出的音乐讨论表现在韩国民歌的实际现场录音。摘要：In this paper, we introduce a computational analysis of the field recording dataset of approximately 700 hours of Korean folk songs, which were recorded around 1980-90s. Because most of the songs were sung by non-expert musicians without accompaniment, the dataset provides several challenges. To address this challenge, we utilized self-supervised learning with convolutional neural network based on pitch contour, then analyzed how the musical concept of tori, a classification system defined by a specific scale, ornamental notes, and an idiomatic melodic contour, is captured by the model. The experimental result shows that our approach can better capture the characteristics of tori compared to traditional pitch histograms. Using our approaches, we have examined how musical discussions proposed in existing academia manifest in the actual field recordings of Korean folk songs.
【3】 Emo-DNA: Emotion Decoupling and Alignment Learning for Cross-Corpus  Speech Emotion Recognition标题：EMO-DNA：跨语料库语音情感识别的情感解耦和对齐学习链接：https://arxiv.org/abs/2308.02190
作者：Jiaxin Ye,Yujie Wei,Xin-Cheng Wen,Chenglong Ma,Zhizhong Huang,Kunhong Liu,Hongming Shan机构：ISTBI, Fudan University, Harbin Institute of Technology, School of CS, Fudan University, School of Film, Xiamen University, ISTBI & MOE FCBS, Fudan University备注：Accepted by ACM MM 2023摘要：跨语料库语音情感识别（SER）旨在将从标注良好的语料库中推断语音情感的能力推广到未标注的语料库中，由于两个语料库之间存在显著差异，这是一项相当具有挑战性的任务。现有的方法，通常是基于无监督域自适应（UDA），努力学习语料库不变的功能，通过全球分布对齐，但不幸的是，所得到的功能是混合与语料库特定的功能或不类歧视。为了解决这些挑战，我们提出了一种新的情感解耦aNd对齐学习框架（EMO-DNA）的跨语料库SER，一种新的UDA方法来学习情感相关的语料库不变的功能。EMO-DNA的新颖性有两个方面：对比性情绪解耦和双水平情绪对齐。一方面，我们的对比情感解耦通过对比解耦损失来实现解耦学习，以加强情感相关特征与语料库特定特征的可分离性。另一方面，我们的双层情感对齐引入了自适应阈值伪标记来选择自信的目标样本进行类级对齐，并进行语料库级对齐，以共同指导模型学习跨语料库的类判别语料库不变特征。大量的实验结果表明，在几个跨语料库的情况下，EMO-DNA的性能优越的国家的最先进的方法。源代码可在https://github.com/Jiaxin-Ye/Emo-DNA获得。摘要：Cross-corpus speech emotion recognition (SER) seeks to generalize the ability of inferring speech emotion from a well-labeled corpus to an unlabeled one, which is a rather challenging task due to the significant discrepancy between two corpora. Existing methods, typically based on unsupervised domain adaptation (UDA), struggle to learn corpus-invariant features by global distribution alignment, but unfortunately, the resulting features are mixed with corpus-specific features or not class-discriminative. To tackle these challenges, we propose a novel Emotion Decoupling aNd Alignment learning framework (EMO-DNA) for cross-corpus SER, a novel UDA method to learn emotion-relevant corpus-invariant features. The novelties of EMO-DNA are two-fold: contrastive emotion decoupling and dual-level emotion alignment. On one hand, our contrastive emotion decoupling achieves decoupling learning via a contrastive decoupling loss to strengthen the separability of emotion-relevant features from corpus-specific ones. On the other hand, our dual-level emotion alignment introduces an adaptive threshold pseudo-labeling to select confident target samples for class-level alignment, and performs corpus-level alignment to jointly guide model for learning class-discriminative corpus-invariant features across corpora. Extensive experimental results demonstrate the superior performance of EMO-DNA over the state-of-the-art methods in several cross-corpus scenarios. Source code is available at https://github.com/Jiaxin-Ye/Emo-DNA.
【4】 N-gram Boosting: Improving Contextual Biasing with Normalized N-gram  Targets标题：N元语法增强：利用归一化N元语法目标改进上下文偏向链接：https://arxiv.org/abs/2308.02092
作者：Wang Yau Li,Shreekantha Nadig,Karol Chang,Zafarullah Mahmood,Riqiang Wang,Simon Vandieken,Jonas Robertson,Fred Mailhot机构：Dialpad Inc., Canada摘要：专有名称和技术术语的准确转录在用于商业对话的语音到文本应用中尤其重要。这些对于理解对话至关重要的单词通常很少见，因此在文本和音频训练数据中可能代表性不足，这在该领域带来了重大挑战。我们提出了一种两步关键字提升机制，该机制成功地适用于归一化的unigrams和n-gram，而不仅仅是单个令牌，这消除了提升原始目标的遗漏命中问题。此外，我们还展示了如何调整提升权重逻辑，避免过度提升多令牌关键字。这将我们的关键字识别率提高了26%，相对于我们专有的域内数据集和LibriSpeech的2%。此方法对于涉及非字母字符或具有非标准发音的目标特别有用。摘要：Accurate transcription of proper names and technical terms is particularly important in speech-to-text applications for business conversations. These words, which are essential to understanding the conversation, are often rare and therefore likely to be under-represented in text and audio training data, creating a significant challenge in this domain. We present a two-step keyword boosting mechanism that successfully works on normalized unigrams and n-grams rather than just single tokens, which eliminates missing hits issues with boosting raw targets. In addition, we show how adjusting the boosting weight logic avoids over-boosting multi-token keywords. This improves our keyword recognition rate by 26% relative on our proprietary in-domain dataset and 2% on LibriSpeech. This method is particularly useful on targets that involve non-alphabetic characters or have non-standard pronunciations.
【5】 Federated Representation Learning for Automatic Speech Recognition标题：用于自动语音识别的联合表示学习链接：https://arxiv.org/abs/2308.02013
作者：Guruprasad V Rames,Gopinath Chennupati,Milind Rao,Anit Kumar Sahu,Ariya Rastrow,Jasha Droppo机构：University of Wisconsin-Madison, Amazon Alexa USA备注：Accepted at ISCA SPSC Symposium 3rd Symposium on Security and Privacy in Speech Communication, 2023摘要：联合学习（FL）是一种隐私保护范式，允许边缘设备在不共享数据的情况下协作学习。像Alexa和Siri这样的边缘设备是未标记音频数据的潜在来源，可以利用这些数据来学习强大的音频表示。在这项工作中，我们将自监督学习（SSL）和FL结合在一起，学习自动语音识别的表示尊重数据隐私约束。我们使用未标记语音数据集Libri-Light中的说话者和章节信息来模拟非IID说话者孤立的数据分布，并使用FedSGD的对比预测编码框架预训练LSTM编码器。我们表明，FL中的预训练ASR编码器的性能与中央预训练模型一样好，与没有预训练相比，提高了12-15%（WER）。我们进一步将联邦预训练模型适应于新的语言法语，并显示出比没有预训练提高20%（WER）。摘要：Federated Learning (FL) is a privacy-preserving paradigm, allowing edge devices to learn collaboratively without sharing data. Edge devices like Alexa and Siri are prospective sources of unlabeled audio data that can be tapped to learn robust audio representations. In this work, we bring Self-supervised Learning (SSL) and FL together to learn representations for Automatic Speech Recognition respecting data privacy constraints. We use the speaker and chapter information in the unlabeled speech dataset, Libri-Light, to simulate non-IID speaker-siloed data distributions and pre-train an LSTM encoder with the Contrastive Predictive Coding framework with FedSGD. We show that the pre-trained ASR encoder in FL performs as well as a centrally pre-trained model and produces an improvement of 12-15% (WER) compared to no pre-training. We further adapt the federated pre-trained models to a new language, French, and show a 20% (WER) improvement over no pre-training.
【6】 DCTM: Dilated Convolutional Transformer Model for Multimodal Engagement  Estimation in Conversation标题：DCTM：会话中多通道参与度估计的扩展卷积变换模型链接：https://arxiv.org/abs/2308.01966
作者：Vu Ngoc Tu,Van Thong Huynh,Hyung-Jeong Yang,M. Zaigham Zaheer,Shah Nawaz,Karthik Nandakumar,Soo-Hyung Kim机构：Chonnam National University, Gwangju, South Korea, Mohamed bin Zayed University of AI, Masdar City, Abu Dhabi, UAE, Deutsches Elektronen-Synchrotron, Hamburg, Germany备注：Accepted in ACMM Grand Challenge摘要：会话参与度估计是一个回归问题，需要识别的有利的注意力和参与的参与者在对话中。这项任务作为一个关键的追求，以获得深入了解人类的互动动态和行为模式中的对话。在这项研究中，我们引入了一个扩展的卷积Transformer，用于建模和估计MULTIMEDIATE 2023竞赛中的人类参与。我们提出的系统超越了基线模型，表现出显着的7 $\%的改进测试集和4 $\%的验证集。此外，我们采用不同的模态融合机制，并表明，对于这种类型的数据，一个简单的级联方法与自注意力融合获得最佳性能。摘要：Conversational engagement estimation is posed as a regression problem, entailing the identification of the favorable attention and involvement of the participants in the conversation. This task arises as a crucial pursuit to gain insights into human's interaction dynamics and behavior patterns within a conversation. In this research, we introduce a dilated convolutional Transformer for modeling and estimating human engagement in the MULTIMEDIATE 2023 competition. Our proposed system surpasses the baseline models, exhibiting a noteworthy $7$\% improvement on test set and $4$\% on validation set. Moreover, we employ different modality fusion mechanism and show that for this type of data, a simple concatenated method with self-attention fusion gains the best performance.
eess.AS音频处理
【1】 Differentiable adaptive short-time Fourier transform with respect to the  window length标题：关于窗口长度的可微自适应短时傅里叶变换链接：https://arxiv.org/abs/2308.02418
作者：Maxime Leiber,Yosra Marnissi,Axel Barrau,Mohammed El Badaoui机构：∗ INRIA, DIENS, PSL Research University, † Safran Tech, Digital Sciences & Technologies, ‡ Offroad, § Univ Lyon, UJM-St-Etienne, LASPI备注：Accepted for IEEE International Conference on Acoustics, Speech, and SIgnal Processing (ICASSP) 2023摘要：本文提出了一种基于梯度的方法对每帧和每频率的窗口长度的短时傅里叶变换（STFT），相关的前期工作中，我们开发了一个可微版本的STFT通过使窗口长度的连续参数的飞行优化。由此产生的可微自适应STFT具有值得称道的属性，如能够适应在相同的时间-频率表示的瞬态和稳态分量，同时很容易通过梯度下降优化。我们验证了我们的方法在振动分析的性能。摘要：This paper presents a gradient-based method for on-the-fly optimization for both per-frame and per-frequency window length of the short-time Fourier transform (STFT), related to previous work in which we developed a differentiable version of STFT by making the window length a continuous parameter. The resulting differentiable adaptive STFT possesses commendable properties, such as the ability to adapt in the same time-frequency representation to both transient and stationary components, while being easily optimized by gradient descent. We validate the performance of our method in vibration analysis.
【2】 Efficient Monaural Speech Enhancement using Spectrum Attention Fusion标题：基于频谱注意力融合的高效单声道语音增强链接：https://arxiv.org/abs/2308.02263
作者：Jinyu Long,Jetic Gū,Binhao Bai,Zhibo Yang,Ping Wei,Junli Li机构：School of Computing Science, Sichuan Normal University, Chengdu , China,  School of Computing Science, Simon Fraser University, Burnaby, BC V,A ,S, Canada,  Visual Computing and Virtual Reality Key Laboratory of Sichuan, Sichuan Normal摘要：语音增强是自动语音处理流水线中的一项要求很高的任务，其重点是将干净的语音与噪声通道分离。基于Transformer的模型最近在语音增强方面击败了RNN和CNN模型，但同时它们的计算成本要高得多，并且需要更多高质量的训练数据，这总是很难获得。在本文中，我们提出了一种改进的语音增强模型，保持自我关注的表现力，同时显着降低模型的复杂性，我们称之为频谱注意力融合。我们仔细构建了一个卷积模块来替换语音Transformer中的几个自注意层，使模型能够更有效地融合频谱特征。我们提出的模型能够实现与SOTA模型相当或更好的结果，但在Voice Bank + DEMAND数据集上具有显着较小的参数（0.58M）。摘要：Speech enhancement is a demanding task in automated speech processing pipelines, focusing on separating clean speech from noisy channels. Transformer based models have recently bested RNN and CNN models in speech enhancement, however at the same time they are much more computationally expensive and require much more high quality training data, which is always hard to come by. In this paper, we present an improvement for speech enhancement models that maintains the expressiveness of self-attention while significantly reducing model complexity, which we have termed Spectrum Attention Fusion. We carefully construct a convolutional module to replace several self-attention layers in a speech Transformer, allowing the model to more efficiently fuse spectral features. Our proposed model is able to achieve comparable or better results against SOTA models but with significantly smaller parameters (0.58M) on the Voice Bank + DEMAND dataset.
【3】 Finding Tori: Self-supervised Learning for Analyzing Korean Folk Song标题：寻找Tori：分析朝鲜族民歌的自我监督学习链接：https://arxiv.org/abs/2308.02249
作者：Danbinaerin Han,Rafael Caro Repetto,Dasaem Jeong机构：Department of Art & Technology, Sogang University, Kunstuniversität Graz备注：Accepted at 24th International Society for Music Information Retrieval Conference (ISMIR 2023)摘要：在本文中，我们介绍了大约700小时的韩国民歌，这是在20世纪80 - 90年代录制的现场录音数据集的计算分析。由于大多数歌曲都是由非专业音乐家在没有伴奏的情况下演唱的，因此数据集提供了一些挑战。为了解决这一挑战，我们利用基于音高轮廓的卷积神经网络进行自监督学习，然后分析了模型如何捕获tori的音乐概念，tori是一个由特定音阶，装饰音符和惯用旋律轮廓定义的分类系统。实验结果表明，我们的方法可以更好地捕捉环面的特点相比，传统的基音直方图。使用我们的方法，我们已经研究了如何在现有的学术界提出的音乐讨论表现在韩国民歌的实际现场录音。摘要：In this paper, we introduce a computational analysis of the field recording dataset of approximately 700 hours of Korean folk songs, which were recorded around 1980-90s. Because most of the songs were sung by non-expert musicians without accompaniment, the dataset provides several challenges. To address this challenge, we utilized self-supervised learning with convolutional neural network based on pitch contour, then analyzed how the musical concept of tori, a classification system defined by a specific scale, ornamental notes, and an idiomatic melodic contour, is captured by the model. The experimental result shows that our approach can better capture the characteristics of tori compared to traditional pitch histograms. Using our approaches, we have examined how musical discussions proposed in existing academia manifest in the actual field recordings of Korean folk songs.
【4】 Emo-DNA: Emotion Decoupling and Alignment Learning for Cross-Corpus  Speech Emotion Recognition标题：EMO-DNA：跨语料库语音情感识别的情感解耦和对齐学习链接：https://arxiv.org/abs/2308.02190
作者：Jiaxin Ye,Yujie Wei,Xin-Cheng Wen,Chenglong Ma,Zhizhong Huang,Kunhong Liu,Hongming Shan机构：ISTBI, Fudan University, Harbin Institute of Technology, School of CS, Fudan University, School of Film, Xiamen University, ISTBI & MOE FCBS, Fudan University备注：Accepted by ACM MM 2023摘要：跨语料库语音情感识别（SER）旨在将从标注良好的语料库中推断语音情感的能力推广到未标注的语料库中，由于两个语料库之间存在显著差异，这是一项相当具有挑战性的任务。现有的方法，通常是基于无监督域自适应（UDA），努力学习语料库不变的功能，通过全球分布对齐，但不幸的是，所得到的功能是混合与语料库特定的功能或不类歧视。为了解决这些挑战，我们提出了一种新的情感解耦aNd对齐学习框架（EMO-DNA）的跨语料库SER，一种新的UDA方法来学习情感相关的语料库不变的功能。EMO-DNA的新颖性有两个方面：对比性情绪解耦和双水平情绪对齐。一方面，我们的对比情感解耦通过对比解耦损失来实现解耦学习，以加强情感相关特征与语料库特定特征的可分离性。另一方面，我们的双层情感对齐引入了自适应阈值伪标记来选择自信的目标样本进行类级对齐，并进行语料库级对齐，以共同指导模型学习跨语料库的类判别语料库不变特征。大量的实验结果表明，在几个跨语料库的情况下，EMO-DNA的性能优越的国家的最先进的方法。源代码可在https://github.com/Jiaxin-Ye/Emo-DNA获得。摘要：Cross-corpus speech emotion recognition (SER) seeks to generalize the ability of inferring speech emotion from a well-labeled corpus to an unlabeled one, which is a rather challenging task due to the significant discrepancy between two corpora. Existing methods, typically based on unsupervised domain adaptation (UDA), struggle to learn corpus-invariant features by global distribution alignment, but unfortunately, the resulting features are mixed with corpus-specific features or not class-discriminative. To tackle these challenges, we propose a novel Emotion Decoupling aNd Alignment learning framework (EMO-DNA) for cross-corpus SER, a novel UDA method to learn emotion-relevant corpus-invariant features. The novelties of EMO-DNA are two-fold: contrastive emotion decoupling and dual-level emotion alignment. On one hand, our contrastive emotion decoupling achieves decoupling learning via a contrastive decoupling loss to strengthen the separability of emotion-relevant features from corpus-specific ones. On the other hand, our dual-level emotion alignment introduces an adaptive threshold pseudo-labeling to select confident target samples for class-level alignment, and performs corpus-level alignment to jointly guide model for learning class-discriminative corpus-invariant features across corpora. Extensive experimental results demonstrate the superior performance of EMO-DNA over the state-of-the-art methods in several cross-corpus scenarios. Source code is available at https://github.com/Jiaxin-Ye/Emo-DNA.
【5】 N-gram Boosting: Improving Contextual Biasing with Normalized N-gram  Targets标题：N元语法增强：利用归一化N元语法目标改进上下文偏向链接：https://arxiv.org/abs/2308.02092
作者：Wang Yau Li,Shreekantha Nadig,Karol Chang,Zafarullah Mahmood,Riqiang Wang,Simon Vandieken,Jonas Robertson,Fred Mailhot机构：Dialpad Inc., Canada摘要：专有名称和技术术语的准确转录在用于商业对话的语音到文本应用中尤其重要。这些对于理解对话至关重要的单词通常很少见，因此在文本和音频训练数据中可能代表性不足，这在该领域带来了重大挑战。我们提出了一种两步关键字提升机制，该机制成功地适用于归一化的unigrams和n-gram，而不仅仅是单个令牌，这消除了提升原始目标的遗漏命中问题。此外，我们还展示了如何调整提升权重逻辑，避免过度提升多令牌关键字。这将我们的关键字识别率提高了26%，相对于我们专有的域内数据集和LibriSpeech的2%。此方法对于涉及非字母字符或具有非标准发音的目标特别有用。摘要：Accurate transcription of proper names and technical terms is particularly important in speech-to-text applications for business conversations. These words, which are essential to understanding the conversation, are often rare and therefore likely to be under-represented in text and audio training data, creating a significant challenge in this domain. We present a two-step keyword boosting mechanism that successfully works on normalized unigrams and n-grams rather than just single tokens, which eliminates missing hits issues with boosting raw targets. In addition, we show how adjusting the boosting weight logic avoids over-boosting multi-token keywords. This improves our keyword recognition rate by 26% relative on our proprietary in-domain dataset and 2% on LibriSpeech. This method is particularly useful on targets that involve non-alphabetic characters or have non-standard pronunciations.
【6】 Federated Representation Learning for Automatic Speech Recognition标题：用于自动语音识别的联合表示学习链接：https://arxiv.org/abs/2308.02013
作者：Guruprasad V Rames,Gopinath Chennupati,Milind Rao,Anit Kumar Sahu,Ariya Rastrow,Jasha Droppo机构：University of Wisconsin-Madison, Amazon Alexa USA备注：Accepted at ISCA SPSC Symposium 3rd Symposium on Security and Privacy in Speech Communication, 2023摘要：联合学习（FL）是一种隐私保护范式，允许边缘设备在不共享数据的情况下协作学习。像Alexa和Siri这样的边缘设备是未标记音频数据的潜在来源，可以利用这些数据来学习强大的音频表示。在这项工作中，我们将自监督学习（SSL）和FL结合在一起，学习自动语音识别的表示尊重数据隐私约束。我们使用未标记语音数据集Libri-Light中的说话者和章节信息来模拟非IID说话者孤立的数据分布，并使用FedSGD的对比预测编码框架预训练LSTM编码器。我们表明，FL中的预训练ASR编码器的性能与中央预训练模型一样好，与没有预训练相比，提高了12-15%（WER）。我们进一步将联邦预训练模型适应于新的语言法语，并显示出比没有预训练提高20%（WER）。摘要：Federated Learning (FL) is a privacy-preserving paradigm, allowing edge devices to learn collaboratively without sharing data. Edge devices like Alexa and Siri are prospective sources of unlabeled audio data that can be tapped to learn robust audio representations. In this work, we bring Self-supervised Learning (SSL) and FL together to learn representations for Automatic Speech Recognition respecting data privacy constraints. We use the speaker and chapter information in the unlabeled speech dataset, Libri-Light, to simulate non-IID speaker-siloed data distributions and pre-train an LSTM encoder with the Contrastive Predictive Coding framework with FedSGD. We show that the pre-trained ASR encoder in FL performs as well as a centrally pre-trained model and produces an improvement of 12-15% (WER) compared to no pre-training. We further adapt the federated pre-trained models to a new language, French, and show a 20% (WER) improvement over no pre-training.
【7】 DCTM: Dilated Convolutional Transformer Model for Multimodal Engagement  Estimation in Conversation标题：DCTM：会话中多通道参与度估计的扩展卷积变换模型链接：https://arxiv.org/abs/2308.01966
作者：Vu Ngoc Tu,Van Thong Huynh,Hyung-Jeong Yang,M. Zaigham Zaheer,Shah Nawaz,Karthik Nandakumar,Soo-Hyung Kim机构：Chonnam National University, Gwangju, South Korea, Mohamed bin Zayed University of AI, Masdar City, Abu Dhabi, UAE, Deutsches Elektronen-Synchrotron, Hamburg, Germany备注：Accepted in ACMM Grand Challenge摘要：会话参与度估计是一个回归问题，需要识别的有利的注意力和参与的参与者在对话中。这项任务作为一个关键的追求，以获得深入了解人类的互动动态和行为模式中的对话。在这项研究中，我们引入了一个扩展的卷积Transformer，用于建模和估计MULTIMEDIATE 2023竞赛中的人类参与。我们提出的系统超越了基线模型，表现出显着的7 $\%的改进测试集和4 $\%的验证集。此外，我们采用不同的模态融合机制，并表明，对于这种类型的数据，一个简单的级联方法与自注意力融合获得最佳性能。摘要：Conversational engagement estimation is posed as a regression problem, entailing the identification of the favorable attention and involvement of the participants in the conversation. This task arises as a crucial pursuit to gain insights into human's interaction dynamics and behavior patterns within a conversation. In this research, we introduce a dilated convolutional Transformer for modeling and estimating human engagement in the MULTIMEDIATE 2023 competition. Our proposed system surpasses the baseline models, exhibiting a noteworthy $7$\% improvement on test set and $4$\% on validation set. Moreover, we employ different modality fusion mechanism and show that for this type of data, a simple concatenated method with self-attention fusion gains the best performance.
机器翻译，仅供参考