今日论文合集：cs.SD语音6篇，eess.AS音频处理6篇。     

【1】 Personalized Lightweight Text-to-Speech: Voice Cloning with Adaptive Structured Pruning标题：个性化轻量级文语转换：自适应结构化剪枝的语音克隆链接：http://arxiv.org/pdf/2303.11816v1

作者：Sung-Feng Huang,Chia-ping Chen,Zhi-Sheng Chen,Yu-Pao Tsai,Hung-yi Lee机构：National Taiwan University,Intelligo Technology Inc.备注：ICASSP 2023摘要：个性化TTS是一个令人兴奋和高度期望的应用程序，允许用户只使用几个录音来训练他们的TTS语音。然而，TTS训练通常需要许多小时的记录和大型模型，使得其不适合在移动设备上部署。为了克服这一限制，相关工作通常需要微调预训练的TTS模型，以保持其生成高质量音频样本的能力，同时适应目标说话者的语音。这个过程通常被称为“语音克隆”。尽管相关的工作在改变TTS模型的语音方面已经取得了显著的成功，但是它们仍然需要从大的预先训练的模型进行微调，从而导致语音克隆模型的显著大小。在本文中，我们提出了应用可训练的结构化修剪语音克隆。通过使用语音克隆数据训练结构化修剪掩码，我们可以为每个目标说话人产生唯一的修剪模型。我们的实验表明，使用可学习的结构化修剪，我们可以压缩模型的大小到7倍小，同时实现可比的语音克隆性能。摘要：Personalized TTS is an exciting and highly desired application that allows users to train their TTS voice using only a few recordings. However, TTS training typically requires many hours of recording and a large model, making it unsuitable for deployment on mobile devices. To overcome this limitation, related works typically require fine-tuning a pre-trained TTS model to preserve its ability to generate high-quality audio samples while adapting to the target speaker's voice. This process is commonly referred to as voice cloning.'' Although related works have achieved significant success in changing the TTS model's voice, they are still required to fine-tune from a large pre-trained model, resulting in a significant size for the voice-cloned model. In this paper, we propose applying trainable structured pruning to voice cloning. By training the structured pruning masks with voice-cloning data, we can produce a unique pruned model for each target speaker. Our experiments demonstrate that using learnable structured pruning, we can compress the model size to 7 times smaller while achieving comparable voice-cloning performance.

【2】 ByteCover3: Accurate Cover Song Identification on Short Queries标题：ByteCover3：在短查询上准确识别翻唱歌曲链接：http://arxiv.org/pdf/2303.11692v1

作者：Xingjian Du,Zijie Wang,Xia Liang,Huidong Liang,Bilei Zhu,Zejun Ma机构：ByteDance, Zhejiang University, University of Oxford备注：Accepeted by ICASSP 2023摘要：近年来，基于深度学习的方法已经成为封面歌曲识别（CSI）的范例，其中ByteCover系统在CSI的所有主流数据集上都取得了最先进的结果。然而，随着短视频的蓬勃发展，许多现实世界的应用需要将短音乐摘录与数据库中的全长音乐曲目进行匹配，这仍然是未开发的，并等待工业级的解决方案。在本文中，我们将以前的ByteCover系统升级到ByteCover3，利用本地功能，以进一步提高识别性能的短音乐查询。ByteCover3设计了一个局部对齐丢失（LAL）模块和一个两阶段的特征检索管道，允许系统以更精确和有效的方式执行CSI。我们在具有不同基准设置的多个数据集上评估了ByteCover3，其中ByteCover3击败了所有比较的方法，包括其以前的版本。摘要：Deep learning based methods have become a paradigm for cover song identification (CSI) in recent years, where the ByteCover systems have achieved state-of-the-art results on all the mainstream datasets of CSI. However, with the burgeon of short videos, many real-world applications require matching short music excerpts to full-length music tracks in the database, which is still under-explored and waiting for an industrial-level solution. In this paper, we upgrade the previous ByteCover systems to ByteCover3 that utilizes local features to further improve the identification performance of short music queries. ByteCover3 is designed with a local alignment loss (LAL) module and a two-stage feature retrieval pipeline, allowing the system to perform CSI in a more precise and efficient way. We evaluated ByteCover3 on multiple datasets with different benchmark settings, where ByteCover3 beat all the compared methods including its previous versions.

【3】 Transformers in Speech Processing: A Survey标题：语音处理中的转换器：综述链接：http://arxiv.org/pdf/2303.11607v1

作者：Siddique Latif,Aun Zaidi,Heriberto Cuayahuitl,Fahad Shamshad,Moazzam Shoukat,Junaid Qadir机构：Queensland University of Technology (QUT), Australia, Information Technology University, Pakistan, University of Lincoln, UK, Mohamed bin Zayed University of Artificial Intelligence, EmulationAI, Qatar University, Doha备注：under-review摘要：Transformers在自然语言处理领域的显着成功，引发了语音处理社区的兴趣，导致他们的潜力建模语音序列中的远程依赖关系的探索。最近，Transformers在各种语音相关领域中获得了突出地位，包括自动语音识别、语音合成、语音翻译、语音辅助语言学、语音增强、口语对话系统和许多多模式应用。在本文中，我们提出了一个全面的调查，旨在桥梁语音技术的不同子领域的研究。通过整合整个语音技术领域的研究结果，我们为有兴趣利用Transformers的力量来推进该领域的研究人员提供了宝贵的资源。我们确定了Transformers在语音处理中遇到的挑战，同时也提供了解决这些问题的潜在解决方案的见解。摘要：The remarkable success of transformers in the field of natural language processing has sparked the interest of the speech-processing community, leading to an exploration of their potential for modeling long-range dependencies within speech sequences. Recently, transformers have gained prominence across various speech-related domains, including automatic speech recognition, speech synthesis, speech translation, speech para-linguistics, speech enhancement, spoken dialogue systems, and numerous multimodal applications. In this paper, we present a comprehensive survey that aims to bridge research studies from diverse subfields within speech technology. By consolidating findings from across the speech technology landscape, we provide a valuable resource for researchers interested in harnessing the power of transformers to advance the field. We identify the challenges encountered by transformers in speech processing while also offering insights into potential solutions to address these issues.

【4】 ModEFormer: Modality-Preserving Embedding for Audio-Video Synchronization using Transformers标题：ModEFormer：基于变形器音视频同步的保形嵌入链接：http://arxiv.org/pdf/2303.11551v1

作者：Akash Gupta,Rohun Tripathi,Wondong Jang机构：⋆ New York University, † Amazon Studios备注：Paper accepted at ICASSP 2023摘要：缺乏音频-视频同步是电视广播和视频会议期间的常见问题，导致不令人满意的观看体验。广泛接受的范例是创建错误检测机制，其识别音频领先或滞后的情况。我们提出了ModEFormer，它独立地提取音频和视频嵌入使用特定于模态的Transformers。与其他基于变换器的方法不同，ModEFormer保留了输入流的模态，这允许我们使用更大的批量大小和更多的负音频样本进行对比学习。进一步地，我们建议在一批中的负样本数和唯一样本数之间进行折衷，以显著地超过先前方法的性能。实验结果表明，ModEFormer实现了最先进的性能，94.5%的LRS 2和90.9%的LRS 3。最后，我们演示了如何ModEFormer可以用于偏移检测测试剪辑。摘要：Lack of audio-video synchronization is a common problem during television broadcasts and video conferencing, leading to an unsatisfactory viewing experience. A widely accepted paradigm is to create an error detection mechanism that identifies the cases when audio is leading or lagging. We propose ModEFormer, which independently extracts audio and video embeddings using modality-specific transformers. Different from the other transformer-based approaches, ModEFormer preserves the modality of the input streams which allows us to use a larger batch size with more negative audio samples for contrastive learning. Further, we propose a trade-off between the number of negative samples and number of unique samples in a batch to significantly exceed the performance of previous methods. Experimental results show that ModEFormer achieves state-of-the-art performance, 94.5% for LRS2 and 90.9% for LRS3. Finally, we demonstrate how ModEFormer can be used for offset detection for test clips.

【5】 ICASSP 2023 Deep Speech Enhancement Challenge标题：ICASSP 2023深度语音增强挑战赛链接：http://arxiv.org/pdf/2303.11510v1

作者：Harishchandra Dubey,Ashkan Aazami,Vishak Gopal,Babak Naderi,Sebastian Braun,Ross Cutler,Alex Ju,Mehdi Zohourian,Min Tang,Hannes Gamper,Mehrsa Golestaneh,Robert Aichner机构：Microsoft Corporation, Redmond, USA备注：6 pages, 1 figure. arXiv admin note: text overlap with arXiv:2202.13288摘要：深度语音增强挑战赛是ICASSP 2023信号处理大挑战赛组织的第五届深度噪声抑制（DNS）挑战赛。在2019-2023年期间组织了DNS挑战，以刺激深度语音增强（DSE）的研究。之前的DNS挑战赛是在INTERSPEECH 2020、ICASSP 2021、INTERSPEECH 2021和ICASSP 2022上组织的。从以前的版本中，我们了解到，提高信号质量（SIG）是具有挑战性的，特别是在存在同时活跃的干扰说话者和噪声的情况下。这项挑战旨在开发联合降噪，去混响和抑制干扰说话者的模型。当主要讲话者佩戴耳机时，他们的语音的某些声学特性（诸如直接混响（DRR）、信噪比（SNR）等）使得即使在没有主要讲话者的登记数据的情况下也可以抑制相邻讲话者。这促使我们为这项挑战创建了两条赛道：（i）1轨耳机;（二）轨道2扬声器。两个轨道都具有全频带（48 kHz）训练数据和测试集，并且每个测试剪辑具有针对主要讲话者的对应登记数据（10- 30 s持续时间）。每个轨道邀请提交个性化和非个性化模型，所有这些都通过相同的主观评估进行评估。提交挑战的大多数模型是个性化模型，同一个团队在两个轨道上都是赢家，其中最好的模型与噪声盲测试集相比，挑战得分提高了0.145和0.141。摘要：Deep Speech Enhancement Challenge is the 5th edition of deep noise suppression (DNS) challenges organized at ICASSP 2023 Signal Processing Grand Challenges. DNS challenges were organized during 2019-2023 to stimulate research in deep speech enhancement (DSE). Previous DNS challenges were organized at INTERSPEECH 2020, ICASSP 2021, INTERSPEECH 2021, and ICASSP 2022. From prior editions, we learnt that improving signal quality (SIG) is challenging particularly in presence of simultaneously active interfering talkers and noise. This challenge aims to develop models for joint denosing, dereverberation and suppression of interfering talkers. When primary talker wears a headphone, certain acoustic properties of their speech such as direct-to-reverberation (DRR), signal to noise ratio (SNR) etc. make it possible to suppress neighboring talkers even without enrollment data for primary talker. This motivated us to create two tracks for this challenge: (i) Track-1 Headset; (ii) Track-2 Speakerphone. Both tracks has fullband (48kHz) training data and testset, and each testclips has a corresponding enrollment data (10-30s duration) for primary talker. Each track invited submissions of personalized and non-personalized models all of which are evaluated through same subjective evaluation. Most models submitted to challenge were personalized models, same team is winner in both tracks where the best models has improvement of 0.145 and 0.141 in challenge's Score as compared to noisy blind testset.

【6】 End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations标题：端到端集成语音分离和语音活动检测的电话通话低延迟二值化链接：http://arxiv.org/pdf/2303.12002v1

作者：Giovanni Morrone,Samuele Cornell,Luca Serafini,Enrico Zovato,Alessio Brutti,Stefano Squartini机构：Ancona, Italy, PerVoice S.p.A., Trento, Italy, Fondazione Bruno Kessler, Trento, Italy, A R T I C L E I N F O摘要：最近的工作表明，语音分离引导日记（SSGD）是一个越来越有前途的方向，主要是由于最近的进展，在语音分离。它通过首先分离扬声器，然后对每个分离的流应用语音活动检测（VAD）来执行日志化。在这项工作中，我们进行了深入的研究，SSGD在会话电话语音（CTS）域，主要集中在低延迟流日记应用程序。我们考虑三个国家的最先进的语音分离（SSEP）算法，并研究其性能在在线和离线的情况下，考虑非因果关系和因果关系的实现，以及连续的SSEP（CSS）窗口推理。我们在两个广泛使用的CTS数据集上比较不同的SSGD算法：CALLHOME和Fisher语料库（第1部分和第2部分），并评估分离和日志化性能。为了提高性能，提出了一种新的，因果关系和计算效率的泄漏去除算法，显着减少误报。我们还首次探索了SSep和VAD模块之间的完全端到端SSGD集成。至关重要的是，这可以对Oracle Speakers源不可用的真实数据进行微调。特别地，我们的最佳模型在CALLHOME上实现了8.8%的DER，其优于当前最先进的端到端神经日志化模型，尽管在数量级更少的数据上训练并且具有显著更低的延迟，即，0.1对比1秒。最后，我们还表明，分离的信号也可以很容易地用于自动语音识别，在某些配置中达到接近使用Oracle源的性能。摘要：Recent works show that speech separation guided diarization (SSGD) is an increasingly promising direction, mainly thanks to the recent progress in speech separation. It performs diarization by first separating the speakers and then applying voice activity detection (VAD) on each separated stream. In this work we conduct an in-depth study of SSGD in the conversational telephone speech (CTS) domain, focusing mainly on low-latency streaming diarization applications. We consider three state-of-the-art speech separation (SSep) algorithms and study their performance both in online and offline scenarios, considering non-causal and causal implementations as well as continuous SSep (CSS) windowed inference. We compare different SSGD algorithms on two widely used CTS datasets: CALLHOME and Fisher Corpus (Part 1 and 2) and evaluate both separation and diarization performance. To improve performance, a novel, causal and computationally efficient leakage removal algorithm is proposed, which significantly decreases false alarms. We also explore, for the first time, fully end-to-end SSGD integration between SSep and VAD modules. Crucially, this enables fine-tuning on real-world data for which oracle speakers sources are not available. In particular, our best model achieves 8.8% DER on CALLHOME, which outperforms the current state-of-the-art end-to-end neural diarization model, despite being trained on an order of magnitude less data and having significantly lower latency, i.e., 0.1 vs. 1 seconds. Finally, we also show that the separated signals can be readily used also for automatic speech recognition, reaching performance close to using oracle sources in some configurations.
eess.AS音频处理

【1】 End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations标题：端到端集成语音分离和语音活动检测的电话通话低延迟二值化链接：http://arxiv.org/pdf/2303.12002v1

*与cs.SD语音【6】为同一篇

作者：Giovanni Morrone,Samuele Cornell,Luca Serafini,Enrico Zovato,Alessio Brutti,Stefano Squartini机构：Ancona, Italy, PerVoice S.p.A., Trento, Italy, Fondazione Bruno Kessler, Trento, Italy, A R T I C L E I N F O摘要：最近的工作表明，语音分离引导日记（SSGD）是一个越来越有前途的方向，主要是由于最近的进展，在语音分离。它通过首先分离扬声器，然后对每个分离的流应用语音活动检测（VAD）来执行日志化。在这项工作中，我们进行了深入的研究，SSGD在会话电话语音（CTS）域，主要集中在低延迟流日记应用程序。我们考虑三个国家的最先进的语音分离（SSEP）算法，并研究其性能在在线和离线的情况下，考虑非因果关系和因果关系的实现，以及连续的SSEP（CSS）窗口推理。我们在两个广泛使用的CTS数据集上比较不同的SSGD算法：CALLHOME和Fisher语料库（第1部分和第2部分），并评估分离和日志化性能。为了提高性能，提出了一种新的，因果关系和计算效率的泄漏去除算法，显着减少误报。我们还首次探索了SSep和VAD模块之间的完全端到端SSGD集成。至关重要的是，这可以对Oracle Speakers源不可用的真实数据进行微调。特别地，我们的最佳模型在CALLHOME上实现了8.8%的DER，其优于当前最先进的端到端神经日志化模型，尽管在数量级更少的数据上训练并且具有显著更低的延迟，即，0.1对比1秒。最后，我们还表明，分离的信号也可以很容易地用于自动语音识别，在某些配置中达到接近使用Oracle源的性能。摘要：Recent works show that speech separation guided diarization (SSGD) is an increasingly promising direction, mainly thanks to the recent progress in speech separation. It performs diarization by first separating the speakers and then applying voice activity detection (VAD) on each separated stream. In this work we conduct an in-depth study of SSGD in the conversational telephone speech (CTS) domain, focusing mainly on low-latency streaming diarization applications. We consider three state-of-the-art speech separation (SSep) algorithms and study their performance both in online and offline scenarios, considering non-causal and causal implementations as well as continuous SSep (CSS) windowed inference. We compare different SSGD algorithms on two widely used CTS datasets: CALLHOME and Fisher Corpus (Part 1 and 2) and evaluate both separation and diarization performance. To improve performance, a novel, causal and computationally efficient leakage removal algorithm is proposed, which significantly decreases false alarms. We also explore, for the first time, fully end-to-end SSGD integration between SSep and VAD modules. Crucially, this enables fine-tuning on real-world data for which oracle speakers sources are not available. In particular, our best model achieves 8.8% DER on CALLHOME, which outperforms the current state-of-the-art end-to-end neural diarization model, despite being trained on an order of magnitude less data and having significantly lower latency, i.e., 0.1 vs. 1 seconds. Finally, we also show that the separated signals can be readily used also for automatic speech recognition, reaching performance close to using oracle sources in some configurations.

【2】 Personalized Lightweight Text-to-Speech: Voice Cloning with Adaptive Structured Pruning标题：个性化轻量级文语转换：自适应结构化剪枝的语音克隆链接：http://arxiv.org/pdf/2303.11816v1

*与cs.SD语音【1】为同一篇

作者：Sung-Feng Huang,Chia-ping Chen,Zhi-Sheng Chen,Yu-Pao Tsai,Hung-yi Lee机构：National Taiwan University,Intelligo Technology Inc.备注：ICASSP 2023摘要：个性化TTS是一个令人兴奋和高度期望的应用程序，允许用户只使用几个录音来训练他们的TTS语音。然而，TTS训练通常需要许多小时的记录和大型模型，使得其不适合在移动设备上部署。为了克服这一限制，相关工作通常需要微调预训练的TTS模型，以保持其生成高质量音频样本的能力，同时适应目标说话者的语音。这个过程通常被称为“语音克隆”。尽管相关的工作在改变TTS模型的语音方面已经取得了显著的成功，但是它们仍然需要从大的预先训练的模型进行微调，从而导致语音克隆模型的显著大小。在本文中，我们提出了应用可训练的结构化修剪语音克隆。通过使用语音克隆数据训练结构化修剪掩码，我们可以为每个目标说话人产生唯一的修剪模型。我们的实验表明，使用可学习的结构化修剪，我们可以压缩模型的大小到7倍小，同时实现可比的语音克隆性能。摘要：Personalized TTS is an exciting and highly desired application that allows users to train their TTS voice using only a few recordings. However, TTS training typically requires many hours of recording and a large model, making it unsuitable for deployment on mobile devices. To overcome this limitation, related works typically require fine-tuning a pre-trained TTS model to preserve its ability to generate high-quality audio samples while adapting to the target speaker's voice. This process is commonly referred to as voice cloning.'' Although related works have achieved significant success in changing the TTS model's voice, they are still required to fine-tune from a large pre-trained model, resulting in a significant size for the voice-cloned model. In this paper, we propose applying trainable structured pruning to voice cloning. By training the structured pruning masks with voice-cloning data, we can produce a unique pruned model for each target speaker. Our experiments demonstrate that using learnable structured pruning, we can compress the model size to 7 times smaller while achieving comparable voice-cloning performance.

【3】 ByteCover3: Accurate Cover Song Identification on Short Queries标题：ByteCover3：在短查询上准确识别翻唱歌曲链接：http://arxiv.org/pdf/2303.11692v1

*与cs.SD语音【2】为同一篇

作者：Xingjian Du,Zijie Wang,Xia Liang,Huidong Liang,Bilei Zhu,Zejun Ma机构：ByteDance, Zhejiang University, University of Oxford备注：Accepeted by ICASSP 2023摘要：近年来，基于深度学习的方法已经成为封面歌曲识别（CSI）的范例，其中ByteCover系统在CSI的所有主流数据集上都取得了最先进的结果。然而，随着短视频的蓬勃发展，许多现实世界的应用需要将短音乐摘录与数据库中的全长音乐曲目进行匹配，这仍然是未开发的，并等待工业级的解决方案。在本文中，我们将以前的ByteCover系统升级到ByteCover3，利用本地功能，以进一步提高识别性能的短音乐查询。ByteCover3设计了一个局部对齐丢失（LAL）模块和一个两阶段的特征检索管道，允许系统以更精确和有效的方式执行CSI。我们在具有不同基准设置的多个数据集上评估了ByteCover3，其中ByteCover3击败了所有比较的方法，包括其以前的版本。摘要：Deep learning based methods have become a paradigm for cover song identification (CSI) in recent years, where the ByteCover systems have achieved state-of-the-art results on all the mainstream datasets of CSI. However, with the burgeon of short videos, many real-world applications require matching short music excerpts to full-length music tracks in the database, which is still under-explored and waiting for an industrial-level solution. In this paper, we upgrade the previous ByteCover systems to ByteCover3 that utilizes local features to further improve the identification performance of short music queries. ByteCover3 is designed with a local alignment loss (LAL) module and a two-stage feature retrieval pipeline, allowing the system to perform CSI in a more precise and efficient way. We evaluated ByteCover3 on multiple datasets with different benchmark settings, where ByteCover3 beat all the compared methods including its previous versions.

【4】 Transformers in Speech Processing: A Survey标题：语音处理中的转换器：综述链接：http://arxiv.org/pdf/2303.11607v1

*与cs.SD语音【3】为同一篇

作者：Siddique Latif,Aun Zaidi,Heriberto Cuayahuitl,Fahad Shamshad,Moazzam Shoukat,Junaid Qadir机构：Queensland University of Technology (QUT), Australia, Information Technology University, Pakistan, University of Lincoln, UK, Mohamed bin Zayed University of Artificial Intelligence, EmulationAI, Qatar University, Doha备注：under-review摘要：Transformers在自然语言处理领域的显着成功，引发了语音处理社区的兴趣，导致他们的潜力建模语音序列中的远程依赖关系的探索。最近，Transformers在各种语音相关领域中获得了突出地位，包括自动语音识别、语音合成、语音翻译、语音辅助语言学、语音增强、口语对话系统和许多多模式应用。在本文中，我们提出了一个全面的调查，旨在桥梁语音技术的不同子领域的研究。通过整合整个语音技术领域的研究结果，我们为有兴趣利用Transformers的力量来推进该领域的研究人员提供了宝贵的资源。我们确定了Transformers在语音处理中遇到的挑战，同时也提供了解决这些问题的潜在解决方案的见解。摘要：The remarkable success of transformers in the field of natural language processing has sparked the interest of the speech-processing community, leading to an exploration of their potential for modeling long-range dependencies within speech sequences. Recently, transformers have gained prominence across various speech-related domains, including automatic speech recognition, speech synthesis, speech translation, speech para-linguistics, speech enhancement, spoken dialogue systems, and numerous multimodal applications. In this paper, we present a comprehensive survey that aims to bridge research studies from diverse subfields within speech technology. By consolidating findings from across the speech technology landscape, we provide a valuable resource for researchers interested in harnessing the power of transformers to advance the field. We identify the challenges encountered by transformers in speech processing while also offering insights into potential solutions to address these issues.

【5】 ModEFormer: Modality-Preserving Embedding for Audio-Video Synchronization using Transformers标题：ModEFormer：基于变形器音视频同步的保形嵌入链接：http://arxiv.org/pdf/2303.11551v1

*与cs.SD语音【4】为同一篇

作者：Akash Gupta,Rohun Tripathi,Wondong Jang机构：⋆ New York University, † Amazon Studios备注：Paper accepted at ICASSP 2023摘要：缺乏音频-视频同步是电视广播和视频会议期间的常见问题，导致不令人满意的观看体验。广泛接受的范例是创建错误检测机制，其识别音频领先或滞后的情况。我们提出了ModEFormer，它独立地提取音频和视频嵌入使用特定于模态的Transformers。与其他基于变换器的方法不同，ModEFormer保留了输入流的模态，这允许我们使用更大的批量大小和更多的负音频样本进行对比学习。进一步地，我们建议在一批中的负样本数和唯一样本数之间进行折衷，以显著地超过先前方法的性能。实验结果表明，ModEFormer实现了最先进的性能，94.5%的LRS 2和90.9%的LRS 3。最后，我们演示了如何ModEFormer可以用于偏移检测测试剪辑。摘要：Lack of audio-video synchronization is a common problem during television broadcasts and video conferencing, leading to an unsatisfactory viewing experience. A widely accepted paradigm is to create an error detection mechanism that identifies the cases when audio is leading or lagging. We propose ModEFormer, which independently extracts audio and video embeddings using modality-specific transformers. Different from the other transformer-based approaches, ModEFormer preserves the modality of the input streams which allows us to use a larger batch size with more negative audio samples for contrastive learning. Further, we propose a trade-off between the number of negative samples and number of unique samples in a batch to significantly exceed the performance of previous methods. Experimental results show that ModEFormer achieves state-of-the-art performance, 94.5% for LRS2 and 90.9% for LRS3. Finally, we demonstrate how ModEFormer can be used for offset detection for test clips.

【6】 ICASSP 2023 Deep Speech Enhancement Challenge标题：ICASSP 2023深度语音增强挑战赛链接：http://arxiv.org/pdf/2303.11510v1

*与cs.SD语音【5】为同一篇

作者：Harishchandra Dubey,Ashkan Aazami,Vishak Gopal,Babak Naderi,Sebastian Braun,Ross Cutler,Alex Ju,Mehdi Zohourian,Min Tang,Hannes Gamper,Mehrsa Golestaneh,Robert Aichner机构：Microsoft Corporation, Redmond, USA备注：6 pages, 1 figure. arXiv admin note: text overlap with arXiv:2202.13288摘要：深度语音增强挑战赛是ICASSP 2023信号处理大挑战赛组织的第五届深度噪声抑制（DNS）挑战赛。在2019-2023年期间组织了DNS挑战，以刺激深度语音增强（DSE）的研究。之前的DNS挑战赛是在INTERSPEECH 2020、ICASSP 2021、INTERSPEECH 2021和ICASSP 2022上组织的。从以前的版本中，我们了解到，提高信号质量（SIG）是具有挑战性的，特别是在存在同时活跃的干扰说话者和噪声的情况下。这项挑战旨在开发联合降噪，去混响和抑制干扰说话者的模型。当主要讲话者佩戴耳机时，他们的语音的某些声学特性（诸如直接混响（DRR）、信噪比（SNR）等）使得即使在没有主要讲话者的登记数据的情况下也可以抑制相邻讲话者。这促使我们为这项挑战创建了两条赛道：（i）1轨耳机;（二）轨道2扬声器。两个轨道都具有全频带（48 kHz）训练数据和测试集，并且每个测试剪辑具有针对主要讲话者的对应登记数据（10- 30 s持续时间）。每个轨道邀请提交个性化和非个性化模型，所有这些都通过相同的主观评估进行评估。提交挑战的大多数模型是个性化模型，同一个团队在两个轨道上都是赢家，其中最好的模型与噪声盲测试集相比，挑战得分提高了0.145和0.141。摘要：Deep Speech Enhancement Challenge is the 5th edition of deep noise suppression (DNS) challenges organized at ICASSP 2023 Signal Processing Grand Challenges. DNS challenges were organized during 2019-2023 to stimulate research in deep speech enhancement (DSE). Previous DNS challenges were organized at INTERSPEECH 2020, ICASSP 2021, INTERSPEECH 2021, and ICASSP 2022. From prior editions, we learnt that improving signal quality (SIG) is challenging particularly in presence of simultaneously active interfering talkers and noise. This challenge aims to develop models for joint denosing, dereverberation and suppression of interfering talkers. When primary talker wears a headphone, certain acoustic properties of their speech such as direct-to-reverberation (DRR), signal to noise ratio (SNR) etc. make it possible to suppress neighboring talkers even without enrollment data for primary talker. This motivated us to create two tracks for this challenge: (i) Track-1 Headset; (ii) Track-2 Speakerphone. Both tracks has fullband (48kHz) training data and testset, and each testclips has a corresponding enrollment data (10-30s duration) for primary talker. Each track invited submissions of personalized and non-personalized models all of which are evaluated through same subjective evaluation. Most models submitted to challenge were personalized models, same team is winner in both tracks where the best models has improvement of 0.145 and 0.141 in challenge's Score as compared to noisy blind testset.

机器翻译，仅供参考