今日论文合集：cs.SD语音6篇，eess.AS音频处理6篇。
cs.SD语音

【1】 Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples标题：印地语作为第二语言：用语义相似的样本改进视觉基础语音链接：http://arxiv.org/pdf/2303.17517v1

作者：Hyeonggon Ryu,Arda Senocak,In So Kweon,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, South Korea备注：ICASSP 2023摘要：本文的目的是从多语言的角度探讨视觉接地语音模型（VGS）的学习。双语VGS模型通常使用来自两种语言的相同数量的口语字幕进行训练。然而，在现实中，对于可用的口头字幕，在语言之间可能存在不平衡。我们在这项工作中的主要贡献是利用双语视觉接地语音模型中的高资源语言的能力来提高低资源语言的性能。我们介绍了两种将高资源语言的知识提取为低资源语言的方法：（1）结合强大的预先训练的高资源语言编码器，以及（2）使用语义相似的口头字幕。我们的实验表明，结合这两种方法有效地使低资源的语言，以超越单语和双语同行的跨模态检索任务的性能。摘要：The objective of this work is to explore the learning of visually grounded speech models (VGS) from multilingual perspective. Bilingual VGS models are generally trained with an equal number of spoken captions from both languages. However, in reality, there can be an imbalance among the languages for the available spoken captions. Our key contribution in this work is to leverage the power of a high-resource language in a bilingual visually grounded speech model to improve the performance of a low-resource language. We introduce two methods to distill the knowledge of high-resource language into low-resource languages: (1) incorporating a strong pre-trained high-resource language encoder and (2) using semantically similar spoken captions. Our experiments show that combining these two approaches effectively enables the low-resource language to surpass the performances of monolingual and bilingual counterparts for cross-modal retrieval tasks.

【2】 Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment标题：基于视听潜伏期对准的视听场景生成链接：http://arxiv.org/pdf/2303.17490v1

作者：Kim Sung-Bin,Arda Senocak,Hyunwoo Ha,Andrew Owens,Tae-Hyun Oh机构：Dept. of Electrical Engineering and ,Grad. School of Artificial Intelligence, POSTECH, Dept. of Electrical Engineering, KAIST, University of Michigan, Institute for Convergence Research and Education in Advanced Technology, Yonsei University备注：CVPR 2023摘要：音频如何描述我们周围的世界？在本文中，我们提出了一种方法，用于生成一个场景的声音的图像。我们的方法解决了处理视觉和声音之间经常存在的巨大差距的挑战。我们设计了一个模型，其工作原理是通过调度每个模型组件的学习过程相关联的视听方式，尽管他们的信息差距。其关键思想是通过学习将音频与视觉潜在空间对齐来丰富具有视觉信息的音频特征。我们将输入音频转换为视觉特征，然后使用预先训练的生成器生成图像。为了进一步提高我们生成的图像的质量，我们使用声源定位来选择具有较强的跨模态相关性的视听对。我们在VEGAS和VGGSound数据集上获得了比以前方法更好的结果。我们还表明，我们可以通过对输入波形或潜在空间进行简单的操作来控制模型的预测。摘要：How does audio describe the world around us? In this paper, we propose a method for generating an image of a scene from sound. Our method addresses the challenges of dealing with the large gaps that often exist between sight and sound. We design a model that works by scheduling the learning procedure of each model component to associate audio-visual modalities despite their information gaps. The key idea is to enrich the audio features with visual information by learning to align audio to visual latent space. We translate the input audio to visual features, then use a pre-trained generator to produce an image. To further improve the quality of our generated images, we use sound source localization to select the audio-visual pairs that have strong cross-modal correlations. We obtain substantially better results on the VEGAS and VGGSound datasets than prior approaches. We also show that we can control our model's predictions by applying simple manipulations to the input waveform, or to the latent space.

【3】 SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision标题：SynthVSR：使用合成监控扩展视觉语音识别链接：http://arxiv.org/pdf/2303.17200v1

作者：Xubo Liu,Egor Lakomkin,Konstantinos Vougioukas,Pingchuan Ma,Honglie Chen,Ruiming Xie,Morrie Doulaty,Niko Moritz,Jáchym Kolář,Stavros Petridis,Maja Pantic,Christian Fuegen机构：University of Surrey, Meta AI, . Synthetic Dataset Generation, Transcribed Speech Dataset, "How are you?", "So cool!", . Speech-Driven Lip Animation, Lip Image, Speech Utterance, Synthetic Video, . Semi-Supervised VSR Learning, Visual Speech, Recognition, "That's, awesome!", Real备注：CVPR 2023摘要：最近报道的视觉语音识别（VSR）的最新结果通常依赖于越来越大量的视频数据，而公开可用的转录视频数据集在大小上是有限的。在本文中，第一次，我们研究了利用合成视觉数据的VSR的潜力。我们的方法，称为SynthVSR，大大提高了VSR系统的性能与合成嘴唇的运动。SynthVSR背后的关键思想是利用语音驱动的嘴唇动画模型，该模型根据输入语音生成嘴唇运动。语音驱动的嘴唇动画模型是在未标记的视听数据集上训练的，并且当标记的视频可用时，可以进一步优化为预训练的VSR模型。由于大量的转录声学数据和人脸图像是可用的，我们能够生成大规模的合成数据，使用建议的嘴唇动画模型的半监督VSR训练。我们评估我们的方法在最大的公共VSR基准-唇读句子3（LRS 3）的性能。SynthVSR仅用30小时的真实标记数据就实现了43.3%的WER，优于使用数千小时视频的现成方法。当使用来自LRS 3的所有438小时标记数据时，WER进一步降低到27.9%，这与最先进的自监督AV-HuBERT方法相当。此外，当与大规模伪标记的视听数据相结合时，SynthVSR仅使用公开可用的数据产生了16.9%的新的最先进的VSR WER，超过了最近使用29倍的非公开机器转录视频数据（90，000小时）训练的最先进的方法。最后，我们进行了广泛的消融研究，以了解我们提出的方法中的每个组件的效果。摘要：Recently reported state-of-the-art results in visual speech recognition (VSR) often rely on increasingly large amounts of video data, while the publicly available transcribed video datasets are limited in size. In this paper, for the first time, we study the potential of leveraging synthetic visual data for VSR. Our method, termed SynthVSR, substantially improves the performance of VSR systems with synthetic lip movements. The key idea behind SynthVSR is to leverage a speech-driven lip animation model that generates lip movements conditioned on the input speech. The speech-driven lip animation model is trained on an unlabeled audio-visual dataset and could be further optimized towards a pre-trained VSR model when labeled videos are available. As plenty of transcribed acoustic data and face images are available, we are able to generate large-scale synthetic data using the proposed lip animation model for semi-supervised VSR training. We evaluate the performance of our approach on the largest public VSR benchmark - Lip Reading Sentences 3 (LRS3). SynthVSR achieves a WER of 43.3% with only 30 hours of real labeled data, outperforming off-the-shelf approaches using thousands of hours of video. The WER is further reduced to 27.9% when using all 438 hours of labeled data from LRS3, which is on par with the state-of-the-art self-supervised AV-HuBERT method. Furthermore, when combined with large-scale pseudo-labeled audio-visual data SynthVSR yields a new state-of-the-art VSR WER of 16.9% using publicly available data only, surpassing the recent state-of-the-art approaches trained with 29 times more non-public machine-transcribed video data (90,000 hours). Finally, we perform extensive ablation studies to understand the effect of each component in our proposed method.

【4】 Prefix tuning for automated audio captioning标题：用于自动音频字幕的前缀调整链接：http://arxiv.org/pdf/2303.17489v1

作者：Minkyu Kim,Kim Sung-Bin2,Tae-Hyun Oh机构：Grad. School of Artificial Intelligence and ,Dept. of Electrical Engineering, POSTECH, Korea., Institute for Convergence Research and Education in Advanced Technology, Yonsei University, Korea.备注：ICASSP 2023摘要：音频字幕旨在从环境声音中生成文本描述。音频字幕的一个挑战是由于缺乏音频-文本配对训练数据而导致的泛化困难。在这项工作中，我们提出了一种简单而有效的方法，通过利用预先训练的语言模型来处理小规模数据集。我们保持语言模型冻结以保持文本生成的表达能力，并且我们只学习从输入音频中提取全局和时间特征。为了弥合音频特征和语言模型之间的模态差距，我们采用映射网络，将音频特征转换为语言模型可以理解的连续向量，称为前缀。我们在Clotho和AudioCaps数据集上评估了我们提出的方法，并表明我们的方法在不同的实验环境中优于现有技术。摘要：Audio captioning aims to generate text descriptions from environmental sounds. One challenge of audio captioning is the difficulty of the generalization due to the lack of audio-text paired training data. In this work, we propose a simple yet effective method of dealing with small-scaled datasets by leveraging a pre-trained language model. We keep the language model frozen to maintain the expressivity for text generation, and we only learn to extract global and temporal features from the input audio. To bridge a modality gap between the audio features and the language model, we employ mapping networks that translate audio features to the continuous vectors the language model can understand, called prefixes. We evaluate our proposed method on the Clotho and AudioCaps dataset and show our method outperforms prior arts in diverse experimental settings.

【5】 WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research标题：WavCaps：一种ChatGPT辅助的弱标注音频字幕数据集链接：http://arxiv.org/pdf/2303.17395v1

作者：Xinhao Mei,Chutong Meng,Haohe Liu,Qiuqiang Kong,Tom Ko,Chengqi Zhao,Mark D. Plumbley,Yuexian Zou,Wenwu Wang备注：12 pages摘要：近年来，音频-语言（AL）多模态学习任务的进展非常显著。然而，研究人员面临的挑战，由于现有的音频语言数据集，这是有限的大小昂贵和耗时的收集过程。为了解决这个数据稀缺的问题，我们引入了WavCaps，这是第一个大规模的弱标记音频字幕数据集，包括大约40万个带有配对字幕的音频片段。我们从网络来源和声音事件检测数据集获取音频片段及其原始描述。然而，在线收获的原始描述是非常嘈杂的，不适合直接用于任务，如自动音频字幕。为了克服这个问题，我们提出了一个三阶段的处理管道，用于过滤噪声数据和生成高质量的字幕，其中ChatGPT，一个大型的语言模型，利用过滤和转换的原始描述自动。我们对WavCaps数据集的特征进行了全面分析，并在多个下游音频语言多模态学习任务上进行了评估。在WavCaps上训练的系统性能明显优于以前的最先进（SOTA）模型。我们的愿望是我们提出的WavCaps数据集，以促进音频语言多模态学习的研究，并展示利用ChatGPT加强学术研究的潜力。我们的数据集和代码可在www.example.com上获得https: github.com XinhaoMei WavCaps。摘要：The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years. However, researchers face challenges due to the costly and time-consuming collection process of existing audio-language datasets, which are limited in size. To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing ChatGPT to enhance academic research. Our dataset and codes are available at https: github.com XinhaoMei WavCaps.

【6】 PROCTER: PROnunciation-aware ConTextual adaptER for personalized speech recognition in neural transducers标题：Procter：神经传感器中用于个性化语音识别的发音感知上下文适配器链接：http://arxiv.org/pdf/2303.17131v1

作者：Rahul Pandey,Roger Ren,Qi Luo,Jing Liu,Ariya Rastrow,Ankur Gandhe,Denis Filimonov,Grant Strimel,Andreas Stolcke,Ivan Bulyko机构：Amazon Alexa AI, USA, George Mason University, USA备注：To appear in Proc. IEEE ICASSP摘要：在语音助理中使用的端到端（E2 E）自动语音识别（ASR）系统通常难以识别针对用户个性化的不频繁词语，诸如姓名和地点。稀有单词通常具有非平凡的发音，并且在这种情况下，以发音词典的形式的人类知识可能是有用的。我们提出了一个PROnunciation-aware上下文adaptER（PROCTER），通过添加音素嵌入和文本嵌入，动态地将词典知识注入RNN-T模型。实验结果表明，所提出的PROCTER架构优于基线RNN-T模型，分别在个性化实体和个性化稀有实体上测量时，将单词错误率（WER）提高了44%和57%，同时仅将模型大小（可训练参数的数量）增加了1%。此外，当在zero-shot设置中进行评估以识别个性化设备名称时，我们观察到使用PROCTER的7% WER改善，而使用纯文本上下文注意力的WER改善仅为1%。摘要：End-to-End (E2E) automatic speech recognition (ASR) systems used in voice assistants often have difficulties recognizing infrequent words personalized to the user, such as names and places. Rare words often have non-trivial pronunciations, and in such cases, human knowledge in the form of a pronunciation lexicon can be useful. We propose a PROnunCiation-aware conTextual adaptER (PROCTER) that dynamically injects lexicon knowledge into an RNN-T model by adding a phonemic embedding along with a textual embedding. The experimental results show that the proposed PROCTER architecture outperforms the baseline RNN-T model by improving the word error rate (WER) by 44% and 57% when measured on personalized entities and personalized rare entities, respectively, while increasing the model size (number of trainable parameters) by only 1%. Furthermore, when evaluated in a zero-shot setting to recognize personalized device names, we observe 7% WER improvement with PROCTER, as compared to only 1% WER improvement with text-only contextual attention.
eess.AS音频处理

【1】 Prefix tuning for automated audio captioning标题：用于自动音频字幕的前缀调整链接：http://arxiv.org/pdf/2303.17489v1

*与cs.SD语音【4】为同一篇

作者：Minkyu Kim,Kim Sung-Bin2,Tae-Hyun Oh机构：Grad. School of Artificial Intelligence and ,Dept. of Electrical Engineering, POSTECH, Korea., Institute for Convergence Research and Education in Advanced Technology, Yonsei University, Korea.备注：ICASSP 2023摘要：音频字幕旨在从环境声音中生成文本描述。音频字幕的一个挑战是由于缺乏音频-文本配对训练数据而导致的泛化困难。在这项工作中，我们提出了一种简单而有效的方法，通过利用预先训练的语言模型来处理小规模数据集。我们保持语言模型冻结以保持文本生成的表达能力，并且我们只学习从输入音频中提取全局和时间特征。为了弥合音频特征和语言模型之间的模态差距，我们采用映射网络，将音频特征转换为语言模型可以理解的连续向量，称为前缀。我们在Clotho和AudioCaps数据集上评估了我们提出的方法，并表明我们的方法在不同的实验环境中优于现有技术。摘要：Audio captioning aims to generate text descriptions from environmental sounds. One challenge of audio captioning is the difficulty of the generalization due to the lack of audio-text paired training data. In this work, we propose a simple yet effective method of dealing with small-scaled datasets by leveraging a pre-trained language model. We keep the language model frozen to maintain the expressivity for text generation, and we only learn to extract global and temporal features from the input audio. To bridge a modality gap between the audio features and the language model, we employ mapping networks that translate audio features to the continuous vectors the language model can understand, called prefixes. We evaluate our proposed method on the Clotho and AudioCaps dataset and show our method outperforms prior arts in diverse experimental settings.

【2】 WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research标题：WavCaps：一种ChatGPT辅助的弱标注音频字幕数据集链接：http://arxiv.org/pdf/2303.17395v1

*与cs.SD语音【5】为同一篇

作者：Xinhao Mei,Chutong Meng,Haohe Liu,Qiuqiang Kong,Tom Ko,Chengqi Zhao,Mark D. Plumbley,Yuexian Zou,Wenwu Wang备注：12 pages摘要：近年来，音频-语言（AL）多模态学习任务的进展非常显著。然而，研究人员面临的挑战，由于现有的音频语言数据集，这是有限的大小昂贵和耗时的收集过程。为了解决这个数据稀缺的问题，我们引入了WavCaps，这是第一个大规模的弱标记音频字幕数据集，包括大约40万个带有配对字幕的音频片段。我们从网络来源和声音事件检测数据集获取音频片段及其原始描述。然而，在线收获的原始描述是非常嘈杂的，不适合直接用于任务，如自动音频字幕。为了克服这个问题，我们提出了一个三阶段的处理管道，用于过滤噪声数据和生成高质量的字幕，其中ChatGPT，一个大型的语言模型，利用过滤和转换的原始描述自动。我们对WavCaps数据集的特征进行了全面分析，并在多个下游音频语言多模态学习任务上进行了评估。在WavCaps上训练的系统性能明显优于以前的最先进（SOTA）模型。我们的愿望是我们提出的WavCaps数据集，以促进音频语言多模态学习的研究，并展示利用ChatGPT加强学术研究的潜力。我们的数据集和代码可在www.example.com上获得https: github.com XinhaoMei WavCaps。摘要：The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years. However, researchers face challenges due to the costly and time-consuming collection process of existing audio-language datasets, which are limited in size. To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing ChatGPT to enhance academic research. Our dataset and codes are available at https: github.com XinhaoMei WavCaps.

【3】 PROCTER: PROnunciation-aware ConTextual adaptER for personalized speech recognition in neural transducers标题：Procter：神经传感器中用于个性化语音识别的发音感知上下文适配器链接：http://arxiv.org/pdf/2303.17131v1

*与cs.SD语音【6】为同一篇

作者：Rahul Pandey,Roger Ren,Qi Luo,Jing Liu,Ariya Rastrow,Ankur Gandhe,Denis Filimonov,Grant Strimel,Andreas Stolcke,Ivan Bulyko机构：Amazon Alexa AI, USA, George Mason University, USA备注：To appear in Proc. IEEE ICASSP摘要：在语音助理中使用的端到端（E2 E）自动语音识别（ASR）系统通常难以识别针对用户个性化的不频繁词语，诸如姓名和地点。稀有单词通常具有非平凡的发音，并且在这种情况下，以发音词典的形式的人类知识可能是有用的。我们提出了一个PROnunciation-aware上下文adaptER（PROCTER），通过添加音素嵌入和文本嵌入，动态地将词典知识注入RNN-T模型。实验结果表明，所提出的PROCTER架构优于基线RNN-T模型，分别在个性化实体和个性化稀有实体上测量时，将单词错误率（WER）提高了44%和57%，同时仅将模型大小（可训练参数的数量）增加了1%。此外，当在zero-shot设置中进行评估以识别个性化设备名称时，我们观察到使用PROCTER的7% WER改善，而使用纯文本上下文注意力的WER改善仅为1%。摘要：End-to-End (E2E) automatic speech recognition (ASR) systems used in voice assistants often have difficulties recognizing infrequent words personalized to the user, such as names and places. Rare words often have non-trivial pronunciations, and in such cases, human knowledge in the form of a pronunciation lexicon can be useful. We propose a PROnunCiation-aware conTextual adaptER (PROCTER) that dynamically injects lexicon knowledge into an RNN-T model by adding a phonemic embedding along with a textual embedding. The experimental results show that the proposed PROCTER architecture outperforms the baseline RNN-T model by improving the word error rate (WER) by 44% and 57% when measured on personalized entities and personalized rare entities, respectively, while increasing the model size (number of trainable parameters) by only 1%. Furthermore, when evaluated in a zero-shot setting to recognize personalized device names, we observe 7% WER improvement with PROCTER, as compared to only 1% WER improvement with text-only contextual attention.

【4】 Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples标题：印地语作为第二语言：用语义相似的样本改进视觉基础语音链接：http://arxiv.org/pdf/2303.17517v1

*与cs.SD语音【1】为同一篇

作者：Hyeonggon Ryu,Arda Senocak,In So Kweon,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, South Korea备注：ICASSP 2023摘要：本文的目的是从多语言的角度探讨视觉接地语音模型（VGS）的学习。双语VGS模型通常使用来自两种语言的相同数量的口语字幕进行训练。然而，在现实中，对于可用的口头字幕，在语言之间可能存在不平衡。我们在这项工作中的主要贡献是利用双语视觉接地语音模型中的高资源语言的能力来提高低资源语言的性能。我们介绍了两种将高资源语言的知识提取为低资源语言的方法：（1）结合强大的预先训练的高资源语言编码器，以及（2）使用语义相似的口头字幕。我们的实验表明，结合这两种方法有效地使低资源的语言，以超越单语和双语同行的跨模态检索任务的性能。摘要：The objective of this work is to explore the learning of visually grounded speech models (VGS) from multilingual perspective. Bilingual VGS models are generally trained with an equal number of spoken captions from both languages. However, in reality, there can be an imbalance among the languages for the available spoken captions. Our key contribution in this work is to leverage the power of a high-resource language in a bilingual visually grounded speech model to improve the performance of a low-resource language. We introduce two methods to distill the knowledge of high-resource language into low-resource languages: (1) incorporating a strong pre-trained high-resource language encoder and (2) using semantically similar spoken captions. Our experiments show that combining these two approaches effectively enables the low-resource language to surpass the performances of monolingual and bilingual counterparts for cross-modal retrieval tasks.

【5】 Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment标题：基于视听潜伏期对准的视听场景生成链接：http://arxiv.org/pdf/2303.17490v1

*与cs.SD语音【2】为同一篇

作者：Kim Sung-Bin,Arda Senocak,Hyunwoo Ha,Andrew Owens,Tae-Hyun Oh机构：Dept. of Electrical Engineering and ,Grad. School of Artificial Intelligence, POSTECH, Dept. of Electrical Engineering, KAIST, University of Michigan, Institute for Convergence Research and Education in Advanced Technology, Yonsei University备注：CVPR 2023摘要：音频如何描述我们周围的世界？在本文中，我们提出了一种方法，用于生成一个场景的声音的图像。我们的方法解决了处理视觉和声音之间经常存在的巨大差距的挑战。我们设计了一个模型，其工作原理是通过调度每个模型组件的学习过程相关联的视听方式，尽管他们的信息差距。其关键思想是通过学习将音频与视觉潜在空间对齐来丰富具有视觉信息的音频特征。我们将输入音频转换为视觉特征，然后使用预先训练的生成器生成图像。为了进一步提高我们生成的图像的质量，我们使用声源定位来选择具有较强的跨模态相关性的视听对。我们在VEGAS和VGGSound数据集上获得了比以前方法更好的结果。我们还表明，我们可以通过对输入波形或潜在空间进行简单的操作来控制模型的预测。摘要：How does audio describe the world around us? In this paper, we propose a method for generating an image of a scene from sound. Our method addresses the challenges of dealing with the large gaps that often exist between sight and sound. We design a model that works by scheduling the learning procedure of each model component to associate audio-visual modalities despite their information gaps. The key idea is to enrich the audio features with visual information by learning to align audio to visual latent space. We translate the input audio to visual features, then use a pre-trained generator to produce an image. To further improve the quality of our generated images, we use sound source localization to select the audio-visual pairs that have strong cross-modal correlations. We obtain substantially better results on the VEGAS and VGGSound datasets than prior approaches. We also show that we can control our model's predictions by applying simple manipulations to the input waveform, or to the latent space.

【6】 SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision标题：SynthVSR：使用合成监控扩展视觉语音识别链接：http://arxiv.org/pdf/2303.17200v1

*与cs.SD语音【3】为同一篇

作者：Xubo Liu,Egor Lakomkin,Konstantinos Vougioukas,Pingchuan Ma,Honglie Chen,Ruiming Xie,Morrie Doulaty,Niko Moritz,Jáchym Kolář,Stavros Petridis,Maja Pantic,Christian Fuegen机构：University of Surrey, Meta AI, . Synthetic Dataset Generation, Transcribed Speech Dataset, "How are you?", "So cool!", . Speech-Driven Lip Animation, Lip Image, Speech Utterance, Synthetic Video, . Semi-Supervised VSR Learning, Visual Speech, Recognition, "That's, awesome!", Real备注：CVPR 2023摘要：最近报道的视觉语音识别（VSR）的最新结果通常依赖于越来越大量的视频数据，而公开可用的转录视频数据集在大小上是有限的。在本文中，第一次，我们研究了利用合成视觉数据的VSR的潜力。我们的方法，称为SynthVSR，大大提高了VSR系统的性能与合成嘴唇的运动。SynthVSR背后的关键思想是利用语音驱动的嘴唇动画模型，该模型根据输入语音生成嘴唇运动。语音驱动的嘴唇动画模型是在未标记的视听数据集上训练的，并且当标记的视频可用时，可以进一步优化为预训练的VSR模型。由于大量的转录声学数据和人脸图像是可用的，我们能够生成大规模的合成数据，使用建议的嘴唇动画模型的半监督VSR训练。我们评估我们的方法在最大的公共VSR基准-唇读句子3（LRS 3）的性能。SynthVSR仅用30小时的真实标记数据就实现了43.3%的WER，优于使用数千小时视频的现成方法。当使用来自LRS 3的所有438小时标记数据时，WER进一步降低到27.9%，这与最先进的自监督AV-HuBERT方法相当。此外，当与大规模伪标记的视听数据相结合时，SynthVSR仅使用公开可用的数据产生了16.9%的新的最先进的VSR WER，超过了最近使用29倍的非公开机器转录视频数据（90，000小时）训练的最先进的方法。最后，我们进行了广泛的消融研究，以了解我们提出的方法中的每个组件的效果。摘要：Recently reported state-of-the-art results in visual speech recognition (VSR) often rely on increasingly large amounts of video data, while the publicly available transcribed video datasets are limited in size. In this paper, for the first time, we study the potential of leveraging synthetic visual data for VSR. Our method, termed SynthVSR, substantially improves the performance of VSR systems with synthetic lip movements. The key idea behind SynthVSR is to leverage a speech-driven lip animation model that generates lip movements conditioned on the input speech. The speech-driven lip animation model is trained on an unlabeled audio-visual dataset and could be further optimized towards a pre-trained VSR model when labeled videos are available. As plenty of transcribed acoustic data and face images are available, we are able to generate large-scale synthetic data using the proposed lip animation model for semi-supervised VSR training. We evaluate the performance of our approach on the largest public VSR benchmark - Lip Reading Sentences 3 (LRS3). SynthVSR achieves a WER of 43.3% with only 30 hours of real labeled data, outperforming off-the-shelf approaches using thousands of hours of video. The WER is further reduced to 27.9% when using all 438 hours of labeled data from LRS3, which is on par with the state-of-the-art self-supervised AV-HuBERT method. Furthermore, when combined with large-scale pseudo-labeled audio-visual data SynthVSR yields a new state-of-the-art VSR WER of 16.9% using publicly available data only, surpassing the recent state-of-the-art approaches trained with 29 times more non-public machine-transcribed video data (90,000 hours). Finally, we perform extensive ablation studies to understand the effect of each component in our proposed method.

机器翻译，仅供参考