今日论文合集：cs.SD语音5篇，eess.AS音频处理7篇。
cs.SD语音
【1】 AnuraSet: A dataset for benchmarking Neotropical anuran calls  identification in passive acoustic monitoring标题：AnuraSet：被动声监测中新热带无尾两栖动物叫声识别的基准数据集链接：https://arxiv.org/abs/2307.06860
作者：Juan Sebastián Cañas,Maria Paula Toro-Gómez,Larissa Sayuri Moreira Sugai,Hernán Darío Benítez Restrepo,Jorge Rudas,Breyner Posso Bautista,Luís Felipe Toledo,Simone Dena,Adão Henrique Rosa Domingos,Franco Leandro de Souza,Selvino Neckel-Oliveira,Anderson da Rosa,Vítor Carvalho-Rocha,José Vinícius Bernardy,José Luiz Massao Moreira Sugai,Carolina Emília dos Santos,Rogério Pereira Bastos,Diego Llusia,Juan Sebastián Ulloa机构：Ulloa, Affiliations, . Instituto de Investigación de Recursos Biológicos Alexander von Humboldt, Avenida Paseo, Bolívar ,-, Bogotá, Colombia; ,. K Lisa Yang Center for Conservation Bioacoustics, Cornell摘要：全球变化的预测，诱导无尾两栖动物的声学行为，这可以通过被动声学监测（PAM）进行研究的转变。了解呼叫行为的变化需要确定无尾两栖动物的物种，这是具有挑战性的，由于新热带区音景的特殊特征。在本文中，我们介绍了一个大规模的多物种数据集的无尾两栖动物的电话记录的PAM，包括27小时的专家注释42个不同的物种从两个巴西生物群系。我们提供了开放访问的数据集，包括原始记录，实验设置代码，和一个基准与细粒度分类问题的基线模型。此外，我们强调了数据集的挑战，以鼓励机器学习研究人员解决无尾两栖类对保护政策的呼叫识别问题。我们所有的实验和资源都可以在我们的GitHub存储库https://github.com/soundclim/anuraset上找到。摘要：Global change is predicted to induce shifts in anuran acoustic behavior, which can be studied through passive acoustic monitoring (PAM). Understanding changes in calling behavior requires the identification of anuran species, which is challenging due to the particular characteristics of neotropical soundscapes. In this paper, we introduce a large-scale multi-species dataset of anuran amphibians calls recorded by PAM, that comprises 27 hours of expert annotations for 42 different species from two Brazilian biomes. We provide open access to the dataset, including the raw recordings, experimental setup code, and a benchmark with a baseline model of the fine-grained categorization problem. Additionally, we highlight the challenges of the dataset to encourage machine learning researchers to solve the problem of anuran call identification towards conservation policy. All our experiments and resources can be found on our GitHub repository https://github.com/soundclim/anuraset.
【2】 Uncovering the Deceptions: An Analysis on Audio Spoofing Detection and  Future Prospects标题：揭开骗局：音频欺骗检测分析及未来展望链接：https://arxiv.org/abs/2307.06669
作者：Rishabh Ranjan,Mayank Vatsa,Richa Singh机构：Indian Institute of Technology, Jodhpur, India备注：Accepted in IJCAI 2023摘要：音频已经成为一种越来越重要的生物识别方式，因为它能够为人类提供与机器交互的直观方式。它目前被用于一系列应用，包括个人身份验证到银行到虚拟助理。研究表明，这些系统也容易受到欺骗和攻击。因此，保护音频处理系统免受诸如身份盗窃、金融欺诈和传播错误信息之类的欺诈活动的影响是至关重要的。本文回顾了当前国家的最先进的技术检测音频欺骗，并讨论了当前的挑战，以及开放的研究问题。本文进一步强调了考虑音频欺骗检测系统的道德和隐私影响的重要性。最后，这项工作的目的是强调需要建立更强大的和可推广的方法，自动说话人验证和对策系统的集成，以及更好的评估协议。摘要：Audio has become an increasingly crucial biometric modality due to its ability to provide an intuitive way for humans to interact with machines. It is currently being used for a range of applications, including person authentication to banking to virtual assistants. Research has shown that these systems are also susceptible to spoofing and attacks. Therefore, protecting audio processing systems against fraudulent activities, such as identity theft, financial fraud, and spreading misinformation, is of paramount importance. This paper reviews the current state-of-the-art techniques for detecting audio spoofing and discusses the current challenges along with open research problems. The paper further highlights the importance of considering the ethical and privacy implications of audio spoofing detection systems. Lastly, the work aims to accentuate the need for building more robust and generalizable methods, the integration of automatic speaker verification and countermeasure systems, and better evaluation protocols.
【3】 Exploring the Integration of Large Language Models into Automatic Speech  Recognition Systems: An Empirical Study标题：探索将大型语言模型集成到自动语音识别系统中：一项实证研究链接：https://arxiv.org/abs/2307.06530
作者：Zeping Min,Jinbo Wang机构： Peking University, No., Yiheyuan Road, Haidian District, Beijing , P.R.China摘要：本文探讨了大语言模型（LLM）到自动语音识别（ASR）系统的集成，以提高转录的准确性。LLM的日益复杂性，以及它们的上下文学习能力和指令遵循行为，在自然语言处理（NLP）领域引起了极大的关注。我们的主要重点是调查使用LLM的上下文学习能力，以提高ASR系统的性能，目前面临的挑战，如环境噪声，扬声器口音和复杂的语言环境的潜力。我们使用Aishell-1和LibriSpeech数据集设计了一项研究，ChatGPT和GPT-4作为LLM功能的基准。不幸的是，我们最初的实验并没有产生有希望的结果，表明利用LLM的上下文学习ASR应用程序的复杂性。尽管有不同的设置和模型的进一步探索，从LLM校正的句子经常导致较高的词错误率（WER），表明在语音应用LLM的局限性。本文提供了这些实验的详细概述，他们的结果和影响，建立使用LLM的上下文学习能力，以纠正语音识别转录中的潜在错误，在当前阶段仍然是一个具有挑战性的任务。摘要：This paper explores the integration of Large Language Models (LLMs) into Automatic Speech Recognition (ASR) systems to improve transcription accuracy. The increasing sophistication of LLMs, with their in-context learning capabilities and instruction-following behavior, has drawn significant attention in the field of Natural Language Processing (NLP). Our primary focus is to investigate the potential of using an LLM's in-context learning capabilities to enhance the performance of ASR systems, which currently face challenges such as ambient noise, speaker accents, and complex linguistic contexts. We designed a study using the Aishell-1 and LibriSpeech datasets, with ChatGPT and GPT-4 serving as benchmarks for LLM capabilities. Unfortunately, our initial experiments did not yield promising results, indicating the complexity of leveraging LLM's in-context learning for ASR applications. Despite further exploration with varied settings and models, the corrected sentences from the LLMs frequently resulted in higher Word Error Rates (WER), demonstrating the limitations of LLMs in speech applications. This paper provides a detailed overview of these experiments, their results, and implications, establishing that using LLMs' in-context learning capabilities to correct potential errors in speech recognition transcriptions is still a challenging task at the current stage.
【4】 Temporal Label-Refinement for Weakly-Supervised Audio-Visual Event  Localization标题：用于弱监督视听事件定位的时间标签细化算法链接：https://arxiv.org/abs/2307.06385
作者：Kalyan Ramakrishnan机构：Temporal Label-Refinement for Weakly-SupervisedAudio-Visual Event LocalizationKalyan RIndian Institute of Technology Madraskalyan08 2 1摘要：视听事件定位（AVEL）是对视听事件进行时间定位和分类的任务，即视频中同时可见和可听的事件。在本文中，我们解决AVEL在一个弱监督设置，其中只有视频级别的事件标签（他们的存在/不存在，但不是他们的位置在时间）可作为监督培训。我们的想法是使用一个基本模型来估计训练数据上的标签，在一个更精细的时间分辨率比在视频级别，并重新训练模型与这些标签。即，我们通过（i）用来自在视频级标签中不具有重叠的第二视频的那些帧替换切片外的帧，以及（ii）将该合成视频馈送到基本模型中以仅提取所讨论的切片的标签来确定训练视频中的帧的每个切片的标签子集。为了处理我们的合成视频的分布外的性质，我们提出了一个辅助目标的基础模型，诱导更可靠的预测本地化的事件标签所需。我们的三个阶段的流水线优于现有的AVEL方法，没有架构上的变化，并提高了相关的弱监督任务的性能。摘要：Audio-Visual Event Localization (AVEL) is the task of temporally localizing and classifying \emph{audio-visual events}, i.e., events simultaneously visible and audible in a video. In this paper, we solve AVEL in a weakly-supervised setting, where only video-level event labels (their presence/absence, but not their locations in time) are available as supervision for training. Our idea is to use a base model to estimate labels on the training data at a finer temporal resolution than at the video level and re-train the model with these labels. I.e., we determine the subset of labels for each \emph{slice} of frames in a training video by (i) replacing the frames outside the slice with those from a second video having no overlap in video-level labels, and (ii) feeding this synthetic video into the base model to extract labels for just the slice in question. To handle the out-of-distribution nature of our synthetic videos, we propose an auxiliary objective for the base model that induces more reliable predictions of the localized event labels as desired. Our three-stage pipeline outperforms several existing AVEL methods with no architectural changes and improves performance on a related weakly-supervised task as well.
【5】 An Improved Metric of Informational Masking for Perceptual Audio Quality  Measurement标题：一种改进的感知音频质量信息掩蔽度量方法链接：https://arxiv.org/abs/2307.06656
作者：Pablo M. Delgado,Jürgen Herre机构：⋆ International Audio Laboratories Erlangen ‡, Am Wolfsmantel , Erlangen, Germany, † Fraunhofer IIS, Am Wolfsmantel , Erlangen, Germany备注：Accepted Publication for WASPAA 2023 - IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, Mohonk Mountain House, New Paltz, NY, USA, Oct 22-25, 2023摘要：感知音频质量测量系统在算法上分析音频处理系统的输出，以使用人类听觉的感知模型来估计可能的感知质量劣化。以这种方式，它们节省了与听力测试（LT）的设计和执行相关联的时间和资源。在质量测量系统中预测外围听觉掩蔽的干扰可听度模型已经显著地增加了由感知音频编解码器处理的信号的主观质量预测性能。此外，认知效应也被称为调节感知失真的严重性，通过影响他们的显着性。然而，由于在质量测量系统中的认知效应模型的性能增益是不一致的，到目前为止，特别是对于音乐信号。首先，本文提出了一种改进的模型信息掩蔽（IM）-质量感知的一个重要的认知效应-考虑干扰信息的复杂性周围的掩蔽阈值。其次，我们将建议的IM度量到一个质量测量系统，使用一种新的认知效果和失真度量之间的交互分析程序。该程序建立认知效应和失真指标之间的相互作用，使用LT数据。所提出的IM指标优于先前提出的IM指标在验证任务中对主观质量分数从大型和多样化的LT数据库。特别地，所提出的系统示出了用带宽扩展技术编码的音乐信号的增加的质量预测，其中其他模型经常失败。摘要：Perceptual audio quality measurement systems algorithmically analyze the output of audio processing systems to estimate possible perceived quality degradation using perceptual models of human audition. In this manner, they save the time and resources associated with the design and execution of listening tests (LTs). Models of disturbance audibility predicting peripheral auditory masking in quality measurement systems have considerably increased subjective quality prediction performance of signals processed by perceptual audio codecs. Additionally, cognitive effects have also been known to regulate perceived distortion severity by influencing their salience. However, the performance gains due to cognitive effect models in quality measurement systems were inconsistent so far, particularly for music signals. Firstly, this paper presents an improved model of informational masking (IM) -- an important cognitive effect in quality perception -- that considers disturbance information complexity around the masking threshold. Secondly, we incorporate the proposed IM metric into a quality measurement systems using a novel interaction analysis procedure between cognitive effects and distortion metrics. The procedure establishes interactions between cognitive effects and distortion metrics using LT data. The proposed IM metric is shown to outperform previously proposed IM metrics in a validation task against subjective quality scores from large and diverse LT databases. Particularly, the proposed system showed an increased quality prediction of music signals coded with bandwidth extension techniques, where other models frequently fail.
eess.AS音频处理
【1】 Personalization for BERT-based Discriminative Speech Recognition  Rescoring标题：基于BERT的区分语音识别的个性化研究链接：https://arxiv.org/abs/2307.06832
作者：Jari Kolehmainen,Yile Gu,Aditya Gourav,Prashanth Gurunath Shivakumar,Ankur Gandhe,Ariya Rastrow,Ivan Bulyko机构：Amazon Alexa摘要：个性化内容的识别仍然是端到端语音识别中的挑战。我们探索了三种新方法，它们在神经重新评分步骤中使用个性化内容来提高识别率：地名录、提示和基于交叉注意的编码器-解码器模型。我们使用内部去识别en-US数据与虚拟语音助手的交互，并辅以个性化的命名实体来比较这些方法。在具有个性化命名实体的测试集上，我们表明，这些方法中的每一种都将单词错误率提高了10%以上，与神经重新评分基线相比。我们还表明，在这个测试集上，自然语言提示可以在没有任何训练的情况下将单词错误率提高7%，并且在泛化方面略有损失。总体而言，地名录被认为是表现最好的字错误率（WER）的10%的改善，同时也提高WER的一般测试集的1%。摘要：Recognition of personalized content remains a challenge in end-to-end speech recognition. We explore three novel approaches that use personalized content in a neural rescoring step to improve recognition: gazetteers, prompting, and a cross-attention based encoder-decoder model. We use internal de-identified en-US data from interactions with a virtual voice assistant supplemented with personalized named entities to compare these approaches. On a test set with personalized named entities, we show that each of these approaches improves word error rate by over 10%, against a neural rescoring baseline. We also show that on this test set, natural language prompts can improve word error rate by 7% without any training and with a marginal loss in generalization. Overall, gazetteers were found to perform the best with a 10% improvement in word error rate (WER), while also improving WER on a general test set by 1%.
【2】 An Improved Metric of Informational Masking for Perceptual Audio Quality  Measurement标题：一种改进的感知音频质量信息掩蔽度量方法链接：https://arxiv.org/abs/2307.06656
作者：Pablo M. Delgado,Jürgen Herre机构：⋆ International Audio Laboratories Erlangen ‡, Am Wolfsmantel , Erlangen, Germany, † Fraunhofer IIS, Am Wolfsmantel , Erlangen, Germany备注：Accepted Publication for WASPAA 2023 - IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, Mohonk Mountain House, New Paltz, NY, USA, Oct 22-25, 2023摘要：感知音频质量测量系统在算法上分析音频处理系统的输出，以使用人类听觉的感知模型来估计可能的感知质量劣化。以这种方式，它们节省了与听力测试（LT）的设计和执行相关联的时间和资源。在质量测量系统中预测外围听觉掩蔽的干扰可听度模型已经显著地增加了由感知音频编解码器处理的信号的主观质量预测性能。此外，认知效应也被称为调节感知失真的严重性，通过影响他们的显着性。然而，由于在质量测量系统中的认知效应模型的性能增益是不一致的，到目前为止，特别是对于音乐信号。首先，本文提出了一种改进的模型信息掩蔽（IM）-质量感知的一个重要的认知效应-考虑干扰信息的复杂性周围的掩蔽阈值。其次，我们将建议的IM度量到一个质量测量系统，使用一种新的认知效果和失真度量之间的交互分析程序。该程序建立认知效应和失真指标之间的相互作用，使用LT数据。所提出的IM指标优于先前提出的IM指标在验证任务中对主观质量分数从大型和多样化的LT数据库。特别地，所提出的系统示出了用带宽扩展技术编码的音乐信号的增加的质量预测，其中其他模型经常失败。摘要：Perceptual audio quality measurement systems algorithmically analyze the output of audio processing systems to estimate possible perceived quality degradation using perceptual models of human audition. In this manner, they save the time and resources associated with the design and execution of listening tests (LTs). Models of disturbance audibility predicting peripheral auditory masking in quality measurement systems have considerably increased subjective quality prediction performance of signals processed by perceptual audio codecs. Additionally, cognitive effects have also been known to regulate perceived distortion severity by influencing their salience. However, the performance gains due to cognitive effect models in quality measurement systems were inconsistent so far, particularly for music signals. Firstly, this paper presents an improved model of informational masking (IM) -- an important cognitive effect in quality perception -- that considers disturbance information complexity around the masking threshold. Secondly, we incorporate the proposed IM metric into a quality measurement systems using a novel interaction analysis procedure between cognitive effects and distortion metrics. The procedure establishes interactions between cognitive effects and distortion metrics using LT data. The proposed IM metric is shown to outperform previously proposed IM metrics in a validation task against subjective quality scores from large and diverse LT databases. Particularly, the proposed system showed an increased quality prediction of music signals coded with bandwidth extension techniques, where other models frequently fail.
【3】 LACE: A light-weight, causal model for enhancing coded speech through  adaptive convolutions标题：LACE：一种通过自适应卷积增强编码语音的轻量级因果模型链接：https://arxiv.org/abs/2307.06610
作者：Jan Büthe,Jean-Marc Valin,Ahmed Mustafa机构：Amazon Web Services, Palo Alto, USA备注：5 pages, accepted at WASPAA 2023摘要：经典的语音编码使用具有零前视的低复杂度后置滤波器来提高编码语音的质量，但其有效性受到其简单性的限制。深度神经网络（DNN）可能更有效，但需要高复杂性和模型大小，或增加延迟。我们提出了一种DNN模型，该模型在每帧的基础上生成经典的滤波器内核，模型的参数仅为300~K，复杂度为100~MFLOPS，这对于桌面或移动终端CPU来说是一个实际的复杂度。缺乏增加的延迟允许它被集成到Opus编解码器，我们证明，它使有效的宽带编码的比特率下降到6 kb/s。摘要：Classical speech coding uses low-complexity postfilters with zero lookahead to enhance the quality of coded speech, but their effectiveness is limited by their simplicity. Deep Neural Networks (DNNs) can be much more effective, but require high complexity and model size, or added delay. We propose a DNN model that generates classical filter kernels on a per-frame basis with a model of just 300~K parameters and 100~MFLOPS complexity, which is a practical complexity for desktop or mobile device CPUs. The lack of added delay allows it to be integrated into the Opus codec, and we demonstrate that it enables effective wideband encoding for bitrates down to 6 kb/s.
【4】 AnuraSet: A dataset for benchmarking Neotropical anuran calls  identification in passive acoustic monitoring标题：AnuraSet：被动声监测中新热带无尾两栖动物叫声识别的基准数据集链接：https://arxiv.org/abs/2307.06860
作者：Juan Sebastián Cañas,Maria Paula Toro-Gómez,Larissa Sayuri Moreira Sugai,Hernán Darío Benítez Restrepo,Jorge Rudas,Breyner Posso Bautista,Luís Felipe Toledo,Simone Dena,Adão Henrique Rosa Domingos,Franco Leandro de Souza,Selvino Neckel-Oliveira,Anderson da Rosa,Vítor Carvalho-Rocha,José Vinícius Bernardy,José Luiz Massao Moreira Sugai,Carolina Emília dos Santos,Rogério Pereira Bastos,Diego Llusia,Juan Sebastián Ulloa机构：Ulloa, Affiliations, . Instituto de Investigación de Recursos Biológicos Alexander von Humboldt, Avenida Paseo, Bolívar ,-, Bogotá, Colombia; ,. K Lisa Yang Center for Conservation Bioacoustics, Cornell摘要：全球变化的预测，诱导无尾两栖动物的声学行为，这可以通过被动声学监测（PAM）进行研究的转变。了解呼叫行为的变化需要确定无尾两栖动物的物种，这是具有挑战性的，由于新热带区音景的特殊特征。在本文中，我们介绍了一个大规模的多物种数据集的无尾两栖动物的电话记录的PAM，包括27小时的专家注释42个不同的物种从两个巴西生物群系。我们提供了开放访问的数据集，包括原始记录，实验设置代码，和一个基准与细粒度分类问题的基线模型。此外，我们强调了数据集的挑战，以鼓励机器学习研究人员解决无尾两栖类对保护政策的呼叫识别问题。我们所有的实验和资源都可以在我们的GitHub存储库https://github.com/soundclim/anuraset上找到。摘要：Global change is predicted to induce shifts in anuran acoustic behavior, which can be studied through passive acoustic monitoring (PAM). Understanding changes in calling behavior requires the identification of anuran species, which is challenging due to the particular characteristics of neotropical soundscapes. In this paper, we introduce a large-scale multi-species dataset of anuran amphibians calls recorded by PAM, that comprises 27 hours of expert annotations for 42 different species from two Brazilian biomes. We provide open access to the dataset, including the raw recordings, experimental setup code, and a benchmark with a baseline model of the fine-grained categorization problem. Additionally, we highlight the challenges of the dataset to encourage machine learning researchers to solve the problem of anuran call identification towards conservation policy. All our experiments and resources can be found on our GitHub repository https://github.com/soundclim/anuraset.
【5】 Uncovering the Deceptions: An Analysis on Audio Spoofing Detection and  Future Prospects标题：揭开骗局：音频欺骗检测分析及未来展望链接：https://arxiv.org/abs/2307.06669
作者：Rishabh Ranjan,Mayank Vatsa,Richa Singh机构：Indian Institute of Technology, Jodhpur, India备注：Accepted in IJCAI 2023摘要：音频已经成为一种越来越重要的生物识别方式，因为它能够为人类提供与机器交互的直观方式。它目前被用于一系列应用，包括个人身份验证到银行到虚拟助理。研究表明，这些系统也容易受到欺骗和攻击。因此，保护音频处理系统免受诸如身份盗窃、金融欺诈和传播错误信息之类的欺诈活动的影响是至关重要的。本文回顾了当前国家的最先进的技术检测音频欺骗，并讨论了当前的挑战，以及开放的研究问题。本文进一步强调了考虑音频欺骗检测系统的道德和隐私影响的重要性。最后，这项工作的目的是强调需要建立更强大的和可推广的方法，自动说话人验证和对策系统的集成，以及更好的评估协议。摘要：Audio has become an increasingly crucial biometric modality due to its ability to provide an intuitive way for humans to interact with machines. It is currently being used for a range of applications, including person authentication to banking to virtual assistants. Research has shown that these systems are also susceptible to spoofing and attacks. Therefore, protecting audio processing systems against fraudulent activities, such as identity theft, financial fraud, and spreading misinformation, is of paramount importance. This paper reviews the current state-of-the-art techniques for detecting audio spoofing and discusses the current challenges along with open research problems. The paper further highlights the importance of considering the ethical and privacy implications of audio spoofing detection systems. Lastly, the work aims to accentuate the need for building more robust and generalizable methods, the integration of automatic speaker verification and countermeasure systems, and better evaluation protocols.
【6】 Exploring the Integration of Large Language Models into Automatic Speech  Recognition Systems: An Empirical Study标题：探索将大型语言模型集成到自动语音识别系统中：一项实证研究链接：https://arxiv.org/abs/2307.06530
作者：Zeping Min,Jinbo Wang机构： Peking University, No., Yiheyuan Road, Haidian District, Beijing , P.R.China摘要：本文探讨了大语言模型（LLM）到自动语音识别（ASR）系统的集成，以提高转录的准确性。LLM的日益复杂性，以及它们的上下文学习能力和指令遵循行为，在自然语言处理（NLP）领域引起了极大的关注。我们的主要重点是调查使用LLM的上下文学习能力，以提高ASR系统的性能，目前面临的挑战，如环境噪声，扬声器口音和复杂的语言环境的潜力。我们使用Aishell-1和LibriSpeech数据集设计了一项研究，ChatGPT和GPT-4作为LLM功能的基准。不幸的是，我们最初的实验并没有产生有希望的结果，表明利用LLM的上下文学习ASR应用程序的复杂性。尽管有不同的设置和模型的进一步探索，从LLM校正的句子经常导致较高的词错误率（WER），表明在语音应用LLM的局限性。本文提供了这些实验的详细概述，他们的结果和影响，建立使用LLM的上下文学习能力，以纠正语音识别转录中的潜在错误，在当前阶段仍然是一个具有挑战性的任务。摘要：This paper explores the integration of Large Language Models (LLMs) into Automatic Speech Recognition (ASR) systems to improve transcription accuracy. The increasing sophistication of LLMs, with their in-context learning capabilities and instruction-following behavior, has drawn significant attention in the field of Natural Language Processing (NLP). Our primary focus is to investigate the potential of using an LLM's in-context learning capabilities to enhance the performance of ASR systems, which currently face challenges such as ambient noise, speaker accents, and complex linguistic contexts. We designed a study using the Aishell-1 and LibriSpeech datasets, with ChatGPT and GPT-4 serving as benchmarks for LLM capabilities. Unfortunately, our initial experiments did not yield promising results, indicating the complexity of leveraging LLM's in-context learning for ASR applications. Despite further exploration with varied settings and models, the corrected sentences from the LLMs frequently resulted in higher Word Error Rates (WER), demonstrating the limitations of LLMs in speech applications. This paper provides a detailed overview of these experiments, their results, and implications, establishing that using LLMs' in-context learning capabilities to correct potential errors in speech recognition transcriptions is still a challenging task at the current stage.
【7】 Temporal Label-Refinement for Weakly-Supervised Audio-Visual Event  Localization标题：用于弱监督视听事件定位的时间标签细化算法链接：https://arxiv.org/abs/2307.06385
作者：Kalyan Ramakrishnan机构：Temporal Label-Refinement for Weakly-SupervisedAudio-Visual Event LocalizationKalyan RIndian Institute of Technology Madraskalyan08 2 1摘要：视听事件定位（AVEL）是对视听事件进行时间定位和分类的任务，即视频中同时可见和可听的事件。在本文中，我们解决AVEL在一个弱监督设置，其中只有视频级别的事件标签（他们的存在/不存在，但不是他们的位置在时间）可作为监督培训。我们的想法是使用一个基本模型来估计训练数据上的标签，在一个更精细的时间分辨率比在视频级别，并重新训练模型与这些标签。即，我们通过（i）用来自在视频级标签中不具有重叠的第二视频的那些帧替换切片外的帧，以及（ii）将该合成视频馈送到基本模型中以仅提取所讨论的切片的标签来确定训练视频中的帧的每个切片的标签子集。为了处理我们的合成视频的分布外的性质，我们提出了一个辅助目标的基础模型，诱导更可靠的预测本地化的事件标签所需。我们的三个阶段的流水线优于现有的AVEL方法，没有架构上的变化，并提高了相关的弱监督任务的性能。摘要：Audio-Visual Event Localization (AVEL) is the task of temporally localizing and classifying \emph{audio-visual events}, i.e., events simultaneously visible and audible in a video. In this paper, we solve AVEL in a weakly-supervised setting, where only video-level event labels (their presence/absence, but not their locations in time) are available as supervision for training. Our idea is to use a base model to estimate labels on the training data at a finer temporal resolution than at the video level and re-train the model with these labels. I.e., we determine the subset of labels for each \emph{slice} of frames in a training video by (i) replacing the frames outside the slice with those from a second video having no overlap in video-level labels, and (ii) feeding this synthetic video into the base model to extract labels for just the slice in question. To handle the out-of-distribution nature of our synthetic videos, we propose an auxiliary objective for the base model that induces more reliable predictions of the localized event labels as desired. Our three-stage pipeline outperforms several existing AVEL methods with no architectural changes and improves performance on a related weakly-supervised task as well.
机器翻译，仅供参考