今日论文合集：cs.SD语音12篇，eess.AS音频处理12篇。
cs.SD语音
【1】 Siamese SIREN: Audio Compression with Implicit Neural Representations标题：暹罗：隐式神经表示的音频压缩链接：https://arxiv.org/abs/2306.12957
作者：Luca A. Lanzendörfer,Roger Wattenhofer备注：Published as a workshop paper at ICML 2023 neural compression workshop摘要：隐式神经表示（INR）已经成为一种很有前途的方法，用于表示各种数据模态，包括3D形状，图像和音频。虽然最近的研究已经证明了INR在图像和3D形状压缩中的成功应用，但它们用于音频压缩的潜力在很大程度上仍未被开发。出于这一点，我们提出了一个初步的调查使用的INR音频压缩。我们的研究介绍了Siamese SIREN，一种基于流行的SIREN架构的新方法。我们的实验结果表明，暹罗SIREN实现了卓越的音频重建保真度，同时利用较少的网络参数相比，以前的INR架构。摘要：Implicit Neural Representations (INRs) have emerged as a promising method for representing diverse data modalities, including 3D shapes, images, and audio. While recent research has demonstrated successful applications of INRs in image and 3D shape compression, their potential for audio compression remains largely unexplored. Motivated by this, we present a preliminary investigation into the use of INRs for audio compression. Our study introduces Siamese SIREN, a novel approach based on the popular SIREN architecture. Our experimental results indicate that Siamese SIREN achieves superior audio reconstruction fidelity while utilizing fewer network parameters compared to previous INR architectures.
【2】 AudioPaLM: A Large Language Model That Can Speak and Listen标题：AudioPaLM：一个能说能听的大型语言模型链接：https://arxiv.org/abs/2306.12925
作者：Paul K. Rubenstein,Chulayuth Asawaroengchai,Duc Dung Nguyen,Ankur Bapna,Zalán Borsos,Félix de Chaumont Quitry,Peter Chen,Dalia El Badawy,Wei Han,Eugene Kharitonov,Hannah Muckenhirn,Dirk Padfield,James Qin,Danny Rozenberg,Tara Sainath,Johan Schalkwyk,Matt Sharifi,Michelle Tadmor,Ramanovich,Marco Tagliasacchi,Alexandru Tudor,Mihajlo Velimirović,Damien Vincent,Jiahui Yu,Yongqiang Wang,Vicky Zayats,Neil Zeghidour,Yu Zhang,Zhishuai Zhang,Lukas Zilka,Christian Frank机构：Mihajlo Velimirovi´c, Google备注：Technical report摘要：我们介绍AudioPaLM，一个大型的语言模型的语音理解和生成。AudioPaLM融合基于文本和基于语音的语言模型，PaLM-2 [Anil等人，2023]和AudioLM [Borsos等人，2022]，转换为统一的多模式架构，该架构可以处理和生成文本和语音，其中应用包括语音识别和语音到语音翻译。AudioPaLM继承了保留副语言信息的能力，例如来自AudioLM的说话者身份和语调，以及仅存在于文本大型语言模型（如PaLM-2）中的语言知识。我们证明，初始化AudioPaLM与一个纯文本的大型语言模型的权重，提高语音处理，成功地利用大量的文本训练数据用于预训练，以协助语音任务。由此产生的模型显着优于现有的系统的语音翻译任务，并有能力执行zero-shot语音到文本翻译的许多语言，输入/目标语言的组合没有看到在训练中。AudioPaLM还演示了音频语言模型的功能，例如基于简短的口头提示跨语言传输语音。我们在www.example.com上发布了我们方法的示例https://google-research.github.io/seanet/audiopalm/examples摘要：We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt. We release examples of our method at https://google-research.github.io/seanet/audiopalm/examples
【3】 NoisyILRMA: Diffuse-Noise-Aware Independent Low-Rank Matrix Analysis for  Fast Blind Source Extraction标题：NoisyILRMA：快速盲源提取的扩散噪声感知独立低秩矩阵法链接：https://arxiv.org/abs/2306.12820
作者：Koki Nishida,Norihiro Takamune,Rintaro Ikeshita,Daichi Kitamura,Hiroshi Saruwatari,Tomohiro Nakatani机构：†The University of Tokyo, Graduate School of Information Science and Technology, Tokyo, Japan, ‡NTT Communication Science Laboratories, NTT Corporation, Kyoto, Japan, §National Institute of Technology, Kagawa College, Kagawa, Japan备注：5 pages, 3 figures, accepted for European Signal Processing Conference 2023 (EUSIPCO 2023)摘要：在本文中，我们解决了多通道盲源提取（BSE）的一个单一的源在扩散噪声环境中。为了解决这个问题，甚至比快速多通道非负矩阵分解（FastMNMF）及其变体更快，我们提出了一种称为NoisyILRMA的BSE方法，这是一种修改的独立低秩矩阵分析（ILRMA），以考虑扩散噪声。NoisyILRMA可以实现相当快的BSE通过结合一个算法开发的独立矢量提取。此外，为了提高NoisyILRMA的BSE性能，我们提出了一种机制，切换源模型与ILRMA类非负矩阵分解到一个更有表达力的源模型在优化过程中。在实验中，我们表明，NoisyILRMA运行速度比FastMNMF算法，同时保持BSE性能。我们还确认，切换机制提高了NoisyILRMA的BSE性能。摘要：In this paper, we address the multichannel blind source extraction (BSE) of a single source in diffuse noise environments. To solve this problem even faster than by fast multichannel nonnegative matrix factorization (FastMNMF) and its variant, we propose a BSE method called NoisyILRMA, which is a modification of independent low-rank matrix analysis (ILRMA) to account for diffuse noise. NoisyILRMA can achieve considerably fast BSE by incorporating an algorithm developed for independent vector extraction. In addition, to improve the BSE performance of NoisyILRMA, we propose a mechanism to switch the source model with ILRMA-like nonnegative matrix factorization to a more expressive source model during optimization. In the experiment, we show that NoisyILRMA runs faster than a FastMNMF algorithm while maintaining the BSE performance. We also confirm that the switching mechanism improves the BSE performance of NoisyILRMA.
【4】 Russian assimilatory palatalization is incomplete neutralization标题：俄语同化腭化是不完全的中和链接：https://arxiv.org/abs/2306.12789
作者：Sejin Oh,Jason A. Shaw,Karthik Durvasula,Alexei Kochotov摘要：不完全中和是指在语音中和语境中潜在对比的语音痕迹。本研究报告审查了这样一种情况：C+j序列中的俄语同化腭化。俄语对比了普通辅音和腭化辅音，普通辅音具有涉及舌背回缩（软腭化/小舌化）的次级发音。然而，俄语也有止滑序列，形成几乎最小的对腭化停止。在腭滑音之前的环境中，由于普通塞音的腭化（同化腭化），腭化辅音和普通辅音之间的对比被中和。本研究的目的是探索中和是否完全。要做到这一点，我们进行了一个电磁articulography（EMA）的实验，检查时间的协调和空间位置的舌体在underlyingly腭化辅音和那些来自同化腭化。来自四位俄语母语者的发音结果显示，两种情况下的手势都是作为复杂的片段协调的，即，它们是腭化辅音。然而，存在与在平原阻塞物中的舌背回缩手势的残余存在一致的条件之间的差异。我们认为俄语平腭对比的中和是不完全的;同化腭化条件下的辅音表现出腭化辅音的手势间协调特征，以及潜在的舌背缩回（腭化/悬雍垂化）手势的残余证据。摘要：Incomplete neutralization refers to phonetic traces of underlying contrasts in phonologically neutralizing contexts. The present study examines one such context: Russian assimilatory palatalization in C+j sequences. Russian contrasts plain and palatalized consonants, with the plain consonants having a secondary articulation involving retraction of the tongue dorsum (velarization/uvularization). However, Russian also has stop-glide sequences that form near-minimal pairs with palatalized stops. In the environment preceding palatal glides, the contrast between palatalized and plain consonants is neutralized, due to the palatalization of the plain stop (assimilatory palatalization). The purpose of the study is to explore whether the neutralization is complete. To do so, we conducted an electromagnetic articulography (EMA) experiment examining temporal coordination and the spatial position of the tongue body in underlyingly palatalized consonants and those derived from assimilatory palatalization. Articulatory results from four native speakers of Russian revealed that gestures in both conditions are coordinated as complex segments, i.e., they are palatalized consonants. However, there are differences across conditions consistent with the residual presence of a tongue dorsum retraction gesture in the plain obstruents. We conclude that neutralization of the plain-palatal contrast in Russian is incomplete; consonants in the assimilatory palatalization condition exhibit inter-gestural coordination characteristic of palatalized consonants along with residual evidence of an underlying tongue dorsum retraction (velarization/uvularization) gesture.
【5】 MFCCGAN: A Novel MFCC-Based Speech Synthesizer Using Adversarial  Learning标题：MFCCGAN：一种基于MFCC的对抗性学习语音合成器链接：https://arxiv.org/abs/2306.12785
作者：Mohammad Reza Hasanabadi Majid Behdad Davood Gharavian机构：Davood Gharavian Shahid Beheshti University,  4 kbps is a very low-bit rate codec selected by the United States Department of Defense [8备注：ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：在本文中，我们介绍了MFCCGAN作为一种新型的语音合成器的基础上对抗学习，采用MFCC作为输入，并产生原始的语音波形。受益于GAN模型功能，它产生的语音比基于规则的基于MFCC的语音合成器WORLD具有更高的可懂度。我们评估的模型的基础上流行的侵入性客观语音清晰度测量（STOI）和质量（NISQA评分）。实验结果表明，我们提出的系统优于Librosa MFCC-反转（增加约26%至53%的STOI和16%至78%的NISQA分数）和约10%的可懂度和约4%的自然度相比，传统的基于规则的声码器世界中使用的CycleGAN-VC家庭的上升。然而，WORLD需要额外的数据，如F0。最后，在基于STOI的鉴别器中使用感知损失可以进一步提高质量。基于WebMUSHRA的主观测试也显示了所提出的方法的质量。摘要：In this paper, we introduce MFCCGAN as a novel speech synthesizer based on adversarial learning that adopts MFCCs as input and generates raw speech waveforms. Benefiting the GAN model capabilities, it produces speech with higher intelligibility than a rule-based MFCC-based speech synthesizer WORLD. We evaluated the model based on a popular intrusive objective speech intelligibility measure (STOI) and quality (NISQA score). Experimental results show that our proposed system outperforms Librosa MFCC- inversion (by an increase of about 26% up to 53% in STOI and 16% up to 78% in NISQA score) and a rise of about 10% in intelligibility and about 4% in naturalness in comparison with conventional rule-based vocoder WORLD that used in the CycleGAN-VC family. However, WORLD needs additional data like F0. Finally, using perceptual loss in discriminators based on STOI could improve the quality more. WebMUSHRA-based subjective tests also show the quality of the proposed approach.
【6】 Toward Leveraging Pre-Trained Self-Supervised Frontends for Automatic  Singing Voice Understanding Tasks: Three Case Studies标题：利用预先训练的自我监督前端自动完成歌唱声音理解任务：三个案例研究链接：https://arxiv.org/abs/2306.12714
作者：Yuya Yamamoto机构：University of Tsukuba, Tsukuba, Japan备注：Submitted to APSIPA 2023摘要：自动歌唱声音理解任务，如歌手识别、歌唱声音转录和歌唱技术分类，受益于利用深度学习技术的数据驱动方法。这些方法的工作，即使在丰富的多样性的声音和噪声样本，由于他们的代表性能力。然而，有限的可用性标记的数据仍然是一个重大的障碍，以实现令人满意的性能。近年来，在语音处理和音乐分类领域中，已经使用大量未标记的数据来训练自监督学习模型（SSL模型）。通过针对目标任务对这些模型进行微调，可以在有限的训练数据下实现与传统监督学习相当的性能。因此，在本文中，我们调查的有效性SSL模型的各种唱歌的声音识别任务。我们报告了比较SSL模型用于三种不同任务的实验结果（即，歌唱者识别，歌唱声音转录和歌唱技术分类）作为初步探索和讨论这些发现的目的。实验结果表明，每个SSL模型实现了相当的性能，有时优于国家的最先进的方法相比，对每个任务。我们还进行了逐层分析，以进一步了解SSL模型的行为。摘要：Automatic singing voice understanding tasks, such as singer identification, singing voice transcription, and singing technique classification, benefit from data-driven approaches that utilize deep learning techniques. These approaches work well even under the rich diversity of vocal and noisy samples owing to their representation ability. However, the limited availability of labeled data remains a significant obstacle to achieving satisfactory performance. In recent years, self-supervised learning models (SSL models) have been trained using large amounts of unlabeled data in the field of speech processing and music classification. By fine-tuning these models for the target tasks, comparable performance to conventional supervised learning can be achieved with limited training data. Therefore, in this paper, we investigate the effectiveness of SSL models for various singing voice recognition tasks. We report the results of experiments comparing SSL models for three different tasks (i.e., singer identification, singing voice transcription, and singing technique classification) as initial exploration and aim to discuss these findings. Experimental results show that each SSL model achieves comparable performance and sometimes outperforms compared to state-of-the-art methods on each task. We also conducted a layer-wise analysis to further understand the behavior of the SSL models.
【7】 NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition  via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning标题：NoRefer：半监督语言模型精调对比学习自动语音识别的无参考质量度量链接：https://arxiv.org/abs/2306.12577
作者：Kamer Ali Yuksel,Thiago Ferreira,Golara Javadi,Mohamed El-Badrashiny,Ahmet Gunduz机构：aiXplain Inc., Los Gatos, CA, USA摘要：本文介绍了NoRefER，自动语音识别（ASR）系统的一种新的无参考质量度量。用于评估ASR系统的传统的基于参考的指标需要昂贵的地面实况转录。NoRefER通过使用Siamese网络架构的对比学习来微调用于成对排序ASR假设的多语言语言模型，从而克服了这一限制。自监督NoRefER利用来自ASR的多个压缩级别的假设之间的已知质量关系来学习按质量对样本内假设进行排名，这对于模型比较是必不可少的。半监督版本还使用参考数据集来改善其样本间质量排名，这对于选择潜在错误的样本至关重要。结果表明，NoRefER与基于参考的指标及其样本内排名高度相关，表明无参考ASR评估或a/b测试的高潜力。摘要：This paper introduces NoRefER, a novel referenceless quality metric for automatic speech recognition (ASR) systems. Traditional reference-based metrics for evaluating ASR systems require costly ground-truth transcripts. NoRefER overcomes this limitation by fine-tuning a multilingual language model for pair-wise ranking ASR hypotheses using contrastive learning with Siamese network architecture. The self-supervised NoRefER exploits the known quality relationships between hypotheses from multiple compression levels of an ASR for learning to rank intra-sample hypotheses by quality, which is essential for model comparisons. The semi-supervised version also uses a referenced dataset to improve its inter-sample quality ranking, which is crucial for selecting potentially erroneous samples. The results indicate that NoRefER correlates highly with reference-based metrics and their intra-sample ranks, indicating a high potential for referenceless ASR evaluation or a/b testing.
【8】 Exploring the Role of Audio in Video Captioning标题：探索音频在视频字幕中的作用链接：https://arxiv.org/abs/2306.12559
作者：Yuhan Shen,Linjie Yang,Longyin Wen,Haichao Yu,Ehsan Elhamifar,Heng Wang机构：† Northeastern University, ‡ ByteDance摘要：视频字幕最近的重点是设计可以同时使用视频和文本模式的架构，并使用大规模视频数据集和文本转录进行预训练，例如HowTo 100 M。虽然这些方法已经取得了显着的改善，音频模态往往被忽略的视频字幕。在这项工作中，我们提出了一个视听框架，其目的是充分利用潜在的音频形式的字幕。我们认为，与依赖于通过自动语音识别（ASR）提取的文本转录相比，使用原始音频信号进行学习可能更有益，因为音频具有包括声学事件、说话者身份等在内的附加信息。我们的贡献是双重的。首先，我们观察到，当用视频和音频模态两者进行预训练时，模型过度专用于音频模态，因为地面实况（即，文本抄本）可以仅使用音频来预测。我们提出了一种模态平衡预训练（MBP）损失来缓解这个问题，并显着提高下游任务的性能。其次，我们切片和切片的跨模态模块，这可能会成为一个信息瓶颈，并产生劣质的结果不同的设计选择。我们提出了新的局部-全局融合机制，以提高跨音频和视频的信息交换。我们通过利用四个数据集上的音频模态展示了显着的改进，甚至在一些指标上优于现有技术，而不依赖于文本模态作为输入。摘要：Recent focus in video captioning has been on designing architectures that can consume both video and text modalities, and using large-scale video datasets with text transcripts for pre-training, such as HowTo100M. Though these approaches have achieved significant improvement, the audio modality is often ignored in video captioning. In this work, we present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. Instead of relying on text transcripts extracted via automatic speech recognition (ASR), we argue that learning with raw audio signals can be more beneficial, as audio has additional information including acoustic events, speaker identity, etc. Our contributions are twofold. First, we observed that the model overspecializes to the audio modality when pre-training with both video and audio modality, since the ground truth (i.e., text transcripts) can be solely predicted using audio. We proposed a Modality Balanced Pre-training (MBP) loss to mitigate this issue and significantly improve the performance on downstream tasks. Second, we slice and dice different design choices of the cross-modal module, which may become an information bottleneck and generate inferior results. We proposed new local-global fusion mechanisms to improve information exchange across audio and video. We demonstrate significant improvements by leveraging the audio modality on four datasets, and even outperform the state of the art on some metrics without relying on the text modality as the input.
【9】 Implicit spoken language diarization标题：内隐口语二元化链接：https://arxiv.org/abs/2306.12913
作者：Jagabandhu Mishra,Amartya Chowdhury,S. R. Mahadeva Prasanna机构：Department of Electrical Engineering, Indian Institute of Technology (IIT) Dharwad, India摘要：口语日记化和相关任务主要是使用语音定位方法进行研究的。语音定位方法大多使用显式的语言建模方式，因此需要中间音素建模和转录数据。或者，深度学习方法对时间动态建模的能力可以有助于通过深度嵌入向量对语言信息进行隐式建模。因此，这项工作初步探讨了可用的扬声器日记框架，捕捉说话人的信息隐式地执行LD任务。LD系统使用端到端x向量方法对合成码切换数据的性能分别为6.78%和7.06%，并且对于实际数据，在日志化错误率和Jaccard错误率（JER）方面分别为22.50%和60.38%。性能下降是由于数据不平衡，并在一定程度上通过使用预训练的wave2vec嵌入来解决，该嵌入在JER方面提供了30.74%的相对改善。摘要：Spoken language diarization (LD) and related tasks are mostly explored using the phonotactic approach. Phonotactic approaches mostly use explicit way of language modeling, hence requiring intermediate phoneme modeling and transcribed data. Alternatively, the ability of deep learning approaches to model temporal dynamics may help for the implicit modeling of language information through deep embedding vectors. Hence this work initially explores the available speaker diarization frameworks that capture speaker information implicitly to perform LD tasks. The performance of the LD system on synthetic code-switch data using the end-to-end x-vector approach is 6.78% and 7.06%, and for practical data is 22.50% and 60.38%, in terms of diarization error rate and Jaccard error rate (JER), respectively. The performance degradation is due to the data imbalance and resolved to some extent by using pre-trained wave2vec embeddings that provide a relative improvement of 30.74% in terms of JER.
【10】 Wind Noise Reduction with a Diffusion-based Stochastic Regeneration  Model标题：基于扩散的随机再生模型的风噪声抑制链接：https://arxiv.org/abs/2306.12867
作者：Jean-Marie Lemercier,Joachim Thiemann,Raphael Koning,Timo Gerkmann机构：Universität Hamburg备注：Submitted to VDE 15th ITG conference on Speech Communication摘要：在本文中，我们提出了一种方法，单通道风噪声降低，使用我们以前提出的基于扩散的随机再生模型相结合的预测和生成建模。我们引入了一个非加性的语音噪声模型占的非线性变形的膜所造成的风流和可能的剪辑。我们表明，我们的随机再生模型优于其他基于神经网络的风噪声降低方法，以及纯粹的预测和生成模型，使用模拟和真实记录的风噪声的数据集。我们进一步表明，所提出的方法推广以及通过测试一个看不见的数据集与真实记录的风噪声。所提出的方法的音频样本、数据生成脚本和代码可以在网上找到（https：//uhh.de/inf-sp-storm-wind）。摘要：In this paper we present a method for single-channel wind noise reduction using our previously proposed diffusion-based stochastic regeneration model combining predictive and generative modelling. We introduce a non-additive speech in noise model to account for the non-linear deformation of the membrane caused by the wind flow and possible clipping. We show that our stochastic regeneration model outperforms other neural-network-based wind noise reduction methods as well as purely predictive and generative models, on a dataset using simulated and real-recorded wind noise. We further show that the proposed method generalizes well by testing on an unseen dataset with real-recorded wind noise. Audio samples, data generation scripts and code for the proposed methods can be found online (https://uhh.de/inf-sp-storm-wind).
【11】 Factors Affecting the Performance of Automated Speaker Verification in  Alzheimer's Disease Clinical Trials标题：阿尔茨海默病临床试验中影响自动说话人识别性能的因素链接：https://arxiv.org/abs/2306.12444
作者：Malikeh Ehghaghi,Marija Stanojevic,Ali Akram,Jekaterina Novikova机构：Winterlight Labs Inc., Toronto, Canada备注：Accepted to the 5th Clinical Natural Language Processing Workshop (ClinicalNLP) at ACL 2023摘要：检测重复患者参与临床试验是一项重大挑战，因为重复患者可能会破坏试验结果的可信度和准确性，并导致重大的健康和财务风险。开发准确的自动说话人验证（ASV）模型对于验证注册个体的身份并删除重复项至关重要，但数据的大小和质量会影响ASV性能。然而，对临床环境中可能影响ASV能力的因素的研究有限。在本文中，我们通过分析参与者的人口统计学特征，音频质量标准和阿尔茨海默氏病（AD）的严重程度如何影响ASV的性能，利用数据集的语音记录从659名参与者与不同水平的AD，通过多个语音任务获得。我们的结果表明ASV性能：1）在男性说话者上略好于女性说话者; 2）对于70岁以上的个体降低;（3）非英语母语者比英语母语者更好; 4）受临床医生干扰、噪声背景和不清晰的参与者语音的负面影响; 5）倾向于随着AD严重程度的增加而降低。我们的研究发现，语音生物识别技术提高公平性的问题，某些亚群表现出不同的ASV性能，由于其固有的语音特性。此外，ASV的性能受到语音记录质量的影响，这强调了改善临床试验中数据收集设置的重要性。摘要：Detecting duplicate patient participation in clinical trials is a major challenge because repeated patients can undermine the credibility and accuracy of the trial's findings and result in significant health and financial risks. Developing accurate automated speaker verification (ASV) models is crucial to verify the identity of enrolled individuals and remove duplicates, but the size and quality of data influence ASV performance. However, there has been limited investigation into the factors that can affect ASV capabilities in clinical environments. In this paper, we bridge the gap by conducting analysis of how participant demographic characteristics, audio quality criteria, and severity level of Alzheimer's disease (AD) impact the performance of ASV utilizing a dataset of speech recordings from 659 participants with varying levels of AD, obtained through multiple speech tasks. Our results indicate that ASV performance: 1) is slightly better on male speakers than on female speakers; 2) degrades for individuals who are above 70 years old; 3) is comparatively better for non-native English speakers than for native English speakers; 4) is negatively affected by clinician interference, noisy background, and unclear participant speech; 5) tends to decrease with an increase in the severity level of AD. Our study finds that voice biometrics raise fairness concerns as certain subgroups exhibit different ASV performances owing to their inherent voice characteristics. Moreover, the performance of ASV is influenced by the quality of speech recordings, which underscores the importance of improving the data collection settings in clinical trials.
【12】 DEPAC: a Corpus for Depression and Anxiety Detection from Speech标题：DEPAC：一个从语音中检测抑郁和焦虑的语料库链接：https://arxiv.org/abs/2306.12443
作者：Mashrura Tasnim,Malikeh Ehghaghi,Brian Diep,Jekaterina Novikova机构：Winterlight Labs, Toronto, Canada备注：Accepted to the Eighth Workshop on Computational Linguistics and Clinical Psychology (CLPsych) at NAACL 2022摘要：抑郁和焦虑等精神痛苦在全球疾病负担中所占比例最大。这种疾病的自动诊断系统，由人工智能的最新创新所赋予的权力，可以为减少受影响个人的痛苦铺平道路。开发这样的系统需要信息丰富和平衡的语料库。在这项工作中，我们介绍了一种新的精神困扰分析音频数据集DEPAC，标记的基础上建立的阈值抑郁和焦虑的标准筛选工具。这个大数据集包括每个人的多个语音任务以及相关的人口统计信息。此外，我们还提出了一个由手工策划的声学和语言特征组成的特征集，这些特征在识别人类语音中的精神疾病迹象方面是有效的。最后，我们通过比较在此数据集上构建的基线机器学习模型与在其他知名抑郁症语料库上训练的基线模型的性能，证明了我们提出的音频语料库和特征集在预测抑郁症严重程度方面的质量和有效性。摘要：Mental distress like depression and anxiety contribute to the largest proportion of the global burden of diseases. Automated diagnosis systems of such disorders, empowered by recent innovations in Artificial Intelligence, can pave the way to reduce the sufferings of the affected individuals. Development of such systems requires information-rich and balanced corpora. In this work, we introduce a novel mental distress analysis audio dataset DEPAC, labeled based on established thresholds on depression and anxiety standard screening tools. This large dataset comprises multiple speech tasks per individual, as well as relevant demographic information. Alongside, we present a feature set consisting of hand-curated acoustic and linguistic features, which were found effective in identifying signs of mental illnesses in human speech. Finally, we justify the quality and effectiveness of our proposed audio corpus and feature set in predicting depression severity by comparing the performance of baseline machine learning models built on this dataset with baseline models trained on other well-known depression corpora.
eess.AS音频处理
【1】 Implicit spoken language diarization标题：内隐口语二元化链接：https://arxiv.org/abs/2306.12913
作者：Jagabandhu Mishra,Amartya Chowdhury,S. R. Mahadeva Prasanna机构：Department of Electrical Engineering, Indian Institute of Technology (IIT) Dharwad, India摘要：口语日记化和相关任务主要是使用语音定位方法进行研究的。语音定位方法大多使用显式的语言建模方式，因此需要中间音素建模和转录数据。或者，深度学习方法对时间动态建模的能力可以有助于通过深度嵌入向量对语言信息进行隐式建模。因此，这项工作初步探讨了可用的扬声器日记框架，捕捉说话人的信息隐式地执行LD任务。LD系统使用端到端x向量方法对合成码切换数据的性能分别为6.78%和7.06%，并且对于实际数据，在日志化错误率和Jaccard错误率（JER）方面分别为22.50%和60.38%。性能下降是由于数据不平衡，并在一定程度上通过使用预训练的wave2vec嵌入来解决，该嵌入在JER方面提供了30.74%的相对改善。摘要：Spoken language diarization (LD) and related tasks are mostly explored using the phonotactic approach. Phonotactic approaches mostly use explicit way of language modeling, hence requiring intermediate phoneme modeling and transcribed data. Alternatively, the ability of deep learning approaches to model temporal dynamics may help for the implicit modeling of language information through deep embedding vectors. Hence this work initially explores the available speaker diarization frameworks that capture speaker information implicitly to perform LD tasks. The performance of the LD system on synthetic code-switch data using the end-to-end x-vector approach is 6.78% and 7.06%, and for practical data is 22.50% and 60.38%, in terms of diarization error rate and Jaccard error rate (JER), respectively. The performance degradation is due to the data imbalance and resolved to some extent by using pre-trained wave2vec embeddings that provide a relative improvement of 30.74% in terms of JER.
【2】 Wind Noise Reduction with a Diffusion-based Stochastic Regeneration  Model标题：基于扩散的随机再生模型的风噪声抑制链接：https://arxiv.org/abs/2306.12867
作者：Jean-Marie Lemercier,Joachim Thiemann,Raphael Koning,Timo Gerkmann机构：Universität Hamburg备注：Submitted to VDE 15th ITG conference on Speech Communication摘要：在本文中，我们提出了一种方法，单通道风噪声降低，使用我们以前提出的基于扩散的随机再生模型相结合的预测和生成建模。我们引入了一个非加性的语音噪声模型占的非线性变形的膜所造成的风流和可能的剪辑。我们表明，我们的随机再生模型优于其他基于神经网络的风噪声降低方法，以及纯粹的预测和生成模型，使用模拟和真实记录的风噪声的数据集。我们进一步表明，所提出的方法推广以及通过测试一个看不见的数据集与真实记录的风噪声。所提出的方法的音频样本、数据生成脚本和代码可以在网上找到（https：//uhh.de/inf-sp-storm-wind）。摘要：In this paper we present a method for single-channel wind noise reduction using our previously proposed diffusion-based stochastic regeneration model combining predictive and generative modelling. We introduce a non-additive speech in noise model to account for the non-linear deformation of the membrane caused by the wind flow and possible clipping. We show that our stochastic regeneration model outperforms other neural-network-based wind noise reduction methods as well as purely predictive and generative models, on a dataset using simulated and real-recorded wind noise. We further show that the proposed method generalizes well by testing on an unseen dataset with real-recorded wind noise. Audio samples, data generation scripts and code for the proposed methods can be found online (https://uhh.de/inf-sp-storm-wind).
【3】 Factors Affecting the Performance of Automated Speaker Verification in  Alzheimer's Disease Clinical Trials标题：阿尔茨海默病临床试验中影响自动说话人识别性能的因素链接：https://arxiv.org/abs/2306.12444
作者：Malikeh Ehghaghi,Marija Stanojevic,Ali Akram,Jekaterina Novikova机构：Winterlight Labs Inc., Toronto, Canada备注：Accepted to the 5th Clinical Natural Language Processing Workshop (ClinicalNLP) at ACL 2023摘要：检测重复患者参与临床试验是一项重大挑战，因为重复患者可能会破坏试验结果的可信度和准确性，并导致重大的健康和财务风险。开发准确的自动说话人验证（ASV）模型对于验证注册个体的身份并删除重复项至关重要，但数据的大小和质量会影响ASV性能。然而，对临床环境中可能影响ASV能力的因素的研究有限。在本文中，我们通过分析参与者的人口统计学特征，音频质量标准和阿尔茨海默氏病（AD）的严重程度如何影响ASV的性能，利用数据集的语音记录从659名参与者与不同水平的AD，通过多个语音任务获得。我们的结果表明ASV性能：1）在男性说话者上略好于女性说话者; 2）对于70岁以上的个体降低;（3）非英语母语者比英语母语者更好; 4）受临床医生干扰、噪声背景和不清晰的参与者语音的负面影响; 5）倾向于随着AD严重程度的增加而降低。我们的研究发现，语音生物识别技术提高公平性的问题，某些亚群表现出不同的ASV性能，由于其固有的语音特性。此外，ASV的性能受到语音记录质量的影响，这强调了改善临床试验中数据收集设置的重要性。摘要：Detecting duplicate patient participation in clinical trials is a major challenge because repeated patients can undermine the credibility and accuracy of the trial's findings and result in significant health and financial risks. Developing accurate automated speaker verification (ASV) models is crucial to verify the identity of enrolled individuals and remove duplicates, but the size and quality of data influence ASV performance. However, there has been limited investigation into the factors that can affect ASV capabilities in clinical environments. In this paper, we bridge the gap by conducting analysis of how participant demographic characteristics, audio quality criteria, and severity level of Alzheimer's disease (AD) impact the performance of ASV utilizing a dataset of speech recordings from 659 participants with varying levels of AD, obtained through multiple speech tasks. Our results indicate that ASV performance: 1) is slightly better on male speakers than on female speakers; 2) degrades for individuals who are above 70 years old; 3) is comparatively better for non-native English speakers than for native English speakers; 4) is negatively affected by clinician interference, noisy background, and unclear participant speech; 5) tends to decrease with an increase in the severity level of AD. Our study finds that voice biometrics raise fairness concerns as certain subgroups exhibit different ASV performances owing to their inherent voice characteristics. Moreover, the performance of ASV is influenced by the quality of speech recordings, which underscores the importance of improving the data collection settings in clinical trials.
【4】 DEPAC: a Corpus for Depression and Anxiety Detection from Speech标题：DEPAC：一个从语音中检测抑郁和焦虑的语料库链接：https://arxiv.org/abs/2306.12443
作者：Mashrura Tasnim,Malikeh Ehghaghi,Brian Diep,Jekaterina Novikova机构：Winterlight Labs, Toronto, Canada备注：Accepted to the Eighth Workshop on Computational Linguistics and Clinical Psychology (CLPsych) at NAACL 2022摘要：抑郁和焦虑等精神痛苦在全球疾病负担中所占比例最大。这种疾病的自动诊断系统，由人工智能的最新创新所赋予的权力，可以为减少受影响个人的痛苦铺平道路。开发这样的系统需要信息丰富和平衡的语料库。在这项工作中，我们介绍了一种新的精神困扰分析音频数据集DEPAC，标记的基础上建立的阈值抑郁和焦虑的标准筛选工具。这个大数据集包括每个人的多个语音任务以及相关的人口统计信息。此外，我们还提出了一个由手工策划的声学和语言特征组成的特征集，这些特征在识别人类语音中的精神疾病迹象方面是有效的。最后，我们通过比较在此数据集上构建的基线机器学习模型与在其他知名抑郁症语料库上训练的基线模型的性能，证明了我们提出的音频语料库和特征集在预测抑郁症严重程度方面的质量和有效性。摘要：Mental distress like depression and anxiety contribute to the largest proportion of the global burden of diseases. Automated diagnosis systems of such disorders, empowered by recent innovations in Artificial Intelligence, can pave the way to reduce the sufferings of the affected individuals. Development of such systems requires information-rich and balanced corpora. In this work, we introduce a novel mental distress analysis audio dataset DEPAC, labeled based on established thresholds on depression and anxiety standard screening tools. This large dataset comprises multiple speech tasks per individual, as well as relevant demographic information. Alongside, we present a feature set consisting of hand-curated acoustic and linguistic features, which were found effective in identifying signs of mental illnesses in human speech. Finally, we justify the quality and effectiveness of our proposed audio corpus and feature set in predicting depression severity by comparing the performance of baseline machine learning models built on this dataset with baseline models trained on other well-known depression corpora.
【5】 Siamese SIREN: Audio Compression with Implicit Neural Representations标题：暹罗：隐式神经表示的音频压缩链接：https://arxiv.org/abs/2306.12957
作者：Luca A. Lanzendörfer,Roger Wattenhofer备注：Published as a workshop paper at ICML 2023 neural compression workshop摘要：隐式神经表示（INR）已经成为一种很有前途的方法，用于表示各种数据模态，包括3D形状，图像和音频。虽然最近的研究已经证明了INR在图像和3D形状压缩中的成功应用，但它们用于音频压缩的潜力在很大程度上仍未被开发。出于这一点，我们提出了一个初步的调查使用的INR音频压缩。我们的研究介绍了Siamese SIREN，一种基于流行的SIREN架构的新方法。我们的实验结果表明，暹罗SIREN实现了卓越的音频重建保真度，同时利用较少的网络参数相比，以前的INR架构。摘要：Implicit Neural Representations (INRs) have emerged as a promising method for representing diverse data modalities, including 3D shapes, images, and audio. While recent research has demonstrated successful applications of INRs in image and 3D shape compression, their potential for audio compression remains largely unexplored. Motivated by this, we present a preliminary investigation into the use of INRs for audio compression. Our study introduces Siamese SIREN, a novel approach based on the popular SIREN architecture. Our experimental results indicate that Siamese SIREN achieves superior audio reconstruction fidelity while utilizing fewer network parameters compared to previous INR architectures.
【6】 AudioPaLM: A Large Language Model That Can Speak and Listen标题：AudioPaLM：一个能说能听的大型语言模型链接：https://arxiv.org/abs/2306.12925
作者：Paul K. Rubenstein,Chulayuth Asawaroengchai,Duc Dung Nguyen,Ankur Bapna,Zalán Borsos,Félix de Chaumont Quitry,Peter Chen,Dalia El Badawy,Wei Han,Eugene Kharitonov,Hannah Muckenhirn,Dirk Padfield,James Qin,Danny Rozenberg,Tara Sainath,Johan Schalkwyk,Matt Sharifi,Michelle Tadmor,Ramanovich,Marco Tagliasacchi,Alexandru Tudor,Mihajlo Velimirović,Damien Vincent,Jiahui Yu,Yongqiang Wang,Vicky Zayats,Neil Zeghidour,Yu Zhang,Zhishuai Zhang,Lukas Zilka,Christian Frank机构：Mihajlo Velimirovi´c, Google备注：Technical report摘要：我们介绍AudioPaLM，一个大型的语言模型的语音理解和生成。AudioPaLM融合基于文本和基于语音的语言模型，PaLM-2 [Anil等人，2023]和AudioLM [Borsos等人，2022]，转换为统一的多模式架构，该架构可以处理和生成文本和语音，其中应用包括语音识别和语音到语音翻译。AudioPaLM继承了保留副语言信息的能力，例如来自AudioLM的说话者身份和语调，以及仅存在于文本大型语言模型（如PaLM-2）中的语言知识。我们证明，初始化AudioPaLM与一个纯文本的大型语言模型的权重，提高语音处理，成功地利用大量的文本训练数据用于预训练，以协助语音任务。由此产生的模型显着优于现有的系统的语音翻译任务，并有能力执行zero-shot语音到文本翻译的许多语言，输入/目标语言的组合没有看到在训练中。AudioPaLM还演示了音频语言模型的功能，例如基于简短的口头提示跨语言传输语音。我们在www.example.com上发布了我们方法的示例https://google-research.github.io/seanet/audiopalm/examples摘要：We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt. We release examples of our method at https://google-research.github.io/seanet/audiopalm/examples
【7】 NoisyILRMA: Diffuse-Noise-Aware Independent Low-Rank Matrix Analysis for  Fast Blind Source Extraction标题：NoisyILRMA：快速盲源提取的扩散噪声感知独立低秩矩阵法链接：https://arxiv.org/abs/2306.12820
作者：Koki Nishida,Norihiro Takamune,Rintaro Ikeshita,Daichi Kitamura,Hiroshi Saruwatari,Tomohiro Nakatani机构：†The University of Tokyo, Graduate School of Information Science and Technology, Tokyo, Japan, ‡NTT Communication Science Laboratories, NTT Corporation, Kyoto, Japan, §National Institute of Technology, Kagawa College, Kagawa, Japan备注：5 pages, 3 figures, accepted for European Signal Processing Conference 2023 (EUSIPCO 2023)摘要：在本文中，我们解决了多通道盲源提取（BSE）的一个单一的源在扩散噪声环境中。为了解决这个问题，甚至比快速多通道非负矩阵分解（FastMNMF）及其变体更快，我们提出了一种称为NoisyILRMA的BSE方法，这是一种修改的独立低秩矩阵分析（ILRMA），以考虑扩散噪声。NoisyILRMA可以实现相当快的BSE通过结合一个算法开发的独立矢量提取。此外，为了提高NoisyILRMA的BSE性能，我们提出了一种机制，切换源模型与ILRMA类非负矩阵分解到一个更有表达力的源模型在优化过程中。在实验中，我们表明，NoisyILRMA运行速度比FastMNMF算法，同时保持BSE性能。我们还确认，切换机制提高了NoisyILRMA的BSE性能。摘要：In this paper, we address the multichannel blind source extraction (BSE) of a single source in diffuse noise environments. To solve this problem even faster than by fast multichannel nonnegative matrix factorization (FastMNMF) and its variant, we propose a BSE method called NoisyILRMA, which is a modification of independent low-rank matrix analysis (ILRMA) to account for diffuse noise. NoisyILRMA can achieve considerably fast BSE by incorporating an algorithm developed for independent vector extraction. In addition, to improve the BSE performance of NoisyILRMA, we propose a mechanism to switch the source model with ILRMA-like nonnegative matrix factorization to a more expressive source model during optimization. In the experiment, we show that NoisyILRMA runs faster than a FastMNMF algorithm while maintaining the BSE performance. We also confirm that the switching mechanism improves the BSE performance of NoisyILRMA.
【8】 Russian assimilatory palatalization is incomplete neutralization标题：俄语同化腭化是不完全的中和链接：https://arxiv.org/abs/2306.12789
作者：Sejin Oh,Jason A. Shaw,Karthik Durvasula,Alexei Kochotov摘要：不完全中和是指在语音中和语境中潜在对比的语音痕迹。本研究报告审查了这样一种情况：C+j序列中的俄语同化腭化。俄语对比了普通辅音和腭化辅音，普通辅音具有涉及舌背回缩（软腭化/小舌化）的次级发音。然而，俄语也有止滑序列，形成几乎最小的对腭化停止。在腭滑音之前的环境中，由于普通塞音的腭化（同化腭化），腭化辅音和普通辅音之间的对比被中和。本研究的目的是探索中和是否完全。要做到这一点，我们进行了一个电磁articulography（EMA）的实验，检查时间的协调和空间位置的舌体在underlyingly腭化辅音和那些来自同化腭化。来自四位俄语母语者的发音结果显示，两种情况下的手势都是作为复杂的片段协调的，即，它们是腭化辅音。然而，存在与在平原阻塞物中的舌背回缩手势的残余存在一致的条件之间的差异。我们认为俄语平腭对比的中和是不完全的;同化腭化条件下的辅音表现出腭化辅音的手势间协调特征，以及潜在的舌背缩回（腭化/悬雍垂化）手势的残余证据。摘要：Incomplete neutralization refers to phonetic traces of underlying contrasts in phonologically neutralizing contexts. The present study examines one such context: Russian assimilatory palatalization in C+j sequences. Russian contrasts plain and palatalized consonants, with the plain consonants having a secondary articulation involving retraction of the tongue dorsum (velarization/uvularization). However, Russian also has stop-glide sequences that form near-minimal pairs with palatalized stops. In the environment preceding palatal glides, the contrast between palatalized and plain consonants is neutralized, due to the palatalization of the plain stop (assimilatory palatalization). The purpose of the study is to explore whether the neutralization is complete. To do so, we conducted an electromagnetic articulography (EMA) experiment examining temporal coordination and the spatial position of the tongue body in underlyingly palatalized consonants and those derived from assimilatory palatalization. Articulatory results from four native speakers of Russian revealed that gestures in both conditions are coordinated as complex segments, i.e., they are palatalized consonants. However, there are differences across conditions consistent with the residual presence of a tongue dorsum retraction gesture in the plain obstruents. We conclude that neutralization of the plain-palatal contrast in Russian is incomplete; consonants in the assimilatory palatalization condition exhibit inter-gestural coordination characteristic of palatalized consonants along with residual evidence of an underlying tongue dorsum retraction (velarization/uvularization) gesture.
【9】 MFCCGAN: A Novel MFCC-Based Speech Synthesizer Using Adversarial  Learning标题：MFCCGAN：一种基于MFCC的对抗性学习语音合成器链接：https://arxiv.org/abs/2306.12785
作者：Mohammad Reza Hasanabadi Majid Behdad Davood Gharavian机构：Davood Gharavian Shahid Beheshti University,  4 kbps is a very low-bit rate codec selected by the United States Department of Defense [8备注：ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：在本文中，我们介绍了MFCCGAN作为一种新型的语音合成器的基础上对抗学习，采用MFCC作为输入，并产生原始的语音波形。受益于GAN模型功能，它产生的语音比基于规则的基于MFCC的语音合成器WORLD具有更高的可懂度。我们评估的模型的基础上流行的侵入性客观语音清晰度测量（STOI）和质量（NISQA评分）。实验结果表明，我们提出的系统优于Librosa MFCC-反转（增加约26%至53%的STOI和16%至78%的NISQA分数）和约10%的可懂度和约4%的自然度相比，传统的基于规则的声码器世界中使用的CycleGAN-VC家庭的上升。然而，WORLD需要额外的数据，如F0。最后，在基于STOI的鉴别器中使用感知损失可以进一步提高质量。基于WebMUSHRA的主观测试也显示了所提出的方法的质量。摘要：In this paper, we introduce MFCCGAN as a novel speech synthesizer based on adversarial learning that adopts MFCCs as input and generates raw speech waveforms. Benefiting the GAN model capabilities, it produces speech with higher intelligibility than a rule-based MFCC-based speech synthesizer WORLD. We evaluated the model based on a popular intrusive objective speech intelligibility measure (STOI) and quality (NISQA score). Experimental results show that our proposed system outperforms Librosa MFCC- inversion (by an increase of about 26% up to 53% in STOI and 16% up to 78% in NISQA score) and a rise of about 10% in intelligibility and about 4% in naturalness in comparison with conventional rule-based vocoder WORLD that used in the CycleGAN-VC family. However, WORLD needs additional data like F0. Finally, using perceptual loss in discriminators based on STOI could improve the quality more. WebMUSHRA-based subjective tests also show the quality of the proposed approach.
【10】 Toward Leveraging Pre-Trained Self-Supervised Frontends for Automatic  Singing Voice Understanding Tasks: Three Case Studies标题：利用预先训练的自我监督前端自动完成歌唱声音理解任务：三个案例研究链接：https://arxiv.org/abs/2306.12714
作者：Yuya Yamamoto机构：University of Tsukuba, Tsukuba, Japan备注：Submitted to APSIPA 2023摘要：自动歌唱声音理解任务，如歌手识别、歌唱声音转录和歌唱技术分类，受益于利用深度学习技术的数据驱动方法。这些方法的工作，即使在丰富的多样性的声音和噪声样本，由于他们的代表性能力。然而，有限的可用性标记的数据仍然是一个重大的障碍，以实现令人满意的性能。近年来，在语音处理和音乐分类领域中，已经使用大量未标记的数据来训练自监督学习模型（SSL模型）。通过针对目标任务对这些模型进行微调，可以在有限的训练数据下实现与传统监督学习相当的性能。因此，在本文中，我们调查的有效性SSL模型的各种唱歌的声音识别任务。我们报告了比较SSL模型用于三种不同任务的实验结果（即，歌唱者识别，歌唱声音转录和歌唱技术分类）作为初步探索和讨论这些发现的目的。实验结果表明，每个SSL模型实现了相当的性能，有时优于国家的最先进的方法相比，对每个任务。我们还进行了逐层分析，以进一步了解SSL模型的行为。摘要：Automatic singing voice understanding tasks, such as singer identification, singing voice transcription, and singing technique classification, benefit from data-driven approaches that utilize deep learning techniques. These approaches work well even under the rich diversity of vocal and noisy samples owing to their representation ability. However, the limited availability of labeled data remains a significant obstacle to achieving satisfactory performance. In recent years, self-supervised learning models (SSL models) have been trained using large amounts of unlabeled data in the field of speech processing and music classification. By fine-tuning these models for the target tasks, comparable performance to conventional supervised learning can be achieved with limited training data. Therefore, in this paper, we investigate the effectiveness of SSL models for various singing voice recognition tasks. We report the results of experiments comparing SSL models for three different tasks (i.e., singer identification, singing voice transcription, and singing technique classification) as initial exploration and aim to discuss these findings. Experimental results show that each SSL model achieves comparable performance and sometimes outperforms compared to state-of-the-art methods on each task. We also conducted a layer-wise analysis to further understand the behavior of the SSL models.
【11】 NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition  via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning标题：NoRefer：半监督语言模型精调对比学习自动语音识别的无参考质量度量链接：https://arxiv.org/abs/2306.12577
作者：Kamer Ali Yuksel,Thiago Ferreira,Golara Javadi,Mohamed El-Badrashiny,Ahmet Gunduz机构：aiXplain Inc., Los Gatos, CA, USA摘要：本文介绍了NoRefER，自动语音识别（ASR）系统的一种新的无参考质量度量。用于评估ASR系统的传统的基于参考的指标需要昂贵的地面实况转录。NoRefER通过使用Siamese网络架构的对比学习来微调用于成对排序ASR假设的多语言语言模型，从而克服了这一限制。自监督NoRefER利用来自ASR的多个压缩级别的假设之间的已知质量关系来学习按质量对样本内假设进行排名，这对于模型比较是必不可少的。半监督版本还使用参考数据集来改善其样本间质量排名，这对于选择潜在错误的样本至关重要。结果表明，NoRefER与基于参考的指标及其样本内排名高度相关，表明无参考ASR评估或a/b测试的高潜力。摘要：This paper introduces NoRefER, a novel referenceless quality metric for automatic speech recognition (ASR) systems. Traditional reference-based metrics for evaluating ASR systems require costly ground-truth transcripts. NoRefER overcomes this limitation by fine-tuning a multilingual language model for pair-wise ranking ASR hypotheses using contrastive learning with Siamese network architecture. The self-supervised NoRefER exploits the known quality relationships between hypotheses from multiple compression levels of an ASR for learning to rank intra-sample hypotheses by quality, which is essential for model comparisons. The semi-supervised version also uses a referenced dataset to improve its inter-sample quality ranking, which is crucial for selecting potentially erroneous samples. The results indicate that NoRefER correlates highly with reference-based metrics and their intra-sample ranks, indicating a high potential for referenceless ASR evaluation or a/b testing.
【12】 Exploring the Role of Audio in Video Captioning标题：探索音频在视频字幕中的作用链接：https://arxiv.org/abs/2306.12559
作者：Yuhan Shen,Linjie Yang,Longyin Wen,Haichao Yu,Ehsan Elhamifar,Heng Wang机构：† Northeastern University, ‡ ByteDance摘要：视频字幕最近的重点是设计可以同时使用视频和文本模式的架构，并使用大规模视频数据集和文本转录进行预训练，例如HowTo 100 M。虽然这些方法已经取得了显着的改善，音频模态往往被忽略的视频字幕。在这项工作中，我们提出了一个视听框架，其目的是充分利用潜在的音频形式的字幕。我们认为，与依赖于通过自动语音识别（ASR）提取的文本转录相比，使用原始音频信号进行学习可能更有益，因为音频具有包括声学事件、说话者身份等在内的附加信息。我们的贡献是双重的。首先，我们观察到，当用视频和音频模态两者进行预训练时，模型过度专用于音频模态，因为地面实况（即，文本抄本）可以仅使用音频来预测。我们提出了一种模态平衡预训练（MBP）损失来缓解这个问题，并显着提高下游任务的性能。其次，我们切片和切片的跨模态模块，这可能会成为一个信息瓶颈，并产生劣质的结果不同的设计选择。我们提出了新的局部-全局融合机制，以提高跨音频和视频的信息交换。我们通过利用四个数据集上的音频模态展示了显着的改进，甚至在一些指标上优于现有技术，而不依赖于文本模态作为输入。摘要：Recent focus in video captioning has been on designing architectures that can consume both video and text modalities, and using large-scale video datasets with text transcripts for pre-training, such as HowTo100M. Though these approaches have achieved significant improvement, the audio modality is often ignored in video captioning. In this work, we present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. Instead of relying on text transcripts extracted via automatic speech recognition (ASR), we argue that learning with raw audio signals can be more beneficial, as audio has additional information including acoustic events, speaker identity, etc. Our contributions are twofold. First, we observed that the model overspecializes to the audio modality when pre-training with both video and audio modality, since the ground truth (i.e., text transcripts) can be solely predicted using audio. We proposed a Modality Balanced Pre-training (MBP) loss to mitigate this issue and significantly improve the performance on downstream tasks. Second, we slice and dice different design choices of the cross-modal module, which may become an information bottleneck and generate inferior results. We proposed new local-global fusion mechanisms to improve information exchange across audio and video. We demonstrate significant improvements by leveraging the audio modality on four datasets, and even outperform the state of the art on some metrics without relying on the text modality as the input.
机器翻译，仅供参考