今日论文合集：cs.SD语音11篇，eess.AS音频处理14篇。cs.SD语音【1】 DCHT: Deep Complex Hybrid Transformer for Speech Enhancement标题：DCHT：用于语音增强的深度复数混合Transformer链接：https://arxiv.org/abs/2310.19602作者：Jialu Li,Junhui Li,Pu Wang,Youshan Zhang备注：IEEE DDP conference摘要：目前大多数基于深度学习的语音增强方法仅在频谱图或波形域中操作。虽然已经提出了组合波形域和频谱域输入的跨域Transformer，但是其性能可以进一步改进。在本文中，我们提出了一种新的深度复杂的混合Transformer，集成了频谱和波形域的方法，以提高语音增强的性能。该模型由两部分组成：一个复杂的Swin-Unet在频谱域和一个双路Transformer网络（DPTnet）在波形域。我们首先在频谱图域中构造一个复Swin-Unet网络，并在复音频频谱中进行语音增强。然后，我们引入改进的DPT通过添加内存压缩的注意。我们的模型能够学习多域特征，以互补的方式减少不同域上的现有噪声。在BirdSoundsDenoising数据集和VCTK+DEMAND数据集上的实验结果表明，与现有方法相比，我们的方法可以实现更好的性能。摘要：Most of the current deep learning-based approaches for speech enhancement only operate in the spectrogram or waveform domain. Although a cross-domain transformer combining waveform- and spectrogram-domain inputs has been proposed, its performance can be further improved. In this paper, we present a novel deep complex hybrid transformer that integrates both spectrogram and waveform domains approaches to improve the performance of speech enhancement. The proposed model consists of two parts: a complex Swin-Unet in the spectrogram domain and a dual-path transformer network (DPTnet) in the waveform domain. We first construct a complex Swin-Unet network in the spectrogram domain and perform speech enhancement in the complex audio spectrum. We then introduce improved DPT by adding memory-compressed attention. Our model is capable of learning multi-domain features to reduce existing noise on different domains in a complementary way. The experimental results on the BirdSoundsDenoising dataset and the VCTK+DEMAND dataset indicate that our method can achieve better performance compared to state-of-the-art methods.【2】 DPATD: Dual-Phase Audio Transformer for Denoising标题：DPATD：用于去噪的双相音频Transformer链接：https://arxiv.org/abs/2310.19588作者：Junhui Li,Pu Wang,Jialu Li,Xinzhe Wang,Youshan Zhang备注：IEEE DDP摘要：最近的高性能的基于变换的语音增强模型表明，时域方法可以达到类似的性能，时频域方法。然而，时域语音增强系统通常接收由大量时间步长组成的输入音频序列，这使得对极长的序列进行建模并训练模型以充分执行具有挑战性。在本文中，我们利用较小的音频块作为输入，以实现音频信息的有效利用，以解决上述挑战。我们提出了一种双相音频去噪Transformer（DPTD），这是一种新的模型，用于在深层结构中组织Transformer层，以学习干净的音频序列进行去噪。DPATD将音频输入分割成更小的块，其中输入长度可以与原始序列长度的平方根成比例。我们的记忆压缩可解释的注意力是有效的，收敛速度比常用的自我注意力模块。大量的实验表明，我们的模型优于国家的最先进的方法。摘要：Recent high-performance transformer-based speech enhancement models demonstrate that time domain methods could achieve similar performance as time-frequency domain methods. However, time-domain speech enhancement systems typically receive input audio sequences consisting of a large number of time steps, making it challenging to model extremely long sequences and train models to perform adequately. In this paper, we utilize smaller audio chunks as input to achieve efficient utilization of audio information to address the above challenges. We propose a dual-phase audio transformer for denoising (DPATD), a novel model to organize transformer layers in a deep structure to learn clean audio sequences for denoising. DPATD splits the audio input into smaller chunks, where the input length can be proportional to the square root of the original sequence length. Our memory-compressed explainable attention is efficient and converges faster compared to the frequently used self-attention module. Extensive experiments demonstrate that our model outperforms state-of-the-art methods.【3】 Sound of Story: Multi-modal Storytelling with Audio标题：故事之声：使用音频的多模态讲故事链接：https://arxiv.org/abs/2310.19264作者：Jaeyeon Bae,Seokhoon Jeong,Seokun Kang,Namgi Han,Jae-Yon Lee,Hyounghun Kim,Taehwan Kim备注：Findings of EMNLP 2023, project: this https URL摘要：故事在现实世界中是多模态的。当一个人讲述一个故事时，他可以使用所有的可视化和声音以及故事本身。然而，以前的研究讲故事的数据集和任务很少关注声音，即使声音也传达有意义的语义的故事。因此，我们建议通过建立一个新的组件称为“背景声音”，这是基于故事上下文的音频没有任何语言信息，以扩展故事的理解和讲述领域。为此，我们引入了一个新的数据集，称为“故事之声（SoS）”，它有配对的图像和文本序列与相应的声音或背景音乐的故事。据我们所知，这是最大的精心策划的有声故事数据集。我们的SoS数据集由27，354个故事组成，每个故事有19.6张图像和984小时的语音解耦音频，如背景音乐和其他声音。作为用声音和数据集讲故事的基准任务，我们提出了模态之间的检索任务，以及来自图像-文本序列的音频生成任务，为它们引入了强大的基线。我们相信，所提出的数据集和任务可能会揭示在声音方面讲故事的多模态理解。下载每个任务的数据集和基线代码将在链接中发布：https://github.com/Sosdatasets/SoS_Dataset。摘要：Storytelling is multi-modal in the real world. When one tells a story, one may use all of the visualizations and sounds along with the story itself. However, prior studies on storytelling datasets and tasks have paid little attention to sound even though sound also conveys meaningful semantics of the story. Therefore, we propose to extend story understanding and telling areas by establishing a new component called "background sound" which is story context-based audio without any linguistic information. For this purpose, we introduce a new dataset, called "Sound of Story (SoS)", which has paired image and text sequences with corresponding sound or background music for a story. To the best of our knowledge, this is the largest well-curated dataset for storytelling with sound. Our SoS dataset consists of 27,354 stories with 19.6 images per story and 984 hours of speech-decoupled audio such as background music and other sounds. As benchmark tasks for storytelling with sound and the dataset, we propose retrieval tasks between modalities, and audio generation tasks from image-text sequences, introducing strong baselines for them. We believe the proposed dataset and tasks may shed light on the multi-modal understanding of storytelling in terms of sound. Downloading the dataset and baseline codes for each task will be released in the link: https://github.com/Sosdatasets/SoS_Dataset.【4】 JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation标题：Jen-1 Composer：高保真多轨音乐生成的统一框架链接：https://arxiv.org/abs/2310.19180作者：Yao Yao,Peike Li,Boyu Chen,Alex Wang备注：Preprints摘要：随着生成式人工智能的快速发展，文本到音乐合成任务已经成为从头开始生成音乐的一个有前途的方向。然而，对多轨生成的更细粒度控制仍然是一个开放的挑战。现有的模型表现出强大的原始生成能力，但缺乏灵活性，无法以可控的方式组成单独的轨道并将其组合，这与人类作曲家的典型工作流程不同。为了解决这个问题，我们提出了JEN-1 Composer，这是一个统一的框架，可以通过单个模型有效地对多轨音乐的边缘，条件和联合分布进行建模。JEN-1 Composer框架展示了无缝整合任何基于扩散的音乐生成系统的能力，\textit{e.g.} Jen-1，增强其多功能多轨音乐生成能力。我们引入了一个课程培训策略，旨在逐步指导模型从单轨道生成到多轨道组合的灵活生成的过渡。在推理过程中，用户能够迭代地制作和选择符合其偏好的音乐曲目，随后按照所提出的人类-人工智能联合作曲工作流程逐步创建整个音乐作品。定量和定性评估展示了可控和高保真多轨音乐合成的最新性能。JEN-1 Composer代表了交互式AI促进音乐创作和作曲的重大进步。演示将在https://jenmusic.ai/audio-demos上提供。摘要：With rapid advances in generative artificial intelligence, the text-to-music synthesis task has emerged as a promising direction for music generation from scratch. However, finer-grained control over multi-track generation remains an open challenge. Existing models exhibit strong raw generation capability but lack the flexibility to compose separate tracks and combine them in a controllable manner, differing from typical workflows of human composers. To address this issue, we propose JEN-1 Composer, a unified framework to efficiently model marginal, conditional, and joint distributions over multi-track music via a single model. JEN-1 Composer framework exhibits the capacity to seamlessly incorporate any diffusion-based music generation system, \textit{e.g.} Jen-1, enhancing its capacity for versatile multi-track music generation. We introduce a curriculum training strategy aimed at incrementally instructing the model in the transition from single-track generation to the flexible generation of multi-track combinations. During the inference, users have the ability to iteratively produce and choose music tracks that meet their preferences, subsequently creating an entire musical composition incrementally following the proposed Human-AI co-composition workflow. Quantitative and qualitative assessments demonstrate state-of-the-art performance in controllable and high-fidelity multi-track music synthesis. The proposed JEN-1 Composer represents a significant advance toward interactive AI-facilitated music creation and composition. Demos will be available at https://jenmusic.ai/audio-demos.【5】 Deep Audio Analyzer: a Framework to Industrialize the Research on Audio Forensics标题：深度音频分析仪：音频取证研究产业化的框架链接：https://arxiv.org/abs/2310.19081作者：Valerio Francesco Puglisi,Oliver Giudice,Sebastiano Battiato摘要：Deep Audio Analyzer是一个开源语音框架，旨在简化神经语音处理管道的研究和开发过程，允许用户以快速和可复制的方式构思，比较和共享结果。本文介绍了核心架构，旨在支持几个共同感兴趣的音频取证领域的任务，从而定制框架创建新的任务的可能性。借助Deep Audio Analyzer，取证人员（即来自执法机构的人员）和研究人员将能够可视化音频特征，轻松评估预训练模型的性能，通过结合深度神经网络模型来创建，导出和共享新的音频分析工作流程。该工具的优势之一是加快音频取证分析领域的研究和实践实验，从而通过导出和共享管道来提高实验再现性。所有功能都是在用户可通过图形用户界面访问的模块中开发的。索引术语：语音处理，深度学习音频，深度学习音频管道创建，音频取证。摘要：Deep Audio Analyzer is an open source speech framework that aims to simplify the research and the development process of neural speech processing pipelines, allowing users to conceive, compare and share results in a fast and reproducible way. This paper describes the core architecture designed to support several tasks of common interest in the audio forensics field, showing possibility of creating new tasks thus customizing the framework. By means of Deep Audio Analyzer, forensics examiners (i.e. from Law Enforcement Agencies) and researchers will be able to visualize audio features, easily evaluate performances on pretrained models, to create, export and share new audio analysis workflows by combining deep neural network models with few clicks. One of the advantages of this tool is to speed up research and practical experimentation, in the field of audio forensics analysis thus also improving experimental reproducibility by exporting and sharing pipelines. All features are developed in modules accessible by the user through a Graphic User Interface. Index Terms: Speech Processing, Deep Learning Audio, Deep Learning Audio Pipeline creation, Audio Forensics.【6】 Feature Aggregation in Joint Sound Classification and Localization Neural Networks标题：联合声音分类和定位神经网络中的特征聚合链接：https://arxiv.org/abs/2310.19063作者：Brendan Healy,Patrick McNamee,Zahra Nili Ahmadabadi摘要：本研究探讨了深度学习技术在联合声音信号分类和定位网络中的应用。目前最先进的声源定位深度学习网络在其架构中缺乏特征聚合。特征聚合通过整合来自不同特征尺度的信息来增强模型性能，从而提高特征的鲁棒性和不变性。这在SSL网络中尤其重要，因为SSL网络必须区分直接和间接声学信号。为了解决这一差距，我们将计算机视觉神经网络的特征聚合技术应用于信号检测神经网络。此外，我们还提出了用于特征聚合的尺度编码网络（SEN），以编码来自各种尺度的特征，压缩网络以实现更高效的计算聚合。为了评估SSL网络中特征聚合的有效性，我们将以下计算机视觉特征聚合子架构集成到SSL控制架构中：路径聚合网络（PANet），加权双向特征金字塔网络（BiFPN）和SEN。这些子架构使用两个信号分类指标和两个到达方向回归指标进行评估。PANet和BiFPN是计算机视觉模型中已建立的聚合器，而所提出的SEN是一个更紧凑的聚合器。结果表明，模型结合功能聚合优于控制模型，声音事件定位和检测网络（SELDnet），在声音信号分类和定位。特征聚合技术提高了声音检测神经网络的性能，特别是在到达方向回归中。摘要：This study addresses the application of deep learning techniques in joint sound signal classification and localization networks. Current state-of-the-art sound source localization deep learning networks lack feature aggregation within their architecture. Feature aggregation enhances model performance by enabling the consolidation of information from different feature scales, thereby improving feature robustness and invariance. This is particularly important in SSL networks, which must differentiate direct and indirect acoustic signals. To address this gap, we adapt feature aggregation techniques from computer vision neural networks to signal detection neural networks. Additionally, we propose the Scale Encoding Network (SEN) for feature aggregation to encode features from various scales, compressing the network for more computationally efficient aggregation. To evaluate the efficacy of feature aggregation in SSL networks, we integrated the following computer vision feature aggregation sub-architectures into a SSL control architecture: Path Aggregation Network (PANet), Weighted Bi-directional Feature Pyramid Network (BiFPN), and SEN. These sub-architectures were evaluated using two metrics for signal classification and two metrics for direction-of-arrival regression. PANet and BiFPN are established aggregators in computer vision models, while the proposed SEN is a more compact aggregator. The results suggest that models incorporating feature aggregations outperformed the control model, the Sound Event Localization and Detection network (SELDnet), in both sound signal classification and localization. The feature aggregation techniques enhance the performance of sound detection neural networks, particularly in direction-of-arrival regression.【7】 Exploring the Emotional Landscape of Music: An Analysis of Valence Trends and Genre Variations in Spotify Music Data标题：探索音乐的情感景观：Spotify音乐数据中的配价趋势和体裁变化分析链接：https://arxiv.org/abs/2310.19052作者：Shruti Dutta,Shashwat Mookherjee备注：6 pages, Accepted at the 18th International Conference for Internet Technology and Secured Transactions, 13-15 November, 2023, St Anne's College, Oxford, UK摘要：本文使用Spotify音乐数据对音乐情感和趋势进行了复杂的分析，包括通过Spotipi API提取的音频特征和效价分数。运用回归模型，时间分析，情绪转换和体裁调查，研究揭示了音乐情感关系的模式。回归模型线性，支持向量，随机森林，岭，预测效价得分。时间分析揭示了随着时间的推移，效价分布的变化，而情绪转换探索阐明播放列表内的情感动态。这项研究有助于对音乐的情感结构进行细致入微的洞察，多年来增强了对音乐与情感之间相互作用的理解。摘要：This paper conducts an intricate analysis of musical emotions and trends using Spotify music data, encompassing audio features and valence scores extracted through the Spotipi API. Employing regression modeling, temporal analysis, mood transitions, and genre investigation, the study uncovers patterns within music-emotion relationships. Regression models linear, support vector, random forest, and ridge, are employed to predict valence scores. Temporal analysis reveals shifts in valence distribution over time, while mood transition exploration illuminates emotional dynamics within playlists. The research contributes to nuanced insights into music's emotional fabric, enhancing comprehension of the interplay between music and emotions through years.【8】 Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition标题：预先训练的语音处理模型包含传播到语音情感识别的类似人类的偏见链接：https://arxiv.org/abs/2310.18877作者：Isaac Slaughter,Craig Greenberg,Reva Schwartz,Aylin Caliskan摘要：以前的工作已经确定，一个人的人口统计数据和讲话风格会影响语音处理模型的性能。但这种偏见从何而来？在这项工作中，我们提出了语音嵌入关联测试（SpEAT），一种用于检测用于许多语音任务的一种类型的模型中的偏差的方法：预训练模型。SpEAT的灵感来自自然语言处理中的词嵌入关联测试，该测试量化了模型对不同概念的表示中的内在偏见，例如种族或效价（某种东西的愉快或不愉快），并捕捉了在大规模社会文化数据上训练的模型学习人类偏见的程度。使用SpEAT，我们在16个英语语音模型（包括4个也在多语言数据上训练的模型）中测试了六种类型的偏差，这些模型来自wav2vec 2.0，HuBERT，WavLM和Whisper模型系列。我们发现，14个或更多的模型显示出积极的效价（愉快）协会与健全的人比残疾人，与欧洲裔美国人比非洲裔美国人，与女性比男性，与美国口音的发言人比非美国口音的发言人，与年轻人比老年人。除了确定预训练的语音模型包含这些偏见之外，我们还表明它们可以产生真实世界的影响。我们将预先训练的模型中发现的偏差与适应语音情感识别（SER）任务的下游模型中的偏差进行了比较，发现在进行的96次测试中，有66次（69%）与SpEAT所指示的正效价更相关的组也倾向于被下游模型预测为具有更高效价。我们的工作提供的证据表明，与基于文本和图像的模型一样，预先训练的基于语音的模型经常学习类似人类的偏见。我们的工作还表明，在预训练模型中发现的偏差可以传播到SER的下游任务。摘要：Previous work has established that a person's demographics and speech style affect how well speech processing models perform for them. But where does this bias come from? In this work, we present the Speech Embedding Association Test (SpEAT), a method for detecting bias in one type of model used for many speech tasks: pre-trained models. The SpEAT is inspired by word embedding association tests in natural language processing, which quantify intrinsic bias in a model's representations of different concepts, such as race or valence (something's pleasantness or unpleasantness) and capture the extent to which a model trained on large-scale socio-cultural data has learned human-like biases. Using the SpEAT, we test for six types of bias in 16 English speech models (including 4 models also trained on multilingual data), which come from the wav2vec 2.0, HuBERT, WavLM, and Whisper model families. We find that 14 or more models reveal positive valence (pleasantness) associations with abled people over disabled people, with European-Americans over African-Americans, with females over males, with U.S. accented speakers over non-U.S. accented speakers, and with younger people over older people. Beyond establishing that pre-trained speech models contain these biases, we also show that they can have real world effects. We compare biases found in pre-trained models to biases in downstream models adapted to the task of Speech Emotion Recognition (SER) and find that in 66 of the 96 tests performed (69%), the group that is more associated with positive valence as indicated by the SpEAT also tends to be predicted as speaking with higher valence by the downstream model. Our work provides evidence that, like text and image-based models, pre-trained speech based-models frequently learn human-like biases. Our work also shows that bias found in pre-trained models can propagate to the downstream task of SER.【9】 MUST: A Multilingual Student-Teacher Learning approach for low-resource speech recognition标题：MASH：一种适用于低资源语音识别的多语言师生学习方法链接：https://arxiv.org/abs/2310.18865作者：Muhammad Umar Farooq,Rehan Ahmad,Thomas Hain备注：Accepted for IEEE ASRU 2023摘要：学生-教师学习或知识蒸馏（KD）先前已用于解决语音识别（ASR）系统训练的数据稀缺问题。然而，KD训练的局限性在于学生模型类必须是教师模型类的适当或不适当子集。如果字符集不相同，它甚至会阻止从声学上相似的语言中提炼出来。在这项工作中，上述限制是通过提出一个多语言的学生-教师（MUST）学习，利用后验映射方法来解决。预训练的映射模型用于将后验从教师语言映射到学生语言ASR。这些映射后验被用作KD学习的软标签。各种教师集成方案进行了实验，训练的ASR模型，为低资源的语言。与基线单语ASR相比，使用MUST学习训练的模型将相对字符错误率（CER）降低了9.5%。摘要：Student-teacher learning or knowledge distillation (KD) has been previously used to address data scarcity issue for training of speech recognition (ASR) systems. However, a limitation of KD training is that the student model classes must be a proper or improper subset of the teacher model classes. It prevents distillation from even acoustically similar languages if the character sets are not same. In this work, the aforementioned limitation is addressed by proposing a MUltilingual Student-Teacher (MUST) learning which exploits a posteriors mapping approach. A pre-trained mapping model is used to map posteriors from a teacher language to the student language ASR. These mapped posteriors are used as soft labels for KD learning. Various teacher ensemble schemes are experimented to train an ASR model for low-resource languages. A model trained with MUST learning reduces relative character error rate (CER) up to 9.5% in comparison with a baseline monolingual ASR.【10】 Audio-Visual Instance Segmentation标题：视听实例分割链接：https://arxiv.org/abs/2310.18709作者：Ruohao Guo,Yaru Chen,Yanyu Qi,Wenzhen Yue,Dantong Niu,Xianghua Ying摘要：在本文中，我们提出了一个新的多模态任务，即视听实例分割（AVIS），其目标是识别，分割和跟踪个人发声对象实例在可听视频，同时。据我们所知，这是第一次，实例分割已扩展到视听领域。为了更好地促进这项研究，我们构建了第一个视听实例分割基准（AVISeg）。具体来说，AVISeg由来自YouTube和公共视听数据集的平均持续时间为62.6秒的1，258个视频组成，其中117个视频通过使用基于分段任何模型（SAM）的交互式半自动标记工具进行了注释。此外，我们提出了一个简单的基线模型的AVIS任务。我们的新模型在Mask2Former中引入了音频分支和跨模态融合模块来定位所有发声对象。最后，我们评估所提出的方法使用两个骨干AVISeg。我们相信，AVIS将激发社区对更全面的多模式理解。摘要：In this paper, we propose a new multi-modal task, namely audio-visual instance segmentation (AVIS), in which the goal is to identify, segment, and track individual sounding object instances in audible videos, simultaneously. To our knowledge, it is the first time that instance segmentation has been extended into the audio-visual domain. To better facilitate this research, we construct the first audio-visual instance segmentation benchmark (AVISeg). Specifically, AVISeg consists of 1,258 videos with an average duration of 62.6 seconds from YouTube and public audio-visual datasets, where 117 videos have been annotated by using an interactive semi-automatic labeling tool based on the Segment Anything Model (SAM). In addition, we present a simple baseline model for the AVIS task. Our new model introduces an audio branch and a cross-modal fusion module to Mask2Former to locate all sounding objects. Finally, we evaluate the proposed method using two backbones on AVISeg. We believe that AVIS will inspire the community towards a more comprehensive multi-modal understanding.【11】 Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Model标题：看透对话：基于扩散模型的视听语音分离链接：https://arxiv.org/abs/2310.19581作者：Suyeon Lee,Chaeyoung Jung,Youngjoon Jang,Jaehun Kim,Joon Son Chung备注：Project page with demo: this https URL摘要：这项工作的目标是提取目标说话人的声音从混合的声音使用视觉线索。现有的视听语音分离工作已经证明了它们具有良好的可懂度，但保持自然度仍然是一个挑战。为了解决这个问题，我们提出了AVDiffuSS，视听语音分离模型的基础上的扩散机制，其能力在生成自然样本。为了有效地融合两种扩散方式，我们还提出了一种基于交叉注意的特征融合机制。该机制是专门为语音域定制的，以在语音生成中整合来自视听对应的语音信息。以这种方式，融合过程保持了特征的高时间分辨率，而没有过多的计算要求。我们证明了所提出的框架在两个基准测试中取得了最先进的结果，包括VoxCeleb2和LRS3，产生的语音具有更好的自然度。摘要：The objective of this work is to extract target speaker's voice from a mixture of voices using visual cues. Existing works on audio-visual speech separation have demonstrated their performance with promising intelligibility, but maintaining naturalness remains a challenge. To address this issue, we propose AVDiffuSS, an audio-visual speech separation model based on a diffusion mechanism known for its capability in generating natural samples. For an effective fusion of the two modalities for diffusion, we also propose a cross-attention-based feature fusion mechanism. This mechanism is specifically tailored for the speech domain to integrate the phonetic information from audio-visual correspondence in speech generation. In this way, the fusion process maintains the high temporal resolution of the features, without excessive computational requirements. We demonstrate that the proposed framework achieves state-of-the-art results on two benchmarks, including VoxCeleb2 and LRS3, producing speech with notably better naturalness.eess.AS音频处理【1】 Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction标题：用于目标语音提取的情景感知视听TF-GridNet链接：https://arxiv.org/abs/2310.19644作者：Zexu Pan,Gordon Wichern,Yoshiki Masuyama,Francois G. Germain,Sameer Khurana,Chiori Hori,Jonathan Le Roux备注：Accepted by ASRU 2023摘要：目标语音提取的目的是基于给定的条件提示，提取被干扰源（诸如噪声或竞争说话者）破坏的目标语音信号。基于最先进的（SOTA）时频扬声器分离模型TF-GridNet的成就，我们提出了AV-GridNet，一个视觉接地的变体，在提取过程中将目标扬声器的面部记录作为调节因素。认识到语音和噪声信号作为干扰源之间的固有差异，我们还提出了SAV-GridNet，这是一种基于网络的感知模型，它首先识别干扰场景的类型，然后应用专门为该场景训练的专用专家模型。我们提出的模型在第二次COG-MHEAR视听语音增强挑战赛上取得了SOTA结果，在客观上和听力测试中均显著优于其他模型。我们还对这两种情况下的结果进行了广泛的分析。摘要：Target speech extraction aims to extract, based on a given conditioning cue, a target speech signal that is corrupted by interfering sources, such as noise or competing speakers. Building upon the achievements of the state-of-the-art (SOTA) time-frequency speaker separation model TF-GridNet, we propose AV-GridNet, a visual-grounded variant that incorporates the face recording of a target speaker as a conditioning factor during the extraction process. Recognizing the inherent dissimilarities between speech and noise signals as interfering sources, we also propose SAV-GridNet, a scenario-aware model that identifies the type of interfering scenario first and then applies a dedicated expert model trained specifically for that scenario. Our proposed model achieves SOTA results on the second COG-MHEAR Audio-Visual Speech Enhancement Challenge, outperforming other models by a significant margin, objectively and in a listening test. We also perform an extensive analysis of the results under the two scenarios.【2】 Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Model标题：看透对话：基于扩散模型的视听语音分离链接：https://arxiv.org/abs/2310.19581作者：Suyeon Lee,Chaeyoung Jung,Youngjoon Jang,Jaehun Kim,Joon Son Chung备注：Project page with demo: this https URL摘要：这项工作的目标是提取目标说话人的声音从混合的声音使用视觉线索。现有的视听语音分离工作已经证明了它们具有良好的可懂度，但保持自然度仍然是一个挑战。为了解决这个问题，我们提出了AVDiffuSS，视听语音分离模型的基础上的扩散机制，其能力在生成自然样本。为了有效地融合两种扩散方式，我们还提出了一种基于交叉注意的特征融合机制。该机制是专门为语音域定制的，以在语音生成中整合来自视听对应的语音信息。以这种方式，融合过程保持了特征的高时间分辨率，而没有过多的计算要求。我们证明了所提出的框架在两个基准测试中取得了最先进的结果，包括VoxCeleb2和LRS3，产生的语音具有更好的自然度。摘要：The objective of this work is to extract target speaker's voice from a mixture of voices using visual cues. Existing works on audio-visual speech separation have demonstrated their performance with promising intelligibility, but maintaining naturalness remains a challenge. To address this issue, we propose AVDiffuSS, an audio-visual speech separation model based on a diffusion mechanism known for its capability in generating natural samples. For an effective fusion of the two modalities for diffusion, we also propose a cross-attention-based feature fusion mechanism. This mechanism is specifically tailored for the speech domain to integrate the phonetic information from audio-visual correspondence in speech generation. In this way, the fusion process maintains the high temporal resolution of the features, without excessive computational requirements. We demonstrate that the proposed framework achieves state-of-the-art results on two benchmarks, including VoxCeleb2 and LRS3, producing speech with notably better naturalness.【3】 Improved Lossless Coding for Storage and Transmission of Multichannel Immersive Audio标题：用于多声道沉浸式音频存储和传输的改进无损编码链接：https://arxiv.org/abs/2310.18461作者：Toni Hirvonen,Mahmoud Namazi摘要：在本文中，用于改善多通道无损编码的技术进行检查。提出了一种用于对相同内容的两个或更多个不同呈现（混合）进行同时编码的方法。信号模型使用上混的过去样本和下混样本的当前时间样本来预测上混。模型参数通过通用线性求解器进行优化，预测残差采用Rice编码。另外，提出了在残差编码之前使用SVD投影。与各种基线进行了比较，包括FLAC。所提出的方法显示出改进的压缩比的沉浸式音频的存储和传输。摘要：In this paper, techniques for improving multichannel lossless coding are examined. A method is proposed for the simultaneous coding of two or more different renderings (mixes) of the same content. The signal model uses both past samples of the upmix, and the current time samples of downmix samples to predict the upmix. Model parameters are optimized via a general linear solver, and the prediction residual is Rice coded. Additionally, the use of an SVD projection prior to residual coding is proposed. A comparison is made against various baselines, including FLAC. The proposed methods show improved compression ratios for the storage and transmission of immersive audio.【4】 MixRep: Hidden Representation Mixup for Low-Resource Speech Recognition标题：MixRep：低资源语音识别的隐藏表示混淆链接：https://arxiv.org/abs/2310.18450作者：Jiamin Xie,John H. L. Hansen备注：Accepted to Interspeech 2023摘要：在本文中，我们提出了MixRep，一个简单而有效的数据增强策略的基础上混合低资源ASR。MixRep在神经网络中插入隐藏表示的特征维度，可以应用于每层的声学特征输入和输出，这概括了以前的MixSpeech方法。此外，我们建议将混合与沿输入的时间轴的正则化相结合，这被显示为互补的。我们将MixRep应用于使用联合CTC损失训练的E2E LAS架构的Conformer编码器。我们实验WSJ数据集和SWB数据集的子集，包括阅读和电话会话语音。实验结果表明，MixRep在低资源ASR中的性能始终优于其他正则化方法。与强大的SpecAugment基线相比，MixRep在eval92集合和eval'2000集合的Callhome部分上实现了+6.5\%和+6.7\%的相对WER减少。摘要：In this paper, we present MixRep, a simple and effective data augmentation strategy based on mixup for low-resource ASR. MixRep interpolates the feature dimensions of hidden representations in the neural network that can be applied to both the acoustic feature input and the output of each layer, which generalizes the previous MixSpeech method. Further, we propose to combine the mixup with a regularization along the time axis of the input, which is shown as complementary. We apply MixRep to a Conformer encoder of an E2E LAS architecture trained with a joint CTC loss. We experiment on the WSJ dataset and subsets of the SWB dataset, covering reading and telephony conversational speech. Experimental results show that MixRep consistently outperforms other regularization methods for low-resource ASR. Compared to a strong SpecAugment baseline, MixRep achieves a +6.5\% and a +6.7\% relative WER reduction on the eval92 set and the Callhome part of the eval'2000 set.【5】 DCHT: Deep Complex Hybrid Transformer for Speech Enhancement标题：DCHT：用于语音增强的深度复数混合转换器链接：https://arxiv.org/abs/2310.19602作者：Jialu Li,Junhui Li,Pu Wang,Youshan Zhang备注：IEEE DDP conference摘要：目前大多数基于深度学习的语音增强方法仅在频谱图或波形域中操作。虽然已经提出了组合波形域和频谱域输入的跨域Transformer，但是其性能可以进一步改进。在本文中，我们提出了一种新的深度复杂的混合Transformer，集成了频谱和波形域的方法，以提高语音增强的性能。该模型由两部分组成：一个复杂的Swin-Unet在频谱域和一个双路Transformer网络（DPTnet）在波形域。我们首先在频谱图域中构造一个复Swin-Unet网络，并在复音频频谱中进行语音增强。然后，我们引入改进的DPT通过添加内存压缩的注意。我们的模型能够学习多域特征，以互补的方式减少不同域上的现有噪声。在BirdSoundsDenoising数据集和VCTK+DEMAND数据集上的实验结果表明，与现有方法相比，我们的方法可以实现更好的性能。摘要：Most of the current deep learning-based approaches for speech enhancement only operate in the spectrogram or waveform domain. Although a cross-domain transformer combining waveform- and spectrogram-domain inputs has been proposed, its performance can be further improved. In this paper, we present a novel deep complex hybrid transformer that integrates both spectrogram and waveform domains approaches to improve the performance of speech enhancement. The proposed model consists of two parts: a complex Swin-Unet in the spectrogram domain and a dual-path transformer network (DPTnet) in the waveform domain. We first construct a complex Swin-Unet network in the spectrogram domain and perform speech enhancement in the complex audio spectrum. We then introduce improved DPT by adding memory-compressed attention. Our model is capable of learning multi-domain features to reduce existing noise on different domains in a complementary way. The experimental results on the BirdSoundsDenoising dataset and the VCTK+DEMAND dataset indicate that our method can achieve better performance compared to state-of-the-art methods.【6】 DPATD: Dual-Phase Audio Transformer for Denoising标题：DPATD：用于去噪的双相音频Transformer链接：https://arxiv.org/abs/2310.19588作者：Junhui Li,Pu Wang,Jialu Li,Xinzhe Wang,Youshan Zhang备注：IEEE DDP摘要：最近的高性能的基于变换的语音增强模型表明，时域方法可以达到类似的性能，时频域方法。然而，时域语音增强系统通常接收由大量时间步长组成的输入音频序列，这使得对极长的序列进行建模并训练模型以充分执行具有挑战性。在本文中，我们利用较小的音频块作为输入，以实现音频信息的有效利用，以解决上述挑战。我们提出了一种双相音频去噪Transformer（DPTD），这是一种新的模型，用于在深层结构中组织Transformer层，以学习干净的音频序列进行去噪。DPATD将音频输入分割成更小的块，其中输入长度可以与原始序列长度的平方根成比例。我们的记忆压缩可解释的注意力是有效的，收敛速度比常用的自我注意力模块。大量的实验表明，我们的模型优于国家的最先进的方法。摘要：Recent high-performance transformer-based speech enhancement models demonstrate that time domain methods could achieve similar performance as time-frequency domain methods. However, time-domain speech enhancement systems typically receive input audio sequences consisting of a large number of time steps, making it challenging to model extremely long sequences and train models to perform adequately. In this paper, we utilize smaller audio chunks as input to achieve efficient utilization of audio information to address the above challenges. We propose a dual-phase audio transformer for denoising (DPATD), a novel model to organize transformer layers in a deep structure to learn clean audio sequences for denoising. DPATD splits the audio input into smaller chunks, where the input length can be proportional to the square root of the original sequence length. Our memory-compressed explainable attention is efficient and converges faster compared to the frequently used self-attention module. Extensive experiments demonstrate that our model outperforms state-of-the-art methods.【7】 Sound of Story: Multi-modal Storytelling with Audio标题：故事之声：多模式有声讲故事链接：https://arxiv.org/abs/2310.19264作者：Jaeyeon Bae,Seokhoon Jeong,Seokun Kang,Namgi Han,Jae-Yon Lee,Hyounghun Kim,Taehwan Kim备注：Findings of EMNLP 2023, project: this https URL摘要：故事在现实世界中是多模态的。当一个人讲述一个故事时，他可以使用所有的可视化和声音以及故事本身。然而，以前的研究讲故事的数据集和任务很少关注声音，即使声音也传达有意义的语义的故事。因此，我们建议通过建立一个新的组件称为“背景声音”，这是基于故事上下文的音频没有任何语言信息，以扩展故事的理解和讲述领域。为此，我们引入了一个新的数据集，称为“故事之声（SoS）”，它有配对的图像和文本序列与相应的声音或背景音乐的故事。据我们所知，这是最大的精心策划的有声故事数据集。我们的SoS数据集由27，354个故事组成，每个故事有19.6张图像和984小时的语音解耦音频，如背景音乐和其他声音。作为用声音和数据集讲故事的基准任务，我们提出了模态之间的检索任务，以及来自图像-文本序列的音频生成任务，为它们引入了强大的基线。我们相信，所提出的数据集和任务可能会揭示在声音方面讲故事的多模态理解。下载每个任务的数据集和基线代码将在链接中发布：https://github.com/Sosdatasets/SoS_Dataset。摘要：Storytelling is multi-modal in the real world. When one tells a story, one may use all of the visualizations and sounds along with the story itself. However, prior studies on storytelling datasets and tasks have paid little attention to sound even though sound also conveys meaningful semantics of the story. Therefore, we propose to extend story understanding and telling areas by establishing a new component called "background sound" which is story context-based audio without any linguistic information. For this purpose, we introduce a new dataset, called "Sound of Story (SoS)", which has paired image and text sequences with corresponding sound or background music for a story. To the best of our knowledge, this is the largest well-curated dataset for storytelling with sound. Our SoS dataset consists of 27,354 stories with 19.6 images per story and 984 hours of speech-decoupled audio such as background music and other sounds. As benchmark tasks for storytelling with sound and the dataset, we propose retrieval tasks between modalities, and audio generation tasks from image-text sequences, introducing strong baselines for them. We believe the proposed dataset and tasks may shed light on the multi-modal understanding of storytelling in terms of sound. Downloading the dataset and baseline codes for each task will be released in the link: https://github.com/Sosdatasets/SoS_Dataset.【8】 JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation标题：Jen-1 Composer：高保真多轨音乐生成的统一框架链接：https://arxiv.org/abs/2310.19180作者：Yao Yao,Peike Li,Boyu Chen,Alex Wang备注：Preprints摘要：随着生成式人工智能的快速发展，文本到音乐合成任务已经成为从头开始生成音乐的一个有前途的方向。然而，对多轨生成的更细粒度控制仍然是一个开放的挑战。现有的模型表现出强大的原始生成能力，但缺乏灵活性，无法以可控的方式组成单独的轨道并将其组合，这与人类作曲家的典型工作流程不同。为了解决这个问题，我们提出了JEN-1 Composer，这是一个统一的框架，可以通过单个模型有效地对多轨音乐的边缘，条件和联合分布进行建模。JEN-1 Composer框架展示了无缝整合任何基于扩散的音乐生成系统的能力，\textit{e.g.} Jen-1，增强其多功能多轨音乐生成能力。我们引入了一个课程培训策略，旨在逐步指导模型从单轨道生成到多轨道组合的灵活生成的过渡。在推理过程中，用户能够迭代地制作和选择符合其偏好的音乐曲目，随后按照所提出的人类-人工智能联合作曲工作流程逐步创建整个音乐作品。定量和定性评估展示了可控和高保真多轨音乐合成的最新性能。JEN-1 Composer代表了交互式AI促进音乐创作和作曲的重大进步。演示将在https://jenmusic.ai/audio-demos上提供。摘要：With rapid advances in generative artificial intelligence, the text-to-music synthesis task has emerged as a promising direction for music generation from scratch. However, finer-grained control over multi-track generation remains an open challenge. Existing models exhibit strong raw generation capability but lack the flexibility to compose separate tracks and combine them in a controllable manner, differing from typical workflows of human composers. To address this issue, we propose JEN-1 Composer, a unified framework to efficiently model marginal, conditional, and joint distributions over multi-track music via a single model. JEN-1 Composer framework exhibits the capacity to seamlessly incorporate any diffusion-based music generation system, \textit{e.g.} Jen-1, enhancing its capacity for versatile multi-track music generation. We introduce a curriculum training strategy aimed at incrementally instructing the model in the transition from single-track generation to the flexible generation of multi-track combinations. During the inference, users have the ability to iteratively produce and choose music tracks that meet their preferences, subsequently creating an entire musical composition incrementally following the proposed Human-AI co-composition workflow. Quantitative and qualitative assessments demonstrate state-of-the-art performance in controllable and high-fidelity multi-track music synthesis. The proposed JEN-1 Composer represents a significant advance toward interactive AI-facilitated music creation and composition. Demos will be available at https://jenmusic.ai/audio-demos.【9】 Deep Audio Analyzer: a Framework to Industrialize the Research on Audio Forensics标题：深度音频分析仪：音频取证研究产业化的框架链接：https://arxiv.org/abs/2310.19081作者：Valerio Francesco Puglisi,Oliver Giudice,Sebastiano Battiato摘要：Deep Audio Analyzer是一个开源语音框架，旨在简化神经语音处理管道的研究和开发过程，允许用户以快速和可复制的方式构思，比较和共享结果。本文介绍了核心架构，旨在支持几个共同感兴趣的音频取证领域的任务，从而定制框架创建新的任务的可能性。借助Deep Audio Analyzer，取证人员（即来自执法机构的人员）和研究人员将能够可视化音频特征，轻松评估预训练模型的性能，通过结合深度神经网络模型来创建，导出和共享新的音频分析工作流程。该工具的优势之一是加快音频取证分析领域的研究和实践实验，从而通过导出和共享管道来提高实验再现性。所有功能都是在用户可通过图形用户界面访问的模块中开发的。索引术语：语音处理，深度学习音频，深度学习音频管道创建，音频取证。摘要：Deep Audio Analyzer is an open source speech framework that aims to simplify the research and the development process of neural speech processing pipelines, allowing users to conceive, compare and share results in a fast and reproducible way. This paper describes the core architecture designed to support several tasks of common interest in the audio forensics field, showing possibility of creating new tasks thus customizing the framework. By means of Deep Audio Analyzer, forensics examiners (i.e. from Law Enforcement Agencies) and researchers will be able to visualize audio features, easily evaluate performances on pretrained models, to create, export and share new audio analysis workflows by combining deep neural network models with few clicks. One of the advantages of this tool is to speed up research and practical experimentation, in the field of audio forensics analysis thus also improving experimental reproducibility by exporting and sharing pipelines. All features are developed in modules accessible by the user through a Graphic User Interface. Index Terms: Speech Processing, Deep Learning Audio, Deep Learning Audio Pipeline creation, Audio Forensics.【10】 Feature Aggregation in Joint Sound Classification and Localization Neural Networks标题：联合声音分类和定位神经网络中的特征聚合链接：https://arxiv.org/abs/2310.19063作者：Brendan Healy,Patrick McNamee,Zahra Nili Ahmadabadi摘要：本研究探讨了深度学习技术在联合声音信号分类和定位网络中的应用。目前最先进的声源定位深度学习网络在其架构中缺乏特征聚合。特征聚合通过整合来自不同特征尺度的信息来增强模型性能，从而提高特征的鲁棒性和不变性。这在SSL网络中尤其重要，因为SSL网络必须区分直接和间接声学信号。为了解决这一差距，我们将计算机视觉神经网络的特征聚合技术应用于信号检测神经网络。此外，我们还提出了用于特征聚合的尺度编码网络（SEN），以编码来自各种尺度的特征，压缩网络以实现更高效的计算聚合。为了评估SSL网络中特征聚合的有效性，我们将以下计算机视觉特征聚合子架构集成到SSL控制架构中：路径聚合网络（PANet），加权双向特征金字塔网络（BiFPN）和SEN。这些子架构使用两个信号分类指标和两个到达方向回归指标进行评估。PANet和BiFPN是计算机视觉模型中已建立的聚合器，而所提出的SEN是一个更紧凑的聚合器。结果表明，模型结合功能聚合优于控制模型，声音事件定位和检测网络（SELDnet），在声音信号分类和定位。特征聚合技术提高了声音检测神经网络的性能，特别是在到达方向回归中。摘要：This study addresses the application of deep learning techniques in joint sound signal classification and localization networks. Current state-of-the-art sound source localization deep learning networks lack feature aggregation within their architecture. Feature aggregation enhances model performance by enabling the consolidation of information from different feature scales, thereby improving feature robustness and invariance. This is particularly important in SSL networks, which must differentiate direct and indirect acoustic signals. To address this gap, we adapt feature aggregation techniques from computer vision neural networks to signal detection neural networks. Additionally, we propose the Scale Encoding Network (SEN) for feature aggregation to encode features from various scales, compressing the network for more computationally efficient aggregation. To evaluate the efficacy of feature aggregation in SSL networks, we integrated the following computer vision feature aggregation sub-architectures into a SSL control architecture: Path Aggregation Network (PANet), Weighted Bi-directional Feature Pyramid Network (BiFPN), and SEN. These sub-architectures were evaluated using two metrics for signal classification and two metrics for direction-of-arrival regression. PANet and BiFPN are established aggregators in computer vision models, while the proposed SEN is a more compact aggregator. The results suggest that models incorporating feature aggregations outperformed the control model, the Sound Event Localization and Detection network (SELDnet), in both sound signal classification and localization. The feature aggregation techniques enhance the performance of sound detection neural networks, particularly in direction-of-arrival regression.【11】 Exploring the Emotional Landscape of Music: An Analysis of Valence Trends and Genre Variations in Spotify Music Data标题：探索音乐的情感景观：Spotify音乐数据中的配价趋势和体裁变化分析链接：https://arxiv.org/abs/2310.19052作者：Shruti Dutta,Shashwat Mookherjee备注：6 pages, Accepted at the 18th International Conference for Internet Technology and Secured Transactions, 13-15 November, 2023, St Anne's College, Oxford, UK摘要：本文使用Spotify音乐数据对音乐情感和趋势进行了复杂的分析，包括通过Spotipi API提取的音频特征和效价分数。运用回归模型，时间分析，情绪转换和体裁调查，研究揭示了音乐情感关系的模式。回归模型线性，支持向量，随机森林，岭，预测效价得分。时间分析揭示了随着时间的推移，效价分布的变化，而情绪转换探索阐明播放列表内的情感动态。这项研究有助于对音乐的情感结构进行细致入微的洞察，多年来增强了对音乐与情感之间相互作用的理解。摘要：This paper conducts an intricate analysis of musical emotions and trends using Spotify music data, encompassing audio features and valence scores extracted through the Spotipi API. Employing regression modeling, temporal analysis, mood transitions, and genre investigation, the study uncovers patterns within music-emotion relationships. Regression models linear, support vector, random forest, and ridge, are employed to predict valence scores. Temporal analysis reveals shifts in valence distribution over time, while mood transition exploration illuminates emotional dynamics within playlists. The research contributes to nuanced insights into music's emotional fabric, enhancing comprehension of the interplay between music and emotions through years.【12】 Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition标题：预先训练的语音处理模型包含传播到语音情感识别的类似人类的偏见链接：https://arxiv.org/abs/2310.18877作者：Isaac Slaughter,Craig Greenberg,Reva Schwartz,Aylin Caliskan摘要：以前的工作已经确定，一个人的人口统计数据和讲话风格会影响语音处理模型的性能。但这种偏见从何而来？在这项工作中，我们提出了语音嵌入关联测试（SpEAT），一种用于检测用于许多语音任务的一种类型的模型中的偏差的方法：预训练模型。SpEAT的灵感来自自然语言处理中的词嵌入关联测试，该测试量化了模型对不同概念的表示中的内在偏见，例如种族或效价（某种东西的愉快或不愉快），并捕捉了在大规模社会文化数据上训练的模型学习人类偏见的程度。使用SpEAT，我们在16个英语语音模型（包括4个也在多语言数据上训练的模型）中测试了六种类型的偏差，这些模型来自wav2vec 2.0，HuBERT，WavLM和Whisper模型系列。我们发现，14个或更多的模型显示出积极的效价（愉快）协会与健全的人比残疾人，与欧洲裔美国人比非洲裔美国人，与女性比男性，与美国口音的发言人比非美国口音的发言人，与年轻人比老年人。除了确定预训练的语音模型包含这些偏见之外，我们还表明它们可以产生真实世界的影响。我们将预先训练的模型中发现的偏差与适应语音情感识别（SER）任务的下游模型中的偏差进行了比较，发现在进行的96次测试中，有66次（69%）与SpEAT所指示的正效价更相关的组也倾向于被下游模型预测为具有更高效价。我们的工作提供的证据表明，与基于文本和图像的模型一样，预先训练的基于语音的模型经常学习类似人类的偏见。我们的工作还表明，在预训练模型中发现的偏差可以传播到SER的下游任务。摘要：Previous work has established that a person's demographics and speech style affect how well speech processing models perform for them. But where does this bias come from? In this work, we present the Speech Embedding Association Test (SpEAT), a method for detecting bias in one type of model used for many speech tasks: pre-trained models. The SpEAT is inspired by word embedding association tests in natural language processing, which quantify intrinsic bias in a model's representations of different concepts, such as race or valence (something's pleasantness or unpleasantness) and capture the extent to which a model trained on large-scale socio-cultural data has learned human-like biases. Using the SpEAT, we test for six types of bias in 16 English speech models (including 4 models also trained on multilingual data), which come from the wav2vec 2.0, HuBERT, WavLM, and Whisper model families. We find that 14 or more models reveal positive valence (pleasantness) associations with abled people over disabled people, with European-Americans over African-Americans, with females over males, with U.S. accented speakers over non-U.S. accented speakers, and with younger people over older people. Beyond establishing that pre-trained speech models contain these biases, we also show that they can have real world effects. We compare biases found in pre-trained models to biases in downstream models adapted to the task of Speech Emotion Recognition (SER) and find that in 66 of the 96 tests performed (69%), the group that is more associated with positive valence as indicated by the SpEAT also tends to be predicted as speaking with higher valence by the downstream model. Our work provides evidence that, like text and image-based models, pre-trained speech based-models frequently learn human-like biases. Our work also shows that bias found in pre-trained models can propagate to the downstream task of SER.【13】 MUST: A Multilingual Student-Teacher Learning approach for low-resource speech recognition标题：MASH：一种适用于低资源语音识别的多语言师生学习方法链接：https://arxiv.org/abs/2310.18865作者：Muhammad Umar Farooq,Rehan Ahmad,Thomas Hain备注：Accepted for IEEE ASRU 2023摘要：学生-教师学习或知识蒸馏（KD）先前已用于解决语音识别（ASR）系统训练的数据稀缺问题。然而，KD训练的局限性在于学生模型类必须是教师模型类的适当或不适当子集。如果字符集不相同，它甚至会阻止从声学上相似的语言中提炼出来。在这项工作中，上述限制是通过提出一个多语言的学生-教师（MUST）学习，利用后验映射方法来解决。预训练的映射模型用于将后验从教师语言映射到学生语言ASR。这些映射后验被用作KD学习的软标签。各种教师集成方案进行了实验，训练的ASR模型，为低资源的语言。与基线单语ASR相比，使用MUST学习训练的模型将相对字符错误率（CER）降低了9.5%。摘要：Student-teacher learning or knowledge distillation (KD) has been previously used to address data scarcity issue for training of speech recognition (ASR) systems. However, a limitation of KD training is that the student model classes must be a proper or improper subset of the teacher model classes. It prevents distillation from even acoustically similar languages if the character sets are not same. In this work, the aforementioned limitation is addressed by proposing a MUltilingual Student-Teacher (MUST) learning which exploits a posteriors mapping approach. A pre-trained mapping model is used to map posteriors from a teacher language to the student language ASR. These mapped posteriors are used as soft labels for KD learning. Various teacher ensemble schemes are experimented to train an ASR model for low-resource languages. A model trained with MUST learning reduces relative character error rate (CER) up to 9.5% in comparison with a baseline monolingual ASR.【14】 Audio-Visual Instance Segmentation标题：视听实例分割链接：https://arxiv.org/abs/2310.18709作者：Ruohao Guo,Yaru Chen,Yanyu Qi,Wenzhen Yue,Dantong Niu,Xianghua Ying摘要：在本文中，我们提出了一个新的多模态任务，即视听实例分割（AVIS），其目标是识别，分割和跟踪个人发声对象实例在可听视频，同时。据我们所知，这是第一次，实例分割已扩展到视听领域。为了更好地促进这项研究，我们构建了第一个视听实例分割基准（AVISeg）。具体来说，AVISeg由来自YouTube和公共视听数据集的平均持续时间为62.6秒的1，258个视频组成，其中117个视频通过使用基于分段任何模型（SAM）的交互式半自动标记工具进行了注释。此外，我们提出了一个简单的基线模型的AVIS任务。我们的新模型在Mask2Former中引入了音频分支和跨模态融合模块来定位所有发声对象。最后，我们评估所提出的方法使用两个骨干AVISeg。我们相信，AVIS将激发社区对更全面的多模式理解。摘要：In this paper, we propose a new multi-modal task, namely audio-visual instance segmentation (AVIS), in which the goal is to identify, segment, and track individual sounding object instances in audible videos, simultaneously. To our knowledge, it is the first time that instance segmentation has been extended into the audio-visual domain. To better facilitate this research, we construct the first audio-visual instance segmentation benchmark (AVISeg). Specifically, AVISeg consists of 1,258 videos with an average duration of 62.6 seconds from YouTube and public audio-visual datasets, where 117 videos have been annotated by using an interactive semi-automatic labeling tool based on the Segment Anything Model (SAM). In addition, we present a simple baseline model for the AVIS task. Our new model introduces an audio branch and a cross-modal fusion module to Mask2Former to locate all sounding objects. Finally, we evaluate the proposed method using two backbones on AVISeg. We believe that AVIS will inspire the community towards a more comprehensive multi-modal understanding.
每日语音速递
