今日论文合集：cs.SD语音4篇，eess.AS音频处理5篇。
cs.SD语音
【1】 End-to-end spoken language understanding using joint CTC loss and  self-supervised, pretrained acoustic encoders标题：使用联合CTC Lost和自我监督、预先训练的声学编码器的端到端口语理解链接：https://arxiv.org/abs/2305.02937
作者：Jixuan Wang,Martin Radfar,Kai Wei,Clement Chung机构：Amazon Alexa AI备注：ICASSP 2023摘要：在口语理解中，由于文本信息的缺乏，直接从音频信号中提取语义是一个挑战。流行的端到端（E2E）SLU模型利用序列到序列自动语音识别（ASR）模型来提取文本嵌入作为输入以推断语义，然而，这需要计算上昂贵的自回归解码。在这项工作中，我们利用自监督声学编码器微调与连接主义时间分类（CTC）提取文本嵌入和使用联合CTC和SLU损失的话语级SLU任务。实验表明，我们的模型实现了4%的绝对改善的最先进的（SOTA）的DSTC2数据集上的对话行为分类模型和1.3%的绝对改善的SOTA SLU模型的SLURP数据集。摘要：It is challenging to extract semantic meanings directly from audio signals in spoken language understanding (SLU), due to the lack of textual information. Popular end-to-end (E2E) SLU models utilize sequence-to-sequence automatic speech recognition (ASR) models to extract textual embeddings as input to infer semantics, which, however, require computationally expensive auto-regressive decoding. In this work, we leverage self-supervised acoustic encoders fine-tuned with Connectionist Temporal Classification (CTC) to extract textual embeddings and use joint CTC and SLU losses for utterance-level SLU tasks. Experiments show that our model achieves 4% absolute improvement over the the state-of-the-art (SOTA) dialogue act classification model on the DSTC2 dataset and 1.3% absolute improvement over the SOTA SLU model on the SLURP dataset.
【2】 HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio  Codec标题：高保真音频编解码器：高保真音频编解码器的分组残差矢量量化链接：https://arxiv.org/abs/2305.02765
作者：Dongchao Yang,Songxiang Liu,Rongjie Huang,Jinchuan Tian,Chao Weng,Yuexian Zou机构：Peking University, China,  Tencent AI Lab, Shenzhen, China,  Zhejiang University, China备注：The first version of HiFi-Codec. arXiv admin note: text overlap with arXiv:2210.13438 by other authors摘要：音频编解码器模型作为用于将音频压缩成离散表示的关键技术而广泛用于音频通信中。如今，音频编解码器模型作为中间表示越来越多地用于生成领域。例如，AudioLM是使用SoundStream的离散表示作为训练目标的音频生成模型，而VALL-E采用Encodec模型作为中间特征来辅助TTS任务。尽管它们很有用，但仍然存在两个挑战：（1）训练这些音频编解码器模型可能是困难的，这是由于缺乏公共可用的训练过程以及对大规模数据和GPU的需要;（2）实现良好的重构性能需要许多码本，这增加了生成模型的负担。在这项研究中，我们提出了一种组残差矢量量化（GRVQ）技术，并使用它来开发一种新的 textbf{Hi}gh textbf{Fi}delity音频编解码器模型，HiFi-Codec，它只需要4个码本。我们使用LibriTTS，VCTK，AISHELL等公开可用的TTS数据来训练所有模型，总时长超过1000小时，使用8个GPU。我们的实验结果表明，HiFi-Codec优于Encodec的重建性能，尽管只需要4个码本。为了促进音频编解码器和生成的研究，我们介绍了AcademiCodec，这是第一个开源音频编解码器工具包，为Encodec，SoundStream和HiFi-Codec提供训练代码和预训练模型。代码和预训练模型可以在以下网站找到：href{https：github.com yangdongchao AcademiCodec}{https：github.com yangdongchao AcademiCodec}摘要：Audio codec models are widely used in audio communication as a crucial technique for compressing audio into discrete representations. Nowadays, audio codec models are increasingly utilized in generation fields as intermediate representations. For instance, AudioLM is an audio generation model that uses the discrete representation of SoundStream as a training target, while VALL-E employs the Encodec model as an intermediate feature to aid TTS tasks. Despite their usefulness, two challenges persist: (1) training these audio codec models can be difficult due to the lack of publicly available training processes and the need for large-scale data and GPUs; (2) achieving good reconstruction performance requires many codebooks, which increases the burden on generation models. In this study, we propose a group-residual vector quantization (GRVQ) technique and use it to develop a novel textbf{Hi}gh textbf{Fi}delity Audio Codec model, HiFi-Codec, which only requires 4 codebooks. We train all the models using publicly available TTS data such as LibriTTS, VCTK, AISHELL, and more, with a total duration of over 1000 hours, using 8 GPUs. Our experimental results show that HiFi-Codec outperforms Encodec in terms of reconstruction performance despite requiring only 4 codebooks. To facilitate research in audio codec and generation, we introduce AcademiCodec, the first open-source audio codec toolkit that offers training codes and pre-trained models for Encodec, SoundStream, and HiFi-Codec. Code and pre-trained model can be found on: href{https: github.com yangdongchao AcademiCodec}{https: github.com yangdongchao AcademiCodec}
【3】 Learning to Detect Novel and Fine-Grained Acoustic Sequences Using  Pretrained Audio Representations标题：使用预先训练的音频表示学习检测新的和细粒度的声音序列链接：https://arxiv.org/abs/2305.02382
作者：Vasudha Kowtha,Miquel Espi Marques,Jonathan Huang,Yichi Zhang,Carlos Avendano机构：Apple备注：IEEE ICASSP 2023摘要：这项工作研究了预训练的音频表示的Few Shot的声音事件检测。我们专门解决的任务，Few Shot检测新的声学序列，或声音事件的语义有意义的时间结构，而不假设访问非目标音频。我们开发了预训练合适的表示的程序，并将其转移到我们的Few Shot学习场景的方法。我们的实验评估了我们在AudioSet上的预训练表示的通用实用性，以及通过从现实世界的声学序列构建的任务提出的Few Shot方法的实用性。我们的预训练嵌入适用于所提出的任务，并实现了我们的Few Shot框架的多个方面。摘要：This work investigates pretrained audio representations for few shot Sound Event Detection. We specifically address the task of few shot detection of novel acoustic sequences, or sound events with semantically meaningful temporal structure, without assuming access to non-target audio. We develop procedures for pretraining suitable representations, and methods which transfer them to our few shot learning scenario. Our experiments evaluate the general purpose utility of our pretrained representations on AudioSet, and the utility of proposed few shot methods via tasks constructed from real-world acoustic sequences. Our pretrained embeddings are suitable to the proposed task, and enable multiple aspects of our few shot framework.
【4】 Hybrid AHS: A Hybrid of Kalman Filter and Deep Learning for Acoustic  Howling Suppression标题：混合AHS：卡尔曼滤波和深度学习相结合的声学咆哮抑制方法链接：https://arxiv.org/abs/2305.02583
作者：Hao Zhang,Meng Yu,Yuzhong Wu,Tao Yu,Dong Yu机构：Tencent AI Lab, Bellevue, WA, USA, Tencent Ethereal Audio Lab, Shenzhen, Guangdong, China备注：submitted to INTERSPEECH 2023摘要：深度学习最近被引入用于有效的声学啸叫抑制（AHS）。然而，啸叫的重复性在离线训练和流式推理之间产生不匹配，从而限制了增强语音的质量。为了解决这一限制，我们提出了一种混合方法，结合了卡尔曼滤波器与自关注递归神经网络（SARNN），以利用各自的优势，强大的AHS。在离线训练期间，从卡尔曼滤波器获得的预处理信号和经由教师强制训练策略生成的理想麦克风信号用于训练深度神经网络（DNN）。在流式推理期间，DNN的参数是固定的，而其输出用作用于更新卡尔曼滤波器的参考信号。使用模拟和真实记录的数据在离线和流式推理场景中的评估表明，所提出的方法有效地抑制啸叫，并始终优于基线。摘要：Deep learning has been recently introduced for efficient acoustic howling suppression (AHS). However, the recurrent nature of howling creates a mismatch between offline training and streaming inference, limiting the quality of enhanced speech. To address this limitation, we propose a hybrid method that combines a Kalman filter with a self-attentive recurrent neural network (SARNN) to leverage their respective advantages for robust AHS. During offline training, a pre-processed signal obtained from the Kalman filter and an ideal microphone signal generated via teacher-forced training strategy are used to train the deep neural network (DNN). During streaming inference, the DNN's parameters are fixed while its output serves as a reference signal for updating the Kalman filter. Evaluation in both offline and streaming inference scenarios using simulated and real-recorded data shows that the proposed method efficiently suppresses howling and consistently outperforms baselines.
eess.AS音频处理
【1】 An Acoustic Simulation Framework to Support Indoor Positioning and Data  Driven Signal Processing Assessments标题：支持室内定位和数据驱动信号处理评估的声学仿真框架链接：https://arxiv.org/abs/2305.02715
作者：Daan Delabie,Chesney Buyle,Bert Cox,Liesbet Van der Perre,Lieven De Strycker机构：KU Leuven, WaveCore, Department of Electrical Engineering (ESAT), KU Leuven Ghent, Ghent, Belgium摘要：我们提出了一个室内声学模拟框架，同时支持超声波和可听信号。该框架为快速室内声学数据生成和定位开发提供了机会。改进的Pyroomacoustics为基础的物理模型，包括图像源模型（ISM）和光线跟踪方法来模拟声学信号在几何空间，扩展典型的鞋盒房间。此外，它提供了便利以促进具有不同方向性图案的多个扬声器和麦克风。除了温度和空气吸收之外，还考虑了以RT 60值或建筑材料的组合为特征的房间混响。可以通过后处理和 或额外扬声器来添加附加噪声源。在模拟评估的室内定位方法进行了比较，在一个测试平台，称为“Techtile”的真实测量。该分析证实，仿真结果接近测量结果，并形成现实的真实表示。仿真框架以模块化的方式构建，各部分可以替换或修改以支持不同的应用领域。该代码是开放源代码。摘要：We present an indoor acoustic simulation framework that supports both ultrasonic and audible signaling. The framework opens the opportunity for fast indoor acoustic data generation and positioning development. The improved Pyroomacoustics-based physical model includes both an image-source model (ISM) and ray tracing method to simulate acoustic signaling in geometric spaces that extend typical shoe-box rooms. Moreover, it offers the convenience to facilitate multiple speakers and microphones with different directivity patterns. In addition to temperature and air absorption, the room reverberation is taken into account characterized by the RT60 value or the combination of building materials. Additional noise sources can be added by means of post processing and or extra speakers. Indoor positioning methods assessed in simulation are compared with real measurements in a testbed, called 'Techtile'. This analysis confirms that the simulation results are close to the measurements and form a realistic representation of the reality. The simulation framework is constructed in a modular way, and parts can be replaced or modified to support different application domains. The code is made available open source.
【2】 Hybrid AHS: A Hybrid of Kalman Filter and Deep Learning for Acoustic  Howling Suppression标题：混合AHS：卡尔曼滤波和深度学习相结合的声学咆哮抑制方法链接：https://arxiv.org/abs/2305.02583
*与cs.SD语音【4】为同一篇
作者：Hao Zhang,Meng Yu,Yuzhong Wu,Tao Yu,Dong Yu机构：Tencent AI Lab, Bellevue, WA, USA, Tencent Ethereal Audio Lab, Shenzhen, Guangdong, China备注：submitted to INTERSPEECH 2023摘要：深度学习最近被引入用于有效的声学啸叫抑制（AHS）。然而，啸叫的重复性在离线训练和流式推理之间产生不匹配，从而限制了增强语音的质量。为了解决这一限制，我们提出了一种混合方法，结合了卡尔曼滤波器与自关注递归神经网络（SARNN），以利用各自的优势，强大的AHS。在离线训练期间，从卡尔曼滤波器获得的预处理信号和经由教师强制训练策略生成的理想麦克风信号用于训练深度神经网络（DNN）。在流式推理期间，DNN的参数是固定的，而其输出用作用于更新卡尔曼滤波器的参考信号。使用模拟和真实记录的数据在离线和流式推理场景中的评估表明，所提出的方法有效地抑制啸叫，并始终优于基线。摘要：Deep learning has been recently introduced for efficient acoustic howling suppression (AHS). However, the recurrent nature of howling creates a mismatch between offline training and streaming inference, limiting the quality of enhanced speech. To address this limitation, we propose a hybrid method that combines a Kalman filter with a self-attentive recurrent neural network (SARNN) to leverage their respective advantages for robust AHS. During offline training, a pre-processed signal obtained from the Kalman filter and an ideal microphone signal generated via teacher-forced training strategy are used to train the deep neural network (DNN). During streaming inference, the DNN's parameters are fixed while its output serves as a reference signal for updating the Kalman filter. Evaluation in both offline and streaming inference scenarios using simulated and real-recorded data shows that the proposed method efficiently suppresses howling and consistently outperforms baselines.
【3】 End-to-end spoken language understanding using joint CTC loss and  self-supervised, pretrained acoustic encoders标题：使用联合CTC Lost和自我监督、预先训练的声学编码器的端到端口语理解链接：https://arxiv.org/abs/2305.02937
*与cs.SD语音【1】为同一篇
作者：Jixuan Wang,Martin Radfar,Kai Wei,Clement Chung机构：Amazon Alexa AI备注：ICASSP 2023摘要：在口语理解中，由于文本信息的缺乏，直接从音频信号中提取语义是一个挑战。流行的端到端（E2E）SLU模型利用序列到序列自动语音识别（ASR）模型来提取文本嵌入作为输入以推断语义，然而，这需要计算上昂贵的自回归解码。在这项工作中，我们利用自监督声学编码器微调与连接主义时间分类（CTC）提取文本嵌入和使用联合CTC和SLU损失的话语级SLU任务。实验表明，我们的模型实现了4%的绝对改善的最先进的（SOTA）的DSTC2数据集上的对话行为分类模型和1.3%的绝对改善的SOTA SLU模型的SLURP数据集。摘要：It is challenging to extract semantic meanings directly from audio signals in spoken language understanding (SLU), due to the lack of textual information. Popular end-to-end (E2E) SLU models utilize sequence-to-sequence automatic speech recognition (ASR) models to extract textual embeddings as input to infer semantics, which, however, require computationally expensive auto-regressive decoding. In this work, we leverage self-supervised acoustic encoders fine-tuned with Connectionist Temporal Classification (CTC) to extract textual embeddings and use joint CTC and SLU losses for utterance-level SLU tasks. Experiments show that our model achieves 4% absolute improvement over the the state-of-the-art (SOTA) dialogue act classification model on the DSTC2 dataset and 1.3% absolute improvement over the SOTA SLU model on the SLURP dataset.
【4】 HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio  Codec标题：高保真音频编解码器：高保真音频编解码器的分组残差矢量量化链接：https://arxiv.org/abs/2305.02765
*与cs.SD语音【2】为同一篇
作者：Dongchao Yang,Songxiang Liu,Rongjie Huang,Jinchuan Tian,Chao Weng,Yuexian Zou机构：Peking University, China,  Tencent AI Lab, Shenzhen, China,  Zhejiang University, China备注：The first version of HiFi-Codec. arXiv admin note: text overlap with arXiv:2210.13438 by other authors摘要：音频编解码器模型作为用于将音频压缩成离散表示的关键技术而广泛用于音频通信中。如今，音频编解码器模型作为中间表示越来越多地用于生成领域。例如，AudioLM是使用SoundStream的离散表示作为训练目标的音频生成模型，而VALL-E采用Encodec模型作为中间特征来辅助TTS任务。尽管它们很有用，但仍然存在两个挑战：（1）训练这些音频编解码器模型可能是困难的，这是由于缺乏公共可用的训练过程以及对大规模数据和GPU的需要;（2）实现良好的重构性能需要许多码本，这增加了生成模型的负担。在这项研究中，我们提出了一种组残差矢量量化（GRVQ）技术，并使用它来开发一种新的 textbf{Hi}gh textbf{Fi}delity音频编解码器模型，HiFi-Codec，它只需要4个码本。我们使用LibriTTS，VCTK，AISHELL等公开可用的TTS数据来训练所有模型，总时长超过1000小时，使用8个GPU。我们的实验结果表明，HiFi-Codec优于Encodec的重建性能，尽管只需要4个码本。为了促进音频编解码器和生成的研究，我们介绍了AcademiCodec，这是第一个开源音频编解码器工具包，为Encodec，SoundStream和HiFi-Codec提供训练代码和预训练模型。代码和预训练模型可以在以下网站找到：href{https：github.com yangdongchao AcademiCodec}{https：github.com yangdongchao AcademiCodec}摘要：Audio codec models are widely used in audio communication as a crucial technique for compressing audio into discrete representations. Nowadays, audio codec models are increasingly utilized in generation fields as intermediate representations. For instance, AudioLM is an audio generation model that uses the discrete representation of SoundStream as a training target, while VALL-E employs the Encodec model as an intermediate feature to aid TTS tasks. Despite their usefulness, two challenges persist: (1) training these audio codec models can be difficult due to the lack of publicly available training processes and the need for large-scale data and GPUs; (2) achieving good reconstruction performance requires many codebooks, which increases the burden on generation models. In this study, we propose a group-residual vector quantization (GRVQ) technique and use it to develop a novel textbf{Hi}gh textbf{Fi}delity Audio Codec model, HiFi-Codec, which only requires 4 codebooks. We train all the models using publicly available TTS data such as LibriTTS, VCTK, AISHELL, and more, with a total duration of over 1000 hours, using 8 GPUs. Our experimental results show that HiFi-Codec outperforms Encodec in terms of reconstruction performance despite requiring only 4 codebooks. To facilitate research in audio codec and generation, we introduce AcademiCodec, the first open-source audio codec toolkit that offers training codes and pre-trained models for Encodec, SoundStream, and HiFi-Codec. Code and pre-trained model can be found on: href{https: github.com yangdongchao AcademiCodec}{https: github.com yangdongchao AcademiCodec}
【5】 Learning to Detect Novel and Fine-Grained Acoustic Sequences Using  Pretrained Audio Representations标题：使用预先训练的音频表示学习检测新的和细粒度的声音序列链接：https://arxiv.org/abs/2305.02382
*与cs.SD语音【3】为同一篇
作者：Vasudha Kowtha,Miquel Espi Marques,Jonathan Huang,Yichi Zhang,Carlos Avendano机构：Apple备注：IEEE ICASSP 2023摘要：这项工作研究了预训练的音频表示的Few Shot的声音事件检测。我们专门解决的任务，Few Shot检测新的声学序列，或声音事件的语义有意义的时间结构，而不假设访问非目标音频。我们开发了预训练合适的表示的程序，并将其转移到我们的Few Shot学习场景的方法。我们的实验评估了我们在AudioSet上的预训练表示的通用实用性，以及通过从现实世界的声学序列构建的任务提出的Few Shot方法的实用性。我们的预训练嵌入适用于所提出的任务，并实现了我们的Few Shot框架的多个方面。摘要：This work investigates pretrained audio representations for few shot Sound Event Detection. We specifically address the task of few shot detection of novel acoustic sequences, or sound events with semantically meaningful temporal structure, without assuming access to non-target audio. We develop procedures for pretraining suitable representations, and methods which transfer them to our few shot learning scenario. Our experiments evaluate the general purpose utility of our pretrained representations on AudioSet, and the utility of proposed few shot methods via tasks constructed from real-world acoustic sequences. Our pretrained embeddings are suitable to the proposed task, and enable multiple aspects of our few shot framework.
机器翻译，仅供参考