【1】 Efficient Similarity-based Passive Filter Pruning for Compressing CNNs标题：基于相似度的高效无源滤波剪枝压缩CNN链接：https://arxiv.org/abs/2210.17416
作者：Arshdeep Singh,Mark D. Plumbley机构：Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK备注：Submitted to ICASSP 2023摘要：卷积神经网络（CNNs）在各种应用中显示出巨大的成功。然而，CNNs的计算复杂度和存储器存储是其在资源受限设备上部署的瓶颈。近年来，人们致力于降低神经网络的计算成本和存储开销，包括基于相似性的被动滤波器修剪方法。基于相似度的被动滤波器剪枝方法计算滤波器的成对相似度矩阵，并剔除少数相似的滤波器以获得较小的剪枝CNN。然而，计算成对相似性矩阵的计算复杂度很高，特别是当卷积层具有许多滤波器时。为了降低计算复杂度，我们提出了一种有效的方法，即利用Nystr om近似方法，仅从几个列来近似完全的成对相似矩阵.与计算完全成对相似度矩阵的基于相似度的修剪方法相比，所提出的基于相似度的有效被动滤波器修剪方法在相同的计算量减少的情况下，速度提高了3倍，并给出了相同的精度。除此之外，所提出的基于相似度的剪枝方法的性能与现有的基于范数的剪枝方法相当或更好.在DCASE 2021 Task 1A baseline网络和一个用于声学场景分类的VGGish网络上对所提出的剪枝方法的有效性进行了评估。摘要：Convolution neural networks (CNNs) have shown great success in various applications. However, the computational complexity and memory storage of CNNs is a bottleneck for their deployment on resource-constrained devices. Recent efforts towards reducing the computation cost and the memory overhead of CNNs involve similarity-based passive filter pruning methods. Similarity-based passive filter pruning methods compute a pairwise similarity matrix for the filters and eliminate a few similar filters to obtain a small pruned CNN. However, the computational complexity of computing the pairwise similarity matrix is high, particularly when a convolutional layer has many filters. To reduce the computational complexity in obtaining the pairwise similarity matrix, we propose to use an efficient method where the complete pairwise similarity matrix is approximated from only a few of its columns by using a Nystr\"om approximation method. The proposed efficient similarity-based passive filter pruning method is 3 times faster and gives same accuracy at the same reduction in computations for CNNs compared to that of the similarity-based pruning method that computes a complete pairwise similarity matrix. Apart from this, the proposed efficient similarity-based pruning method performs similarly or better than the existing norm-based pruning methods. The efficacy of the proposed pruning method is evaluated on CNNs such as DCASE 2021 Task 1A baseline network and a VGGish network designed for acoustic scene classification.
【2】 Analysis and Detection of Singing Techniques in Repertoires of J-POP Solo Singers标题：J-POP独唱曲目演唱技巧的分析与检测链接：https://arxiv.org/abs/2210.17367
作者：Yuya Yamamoto,Juhan Nam,Hiroko Terasawa机构：Doctoral Program in Informatics, University of Tsukuba, Japan, Graduate School of Culture Technology, KAIST, South Korea备注：Accepted at ISMIR 2022摘要：本文主要对歌唱技法范围内的音乐信息检索进行研究。我们使用日本流行音乐歌曲（J-POP）中著名独唱歌手的真实世界录音来调查歌手如何使用演唱技巧。首先，我们建立了一个新的歌唱技巧数据集。该数据集包括168首商业J-POP歌曲，每首歌曲都使用各种演唱技术进行注释，并带有时间戳和音高轮廓。我们还提出了歌唱技巧的数据集上的描述性统计，以澄清什么和如何经常歌唱技巧出现。我们进一步探讨了使用先前提出的机器学习技术自动检测歌唱技巧的困难。在检测中，我们还研究了辅助信息（即，音调和标签持续时间的分布），而不仅仅是提供基线。在9路多类检测上，最佳的宏平均F测度达到40.4%。我们在附录网站0上提供了数据集的注释及其详细信息。摘要：In this paper, we focus on singing techniques within the scope of music information retrieval research. We investigate how singers use singing techniques using real-world recordings of famous solo singers in Japanese popular music songs (J-POP). First, we built a new dataset of singing techniques. The dataset consists of 168 commercial J-POP songs, and each song is annotated using various singing techniques with timestamps and vocal pitch contours. We also present descriptive statistics of singing techniques on the dataset to clarify what and how often singing techniques appear. We further explored the difficulty of the automatic detection of singing techniques using previously proposed machine learning techniques. In the detection, we also investigate the effectiveness of auxiliary information (i.e., pitch and distribution of label duration), not only providing the baseline. The best result achieves 40.4% at macro-average F-measure on nine-way multi-class detection. We provide the annotation of the dataset and its detail on the appendix website 0 .
【3】 Robust MelGAN: A robust universal neural vocoder for high-fidelity TTS标题：Robust Melgan：一种适用于高保真TTS的健壮通用神经声码器链接：https://arxiv.org/abs/2210.17349
作者：Kun Song,Jian Cong,Yongmao Zhang,Lei Xie,Ning Jiang,Haiying Wu机构：Northwestern Polytechnical University, Xi’an, China, Mashang Consumer Finance Co., Ltd.摘要：在当前的两阶段神经文本到语音（TTS）范式中，理想的是具有一个一旦训练好就对从声学模型预测的不完美Mel谱图具有鲁棒性的通用神经声码器。为此，本文提出了一种鲁棒MelGAN声码器，解决了原有多带MelGAN的金属音问题，提高了其泛化能力。具体地说，我们在发生器中引入了细粒度的网络丢弃策略。通过专门设计的过平滑处理器分离语音信号的周期性和非周期性分量，只对周期性分量进行网络丢弃，既减轻了金属音的影响，又保持了良好的说话人相似度。为了进一步提高泛化能力，在鉴别器中引入了几种数据增强方法来增强伪数据，包括谐波偏移、谐波噪声和相位噪声。实验表明，Robust MelGAN可以作为一种通用的声码器，显著提高基于各种类型数据的TTS系统的音质。摘要：In current two-stage neural text-to-speech (TTS) paradigm, it is ideal to have a universal neural vocoder, once trained, which is robust to imperfect mel-spectrogram predicted from the acoustic model. To this end, we propose Robust MelGAN vocoder by solving the original multi-band MelGAN's metallic sound problem and increasing its generalization ability. Specifically, we introduce a fine-grained network dropout strategy to the generator. With a specifically designed over-smooth handler which separates speech signal intro periodic and aperiodic components, we only perform network dropout to the aperodic components, which alleviates metallic sounding and maintains good speaker similarity. To further improve generalization ability, we introduce several data augmentation methods to augment fake data in the discriminator, including harmonic shift, harmonic noise and phase noise. Experiments show that Robust MelGAN can be used as a universal vocoder, significantly improving sound quality in TTS systems built on various types of data.
【4】 AccentSpeech: Learning Accent from Crowd-sourced Data for Target Speaker TTS with Accents标题：AccentSpeech：从众包数据中学习带有口音的目标说话人TTS的口音链接：https://arxiv.org/abs/2210.17305
作者：Yongmao Zhang,Zhichao Wang,Peiji Yang,Hongshen Sun,Zhisheng Wang,Lei Xie机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, Tencent, Shenzhen, China备注：Accepted by ISCSLP2022摘要：从众包数据中学习重音是实现目标语者TTS系统合成重音语音的可行方法。为此，有两个具有挑战性的问题需要解决。首先，在重音转移中直接使用声学质量较差的众包数据和目标说话人数据将明显导致合成语音质量下降。针对这一问题，本文提出了一种基于瓶颈特征（BN）的TTS方法，该方法将TTS分解为用于重音学习的T2 BN（Text-to-BN）模块和用于说话人音色学习的BN 2 Mel（BN 2 Mel）模块，并以神经网络的BN特征作为中间表示，对噪声干扰具有鲁棒性。第二，在两阶段系统中使用众包数据直接训练T2 BN会产生韵律较差的目标说话人重音语音。这是因为众包录音是由普通的非专业发言人贡献的。为了解决这个问题，我们将两阶段方法改进为一种新的三阶段方法，其中T2 BN和BN 2 Mel使用高质量的目标说话人数据进行训练，并且在两个模块之间插入一个新的BN-to-BN模块来执行重音迁移。为了训练BN 2BN模块，通过提出的数据扩充过程获得平行的非重音和重音BN特征。最后，由于该方法继承了专业目标说话人的韵律模式，同时通过BN 2BN模块实现重音迁移，因此该方法能够为目标说话人生成具有良好韵律的重音语音。该方法被命名为AccentSpeech，并在普通话TTS重音迁移任务中得到了验证。摘要：Learning accent from crowd-sourced data is a feasible way to achieve a target speaker TTS system that can synthesize accent speech. To this end, there are two challenging problems to be solved. First, direct use of the poor acoustic quality crowd-sourced data and the target speaker data in accent transfer will apparently lead to synthetic speech with degraded quality. To mitigate this problem, we take a bottleneck feature (BN) based TTS approach, in which TTS is decomposed into a Text-to-BN (T2BN) module to learn accent and a BN-to-Mel (BN2Mel) module to learn speaker timbre, where neural network based BN feature serves as the intermediate representation that are robust to noise interference. Second, direct training T2BN using the crowd-sourced data in the two-stage system will produce accent speech of target speaker with poor prosody. This is because the the crowd-sourced recordings are contributed from the ordinary unprofessional speakers. To tackle this problem, we update the two-stage approach to a novel three-stage approach, where T2BN and BN2Mel are trained using the high-quality target speaker data and a new BN-to-BN module is plugged in between the two modules to perform accent transfer. To train the BN2BN module, the parallel unaccented and accented BN features are obtained by a proposed data augmentation procedure. Finally the proposed three-stage approach manages to produce accent speech for the target speaker with good prosody, as the prosody pattern is inherited from the professional target speaker and accent transfer is achieved by the BN2BN module at the same time. The proposed approach, named as AccentSpeech, is validated in a Mandarin TTS accent transfer task.
【5】 Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation标题：基于流的语音转换的跨语言文本到语音转换，以改善发音链接：https://arxiv.org/abs/2210.17264
作者：Nikolaos Ellinas,Georgios Vamvoukakis,Konstantinos Markopoulos,Georgia Maniati,Panos Kakoulidis,June Sig Sung,Inchul Hwang,Spyros Raptis,Aimilios Chalamandaris,Pirros Tsiakoulis机构：Innoetics, Samsung Electronics, Greece, Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：本文提出了一种端到端跨语言的文语转换（TTS）方法，该方法的目标是在不考虑说话人语言的情况下，保留目标语言的发音。所使用的模型基于非注意Tacotron架构，其中解码器已被以说话者身份为条件的归一化流网络所取代，由于固有的语言内容和说话者身份的解纠缠，允许TTS和语音转换（VC）两者由同一模型执行。当在跨语言设置中使用时，声学特征最初由目标语言的母语说话者产生，然后由相同的模型应用话音转换，以便将这些特征转换成目标说话者的话音。我们通过客观和主观评估验证了我们的方法与基线跨语言合成相比具有优势。通过包括平均发言7.5分钟的发言者，我们也提出了低资源情况下的积极结果。摘要：This paper presents a method for end-to-end cross-lingual text-to-speech (TTS) which aims to preserve the target language's pronunciation regardless of the original speaker's language. The model used is based on a non-attentive Tacotron architecture, where the decoder has been replaced with a normalizing flow network conditioned on the speaker identity, allowing both TTS and voice conversion (VC) to be performed by the same model due to the inherent linguistic content and speaker identity disentanglement. When used in a cross-lingual setting, acoustic features are initially produced with a native speaker of the target language and then voice conversion is applied by the same model in order to convert these features to the target speaker's voice. We verify through objective and subjective evaluations that our method can have benefits compared to baseline cross-lingual synthesis. By including speakers averaging 7.5 minutes of speech, we also present positive results on low-resource scenarios.
【6】 Combining Automatic Speaker Verification and Prosody Analysis for Synthetic Speech Detection标题：自动说话人识别与韵律分析相结合的合成语音检测链接：https://arxiv.org/abs/2210.17222
作者：Luigi Attorresi,Davide Salvi,Clara Borrelli,Paolo Bestagini,Stefano Tubaro机构：Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, Italy摘要：媒体 内容 合成 技术 的 快速 传播 以及 音频 和 视频 深度 伪造 对 人们 生活 的 潜在 破坏 性 影响 已经 提出 了 实现 能够 自动 检测 这些 伪造 的 系统 的 需要 。在 本文 中 ， 我们 提出 了 一 种 新 的 合成 语音 检测 方法 ， 利用 人类 语音 的 两 个 高级 语义 属性 的 组合 。一方面 ， 我们 关注 说话 人 身份 线索 ， 并 将 其 表示 为 使用 最 先进 的 方法 提取 的 说话 人 嵌入 ， 用于 自动 说话 人 确认 任务 。另 一方面 ， 语音 韵律 ， 意 为 语音 中 节奏 、 音高 或 重音 的 变化 ， 通过 专门 的 编码 器 提取 。我们 证明 了 这 两 种 嵌入 的 组合 馈送 到 监督 二 进制 分类 器 允许 检测 由 文本 到 语音 和 语音 转换 技术 生成 的 deepfake 语音 。我们 的 结果 显示 了 相对 于 所 考虑 的 基线 的 改进 、 在 多 个 数据 集 上 的 良好 泛化 特性 以及 对 音频 压缩 的 鲁 棒 性 。摘要：The rapid spread of media content synthesis technology and the potentially damaging impact of audio and video deepfakes on people's lives have raised the need to implement systems able to detect these forgeries automatically. In this work we present a novel approach for synthetic speech detection, exploiting the combination of two high-level semantic properties of the human voice. On one side, we focus on speaker identity cues and represent them as speaker embeddings extracted using a state-of-the-art method for the automatic speaker verification task. On the other side, voice prosody, intended as variations in rhythm, pitch or accent in speech, is extracted through a specialized encoder. We show that the combination of these two embeddings fed to a supervised binary classifier allows the detection of deepfake speech generated with both Text-to-Speech and Voice Conversion techniques. Our results show improvements over the considered baselines, good generalization properties over multiple datasets and robustness to audio compression.
【7】 Self-Supervised Hierarchical Metrical Structure Modeling标题：自监督层次度量结构建模链接：https://arxiv.org/abs/2210.17183
作者：Junyan Jiang,Gus Xia机构：Music X Lab, NYU Shanghai, Mohamed bin Zayed University of Artificial Intelligence摘要：我们提出一种新的方法，以最小领域知识的自监督方式为符号音乐和音频信号建模分层韵律结构。该模型对节拍对齐的音乐信号进行训练和推理，并预测出一个从节拍、小节到小节的8层韵律树。训练过程不需要任何等级韵律标记，除了节拍，纯粹依赖于韵律规则性和语音间一致性的性质作为归纳偏差。实验表明，该方法在符号音乐和音频信号的多韵律结构分析任务中取得了与监督基线相当的性能。所有的演示、源代码和预先训练的模型都可以在GitHub上公开获得。摘要：We propose a novel method to model hierarchical metrical structures for both symbolic music and audio signals in a self-supervised manner with minimal domain knowledge. The model trains and inferences on beat-aligned music signals and predicts an 8-layer hierarchical metrical tree from beat, measure to the section level. The training procedural does not require any hierarchical metrical labeling except for beats, purely relying on the nature of metrical regularity and inter-voice consistency as inductive biases. We show in experiments that the method achieves comparable performance with supervised baselines on multiple metrical structure analysis tasks on both symbolic music and audio signals. All demos, source code and pre-trained models are publicly available on GitHub.
【8】 Audio Time-Scale Modification with Temporal Compressing Networks标题：基于时间压缩网络的音频时间尺度修正链接：https://arxiv.org/abs/2210.17152
作者：Ernie Chu,Ju-Ting Chen,Chia-Ping Chen机构：Research Center for Information Technology Innovation, Academia Sinica, National Cheng Kung University, National Sun Yat-sen University摘要：提出了一种新的音频信号时间尺度修正方法。传统方法采用分帧技术，而谱方法采用短时傅里叶变换，在时间拉伸过程中保持频率不变。TSM-Net，我们的神经网络模型将原始音频编码为高级潜在表示。我们称之为Neuralgram，其中一个向量代表1024个音频样本。它的灵感来自于帧技术，但解决了剪切伪像。神经图是具有实值的二维矩阵，我们可以将一些现有的图像缩放技术应用于神经图，并使用我们的神经解码器对其进行解码，以获得时间缩放的音频。编码器和解码器都是用GANs训练的，它对缩放后的神经图表现出良好的泛化能力。该方法产生的伪影少，为现代时间尺度修正研究开辟了新的可能性。音频样本可在www.example.com上找到https://ernestchu.github.io/tsm-net-demo/摘要：We proposed a novel approach in the field of time-scale modification on audio signals. While traditional methods use the framing technique, spectral approach uses the short-time Fourier transform to preserve the frequency during temporal stretching. TSM-Net, our neural-network model encodes the raw audio into a high-level latent representation. We call it Neuralgram, in which one vector represents 1024 audio samples. It is inspired by the framing technique but addresses the clipping artifacts. The Neuralgram is a two-dimensional matrix with real values, we can apply some existing image resizing techniques on the Neuralgram and decode it using our neural decoder to obtain the time-scaled audio. Both the encoder and decoder are trained with GANs, which shows fair generalization ability on the scaled Neuralgrams. Our method yields little artifacts and opens a new possibility in the research of modern time-scale modification. The audio samples can be found on https://ernestchu.github.io/tsm-net-demo/
【9】 Improving Audio-Language Learning with MixGen and Multi-Level Test-Time Augmentation标题：用MixGen和多级测试时间延长改善听力学习链接：https://arxiv.org/abs/2210.17143
作者：Eungbeom Kim,Jinhee Kim,Yoori Oh,Kyungsu Kim,Minju Park,Jaeheon Sim,Jinwoo Lee,Kyogu Lee机构：Interdisciplinary Program in Artificial Intelligence, Seoul National University, Department of Intelligence and Information, Seoul National University, Artificial Intelligence Institute, Seoul National University备注：Submitted to ICASSP 2023摘要：本文提出了两种新的增强方法：1）有声语言混合生成（AL-MixGen）和2）多水平测试时间增强（Multi-TTA）。受最初应用于视觉语言学习的MixGen的启发，我们引入了一种用于听觉语言领域的增强方法。我们还探讨了测试时间扩展的影响，并提出了Multi-TTA，它将测试时间扩展推广到深度学习模型的多个层次。将AL-MixGen和Multi-TTA整合到基线中，音频字幕的SPIDEr达到47.5，比基线高出18.2%，比最先进的方法（模型缩小5倍）性能更佳。在音频文本检索中，所提方法的性能也超过了基线.摘要：In this paper, we propose two novel augmentation methods 1) audio-language MixGen (AL-MixGen) and 2) multi-level test-time augmentation (Multi-TTA) for audio-language learning. Inspired by MixGen, which is originally applied to vision-language learning, we introduce an augmentation method for the audio-language domain. We also explore the impact of test-time augmentations and present Multi-TTA which generalizes test-time augmentation over multiple layers of a deep learning model. Incorporating AL-MixGen and Multi-TTA into the baseline achieves 47.5 SPIDEr on audio captioning, which is an +18.2% over the baseline and outperforms the state-of-the-art approach with a 5x smaller model. In audio-text retrieval, the proposed methods surpass the baseline performance as well.
【10】 Structured State Space Decoder for Speech Recognition and Synthesis标题：用于语音识别与合成的结构化状态空间解码器链接：https://arxiv.org/abs/2210.17098
作者：Koichi Miyazaki,Masato Murata,Tomoki Koriyama机构：CyberAgent, Inc.备注：Submitted to ICASSP 2023摘要：近年来开发的自动语音识别（ASR）系统已经显示出具有自注意力模型（例如，Transformer和Conformer），它们正在取代传统的递归神经网络。同时，最近已经提出了一种结构化状态空间模型（S4），其对于各种长序列建模任务产生了有希望的结果，包括原始语音分类。S4模型可以并行训练，与Transformer模型相同。在本研究中，我们将S4作为ASR和文本到语音（TTS）任务的解码器，并与Transformer解码器进行比较。实验结果表明，该模型在LibriSpeech测试-清洁/测试-其他集上的竞争词错误率（WER）为1.88%/4.25%，在CSJ评价1/评价2/评价3集上的字符错误率（CER）为3.80%/2.63%/2.98%.此外，与标准Transformer模型相比，该模型具有更好的鲁棒性，尤其是在两个数据集上的长格式语音。对于TTS任务，所提出的方法优于Transformer基线。摘要：Automatic speech recognition (ASR) systems developed in recent years have shown promising results with self-attention models (e.g., Transformer and Conformer), which are replacing conventional recurrent neural networks. Meanwhile, a structured state space model (S4) has been recently proposed, producing promising results for various long-sequence modeling tasks, including raw speech classification. The S4 model can be trained in parallel, same as the Transformer model. In this study, we applied S4 as a decoder for ASR and text-to-speech (TTS) tasks by comparing it with the Transformer decoder. For the ASR task, our experimental results demonstrate that the proposed model achieves a competitive word error rate (WER) of 1.88%/4.25% on LibriSpeech test-clean/test-other set and a character error rate (CER) of 3.80%/2.63%/2.98% on the CSJ eval1/eval2/eval3 set. Furthermore, the proposed model is more robust than the standard Transformer model, particularly for long-form speech on both the datasets. For the TTS task, the proposed method outperforms the Transformer baseline.
【11】 FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition标题：FusionFormer：在Transformer中融合操作以实现高效的流语音识别链接：https://arxiv.org/abs/2210.17079
作者：Xingchen Song,Di Wu,Binbin Zhang,Zhiyong Wu,Wenpeng Li,Dongfang Li,Pengshen Zhang,Zhendong Peng,Fuping Pan,Changbao Zhu,Zhongqin Wu机构：Tsinghua University, Beijing, China , Horizon Robtics, Beijing, China, WeNet Open Source Community备注：8 pages, plus 3 appendix摘要：最近提出的Conformer体系结构将卷积与注意力相结合，以捕获局部和全局依赖性，该体系结构已成为自动语音识别（ASR）的\textit{de facto}主干模型。该体系结构继承了自然语言处理（NLP）任务，采用层规范化（LN）作为默认的规范化技术。然而，通过一系列系统的研究发现，LN虽然只贡献了0.1%的FLOP，但却可能占用了10%的推理时间.这促使我们用其他归一化技术来代替LN，例如：批归一化（BN），利用算子融合方法加快推理速度，避免了在推理过程中计算均值和方差统计量。在检查了直接去除所有LN层或在相同位置用BN代替它们的几种简单尝试之后，我们发现发散问题主要是由不稳定的层输出引起的。因此，我们提出将BN层附加到每个线性或卷积层，其中观察到稳定的训练结果。我们还建议简化Conformer中的激活，如Swish和GLU，通过用ReLU代替它们。所有这些被交换的模块可以被融合到相邻线性/卷积层的权重中，因此具有零推理成本。因此，我们将其命名为FusionFormer。实验表明，FusionFormer与基于LN的Conformer一样有效，并且速度快约10%。摘要：The recently proposed Conformer architecture which combines convolution with attention to capture both local and global dependencies has become the \textit{de facto} backbone model for Automatic Speech Recognition~(ASR). Inherited from the Natural Language Processing (NLP) tasks, the architecture takes Layer Normalization~(LN) as a default normalization technique. However, through a series of systematic studies, we find that LN might take 10\% of the inference time despite that it only contributes to 0.1\% of the FLOPs. This motivates us to replace LN with other normalization techniques, e.g., Batch Normalization~(BN), to speed up inference with the help of operator fusion methods and the avoidance of calculating the mean and variance statistics during inference. After examining several plain attempts which directly remove all LN layers or replace them with BN in the same place, we find that the divergence issue is mainly caused by the unstable layer output. We therefore propose to append a BN layer to each linear or convolution layer where stabilized training results are observed. We also propose to simplify the activations in Conformer, such as Swish and GLU, by replacing them with ReLU. All these exchanged modules can be fused into the weights of the adjacent linear/convolution layers and hence have zero inference cost. Therefore, we name it FusionFormer. Our experiments indicate that FusionFormer is as effective as the LN-based Conformer and is about 10\% faster.
【12】 Modular Hybrid Autoregressive Transducer标题：模块化混合式自回归传感器链接：https://arxiv.org/abs/2210.17049
作者：Zhong Meng,Tongzhou Chen,Rohit Prabhavalkar,Yu Zhang,Gary Wang,Kartik Audhkhasi,Jesse Emond,Trevor Strohman,Bhuvana Ramabhadran,W. Ronny Huang,Ehsan Variani,Yinghui Huang,Pedro J. Moreno机构：Google LLC., USA备注：None摘要：对于端到端语音识别来说，换能器模型的纯文本自适应仍然是挑战性的，因为换能器不具有清楚分离的声学模型（AM）、语言模型（LM）或空白模型。在这项工作中，我们提出了一个模块化混合自回归换能器（MHAT），它具有结构上分离的标签和空白解码器，分别预测标签和空白分布，以及共享的声学编码器。编码器和标签解码器的输出被直接投影到AM和内部LM得分，然后被相加以计算标签后验。我们用内部LM丢失和HAT丢失来训练MHAT，以确保其内部LM成为能够有效适应文本的独立神经LM。此外，MHAT的文本适应促进了比基于内部LM减法的方法更好的LM融合。在Google的大规模生产数据上，根据400K小时训练的HAT，采用100B句子的多域MHAT在没有LM融合的情况下实现了高达12.4%的相对WER减少，而在有LM融合的情况下实现了21.5%的相对WER减少。摘要：Text-only adaptation of a transducer model remains challenging for end-to-end speech recognition since the transducer has no clearly separated acoustic model (AM), language model (LM) or blank model. In this work, we propose a modular hybrid autoregressive transducer (MHAT) that has structurally separated label and blank decoders to predict label and blank distributions, respectively, along with a shared acoustic encoder. The encoder and label decoder outputs are directly projected to AM and internal LM scores and then added to compute label posteriors. We train MHAT with an internal LM loss and a HAT loss to ensure that its internal LM becomes a standalone neural LM that can be effectively adapted to text. Moreover, text adaptation of MHAT fosters a much better LM fusion than internal LM subtraction-based methods. On Google's large-scale production data, a multi-domain MHAT adapted with 100B sentences achieves relative WER reductions of up to 12.4% without LM fusion and 21.5% with LM fusion from 400K-hour trained HAT.
【13】 Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation标题：结合语音和双语文本的直接语音到语音翻译的联合预训练链接：https://arxiv.org/abs/2210.17027
作者：Kun Wei,Long Zhou,Ziqiang Zhang,Liping Chen,Shujie Liu,Lei He,Jinyu Li,Furu Wei机构：Northwestern Polytechnical University, Xian, China, Microsoft Corporation备注：Submitted to ICASSP 2023摘要：直接语音到语音翻译（S2 ST）与级联S2 ST相比具有许多优点，是一个很有吸引力的研究课题。然而，直接S2 ST面临着数据稀缺的问题，因为从源语言到目标语言的语料库非常稀少。针对这一问题，本文提出了一种Speech 2S模型，该模型采用非成对语音和双语文本数据进行联合预训练，用于直接语音到语音的翻译任务。通过有效地利用成对的文本数据，Speech 2S能够对从源语言到目标语言的跨语言语音转换进行建模。我们在Europarl-ST和VoxPopuli数据集上验证了Speech 2S的性能。实验结果表明，与仅使用编码器的预训练模型相比，Speech 2S获得了约5个BLEU分数的改进，并且实现了与现有最先进模型相比具有竞争力甚至更好的性能1。摘要：Direct speech-to-speech translation (S2ST) is an attractive research topic with many advantages compared to cascaded S2ST. However, direct S2ST suffers from the data scarcity problem because the corpora from speech of the source language to speech of the target language are very rare. To address this issue, we propose in this paper a Speech2S model, which is jointly pre-trained with unpaired speech and bilingual text data for direct speech-to-speech translation tasks. By effectively leveraging the paired text data, Speech2S is capable of modeling the cross-lingual speech conversion from source to target language. We verify the performance of the proposed Speech2S on Europarl-ST and VoxPopuli datasets. Experimental results demonstrate that Speech2S gets an improvement of about 5 BLEU scores compared to encoder-only pre-training models, and achieves a competitive or even better performance than existing state-of-the-art models1.
【14】 Blank Collapse: Compressing CTC emission for the faster decoding标题：空白折叠：压缩CTC发射以实现更快的解码链接：https://arxiv.org/abs/2210.17017
作者：Minkyu Jung,Ohhyeok Kwon,Seunghyun Seo,Soonshin Seo机构：NAVER Corporation, South Korea备注：Submitted to ICASSP 2023摘要：连接主义时态分类（CTC）模型是一种非常有效的序列建模方法，特别是对语音数据。为了使用CTC模型作为自动语音识别（ASR）任务，需要使用外部语言模型（如n元LM）进行波束搜索解码以获得合理的结果。本文对CTC波束搜索中的空白标签进行了深入的分析，提出了一种非常简单的方法来减少计算量，从而加快波束搜索的解码速度。在LibriSpeech数据集上，与普通波束搜索解码相比，该方法在解码精度损失很小的情况下，解码速度提高了78%。通过实验和数学推导，从理论上证明了该方法的有效性。我们还观察到，如果模型的精度更高，这种减少更明显。摘要：Connectionist Temporal Classification (CTC) model is a very efficient method for modeling sequences, especially for speech data. In order to use CTC model as an Automatic Speech Recognition (ASR) task, the beam search decoding with an external language model like n-gram LM is necessary to obtain reasonable results. In this paper we analyze the blank label in CTC beam search deeply and propose a very simple method to reduce the amount of calculation resulting in faster beam search decoding speed. With this method, we can get up to 78% faster decoding speed than ordinary beam search decoding with a very small loss of accuracy in LibriSpeech datasets. We prove this method is effective not only practically by experiments but also theoretically by mathematical reasoning. We also observe that this reduction is more obvious if the accuracy of the model is higher.
【15】 Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit标题：Wespeaker：面向研发和生产的扬声器嵌入式学习工具包链接：https://arxiv.org/abs/2210.17016
作者：Hongji Wang,Chengdong Liang,Shuai Wang,Zhengyang Chen,Binbin Zhang,Xu Xiang,Yanlei Deng,Yanmin Qian机构：X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China, Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China, Horizon Robotics, Beijing, China,AISpeech Ltd, Suzhou, China,NVIDIA, Santa Clara, USA, WeNet Open Source Community摘要：说话人建模是说话人识别、说话人日记化等许多相关任务的基础。主要的建模方法是固定维向量表示，即，扬声器嵌入。本文介绍了一个面向研究和生产的说话人嵌入式学习工具包Wespeaker。Wespeaker包含可扩展数据管理、最先进的说话人嵌入模型、损失函数和评分后端的实现，通过结构化配方实现了极具竞争力的结果，这些配方在几个说话人验证挑战中的获胜系统中被采用。其他下游任务的应用，如扬声器diarization，也在相关的食谱中展示。此外，还集成了CPU和GPU兼容的部署代码，以便进行面向生产的开发。该工具包可在www.example.com上公开获得https://github.com/wenet-e2e/wespeaker。摘要：Speaker modeling is essential for many related tasks, such as speaker recognition and speaker diarization. The dominant modeling approach is fixed-dimensional vector representation, i.e., speaker embedding. This paper introduces a research and production oriented speaker embedding learning toolkit, Wespeaker. Wespeaker contains the implementation of scalable data management, state-of-the-art speaker embedding models, loss functions, and scoring back-ends, with highly competitive results achieved by structured recipes which were adopted in the winning systems in several speaker verification challenges. The application to other downstream tasks such as speaker diarization is also exhibited in the related recipe. Moreover, CPU- and GPU-compatible deployment codes are integrated for production-oriented development. The toolkit is publicly available at https://github.com/wenet-e2e/wespeaker.
【16】 Synthesizer Preset Interpolation using Transformer Auto-Encoders标题：使用Transformer自动编码器的合成器预置内插链接：https://arxiv.org/abs/2210.16984
作者：Gwendal Le Vaillant,Thierry Dutoit机构：Information, Signal and Artificial Intelligence (ISIA), University of Mons, Belgium, HE,B-ISIB Research Institute, Brussels, Belgium摘要：声音合成器在现代音乐制作中广泛使用，但越来越需要掌握专业技能。这项工作集中在预置之间的插值，即，所有声音合成参数的值集，以便能够从现有声音直观地创建新声音。 我们介绍了一个双模态自动编码器神经网络，它同时处理预置使用多头注意块，和音频使用卷积。该模型已在一个具有一百多个参数的常用调频合成器上进行了测试。实验将该模型与相关体系结构和方法进行了比较，并证明了它执行了更平滑的插值。在训练之后，可以将所提出的模型集成到商业合成器中，用于现场插值或声音设计任务。摘要：Sound synthesizers are widespread in modern music production but they increasingly require expert skills to be mastered. This work focuses on interpolation between presets, i.e., sets of values of all sound synthesis parameters, to enable the intuitive creation of new sounds from existing ones. We introduce a bimodal auto-encoder neural network, which simultaneously processes presets using multi-head attention blocks, and audio using convolutions. This model has been tested on a popular frequency modulation synthesizer with more than one hundred parameters. Experiments have compared the model to related architectures and methods, and have demonstrated that it performs smoother interpolations. After training, the proposed model can be integrated into commercial synthesizers for live interpolation or sound design tasks.
【17】 TT-Net: Dual-path transformer based sound field translation in the spherical harmonic domain标题：TT-Net：基于双径Transformer的球谐波域声场转换链接：https://arxiv.org/abs/2210.16849
作者：Yiwen Wang,Zijian Lan,Xihong Wu,Tianshu Qu机构：Key Laboratory on Machine Perception (Ministry of Education), School of Intelligence Science and Technology, Peking University, Beijing, China备注：Submitted to ICASSP 2023摘要：在现有的基于球谐分析的声场平移方法中，基于加法定理的求解方法通常面临矩阵条件数过大而导致的奇异值问题。球面径向函数的不同距离和频率对平移矩阵稳定性的影响将影响选定点处SH系数的精度。针对上述问题，本文提出了一种基于双路变换器的神经网络方案。更具体地说，双路径网络是由自注意模块沿着频率和阶次轴两个维度构造的。网络中引入了变换-平均-级联层和升阶层，为多采样点和升阶提供了解决方案。数值仿真结果表明，该方法不仅拓宽了系统的工作频率范围，而且拓宽了系统的平移距离范围。利用所提出的双路径网络可以获得更精确的高阶SH系数。摘要：In the current method for the sound field translation tasks based on spherical harmonic (SH) analysis, the solution based on the additive theorem usually faces the problem of singular values caused by large matrix condition numbers. The influence of different distances and frequencies of the spherical radial function on the stability of the translation matrix will affect the accuracy of the SH coefficients at the selected point. Due to the problems mentioned above, we propose a neural network scheme based on the dual-path transformer. More specifically, the dual-path network is constructed by the self-attention module along the two dimensions of the frequency and order axes. The transform-average-concatenate layer and upscaling layer are introduced in the network, which provides solutions for multiple sampling points and upscaling. Numerical simulation results indicate that both the working frequency range and the distance range of the translation are extended. More accurate higher-order SH coefficients are obtained with the proposed dual-path network.
【18】 SRTNet: Time Domain Speech Enhancement Via Stochastic Refinement标题：SRTNet：基于随机求精的时域语音增强链接：https://arxiv.org/abs/2210.16805
作者：Zhibin Qiu,Mengfan Fu,Yinfeng Yu,LiLi Yin,Fuchun Sun,Hao Huang机构：School of Information Science and Engineering, Xinjiang University, China, Department of Computer Science and Technology, Tsinghua University, China摘要：扩散模型作为一种新的生成模型，在图像生成和音频合成中得到了广泛的应用，但在语音增强中却很少得到应用。在本文中，我们使用扩散模型作为随机精化的一个模块。本文提出了一种新的基于全时域随机求精的语音增强方法SRTNet。具体地说，我们设计了一个由确定性模块和随机性模块组成的联合网络，它构成了"增强-精炼“范式。从理论上证明了该方法的可行性，实验证明该方法具有训练速度快、采样速度快和质量高等优点。您可以在www.example.com取得我们的程式码和增强的范例https://github.com/zhibinQiu/SRTNet.git。摘要：Diffusion model, as a new generative model which is very popular in image generation and audio synthesis, is rarely used in speech enhancement. In this paper, we use the diffusion model as a module for stochastic refinement. We propose SRTNet, a novel method for speech enhancement via Stochastic Refinement in complete Time domain. Specifically, we design a joint network consisting of a deterministic module and a stochastic module, which makes up the ``enhance-and-refine'' paradigm. We theoretically demonstrate the feasibility of our method and experimentally prove that our method achieves faster training, faster sampling and higher quality. Our code and enhanced samples are available at https://github.com/zhibinQiu/SRTNet.git.
【19】 Adaptive Speech Quality Aware Complex Neural Network for Acoustic Echo Cancellation with Supervised Contrastive Learning标题：基于有监督对比学习的自适应语音质量感知复合神经网络回声抵消链接：https://arxiv.org/abs/2210.16791
作者：Bozhong Liu,Xiaoxi Yu,Hantao Huang机构：MediaTek Singapore摘要：声学回声消除（AEC）设计用于从麦克风信号中消除回声、混响和不需要的附加声音，同时保持近端说话者的语音质量。本文提出了一种自适应语音质量复合神经网络，以专注于实时声学回声消除的特定任务。具体地说，我们提出了一个复杂的模块化神经网络，它有不同的阶段来关注，特征提取，声学分离和掩模优化。此外，我们采用对比学习架构和新颖的语音质量感知损失函数，以进一步改善性能。该模型的预训练时间为72小时，然后进行72小时的微调。所提出的模型优于现有的性能。摘要：Acoustic echo cancellation (AEC) is designed to remove echoes, reverberation, and unwanted added sounds from the microphone signal while maintaining the quality of the near-end speaker's speech. This paper proposes adaptive speech quality complex neural networks to focus on specific tasks for real-time acoustic echo cancellation. In specific, we propose a complex modularize neural network with different stages to focus on, feature extraction acoustic separation, and mask optimization receptively. Furthermore, we adopt the contrastive learning framework and novel speech quality aware loss functions to further improve the performance. The model is trained with 72 hours for pre-training and then 72 hours for fine-tuning. The proposed model outperforms the state-of-the-art performance.
【20】 Symmetric Saliency-based Adversarial Attack To Speaker Identification标题：基于对称显著性的说话人识别对抗性攻击链接：https://arxiv.org/abs/2210.16777
作者：Jiadi Yao,Xing Chen,Xiao-Lei Zhang,Wei-Qiang Zhang,Kunde Yang摘要：目前，对抗性攻击的说话人识别方法要么计算量大，要么效率不高。针对这一问题，本文提出了一种基于生成网络的对称显著性编解码方法（SSED），用于生成对抗性语音样本进行说话人识别。它包含两种新的组分。首先，利用一种新颖的显著图解码器学习语音样本对目标说话人识别系统决策的重要性，从而使攻击者集中精力对重要样本产生人工噪声。同时提出了一种角损耗函数，使嵌入说话人远离源说话人。实验结果表明，SSED算法在开放集和封闭集说话人识别任务中均具有97%以上的目标攻击成功率和39 dB以上的信噪比，且计算量小.摘要：Adversarial attack approaches to speaker identification either need high computational cost or are not very effective, to our knowledge. To address this issue, in this paper, we propose a novel generation-network-based approach, called symmetric saliency-based encoder-decoder (SSED), to generate adversarial voice examples to speaker identification. It contains two novel components. First, it uses a novel saliency map decoder to learn the importance of speech samples to the decision of a targeted speaker identification system, so as to make the attacker focus on generating artificial noise to the important samples. It also proposes an angular loss function to push the speaker embedding far away from the source speaker. Our experimental results demonstrate that the proposed SSED yields the state-of-the-art performance, i.e. over 97% targeted attack success rate and a signal-to-noise level of over 39 dB on both the open-set and close-set speaker identification tasks, with a low computational cost.
【21】 token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text标题：Token2vec：一种基于非配对语音和文本的联合自监督预训练框架链接：https://arxiv.org/abs/2210.16755
作者：Xianghu Yue,Junyi Ao,Xiaoxue Gao,Haizhou Li机构：Department of Electrical and Computer Engineering, National University of Singapore, Singapore, School of Data Science, The Chinese University of Hong Kong, Shenzhen, China备注：Submitted to ICASSP 2023摘要：自监督预训练在文本和语音处理中都取得了成功。语音和文本提供不同但互补的信息。问题是我们是否能够对不成对的语音和文本执行语音-文本联合预训练。本文将自监督预训练的思想进一步发展，提出了一种新的基于语音离散表示的非成对语音和文本联合预训练框架--token 2 vec。首先，由于语音和文本模态的不同特性，即语音是连续的，文本是离散的，我们首先将语音离散化为一系列离散的语音标记来解决模态失配问题。其次，为了解决语音序列比文本序列长得多的长度失配问题，我们将文本中的单词转换为音素序列，并随机重复序列中的每个音素。最后，我们将离散的语音和文本标记输入到模态不可知的Transformer编码器中，并使用标记级掩蔽语言建模（tMLM）进行预训练。实验结果表明，token 2 vec算法的WER相对降低了17.7%，明显优于各种纯语音预训练基线。Token 2 vec模型也在非ASR任务上进行了验证，即，口语意图分类，并表现出良好的可迁移性。摘要：Self-supervised pre-training has been successful in both text and speech processing. Speech and text offer different but complementary information. The question is whether we are able to perform a speech-text joint pre-training on unpaired speech and text. In this paper, we take the idea of self-supervised pre-training one step further and propose token2vec, a novel joint pre-training framework for unpaired speech and text based on discrete representations of speech. Firstly, due to the distinct characteristics between speech and text modalities, where speech is continuous while text is discrete, we first discretize speech into a sequence of discrete speech tokens to solve the modality mismatch problem. Secondly, to solve the length mismatch problem, where the speech sequence is usually much longer than text sequence, we convert the words of text into phoneme sequences and randomly repeat each phoneme in the sequences. Finally, we feed the discrete speech and text tokens into a modality-agnostic Transformer encoder and pre-train with token-level masking language modeling (tMLM). Experiments show that token2vec is significantly superior to various speech-only pre-training baselines, with up to 17.7% relative WER reduction. Token2vec model is also validated on a non-ASR task, i.e., spoken intent classification, and shows good transferability.
【22】 XNOR-FORMER: Learning Accurate Approximations in Long Speech Transformers标题：XNOR-Form：在长语音转换器中学习精确的近似链接：https://arxiv.org/abs/2210.16643
作者：Roshan Sharma,Bhiksha Raj机构：Electrical and Computer Engineering,Language Technologies Institute, Carnegie Mellon University备注：Under review at ICASSP 2023摘要：变换器是用于语音、视觉和自然语言处理等中的许多任务的最新技术。自我注意力是该性能的关键贡献者，其具有二次方计算复杂度，这使得对较长输入序列的训练具有挑战性。先前的工作已经产生了具有线性注意力的最先进的变压器变体，然而，当前的模型牺牲了性能来实现高效的实施。在本研究中，我们借由检视自我注意中的键-查询乘积性质，发展出一种新颖的线性转换器。我们的模型在语音识别和语音摘要方面优于最先进的方法，导致在Librispeech-100语音识别基准和新的INTERVIEW语音识别基准上1%的绝对WER改善，以及在使用How2进行摘要的ROUGE上5分。摘要：Transformers are among the state of the art for many tasks in speech, vision, and natural language processing, among others. Self-attentions, which are crucial contributors to this performance have quadratic computational complexity, which makes training on longer input sequences challenging. Prior work has produced state-of-the-art transformer variants with linear attention, however, current models sacrifice performance to achieve efficient implementations. In this work, we develop a novel linear transformer by examining the properties of the key-query product within self-attentions. Our model outperforms state of the art approaches on speech recognition and speech summarization, resulting in 1 % absolute WER improvement on the Librispeech-100 speech recognition benchmark and a new INTERVIEW speech recognition benchmark, and 5 points on ROUGE for summarization with How2.
【23】 Unifying the Discrete and Continuous Emotion labels for Speech Emotion Recognition标题：语音情感识别中离散情感标签与连续情感标签的统一链接：https://arxiv.org/abs/2210.16642
作者：Roshan Sharma,Hira Dhamyal,Bhiksha Raj,Rita Singh机构：Electrical and Computer Engineering,Language Technologies Institute, Carnegie Mellon University备注：Under Review at ICASSP 2023摘要：传统上，在用于从语音中检测情感的副语言分析中，已经用离散的或维度的（连续值的）标签来识别情感。因此，已经提出的用于情感检测的模型使用这些标签类型中的一个或另一个。然而，像罗素和普卢奇克这样的心理学家提出了统一这些观点的理论和模型，认为这些表征具有共享和互补的信息。本文试图通过计算验证这些观点。为此，我们提出了一个模型来联合预测连续和离散情绪属性，并展示了如何利用这些属性之间的关系来提高情绪识别任务的鲁棒性和性能。我们的方法包括多任务和分层多任务学习框架，它们联合建模连续值情感标签和离散情感标签之间的关系。在IEMOCAP和MSPPodcast两个常用的语音情感识别数据集上的实验结果表明，与非统一的强基线方法相比，该模型在统计上有显著的性能改善.我们还证明，使用一种类型的标签（离散或连续值）进行训练，可以提高使用另一种类型标签的任务的识别性能。实验结果和推理的这种方法（称为失配训练方法）也给出了。摘要：Traditionally, in paralinguistic analysis for emotion detection from speech, emotions have been identified with discrete or dimensional (continuous-valued) labels. Accordingly, models that have been proposed for emotion detection use one or the other of these label types. However, psychologists like Russell and Plutchik have proposed theories and models that unite these views, maintaining that these representations have shared and complementary information. This paper is an attempt to validate these viewpoints computationally. To this end, we propose a model to jointly predict continuous and discrete emotional attributes and show how the relationship between these can be utilized to improve the robustness and performance of emotion recognition tasks. Our approach comprises multi-task and hierarchical multi-task learning frameworks that jointly model the relationships between continuous-valued and discrete emotion labels. Experimental results on two widely used datasets (IEMOCAP and MSPPodcast) for speech-based emotion recognition show that our model results in statistically significant improvements in performance over strong baselines with non-unified approaches. We also demonstrate that using one type of label (discrete or continuous-valued) for training improves recognition performance in tasks that use the other type of label. Experimental results and reasoning for this approach (called the mismatched training approach) are also presented.
【24】 Relating Human Perception of Musicality to Prediction in a Predictive Coding Model标题：在预测编码模型中将人类对音乐性的感知与预测联系起来链接：https://arxiv.org/abs/2210.16587
作者：Nikolas McNeal,Jennifer Huang,Aniekan Umoren,Shuqi Dai,Roger Dannenberg,Richard Randall,Tai Sing Lee机构：The Ohio State University, Carnegie Mellon University, Massachussetts Institute of Technology备注：5 pages, 5 figures, currently in peer review摘要：我们探索使用一个神经网络的启发预测编码建模人类音乐感知。这个网络是基于计算神经科学理论发展起来的，该理论认为视觉皮层的层级结构中存在着反复的交互作用。当使用视频数据使用自监督学习进行训练时，该模型表现出与人类视觉错觉一致的行为。在这里，我们将这个网络用于模拟分级听觉系统，并研究它是否会对一组随机音调序列的音乐性做出与人类相似的选择。当用大量的器乐古典音乐和呈现为梅尔谱图的流行旋律来训练该模型时，它对于被人类对象评价为音乐性较低的随机音调序列表现出较大的预测误差。我们发现，预测误差取决于关于后续音符、音高间隔和时间上下文的信息量。我们的研究结果表明，可预测性与人类对音乐性的感知相关，并且在音乐上训练的预测编码神经网络可以用来表征有助于人类对音乐感知的特征和主题。摘要：We explore the use of a neural network inspired by predictive coding for modeling human music perception. This network was developed based on the computational neuroscience theory of recurrent interactions in the hierarchical visual cortex. When trained with video data using self-supervised learning, the model manifests behaviors consistent with human visual illusions. Here, we adapt this network to model the hierarchical auditory system and investigate whether it will make similar choices to humans regarding the musicality of a set of random pitch sequences. When the model is trained with a large corpus of instrumental classical music and popular melodies rendered as mel spectrograms, it exhibits greater prediction errors for random pitch sequences that are rated less musical by human subjects. We found that the prediction error depends on the amount of information regarding the subsequent note, the pitch interval, and the temporal context. Our findings suggest that predictability is correlated with human perception of musicality and that a predictive coding neural network trained on music can be used to characterize the features and motifs contributing to human perception of music.
【25】 End-to-end Spoken Language Understanding with Tree-constrained Pointer Generator标题：使用树约束指针生成器实现端到端的口语理解链接：https://arxiv.org/abs/2210.16554
作者：Guangzhi Sun,Chao Zhang,Philip C. Woodland机构：Cambridge University Engineering Dept., Trumpington St., Cambridge, CB,PZ U.K.备注：5 pages, submitted to ICASSP 2023摘要：端到端口语理解（SLU）存在长尾词问题。在端到端的SLU系统中，本文利用上下文偏差技术来提高稀有词的语音识别率。研究了一种功能强大、效率较高的偏置模型组件--树约束指针生成器（TCPGen），它利用一个包含相应实体的时隙候选列表来提取偏置列表.同时，为了偏置SLU模型输出时隙分布，提出了时隙概率偏置（SPB）机制，用于从TCPGen计算时隙分布。在SLURP数据集上的实验表明，使用TCPGen和SPB的SLU-F1改进是一致的，尤其是在看不见的实体上。在一个新的划分上，通过为测试提供5种槽类型，使用SPB的TCPGen实现了零射击学习，与无法处理它的基线相比，SLU-F1得分超过50%。除了槽填充外，意图分类准确率也得到了提高。摘要：End-to-end spoken language understanding (SLU) suffers from the long-tail word problem. This paper exploits contextual biasing, a technique to improve the speech recognition of rare words, in end-to-end SLU systems. Specifically, a tree-constrained pointer generator (TCPGen), a powerful and efficient biasing model component, is studied, which leverages a slot shortlist with corresponding entities to extract biasing lists. Meanwhile, to bias the SLU model output slot distribution, a slot probability biasing (SPB) mechanism is proposed to calculate a slot distribution from TCPGen. Experiments on the SLURP dataset showed consistent SLU-F1 improvements using TCPGen and SPB, especially on unseen entities. On a new split by holding out 5 slot types for the test, TCPGen with SPB achieved zero-shot learning with an SLU-F1 score over 50% compared to baselines which can not deal with it. In addition to slot filling, the intent classification accuracy was also improved.
【26】 Exploiting prompt learning with pre-trained language models for Alzheimer's Disease detection标题：利用快速学习和预先训练的语言模型检测阿尔茨海默病链接：https://arxiv.org/abs/2210.16539
作者：Yi Wang,Jiajun Deng,Tianzi Wang,Bo Zheng,Shoukang Hu,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Nanyang Technological University, Singapore备注：Submitted to ICASSP 2023摘要：阿尔茨海默病（AD）的早期诊断对于促进预防性护理和延缓进一步进展至关重要。基于语音的自动AD筛查系统提供了非侵入性的并且比其他临床筛查技术更可扩展的替代方案。由诸如BERT的预训练语言模型（PLM）产生的文本嵌入特征被广泛地用于这样的系统中。然而，PLM域微调通常基于与后端AD检测任务不一致的屏蔽词或句子预测成本。为此，本文研究了PLM的基于提示的微调的使用，其一致地使用AD分类错误作为训练目标函数。在PLM微调期间，基于停顿或停顿填充符令牌频率的不流畅特征被进一步并入提示短语。利用基于BERT或RoBERTa的PLM之间的互补性，进一步应用了它们之间的基于决策投票的系统组合，其中基于BERT或RoBERTa的PLM或者是即时学习微调的，或者是使用传统的屏蔽词或句子预测成本优化的。采用平均值、标准差和15次实验中准确度得分的最大值作为AD检测系统的性能度量。在由48名老年人组成的ADReSS20测试集上，使用人工和ASR语音转录本分别获得了84.20%（标准差为2.09%，最佳为87.5%）和82.64%（标准差为4.0%，最佳为89.58%）的平均检测准确率。摘要：Early diagnosis of Alzheimer's disease (AD) is crucial in facilitating preventive care and to delay further progression. Speech based automatic AD screening systems provide a non-intrusive and more scalable alternative to other clinical screening techniques. Textual embedding features produced by pre-trained language models (PLMs) such as BERT are widely used in such systems. However, PLM domain fine-tuning is commonly based on the masked word or sentence prediction costs that are inconsistent with the back-end AD detection task. To this end, this paper investigates the use of prompt-based fine-tuning of PLMs that consistently uses AD classification errors as the training objective function. Disfluency features based on hesitation or pause filler token frequencies are further incorporated into prompt phrases during PLM fine-tuning. The exploit of the complementarity between BERT or RoBERTa based PLMs that are either prompt learning fine-tuned, or optimized using conventional masked word or sentence prediction costs, decision voting based system combination between them is further applied. Mean, standard deviation and the maximum among accuracy scores over 15 experiment runs are adopted as performance measurements for the AD detection system. Mean detection accuracy of 84.20% (with std 2.09%, best 87.5%) and 82.64% (with std 4.0%, best 89.58%) were obtained using manual and ASR speech transcripts respectively on the ADReSS20 test set consisting of 48 elderly speakers.
【27】 Phonemic Representation and Transcription for Speech to Text Applications for Under-resourced Indigenous African Languages: The Case of Kiswahili标题：资源不足的非洲土著语言的语音到文本应用的音素表示和转录：斯瓦希里语的情况链接：https://arxiv.org/abs/2210.16537
作者：Ebbie Awino,Lilian Wanzare,Lawrence Muchemi,Barack Wanjawa,Edward Ombui,Florence Indede,Owen McOnyango,Benard Okal机构：2 – affiliation University of Nairobi, 3 – affiliation Maseno University, 4 – affiliation Africa Nazarene University摘要：建立自动语音识别系统是一项具有挑战性的任务，特别是对于资源不足的语言，需要几乎从头开始构建语料库，缺乏足够的训练数据。现已发现，包括斯瓦希里语在内的几种非洲土著语言在技术上资源不足。ASR系统是至关重要的，特别是对于听力受损的人，他们可以受益于他们的母语抄本。然而，转录语音数据集的缺乏使得为这些土著语言开发ASR模型的努力变得复杂。本文探讨了斯瓦希里语语音语料库的转录过程和开发过程，该语料库包括朗读文本和斯瓦希里语母语者的自发语音数据。该研究还讨论了斯瓦希里语的元音和辅音，并为ASR模型提供了一个更新的斯瓦希里语音素词典，该词典是使用CMU Sphinx语音识别工具箱（一个开放源代码的语音识别工具箱）创建的。ASR模型使用扩展的语音集进行训练，得到的WER和SER分别为18.87%和49.5%，比之前针对资源不足语言的类似研究的性能有所提高。摘要：Building automatic speech recognition (ASR) systems is a challenging task, especially for under-resourced languages that need to construct corpora nearly from scratch and lack sufficient training data. It has emerged that several African indigenous languages, including Kiswahili, are technologically under-resourced. ASR systems are crucial, particularly for the hearing-impaired persons who can benefit from having transcripts in their native languages. However, the absence of transcribed speech datasets has complicated efforts to develop ASR models for these indigenous languages. This paper explores the transcription process and the development of a Kiswahili speech corpus, which includes both read-out texts and spontaneous speech data from native Kiswahili speakers. The study also discusses the vowels and consonants in Kiswahili and provides an updated Kiswahili phoneme dictionary for the ASR model that was created using the CMU Sphinx speech recognition toolbox, an open-source speech recognition toolkit. The ASR model was trained using an extended phonetic set that yielded a WER and SER of 18.87% and 49.5%, respectively, an improved performance than previous similar research for under-resourced languages.
【28】 Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation标题：利用场景图学习音频源分离的视听动力学链接：https://arxiv.org/abs/2210.16472
作者：Moitreya Chatterjee,Narendra Ahuja,Anoop Cherian机构：University of Illinois, Urbana-Champaign, Urbana, IL, Mitsubishi Electric Research Labs, Cambridge, MA备注：Accepted at NeurIPS 2022摘要：None摘要：There exists an unequivocal distinction between the sound produced by a static source and that produced by a moving one, especially when the source moves towards or away from the microphone. In this paper, we propose to use this connection between audio and visual dynamics for solving two challenging tasks simultaneously, namely: (i) separating audio sources from a mixture using visual cues, and (ii) predicting the 3D visual motion of a sounding source using its separated audio. Towards this end, we present Audio Separator and Motion Predictor (ASMP) -- a deep learning framework that leverages the 3D structure of the scene and the motion of sound sources for better audio source separation. At the heart of ASMP is a 2.5D scene graph capturing various objects in the video and their pseudo-3D spatial proximities. This graph is constructed by registering together 2.5D monocular depth predictions from the 2D video frames and associating the 2.5D scene regions with the outputs of an object detector applied on those frames. The ASMP task is then mathematically modeled as the joint problem of: (i) recursively segmenting the 2.5D scene graph into several sub-graphs, each associated with a constituent sound in the input audio mixture (which is then separated) and (ii) predicting the 3D motions of the corresponding sound sources from the separated audio. To empirically evaluate ASMP, we present experiments on two challenging audio-visual datasets, viz. Audio Separation in the Wild (ASIW) and Audio Visual Event (AVE). Our results demonstrate that ASMP achieves a clear improvement in source separation quality, outperforming prior works on both datasets, while also estimating the direction of motion of the sound sources better than other methods.
【29】 HeartSiam: A Domain Invariant Model for Heart Sound Classification标题：HeartSiam：一种心音分类领域不变模型链接：https://arxiv.org/abs/2210.16394
作者：Reza Yousefi Mashhoor,Ahmad Ayatollahi机构：Electrical Engineering Dept., Sharif University of Technology, Tehran, Iran, Iran University of Science and Technology摘要：根据WHO，心血管疾病是导致死亡的主要原因之一。心音图（PCG）是一种经济、无创的心脏监测方法。这项工作的主要目的是将心音分类为正常/异常类别。使用不同的听诊器记录心音，因此在该域中不同。根据最近的研究，这种变异性可以影响心音分类。本文提出了一种用于学习正常信号与正常信号或异常信号与异常信号之间的相似性以及正常信号与异常信号之间的差异的连体网络结构。通过在所有域上应用这种相似性和差异学习，可以很好地实现域不变心音分类的任务。我们使用了多域2016 Physionet/CinC挑战数据集作为评价方法。结果：在挑战提供的评估集上，我们获得了82.8%的灵敏度、75.3%的特异性和79.1%的平均准确性。在克服多域问题的同时，所提出的方法在高达10.9%的特异性和高达5.6%的平均准确性方面超过了Physionet挑战的第一名方法。此外，与类似的最先进的域不变方法相比，我们的模型收敛更快，在相同数量的学习历元的特异性（4.1%）和平均准确性（1.5%）方面表现更好。摘要：Cardiovascular disease is one of the leading causes of death according to WHO. Phonocardiography (PCG) is a costeffective, non-invasive method suitable for heart monitoring. The main aim of this work is to classify heart sounds into normal/abnormal categories. Heart sounds are recorded using different stethoscopes, thus varying in the domain. Based on recent studies, this variability can affect heart sound classification. This work presents a Siamese network architecture for learning the similarity between normal vs. normal or abnormal vs. abnormal signals and the difference between normal vs. abnormal signals. By applying this similarity and difference learning across all domains, the task of domain invariant heart sound classification can be well achieved. We have used the multi-domain 2016 Physionet/CinC challenge dataset for the evaluation method. Results: On the evaluation set provided by the challenge, we have achieved a sensitivity of 82.8%, specificity of 75.3%, and mean accuracy of 79.1%. While overcoming the multi-domain problem, the proposed method has surpassed the first-place method of the Physionet challenge in terms of specificity up to 10.9% and mean accuracy up to 5.6%. Also, compared with similar state-of-the-art domain invariant methods, our model converges faster and performs better in specificity (4.1%) and mean accuracy (1.5%) with an equal number of epochs learned.
【30】 Filter and evolve: progressive pseudo label refining for semi-supervised automatic speech recognition标题：过滤和进化：半监督自动语音识别的渐进式伪标签精化链接：https://arxiv.org/abs/2210.16318
作者：Zezhong Jin,Dading Zhong,Xiao Song,Zhaoyi Liu,Naipeng Ye,Qingcheng Zeng机构：Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China, School of Electronic and Computer Engineering, Peking University, China, imec-Distrinet, Computer Science, KU Leuven, Leuven, Belgium摘要：利用伪标签对自监督预训练模型进行微调可以有效地提高语音识别性能。但是，低质量伪标签会误导判定边界并降低性能。我们提出一个简单而有效的策略来过滤低质量的伪标签以缓解这个问题。具体地，在整个训练集上产生伪标签，并且经由从模型输出计算的平均概率得分来过滤伪标签。随后，具有高概率分数的最佳百分比的话语被认为是具有可信标签的可靠训练数据。该模型被迭代地更新以校正不可靠的伪标签，从而最小化噪声标签的影响。重复上述过程，直到不可靠的伪标记已被充分校正。在LibriSpeech上的大量实验表明，这些滤波后的样本使得改进后的模型能够产生更正确的预测，从而在各种实验设置下获得更好的ASR性能。摘要：Fine tuning self supervised pretrained models using pseudo labels can effectively improve speech recognition performance. But, low quality pseudo labels can misguide decision boundaries and degrade performance. We propose a simple yet effective strategy to filter low quality pseudo labels to alleviate this problem. Specifically, pseudo-labels are produced over the entire training set and filtered via average probability scores calculated from the model output. Subsequently, an optimal percentage of utterances with high probability scores are considered reliable training data with trustworthy labels. The model is iteratively updated to correct the unreliable pseudo labels to minimize the effect of noisy labels. The process above is repeated until unreliable pseudo abels have been adequately corrected. Extensive experiments on LibriSpeech show that these filtered samples enable the refined model to yield more correct predictions, leading to better ASR performances under various experimental settings.
【31】 Audio-Visual Speech Enhancement and Separation by Leveraging Multi-Modal Self-Supervised Embeddings标题：利用多模式自监督嵌入的视听语音增强和分离链接：https://arxiv.org/abs/2210.17456
作者：I-Chun Chern,Kuo-Hsuan Hung,Yi-Ting Chen,Tassadaq Hussain,Mandar Gogate,Amir Hussain,Yu Tsao,Jen-Cheng Hou机构：Carnegie Mellon University, USA ,National Taiwan University, Taiwan, Academia Sinica, Taiwan ,Edinburgh Napier University, UK备注：Under peer review摘要：AV-HuBERT是一种多模态自监督学习模型，已被证明对自动语音识别和唇读等分类问题是有效的。这表明通过利用多模态自监督嵌入可以获得有用的视听语音表示。然而，不清楚这种表示是否可以被推广到解决真实世界的多模态AV回归任务，诸如视听语音增强（AVSE）和视听语音分离（AVSS）。在本研究中，我们利用了预训练的AV-HuBERT模型，随后是AVSE和AVSS的SE模块。对比实验结果表明，该模型的性能优于现有的AVSE模型和传统的纯音频SE模型。总之，我们的结果证实了我们提出的模型在适当的微调策略下对AVSS任务的有效性，证明了从AV-HUBERT获得的多模态自监督嵌入可以推广到视听回归任务。摘要：AV-HuBERT, a multi-modal self-supervised learning model, has been shown to be effective for categorical problems such as automatic speech recognition and lip-reading. This suggests that useful audio-visual speech representations can be obtained via utilizing multi-modal self-supervised embeddings. Nevertheless, it is unclear if such representations can be generalized to solve real-world multi-modal AV regression tasks, such as audio-visual speech enhancement (AVSE) and audio-visual speech separation (AVSS). In this study, we leveraged the pre-trained AV-HuBERT model followed by an SE module for AVSE and AVSS. Comparative experimental results demonstrate that our proposed model performs better than the state-of-the-art AVSE and traditional audio-only SE models. In summary, our results confirm the effectiveness of our proposed model for the AVSS task with proper fine-tuning strategies, demonstrating that multi-modal self-supervised embeddings obtained from AV-HUBERT can be generalized to audio-visual regression tasks.
【32】 VoicePrivacy 2022 System Description: Speaker Anonymization with Feature-matched F0 Trajectories标题：VoicePrivacy 2022系统描述：特征匹配F0轨迹的说话人匿名化链接：https://arxiv.org/abs/2210.17338
作者：Ünal Ege Gaznepoglu,Anna Leschanowsky,Nils Peters机构：Friedrich-Alexander-Universit¨at, International Audio Laboratories Erlangen, Germany, Fraunhofer IIS, Erlangen, Germany备注：4 pages, 4 figures, 2 tables, submitted to VoicePrivacy Challenge 2022摘要：我们介绍了一种新的方法来提高VoicePrivacy Challenge 2022基线B1变体的性能。在基于x向量的匿名化系统的已知缺陷中，输入特征的不充分的去纠缠。特别是基频（F0）轨迹，其用于语音合成而没有任何修改。特别是在跨性别转换中，这种情况会导致不自然的声音，增加单词错误率（WER），以及个人信息泄漏。我们的提交通过合成F0轨迹克服了这个问题，该轨迹更好地与匿名x矢量协调。我们利用低复杂度的深度神经网络，使用来自瓶颈特征（BN）的语言内容和匿名化的x向量来估计每帧的适当F0值。我们的方法导致显著改进的匿名化系统和增加的合成语音的自然度。因此，我们的结果表明，语音匿名化不需要F0提取。摘要：We introduce a novel method to improve the performance of the VoicePrivacy Challenge 2022 baseline B1 variants. Among the known deficiencies of x-vector-based anonymization systems is the insufficient disentangling of the input features. In particular, the fundamental frequency (F0) trajectories, which are used for voice synthesis without any modifications. Especially in cross-gender conversion, this situation causes unnatural sounding voices, increases word error rates (WERs), and personal information leakage. Our submission overcomes this problem by synthesizing an F0 trajectory, which better harmonizes with the anonymized x-vector. We utilized a low-complexity deep neural network to estimate an appropriate F0 value per frame, using the linguistic content from the bottleneck features (BN) and the anonymized x-vector. Our approach results in a significantly improved anonymization system and increased naturalness of the synthesized voice. Consequently, our results suggest that F0 extraction is not required for voice anonymization.
【33】 Diffusion-based Generative Speech Source Separation标题：基于扩散的生成性语音源分离链接：https://arxiv.org/abs/2210.17327
作者：Robin Scheibler,Youna Ji,Soo-Whan Chung,Jaeuk Byun,Soyeon Choe,Min-Seok Choi机构：LINE Corporation, Tokyo, Japan, Naver Corporation, South Korea备注：5 pages, 3 figures, 2 tables. Submitted to ICASSP 2023摘要：提出了一种基于随机微分方程（SDE）分数匹配的单通道信号分离方法。我们设计了一个定制的连续时间扩散混合过程，从分离的源开始，收敛到以它们的混合物为中心的高斯分布。这个公式让我们可以应用基于得分的生成式建模机制。首先，我们训练一个神经网络来近似边缘概率的得分函数或扩散混合过程。然后，我们用它来求解逆时随机微分方程，该方程从混合源开始逐步分离源。我们提出一种改进的训练策略来处理模型失配和源排列歧义。在WSJ 0 2 mix数据集上的实验验证了该方法的有效性.此外，该方法也适用于语音增强，并在VoiceBank-DEMAND数据集上表现出与现有工作相当的性能。摘要：We propose a new single channel source separation method based on score-matching of a stochastic differential equation (SDE). We craft a tailored continuous time diffusion-mixing process starting from the separated sources and converging to a Gaussian distribution centered on their mixture. This formulation lets us apply the machinery of score-based generative modelling. First, we train a neural network to approximate the score function of the marginal probabilities or the diffusion-mixing process. Then, we use it to solve the reverse time SDE that progressively separates the sources starting from their mixture. We propose a modified training strategy to handle model mismatch and source permutation ambiguity. Experiments on the WSJ0 2mix dataset demonstrate the potential of the method. Furthermore, the method is also suitable for speech enhancement and shows performance competitive with prior work on the VoiceBank-DEMAND dataset.
【34】 Model Compression for DNN-Based Text-Independent Speaker Verification Using Weight Quantization标题：基于加权量化的DNN文本无关说话人确认模型压缩链接：https://arxiv.org/abs/2210.17326
作者：Jingyu Li,Zhaoyang Zhang,Jiong Wang,Tan Lee机构：Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China摘要：基于DNN的模型在说话人确认（SV）任务中以相当大的计算成本实现了高性能。模型大小是在资源受限设备上应用模型时的一个重要考虑因素，而SV模型的模型压缩在以前的工作中没有被广泛研究。本文采用权值量化的方法对基于DNN的说话人嵌入提取模型进行压缩。实验中采用了均匀量子化和2的幂次量子化。在VoxCelib上的实验结果表明，权重量化可以在不影响性能的情况下将ECAPA-TDNN和ResNet的规模减小4倍。量化的4位ResNet以小8倍的大小实现了与原始模型类似的性能。实验结果表明，ECAPA-TDNN的性能比ResNet对量化更为敏感，这是因为ECAPA-TDNN的权重分布不同。在CN-Celeb上的实验也证明了量化模型在语言不匹配的情况下对SV是鲁棒的。摘要：DNN-based models achieve high performance in the speaker verification (SV) task with substantial computation costs. The model size is an essential concern in applying models on resource-constrained devices, while model compression for SV models has not been studied extensively in previous works. Weight quantization is exploited to compress DNN-based speaker embedding extraction models in this paper. Uniform and Powers-of-Two quantization are utilized in the experiments. The results on VoxCeleb show that the weight quantization can decrease the size of ECAPA-TDNN and ResNet by 4 times with insignificant performance decline. The quantized 4-bit ResNet achieves similar performance to the original model with an 8 times smaller size. We empirically show that the performance of ECAPA-TDNN is more sensitive than ResNet to quantization due to the difference in weight distribution. The experiments on CN-Celeb also demonstrate that quantized models are robust for SV in the language mismatch scenario.
【35】 There is more than one kind of robustness: Fooling Whisper with adversarial examples标题：健壮性不止一种：用对抗性的例子愚弄耳语链接：https://arxiv.org/abs/2210.17316
作者：Raphael Olivier,Bhiksha Raj机构：Carnegie Mellon University摘要：Whisper是一种最新的自动语音识别（ASR）模型，它对非分布输入和随机噪声都表现出令人印象深刻的鲁棒性。在这项工作中，我们证明了这种鲁棒性不会延续到对抗噪声。我们产生非常小的输入扰动，信噪比高达45dB，这样我们就可以大大降低Whisper的性能，甚至转录我们选择的目标句子。我们还表明，通过欺骗Whisper语言检测器，我们可以很容易地降低多语言模型的性能。广泛流行的开放源代码模型的这些漏洞具有实际的安全含义，并强调了对相反健壮的ASR的需要。摘要：Whisper is a recent Automatic Speech Recognition (ASR) model displaying impressive robustness to both out-of-distribution inputs and random noise. In this work, we show that this robustness does not carry over to adversarial noise. We generate very small input perturbations with Signal Noise Ratio of up to 45dB, with which we can degrade Whisper performance dramatically, or even transcribe a target sentence of our choice. We also show that by fooling the Whisper language detector we can very easily degrade the performance of multilingual models. These vulnerabilities of a widely popular open-source model have practical security implications, and emphasize the need for adversarially robust ASR.
【36】 Convolution-Based Channel-Frequency Attention for Text-Independent Speaker Verification标题：基于卷积的声频关注度的文本无关说话人确认链接：https://arxiv.org/abs/2210.17310
作者：Jingyu Li,Yusheng Tian,Tan Lee机构：Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong摘要：深度卷积神经网络（CNNs）已被应用于提取说话人嵌入，并在说话人确认中取得了显著的成功。引入注意力机制已被证明在提高模型性能方面是有效的。本文提出了一种基于二维卷积的注意力模型C2 D-Att。在注意力计算中，通过轻量级卷积层，将卷积通道与频率的交互作用纳入注意力计算。这只需要少量的参数。产生细粒度的注意力权重以表示信道和频率特定的信息。通过对输入特征进行加权，提高了说话人建模的表示能力。C2 D-Att被集成到ResNet的修改版本中用于说话人嵌入提取。实验在VoxCeleb数据集上进行。实验结果表明，C2 DAtt算法能够有效地生成区分性注意图，且优于其他注意方法。该模型在不同尺度下均表现出较强的鲁棒性，取得了较好的结果。摘要：Deep convolutional neural networks (CNNs) have been applied to extracting speaker embeddings with significant success in speaker verification. Incorporating the attention mechanism has shown to be effective in improving the model performance. This paper presents an efficient two-dimensional convolution-based attention module, namely C2D-Att. The interaction between the convolution channel and frequency is involved in the attention calculation by lightweight convolution layers. This requires only a small number of parameters. Fine-grained attention weights are produced to represent channel and frequency-specific information. The weights are imposed on the input features to improve the representation ability for speaker modeling. The C2D-Att is integrated into a modified version of ResNet for speaker embedding extraction. Experiments are conducted on VoxCeleb datasets. The results show that C2DAtt is effective in generating discriminative attention maps and outperforms other attention methods. The proposed model shows robust performance with different scales of model size and achieves state-of-the-art results.
【37】 A Versatile Diffusion-based Generative Refiner for Speech Enhancement标题：一种通用的基于扩散的生成式语音增强精化器链接：https://arxiv.org/abs/2210.17287
作者：Ryosuke Sawata,Naoki Murata,Yuhta Takida,Toshimitsu Uesaka,Takashi Shibuya,Shusuke Takahashi,Yuki Mitsufuji机构：Sony Group Corporation, Tokyo, Japan备注：Submitted to ICASSP 2023摘要：虽然基于深度神经网络的语音增强方法优于以往的非深度神经网络语音增强方法，但它们往往会降低语音的感知质量。针对这一问题，本文提出了一种基于DNN的生成式精炼器，旨在提高SE方法预处理后的感知语音质量。作为细化器，我们通过利用仅由干净语音组成的数据集来训练基于扩散的生成模型。然后，通过去噪扩散恢复，用新生成的干净部分替换由SE方法引起的退化和失真部分。一旦我们的精炼器在一组干净的语音上被训练，它就可以被应用于各种SE方法，而不需要为每个SE模块专门进行额外的训练。因此，我们的磨浆机可以是一个多功能的后处理模块w.r.t. SE方法，并在模块化方面具有很高的潜力。实验结果表明，无论采用何种SE方法，该方法都能提高语音的感知质量。摘要：Although deep neural network (DNN)-based speech enhancement (SE) methods outperform the previous non-DNN-based ones, they often degrade the perceptual quality of generated outputs. To tackle this problem, We introduce a DNN-based generative refiner aiming to improve perceptual speech quality pre-processed by an SE method. As the refiner, we train a diffusion-based generative model by utilizing a dataset consisting of clean speech only. Then, the model replaces the degraded and distorted parts caused by a preceding SE method with newly generated clean parts by denoising diffusion restoration. Once our refiner is trained on a set of clean speech, it can be applied to various SE methods without additional training specialized for each SE module. Therefore, our refiner can be a versatile post-processing module w.r.t. SE methods and has high potential in terms of modularity. Experimental results show that our method improved perceptual speech quality regardless of the preceding SE methods used.
【38】 DiaCorrect: End-to-end error correction for speaker diarization标题：DiaEqut：针对扬声器二元化的端到端纠错链接：https://arxiv.org/abs/2210.17189
作者：Jiangyu Han,Yuhang Cao,Heng Lu,Yanhua Long机构：Ximalaya Inc., Shanghai, China, Shanghai Normal University, Shanghai, China备注：submitted to ICASSP摘要：近年来，说话人日记化引起了广泛的关注。为了获得更好的性能，一些研究提出在多个阶段中日记化语音。虽然这些方法可能带来额外的好处，但它们中的大多数都相当复杂。受自动语音识别（ASR）中拼写纠正的启发，本文提出了一个端到端的纠错框架DiaCorrect，以一种简单而有效的方式改进初始diarization结果。DiaCorrect通过利用输入混合信号与其对应的说话人活动之间的声学交互作用，自动调整初始说话人活动以最小化diarization误差。在不增加额外负担的情况下，对基于LibriSpeech的2人会议类数据的实验表明，使用DiaCorrect的自注意端到端神经网络日记化（SA-EEND）基线可以将日记化错误率（DER）从12.31%降低到4.63%，降低幅度超过62.4%。我们的源代码可从www.example.com在线获取https://github.com/jyhan03/diacorrect。摘要：In recent years, speaker diarization has attracted widespread attention. To achieve better performance, some studies propose to diarize speech in multiple stages. Although these methods might bring additional benefits, most of them are quite complex. Motivated by spelling correction in automatic speech recognition (ASR), in this paper, we propose an end-to-end error correction framework, termed DiaCorrect, to refine the initial diarization results in a simple but efficient way. By exploiting the acoustic interactions between input mixture and its corresponding speaker activity, DiaCorrect could automatically adapt the initial speaker activity to minimize the diarization errors. Without bells and whistles, experiments on LibriSpeech based 2-speaker meeting-like data show that, the self-attentitive end-to-end neural diarization (SA-EEND) baseline with DiaCorrect could reduce its diarization error rate (DER) by over 62.4% from 12.31% to 4.63%. Our source code is available online at https://github.com/jyhan03/diacorrect.
【39】 Minimum Processing Near-end Listening Enhancement标题：最低处理能力近端监听增强链接：https://arxiv.org/abs/2210.17154
作者：Andreas Jonas Fuglsig,Jesper Jensen,Zheng-Hua Tan,Lars Søndergaard Bertelsen,Jens Christian Lindof,Jan Østergaard摘要：来自移动电话或公共广播系统的语音的可懂度和质量经常受到收听环境中的背景噪声的影响。通过对语音信号进行预处理，可以提高语音的可懂度和质量--这被称为近端收听增强（NLE）。虽然，现有的NLE技术能够在恶劣的噪声环境中极大地提高可懂度，但是在有利的噪声条件下，语音的可懂度达到了不能进一步提高的上限。实际上，现有方法仅仅关注于提高可懂度，这导致了对语音信号的不必要的处理，并导致语音失真和质量下降。在这篇论文中，我们提出了一种新的NLE理论，在一定的性能约束条件下，可理解性。本文提出了一种封闭形式的解决方案，其中性能标准是基于近似的语音可懂度指数的可懂度估计器，处理惩罚是处理后的语音与干净语音之间的均方误差。这产生了一种NLE方法，该方法通过将处理限制到实现期望的可懂度所需的最小值，而同时通过最小化语音失真量来关注有利的噪声情况下的质量，从而经由简单的增益规则来适应于变化的噪声条件。通过仿真研究，我们表明所提出的方法在客观测量和主观听音测试中获得了与现有方法相当或更好的语音质量，同时仍保持与现有方法相当的客观语音可懂度性能。摘要：The intelligibility and quality of speech from a mobile phone or public announcement system are often affected by background noise in the listening environment. By pre-processing the speech signal it is possible to improve the speech intelligibility and quality -- this is known as near-end listening enhancement (NLE). Although, existing NLE techniques are able to greatly increase intelligibility in harsh noise environments, in favorable noise conditions the intelligibility of speech reaches a ceiling where it cannot be further enhanced. Actually, the focus of existing methods solely on improving the intelligibility causes unnecessary processing of the speech signal and leads to speech distortions and quality degradations. In this paper, we provide a new rationale for NLE, where the target speech is minimally processed in terms of a processing penalty, provided that a certain performance constraint, e.g., intelligibility, is satisfied. We present a closed-form solution for the case where the performance criterion is an intelligibility estimator based on the approximated speech intelligibility index and the processing penalty is the mean-square error between the processed and the clean speech. This produces an NLE method that adapts to changing noise conditions via a simple gain rule by limiting the processing to the minimum necessary to achieve a desired intelligibility, while at the same time focusing on quality in favorable noise situations by minimizing the amount of speech distortions. Through simulation studies, we show the proposed method attains speech quality on par or better than existing methods in both objective measurements and subjective listening tests, whilst still sustaining objective speech intelligibility performance on par with existing methods.
【40】 The Importance of Accurate Alignments in End-to-End Speech Synthesis标题：准确对齐在端到端语音合成中的重要性链接：https://arxiv.org/abs/2210.17153
作者：Anusha Prakash,Hema A Murthy机构：Indian Institute of Technology Madras, India备注：Version 1 uploaded摘要：由于拼接的特性，单元选择合成系统需要对语音信号进行精确的分段和标记。基于隐马尔可夫模型的语音合成适应一些转录误差，但后来表明，准确的转录用更少量的训练数据产生高度可理解的语音。随着端到端（E2E）系统的到来，观察到可以用大量数据合成非常好质量的语音。随着端到端合成从Tacotron发展到FastSpeech2，表示韵律的特征对于高质量合成的重要性已经变得迫在眉睫。特别地，子字单元的持续时间是重要的。FastSpeech的变体使用教师模型或强制对齐来获得高质量的合成。在本文中，我们将重点放在持续时间预测上，使用信号处理线索与强制对齐相结合，在训练过程中产生准确的音素持续时间。 目前的工作旨在强调精确比对对于高质量合成的重要性。尝试用精确标记的数据训练E2E系统，并将其与近似标记的数据进行比较。摘要：Unit selection synthesis systems required accurate segmentation and labeling of the speech signal owing to the concatenative nature. Hidden Markov model-based speech synthesis accommodates some transcription errors, but it was later shown that accurate transcriptions yield highly intelligible speech with smaller amounts of training data. With the arrival of end-to-end (E2E) systems, it was observed that very good quality speech could be synthesised with large amounts of data. As end-to-end synthesis progressed from Tacotron to FastSpeech2, it has become imminent that features that represent prosody are important for good-quality synthesis. In particular, durations of the sub-word units are important. Variants of FastSpeech use a teacher model or forced alignments to obtain good-quality synthesis. In this paper, we focus on duration prediction, using signal processing cues in tandem with forced alignment to produce accurate phone durations during training. The current work aims to highlight the importance of accurate alignments for good-quality synthesis. An attempt is made to train the E2E systems with accurately labeled data, and compare the same with approximately labeled data.
【41】 Real-Time MRI Video synthesis from time aligned phonemes with sequence-to-sequence networks标题：基于序列到序列网络的时间序列音素实时MRI视频合成链接：https://arxiv.org/abs/2210.16881
作者：Sathvik Udupa,Prasanta Kumar Ghosh机构：Electrical Engineering Department, Indian Institute of Science (IISc), Bangalore - , India.备注：submitted to ICASSP 2023摘要：口腔正中矢状面的实时磁共振成像（rtMRI）对于言语产生研究是有意义的。在这项工作中，我们的重点是从语音音素序列估计话语级rtMRI视频。我们从强制对齐中获得时间对齐的音素，以获得与rtMRI帧对齐的帧级音素序列。我们提出了一个具有Transformer音素编码器和卷积帧解码器的序列到序列学习模型。然后，我们通过使用从预先训练的音素条件变分自动编码器（CVAE）采样获得的中间特征来修改学习。我们以特定主题的方式对8个主题进行了训练，并通过主观测试证明了性能。我们还使用空气组织边界（ATB）分割的辅助任务来获得所提出的模型的客观评分。我们表明，所提出的方法能够为不可见的话语生成逼真的rtMRI视频，并且添加CVAE有利于学习难以学习映射的受试者的序列到序列映射。摘要：Real-Time Magnetic resonance imaging (rtMRI) of the midsagittal plane of the mouth is of interest for speech production research. In this work, we focus on estimating utterance level rtMRI video from the spoken phoneme sequence. We obtain time-aligned phonemes from forced alignment, to obtain frame-level phoneme sequences which are aligned with rtMRI frames. We propose a sequence-to-sequence learning model with a transformer phoneme encoder and convolutional frame decoder. We then modify the learning by using intermediary features obtained from sampling from a pretrained phoneme-conditioned variational autoencoder (CVAE). We train on 8 subjects in a subject-specific manner and demonstrate the performance with a subjective test. We also use an auxiliary task of air tissue boundary (ATB) segmentation to obtain the objective scores on the proposed models. We show that the proposed method is able to generate realistic rtMRI video for unseen utterances, and adding CVAE is beneficial for learning the sequence-to-sequence mapping for subjects where the mapping is hard to learn.
【42】 Improved acoustic-to-articulatory inversion using representations from pretrained self-supervised learning models标题：使用预先训练的自监督学习模型表示的改进的声学到发音转换链接：https://arxiv.org/abs/2210.16871
作者：Sathvik Udupa,Siddarth C,Prasanta Kumar Ghosh机构：Electrical Engineering Department, Indian Institute of Science (IISc), Bangalore - , India., Robert Bosch Centre for Data Science and AI, Indian Institute of Technology Madras - , India.备注：submitted to ICASSP 2023摘要：在这项工作中，我们研究了预训练的自监督学习（SSL）特征对学习声学到发音倒置（AAI）映射的有效性。基于信号处理的声学特征（如MFCC）已主要用于具有深度神经网络的AAI任务。由于SSL特征对于诸如语音识别、情感分类等各种其他语音任务工作良好，我们实验了其对AAI有效性。我们使用3种不同模型复杂度的基于变压器神经网络的AAI模型对SSL特征进行训练，并使用来自10个被试的数据在主题特定（subject-specific，SS）、合并和微调（pooled and fine-tuned，FT）配置下将其性能与MFCC进行比较，并使用相关系数（correlation coefficient，CC）评分对未见句子测试集进行评估。我们发现，基于声学特征重构目标的SSL特征（如TERA和DeCoAR）对于AAI效果良好，这些SSL特征的SS CC接近于MFCC的最佳FT CC。我们还发现不同模型尺寸的结果一致。摘要：In this work, we investigate the effectiveness of pretrained Self-Supervised Learning (SSL) features for learning the mapping for acoustic to articulatory inversion (AAI). Signal processing-based acoustic features such as MFCCs have been predominantly used for the AAI task with deep neural networks. With SSL features working well for various other speech tasks such as speech recognition, emotion classification, etc., we experiment with its efficacy for AAI. We train on SSL features with transformer neural networks-based AAI models of 3 different model complexities and compare its performance with MFCCs in subject-specific (SS), pooled and fine-tuned (FT) configurations with data from 10 subjects, and evaluate with correlation coefficient (CC) score on the unseen sentence test set. We find that acoustic feature reconstruction objective-based SSL features such as TERA and DeCoAR work well for AAI, with SS CCs of these SSL features reaching close to the best FT CCs of MFCC. We also find the results consistent across different model sizes.
【43】 WeKws: A production first small-footprint end-to-end Keyword Spotting Toolkit标题：WeKws：首个产品化小规模端到端关键词识别工具包链接：https://arxiv.org/abs/2210.16743
作者：Jie Wang,Menglong Xu,Jingyong Hou,Binbin Zhang,Xiao-Lei Zhang,Lei Xie,Fuping Pan机构：School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, China, School of Computer Science, Northwestern Polytechnical University, Xi’an, China, Horizon Robotics, Beijing, China, WeNet Open Source Community摘要：关键词识别（KWS）实现了基于语音的用户交互，并逐渐成为智能设备不可或缺的组成部分。最近，端到端（E2E）方法已经成为设备上KWS任务的最流行的方法。然而，E2E KWS方法的研究和部署仍然存在差距。本文介绍了一个具有产品级质量、易于构建、便于应用的E2E知识服务工具包WeKws。WeKws包含了几个最先进的主干网络的实现，使其在三个公开可用的数据集上取得了非常有竞争力的结果。为了使WeKws成为一个纯粹的E2E工具包，我们利用一个改进的最大池损失，使模型能够自己学习关键词的结束位置，这大大简化了训练流水线，使WeKws能够非常高效地应用于现实世界的场景。该工具包可在www.example.com上公开获得https://github.com/wenet-e2e/wekws。摘要：Keyword spotting (KWS) enables speech-based user interaction and gradually becomes an indispensable component of smart devices. Recently, end-to-end (E2E) methods have become the most popular approach for on-device KWS tasks. However, there is still a gap between the research and deployment of E2E KWS methods. In this paper, we introduce WeKws, a production-quality, easy-to-build, and convenient-to-be-applied E2E KWS toolkit. WeKws contains the implementations of several state-of-the-art backbone networks, making it achieve highly competitive results on three publicly available datasets. To make WeKws a pure E2E toolkit, we utilize a refined max-pooling loss to make the model learn the ending position of the keyword by itself, which significantly simplifies the training pipeline and makes WeKws very efficient to be applied in real-world scenarios. The toolkit is publicly available at https://github.com/wenet-e2e/wekws.
【44】 DuDe: Dual-Decoder Multilingual ASR for Indian Languages using Common Label Set标题：DUD：使用通用标签集的印度语言双解码多语言ASR链接：https://arxiv.org/abs/2210.16739
作者：Arunkumar A,Mudit Batra,Umesh S机构：Speech Lab, Indian Institute of Technology Madras备注：Submitted to ICASSP 2023摘要：None摘要：In a multilingual country like India, multilingual Automatic Speech Recognition (ASR) systems have much scope. Multilingual ASR systems exhibit many advantages like scalability, maintainability, and improved performance over the monolingual ASR systems. However, building multilingual systems for Indian languages is challenging since different languages use different scripts for writing. On the other hand, Indian languages share a lot of common sounds. Common Label Set (CLS) exploits this idea and maps graphemes of various languages with similar sounds to common labels. Since Indian languages are mostly phonetic, building a parser to convert from native script to CLS is easy. In this paper, we explore various approaches to build multilingual ASR models. We also propose a novel architecture called Encoder-Decoder-Decoder for building multilingual systems that use both CLS and native script labels. We also analyzed the effectiveness of CLS-based multilingual systems combined with machine transliteration.
【45】 Improvements to Embedding-Matching Acoustic-to-Word ASR Using Multiple-Hypothesis Pronunciation-Based Embeddings标题：基于多假设发音的嵌入匹配声词ASR的改进链接：https://arxiv.org/abs/2210.16726
作者：Hao Yen,Woojay Jeon机构：Georgia Institute of Technology, School of Electrical and Computer Engineering, Atlanta, GA, USA, One Apple Parkway, Cupertino, CA, USA摘要：在嵌入匹配声-字（A2 W）ASR中，词汇表中的每个字由固定维嵌入向量表示，该固定维嵌入向量可以独立于系统的其余部分而被添加或移除。该方法是解决动态词汇表外（OOV）单词问题的一种很好的方法，在该问题中，说话人和上下文相关的命名实体（如联系人姓名）必须在测试时针对每个语音话语动态地合并到ASR中。然而，在提高嵌入匹配A2 W的整体准确性方面仍然存在挑战。本文提出了两种提高嵌入匹配A2 W算法精度的方法。首先，我们提出在每个时刻内部产生多个嵌入，而不是单个嵌入，这允许A2 W模型在音频中的多个时间段上提出更丰富的假设集。第二，我们建议使用单词发音嵌入而不是单词拼写嵌入来减少由具有多于一个声音的单词引入的歧义。我们表明，在动态OOV词起关键作用的场景中，上述思想在相同的训练数据和几乎相同的模型大小的情况下给出了显著的准确性改进。在对基于语音的数字助理的各种查询的数据集上，包括许多依赖于用户的联系人姓名，我们观察到使用所提出的改进在单词错误率上减少了高达18%。摘要：In embedding-matching acoustic-to-word (A2W) ASR, every word in the vocabulary is represented by a fixed-dimension embedding vector that can be added or removed independently of the rest of the system. The approach is potentially an elegant solution for the dynamic out-of-vocabulary (OOV) words problem, where speaker- and context-dependent named entities like contact names must be incorporated into the ASR on-the-fly for every speech utterance at testing time. Challenges still remain, however, in improving the overall accuracy of embedding-matching A2W. In this paper, we contribute two methods that improve the accuracy of embedding-matching A2W. First, we propose internally producing multiple embeddings, instead of a single embedding, at each instance in time, which allows the A2W model to propose a richer set of hypotheses over multiple time segments in the audio. Second, we propose using word pronunciation embeddings rather than word orthography embeddings to reduce ambiguities introduced by words that have more than one sound. We show that the above ideas give significant accuracy improvement, with the same training data and nearly identical model size, in scenarios where dynamic OOV words play a crucial role. On a dataset of various queries to a speech-based digital assistant that include many user-dependent contact names, we observe up to 18% decrease in word error rate using the proposed improvements.
【46】 Speaker Representation Learning via Contrastive Loss with Maximal Speaker Separability标题：基于最大说话人可分性的对比损失说话人表征学习链接：https://arxiv.org/abs/2210.16636
作者：Zhe Li,Man-Wai Mak机构：The Hong Kong Polytechnic University, Hong Kong SAR备注：Accept by APSIPA ASC 2022, 6 pages, 2 figures摘要：使用深度模型进行说话人表征学习的一个巨大挑战是设计学习目标，以增强在不可见域下不可见说话人的区分。本文提出了一种有监督的对比学习目标，通过有效利用训练数据中的标签信息来学习说话人嵌入空间。在这样的空间中，由相同或相似说话者说出的话语对将保持接近，而由不同说话者说出的话语对将远离。对于每个训练说话人，我们在他们的话语上执行随机数据扩充以形成正对，并且来自不同说话人的话语形成负对。为了最大化说话人在嵌入空间中的可分离性，我们将附加的角边缘损失加入对比学习目标中。在CN-Celeb上的实验结果表明，新的学习目标能使ECAPA-TDNN找到一个具有较高说话人识别率的嵌入空间。对比学习目标很容易实现，我们在www.example.com上提供了PyTorch代码https://github.com/shanmon110/AAMSupCon。摘要：A great challenge in speaker representation learning using deep models is to design learning objectives that can enhance the discrimination of unseen speakers under unseen domains. This work proposes a supervised contrastive learning objective to learn a speaker embedding space by effectively leveraging the label information in the training data. In such a space, utterance pairs spoken by the same or similar speakers will stay close, while utterance pairs spoken by different speakers will be far apart. For each training speaker, we perform random data augmentation on their utterances to form positive pairs, and utterances from different speakers form negative pairs. To maximize speaker separability in the embedding space, we incorporate the additive angular-margin loss into the contrastive learning objective. Experimental results on CN-Celeb show that this new learning objective can cause ECAPA-TDNN to find an embedding space that exhibits great speaker discrimination. The contrastive learning objective is easy to implement, and we provide PyTorch code at https://github.com/shanmon110/AAMSupCon.
【47】 Discriminative Speaker Representation via Contrastive Learning with Class-Aware Attention in Angular Space标题：角空间类意识注意对比学习的辨别性说话人表征链接：https://arxiv.org/abs/2210.16622
作者：Zhe Li,Man-Wai Mak,Helen Mei-Ling Meng机构：Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong备注：Submitted to ICASSP 2023, 5 pages, 2 figures摘要：对比学习在说话人确认中的应用面临的挑战是基于softmax的对比损失缺乏区分能力，以及硬否定对容易影响学习。为了克服这些挑战，我们提出了一种对比学习SV框架，在监督对比损失的基础上加入一个附加的角度边缘。该间隔提高了说话人表示的辨别能力。我们引入了一种类别感知的注意机制，通过这种机制，硬否定样本对监督对比损失的贡献较小。我们还采用了一种基于梯度的多目标优化方法来平衡分类和对比损失。在CN-Celeb和Voxceleb 1上的实验结果表明，这种新的学习目标可以使编码器找到一个在不同语言之间表现出很大说话人区分度的嵌入空间。摘要：The challenges in applying contrastive learning to speaker verification (SV) are that the softmax-based contrastive loss lacks discriminative power and that the hard negative pairs can easily influence learning. To overcome these challenges, we propose a contrastive learning SV framework incorporating an additive angular margin into the supervised contrastive loss. The margin improves the speaker representation's discrimination ability. We introduce a class-aware attention mechanism through which hard negative samples contribute less significantly to the supervised contrastive loss. We also employed a gradient-based multi-objective optimization approach to balance the classification and contrastive loss. Experimental results on CN-Celeb and Voxceleb1 show that this new learning objective can cause the encoder to find an embedding space that exhibits great speaker discrimination across languages.
【48】 Application of Knowledge Distillation to Multi-task Speech Representation Learning标题：知识提取在多任务语音表征学习中的应用链接：https://arxiv.org/abs/2210.16611
作者：Mine Kerpicci,Van Nguyen,Shuhua Zhang,Erik Visser机构：Georgia Institue of Technology, Atlanta, Georgia, Qualcomm Technologies, Inc., San Diego, California备注：Speech representation learning, multitask learning, wav2vec, HuBERT, knowledge distillation摘要：已经提出了诸如wav2vec 2.0和HuBERT之类的模型体系结构来以自监督方式从音频波形学习语音表示。当这些模型与语音识别等下游任务相结合时，它们已被证明可提供最先进的性能。但是，这些模型使用了大量的参数，其最小版本具有大约9500万个参数。这对边缘AI设备部署构成了挑战。在本文中，我们使用知识蒸馏将原始模型大小减少了大约75%，同时保持了相似的性能水平。此外，我们还使用wav2vec 2.0和HuBERT模型进行提取，并通过实验分别在单任务和多任务框架下对提取的模型进行微调，给出了全面的性能分析。特别地，我们的实验表明，在关键词识别和说话人确认任务上微调提取的模型分别导致仅0.1%的准确率和0.9%的等错误率下降。摘要：Model architectures such as wav2vec 2.0 and HuBERT have been proposed to learn speech representations from audio waveforms in a self-supervised manner. When these models are combined with downstream tasks such as speech recognition, they have been shown to provide state-of-the-art performance. However, these models use a large number of parameters, the smallest version of which has about 95 million parameters. This constitutes a challenge for edge AI device deployments. In this paper, we use knowledge distillation to reduce the original model size by about 75% while maintaining similar performance levels. Moreover, we use wav2vec 2.0 and HuBERT models for distillation and present a comprehensive performance analysis through our experiments where we fine-tune the distilled models on single task and multi-task frameworks separately. In particular, our experiments show that fine-tuning the distilled models on keyword spotting and speaker verification tasks result in only 0.1% accuracy and 0.9% equal error rate degradations, respectively.
【49】 Articulatory Representation Learning Via Joint Factor Analysis and Neural Matrix Factorization标题：基于联合因子分析和神经矩阵分解的发音表征学习链接：https://arxiv.org/abs/2210.16498
作者：Jiachen Lian,Alan W Black,Yijing Lu,Louis Goldstein,Shinji Watanabe,Gopala K. Anumanchipalli机构：UC Berkeley , Carnegie Mellon University , University of Southern California备注：Submitted to 2023 ICASSP摘要：发音表征学习是神经语音产生系统建模的基础研究。我们之前的工作建立了一个深度范式，将发音运动学数据分解为手势，明确地建模了人类语音产生机制编码的语音和语言结构，以及相应的手势得分。我们通过提出两个关切继续这一工作路线：（1）在原始算法中发音器纠缠在一起，使得一些发音器不利用有效的移动模式，这限制了姿势和姿势分数的可解释性;（2）EMA数据是从发音者中稀疏采样的，这限制了习得表征的可理解性。本文提出了一种新的发音表征分解算法，该算法利用引导因子分析的优势来获得发音特征因子和因子得分。然后对因子得分采用神经卷积矩阵因式分解算法以导出新的姿势和姿态得分。我们使用rtMRI语料库进行实验，该语料库捕获了细粒度的声道轮廓。主观和客观的评价结果表明，新提出的系统提供了可理解、可概括、有效和可解释的发音表征。摘要：Articulatory representation learning is the fundamental research in modeling neural speech production system. Our previous work has established a deep paradigm to decompose the articulatory kinematics data into gestures, which explicitly model the phonological and linguistic structure encoded with human speech production mechanism, and corresponding gestural scores. We continue with this line of work by raising two concerns: (1) The articulators are entangled together in the original algorithm such that some of the articulators do not leverage effective moving patterns, which limits the interpretability of both gestures and gestural scores; (2) The EMA data is sparsely sampled from articulators, which limits the intelligibility of learned representations. In this work, we propose a novel articulatory representation decomposition algorithm that takes the advantage of guided factor analysis to derive the articulatory-specific factors and factor scores. A neural convolutive matrix factorization algorithm is then employed on the factor scores to derive the new gestures and gestural scores. We experiment with the rtMRI corpus that captures the fine-grained vocal tract contours. Both subjective and objective evaluation results suggest that the newly proposed system delivers the articulatory representations that are intelligible, generalizable, efficient and interpretable.
【50】 Accelerating RNN-T Training and Inference Using CTC guidance标题：利用CTC制导加速RNN-T训练和推理链接：https://arxiv.org/abs/2210.16481
作者：Yongqiang Wang,Zhehuai Chen,Chengjian Zheng,Yu Zhang,Wei Han,Parisa Haghani机构：Google备注：submitted to ICASSP 2023摘要：提出了一种基于联合训练的连接主义时态分类（CTC）模型指导的递归神经网络转换器（RNN-T）训练和推理过程的加速方法.我们做出了一个关键假设，即如果编码器嵌入帧被CTC模型分类为空白帧，则对于RNN-T中的所有部分对准或假设，该帧很可能将被对准到空白，并且可以从解码器输入中丢弃该帧。我们还表明，这种帧缩减操作可以应用在编码器的中间，这导致显著加快了RNN-T中的训练和推理。我们进一步表明，CTC对齐，CTC解码器的副产品，也可以用于在训练期间对RNN-T执行格基约简。在Librispeech和SpeechStew任务上对该方法进行了评估.我们证明了所提出的方法能够将RNN-T推理加速2.2倍，同时具有相似或略好的误字率（WER）。摘要：We propose a novel method to accelerate training and inference process of recurrent neural network transducer (RNN-T) based on the guidance from a co-trained connectionist temporal classification (CTC) model. We made a key assumption that if an encoder embedding frame is classified as a blank frame by the CTC model, it is likely that this frame will be aligned to blank for all the partial alignments or hypotheses in RNN-T and it can be discarded from the decoder input. We also show that this frame reduction operation can be applied in the middle of the encoder, which result in significant speed up for the training and inference in RNN-T. We further show that the CTC alignment, a by-product of the CTC decoder, can also be used to perform lattice reduction for RNN-T during training. Our method is evaluated on the Librispeech and SpeechStew tasks. We demonstrate that the proposed method is able to accelerate the RNN-T inference by 2.2 times with similar or slightly better word error rates (WER).
【51】 Learning to Compute the Articulatory Representations of Speech with the MIRRORNET标题：用MIRRORNET学习计算语音的节律表征链接：https://arxiv.org/abs/2210.16454
作者：Yashish M. Siriwardena,Carol Espy-Wilson,Shihab Shamma机构：Institute for Systems Research, University of Maryland College Park, USA摘要：包括人类在内的大多数生物体通过协调和整合感觉信号与运动动作来生存和完成所需的任务。学习这些复杂的感觉运动映射是同时进行的，并且通常是以无监督或半监督的方式进行。本研究探讨了一种受此感觉运动学习范式启发的自动编码器架构（MirrorNet），以学习如何控制发音合成器。该合成器将由六个声道变量（TV）和源特征（发声指示符和音高）组成的控制信号作为输入，并生成相应的声谱图。由于合成器的非线性结构，产生目标语音信号的控制参数不容易计算，也不总是唯一的。在这里，我们演示如何初始化MirrorNet学习，以便产生有意义的发音值范围。经过训练后，MirrorNet成功地估计出合成任意语音所需的TV和源特征。在Wisconsin X射线微束（XRMB）数据集上，这种方法的性能优于以前设计的最好的“语音反演”系统。摘要：Most organisms including humans function by coordinating and integrating sensory signals with motor actions to survive and accomplish desired tasks. Learning these complex sensorimotor mappings proceeds simultaneously and often in an unsupervised or semi-supervised fashion. An autoencoder architecture (MirrorNet) inspired by this sensorimotor learning paradigm is explored in this work to learn how to control an articulatory synthesizer. The synthesizer takes as input control signals consisting of six vocal Tract Variables (TVs) and source features (voicing indicators and pitch), and generates the corresponding auditory spectrograms. Due to the non-linear structure of the synthesizer, the control parameters that produce a target speech signal are not readily computable nor are they always unique. Here we demonstrate how to initialize the MirrorNet learning so as to produce a meaningful range of articulatory values. Once trained, the MirrorNet successfully estimates the TVs and source features needed to synthesize any arbitrary speech utterance. This approach outperforms the best previously designed `speech inversion' systems on the Wisconsin X-ray microbeam (XRMB) dataset.
【52】 The Secret Source : Incorporating Source Features to Improve Acoustic-to-Articulatory Speech Inversion标题：秘密来源：结合声源特征改善声学到节律性语音倒置链接：https://arxiv.org/abs/2210.16450
作者：Yashish M. Siriwardena,Carol Espy-Wilson机构：Institute for Systems Research, University of Maryland College Park, USA摘要：在这项工作中，我们将声学衍生的源特征、非周期性、周期性和基音作为附加目标引入到声学到发音语音反演（SI）系统中。本文还提出了一种基于时域卷积的SI系统，该系统利用听觉频谱图作为输入语音的表示，学习源语音和声道之间的长距离依赖关系和复杂的相互作用，从而提高SI的性能。实验使用Wisconsin X射线微束（XRMB）和Haskins生产率比较（HPRC）数据集进行，并相对于三个基线SI模型架构进行比较。当源特征被用作附加目标时，所提出的具有HPRC数据集的SI系统获得了接近28%的改进。在XRMB数据集上，相同的SI系统比当前性能最好的SI模型的性能高出约9%。摘要：In this work, we incorporated acoustically derived source features, aperiodicity, periodicity and pitch as additional targets to an acoustic-to-articulatory speech inversion (SI) system. We also propose a Temporal Convolution based SI system, which uses auditory spectrograms as the input speech representation, to learn long-range dependencies and complex interactions between the source and vocal tract, to improve the SI task. The experiments are conducted with both the Wisconsin X-ray microbeam (XRMB) and Haskins Production Rate Comparison (HPRC) datasets, with comparisons done with respect to three baseline SI model architectures. The proposed SI system with the HPRC dataset gains an improvement of close to 28% when the source features are used as additional targets. The same SI system outperforms the current best performing SI models by around 9% on the XRMB dataset.
【53】 Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention标题：具有自适应视听注意的视觉感知音频字幕链接：https://arxiv.org/abs/2210.16428
作者：Xubo Liu,Qiushi Huang,Xinhao Mei,Haohe Liu,Qiuqiang Kong,Jianyuan Sun,Shengchen Li,Tom Ko,Yu Zhang,Lilian H. Tang,Mark D. Plumbley,Volkan Kılıç,Wenwu Wang机构：University of Surrey, UK, ByteDance, China, Xi’an Jiaotong-Liverpool University, China, Southern University of Science and Technology, China, Izmir Katip Celebi University, Turkey备注：Submitted to ICASSP 2023摘要：音频字幕是生成描述音频剪辑内容的字幕的任务。在现实世界中，许多对象会产生类似的声音。仅利用音频信息难以识别这些听觉模糊声音事件。如何准确地识别歧义声音是音频字幕系统面临的主要挑战。受人类视听多模态感知的启发，本文提出了视觉感知的音频字幕，利用视觉信息帮助识别模糊的发声对象。具体地说，我们引入了现成的视觉编码器来处理视频输入，并将提取的视觉特征融入到音频字幕系统中。此外，为了更好地利用冗余音视频流中的互补上下文，我们提出了一种音视频注意机制，该机制根据音频和视频信息的置信度自适应地集成音频和视频信息。在最大的音频字幕数据集AudioCaps上的实验结果表明，与现有的强基线音频字幕系统相比，该方法取得了显著的改进，与现有技术的结果相当。摘要：Audio captioning is the task of generating captions that describe the content of audio clips. In the real world, many objects produce similar sounds. It is difficult to identify these auditory ambiguous sound events with access to audio information only. How to accurately recognize ambiguous sounds is a major challenge for audio captioning systems. In this work, inspired by the audio-visual multi-modal perception of human beings, we propose visually-aware audio captioning, which makes use of visual information to help the recognition of ambiguous sounding objects. Specifically, we introduce an off-the-shelf visual encoder to process the video inputs, and incorporate the extracted visual features into an audio captioning system. Furthermore, to better exploit complementary contexts from redundant audio-visual streams, we propose an audio-visual attention mechanism that integrates audio and visual information adaptively according to their confidence levels. Experimental results on AudioCaps, the largest publicly available audio captioning dataset, show that the proposed method achieves significant improvement over a strong baseline audio captioning system and is on par with the state-of-the-art result.
eess.AS音频处理的前53篇论文与cs.SD 方向53的篇一致，因篇幅有限不再赘述，下面是单独不一样的论文：
【54】 BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model标题：BERT会见CTC：使用预训练掩蔽语言模型的端到端语音识别新公式链接：https://arxiv.org/abs/2210.16663
作者：Yosuke Higuchi,Brian Yan,Siddhant Arora,Tetsuji Ogawa,Tetsunori Kobayashi,Shinji Watanabe机构：Carnegie Mellon Univeristy,Waseda University备注：Accepted to Findings of EMNLP2022摘要：提出了一种新的端到端语音识别算法BERT-CTC，该算法将BERT算法应用于连接主义时态分类（CTC）。我们的公式放宽了传统CTC中使用的条件独立性假设，并通过BERT上下文嵌入获得的显式输出依赖关系来融入语言知识。BERT-CTC通过自注意机制关注输入和假设输出序列的全部上下文。该机制鼓励模型学习音频和令牌表示之间的内部/相互依赖性，同时保持CTC的训练效率。在推理过程中，BERT-CTC将屏蔽预测算法与CTC解码相结合，迭代地细化输出序列。实验结果表明，BERT-CTC在不同的说话风格和语言方面都优于传统的方法。最后，我们证明BERT-CTC中的语义表示对后续的口语理解任务是有益的。摘要：This paper presents BERT-CTC, a novel formulation of end-to-end speech recognition that adapts BERT for connectionist temporal classification (CTC). Our formulation relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through the explicit output dependency obtained by BERT contextual embedding. BERT-CTC attends to the full contexts of the input and hypothesized output sequences via the self-attention mechanism. This mechanism encourages a model to learn inner/inter-dependencies between the audio and token representations while maintaining CTC's training efficiency. During inference, BERT-CTC combines a mask-predict algorithm with CTC decoding, which iteratively refines an output sequence. The experimental results reveal that BERT-CTC improves over conventional approaches across variations in speaking styles and languages. Finally, we show that the semantic representations in BERT-CTC are beneficial towards downstream spoken language understanding tasks.
机器翻译，仅供参考
每日语音速递