今天跟大家分享一篇语音相关的论文合集：cs.SD语音17篇，eess.AS音频处理20篇。
【1】 Pay Attention to Hard Trials
标题：重视艰苦的考验
链接：https://arxiv.org/abs/2209.04687
作者：Lantian Li,Di Wang,Dong Wang机构：∗ Center for Speech and Language Technologies, BNRist, Tsinghua University, China, † School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China摘要：说话人识别系统的性能是在测试试验中评估的。尽管试验就像裁缝的尺子一样重要，但迄今为止还没有得到仔细的处理，大多数现有的基准测试都是通过简单的交叉配对来组成试验的。在本文中，我们认为交叉配对方法产生了压倒性的容易的试验，这反过来又导致了系统和技术比较中的潜在偏倚。解决问题，我们提倡更加注重硬审。我们提出了一种基于支持向量机的方法来识别硬试验，并使用它来构造新的评估集VoxCeleb 1和SITW。有了这些新的数据集，我们可以重新评估一些最新技术的贡献。代码和确定的困难试验将在线发布在www.example.com上http://project.cslt.org。摘要：Performance of speaker recognition systems is evaluated on test trials. Although as crucial as rulers for tailors, trials have not been carefully treated so far, and most existing benchmarks compose trials by naive cross-pairing. In this paper, we argue that the cross-pairing approach produces overwhelming easy trials, which in turn leads to potential bias in system and technique comparison. To solve the problem, we advocate more attention to hard trials. We present an SVM-based approach to identifying hard trials and use it to construct new evaluation sets for VoxCeleb1 and SITW. With the new sets, we can re-evaluate the contribution of some recent technologies. The code and the identified hard trials will be published online at http://project.cslt.org.【2】 Defend Data Poisoning Attacks on Voice Authentication
标题：防御语音认证中的数据中毒攻击
链接：https://arxiv.org/abs/2209.04547
作者：Ke Li,Cameron Baird,Dan Lin摘要：随着深度学习技术的发展，说话人验证已经达到了非常高的准确性，并且作为一种生物特征认证选项在我们日常生活的许多场景中越来越受欢迎，特别是在不断增长的网络服务市场中。与传统密码相比，“语音密码”方便得多，因为它减轻了人们记忆不同密码的负担。然而，新的机器学习攻击正在使这些语音认证系统面临风险。在没有强有力的安全保证的情况下，攻击者可以通过欺骗基于深度神经网络（DNN）的语音识别模型来访问合法用户的Web帐户。针对语音认证系统，提出了一种简单易行的数据中毒攻击方法，该方法很难被现有的防御机制所捕获。因此，我们提出了一种更鲁棒的防御方法，称为Guardian，它是一种基于卷积神经网络的鉴别器。Guardian鉴别器集成了一系列新颖的技术，包括偏差减小、输入增强和集成学习。我们的方法能够区分大约95%的被攻击账户和正常账户，这比现有的只有60%准确率的方法要有效得多。摘要：With the advances in deep learning, speaker verification has achieved very high accuracy and is gaining popularity as a type of biometric authentication option in many scenes of our daily life, especially the growing market of web services. Compared to traditional passwords, "vocal passwords" are much more convenient as they relieve people from memorizing different passwords. However, new machine learning attacks are putting these voice authentication systems at risk. Without a strong security guarantee, attackers could access legitimate users' web accounts by fooling the deep neural network (DNN) based voice recognition models. In this paper, we demonstrate an easy-to-implement data poisoning attack to the voice authentication system, which can hardly be captured by existing defense mechanisms. Thus, we propose a more robust defense method, called Guardian, which is a convolutional neural network-based discriminator. The Guardian discriminator integrates a series of novel techniques including bias reduction, input augmentation, and ensemble learning. Our approach is able to distinguish about 95% of attacked accounts from normal accounts, which is much more effective than existing approaches with only 60% accuracy.【3】 DeID-VC: Speaker De-identification via Zero-shot Pseudo Voice Conversion
标题：DEID-VC：基于零点伪音转换的说话人识别
链接：https://arxiv.org/abs/2209.04530
作者：Ruibin Yuan,Yuxuan Wu,Jacob Li,Jaxter Kim机构：Carnegie Mellon University, United States备注：Accepted by Interspeech 2022摘要：基于语音的在线服务的广泛采用引起了关于它们使用和共享的数据的安全和隐私问题。如果数据被破坏，攻击者可以利用用户的语音绕过说话者验证系统，甚至冒充用户。为了缓解这一问题，我们提出了DeID-VC，一个说话人去识别系统，它将真实说话人转换为伪说话人，从而从语音中去除或混淆说话人相关的属性。DeID-VC的关键组件包括基于变分自动编码器（VAE）的伪说话人生成器（PSG）和zero-shot设置下的语音转换自动编码器（AE）。在PSG的帮助下，DeID-VC可以在说话人级别甚至在话语级别分配唯一的伪说话人。此外，加入了两个新的学习目标，以弥补zero-shot语音转换的训练和推理之间的差距。我们给出了我们的实验结果，字错误率（WER）和等错误率（EER），以及三个主观指标来评估DeID-VC生成的输出。结果表明，与我们的基线相比，我们的方法大大提高了可懂度（WER降低10%）和去识别有效性（EER提高5%）。代码和收听演示：https://github.com/a43992899/DeID-VC摘要：The widespread adoption of speech-based online services raises security and privacy concerns regarding the data that they use and share. If the data were compromised, attackers could exploit user speech to bypass speaker verification systems or even impersonate users. To mitigate this, we propose DeID-VC, a speaker de-identification system that converts a real speaker to pseudo speakers, thus removing or obfuscating the speaker-dependent attributes from a spoken voice. The key components of DeID-VC include a Variational Autoencoder (VAE) based Pseudo Speaker Generator (PSG) and a voice conversion Autoencoder (AE) under zero-shot settings. With the help of PSG, DeID-VC can assign unique pseudo speakers at speaker level or even at utterance level. Also, two novel learning objectives are added to bridge the gap between training and inference of zero-shot voice conversion. We present our experimental results with word error rate (WER) and equal error rate (EER), along with three subjective metrics to evaluate the generated output of DeID-VC. The result shows that our method substantially improved intelligibility (WER 10% lower) and de-identification effectiveness (EER 5% higher) compared to our baseline. Code and listening demo: https://github.com/a43992899/DeID-VC【4】 Modeling Dependent Structure for Utterances in ASR Evaluation
标题：ASR评价中话语依赖结构的建模
链接：https://arxiv.org/abs/2209.05281
机构：Meta AI, Menlo Park, CA, USA摘要：在自动语音识别（ASR）评测中，Bootstrap重采样方法被广泛用于对误词率（WER）进行显著性分析。为了解决语音数据的相关性问题，提出了分块自举方法，通过将语音数据划分为不相关的块，对这些块进行重采样，而不是对原始数据进行重采样。然而，揭示话语间的依存结构往往是一件很不容易的事情，这可能会导致统计测试中的主观发现。本文提出了一种基于图形套索的方法来显式地建模这种依赖性，并以严格的方式估计独立的话语块。然后，在推断的块的顶部应用逐块引导。我们证明了所得WER的方差估计在温和条件下是一致的。在LibriSpeech语料库上验证了该方法的有效性。摘要：The bootstrap resampling method has been popular for performing significance analysis on word error rate (WER) in automatic speech recognition (ASR) evaluations. To deal with the issue of dependent speech data, the blockwise bootstrap approach is also proposed that by dividing utterances into uncorrelated blocks, it resamples these blocks instead of original data. However, it is always nontrivial to uncover the dependent structure among utterances, which could lead to subjective findings in statistical testing. In this paper, we present graphical lasso based methods to explicitly model such dependency and estimate the independent blocks of utterances in a rigorous way. Then the blockwise bootstrap is applied on top of the inferred blocks. We show that the resulting variance estimator for WER is consistent under mild conditions. We also demonstrate the validity of proposed approach on LibriSpeech data.【5】 Continuous head-related transfer function representation based on hyperspherical harmonics
标题：基于超球面调和的头部相关传递函数连续表示
链接：https://arxiv.org/abs/2209.05110
机构：AGH University of Science and Technology, Department of Robotics and Mechatronics备注：Submitted to Archives of Acoustics 4.06.2022摘要：在球谐域中表达头部相关传递函数（HRTF）是一种获得空间连续性的方法。然而，HRTF不仅是方向的函数，而且是频率的函数。本文提出了一种基于超球谐函数的扩展方法，利用超球谐函数（HSHs）获得在空间和频率上都连续的HRTF表示。HSH近似的应用导致相对小的系数集合，其可以在任何方向和频率上被解码为HRTF值。本文讨论了通过将该方法应用于从示例性HRTF测量提取的幅度谱而获得的结果。基于SH和HSH的HRTF表示具有相似的再现精度，而后者具有在空间和频率上的连续性，并且需要更少的系数。所开发的基于HSH的连续函数模型可以用于多种目的，例如机器学习应用的插值、压缩或参数化。摘要：Expressing head-related transfer functions (HRTFs) in spherical harmonic (SH) domain has been thoroughly studied as a method of obtaining continuity over space. However, HRTFs are functions not only of direction but also of frequency. This paper presents an extension of the SH-based method, utilizing hyperspherical harmonics (HSHs) to obtain an HRTF representation that is continuous over both space and frequency. The application of the HSH approximation results in a relatively small set of coefficients which can be decoded into HRTF values at any direction and frequency. The paper discusses results obtained by applying the method to magnitude spectra extracted from exemplary HRTF measurements. The HRTF representations based on SHs and HSHs exhibit similar reproduction accuracy, with the latter one featuring continuity over both space and frequency and requiring much lower number of coefficients. The developed HSH-based continuous functional model can serve multiple purposes, such as interpolation, compression or parametrization for machine-learning applications.【6】 VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition
标题：VarArray与t-SOT相遇：推动远程语音识别流技术的发展
链接：https://arxiv.org/abs/2209.04974
作者：Naoyuki Kanda,Jian Wu,Xiaofei Wang,Zhuo Chen,Jinyu Li,Takuya Yoshioka机构：Microsoft, One Microsoft Way, Redmond, WA, USA摘要：提出了一种新的流式自动语音识别（ASR）框架，用于处理由任意几何形状的麦克风阵列捕获的多人重叠语音.我们的框架，命名为t-SOT-VA，利用了独立开发的两项最新技术：阵列几何不可知的连续语音分离，或VarArray，以及基于令牌级串行化输出训练（t-SOT）的流式多说话者ASR。为了结合这两种技术的优点，我们新设计了一种基于t-SOT的ASR模型，该模型基于来自VarArray的两个分离的语音信号生成串行化的多说话者转录。我们还提出了一种针对这种ASR模型的预训练方案，其中，我们基于单耳单说话者ASR训练数据来模拟VarArray的输出信号。在AMI会议语料库上进行的会话转录实验表明，基于该框架的系统性能明显优于传统系统。在多个远距离麦克风设置中，我们的系统对于AMI开发集和评估集分别实现了13.7%和15.5%的最先进的字错误率，同时保留了流推理能力。摘要：This paper presents a novel streaming automatic speech recognition (ASR) framework for multi-talker overlapping speech captured by a distant microphone array with an arbitrary geometry. Our framework, named t-SOT-VA, capitalizes on independently developed two recent technologies; array-geometry-agnostic continuous speech separation, or VarArray, and streaming multi-talker ASR based on token-level serialized output training (t-SOT). To combine the best of both technologies, we newly design a t-SOT-based ASR model that generates a serialized multi-talker transcription based on two separated speech signals from VarArray. We also propose a pre-training scheme for such an ASR model where we simulate VarArray's output signals based on monaural single-talker ASR training data. Conversation transcription experiments using the AMI meeting corpus show that the system based on the proposed framework significantly outperforms conventional ones. Our system achieves the state-of-the-art word error rates of 13.7% and 15.5% for the AMI development and evaluation sets, respectively, in the multiple-distant-microphone setting while retaining the streaming inference capability.【7】 Reconstructing the Dynamic Directivity of Unconstrained Speech
标题：重构无约束语音的动态指向性
链接：https://arxiv.org/abs/2209.04473
作者：Camille Noufi,Dejan Markovic,Peter Dodds备注：19 pages, 8 figures, 3 tables. Internally-reviewed manuscript approved for public release by Facebook Inc. Researched uses proprietary dataset not available for public release摘要：自然 语音 指向 性 的 精确 模型 是 在 虚拟 通信 环境 中 实现 真实 声音 存在 的 重要 步骤 . 本文 提出 了 一 种 估计 和 重建 自然 、 无 约束 语音 的 空间 能量 分布 模式 的 方法 . 我们 将 我们 的 方法 详细 描述 为 两 个 阶段 . 使用 真实 、 静态 麦克风 阵列 捕获 的 语音 记录 ，我们 创建 了 一 个 随 时间 跟踪 说话 者 运动 的 虚拟 阵列 。我们 使用 这个 自我 中心 的 虚拟 阵列 来 测量 和 编码 语音 信号 的 高 分辨率 方向 性 图案 ， 因为 它 随着 自然 语音 和 运动 而 动态 演变 。利用 这个 编码 的 方向 性 表示 ，我们 训练 一 个 机器 学习 模型 ， 当 给定 一 组 有限 的 语音 信号 时 ， 该 模型 用于 估计 完整 的 、 动态 的 方向 性 模式 ， 就 像 使用 头 戴 式 显示 器 上 的 麦克风 记录 语音 时 的 情况 一样我们 研究 了 各种 模型 体系 结构 和 训练 范例 ，和 实用 性 ， 结果 表明 ， 神经 网络 可以 从 有限 的 语音 信息 回归 到 精确 的 、 动态 的 全 方向 性 图 的 估计 。摘要：An accurate model of natural speech directivity is an important step toward achieving realistic vocal presence within a virtual communication setting. In this article, we propose a method to estimate and reconstruct the spatial energy distribution pattern of natural, unconstrained speech. We detail our method in two stages. Using recordings of speech captured by a real, static microphone array, we create a virtual array that tracks with the movement of the speaker over time. We use this egocentric virtual array to measure and encode the high-resolution directivity pattern of the speech signal as it dynamically evolves with natural speech and movement. Utilizing this encoded directivity representation, we train a machine learning model that leverages to estimate the full, dynamic directivity pattern when given a limited set of speech signals, as would be the case when speech is recorded using the microphones on a head-mounted display (HMD). We examine a variety of model architectures and training paradigms, and discuss the utility and practicality of each implementation. Our results demonstrate that neural networks can be used to regress from limited speech information to an accurate, dynamic estimation of the full directivity pattern.【8】 A Semi-Supervised Algorithm for Improving the Consistency of Crowdsourced Datasets: The COVID-19 Case Study on Respiratory Disorder Classification
标题：一种提高众包数据集一致性的半监督算法--以新冠肺炎呼吸系统疾病分类为例
链接：https://arxiv.org/abs/2209.04360
作者：Lara Orlandic,Tomas Teijeiro,David Atienza机构：Embedded Systems Laboratory (ESL), EPFL, Lausanne, Switzerland, Department of Mathematics, University of the Basque Country (UPVEHU), Spain摘要：咳嗽 音频 信号 分类 是 筛查 呼吸 系统 疾病 （ 如 COVID - 19 ） 的 一 种 潜在 有用 工具 。由于 从 患有 此类 传染 病 的 患者 身上 收集 数据 是 危险 的 ， 许多 研究 团队 转向 众 包 来 快速 收集 咳嗽 声音 数据 ， 就 像 生成 COUGHVID 数据 集 一样 。COUGHVID 数据 集 招募 了 专家 医生 来 诊断 有限 数量 的 上传 记录 中 存在 的 基础 疾病 。然而 ， 这种 方法 存在 咳嗽 的 潜在 错误 标记 ， 以及 专家 之间 的 明显 分歧 。在 这项 工作 中 ， 我们 使用 半 监督 学习 （ SSL ） 方法 来 提高 COUGHVID 数据 集 的 标签 一致 性 ， 以及 COVID - 19 与 健康 咳嗽 音 分类 的 鲁 棒 性 。首先 ， 我们 利用 现有 的 SSL 专家 知识 聚合 技术 来 克服 数据 集中 标签 的 不 一致 性 和 稀疏 性 。接下来 ， 我们 的 SSL 方法 被 用于 识别 重新 标记 的 COUGHVID 音频 样本 的 子 样本 ， 其 可以 被 用于 训练 或 增强 未来 的 咳嗽 分类 模型 。尽管 原始 数据 集中 存在 专家 标签 不 一致 性 ， 但 重新 标记 数据 的 一致 性 表现 为 高度 的 类 可分性 ， 比 用户 标记 数据 的 类 可分性 高 3 倍 。此外 ， 用户 标记 的 音频 片段 中 的 频谱 差异 在 重新 标记 的 数据 中 被 放大 ， 导致 健康 咳嗽 和 COVID - 19 咳嗽 之间 的 功率 谱 密度 显著 不同 ， 这 既 证明 了 新 数据 集 的 一致 性 增强 ， 也 证明 了 其 从 声学 角度 的 可 解释 性 。最 后 ， 我们 演示 了 如何 使用 重新 标记 的 数据 集 来 训练 咳嗽 分类 器 。该 SSL 方法 可 用于 组合 多 个 专家 的 医学 知识 ， 以 提高 任何 诊断 分类 任务 的 数据 库 一致 性 。摘要：Cough audio signal classification is a potentially useful tool in screening for respiratory disorders, such as COVID-19. Since it is dangerous to collect data from patients with such contagious diseases, many research teams have turned to crowdsourcing to quickly gather cough sound data, as it was done to generate the COUGHVID dataset. The COUGHVID dataset enlisted expert physicians to diagnose the underlying diseases present in a limited number of uploaded recordings. However, this approach suffers from potential mislabeling of the coughs, as well as notable disagreement between experts. In this work, we use a semi-supervised learning (SSL) approach to improve the labeling consistency of the COUGHVID dataset and the robustness of COVID-19 versus healthy cough sound classification. First, we leverage existing SSL expert knowledge aggregation techniques to overcome the labeling inconsistencies and sparsity in the dataset. Next, our SSL approach is used to identify a subsample of re-labeled COUGHVID audio samples that can be used to train or augment future cough classification models. The consistency of the re-labeled data is demonstrated in that it exhibits a high degree of class separability, 3x higher than that of the user-labeled data, despite the expert label inconsistency present in the original dataset. Furthermore, the spectral differences in the user-labeled audio segments are amplified in the re-labeled data, resulting in significantly different power spectral densities between healthy and COVID-19 coughs, which demonstrates both the increased consistency of the new dataset and its explainability from an acoustic perspective. Finally, we demonstrate how the re-labeled dataset can be used to train a cough classifier. This SSL approach can be used to combine the medical knowledge of several experts to improve the database consistency for any diagnostic classification task.【9】 Overlapped speech and gender detection with WavLM pre-trained features
标题：利用WavLM预训练特征进行重叠语音和性别检测
链接：https://arxiv.org/abs/2209.04167
作者：Martin Lebourdais,Marie Tahon,Antoine Laurent,Sylvain Meignier机构：LIUM, Le Mans Universit´e, France备注：Submitted and accepted to Interspeech 2022摘要：本文以重叠言语和性别检测为研究对象，研究法国视听媒体中男女互动的情况（性别平等监测项目）。在这种应用背景下，我们需要根据说话者的性别自动分割语音信号，并且识别何时至少有两个说话者同时说话。本文提出利用WavLM模型在大量语音数据上预先训练的优点，建立重叠语音检测（OSD）和性别检测（GD）系统。在本研究中，我们使用了两个不同的语料库。DIHARD III语料库很好地适应了OSD任务，但缺乏性别信息。ALLIES语料库符合项目应用背景。我们最好的OSD系统是一个时间卷积网络（TCN）与WavLM预训练的特征作为输入，它达到了一个新的最先进的F1成绩的性能在DIHARD。在法国广播新闻ALLIES数据的性别平衡子集上，用WavLM输入训练神经GD，并获得97.9%的准确率。这项工作为人文科学研究人员关于法国媒体中女性和男性代表性差异的新视角。摘要：This article focuses on overlapped speech and gender detection in order to study interactions between women and men in French audiovisual media (Gender Equality Monitoring project). In this application context, we need to automatically segment the speech signal according to speakers gender, and to identify when at least two speakers speak at the same time. We propose to use WavLM model which has the advantage of being pre-trained on a huge amount of speech data, to build an overlapped speech detection (OSD) and a gender detection (GD) systems. In this study, we use two different corpora. The DIHARD III corpus which is well adapted for the OSD task but lack gender information. The ALLIES corpus fits with the project application context. Our best OSD system is a Temporal Convolutional Network (TCN) with WavLM pre-trained features as input, which reaches a new state-of-the-art F1-score performance on DIHARD. A neural GD is trained with WavLM inputs on a gender balanced subset of the French broadcast news ALLIES data, and obtains an accuracy of 97.9%. This work opens new perspectives for human science researchers regarding the differences of representation between women and men in French media.【10】 MATT: A Multiple-instance Attention Mechanism for Long-tail Music Genre Classification
标题：Matt：一种长尾音乐体裁分类的多实例注意机制
链接：https://arxiv.org/abs/2209.04109
作者：Xiaokai Liu,Menghua Zhang机构：∗School of Cyber Science and Engineering, § School of Energy and Power Engineering, Huazhong University of Science and Technology, Wuhan, China摘要：不平衡音乐类型分类是音乐信息检索领域的一项重要任务，它是基于相关音乐音频片段识别长尾、数据贫乏的音乐类型，在现实生活中非常普遍。现有的音乐分类模型大多针对类别均衡的音乐数据集，在识别分布尾部的音乐类型时，其准确性和泛化能力较差.受多实例学习（Multi-instance Learning，MIL）在各类分类任务中成功应用的启发，提出了一种新的多实例注意（Multi-instance Attention，MATT）机制来提高尾类识别的性能.具体地说，我们首先通过生成专辑-艺术家对包来构造包级数据集。第二，我们利用神经网络来编码音乐音频片段。最后，在多实例注意机制的指导下，神经网络模型可以选择信息量最大的流派来匹配给定的音乐片段。在大规模长尾分布音乐流派基准数据集上的综合实验结果表明，MATT算法的性能明显优于其他现有的基准算法.摘要：Imbalanced music genre classification is a crucial task in the Music Information Retrieval (MIR) field for identifying the long-tail, data-poor genre based on the related music audio segments, which is very prevalent in real-world scenarios. Most of the existing models are designed for class-balanced music datasets, resulting in poor performance in accuracy and generalization when identifying the music genres at the tail of the distribution. Inspired by the success of introducing Multi-instance Learning (MIL) in various classification tasks, we propose a novel mechanism named Multi-instance Attention (MATT) to boost the performance for identifying tail classes. Specifically, we first construct the bag-level datasets by generating the album-artist pair bags. Second, we leverage neural networks to encode the music audio segments. Finally, under the guidance of a multi-instance attention mechanism, the neural network-based models could select the most informative genre to match the given music segment. Comprehensive experimental results on a large-scale music genre benchmark dataset with long-tail distribution demonstrate MATT significantly outperforms other state-of-the-art baselines.【11】 Learning Audio-Visual embedding for Wild Person Verification
标题：基于学习视听嵌入的野人身份验证
链接：https://arxiv.org/abs/2209.04093
作者：Peiwen Sun,Shanshan Zhang,Zishan Liu,Yougen Yuan,Taotao Zhang,Honggang Zhang,Pengfei Hu机构：School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China, TEG AI, Tencent Inc, Beijing, China摘要：已经观察到，可以从这两种模态中提取音频-视频嵌入，以获得用于个人验证的鲁棒性。然而，用于从每个帧生成单个话语表示的聚合器似乎没有被很好地探索。在本文中，我们提出了一个从融合的角度考虑聚合器的视听网络。在人脸验证中首次引入了改进的注意统计池算法。然后发现在合并过程中模态间存在很强的相关性，提出了一种包含循环一致性的联合注意力合并算法来学习隐含的帧间权重。最后，将模态与门控注意机制相融合。在VoxCeleb 2 dev数据集上对所提出的模型进行了训练，最佳系统在VoxCeleb 1的三个官方测试列表上分别获得了0.18%，0.27%和0.49%的EER，这是我们所知道的发表的最好的个人验证结果.作为分析，生成可视化图以解释该系统如何在模态之间交互。摘要：It has already been observed that audio-visual embedding can be extracted from these two modalities to gain robustness for person verification. However, the aggregator that used to generate a single utterance representation from each frame does not seem to be well explored. In this article, we proposed an audio-visual network that considers aggregator from a fusion perspective. We introduced improved attentive statistics pooling for the first time in face verification. Then we find that strong correlation exists between modalities during pooling, so joint attentive pooling is proposed which contains cycle consistency to learn the implicit inter-frame weight. Finally, fuse the modality with a gated attention mechanism. All the proposed models are trained on the VoxCeleb2 dev dataset and the best system obtains 0.18\%, 0.27\%, and 0.49\% EER on three official trail lists of VoxCeleb1 respectively, which is to our knowledge the best-published results for person verification. As an analysis, visualization maps are generated to explain how this system interact between modalities.【12】 Prediction method of Soundscape Impressions using Environmental Sounds and Aerial Photographs
标题：利用环境声和航拍照片预测声景观印象的方法
链接：https://arxiv.org/abs/2209.04077
作者：Yusuke Ono,Sunao Hara,Masanobu Abe机构：∗ Okayama University, Okayama, Japan备注：Submitted APSIPA ASC 2022摘要：我们研究了一种基于声音环境印象来量化城市特征的方法。本研究利用云传感法采集的声音数据，对城市的两种声景印象--愉悦感和事件感进行了预测。所收集的声音包括使用全球定位系统的记录位置的元信息。此外，通过使用瑞典声景质量协议（Swedish Soundscape-Quality Protocol）定义的评估，将声景印象和声源特征分别分配给云感测声音，从而评估声学环境的质量。利用多层感知器的深度神经网络对10秒的声音输入和其位置的航拍照片建立预测模型。声学特征包括等效噪声级和倍频程滤波器每秒的输出，以及它们在10~s内的统计。使用ResNet-50和自动编码器架构从航空照片中提取图像特征。我们进行了对比实验来演示每种特性的优势。作为比较的结果，航空照片和声源特征对于预测印象信息是有效的。另外，即使使用声学和图像特征来预测声源特征，这些特征也显示出良好的结果，以预测接近于甲骨文声源特征的结果的声景印象。摘要：We investigate an method for quantifying city characteristics based on impressions of a sound environment. The quantification of the city characteristics will be beneficial to government policy planning, tourism projects, etc. In this study, we try to predict two soundscape impressions, meaning pleasantness and eventfulness, using sound data collected by the cloud-sensing method. The collected sounds comprise meta information of recording location using Global Positioning System. Furthermore, the soundscape impressions and sound-source features are separately assigned to the cloud-sensing sounds by assessments defined using Swedish Soundscape-Quality Protocol, assessing the quality of the acoustic environment. The prediction models are built using deep neural networks with multi-layer perceptron for the input of 10-second sound and the aerial photographs of its location. An acoustic feature comprises equivalent noise level and outputs of octave-band filters every second, and statistics of them in 10~s. An image feature is extracted from an aerial photograph using ResNet-50 and autoencoder architecture. We perform comparison experiments to demonstrate the benefit of each feature. As a result of the comparison, aerial photographs and sound-source features are efficient to predict impression information. Additionally, even if the sound-source features are predicted using acoustic and image features, the features also show fine results to predict the soundscape impression close to the result of oracle sound-source features.【13】 Improving the Environmental Perception of Autonomous Vehicles using Deep Learning-based Audio Classification
标题：基于深度学习的音频分类方法改善自主车辆的环境感知
链接：https://arxiv.org/abs/2209.04075
作者：Finley Walden,Sagar Dasgupta,Mizanur Rahman,Mhafuzul Islam机构：Undergraduate Student, Department of Computer Science, The University of Alabama, Cyber Hall, Box , Kirkbride Lane, Tuscaloosa, AL , Ph.D. Student, Department of Civil, Construction & Environmental Engineering, Assistant Professor, Senior Research Scientist摘要：听觉 是 自动 驾驶 车辆 感知 周围 环境 的 关键 。虽然 AV 的 视觉 传感 器 ， 如 摄像 机 、 激光 雷达 和 雷达 ， 有助于 看到 其 周围 环境 ， 但 AV 不能 看到 这些 传感 器 的 视线 之外 。另 一方面 ， AV 的 听觉 不能 被 视线 所 阻碍 。例如 ， 即使 紧急 车辆 不在 AV 的 视线 内 ， AV 也 可以 通过 音频 分类 来 识别 紧急 车辆 的 警笛 。因此 ， 听觉 感知 是 对 相机 、 激光 雷达 和 基于 雷达 的 感知 系统 的 补充 。提出 了 一 种 基于 深度 学习 的 鲁 棒 音频 分类 框架 ， 旨在 提高 AV 的 环境 感知 能力 。所 提出 的 框架 利用 深度 卷积 神经 网络 （ CNN ） 来 分类 不同 的 音频 类别 。使用 城市 环境 数据 集 UrbanSound8k 对 所 开发 的 框架 进行 训练 和 测试 。七 个 音频 类 ，空调 、 汽车 喇叭 、 儿童 玩耍 、 狗 叫 、 引擎 空转 、 枪声 和 警笛 ， 由于 它们 与 AV 的 相关 性 而 从 UrbanSound8k 数据 集中 识别 。我们 的 框架 可以 对 不同 的 音频 类别 进行 分类 ， 准确 率 为 97.82% 。此外 ， 给出 了 所有 10 类 的 音频 分类 准确 率 ， 这 证明 了 与 现有 的 音频 分类 框架 相比 ， 我们 的 框架 在 AV 相关 声音 的 情况 下 表现 得 更 好 。摘要：Sense of hearing is crucial for autonomous vehicles (AVs) to better perceive its surrounding environment. Although visual sensors of an AV, such as camera, lidar, and radar, help to see its surrounding environment, an AV cannot see beyond those sensors line of sight. On the other hand, an AV s sense of hearing cannot be obstructed by line of sight. For example, an AV can identify an emergency vehicle s siren through audio classification even though the emergency vehicle is not within the line of sight of the AV. Thus, auditory perception is complementary to the camera, lidar, and radar-based perception systems. This paper presents a deep learning-based robust audio classification framework aiming to achieve improved environmental perception for AVs. The presented framework leverages a deep Convolution Neural Network (CNN) to classify different audio classes. UrbanSound8k, an urban environment dataset, is used to train and test the developed framework. Seven audio classes i.e., air conditioner, car horn, children playing, dog bark, engine idling, gunshot, and siren, are identified from the UrbanSound8k dataset because of their relevancy related to AVs. Our framework can classify different audio classes with 97.82% accuracy. Moreover, the audio classification accuracies with all ten classes are presented, which proves that our framework performed better in the case of AV-related sounds compared to the existing audio classification frameworks.【14】 Audio Analytics-based Human Trafficking Detection Framework for Autonomous Vehicles
标题：基于音频分析的自动车辆人口贩运检测框架
链接：https://arxiv.org/abs/2209.04071
作者：Sagar Dasgupta,Kazi Shakib,Mizanur Rahman,Silvana V Croope,Steven Jones机构：Ph.D. Student, The University of Alabama, Cyber Hall, Box , Kirkbride Lane, Tuscaloosa, AL , Assistant Professor, Resilience Research Engineer, Alabama Transportation Institute, Director, Transportation Policy Research Center摘要：人口 贩运 是 一 个 普遍 问题 ， 尽管 在 全球 范围 内 作出 了 许多 努力 来 打击 这 一 问题 ， 但 这 一 问题 依然 存在 。任何 年龄 、 种族 、 民族 、 性别 、 性别 认同 、 性 取向 、 国籍 、 移民 身份 、 文化 背景 、 宗教 、 社会 经济 阶层 和 教育 程度 的 个人 都 可能 成为 人口 贩运 的 受害 者 。随着 技术 的 进步 和 自动 驾驶 车辆 的 引入 ， 人口 贩运 者 将 采用 新 的 方式 运送 受害 者 ， 这 可能 会 加速 有 组织 人口 贩运 网络 的 发展 ， 从而 使 执法 机构 在 侦查 人口 贩运 活动 时 面临 更 大 的 挑战 。本 研究 的 目的 是 开发 一 种 创新 的 基于 音频 分析 的 自动 驾驶 车辆 人口 贩卖 检测 框架 。本 研究 的 主要 贡献 是 ：（ i ） 为 反 车辆 组织 确定 四 种 重要 、 可行 和 现实 的 人口 贩运 情景 ;创建 一 个 新 的 与 人口 贩运 有关 的 综合 音频 数据 集 ， 分为 五 类 ， 即 ：哭泣 、 尖叫 、 车门 撞击 、 汽车 噪音 和 谈话 ;以及 （ iii ） 开发 用于 与 人口 贩运 有关 的 音频 数据 分类 的 深度 1 - D 卷积 神经 网络 （ CNN ） 架构 。我们 还 使用 新 的 音频 数据 集 进行 了 案例 研究 ， 并 评估 了 深度 1 - D CNN 的 音频 分类 性能 。我们 的 分析 表明 ， 深度 一维 CNN 能够 以 95% 的 准确 率 区分 来自 人口 贩运 受害 者 的 声音 和 非 人口 贩运 的 声音 ， 这 证明 了 我们 的 框架 的 有效 性 。摘要：Human trafficking is a universal problem, persistent despite numerous efforts to combat it globally. Individuals of any age, race, ethnicity, sex, gender identity, sexual orientation, nationality, immigration status, cultural background, religion, socioeconomic class, and education can be a victim of human trafficking. With the advancements in technology and the introduction of autonomous vehicles (AVs), human traffickers will adopt new ways to transport victims, which could accelerate the growth of organized human trafficking networks, which can make the detection of trafficking in persons more challenging for law enforcement agencies. The objective of this study is to develop an innovative audio analytics-based human trafficking detection framework for autonomous vehicles. The primary contributions of this study are to: (i) define four non-trivial, feasible, and realistic human trafficking scenarios for AVs; (ii) create a new and comprehensive audio dataset related to human trafficking with five classes i.e., crying, screaming, car door banging, car noise, and conversation; and (iii) develop a deep 1-D Convolution Neural Network (CNN) architecture for audio data classification related to human trafficking. We have also conducted a case study using the new audio dataset and evaluated the audio classification performance of the deep 1-D CNN. Our analyses reveal that the deep 1-D CNN can distinguish sound coming from a human trafficking victim from a non-human trafficking sound with an accuracy of 95%, which proves the efficacy of our framework.【15】 Non-autoregressive Error Correction for CTC-based ASR with Phone-conditioned Masked LM
标题：基于CTC的自动语音识别的非自回归纠错
链接：https://arxiv.org/abs/2209.04062
作者：Hayato Futami,Hirofumi Inaguma,Sei Ueno,Masato Mimura,Shinsuke Sakai,Tatsuya Kawahara机构：Graduate School of Informatics, Kyoto University, Sakyo-ku, Kyoto, Japan备注：Accepted in Interspeech2022摘要：基于连接主义时态分类（CTC）的模型由于其非自回归性质而在自动语音识别（ASR）中具有吸引力。为了充分利用纯文本数据，语言模型（LM）集成方法（如重新评分和浅融合）已被广泛用于CTC。然而，由于需要波束搜索，它们失去了CTC的非自回归性质，这降低了推理速度。在本研究中，我们提出了一种使用音素条件掩蔽LM（PC-MLM）的错误更正方法。在该方法中，对CTC贪婪解码输出中的低置信度单词标记进行屏蔽。然后，PC-MLM在给定从CTC补充预测的未屏蔽单词和音素的情况下预测这些屏蔽单词标记。我们进一步将其扩展为可删除PC-MLM，以解决插入错误。由于CTC和PC-MLM都是非自回归模型，因此该方法能够实现快速LM积分。在CSJ和TED-LIUM 2语料库上的实验结果表明，该方法在推理速度和CSJ上的识别准确率方面均优于重评分和浅融合方法。摘要：Connectionist temporal classification (CTC) -based models are attractive in automatic speech recognition (ASR) because of their non-autoregressive nature. To take advantage of text-only data, language model (LM) integration approaches such as rescoring and shallow fusion have been widely used for CTC. However, they lose CTC's non-autoregressive nature because of the need for beam search, which slows down the inference speed. In this study, we propose an error correction method with phone-conditioned masked LM (PC-MLM). In the proposed method, less confident word tokens in a greedy decoded output from CTC are masked. PC-MLM then predicts these masked word tokens given unmasked words and phones supplementally predicted from CTC. We further extend it to Deletable PC-MLM in order to address insertion errors. Since both CTC and PC-MLM are non-autoregressive models, the method enables fast LM integration. Experimental evaluations on the Corpus of Spontaneous Japanese (CSJ) and TED-LIUM2 in domain adaptation setting shows that our proposed method outperformed rescoring and shallow fusion in terms of inference speed, and also in terms of recognition accuracy on CSJ.【16】 Longitudinal Acoustic Speech Tracking Following Pediatric Traumatic Brain Injury
标题：儿童创伤性脑损伤后的纵向声学语音跟踪
链接：https://arxiv.org/abs/2209.04406
作者：Camille Noufi,Adam C. Lammert,Daryush D. Mehta,James R. Williamson,Gregory Ciccarelli,Douglas Sturim,Jordan R. Green,Thomas F. Quatieri,Thomas F. Campbell机构：Stanford University, Stanford, CA, USA, MIT Lincoln Laboratory, Lexington, MA, USA, Worcester Polytechnic Institute, Worcester, MA, USA, Mass General Hospital, Boston, MA, USA, University of Texas at Dallas, Dallas, TX, USA摘要：儿童创伤性脑损伤（TBI）后常见结局指标的建议支持在恢复和治疗计划中整合仪器测量和感知评估。因此，在评估儿童TBI后语音特征随时间的变化时，一套全面的灵敏、稳健和非侵入性测量是必不可少的。在这篇文章中，我们研究了10名被诊断为严重TBI的儿童队列受试者的声学言语模式的变化。我们从儿童产生可理解的单词后的一年中记录的儿童语音中提取了一组不同的已知和新颖的声学特征。对这些特征进行单独分析，并按语音子系统、受试者内和队列进行分析。作为一个群体，年龄较大的儿童表现出高度显著的（p〈0.01）音高变化和音素多样性增加，停顿长度缩短，发音速率变化稳定。年龄较小的儿童表现出类似的稳定速率变化，同时基于共振峰的发音复杂度增加。对特征集与年龄的相关性分析以及与正常发育数据的比较证实，受伤时的年龄在构建恢复轨迹方面起着重要作用。作为一个整体，几乎所有的语音特征都发生了显著变化（p〈0.05），这证实了需要声学测量补充感知评估，以确定TBI后语音治疗的有效治疗目标。摘要：Recommendations for common outcome measures following pediatric traumatic brain injury (TBI) support the integration of instrumental measurements alongside perceptual assessment in recovery and treatment plans. A comprehensive set of sensitive, robust and non-invasive measurements is therefore essential in assessing variations in speech characteristics over time following pediatric TBI. In this article, we study the changes in the acoustic speech patterns of a pediatric cohort of ten subjects diagnosed with severe TBI. We extract a diverse set of both well-known and novel acoustic features from child speech recorded throughout the year after the child produced intelligible words. These features are analyzed individually and by speech subsystem, within-subject and across the cohort. As a group, older children exhibit highly significant (p<0.01) increases in pitch variation and phoneme diversity, shortened pause length, and steadying articulation rate variability. Younger children exhibit similar steadied rate variability alongside an increase in formant-based articulation complexity. Correlation analysis of the feature set with age and comparisons to normative developmental data confirm that age at injury plays a significant role in framing the recovery trajectory. Nearly all speech features significantly change (p<0.05) for the cohort as a whole, confirming that acoustic measures supplementing perceptual assessment are needed to identify efficacious treatment targets for speech therapy following TBI.【17】 Streaming End-to-End Target Speaker ASR
标题：端到端流传输目标扬声器ASR
链接：https://arxiv.org/abs/2209.04175
作者：Takafumi Moriya,Hiroshi Sato,Tsubasa Ochiai,Marc Delcroix,Takahiro Shinozaki机构：NTT Corporation, Japan; ,Tokyo Institute of Technology, Japan备注：Accepted to Interspeech 2022摘要：尽管深度学习技术的最新进展提高了单说话者情况下的自动语音识别（ASR）性能，但识别多个声音重叠的多说话者语音仍然很困难。解决该问题的一种传统方法是使用语音分离或目标语音提取前端与ASR后端的级联。然而，前端模块的额外计算成本是快速响应的关键障碍，特别是对于流式ASR。本文提出了一种目标说话人ASR（TS-ASR）系统，该系统将目标语音提取功能隐式地集成在流式端到端（E2 E）ASR系统中，即递归神经网络-换能器（RNNT）。我们的系统使用与目标语音提取所采用的类似的思想，但是直接在RNNT的编码器级实现它。这允许实现TS-ASR，而无需在前端上放置额外的计算成本。注意，本研究呈现了关于E2 E TS-ASR的先前研究之间的两个主要差异;我们研究了流模型，并将我们的研究基于Conformer模型，而先前的研究使用基于RNN的系统，并仅考虑离线处理。实验结果表明，该算法在离线环境下具有与传统级联系统相当的识别性能，同时降低了计算开销，实现了流式TS-ASR.摘要：Although recent advances in deep learning technology have boosted automatic speech recognition (ASR) performance in the single-talker case, it remains difficult to recognize multi-talker speech in which many voices overlap. One conventional approach to tackle this problem is to use a cascade of a speech separation or target speech extraction front-end with an ASR back-end. However, the extra computation costs of the front-end module are a critical barrier to quick response, especially for streaming ASR. In this paper, we propose a target-speaker ASR (TS-ASR) system that implicitly integrates the target speech extraction functionality within a streaming end-to-end (E2E) ASR system, i.e. recurrent neural network-transducer (RNNT). Our system uses a similar idea as adopted for target speech extraction, but implements it directly at the level of the encoder of RNNT. This allows TS-ASR to be realized without placing extra computation costs on the front-end. Note that this study presents two major differences between prior studies on E2E TS-ASR; we investigate streaming models and base our study on Conformer models, whereas prior studies used RNN-based systems and considered only offline processing. We confirm in experiments that our TS-ASR achieves comparable recognition performance with conventional cascade systems in the offline setting, while reducing computation costs and realizing streaming TS-ASR.
【1】 Modeling Dependent Structure for Utterances in ASR Evaluation
标题：ASR评价中话语依赖结构的建模
链接：https://arxiv.org/abs/2209.05281
* 与cs.SD语音【4】为同一篇
机构：Meta AI, Menlo Park, CA, USA摘要：在自动语音识别（ASR）评测中，Bootstrap重采样方法被广泛用于对误词率（WER）进行显著性分析。为了解决语音数据的相关性问题，提出了分块自举方法，通过将语音数据划分为不相关的块，对这些块进行重采样，而不是对原始数据进行重采样。然而，揭示话语间的依存结构往往是一件很不容易的事情，这可能会导致统计测试中的主观发现。本文提出了一种基于图形套索的方法来显式地建模这种依赖性，并以严格的方式估计独立的话语块。然后，在推断的块的顶部应用逐块引导。我们证明了所得WER的方差估计在温和条件下是一致的。在LibriSpeech语料库上验证了该方法的有效性。摘要：The bootstrap resampling method has been popular for performing significance analysis on word error rate (WER) in automatic speech recognition (ASR) evaluations. To deal with the issue of dependent speech data, the blockwise bootstrap approach is also proposed that by dividing utterances into uncorrelated blocks, it resamples these blocks instead of original data. However, it is always nontrivial to uncover the dependent structure among utterances, which could lead to subjective findings in statistical testing. In this paper, we present graphical lasso based methods to explicitly model such dependency and estimate the independent blocks of utterances in a rigorous way. Then the blockwise bootstrap is applied on top of the inferred blocks. We show that the resulting variance estimator for WER is consistent under mild conditions. We also demonstrate the validity of proposed approach on LibriSpeech data.【2】 The 2022 Far-field Speaker Verification Challenge: Exploring domain mismatch and semi-supervised learning under the far-field scenarios
标题：2022年远场说话人验证挑战赛：探索远场场景下的域失配和半监督学习
链接：https://arxiv.org/abs/2209.05273
作者：Xiaoyi Qin,Ming Li,Hui Bu,Shrikanth Narayanan,Haizhou Li机构：Data Science Research Center, Duke Kunshan University, Kunshan, China, Department of Electrical & Computer Engineering, National University of Singapore, Singapore, Signal Analysis and Interpretation Lab, University of Southern California, Los Angeles, USA摘要：FFSVC 2022是远场说话人验证的第二个挑战。FFSVC 2022提供了全监督远场说话人确认，进一步探索了远场场景，并提出了半监督远场说话人确认。与FFSVC 2020相比，FFSVC 2022侧重于单通道场景。此外，今年还发布了FFSVC 2020数据集的补充集。补充集包括更多记录设备，数据分布与FFSVC 2022评估集相同。本文对FFSVC 2022进行了总结，包括任务描述、试验设计细节、基线系统和挑战结果总结。挑战结果表明在现场取得了实质性进展，但也表明远场方案仍存在困难。摘要：FFSVC2022 is the second challenge of far-field speaker verification. FFSVC2022 provides the fully-supervised far-field speaker verification to further explore the far-field scenario and proposes semi-supervised far-field speaker verification. In contrast to FFSVC2020, FFSVC2022 focus on the single-channel scenario. In addition, a supplementary set for the FFSVC2020 dataset is released this year. The supplementary set consists of more recording devices and has the same data distribution as the FFSVC2022 evaluation set. This paper summarizes the FFSVC 2022, including tasks description, trial designing details, a baseline system and a summary of challenge results. The challenge results indicate substantial progress made in the field but also present that there are still difficulties with the far-field scenario.【3】 How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models
标题：韵律对话轮转换有多大帮助？使用语音活动投影模型进行调查
链接：https://arxiv.org/abs/2209.05161
作者：Erik Ekstedt,Gabriel Skantze机构：KTH Speech, Music and Hearing, Stockholm, Sweden备注：SIGDIAL 2022 Best Paper Award Winner摘要：话轮转换是人类交流的一个基本方面，可以被描述为在整个对话过程中在适当的位置轮流、预测即将到来的话轮转换和提供反向通道的能力。在本研究中，我们使用最近提出的语音活动投射模型来研究韵律在话轮转换中的作用。该模型以自我监督的方式对对话者即将到来的言语活动进行增量建模，而不依赖于话轮转换事件的显式注释或韵律特征的显式建模。通过对语音信号的处理，我们研究了这些模型如何隐含地利用韵律信息。我们发现，这些系统学会了利用语音的各种韵律方面，既包括长形式会话的聚合定量度量，也包括专门设计为依赖于韵律的单个话语。摘要：Turn-taking is a fundamental aspect of human communication and can be described as the ability to take turns, project upcoming turn shifts, and supply backchannels at appropriate locations throughout a conversation. In this work, we investigate the role of prosody in turn-taking using the recently proposed Voice Activity Projection model, which incrementally models the upcoming speech activity of the interlocutors in a self-supervised manner, without relying on explicit annotation of turn-taking events, or the explicit modeling of prosodic features. Through manipulation of the speech signal, we investigate how these models implicitly utilize prosodic information. We show that these systems learn to utilize various prosodic aspects of speech both on aggregate quantitative metrics of long-form conversations and on single utterances specifically designed to depend on prosody.【4】 Continuous head-related transfer function representation based on hyperspherical harmonics
标题：基于超球面调和的头部相关传递函数连续表示
链接：https://arxiv.org/abs/2209.05110
* 与cs.SD语音【5】为同一篇
机构：AGH University of Science and Technology, Department of Robotics and Mechatronics备注：Submitted to Archives of Acoustics 4.06.2022摘要：在球谐域中表达头部相关传递函数（HRTF）是一种获得空间连续性的方法。然而，HRTF不仅是方向的函数，而且是频率的函数。本文提出了一种基于超球谐函数的扩展方法，利用超球谐函数（HSHs）获得在空间和频率上都连续的HRTF表示。HSH近似的应用导致相对小的系数集合，其可以在任何方向和频率上被解码为HRTF值。本文讨论了通过将该方法应用于从示例性HRTF测量提取的幅度谱而获得的结果。基于SH和HSH的HRTF表示具有相似的再现精度，而后者具有在空间和频率上的连续性，并且需要更少的系数。所开发的基于HSH的连续函数模型可以用于多种目的，例如机器学习应用的插值、压缩或参数化。摘要：Expressing head-related transfer functions (HRTFs) in spherical harmonic (SH) domain has been thoroughly studied as a method of obtaining continuity over space. However, HRTFs are functions not only of direction but also of frequency. This paper presents an extension of the SH-based method, utilizing hyperspherical harmonics (HSHs) to obtain an HRTF representation that is continuous over both space and frequency. The application of the HSH approximation results in a relatively small set of coefficients which can be decoded into HRTF values at any direction and frequency. The paper discusses results obtained by applying the method to magnitude spectra extracted from exemplary HRTF measurements. The HRTF representations based on SHs and HSHs exhibit similar reproduction accuracy, with the latter one featuring continuity over both space and frequency and requiring much lower number of coefficients. The developed HSH-based continuous functional model can serve multiple purposes, such as interpolation, compression or parametrization for machine-learning applications.【5】 VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition
标题：VarArray与t-SOT相遇：推动远程语音识别流技术的发展
链接：https://arxiv.org/abs/2209.04974
* 与cs.SD语音【6】为同一篇
作者：Naoyuki Kanda,Jian Wu,Xiaofei Wang,Zhuo Chen,Jinyu Li,Takuya Yoshioka机构：Microsoft, One Microsoft Way, Redmond, WA, USA摘要：提出了一种新的流式自动语音识别（ASR）框架，用于处理由任意几何形状的麦克风阵列捕获的多人重叠语音.我们的框架，命名为t-SOT-VA，利用了独立开发的两项最新技术：阵列几何不可知的连续语音分离，或VarArray，以及基于令牌级串行化输出训练（t-SOT）的流式多说话者ASR。为了结合这两种技术的优点，我们新设计了一种基于t-SOT的ASR模型，该模型基于来自VarArray的两个分离的语音信号生成串行化的多说话者转录。我们还提出了一种针对这种ASR模型的预训练方案，其中，我们基于单耳单说话者ASR训练数据来模拟VarArray的输出信号。在AMI会议语料库上进行的会话转录实验表明，基于该框架的系统性能明显优于传统系统。在多个远距离麦克风设置中，我们的系统对于AMI开发集和评估集分别实现了13.7%和15.5%的最先进的字错误率，同时保留了流推理能力。摘要：This paper presents a novel streaming automatic speech recognition (ASR) framework for multi-talker overlapping speech captured by a distant microphone array with an arbitrary geometry. Our framework, named t-SOT-VA, capitalizes on independently developed two recent technologies; array-geometry-agnostic continuous speech separation, or VarArray, and streaming multi-talker ASR based on token-level serialized output training (t-SOT). To combine the best of both technologies, we newly design a t-SOT-based ASR model that generates a serialized multi-talker transcription based on two separated speech signals from VarArray. We also propose a pre-training scheme for such an ASR model where we simulate VarArray's output signals based on monaural single-talker ASR training data. Conversation transcription experiments using the AMI meeting corpus show that the system based on the proposed framework significantly outperforms conventional ones. Our system achieves the state-of-the-art word error rates of 13.7% and 15.5% for the AMI development and evaluation sets, respectively, in the multiple-distant-microphone setting while retaining the streaming inference capability.【6】 Reconstructing the Dynamic Directivity of Unconstrained Speech
标题：重构无约束语音的动态指向性
链接：https://arxiv.org/abs/2209.04473
* 与cs.SD语音【7】为同一篇
作者：Camille Noufi,Dejan Markovic,Peter Dodds备注：19 pages, 8 figures, 3 tables. Internally-reviewed manuscript approved for public release by Facebook Inc. Researched uses proprietary dataset not available for public release摘要：自然语音方向性的精确模型是朝向在虚拟通信设置内实现真实声音存在的重要步骤。在这篇文章中，我们提出了一种方法来估计和重建自然的，无约束的语音的空间能量分布模式。我们分两个阶段详细说明我们的方法。利用真实的静态麦克风阵列捕获的语音录音，我们创建了一个虚拟阵列，它可以跟踪说话者随时间的移动。我们使用这个自我中心的虚拟阵列来测量和编码语音信号的高分辨率方向性模式，因为它随着自然语音和运动而动态演变。利用该编码的方向性表示，我们训练机器学习模型，当给定有限的语音信号集时，该机器学习模型用于估计完整的动态方向性图案，如使用头戴式显示器（HMD）上的麦克风记录语音时的情况。我们研究了各种模型体系结构和训练范例，并讨论了每种实现的实用性和实用性。我们的结果表明，神经网络可以用于从有限的语音信息回归到完整方向性图案的准确、动态估计。摘要：An accurate model of natural speech directivity is an important step toward achieving realistic vocal presence within a virtual communication setting. In this article, we propose a method to estimate and reconstruct the spatial energy distribution pattern of natural, unconstrained speech. We detail our method in two stages. Using recordings of speech captured by a real, static microphone array, we create a virtual array that tracks with the movement of the speaker over time. We use this egocentric virtual array to measure and encode the high-resolution directivity pattern of the speech signal as it dynamically evolves with natural speech and movement. Utilizing this encoded directivity representation, we train a machine learning model that leverages to estimate the full, dynamic directivity pattern when given a limited set of speech signals, as would be the case when speech is recorded using the microphones on a head-mounted display (HMD). We examine a variety of model architectures and training paradigms, and discuss the utility and practicality of each implementation. Our results demonstrate that neural networks can be used to regress from limited speech information to an accurate, dynamic estimation of the full directivity pattern.【7】 Pay Attention to Hard Trials
标题：重视艰苦的考验
链接：https://arxiv.org/abs/2209.04687
* 与cs.SD语音【1】为同一篇
作者：Lantian Li,Di Wang,Dong Wang机构：∗ Center for Speech and Language Technologies, BNRist, Tsinghua University, China, † School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China摘要：说话人识别系统的性能是在测试试验中评估的。尽管试验就像裁缝的尺子一样重要，但迄今为止还没有得到仔细的处理，大多数现有的基准测试都是通过简单的交叉配对来组成试验的。在本文中，我们认为交叉配对方法产生了压倒性的容易的试验，这反过来又导致了系统和技术比较中的潜在偏倚。解决问题，我们提倡更加注重硬审。我们提出了一种基于支持向量机的方法来识别硬试验，并使用它来构造新的评估集VoxCeleb 1和SITW。有了这些新的数据集，我们可以重新评估一些最新技术的贡献。代码和确定的困难试验将在线发布在www.example.com上http://project.cslt.org。摘要：Performance of speaker recognition systems is evaluated on test trials. Although as crucial as rulers for tailors, trials have not been carefully treated so far, and most existing benchmarks compose trials by naive cross-pairing. In this paper, we argue that the cross-pairing approach produces overwhelming easy trials, which in turn leads to potential bias in system and technique comparison. To solve the problem, we advocate more attention to hard trials. We present an SVM-based approach to identifying hard trials and use it to construct new evaluation sets for VoxCeleb1 and SITW. With the new sets, we can re-evaluate the contribution of some recent technologies. The code and the identified hard trials will be published online at http://project.cslt.org.【8】 Defend Data Poisoning Attacks on Voice Authentication
标题：防御语音认证中的数据中毒攻击
链接：https://arxiv.org/abs/2209.04547
* 与cs.SD语音【2】为同一篇
作者：Ke Li,Cameron Baird,Dan Lin摘要：随着深度学习技术的发展，说话人验证已经达到了非常高的准确性，并且作为一种生物特征认证选项在我们日常生活的许多场景中越来越受欢迎，特别是在不断增长的网络服务市场中。与传统密码相比，“语音密码”方便得多，因为它减轻了人们记忆不同密码的负担。然而，新的机器学习攻击正在使这些语音认证系统面临风险。在没有强有力的安全保证的情况下，攻击者可以通过欺骗基于深度神经网络（DNN）的语音识别模型来访问合法用户的Web帐户。针对语音认证系统，提出了一种简单易行的数据中毒攻击方法，该方法很难被现有的防御机制所捕获。因此，我们提出了一种更鲁棒的防御方法，称为Guardian，它是一种基于卷积神经网络的鉴别器。Guardian鉴别器集成了一系列新颖的技术，包括偏差减小、输入增强和集成学习。我们的方法能够区分大约95%的被攻击账户和正常账户，这比现有的只有60%准确率的方法要有效得多。摘要：With the advances in deep learning, speaker verification has achieved very high accuracy and is gaining popularity as a type of biometric authentication option in many scenes of our daily life, especially the growing market of web services. Compared to traditional passwords, "vocal passwords" are much more convenient as they relieve people from memorizing different passwords. However, new machine learning attacks are putting these voice authentication systems at risk. Without a strong security guarantee, attackers could access legitimate users' web accounts by fooling the deep neural network (DNN) based voice recognition models. In this paper, we demonstrate an easy-to-implement data poisoning attack to the voice authentication system, which can hardly be captured by existing defense mechanisms. Thus, we propose a more robust defense method, called Guardian, which is a convolutional neural network-based discriminator. The Guardian discriminator integrates a series of novel techniques including bias reduction, input augmentation, and ensemble learning. Our approach is able to distinguish about 95% of attacked accounts from normal accounts, which is much more effective than existing approaches with only 60% accuracy.【9】 DeID-VC: Speaker De-identification via Zero-shot Pseudo Voice Conversion
标题：DEID-VC：基于零点伪音转换的说话人识别
链接：https://arxiv.org/abs/2209.04530
* 与cs.SD语音【3】为同一篇
作者：Ruibin Yuan,Yuxuan Wu,Jacob Li,Jaxter Kim机构：Carnegie Mellon University, United States备注：Accepted by Interspeech 2022摘要：基于语音的在线服务的广泛采用引起了关于它们使用和共享的数据的安全和隐私问题。如果数据被破坏，攻击者可以利用用户的语音绕过说话者验证系统，甚至冒充用户。为了缓解这一问题，我们提出了DeID-VC，一个说话人去识别系统，它将真实说话人转换为伪说话人，从而从语音中去除或混淆说话人相关的属性。DeID-VC的关键组件包括基于变分自动编码器（VAE）的伪说话人生成器（PSG）和zero-shot设置下的语音转换自动编码器（AE）。在PSG的帮助下，DeID-VC可以在说话人级别甚至在话语级别分配唯一的伪说话人。此外，加入了两个新的学习目标，以弥补zero-shot语音转换的训练和推理之间的差距。我们给出了我们的实验结果，字错误率（WER）和等错误率（EER），以及三个主观指标来评估DeID-VC生成的输出。结果表明，与我们的基线相比，我们的方法大大提高了可懂度（WER降低10%）和去识别有效性（EER提高5%）。代码和收听演示：https://github.com/a43992899/DeID-VC摘要：The widespread adoption of speech-based online services raises security and privacy concerns regarding the data that they use and share. If the data were compromised, attackers could exploit user speech to bypass speaker verification systems or even impersonate users. To mitigate this, we propose DeID-VC, a speaker de-identification system that converts a real speaker to pseudo speakers, thus removing or obfuscating the speaker-dependent attributes from a spoken voice. The key components of DeID-VC include a Variational Autoencoder (VAE) based Pseudo Speaker Generator (PSG) and a voice conversion Autoencoder (AE) under zero-shot settings. With the help of PSG, DeID-VC can assign unique pseudo speakers at speaker level or even at utterance level. Also, two novel learning objectives are added to bridge the gap between training and inference of zero-shot voice conversion. We present our experimental results with word error rate (WER) and equal error rate (EER), along with three subjective metrics to evaluate the generated output of DeID-VC. The result shows that our method substantially improved intelligibility (WER 10% lower) and de-identification effectiveness (EER 5% higher) compared to our baseline. Code and listening demo: https://github.com/a43992899/DeID-VC【10】 Streaming End-to-End Target Speaker ASR
标题：端到端流传输目标扬声器ASR
链接：https://arxiv.org/abs/2209.04175
* 与cs.SD语音【17】为同一篇
作者：Takafumi Moriya,Hiroshi Sato,Tsubasa Ochiai,Marc Delcroix,Takahiro Shinozaki机构：NTT Corporation, Japan; ,Tokyo Institute of Technology, Japan备注：Accepted to Interspeech 2022摘要：尽管深度学习技术的最新进展提高了单说话者情况下的自动语音识别（ASR）性能，但识别多个声音重叠的多说话者语音仍然很困难。解决该问题的一种传统方法是使用语音分离或目标语音提取前端与ASR后端的级联。然而，前端模块的额外计算成本是快速响应的关键障碍，特别是对于流式ASR。本文提出了一种目标说话人ASR（TS-ASR）系统，该系统将目标语音提取功能隐式地集成在流式端到端（E2 E）ASR系统中，即递归神经网络-换能器（RNNT）。我们的系统使用与目标语音提取所采用的类似的思想，但是直接在RNNT的编码器级实现它。这允许实现TS-ASR，而无需在前端上放置额外的计算成本。注意，本研究呈现了关于E2 E TS-ASR的先前研究之间的两个主要差异;我们研究了流模型，并将我们的研究基于Conformer模型，而先前的研究使用基于RNN的系统，并仅考虑离线处理。实验结果表明，该算法在离线环境下具有与传统级联系统相当的识别性能，同时降低了计算开销，实现了流式TS-ASR.摘要：Although recent advances in deep learning technology have boosted automatic speech recognition (ASR) performance in the single-talker case, it remains difficult to recognize multi-talker speech in which many voices overlap. One conventional approach to tackle this problem is to use a cascade of a speech separation or target speech extraction front-end with an ASR back-end. However, the extra computation costs of the front-end module are a critical barrier to quick response, especially for streaming ASR. In this paper, we propose a target-speaker ASR (TS-ASR) system that implicitly integrates the target speech extraction functionality within a streaming end-to-end (E2E) ASR system, i.e. recurrent neural network-transducer (RNNT). Our system uses a similar idea as adopted for target speech extraction, but implements it directly at the level of the encoder of RNNT. This allows TS-ASR to be realized without placing extra computation costs on the front-end. Note that this study presents two major differences between prior studies on E2E TS-ASR; we investigate streaming models and base our study on Conformer models, whereas prior studies used RNN-based systems and considered only offline processing. We confirm in experiments that our TS-ASR achieves comparable recognition performance with conventional cascade systems in the offline setting, while reducing computation costs and realizing streaming TS-ASR.【11】 Longitudinal Acoustic Speech Tracking Following Pediatric Traumatic Brain Injury
标题：儿童创伤性脑损伤后的纵向声学语音跟踪
链接：https://arxiv.org/abs/2209.04406
* 与cs.SD语音【16】为同一篇
作者：Camille Noufi,Adam C. Lammert,Daryush D. Mehta,James R. Williamson,Gregory Ciccarelli,Douglas Sturim,Jordan R. Green,Thomas F. Quatieri,Thomas F. Campbell机构：Stanford University, Stanford, CA, USA, MIT Lincoln Laboratory, Lexington, MA, USA, Worcester Polytechnic Institute, Worcester, MA, USA, Mass General Hospital, Boston, MA, USA, University of Texas at Dallas, Dallas, TX, USA摘要：儿童创伤性脑损伤（TBI）后常见结局指标的建议支持在恢复和治疗计划中整合仪器测量和感知评估。因此，在评估儿童TBI后语音特征随时间的变化时，一套全面的灵敏、稳健和非侵入性测量是必不可少的。在这篇文章中，我们研究了10名被诊断为严重TBI的儿童队列受试者的声学言语模式的变化。我们从儿童产生可理解的单词后的一年中记录的儿童语音中提取了一组不同的已知和新颖的声学特征。对这些特征进行单独分析，并按语音子系统、受试者内和队列进行分析。作为一个群体，年龄较大的儿童表现出高度显著的（p〈0.01）音高变化和音素多样性增加，停顿长度缩短，发音速率变化稳定。年龄较小的儿童表现出类似的稳定速率变化，同时基于共振峰的发音复杂度增加。对特征集与年龄的相关性分析以及与正常发育数据的比较证实，受伤时的年龄在构建恢复轨迹方面起着重要作用。作为一个整体，几乎所有的语音特征都发生了显著变化（p〈0.05），这证实了需要声学测量补充感知评估，以确定TBI后语音治疗的有效治疗目标。摘要：Recommendations for common outcome measures following pediatric traumatic brain injury (TBI) support the integration of instrumental measurements alongside perceptual assessment in recovery and treatment plans. A comprehensive set of sensitive, robust and non-invasive measurements is therefore essential in assessing variations in speech characteristics over time following pediatric TBI. In this article, we study the changes in the acoustic speech patterns of a pediatric cohort of ten subjects diagnosed with severe TBI. We extract a diverse set of both well-known and novel acoustic features from child speech recorded throughout the year after the child produced intelligible words. These features are analyzed individually and by speech subsystem, within-subject and across the cohort. As a group, older children exhibit highly significant (p<0.01) increases in pitch variation and phoneme diversity, shortened pause length, and steadying articulation rate variability. Younger children exhibit similar steadied rate variability alongside an increase in formant-based articulation complexity. Correlation analysis of the feature set with age and comparisons to normative developmental data confirm that age at injury plays a significant role in framing the recovery trajectory. Nearly all speech features significantly change (p<0.05) for the cohort as a whole, confirming that acoustic measures supplementing perceptual assessment are needed to identify efficacious treatment targets for speech therapy following TBI.【12】 A Semi-Supervised Algorithm for Improving the Consistency of Crowdsourced Datasets: The COVID-19 Case Study on Respiratory Disorder Classification
标题：一种提高众包数据集一致性的半监督算法--以新冠肺炎呼吸系统疾病分类为例
链接：https://arxiv.org/abs/2209.04360
* 与cs.SD语音【8】为同一篇
作者：Lara Orlandic,Tomas Teijeiro,David Atienza机构：Embedded Systems Laboratory (ESL), EPFL, Lausanne, Switzerland, Department of Mathematics, University of the Basque Country (UPVEHU), Spain摘要：咳嗽 音频 信号 分类 是 筛查 呼吸 系统 疾病 （ 如 COVID - 19 ） 的 一 种 潜在 有用 工具 。由于 从 患有 此类 传染 病 的 患者 身上 收集 数据 是 危险 的 ， 许多 研究 团队 转向 众 包 来 快速 收集 咳嗽 声音 数据 ， 就 像 生成 COUGHVID 数据 集 一样 。COUGHVID 数据 集 招募 了 专家 医生 来 诊断 有限 数量 的 上传 记录 中 存在 的 基础 疾病 。然而 ， 这种 方法 存在 咳嗽 的 潜在 错误 标记 ， 以及 专家 之间 的 明显 分歧 。在 这项 工作 中 ， 我们 使用 半 监督 学习 （ SSL ） 方法 来 提高 COUGHVID 数据 集 的 标签 一致 性 ， 以及 COVID - 19 与 健康 咳嗽 音 分类 的 鲁 棒 性 。首先 ， 我们 利用 现有 的 SSL 专家 知识 聚合 技术 来 克服 数据 集中 标签 的 不 一致 性 和 稀疏 性 。接下来 ， 我们 的 SSL 方法 被 用于 识别 重新 标记 的 COUGHVID 音频 样本 的 子 样本 ， 其 可以 被 用于 训练 或 增强 未来 的 咳嗽 分类 模型 。尽管 原始 数据 集中 存在 专家 标签 不 一致 性 ， 但 重新 标记 数据 的 一致 性 表现 为 高度 的 类 可分性 ， 比 用户 标记 数据 的 类 可分性 高 3 倍 。此外 ， 用户 标记 的 音频 片段 中 的 频谱 差异 在 重新 标记 的 数据 中 被 放大 ， 导致 健康 咳嗽 和 COVID - 19 咳嗽 之间 的 功率 谱 密度 显著 不同 ， 这 既 证明 了 新 数据 集 的 一致 性 增强 ， 也 证明 了 其 从 声学 角度 的 可 解释 性 。最 后 ， 我们 演示 了 如何 使用 重新 标记 的 数据 集 来 训练 咳嗽 分类 器 。该 SSL 方法 可 用于 组合 多 个 专家 的 医学 知识 ， 以 提高 任何 诊断 分类 任务 的 数据 库 一致 性 。摘要：Cough audio signal classification is a potentially useful tool in screening for respiratory disorders, such as COVID-19. Since it is dangerous to collect data from patients with such contagious diseases, many research teams have turned to crowdsourcing to quickly gather cough sound data, as it was done to generate the COUGHVID dataset. The COUGHVID dataset enlisted expert physicians to diagnose the underlying diseases present in a limited number of uploaded recordings. However, this approach suffers from potential mislabeling of the coughs, as well as notable disagreement between experts. In this work, we use a semi-supervised learning (SSL) approach to improve the labeling consistency of the COUGHVID dataset and the robustness of COVID-19 versus healthy cough sound classification. First, we leverage existing SSL expert knowledge aggregation techniques to overcome the labeling inconsistencies and sparsity in the dataset. Next, our SSL approach is used to identify a subsample of re-labeled COUGHVID audio samples that can be used to train or augment future cough classification models. The consistency of the re-labeled data is demonstrated in that it exhibits a high degree of class separability, 3x higher than that of the user-labeled data, despite the expert label inconsistency present in the original dataset. Furthermore, the spectral differences in the user-labeled audio segments are amplified in the re-labeled data, resulting in significantly different power spectral densities between healthy and COVID-19 coughs, which demonstrates both the increased consistency of the new dataset and its explainability from an acoustic perspective. Finally, we demonstrate how the re-labeled dataset can be used to train a cough classifier. This SSL approach can be used to combine the medical knowledge of several experts to improve the database consistency for any diagnostic classification task.【13】 Overlapped speech and gender detection with WavLM pre-trained features
标题：利用WavLM预训练特征进行重叠语音和性别检测
链接：https://arxiv.org/abs/2209.04167
* 与cs.SD语音【9】为同一篇
作者：Martin Lebourdais,Marie Tahon,Antoine Laurent,Sylvain Meignier机构：LIUM, Le Mans Universit´e, France备注：Submitted and accepted to Interspeech 2022摘要：本文以重叠言语和性别检测为研究对象，研究法国视听媒体中男女互动的情况（性别平等监测项目）。在这种应用背景下，我们需要根据说话者的性别自动分割语音信号，并且识别何时至少有两个说话者同时说话。本文提出利用WavLM模型在大量语音数据上预先训练的优点，建立重叠语音检测（OSD）和性别检测（GD）系统。在本研究中，我们使用了两个不同的语料库。DIHARD III语料库很好地适应了OSD任务，但缺乏性别信息。ALLIES语料库符合项目应用背景。我们最好的OSD系统是一个时间卷积网络（TCN）与WavLM预训练的特征作为输入，它达到了一个新的最先进的F1成绩的性能在DIHARD。在法国广播新闻ALLIES数据的性别平衡子集上，用WavLM输入训练神经GD，并获得97.9%的准确率。这项工作为人文科学研究人员关于法国媒体中女性和男性代表性差异的新视角。摘要：This article focuses on overlapped speech and gender detection in order to study interactions between women and men in French audiovisual media (Gender Equality Monitoring project). In this application context, we need to automatically segment the speech signal according to speakers gender, and to identify when at least two speakers speak at the same time. We propose to use WavLM model which has the advantage of being pre-trained on a huge amount of speech data, to build an overlapped speech detection (OSD) and a gender detection (GD) systems. In this study, we use two different corpora. The DIHARD III corpus which is well adapted for the OSD task but lack gender information. The ALLIES corpus fits with the project application context. Our best OSD system is a Temporal Convolutional Network (TCN) with WavLM pre-trained features as input, which reaches a new state-of-the-art F1-score performance on DIHARD. A neural GD is trained with WavLM inputs on a gender balanced subset of the French broadcast news ALLIES data, and obtains an accuracy of 97.9%. This work opens new perspectives for human science researchers regarding the differences of representation between women and men in French media.【14】 MATT: A Multiple-instance Attention Mechanism for Long-tail Music Genre Classification
标题：Matt：一种长尾音乐体裁分类的多实例注意机制
链接：https://arxiv.org/abs/2209.04109
* 与cs.SD语音【10】为同一篇
作者：Xiaokai Liu,Menghua Zhang机构：∗School of Cyber Science and Engineering, § School of Energy and Power Engineering, Huazhong University of Science and Technology, Wuhan, China摘要：不平衡音乐类型分类是音乐信息检索领域的一项重要任务，它是基于相关音乐音频片段识别长尾、数据贫乏的音乐类型，在现实生活中非常普遍。现有的音乐分类模型大多针对类别均衡的音乐数据集，在识别分布尾部的音乐类型时，其准确性和泛化能力较差.受多实例学习（Multi-instance Learning，MIL）在各类分类任务中成功应用的启发，提出了一种新的多实例注意（Multi-instance Attention，MATT）机制来提高尾类识别的性能.具体地说，我们首先通过生成专辑-艺术家对包来构造包级数据集。第二，我们利用神经网络来编码音乐音频片段。最后，在多实例注意机制的指导下，神经网络模型可以选择信息量最大的流派来匹配给定的音乐片段。在大规模长尾分布音乐流派基准数据集上的综合实验结果表明，MATT算法的性能明显优于其他现有的基准算法.摘要：Imbalanced music genre classification is a crucial task in the Music Information Retrieval (MIR) field for identifying the long-tail, data-poor genre based on the related music audio segments, which is very prevalent in real-world scenarios. Most of the existing models are designed for class-balanced music datasets, resulting in poor performance in accuracy and generalization when identifying the music genres at the tail of the distribution. Inspired by the success of introducing Multi-instance Learning (MIL) in various classification tasks, we propose a novel mechanism named Multi-instance Attention (MATT) to boost the performance for identifying tail classes. Specifically, we first construct the bag-level datasets by generating the album-artist pair bags. Second, we leverage neural networks to encode the music audio segments. Finally, under the guidance of a multi-instance attention mechanism, the neural network-based models could select the most informative genre to match the given music segment. Comprehensive experimental results on a large-scale music genre benchmark dataset with long-tail distribution demonstrate MATT significantly outperforms other state-of-the-art baselines.【15】 Learning Audio-Visual embedding for Wild Person Verification
标题：基于学习视听嵌入的野人身份验证
链接：https://arxiv.org/abs/2209.04093
* 与cs.SD语音【11】为同一篇
作者：Peiwen Sun,Shanshan Zhang,Zishan Liu,Yougen Yuan,Taotao Zhang,Honggang Zhang,Pengfei Hu机构：School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China, TEG AI, Tencent Inc, Beijing, China摘要：已经观察到，可以从这两种模态中提取音频-视频嵌入，以获得用于个人验证的鲁棒性。然而，用于从每个帧生成单个话语表示的聚合器似乎没有被很好地探索。在本文中，我们提出了一个从融合的角度考虑聚合器的视听网络。在人脸验证中首次引入了改进的注意统计池算法。然后发现在合并过程中模态间存在很强的相关性，提出了一种包含循环一致性的联合注意力合并算法来学习隐含的帧间权重。最后，将模态与门控注意机制相融合。在VoxCeleb 2 dev数据集上对所提出的模型进行了训练，最佳系统在VoxCeleb 1的三个官方测试列表上分别获得了0.18%，0.27%和0.49%的EER，这是我们所知道的发表的最好的个人验证结果.作为分析，生成可视化图以解释该系统如何在模态之间交互。摘要：It has already been observed that audio-visual embedding can be extracted from these two modalities to gain robustness for person verification. However, the aggregator that used to generate a single utterance representation from each frame does not seem to be well explored. In this article, we proposed an audio-visual network that considers aggregator from a fusion perspective. We introduced improved attentive statistics pooling for the first time in face verification. Then we find that strong correlation exists between modalities during pooling, so joint attentive pooling is proposed which contains cycle consistency to learn the implicit inter-frame weight. Finally, fuse the modality with a gated attention mechanism. All the proposed models are trained on the VoxCeleb2 dev dataset and the best system obtains 0.18\%, 0.27\%, and 0.49\% EER on three official trail lists of VoxCeleb1 respectively, which is to our knowledge the best-published results for person verification. As an analysis, visualization maps are generated to explain how this system interact between modalities.【16】 Prediction method of Soundscape Impressions using Environmental Sounds and Aerial Photographs
标题：利用环境声和航拍照片预测声景观印象的方法
链接：https://arxiv.org/abs/2209.04077
* 与cs.SD语音【12】为同一篇
作者：Yusuke Ono,Sunao Hara,Masanobu Abe机构：∗ Okayama University, Okayama, Japan备注：Submitted APSIPA ASC 2022摘要：我们研究了一种基于声音环境印象来量化城市特征的方法。本研究利用云传感法采集的声音数据，对城市的两种声景印象--愉悦感和事件感进行了预测。所收集的声音包括使用全球定位系统的记录位置的元信息。此外，通过使用瑞典声景质量协议（Swedish Soundscape-Quality Protocol）定义的评估，将声景印象和声源特征分别分配给云感测声音，从而评估声学环境的质量。利用多层感知器的深度神经网络对10秒的声音输入和其位置的航拍照片建立预测模型。声学特征包括等效噪声级和倍频程滤波器每秒的输出，以及它们在10~s内的统计。使用ResNet-50和自动编码器架构从航空照片中提取图像特征。我们进行了对比实验来演示每种特性的优势。作为比较的结果，航空照片和声源特征对于预测印象信息是有效的。另外，即使使用声学和图像特征来预测声源特征，这些特征也显示出良好的结果，以预测接近于甲骨文声源特征的结果的声景印象。摘要：We investigate an method for quantifying city characteristics based on impressions of a sound environment. The quantification of the city characteristics will be beneficial to government policy planning, tourism projects, etc. In this study, we try to predict two soundscape impressions, meaning pleasantness and eventfulness, using sound data collected by the cloud-sensing method. The collected sounds comprise meta information of recording location using Global Positioning System. Furthermore, the soundscape impressions and sound-source features are separately assigned to the cloud-sensing sounds by assessments defined using Swedish Soundscape-Quality Protocol, assessing the quality of the acoustic environment. The prediction models are built using deep neural networks with multi-layer perceptron for the input of 10-second sound and the aerial photographs of its location. An acoustic feature comprises equivalent noise level and outputs of octave-band filters every second, and statistics of them in 10~s. An image feature is extracted from an aerial photograph using ResNet-50 and autoencoder architecture. We perform comparison experiments to demonstrate the benefit of each feature. As a result of the comparison, aerial photographs and sound-source features are efficient to predict impression information. Additionally, even if the sound-source features are predicted using acoustic and image features, the features also show fine results to predict the soundscape impression close to the result of oracle sound-source features.【17】 Improving the Environmental Perception of Autonomous Vehicles using Deep Learning-based Audio Classification
标题：基于深度学习的音频分类方法改善自主车辆的环境感知
链接：https://arxiv.org/abs/2209.04075
* 与cs.SD语音【13】为同一篇
作者：Finley Walden,Sagar Dasgupta,Mizanur Rahman,Mhafuzul Islam机构：Undergraduate Student, Department of Computer Science, The University of Alabama, Cyber Hall, Box , Kirkbride Lane, Tuscaloosa, AL , Ph.D. Student, Department of Civil, Construction & Environmental Engineering, Assistant Professor, Senior Research Scientist摘要：听觉 是 自动 驾驶 车辆 感知 周围 环境 的 关键 。虽然 AV 的 视觉 传感 器 ， 如 摄像 机 、 激光 雷达 和 雷达 ， 有助于 看到 其 周围 环境 ， 但 AV 不能 看到 这些 传感 器 的 视线 之外 。另 一方面 ， AV 的 听觉 不能 被 视线 所 阻碍 。例如 ， 即使 紧急 车辆 不在 AV 的 视线 内 ， AV 也 可以 通过 音频 分类 来 识别 紧急 车辆 的 警笛 。因此 ， 听觉 感知 是 对 相机 、 激光 雷达 和 基于 雷达 的 感知 系统 的 补充 。提出 了 一 种 基于 深度 学习 的 鲁 棒 音频 分类 框架 ， 旨在 提高 AV 的 环境 感知 能力 。所 提出 的 框架 利用 深度 卷积 神经 网络 （ CNN ） 来 分类 不同 的 音频 类别 。使用 城市 环境 数据 集 UrbanSound8k 对 所 开发 的 框架 进行 训练 和 测试 。七 个 音频 类 ，空调 、 汽车 喇叭 、 儿童 玩耍 、 狗 叫 、 引擎 空转 、 枪声 和 警笛 ， 由于 它们 与 AV 的 相关 性 而 从 UrbanSound8k 数据 集中 识别 。我们 的 框架 可以 对 不同 的 音频 类别 进行 分类 ， 准确 率 为 97.82% 。此外 ， 给出 了 所有 10 类 的 音频 分类 准确 率 ， 这 证明 了 与 现有 的 音频 分类 框架 相比 ， 我们 的 框架 在 AV 相关 声音 的 情况 下 表现 得 更 好 。摘要：Sense of hearing is crucial for autonomous vehicles (AVs) to better perceive its surrounding environment. Although visual sensors of an AV, such as camera, lidar, and radar, help to see its surrounding environment, an AV cannot see beyond those sensors line of sight. On the other hand, an AV s sense of hearing cannot be obstructed by line of sight. For example, an AV can identify an emergency vehicle s siren through audio classification even though the emergency vehicle is not within the line of sight of the AV. Thus, auditory perception is complementary to the camera, lidar, and radar-based perception systems. This paper presents a deep learning-based robust audio classification framework aiming to achieve improved environmental perception for AVs. The presented framework leverages a deep Convolution Neural Network (CNN) to classify different audio classes. UrbanSound8k, an urban environment dataset, is used to train and test the developed framework. Seven audio classes i.e., air conditioner, car horn, children playing, dog bark, engine idling, gunshot, and siren, are identified from the UrbanSound8k dataset because of their relevancy related to AVs. Our framework can classify different audio classes with 97.82% accuracy. Moreover, the audio classification accuracies with all ten classes are presented, which proves that our framework performed better in the case of AV-related sounds compared to the existing audio classification frameworks.【18】 Audio Analytics-based Human Trafficking Detection Framework for Autonomous Vehicles
标题：基于音频分析的自动车辆人口贩运检测框架
链接：https://arxiv.org/abs/2209.04071
* 与cs.SD语音【14】为同一篇
作者：Sagar Dasgupta,Kazi Shakib,Mizanur Rahman,Silvana V Croope,Steven Jones机构：Ph.D. Student, The University of Alabama, Cyber Hall, Box , Kirkbride Lane, Tuscaloosa, AL , Assistant Professor, Resilience Research Engineer, Alabama Transportation Institute, Director, Transportation Policy Research Center摘要：人口 贩运 是 一 个 普遍 问题 ， 尽管 在 全球 范围 内 作出 了 许多 努力 来 打击 这 一 问题 ， 但 这 一 问题 依然 存在 。任何 年龄 、 种族 、 民族 、 性别 、 性别 认同 、 性 取向 、 国籍 、 移民 身份 、 文化 背景 、 宗教 、 社会 经济 阶层 和 教育 程度 的 个人 都 可能 成为 人口 贩运 的 受害 者 。随着 技术 的 进步 和 自动 驾驶 车辆 的 引入 ， 人口 贩运 者 将 采用 新 的 方式 运送 受害 者 ， 这 可能 会 加速 有 组织 人口 贩运 网络 的 发展 ， 从而 使 执法 机构 在 侦查 人口 贩运 活动 时 面临 更 大 的 挑战 。本 研究 的 目的 是 开发 一 种 创新 的 基于 音频 分析 的 自动 驾驶 车辆 人口 贩卖 检测 框架 。本 研究 的 主要 贡献 是 ：（ i ） 为 反 车辆 组织 确定 四 种 重要 、 可行 和 现实 的 人口 贩运 情景 ;创建 一 个 新 的 与 人口 贩运 有关 的 综合 音频 数据 集 ， 分为 五 类 ， 即 ：哭泣 、 尖叫 、 车门 撞击 、 汽车 噪音 和 谈话 ;以及 （ iii ） 开发 用于 与 人口 贩运 有关 的 音频 数据 分类 的 深度 1 - D 卷积 神经 网络 （ CNN ） 架构 。我们 还 使用 新 的 音频 数据 集 进行 了 案例 研究 ， 并 评估 了 深度 1 - D CNN 的 音频 分类 性能 。我们 的 分析 表明 ， 深度 一维 CNN 能够 以 95% 的 准确 率 区分 来自 人口 贩运 受害 者 的 声音 和 非 人口 贩运 的 声音 ， 这 证明 了 我们 的 框架 的 有效 性 。摘要：Human trafficking is a universal problem, persistent despite numerous efforts to combat it globally. Individuals of any age, race, ethnicity, sex, gender identity, sexual orientation, nationality, immigration status, cultural background, religion, socioeconomic class, and education can be a victim of human trafficking. With the advancements in technology and the introduction of autonomous vehicles (AVs), human traffickers will adopt new ways to transport victims, which could accelerate the growth of organized human trafficking networks, which can make the detection of trafficking in persons more challenging for law enforcement agencies. The objective of this study is to develop an innovative audio analytics-based human trafficking detection framework for autonomous vehicles. The primary contributions of this study are to: (i) define four non-trivial, feasible, and realistic human trafficking scenarios for AVs; (ii) create a new and comprehensive audio dataset related to human trafficking with five classes i.e., crying, screaming, car door banging, car noise, and conversation; and (iii) develop a deep 1-D Convolution Neural Network (CNN) architecture for audio data classification related to human trafficking. We have also conducted a case study using the new audio dataset and evaluated the audio classification performance of the deep 1-D CNN. Our analyses reveal that the deep 1-D CNN can distinguish sound coming from a human trafficking victim from a non-human trafficking sound with an accuracy of 95%, which proves the efficacy of our framework.【19】 Non-autoregressive Error Correction for CTC-based ASR with Phone-conditioned Masked LM
标题：基于CTC的自动语音识别的非自回归纠错
链接：https://arxiv.org/abs/2209.04062
* 与cs.SD语音【15】为同一篇
作者：Hayato Futami,Hirofumi Inaguma,Sei Ueno,Masato Mimura,Shinsuke Sakai,Tatsuya Kawahara机构：Graduate School of Informatics, Kyoto University, Sakyo-ku, Kyoto, Japan备注：Accepted in Interspeech2022摘要：基于连接主义时态分类（CTC）的模型由于其非自回归性质而在自动语音识别（ASR）中具有吸引力。为了充分利用纯文本数据，语言模型（LM）集成方法（如重新评分和浅融合）已被广泛用于CTC。然而，由于需要波束搜索，它们失去了CTC的非自回归性质，这降低了推理速度。在本研究中，我们提出了一种使用音素条件掩蔽LM（PC-MLM）的错误更正方法。在该方法中，对CTC贪婪解码输出中的低置信度单词标记进行屏蔽。然后，PC-MLM在给定从CTC补充预测的未屏蔽单词和音素的情况下预测这些屏蔽单词标记。我们进一步将其扩展为可删除PC-MLM，以解决插入错误。由于CTC和PC-MLM都是非自回归模型，因此该方法能够实现快速LM积分。在CSJ和TED-LIUM 2语料库上的实验结果表明，该方法在推理速度和CSJ上的识别准确率方面均优于重评分和浅融合方法。摘要：Connectionist temporal classification (CTC) -based models are attractive in automatic speech recognition (ASR) because of their non-autoregressive nature. To take advantage of text-only data, language model (LM) integration approaches such as rescoring and shallow fusion have been widely used for CTC. However, they lose CTC's non-autoregressive nature because of the need for beam search, which slows down the inference speed. In this study, we propose an error correction method with phone-conditioned masked LM (PC-MLM). In the proposed method, less confident word tokens in a greedy decoded output from CTC are masked. PC-MLM then predicts these masked word tokens given unmasked words and phones supplementally predicted from CTC. We further extend it to Deletable PC-MLM in order to address insertion errors. Since both CTC and PC-MLM are non-autoregressive models, the method enables fast LM integration. Experimental evaluations on the Corpus of Spontaneous Japanese (CSJ) and TED-LIUM2 in domain adaptation setting shows that our proposed method outperformed rescoring and shallow fusion in terms of inference speed, and also in terms of recognition accuracy on CSJ.【20】 The Role of Voice Persona in Expressive Communication:An Argument for Relevance in Speech Synthesis Design
标题：语音人物角色在表现力交流中的作用：语音合成设计中的关联性论证
链接：https://arxiv.org/abs/2209.02855
作者：Camille Noufi,Lloyd May,Jonathan Berger机构：Center for Computer Research in Music and Acoustics, Stanford University, USA摘要：我们提出了一种在合成语音中注入表现力的方法，通过对10位声乐研究和表演专家的访谈进行主题分析，为实时、交互式声乐角色的设计框架提供信息，该角色将产生令人信服的和适当的上下文相关的表达。所得到的语音音调被定义为存在于连续的、上下文相关的概率空间内的点。在合成语音中包括语音角色在广泛的应用中是重要的。特别令人感兴趣的是在增强和辅助通信（AAC）社区的潜在影响。最后，我们将介绍我们正在进行的研究，调查声音人物角色的主题，以及他们如何可以继续告知所提出的表达性语音合成设计框架。摘要：We present an approach to imbuing expressivity in a synthesized voice by acquiring a thematic analysis of 10 interviews with vocal studies and performance experts to inform the design framework for a real-time, interactive vocal persona that would generate compelling and appropriate contextually-dependent expression. The resultant tone of voice is defined as a point existing within a continuous, contextually-dependent probability space. The inclusion of voice persona in synthesized voice can be significant in a broad range of applications. Of particular interest is the potential impact in augmentative and assistive communication (AAC) community. Finally, we conclude with an introduction to our ongoing research investigating the themes of vocal persona and how they may continue to inform proposed expressive speech synthesis design frameworks.
机器翻译，仅供参考
每日语音速递