今日论文合集：cs.SD语音23篇，eess.AS音频处理24篇。【1】 Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation标题：从运动中定位声音：联合学习声音方向和摄像机旋转链接：http://arxiv.org/pdf/2303.11329v1
作者：Ziyang Chen,Shengyi Qian,Andrew Owens机构：University of Michigan备注：Project site: this https URL摘要：当我们转动头部时，我们感知到的图像和声音会发生微妙但几何学上一致的变化。在本文中，我们使用这些线索来解决一个我们称之为运动声音定位（SLfM）的问题：联合估计摄像机旋转和定位声源。我们学会了仅仅通过自我监督来解决这些任务。视觉模型根据一对图像预测相机旋转，而音频模型根据双耳声音预测声源的方向。我们训练这些模型以产生彼此一致的预测。在测试时，模型可以独立部署。为了获得非常适合于解决这个挑战性问题的特征表示，我们还提出了一种用于通过跨视图双耳化来学习视听表示的方法：从一个视图估计双耳声，给定来自另一视图的图像和声音。我们的模型可以成功地估计真实场景和合成场景中的精确旋转，并以与最先进的自监督方法相竞争的精度定位声源。项目地点：https: ificl.github.io SLfM摘要：The images and sounds that we perceive undergo subtle but geometrically consistent changes as we rotate our heads. In this paper, we use these cues to solve a problem we call Sound Localization from Motion (SLfM): jointly estimating camera rotation and localizing sound sources. We learn to solve these tasks solely through self-supervision. A visual model predicts camera rotation from a pair of images, while an audio model predicts the direction of sound sources from binaural sounds. We train these models to generate predictions that agree with one another. At test time, the models can be deployed independently. To obtain a feature representation that is well-suited to solving this challenging problem, we also propose a method for learning an audio-visual representation through cross-view binauralization: estimating binaural sound from one view, given images and sound from another. Our model can successfully estimate accurate rotations on both real and synthetic scenes, and localize sound sources with accuracy competitive with state-of-the-art self-supervised approaches. Project site: https: ificl.github.io SLfM.【2】 Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speech标题：鸡尾酒休BERT：混合语音和单源语音的广义自监督预训练链接：http://arxiv.org/pdf/2303.11131v1
作者：Maryam Fazel-Zarandi,Wei-Ning Hsu机构：Meta AI - FAIR备注：ICASSP 2023摘要：自监督学习有效地利用了未标记数据，提高了标记效率和对没有标记数据的领域的泛化能力。虽然最近的工作已经研究了对更多声学 语言学领域、语言和模态的概括，但这些研究仅限于录音中有一个主要说话者的单源语音。本文提出了一种自监督学习框架Cocktail HuBERT，它使用掩蔽伪源分离目标推广到混合语音。该目标鼓励模型识别源的数量，分离和理解上下文，并推断表示为发现单元的掩蔽区域的内容。鸡尾酒HuBERT优于最先进的结果，在多扬声器ASR上WER降低69%，在diarization上DER降低31%，在SUPERB的单扬声器和多扬声器任务上具有竞争力。摘要：Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data. While recent work has studied generalization to more acoustic linguistic domains, languages, and modalities, these investigations are limited to single-source speech with one primary speaker in the recording. This paper presents Cocktail HuBERT, a self-supervised learning framework that generalizes to mixture speech using a masked pseudo source separation objective. This objective encourages the model to identify the number of sources, separate and understand the context, and infer the content of masked regions represented as discovered units. Cocktail HuBERT outperforms state-of-the-art results with 69% lower WER on multi-speaker ASR, 31% lower DER on diarization, and is competitive on single- and multi-speaker tasks from SUPERB.【3】 EmoTalk: Speech-driven emotional disentanglement for 3D face animation标题：EmoTalk：语音驱动的3D人脸动画情感解脱链接：http://arxiv.org/pdf/2303.11089v1
作者：Ziqiao Peng,Haoyu Wu,Zhenbo Song,Hao Xu,Xiangyu Zhu,Hongyan Liu,Jun He,Zhaoxin Fan机构：Renmin University of China, Nanjing University of Science and Technology, The Hong Kong University of Science and Technology, Chinese Academy of Sciences, Tsinghua University, Psyche AI Inc.摘要：Transformer在多人自注意的支持下，在语音情感识别（SER）中取得了显著的效果。与原有的全注意机制相比，基于窗口的注意机制在学习细粒度特征时更有效，同时大大减少了模型冗余。然而，情感线索以多粒度方式呈现，使得预定义固定窗口可严重降低模型灵活性。此外，很难手动获得最佳窗口设置。本文提出了一种用于SER任务的可变形语音Transformer DST。DST通过一个轻量级的判决网络确定以输入语音为条件的窗口大小的使用。同时，利用基于声学特征的数据相关偏移量来调整注意窗口的位置，使DST能够自适应地发现和注意语音中的有价值信息。在IEMOCAP和MELD上的大量实验证明了DST的优越性。摘要：Enabled by multi-head self-attention, Transformer has exhibited remarkable results in speech emotion recognition (SER). Compared to the original full attention mechanism, window-based attention is more effective in learning fine-grained features while greatly reducing model redundancy. However, emotional cues are present in a multi-granularity manner such that the pre-defined fixed window can severely degrade the model flexibility. In addition, it is difficult to obtain the optimal window settings manually. In this paper, we propose a Deformable Speech Transformer, named DST, for SER task. DST determines the usage of window sizes conditioned on input speech via a light-weight decision network. Meanwhile, data-dependent offsets derived from acoustic features are utilized to adjust the positions of the attention windows, allowing DST to adaptively discover and attend to the valuable information embedded in the speech. Extensive experiments on IEMOCAP and MELD demonstrate the superiority of DST.【4】 Dual-stream Time-Delay Neural Network with Dynamic Global Filter for Speaker Verification标题：基于动态全局滤波的双流时延神经网络说话人确认链接：http://arxiv.org/pdf/2303.11020v1
作者：Yangfu Li,Xiaodan Lin机构：School of Information Science and Engineering, Huaqiao University备注：13 pages 4 figures摘要：时延神经网络（TDNN）是文本无关说话人确认的最新模型之一。然而，传统的TDNN难以捕捉全局上下文，而全局上下文在许多最近的工作中已经被证明是鲁棒的说话人表示和长时间说话人确认的关键。此外，常见的解决方案，例如，自注意，对于输入标记具有二次复杂度，这使得它们在应用于TDNN中具有大尺寸的特征图时在计算上难以承受。为了解决这些问题，我们提出了TDNN的全局滤波器，它应用对数线性复杂度FFT IFFT和一组可微分频域滤波器来有效地建模语音中的长期相关性。此外，为了提高全局滤波器的性能，防止过拟合，还特别设计了动态滤波策略和稀疏正则化方法。此外，我们还构造了一种双流TDNN（DS-TDNN），它通过分割基本通道来降低复杂度，并使用全局滤波器来提高识别性能。在Voxceleb和SITW数据库上的实验结果表明，DS-TDNN与ECAPA-TDNN相比，在复杂度和参数上分别降低了28%和15%，性能提高了约10%。此外，在面对长时间语音时，与其他主流的基线系统相比，它在效率和有效性之间具有最佳的折衷。最后，可视化和详细的消融研究进一步揭示了DS-TDNN的优势。摘要：The time-delay neural network (TDNN) is one of the state-of-the-art models for text-independent speaker verification. However, it is difficult for conventional TDNN to capture global context that has been proven critical for robust speaker representations and long-duration speaker verification in many recent works. Besides, the common solutions, e.g., self-attention, have quadratic complexity for input tokens, which makes them computationally unaffordable when applied to the feature maps with large sizes in TDNN. To address these issues, we propose the Global Filter for TDNN, which applies log-linear complexity FFT IFFT and a set of differentiable frequency-domain filters to efficiently model the long-term dependencies in speech. Besides, a dynamic filtering strategy, and a sparse regularization method are specially designed to enhance the performance of the global filter and prevent it from overfitting. Furthermore, we construct a dual-stream TDNN (DS-TDNN), which splits the basic channels for complexity reduction and employs the global filter to increase recognition performance. Experiments on Voxceleb and SITW databases show that the DS-TDNN achieves approximate 10% improvement with a decline over 28% and 15% in complexity and parameters compared with the ECAPA-TDNN. Besides, it has the best trade-off between efficiency and effectiveness compared with other popular baseline systems when facing long-duration speech. Finally, visualizations and a detailed ablation study further reveal the advantages of the DS-TDNN.【5】 On-the-fly Text Retrieval for End-to-End ASR Adaptation标题：用于端到端ASR适配的即时文本检索链接：http://arxiv.org/pdf/2303.10942v1
作者：Bolaji Yusuf,Aditya Gourav,Ankur Gandhe,Ivan Bulyko机构：Amazon Alexa, USA, Bo˘gazic¸i University, Department of Electrical and Electronics Engineering, Istanbul, Turkey备注：Accepted to ICASSP 2023; Appendix added to include ablations that could not fit into the conference 4-page limit摘要：端到端语音识别模型通过合并外部文本源来改进，典型地通过与外部语言模型融合来改进。每当感兴趣的语料库改变时，这种语言模型必须被重新训练。此外，由于它们将整个语料库存储在其参数中，罕见单词可能难以回忆。在这项工作中，我们提出用检索语言模型来增强基于转换器的ASR模型，该检索语言模型直接从外部文本语料库中检索部分ASR假设的似然完成。然后，这些完成由适配器集成到后续预测中，适配器被训练一次，使得感兴趣语料库可以被切换而不引起再训练的计算开销。实验结果表明，该模型在两个问答数据集上显著提高了传感器基线的性能。此外，它在命名实体的识别上比浅融合好大约7个相对值;当两者结合时，相对改善增加到13%。摘要：End-to-end speech recognition models are improved by incorporating external text sources, typically by fusion with an external language model. Such language models have to be retrained whenever the corpus of interest changes. Furthermore, since they store the entire corpus in their parameters, rare words can be challenging to recall. In this work, we propose augmenting a transducer-based ASR model with a retrieval language model, which directly retrieves from an external text corpus plausible completions for a partial ASR hypothesis. These completions are then integrated into subsequent predictions by an adapter, which is trained once, so that the corpus of interest can be switched without incurring the computational overhead of retraining. Our experiments show that the proposed model significantly improves the performance of a transducer baseline on a pair of question-answering datasets. Further, it outperforms shallow fusion on recognition of named entities by about 7 relative; when the two are combined, the relative improvement increases to 13%.【6】 Exploring Representation Learning for Small-Footprint Keyword Spotting标题：基于表征学习的小足迹关键词识别方法研究链接：http://arxiv.org/pdf/2303.10912v1
作者：Fan Cui,Liyong Guo,Quandong Wang,Peng Gao,Yujun Wang机构：Xiaomi, Beijing, China摘要：本文研究了面向低资源关键词识别的表示学习算法。KWS的主要挑战是有限的标记数据和有限的可用设备资源。为了解决这些问题，我们通过自我监督的对比学习和预训练模型的自我训练来探索知识工作系统的表征学习。首先，设计了局部-全局对比度连体网络（LGCSiam），通过局部-全局对比度损失学习相似音频样本的相似话语级别表示，而不需要地面真实值。其次，采用自监督预训练的Wav2Vec2.0模型作为约束模块（WVC），强制KWS模型学习帧级声学表示。通过LGCSiam和WVC模块，可以使用未标记数据对小足迹KWS模型进行预训练。在语音命令数据集上的实验表明，自训练WVC模块和自监督LGCSiam模块显著提高了准确率，尤其是在小标注数据集上训练时。摘要：In this paper, we investigate representation learning for low-resource keyword spotting (KWS). The main challenges of KWS are limited labeled data and limited available device resources. To address those challenges, we explore representation learning for KWS by self-supervised contrastive learning and self-training with pretrained model. First, local-global contrastive siamese networks (LGCSiam) are designed to learn similar utterance-level representations for similar audio samplers by proposed local-global contrastive loss without requiring ground-truth. Second, a self-supervised pretrained Wav2Vec 2.0 model is applied as a constraint module (WVC) to force the KWS model to learn frame-level acoustic representations. By the LGCSiam and WVC modules, the proposed small-footprint KWS model can be pretrained with unlabeled data. Experiments on speech commands dataset show that the self-training WVC module and the self-supervised LGCSiam module significantly improve accuracy, especially in the case of training on a small labeled dataset.【7】 Relate auditory speech to EEG by shallow-deep attention-based network标题：基于浅-深注意力网络的听觉语音与脑电信号关联链接：http://arxiv.org/pdf/2303.10897v1
作者：Fan Cui,Liyong Guo,Lang He,Jiyao Liu,ErCheng Pei,Yujun Wang,Dongmei Jiang机构：Xiaomi Corp., Beijing, China, Xi’an University of Posts and Telecommunications, Xi’an, China, Northwestern Polytechnical University, Xi’an, China, Peng Cheng Laboratory, ShenZhen, China摘要：脑电图（EEG）在检测大脑对不同刺激的反应方面起着至关重要的作用。本文提出了一种新的基于浅-深注意的神经网络（SDANet），用于对诱发脑电信号的听觉刺激进行分类。该算法采用基于注意力的相关模块（ACM）从全局角度发现听觉语音和脑电之间的联系，并采用深浅相似度分类模块（SDSCM）通过对浅层和深层的嵌入学习来确定分类结果。此外，采用多种训练策略和数据扩充来增强模型的鲁棒性。实验在听觉脑电挑战赛（ICASSP Signal Processing Grand Challenge 2023）提供的数据集上进行。结果表明，该模型在匹配-失配跟踪上相对于基线有明显的增益。摘要：Electroencephalography (EEG) plays a vital role in detecting how brain responses to different stimulus. In this paper, we propose a novel Shallow-Deep Attention-based Network (SDANet) to classify the correct auditory stimulus evoking the EEG signal. It adopts the Attention-based Correlation Module (ACM) to discover the connection between auditory speech and EEG from global aspect, and the Shallow-Deep Similarity Classification Module (SDSCM) to decide the classification result via the embeddings learned from the shallow and deep layers. Moreover, various training strategies and data augmentation are used to boost the model robustness. Experiments are conducted on the dataset provided by Auditory EEG challenge (ICASSP Signal Processing Grand Challenge 2023). Results show that the proposed model has a significant gain over the baseline on the match-mismatch track.【8】 Multiscale Audio Spectrogram Transformer for Efficient Audio Classification标题：用于高效音频分类的多尺度音频频谱转换器链接：http://arxiv.org/pdf/2303.10757v1
作者：Wentao Zhu,Mohamed Omar机构：⋆ Amazon备注：ICASSP 2023摘要：音频事件在时间和频率上都具有层次结构，可以组合在一起构造更抽象的语义音频类。在这项工作中，我们开发了一个多尺度音频谱图Transformer（MAST），它采用层次表示学习来进行有效的音频分类。MAST算法在不同阶段沿时域（频域）采用一维（和二维）池化算子，逐步减少特征点数量，增加特征维数.在没有外部训练数据的情况下，MAST在Kinetics-Sounds、Epic-Kitchens-100和VGGSound上的准确率分别比AST高22.2%、4.4%和4.7%.在下载的AudioSet数据集上，有超过20%的音频丢失，MAST也实现了比AST略好的准确性。此外，MAST在乘加（MACs）方面比AST效率高5倍，参数数量减少42%。通过聚类度量和可视化，我们证明了所提出的MAST可以从音频信号中学习语义上更可分离的特征表示。摘要：Audio event has a hierarchical architecture in both time and frequency and can be grouped together to construct more abstract semantic audio classes. In this work, we develop a multiscale audio spectrogram Transformer (MAST) that employs hierarchical representation learning for efficient audio classification. Specifically, MAST employs one-dimensional (and two-dimensional) pooling operators along the time (and frequency domains) in different stages, and progressively reduces the number of tokens and increases the feature dimensions. MAST significantly outperforms AST~ cite{gong2021ast} by 22.2 %, 4.4 % and 4.7 % on Kinetics-Sounds, Epic-Kitchens-100 and VGGSound in terms of the top-1 accuracy without external training data. On the downloaded AudioSet dataset, which has over 20 % missing audios, MAST also achieves slightly better accuracy than AST. In addition, MAST is 5x more efficient in terms of multiply-accumulates (MACs) with 42 % reduction in the number of parameters compared to AST. Through clustering metrics and visualizations, we demonstrate that the proposed MAST can learn semantically more separable feature representations from audio signals.【9】 ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement标题：ERSAM：寻找节能、实时的社会环境测量的神经结构链接：http://arxiv.org/pdf/2303.10727v1
作者：Chaojian Li,Wenwan Chen,Jiayi Yuan,Yingyan,Lin,Ashutosh Sabharwal机构：Georgia Institute of Technology, Atlanta, GA, Rice University, Houston, TX备注：Accepted by ICASSP'23摘要：社会氛围描述了社会互动发生的背景，并且可以使用语音音频通过计数同时说话的人数来测量。这种测量方法支持各种心理健康跟踪和以人为中心的物联网应用。虽然设备上的社会氛围测量（SAM）对于确保用户隐私并因此促进上述应用的广泛采用是非常期望的，但是由最先进的深度神经网络（DNN）供电的SAM解决方案所需的计算复杂性与移动设备上经常受限的资源相矛盾。此外，由于各种隐私约束和所需的人力，当涉及临床设置下的SAM时，只有有限的标记数据可用或实用，这进一步挑战了设备上SAM解决方案的可实现精度。为此，提出了一种专用于节能实时SAM的神经结构搜索框架（ERSAM）。具体而言，我们的ERSAM框架可以自动搜索DNN，以推进移动SAM解决方案的可实现精度与硬件效率的边界。例如，ERSAM交付的DNN在Pixel 3电话上5秒音频段仅消耗40 mW x 12 h能量和0.05秒处理延迟，而在LibriSpeech生成的社交氛围数据集上仅实现14.3%的错误率。我们可以期待ERSAM框架能够为需求日益增长的无处不在的设备上SAM解决方案铺平道路。摘要：Social ambiance describes the context in which social interactions happen, and can be measured using speech audio by counting the number of concurrent speakers. This measurement has enabled various mental health tracking and human-centric IoT applications. While on-device Socal Ambiance Measure (SAM) is highly desirable to ensure user privacy and thus facilitate wide adoption of the aforementioned applications, the required computational complexity of state-of-the-art deep neural networks (DNNs) powered SAM solutions stands at odds with the often constrained resources on mobile devices. Furthermore, only limited labeled data is available or practical when it comes to SAM under clinical settings due to various privacy constraints and the required human effort, further challenging the achievable accuracy of on-device SAM solutions. To this end, we propose a dedicated neural architecture search framework for Energy-efficient and Real-time SAM (ERSAM). Specifically, our ERSAM framework can automatically search for DNNs that push forward the achievable accuracy vs. hardware efficiency frontier of mobile SAM solutions. For example, ERSAM-delivered DNNs only consume 40 mW x 12 h energy and 0.05 seconds processing latency for a 5 seconds audio segment on a Pixel 3 phone, while only achieving an error rate of 14.3% on a social ambiance dataset generated by LibriSpeech. We can expect that our ERSAM framework can pave the way for ubiquitous on-device SAM solutions which are in growing demand.【10】 Right the docs: Characterising voice dataset documentation practices used in machine learning标题：正确的文档：描述机器学习中使用的语音数据集文档实践链接：http://arxiv.org/pdf/2303.10721v1
作者：Kathy Reid,Elizabeth T. Williams机构：School of Cybernetics, Australian National University, Canberra, Australia备注：16 pages, 3 tables, preprint of a submission to AIES 2023摘要：语音使能技术正迅速变得普遍存在，并且由诸如语音识别和语音活动检测的机器学习（ML）使能组件构成。然而，这些系统并不适用于所有人。他们在年龄、性别和口音等方面表现出偏见----系统和不公平地歧视个人或群体，而偏袒他人（Friedman & Nissembaum，1996年）。ML依赖于大数据集进行训练。数据集文档旨在让ML从业者（MLP）更好地理解数据集的特征。然而，目前还缺乏专门针对语音数据集文档的实证研究。此外，虽然MLP是公平性研究的频繁参与者，但很少有工作关注那些处理语音数据的人。我们的工作对这一差距做出了经验性的贡献。在此，我们将两种方法结合起来，形成一个探索性研究。首先，我们进行了13次半结构化访谈，探索语音数据集文档实践的多个角度。使用开放和轴向编码方法，我们通过角色和权衡的镜头来探索MLP的实践。从这项工作中，我们然后有目的地采样9个语音数据集的语音数据集文档（VDD）。我们的发现然后三角这两种方法，使用镜头的MLP的作用和权衡。我们发现，目前的VDD做法是不成熟的，不充分的和不相称的。语音数据集的特征是以碎片化、不相交的方式编码的，这通常不满足MLP的需要。此外，它们不容易比较，这对从业人员减少偏倚的努力构成了障碍。然后，我们讨论了这些发现对语音数据和语音技术中的偏见实践的影响。我们最后提出了一个未来工作的计划来解决这些发现--也就是说，我们如何“纠正这些文档”。摘要：Voice-enabled technology is quickly becoming ubiquitous, and is constituted from machine learning (ML)-enabled components such as speech recognition and voice activity detection. However, these systems don't yet work well for everyone. They exhibit bias - the systematic and unfair discrimination against individuals or cohorts of individuals in favour of others (Friedman & Nissembaum, 1996) - across axes such as age, gender and accent. ML is reliant on large datasets for training. Dataset documentation is designed to give ML Practitioners (MLPs) a better understanding of a dataset's characteristics. However, there is a lack of empirical research on voice dataset documentation specifically. Additionally, while MLPs are frequent participants in fairness research, little work focuses on those who work with voice data. Our work makes an empirical contribution to this gap. Here, we combine two methods to form an exploratory study. First, we undertake 13 semi-structured interviews, exploring multiple perspectives of voice dataset documentation practice. Using open and axial coding methods, we explore MLPs' practices through the lenses of roles and tradeoffs. Drawing from this work, we then purposively sample voice dataset documents (VDDs) for 9 voice datasets. Our findings then triangulate these two methods, using the lenses of MLP roles and trade-offs. We find that current VDD practices are inchoate, inadequate and incommensurate. The characteristics of voice datasets are codified in fragmented, disjoint ways that often do not meet the needs of MLPs. Moreover, they cannot be readily compared, presenting a barrier to practitioners' bias reduction efforts. We then discuss the implications of these findings for bias practices in voice data and speech technologies. We conclude by setting out a program of future work to address these findings -- that is, how we may "right the docs".【11】 Audio-Text Models Do Not Yet Leverage Natural Language标题：音频文本模型还没有利用自然语言链接：http://arxiv.org/pdf/2303.10667v1
作者：Ho-Hsiang Wu,Oriol Nieto,Juan Pablo Bello,Justin Salamon机构：Music and Audio Research Laboratory, New York University, New York, NY, USA, Adobe Research, San Francisco, CA, USA备注：Copyright 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprintingrepublishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：多模态对比学习技术在音频-文本领域迅速成为一个非常活跃的研究领域。大多数作品是用标准音频检索和分类基准来评估的，假设（i）这些模型能够利用自然语言中包含的丰富信息，以及（ii）当前基准能够捕捉这种信息的细微差别。在这项工作中，我们表明，最先进的音频文本模型还没有真正理解自然语言，特别是上下文概念，如顺序或并发排序的声音事件。我们的结果表明，现有的基准不足以评估这些模型的能力，以匹配复杂的上下文从音频和文本模态。我们提出了一种基于Transformer的架构，并证明了与现有技术不同，该架构能够在给定适当基准数据的情况下，对文本和音频中声音事件之间的顺序关系进行建模。我们提倡收集或生成额外的、多样化的数据，以允许未来的研究充分利用自然语言进行音频文本建模。摘要：Multi-modal contrastive learning techniques in the audio-text domain have quickly become a highly active area of research. Most works are evaluated with standard audio retrieval and classification benchmarks assuming that (i) these models are capable of leveraging the rich information contained in natural language, and (ii) current benchmarks are able to capture the nuances of such information. In this work, we show that state-of-the-art audio-text models do not yet really understand natural language, especially contextual concepts such as sequential or concurrent ordering of sound events. Our results suggest that existing benchmarks are not sufficient to assess these models' capabilities to match complex contexts from the audio and text modalities. We propose a Transformer-based architecture and show that, unlike prior work, it is capable of modeling the sequential relationship between sound events in the text and audio, given appropriate benchmark data. We advocate for the collection or generation of additional, diverse, data to allow future research to fully leverage natural language for audio-text modeling.【12】 Textless Speech-to-Music Retrieval Using Emotion Similarity标题：基于情感相似度的无文本语音到音乐检索链接：http://arxiv.org/pdf/2303.10539v1
作者：SeungHeon Doh,Minz Won,Keunwoo Choi,Juhan Nam机构：♭Graduate School of Culture Technology, KAIST, South Korea, ♮ByteDance, USA, ♯Gaudio Lab, South Korea备注：To Appear IEEE ICASSP 2023摘要：我们介绍了一个基于语音情感推荐音乐的框架。在内容创作和日常生活中，语音包含着人类情感的信息，音乐可以增强这些信息。我们的架构着重于一个跨领域的检索系统，透过情感标签来桥接语音与音乐之间的鸿沟。我们探讨了不同的言语表征，并报告了它们对不同言语类型的影响，包括行动声和唤醒词。我们还提出了一个情感相似性规则化项用于跨领域检索任务。通过将正则化项合并到训练中，情感空间中相似的语音和音乐对在联合嵌入空间中更接近。实验结果表明，该模型在无文本语音音乐检索中是有效的。摘要：We introduce a framework that recommends music based on the emotions of speech. In content creation and daily life, speech contains information about human emotions, which can be enhanced by music. Our framework focuses on a cross-domain retrieval system to bridge the gap between speech and music via emotion labels. We explore different speech representations and report their impact on different speech types, including acting voice and wake-up words. We also propose an emotion similarity regularization term in cross-domain retrieval tasks. By incorporating the regularization term into training, similar speech-and-music pairs in the emotion space are closer in the joint embedding space. Our comprehensive experimental results show that the proposed model is effective in textless speech-to-music retrieval.【13】 A Deep Learning System for Domain-specific speech Recognition标题：一种面向领域语音识别的深度学习系统链接：http://arxiv.org/pdf/2303.10510v1
作者：Yanan Jia机构：Businessolver备注：4th International Conference on Natural Language Processing and Computational Linguistics (NLPCL 2023)摘要：随着人机语音接口提供对日益智能化的机器的容易访问，提出了许多最先进的自动语音识别（ASR）系统。然而，商业ASR系统通常对特定领域语音的性能较差，尤其是在低资源环境下。作者使用预先训练的DeepSpeech 2和Wav 2 Vec 2声学模型来开发特定于受益的ASR系统。特定领域的数据使用所提出的半监督学习标注来收集，几乎没有人为干预。最佳性能来自于带有外部KenLM的微调Wav 2 Vec 2-Large-LV 60声学模型，该模型在特定利益语音方面超过了Google和AWS ASR系统。使用易错ASR转录作为口语理解（SLU）的一部分的可行性也进行了调查。特定收益自然语言理解（NLU）任务的测试结果表明，即使在文本误词率（WER）较高的情况下，特定领域的精调ASR系统仍能优于商用ASR系统，且精调ASR与人工文本的测试结果相似。摘要：As human-machine voice interfaces provide easy access to increasingly intelligent machines, many state-of-the-art automatic speech recognition (ASR) systems are proposed. However, commercial ASR systems usually have poor performance on domain-specific speech especially under low-resource settings. The author works with pre-trained DeepSpeech2 and Wav2Vec2 acoustic models to develop benefit-specific ASR systems. The domain-specific data are collected using proposed semi-supervised learning annotation with little human intervention. The best performance comes from a fine-tuned Wav2Vec2-Large-LV60 acoustic model with an external KenLM, which surpasses the Google and AWS ASR systems on benefit-specific speech. The viability of using error prone ASR transcriptions as part of spoken language understanding (SLU) is also investigated. Results of a benefit-specific natural language understanding (NLU) task show that the domain-specific fine-tuned ASR system can outperform the commercial ASR systems even when its transcriptions have higher word error rate (WER), and the results between fine-tuned ASR and human transcriptions are similar.【14】 A Content Adaptive Learnable Time-Frequency Representation For Audio Signal Processing标题：一种用于音频信号处理的内容自适应学习时频表示链接：http://arxiv.org/pdf/2303.10446v1
作者：Prateek Verma,Chris Chafe机构：Stanford University备注：5 pages, 4 figures. 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing, Rhodes, Greece摘要：我们提出一个可学习的内容适应性音讯处理前端。在现代深度学习出现之前，我们使用固定表示的不可学习前端，如带 不带神经架构的谱图或梅尔谱图。随着卷积架构支持ASR和声学场景理解等各种应用，出现了向可学习前端的转变，其中基函数的类型和权重都是从头学习的，并针对感兴趣的特定任务进行优化。随着向不存在卷积块的基于变压器的架构的转变，线性层将小波形补丁投影到小的潜在维度上，然后将它们馈送到变压器架构。在这项工作中，我们提出了一种计算内容自适应可学习时频表示的方法。我们让每个音频信号通过一组卷积滤波器，每个滤波器给出一个固定维向量。这类似于学习一组有限脉冲响应滤波器组并根据输入信号的内容使输入信号通过最佳滤波器组。内容自适应的可学习时频表示可能有更广泛的应用，超出了本文的实验。摘要：We propose a learnable content adaptive front end for audio signal processing. Before the modern advent of deep learning, we used fixed representation non-learnable front-ends like spectrogram or mel-spectrogram with without neural architectures. With convolutional architectures supporting various applications such as ASR and acoustic scene understanding, a shift to a learnable front ends occurred in which both the type of basis functions and the weight were learned from scratch and optimized for the particular task of interest. With the shift to transformer-based architectures with no convolutional blocks present, a linear layer projects small waveform patches onto a small latent dimension before feeding them to a transformer architecture. In this work, we propose a way of computing a content-adaptive learnable time-frequency representation. We pass each audio signal through a bank of convolutional filters, each giving a fixed-dimensional vector. It is akin to learning a bank of finite impulse-response filterbanks and passing the input signal through the optimum filter bank depending on the content of the input signal. A content-adaptive learnable time-frequency representation may be more broadly applicable, beyond the experiments in this paper.【15】 EarCough: Enabling Continuous Subject Cough Event Detection on Hearables标题：EarCough：在可听对象上启用连续主题咳嗽事件检测链接：http://arxiv.org/pdf/2303.10445v1
作者：Xiyuxing Zhang,Yuntao Wang,Jingru Zhang,Yaqing Yang,Shwetak Patel,Yuanchun Shi机构：Department of Computer Science and, Technology, Tsinghua University, China, University of Washington, United States备注：This paper has been accepted by ACM CHI 2023摘要：咳嗽监测可以实现新的个人肺部健康应用。受试者咳嗽事件检测是连续咳嗽监测的基础。最近，智能可听设备的快速增长为此类需求打开了新的机遇。提出了EarCough算法，利用始终开启的主动噪声消除（ANC）麦克风，实现边缘计算可闻设备上的连续咳嗽事件检测。具体而言，我们提出了一种轻量级的端到端神经网络模型-- EarCoughNet。为了评估该方法的有效性，我们通过用户研究构建了一个同步的运动和音频数据集。结果表明，EarCough的准确率为95.4%，F1评分为92.9%，所需空间仅为385 kB。我们将EarCough视为未来助听器的低成本附加设备，以实现连续受试者咳嗽事件检测。摘要：Cough monitoring can enable new individual pulmonary health applications. Subject cough event detection is the foundation for continuous cough monitoring. Recently, the rapid growth in smart hearables has opened new opportunities for such needs. This paper proposes EarCough, which enables continuous subject cough event detection on edge computing hearables by leveraging the always-on active noise cancellation (ANC) microphones. Specifically, we proposed a lightweight end-to-end neural network model -- EarCoughNet. To evaluate the effectiveness of our method, we constructed a synchronous motion and audio dataset through a user study. Results show that EarCough achieved an accuracy of 95.4% and an F1-score of 92.9% with a space requirement of only 385 kB. We envision EarCough as a low-cost add-on for future hearables to enable continuous subject cough event detection.【16】 Weight-sharing Supernet for Searching Specialized Acoustic Event Classification Networks Across Device Constraints标题：跨设备约束搜索专业声事件分类网络的加权超网链接：http://arxiv.org/pdf/2303.10351v1
作者：Guan-Ting Lin,Qingming Tang,Chieh-Chi Kao,Viktor Rozgic,Chao Wang机构：National Taiwan University, Amazon Alexa备注：Accepted by ICASSP 2023摘要：声学事件分类（AEC）已广泛应用于智能扬声器和移动电话等设备中，以实现家庭安全或无障碍支持。随着AEC模型在越来越多具有不同计算资源约束的设备上运行，开发针对每个给定计算资源约束进行调整以实现最佳精度 计算权衡的模型变得越来越昂贵。本文介绍了一种用于AEC的一次性（OFA）神经结构搜索（NAS）框架。具体来说，我们首先训练一个支持不同模型架构的权重共享超网，然后在给定特定计算资源约束的情况下自动搜索模型。我们的实验结果表明，通过仅仅训练一次，从NAS得到的模型显著优于从零开始单独训练的模型和知识蒸馏的模型（25.4%和7.3%相对改进）。我们还发现，超小模型的权重共享超网训练的好处不仅来自搜索，而且来自优化。摘要：Acoustic Event Classification (AEC) has been widely used in devices such as smart speakers and mobile phones for home safety or accessibility support. As AEC models run on more and more devices with diverse computation resource constraints, it became increasingly expensive to develop models that are tuned to achieve optimal accuracy computation trade-off for each given computation resource constraint. In this paper, we introduce a Once-For-All (OFA) Neural Architecture Search (NAS) framework for AEC. Specifically, we first train a weight-sharing supernet that supports different model architectures, followed by automatically searching for a model given specific computational resource constraints. Our experimental results showed that by just training once, the resulting model from NAS significantly outperforms both models trained individually from scratch and knowledge distillation (25.4% and 7.3% relative improvement). We also found that the benefit of weight-sharing supernet training of ultra-small models comes not only from searching but from optimization.【17】 Multimodal Continuous Emotion Recognition: A Technical Report for ABAW5标题：多模式连续情感识别：ABAW5的技术报告链接：http://arxiv.org/pdf/2303.10335v1
作者：Su Zhang,Ziyuan Zhao,Cuntai Guan机构：Nanyang Technological University备注：4 pages. arXiv admin note: substantial text overlap with arXiv:2203.13031摘要：我们使用两个多模态模型，利用视觉、听觉和语言信息进行连续效价唤醒识别。第一个模型与我们在ABAW 2和ABAW 3中使用的相同，它使用了领导者-跟随者注意。第二模型具有用于空间和时间编码的相同架构。至于融合块，它采用了一种紧凑而直接的通道关注，这是从End 2 You工具包中借来的。与我们之前直接使用VGG特征作为音频特征的尝试不同，这次我们使用logmel谱图来馈送预先训练的VGG模型，并在训练期间对其进行微调。为了充分利用数据并避免过拟合，进行了交叉验证。选择一致性相关系数最高的倍数进行提交。该代码可在www.example.com上获得https: github.com sucv ABAW5。摘要：We used two multimodal models for continuous valence-arousal recognition using visual, audio, and linguistic information. The first model is the same as we used in ABAW2 and ABAW3, which employs the leader-follower attention. The second model has the same architecture for spatial and temporal encoding. As for the fusion block, it employs a compact and straightforward channel attention, borrowed from the End2You toolkit. Unlike our previous attempts that use Vggish feature directly as the audio feature, this time we feed the pre-trained VGG model using logmel-spectrogram and finetune it during the training. To make full use of the data and alleviate over-fitting, cross-validation is carried out. The fold with the highest concordance correlation coefficient is selected for submission. The code is to be available at https: github.com sucv ABAW5.【18】 Zero-shot Sound Event Classification Using a Sound Attribute Vector with Global and Local Feature Learning标题：基于全局和局部特征学习的声音属性向量零发声事件分类链接：http://arxiv.org/pdf/2303.10316v1
作者：Yi-Han Lin,Xunquan Chen,Ryoichi Takashima,Tetsuya Takiguchi机构：Graduate School of System Informatics, Kobe University, Japan备注：Accepted by ICASSP2023摘要：提出了一种zero-shot声音事件分类（ZS-SEC）方法，用于识别训练数据中从未出现过的声音事件。在我们之前的工作中，我们提出了一种使用声音属性向量（SAVs）的ZS-SEC方法，其中深度神经网络模型推断描述事件类声音的属性信息，而不是直接推断其类标签。我们以前的方法表明，它可以在一定程度上分类看不见的事件;然而，对于看不见的事件的准确度远低于对于看到的事件的准确度。本文提出了一种新的ZS-SEC方法，该方法可以同时学习判别性全局特征和局部特征，以增强基于SAV的ZS-SEC。该方法在学习全局特征以区分训练数据中的事件类别的同时，学习时空局部特征以利用属性原型回归属性信息。实验结果表明，该方法可以提高SAV的ZS-SEC的准确性，并能可视化谱图中与各属性相关的区域。摘要：This paper introduces a zero-shot sound event classification (ZS-SEC) method to identify sound events that have never occurred in training data. In our previous work, we proposed a ZS-SEC method using sound attribute vectors (SAVs), where a deep neural network model infers attribute information that describes the sound of an event class instead of inferring its class label directly. Our previous method showed that it could classify unseen events to some extent; however, the accuracy for unseen events was far inferior to that for seen events. In this paper, we propose a new ZS-SEC method that can learn discriminative global features and local features simultaneously to enhance SAV-based ZS-SEC. In the proposed method, while the global features are learned in order to discriminate the event classes in the training data, the spectro-temporal local features are learned in order to regress the attribute information using attribute prototypes. The experimental results show that our proposed method can improve the accuracy of SAV-based ZS-SEC and can visualize the region in the spectrogram related to each attribute.【19】 Code-Switching Text Generation and Injection in Mandarin-English ASR标题：汉英ASR中码转换文本的生成与注入链接：http://arxiv.org/pdf/2303.10949v1
作者：Haibin Yu,Yuxuan Hu,Yao Qian,Ma Jin,Linquan Liu,Shujie Liu,Yu Shi,Yanmin Qian,Edward Lin,Michael Zeng机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Microsoft Corporation备注：Accepted by ICASSP 2023摘要：语码转换言语是指在同一话语中混合使用两种或两种以上语言的一种表达方式。由于缺乏数据，对这种语音进行端到端（E2 E）建模的自动语音识别（ASR）可能是一项具有挑战性的任务。在本研究中，我们研究了文本生成和注入，以提高工业上常用的流模型，转换器-转换器（T-T），在汉语-英语代码切换语音识别的性能。首先提出了一种生成语码转换文本数据的策略，然后研究了通过文本到语音（TTS）转换显式地或通过绑定语音和文本潜在空间隐式地将生成的文本注入到T-T模型中。在包含1，800小时真实汉语-英语语码转换语音的数据集上训练的T-T模型的实验结果表明，我们的注入生成的语码转换文本的方法显著提高了T-T模型的性能，即：在三个评估集上平均相对降低了16%的基于标记的错误率（TER），并且在包含与训练集更同质的数据的评估集上，捆绑语音和文本潜在空间的方法优于TTS转换。摘要：Code-switching speech refers to a means of expression by mixing two or more languages within a single utterance. Automatic Speech Recognition (ASR) with End-to-End (E2E) modeling for such speech can be a challenging task due to the lack of data. In this study, we investigate text generation and injection for improving the performance of an industry commonly-used streaming model, Transformer-Transducer (T-T), in Mandarin-English code-switching speech recognition. We first propose a strategy to generate code-switching text data and then investigate injecting generated text into T-T model explicitly by Text-To-Speech (TTS) conversion or implicitly by tying speech and text latent spaces. Experimental results on the T-T model trained with a dataset containing 1,800 hours of real Mandarin-English code-switched speech show that our approaches to inject generated code-switching text significantly boost the performance of T-T models, i.e., 16% relative Token-based Error Rate (TER) reduction averaged on three evaluation sets, and the approach of tying speech and text latent spaces is superior to that of TTS conversion on the evaluation set which contains more homogeneous data with the training set.【20】 Approaching an unknown communication system by latent space exploration and causal inference标题：用潜在空间探索和因果推理接近未知的通信系统链接：http://arxiv.org/pdf/2303.10931v1
作者：Gašper Beguš,Andrej Leban,Shane Gero机构：University of California, Berkeley, Berkeley, CA, United States, Department of Biology, Carleton University, Ottawa, Canada, Project CETI, New York, NY, United States, The Dominica Sperm Whale Project, Roseau, Dominica备注：25 pages, 23 figures摘要：提出了一种通过挖掘无监督深度生成模型的潜在空间来发现数据中有意义属性的方法。我们将单个潜在变量操纵到训练范围之外的极值与受因果推理启发的方法结合到一种我们称为极值因果解纠缠（CDEV）的方法中，并表明这种方法产生了对模型可解释性的见解。使用这种技术，我们可以推断模型将未知数据的哪些属性编码为有意义的。我们应用这种方法来测试抹香鲸的交流系统中有意义的东西，抹香鲸是最有趣和研究不足的动物交流系统之一。我们训练一个网络，这个网络已经被证明能够学习有意义的语音表征，并测试我们是否能够利用这种无监督的学习来破译另一个我们没有地面真理的语音通信系统的属性。这项提议的技术表明，抹香鲸利用序列中的滴答声数量、时间的规律性以及序列的频谱平均值和声学规律性等音频特性来编码信息。其中一些发现与现有假设一致，而另一些则是首次提出。我们还认为，我们的模型揭示了抹香鲸通信系统中控制通信单元结构的规则，并在生成训练期间没有显示的创新数据时应用这些规则。本文认为，用因果方法解释深度神经网络的输出可能是接近未知数据的可行策略，并介绍了深度学习如何限制假设空间的另一个案例。最后，将潜在空间操作和因果推理相结合的方法可以推广到其他体系结构和任意数据集。摘要：This paper proposes a methodology for discovering meaningful properties in data by exploring the latent space of unsupervised deep generative models. We combine manipulation of individual latent variables to extreme values outside the training range with methods inspired by causal inference into an approach we call causal disentanglement with extreme values (CDEV) and show that this approach yields insights for model interpretability. Using this technique, we can infer what properties of unknown data the model encodes as meaningful. We apply the methodology to test what is meaningful in the communication system of sperm whales, one of the most intriguing and understudied animal communication systems. We train a network that has been shown to learn meaningful representations of speech and test whether we can leverage such unsupervised learning to decipher the properties of another vocal communication system for which we have no ground truth. The proposed technique suggests that sperm whales encode information using the number of clicks in a sequence, the regularity of their timing, and audio properties such as the spectral mean and the acoustic regularity of the sequences. Some of these findings are consistent with existing hypotheses, while others are proposed for the first time. We also argue that our models uncover rules that govern the structure of communication units in the sperm whale communication system and apply them while generating innovative data not shown during training. This paper suggests that an interpretation of the outputs of deep neural networks with causal methodology can be a viable strategy for approaching data about which little is known and presents another case of how deep learning can limit the hypothesis space. Finally, the proposed approach combining latent space manipulation and causal inference can be extended to other architectures and arbitrary datasets.【21】 Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition标题：端到端语音识别中多个基础模型的知识提取链接：http://arxiv.org/pdf/2303.10917v1
作者：Xiaoyu Yang,Qiujia Li,Chao Zhang,Philip C. Woodland摘要：尽管通过自监督学习预先训练的大型基础模型在包括自动语音识别（ASR）的许多任务中已经实现了最先进的性能，但是在实践中经常需要知识提炼（KD），以将由大型教师模型学习的知识转移到具有可负担的计算和存储成本的小得多的学生模型中。本文提出了一种新的两阶段知识发现框架，将多个语音基础模型中的知识提取到单个学生神经元模型中，用于ASR。在第一阶段，使用从多个教师模型提取的嵌入来预训练学生模型编码器。在第二阶段中，基于ASR任务用音频-文本对微调学生编码器。在LibriSpeech 100小时子集上的实验结果表明，在只有一个教师的情况下，所提出的KD框架能够提高流媒体和非流媒体学生模型的性能。当多个教师联合使用时，学生模型的性能可以进一步提高，实现了17.5%和10.6%的单词错误率减少（WERR）。我们提出的框架可以与其他现有的知识发现方法相结合，以实现进一步的改进。通过在编码器预训练期间合并额外的未标记数据获得了进一步的WERR，导致非流式学生模型上的总相对WERR为55.0%。摘要：Although large foundation models pre-trained by self-supervised learning have achieved state-of-the-art performance in many tasks including automatic speech recognition (ASR), knowledge distillation (KD) is often required in practice to transfer the knowledge learned by large teacher models into much smaller student models with affordable computation and memory costs. This paper proposes a novel two-stage KD framework to distil the knowledge from multiple speech foundation models as teachers into a single student neural transducer model for ASR. In the first stage, the student model encoder is pre-trained using the embeddings extracted from multiple teacher models. In the second stage, the student encoder is fine-tuned with the audio-text pairs based on the ASR task. Experiments on the LibriSpeech 100-hour subset show that the proposed KD framework improves the performance of both streaming and non-streaming student models when using only one teacher. The performance of the student model can be further enhanced when multiple teachers are used jointly, achieving word error rate reductions (WERRs) of 17.5% and 10.6%. Our proposed framework can be combined with other existing KD methods to achieve further improvements. Further WERRs were obtained by incorporating extra unlabelled data during encoder pre-training, leading to a total relative WERR of 55.0% on the non-streaming student model.【22】 The Graph feature fusion technique for speaker recognition based on wav2vec2.0 framework标题：基于Wave2ve2.0框架的说话人识别图特征融合技术链接：http://arxiv.org/pdf/2303.10556v1
作者：Zirui Ge,Haiyan Guo,Zhen Yang机构：School of Communication and Information Engineering, Nanjing University of Posts and, Telecommunications, Nanjing , China摘要：预训练的wav2vec2. 0模型在说话人识别中的有效性得到了验证。然而，现有的特征处理方法主要集中在对wav2vec2. 0模型训练后的输出特征进行经典的合并，如均值合并、最大值合并等，这些方法将特征作为独立的、不相关的单元，忽略了各特征之间的相互关系，没有将特征作为说话人的整体表示。GRU（Gated Recurrent Unit）作为一种特征融合方法，也可以看作是一种复杂的池化技术，它主要关注时间信息，在一些主要信息不在时间维度上的情况下性能可能会很差。本文研究了基于wav2vec2. 0框架的图形神经网络（GNN）后端处理模块，为上述问题提供了一种解决方案。GNN将输出的所有特征作为图信号数据，提取特征的相关图结构信息，用于说话人识别。具体地说，我们首先从理论上简单地证明了GNN特征融合方法优于均值、最大值、随机合并等方法。然后，将wav2vec2. 0的输出特征建模为图的顶点，利用图注意网络（GAT）构造图的邻接矩阵.最后，我们依据讯息传递类神经网路设计讯息函式、顶点更新函式及读出函式，将说话人特徴转换成图特徴。实验表明，与基线方法相比，我们的性能可以提供相对的改进。代码可在xxx获得。摘要：Pre-trained wav2vec2.0 model has been proved its effectiveness for speaker recognition. However, current feature processing methods are focusing on classical pooling on the output features of the pre-trained wav2vec2.0 model, such as mean pooling, max pooling etc. That methods take the features as the independent and irrelevant units, ignoring the inter-relationship among all the features, and do not take the features as an overall representation of a speaker. Gated Recurrent Unit (GRU), as a feature fusion method, can also be considered as a complicated pooling technique, mainly focuses on the temporal information, which may show poor performance in some situations that the main information is not on the temporal dimension. In this paper, we investigate the graph neural network (GNN) as a backend processing module based on wav2vec2.0 framework to provide a solution for the mentioned matters. The GNN takes all the output features as the graph signal data and extracts the related graph structure information of features for speaker recognition. Specifically, we first give a simple proof that the GNN feature fusion method can outperform than the mean, max, random pooling methods and so on theoretically. Then, we model the output features of wav2vec2.0 as the vertices of a graph, and construct the graph adjacency matrix by graph attention network (GAT). Finally, we follow the message passing neural network (MPNN) to design our message function, vertex update function and readout function to transform the speaker features into the graph features. The experiments show our performance can provide a relative improvement compared to the baseline methods. Code is available at xxx.【23】 Powerful and Extensible WFST Framework for RNN-Transducer Losses标题：功能强大且可扩展的RNN换能器损耗WFST框架链接：http://arxiv.org/pdf/2303.10384v1
作者：Aleksandr Laptev,Vladimir Bataev,Igor Gitman,Boris Ginsburg机构：NVIDIA, USA, ITMO University, St. Petersburg, Russia, University of London, London, UK备注：To appear in Proc. ICASSP 2023, June 04-10, 2023, Rhodes island, Greece. 5 pages, 5 figures, 3 tables摘要：本文提出了一种基于加权有限状态变换器（WFST）的框架，以简化RNN变换器（RNN-T）损耗修正的开发。RNN-T的现有实现使用CUDA相关代码，这很难扩展和调试。WFST易于构造和扩展，并允许通过可视化进行调试。我们介绍了两种支持WFST的RNN-T实现：（1）“合成-转换器”，基于来自声学和文本图式的WFST图的合成-在计算上有竞争力并且易于修改;（2）“网格转换器”，它直接构造用于进一步计算的网格--最紧凑，并且计算效率高。我们通过引入一个新的W转换器损耗--用通配符修改连接主义时态分类来说明可扩展性的简易性。W-转换器（W-RNNT）在弱监督数据设置中始终优于标准RNN-T，该弱监督数据设置在话语的开始和结束处具有缺失的转录部分。所有RNN-T损失都是用k2框架实现的，并可在NeMo工具包中获得。摘要：This paper presents a framework based on Weighted Finite-State Transducers (WFST) to simplify the development of modifications for RNN-Transducer (RNN-T) loss. Existing implementations of RNN-T use CUDA-related code, which is hard to extend and debug. WFSTs are easy to construct and extend, and allow debugging through visualization. We introduce two WFST-powered RNN-T implementations: (1) "Compose-Transducer", based on a composition of the WFST graphs from acoustic and textual schema -- computationally competitive and easy to modify; (2) "Grid-Transducer", which constructs the lattice directly for further computations -- most compact, and computationally efficient. We illustrate the ease of extensibility through introduction of a new W-Transducer loss -- the adaptation of the Connectionist Temporal Classification with Wild Cards. W-Transducer (W-RNNT) consistently outperforms the standard RNN-T in a weakly-supervised data setup with missing parts of transcriptions at the beginning and end of utterances. All RNN-T losses are implemented with the k2 framework and are available in the NeMo toolkit.【1】 Code-Switching Text Generation and Injection in Mandarin-English ASR标题：汉英ASR中码转换文本的生成与注入链接：http://arxiv.org/pdf/2303.10949v1
*与cs.SD语音【19】为同一篇
作者：Haibin Yu,Yuxuan Hu,Yao Qian,Ma Jin,Linquan Liu,Shujie Liu,Yu Shi,Yanmin Qian,Edward Lin,Michael Zeng机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Microsoft Corporation备注：Accepted by ICASSP 2023摘要：语码转换言语是指在同一话语中混合使用两种或两种以上语言的一种表达方式。由于缺乏数据，对这种语音进行端到端（E2 E）建模的自动语音识别（ASR）可能是一项具有挑战性的任务。在本研究中，我们研究了文本生成和注入，以提高工业上常用的流模型，转换器-转换器（T-T），在汉语-英语代码切换语音识别的性能。首先提出了一种生成语码转换文本数据的策略，然后研究了通过文本到语音（TTS）转换显式地或通过绑定语音和文本潜在空间隐式地将生成的文本注入到T-T模型中。在包含1，800小时真实汉语-英语语码转换语音的数据集上训练的T-T模型的实验结果表明，我们的注入生成的语码转换文本的方法显著提高了T-T模型的性能，即：在三个评估集上平均相对降低了16%的基于标记的错误率（TER），并且在包含与训练集更同质的数据的评估集上，捆绑语音和文本潜在空间的方法优于TTS转换。摘要：Code-switching speech refers to a means of expression by mixing two or more languages within a single utterance. Automatic Speech Recognition (ASR) with End-to-End (E2E) modeling for such speech can be a challenging task due to the lack of data. In this study, we investigate text generation and injection for improving the performance of an industry commonly-used streaming model, Transformer-Transducer (T-T), in Mandarin-English code-switching speech recognition. We first propose a strategy to generate code-switching text data and then investigate injecting generated text into T-T model explicitly by Text-To-Speech (TTS) conversion or implicitly by tying speech and text latent spaces. Experimental results on the T-T model trained with a dataset containing 1,800 hours of real Mandarin-English code-switched speech show that our approaches to inject generated code-switching text significantly boost the performance of T-T models, i.e., 16% relative Token-based Error Rate (TER) reduction averaged on three evaluation sets, and the approach of tying speech and text latent spaces is superior to that of TTS conversion on the evaluation set which contains more homogeneous data with the training set.【2】 Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition标题：端到端语音识别中多个基础模型的知识提取链接：http://arxiv.org/pdf/2303.10917v1
*与cs.SD语音【21】为同一篇
作者：Xiaoyu Yang,Qiujia Li,Chao Zhang,Philip C. Woodland摘要：尽管通过自监督学习预先训练的大型基础模型在包括自动语音识别（ASR）的许多任务中已经实现了最先进的性能，但是在实践中经常需要知识提炼（KD），以将由大型教师模型学习的知识转移到具有可负担的计算和存储成本的小得多的学生模型中。本文提出了一种新的两阶段知识发现框架，将多个语音基础模型中的知识提取到单个学生神经元模型中，用于ASR。在第一阶段，使用从多个教师模型提取的嵌入来预训练学生模型编码器。在第二阶段中，基于ASR任务用音频-文本对微调学生编码器。在LibriSpeech 100小时子集上的实验结果表明，在只有一个教师的情况下，所提出的KD框架能够提高流媒体和非流媒体学生模型的性能。当多个教师联合使用时，学生模型的性能可以进一步提高，实现了17.5%和10.6%的单词错误率减少（WERR）。我们提出的框架可以与其他现有的知识发现方法相结合，以实现进一步的改进。通过在编码器预训练期间合并额外的未标记数据获得了进一步的WERR，导致非流式学生模型上的总相对WERR为55.0%。摘要：Although large foundation models pre-trained by self-supervised learning have achieved state-of-the-art performance in many tasks including automatic speech recognition (ASR), knowledge distillation (KD) is often required in practice to transfer the knowledge learned by large teacher models into much smaller student models with affordable computation and memory costs. This paper proposes a novel two-stage KD framework to distil the knowledge from multiple speech foundation models as teachers into a single student neural transducer model for ASR. In the first stage, the student model encoder is pre-trained using the embeddings extracted from multiple teacher models. In the second stage, the student encoder is fine-tuned with the audio-text pairs based on the ASR task. Experiments on the LibriSpeech 100-hour subset show that the proposed KD framework improves the performance of both streaming and non-streaming student models when using only one teacher. The performance of the student model can be further enhanced when multiple teachers are used jointly, achieving word error rate reductions (WERRs) of 17.5% and 10.6%. Our proposed framework can be combined with other existing KD methods to achieve further improvements. Further WERRs were obtained by incorporating extra unlabelled data during encoder pre-training, leading to a total relative WERR of 55.0% on the non-streaming student model.【3】 The Graph feature fusion technique for speaker recognition based on wav2vec2.0 framework标题：基于Wave2ve2.0框架的说话人识别图特征融合技术链接：http://arxiv.org/pdf/2303.10556v1
*与cs.SD语音【22】为同一篇
作者：Zirui Ge,Haiyan Guo,Zhen Yang机构：School of Communication and Information Engineering, Nanjing University of Posts and, Telecommunications, Nanjing , China摘要：预训练的wav2vec2. 0模型在说话人识别中的有效性得到了验证。然而，现有的特征处理方法主要集中在对wav2vec2. 0模型训练后的输出特征进行经典的合并，如均值合并、最大值合并等，这些方法将特征作为独立的、不相关的单元，忽略了各特征之间的相互关系，没有将特征作为说话人的整体表示。GRU（Gated Recurrent Unit）作为一种特征融合方法，也可以看作是一种复杂的池化技术，它主要关注时间信息，在一些主要信息不在时间维度上的情况下性能可能会很差。本文研究了基于wav2vec2. 0框架的图形神经网络（GNN）后端处理模块，为上述问题提供了一种解决方案。GNN将输出的所有特征作为图信号数据，提取特征的相关图结构信息，用于说话人识别。具体地说，我们首先从理论上简单地证明了GNN特征融合方法优于均值、最大值、随机合并等方法。然后，将wav2vec2. 0的输出特征建模为图的顶点，利用图注意网络（GAT）构造图的邻接矩阵.最后，我们依据讯息传递类神经网路设计讯息函式、顶点更新函式及读出函式，将说话人特徴转换成图特徴。实验表明，与基线方法相比，我们的性能可以提供相对的改进。代码可在xxx获得。摘要：Pre-trained wav2vec2.0 model has been proved its effectiveness for speaker recognition. However, current feature processing methods are focusing on classical pooling on the output features of the pre-trained wav2vec2.0 model, such as mean pooling, max pooling etc. That methods take the features as the independent and irrelevant units, ignoring the inter-relationship among all the features, and do not take the features as an overall representation of a speaker. Gated Recurrent Unit (GRU), as a feature fusion method, can also be considered as a complicated pooling technique, mainly focuses on the temporal information, which may show poor performance in some situations that the main information is not on the temporal dimension. In this paper, we investigate the graph neural network (GNN) as a backend processing module based on wav2vec2.0 framework to provide a solution for the mentioned matters. The GNN takes all the output features as the graph signal data and extracts the related graph structure information of features for speaker recognition. Specifically, we first give a simple proof that the GNN feature fusion method can outperform than the mean, max, random pooling methods and so on theoretically. Then, we model the output features of wav2vec2.0 as the vertices of a graph, and construct the graph adjacency matrix by graph attention network (GAT). Finally, we follow the message passing neural network (MPNN) to design our message function, vertex update function and readout function to transform the speaker features into the graph features. The experiments show our performance can provide a relative improvement compared to the baseline methods. Code is available at xxx.【4】 Powerful and Extensible WFST Framework for RNN-Transducer Losses标题：功能强大且可扩展的RNN换能器损耗WFST框架链接：http://arxiv.org/pdf/2303.10384v1
*与cs.SD语音【23】为同一篇
作者：Aleksandr Laptev,Vladimir Bataev,Igor Gitman,Boris Ginsburg机构：NVIDIA, USA, ITMO University, St. Petersburg, Russia, University of London, London, UK备注：To appear in Proc. ICASSP 2023, June 04-10, 2023, Rhodes island, Greece. 5 pages, 5 figures, 3 tables摘要：本文提出了一种基于加权有限状态变换器（WFST）的框架，以简化RNN变换器（RNN-T）损耗修正的开发。RNN-T的现有实现使用CUDA相关代码，这很难扩展和调试。WFST易于构造和扩展，并允许通过可视化进行调试。我们介绍了两种支持WFST的RNN-T实现：（1）“合成-转换器”，基于来自声学和文本图式的WFST图的合成-在计算上有竞争力并且易于修改;（2）“网格转换器”，它直接构造用于进一步计算的网格--最紧凑，并且计算效率高。我们通过引入一个新的W转换器损耗--用通配符修改连接主义时态分类来说明可扩展性的简易性。W-转换器（W-RNNT）在弱监督数据设置中始终优于标准RNN-T，该弱监督数据设置在话语的开始和结束处具有缺失的转录部分。所有RNN-T损失都是用k2框架实现的，并可在NeMo工具包中获得。摘要：This paper presents a framework based on Weighted Finite-State Transducers (WFST) to simplify the development of modifications for RNN-Transducer (RNN-T) loss. Existing implementations of RNN-T use CUDA-related code, which is hard to extend and debug. WFSTs are easy to construct and extend, and allow debugging through visualization. We introduce two WFST-powered RNN-T implementations: (1) "Compose-Transducer", based on a composition of the WFST graphs from acoustic and textual schema -- computationally competitive and easy to modify; (2) "Grid-Transducer", which constructs the lattice directly for further computations -- most compact, and computationally efficient. We illustrate the ease of extensibility through introduction of a new W-Transducer loss -- the adaptation of the Connectionist Temporal Classification with Wild Cards. W-Transducer (W-RNNT) consistently outperforms the standard RNN-T in a weakly-supervised data setup with missing parts of transcriptions at the beginning and end of utterances. All RNN-T losses are implemented with the k2 framework and are available in the NeMo toolkit.【5】 Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation标题：从运动中定位声音：联合学习声音方向和摄像机旋转链接：http://arxiv.org/pdf/2303.11329v1
*与cs.SD语音【1】为同一篇
作者：Ziyang Chen,Shengyi Qian,Andrew Owens机构：University of Michigan备注：Project site: this https URL摘要：当我们转动头部时，我们感知到的图像和声音会发生微妙但几何学上一致的变化。在本文中，我们使用这些线索来解决一个我们称之为运动声音定位（SLfM）的问题：联合估计摄像机旋转和定位声源。我们学会了仅仅通过自我监督来解决这些任务。视觉模型根据一对图像预测相机旋转，而音频模型根据双耳声音预测声源的方向。我们训练这些模型以产生彼此一致的预测。在测试时，模型可以独立部署。为了获得非常适合于解决这个挑战性问题的特征表示，我们还提出了一种用于通过跨视图双耳化来学习视听表示的方法：从一个视图估计双耳声，给定来自另一视图的图像和声音。我们的模型可以成功地估计真实场景和合成场景中的精确旋转，并以与最先进的自监督方法相竞争的精度定位声源。项目地点：https: ificl.github.io SLfM摘要：The images and sounds that we perceive undergo subtle but geometrically consistent changes as we rotate our heads. In this paper, we use these cues to solve a problem we call Sound Localization from Motion (SLfM): jointly estimating camera rotation and localizing sound sources. We learn to solve these tasks solely through self-supervision. A visual model predicts camera rotation from a pair of images, while an audio model predicts the direction of sound sources from binaural sounds. We train these models to generate predictions that agree with one another. At test time, the models can be deployed independently. To obtain a feature representation that is well-suited to solving this challenging problem, we also propose a method for learning an audio-visual representation through cross-view binauralization: estimating binaural sound from one view, given images and sound from another. Our model can successfully estimate accurate rotations on both real and synthetic scenes, and localize sound sources with accuracy competitive with state-of-the-art self-supervised approaches. Project site: https: ificl.github.io SLfM.【6】 Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speech标题：鸡尾酒休BERT：混合语音和单源语音的广义自监督预训练链接：http://arxiv.org/pdf/2303.11131v1
*与cs.SD语音【2】为同一篇
作者：Maryam Fazel-Zarandi,Wei-Ning Hsu机构：Meta AI - FAIR备注：ICASSP 2023摘要：自监督学习有效地利用了未标记数据，提高了标记效率和对没有标记数据的领域的泛化能力。虽然最近的工作已经研究了对更多声学 语言学领域、语言和模态的概括，但这些研究仅限于录音中有一个主要说话者的单源语音。本文提出了一种自监督学习框架Cocktail HuBERT，它使用掩蔽伪源分离目标推广到混合语音。该目标鼓励模型识别源的数量，分离和理解上下文，并推断表示为发现单元的掩蔽区域的内容。鸡尾酒HuBERT优于最先进的结果，在多扬声器ASR上WER降低69%，在diarization上DER降低31%，在SUPERB的单扬声器和多扬声器任务上具有竞争力。摘要：Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data. While recent work has studied generalization to more acoustic linguistic domains, languages, and modalities, these investigations are limited to single-source speech with one primary speaker in the recording. This paper presents Cocktail HuBERT, a self-supervised learning framework that generalizes to mixture speech using a masked pseudo source separation objective. This objective encourages the model to identify the number of sources, separate and understand the context, and infer the content of masked regions represented as discovered units. Cocktail HuBERT outperforms state-of-the-art results with 69% lower WER on multi-speaker ASR, 31% lower DER on diarization, and is competitive on single- and multi-speaker tasks from SUPERB.【7】 EmoTalk: Speech-driven emotional disentanglement for 3D face animation标题：EmoTalk：语音驱动的3D人脸动画情感解脱链接：http://arxiv.org/pdf/2303.11089v1
*与cs.SD语音【3】为同一篇
作者：Ziqiao Peng,Haoyu Wu,Zhenbo Song,Hao Xu,Xiangyu Zhu,Hongyan Liu,Jun He,Zhaoxin Fan机构：Renmin University of China, Nanjing University of Science and Technology, The Hong Kong University of Science and Technology, Chinese Academy of Sciences, Tsinghua University, Psyche AI Inc.摘要：Transformer在多人自注意的支持下，在语音情感识别（SER）中取得了显著的效果。与原有的全注意机制相比，基于窗口的注意机制在学习细粒度特征时更有效，同时大大减少了模型冗余。然而，情感线索以多粒度方式呈现，使得预定义固定窗口可严重降低模型灵活性。此外，很难手动获得最佳窗口设置。本文提出了一种用于SER任务的可变形语音Transformer DST。DST通过一个轻量级的判决网络确定以输入语音为条件的窗口大小的使用。同时，利用基于声学特征的数据相关偏移量来调整注意窗口的位置，使DST能够自适应地发现和注意语音中的有价值信息。在IEMOCAP和MELD上的大量实验证明了DST的优越性。摘要：Enabled by multi-head self-attention, Transformer has exhibited remarkable results in speech emotion recognition (SER). Compared to the original full attention mechanism, window-based attention is more effective in learning fine-grained features while greatly reducing model redundancy. However, emotional cues are present in a multi-granularity manner such that the pre-defined fixed window can severely degrade the model flexibility. In addition, it is difficult to obtain the optimal window settings manually. In this paper, we propose a Deformable Speech Transformer, named DST, for SER task. DST determines the usage of window sizes conditioned on input speech via a light-weight decision network. Meanwhile, data-dependent offsets derived from acoustic features are utilized to adjust the positions of the attention windows, allowing DST to adaptively discover and attend to the valuable information embedded in the speech. Extensive experiments on IEMOCAP and MELD demonstrate the superiority of DST.【8】 Dual-stream Time-Delay Neural Network with Dynamic Global Filter for Speaker Verification标题：基于动态全局滤波的双流时延神经网络说话人确认链接：http://arxiv.org/pdf/2303.11020v1
*与cs.SD语音【4】为同一篇
作者：Yangfu Li,Xiaodan Lin机构：School of Information Science and Engineering, Huaqiao University备注：13 pages 4 figures摘要：时延神经网络（TDNN）是文本无关说话人确认的最新模型之一。然而，传统的TDNN难以捕捉全局上下文，而全局上下文在许多最近的工作中已经被证明是鲁棒的说话人表示和长时间说话人确认的关键。此外，常见的解决方案，例如，自注意，对于输入标记具有二次复杂度，这使得它们在应用于TDNN中具有大尺寸的特征图时在计算上难以承受。为了解决这些问题，我们提出了TDNN的全局滤波器，它应用对数线性复杂度FFT IFFT和一组可微分频域滤波器来有效地建模语音中的长期相关性。此外，为了提高全局滤波器的性能，防止过拟合，还特别设计了动态滤波策略和稀疏正则化方法。此外，我们还构造了一种双流TDNN（DS-TDNN），它通过分割基本通道来降低复杂度，并使用全局滤波器来提高识别性能。在Voxceleb和SITW数据库上的实验结果表明，DS-TDNN与ECAPA-TDNN相比，在复杂度和参数上分别降低了28%和15%，性能提高了约10%。此外，在面对长时间语音时，与其他主流的基线系统相比，它在效率和有效性之间具有最佳的折衷。最后，可视化和详细的消融研究进一步揭示了DS-TDNN的优势。摘要：The time-delay neural network (TDNN) is one of the state-of-the-art models for text-independent speaker verification. However, it is difficult for conventional TDNN to capture global context that has been proven critical for robust speaker representations and long-duration speaker verification in many recent works. Besides, the common solutions, e.g., self-attention, have quadratic complexity for input tokens, which makes them computationally unaffordable when applied to the feature maps with large sizes in TDNN. To address these issues, we propose the Global Filter for TDNN, which applies log-linear complexity FFT IFFT and a set of differentiable frequency-domain filters to efficiently model the long-term dependencies in speech. Besides, a dynamic filtering strategy, and a sparse regularization method are specially designed to enhance the performance of the global filter and prevent it from overfitting. Furthermore, we construct a dual-stream TDNN (DS-TDNN), which splits the basic channels for complexity reduction and employs the global filter to increase recognition performance. Experiments on Voxceleb and SITW databases show that the DS-TDNN achieves approximate 10% improvement with a decline over 28% and 15% in complexity and parameters compared with the ECAPA-TDNN. Besides, it has the best trade-off between efficiency and effectiveness compared with other popular baseline systems when facing long-duration speech. Finally, visualizations and a detailed ablation study further reveal the advantages of the DS-TDNN.【9】 On-the-fly Text Retrieval for End-to-End ASR Adaptation标题：用于端到端ASR适配的即时文本检索链接：http://arxiv.org/pdf/2303.10942v1
*与cs.SD语音【5】为同一篇
作者：Bolaji Yusuf,Aditya Gourav,Ankur Gandhe,Ivan Bulyko机构：Amazon Alexa, USA, Bo˘gazic¸i University, Department of Electrical and Electronics Engineering, Istanbul, Turkey备注：Accepted to ICASSP 2023; Appendix added to include ablations that could not fit into the conference 4-page limit摘要：端到端语音识别模型通过合并外部文本源来改进，典型地通过与外部语言模型融合来改进。每当感兴趣的语料库改变时，这种语言模型必须被重新训练。此外，由于它们将整个语料库存储在其参数中，罕见单词可能难以回忆。在这项工作中，我们提出用检索语言模型来增强基于转换器的ASR模型，该检索语言模型直接从外部文本语料库中检索部分ASR假设的似然完成。然后，这些完成由适配器集成到后续预测中，适配器被训练一次，使得感兴趣语料库可以被切换而不引起再训练的计算开销。实验结果表明，该模型在两个问答数据集上显著提高了传感器基线的性能。此外，它在命名实体的识别上比浅融合好大约7个相对值;当两者结合时，相对改善增加到13%。摘要：End-to-end speech recognition models are improved by incorporating external text sources, typically by fusion with an external language model. Such language models have to be retrained whenever the corpus of interest changes. Furthermore, since they store the entire corpus in their parameters, rare words can be challenging to recall. In this work, we propose augmenting a transducer-based ASR model with a retrieval language model, which directly retrieves from an external text corpus plausible completions for a partial ASR hypothesis. These completions are then integrated into subsequent predictions by an adapter, which is trained once, so that the corpus of interest can be switched without incurring the computational overhead of retraining. Our experiments show that the proposed model significantly improves the performance of a transducer baseline on a pair of question-answering datasets. Further, it outperforms shallow fusion on recognition of named entities by about 7 relative; when the two are combined, the relative improvement increases to 13%.【10】 Approaching an unknown communication system by latent space exploration and causal inference标题：用潜在空间探索和因果推理接近未知的通信系统链接：http://arxiv.org/pdf/2303.10931v1
*与cs.SD语音【20】为同一篇
作者：Gašper Beguš,Andrej Leban,Shane Gero机构：University of California, Berkeley, Berkeley, CA, United States, Department of Biology, Carleton University, Ottawa, Canada, Project CETI, New York, NY, United States, The Dominica Sperm Whale Project, Roseau, Dominica备注：25 pages, 23 figures摘要：提出了一种通过挖掘无监督深度生成模型的潜在空间来发现数据中有意义属性的方法。我们将单个潜在变量操纵到训练范围之外的极值与受因果推理启发的方法结合到一种我们称为极值因果解纠缠（CDEV）的方法中，并表明这种方法产生了对模型可解释性的见解。使用这种技术，我们可以推断模型将未知数据的哪些属性编码为有意义的。我们应用这种方法来测试抹香鲸的交流系统中有意义的东西，抹香鲸是最有趣和研究不足的动物交流系统之一。我们训练一个网络，这个网络已经被证明能够学习有意义的语音表征，并测试我们是否能够利用这种无监督的学习来破译另一个我们没有地面真理的语音通信系统的属性。这项提议的技术表明，抹香鲸利用序列中的滴答声数量、时间的规律性以及序列的频谱平均值和声学规律性等音频特性来编码信息。其中一些发现与现有假设一致，而另一些则是首次提出。我们还认为，我们的模型揭示了抹香鲸通信系统中控制通信单元结构的规则，并在生成训练期间没有显示的创新数据时应用这些规则。本文认为，用因果方法解释深度神经网络的输出可能是接近未知数据的可行策略，并介绍了深度学习如何限制假设空间的另一个案例。最后，将潜在空间操作和因果推理相结合的方法可以推广到其他体系结构和任意数据集。摘要：This paper proposes a methodology for discovering meaningful properties in data by exploring the latent space of unsupervised deep generative models. We combine manipulation of individual latent variables to extreme values outside the training range with methods inspired by causal inference into an approach we call causal disentanglement with extreme values (CDEV) and show that this approach yields insights for model interpretability. Using this technique, we can infer what properties of unknown data the model encodes as meaningful. We apply the methodology to test what is meaningful in the communication system of sperm whales, one of the most intriguing and understudied animal communication systems. We train a network that has been shown to learn meaningful representations of speech and test whether we can leverage such unsupervised learning to decipher the properties of another vocal communication system for which we have no ground truth. The proposed technique suggests that sperm whales encode information using the number of clicks in a sequence, the regularity of their timing, and audio properties such as the spectral mean and the acoustic regularity of the sequences. Some of these findings are consistent with existing hypotheses, while others are proposed for the first time. We also argue that our models uncover rules that govern the structure of communication units in the sperm whale communication system and apply them while generating innovative data not shown during training. This paper suggests that an interpretation of the outputs of deep neural networks with causal methodology can be a viable strategy for approaching data about which little is known and presents another case of how deep learning can limit the hypothesis space. Finally, the proposed approach combining latent space manipulation and causal inference can be extended to other architectures and arbitrary datasets.【11】 Exploring Representation Learning for Small-Footprint Keyword Spotting标题：基于表征学习的小足迹关键词识别方法研究链接：http://arxiv.org/pdf/2303.10912v1
*与cs.SD语音【6】为同一篇
作者：Fan Cui,Liyong Guo,Quandong Wang,Peng Gao,Yujun Wang机构：Xiaomi, Beijing, China摘要：本文研究了面向低资源关键词识别的表示学习算法。KWS的主要挑战是有限的标记数据和有限的可用设备资源。为了解决这些问题，我们通过自我监督的对比学习和预训练模型的自我训练来探索知识工作系统的表征学习。首先，设计了局部-全局对比度连体网络（LGCSiam），通过局部-全局对比度损失学习相似音频样本的相似话语级别表示，而不需要地面真实值。其次，采用自监督预训练的Wav2Vec2.0模型作为约束模块（WVC），强制KWS模型学习帧级声学表示。通过LGCSiam和WVC模块，可以使用未标记数据对小足迹KWS模型进行预训练。在语音命令数据集上的实验表明，自训练WVC模块和自监督LGCSiam模块显著提高了准确率，尤其是在小标注数据集上训练时。摘要：In this paper, we investigate representation learning for low-resource keyword spotting (KWS). The main challenges of KWS are limited labeled data and limited available device resources. To address those challenges, we explore representation learning for KWS by self-supervised contrastive learning and self-training with pretrained model. First, local-global contrastive siamese networks (LGCSiam) are designed to learn similar utterance-level representations for similar audio samplers by proposed local-global contrastive loss without requiring ground-truth. Second, a self-supervised pretrained Wav2Vec 2.0 model is applied as a constraint module (WVC) to force the KWS model to learn frame-level acoustic representations. By the LGCSiam and WVC modules, the proposed small-footprint KWS model can be pretrained with unlabeled data. Experiments on speech commands dataset show that the self-training WVC module and the self-supervised LGCSiam module significantly improve accuracy, especially in the case of training on a small labeled dataset.【12】 Relate auditory speech to EEG by shallow-deep attention-based network标题：基于浅-深注意力网络的听觉语音与脑电信号关联链接：http://arxiv.org/pdf/2303.10897v1
*与cs.SD语音【7】为同一篇
作者：Fan Cui,Liyong Guo,Lang He,Jiyao Liu,ErCheng Pei,Yujun Wang,Dongmei Jiang机构：Xiaomi Corp., Beijing, China, Xi’an University of Posts and Telecommunications, Xi’an, China, Northwestern Polytechnical University, Xi’an, China, Peng Cheng Laboratory, ShenZhen, China摘要：脑电图（EEG）在检测大脑对不同刺激的反应方面起着至关重要的作用。本文提出了一种新的基于浅-深注意的神经网络（SDANet），用于对诱发脑电信号的听觉刺激进行分类。该算法采用基于注意力的相关模块（ACM）从全局角度发现听觉语音和脑电之间的联系，并采用深浅相似度分类模块（SDSCM）通过对浅层和深层的嵌入学习来确定分类结果。此外，采用多种训练策略和数据扩充来增强模型的鲁棒性。实验在听觉脑电挑战赛（ICASSP Signal Processing Grand Challenge 2023）提供的数据集上进行。结果表明，该模型在匹配-失配跟踪上相对于基线有明显的增益。摘要：Electroencephalography (EEG) plays a vital role in detecting how brain responses to different stimulus. In this paper, we propose a novel Shallow-Deep Attention-based Network (SDANet) to classify the correct auditory stimulus evoking the EEG signal. It adopts the Attention-based Correlation Module (ACM) to discover the connection between auditory speech and EEG from global aspect, and the Shallow-Deep Similarity Classification Module (SDSCM) to decide the classification result via the embeddings learned from the shallow and deep layers. Moreover, various training strategies and data augmentation are used to boost the model robustness. Experiments are conducted on the dataset provided by Auditory EEG challenge (ICASSP Signal Processing Grand Challenge 2023). Results show that the proposed model has a significant gain over the baseline on the match-mismatch track.【13】 Multiscale Audio Spectrogram Transformer for Efficient Audio Classification标题：用于高效音频分类的多尺度音频频谱转换器链接：http://arxiv.org/pdf/2303.10757v1
*与cs.SD语音【8】为同一篇
作者：Wentao Zhu,Mohamed Omar机构：⋆ Amazon备注：ICASSP 2023摘要：音频事件在时间和频率上都具有层次结构，可以组合在一起构造更抽象的语义音频类。在这项工作中，我们开发了一个多尺度音频谱图Transformer（MAST），它采用层次表示学习来进行有效的音频分类。MAST算法在不同阶段沿时域（频域）采用一维（和二维）池化算子，逐步减少特征点数量，增加特征维数.在没有外部训练数据的情况下，MAST在Kinetics-Sounds、Epic-Kitchens-100和VGGSound上的准确率分别比AST高22.2%、4.4%和4.7%.在下载的AudioSet数据集上，有超过20%的音频丢失，MAST也实现了比AST略好的准确性。此外，MAST在乘加（MACs）方面比AST效率高5倍，参数数量减少42%。通过聚类度量和可视化，我们证明了所提出的MAST可以从音频信号中学习语义上更可分离的特征表示。摘要：Audio event has a hierarchical architecture in both time and frequency and can be grouped together to construct more abstract semantic audio classes. In this work, we develop a multiscale audio spectrogram Transformer (MAST) that employs hierarchical representation learning for efficient audio classification. Specifically, MAST employs one-dimensional (and two-dimensional) pooling operators along the time (and frequency domains) in different stages, and progressively reduces the number of tokens and increases the feature dimensions. MAST significantly outperforms AST~ cite{gong2021ast} by 22.2 %, 4.4 % and 4.7 % on Kinetics-Sounds, Epic-Kitchens-100 and VGGSound in terms of the top-1 accuracy without external training data. On the downloaded AudioSet dataset, which has over 20 % missing audios, MAST also achieves slightly better accuracy than AST. In addition, MAST is 5x more efficient in terms of multiply-accumulates (MACs) with 42 % reduction in the number of parameters compared to AST. Through clustering metrics and visualizations, we demonstrate that the proposed MAST can learn semantically more separable feature representations from audio signals.【14】 ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement标题：ERSAM：寻找节能、实时的社会环境测量的神经结构链接：http://arxiv.org/pdf/2303.10727v1
*与cs.SD语音【9】为同一篇
作者：Chaojian Li,Wenwan Chen,Jiayi Yuan,Yingyan,Lin,Ashutosh Sabharwal机构：Georgia Institute of Technology, Atlanta, GA, Rice University, Houston, TX备注：Accepted by ICASSP'23摘要：社会氛围描述了社会互动发生的背景，并且可以使用语音音频通过计数同时说话的人数来测量。这种测量方法支持各种心理健康跟踪和以人为中心的物联网应用。虽然设备上的社会氛围测量（SAM）对于确保用户隐私并因此促进上述应用的广泛采用是非常期望的，但是由最先进的深度神经网络（DNN）供电的SAM解决方案所需的计算复杂性与移动设备上经常受限的资源相矛盾。此外，由于各种隐私约束和所需的人力，当涉及临床设置下的SAM时，只有有限的标记数据可用或实用，这进一步挑战了设备上SAM解决方案的可实现精度。为此，提出了一种专用于节能实时SAM的神经结构搜索框架（ERSAM）。具体而言，我们的ERSAM框架可以自动搜索DNN，以推进移动SAM解决方案的可实现精度与硬件效率的边界。例如，ERSAM交付的DNN在Pixel 3电话上5秒音频段仅消耗40 mW x 12 h能量和0.05秒处理延迟，而在LibriSpeech生成的社交氛围数据集上仅实现14.3%的错误率。我们可以期待ERSAM框架能够为需求日益增长的无处不在的设备上SAM解决方案铺平道路。摘要：Social ambiance describes the context in which social interactions happen, and can be measured using speech audio by counting the number of concurrent speakers. This measurement has enabled various mental health tracking and human-centric IoT applications. While on-device Socal Ambiance Measure (SAM) is highly desirable to ensure user privacy and thus facilitate wide adoption of the aforementioned applications, the required computational complexity of state-of-the-art deep neural networks (DNNs) powered SAM solutions stands at odds with the often constrained resources on mobile devices. Furthermore, only limited labeled data is available or practical when it comes to SAM under clinical settings due to various privacy constraints and the required human effort, further challenging the achievable accuracy of on-device SAM solutions. To this end, we propose a dedicated neural architecture search framework for Energy-efficient and Real-time SAM (ERSAM). Specifically, our ERSAM framework can automatically search for DNNs that push forward the achievable accuracy vs. hardware efficiency frontier of mobile SAM solutions. For example, ERSAM-delivered DNNs only consume 40 mW x 12 h energy and 0.05 seconds processing latency for a 5 seconds audio segment on a Pixel 3 phone, while only achieving an error rate of 14.3% on a social ambiance dataset generated by LibriSpeech. We can expect that our ERSAM framework can pave the way for ubiquitous on-device SAM solutions which are in growing demand.【15】 Right the docs: Characterising voice dataset documentation practices used in machine learning标题：正确的文档：描述机器学习中使用的语音数据集文档实践链接：http://arxiv.org/pdf/2303.10721v1
*与cs.SD语音【10】为同一篇
作者：Kathy Reid,Elizabeth T. Williams机构：School of Cybernetics, Australian National University, Canberra, Australia备注：16 pages, 3 tables, preprint of a submission to AIES 2023摘要：语音使能技术正迅速变得普遍存在，并且由诸如语音识别和语音活动检测的机器学习（ML）使能组件构成。然而，这些系统并不适用于所有人。他们在年龄、性别和口音等方面表现出偏见----系统和不公平地歧视个人或群体，而偏袒他人（Friedman & Nissembaum，1996年）。ML依赖于大数据集进行训练。数据集文档旨在让ML从业者（MLP）更好地理解数据集的特征。然而，目前还缺乏专门针对语音数据集文档的实证研究。此外，虽然MLP是公平性研究的频繁参与者，但很少有工作关注那些处理语音数据的人。我们的工作对这一差距做出了经验性的贡献。在此，我们将两种方法结合起来，形成一个探索性研究。首先，我们进行了13次半结构化访谈，探索语音数据集文档实践的多个角度。使用开放和轴向编码方法，我们通过角色和权衡的镜头来探索MLP的实践。从这项工作中，我们然后有目的地采样9个语音数据集的语音数据集文档（VDD）。我们的发现然后三角这两种方法，使用镜头的MLP的作用和权衡。我们发现，目前的VDD做法是不成熟的，不充分的和不相称的。语音数据集的特征是以碎片化、不相交的方式编码的，这通常不满足MLP的需要。此外，它们不容易比较，这对从业人员减少偏倚的努力构成了障碍。然后，我们讨论了这些发现对语音数据和语音技术中的偏见实践的影响。我们最后提出了一个未来工作的计划来解决这些发现--也就是说，我们如何“纠正这些文档”。摘要：Voice-enabled technology is quickly becoming ubiquitous, and is constituted from machine learning (ML)-enabled components such as speech recognition and voice activity detection. However, these systems don't yet work well for everyone. They exhibit bias - the systematic and unfair discrimination against individuals or cohorts of individuals in favour of others (Friedman & Nissembaum, 1996) - across axes such as age, gender and accent. ML is reliant on large datasets for training. Dataset documentation is designed to give ML Practitioners (MLPs) a better understanding of a dataset's characteristics. However, there is a lack of empirical research on voice dataset documentation specifically. Additionally, while MLPs are frequent participants in fairness research, little work focuses on those who work with voice data. Our work makes an empirical contribution to this gap. Here, we combine two methods to form an exploratory study. First, we undertake 13 semi-structured interviews, exploring multiple perspectives of voice dataset documentation practice. Using open and axial coding methods, we explore MLPs' practices through the lenses of roles and tradeoffs. Drawing from this work, we then purposively sample voice dataset documents (VDDs) for 9 voice datasets. Our findings then triangulate these two methods, using the lenses of MLP roles and trade-offs. We find that current VDD practices are inchoate, inadequate and incommensurate. The characteristics of voice datasets are codified in fragmented, disjoint ways that often do not meet the needs of MLPs. Moreover, they cannot be readily compared, presenting a barrier to practitioners' bias reduction efforts. We then discuss the implications of these findings for bias practices in voice data and speech technologies. We conclude by setting out a program of future work to address these findings -- that is, how we may "right the docs".【16】 Audio-Text Models Do Not Yet Leverage Natural Language标题：音频文本模型还没有利用自然语言链接：http://arxiv.org/pdf/2303.10667v1
*与cs.SD语音【11】为同一篇
作者：Ho-Hsiang Wu,Oriol Nieto,Juan Pablo Bello,Justin Salamon机构：Music and Audio Research Laboratory, New York University, New York, NY, USA, Adobe Research, San Francisco, CA, USA备注：Copyright 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprintingrepublishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：多模态对比学习技术在音频-文本领域迅速成为一个非常活跃的研究领域。大多数作品是用标准音频检索和分类基准来评估的，假设（i）这些模型能够利用自然语言中包含的丰富信息，以及（ii）当前基准能够捕捉这种信息的细微差别。在这项工作中，我们表明，最先进的音频文本模型还没有真正理解自然语言，特别是上下文概念，如顺序或并发排序的声音事件。我们的结果表明，现有的基准不足以评估这些模型的能力，以匹配复杂的上下文从音频和文本模态。我们提出了一种基于Transformer的架构，并证明了与现有技术不同，该架构能够在给定适当基准数据的情况下，对文本和音频中声音事件之间的顺序关系进行建模。我们提倡收集或生成额外的、多样化的数据，以允许未来的研究充分利用自然语言进行音频文本建模。摘要：Multi-modal contrastive learning techniques in the audio-text domain have quickly become a highly active area of research. Most works are evaluated with standard audio retrieval and classification benchmarks assuming that (i) these models are capable of leveraging the rich information contained in natural language, and (ii) current benchmarks are able to capture the nuances of such information. In this work, we show that state-of-the-art audio-text models do not yet really understand natural language, especially contextual concepts such as sequential or concurrent ordering of sound events. Our results suggest that existing benchmarks are not sufficient to assess these models' capabilities to match complex contexts from the audio and text modalities. We propose a Transformer-based architecture and show that, unlike prior work, it is capable of modeling the sequential relationship between sound events in the text and audio, given appropriate benchmark data. We advocate for the collection or generation of additional, diverse, data to allow future research to fully leverage natural language for audio-text modeling.【17】 Textless Speech-to-Music Retrieval Using Emotion Similarity标题：基于情感相似度的无文本语音到音乐检索链接：http://arxiv.org/pdf/2303.10539v1
*与cs.SD语音【12】为同一篇
作者：SeungHeon Doh,Minz Won,Keunwoo Choi,Juhan Nam机构：♭Graduate School of Culture Technology, KAIST, South Korea, ♮ByteDance, USA, ♯Gaudio Lab, South Korea备注：To Appear IEEE ICASSP 2023摘要：我们介绍了一个基于语音情感推荐音乐的框架。在内容创作和日常生活中，语音包含着人类情感的信息，音乐可以增强这些信息。我们的架构着重于一个跨领域的检索系统，透过情感标签来桥接语音与音乐之间的鸿沟。我们探讨了不同的言语表征，并报告了它们对不同言语类型的影响，包括行动声和唤醒词。我们还提出了一个情感相似性规则化项用于跨领域检索任务。通过将正则化项合并到训练中，情感空间中相似的语音和音乐对在联合嵌入空间中更接近。实验结果表明，该模型在无文本语音音乐检索中是有效的。摘要：We introduce a framework that recommends music based on the emotions of speech. In content creation and daily life, speech contains information about human emotions, which can be enhanced by music. Our framework focuses on a cross-domain retrieval system to bridge the gap between speech and music via emotion labels. We explore different speech representations and report their impact on different speech types, including acting voice and wake-up words. We also propose an emotion similarity regularization term in cross-domain retrieval tasks. By incorporating the regularization term into training, similar speech-and-music pairs in the emotion space are closer in the joint embedding space. Our comprehensive experimental results show that the proposed model is effective in textless speech-to-music retrieval.【18】 A Deep Learning System for Domain-specific speech Recognition标题：一种面向领域语音识别的深度学习系统链接：http://arxiv.org/pdf/2303.10510v1
*与cs.SD语音【13】为同一篇
作者：Yanan Jia机构：Businessolver备注：4th International Conference on Natural Language Processing and Computational Linguistics (NLPCL 2023)摘要：随着人机语音接口提供对日益智能化的机器的容易访问，提出了许多最先进的自动语音识别（ASR）系统。然而，商业ASR系统通常对特定领域语音的性能较差，尤其是在低资源环境下。作者使用预先训练的DeepSpeech 2和Wav 2 Vec 2声学模型来开发特定于受益的ASR系统。特定领域的数据使用所提出的半监督学习标注来收集，几乎没有人为干预。最佳性能来自于带有外部KenLM的微调Wav 2 Vec 2-Large-LV 60声学模型，该模型在特定利益语音方面超过了Google和AWS ASR系统。使用易错ASR转录作为口语理解（SLU）的一部分的可行性也进行了调查。特定收益自然语言理解（NLU）任务的测试结果表明，即使在文本误词率（WER）较高的情况下，特定领域的精调ASR系统仍能优于商用ASR系统，且精调ASR与人工文本的测试结果相似。摘要：As human-machine voice interfaces provide easy access to increasingly intelligent machines, many state-of-the-art automatic speech recognition (ASR) systems are proposed. However, commercial ASR systems usually have poor performance on domain-specific speech especially under low-resource settings. The author works with pre-trained DeepSpeech2 and Wav2Vec2 acoustic models to develop benefit-specific ASR systems. The domain-specific data are collected using proposed semi-supervised learning annotation with little human intervention. The best performance comes from a fine-tuned Wav2Vec2-Large-LV60 acoustic model with an external KenLM, which surpasses the Google and AWS ASR systems on benefit-specific speech. The viability of using error prone ASR transcriptions as part of spoken language understanding (SLU) is also investigated. Results of a benefit-specific natural language understanding (NLU) task show that the domain-specific fine-tuned ASR system can outperform the commercial ASR systems even when its transcriptions have higher word error rate (WER), and the results between fine-tuned ASR and human transcriptions are similar.【19】 A Content Adaptive Learnable Time-Frequency Representation For Audio Signal Processing标题：一种用于音频信号处理的内容自适应学习时频表示链接：http://arxiv.org/pdf/2303.10446v1
*与cs.SD语音【14】为同一篇
作者：Prateek Verma,Chris Chafe机构：Stanford University备注：5 pages, 4 figures. 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing, Rhodes, Greece摘要：我们提出一个可学习的内容适应性音讯处理前端。在现代深度学习出现之前，我们使用固定表示的不可学习前端，如带 不带神经架构的谱图或梅尔谱图。随着卷积架构支持ASR和声学场景理解等各种应用，出现了向可学习前端的转变，其中基函数的类型和权重都是从头学习的，并针对感兴趣的特定任务进行优化。随着向不存在卷积块的基于变压器的架构的转变，线性层将小波形补丁投影到小的潜在维度上，然后将它们馈送到变压器架构。在这项工作中，我们提出了一种计算内容自适应可学习时频表示的方法。我们让每个音频信号通过一组卷积滤波器，每个滤波器给出一个固定维向量。这类似于学习一组有限脉冲响应滤波器组并根据输入信号的内容使输入信号通过最佳滤波器组。内容自适应的可学习时频表示可能有更广泛的应用，超出了本文的实验。摘要：We propose a learnable content adaptive front end for audio signal processing. Before the modern advent of deep learning, we used fixed representation non-learnable front-ends like spectrogram or mel-spectrogram with without neural architectures. With convolutional architectures supporting various applications such as ASR and acoustic scene understanding, a shift to a learnable front ends occurred in which both the type of basis functions and the weight were learned from scratch and optimized for the particular task of interest. With the shift to transformer-based architectures with no convolutional blocks present, a linear layer projects small waveform patches onto a small latent dimension before feeding them to a transformer architecture. In this work, we propose a way of computing a content-adaptive learnable time-frequency representation. We pass each audio signal through a bank of convolutional filters, each giving a fixed-dimensional vector. It is akin to learning a bank of finite impulse-response filterbanks and passing the input signal through the optimum filter bank depending on the content of the input signal. A content-adaptive learnable time-frequency representation may be more broadly applicable, beyond the experiments in this paper.【20】 EarCough: Enabling Continuous Subject Cough Event Detection on Hearables标题：EarCough：在可听对象上启用连续主题咳嗽事件检测链接：http://arxiv.org/pdf/2303.10445v1
*与cs.SD语音【15】为同一篇
作者：Xiyuxing Zhang,Yuntao Wang,Jingru Zhang,Yaqing Yang,Shwetak Patel,Yuanchun Shi机构：Department of Computer Science and, Technology, Tsinghua University, China, University of Washington, United States备注：This paper has been accepted by ACM CHI 2023摘要：咳嗽监测可以实现新的个人肺部健康应用。受试者咳嗽事件检测是连续咳嗽监测的基础。最近，智能可听设备的快速增长为此类需求打开了新的机遇。提出了EarCough算法，利用始终开启的主动噪声消除（ANC）麦克风，实现边缘计算可闻设备上的连续咳嗽事件检测。具体而言，我们提出了一种轻量级的端到端神经网络模型-- EarCoughNet。为了评估该方法的有效性，我们通过用户研究构建了一个同步的运动和音频数据集。结果表明，EarCough的准确率为95.4%，F1评分为92.9%，所需空间仅为385 kB。我们将EarCough视为未来助听器的低成本附加设备，以实现连续受试者咳嗽事件检测。摘要：Cough monitoring can enable new individual pulmonary health applications. Subject cough event detection is the foundation for continuous cough monitoring. Recently, the rapid growth in smart hearables has opened new opportunities for such needs. This paper proposes EarCough, which enables continuous subject cough event detection on edge computing hearables by leveraging the always-on active noise cancellation (ANC) microphones. Specifically, we proposed a lightweight end-to-end neural network model -- EarCoughNet. To evaluate the effectiveness of our method, we constructed a synchronous motion and audio dataset through a user study. Results show that EarCough achieved an accuracy of 95.4% and an F1-score of 92.9% with a space requirement of only 385 kB. We envision EarCough as a low-cost add-on for future hearables to enable continuous subject cough event detection.【21】 Weight-sharing Supernet for Searching Specialized Acoustic Event Classification Networks Across Device Constraints标题：跨设备约束搜索专业声事件分类网络的加权超网链接：http://arxiv.org/pdf/2303.10351v1
*与cs.SD语音【16】为同一篇
作者：Guan-Ting Lin,Qingming Tang,Chieh-Chi Kao,Viktor Rozgic,Chao Wang机构：National Taiwan University, Amazon Alexa备注：Accepted by ICASSP 2023摘要：声学事件分类（AEC）已广泛应用于智能扬声器和移动电话等设备中，以实现家庭安全或无障碍支持。随着AEC模型在越来越多具有不同计算资源约束的设备上运行，开发针对每个给定计算资源约束进行调整以实现最佳精度 计算权衡的模型变得越来越昂贵。本文介绍了一种用于AEC的一次性（OFA）神经结构搜索（NAS）框架。具体来说，我们首先训练一个支持不同模型架构的权重共享超网，然后在给定特定计算资源约束的情况下自动搜索模型。我们的实验结果表明，通过仅仅训练一次，从NAS得到的模型显著优于从零开始单独训练的模型和知识蒸馏的模型（25.4%和7.3%相对改进）。我们还发现，超小模型的权重共享超网训练的好处不仅来自搜索，而且来自优化。摘要：Acoustic Event Classification (AEC) has been widely used in devices such as smart speakers and mobile phones for home safety or accessibility support. As AEC models run on more and more devices with diverse computation resource constraints, it became increasingly expensive to develop models that are tuned to achieve optimal accuracy computation trade-off for each given computation resource constraint. In this paper, we introduce a Once-For-All (OFA) Neural Architecture Search (NAS) framework for AEC. Specifically, we first train a weight-sharing supernet that supports different model architectures, followed by automatically searching for a model given specific computational resource constraints. Our experimental results showed that by just training once, the resulting model from NAS significantly outperforms both models trained individually from scratch and knowledge distillation (25.4% and 7.3% relative improvement). We also found that the benefit of weight-sharing supernet training of ultra-small models comes not only from searching but from optimization.【22】 Multimodal Continuous Emotion Recognition: A Technical Report for ABAW5标题：多模式连续情感识别：ABAW5的技术报告链接：http://arxiv.org/pdf/2303.10335v1
*与cs.SD语音【17】为同一篇
作者：Su Zhang,Ziyuan Zhao,Cuntai Guan机构：Nanyang Technological University备注：4 pages. arXiv admin note: substantial text overlap with arXiv:2203.13031摘要：我们使用两个多模态模型，利用视觉、听觉和语言信息进行连续效价唤醒识别。第一个模型与我们在ABAW 2和ABAW 3中使用的相同，它使用了领导者-跟随者注意。第二模型具有用于空间和时间编码的相同架构。至于融合块，它采用了一种紧凑而直接的通道关注，这是从End 2 You工具包中借来的。与我们之前直接使用VGG特征作为音频特征的尝试不同，这次我们使用logmel谱图来馈送预先训练的VGG模型，并在训练期间对其进行微调。为了充分利用数据并避免过拟合，进行了交叉验证。选择一致性相关系数最高的倍数进行提交。该代码可在www.example.com上获得https: github.com sucv ABAW5。摘要：We used two multimodal models for continuous valence-arousal recognition using visual, audio, and linguistic information. The first model is the same as we used in ABAW2 and ABAW3, which employs the leader-follower attention. The second model has the same architecture for spatial and temporal encoding. As for the fusion block, it employs a compact and straightforward channel attention, borrowed from the End2You toolkit. Unlike our previous attempts that use Vggish feature directly as the audio feature, this time we feed the pre-trained VGG model using logmel-spectrogram and finetune it during the training. To make full use of the data and alleviate over-fitting, cross-validation is carried out. The fold with the highest concordance correlation coefficient is selected for submission. The code is to be available at https: github.com sucv ABAW5.【23】 Zero-shot Sound Event Classification Using a Sound Attribute Vector with Global and Local Feature Learning标题：基于全局和局部特征学习的声音属性向量零发声事件分类链接：http://arxiv.org/pdf/2303.10316v1
*与cs.SD语音【18】为同一篇
作者：Yi-Han Lin,Xunquan Chen,Ryoichi Takashima,Tetsuya Takiguchi机构：Graduate School of System Informatics, Kobe University, Japan备注：Accepted by ICASSP2023摘要：提出了一种zero-shot声音事件分类（ZS-SEC）方法，用于识别训练数据中从未出现过的声音事件。在我们之前的工作中，我们提出了一种使用声音属性向量（SAVs）的ZS-SEC方法，其中深度神经网络模型推断描述事件类声音的属性信息，而不是直接推断其类标签。我们以前的方法表明，它可以在一定程度上分类看不见的事件;然而，对于看不见的事件的准确度远低于对于看到的事件的准确度。本文提出了一种新的ZS-SEC方法，该方法可以同时学习判别性全局特征和局部特征，以增强基于SAV的ZS-SEC。该方法在学习全局特征以区分训练数据中的事件类别的同时，学习时空局部特征以利用属性原型回归属性信息。实验结果表明，该方法可以提高SAV的ZS-SEC的准确性，并能可视化谱图中与各属性相关的区域。摘要：This paper introduces a zero-shot sound event classification (ZS-SEC) method to identify sound events that have never occurred in training data. In our previous work, we proposed a ZS-SEC method using sound attribute vectors (SAVs), where a deep neural network model infers attribute information that describes the sound of an event class instead of inferring its class label directly. Our previous method showed that it could classify unseen events to some extent; however, the accuracy for unseen events was far inferior to that for seen events. In this paper, we propose a new ZS-SEC method that can learn discriminative global features and local features simultaneously to enhance SAV-based ZS-SEC. In the proposed method, while the global features are learned in order to discriminate the event classes in the training data, the spectro-temporal local features are learned in order to regress the attribute information using attribute prototypes. The experimental results show that our proposed method can improve the accuracy of SAV-based ZS-SEC and can visualize the region in the spectrogram related to each attribute.【24】 Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows标题：微小、始终在线和脆弱：通过设备上机器学习工作流中的设计选择传播偏差链接：http://arxiv.org/pdf/2201.07677v4
作者：Wiebke Toussaint,Aaron Yi Ding,Fahim Kawsar,Akhil Mathur机构：Netherlands备注：To be published in ACM Transactions on Software Engineering and Methodology摘要：数十亿个分布式、异构和资源受限的物联网设备部署了设备上机器学习（ML），以实现对个人数据的私密、快速和离线推理。设备上ML高度依赖于上下文，并且对用户、使用、硬件和环境属性敏感。ML的这种敏感性和偏倚倾向使得在器械上设置中研究偏倚非常重要。本研究是该新兴领域中最早的偏倚研究之一，为构建更公平的设备上ML奠定了重要基础。我们应用软件工程的视角，通过设备上ML工作流程中的设计选择调查偏差的传播。我们首先确认可靠性偏差是不公平的来源，并提出一种量化方法，然后通过关键词识别任务进行实证实验，以显示复杂和相互作用的技术设计选择如何放大和传播可靠性偏差。我们的结果验证了模型训练期间所做的设计选择，如采样率和输入特征类型，以及优化模型所做的选择，如轻量级架构、修剪学习率和修剪稀疏度，可以导致男性和女性组之间不同的预测性能。基于我们的发现，我们建议工程师采取低工作量策略来缓解器械上ML的偏倚。摘要：Billions of distributed, heterogeneous and resource constrained IoT devices deploy on-device machine learning (ML) for private, fast and offline inference on personal data. On-device ML is highly context dependent, and sensitive to user, usage, hardware and environment attributes. This sensitivity and the propensity towards bias in ML makes it important to study bias in on-device settings. Our study is one of the first investigations of bias in this emerging domain, and lays important foundations for building fairer on-device ML. We apply a software engineering lens, investigating the propagation of bias through design choices in on-device ML workflows. We first identify reliability bias as a source of unfairness and propose a measure to quantify it. We then conduct empirical experiments for a keyword spotting task to show how complex and interacting technical design choices amplify and propagate reliability bias. Our results validate that design choices made during model training, like the sample rate and input feature type, and choices made to optimize models, like light-weight architectures, the pruning learning rate and pruning sparsity, can result in disparate predictive performance across male and female groups. Based on our findings we suggest low effort strategies for engineers to mitigate bias in on-device ML.机器翻译，仅供参考
每日语音速递