今日论文合集：cs.SD语音8篇，eess.AS音频处理12篇。
cs.SD语音
【1】 CryCeleb: A Speaker Verification Dataset Based on Infant Cry Sounds标题：CryCeleb：基于婴儿哭声的说话人确认数据集链接：https://arxiv.org/abs/2305.00969
作者：David Budaghyan,Arsenii Gorin,Cem Subakan,Charles C. Onu机构：Ubenwa Health,Mila-Quebec AI Institute,Universit´e Laval,Concordia University摘要：本文描述了Ubenwa CryCeleb数据集-婴儿哭声的标记集合，以及伴随的CryCeleb 2023任务-基于婴儿哭声的公共扬声器验证挑战。我们从786名新生儿中发布了超过6小时的手动分段哭声用于学术用途，以鼓励婴儿哭声分析的研究。摘要：This paper describes the Ubenwa CryCeleb dataset - a labeled collection of infant cries, and the accompanying CryCeleb 2023 task - a public speaker verification challenge based on infant cry sounds. We release for academic usage more than 6 hours of manually segmented cry sounds from 786 newborns to encourage research in infant cry analysis.
【2】 Joint Modelling of Spoken Language Understanding Tasks with Integrated  Dialog History标题：综合对话历史的口语理解任务的联合建模链接：https://arxiv.org/abs/2305.00926
作者：Siddhant Arora,Hayato Futami,Emiru Tsunoo,Brian Yan,Shinji Watanabe机构：Carnegie Mellon University,Sony Group Corporation, Japan备注：Accepted at ICASSP 20223摘要：大多数人类交互以口语对话的形式发生，其中给定话语的语义含义取决于上下文。口语对话中的每个话语可以由许多语义和说话者属性表示，并且已经对构建用于自动预测这些属性的口语理解（SLU）系统感兴趣。最近的工作表明，纳入对话历史可以帮助提高SLU的性能。然而，单独的模型用于每个SLU任务，导致增加的推理时间和计算成本。受此启发，我们的目标是：我们是否可以联合建模所有SLU任务，同时结合上下文以促进低延迟和轻量级推理？为了回答这个问题，我们提出了一种新的模型架构，学习对话上下文，共同预测的意图，对话行为，扬声器的作用，和情感的口头发言。请注意，我们的联合预测是基于自回归模型，我们需要决定对话属性的预测顺序，这并不简单。为了缓解这个问题，我们还提出了一个顺序不可知的训练方法。我们的实验表明，我们的联合模型取得了类似的结果，特定于任务的分类器，可以有效地整合对话上下文，以进一步提高SLU的性能。摘要：Most human interactions occur in the form of spoken conversations where the semantic meaning of a given utterance depends on the context. Each utterance in spoken conversation can be represented by many semantic and speaker attributes, and there has been an interest in building Spoken Language Understanding (SLU) systems for automatically predicting these attributes. Recent work has shown that incorporating dialogue history can help advance SLU performance. However, separate models are used for each SLU task, leading to an increase in inference time and computation cost. Motivated by this, we aim to ask: can we jointly model all the SLU tasks while incorporating context to facilitate low-latency and lightweight inference? To answer this, we propose a novel model architecture that learns dialog context to jointly predict the intent, dialog act, speaker role, and emotion for the spoken utterance. Note that our joint prediction is based on an autoregressive model and we need to decide the prediction order of dialog attributes, which is not trivial. To mitigate the issue, we also propose an order agnostic training method. Our experiments show that our joint model achieves similar results to task-specific classifiers and can effectively integrate dialog context to further improve the SLU performance.
【3】 Emotions Beyond Words: Non-Speech Audio Emotion Recognition With Edge  Computing标题：文字之外的情感：基于边缘计算的非语音音频情感识别链接：https://arxiv.org/abs/2305.00725
作者：Ibrahim Malik,Siddique Latif,Sanaullah Manzoor,Muhammad Usama,Junaid Qadir,Raja Jurdak机构：EmulationAI, Queensland University of Technology (QUT), Brisbane, Australia, University of the West Scotland, United Kingdom, National University of Computer and Emerging Sciences (NUCES), Pakistan, Qatar University, Doha备注：Under review摘要：非语音情感识别具有广泛的应用，包括医疗保健、犯罪控制和救援以及娱乐等。使用边缘计算提供这些应用具有巨大的潜力，然而，最近的研究集中在使用复杂架构的语音情感识别上。本文提出了一种基于非语音的情感识别系统，该系统可以依靠边缘计算来分析通过尖叫和哭泣等非语音表达传达的情感。特别是，我们探索知识蒸馏设计一个计算效率高的系统，可以部署在边缘设备上，资源有限，而不会显着降低性能。我们全面评估我们提出的框架，使用两个公开的数据集，并通过与著名的MobileNet模型的结果进行比较，突出其有效性。我们的研究结果证明了使用边缘计算进行非语音情感检测的可行性和有效性，这可能会改善通信网络中依赖情感检测的应用程序。据我们所知，这是第一个基于边缘计算的框架，用于检测非语音音频中的情绪，为未来的研究提供了有希望的方向。摘要：Non-speech emotion recognition has a wide range of applications including healthcare, crime control and rescue, and entertainment, to name a few. Providing these applications using edge computing has great potential, however, recent studies are focused on speech-emotion recognition using complex architectures. In this paper, a non-speech-based emotion recognition system is proposed, which can rely on edge computing to analyse emotions conveyed through non-speech expressions like screaming and crying. In particular, we explore knowledge distillation to design a computationally efficient system that can be deployed on edge devices with limited resources without degrading the performance significantly. We comprehensively evaluate our proposed framework using two publicly available datasets and highlight its effectiveness by comparing the results with the well-known MobileNet model. Our results demonstrate the feasibility and effectiveness of using edge computing for non-speech emotion detection, which can potentially improve applications that rely on emotion detection in communication networks. To the best of our knowledge, this is the first work on an edge-computing-based framework for detecting emotions in non-speech audio, offering promising directions for future research.
【4】 Transfer of knowledge among instruments in automatic music transcription标题：音乐自动改编中乐器间的知识传递链接：https://arxiv.org/abs/2305.00426
作者：Michał Leś,Michał Woźniak机构：Wrocław University of Science and Technology, Wrocław, Poland摘要：自动音乐转录（AMT）是音乐信息检索领域中最具挑战性的任务之一。它是将音乐的音频记录转换为包含关于音符、和弦和节奏的信息的符号表示的过程。目前该领域的研究主要集中在开发基于Transformer架构的新模型或使用方法来执行半监督训练，这给出了出色的结果，但训练此类模型的计算成本是巨大的。  这项工作显示了如何使用软件合成器生成的易于生成的合成音频数据来训练通用模型。这为进一步迁移学习打下了良好的基础，使转录模型快速适应其他仪器。实现的结果证明，使用合成数据进行训练可能是预训练通用模型的一个很好的基础，其中转录的任务并不集中在一个仪器上。摘要：Automatic music transcription (AMT) is one of the most challenging tasks in the music information retrieval domain. It is the process of converting an audio recording of music into a symbolic representation containing information about the notes, chords, and rhythm. Current research in this domain focuses on developing new models based on transformer architecture or using methods to perform semi-supervised training, which gives outstanding results, but the computational cost of training such models is enormous.  This work shows how to employ easily generated synthesized audio data produced by software synthesizers to train a universal model. It is a good base for further transfer learning to quickly adapt transcription model for other instruments. Achieved results prove that using synthesized data for training may be a good base for pretraining general-purpose models, where the task of transcription is not focused on one instrument.
【5】 Transformer-based Sequence Labeling for Audio Classification based on  MFCCs标题：基于MFCC的音频分类中基于Transformer的序列标注链接：https://arxiv.org/abs/2305.00417
作者：C. S. Sonali,Chinmayi B S,Ahana Balasubramanian机构：Department of Electronics and Communication Engineering, Ramaiah Institute of Technology, Bengaluru , India备注：5 pages, 2 figures摘要：音频分类在语音和音乐识别等领域至关重要。从音频信号中提取特征，如Mel谱图和MFCC，是音频分类的关键步骤。这些特征被转换成用于分类的谱图。研究人员已经探索了各种技术，包括传统的机器和深度学习方法来对频谱图进行分类，但这些技术在计算上可能非常昂贵。为了简化该过程，可以使用受NLP中的序列分类启发的更直接的方法。提出了一种基于变换器-编码器的MFCC音频分类模型。该模型以ESC-50，Speech Commands v0为基准。02和UrbanSound 8 k数据集，并表现出强劲的性能，最高准确率为95.在UrbanSound 8 k数据集上训练模型时获得2%。该模型仅由127，544个总参数组成，使其在音频分类任务中重量轻但效率高。摘要：Audio classification is vital in areas such as speech and music recognition. Feature extraction from the audio signal, such as Mel-Spectrograms and MFCCs, is a critical step in audio classification. These features are transformed into spectrograms for classification. Researchers have explored various techniques, including traditional machine and deep learning methods to classify spectrograms, but these can be computationally expensive. To simplify this process, a more straightforward approach inspired by sequence classification in NLP can be used. This paper proposes a Transformer-encoder-based model for audio classification using MFCCs. The model was benchmarked against the ESC-50, Speech Commands v0.02 and UrbanSound8k datasets and has shown strong performance, with the highest accuracy of 95.2% obtained upon training the model on the UrbanSound8k dataset. The model consisted of a mere 127,544 total parameters, making it light-weight yet highly efficient at the audio classification task.
【6】 Environmental sound conversion from vocal imitations and sound event  labels标题：从声乐模拟和声音事件标签进行环境声音转换链接：https://arxiv.org/abs/2305.00302
作者：Yuki Okamoto,Keisuke Imoto,Shinnosuke Takamichi,Ryotaro Nagase,Takahiro Fukumori,Yoichi Yamashita机构：Ritsumeikan University, Japan,  Doshisha University, Japan,  The University of Tokyo, Japan摘要：表达环境声音的一种方式是使用声音模仿，其涉及通过声音复制或模仿声音的节奏和音高的过程。在环境声音合成模型中，我们可以使用声音模仿来有效地表达环境声音的特征，例如节奏和音高，这些特征不能通过传统的输入信息来表达，例如声音事件标签、图像和文本。因此，使用人声模仿作为环境声音合成的输入将使我们能够控制声音的音高和节奏，并产生不同的声音。因此，在本文中，我们提出了一个框架，环境声音转换从声乐模仿，以产生不同的声音。我们还提出了一种环境声音合成的方法，从人声模仿和声音事件的标签。预期使用声音事件标签来控制合成声音的声音事件类别，这不能仅通过声音模仿来控制。我们的客观和主观的实验结果表明，声乐模仿有效地控制声音的音高和节奏，并产生多样化的声音。摘要：One way of expressing an environmental sound is using vocal imitations, which involve the process of replicating or mimicking the rhythms and pitches of sounds by voice. We can effectively express the features of environmental sounds, such as rhythms and pitches, using vocal imitations, which cannot be expressed by conventional input information, such as sound event labels, images, and texts, in an environmental sound synthesis model. Therefore, using vocal imitations as input for environmental sound synthesis will enable us to control the pitches and rhythms of sounds and generate diverse sounds. In this paper, we thus propose a framework for environmental sound conversion from vocal imitations to generate diverse sounds. We also propose a method of environmental sound synthesis from vocal imitations and sound event labels. Using sound event labels is expected to control the sound event class of the synthesized sound, which cannot be controlled by only vocal imitations. Our objective and subjective experimental results show that vocal imitations effectively control the pitches and rhythms of sounds and generate diverse sounds.
【7】 Enhancing multilingual speech recognition in air traffic control by  sentence-level language identification标题：语句级语言识别增强空中交通管制多语种语音识别链接：https://arxiv.org/abs/2305.00170
作者：Peng Fan,Dongyue Guo,JianWei Zhang,Bo Yang,Yi Lin机构：National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, wangjiang road, Chengdu, Sichuan, China, College of Computer Science, Sichuan University, wangjiang road, Chengdu, Sichuan, China摘要：自动语音识别（ASR）技术在提高空中交通管制（ATC）工作效率和安全性方面得到了越来越广泛的应用。然而，管制员与飞行员之间的多语种语音对话给建立高精度的自动语音识别系统带来了巨大的挑战。在这项工作中，我们提出了一个两阶段的多语言ASR框架。第一阶段是训练语言识别器（LID），其基于递归神经网络（RNN）以独热编码的形式获得句子语言识别。第二阶段旨在训练基于RNN的端到端多语言识别模型，该模型利用LID生成的句子语言特征来增强输入特征。在这项工作中，我们引入特征线性调制（薄膜），以提高性能的多语言ASR利用句子语言识别。此外，我们引入了一个新的句子语言识别学习模块称为SLIL，它由一个电影层和挤压和激励网络层。在ATCPeech数据集上的大量实验表明，我们提出的方法优于基线模型。相比香草FiLMed骨干模型，建议的多语言ASR模型获得约7。50%的字符错误率相对性能改进。摘要：Automatic speech recognition (ASR) technique is becoming increasingly popular to improve the efficiency and safety of air traffic control (ATC) operations. However, the conversation between ATC controllers and pilots using multilingual speech brings a great challenge to building high-accuracy ASR systems. In this work, we present a two-stage multilingual ASR framework. The first stage is to train a language identifier (LID), that based on a recurrent neural network (RNN) to obtain sentence language identification in the form of one-hot encoding. The second stage aims to train an RNN-based end-to-end multilingual recognition model that utilizes sentence language features generated by LID to enhance input features. In this work, We introduce Featurewise Linear Modulation (FiLM) to improve the performance of multilingual ASR by utilizing sentence language identification. Furthermore, we introduce a new sentence language identification learning module called SLIL, which consists of a FiLM layer and a Squeeze-and-Excitation Networks layer. Extensive experiments on the ATCSpeech dataset show that our proposed method outperforms the baseline model. Compared to the vanilla FiLMed backbone model, the proposed multilingual ASR model obtains about 7.50% character error rate relative performance improvement.
【8】 Adversarial Representation Learning for Robust Privacy Preservation in  Audio标题：音频中稳健隐私保护的对抗性表征学习链接：https://arxiv.org/abs/2305.00011
作者：Shayan Gharib,Minh Tran,Diep Luong,Konstantinos Drossos,Tuomas Virtanen机构：University of Helsinki, Helsinki, Finland,  Tampere University, Tampere, Finland,  Nokia Tech, Espoo, Finland摘要：声音事件检测系统广泛用于各种应用中，例如监视和环境监测，其中数据被自动收集、处理并发送到云以进行声音识别。然而，该过程可能会无意中泄露有关用户或其周围环境的敏感信息，从而引起隐私问题。在这项研究中，我们提出了一种新的对抗性训练方法，用于学习录音的表示，有效地防止从录音的潜在特征中检测到语音活动。所提出的方法训练模型以生成不能由语音分类器与非语音记录区分开的包含语音的音频记录的不变的潜在表示。我们的工作的新颖之处在于优化算法，其中语音分类器的权重定期替换为以监督方式训练的分类器的权重。这在对抗训练期间不断增加语音分类器的辨别能力，激励模型生成语音不可区分的潜在表示，即使使用在对抗训练循环之外训练的新语音分类器。所提出的方法进行评估，对基线方法没有隐私措施和先前的对抗性训练方法，表明显着减少隐私侵犯相比，基线方法。此外，我们表明，事先对抗的方法实际上是无效的，为此目的。摘要：Sound event detection systems are widely used in various applications such as surveillance and environmental monitoring where data is automatically collected, processed, and sent to a cloud for sound recognition. However, this process may inadvertently reveal sensitive information about users or their surroundings, hence raising privacy concerns. In this study, we propose a novel adversarial training method for learning representations of audio recordings that effectively prevents the detection of speech activity from the latent features of the recordings. The proposed method trains a model to generate invariant latent representations of speech-containing audio recordings that cannot be distinguished from non-speech recordings by a speech classifier. The novelty of our work is in the optimization algorithm, where the speech classifier's weights are regularly replaced with the weights of classifiers trained in a supervised manner. This increases the discrimination power of the speech classifier constantly during the adversarial training, motivating the model to generate latent representations in which speech is not distinguishable, even using new speech classifiers trained outside the adversarial training loop. The proposed method is evaluated against a baseline approach with no privacy measures and a prior adversarial training method, demonstrating a significant reduction in privacy violations compared to the baseline approach. Additionally, we show that the prior adversarial method is practically ineffective for this purpose.
eess.AS音频处理
【1】 A Review of Deep Learning Techniques for Speech Processing标题：语音处理中的深度学习技术综述链接：https://arxiv.org/abs/2305.00359
作者：Ambuj Mehrish,Navonil Majumder,Rishabh Bhardwaj,Soujanya Poria机构：RADA MIHALCEA, University of Michigan, USA, HMM + GMM, • LSTM, • GRU, • Wav,Vec ,., • FastSpeech, • Conformer, • ContextNet, Time, Performance, Whisper, Vall-E, • HuBERT, • Speechstew摘要：随着深度学习的出现，语音处理领域经历了一场变革性的转变。多个处理层的使用使得能够创建能够从语音数据中提取复杂特征的模型。这一发展为语音识别、文本到语音合成、自动语音识别和情感识别的空前进步铺平了道路，将这些任务的性能推向了前所未有的高度。深度学习技术的强大功能为语音处理领域的研究和创新开辟了新的途径，对一系列行业和应用产生了深远的影响。本文全面概述了关键的深度学习模型及其在语音处理任务中的应用。我们首先追溯语音处理研究的发展，从早期的方法，如MFCC和HMM，到深度学习架构的最新进展，如CNN，RNN，Transformers，构象和扩散模型。我们对这些方法进行分类，并比较它们在解决语音处理任务方面的优缺点。此外，我们广泛涵盖了文献中使用的各种语音处理任务、数据集和基准测试，并描述了如何利用不同的深度学习网络来处理这些任务。此外，我们还讨论了深度学习在语音处理中的挑战和未来发展方向，包括对更有效的参数，可解释的模型的需求以及深度学习在多模态语音处理中的潜力。通过研究该领域的演变，比较和对比不同的方法，并强调未来的方向和挑战，我们希望激发在这个令人兴奋和快速发展的领域的进一步研究。摘要：The field of speech processing has undergone a transformative shift with the advent of deep learning. The use of multiple processing layers has enabled the creation of models capable of extracting intricate features from speech data. This development has paved the way for unparalleled advancements in speech recognition, text-to-speech synthesis, automatic speech recognition, and emotion recognition, propelling the performance of these tasks to unprecedented heights. The power of deep learning techniques has opened up new avenues for research and innovation in the field of speech processing, with far-reaching implications for a range of industries and applications. This review paper provides a comprehensive overview of the key deep learning models and their applications in speech-processing tasks. We begin by tracing the evolution of speech processing research, from early approaches, such as MFCC and HMM, to more recent advances in deep learning architectures, such as CNNs, RNNs, transformers, conformers, and diffusion models. We categorize the approaches and compare their strengths and weaknesses for solving speech-processing tasks. Furthermore, we extensively cover various speech-processing tasks, datasets, and benchmarks used in the literature and describe how different deep-learning networks have been utilized to tackle these tasks. Additionally, we discuss the challenges and future directions of deep learning in speech processing, including the need for more parameter-efficient, interpretable models and the potential of deep learning for multimodal speech processing. By examining the field's evolution, comparing and contrasting different approaches, and highlighting future directions and challenges, we hope to inspire further research in this exciting and rapidly advancing field.
【2】 Towards Better Domain Adaptation for Self-supervised Models: A Case  Study of Child ASR标题：走向更好的自我监督模型的领域适应：儿童ASR案例研究链接：https://arxiv.org/abs/2305.00115
作者：Ruchao Fan,Yunzheng Zhu,Jinhan Wang,Abeer Alwan备注：Published in IEEE Journal of Selected Topics in Signal Processing, ICASSP Journal Poster Presentation摘要：最近，来自未标记语音数据的自监督学习（SSL）在自动语音识别（ASR）社区中获得了越来越多的关注。典型的SSL方法包括自回归预测编码（APC）、Wav 2 vec 2。0和隐藏单元BERT（HuBERT）。然而，SSL模型偏向于预训练数据。当SSL模型与来自另一个域的数据进行微调时，会发生域转移，并可能导致下游任务的知识转移有限。在本文中，我们提出了一个新的框架，域负责的适应和微调（草案），以减少域转移预训练的语音模型，并评估它的因果和非因果Transformer。对于因果Transformer，提出了APC的扩展（E-APC），通过使用多个时间移位序列来执行预测，从未标记的数据中学习更丰富的信息。对于非因果Transformer，使用双向APC（Bi-APC）的各种解决方案进行了研究。此外，草案框架检查Wav 2 vec 2。0和HuBERT方法，它们使用非因果Transformers作为主干。实验进行儿童ASR（使用OGI和MyST数据库）使用SSL模型训练的未标记的成人语音数据从Librispeech。相对WER改善高达19。7%的两个孩子的任务相比，没有适应的预训练模型。与不使用预训练方法的模型相比，使用所提出的方法（E-APC和DRAFT），相对WER改进甚至更大（OGI和MyST数据分别为30%和19%）。摘要：Recently, self-supervised learning (SSL) from unlabelled speech data has gained increased attention in the automatic speech recognition (ASR) community. Typical SSL methods include autoregressive predictive coding (APC), Wav2vec2.0, and hidden unit BERT (HuBERT). However, SSL models are biased to the pretraining data. When SSL models are finetuned with data from another domain, domain shifting occurs and might cause limited knowledge transfer for downstream tasks. In this paper, we propose a novel framework, domain responsible adaptation and finetuning (DRAFT), to reduce domain shifting in pretrained speech models, and evaluate it for a causal and non-causal transformer. For the causal transformer, an extension of APC (E-APC) is proposed to learn richer information from unlabelled data by using multiple temporally-shifted sequences to perform prediction. For the non-causal transformer, various solutions for using the bidirectional APC (Bi-APC) are investigated. In addition, the DRAFT framework is examined for Wav2vec2.0 and HuBERT methods, which use non-causal transformers as the backbone. The experiments are conducted on child ASR (using the OGI and MyST databases) using SSL models trained with unlabelled adult speech data from Librispeech. The relative WER improvements of up to 19.7% on the two child tasks are observed when compared to the pretrained models without adaptation. With the proposed methods (E-APC and DRAFT), the relative WER improvements are even larger (30% and 19% on the OGI and MyST data, respectively) when compared to the models without using pretraining methods.
【3】 CryCeleb: A Speaker Verification Dataset Based on Infant Cry Sounds标题：CryCeleb：基于婴儿哭声的说话人确认数据集链接：https://arxiv.org/abs/2305.00969
*与cs.SD语音【1】为同一篇
作者：David Budaghyan,Arsenii Gorin,Cem Subakan,Charles C. Onu机构：Ubenwa Health,Mila-Quebec AI Institute,Universit´e Laval,Concordia University摘要：本文描述了Ubenwa CryCeleb数据集-婴儿哭声的标记集合，以及伴随的CryCeleb 2023任务-基于婴儿哭声的公共扬声器验证挑战。我们从786名新生儿中发布了超过6小时的手动分段哭声用于学术用途，以鼓励婴儿哭声分析的研究。摘要：This paper describes the Ubenwa CryCeleb dataset - a labeled collection of infant cries, and the accompanying CryCeleb 2023 task - a public speaker verification challenge based on infant cry sounds. We release for academic usage more than 6 hours of manually segmented cry sounds from 786 newborns to encourage research in infant cry analysis.
【4】 Joint Modelling of Spoken Language Understanding Tasks with Integrated  Dialog History标题：综合对话历史的口语理解任务的联合建模链接：https://arxiv.org/abs/2305.00926
*与cs.SD语音【2】为同一篇
作者：Siddhant Arora,Hayato Futami,Emiru Tsunoo,Brian Yan,Shinji Watanabe机构：Carnegie Mellon University,Sony Group Corporation, Japan备注：Accepted at ICASSP 20223摘要：大多数人类交互以口语对话的形式发生，其中给定话语的语义含义取决于上下文。口语对话中的每个话语可以由许多语义和说话者属性表示，并且已经对构建用于自动预测这些属性的口语理解（SLU）系统感兴趣。最近的工作表明，纳入对话历史可以帮助提高SLU的性能。然而，单独的模型用于每个SLU任务，导致增加的推理时间和计算成本。受此启发，我们的目标是：我们是否可以联合建模所有SLU任务，同时结合上下文以促进低延迟和轻量级推理？为了回答这个问题，我们提出了一种新的模型架构，学习对话上下文，共同预测的意图，对话行为，扬声器的作用，和情感的口头发言。请注意，我们的联合预测是基于自回归模型，我们需要决定对话属性的预测顺序，这并不简单。为了缓解这个问题，我们还提出了一个顺序不可知的训练方法。我们的实验表明，我们的联合模型取得了类似的结果，特定于任务的分类器，可以有效地整合对话上下文，以进一步提高SLU的性能。摘要：Most human interactions occur in the form of spoken conversations where the semantic meaning of a given utterance depends on the context. Each utterance in spoken conversation can be represented by many semantic and speaker attributes, and there has been an interest in building Spoken Language Understanding (SLU) systems for automatically predicting these attributes. Recent work has shown that incorporating dialogue history can help advance SLU performance. However, separate models are used for each SLU task, leading to an increase in inference time and computation cost. Motivated by this, we aim to ask: can we jointly model all the SLU tasks while incorporating context to facilitate low-latency and lightweight inference? To answer this, we propose a novel model architecture that learns dialog context to jointly predict the intent, dialog act, speaker role, and emotion for the spoken utterance. Note that our joint prediction is based on an autoregressive model and we need to decide the prediction order of dialog attributes, which is not trivial. To mitigate the issue, we also propose an order agnostic training method. Our experiments show that our joint model achieves similar results to task-specific classifiers and can effectively integrate dialog context to further improve the SLU performance.
【5】 Emotions Beyond Words: Non-Speech Audio Emotion Recognition With Edge  Computing标题：文字之外的情感：基于边缘计算的非语音音频情感识别链接：https://arxiv.org/abs/2305.00725
*与cs.SD语音【3】为同一篇
作者：Ibrahim Malik,Siddique Latif,Sanaullah Manzoor,Muhammad Usama,Junaid Qadir,Raja Jurdak机构：EmulationAI, Queensland University of Technology (QUT), Brisbane, Australia, University of the West Scotland, United Kingdom, National University of Computer and Emerging Sciences (NUCES), Pakistan, Qatar University, Doha备注：Under review摘要：非语音情感识别具有广泛的应用，包括医疗保健、犯罪控制和救援以及娱乐等。使用边缘计算提供这些应用具有巨大的潜力，然而，最近的研究集中在使用复杂架构的语音情感识别上。本文提出了一种基于非语音的情感识别系统，该系统可以依靠边缘计算来分析通过尖叫和哭泣等非语音表达传达的情感。特别是，我们探索知识蒸馏设计一个计算效率高的系统，可以部署在边缘设备上，资源有限，而不会显着降低性能。我们全面评估我们提出的框架，使用两个公开的数据集，并通过与著名的MobileNet模型的结果进行比较，突出其有效性。我们的研究结果证明了使用边缘计算进行非语音情感检测的可行性和有效性，这可能会改善通信网络中依赖情感检测的应用程序。据我们所知，这是第一个基于边缘计算的框架，用于检测非语音音频中的情绪，为未来的研究提供了有希望的方向。摘要：Non-speech emotion recognition has a wide range of applications including healthcare, crime control and rescue, and entertainment, to name a few. Providing these applications using edge computing has great potential, however, recent studies are focused on speech-emotion recognition using complex architectures. In this paper, a non-speech-based emotion recognition system is proposed, which can rely on edge computing to analyse emotions conveyed through non-speech expressions like screaming and crying. In particular, we explore knowledge distillation to design a computationally efficient system that can be deployed on edge devices with limited resources without degrading the performance significantly. We comprehensively evaluate our proposed framework using two publicly available datasets and highlight its effectiveness by comparing the results with the well-known MobileNet model. Our results demonstrate the feasibility and effectiveness of using edge computing for non-speech emotion detection, which can potentially improve applications that rely on emotion detection in communication networks. To the best of our knowledge, this is the first work on an edge-computing-based framework for detecting emotions in non-speech audio, offering promising directions for future research.
【6】 Transfer of knowledge among instruments in automatic music transcription标题：音乐自动改编中乐器间的知识传递链接：https://arxiv.org/abs/2305.00426
*与cs.SD语音【4】为同一篇
作者：Michał Leś,Michał Woźniak机构：Wrocław University of Science and Technology, Wrocław, Poland摘要：自动音乐转录（AMT）是音乐信息检索领域中最具挑战性的任务之一。它是将音乐的音频记录转换为包含关于音符、和弦和节奏的信息的符号表示的过程。目前该领域的研究主要集中在开发基于Transformer架构的新模型或使用方法来执行半监督训练，这给出了出色的结果，但训练此类模型的计算成本是巨大的。  这项工作显示了如何使用软件合成器生成的易于生成的合成音频数据来训练通用模型。这为进一步迁移学习打下了良好的基础，使转录模型快速适应其他仪器。实现的结果证明，使用合成数据进行训练可能是预训练通用模型的一个很好的基础，其中转录的任务并不集中在一个仪器上。摘要：Automatic music transcription (AMT) is one of the most challenging tasks in the music information retrieval domain. It is the process of converting an audio recording of music into a symbolic representation containing information about the notes, chords, and rhythm. Current research in this domain focuses on developing new models based on transformer architecture or using methods to perform semi-supervised training, which gives outstanding results, but the computational cost of training such models is enormous.  This work shows how to employ easily generated synthesized audio data produced by software synthesizers to train a universal model. It is a good base for further transfer learning to quickly adapt transcription model for other instruments. Achieved results prove that using synthesized data for training may be a good base for pretraining general-purpose models, where the task of transcription is not focused on one instrument.
【7】 Transformer-based Sequence Labeling for Audio Classification based on  MFCCs标题：基于MFCC的音频分类中基于Transformer的序列标注链接：https://arxiv.org/abs/2305.00417
*与cs.SD语音【5】为同一篇
作者：C. S. Sonali,Chinmayi B S,Ahana Balasubramanian机构：Department of Electronics and Communication Engineering, Ramaiah Institute of Technology, Bengaluru , India备注：5 pages, 2 figures摘要：音频分类在语音和音乐识别等领域至关重要。从音频信号中提取特征，如Mel谱图和MFCC，是音频分类的关键步骤。这些特征被转换成用于分类的谱图。研究人员已经探索了各种技术，包括传统的机器和深度学习方法来对频谱图进行分类，但这些技术在计算上可能非常昂贵。为了简化该过程，可以使用受NLP中的序列分类启发的更直接的方法。提出了一种基于变换器-编码器的MFCC音频分类模型。该模型以ESC-50，Speech Commands v0为基准。02和UrbanSound 8 k数据集，并表现出强劲的性能，最高准确率为95.在UrbanSound 8 k数据集上训练模型时获得2%。该模型仅由127，544个总参数组成，使其在音频分类任务中重量轻但效率高。摘要：Audio classification is vital in areas such as speech and music recognition. Feature extraction from the audio signal, such as Mel-Spectrograms and MFCCs, is a critical step in audio classification. These features are transformed into spectrograms for classification. Researchers have explored various techniques, including traditional machine and deep learning methods to classify spectrograms, but these can be computationally expensive. To simplify this process, a more straightforward approach inspired by sequence classification in NLP can be used. This paper proposes a Transformer-encoder-based model for audio classification using MFCCs. The model was benchmarked against the ESC-50, Speech Commands v0.02 and UrbanSound8k datasets and has shown strong performance, with the highest accuracy of 95.2% obtained upon training the model on the UrbanSound8k dataset. The model consisted of a mere 127,544 total parameters, making it light-weight yet highly efficient at the audio classification task.
【8】 Environmental sound conversion from vocal imitations and sound event  labels标题：从声乐模拟和声音事件标签进行环境声音转换链接：https://arxiv.org/abs/2305.00302
*与cs.SD语音【6】为同一篇
作者：Yuki Okamoto,Keisuke Imoto,Shinnosuke Takamichi,Ryotaro Nagase,Takahiro Fukumori,Yoichi Yamashita机构：Ritsumeikan University, Japan,  Doshisha University, Japan,  The University of Tokyo, Japan摘要：表达环境声音的一种方式是使用声音模仿，其涉及通过声音复制或模仿声音的节奏和音高的过程。在环境声音合成模型中，我们可以使用声音模仿来有效地表达环境声音的特征，例如节奏和音高，这些特征不能通过传统的输入信息来表达，例如声音事件标签、图像和文本。因此，使用人声模仿作为环境声音合成的输入将使我们能够控制声音的音高和节奏，并产生不同的声音。因此，在本文中，我们提出了一个框架，环境声音转换从声乐模仿，以产生不同的声音。我们还提出了一种环境声音合成的方法，从人声模仿和声音事件的标签。预期使用声音事件标签来控制合成声音的声音事件类别，这不能仅通过声音模仿来控制。我们的客观和主观的实验结果表明，声乐模仿有效地控制声音的音高和节奏，并产生多样化的声音。摘要：One way of expressing an environmental sound is using vocal imitations, which involve the process of replicating or mimicking the rhythms and pitches of sounds by voice. We can effectively express the features of environmental sounds, such as rhythms and pitches, using vocal imitations, which cannot be expressed by conventional input information, such as sound event labels, images, and texts, in an environmental sound synthesis model. Therefore, using vocal imitations as input for environmental sound synthesis will enable us to control the pitches and rhythms of sounds and generate diverse sounds. In this paper, we thus propose a framework for environmental sound conversion from vocal imitations to generate diverse sounds. We also propose a method of environmental sound synthesis from vocal imitations and sound event labels. Using sound event labels is expected to control the sound event class of the synthesized sound, which cannot be controlled by only vocal imitations. Our objective and subjective experimental results show that vocal imitations effectively control the pitches and rhythms of sounds and generate diverse sounds.
【9】 Analysis of vocal breath sounds before and after administering  Bronchodilator in Asthmatic patients标题：哮喘患者应用支气管扩张剂前后的嗓音分析链接：https://arxiv.org/abs/2305.00242
作者：Shivani Yadav,Dipanjan Gope,Uma Maheswari K.,Prasanta Kumar Ghosh机构：BioSystems Science and Engineering, Indian Institute of Science, (IISc), Bangalore-, India, Electrical Communication Engineering, Indian Institute of Science, Pulmonary Medicine, St. Johns National Academy of Health, Sciences, Bangalore-, India摘要：哮喘是气道的慢性炎症性疾病之一，其引起胸闷、喘息、呼吸困难和咳嗽。肺活量测定是一种依赖于努力的测试，用于监测和诊断肺部疾病，如哮喘。基于声音呼吸声（VBS）的分析可以是肺活量测定的替代方案，因为VBS特性根据肺部状况而变化。VBS测试消耗更少的时间，也需要更少的努力，不像肺量测定。在这项工作中，VBS的特点进行了分析之前和之后，以受试者依赖性的方式使用线性判别分析（LDA）的支气管扩张剂。我们发现，通过LDA学习的功能显示在本研究中考虑的所有30例受试者中，在给予支气管扩张剂之前和之后记录的VBS之间存在显著差异，而基线功能仅在26例受试者中可以实现VBS之间的显著差异。我们还观察到，所有频率范围对区分使用支气管扩张剂前和使用支气管扩张剂后的情况的贡献并不相等。通过实验，我们发现400- 500 Hz和1480- 1900 Hz两个频率范围对所有受试者的辨别力最大。本文中提出的研究分析了前和后支气管扩张剂的影响，在口吸入声音记录在受试者依赖性的方式。这项工作的结果表明，在口腔记录的吸入声音可以是一个很好的刺激，以区分之前和之后的哮喘患者的支气管扩张剂条件。基于吸气声的支气管扩张剂前后辨别在临床环境中可能具有潜在用途。摘要：Asthma is one of the chronic inflammatory diseases of the airways, which causes chest tightness, wheezing, breathlessness, and cough. Spirometry is an effort-dependent test used to monitor and diagnose lung conditions like Asthma. Vocal breath sound (VBS) based analysis can be an alternative to spirometry as VBS characteristics change depending on the lung condition. VBS test consumes less time, and it also requires less effort, unlike spirometry. In this work, VBS characteristics are analyzed before and after administering bronchodilator in a subject-dependent manner using linear discriminant analysis (LDA). We find that features learned through LDA show a significant difference between VBS recorded before and after administering bronchodilator in all 30 subjects considered in this work, whereas the baseline features could achieve a significant difference between VBS only for 26 subjects. We also observe that all frequency ranges do not contribute equally to the discrimination between pre and post bronchodilator conditions. From experiments, we find that two frequency ranges, namely 400-500Hz and 1480-1900Hz, maximally contribute to the discrimination of all the subjects. The study presented in this paper analyzes the pre and post-bronchodilator effect on the inhalation sound recorded at the mouth in a subject-dependent manner. The findings of this work suggest that, inhalation sound recorded at mouth can be a good stimulus to discriminate pre and post-bronchodilator conditions in asthmatic subjects. Inhale sound-based pre and post-bronchodilator discrimination can be of potential use in clinical settings.
【10】 Enhancing multilingual speech recognition in air traffic control by  sentence-level language identification标题：语句级语言识别增强空中交通管制多语种语音识别链接：https://arxiv.org/abs/2305.00170
*与cs.SD语音【7】为同一篇
作者：Peng Fan,Dongyue Guo,JianWei Zhang,Bo Yang,Yi Lin机构：National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, wangjiang road, Chengdu, Sichuan, China, College of Computer Science, Sichuan University, wangjiang road, Chengdu, Sichuan, China摘要：自动语音识别（ASR）技术在提高空中交通管制（ATC）工作效率和安全性方面得到了越来越广泛的应用。然而，管制员与飞行员之间的多语种语音对话给建立高精度的自动语音识别系统带来了巨大的挑战。在这项工作中，我们提出了一个两阶段的多语言ASR框架。第一阶段是训练语言识别器（LID），其基于递归神经网络（RNN）以独热编码的形式获得句子语言识别。第二阶段旨在训练基于RNN的端到端多语言识别模型，该模型利用LID生成的句子语言特征来增强输入特征。在这项工作中，我们引入特征线性调制（薄膜），以提高性能的多语言ASR利用句子语言识别。此外，我们引入了一个新的句子语言识别学习模块称为SLIL，它由一个电影层和挤压和激励网络层。在ATCPeech数据集上的大量实验表明，我们提出的方法优于基线模型。相比香草FiLMed骨干模型，建议的多语言ASR模型获得约7。50%的字符错误率相对性能改进。摘要：Automatic speech recognition (ASR) technique is becoming increasingly popular to improve the efficiency and safety of air traffic control (ATC) operations. However, the conversation between ATC controllers and pilots using multilingual speech brings a great challenge to building high-accuracy ASR systems. In this work, we present a two-stage multilingual ASR framework. The first stage is to train a language identifier (LID), that based on a recurrent neural network (RNN) to obtain sentence language identification in the form of one-hot encoding. The second stage aims to train an RNN-based end-to-end multilingual recognition model that utilizes sentence language features generated by LID to enhance input features. In this work, We introduce Featurewise Linear Modulation (FiLM) to improve the performance of multilingual ASR by utilizing sentence language identification. Furthermore, we introduce a new sentence language identification learning module called SLIL, which consists of a FiLM layer and a Squeeze-and-Excitation Networks layer. Extensive experiments on the ATCSpeech dataset show that our proposed method outperforms the baseline model. Compared to the vanilla FiLMed backbone model, the proposed multilingual ASR model obtains about 7.50% character error rate relative performance improvement.
【11】 MMViT: Multiscale Multiview Vision Transformers标题：MMViT：多尺度多视角视觉转换器链接：https://arxiv.org/abs/2305.00104
作者：Yuchen Liu,Natasha Ong,Kaiyan Peng,Bo Xiong,Qifan Wang,Rui Hou,Madian Khabsa,Kaiyue Yang,David Liu,Donald S. Williamson,Hanchao Yu机构：Meta Platforms Inc., Indiana University Bloomington, The Ohio State University摘要：我们提出了多尺度多视图Vision Transformers（MMViT），它引入了多尺度特征映射和多视图编码的变压器模型。我们的模型对输入信号的不同视图进行编码，并构建多个通道分辨率特征阶段，以并行处理不同分辨率的输入的多个视图。在每个尺度阶段，我们使用交叉注意块来融合不同视图之间的信息。这使得MMViT模型能够以不同分辨率获取输入的复杂高维表示。该模型可以作为多个领域的骨干模型。我们证明了MMViT在音频和图像分类任务上的有效性，实现了最先进的结果。摘要：We present Multiscale Multiview Vision Transformers (MMViT), which introduces multiscale feature maps and multiview encodings to transformer models. Our model encodes different views of the input signal and builds several channel-resolution feature stages to process the multiple views of the input at different resolutions in parallel. At each scale stage, we use a cross-attention block to fuse information across different views. This enables the MMViT model to acquire complex high-dimensional representations of the input at different resolutions. The proposed model can serve as a backbone model in multiple domains. We demonstrate the effectiveness of MMViT on audio and image classification tasks, achieving state-of-the-art results.
【12】 Adversarial Representation Learning for Robust Privacy Preservation in  Audio标题：音频中稳健隐私保护的对抗性表征学习链接：https://arxiv.org/abs/2305.00011*与cs.SD语音【8】为同一篇
作者：Shayan Gharib,Minh Tran,Diep Luong,Konstantinos Drossos,Tuomas Virtanen机构：University of Helsinki, Helsinki, Finland,  Tampere University, Tampere, Finland,  Nokia Tech, Espoo, Finland摘要：声音事件检测系统广泛用于各种应用中，例如监视和环境监测，其中数据被自动收集、处理并发送到云以进行声音识别。然而，该过程可能会无意中泄露有关用户或其周围环境的敏感信息，从而引起隐私问题。在这项研究中，我们提出了一种新的对抗性训练方法，用于学习录音的表示，有效地防止从录音的潜在特征中检测到语音活动。所提出的方法训练模型以生成不能由语音分类器与非语音记录区分开的包含语音的音频记录的不变的潜在表示。我们的工作的新颖之处在于优化算法，其中语音分类器的权重定期替换为以监督方式训练的分类器的权重。这在对抗训练期间不断增加语音分类器的辨别能力，激励模型生成语音不可区分的潜在表示，即使使用在对抗训练循环之外训练的新语音分类器。所提出的方法进行评估，对基线方法没有隐私措施和先前的对抗性训练方法，表明显着减少隐私侵犯相比，基线方法。此外，我们表明，事先对抗的方法实际上是无效的，为此目的。摘要：Sound event detection systems are widely used in various applications such as surveillance and environmental monitoring where data is automatically collected, processed, and sent to a cloud for sound recognition. However, this process may inadvertently reveal sensitive information about users or their surroundings, hence raising privacy concerns. In this study, we propose a novel adversarial training method for learning representations of audio recordings that effectively prevents the detection of speech activity from the latent features of the recordings. The proposed method trains a model to generate invariant latent representations of speech-containing audio recordings that cannot be distinguished from non-speech recordings by a speech classifier. The novelty of our work is in the optimization algorithm, where the speech classifier's weights are regularly replaced with the weights of classifiers trained in a supervised manner. This increases the discrimination power of the speech classifier constantly during the adversarial training, motivating the model to generate latent representations in which speech is not distinguishable, even using new speech classifiers trained outside the adversarial training loop. The proposed method is evaluated against a baseline approach with no privacy measures and a prior adversarial training method, demonstrating a significant reduction in privacy violations compared to the baseline approach. Additionally, we show that the prior adversarial method is practically ineffective for this purpose.
机器翻译，仅供参考