今日论文合集：cs.SD语音10篇，eess.AS音频处理10篇。cs.SD语音【1】 Physics-Based Acoustic Holograms标题：基于物理的声全息图链接：https://arxiv.org/abs/2305.03625作者：Antonio Stanziola,Ben T. Cox,Bradley E. Treeby,Michael D. Brown机构：Dept. of Medical Physics and Biomedical Engineering, University College London, Gower St, London, WC,E ,BT, Dept. of Neuroscience, Erasmus MC, Rotterdam, Netherlands摘要：增材制造的进步已经使得能够实现廉价的、可扩展的衍射声透镜，其可以用于经由相位和/或振幅调制生成复杂的声场。然而，这些全息图的设计依赖于一个薄的元件近似从光学适应，可以严重限制所实现的声场的保真度。在这里，我们介绍了基于物理的声学全息图与复杂的内部结构。通过优化全息图内的声学特性分布，使用具有制造约束的可微分声学模型来设计结构。全息图可以使用当代3D打印机简单且廉价地制造。实验测量表明，与传统的薄元件全息图相比，显着的改善。摘要：Advances in additive manufacturing have enabled the realisation of inexpensive, scalable, diffractive acoustic lenses that can be used to generate complex acoustic fields via phase and/or amplitude modulation. However, the design of these holograms relies on a thin-element approximation adapted from optics which can severely limit the fidelity of the realised acoustic field. Here, we introduce physics-based acoustic holograms with a complex internal structure. The structures are designed using a differentiable acoustic model with manufacturing constraints via optimisation of the acoustic property distribution within the hologram. The holograms can be fabricated simply and inexpensively using contemporary 3D printers. Experimental measurements demonstrate a significant improvement compared to conventional thin-element holograms.【2】 A Multimodal Dynamical Variational Autoencoder for Audiovisual Speech Representation Learning标题：一种用于视听语音表征学习的多模式动态变分自动编码器链接：https://arxiv.org/abs/2305.03582作者：Samir Sadok,Simon Leglaive,Laurent Girin,Xavier Alameda-Pineda,Renaud Séguier机构：CentraleSupélec, IETR UMR CNRS , France, Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France, Inria, Univ. Grenoble Alpes, CNRS, LJK, France备注：25 pages, 14 figures, this https URL摘要：在本文中，我们提出了一个多模态\textit{和}动态VAE（MDVAE）应用于无监督的视听语音表示学习。潜在空间被构造成将在模态之间共享的潜在动态因素与特定于每个模态的潜在动态因素分离。一个静态的潜在变量也被引入到编码的信息是恒定的，随着时间的推移在视听语音序列。该模型在两个阶段中以无监督的方式在视听情感语音数据集上进行训练。在第一阶段中，矢量量化的VAE（VQ-VAE）针对每个模态独立地学习，而不进行时间建模。第二阶段包括在量化之前在VQ-VAE的中间表示上学习MDVAE模型。静态与动态以及模态特定与模态公共信息之间的解纠缠发生在该第二训练阶段期间。大量的实验进行调查如何视听语音潜在因素编码的MDVAE的潜在空间。这些实验包括操纵视听语音，视听面部图像去噪，和视听语音情感识别。结果表明，MDVAE在其潜空间中有效地融合了听觉和视觉信息。他们还表明，学习的视听语音的静态表示可以用于情感识别与少数标记的数据，并具有更好的准确性相比，单峰基线和最先进的监督模型的基础上的视听Transformer架构。摘要：In this paper, we present a multimodal \textit{and} dynamical VAE (MDVAE) applied to unsupervised audio-visual speech representation learning. The latent space is structured to dissociate the latent dynamical factors that are shared between the modalities from those that are specific to each modality. A static latent variable is also introduced to encode the information that is constant over time within an audiovisual speech sequence. The model is trained in an unsupervised manner on an audiovisual emotional speech dataset, in two stages. In the first stage, a vector quantized VAE (VQ-VAE) is learned independently for each modality, without temporal modeling. The second stage consists in learning the MDVAE model on the intermediate representation of the VQ-VAEs before quantization. The disentanglement between static versus dynamical and modality-specific versus modality-common information occurs during this second training stage. Extensive experiments are conducted to investigate how audiovisual speech latent factors are encoded in the latent space of MDVAE. These experiments include manipulating audiovisual speech, audiovisual facial image denoising, and audiovisual speech emotion recognition. The results show that MDVAE effectively combines the audio and visual information in its latent space. They also show that the learned static representation of audiovisual speech can be used for emotion recognition with few labeled data, and with better accuracy compared with unimodal baselines and a state-of-the-art supervised model based on an audiovisual transformer architecture.【3】 A vector quantized masked autoencoder for audiovisual speech emotion recognition标题：一种用于视听语音情感识别的矢量量化掩码自动编码器链接：https://arxiv.org/abs/2305.03568作者：Samir Sadok,Simon Leglaive,Renaud Séguier机构：CentraleSupélec, IETR UMR CNRS , Rennes备注：14 pages, 4 figures, this https URL摘要：虽然全监督模型已被证明是有效的视听语音情感识别（SER），有限的可用性标记的数据仍然是该领域的一个主要挑战。为了解决这个问题，自监督学习方法，如掩码自动编码器（MAE），已获得普及作为潜在的解决方案。在本文中，我们提出了VQ-MAE-AV模型，一个矢量量化MAE专门设计用于视听语音自监督表示学习。与现有的多模态MAE依赖于处理的原始视听语音数据，所提出的方法采用了一个自我监督的范例，基于离散的音频和视觉语音表示学习两个预先训练的矢量量化变分自动编码器。实验结果表明，所提出的方法，这是预先训练的VoxCeleb 2数据库和微调标准的情感视听语音数据集，优于国家的最先进的视听SER方法。摘要：While fully-supervised models have been shown to be effective for audiovisual speech emotion recognition (SER), the limited availability of labeled data remains a major challenge in the field. To address this issue, self-supervised learning approaches, such as masked autoencoders (MAEs), have gained popularity as potential solutions. In this paper, we propose the VQ-MAE-AV model, a vector quantized MAE specifically designed for audiovisual speech self-supervised representation learning. Unlike existing multimodal MAEs that rely on the processing of the raw audiovisual speech data, the proposed method employs a self-supervised paradigm based on discrete audio and visual speech representations learned by two pre-trained vector quantized variational autoencoders. Experimental results show that the proposed approach, which is pre-trained on the VoxCeleb2 database and fine-tuned on standard emotional audiovisual speech datasets, outperforms the state-of-the-art audiovisual SER methods.【4】 Exploring Softly Masked Language Modelling for Controllable Symbolic Music Generation标题：探索可控符号音乐生成的软屏蔽语言建模链接：https://arxiv.org/abs/2305.03530作者：Nicolas Jonason,Bob L. T. Sturm机构：Version备注：First version摘要：本文介绍了一些早期的探索应用软屏蔽语言建模（SMLM）的符号音乐生成。SMLM可以被视为掩蔽语言建模（MLM）的概括，其中输入集的每个元素不是已知的或未知的，而是部分已知的。我们展示了一些结果，应用SMLM约束的符号音乐生成使用Transformer编码器架构。一些音频示例可在www.example.com上获得https://erl-j.github.io/smlm-web-supplement/摘要：This document presents some early explorations of applying Softly Masked Language Modelling (SMLM) to symbolic music generation. SMLM can be seen as a generalisation of masked language modelling (MLM), where instead of each element of the input set being either known or unknown, elements can be partly known. We demonstrate some results of applying SMLM to constrained symbolic music generation using a transformer encoder architecture. Several audio examples are available at https://erl-j.github.io/smlm-web-supplement/【5】 Compressing audio CNNs with graph centrality based filter pruning标题：基于图中心性的过滤剪枝压缩音频CNN链接：https://arxiv.org/abs/2305.03391作者：James A King,Arshdeep Singh,Mark D. Plumbley机构：Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK.摘要：卷积神经网络（CNN）在许多现实问题的高性能解决方案中很常见，例如音频分类。CNN有许多参数和滤波器，其中一些对性能的影响比其他的更大。这意味着网络可能包含许多不必要的滤波器，增加了CNN的计算和内存需求，同时提供有限的性能优势。为了使CNN更有效，我们提出了一个修剪框架，消除具有最高“共性”的过滤器。我们使用图论的“中心性”概念来衡量这种共性。我们假设具有高中心性的过滤器应该被消除，因为它代表了通用性，并且可以被其他过滤器取代，而不会对网络的性能产生太大影响。声学场景分类和音频标记的建议框架的实验评估。在DCASE 2021任务1A基线网络上，我们提出的方法减少了71\%的计算每推理与50\%的参数减少在不到两个百分点的精度下降相比，原始网络。对于大规模的CNN，如PANN设计的音频标记，我们的方法减少了24\%的计算，每个推理少41\%的参数，在性能略有改善。摘要：Convolutional neural networks (CNNs) are commonplace in high-performing solutions to many real-world problems, such as audio classification. CNNs have many parameters and filters, with some having a larger impact on the performance than others. This means that networks may contain many unnecessary filters, increasing a CNN's computation and memory requirements while providing limited performance benefits. To make CNNs more efficient, we propose a pruning framework that eliminates filters with the highest "commonality". We measure this commonality using the graph-theoretic concept of "centrality". We hypothesise that a filter with a high centrality should be eliminated as it represents commonality and can be replaced by other filters without affecting the performance of a network much. An experimental evaluation of the proposed framework is performed on acoustic scene classification and audio tagging. On the DCASE 2021 Task 1A baseline network, our proposed method reduces computations per inference by 71\% with 50\% fewer parameters at less than a two percentage point drop in accuracy compared to the original network. For large-scale CNNs such as PANNs designed for audio tagging, our method reduces 24\% computations per inference with 41\% fewer parameters at a slight improvement in performance.【6】 Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks标题：基于传感器和注意力的语音到文本混合编解码器建模链接：https://arxiv.org/abs/2305.03101作者：Yun Tang,Anna Y. Sun,Hirofumi Inaguma,Xinyue Chen,Ning Dong,Xutai Ma,Paden D. Tomasello,Juan Pino机构：Meta AI⋆, Carnegie Mellon University♦备注：ACL 2023 main conference摘要：Transducer和Attention Based Encoder-Decoder（AED）是两种广泛用于语音到文本任务的框架。它们是为不同的目的而设计的，对于语音到文本的任务来说，每个都有自己的优点和缺点。为了利用这两种建模方法的优势，我们提出了一个解决方案，通过结合换能器和基于注意力的编码器-解码器（TAED）的语音到文本的任务。该新方法利用AED在非单调序列中的强度来进行序列学习，同时保留换能器的流属性。在所提出的框架中，换能器和AED共享相同的语音编码器。换能器中的预测器由AED模型中的解码器代替，并且解码器的输出以语音输入为条件，而不是来自无条件语言模型的输出。所提出的解决方案确保了模型通过覆盖所有可能的读/写场景进行优化，并为流应用程序创建匹配的环境。我们在\textsc{MuST-C}数据集上评估了所提出的方法，结果表明TAED在离线自动语音识别（ASR）和语音到文本翻译（ST）任务中的表现明显优于Transducer。在流式传输情况下，TAED在ASR任务和一个ST方向上优于换能器，而在另一个平移方向上获得了相当的结果。摘要：Transducer and Attention based Encoder-Decoder (AED) are two widely used frameworks for speech-to-text tasks. They are designed for different purposes and each has its own benefits and drawbacks for speech-to-text tasks. In order to leverage strengths of both modeling methods, we propose a solution by combining Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks. The new method leverages AED's strength in non-monotonic sequence to sequence learning while retaining Transducer's streaming property. In the proposed framework, Transducer and AED share the same speech encoder. The predictor in Transducer is replaced by the decoder in the AED model, and the outputs of the decoder are conditioned on the speech inputs instead of outputs from an unconditioned language model. The proposed solution ensures that the model is optimized by covering all possible read/write scenarios and creates a matched environment for streaming applications. We evaluate the proposed approach on the \textsc{MuST-C} dataset and the findings demonstrate that TAED performs significantly better than Transducer for offline automatic speech recognition (ASR) and speech-to-text translation (ST) tasks. In the streaming case, TAED outperforms Transducer in the ASR task and one ST direction while comparable results are achieved in another translation direction.【7】 Blind identification of Ambisonic reduced room impulse response标题：弱化室内冲激反应的盲辨识链接：https://arxiv.org/abs/2305.03558作者：Srđan Kitić,Jérôme Daniel备注：Submitted摘要：最近提出的广义时域速度向量（GTVV）是球谐（又名高保真度立体声）域中的相对房间脉冲响应的概括，其允许早期回波参数的盲估计：各个反射的方向和相对延迟。然而，推导出的GTVV的封闭形式的表达式要求几个假设，最重要的是，参考信号的脉冲响应需要是一个最小相位滤波器。在实践中，通过朝向源的到达方向的空间滤波来获得参考，并且上述条件受所应用的波束形成器的性能（并且因此受高保真度立体声阵列阶数）的限制。在目前的工作中，我们建议绕过这个问题，通过适当的建模GTVV时间序列，这不仅允许放松的初始假设，但也提取其中的信息是一个更一致和有效的方式，进入盲系统识别的领域。使用实测房间脉冲响应的实验证实了所提出的方法的有效性。摘要：Recently proposed Generalized Time-domain Velocity Vector (GTVV) is a generalization of relative room impulse response in spherical harmonic (aka Ambisonic) domain that allows for blind estimation of early-echo parameters: the directions and relative delays of individual reflections. However, the derived closed-form expression of GTVV mandates few assumptions to hold, most important being that the impulse response of the reference signal needs to be a minimum-phase filter. In practice, the reference is obtained by spatial filtering towards the Direction-of-Arrival of the source, and the aforementioned condition is bounded by the performance of the applied beamformer (and thus, by the Ambisonic array order). In the present work, we suggest to circumvent this problem by properly modelling the GTVV time series, which permits not only to relax the initial assumptions, but also to extract the information therein is a more consistent and efficient manner, entering the realm of blind system identification. Experiments using measured room impulse responses confirm the effectiveness of the proposed approach.【8】 Time-weighted Frequency Domain Audio Representation with GMM Estimator for Anomalous Sound Detection标题：基于GMM估计器的时间加权频域音频表示及其在异声检测中的应用链接：https://arxiv.org/abs/2305.03328作者：Jian Guan,Youde Liu,Qiaoxi Zhu,Tieran Zheng,Jiqing Han,Wenwu Wang机构：Group of Intelligent Signal Processing, College of Computer Science and Technology, Harbin Engineering University, China, School of Computer Science and Technology, Harbin Institute of Technology, China备注：To appear at ICASSP 2023摘要：虽然深度学习是无监督异常声音检测的主流方法，但以统计音频表示作为输入的高斯混合模型（GMM）可以以更低的模型复杂度和更少的参数获得类似的结果。现有的统计频率表示，例如，对数梅尔频谱图的平均值或最大值随着时间的推移，并不总是工作以及不同的机器。本文提出了一种基于时间加权频域表示的GMM方法（TWFR-GMM），用于异常声检测。TWFR是一种通用的统计频域表示，可以适应不同的机器类型，使用时域上的全局加权排名池化。这使得GMM估计器识别异常，即使在域移位条件下，如用基于马氏距离的度量可视化的。在DCASE 2022 Challenge Task 2数据集上的实验表明，我们的方法比最近的深度学习方法具有更好的检测性能。TWFR-GMM是我们提交的核心，在DCASE 2022挑战任务2中获得第三名。摘要：Although deep learning is the mainstream method in unsupervised anomalous sound detection, Gaussian Mixture Model (GMM) with statistical audio frequency representation as input can achieve comparable results with much lower model complexity and fewer parameters. Existing statistical frequency representations, e.g, the log-Mel spectrogram's average or maximum over time, do not always work well for different machines. This paper presents Time-Weighted Frequency Domain Representation (TWFR) with the GMM method (TWFR-GMM) for anomalous sound detection. The TWFR is a generalized statistical frequency domain representation that can adapt to different machine types, using the global weighted ranking pooling over time-domain. This allows GMM estimator to recognize anomalies, even under domain-shift conditions, as visualized with a Mahalanobis distance-based metric. Experiments on DCASE 2022 Challenge Task2 dataset show that our method has better detection performance than recent deep learning methods. TWFR-GMM is the core of our submission that achieved the 3rd place in DCASE 2022 Challenge Task2.【9】 Employing Hybrid Deep Neural Networks on Dari Speech标题：混合深度神经网络在达里语语音中的应用链接：https://arxiv.org/abs/2305.03200作者：Jawid Ahmad Baktash,Mursal Dawodi机构：LIA, Avignon University, Avignon, France备注：this https URL摘要：这篇论文是我们以前会议论文的延伸。近年来，研究人员对开发和改进语音识别系统以促进和增强人机交互越来越感兴趣。如今，自动语音识别（ASR）系统已经变得无处不在，从游戏到翻译系统、机器人等等，几乎无处不在。然而，对于低资源语言的语音识别系统仍需要大量的研究。本文重点介绍使用Mel频率倒谱系数（MFCC）特征提取方法和三种不同的深度神经网络模型识别达里语中的单个单词：卷积神经网络（CNN）、递归神经网络（RNN）和多层感知器（MLP），以及结合CNN和RNN的两种混合模型。我们评估这些模型使用一个孤立的达里语单词语料库，我们已经创建，包括1000话语20短达里语。我们的研究取得了令人印象深刻的平均准确率为98.365%。摘要：This paper is an extension of our previous conference paper. In recent years, there has been a growing interest among researchers in developing and improving speech recognition systems to facilitate and enhance human-computer interaction. Today, Automatic Speech Recognition (ASR) systems have become ubiquitous, used in everything from games to translation systems, robots, and more. However, much research is still needed on speech recognition systems for low-resource languages. This article focuses on the recognition of individual words in the Dari language using the Mel-frequency cepstral coefficients (MFCCs) feature extraction method and three different deep neural network models: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Multilayer Perceptron (MLP), as well as two hybrid models combining CNN and RNN. We evaluate these models using an isolated Dari word corpus that we have created, consisting of 1000 utterances for 20 short Dari terms. Our study achieved an impressive average accuracy of 98.365%.【10】 Plug-and-Play Multilingual Few-shot Spoken Words Recognition标题：即插即用多语种Few-Shot语音识别链接：https://arxiv.org/abs/2305.03058作者：Aaqib Saeed,Vasileios Tsouvalas机构：Eindhoven University of Technology备注：Code: this https URL摘要：随着技术的进步和数字设备的普及，无缝人机通信越来越重要。移动、可穿戴和其他物联网（IoT）设备的日益普及改变了我们与这些智能设备的交互方式，使准确的口语识别成为有效交互的关键组成部分。然而，构建可以处理新关键字的鲁棒口语词检测系统仍然具有挑战性，特别是对于训练数据有限的低资源语言。在这里，我们提出了PLiX，这是一个多语言和即插即用的关键字识别系统，它利用Few-Shot学习来利用大量的真实数据，并在测试时识别看不见的口语单词。我们的Few-Shot深度模型是通过20种语言的数百万个一秒音频片段学习的，在高效的同时实现了最先进的性能。广泛的评估表明，PLiX可以推广到新的口语单词，只要一个支持的例子，并表现良好的看不见的语言开箱即用。我们发布模型和推理代码，作为未来研究和新兴设备语音用户界面开发的基础。摘要：As technology advances and digital devices become prevalent, seamless human-machine communication is increasingly gaining significance. The growing adoption of mobile, wearable, and other Internet of Things (IoT) devices has changed how we interact with these smart devices, making accurate spoken words recognition a crucial component for effective interaction. However, building robust spoken words detection system that can handle novel keywords remains challenging, especially for low-resource languages with limited training data. Here, we propose PLiX, a multilingual and plug-and-play keyword spotting system that leverages few-shot learning to harness massive real-world data and enable the recognition of unseen spoken words at test-time. Our few-shot deep models are learned with millions of one-second audio clips across 20 languages, achieving state-of-the-art performance while being highly efficient. Extensive evaluations show that PLiX can generalize to novel spoken words given as few as just one support example and performs well on unseen languages out of the box. We release models and inference code to serve as a foundation for future research and voice-enabled user interface development for emerging devices.eess.AS音频处理【1】 Blind identification of Ambisonic reduced room impulse response标题：弱化室内冲激反应的盲辨识链接：https://arxiv.org/abs/2305.03558*与cs.SD语音【7】为同一篇作者：Srđan Kitić,Jérôme Daniel备注：Submitted摘要：最近提出的广义时域速度向量（GTVV）是球谐（又名高保真度立体声）域中的相对房间脉冲响应的概括，其允许早期回波参数的盲估计：各个反射的方向和相对延迟。然而，推导出的GTVV的封闭形式的表达式要求几个假设，最重要的是，参考信号的脉冲响应需要是一个最小相位滤波器。在实践中，通过朝向源的到达方向的空间滤波来获得参考，并且上述条件受所应用的波束形成器的性能（并且因此受高保真度立体声阵列阶数）的限制。在目前的工作中，我们建议绕过这个问题，通过适当的建模GTVV时间序列，这不仅允许放松的初始假设，但也提取其中的信息是一个更一致和有效的方式，进入盲系统识别的领域。使用实测房间脉冲响应的实验证实了所提出的方法的有效性。摘要：Recently proposed Generalized Time-domain Velocity Vector (GTVV) is a generalization of relative room impulse response in spherical harmonic (aka Ambisonic) domain that allows for blind estimation of early-echo parameters: the directions and relative delays of individual reflections. However, the derived closed-form expression of GTVV mandates few assumptions to hold, most important being that the impulse response of the reference signal needs to be a minimum-phase filter. In practice, the reference is obtained by spatial filtering towards the Direction-of-Arrival of the source, and the aforementioned condition is bounded by the performance of the applied beamformer (and thus, by the Ambisonic array order). In the present work, we suggest to circumvent this problem by properly modelling the GTVV time series, which permits not only to relax the initial assumptions, but also to extract the information therein is a more consistent and efficient manner, entering the realm of blind system identification. Experiments using measured room impulse responses confirm the effectiveness of the proposed approach.【2】 Time-weighted Frequency Domain Audio Representation with GMM Estimator for Anomalous Sound Detection标题：基于GMM估计器的时间加权频域音频表示及其在异声检测中的应用链接：https://arxiv.org/abs/2305.03328*与cs.SD语音【8】为同一篇作者：Jian Guan,Youde Liu,Qiaoxi Zhu,Tieran Zheng,Jiqing Han,Wenwu Wang机构：Group of Intelligent Signal Processing, College of Computer Science and Technology, Harbin Engineering University, China, School of Computer Science and Technology, Harbin Institute of Technology, China备注：To appear at ICASSP 2023摘要：虽然深度学习是无监督异常声音检测的主流方法，但以统计音频表示作为输入的高斯混合模型（GMM）可以以更低的模型复杂度和更少的参数获得类似的结果。现有的统计频率表示，例如，对数梅尔频谱图的平均值或最大值随着时间的推移，并不总是工作以及不同的机器。本文提出了一种基于时间加权频域表示的GMM方法（TWFR-GMM），用于异常声检测。TWFR是一种通用的统计频域表示，可以适应不同的机器类型，使用时域上的全局加权排名池化。这使得GMM估计器识别异常，即使在域移位条件下，如用基于马氏距离的度量可视化的。在DCASE 2022 Challenge Task 2数据集上的实验表明，我们的方法比最近的深度学习方法具有更好的检测性能。TWFR-GMM是我们提交的核心，在DCASE 2022挑战任务2中获得第三名。摘要：Although deep learning is the mainstream method in unsupervised anomalous sound detection, Gaussian Mixture Model (GMM) with statistical audio frequency representation as input can achieve comparable results with much lower model complexity and fewer parameters. Existing statistical frequency representations, e.g, the log-Mel spectrogram's average or maximum over time, do not always work well for different machines. This paper presents Time-Weighted Frequency Domain Representation (TWFR) with the GMM method (TWFR-GMM) for anomalous sound detection. The TWFR is a generalized statistical frequency domain representation that can adapt to different machine types, using the global weighted ranking pooling over time-domain. This allows GMM estimator to recognize anomalies, even under domain-shift conditions, as visualized with a Mahalanobis distance-based metric. Experiments on DCASE 2022 Challenge Task2 dataset show that our method has better detection performance than recent deep learning methods. TWFR-GMM is the core of our submission that achieved the 3rd place in DCASE 2022 Challenge Task2.【3】 Employing Hybrid Deep Neural Networks on Dari Speech标题：混合深度神经网络在达里语语音中的应用链接：https://arxiv.org/abs/2305.03200*与cs.SD语音【9】为同一篇作者：Jawid Ahmad Baktash,Mursal Dawodi机构：LIA, Avignon University, Avignon, France备注：this https URL摘要：这篇论文是我们以前会议论文的延伸。近年来，研究人员对开发和改进语音识别系统以促进和增强人机交互越来越感兴趣。如今，自动语音识别（ASR）系统已经变得无处不在，从游戏到翻译系统、机器人等等，几乎无处不在。然而，对于低资源语言的语音识别系统仍需要大量的研究。本文重点介绍使用Mel频率倒谱系数（MFCC）特征提取方法和三种不同的深度神经网络模型识别达里语中的单个单词：卷积神经网络（CNN）、递归神经网络（RNN）和多层感知器（MLP），以及结合CNN和RNN的两种混合模型。我们评估这些模型使用一个孤立的达里语单词语料库，我们已经创建，包括1000话语20短达里语。我们的研究取得了令人印象深刻的平均准确率为98.365%。摘要：This paper is an extension of our previous conference paper. In recent years, there has been a growing interest among researchers in developing and improving speech recognition systems to facilitate and enhance human-computer interaction. Today, Automatic Speech Recognition (ASR) systems have become ubiquitous, used in everything from games to translation systems, robots, and more. However, much research is still needed on speech recognition systems for low-resource languages. This article focuses on the recognition of individual words in the Dari language using the Mel-frequency cepstral coefficients (MFCCs) feature extraction method and three different deep neural network models: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Multilayer Perceptron (MLP), as well as two hybrid models combining CNN and RNN. We evaluate these models using an isolated Dari word corpus that we have created, consisting of 1000 utterances for 20 short Dari terms. Our study achieved an impressive average accuracy of 98.365%.【4】 Plug-and-Play Multilingual Few-shot Spoken Words Recognition标题：即插即用多语种Few-Shot语音识别链接：https://arxiv.org/abs/2305.03058*与cs.SD语音【10】为同一篇作者：Aaqib Saeed,Vasileios Tsouvalas机构：Eindhoven University of Technology备注：Code: this https URL摘要：随着技术的进步和数字设备的普及，无缝人机通信越来越重要。移动、可穿戴和其他物联网（IoT）设备的日益普及改变了我们与这些智能设备的交互方式，使准确的口语识别成为有效交互的关键组成部分。然而，构建可以处理新关键字的鲁棒口语词检测系统仍然具有挑战性，特别是对于训练数据有限的低资源语言。在这里，我们提出了PLiX，这是一个多语言和即插即用的关键字识别系统，它利用Few-Shot学习来利用大量的真实数据，并在测试时识别看不见的口语单词。我们的Few-Shot深度模型是通过20种语言的数百万个一秒音频片段学习的，在高效的同时实现了最先进的性能。广泛的评估表明，PLiX可以推广到新的口语单词，只要一个支持的例子，并表现良好的看不见的语言开箱即用。我们发布模型和推理代码，作为未来研究和新兴设备语音用户界面开发的基础。摘要：As technology advances and digital devices become prevalent, seamless human-machine communication is increasingly gaining significance. The growing adoption of mobile, wearable, and other Internet of Things (IoT) devices has changed how we interact with these smart devices, making accurate spoken words recognition a crucial component for effective interaction. However, building robust spoken words detection system that can handle novel keywords remains challenging, especially for low-resource languages with limited training data. Here, we propose PLiX, a multilingual and plug-and-play keyword spotting system that leverages few-shot learning to harness massive real-world data and enable the recognition of unseen spoken words at test-time. Our few-shot deep models are learned with millions of one-second audio clips across 20 languages, achieving state-of-the-art performance while being highly efficient. Extensive evaluations show that PLiX can generalize to novel spoken words given as few as just one support example and performs well on unseen languages out of the box. We release models and inference code to serve as a foundation for future research and voice-enabled user interface development for emerging devices.【5】 Physics-Based Acoustic Holograms标题：基于物理的声全息图链接：https://arxiv.org/abs/2305.03625*与cs.SD语音【1】为同一篇作者：Antonio Stanziola,Ben T. Cox,Bradley E. Treeby,Michael D. Brown机构：Dept. of Medical Physics and Biomedical Engineering, University College London, Gower St, London, WC,E ,BT, Dept. of Neuroscience, Erasmus MC, Rotterdam, Netherlands摘要：增材制造的进步已经使得能够实现廉价的、可扩展的衍射声透镜，其可以用于经由相位和/或振幅调制生成复杂的声场。然而，这些全息图的设计依赖于一个薄的元件近似从光学适应，可以严重限制所实现的声场的保真度。在这里，我们介绍了基于物理的声学全息图与复杂的内部结构。通过优化全息图内的声学特性分布，使用具有制造约束的可微分声学模型来设计结构。全息图可以使用当代3D打印机简单且廉价地制造。实验测量表明，与传统的薄元件全息图相比，显着的改善。摘要：Advances in additive manufacturing have enabled the realisation of inexpensive, scalable, diffractive acoustic lenses that can be used to generate complex acoustic fields via phase and/or amplitude modulation. However, the design of these holograms relies on a thin-element approximation adapted from optics which can severely limit the fidelity of the realised acoustic field. Here, we introduce physics-based acoustic holograms with a complex internal structure. The structures are designed using a differentiable acoustic model with manufacturing constraints via optimisation of the acoustic property distribution within the hologram. The holograms can be fabricated simply and inexpensively using contemporary 3D printers. Experimental measurements demonstrate a significant improvement compared to conventional thin-element holograms.【6】 A Multimodal Dynamical Variational Autoencoder for Audiovisual Speech Representation Learning标题：一种用于视听语音表征学习的多模式动态变分自动编码器链接：https://arxiv.org/abs/2305.03582*与cs.SD语音【2】为同一篇作者：Samir Sadok,Simon Leglaive,Laurent Girin,Xavier Alameda-Pineda,Renaud Séguier机构：CentraleSupélec, IETR UMR CNRS , France, Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France, Inria, Univ. Grenoble Alpes, CNRS, LJK, France备注：25 pages, 14 figures, this https URL摘要：在本文中，我们提出了一个多模态\textit{和}动态VAE（MDVAE）应用于无监督的视听语音表示学习。潜在空间被构造成将在模态之间共享的潜在动态因素与特定于每个模态的潜在动态因素分离。一个静态的潜在变量也被引入到编码的信息是恒定的，随着时间的推移在视听语音序列。该模型在两个阶段中以无监督的方式在视听情感语音数据集上进行训练。在第一阶段中，矢量量化的VAE（VQ-VAE）针对每个模态独立地学习，而不进行时间建模。第二阶段包括在量化之前在VQ-VAE的中间表示上学习MDVAE模型。静态与动态以及模态特定与模态公共信息之间的解纠缠发生在该第二训练阶段期间。大量的实验进行调查如何视听语音潜在因素编码的MDVAE的潜在空间。这些实验包括操纵视听语音，视听面部图像去噪，和视听语音情感识别。结果表明，MDVAE在其潜空间中有效地融合了听觉和视觉信息。他们还表明，学习的视听语音的静态表示可以用于情感识别与少数标记的数据，并具有更好的准确性相比，单峰基线和最先进的监督模型的基础上的视听Transformer架构。摘要：In this paper, we present a multimodal \textit{and} dynamical VAE (MDVAE) applied to unsupervised audio-visual speech representation learning. The latent space is structured to dissociate the latent dynamical factors that are shared between the modalities from those that are specific to each modality. A static latent variable is also introduced to encode the information that is constant over time within an audiovisual speech sequence. The model is trained in an unsupervised manner on an audiovisual emotional speech dataset, in two stages. In the first stage, a vector quantized VAE (VQ-VAE) is learned independently for each modality, without temporal modeling. The second stage consists in learning the MDVAE model on the intermediate representation of the VQ-VAEs before quantization. The disentanglement between static versus dynamical and modality-specific versus modality-common information occurs during this second training stage. Extensive experiments are conducted to investigate how audiovisual speech latent factors are encoded in the latent space of MDVAE. These experiments include manipulating audiovisual speech, audiovisual facial image denoising, and audiovisual speech emotion recognition. The results show that MDVAE effectively combines the audio and visual information in its latent space. They also show that the learned static representation of audiovisual speech can be used for emotion recognition with few labeled data, and with better accuracy compared with unimodal baselines and a state-of-the-art supervised model based on an audiovisual transformer architecture.【7】 A vector quantized masked autoencoder for audiovisual speech emotion recognition标题：一种用于视听语音情感识别的矢量量化掩码自动编码器链接：https://arxiv.org/abs/2305.03568*与cs.SD语音【3】为同一篇作者：Samir Sadok,Simon Leglaive,Renaud Séguier机构：CentraleSupélec, IETR UMR CNRS , Rennes备注：14 pages, 4 figures, this https URL摘要：虽然全监督模型已被证明是有效的视听语音情感识别（SER），有限的可用性标记的数据仍然是该领域的一个主要挑战。为了解决这个问题，自监督学习方法，如掩码自动编码器（MAE），已获得普及作为潜在的解决方案。在本文中，我们提出了VQ-MAE-AV模型，一个矢量量化MAE专门设计用于视听语音自监督表示学习。与现有的多模态MAE依赖于处理的原始视听语音数据，所提出的方法采用了一个自我监督的范例，基于离散的音频和视觉语音表示学习两个预先训练的矢量量化变分自动编码器。实验结果表明，所提出的方法，这是预先训练的VoxCeleb 2数据库和微调标准的情感视听语音数据集，优于国家的最先进的视听SER方法。摘要：While fully-supervised models have been shown to be effective for audiovisual speech emotion recognition (SER), the limited availability of labeled data remains a major challenge in the field. To address this issue, self-supervised learning approaches, such as masked autoencoders (MAEs), have gained popularity as potential solutions. In this paper, we propose the VQ-MAE-AV model, a vector quantized MAE specifically designed for audiovisual speech self-supervised representation learning. Unlike existing multimodal MAEs that rely on the processing of the raw audiovisual speech data, the proposed method employs a self-supervised paradigm based on discrete audio and visual speech representations learned by two pre-trained vector quantized variational autoencoders. Experimental results show that the proposed approach, which is pre-trained on the VoxCeleb2 database and fine-tuned on standard emotional audiovisual speech datasets, outperforms the state-of-the-art audiovisual SER methods.【8】 Exploring Softly Masked Language Modelling for Controllable Symbolic Music Generation标题：探索可控符号音乐生成的软屏蔽语言建模链接：https://arxiv.org/abs/2305.03530*与cs.SD语音【4】为同一篇作者：Nicolas Jonason,Bob L. T. Sturm机构：Version备注：First version摘要：本文介绍了一些早期的探索应用软屏蔽语言建模（SMLM）的符号音乐生成。SMLM可以被视为掩蔽语言建模（MLM）的概括，其中输入集的每个元素不是已知的或未知的，而是部分已知的。我们展示了一些结果，应用SMLM约束的符号音乐生成使用Transformer编码器架构。一些音频示例可在www.example.com上获得https://erl-j.github.io/smlm-web-supplement/摘要：This document presents some early explorations of applying Softly Masked Language Modelling (SMLM) to symbolic music generation. SMLM can be seen as a generalisation of masked language modelling (MLM), where instead of each element of the input set being either known or unknown, elements can be partly known. We demonstrate some results of applying SMLM to constrained symbolic music generation using a transformer encoder architecture. Several audio examples are available at https://erl-j.github.io/smlm-web-supplement/【9】 Compressing audio CNNs with graph centrality based filter pruning标题：基于图中心性的过滤剪枝压缩音频CNN链接：https://arxiv.org/abs/2305.03391*与cs.SD语音【5】为同一篇作者：James A King,Arshdeep Singh,Mark D. Plumbley机构：Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK.摘要：卷积神经网络（CNN）在许多现实问题的高性能解决方案中很常见，例如音频分类。CNN有许多参数和滤波器，其中一些对性能的影响比其他的更大。这意味着网络可能包含许多不必要的滤波器，增加了CNN的计算和内存需求，同时提供有限的性能优势。为了使CNN更有效，我们提出了一个修剪框架，消除具有最高“共性”的过滤器。我们使用图论的“中心性”概念来衡量这种共性。我们假设具有高中心性的过滤器应该被消除，因为它代表了通用性，并且可以被其他过滤器取代，而不会对网络的性能产生太大影响。声学场景分类和音频标记的建议框架的实验评估。在DCASE 2021任务1A基线网络上，我们提出的方法减少了71\%的计算每推理与50\%的参数减少在不到两个百分点的精度下降相比，原始网络。对于大规模的CNN，如PANN设计的音频标记，我们的方法减少了24\%的计算，每个推理少41\%的参数，在性能略有改善。摘要：Convolutional neural networks (CNNs) are commonplace in high-performing solutions to many real-world problems, such as audio classification. CNNs have many parameters and filters, with some having a larger impact on the performance than others. This means that networks may contain many unnecessary filters, increasing a CNN's computation and memory requirements while providing limited performance benefits. To make CNNs more efficient, we propose a pruning framework that eliminates filters with the highest "commonality". We measure this commonality using the graph-theoretic concept of "centrality". We hypothesise that a filter with a high centrality should be eliminated as it represents commonality and can be replaced by other filters without affecting the performance of a network much. An experimental evaluation of the proposed framework is performed on acoustic scene classification and audio tagging. On the DCASE 2021 Task 1A baseline network, our proposed method reduces computations per inference by 71\% with 50\% fewer parameters at less than a two percentage point drop in accuracy compared to the original network. For large-scale CNNs such as PANNs designed for audio tagging, our method reduces 24\% computations per inference with 41\% fewer parameters at a slight improvement in performance.【10】 Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks标题：基于传感器和注意力的语音到文本混合编解码器建模链接：https://arxiv.org/abs/2305.03101*与cs.SD语音【6】为同一篇作者：Yun Tang,Anna Y. Sun,Hirofumi Inaguma,Xinyue Chen,Ning Dong,Xutai Ma,Paden D. Tomasello,Juan Pino机构：Meta AI⋆, Carnegie Mellon University♦备注：ACL 2023 main conference摘要：Transducer和Attention Based Encoder-Decoder（AED）是两种广泛用于语音到文本任务的框架。它们是为不同的目的而设计的，对于语音到文本的任务来说，每个都有自己的优点和缺点。为了利用这两种建模方法的优势，我们提出了一个解决方案，通过结合换能器和基于注意力的编码器-解码器（TAED）的语音到文本的任务。该新方法利用AED在非单调序列中的强度来进行序列学习，同时保留换能器的流属性。在所提出的框架中，换能器和AED共享相同的语音编码器。换能器中的预测器由AED模型中的解码器代替，并且解码器的输出以语音输入为条件，而不是来自无条件语言模型的输出。所提出的解决方案确保了模型通过覆盖所有可能的读/写场景进行优化，并为流应用程序创建匹配的环境。我们在\textsc{MuST-C}数据集上评估了所提出的方法，结果表明TAED在离线自动语音识别（ASR）和语音到文本翻译（ST）任务中的表现明显优于Transducer。在流式传输情况下，TAED在ASR任务和一个ST方向上优于换能器，而在另一个平移方向上获得了相当的结果。摘要：Transducer and Attention based Encoder-Decoder (AED) are two widely used frameworks for speech-to-text tasks. They are designed for different purposes and each has its own benefits and drawbacks for speech-to-text tasks. In order to leverage strengths of both modeling methods, we propose a solution by combining Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks. The new method leverages AED's strength in non-monotonic sequence to sequence learning while retaining Transducer's streaming property. In the proposed framework, Transducer and AED share the same speech encoder. The predictor in Transducer is replaced by the decoder in the AED model, and the outputs of the decoder are conditioned on the speech inputs instead of outputs from an unconditioned language model. The proposed solution ensures that the model is optimized by covering all possible read/write scenarios and creates a matched environment for streaming applications. We evaluate the proposed approach on the \textsc{MuST-C} dataset and the findings demonstrate that TAED performs significantly better than Transducer for offline automatic speech recognition (ASR) and speech-to-text translation (ST) tasks. In the streaming case, TAED outperforms Transducer in the ASR task and one ST direction while comparable results are achieved in another translation direction.
每日语音速递
