今日论文合集：cs.SD语音13篇，eess.AS音频处理19篇。
cs.SD语音
【1】 Performance Comparison of Pre-trained Models for Speech-to-Text in  Turkish: Whisper-Small and Wav2Vec2-XLS-R-300M标题：用于土耳其语语音转文本的预训练模型的性能比较：Whisper-Small和Wav2Vec2-XLS-R-300M链接：https://arxiv.org/abs/2307.04765
作者：Oyku Berfin Mercan,Sercan Cepni,Davut Emre Tasar,Sukru Ozan备注：10 Pages, in Turkish language, Pre-Print摘要：在这项研究中，针对土耳其语检查了Whisper-Small和Wav 2 Vec 2-XLS-R-300 M模型的性能，这两个模型是两个预先训练的语音到文本的多语言模型。Mozilla Common Voice 11.0版是用土耳其语编写的，是一个开源数据集，用于研究。多语言模型Whisper- Small和Wav 2 Vec 2-XLS-R-300 M使用包含少量数据的该数据集进行微调。比较了两种模型的语音到文本的性能。Wav 2 Vec 2-XLS-R-300 M和Whisper-Small模型的WER值分别计算为0.28和0.16。此外，使用由呼叫中心记录准备的测试数据检查了模型的性能，这些数据未包括在训练和验证数据集中。摘要：In this study, the performances of the Whisper-Small and Wav2Vec2-XLS-R-300M models which are two pre-trained multilingual models for speech to text were examined for the Turkish language. Mozilla Common Voice version 11.0 which is prepared in Turkish language and is an open-source data set, was used in the study. The multilingual models, Whisper- Small and Wav2Vec2-XLS-R-300M were fine-tuned with this data set which contains a small amount of data. The speech to text performance of the two models was compared. WER values are calculated as 0.28 and 0.16 for the Wav2Vec2-XLS- R-300M and the Whisper-Small models respectively. In addition, the performances of the models were examined with the test data prepared with call center records that were not included in the training and validation dataset.
【2】 Learning Spatial Features from Audio-Visual Correspondence in Egocentric  Videos标题：从以自我为中心的视频的视听对应中学习空间特征链接：https://arxiv.org/abs/2307.04760
作者：Sagnik Majumder,Ziad Al-Halah,Kristen Grauman机构：University of Texas at Austin, FAIR, University of Utah摘要：我们提出了一个自我监督的方法，学习表示的基础上，在以自我为中心的视频空间视听对应。特别地，我们的方法利用掩蔽的自动编码框架来通过音频和视觉的协同作用合成掩蔽的双耳音频，从而学习两种模态之间的有用的空间关系。我们使用预训练的特征来处理两个在社交场景中需要空间理解的下游视频任务：活动说话人检测和空间音频去噪。我们通过大量的实验表明，我们的功能是通用的，足以改善多个国家的最先进的基线上两个公共的具有挑战性的以自我为中心的视频数据集，EgoCom和EasyCom。项目：http://vision.cs.utexas.edu/projects/ego_av_corr。摘要：We propose a self-supervised method for learning representations based on spatial audio-visual correspondences in egocentric videos. In particular, our method leverages a masked auto-encoding framework to synthesize masked binaural audio through the synergy of audio and vision, thereby learning useful spatial relationships between the two modalities. We use our pretrained features to tackle two downstream video tasks requiring spatial understanding in social scenarios: active speaker detection and spatial audio denoising. We show through extensive experiments that our features are generic enough to improve over multiple state-of-the-art baselines on two public challenging egocentric video datasets, EgoCom and EasyCom. Project: http://vision.cs.utexas.edu/projects/ego_av_corr.
【3】 Vocal Tract Area Estimation by Gradient Descent标题：用梯度下降法估计声道面积链接：https://arxiv.org/abs/2307.04702
作者：David Südholt,Mateo Cámara,Zhiyuan Xu,Joshua D. Reiss机构：Centre for Digital Music, Queen Mary University of London, London, UK, Information Processing & Telecomm. Center, Universidad Politécnica de Madrid, Madrid, Spain备注：Accepted to DAFx 2023摘要：发音特征可以通过允许用户直接修改诸如声音应变或嘴唇位置的参数来提供用于人类发声的合成的可解释的和灵活的控制。为了通过再合成进行这种操作，我们需要估计直接从音频记录中产生所需发声的特征。在这项工作中，我们提出了一个白盒优化技术估计声门源参数和声道形状从人类元音的音频记录。该方法是基于逆滤波和优化的波导模型的频率响应梯度下降，传播误差梯度通过映射的发音特征到声道面积函数.我们将这种方法应用到匹配的粉红长号，一个交互式发音合成器，一个给定的发声的声音的任务。我们发现，我们的方法准确地恢复粉红长号本身产生的音频控制功能。然后，我们比较我们的技术对进化优化算法和神经网络训练预测控制参数的音频。主观评价发现，我们的方法优于这些黑盒优化基线的任务，再现人类发声。摘要：Articulatory features can provide interpretable and flexible controls for the synthesis of human vocalizations by allowing the user to directly modify parameters like vocal strain or lip position. To make this manipulation through resynthesis possible, we need to estimate the features that result in a desired vocalization directly from audio recordings. In this work, we propose a white-box optimization technique for estimating glottal source parameters and vocal tract shapes from audio recordings of human vowels. The approach is based on inverse filtering and optimizing the frequency response of a wave\-guide model of the vocal tract with gradient descent, propagating error gradients through the mapping of articulatory features to the vocal tract area function. We apply this method to the task of matching the sound of the Pink Trombone, an interactive articulatory synthesizer, to a given vocalization. We find that our method accurately recovers control functions for audio generated by the Pink Trombone itself. We then compare our technique against evolutionary optimization algorithms and a neural network trained to predict control parameters from audio. A subjective evaluation finds that our approach outperforms these black-box optimization baselines on the task of reproducing human vocalizations.
【4】 VampNet: Music Generation via Masked Acoustic Token Modeling标题：VampNet：基于掩蔽声学符号建模的音乐生成链接：https://arxiv.org/abs/2307.04686
作者：Hugo Flores Garcia,Prem Seetharaman,Rithesh Kumar,Bryan Pardo机构：Descript Inc.,  Northwestern University摘要：我们介绍VampNet，一个掩蔽的声学令牌建模方法的音乐合成，压缩，修复和变化。我们在训练过程中使用可变的掩蔽时间表，这允许我们通过在推理过程中应用各种掩蔽方法（称为提示）来从模型中采样连贯的音乐。VampNet是非自回归的，利用双向Transformer架构，在正向传递中处理所有令牌。VampNet只需36次采样，就可以生成连贯的高保真音乐波形。我们表明，通过以各种方式提示VampNet，我们可以将其应用于音乐压缩，修复，outpainting，延续和循环变化（vamping）等任务。适当地提示，VampNet能够保持风格，流派，乐器和音乐的其他高级方面。这种灵活的提示功能使VampNet成为一个强大的音乐共同创作工具。代码和音频示例可在线获得。摘要：We introduce VampNet, a masked acoustic token modeling approach to music synthesis, compression, inpainting, and variation. We use a variable masking schedule during training which allows us to sample coherent music from the model by applying a variety of masking approaches (called prompts) during inference. VampNet is non-autoregressive, leveraging a bidirectional transformer architecture that attends to all tokens in a forward pass. With just 36 sampling passes, VampNet can generate coherent high-fidelity musical waveforms. We show that by prompting VampNet in various ways, we can apply it to tasks like music compression, inpainting, outpainting, continuation, and looping with variation (vamping). Appropriately prompted, VampNet is capable of maintaining style, genre, instrumentation, and other high-level aspects of the music. This flexible prompting capability makes VampNet a powerful music co-creation tool. Code and audio samples are available online.
【5】 The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023  Speech-to-Speech Translation Task标题：面向IWSLT 2023语音翻译任务的NPU-MSXF语音翻译系统链接：https://arxiv.org/abs/2307.04630
作者：Kun Song,Yi lei,Peikun Chen,Yiqing Cao,Kun Wei,Yongmao Zhang,Lei Xie,Ning Jiang,Guoqing Zhao机构：School of Computer Science, Northwestern Polytechnical University, China, Department of Computer Science and Technology, Nanjing University, China, MaShang Consumer Finance Co., Ltd, China备注：IWSLT@ACL 2023 system paper. Our submitted system ranks 1st in the S2ST task of the IWSLT 2023 evaluation campaign摘要：本文描述了用于IWSLT 2023语音到语音翻译（S2 ST）任务的NPU-MSXF系统，该任务旨在将多源英语语音翻译为汉语语音。该系统是建立在一个级联的方式，包括自动语音识别（ASR），机器翻译（MT），文本到语音（TTS）。我们付出了巨大的努力来处理具有挑战性的多源输入。具体来说，为了提高对多源语音输入的鲁棒性，我们采用了各种数据增强策略和基于ROVER的多个ASR模型输出的分数融合。为了更好地处理嘈杂的ASR成绩单，我们引入了一个三阶段的微调策略，以提高翻译的准确性。最后，我们建立了一个高自然度和音质的TTS模型，它利用了一个两阶段的框架，使用网络瓶颈功能作为一个强大的中间表示说话人音色和语言内容解纠缠。基于两阶段的框架，预先训练的说话人嵌入作为一个条件，将源英语语音中的说话人音色转换为翻译的中文语音。实验结果表明，我们的系统具有较高的翻译准确率，语音自然度，声音质量，说话人相似度。此外，它显示出良好的鲁棒性多源数据。摘要：This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech. The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous efforts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER-based score fusion on multiple ASR model outputs. To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity. Moreover, it shows good robustness to multi-source data.
【6】 EchoVest: Real-Time Sound Classification and Depth Perception Expressed  through Transcutaneous Electrical Nerve Stimulation标题：EchoVest：通过经皮神经电刺激表达的实时声音分类和深度感知链接：https://arxiv.org/abs/2307.04604
作者：Jesse Choe,Siddhant Sood,Ryan Park摘要：全世界有超过15亿人患有听力障碍。尽管已经为这些残疾人创造了各种技术，但这些技术中的大多数要么极其昂贵，要么在中低收入国家无法日常使用。为了解决这个问题，我们开发了一种新的辅助设备，EchoVest，让盲人/聋人直观地更加了解他们的环境。EchoVest通过利用基于声源的经皮电神经刺激（TENS）将振动传输到用户的身体。EchoVest还提供各种功能，包括声音定位，声音分类，降噪和深度感知。我们的目标是在准确性和计算成本方面优于基于CNN的机器学习模型，这是分类任务中最常用的机器学习模型。为此，我们开发并采用了一种新的音频流水线，该流水线适应音频频谱图Transformer（AST）模型，这是一种基于注意力的模型，用于我们的声音分类目的，以及用于降噪的快速傅立叶变换。大津法的应用，帮助我们找到最佳阈值的背景噪声的声音过滤，给我们更大的准确性。为了准确地计算方向和深度，我们应用复到达时间差算法和SOTA定位。我们最后的改进是使用盲源分离，使我们的算法适用于多个麦克风输入。最终的算法在许多检查点上取得了最先进的结果，包括在ESC-50数据集上进行环境声音分类的95.7%的准确率。摘要：Over 1.5 billion people worldwide live with hearing impairment. Despite various technologies that have been created for individuals with such disabilities, most of these technologies are either extremely expensive or inaccessible for everyday use in low-medium income countries. In order to combat this issue, we have developed a new assistive device, EchoVest, for blind/deaf people to intuitively become more aware of their environment. EchoVest transmits vibrations to the user's body by utilizing transcutaneous electric nerve stimulation (TENS) based on the source of the sounds. EchoVest also provides various features, including sound localization, sound classification, noise reduction, and depth perception. We aimed to outperform CNN-based machine-learning models, the most commonly used machine learning model for classification tasks, in accuracy and computational costs. To do so, we developed and employed a novel audio pipeline that adapts the Audio Spectrogram Transformer (AST) model, an attention-based model, for our sound classification purposes, and Fast Fourier Transforms for noise reduction. The application of Otsu's Method helped us find the optimal thresholds for background noise sound filtering and gave us much greater accuracy. In order to calculate direction and depth accurately, we applied Complex Time Difference of Arrival algorithms and SOTA localization. Our last improvement was to use blind source separation to make our algorithms applicable to multiple microphone inputs. The final algorithm achieved state-of-the-art results on numerous checkpoints, including a 95.7\% accuracy on the ESC-50 dataset for environmental sound classification.
【7】 HCLAS-X: Hierarchical and Cascaded Lyrics Alignment System Using  Multimodal Cross-Correlation标题：HCLAS-X：基于多模互相关的层次化级联歌词对齐系统链接：https://arxiv.org/abs/2307.04377
作者：Minsung Kang,Soochul Park,Keunwoo Choi机构：Gaudio Lab, Inc.摘要：在这项工作中，我们解决歌词对齐的挑战，这涉及到对齐歌曲的歌词和声乐成分。这个问题需要两种不同的模态，即文本和音频的对齐。为了克服这一挑战，我们提出了一个模型，在监督的方式进行训练，利用人声和歌词之间的潜在表示的互相关矩阵。我们的系统设计在一个层次和级联的方式。它首先在句子级别上预测同步时间，然后在单词级别上预测同步时间。这种设计使得系统能够处理长序列，因为互相关使用关于序列长度的二次存储器。在我们的实验中，我们证明，我们提出的系统实现了显着的改善，平均平均误差，展示其鲁棒性相比，以前的国家的最先进的模型。此外，我们成功地将其部署在几个音乐流媒体服务后，系统进行了定性分析。摘要：In this work, we address the challenge of lyrics alignment, which involves aligning the lyrics and vocal components of songs. This problem requires the alignment of two distinct modalities, namely text and audio. To overcome this challenge, we propose a model that is trained in a supervised manner, utilizing the cross-correlation matrix of latent representations between vocals and lyrics. Our system is designed in a hierarchical and cascaded manner. It predicts synced time first on a sentence-level and subsequently on a word-level. This design enables the system to process long sequences, as the cross-correlation uses quadratic memory with respect to sequence length. In our experiments, we demonstrate that our proposed system achieves a significant improvement in mean average error, showcasing its robustness in comparison to the previous state-of-the-art model. Additionally, we conduct a qualitative analysis of the system after successfully deploying it in several music streaming services.
【8】 Automatic Piano Transcription with Hierarchical Frequency-Time  Transformer标题：采用分级频率-时间转换器的自动钢琴改编链接：https://arxiv.org/abs/2307.04305
作者：Keisuke Toyama,Taketo Akama,Yukara Ikemiya,Yuhta Takida,Wei-Hsiang Liao,Yuki Mitsufuji机构：Sony Group Corporation, Japan,  Sony Computer Science Laboratories, Japan备注：8 pages, 6 figures, to be published in ISMIR2023摘要：考虑到长期的光谱和时间依赖性是钢琴自动转录的必要条件。这在确定复调钢琴内容中的每个音符的精确起始和偏移时特别有帮助。在这种情况下，我们可以依靠Transformers中的自我注意机制的能力来捕获频率和时间轴上的这些长期依赖关系。在这项工作中，我们提出了hFT-Transformer，这是一种自动音乐转录的方法，使用了两级分层频率-时间Transformer架构。第一层次包括时间轴上的卷积块、频率轴上的Transformer编码器和频率轴上的变换器解码器。然后，输出被馈送到第二层次，该层次由时间轴上的另一个Transformer编码器组成。我们使用广泛使用的MAPS和MAESTRO v3.0.0数据集评估了我们的方法，它展示了帧、音符、带偏移的音符以及带偏移和速度估计中所有度量的F1分数的最新性能。摘要：Taking long-term spectral and temporal dependencies into account is essential for automatic piano transcription. This is especially helpful when determining the precise onset and offset for each note in the polyphonic piano content. In this case, we may rely on the capability of self-attention mechanism in Transformers to capture these long-term dependencies in the frequency and time axes. In this work, we propose hFT-Transformer, which is an automatic music transcription method that uses a two-level hierarchical frequency-time Transformer architecture. The first hierarchy includes a convolutional block in the time axis, a Transformer encoder in the frequency axis, and a Transformer decoder that converts the dimension in the frequency axis. The output is then fed into the second hierarchy which consists of another Transformer encoder in the time axis. We evaluated our method with the widely used MAPS and MAESTRO v3.0.0 datasets, and it demonstrated state-of-the-art performance on all the F1-scores of the metrics among Frame, Note, Note with Offset, and Note with Offset and Velocity estimations.
【9】 Edge Storage Management Recipe with Zero-Shot Data Compression for Road  Anomaly Detection标题：基于零点数据压缩的道路异常检测边缘存储管理方案链接：https://arxiv.org/abs/2307.04298
作者：YeongHyeon Park,Uju Gim,Myung Jin Kim机构：SK Planet Co., Ltd., Seongnam, Rep. of Korea备注：5 pages, 3 figures, 4 tables摘要：最近的研究表明，基于边缘计算的道路异常检测系统，也可以同时进行数据收集。然而，边缘计算机的数据存储量很小，但我们需要长时间存储收集的音频样本，以便更新现有模型或开发新方法。因此，我们应该考虑一种高效的存储管理方法，同时保留高保真音频。从硬件角度看的方法（例如使用低分辨率麦克风）是减少文件大小的直观方法，但不建议使用，因为它从根本上切断了高频分量。另一方面，应推荐将收集的高分辨率音频编码为紧凑代码的计算文件压缩方法，因为它还提供了相应的解码方法。受此启发，我们提出了一种简单而有效的基于预训练自动编码器的数据压缩方法。预训练的自动编码器是为了音频超分辨率的目的而训练的，因此它可以用于编码或解码任何任意采样率。此外，它将降低从边缘到中央服务器的数据传输的通信成本。通过对比实验，我们确认zero-shot音频压缩和解压缩高度保留异常检测性能，同时提高存储和传输效率。摘要：Recent studies show edge computing-based road anomaly detection systems which may also conduct data collection simultaneously. However, the edge computers will have small data storage but we need to store the collected audio samples for a long time in order to update existing models or develop a novel method. Therefore, we should consider an approach for efficient storage management methods while preserving high-fidelity audio. A hardware-perspective approach, such as using a low-resolution microphone, is an intuitive way to reduce file size but is not recommended because it fundamentally cuts off high-frequency components. On the other hand, a computational file compression approach that encodes collected high-resolution audio into a compact code should be recommended because it also provides a corresponding decoding method. Motivated by this, we propose a way of simple yet effective pre-trained autoencoder-based data compression method. The pre-trained autoencoder is trained for the purpose of audio super-resolution so it can be utilized to encode or decode any arbitrary sampling rate. Moreover, it will reduce the communication cost for data transmission from the edge to the central server. Via the comparative experiments, we confirm that the zero-shot audio compression and decompression highly preserve anomaly detection performance while enhancing storage and transmission efficiency.
【10】 Can Generative Large Language Models Perform ASR Error Correction?标题：生成性大型语言模型可以执行ASR纠错吗？链接：https://arxiv.org/abs/2307.04172
作者：Rao Ma,Mengjie Qian,Potsawee Manakul,Mark Gales,Kate Knill机构：ALTA Institute, Machine Intelligence Lab, Department of Engineering, Cambridge University, UK摘要：ASR纠错继续充当语音识别系统的后处理的重要部分。传统上，这些模型使用底层ASR系统的解码结果和参考文本进行监督训练。这种方法是计算密集型的，并且在切换底层ASR模型时需要重新训练模型。近年来，大型语言模型及其以zero-shot方式执行自然语言处理任务的能力得到了发展。在本文中，我们以ChatGPT为例，以检查其执行ASR错误校正的能力，在zero-shot或1杆设置。我们使用ASR的N-最好的列表作为模型输入，并提出无约束的错误校正和N-最好的约束错误校正方法。Conformer-Transducer模型和预训练的Whisper模型的结果表明，我们可以使用强大的ChatGPT模型通过纠错大大提高ASR系统的性能。摘要：ASR error correction continues to serve as an important part of post-processing for speech recognition systems. Traditionally, these models are trained with supervised training using the decoding results of the underlying ASR system and the reference text. This approach is computationally intensive and the model needs to be re-trained when switching the underlying ASR model. Recent years have seen the development of large language models and their ability to perform natural language processing tasks in a zero-shot manner. In this paper, we take ChatGPT as an example to examine its ability to perform ASR error correction in the zero-shot or 1-shot settings. We use the ASR N-best list as model input and propose unconstrained error correction and N-best constrained error correction methods. Results on a Conformer-Transducer model and the pre-trained Whisper model show that we can largely improve the ASR system performance with error correction using the powerful ChatGPT model.
【11】 Emotion-Guided Music Accompaniment Generation Based on Variational  Autoencoder标题：基于变分自动编码器的情感制导音乐伴奏生成链接：https://arxiv.org/abs/2307.04015
作者：Qi Wang,Shubing Zhang,Li Zhou机构：China University of Geosciences(Wuhan)备注：Accepted By International Joint Conference on Neural Networks 2023(IJCNN2023)摘要：音乐伴奏的生成是作曲过程中一个至关重要的环节。深度神经网络在这一领域取得了重大进展，但人工智能如何有效地整合人类情感以创造美丽的伴奏仍然是一个挑战。现有模型难以在作曲时有效地表征神经网络模型中的人类情感。为了解决这个问题，我们提出了使用一个易于表示的情绪流模型，价/唤醒曲线，它允许的兼容性的情绪信息的模型内的数据转换和提高情绪因素的解释性，通过利用变分自动编码器的模型结构。此外，我们使用相对自我注意来维持音乐在乐句层次的结构，并结合乐理规则产生更丰富的伴奏。摘要：Music accompaniment generation is a crucial aspect in the composition process. Deep neural networks have made significant strides in this field, but it remains a challenge for AI to effectively incorporate human emotions to create beautiful accompaniments. Existing models struggle to effectively characterize human emotions within neural network models while composing music. To address this issue, we propose the use of an easy-to-represent emotion flow model, the Valence/Arousal Curve, which allows for the compatibility of emotional information within the model through data transformation and enhances interpretability of emotional factors by utilizing a Variational Autoencoder as the model structure. Further, we used relative self-attention to maintain the structure of the music at music phrase level and to generate a richer accompaniment when combined with the rules of music theory.
【12】 Exploiting an External Microphone for Binaural RTF-Vector-Based  Direction of Arrival Estimation for Multiple Speakers标题：利用外置麦克风实现双耳RTF-基于矢量的多说话人波达方向估计链接：https://arxiv.org/abs/2307.04460
作者：Daniel Fejgin,Simon Doclo机构：University of Oldenburg, Department of Medical Physics and Acoustics, and Cluster of Excellence Hearing,all, Oldenburg, Germany备注：Paper accepted for Forum Acusticum 2023摘要：在助听器应用中，一个重要的目标是准确地估计多个扬声器在噪声和混响环境中的到达方向（DOA）。最近，我们提出了一种双耳波达方向估计方法，其中的DOA的扬声器估计通过选择的方向之间的估计的相对传递函数（RTF）向量和原型消声RTF向量的数据库的所谓的厄米角谱最大化。RTF向量估计使用协方差白化（CW）的方法，这需要一个计算复杂的广义特征值分解。通过仅考虑其中一个扬声器可能支配其他扬声器的频率、噪声和混响来获得空间频谱。在这方面的贡献，我们利用的外部麦克风，在空间上分离的助听器麦克风的可用性，并考虑一个低复杂度的RTF矢量估计方法，假设在外部麦克风和助听器麦克风的不希望的组件之间的低空间相干性。在混响较弱、信噪比较低的声环境中，利用两个扬声器的录音和弥散状的串音噪声，仿真结果表明，该方法在较低的计算复杂度下获得了与CW方法相当的DOA估计性能。摘要：In hearing aid applications, an important objective is to accurately estimate the direction of arrival (DOA) of multiple speakers in noisy and reverberant environments. Recently, we proposed a binaural DOA estimation method, where the DOAs of the speakers are estimated by selecting the directions for which the so-called Hermitian angle spectrum between the estimated relative transfer function (RTF) vector and a database of prototype anechoic RTF vectors is maximized. The RTF vector is estimated using the covariance whitening (CW) method, which requires a computationally complex generalized eigenvalue decomposition. The spatial spectrum is obtained by only considering frequencies where it is likely that one speaker dominates over the other speakers, noise and reverberation. In this contribution, we exploit the availability of an external microphone that is spatially separated from the hearing aid microphones and consider a low-complexity RTF vector estimation method that assumes a low spatial coherence between the undesired components in the external microphone and the hearing aid microphones. Using recordings of two speakers and diffuse-like babble noise in acoustic environments with mild reverberation and low signal-to-noise ratio, simulation results show that the proposed method yields a comparable DOA estimation performance as the CW method at a lower computational complexity.
【13】 On decoder-only architecture for speech-to-text and large language model  integration标题：用于语音到文本和大语言模型集成的仅解码器架构链接：https://arxiv.org/abs/2307.03917
作者：Jian Wu,Yashesh Gaur,Zhuo Chen,Long Zhou,Yimeng Zhu,Tianrui Wang,Jinyu Li,Shujie Liu,Bo Ren,Linquan Liu,Yu Wu机构：Microsoft, One Microsoft Way, Redmond, USA摘要：大型语言模型（LLM）在自然语言处理领域取得了显著的成功，使得使用自然语言进行更好的人机交互成为可能。然而，语音信号到LLM的无缝集成还没有被很好地探索。对于语音处理任务，“仅解码器”架构也没有得到很好的研究。在这项研究中，我们介绍了语音LLaMA，一种新的方法，有效地将声学信息到基于文本的大型语言模型。我们的方法利用连接时间分类和一个简单的音频编码器映射压缩的声学特征的连续语义空间的LLM。此外，我们进一步探索解码器的语音到文本的任务，通过训练一个较小规模的随机初始化的语音LLaMA模型，从语音文本配对数据单独的架构。我们对多语言语音到文本翻译任务进行了实验，并在强基线上表现出显着的改进，突出了仅解码器模型用于语音到文本转换的潜在优势。摘要：Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language. However, the seamless integration of speech signals into LLMs has not been explored well. The "decoder-only" architecture has also not been well studied for speech processing tasks. In this research, we introduce Speech-LLaMA, a novel approach that effectively incorporates acoustic information into text-based large language models. Our method leverages Connectionist Temporal Classification and a simple audio encoder to map the compressed acoustic features to the continuous semantic space of the LLM. In addition, we further probe the decoder-only architecture for speech-to-text tasks by training a smaller scale randomly initialized speech-LLaMA model from speech-text paired data alone. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines, highlighting the potential advantages of decoder-only models for speech-to-text conversion.
eess.AS音频处理
【1】 Behavioral Analysis of Pathological Speaker Embeddings of Patients  During Oncological Treatment of Oral Cancer标题：口腔癌肿瘤治疗患者病理性扬声器植入的行为学分析链接：https://arxiv.org/abs/2307.04744
作者：Jenthe Thienpondt,Kris Demuynck机构：IDLab, Department of Electronics and Information Systems, Ghent University - imec, Belgium, Department of Oral and Maxillofacial Surgery and Special Dental Care, University Medical Center, Utrecht, Utrecht University, The Netherlands备注：proceedings of INTERSPEECH 2023摘要：在本文中，我们分析了口腔癌治疗期间患者的说话人嵌入行为。首先，我们发现，治疗前和治疗后的扬声器嵌入显着不同，通知语音特征的实质性变化。然而，在术后12个月后观察到部分恢复到术前的声音特征。其次，在不同的治疗阶段的同一扬声器的相似性是类似的健康的扬声器，这表明嵌入可以捕获的特征，甚至严重受损的语音。最后，说话人确认分析表明，一个稳定的假阳性率和可变的假阴性率时，结合不同的治疗阶段的语音样本。这表明嵌入对其他说话者的鲁棒性，同时仍然在治疗期间捕获变化的语音特性。据我们所知，这是口腔癌患者治疗期间扬声器嵌入的第一次分析。摘要：In this paper, we analyze the behavior of speaker embeddings of patients during oral cancer treatment. First, we found that pre- and post-treatment speaker embeddings differ significantly, notifying a substantial change in voice characteristics. However, a partial recovery to pre-operative voice traits is observed after 12 months post-operation. Secondly, the same-speaker similarity at distinct treatment stages is similar to healthy speakers, indicating that the embeddings can capture characterizing features of even severely impaired speech. Finally, a speaker verification analysis signifies a stable false positive rate and variable false negative rate when combining speech samples of different treatment stages. This indicates robustness of the embeddings towards other speakers, while still capturing the changing voice characteristics during treatment. To the best of our knowledge, this is the first analysis of speaker embeddings during oral cancer treatment of patients.
【2】 Timbre transfer using image-to-image denoising diffusion models标题：基于图像到图像去噪扩散模型的音色传递链接：https://arxiv.org/abs/2307.04586
作者：Luca Comanducci,Fabio Antonacci,Augusto Sarti机构：Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Italy摘要：音色转换技术旨在将由一种乐器产生的音乐作品的声音转换成如同由另一种乐器演奏一样的声音，同时尽可能地保持音乐特征的内容，例如旋律和力度。继他们最近在基于深度学习的生成方面取得突破之后，我们应用去噪扩散模型（DDM）来执行音色传输。具体而言，我们应用最近提出的去噪扩散隐式模型（DDIM），使加快采样过程。受最近的应用程序的DDMs的图像翻译问题，我们制定的音色转移任务类似，首先转换成对数梅尔声谱图的音频轨道，并通过调节所需的音色声谱图的生成，通过输入音色声谱图。我们执行一对一和多对多音色传输，通过转换仅包含单一乐器和多乐器的音频波形，分别。我们比较所提出的技术与现有的国家的最先进的方法，通过听力测试和客观的措施，以证明所提出的模型的有效性。摘要：Timbre transfer techniques aim at converting the sound of a musical piece generated by one instrument into the same one as if it was played by another instrument, while maintaining as much as possible the content in terms of musical characteristics such as melody and dynamics. Following their recent breakthroughs in deep learning-based generation, we apply Denoising Diffusion Models (DDMs) to perform timbre transfer. Specifically, we apply the recently proposed Denoising Diffusion Implicit Models (DDIMs) that enable to accelerate the sampling procedure. Inspired by the recent application of DDMs to image translation problems we formulate the timbre transfer task similarly, by first converting the audio tracks into log mel spectrograms and by conditioning the generation of the desired timbre spectrogram through the input timbre spectrogram. We perform both one-to-one and many-to-many timbre transfer, by converting audio waveforms containing only single instruments and multiple instruments, respectively. We compare the proposed technique with existing state-of-the-art methods both through listening tests and objective measures in order to demonstrate the effectiveness of the proposed model.
【3】 Study on the Correlation between Objective Evaluations and Subjective  Speech Quality and Intelligibility标题：客观评价与主观语音质量和清晰度的相关性研究链接：https://arxiv.org/abs/2307.04517
作者：Hsin-Tien Chiang,Kuo-Hsuan Hung,Szu-Wei Fu,Heng-Cheng Kuo,Ming-Hsueh Tsai,Yu Tsao机构：Academia Sinica,National Taiwan University,NVIDIA, National Academy for Educational Research摘要：主观测试是评价语音质量和可懂度的金标准，但它们耗时且昂贵。因此，符合人类感知的客观措施至关重要。本研究使用中文语音数据集评估常用的客观指标与主观语音质量和可懂度之间的相关性。此外，提出了新的客观指标，结合使用深度学习技术预测主观质量和可懂度的当前客观指标。所提出的深度学习模型减少了训练数据量，而不会显著影响预测性能。我们解释了深度学习模型，以了解客观指标如何反映主观质量和可理解性。我们还探讨了包括主观语音质量评级对语音清晰度预测的影响。我们的研究结果提供了宝贵的见解客观措施和人类的看法之间的关系。摘要：Subjective tests are the gold standard for evaluating speech quality and intelligibility, but they are time-consuming and expensive. Thus, objective measures that align with human perceptions are crucial. This study evaluates the correlation between commonly used objective measures and subjective speech quality and intelligibility using a Chinese speech dataset. Moreover, new objective measures are proposed combining current objective measures using deep learning techniques to predict subjective quality and intelligibility. The proposed deep learning model reduces the amount of training data without significantly impacting prediction performance. We interpret the deep learning model to understand how objective measures reflect subjective quality and intelligibility. We also explore the impact of including subjective speech quality ratings on speech intelligibility prediction. Our findings offer valuable insights into the relationship between objective measures and human perceptions.
【4】 Exploiting an External Microphone for Binaural RTF-Vector-Based  Direction of Arrival Estimation for Multiple Speakers标题：利用外置麦克风实现双耳RTF-基于矢量的多说话人波达方向估计链接：https://arxiv.org/abs/2307.04460
作者：Daniel Fejgin,Simon Doclo机构：University of Oldenburg, Department of Medical Physics and Acoustics, and Cluster of Excellence Hearing,all, Oldenburg, Germany备注：Paper accepted for Forum Acusticum 2023摘要：在助听器应用中，一个重要的目标是准确地估计多个扬声器在噪声和混响环境中的到达方向（DOA）。最近，我们提出了一种双耳波达方向估计方法，其中的DOA的扬声器估计通过选择的方向之间的估计的相对传递函数（RTF）向量和原型消声RTF向量的数据库的所谓的厄米角谱最大化。RTF向量估计使用协方差白化（CW）的方法，这需要一个计算复杂的广义特征值分解。通过仅考虑其中一个扬声器可能支配其他扬声器的频率、噪声和混响来获得空间频谱。在这方面的贡献，我们利用的外部麦克风，在空间上分离的助听器麦克风的可用性，并考虑一个低复杂度的RTF矢量估计方法，假设在外部麦克风和助听器麦克风的不希望的组件之间的低空间相干性。在混响较弱、信噪比较低的声环境中，利用两个扬声器的录音和弥散状的串音噪声，仿真结果表明，该方法在较低的计算复杂度下获得了与CW方法相当的DOA估计性能。摘要：In hearing aid applications, an important objective is to accurately estimate the direction of arrival (DOA) of multiple speakers in noisy and reverberant environments. Recently, we proposed a binaural DOA estimation method, where the DOAs of the speakers are estimated by selecting the directions for which the so-called Hermitian angle spectrum between the estimated relative transfer function (RTF) vector and a database of prototype anechoic RTF vectors is maximized. The RTF vector is estimated using the covariance whitening (CW) method, which requires a computationally complex generalized eigenvalue decomposition. The spatial spectrum is obtained by only considering frequencies where it is likely that one speaker dominates over the other speakers, noise and reverberation. In this contribution, we exploit the availability of an external microphone that is spatially separated from the hearing aid microphones and consider a low-complexity RTF vector estimation method that assumes a low spatial coherence between the undesired components in the external microphone and the hearing aid microphones. Using recordings of two speakers and diffuse-like babble noise in acoustic environments with mild reverberation and low signal-to-noise ratio, simulation results show that the proposed method yields a comparable DOA estimation performance as the CW method at a lower computational complexity.
【5】 A Demand-Driven Perspective on Generative Audio AI标题：需求驱动视角下的生成性音频人工智能链接：https://arxiv.org/abs/2307.04292
作者：Sangshin Oh,Minsung Kang,Hyeongi Moon,Keunwoo Choi,Ben Sangbae Chon备注：10 pages, 7 figures摘要：为了成功部署人工智能研究，了解行业的需求至关重要。在本文中，我们提出了一项与专业音频工程师进行的调查结果，以确定研究的优先级和定义各种研究任务。我们还总结了当前的挑战，在音频质量和可控性的调查的基础上。我们的分析强调，数据集的可用性是目前实现高质量音频生成的主要瓶颈。最后，我们提出了一些潜在的解决方案与经验证据揭示的问题。摘要：To achieve successful deployment of AI research, it is crucial to understand the demands of the industry. In this paper, we present the results of a survey conducted with professional audio engineers, in order to determine research priorities and define various research tasks. We also summarize the current challenges in audio quality and controllability based on the survey. Our analysis emphasizes that the availability of datasets is currently the main bottleneck for achieving high-quality audio generation. Finally, we suggest potential solutions for some revealed issues with empirical evidence.
【6】 IANS: Intelligibility-aware Null-steering Beamforming for  Dual-Microphone Arrays标题：IANS：用于双麦克风阵列的可理解性感知零控波束形成链接：https://arxiv.org/abs/2307.04179
作者：Wen-Yuan Ting,Syu-Siang Wang,Yu Tsao,Borching Su机构：⋆ Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan, † Department of Electrical Engineering, Yuan Ze University, Taoyuan, Taiwan备注：Preprint submitted to IEEE MLSP 2023摘要：波束形成技术由于其有效的空间滤波能力而在语音相关应用中流行。然而，传统的波束形成技术通常严重依赖于目标的到达方向（DOA），相对传递函数（RTF）或协方差矩阵。本文提出了一种新的方法，可懂度感知调零（IANS）波束形成框架，它使用的STOI-Net可懂度预测模型，以提高语音可懂度，而无需先前提到的语音信号参数的先验知识。IANS框架结合了调零波束形成器（NSBF）以生成一组波束形成输出，以及ST 0 I-Net，以确定最佳结果。实验结果表明，IANS可以使用一个小的双麦克风阵列产生可懂度增强的信号。的结果是可比的零转向波束形成器与给定的知识的DOA。摘要：Beamforming techniques are popular in speech-related applications due to their effective spatial filtering capabilities. Nonetheless, conventional beamforming techniques generally depend heavily on either the target's direction-of-arrival (DOA), relative transfer function (RTF) or covariance matrix. This paper presents a new approach, the intelligibility-aware null-steering (IANS) beamforming framework, which uses the STOI-Net intelligibility prediction model to improve speech intelligibility without prior knowledge of the speech signal parameters mentioned earlier. The IANS framework combines a null-steering beamformer (NSBF) to generate a set of beamformed outputs, and STOI-Net, to determine the optimal result. Experimental results indicate that IANS can produce intelligibility-enhanced signals using a small dual-microphone array. The results are comparable to those obtained by null-steering beamformers with given knowledge of DOAs.
【7】 On decoder-only architecture for speech-to-text and large language model  integration标题：用于语音到文本和大语言模型集成的仅解码器架构链接：https://arxiv.org/abs/2307.03917
作者：Jian Wu,Yashesh Gaur,Zhuo Chen,Long Zhou,Yimeng Zhu,Tianrui Wang,Jinyu Li,Shujie Liu,Bo Ren,Linquan Liu,Yu Wu机构：Microsoft, One Microsoft Way, Redmond, USA摘要：大型语言模型（LLM）在自然语言处理领域取得了显著的成功，使得使用自然语言进行更好的人机交互成为可能。然而，语音信号到LLM的无缝集成还没有被很好地探索。对于语音处理任务，“仅解码器”架构也没有得到很好的研究。在这项研究中，我们介绍了语音LLaMA，一种新的方法，有效地将声学信息到基于文本的大型语言模型。我们的方法利用连接时间分类和一个简单的音频编码器映射压缩的声学特征的连续语义空间的LLM。此外，我们进一步探索解码器的语音到文本的任务，通过训练一个较小规模的随机初始化的语音LLaMA模型，从语音文本配对数据单独的架构。我们对多语言语音到文本翻译任务进行了实验，并在强基线上表现出显着的改进，突出了仅解码器模型用于语音到文本转换的潜在优势。摘要：Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language. However, the seamless integration of speech signals into LLMs has not been explored well. The "decoder-only" architecture has also not been well studied for speech processing tasks. In this research, we introduce Speech-LLaMA, a novel approach that effectively incorporates acoustic information into text-based large language models. Our method leverages Connectionist Temporal Classification and a simple audio encoder to map the compressed acoustic features to the continuous semantic space of the LLM. In addition, we further probe the decoder-only architecture for speech-to-text tasks by training a smaller scale randomly initialized speech-LLaMA model from speech-text paired data alone. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines, highlighting the potential advantages of decoder-only models for speech-to-text conversion.
【8】 Performance Comparison of Pre-trained Models for Speech-to-Text in  Turkish: Whisper-Small and Wav2Vec2-XLS-R-300M标题：用于土耳其语语音转文本的预训练模型的性能比较：Whisper-Small和Wav2Vec2-XLS-R-300M链接：https://arxiv.org/abs/2307.04765
作者：Oyku Berfin Mercan,Sercan Cepni,Davut Emre Tasar,Sukru Ozan备注：10 Pages, in Turkish language, Pre-Print摘要：在这项研究中，针对土耳其语检查了Whisper-Small和Wav 2 Vec 2-XLS-R-300 M模型的性能，这两个模型是两个预先训练的语音到文本的多语言模型。Mozilla Common Voice 11.0版是用土耳其语编写的，是一个开源数据集，用于研究。多语言模型Whisper- Small和Wav 2 Vec 2-XLS-R-300 M使用包含少量数据的该数据集进行微调。比较了两种模型的语音到文本的性能。Wav 2 Vec 2-XLS-R-300 M和Whisper-Small模型的WER值分别计算为0.28和0.16。此外，使用由呼叫中心记录准备的测试数据检查了模型的性能，这些数据未包括在训练和验证数据集中。摘要：In this study, the performances of the Whisper-Small and Wav2Vec2-XLS-R-300M models which are two pre-trained multilingual models for speech to text were examined for the Turkish language. Mozilla Common Voice version 11.0 which is prepared in Turkish language and is an open-source data set, was used in the study. The multilingual models, Whisper- Small and Wav2Vec2-XLS-R-300M were fine-tuned with this data set which contains a small amount of data. The speech to text performance of the two models was compared. WER values are calculated as 0.28 and 0.16 for the Wav2Vec2-XLS- R-300M and the Whisper-Small models respectively. In addition, the performances of the models were examined with the test data prepared with call center records that were not included in the training and validation dataset.
【9】 Learning Spatial Features from Audio-Visual Correspondence in Egocentric  Videos标题：从以自我为中心的视频的视听对应中学习空间特征链接：https://arxiv.org/abs/2307.04760
作者：Sagnik Majumder,Ziad Al-Halah,Kristen Grauman机构：University of Texas at Austin, FAIR, University of Utah摘要：我们提出了一个自我监督的方法，学习表示的基础上，在以自我为中心的视频空间视听对应。特别地，我们的方法利用掩蔽的自动编码框架来通过音频和视觉的协同作用合成掩蔽的双耳音频，从而学习两种模态之间的有用的空间关系。我们使用预训练的特征来处理两个在社交场景中需要空间理解的下游视频任务：活动说话人检测和空间音频去噪。我们通过大量的实验表明，我们的功能是通用的，足以改善多个国家的最先进的基线上两个公共的具有挑战性的以自我为中心的视频数据集，EgoCom和EasyCom。项目：http://vision.cs.utexas.edu/projects/ego_av_corr。摘要：We propose a self-supervised method for learning representations based on spatial audio-visual correspondences in egocentric videos. In particular, our method leverages a masked auto-encoding framework to synthesize masked binaural audio through the synergy of audio and vision, thereby learning useful spatial relationships between the two modalities. We use our pretrained features to tackle two downstream video tasks requiring spatial understanding in social scenarios: active speaker detection and spatial audio denoising. We show through extensive experiments that our features are generic enough to improve over multiple state-of-the-art baselines on two public challenging egocentric video datasets, EgoCom and EasyCom. Project: http://vision.cs.utexas.edu/projects/ego_av_corr.
【10】 Vocal Tract Area Estimation by Gradient Descent标题：用梯度下降法估计声道面积链接：https://arxiv.org/abs/2307.04702
作者：David Südholt,Mateo Cámara,Zhiyuan Xu,Joshua D. Reiss机构：Centre for Digital Music, Queen Mary University of London, London, UK, Information Processing & Telecomm. Center, Universidad Politécnica de Madrid, Madrid, Spain备注：Accepted to DAFx 2023摘要：发音特征可以通过允许用户直接修改诸如声音应变或嘴唇位置的参数来提供用于人类发声的合成的可解释的和灵活的控制。为了通过再合成进行这种操作，我们需要估计直接从音频记录中产生所需发声的特征。在这项工作中，我们提出了一个白盒优化技术估计声门源参数和声道形状从人类元音的音频记录。该方法是基于逆滤波和优化的波导模型的频率响应梯度下降，传播误差梯度通过映射的发音特征到声道面积函数.我们将这种方法应用到匹配的粉红长号，一个交互式发音合成器，一个给定的发声的声音的任务。我们发现，我们的方法准确地恢复粉红长号本身产生的音频控制功能。然后，我们比较我们的技术对进化优化算法和神经网络训练预测控制参数的音频。主观评价发现，我们的方法优于这些黑盒优化基线的任务，再现人类发声。摘要：Articulatory features can provide interpretable and flexible controls for the synthesis of human vocalizations by allowing the user to directly modify parameters like vocal strain or lip position. To make this manipulation through resynthesis possible, we need to estimate the features that result in a desired vocalization directly from audio recordings. In this work, we propose a white-box optimization technique for estimating glottal source parameters and vocal tract shapes from audio recordings of human vowels. The approach is based on inverse filtering and optimizing the frequency response of a wave\-guide model of the vocal tract with gradient descent, propagating error gradients through the mapping of articulatory features to the vocal tract area function. We apply this method to the task of matching the sound of the Pink Trombone, an interactive articulatory synthesizer, to a given vocalization. We find that our method accurately recovers control functions for audio generated by the Pink Trombone itself. We then compare our technique against evolutionary optimization algorithms and a neural network trained to predict control parameters from audio. A subjective evaluation finds that our approach outperforms these black-box optimization baselines on the task of reproducing human vocalizations.
【11】 VampNet: Music Generation via Masked Acoustic Token Modeling标题：VampNet：基于掩蔽声学符号建模的音乐生成链接：https://arxiv.org/abs/2307.04686
作者：Hugo Flores Garcia,Prem Seetharaman,Rithesh Kumar,Bryan Pardo机构：Descript Inc.,  Northwestern University摘要：我们介绍VampNet，一个掩蔽的声学令牌建模方法的音乐合成，压缩，修复和变化。我们在训练过程中使用可变的掩蔽时间表，这允许我们通过在推理过程中应用各种掩蔽方法（称为提示）来从模型中采样连贯的音乐。VampNet是非自回归的，利用双向Transformer架构，在正向传递中处理所有令牌。VampNet只需36次采样，就可以生成连贯的高保真音乐波形。我们表明，通过以各种方式提示VampNet，我们可以将其应用于音乐压缩，修复，outpainting，延续和循环变化（vamping）等任务。适当地提示，VampNet能够保持风格，流派，乐器和音乐的其他高级方面。这种灵活的提示功能使VampNet成为一个强大的音乐共同创作工具。代码和音频示例可在线获得。摘要：We introduce VampNet, a masked acoustic token modeling approach to music synthesis, compression, inpainting, and variation. We use a variable masking schedule during training which allows us to sample coherent music from the model by applying a variety of masking approaches (called prompts) during inference. VampNet is non-autoregressive, leveraging a bidirectional transformer architecture that attends to all tokens in a forward pass. With just 36 sampling passes, VampNet can generate coherent high-fidelity musical waveforms. We show that by prompting VampNet in various ways, we can apply it to tasks like music compression, inpainting, outpainting, continuation, and looping with variation (vamping). Appropriately prompted, VampNet is capable of maintaining style, genre, instrumentation, and other high-level aspects of the music. This flexible prompting capability makes VampNet a powerful music co-creation tool. Code and audio samples are available online.
【12】 The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023  Speech-to-Speech Translation Task标题：面向IWSLT 2023语音翻译任务的NPU-MSXF语音翻译系统链接：https://arxiv.org/abs/2307.04630
作者：Kun Song,Yi lei,Peikun Chen,Yiqing Cao,Kun Wei,Yongmao Zhang,Lei Xie,Ning Jiang,Guoqing Zhao机构：School of Computer Science, Northwestern Polytechnical University, China, Department of Computer Science and Technology, Nanjing University, China, MaShang Consumer Finance Co., Ltd, China备注：IWSLT@ACL 2023 system paper. Our submitted system ranks 1st in the S2ST task of the IWSLT 2023 evaluation campaign摘要：本文描述了用于IWSLT 2023语音到语音翻译（S2 ST）任务的NPU-MSXF系统，该任务旨在将多源英语语音翻译为汉语语音。该系统是建立在一个级联的方式，包括自动语音识别（ASR），机器翻译（MT），文本到语音（TTS）。我们付出了巨大的努力来处理具有挑战性的多源输入。具体来说，为了提高对多源语音输入的鲁棒性，我们采用了各种数据增强策略和基于ROVER的多个ASR模型输出的分数融合。为了更好地处理嘈杂的ASR成绩单，我们引入了一个三阶段的微调策略，以提高翻译的准确性。最后，我们建立了一个高自然度和音质的TTS模型，它利用了一个两阶段的框架，使用网络瓶颈功能作为一个强大的中间表示说话人音色和语言内容解纠缠。基于两阶段的框架，预先训练的说话人嵌入作为一个条件，将源英语语音中的说话人音色转换为翻译的中文语音。实验结果表明，我们的系统具有较高的翻译准确率，语音自然度，声音质量，说话人相似度。此外，它显示出良好的鲁棒性多源数据。摘要：This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech. The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous efforts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER-based score fusion on multiple ASR model outputs. To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity. Moreover, it shows good robustness to multi-source data.
【13】 EchoVest: Real-Time Sound Classification and Depth Perception Expressed  through Transcutaneous Electrical Nerve Stimulation标题：EchoVest：通过经皮神经电刺激表达的实时声音分类和深度感知链接：https://arxiv.org/abs/2307.04604
作者：Jesse Choe,Siddhant Sood,Ryan Park摘要：全世界有超过15亿人患有听力障碍。尽管已经为这些残疾人创造了各种技术，但这些技术中的大多数要么极其昂贵，要么在中低收入国家无法日常使用。为了解决这个问题，我们开发了一种新的辅助设备，EchoVest，让盲人/聋人直观地更加了解他们的环境。EchoVest通过利用基于声源的经皮电神经刺激（TENS）将振动传输到用户的身体。EchoVest还提供各种功能，包括声音定位，声音分类，降噪和深度感知。我们的目标是在准确性和计算成本方面优于基于CNN的机器学习模型，这是分类任务中最常用的机器学习模型。为此，我们开发并采用了一种新的音频流水线，该流水线适应音频频谱图Transformer（AST）模型，这是一种基于注意力的模型，用于我们的声音分类目的，以及用于降噪的快速傅立叶变换。大津法的应用，帮助我们找到最佳阈值的背景噪声的声音过滤，给我们更大的准确性。为了准确地计算方向和深度，我们应用复到达时间差算法和SOTA定位。我们最后的改进是使用盲源分离，使我们的算法适用于多个麦克风输入。最终的算法在许多检查点上取得了最先进的结果，包括在ESC-50数据集上进行环境声音分类的95.7%的准确率。摘要：Over 1.5 billion people worldwide live with hearing impairment. Despite various technologies that have been created for individuals with such disabilities, most of these technologies are either extremely expensive or inaccessible for everyday use in low-medium income countries. In order to combat this issue, we have developed a new assistive device, EchoVest, for blind/deaf people to intuitively become more aware of their environment. EchoVest transmits vibrations to the user's body by utilizing transcutaneous electric nerve stimulation (TENS) based on the source of the sounds. EchoVest also provides various features, including sound localization, sound classification, noise reduction, and depth perception. We aimed to outperform CNN-based machine-learning models, the most commonly used machine learning model for classification tasks, in accuracy and computational costs. To do so, we developed and employed a novel audio pipeline that adapts the Audio Spectrogram Transformer (AST) model, an attention-based model, for our sound classification purposes, and Fast Fourier Transforms for noise reduction. The application of Otsu's Method helped us find the optimal thresholds for background noise sound filtering and gave us much greater accuracy. In order to calculate direction and depth accurately, we applied Complex Time Difference of Arrival algorithms and SOTA localization. Our last improvement was to use blind source separation to make our algorithms applicable to multiple microphone inputs. The final algorithm achieved state-of-the-art results on numerous checkpoints, including a 95.7\% accuracy on the ESC-50 dataset for environmental sound classification.
【14】 Read, Look or Listen? What's Needed for Solving a Multimodal Dataset标题：读、看还是听？求解多模式数据集需要什么链接：https://arxiv.org/abs/2307.04532
作者：Netta Madvil,Yonatan Bitton,Roy Schwartz机构：School of Computer Science, The Hebrew University, Jerusalem, Israel摘要：大规模多模态数据集的流行在评估数据集质量方面提出了独特的挑战。我们提出了一个两步的方法来分析多模态数据集，它利用一个小种子的人类注释映射到每个多模态实例所需的处理方式。我们的方法揭示了数据集中不同模态的重要性，以及它们之间的关系。我们将我们的方法应用到TVQA，视频问答数据集，并发现大多数问题可以使用一个单一的模态来回答，没有对任何特定模态的实质性偏见。此外，我们发现超过70%的问题可以使用几种不同的单模态策略来解决，例如，通过观看视频或收听音频，突出TVQA中多种模态的有限集成。我们利用我们的注释和分析的MERLOT储备，发现它的斗争与基于图像的问题相比，文本和音频，但也与听觉扬声器识别。根据我们的观察，我们引入了一个新的测试集，需要多种方式，观察模型性能急剧下降。我们的方法提供了有价值的见解多模态数据集，并强调了开发更强大的模型的需要。摘要：The prevalence of large-scale multimodal datasets presents unique challenges in assessing dataset quality. We propose a two-step method to analyze multimodal datasets, which leverages a small seed of human annotation to map each multimodal instance to the modalities required to process it. Our method sheds light on the importance of different modalities in datasets, as well as the relationship between them. We apply our approach to TVQA, a video question-answering dataset, and discover that most questions can be answered using a single modality, without a substantial bias towards any specific modality. Moreover, we find that more than 70% of the questions are solvable using several different single-modality strategies, e.g., by either looking at the video or listening to the audio, highlighting the limited integration of multiple modalities in TVQA. We leverage our annotation and analyze the MERLOT Reserve, finding that it struggles with image-based questions compared to text and audio, but also with auditory speaker identification. Based on our observations, we introduce a new test set that necessitates multiple modalities, observing a dramatic drop in model performance. Our methodology provides valuable insights into multimodal datasets and highlights the need for the development of more robust models.
【15】 HCLAS-X: Hierarchical and Cascaded Lyrics Alignment System Using  Multimodal Cross-Correlation标题：HCLAS-X：基于多模互相关的层次化级联歌词对齐系统链接：https://arxiv.org/abs/2307.04377
作者：Minsung Kang,Soochul Park,Keunwoo Choi机构：Gaudio Lab, Inc.摘要：在这项工作中，我们解决歌词对齐的挑战，这涉及到对齐歌曲的歌词和声乐成分。这个问题需要两种不同的模态，即文本和音频的对齐。为了克服这一挑战，我们提出了一个模型，在监督的方式进行训练，利用人声和歌词之间的潜在表示的互相关矩阵。我们的系统设计在一个层次和级联的方式。它首先在句子级别上预测同步时间，然后在单词级别上预测同步时间。这种设计使得系统能够处理长序列，因为互相关使用关于序列长度的二次存储器。在我们的实验中，我们证明，我们提出的系统实现了显着的改善，平均平均误差，展示其鲁棒性相比，以前的国家的最先进的模型。此外，我们成功地将其部署在几个音乐流媒体服务后，系统进行了定性分析。摘要：In this work, we address the challenge of lyrics alignment, which involves aligning the lyrics and vocal components of songs. This problem requires the alignment of two distinct modalities, namely text and audio. To overcome this challenge, we propose a model that is trained in a supervised manner, utilizing the cross-correlation matrix of latent representations between vocals and lyrics. Our system is designed in a hierarchical and cascaded manner. It predicts synced time first on a sentence-level and subsequently on a word-level. This design enables the system to process long sequences, as the cross-correlation uses quadratic memory with respect to sequence length. In our experiments, we demonstrate that our proposed system achieves a significant improvement in mean average error, showcasing its robustness in comparison to the previous state-of-the-art model. Additionally, we conduct a qualitative analysis of the system after successfully deploying it in several music streaming services.
【16】 Automatic Piano Transcription with Hierarchical Frequency-Time  Transformer标题：采用分级频率-时间转换器的自动钢琴改编链接：https://arxiv.org/abs/2307.04305
作者：Keisuke Toyama,Taketo Akama,Yukara Ikemiya,Yuhta Takida,Wei-Hsiang Liao,Yuki Mitsufuji机构：Sony Group Corporation, Japan,  Sony Computer Science Laboratories, Japan备注：8 pages, 6 figures, to be published in ISMIR2023摘要：考虑到长期的光谱和时间依赖性是钢琴自动转录的必要条件。这在确定复调钢琴内容中的每个音符的精确起始和偏移时特别有帮助。在这种情况下，我们可以依靠Transformers中的自我注意机制的能力来捕获频率和时间轴上的这些长期依赖关系。在这项工作中，我们提出了hFT-Transformer，这是一种自动音乐转录的方法，使用了两级分层频率-时间Transformer架构。第一层次包括时间轴上的卷积块、频率轴上的Transformer编码器和频率轴上的变换器解码器。然后，输出被馈送到第二层次，该层次由时间轴上的另一个Transformer编码器组成。我们使用广泛使用的MAPS和MAESTRO v3.0.0数据集评估了我们的方法，它展示了帧、音符、带偏移的音符以及带偏移和速度估计中所有度量的F1分数的最新性能。摘要：Taking long-term spectral and temporal dependencies into account is essential for automatic piano transcription. This is especially helpful when determining the precise onset and offset for each note in the polyphonic piano content. In this case, we may rely on the capability of self-attention mechanism in Transformers to capture these long-term dependencies in the frequency and time axes. In this work, we propose hFT-Transformer, which is an automatic music transcription method that uses a two-level hierarchical frequency-time Transformer architecture. The first hierarchy includes a convolutional block in the time axis, a Transformer encoder in the frequency axis, and a Transformer decoder that converts the dimension in the frequency axis. The output is then fed into the second hierarchy which consists of another Transformer encoder in the time axis. We evaluated our method with the widely used MAPS and MAESTRO v3.0.0 datasets, and it demonstrated state-of-the-art performance on all the F1-scores of the metrics among Frame, Note, Note with Offset, and Note with Offset and Velocity estimations.
【17】 Edge Storage Management Recipe with Zero-Shot Data Compression for Road  Anomaly Detection标题：基于零点数据压缩的道路异常检测边缘存储管理方案链接：https://arxiv.org/abs/2307.04298
作者：YeongHyeon Park,Uju Gim,Myung Jin Kim机构：SK Planet Co., Ltd., Seongnam, Rep. of Korea备注：5 pages, 3 figures, 4 tables摘要：最近的研究表明，基于边缘计算的道路异常检测系统，也可以同时进行数据收集。然而，边缘计算机的数据存储量很小，但我们需要长时间存储收集的音频样本，以便更新现有模型或开发新方法。因此，我们应该考虑一种高效的存储管理方法，同时保留高保真音频。从硬件角度看的方法（例如使用低分辨率麦克风）是减少文件大小的直观方法，但不建议使用，因为它从根本上切断了高频分量。另一方面，应推荐将收集的高分辨率音频编码为紧凑代码的计算文件压缩方法，因为它还提供了相应的解码方法。受此启发，我们提出了一种简单而有效的基于预训练自动编码器的数据压缩方法。预训练的自动编码器是为了音频超分辨率的目的而训练的，因此它可以用于编码或解码任何任意采样率。此外，它将降低从边缘到中央服务器的数据传输的通信成本。通过对比实验，我们确认zero-shot音频压缩和解压缩高度保留异常检测性能，同时提高存储和传输效率。摘要：Recent studies show edge computing-based road anomaly detection systems which may also conduct data collection simultaneously. However, the edge computers will have small data storage but we need to store the collected audio samples for a long time in order to update existing models or develop a novel method. Therefore, we should consider an approach for efficient storage management methods while preserving high-fidelity audio. A hardware-perspective approach, such as using a low-resolution microphone, is an intuitive way to reduce file size but is not recommended because it fundamentally cuts off high-frequency components. On the other hand, a computational file compression approach that encodes collected high-resolution audio into a compact code should be recommended because it also provides a corresponding decoding method. Motivated by this, we propose a way of simple yet effective pre-trained autoencoder-based data compression method. The pre-trained autoencoder is trained for the purpose of audio super-resolution so it can be utilized to encode or decode any arbitrary sampling rate. Moreover, it will reduce the communication cost for data transmission from the edge to the central server. Via the comparative experiments, we confirm that the zero-shot audio compression and decompression highly preserve anomaly detection performance while enhancing storage and transmission efficiency.
【18】 Can Generative Large Language Models Perform ASR Error Correction?标题：生成性大型语言模型可以执行ASR纠错吗？链接：https://arxiv.org/abs/2307.04172
作者：Rao Ma,Mengjie Qian,Potsawee Manakul,Mark Gales,Kate Knill机构：ALTA Institute, Machine Intelligence Lab, Department of Engineering, Cambridge University, UK摘要：ASR纠错继续充当语音识别系统的后处理的重要部分。传统上，这些模型使用底层ASR系统的解码结果和参考文本进行监督训练。这种方法是计算密集型的，并且在切换底层ASR模型时需要重新训练模型。近年来，大型语言模型及其以zero-shot方式执行自然语言处理任务的能力得到了发展。在本文中，我们以ChatGPT为例，以检查其执行ASR错误校正的能力，在zero-shot或1杆设置。我们使用ASR的N-最好的列表作为模型输入，并提出无约束的错误校正和N-最好的约束错误校正方法。Conformer-Transducer模型和预训练的Whisper模型的结果表明，我们可以使用强大的ChatGPT模型通过纠错大大提高ASR系统的性能。摘要：ASR error correction continues to serve as an important part of post-processing for speech recognition systems. Traditionally, these models are trained with supervised training using the decoding results of the underlying ASR system and the reference text. This approach is computationally intensive and the model needs to be re-trained when switching the underlying ASR model. Recent years have seen the development of large language models and their ability to perform natural language processing tasks in a zero-shot manner. In this paper, we take ChatGPT as an example to examine its ability to perform ASR error correction in the zero-shot or 1-shot settings. We use the ASR N-best list as model input and propose unconstrained error correction and N-best constrained error correction methods. Results on a Conformer-Transducer model and the pre-trained Whisper model show that we can largely improve the ASR system performance with error correction using the powerful ChatGPT model.
【19】 Emotion-Guided Music Accompaniment Generation Based on Variational  Autoencoder标题：基于变分自动编码器的情感制导音乐伴奏生成链接：https://arxiv.org/abs/2307.04015
作者：Qi Wang,Shubing Zhang,Li Zhou机构：China University of Geosciences(Wuhan)备注：Accepted By International Joint Conference on Neural Networks 2023(IJCNN2023)摘要：音乐伴奏的生成是作曲过程中一个至关重要的环节。深度神经网络在这一领域取得了重大进展，但人工智能如何有效地整合人类情感以创造美丽的伴奏仍然是一个挑战。现有模型难以在作曲时有效地表征神经网络模型中的人类情感。为了解决这个问题，我们提出了使用一个易于表示的情绪流模型，价/唤醒曲线，它允许的兼容性的情绪信息的模型内的数据转换和提高情绪因素的解释性，通过利用变分自动编码器的模型结构。此外，我们使用相对自我注意来维持音乐在乐句层次的结构，并结合乐理规则产生更丰富的伴奏。摘要：Music accompaniment generation is a crucial aspect in the composition process. Deep neural networks have made significant strides in this field, but it remains a challenge for AI to effectively incorporate human emotions to create beautiful accompaniments. Existing models struggle to effectively characterize human emotions within neural network models while composing music. To address this issue, we propose the use of an easy-to-represent emotion flow model, the Valence/Arousal Curve, which allows for the compatibility of emotional information within the model through data transformation and enhances interpretability of emotional factors by utilizing a Variational Autoencoder as the model structure. Further, we used relative self-attention to maintain the structure of the music at music phrase level and to generate a richer accompaniment when combined with the rules of music theory.
机器翻译，仅供参考