今日论文合集：cs.SD语音7篇，eess.AS音频处理11篇。cs.SD语音【1】 M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis标题：M2-CTTS：端到端多尺度多模式对话式文语合成链接：https://arxiv.org/abs/2305.02269作者：Jinlong Xue,Yayue Deng,Fengping Wang,Ya Li,Yingming Gao,Jianhua Tao,Jianqing Sun,Jiaen Liang机构：Beijing University of Posts and Telecommunications, Beijing, China, NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China, Unisound AI Technology Co., Ltd, Beijing, China备注：5 pages, 1 figures, 2 tables. Accepted by ICASSP 2023摘要：会话式文语转换（TTS）的目的是根据历史会话合成具有适当韵律的回复语音。然而，它仍然是一个挑战，全面的对话模型，和大多数的会话TTS系统只专注于提取全球信息和省略本地韵律特征，其中包含重要的细粒度信息，如关键字和重点。而且，仅仅考虑文本特征是不够的，声学特征还包含了各种韵律信息。因此，我们提出了M2-CTTS，一个端到端的多尺度多模态会话的文本到语音系统，旨在综合利用历史会话和增强韵律的表达。更具体地说，我们设计了一个文本上下文模块和一个声学上下文模块的粗粒度和细粒度建模。实验结果表明，我们的模型与细粒度的上下文信息混合，并考虑额外的声学特征，在CMOS测试中实现了更好的韵律性能和自然度。摘要：Conversational text-to-speech (TTS) aims to synthesize speech with proper prosody of reply based on the historical conversation. However, it is still a challenge to comprehensively model the conversation, and a majority of conversational TTS systems only focus on extracting global information and omit local prosody features, which contain important fine-grained information like keywords and emphasis. Moreover, it is insufficient to only consider the textual features, and acoustic features also contain various prosody information. Hence, we propose M2-CTTS, an end-to-end multi-scale multi-modal conversational text-to-speech system, aiming to comprehensively utilize historical conversation and enhance prosodic expression. More specifically, we design a textual context module and an acoustic context module with both coarse-grained and fine-grained modeling. Experimental results demonstrate that our model mixed with fine-grained context information and additionally considering acoustic features achieves better prosody performance and naturalness in CMOS tests.【2】 Considerations for Ethical Speech Recognition Datasets标题：伦理语音识别数据集的考虑因素链接：https://arxiv.org/abs/2305.02081作者：Orestis Papakyriakopoulos,Alice Xiang机构：Sony AI, Switzerland, USA备注：None摘要：语音人工智能技术主要是在公开可用的数据集上进行训练，或者通过大规模的语音网络爬行进行训练。在这两种情况下，数据采集的重点都是尽量减少收集工作，而不一定考虑数据主体的保护或用户需求。这导致模型在用于偏离训练集中的主要人口统计的用户时不稳健，区分具有不同方言、口音、说话风格和不流利的个体。在这次演讲中，我们使用自动语音识别作为案例研究，并研究道德语音数据集应该具备的属性，以实现负责任的人工智能应用。我们展示了多样性问题、包容性实践以及可以改进训练模型的必要考虑因素，同时促进模型的可解释性并保护用户和数据主体。我们主张数据主体的法律和隐私保护，与用户人口统计和需求相对应的有针对性的数据采样，适当的元数据，以确保在模型失败的情况下的可解释性和问责制，以及社会技术 和情境模型设计。我们希望这次演讲能够激励研究人员和从业者在语音技术和其他领域设计和使用更多以人为中心的数据集，以授权和尊重用户的方式，同时提高机器学习模型的鲁棒性和实用性。摘要：Speech AI Technologies are largely trained on publicly available datasets or by the massive web-crawling of speech. In both cases, data acquisition focuses on minimizing collection effort, without necessarily taking the data subjects' protection or user needs into consideration. This results to models that are not robust when used on users who deviate from the dominant demographics in the training set, discriminating individuals having different dialects, accents, speaking styles, and disfluencies. In this talk, we use automatic speech recognition as a case study and examine the properties that ethical speech datasets should possess towards responsible AI applications. We showcase diversity issues, inclusion practices, and necessary considerations that can improve trained models, while facilitating model explainability and protecting users and data subjects. We argue for the legal & privacy protection of data subjects, targeted data sampling corresponding to user demographics & needs, appropriate meta data that ensure explainability & accountability in cases of model failure, and the sociotechnical & situated model design. We hope this talk can inspire researchers & practitioners to design and use more human-centric datasets in speech technologies and other domains, in ways that empower and respect users, while improving machine learning models' robustness and utility.【3】 Diverse and Vivid Sound Generation from Text Descriptions标题：从文本描述中生成多样化和生动的声音链接：https://arxiv.org/abs/2305.01980作者：Guangwei Li,Xuenan Xu,Lingfeng Dai,Mengyue Wu,Kai Yu机构：X-LANCE Lab, Department of Computer Science and Engineering, MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China摘要：以往的音频生成主要集中在语音或音乐等特定的声音类别，其形式和内容受到很大的限制。在本文中，我们超越了特定的音频生成，使用自然语言描述为线索，产生广泛的声音。与视觉信息不同，文本描述本质上是简洁的，但其下隐藏着丰富的含义，这对要生成的音频提出了更高的可能性和复杂性。变异量化GAN用于训练学习频谱图的离散表示的码本。对于给定的文本描述，其预训练的嵌入被馈送到Transformer以对码本索引进行采样，以解码频谱图，以由melgan声码器进一步变换成波形。所生成的波形具有高质量和保真度，同时与给定的文本良好地对应。实验表明，我们提出的方法是能够产生自然，生动的音频，取得了极好的定量和定性的结果。摘要：Previous audio generation mainly focuses on specified sound classes such as speech or music, whose form and content are greatly restricted. In this paper, we go beyond specific audio generation by using natural language description as a clue to generate broad sounds. Unlike visual information, a text description is concise by its nature but has rich hidden meanings beneath, which poses a higher possibility and complexity on the audio to be generated. A Variation-Quantized GAN is used to train a codebook learning discrete representations of spectrograms. For a given text description, its pre-trained embedding is fed to a Transformer to sample codebook indices to decode a spectrogram to be further transformed into waveform by a melgan vocoder. The generated waveform has high quality and fidelity while excellently corresponding to the given text. Experiments show that our proposed method is capable of generating natural, vivid audios, achieving superb quantitative and qualitative results.【4】 Analysing the Impact of Audio Quality on the Use of Naturalistic Long-Form Recordings for Infant-Directed Speech Research标题：分析音频质量对使用自然长格式录音进行婴儿定向言语研究的影响链接：https://arxiv.org/abs/2305.01965作者：María Andrea Cruz Blandón,Alejandrina Cristia,Okko Räsänen机构：D´epartement d’´etudes cognitives, ENS, EHESS, CNRS, PSL University, rue d’Ulm, Paris, France, Unit of Computing Sciences, Tampere University, P.O. Box , FI-, Finland备注：Accepted at the CogSci2023 conference摘要：早期语言习得模型旨在了解婴儿如何引导他们的语言技能。建模包括用于训练模型的输入数据的属性、被测试的认知假设及其算法实现，以及将模型与人类数据进行比较的评估方法。最近的发展使得能够使用更自然的训练数据的计算模型。这也激发了对模型行为的更自然的测试的发展。实现这一目标的关键一步是开发代表性的语音数据集，包括婴儿在自然环境中听到的语音。然而，这种记录的主要缺点是它们通常是嘈杂的，并且目前不清楚声音质量如何影响对这种数据进行的分析和建模实验。在本文中，我们探讨这方面的情况下，婴儿导向的语音（IDS）和成人导向的语音（ADS）分析。首先，我们手动和自动注释的音频质量的话语提取两个语料库的儿童为中心的长形式的录音（英语和法语）。然后，我们比较了IDS和ADS在实验室数据集中的声学特征，以及自然数据的不同音频质量子集。最后，我们评估了音频质量和录音环境如何改变使用最近的自监督学习模型的建模分析的结论。我们的研究结果表明，使用适度和高音频质量的自然语音数据的结果在很大程度上类似的结论IDS和ADS的声学分析和建模实验。我们还发现，一个自动的声音质量评估工具，可以用来筛选出有用的部分，长形式的录音进行更仔细的分析，具有可比的结果，手动质量注释。摘要：Modelling of early language acquisition aims to understand how infants bootstrap their language skills. The modelling encompasses properties of the input data used for training the models, the cognitive hypotheses and their algorithmic implementations being tested, and the evaluation methodologies to compare models to human data. Recent developments have enabled the use of more naturalistic training data for computational models. This also motivates development of more naturalistic tests of model behaviour. A crucial step towards such an aim is to develop representative speech datasets consisting of speech heard by infants in their natural environments. However, a major drawback of such recordings is that they are typically noisy, and it is currently unclear how the sound quality could affect analyses and modelling experiments conducted on such data. In this paper, we explore this aspect for the case of infant-directed speech (IDS) and adult-directed speech (ADS) analysis. First, we manually and automatically annotated audio quality of utterances extracted from two corpora of child-centred long-form recordings (in English and French). We then compared acoustic features of IDS and ADS in an in-lab dataset and across different audio quality subsets of naturalistic data. Finally, we assessed how the audio quality and recording environment may change the conclusions of a modelling analysis using a recent self-supervised learning model. Our results show that the use of modest and high audio quality naturalistic speech data result in largely similar conclusions on IDS and ADS in terms of acoustic analyses and modelling experiments. We also found that an automatic sound quality assessment tool can be used to screen out useful parts of long-form recordings for a closer analysis with comparable results to that of manual quality annotation.【5】 Unsupervised Improvement of Audio-Text Cross-Modal Representations标题：音频-文本跨通道表示的无监督改进链接：https://arxiv.org/abs/2305.01864作者：Zhepei Wang,Cem Subakan,Krishna Subramani,Junkai Wu,Tiago Tavares,Fabio Ayres,Paris Smaragdis机构：♯University of Illinois at Urbana-Champaign, ♭Universit´e Laval, ♮Concordia University, Mila-Quebec AI Institute, †Insper, αAmazon Web Services备注：Submitted to WASPAA 2023摘要：使用语言模型来获得跨模态音频文本表示的最新进展已经克服了使用预定义标签的传统训练方法的局限性。这使社区能够在zero-shot分类等任务中取得进展，否则这是不可能的。然而，学习这样的表示需要大量的人类注释的音频文本对。在本文中，我们研究了无监督的方法，以改善这种表示与未配对的文本和音频的学习框架。我们探索特定领域和特定领域的策展方法来创建音频文本对，我们使用它来进一步改进模型。我们还表明，当特定领域的策展与软标签对比损失结合使用时，我们能够在下游声音事件分类或声学场景分类任务的zero-shot分类性能方面获得显着改善。摘要：Recent advances in using language models to obtain cross-modal audio-text representations have overcome the limitations of conventional training approaches that use predefined labels. This has allowed the community to make progress in tasks like zero-shot classification, which would otherwise not be possible. However, learning such representations requires a large amount of human-annotated audio-text pairs. In this paper, we study unsupervised approaches to improve the learning framework of such representations with unpaired text and audio. We explore domain-unspecific and domain-specific curation methods to create audio-text pairs that we use to further improve the model. We also show that when domain-specific curation is used in conjunction with a soft-labeled contrastive loss, we are able to obtain significant improvement in terms of zero-shot classification performance on downstream sound event classification or acoustic scene classification tasks.【6】 AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation标题：AV-SAM：切分符合视听定位和切分的任何模型链接：https://arxiv.org/abs/2305.01836作者：Shentong Mo,Yapeng Tian机构：Carnegie Mellon University, University of Texas at Dallas摘要：SegmentAnything Model（SAM）最近在视觉分割任务中显示出强大的有效性。然而，关于SAM如何在视听任务中工作，如视觉声音定位和分割的探索较少。在这项工作中，我们提出了一个简单而有效的视听定位和分割框架的基础上的段任何模型，即AV-SAM，可以生成对应于音频的发声对象掩码。具体来说，我们的AV-SAM只是利用来自SAM中预训练图像编码器的音频特征和视觉特征的像素级视听融合来聚合跨模态表示。然后，将聚合的跨模态特征馈送到提示编码器和掩码解码器中以生成最终的视听分割掩码。我们在Flickr-SoundNet和AVSBench数据集上进行了广泛的实验。实验结果表明，所提出的AV-SAM算法在探测目标定位和分割方面具有较好的性能。摘要：Segment Anything Model (SAM) has recently shown its powerful effectiveness in visual segmentation tasks. However, there is less exploration concerning how SAM works on audio-visual tasks, such as visual sound localization and segmentation. In this work, we propose a simple yet effective audio-visual localization and segmentation framework based on the Segment Anything Model, namely AV-SAM, that can generate sounding object masks corresponding to the audio. Specifically, our AV-SAM simply leverages pixel-wise audio-visual fusion across audio features and visual features from the pre-trained image encoder in SAM to aggregate cross-modal representations. Then, the aggregated cross-modal features are fed into the prompt encoder and mask decoder to generate the final audio-visual segmentation masks. We conduct extensive experiments on Flickr-SoundNet and AVSBench datasets. The results demonstrate that the proposed AV-SAM can achieve competitive performance on sounding object localization and segmentation.【7】 SIA-FTP: A Spoken Instruction Aware Flight Trajectory Prediction Framework标题：SIA-FTP：一种语音指令感知的飞行轨迹预测框架链接：https://arxiv.org/abs/2305.01661作者：Dongyue Guo,Jianwei Zhang,Yi Lin机构：National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, China, College of Computer Science, Sichuan University, Chengdu, China摘要：通过语音通信进行地空协商是确保空中交通管制（ATC）安全和效率的重要前提。然而，随着交通流量的增加，人为因素导致的不正确指令给空管安全带来了极大的威胁。现有的飞行轨迹预测（FTP）方法主要依赖于历史轨迹的飞行状态，导致实时机动指令的预测存在较大延迟，不利于冲突检测。一个主要的原因是，口头指令和飞行轨迹在当前的空中交通管制（ATC）系统中以不同的方式呈现，这给考虑FTP任务中的机动指令带来了很大的挑战。在本文中，口语指令感知FTP框架，称为SIA-FTP，创新地提出了支持高机动FTP任务，结合即时口语指令。为了解决模态差距和最小化的数据要求，提出了一个3阶段的学习范式，以渐进的方式实现SIA-FTP框架，包括基于轨迹的FTP预训练，面向意图的指令嵌入学习，和多模态微调。具体地，FTP模型和具有操纵语义的指令嵌入在第一阶段和第二阶段中使用大量资源丰富的轨迹和文本数据进行预训练。接着，提出了一种多模态融合策略，将预训练的指令嵌入到FTP模型中，并将两个预训练的网络集成到联合模型中。最后，联合模型微调使用有限的轨迹指令数据，以提高机动指令场景内的FTP性能。实验结果表明，该框架提出了一个令人印象深刻的性能改善，在高机动的情况下。摘要：Ground-air negotiation via speech communication is a vital prerequisite for ensuring safety and efficiency in air traffic control (ATC) operations. However, with the increase in traffic flow, incorrect instructions caused by human factors bring a great threat to ATC safety. Existing flight trajectory prediction (FTP) approaches primarily rely on the flight status of historical trajectory, leading to significant delays in the prediction of real-time maneuvering instruction, which is not conducive to conflict detection. A major reason is that spoken instructions and flight trajectories are presented in different modalities in the current air traffic control (ATC) system, bringing great challenges to considering the maneuvering instruction in the FTP tasks. In this paper, a spoken instruction-aware FTP framework, called SIA-FTP, is innovatively proposed to support high-maneuvering FTP tasks by incorporating instant spoken instruction. To address the modality gap and minimize the data requirements, a 3-stage learning paradigm is proposed to implement the SIA-FTP framework in a progressive manner, including trajectory-based FTP pretraining, intent-oriented instruction embedding learning, and multi-modal finetuning. Specifically, the FTP model and the instruction embedding with maneuvering semantics are pre-trained using volumes of well-resourced trajectory and text data in the 1st and 2nd stages. In succession, a multi-modal fusion strategy is proposed to incorporate the pre-trained instruction embedding into the FTP model and integrate the two pre-trained networks into a joint model. Finally, the joint model is finetuned using the limited trajectory-instruction data to enhance the FTP performance within maneuvering instruction scenarios. The experimental results demonstrated that the proposed framework presents an impressive performance improvement in high-maneuvering scenarios.eess.AS音频处理【1】 Improved Vocal Effort Transfer Vector Estimation for Vocal Effort-Robust Speaker Verification标题：改进的发声努力传递向量估计用于发声努力-稳健说话人确认链接：https://arxiv.org/abs/2305.02147作者：Iván López-Espejo,Santi Prieto,Alfonso Ortega,Eduardo Lleida机构：Department of Electronic Systems, Aalborg University, Denmark, Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, USA, VeriDas | das-Nano, Navarre, Spain, ViVoLab, Arag´on Institute for Engineering Research (I,A)摘要：尽管现代说话人确认技术已经非常成熟，但在面对非中性发音（例如：非中性发音）的情况下，其性能仍然会显著下降。例如，大声喊叫和低声耳语）讲话。为了解决这个问题，在本文中，我们提出了一种新的扬声器嵌入补偿方法的基础上最小均方误差（MMSE）估计。该方法建模的联合分布的声乐努力转移向量和非中性发声嵌入空间，并在主成分分析域中操作，以应付非中性发声语音数据的稀缺性。实验进行了使用一个先进的说话人验证系统集成了一个强大的自我监督的预训练模型的语音表示。在与一个国家的最先进的嵌入补偿方法相比，建议的MMSE估计器产生优越的和有竞争力的等错误率的结果时，处理大声和低声说话，分别。摘要：Despite the maturity of modern speaker verification technology, its performance still significantly degrades when facing non-neutrally-phonated (e.g., shouted and whispered) speech. To address this issue, in this paper, we propose a new speaker embedding compensation method based on a minimum mean square error (MMSE) estimator. This method models the joint distribution of the vocal effort transfer vector and non-neutrally-phonated embedding spaces and operates in a principal component analysis domain to cope with non-neutrally-phonated speech data scarcity. Experiments are carried out using a cutting-edge speaker verification system integrating a powerful self-supervised pre-trained model for speech representation. In comparison with a state-of-the-art embedding compensation method, the proposed MMSE estimator yields superior and competitive equal error rate results when tackling shouted and whispered speech, respectively.【2】 Sound in occupied open-plan offices: Objective metrics with a review of historical perspectives标题：占用开放式办公室的音质：客观指标与历史观点的回顾链接：https://arxiv.org/abs/2305.01762作者：Manuj Yadav,Densil Cabrera,Jungsoo Kim,Janina Fels,Richard de Dear机构：a Sydney School of Architecture, Design and Planning, The University of Sydney, Sydney, NSW , Australia, b Institute of Technical Acoustics, RWTH Aachen University, Kopernikusstraße , Aachen, Germany, a r t i c l e, i n f o, Article history:摘要：开放式办公室（OPO）已经存在了半个多世纪，记录了工作场所地形的变迁以及其他因素。本文讨论了这样一个因素-在占用OPOs的声音环境中的几个客观的工作场所参数，在当代OPOs的测量和比较，在过去的50年的研究。在典型的工作时间内，在43个办公室进行了全方位和双耳声音测量。结果描述了几个声学和心理声学指标的变化，并提出了统计模型，预测这些指标作为办公室中的工作站的数量的函数。LA，53的等式。对于占用的OPO，典型值为6dB，频谱斜率约为-4dB 倍频程。LA，eq值在所研究的工作场所参数上变化不大（例如：例如，地板面积、工作活动等），-2除外。7 dB和-4。有 无地毯的办公室与有天花板吸收但有 无地毯的办公室之间的差异分别为1 dB;最有可能是由于地板冲击噪声降低导致语音电平降低。声音波动，以公制噪声气候（NCI：LA 10-LA 90）和心理声学波动强度（FS）随着OPO中工作站数量的增加而显著降低。这表明在较大的办公室中听觉干扰较少，这需要进一步调查。就历史趋势而言，OPO多年来变得更安静，特别是被量化为LA 90的背景噪声，尽管有几个微妙之处。总体而言，目前的研究结果可以告知几个OPO设计的角度，包括政策文件，提供值的OPO声学环境的实验室模拟，帮助解释OPO乘员的主观印象，等等。摘要：Open-plan offices (OPOs) have been around for more than half a century now, chronicling the vicissitudes of workplace topography amongst other factors. This paper addresses one such factor - the sound environment in occupied OPOs in relation to several objective workplace parameters, using measurements in contemporary OPOs and comparisons with studies over the last 50 years. Omnidirectional and binaural sound measurements were conducted in 43 offices during typical working hours. The results describe variation in several acoustic and psychoacoustic metrics, and present statistical models that predict these metrics as a function of the number of workstations in offices. LA,eq of 53.6 dB is typical for occupied OPOs, with spectral slope of approximately -4 dB octave. LA,eq values do not vary much over the workplace parameters studied (e.g., floor plate area, work activity, etc), except for -2.7 dB and -4.1 dB differences between offices with without carpeting, and offices with ceiling absorption but with without carpeting, respectively; most likely from reduced floor impact noise leading to speech level reduction. Sound fluctuation, as characterised by the metric Noise Climate (NCl: LA10 - LA90) and the psychoacoustic Fluctuation Strength (FS), decreases significantly with increasing number of workstations in OPOs. This suggests lesser auditory distraction in larger offices, which needs further investigation. In terms of historical trends, OPOs have become quieter over the years, especially background noise quantified as LA90, although there are several subtleties. Overall, current findings can inform several OPO design perspectives including policy documents, provide values for laboratory simulations of OPO acoustic environments, help interpret subjective impressions of OPO occupants, etc.【3】 Evaluation of Speaker Anonymization on Emotional Speech标题：说话人匿名化在情感语音中的评价链接：https://arxiv.org/abs/2305.01759作者：Hubert Nourtel,Pierre Champion,Denis Jouvet,Anthony Larcher,Marie Tahon机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France, LIUM - EA, Le Mans Universit´e, Avenue Olivier Messiaen, LE MANS CEDEX , France, ⋆ equal contribution from authors备注：None摘要：语音数据携带一系列个人信息，例如说话者的身份和情绪状态。这些属性可用于恶意目的。随着虚拟助手的发展，新一代的隐私威胁已经出现。目前的研究已经解决了保护语音隐私的话题。其中之一，VoicePrivacy倡议旨在促进语音技术隐私保护工具的开发。VoicePrivacy 2020 Challenge（VPC）所选的任务是关于说话人匿名化。目标是隐藏源说话人的身份，同时保留语言信息。VPC的基线使用语音转换。本文研究的影响，说话人匿名基线系统的VPC的情感信息存在于语音话语。根据VPC规则对攻击者关于匿名化系统的知识进行评估。我们的研究结果表明，VPC基线系统不抑制扬声器的情绪对知情的攻击者。当比较匿名语音与原始语音时，相对于IEMOCAP数据，情感识别性能下降了15%，类似于用于评估语言信息保存的自动语音识别所观察到的退化。摘要：Speech data carries a range of personal information, such as the speaker's identity and emotional state. These attributes can be used for malicious purposes. With the development of virtual assistants, a new generation of privacy threats has emerged. Current studies have addressed the topic of preserving speech privacy. One of them, the VoicePrivacy initiative aims to promote the development of privacy preservation tools for speech technology. The task selected for the VoicePrivacy 2020 Challenge (VPC) is about speaker anonymization. The goal is to hide the source speaker's identity while preserving the linguistic information. The baseline of the VPC makes use of a voice conversion. This paper studies the impact of the speaker anonymization baseline system of the VPC on emotional information present in speech utterances. Evaluation is performed following the VPC rules regarding the attackers' knowledge about the anonymization system. Our results show that the VPC baseline system does not suppress speakers' emotions against informed attackers. When comparing anonymized speech to original speech, the emotion recognition performance is degraded by 15 % relative to IEMOCAP data, similar to the degradation observed for automatic speech recognition used to evaluate the preservation of the linguistic information.【4】 Adversarial Generative NMF for Single Channel Source Separation标题：用于单通道信源分离的对抗性产生式NMF链接：https://arxiv.org/abs/2305.01758作者：Martin Ludvigsen,Markus Grasmair机构：Department of Mathematical Sciences, NTNU – Norwegian University of Science and Technology, Trondheim, Norway备注：24 pages, 4 figures摘要：正则化泛函的对抗学习的思想最近被引入到更广泛的逆问题背景中。这种方法背后的直觉是认识到，不仅需要学习构成想要表示的一类信号的基本特征，而且甚至更重要的是，在表示中要避免哪些特征。在本文中，我们将应用这种方法的源分离的问题，通过非负矩阵分解（NMF），并提出了一种新的方法的对抗性训练的NMF基地。我们在数值实验中示出，无论是图像和音频分离，这导致重建信号的明显改善，特别是在很少或没有强有力的监督数据的情况下。摘要：The idea of adversarial learning of regularization functionals has recently been introduced in the wider context of inverse problems. The intuition behind this method is the realization that it is not only necessary to learn the basic features that make up a class of signals one wants to represent, but also, or even more so, which features to avoid in the representation. In this paper, we will apply this approach to the problem of source separation by means of non-negative matrix factorization (NMF) and present a new method for the adversarial training of NMF bases. We show in numerical experiments, both for image and audio separation, that this leads to a clear improvement of the reconstructed signals, in particular in the case where little or no strong supervision data is available.【5】 M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis标题：M2-CTTS：端到端多尺度多模式对话式文语合成链接：https://arxiv.org/abs/2305.02269*与cs.SD语音【1】为同一篇作者：Jinlong Xue,Yayue Deng,Fengping Wang,Ya Li,Yingming Gao,Jianhua Tao,Jianqing Sun,Jiaen Liang机构：Beijing University of Posts and Telecommunications, Beijing, China, NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China, Unisound AI Technology Co., Ltd, Beijing, China备注：5 pages, 1 figures, 2 tables. Accepted by ICASSP 2023摘要：会话式文语转换（TTS）的目的是根据历史会话合成具有适当韵律的回复语音。然而，它仍然是一个挑战，全面的对话模型，和大多数的会话TTS系统只专注于提取全球信息和省略本地韵律特征，其中包含重要的细粒度信息，如关键字和重点。而且，仅仅考虑文本特征是不够的，声学特征还包含了各种韵律信息。因此，我们提出了M2-CTTS，一个端到端的多尺度多模态会话的文本到语音系统，旨在综合利用历史会话和增强韵律的表达。更具体地说，我们设计了一个文本上下文模块和一个声学上下文模块的粗粒度和细粒度建模。实验结果表明，我们的模型与细粒度的上下文信息混合，并考虑额外的声学特征，在CMOS测试中实现了更好的韵律性能和自然度。摘要：Conversational text-to-speech (TTS) aims to synthesize speech with proper prosody of reply based on the historical conversation. However, it is still a challenge to comprehensively model the conversation, and a majority of conversational TTS systems only focus on extracting global information and omit local prosody features, which contain important fine-grained information like keywords and emphasis. Moreover, it is insufficient to only consider the textual features, and acoustic features also contain various prosody information. Hence, we propose M2-CTTS, an end-to-end multi-scale multi-modal conversational text-to-speech system, aiming to comprehensively utilize historical conversation and enhance prosodic expression. More specifically, we design a textual context module and an acoustic context module with both coarse-grained and fine-grained modeling. Experimental results demonstrate that our model mixed with fine-grained context information and additionally considering acoustic features achieves better prosody performance and naturalness in CMOS tests.【6】 Considerations for Ethical Speech Recognition Datasets标题：伦理语音识别数据集的考虑因素链接：https://arxiv.org/abs/2305.02081*与cs.SD语音【2】为同一篇作者：Orestis Papakyriakopoulos,Alice Xiang机构：Sony AI, Switzerland, USA备注：None摘要：语音人工智能技术主要是在公开可用的数据集上进行训练，或者通过大规模的语音网络爬行进行训练。在这两种情况下，数据采集的重点都是尽量减少收集工作，而不一定考虑数据主体的保护或用户需求。这导致模型在用于偏离训练集中的主要人口统计的用户时不稳健，区分具有不同方言、口音、说话风格和不流利的个体。在这次演讲中，我们使用自动语音识别作为案例研究，并研究道德语音数据集应该具备的属性，以实现负责任的人工智能应用。我们展示了多样性问题、包容性实践以及可以改进训练模型的必要考虑因素，同时促进模型的可解释性并保护用户和数据主体。我们主张数据主体的法律和隐私保护，与用户人口统计和需求相对应的有针对性的数据采样，适当的元数据，以确保在模型失败的情况下的可解释性和问责制，以及社会技术 和情境模型设计。我们希望这次演讲能够激励研究人员和从业者在语音技术和其他领域设计和使用更多以人为中心的数据集，以授权和尊重用户的方式，同时提高机器学习模型的鲁棒性和实用性。摘要：Speech AI Technologies are largely trained on publicly available datasets or by the massive web-crawling of speech. In both cases, data acquisition focuses on minimizing collection effort, without necessarily taking the data subjects' protection or user needs into consideration. This results to models that are not robust when used on users who deviate from the dominant demographics in the training set, discriminating individuals having different dialects, accents, speaking styles, and disfluencies. In this talk, we use automatic speech recognition as a case study and examine the properties that ethical speech datasets should possess towards responsible AI applications. We showcase diversity issues, inclusion practices, and necessary considerations that can improve trained models, while facilitating model explainability and protecting users and data subjects. We argue for the legal & privacy protection of data subjects, targeted data sampling corresponding to user demographics & needs, appropriate meta data that ensure explainability & accountability in cases of model failure, and the sociotechnical & situated model design. We hope this talk can inspire researchers & practitioners to design and use more human-centric datasets in speech technologies and other domains, in ways that empower and respect users, while improving machine learning models' robustness and utility.【7】 Diverse and Vivid Sound Generation from Text Descriptions标题：从文本描述中生成多样化和生动的声音链接：https://arxiv.org/abs/2305.01980*与cs.SD语音【3】为同一篇作者：Guangwei Li,Xuenan Xu,Lingfeng Dai,Mengyue Wu,Kai Yu机构：X-LANCE Lab, Department of Computer Science and Engineering, MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China摘要：以往的音频生成主要集中在语音或音乐等特定的声音类别，其形式和内容受到很大的限制。在本文中，我们超越了特定的音频生成，使用自然语言描述为线索，产生广泛的声音。与视觉信息不同，文本描述本质上是简洁的，但其下隐藏着丰富的含义，这对要生成的音频提出了更高的可能性和复杂性。变异量化GAN用于训练学习频谱图的离散表示的码本。对于给定的文本描述，其预训练的嵌入被馈送到Transformer以对码本索引进行采样，以解码频谱图，以由melgan声码器进一步变换成波形。所生成的波形具有高质量和保真度，同时与给定的文本良好地对应。实验表明，我们提出的方法是能够产生自然，生动的音频，取得了极好的定量和定性的结果。摘要：Previous audio generation mainly focuses on specified sound classes such as speech or music, whose form and content are greatly restricted. In this paper, we go beyond specific audio generation by using natural language description as a clue to generate broad sounds. Unlike visual information, a text description is concise by its nature but has rich hidden meanings beneath, which poses a higher possibility and complexity on the audio to be generated. A Variation-Quantized GAN is used to train a codebook learning discrete representations of spectrograms. For a given text description, its pre-trained embedding is fed to a Transformer to sample codebook indices to decode a spectrogram to be further transformed into waveform by a melgan vocoder. The generated waveform has high quality and fidelity while excellently corresponding to the given text. Experiments show that our proposed method is capable of generating natural, vivid audios, achieving superb quantitative and qualitative results.【8】 Analysing the Impact of Audio Quality on the Use of Naturalistic Long-Form Recordings for Infant-Directed Speech Research标题：分析音频质量对使用自然长格式录音进行婴儿定向言语研究的影响链接：https://arxiv.org/abs/2305.01965*与cs.SD语音【4】为同一篇作者：María Andrea Cruz Blandón,Alejandrina Cristia,Okko Räsänen机构：D´epartement d’´etudes cognitives, ENS, EHESS, CNRS, PSL University, rue d’Ulm, Paris, France, Unit of Computing Sciences, Tampere University, P.O. Box , FI-, Finland备注：Accepted at the CogSci2023 conference摘要：早期语言习得模型旨在了解婴儿如何引导他们的语言技能。建模包括用于训练模型的输入数据的属性、被测试的认知假设及其算法实现，以及将模型与人类数据进行比较的评估方法。最近的发展使得能够使用更自然的训练数据的计算模型。这也激发了对模型行为的更自然的测试的发展。实现这一目标的关键一步是开发代表性的语音数据集，包括婴儿在自然环境中听到的语音。然而，这种记录的主要缺点是它们通常是嘈杂的，并且目前不清楚声音质量如何影响对这种数据进行的分析和建模实验。在本文中，我们探讨这方面的情况下，婴儿导向的语音（IDS）和成人导向的语音（ADS）分析。首先，我们手动和自动注释的音频质量的话语提取两个语料库的儿童为中心的长形式的录音（英语和法语）。然后，我们比较了IDS和ADS在实验室数据集中的声学特征，以及自然数据的不同音频质量子集。最后，我们评估了音频质量和录音环境如何改变使用最近的自监督学习模型的建模分析的结论。我们的研究结果表明，使用适度和高音频质量的自然语音数据的结果在很大程度上类似的结论IDS和ADS的声学分析和建模实验。我们还发现，一个自动的声音质量评估工具，可以用来筛选出有用的部分，长形式的录音进行更仔细的分析，具有可比的结果，手动质量注释。摘要：Modelling of early language acquisition aims to understand how infants bootstrap their language skills. The modelling encompasses properties of the input data used for training the models, the cognitive hypotheses and their algorithmic implementations being tested, and the evaluation methodologies to compare models to human data. Recent developments have enabled the use of more naturalistic training data for computational models. This also motivates development of more naturalistic tests of model behaviour. A crucial step towards such an aim is to develop representative speech datasets consisting of speech heard by infants in their natural environments. However, a major drawback of such recordings is that they are typically noisy, and it is currently unclear how the sound quality could affect analyses and modelling experiments conducted on such data. In this paper, we explore this aspect for the case of infant-directed speech (IDS) and adult-directed speech (ADS) analysis. First, we manually and automatically annotated audio quality of utterances extracted from two corpora of child-centred long-form recordings (in English and French). We then compared acoustic features of IDS and ADS in an in-lab dataset and across different audio quality subsets of naturalistic data. Finally, we assessed how the audio quality and recording environment may change the conclusions of a modelling analysis using a recent self-supervised learning model. Our results show that the use of modest and high audio quality naturalistic speech data result in largely similar conclusions on IDS and ADS in terms of acoustic analyses and modelling experiments. We also found that an automatic sound quality assessment tool can be used to screen out useful parts of long-form recordings for a closer analysis with comparable results to that of manual quality annotation.【9】 Unsupervised Improvement of Audio-Text Cross-Modal Representations标题：音频-文本跨通道表示的无监督改进链接：https://arxiv.org/abs/2305.01864*与cs.SD语音【5】为同一篇作者：Zhepei Wang,Cem Subakan,Krishna Subramani,Junkai Wu,Tiago Tavares,Fabio Ayres,Paris Smaragdis机构：♯University of Illinois at Urbana-Champaign, ♭Universit´e Laval, ♮Concordia University, Mila-Quebec AI Institute, †Insper, αAmazon Web Services备注：Submitted to WASPAA 2023摘要：使用语言模型来获得跨模态音频文本表示的最新进展已经克服了使用预定义标签的传统训练方法的局限性。这使社区能够在zero-shot分类等任务中取得进展，否则这是不可能的。然而，学习这样的表示需要大量的人类注释的音频文本对。在本文中，我们研究了无监督的方法，以改善这种表示与未配对的文本和音频的学习框架。我们探索特定领域和特定领域的策展方法来创建音频文本对，我们使用它来进一步改进模型。我们还表明，当特定领域的策展与软标签对比损失结合使用时，我们能够在下游声音事件分类或声学场景分类任务的zero-shot分类性能方面获得显着改善。摘要：Recent advances in using language models to obtain cross-modal audio-text representations have overcome the limitations of conventional training approaches that use predefined labels. This has allowed the community to make progress in tasks like zero-shot classification, which would otherwise not be possible. However, learning such representations requires a large amount of human-annotated audio-text pairs. In this paper, we study unsupervised approaches to improve the learning framework of such representations with unpaired text and audio. We explore domain-unspecific and domain-specific curation methods to create audio-text pairs that we use to further improve the model. We also show that when domain-specific curation is used in conjunction with a soft-labeled contrastive loss, we are able to obtain significant improvement in terms of zero-shot classification performance on downstream sound event classification or acoustic scene classification tasks.【10】 AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation标题：AV-SAM：切分符合视听定位和切分的任何模型链接：https://arxiv.org/abs/2305.01836*与cs.SD语音【6】为同一篇作者：Shentong Mo,Yapeng Tian机构：Carnegie Mellon University, University of Texas at Dallas摘要：SegmentAnything Model（SAM）最近在视觉分割任务中显示出强大的有效性。然而，关于SAM如何在视听任务中工作，如视觉声音定位和分割的探索较少。在这项工作中，我们提出了一个简单而有效的视听定位和分割框架的基础上的段任何模型，即AV-SAM，可以生成对应于音频的发声对象掩码。具体来说，我们的AV-SAM只是利用来自SAM中预训练图像编码器的音频特征和视觉特征的像素级视听融合来聚合跨模态表示。然后，将聚合的跨模态特征馈送到提示编码器和掩码解码器中以生成最终的视听分割掩码。我们在Flickr-SoundNet和AVSBench数据集上进行了广泛的实验。实验结果表明，所提出的AV-SAM算法在探测目标定位和分割方面具有较好的性能。摘要：Segment Anything Model (SAM) has recently shown its powerful effectiveness in visual segmentation tasks. However, there is less exploration concerning how SAM works on audio-visual tasks, such as visual sound localization and segmentation. In this work, we propose a simple yet effective audio-visual localization and segmentation framework based on the Segment Anything Model, namely AV-SAM, that can generate sounding object masks corresponding to the audio. Specifically, our AV-SAM simply leverages pixel-wise audio-visual fusion across audio features and visual features from the pre-trained image encoder in SAM to aggregate cross-modal representations. Then, the aggregated cross-modal features are fed into the prompt encoder and mask decoder to generate the final audio-visual segmentation masks. We conduct extensive experiments on Flickr-SoundNet and AVSBench datasets. The results demonstrate that the proposed AV-SAM can achieve competitive performance on sounding object localization and segmentation.【11】 SIA-FTP: A Spoken Instruction Aware Flight Trajectory Prediction Framework标题：SIA-FTP：一种语音指令感知的飞行轨迹预测框架链接：https://arxiv.org/abs/2305.01661*与cs.SD语音【7】为同一篇作者：Dongyue Guo,Jianwei Zhang,Yi Lin机构：National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, China, College of Computer Science, Sichuan University, Chengdu, China摘要：通过语音通信进行地空协商是确保空中交通管制（ATC）安全和效率的重要前提。然而，随着交通流量的增加，人为因素导致的不正确指令给空管安全带来了极大的威胁。现有的飞行轨迹预测（FTP）方法主要依赖于历史轨迹的飞行状态，导致实时机动指令的预测存在较大延迟，不利于冲突检测。一个主要的原因是，口头指令和飞行轨迹在当前的空中交通管制（ATC）系统中以不同的方式呈现，这给考虑FTP任务中的机动指令带来了很大的挑战。在本文中，口语指令感知FTP框架，称为SIA-FTP，创新地提出了支持高机动FTP任务，结合即时口语指令。为了解决模态差距和最小化的数据要求，提出了一个3阶段的学习范式，以渐进的方式实现SIA-FTP框架，包括基于轨迹的FTP预训练，面向意图的指令嵌入学习，和多模态微调。具体地，FTP模型和具有操纵语义的指令嵌入在第一阶段和第二阶段中使用大量资源丰富的轨迹和文本数据进行预训练。接着，提出了一种多模态融合策略，将预训练的指令嵌入到FTP模型中，并将两个预训练的网络集成到联合模型中。最后，联合模型微调使用有限的轨迹指令数据，以提高机动指令场景内的FTP性能。实验结果表明，该框架提出了一个令人印象深刻的性能改善，在高机动的情况下。摘要：Ground-air negotiation via speech communication is a vital prerequisite for ensuring safety and efficiency in air traffic control (ATC) operations. However, with the increase in traffic flow, incorrect instructions caused by human factors bring a great threat to ATC safety. Existing flight trajectory prediction (FTP) approaches primarily rely on the flight status of historical trajectory, leading to significant delays in the prediction of real-time maneuvering instruction, which is not conducive to conflict detection. A major reason is that spoken instructions and flight trajectories are presented in different modalities in the current air traffic control (ATC) system, bringing great challenges to considering the maneuvering instruction in the FTP tasks. In this paper, a spoken instruction-aware FTP framework, called SIA-FTP, is innovatively proposed to support high-maneuvering FTP tasks by incorporating instant spoken instruction. To address the modality gap and minimize the data requirements, a 3-stage learning paradigm is proposed to implement the SIA-FTP framework in a progressive manner, including trajectory-based FTP pretraining, intent-oriented instruction embedding learning, and multi-modal finetuning. Specifically, the FTP model and the instruction embedding with maneuvering semantics are pre-trained using volumes of well-resourced trajectory and text data in the 1st and 2nd stages. In succession, a multi-modal fusion strategy is proposed to incorporate the pre-trained instruction embedding into the FTP model and integrate the two pre-trained networks into a joint model. Finally, the joint model is finetuned using the limited trajectory-instruction data to enhance the FTP performance within maneuvering instruction scenarios. The experimental results demonstrated that the proposed framework presents an impressive performance improvement in high-maneuvering scenarios.
每日语音速递
