今日论文合集：cs.SD语音5篇，eess.AS音频处理8篇。
cs.SD语音

【1】 DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided  Speaker Embedding标题：DiffV2S：视觉引导说话人嵌入的基于扩散的视频到语音合成链接：https://arxiv.org/abs/2308.07787

作者：Jeongsoo Choi,Joanna Hong,Yong Man Ro机构：School of Electrical Engineering, KAIST备注：ICCV 2023摘要：最近的研究已经证明了令人印象深刻的结果，在视频到语音合成，涉及重建语音完全从视觉输入。然而，以前的工作一直在努力准确地合成语音，由于缺乏足够的指导模型推断正确的内容与适当的声音。为了解决这个问题，他们采用了一个额外的扬声器嵌入作为说话风格的指导，从参考听觉信息。然而，并不总是能够从对应的视频输入获得音频信息，特别是在推断时间期间。在本文中，我们提出了一种新的视觉引导的说话人嵌入提取器，使用自监督的预训练模型和即时调整技术。在这样做时，可以仅从输入的视觉信息产生富说话者嵌入信息，并且在推断时间期间不需要额外的音频信息。使用提取的视觉引导的扬声器嵌入表示，我们进一步开发了一个基于扩散的视频到语音合成模型，所谓的DiffV2S，条件是这些扬声器嵌入和从输入视频中提取的视觉表示。所提出的DiffV2S不仅保持输入视频帧中包含的音素细节，而且还创建了一个高度可理解的梅尔频谱图，其中多个扬声器的扬声器身份都被保留。我们的实验结果表明，DiffV2S实现了国家的最先进的性能相比，以前的视频到语音合成技术。摘要：Recent research has demonstrated impressive results in video-to-speech synthesis which involves reconstructing speech solely from visual input. However, previous works have struggled to accurately synthesize speech due to a lack of sufficient guidance for the model to infer the correct content with the appropriate sound. To resolve the issue, they have adopted an extra speaker embedding as a speaking style guidance from a reference auditory information. Nevertheless, it is not always possible to obtain the audio information from the corresponding video input, especially during the inference time. In this paper, we present a novel vision-guided speaker embedding extractor using a self-supervised pre-trained model and prompt tuning technique. In doing so, the rich speaker embedding information can be produced solely from input visual information, and the extra audio information is not necessary during the inference time. Using the extracted vision-guided speaker embedding representations, we further develop a diffusion-based video-to-speech synthesis model, so called DiffV2S, conditioned on those speaker embeddings and the visual representation extracted from the input video. The proposed DiffV2S not only maintains phoneme details contained in the input video frames, but also creates a highly intelligible mel-spectrogram in which the speaker identities of the multiple speakers are all preserved. Our experimental results show that DiffV2S achieves the state-of-the-art performance compared to the previous video-to-speech synthesis technique.
【2】 O-1: Self-training with Oracle and 1-best Hypothesis标题：O-1：甲骨文的自我训练和1-最佳假设链接：https://arxiv.org/abs/2308.07486
作者：Murali Karthick Baskar,Andrew Rosenberg,Bhuvana Ramabhadran,Kartik Audhkhasi机构：Google Inc.摘要：我们引入了O-1，一个新的自我训练目标，以减少训练偏差和统一的语音识别的训练和评估指标。O-1是期望最小贝叶斯风险（EMBR）的一个更快的变体，它增强了oracle假设，可以适应监督和无监督数据。我们证明了我们的方法的有效性，在公开可用的SpeechStew数据集和大规模的内部数据集的识别。在Speechstew上，O-1目标将实际性能和Oracle性能之间的差距缩小了80%，而EMBR将差距缩小了43%。O-1在SpeechStew包含的各种数据集上实现了13%至25%的EMBR相对改进，并且相对于Oracle WER在内部数据集上的EMBR训练减少了12%的相对差距。总体而言，O-1导致WER相对于EMBR有9\%的相对改善，从而说明了针对大规模数据集提出的目标的可扩展性。摘要：We introduce O-1, a new self-training objective to reduce training bias and unify training and evaluation metrics for speech recognition. O-1 is a faster variant of Expected Minimum Bayes Risk (EMBR), that boosts the oracle hypothesis and can accommodate both supervised and unsupervised data. We demonstrate the effectiveness of our approach in terms of recognition on publicly available SpeechStew datasets and a large-scale, in-house data set. On Speechstew, the O-1 objective closes the gap between the actual and oracle performance by 80\% relative compared to EMBR which bridges the gap by 43\% relative. O-1 achieves 13\% to 25\% relative improvement over EMBR on the various datasets that SpeechStew comprises of, and a 12\% relative gap reduction with respect to the oracle WER over EMBR training on the in-house dataset. Overall, O-1 results in a 9\% relative improvement in WER over EMBR, thereby speaking to the scalability of the proposed objective for large-scale datasets.
【3】 Text Injection for Capitalization and Turn-Taking Prediction in Speech  Models标题：语音模型中用于大写和话轮转换预测的文本注入链接：https://arxiv.org/abs/2308.07395
作者：Shaan Bijwadia,Shuo-yiin Chang,Weiran Wang,Zhong Meng,Hao Zhang,Tara N. Sainath机构：Google, USA摘要：用于自动语音识别（ASR）的文本注入（其中未配对的仅文本数据用于补充配对的音频-文本数据）已经显示出对单词错误率的有希望的改进。本研究探讨使用文本注入辅助任务，这是非ASR任务经常执行的E2 E模型。在这项工作中，我们使用联合端到端和内部语言模型训练（JEIT）作为我们的文本注入算法来训练ASR模型，执行两个辅助任务。第一个是大写，这是一个去规范化的任务。第二种是轮转换预测，其尝试识别用户是否已经完成了数字助理交互中的会话轮。我们展示的结果表明，我们的文本注入方法提高了长尾数据的大写性能，并提高了话轮检测召回率。摘要：Text injection for automatic speech recognition (ASR), wherein unpaired text-only data is used to supplement paired audio-text data, has shown promising improvements for word error rate. This study examines the use of text injection for auxiliary tasks, which are the non-ASR tasks often performed by an E2E model. In this work, we use joint end-to-end and internal language model training (JEIT) as our text injection algorithm to train an ASR model which performs two auxiliary tasks. The first is capitalization, which is a de-normalization task. The second is turn-taking prediction, which attempts to identify whether a user has completed their conversation turn in a digital assistant interaction. We show results demonstrating that our text injection method boosts capitalization performance for long-tail data, and improves turn-taking detection recall.
【4】 Using Text Injection to Improve Recognition of Personal Identifiers in  Speech标题：利用文本注入改进语音中个人识别符的识别链接：https://arxiv.org/abs/2308.07393
作者：Yochai Blau,Rohan Agrawal,Lior Madmony,Gary Wang,Andrew Rosenberg,Zhehuai Chen,Zorik Gekhman,Genady Beryozkin,Parisa Haghani,Bhuvana Ramabhadran机构：Google备注：Accepted to Interspeech 2023摘要：在许多自动语音识别（ASR）应用中，准确识别特定类别（诸如人名、日期或其他标识符）是至关重要的。由于这些类别代表了个人信息，因此对这些数据的道德使用，包括收集、转录、培训和评估，需要特别注意。确保个人安全和隐私的一种方法是从收集中删除或完全删除个人身份信息（PII）。然而，这导致ASR模型往往具有这些类别的较低识别准确度。我们使用文本注入，以提高识别的PII类别包括假的文本替代品在训练数据中使用文本注入的方法。我们证明了医疗笔记中的姓名和日期召回的实质性改进，同时提高了整体WER。对于字母数字序列，我们显示字符错误率和句子准确性的改进。摘要：Accurate recognition of specific categories, such as persons' names, dates or other identifiers is critical in many Automatic Speech Recognition (ASR) applications. As these categories represent personal information, ethical use of this data including collection, transcription, training and evaluation demands special care. One way of ensuring the security and privacy of individuals is to redact or eliminate Personally Identifiable Information (PII) from collection altogether. However, this results in ASR models that tend to have lower recognition accuracy of these categories. We use text-injection to improve the recognition of PII categories by including fake textual substitutes of PII categories in the training data using a text injection method. We demonstrate substantial improvement to Recall of Names and Dates in medical notes while improving overall WER. For alphanumeric digit sequences we show improvements to Character Error Rate and Sentence Accuracy.
【5】 Preliminary investigation of the short-term in situ performance of an  automatic masker selection system标题：掩膜自动分选系统的短期现场性能初步研究链接：https://arxiv.org/abs/2308.07767
作者：Bhan Lam,Zhen-Ting Ong,Kenneth Ooi,Wen-Hui Ong,Trevor Wong,Karn N. Watcharasupat,Woon-Seng Gan机构：School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore,  Nanyang Avenue, Singapore , Center for Music Technology, Georgia Institute of Technology, USA,  McMillan Street NW, Atlanta, GA备注：paper submitted to the 52nd International Congress and Exposition on Noise Control Engineering held in Chiba, Greater Tokyo, Japan, on 20-23 August 2023 (Inter-Noise 2023)摘要：声景增强或“掩蔽”将想要的声音引入到声学环境中以改善声学舒适度。通常，掩蔽物选择和回放策略是任意的或基于简单规则（例如，-3dBA），这可能导致动态声学环境的声学舒适度的次优增加或甚至降低。为了减少在掩蔽物的选择中的模糊性，最近开发了掩蔽物自动选择系统（AMSS）。AMSS使用在大规模主观反应数据集上训练的深度学习模型，以最大化导出的ISO愉悦度（ISO 12913-2）。因此，本研究探讨了短期的AMSS在城市公园的凉亭实施的原位性能。首先，从AMSS的预测ISO愉悦度进行评估，在比较原位的主观评价分数。其次，评估不同的掩蔽选择方案对感知的情感质量和适当性的影响。每位参与者总共评估了6种情况：（1）没有掩蔽物的周围环境;（2）AMSS;（3）鸟和（4）来自现有技术的水掩蔽物;（5）从用于训练AMSS的相同掩蔽物库中随机选择;以及（6）基于对用于训练AMSS的数据集的分析来选择最佳性能掩蔽器。摘要：Soundscape augmentation or "masking" introduces wanted sounds into the acoustic environment to improve acoustic comfort. Usually, the masker selection and playback strategies are either arbitrary or based on simple rules (e.g. -3 dBA), which may lead to sub-optimal increment or even reduction in acoustic comfort for dynamic acoustic environments. To reduce ambiguity in the selection of maskers, an automatic masker selection system (AMSS) was recently developed. The AMSS uses a deep-learning model trained on a large-scale dataset of subjective responses to maximize the derived ISO pleasantness (ISO 12913-2). Hence, this study investigates the short-term in situ performance of the AMSS implemented in a gazebo in an urban park. Firstly, the predicted ISO pleasantness from the AMSS is evaluated in comparison to the in situ subjective evaluation scores. Secondly, the effect of various masker selection schemes on the perceived affective quality and appropriateness would be evaluated. In total, each participant evaluated 6 conditions: (1) ambient environment with no maskers; (2) AMSS; (3) bird and (4) water masker from prior art; (5) random selection from same pool of maskers used to train the AMSS; and (6) selection of best-performing maskers based on the analysis of the dataset used to train the AMSS.
eess.AS音频处理

【1】 GIST-AiTeR Speaker Diarization System for VoxCeleb Speaker Recognition  Challenge (VoxSRC) 2023标题：用于VoxCeleb说话人识别挑战赛(VoxSRC)2023的Gist-Aiter说话人二值化系统链接：https://arxiv.org/abs/2308.07788

作者：Dongkeon Park,Ji Won Kim,Kang Ryul Kim,Do Hyun Lee,Hong Kook Kim机构：AI Graduate School,School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology, Gwangju , Republic of Korea备注：2023 VoxSRC Track4摘要：本报告描述了GIST-AiTeR团队为2023 VoxCeleb Speaker Recognition Challenge（VoxSRC）Track 4提交的系统。我们的提交系统专注于实现不同的扬声器日志化（SD）技术，包括ResNet 293和MFA-Conformer与不同的段和跳长的组合。然后，将这些模型组合成集成模型。ResNet 293和MFA-Conformer模型在VAL 46上分别表现出3.65%和3.83%的日志化错误率（DER）。提交的集合模型在VAL 46上提供了3.50%的DER，因此，它在2023 VoxSRC挑战测试集上实现了4.88%的DER。摘要：This report describes the submission system by the GIST-AiTeR team for the 2023 VoxCeleb Speaker Recognition Challenge (VoxSRC) Track 4. Our submission system focuses on implementing diverse speaker diarization (SD) techniques, including ResNet293 and MFA-Conformer with different combinations of segment and hop length. Then, those models are combined into an ensemble model. The ResNet293 and MFA-Conformer models exhibited the diarization error rates (DERs) of 3.65% and 3.83% on VAL46, respectively. The submitted ensemble model provided a DER of 3.50% on VAL46, and consequently, it achieved a DER of 4.88% on the 2023 VoxSRC challenge test set.
【2】 Preliminary investigation of the short-term in situ performance of an  automatic masker selection system标题：掩膜自动分选系统的短期现场性能初步研究链接：https://arxiv.org/abs/2308.07767
作者：Bhan Lam,Zhen-Ting Ong,Kenneth Ooi,Wen-Hui Ong,Trevor Wong,Karn N. Watcharasupat,Woon-Seng Gan机构：School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore,  Nanyang Avenue, Singapore , Center for Music Technology, Georgia Institute of Technology, USA,  McMillan Street NW, Atlanta, GA备注：paper submitted to the 52nd International Congress and Exposition on Noise Control Engineering held in Chiba, Greater Tokyo, Japan, on 20-23 August 2023 (Inter-Noise 2023)摘要：声景增强或“掩蔽”将想要的声音引入到声学环境中以改善声学舒适度。通常，掩蔽物选择和回放策略是任意的或基于简单规则（例如，-3dBA），这可能导致动态声学环境的声学舒适度的次优增加或甚至降低。为了减少在掩蔽物的选择中的模糊性，最近开发了掩蔽物自动选择系统（AMSS）。AMSS使用在大规模主观反应数据集上训练的深度学习模型，以最大化导出的ISO愉悦度（ISO 12913-2）。因此，本研究探讨了短期的AMSS在城市公园的凉亭实施的原位性能。首先，从AMSS的预测ISO愉悦度进行评估，在比较原位的主观评价分数。其次，评估不同的掩蔽选择方案对感知的情感质量和适当性的影响。每位参与者总共评估了6种情况：（1）没有掩蔽物的周围环境;（2）AMSS;（3）鸟和（4）来自现有技术的水掩蔽物;（5）从用于训练AMSS的相同掩蔽物库中随机选择;以及（6）基于对用于训练AMSS的数据集的分析来选择最佳性能掩蔽器。摘要：Soundscape augmentation or "masking" introduces wanted sounds into the acoustic environment to improve acoustic comfort. Usually, the masker selection and playback strategies are either arbitrary or based on simple rules (e.g. -3 dBA), which may lead to sub-optimal increment or even reduction in acoustic comfort for dynamic acoustic environments. To reduce ambiguity in the selection of maskers, an automatic masker selection system (AMSS) was recently developed. The AMSS uses a deep-learning model trained on a large-scale dataset of subjective responses to maximize the derived ISO pleasantness (ISO 12913-2). Hence, this study investigates the short-term in situ performance of the AMSS implemented in a gazebo in an urban park. Firstly, the predicted ISO pleasantness from the AMSS is evaluated in comparison to the in situ subjective evaluation scores. Secondly, the effect of various masker selection schemes on the perceived affective quality and appropriateness would be evaluated. In total, each participant evaluated 6 conditions: (1) ambient environment with no maskers; (2) AMSS; (3) bird and (4) water masker from prior art; (5) random selection from same pool of maskers used to train the AMSS; and (6) selection of best-performing maskers based on the analysis of the dataset used to train the AMSS.
【3】 The DKU-MSXF Diarization System for the VoxCeleb Speaker Recognition  Challenge 2023标题：用于VoxCeleb说话人识别挑战赛2023的DKU-MSXF二值化系统链接：https://arxiv.org/abs/2308.07595
作者：Ming Cheng,Weiqing Wang,Xiaoyi Qin,Yuke Lin,Ning Jiang,Guoqing Zhao,Ming Li机构：Data Science Research Center, Duke Kunshan University, Kunshan, China, Mashang Consumer Finance Co., Ltd.摘要：本文介绍了DKU-MSXF提交到VoxCeleb Speaker Recognition Challenge 2023（VoxSRC-23）的轨道4。我们的系统管道包含语音活动检测，基于聚类的日记，重叠语音检测，和目标说话人的语音活动检测，其中每个程序有一个融合的输出从3个子模型。最后，我们使用DOVER-Lap融合了不同的基于聚类和基于TSVAD的日志化系统，并实现了4.30%的日志化错误率（DER），在挑战排行榜的第4轨道上排名第一。摘要：This paper describes the DKU-MSXF submission to track 4 of the VoxCeleb Speaker Recognition Challenge 2023 (VoxSRC-23). Our system pipeline contains voice activity detection, clustering-based diarization, overlapped speech detection, and target-speaker voice activity detection, where each procedure has a fused output from 3 sub-models. Finally, we fuse different clustering-based and TSVAD-based diarization systems using DOVER-Lap and achieve the 4.30% diarization error rate (DER), which ranks first place on track 4 of the challenge leaderboard.
【4】 DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided  Speaker Embedding标题：DiffV2S：视觉引导说话人嵌入的基于扩散的视频到语音合成链接：https://arxiv.org/abs/2308.07787
作者：Jeongsoo Choi,Joanna Hong,Yong Man Ro机构：School of Electrical Engineering, KAIST备注：ICCV 2023摘要：最近的研究已经证明了令人印象深刻的结果，在视频到语音合成，涉及重建语音完全从视觉输入。然而，以前的工作一直在努力准确地合成语音，由于缺乏足够的指导模型推断正确的内容与适当的声音。为了解决这个问题，他们采用了一个额外的扬声器嵌入作为说话风格的指导，从参考听觉信息。然而，并不总是能够从对应的视频输入获得音频信息，特别是在推断时间期间。在本文中，我们提出了一种新的视觉引导的说话人嵌入提取器，使用自监督的预训练模型和即时调整技术。在这样做时，可以仅从输入的视觉信息产生富说话者嵌入信息，并且在推断时间期间不需要额外的音频信息。使用提取的视觉引导的扬声器嵌入表示，我们进一步开发了一个基于扩散的视频到语音合成模型，所谓的DiffV2S，条件是这些扬声器嵌入和从输入视频中提取的视觉表示。所提出的DiffV2S不仅保持输入视频帧中包含的音素细节，而且还创建了一个高度可理解的梅尔频谱图，其中多个扬声器的扬声器身份都被保留。我们的实验结果表明，DiffV2S实现了国家的最先进的性能相比，以前的视频到语音合成技术。摘要：Recent research has demonstrated impressive results in video-to-speech synthesis which involves reconstructing speech solely from visual input. However, previous works have struggled to accurately synthesize speech due to a lack of sufficient guidance for the model to infer the correct content with the appropriate sound. To resolve the issue, they have adopted an extra speaker embedding as a speaking style guidance from a reference auditory information. Nevertheless, it is not always possible to obtain the audio information from the corresponding video input, especially during the inference time. In this paper, we present a novel vision-guided speaker embedding extractor using a self-supervised pre-trained model and prompt tuning technique. In doing so, the rich speaker embedding information can be produced solely from input visual information, and the extra audio information is not necessary during the inference time. Using the extracted vision-guided speaker embedding representations, we further develop a diffusion-based video-to-speech synthesis model, so called DiffV2S, conditioned on those speaker embeddings and the visual representation extracted from the input video. The proposed DiffV2S not only maintains phoneme details contained in the input video frames, but also creates a highly intelligible mel-spectrogram in which the speaker identities of the multiple speakers are all preserved. Our experimental results show that DiffV2S achieves the state-of-the-art performance compared to the previous video-to-speech synthesis technique.
【5】 AKVSR: Audio Knowledge Empowered Visual Speech Recognition by  Compressing Audio Knowledge of a Pretrained Model标题：AKVSR：通过压缩预训练模型的音频知识来增强视觉语音识别链接：https://arxiv.org/abs/2308.07593
作者：Jeong Hun Yeo,Minsu Kim,Jeongsoo Choi,Dae Hoe Kim,Yong Man Ro摘要：视觉语音识别（VSR）是从无声的嘴唇运动预测口语单词的任务。由于对嘴唇运动的信息不足，VSR被认为是一项具有挑战性的任务。在本文中，我们提出了一个音频知识授权的视觉语音识别框架（AKVSR），以补充不足的语音信息的视觉模态，通过使用音频模态。与先前的方法不同，所提出的AKVSR 1）利用由大规模预训练音频模型编码的丰富音频知识，2）通过经由量化丢弃来自音频的非语言信息来将音频知识的语言信息保存在紧凑音频存储器中，以及3）包括音频桥接模块，其可以从紧凑音频存储器中找到最佳匹配的音频特征。这使得我们的训练可以在没有音频输入的情况下进行，一旦在紧凑的音频存储器被组成之后。我们通过大量的实验验证了所提出的方法的有效性，并实现了新的国家的最先进的性能上广泛使用的数据集，LRS2和LRS3。摘要：Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements. VSR is regarded as a challenging task because of the insufficient information on lip movements. In this paper, we propose an Audio Knowledge empowered Visual Speech Recognition framework (AKVSR) to complement the insufficient speech information of visual modality by using audio modality. Different from the previous methods, the proposed AKVSR 1) utilizes rich audio knowledge encoded by a large-scale pretrained audio model, 2) saves the linguistic information of audio knowledge in compact audio memory by discarding the non-linguistic information from the audio through quantization, and 3) includes Audio Bridging Module which can find the best-matched audio features from the compact audio memory, which makes our training possible without audio inputs, once after the compact audio memory is composed. We validate the effectiveness of the proposed method through extensive experiments, and achieve new state-of-the-art performances on the widely-used datasets, LRS2 and LRS3.
【6】 O-1: Self-training with Oracle and 1-best Hypothesis标题：O-1：甲骨文的自我训练和1-最佳假设链接：https://arxiv.org/abs/2308.07486
作者：Murali Karthick Baskar,Andrew Rosenberg,Bhuvana Ramabhadran,Kartik Audhkhasi机构：Google Inc.摘要：我们引入了O-1，一个新的自我训练目标，以减少训练偏差和统一的语音识别的训练和评估指标。O-1是期望最小贝叶斯风险（EMBR）的一个更快的变体，它增强了oracle假设，可以适应监督和无监督数据。我们证明了我们的方法的有效性，在公开可用的SpeechStew数据集和大规模的内部数据集的识别。在Speechstew上，O-1目标将实际性能和Oracle性能之间的差距缩小了80%，而EMBR将差距缩小了43%。O-1在SpeechStew包含的各种数据集上实现了13%至25%的EMBR相对改进，并且相对于Oracle WER在内部数据集上的EMBR训练减少了12%的相对差距。总体而言，O-1导致WER相对于EMBR有9\%的相对改善，从而说明了针对大规模数据集提出的目标的可扩展性。摘要：We introduce O-1, a new self-training objective to reduce training bias and unify training and evaluation metrics for speech recognition. O-1 is a faster variant of Expected Minimum Bayes Risk (EMBR), that boosts the oracle hypothesis and can accommodate both supervised and unsupervised data. We demonstrate the effectiveness of our approach in terms of recognition on publicly available SpeechStew datasets and a large-scale, in-house data set. On Speechstew, the O-1 objective closes the gap between the actual and oracle performance by 80\% relative compared to EMBR which bridges the gap by 43\% relative. O-1 achieves 13\% to 25\% relative improvement over EMBR on the various datasets that SpeechStew comprises of, and a 12\% relative gap reduction with respect to the oracle WER over EMBR training on the in-house dataset. Overall, O-1 results in a 9\% relative improvement in WER over EMBR, thereby speaking to the scalability of the proposed objective for large-scale datasets.
【7】 Text Injection for Capitalization and Turn-Taking Prediction in Speech  Models标题：语音模型中用于大写和话轮转换预测的文本注入链接：https://arxiv.org/abs/2308.07395
作者：Shaan Bijwadia,Shuo-yiin Chang,Weiran Wang,Zhong Meng,Hao Zhang,Tara N. Sainath机构：Google, USA摘要：用于自动语音识别（ASR）的文本注入（其中未配对的仅文本数据用于补充配对的音频-文本数据）已经显示出对单词错误率的有希望的改进。本研究探讨使用文本注入辅助任务，这是非ASR任务经常执行的E2 E模型。在这项工作中，我们使用联合端到端和内部语言模型训练（JEIT）作为我们的文本注入算法来训练ASR模型，执行两个辅助任务。第一个是大写，这是一个去规范化的任务。第二种是轮转换预测，其尝试识别用户是否已经完成了数字助理交互中的会话轮。我们展示的结果表明，我们的文本注入方法提高了长尾数据的大写性能，并提高了话轮检测召回率。摘要：Text injection for automatic speech recognition (ASR), wherein unpaired text-only data is used to supplement paired audio-text data, has shown promising improvements for word error rate. This study examines the use of text injection for auxiliary tasks, which are the non-ASR tasks often performed by an E2E model. In this work, we use joint end-to-end and internal language model training (JEIT) as our text injection algorithm to train an ASR model which performs two auxiliary tasks. The first is capitalization, which is a de-normalization task. The second is turn-taking prediction, which attempts to identify whether a user has completed their conversation turn in a digital assistant interaction. We show results demonstrating that our text injection method boosts capitalization performance for long-tail data, and improves turn-taking detection recall.
【8】 Using Text Injection to Improve Recognition of Personal Identifiers in  Speech标题：利用文本注入改进语音中个人识别符的识别链接：https://arxiv.org/abs/2308.07393
作者：Yochai Blau,Rohan Agrawal,Lior Madmony,Gary Wang,Andrew Rosenberg,Zhehuai Chen,Zorik Gekhman,Genady Beryozkin,Parisa Haghani,Bhuvana Ramabhadran机构：Google备注：Accepted to Interspeech 2023摘要：在许多自动语音识别（ASR）应用中，准确识别特定类别（诸如人名、日期或其他标识符）是至关重要的。由于这些类别代表了个人信息，因此对这些数据的道德使用，包括收集、转录、培训和评估，需要特别注意。确保个人安全和隐私的一种方法是从收集中删除或完全删除个人身份信息（PII）。然而，这导致ASR模型往往具有这些类别的较低识别准确度。我们使用文本注入，以提高识别的PII类别包括假的文本替代品在训练数据中使用文本注入的方法。我们证明了医疗笔记中的姓名和日期召回的实质性改进，同时提高了整体WER。对于字母数字序列，我们显示字符错误率和句子准确性的改进。摘要：Accurate recognition of specific categories, such as persons' names, dates or other identifiers is critical in many Automatic Speech Recognition (ASR) applications. As these categories represent personal information, ethical use of this data including collection, transcription, training and evaluation demands special care. One way of ensuring the security and privacy of individuals is to redact or eliminate Personally Identifiable Information (PII) from collection altogether. However, this results in ASR models that tend to have lower recognition accuracy of these categories. We use text-injection to improve the recognition of PII categories by including fake textual substitutes of PII categories in the training data using a text injection method. We demonstrate substantial improvement to Recall of Names and Dates in medical notes while improving overall WER. For alphanumeric digit sequences we show improvements to Character Error Rate and Sentence Accuracy.
机器翻译，仅供参考