【1】 SJTU-AISPEECH System for VoxCeleb Speaker Recognition Challenge 2022
标题：2022年VoxCeleb说话人识别挑战赛SJTU-AISPEECH系统
链接：https://arxiv.org/abs/2209.09076
作者：Zhengyang Chen,Bing Han,Xu Xiang,Houjun Huang,Bei Liu,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, Department of Computer Science and Engineering, X-LANCE Lab, Shanghai Jiao Tong University, AISpeech Ltd, Suzhou China备注：System description of VoxSRC 2022摘要：本报告描述了2022年Voxceleb说话人识别挑战赛的SJTU-AISPEECH系统。对于track 1，我们实现了两种系统，在线系统和离线系统。研究了不同的基于ResNet的骨干网和损失函数。我们的最终融合系统在track 1中获得第3名。对于track 3，我们实现了统计自适应和基于联合训练的域自适应。在基于联合训练的领域自适应算法中，采用不同的训练目标对源领域数据集和目标领域数据集进行联合训练，以实现领域自适应。针对目标域数据，研究了两种不同的训练目标：基于自监督学习的角度原型损失和基于半监督学习的伪标签估计分类损失。此外，当目标域目标为分类损失时，采用动态损失门和标签校正（DLG-LC）策略来提高伪标签的质量。我们最终的融合系统在track 3中获得了第4名（非常接近第3名，相对低于1%）。摘要：This report describes the SJTU-AISPEECH system for the Voxceleb Speaker Recognition Challenge 2022. For track1, we implemented two kinds of systems, the online system and the offline system. Different ResNet-based backbones and loss functions are explored. Our final fusion system achieved 3rd place in track1. For track3, we implemented statistic adaptation and jointly training based domain adaptation. In the jointly training based domain adaptation, we jointly trained the source and target domain dataset with different training objectives to do the domain adaptation. We explored two different training objectives for target domain data, self-supervised learning based angular proto-typical loss and semi-supervised learning based classification loss with estimated pseudo labels. Besides, we used the dynamic loss-gate and label correction (DLG-LC) strategy to improve the quality of pseudo labels when the target domain objective is a classification loss. Our final fusion system achieved 4th place (very close to 3rd place, relatively less than 1%) in track3.
【2】 The Royalflush System for VoxCeleb Speaker Recognition Challenge 2022
标题：VoxCeleb说话人识别挑战赛2022的RoyalFlush系统
链接：https://arxiv.org/abs/2209.09010
作者：Jingguang Tian,Xinhui Hu,Xinkang Xu机构：Hithink RoyalFlush AI Research Institute, Zhejiang, China摘要：在本技术报告中，我们描述了2022年VoxCeleb说话人识别挑战赛（VoxSRC-22）的Royalflush提交材料。我们的提交包含用于监督说话人验证的轨道1和用于半监督说话人验证的轨道3。对于音轨1，我们开发了一个功能强大的基于U-Net的说话人嵌入提取器。在验证集上，该系统的EER和MinDCF分别为2.06%和0.1293。与现有的ECAPA-TDNN相比，EER和MinDCF分别提高了20.7%和22.70%。对于音轨3，我们采用源域监督和目标域自监督的联合训练来获得说话人嵌入提取器。随后的聚类过程可以获得目标域伪说话人标签。我们以监督的方式使用所有源和目标域数据来调整说话人嵌入提取器，其中它可以充分利用两个域信息。此外，可以重复聚类和监督域自适应，直到性能收敛于验证集。我们最终提交的是10个模型的融合，并在验证集上实现了7.75%的EER和0.3517的MinDCF。摘要：In this technical report, we describe the Royalflush submissions for the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). Our submissions contain track 1, which is for supervised speaker verification and track 3, which is for semi-supervised speaker verification. For track 1, we develop a powerful U-Net-based speaker embedding extractor with a symmetric architecture. The proposed system achieves 2.06% in EER and 0.1293 in MinDCF on the validation set. Compared with the state-of-the-art ECAPA-TDNN, it obtains a relative improvement of 20.7% in EER and 22.70% in MinDCF. For track 3, we employ the joint training of source domain supervision and target domain self-supervision to get a speaker embedding extractor. The subsequent clustering process can obtain target domain pseudo-speaker labels. We adapt the speaker embedding extractor using all source and target domain data in a supervised manner, where it can fully leverage both domain information. Moreover, clustering and supervised domain adaptation can be repeated until the performance converges on the validation set. Our final submission is a fusion of 10 models and achieves 7.75% EER and 0.3517 MinDCF on the validation set.
【3】 AutoLV: Automatic Lecture Video Generator
标题：AutoLV：讲座视频自动生成器
链接：https://arxiv.org/abs/2209.08795
作者：Wenbin Wang,Yang Song,Sanjay Jha机构：School of Computer Science and Engineering, University of New South Wales, Australia备注：4 pages, 4 figures, ICIP 2022摘要：提出了一种端到端的演讲视频生成系统，该系统可以直接从带注释的幻灯片、教师参考语音和教师参考肖像视频生成真实完整的演讲视频。该系统主要由Few-Shot说话人自适应的语音合成模块和基于对抗学习的说话人生成模块组成。它不仅可以减轻教师的工作量，而且可以改变语言和口音，使学生更容易跟上讲座，并使讲座内容得到更广泛的传播。实验结果表明，该模型在真实性、自然性和准确性方面优于现有的其他方法。下面是一段视频，演示我们的系统如何工作，以及评估和比较的结果：https://youtu.be/cY6TYkI0cog。摘要：We propose an end-to-end lecture video generation system that can generate realistic and complete lecture videos directly from annotated slides, instructor's reference voice and instructor's reference portrait video. Our system is primarily composed of a speech synthesis module with few-shot speaker adaptation and an adversarial learning-based talking-head generation module. It is capable of not only reducing instructors' workload but also changing the language and accent which can help the students follow the lecture more easily and enable a wider dissemination of lecture contents. Our experimental results show that the proposed model outperforms other current approaches in terms of authenticity, naturalness and accuracy. Here is a video demonstration of how our system works, and the outcomes of the evaluation and comparison: https://youtu.be/cY6TYkI0cog.
【4】 Playing Technique Detection by Fusing Note Onset Information in Guzheng Performance
标题：古筝演奏中融合音符起音信息的演奏技巧检测
链接：https://arxiv.org/abs/2209.08774
作者：Dichucheng Li,Yulun Wu,Qinyu Li,Jiahao Zhao,Yi Yu,Fan Xia,Wei Li机构：School of Computer Science and Technology, Fudan University, Shanghai, China, College of Experimental Art, Sichuan Conservatory of Music, Sichuan, China, National Institute of Informatics (NII), Tokyo, Japan备注：Accepted to ISMIR 2022摘要：古筝是一种中国传统乐器，演奏技法多样。乐器演奏技术在音乐演奏中起着重要的作用。然而，大多数现有的用于IPT检测的工作对于可变长度音频表现出低效率，并且由于它们依赖于用于训练和测试的单个声音库而不能提供泛化的保证。在本研究中，我们提出了一个端对端的古筝演奏技术检测系统，使用全卷积网络，可以应用于可变长度的音频。因为每种古筝演奏技巧都应用于一个音符，所以专用的开始检测器被训练为将音频划分为若干音符，并且其预测与逐帧IPT预测相融合。在融合过程中，我们在每个音符内逐帧添加IPT预测，并获得每个音符内具有最高概率的IPT作为该音符的最终输出。我们从多个声音库和真实世界的录音中创建了一个名为GZ_IsoTech的新数据集，用于古筝演奏分析。该方法的帧级准确率和音符级F1-score分别为87.97%和80.76%，大大优于已有的方法，表明了该方法在IPT检测中的有效性.摘要：The Guzheng is a kind of traditional Chinese instruments with diverse playing techniques. Instrument playing techniques (IPT) play an important role in musical performance. However, most of the existing works for IPT detection show low efficiency for variable-length audio and provide no assurance in the generalization as they rely on a single sound bank for training and testing. In this study, we propose an end-to-end Guzheng playing technique detection system using Fully Convolutional Networks that can be applied to variable-length audio. Because each Guzheng playing technique is applied to a note, a dedicated onset detector is trained to divide an audio into several notes and its predictions are fused with frame-wise IPT predictions. During fusion, we add the IPT predictions frame by frame inside each note and get the IPT with the highest probability within each note as the final output of that note. We create a new dataset named GZ_IsoTech from multiple sound banks and real-world recordings for Guzheng performance analysis. Our approach achieves 87.97% in frame-level accuracy and 80.76% in note-level F1-score, outperforming existing works by a large margin, which indicates the effectiveness of our proposed method in IPT detection.
【5】 Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach
标题：谱写和润色：通过两个阶段的方法产生结构良好的钢琴演奏
链接：https://arxiv.org/abs/2209.08212
作者：Shih-Lun Wu,Yi-Hsuan Yang机构：School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, United States, ˇ “( Yating Music Team, Taiwan AI Labs, Taipei, Taiwan, Z Research Center for IT Innovation, Academia Sinica, Taipei, Taiwan备注：Preprint. 4 pages, 2 figures, 4 tables摘要：即使使用像Transformers这样强大的序列模型，生成具有长距离音乐结构的富有表现力的钢琴演奏仍然具有挑战性。同时，还提出了一种合成结构良好的旋律或导奏单（旋律+和弦）的方法，即，更简单的音乐形式，获得了更多的成功。根据以上情况，我们设计了一个两阶段的基于Transformer的框架，首先合成一个前导表，然后用伴奏和表现性的触摸来修饰它。这样的因式分解还使得能够对非钢琴数据进行预训练。我们的客观和主观实验表明，“作曲与修饰”将当前艺术状态与真实表演之间的结构性差距缩小了一半，并改善了音乐的其他方面，如丰富性和连贯性。摘要：Even with strong sequence models like Transformers, generating expressive piano performances with long-range musical structures remains challenging. Meanwhile, methods to compose well-structured melodies or lead sheets (melody + chords), i.e., simpler forms of music, gained more success. Observing the above, we devise a two-stage Transformer-based framework that Composes a lead sheet first, and then Embellishes it with accompaniment and expressive touches. Such a factorization also enables pretraining on non-piano data. Our objective and subjective experiments show that Compose & Embellish shrinks the gap in structureness between a current state of the art and real performances by half, and improves other musical aspects such as richness and coherence as well.
【6】 Representation Learning Strategies to Model Pathological Speech: Effect of Multiple Spectral Resolutions
标题：病态言语模型的表征学习策略：多光谱分辨率的影响
链接：https://arxiv.org/abs/2209.08379
作者：Gabriel Figueiredo Miller,Juan Camilo Vásquez-Correa,Juan Rafael Orozco-Arroyave,Elmar Nöth机构：Pattern Recognition Lab. Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Germany, University of Antioquia UdeA, Medell´ın, Colombia摘要：本文考虑了一种表征学习策略来对帕金森病和唇腭裂患者的语音信号进行建模。特别地，它比较了不同的参数化表示类型，例如宽带和窄带谱图，以及基于小波的尺度图，目的是量化每一种的表示能力。量化方法包括所提出的模型对不同病理和相关疾病严重程度进行分类的能力。此外，本文还提出了一种新的融合策略--多光谱融合，该融合策略利用基于自动编码器的表示学习策略，将宽带和窄带光谱分辨率相结合。该模型对帕金森病患者的语音进行分类，准确率可达95%。所提出的模型还能够评估帕金森病患者的构音障碍严重程度，Spearman相关性高达0.75。这些结果优于那些在文献中观察到的，其中相同的问题是用相同的语料库解决的。摘要：This paper considers a representation learning strategy to model speech signals from patients with Parkinson's disease and cleft lip and palate. In particular, it compares different parametrized representation types such as wideband and narrowband spectrograms, and wavelet-based scalograms, with the goal of quantifying the representation capacity of each. Methods for quantification include the ability of the proposed model to classify different pathologies and the associated disease severity. Additionally, this paper proposes a novel fusion strategy called multi-spectral fusion that combines wideband and narrowband spectral resolutions using a representation learning strategy based on autoencoders. The proposed models are able to classify the speech from Parkinson's disease patients with accuracy up to 95\%. The proposed models were also able to asses the dysarthria severity of Parkinson's disease patients with a Spearman correlation up to 0.75. These results outperform those observed in literature where the same problem was addressed with the same corpus.
【7】 An Automatic Speech Recognition System for Bengali Language based on Wav2Vec2 and Transfer Learning
标题：基于Wav2Vec2和迁移学习的孟加拉语语音自动识别系统
链接：https://arxiv.org/abs/2209.08119
作者：Tushar Talukder Showrav机构：Dept. of Electrical & Electronic Engineering, Bangladesh University of Engineering & Technology, Dhaka, Bangladesh摘要：一种解码和转录口头语音的独立的自动化方法被称为自动语音识别（ASR）。典型的ASR系统从音频记录或流中提取特征，并且运行一个或多个算法以将特征映射到对应的文本。近年来，在语音信号处理领域已经进行了大量的研究。当给予足够的资源时，传统的ASR和新兴的端到端（E2E）语音识别都产生了有希望的结果。然而，对于像孟加拉语这样的低资源语言，ASR的当前状态落后了，尽管低资源状态没有反映出全世界有超过5亿人使用这种语言的事实。尽管孟加拉语很受欢迎，但目前可用的各种开源数据集并不多，这使得对孟加拉语语音识别系统进行研究变得困难。本文是"BUET CSE Fest DL Sprint“竞赛的一部分。本文的目的是在迁移学习框架的基础上，采用E2E结构的语音识别技术，提高孟加拉语语音识别性能。该方法在7747个样本的测试数据集上，仅用1000个训练数据集进行训练，得到了3.819的Levenshtein平均距离.摘要：An independent, automated method of decoding and transcribing oral speech is known as automatic speech recognition (ASR). A typical ASR system extracts featured from audio recordings or streams and run one or more algorithms to map the features to corresponding texts. Numerous of research has been done in the field of speech signal processing in recent years. When given adequate resources, both conventional ASR and emerging end-to-end (E2E) speech recognition have produced promising results. However, for low-resource languages like Bengali, the current state of ASR lags behind, although the low resource state does not reflect upon the fact that this language is spoken by over 500 million people all over the world. Despite its popularity, there aren't many diverse open-source datasets available, which makes it difficult to conduct research on Bengali speech recognition systems. This paper is a part of the competition named `BUET CSE Fest DL Sprint'. The purpose of this paper is to improve the speech recognition performance of the Bengali language by adopting speech recognition technology on the E2E structure based on the transfer learning framework. The proposed method effectively models the Bengali language and achieves 3.819 score in `Levenshtein Mean Distance' on the test dataset of 7747 samples, when only 1000 samples of train dataset were used to train.
【1】 Representation Learning Strategies to Model Pathological Speech: Effect of Multiple Spectral Resolutions
标题：病态言语模型的表征学习策略：多光谱分辨率的影响
链接：https://arxiv.org/abs/2209.08379
* 与cs.SD语音【6】为同一篇
作者：Gabriel Figueiredo Miller,Juan Camilo Vásquez-Correa,Juan Rafael Orozco-Arroyave,Elmar Nöth机构：Pattern Recognition Lab. Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Germany, University of Antioquia UdeA, Medell´ın, Colombia摘要：本文考虑了一种表征学习策略来对帕金森病和唇腭裂患者的语音信号进行建模。特别地，它比较了不同的参数化表示类型，例如宽带和窄带谱图，以及基于小波的尺度图，目的是量化每一种的表示能力。量化方法包括所提出的模型对不同病理和相关疾病严重程度进行分类的能力。此外，本文还提出了一种新的融合策略--多光谱融合，该融合策略利用基于自动编码器的表示学习策略，将宽带和窄带光谱分辨率相结合。该模型对帕金森病患者的语音进行分类，准确率可达95%。所提出的模型还能够评估帕金森病患者的构音障碍严重程度，Spearman相关性高达0.75。这些结果优于那些在文献中观察到的，其中相同的问题是用相同的语料库解决的。摘要：This paper considers a representation learning strategy to model speech signals from patients with Parkinson's disease and cleft lip and palate. In particular, it compares different parametrized representation types such as wideband and narrowband spectrograms, and wavelet-based scalograms, with the goal of quantifying the representation capacity of each. Methods for quantification include the ability of the proposed model to classify different pathologies and the associated disease severity. Additionally, this paper proposes a novel fusion strategy called multi-spectral fusion that combines wideband and narrowband spectral resolutions using a representation learning strategy based on autoencoders. The proposed models are able to classify the speech from Parkinson's disease patients with accuracy up to 95\%. The proposed models were also able to asses the dysarthria severity of Parkinson's disease patients with a Spearman correlation up to 0.75. These results outperform those observed in literature where the same problem was addressed with the same corpus.
【2】 Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition
标题：通过共享用于端到端语音识别的稀疏门控专家来实现参数高效一致性
链接：https://arxiv.org/abs/2209.08326
作者：Ye Bai,Jie Li,Wenjing Han,Hao Ni,Kaituo Xu,Zhuo Zhang,Cheng Yi,Xiaorui Wang机构：Kuaishou Technology Co., Ltd, Beijing, China备注：accepted in INTERSPEECH 2022摘要：虽然Transformers及其变体在语音识别中表现出良好的性能，但参数化特性导致训练和推理过程中大量的存储开销。一些工作使用跨层权重分配来减少模型的参数。然而，不可避免的容量损失损害了模型性能。针对这一问题，提出了一种基于稀疏门控专家共享的参数有效构象器.具体地说，我们使用稀疏门控专家混合（MoE）来扩展构象模块的容量而不增加计算。然后，共享分组的构象异构体块的参数，从而减少参数的数量。接下来，为了确保共享块具有适应不同级别表示的灵活性，我们分别设计了MoE路由器和归一化。此外，我们还使用知识提取来进一步提高性能。实验结果表明，与全参数模型相比，所提模型仅需1/3的编码器参数就能获得较好的性能。摘要：While transformers and their variant conformers show promising performance in speech recognition, the parameterized property leads to much memory cost during training and inference. Some works use cross-layer weight-sharing to reduce the parameters of the model. However, the inevitable loss of capacity harms the model performance. To address this issue, this paper proposes a parameter-efficient conformer via sharing sparsely-gated experts. Specifically, we use sparsely-gated mixture-of-experts (MoE) to extend the capacity of a conformer block without increasing computation. Then, the parameters of the grouped conformer blocks are shared so that the number of parameters is reduced. Next, to ensure the shared blocks with the flexibility of adapting representations at different levels, we design the MoE routers and normalization individually. Moreover, we use knowledge distillation to further improve the performance. Experimental results show that the proposed model achieves competitive performance with 1/3 of the parameters of the encoder, compared with the full-parameter model.
【3】 An Automatic Speech Recognition System for Bengali Language based on Wav2Vec2 and Transfer Learning
标题：基于Wav2Vec2和迁移学习的孟加拉语语音自动识别系统
链接：https://arxiv.org/abs/2209.08119
* 与cs.SD语音【7】为同一篇
作者：Tushar Talukder Showrav机构：Dept. of Electrical & Electronic Engineering, Bangladesh University of Engineering & Technology, Dhaka, Bangladesh摘要：一种解码和转录口头语音的独立的自动化方法被称为自动语音识别（ASR）。典型的ASR系统从音频记录或流中提取特征，并且运行一个或多个算法以将特征映射到对应的文本。近年来，在语音信号处理领域已经进行了大量的研究。当给予足够的资源时，传统的ASR和新兴的端到端（E2E）语音识别都产生了有希望的结果。然而，对于像孟加拉语这样的低资源语言，ASR的当前状态落后了，尽管低资源状态没有反映出全世界有超过5亿人使用这种语言的事实。尽管孟加拉语很受欢迎，但目前可用的各种开源数据集并不多，这使得对孟加拉语语音识别系统进行研究变得困难。本文是"BUET CSE Fest DL Sprint“竞赛的一部分。本文的目的是在迁移学习框架的基础上，采用E2E结构的语音识别技术，提高孟加拉语语音识别性能。该方法在7747个样本的测试数据集上，仅用1000个训练数据集进行训练，得到了3.819的Levenshtein平均距离.摘要：An independent, automated method of decoding and transcribing oral speech is known as automatic speech recognition (ASR). A typical ASR system extracts featured from audio recordings or streams and run one or more algorithms to map the features to corresponding texts. Numerous of research has been done in the field of speech signal processing in recent years. When given adequate resources, both conventional ASR and emerging end-to-end (E2E) speech recognition have produced promising results. However, for low-resource languages like Bengali, the current state of ASR lags behind, although the low resource state does not reflect upon the fact that this language is spoken by over 500 million people all over the world. Despite its popularity, there aren't many diverse open-source datasets available, which makes it difficult to conduct research on Bengali speech recognition systems. This paper is a part of the competition named `BUET CSE Fest DL Sprint'. The purpose of this paper is to improve the speech recognition performance of the Bengali language by adopting speech recognition technology on the E2E structure based on the transfer learning framework. The proposed method effectively models the Bengali language and achieves 3.819 score in `Levenshtein Mean Distance' on the test dataset of 7747 samples, when only 1000 samples of train dataset were used to train.
【4】 SJTU-AISPEECH System for VoxCeleb Speaker Recognition Challenge 2022
标题：2022年VoxCeleb说话人识别挑战赛SJTU-AISPEECH系统
链接：https://arxiv.org/abs/2209.09076
* 与cs.SD语音【1】为同一篇
作者：Zhengyang Chen,Bing Han,Xu Xiang,Houjun Huang,Bei Liu,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, Department of Computer Science and Engineering, X-LANCE Lab, Shanghai Jiao Tong University, AISpeech Ltd, Suzhou China备注：System description of VoxSRC 2022摘要：本报告描述了2022年Voxceleb说话人识别挑战赛的SJTU-AISPEECH系统。对于track 1，我们实现了两种系统，在线系统和离线系统。研究了不同的基于ResNet的骨干网和损失函数。我们的最终融合系统在track 1中获得第3名。对于track 3，我们实现了统计自适应和基于联合训练的域自适应。在基于联合训练的领域自适应算法中，采用不同的训练目标对源领域数据集和目标领域数据集进行联合训练，以实现领域自适应。针对目标域数据，研究了两种不同的训练目标：基于自监督学习的角度原型损失和基于半监督学习的伪标签估计分类损失。此外，当目标域目标为分类损失时，采用动态损失门和标签校正（DLG-LC）策略来提高伪标签的质量。我们最终的融合系统在track 3中获得了第4名（非常接近第3名，相对低于1%）。摘要：This report describes the SJTU-AISPEECH system for the Voxceleb Speaker Recognition Challenge 2022. For track1, we implemented two kinds of systems, the online system and the offline system. Different ResNet-based backbones and loss functions are explored. Our final fusion system achieved 3rd place in track1. For track3, we implemented statistic adaptation and jointly training based domain adaptation. In the jointly training based domain adaptation, we jointly trained the source and target domain dataset with different training objectives to do the domain adaptation. We explored two different training objectives for target domain data, self-supervised learning based angular proto-typical loss and semi-supervised learning based classification loss with estimated pseudo labels. Besides, we used the dynamic loss-gate and label correction (DLG-LC) strategy to improve the quality of pseudo labels when the target domain objective is a classification loss. Our final fusion system achieved 4th place (very close to 3rd place, relatively less than 1%) in track3.
【5】 The Royalflush System for VoxCeleb Speaker Recognition Challenge 2022
标题：VoxCeleb说话人识别挑战赛2022的RoyalFlush系统
链接：https://arxiv.org/abs/2209.09010
* 与cs.SD语音【2】为同一篇
作者：Jingguang Tian,Xinhui Hu,Xinkang Xu机构：Hithink RoyalFlush AI Research Institute, Zhejiang, China摘要：在本技术报告中，我们描述了2022年VoxCeleb说话人识别挑战赛（VoxSRC-22）的Royalflush提交材料。我们的提交包含用于监督说话人验证的轨道1和用于半监督说话人验证的轨道3。对于音轨1，我们开发了一个功能强大的基于U-Net的说话人嵌入提取器。在验证集上，该系统的EER和MinDCF分别为2.06%和0.1293。与现有的ECAPA-TDNN相比，EER和MinDCF分别提高了20.7%和22.70%。对于音轨3，我们采用源域监督和目标域自监督的联合训练来获得说话人嵌入提取器。随后的聚类过程可以获得目标域伪说话人标签。我们以监督的方式使用所有源和目标域数据来调整说话人嵌入提取器，其中它可以充分利用两个域信息。此外，可以重复聚类和监督域自适应，直到性能收敛于验证集。我们最终提交的是10个模型的融合，并在验证集上实现了7.75%的EER和0.3517的MinDCF。摘要：In this technical report, we describe the Royalflush submissions for the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). Our submissions contain track 1, which is for supervised speaker verification and track 3, which is for semi-supervised speaker verification. For track 1, we develop a powerful U-Net-based speaker embedding extractor with a symmetric architecture. The proposed system achieves 2.06% in EER and 0.1293 in MinDCF on the validation set. Compared with the state-of-the-art ECAPA-TDNN, it obtains a relative improvement of 20.7% in EER and 22.70% in MinDCF. For track 3, we employ the joint training of source domain supervision and target domain self-supervision to get a speaker embedding extractor. The subsequent clustering process can obtain target domain pseudo-speaker labels. We adapt the speaker embedding extractor using all source and target domain data in a supervised manner, where it can fully leverage both domain information. Moreover, clustering and supervised domain adaptation can be repeated until the performance converges on the validation set. Our final submission is a fusion of 10 models and achieves 7.75% EER and 0.3517 MinDCF on the validation set.
【6】 AutoLV: Automatic Lecture Video Generator
标题：AutoLV：讲座视频自动生成器
链接：https://arxiv.org/abs/2209.08795
* 与cs.SD语音【3】为同一篇
作者：Wenbin Wang,Yang Song,Sanjay Jha机构：School of Computer Science and Engineering, University of New South Wales, Australia备注：4 pages, 4 figures, ICIP 2022摘要：提出了一种端到端的演讲视频生成系统，该系统可以直接从带注释的幻灯片、教师参考语音和教师参考肖像视频生成真实完整的演讲视频。该系统主要由Few-Shot说话人自适应的语音合成模块和基于对抗学习的说话人生成模块组成。它不仅可以减轻教师的工作量，而且可以改变语言和口音，使学生更容易跟上讲座，并使讲座内容得到更广泛的传播。实验结果表明，该模型在真实性、自然性和准确性方面优于现有的其他方法。下面是一段视频，演示我们的系统如何工作，以及评估和比较的结果：https://youtu.be/cY6TYkI0cog。摘要：We propose an end-to-end lecture video generation system that can generate realistic and complete lecture videos directly from annotated slides, instructor's reference voice and instructor's reference portrait video. Our system is primarily composed of a speech synthesis module with few-shot speaker adaptation and an adversarial learning-based talking-head generation module. It is capable of not only reducing instructors' workload but also changing the language and accent which can help the students follow the lecture more easily and enable a wider dissemination of lecture contents. Our experimental results show that the proposed model outperforms other current approaches in terms of authenticity, naturalness and accuracy. Here is a video demonstration of how our system works, and the outcomes of the evaluation and comparison: https://youtu.be/cY6TYkI0cog.
【7】 Playing Technique Detection by Fusing Note Onset Information in Guzheng Performance
标题：古筝演奏中融合音符起音信息的演奏技巧检测
链接：https://arxiv.org/abs/2209.08774
* 与cs.SD语音【4】为同一篇
作者：Dichucheng Li,Yulun Wu,Qinyu Li,Jiahao Zhao,Yi Yu,Fan Xia,Wei Li机构：School of Computer Science and Technology, Fudan University, Shanghai, China, College of Experimental Art, Sichuan Conservatory of Music, Sichuan, China, National Institute of Informatics (NII), Tokyo, Japan备注：Accepted to ISMIR 2022摘要：古筝是一种中国传统乐器，演奏技法多样。乐器演奏技术在音乐演奏中起着重要的作用。然而，大多数现有的用于IPT检测的工作对于可变长度音频表现出低效率，并且由于它们依赖于用于训练和测试的单个声音库而不能提供泛化的保证。在本研究中，我们提出了一个端对端的古筝演奏技术检测系统，使用全卷积网络，可以应用于可变长度的音频。因为每种古筝演奏技巧都应用于一个音符，所以专用的开始检测器被训练为将音频划分为若干音符，并且其预测与逐帧IPT预测相融合。在融合过程中，我们在每个音符内逐帧添加IPT预测，并获得每个音符内具有最高概率的IPT作为该音符的最终输出。我们从多个声音库和真实世界的录音中创建了一个名为GZ_IsoTech的新数据集，用于古筝演奏分析。该方法的帧级准确率和音符级F1-score分别为87.97%和80.76%，大大优于已有的方法，表明了该方法在IPT检测中的有效性.摘要：The Guzheng is a kind of traditional Chinese instruments with diverse playing techniques. Instrument playing techniques (IPT) play an important role in musical performance. However, most of the existing works for IPT detection show low efficiency for variable-length audio and provide no assurance in the generalization as they rely on a single sound bank for training and testing. In this study, we propose an end-to-end Guzheng playing technique detection system using Fully Convolutional Networks that can be applied to variable-length audio. Because each Guzheng playing technique is applied to a note, a dedicated onset detector is trained to divide an audio into several notes and its predictions are fused with frame-wise IPT predictions. During fusion, we add the IPT predictions frame by frame inside each note and get the IPT with the highest probability within each note as the final output of that note. We create a new dataset named GZ_IsoTech from multiple sound banks and real-world recordings for Guzheng performance analysis. Our approach achieves 87.97% in frame-level accuracy and 80.76% in note-level F1-score, outperforming existing works by a large margin, which indicates the effectiveness of our proposed method in IPT detection.
【8】 Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach
标题：谱写和润色：通过两个阶段的方法产生结构良好的钢琴演奏
链接：https://arxiv.org/abs/2209.08212
* 与cs.SD语音【5】为同一篇
作者：Shih-Lun Wu,Yi-Hsuan Yang机构：School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, United States, ˇ “( Yating Music Team, Taiwan AI Labs, Taipei, Taiwan, Z Research Center for IT Innovation, Academia Sinica, Taipei, Taiwan备注：Preprint. 4 pages, 2 figures, 4 tables摘要：即使使用像Transformers这样强大的序列模型，生成具有长距离音乐结构的富有表现力的钢琴演奏仍然具有挑战性。同时，还提出了一种合成结构良好的旋律或导奏单（旋律+和弦）的方法，即，更简单的音乐形式，获得了更多的成功。根据以上情况，我们设计了一个两阶段的基于Transformer的框架，首先合成一个前导表，然后用伴奏和表现性的触摸来修饰它。这样的因式分解还使得能够对非钢琴数据进行预训练。我们的客观和主观实验表明，“作曲与修饰”将当前艺术状态与真实表演之间的结构性差距缩小了一半，并改善了音乐的其他方面，如丰富性和连贯性。摘要：Even with strong sequence models like Transformers, generating expressive piano performances with long-range musical structures remains challenging. Meanwhile, methods to compose well-structured melodies or lead sheets (melody + chords), i.e., simpler forms of music, gained more success. Observing the above, we devise a two-stage Transformer-based framework that Composes a lead sheet first, and then Embellishes it with accompaniment and expressive touches. Such a factorization also enables pretraining on non-piano data. Our objective and subjective experiments show that Compose & Embellish shrinks the gap in structureness between a current state of the art and real performances by half, and improves other musical aspects such as richness and coherence as well.
机器翻译，仅供参考
每日语音速递