语音/音频处理学术速递[6.11]

AI 总结：提出SARA双流VAE，融合冻结的SSL语义锚点和残差声学编码器，解决语音分词器中声学与语义的权衡，实现高保真重建和零样本TTS的自然合成。

链接：https://arxiv.org/abs/2606.11611

机构：School of Informatics, Xiamen University（厦门大学信息学院）； School of Electronic Science and Engineering, Xiamen University（厦门大学电子科学与技术学院）； DiDi Global Inc.（滴滴全球股份有限公司）

作者：Peijie Chen, Wenhao Guan, Weijie Wu, Kaidi Wang, Daiyu Huang, Zhuanling Zha, Junbo Li, Jun Fang, Qingyang Hong, Lin Li

英文摘要：Zero-shot text-to-speech (TTS) relies on robust speech representations. However, current speech tokenizers face a fundamental trade-off: acoustic codecs preserve high-fidelity audio but lack linguistic constraints, causing content errors during generation, whereas semantic tokens from self-supervised learning (SSL) models ensure precise text alignment but discard some acoustic information. To bridge this gap, we propose SARA, a dual-stream VAE that directly fuses a frozen SSL semantic anchor with a dedicated residual acoustic encoder. This effectively mitigates the dilemma, creating an efficient and compact latent space without relying on complex regularizers. SARA achieves superior reconstruction quality over strong baselines. Furthermore, in downstream zero-shot TTS tasks, it yields highly natural and expressive synthesis quality, and maintains robust generation performance even under accelerated inference, offering a favorable trade-off between synthesis speed and computational cost.

AI 总结：提出质量自适应角度边界学习框架QLung，通过频谱熵和均方根能量推导无参考音频质量边界，自适应缩放角度边界，改善特征泛化，在ICBHI和SPRSound数据集上分别提升2.46%和达到最优分布外性能。

链接：https://arxiv.org/abs/2606.11915

机构：RSC LAB, MODULABS, Republic of Korea（RSC实验室，MODULABS，韩国）； Department of Electronic Engineering, Wonkwang University, Republic of Korea（韩国圆光大学电子工程系）； AI Convergence Research Institute, Wonkwang University, Republic of Korea（韩国圆光大学人工智能融合研究所）

作者：Yoon Tae Kim, Heejoon Koo, Miika Toikkanen, June-Woo Kim

英文摘要：We present a quality-adaptive angular-margin learning framework that improves feature generalization by enforcing intra-class compactness and inter-class separability. Our framework, titled QLung, introduces a no-reference audio quality margin derived from spectral entropy and root-mean-square energy, which adaptively scales angular margins based on recording quality. To this end, we propose a log-scaled angular margin that stabilizes training under severe class imbalance. We also use an angular classifier that normalizes features and class weights, ensuring margin penalties are applied consistently on the unit hypersphere. Our approach improves in-distribution performance on the ICBHI dataset by 2.46\% over the cross-entropy baseline, and most significantly, achieves the strongest out-of-distribution performance on the SPRSound dataset compared to prior state-of-the-art methods. Code is available at https://github.com/RSC-Toolkit/QLung.

AI 总结：针对呼吸音分类中AST模型对局部异常模式不敏感的问题，提出基于状态空间模型的谱感知层正则化和双轴补丁混合对比学习，在ICBHI基准上达到64.48%分数，比AST基线提升5%。

链接：https://arxiv.org/abs/2606.11922

机构：RSC LAB, MODULABS（RSC实验室，MODULABS）； Department of Electronic Engineering, Wonkwang University（圆光大学电子工程系）； AI Convergence Research Institute, Wonkwang University（圆光大学人工智能融合研究所）

作者：Hemansh Shridhar, Miika Toikkanen, June-Woo Kim

英文摘要：Recent respiratory sound classification (RSC) studies largely rely on CLS-token driven self-attention architectures such as the Audio Spectrogram Transformer (AST). While effective at modeling global context, recent analyses suggest a low-pass filtering behavior that may reduce sensitivity to localized abnormal patterns. In this work, we investigate State Space Models (SSMs) as an alternative backbone for RSC. Using the Distilled Audio State Space model, we analyze intermediate representations through spectral response curves and observe stronger preservation of mid-to-high spatial-frequency components. Based on these observations, we introduce spectral-aware layer regularization using Gaussian convolution applied to selected layers. We further propose Dual-Axis Patch-Mix contrastive learning tailored to SSM-based audio models for robust representation learning. Experiments on the ICBHI benchmark show that our approach achieves 64.48% score, outperforming the AST baseline by 5%. Code is available at https://github.com/RSC-Toolkit/Lung-SRAD.

AI 总结：提出Fast-SDE，一种基于子带处理的轻量级单麦克风框架，用于在资源受限的机器人平台上实现高效声源距离估计。

链接：https://arxiv.org/abs/2606.12339

机构：Institute of Science Tokyo（东京科学大学）

作者：Jiang Wang, Runwu Shi, Yaozhong Kang, Benjamin Yen, Takeshi Ashizawa, Kazuhiro Nakadai

英文摘要：Sound source distance estimation (SDE) is a critical capability in human-robot interaction. An inappropriate interaction distance not only reduces the reliability of speech acquisition and understanding, but also compromises the naturalness and comfort of the interaction. Most existing SDE methods rely on microphone arrays, however, multi-microphone systems typically require careful hardware synchronization, geometric calibration, and additional space and computational resources, which limits applicability to size-constrained and computability-limited embodied platforms. To alleviate these issues, we propose Fast-SDE, a lightweight single-microphone SDE framework that is suited for deployment on robot platforms with limited computational resources and strict size constraints. Specifically, Fast-SDE employs a subband-based backbone that decomposes the frequency axis into multiple subbands, rather than processing the entire spectrum with a wide full-band backbone. A shared subband encoder then maps each subband to a compact latent representation and learns the relationship between acoustic structure and time-frequency patterns. Finally, a lightweight regression head converts the fused subband representations into the estimated distance. Extensive simulation and real-world experiments demonstrate the merits of the proposed method. To benefit the broader research community, we have open-sourced our code at https://github.com/JiangWAV/FAST-SDE.

AI 总结：提出StreamMUSE系统，在客户端-服务器架构中实现帧同步流式推理，通过现场音乐伴奏任务验证了不同延迟环境下实时同步的有效性。

链接：https://arxiv.org/abs/2606.11886

机构：MBZUAI（穆罕默德·本·扎耶德人工智能大学）

作者：Bowen Zheng, Andrew H. Yang, Jiaqi Ruan, Jia He, Xinyue Li, Yuan-Hsin Chen, Ziyu Wang, Xiaosong Ma

英文摘要：Language models (LMs) have become one of the most prominent paradigms in modern generative modeling. While making them faster has been the main focus of real-time deployment, speed alone is not enough. Many real-world applications, such as synchronized translation and voice synthesis, also require precise alignment between generation and external signals, both in terms of generation content and timing. We refer to this problem as \textit{frame-synchronous streaming inference}. To address it, we present StreamMUSE, an inference system that performs LM generation in response to an external signal stream within a client-server architecture. The client continuously sends high-frequency inference requests based on the most recent inputs and receives outputs synchronized to the external clock, while the server executes model inference. We demonstrate the framework through a live music accompaniment task, showing how real-time synchronization can be achieved across different deployment environments with varying round-trip latencies. We further model the relationship between system hyperparameters and round-trip latency, and evaluate how different environments affect optimal configurations to achieve real-time performance. Experimental results show a consistent correspondence between system real-time performance and music quality, demonstrating the effectiveness of the proposed framework. The project is open source. Relevant code and the latest updates are available at https://stream-muse-webpage.vercel.app/#audio-library.

AI 总结：针对弱对齐的乐谱-音频数据，提出基于二分图匹配的上下文感知起始点细化方法，显著提升自动音乐转录的起始点对齐和转录精度。

链接：https://arxiv.org/abs/2606.11903

作者：Abhirup Saha, Hans-Ulrich Berendes, Meinard Müller, Ben Maman

英文摘要：Precise note-level annotations are critical for training automatic music transcription (AMT) systems, in particular note-onset labels, which form a core component of many recent AMT systems. However, high-quality annotations for real-world recordings are scarce. Sequence-level score--audio alignment methods such as dynamic time warping provide only coarse correspondence, making a local refinement step necessary. This refinement step, known as snapping, adjusts aligned score onsets using peaks in a neural onset posteriorgram and often determines whether weakly aligned score--audio pairs become usable training data at all. Despite its practical importance, snapping is typically treated as a simple post-processing heuristic and implemented with greedy local decisions. We present a systematic analysis of snapping strategies for training instrument-agnostic transcribers, demonstrating that snapping is essential for learning from weakly aligned data. Building on this, we formulate snapping as a per-pitch assignment problem and solve it via bipartite graph matching, yielding context-aware onset decisions under overlapping refinement windows and uncertain initial alignments. Extensive cross-dataset experiments across piano, chamber, and orchestral recordings show improved onset alignment and transcription accuracy over greedy snapping, with gains increasing for wider snapping windows and coarser initial alignments. Qualitative examples are provided on our project page: https://abhirupsaha8.github.io

AI 总结：提出PianoKontext，一种基于流匹配的钢琴演奏渲染模型，通过动态时间规整对齐乐谱与演奏的潜在表示，生成可变长度的表现力演奏。

链接：https://arxiv.org/abs/2606.12282

作者：Dmitrii Gavrilev

英文摘要：Expressive performance rendering (EPR) aims to generate realistic performances constrained on sequences of notes. However, flow matching audio editing models manipulate only synchronized music samples of the same duration, limiting their understanding of expressive timing. We introduce PianoKontext, a flow matching rendering model for classical piano music that generates variable-length performances in the latent space of a pretrained Music2Latent model. We synthesize MIDI scores into deadpan audio and employ Dynamic Time Warping (DTW) in the latent space to construct paired data for training. The aligned embeddings are concatenated in DiT blocks, allowing for a simple and effective learning of the dependencies between the score and performances. Audio samples are available at our demo page: https://realfolkcode.github.io/pianokontext_demo/.

AI 总结：提出基于指令的向量操控方法，通过对比不同指令下的激活来重定向音频令牌的时间注意力，实现无需训练的声音事件定位，显著优于直接提示和随机基线。

链接：https://arxiv.org/abs/2606.11400

作者：Tsung-En Lin, Hung-Yi Lee

英文摘要：Large Audio-Language Models (LALMs) excel at audio understanding but expose little about where in an audio signal they attend. We introduce instruction-based vector steering, which constructs a steering vector by contrasting activations from differently instructed prompts while keeping the audio fixed. Through a systematic probe of LALM attention, we find that - unlike standard prompting or audio-based steering - this intervention significantly redistributes the temporal attention allocated to audio tokens, concentrating it on acoustically relevant regions. We then show that this attention shift is behaviorally meaningful: in a controlled three-event setting, reading out the temporal position of maximal steering-induced attention change recovers the location of a queried sound event without any training, attaining 60.87% and 68.72% overlap with ground-truth intervals on Qwen2-Audio and Audio Flamingo 3, far above direct prompting (31.84%, 46.75%) and random baselines (27.74%). Our results characterize a mechanistic property of instruction-based steering in LALMs and provide a training-free probe for the latent temporal structure these models encode.

AI 总结：提出RAIL基准，基于CHC认知框架将听觉智能分解为五种核心能力，构建结构化评估任务，系统评测大型音频语言模型的认知行为。

链接：https://arxiv.org/abs/2606.11260

机构：School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）； Faculty of Psychology and Educational Sciences, Alexandru Ioan Cuza University of Iași（亚历山德鲁伊万库扎大学心理学与教育科学学院）； School of Electronic Information, Wuhan University（武汉大学电子信息学院）； School of Public Health, The University of Hong Kong（香港大学公共卫生学院）； School of Computer Science, The University of Auckland（奥克兰大学计算机科学学院）； Department of Data Science and Artificial Intelligence, Monash University（莫纳什大学数据科学与人工智能系）

作者：Hongyu Jin, Siyi Wang, Yang Xiao, Jiaheng Dong, Shihong Tan, Kaiyuan peng, Georgiana Juravle, Shanquan Chen, Gongping Huang, Hong Jia, Eun-Jung Holden, James Bailey, Ting Dang

英文摘要：Humans process rich auditory environments through tightly integrated cognitive capabilities such as audio perception, audio reasoning, and memory. Despite recent progress in large audio-language models (LALMs) across speech understanding and multimodal audio reasoning, current evaluation paradigms remain largely task- or modality-centric, focusing on end performance while overlooking underlying auditory cognitive behaviours. This reveals a fundamental gap between how auditory cognition is understood in humans and how it is evaluated in LALMs, particularly in the lack of frameworks that operationalise cognitive principles beyond task-level metrics to systematically capture model behaviour. In this work, we introduce RAIL, a human-centric evaluation paradigm grounded in the Cattell-Horn-Carroll (CHC) cognitive framework. RAIL formalises auditory cognition into five core capabilities and develop them into structured evaluation tasks that probe how models process, retain, and integrate auditory information. We further construct a cognitively grounded benchmark with principled data curation and human-aligned evaluation protocols. Evaluating 26 state-of-the-art LALMs, we find that current models exhibit highly uneven performance across cognitive abilities. RAIL establishes a new evaluation paradigm that moves beyond task-centric benchmarking toward cognitively grounded assessment of auditory intelligence.

AI 总结：本文提出CS-YODAS数据集，通过可扩展的人机协同流程从多语言YouTube数据中挖掘真实代码转换语音，涵盖7种基质语言共313小时，并分析其分布特征与语言对切换模式。

链接：https://arxiv.org/abs/2606.11514

机构：Carnegie Mellon University（卡内基梅隆大学）； Johns Hopkins University（约翰霍普金斯大学）； University of Texas at Austin（德克萨斯大学奥斯汀分校）； University of Sheffield（谢菲尔德大学）； Brno University of Technology（布尔诺理工大学）； MBZUAI（穆罕默德·本·扎耶德人工智能大学）； Kyoto University（京都大学）

作者：Brian Yan, Qingzheng Wang, Matthew Wiesner, Anuj Diwan, Olga Iakovenko, Alexander Polok, Injy Hamed, Shuichiro Shimizu, Iris Emerman Thomas Hain, David R. Mortensen, Peter Viechnicki, Shinji Watanabe

英文摘要：We present CS-YODAS, a Creative Commons-licensed dataset of in-the-wild code-switched speech mined from multilingual YouTube data. Code-switching (CS), or the alternation between languages within an utterance or conversation, is common in multilingual settings but remains underrepresented in existing CS speech resources, which are typically small, domain-specific, or artificially constructed. Building on the YODAS corpus, we develop a scalable, human-in-the-loop pipeline for identifying and validating naturally occurring code-switching. The resulting dataset, which totals 313 hours and spans 7 matrix languages, provides diverse, real-world examples of spontaneous code-switched speech. We further analyze the distribution and characteristics of code-switching in the wild, examining language-pair frequencies and switching patterns, and report baseline results for spoken language identification. We hope that CS-YODAS will encourage broader and more comprehensive research on code-switched speech. Dataset link: https://huggingface.co/datasets/byan/cs-yodas.

AI 总结：研究比较全局锚定与成对验证在合成语音源追踪中的性能，发现成对验证导致嵌入方差集中、分辨率降低，从而在域内和域外任务中表现更差。

链接：https://arxiv.org/abs/2606.11666

机构：Brno University of Technology（布尔诺理工大学）

作者：Anton Firc, Zbyněk Lička, Vojtěch Staněk, Kamil Malinka

英文摘要：Open-set source tracing is increasingly framed as a verification problem, motivating the use of pairwise metric-learning objectives from biometrics. We thus compare global anchoring and pairwise verification under matched backbones and a fixed data and epoch budget on MLAAD (in-domain) and STOPA (out-of-domain). In our runs, global anchoring yields lower in-domain error (8.61% EER) than pairwise variants (12-15% EER), even with rival mining and XLS-R finetuning. Because pairwise objectives optimize similarity directly, they concentrate variance into fewer embedding directions, reducing resolution among closely related generators. To test if this drives the drop, we impose a similar bottleneck to the globally supervised baseline, yet the baseline remains competitive. Together with an embedding-space analysis ($k_{99}$), these results suggest that the gap is not explained by dimensionality alone, but rather by the pairwise objective's shaping of the retained directions.

AI 总结：提出SpAArSIST，通过稀疏化图池化后端，在保持竞争力的同时降低计算量20.7%、模型大小4.1%，并提升域外鲁棒性。

链接：https://arxiv.org/abs/2606.11674

机构：Brno University of Technology（布尔诺理工大学）

作者：Anton Firc, Vojtěch Staněk, Zbyněk Lička, Kamil Malinka, Martin Perešíni

英文摘要：We present SpAArSIST, a deployment-oriented refinement of the widely used AASIST graph pooling backend for self-supervised learning (SSL) based anti-spoofing. Motivated by redundant operations in public implementations, we replace learned pooling and stack-node attention with explicit, lightweight choices: separate train and inference graph pooling ratios $(k_{\mathrm{tr}},k_{\mathrm{inf}})$, magnitude-based node scoring, and mean aggregation of graph nodes. The best overall configuration (rank 1) cuts backend compute by 20.7% (195.045M $\rightarrow$ 154.706M MACs) and model size by 4.1% (611.8k $\rightarrow$ 586.4k params), while improving out-of-domain robustness on In-the-Wild to 2.82% EER and 0.078 minDCF (from 4.64% and 0.133) and remaining competitive on ASVspoof5. We further provide a composite selection score that summarizes accuracy, calibration, and compute to support balanced deployment-oriented model choice.

AI 总结：提出特征对齐水印方法，通过将水印与原始语音特征分布对齐，在保持不可感知性的同时提高水印能量，增强对语音重建模型的鲁棒性。

链接：https://arxiv.org/abs/2606.11828

机构：Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）； Shenzhen Key Laboratory of Intelligent Media and Content Understanding（深圳市智能媒体与内容理解重点实验室）； Tencent AI Lab（腾讯人工智能实验室）

作者：Haiyun Li, Shuhai Peng, Zhisheng Zhang, Jingran Xie, Xiaofeng Xie, Hanyang Peng, Zhiyong Wu

英文摘要：Audio watermarking aims to embed identifiable information into audio while remaining imperceptible. Existing methods adopt high-fidelity, low-energy designs to preserve perceptual quality, but the resulting watermarks lack robustness under suppression by speech reconstruction models. Improving robustness is challenging due to the inherent robustness-fidelity trade-off in existing designs, where increasing watermark energy improves robustness but reduces fidelity. To address this problem, we propose a feature-aligned watermarking method that aligns the watermark with the original speech feature distribution, allowing higher watermark energy to improve robustness while preserving imperceptibility. We use a pretrained speech codec to generate a pseudo-speech watermark and fuse it into the spectrogram of the input audio, with VAD loss and perceptual losses guiding embedding within voiced regions. Experiments show that our method maintains imperceptibility comparable to existing approaches while substantially improving robustness under both seen and unseen speech reconstruction models.

AI 总结：提出一种基于k-means通道聚类的无数据无训练压缩方法，通过层间不同参数簇数实现细粒度混合稀疏剪枝，在HuBERT-large和Whisper-large-v3上显著降低WER。

链接：https://arxiv.org/abs/2606.11836

机构：The Chinese University of Hong Kong（香港中文大学）； National Research Council Canada（加拿大国家研究委员会）

作者：Haoning Xu, Zhaoqing Li, Huimeng Wang, Youjun Chen, Chengxi Deng, Mengzhe Geng, Xunying Liu

英文摘要：This paper presents a novel data-free and training-free compression approach for speech foundation models using channelwise clustering via k-means. More fine-grained, mixed sparsity pruning by layer-level varying number of parameter clusters is also explored. Experiments conducted on the LibriSpeech dataset suggest that when operating with pruning sparsity of 50% on HuBERT-large, consistent WER reductions of 27.73%/18.61% absolute (34.37%/21.91% relative) over the magnitude-based pruning were obtained on the test-clean and test-other subsets before fine-tuning and 0.19%/0.79% absolute (3.36%/4.62% relative) after fine-tuning with only 3 epochs. Similar WER reductions of 2.86%/5.02% absolute (59.21%/55.29% relative) were observed against magnitudebased pruning on Whisper-large-v3 at 10% sparsity, all with no significant WER increase relative to the uncompressed baseline.