Abstract

Ultrasound-to-Speech aims to synthesize intelligible speech from tongue ultrasound video. Prior approaches rely mainly on purely convolutional or recurrent architectures. We present Tongue2Speech, a lightweight framework combining explicit 3D spatio-temporal encoding with a multi-layer Transformer to model long-range articulatory dynamics. Spectral distortion metrics (such as MSE) commonly used in prior work are weak indicators of lexical accuracy when used in isolation. Therefore, we evaluate performance using word error rate (WER). On the English TaL corpus, Tongue2Speech achieves 15.93% WER in single-speaker and 31.64% in multi-speaker settings, out-performing six strong baselines. Speaker-specific fine-tuning further improves intelligibility for unseen users. These results highlight tongue-only ultrasound as a promising pathway toward practical silent speech synthesis. Audio samples are available at: https://tongue2speech.github.io/

Abstract

Subtitles are essential for accessibility and global dissemination of video content, yet conventional formats often omit critical cues such as speaker identities. Character-Aware Subtitling (CAS) aims to address this gap, however, existing approaches are limited by the absence of public checkpoints, weak pipeline coordination leading to detrimental performance, and insufficient failure analysis. This work presents a systematic study of subtitling pipelines, demonstrating that improved transcript-speech alignment substantially enhances downstream performance, achieving state-of-the-art results on the LLR dataset by 2.77% DER-point zero-shot, and by 4.78% DER-point after VAD finetuning. In addition, we propose design refinements, adopt open, out-of-the-box model alternatives, and introduce a scalable method for curating diverse character priors, thereby improving the robustness and replicability of CAS.

Abstract

Whispered and dysarthric speech hinder effective communication and undermine the reliability of voice-enabled systems. We present CLARIS, a compact speech-to-speech restoration system that turns such atypical input into clear, expressive speech. CLARIS requires no disorder-specific architectural tuning, generalizes across languages, and adapts quickly to new accents and speakers, enabling practical personalization. On whispered English, Hindi, and clinically challenging dysarthric speech, CLARIS delivers state-of-the-art intelligibility and naturalness, with listener studies confirming gains in quality, intelligibility, naturalness, and prosody. The system runs in real time, converting one second of input in about 30ms and enables inclusive, private, and personalized voice interaction. Audio samples are available at https://claris-w2s.github.io/CLARIS/

Abstract

Contrastive vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition yet remain vulnerable to spurious correlations, particularly background over-reliance. We introduce Cluster-based Concept Importance (CCI), a novel interpretability method that uses CLIP's own patch embeddings to group spatial patches into semantically coherent clusters, mask them, and evaluate relative changes in model predictions. CCI sets a new state of the art on faithfulness benchmarks, surpassing prior methods by large margins; for example, it yields more than a twofold improvement on the deletion-AUC metric for MS COCO retrieval. We further propose that CCI, when combined with GroundedSAM, automatically categorizes predictions as foreground- or background-driven, providing a crucial diagnostic ability. Existing benchmarks such as CounterAnimals, however, rely solely on accuracy and implicitly attribute all performance degradation to background correlations. Our analysis shows this assumption to be incomplete, since many errors arise from viewpoint variation, scale shifts, and fine-grained object confusions. To disentangle these effects, we introduce COVAR, a benchmark that systematically varies object foregrounds and backgrounds. Leveraging CCI with COVAR, we present a comprehensive evaluation of eighteen CLIP variants, offering methodological advances and empirical evidence that chart a path toward more robust VLMs

Abstract

Large-scale vision-language models such as CLIP achieve strong zero-shot recognition but struggle with classes that are rarely seen during pretraining, including newly emerging entities and culturally specific categories. We introduce LiteEmbed, a lightweight framework for few-shot personalization of CLIP that enables new classes to be added without retraining its encoders. LiteEmbed performs subspace-guided optimization of text embeddings within CLIP's vocabulary, leveraging a PCA-based decomposition that disentangles coarse semantic directions from fine-grained variations. Two complementary objectives, coarse alignment and fine separation, jointly preserve global semantic consistency while enhancing discriminability among visually similar classes. Once optimized, the embeddings are plug-and-play, seamlessly substituting CLIP's original text features across classification, retrieval, segmentation, and detection tasks. Extensive experiments demonstrate substantial gains over prior methods, establishing LiteEmbed as an effective approach for adapting CLIP to underrepresented, rare, or unseen classes.

Abstract

We propose an alignment-free, end-to-end Non-Audible Murmur (NAM)-to-Speech conversion model. Existing methods rely on large NAM-text pairs per speaker to generate
high-quality alignments for training non-autoregressive models.
However, alignment quality deteriorates when trained on multispeaker data, limiting their ability to generalize and effectively
utilize the available training data. To address this, we introduce
a streamlined autoregressive approach that eliminates the need
for explicit alignment learning. By leveraging multi-speaker
samples, synthetic training pairs, and multitask character recognition training, our method reduces the word error rate (WER)
by 59.19% compared to the state-of-the-art (SOTA) on two public datasets. We demonstrate the model's zero-shot capability and validate the effectiveness of multitask training through
ablation studies. Speech samples are available at https:
//noalignnam.github.io/autoregressiveNAM/

Abstract

Pretrained models are ubiquitous in the current deep learning landscape, offering strong
results on a broad range of tasks. Recent works have shown that models differing in various design choices exhibit categorically diverse generalization behavior, resulting in one
model grasping distinct data-specific insights unavailable to the other. In this paper, we
propose to leverage large publicly available model repositories as an auxiliary source of
model improvements. We introduce a data partitioning strategy where pretrained models
autonomously adopt either the role of a student, seeking knowledge, or that of a teacher,
imparting knowledge. Experiments across various tasks demonstrate the effectiveness of
our proposed approach. In image classification, we improved the performance of ViT-B
by approximately 1.4% through bidirectional knowledge transfer with ViT-T. For semantic
segmentation, our method boosted all evaluation metrics by enabling knowledge transfer
both within and across backbone architectures. In video saliency prediction, our approach
achieved a new state-of-the-art. We further extend our approach to knowledge transfer
between multiple models, leading to considerable performance improvements for all model
participants.

Abstract

To understand a prompt, Vision-Language models (VLMs) must perceive the image, comprehend the text, and build associations within and across both modalities. For instance, given an `image of a red toy car', the model should associate this image to phrases like `car', `red toy', `red object', etc. Prior work proposed the Binding ID mechanism in LLMs, suggesting that the entity and its corresponding attribute tokens share a Binding ID in the model activations. We investigate this for image-text binding in VLMs using a synthetic dataset and task that requires models to associate 3D objects in an image with their descriptions in the text. Our experiments demonstrate that VLMs assign a distinct Binding ID to an object's image tokens and its textual references, enabling in-context association.

Abstract

We motivate weakly supervised learning as an effective learning paradigm for problems where curating perfectly annotated datasets is expensive and may require domain expertise such as fine-grained classification. We focus on Partial Label Learning (PLL), a weakly-supervised learning paradigm where each training instance is paired with a set of candidate labels (partial label), one of which is the true label. Noisy PLL (NPLL) relaxes this constraint by allowing some partial labels to not contain the true label, enhancing the practicality of the problem. Our work centres on NPLL and presents a framework that initially assigns pseudo-labels to images by exploiting the noisy partial labels through a weighted nearest neighbour algorithm. These pseudo-label and image pairs are then used to train a deep neural network classifier with label smoothing. The classifier's features and predictions are subsequently employed to refine and enhance the accuracy of pseudo-labels. We perform thorough experiments on seven datasets and compare against nine NPLL and PLL methods. We achieve state-of-the-art results in all studied settings from the prior literature, obtaining substantial gains in the simulated fine-grained benchmarks. Further, we show the promising generalisation capability of our framework in realistic, fine-grained, crowd-sourced datasets.

Abstract

We consider the problem of single-source domain generalization. Existing methods typically rely on extensive augmentations to synthetically cover diverse domains during training. However, they struggle with semantic shifts (e.g., background and viewpoint changes), as they often learn global features instead of local concepts that tend to be domain invariant. To address this gap, we propose an approach that compels models to leverage such local concepts
during prediction. Given no suitable dataset with per-class concepts and localization maps exists, we first develop a novel pipeline to generate annotations by exploiting the rich features of diffusion and large-language models. Our next innovation is TIDE, a novel training scheme with a concept saliency alignment loss that ensures model focus on the right per-concept regions and a local concept contrastive loss that promotes learning domain-invariant concept representations. This not only gives a robust model but also can be visually interpreted using the predicted concept saliency maps. Given these maps at test time, our final contribution is a new correction algorithm that uses the corresponding local concept representations to iteratively refine the prediction until it aligns with prototypical concept representations that we store at the end of model training. We evaluate our approach extensively on four standard DG benchmark datasets and substantially outperform the current state-of-the-art (12% improvement on average) while also demonstrating that our predictions can be visually interpreted.